A multi-modal contrastive learning remote sensing classification method based on multi-positive sample matching

Abstract

The application discloses a kind of multi-modal contrast learning remote sensing classification methods based on multiple positive sample matching, including constructing multi-modal contrast learning remote sensing classification model;Through joint optimization cross-entropy classification loss and multiple positive sample pair contrast learning loss, the end-to-end training of the multi-modal contrast learning remote sensing classification model is carried out;Based on the model of training completion, fusion classification is carried out to hyperspectral image and its corresponding laser radar data, and the final ground object class classification result is output.The method realizes the depth fusion and stable contrast learning of remote sensing multi-modal data on multiple semantic levels by hierarchical cross-attention fusion module and multiple positive sample matching strategy.The method can effectively improve the information interaction quality and semantic consistency between modes, while reducing the dependence of traditional contrast learning method on negative samples, significantly improving the classification accuracy and robustness of multi-modal remote sensing image.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image technology, and in particular relates to a multimodal contrastive learning remote sensing classification method based on multiple positive sample matching. Background Technology

[0002] With the development of remote sensing technology, the types of data for acquiring surface information are becoming increasingly diversified. Among them, hyperspectral images (HSI) provide rich spectral information due to their abundant spectral bands, while lidar (LiDAR) provides accurate terrain structure and height information. The two have complementary advantages; therefore, how to effectively fuse hyperspectral and lidar multimodal remote sensing data to improve the performance of ground object identification and classification has become a research hotspot in intelligent remote sensing interpretation. Existing research generally adopts multimodal deep learning methods to extract features from different modalities and then fuse and classify them. However, hyperspectral and lidar data have inherent differences in perception mechanisms and data structures: the former is often multidimensional data focusing on spectral information, while the latter is one-dimensional planar data emphasizing spatial geometry. The differences in feature dimensions and distribution between the two are significant, making direct correspondence difficult. Existing methods often employ simple stitching or shallow fusion, but these approaches ignore the differences between modalities, often resulting in large feature alignment errors, excessive redundant information, and difficulty in leveraging the complementarity between multimodal data. Other methods focus only on deep single-layer fusion, which can easily lead to information loss or bias towards a single modality, ultimately resulting in unstable classification results, limited accuracy improvement, and difficulty in meeting the needs of complex application scenarios.

[0003] In addition, the problem of label scarcity in the field of hyperspectral classification still exists. Some works have introduced contrastive learning mechanisms to reduce the need for labels, thereby enhancing discriminative power by narrowing the distance between positive samples and widening the distance between negative samples. However, traditional contrastive learning methods generally rely on a large number of negative samples as training support, which has the following problems: (1) It is difficult to stably select high-quality negative samples: In contrastive learning, high-quality negative samples usually refer to difficult negative samples (data that are similar to positive samples but not of the same class) which help improve the model's discriminative power. However, difficult negative samples are often difficult to distinguish from false negative samples. Although the latter are treated as negative samples, they are of the same class as positive samples, which can easily mislead the model and lead to unstable training; (2) When there is noise in the distribution between samples or semantic shift between modalities, it is easy to cause unstable training or even collapse; (3) When positive and negative samples are difficult to define, performance improvement is limited: positive samples should be the same as the target class, and negative samples should come from different classes. However, in practical applications, data from different classes may be very similar (such as different vegetation), and data from the same class may be very different (such as different features of the same type of land cover under different sensors). This ambiguity makes it difficult for the model to distinguish between positive and negative samples, thus limiting the training effect. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention proposes a multimodal contrastive learning remote sensing classification method based on multi-positive sample matching.

[0005] The technical solution of the present invention is as follows:

[0006] A multimodal contrastive learning remote sensing classification method based on multiple positive sample matching is used to classify land cover categories in hyperspectral images and their corresponding lidar data, including:

[0007] A multimodal contrastive learning remote sensing classification model is constructed. The model extracts low-level features, generates cross-modal fusion features, and extracts high-level semantic features from hyperspectral images and lidar data through two data processing streams in the hierarchical cross-attention fusion module. It also constructs multiple positive sample pairs of the same ground feature in different modalities and at different levels for contrastive learning through a multi-positive sample matching strategy.

[0008] By jointly optimizing the cross-entropy classification loss and the multi-positive sample pair contrastive learning loss, the model focuses on cross-modal feature consistency, thereby enabling end-to-end training of the multimodal contrastive learning remote sensing classification model.

[0009] Based on the trained model, hyperspectral images and their corresponding lidar data are fused and classified to output the final land cover classification results.

[0010] Furthermore, the hierarchical cross-attention fusion module employs the following methods for low-level feature extraction, cross-modal fusion feature generation, and high-level semantic feature extraction from hyperspectral images and lidar data:

[0011] Principal component analysis was used to perform spectral dimensionality reduction preprocessing on the hyperspectral image, while normalization and registration of the lidar data were performed. The feature extraction module was used to extract the low-level features of the hyperspectral image after spectral dimensionality reduction preprocessing and the lidar data after normalization and registration.

[0012] A cross-modal cross-attention mechanism is used to generate two sets of fused features corresponding to the hyperspectral image and the corresponding lidar data, respectively.

[0013] The two sets of fused features generated are respectively input into the semantic extraction module used to extract high-order semantic representations with strong discriminative power, and high-level semantic features are extracted.

[0014] Furthermore, the feature extraction module consists of three sets of convolutional structures, each set of convolutional structures including a two-dimensional convolutional layer, a batch normalization layer and a non-linear activation function;

[0015] In the data processing stream corresponding to the hyperspectral image, the feature extraction module extracts low-level features and enhances spatial local perception capabilities by progressively compressing the channel dimensions.

[0016] In the data processing stream corresponding to the lidar data, the feature extraction module is adapted to the output dimension of the hyperspectral branch and adopts a symmetrical structure to extract the low-level features by gradually expanding the channels.

[0017] Furthermore, the specific method for generating fused features using a cross-modal cross-attention mechanism includes:

[0018] We use the low-level features from one data stream as the query vector Q, learn the importance of the corresponding low-level features from another data stream as key vectors (K and V), calculate attention weights to re-represent the information in the data stream corresponding to the query vector, and obtain the fused features by the dot product similarity between the query vector Q and the key vector K, as follows:

[0019]

[0020] In the formula, F fusion Features of fusion;

[0021] W q W k W v These are the learnable linear mapping matrices corresponding to the query vector Q, the key vector K, and the value vector V in the key-value vector, respectively;

[0022] QW q KW k VW v These are vector representations of the attention weights of the same dimension obtained through projection onto a linear layer; This is the scaling factor;

[0023] QW q (KW k ) T To query the dot product similarity between vector Q and key vector K;

[0024] σ(·) is the Softmax function; Conv(·) is a 1×1 convolution operation; + is the residual connection operation.

[0025] Furthermore, the semantic extraction module includes several fully connected layers, and activation functions and normalization layers sequentially connected to each fully connected layer;

[0026] The fusion features are first projected onto the spatial-semantic distribution through several fully connected layers; then they are processed by activation functions and normalization layers after each fully connected layer.

[0027] Furthermore, the specific method for constructing multiple positive samples of the same land cover across different modalities and levels using a multi-positive-sample matching strategy for contrastive learning includes:

[0028] Construct positive sample pairs, which include high-level-fusion layer positive sample pairs and high-level-low-level positive sample pairs;

[0029] Feature similarity measure, calculating the feature similarity for each positive sample pair;

[0030] Based on the feature similarity of each positive sample pair, a contrastive learning loss function using multiple positive sample pairs is constructed for contrastive learning.

[0031] Furthermore, the high-level-fusion layer positive sample pairs include positive sample pairs formed by pairing high-level semantic features corresponding to hyperspectral images with fusion features corresponding to lidar data, and positive sample pairs formed by pairing high-level semantic features corresponding to lidar data with fusion features corresponding to hyperspectral images; the high-level-low-level positive sample pairs include positive sample pairs formed by pairing high-level semantic features corresponding to hyperspectral images with low-level features corresponding to lidar data, and positive sample pairs formed by pairing high-level semantic features corresponding to lidar data with low-level features corresponding to hyperspectral images.

[0032] Furthermore, the method for calculating the feature similarity is as follows:

[0033] For positive sample pairs (u,v) corresponding to feature representations of two modalities and different semantic layers, the formula for calculating feature similarity is as follows:

[0034]

[0035] In the formula, u and v are feature representations of two modalities and different semantic layers; · represents the vector inner product operation, and ||·|| represents the L2 norm of the feature vector;

[0036] Furthermore, the expression for the multi-positive-sample-pair contrastive learning loss function mPMSloss is:

[0037] mPMS loss = loss1 + loss2

[0038] loss1 = (-(sim_cos(h)) high ,l fusion )+sim_cos(l high ,h fusion ))+2)

[0039] loss2=(-(sim_cos(h high ,l low )+sim_cos(l high ,h low ))+2)

[0040] Where loss1 and loss2 are the first and second loss functions, respectively; hhigh h fusion and h low These represent the high-level semantic features, fusion features, and low-level features corresponding to the hyperspectral image, respectively. high , l fusion and l low These are the high-level semantic features, fusion features, and low-level features corresponding to the lidar data.

[0041] Furthermore, the objective function for end-to-end training of the multimodal contrastive learning remote sensing classification model consists of a cross-entropy classification loss function and a positive sample pair contrastive learning loss function, expressed as follows:

[0042] Loss=CE loss +λ×mPMS loss

[0043]

[0044] In the formula, CE loss For cross-entropy loss, mPMS loss λ is the loss function for contrastive learning of multiple positive sample pairs, where λ is the loss weight coefficient. This represents the label of sample i in the c-th land cover category. express Corresponding probability.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] This invention proposes a multimodal contrastive learning remote sensing classification method based on multi-positive sample matching. This method achieves deep fusion and stable contrastive learning of remote sensing multimodal data across multiple semantic levels through a hierarchical cross-attention fusion module (HCIF) and a multi-positive sample matching strategy (mPMS). This method effectively improves the quality of information interaction and semantic consistency between modalities, while reducing the dependence of traditional contrastive learning methods on negative samples, significantly improving the classification accuracy and robustness of multimodal remote sensing images.

[0047] The hierarchical cross-attention fusion module (HCIF) in this invention extracts multi-scale features within a modality through convolutional structures and establishes multi-layer semantic associations between modalities through a bidirectional cross-attention mechanism. This achieves deep complementarity and feature alignment between hyperspectral images and LiDAR data at different semantic levels. This module significantly improves the discriminativity and consistency of the fused representation, providing more semantically expressive feature inputs for downstream tasks.

[0048] The multi-positive sample matching strategy (mPMS) in this invention constructs multiple hierarchical, multimodal positive sample pairs to replace the traditional contrastive learning method that relies on negative samples, effectively solving problems such as difficulty in selecting negative samples and unstable training. This strategy can learn feature consistency from multiple positive samples, enhance the constraint effect of contrastive loss on the feature space structure, and thus improve the model's convergence speed and generalization ability.

[0049] This invention addresses the problems of insufficient feature fusion, inaccurate modality alignment, and unstable contrastive learning in existing multimodal remote sensing classification methods by constructing an end-to-end optimized training mechanism. This method integrates feature representation and multi-positive-sample consistency modeling, significantly improving fusion classification performance while maintaining feature structure and distinguishability. Attached Figure Description

[0050] Figure 1 This is a flowchart of the multimodal contrastive learning remote sensing classification method based on multiple positive sample matching in the embodiment;

[0051] Figure 2 This is a block diagram illustrating the principle of the multimodal contrastive learning remote sensing classification method based on multiple positive sample matching in the embodiment.

[0052] Figure 3 This describes the implementation process of the attention mechanism in the hierarchical cross-attention fusion module of the embodiment;

[0053] Figure 4 This is a schematic diagram of the multi-positive sample matching strategy in the embodiment; Detailed Implementation

[0054] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0055] Example 1:

[0056] This invention provides a multimodal contrastive learning remote sensing classification method based on multiple positive sample matching, used for land cover classification of hyperspectral images (HSI) and their corresponding LiDAR data, such as... Figure 1 As shown, the method includes:

[0057] A multimodal contrastive learning remote sensing classification model is constructed, which includes a hierarchical cross-attention fusion module (HCIF) and a multi-positive sample matching strategy (mPMS). The HCIF module has two data processing streams that sequentially extract low-level features from hyperspectral images and LiDAR data, generate cross-modal fusion features, and extract high-level semantic features, forming a hierarchical multimodal feature representation. The mPMS strategy models semantic consistency, guiding feature expression alignment. By constructing multiple positive sample pairs of the same ground feature across different modalities and levels for contrastive learning, it replaces the traditional negative sample-dependent contrastive learning process, improving training stability and cross-modal consistency.

[0058] By jointly optimizing the cross-entropy classification loss (CE Loss) and the contrastive learning loss (mPMS Loss), the model focuses on cross-modal feature consistency. This enables end-to-end training of the multimodal contrastive learning remote sensing classification model, thereby improving the model's accuracy in land cover classification.

[0059] Specifically, given a hyperspectral image HSI∈R H×W×B Here, H and W represent the height and width of the hyperspectral image, respectively, and B represents the number of spectral channels. Assume the entire HSI dataset consists of N labeled pixels, X. N ={x1,x2,...,x N}∈R 1 ×1×B and M unlabeled pixels X M ={x1,x2,...,x M}∈R 1×1×B Each labeled pixel has a one-hot encoding class label Y. N ={y1,y2,...,y N}∈Rc, where c represents the number of land cover classes, denoted by X N and X M The new dataset Z is formed by the adjacent cubes centered on the pixels in the dataset. N ={z1,z2,...,z N}∈R w×w×B and Z M ={z1,z2,...,z M}∈R w ×w×B .

[0060] During training, the hyperspectral images and corresponding LiDAR data are simultaneously input into the model. First, HCIF is used to extract hierarchical representations for each modality. Single-modality data sequentially passes through a feature extraction module, a cross-attention module, and a high-level semantic extraction module to obtain low-level features, cross-modal fusion features, and high-level semantic features, respectively. These multimodal, hierarchical features are then concatenated to construct a unified feature representation that integrates different modalities and semantic levels. This comprehensive representation is used as input for the final classification task.

[0061] Based on the trained model, hyperspectral images and their corresponding lidar data are fused and classified to output the final land cover classification results.

[0062] The method of this invention addresses the challenges of feature differences and semantic alignment in multimodal remote sensing data by constructing a unified and optimized fusion and comparison learning process to improve classification performance and feature consistency.

[0063] Example 2:

[0064] This embodiment is further designed based on Embodiment 1, as follows: Figure 2 As shown, in this example, the hierarchical cross-attention fusion module uses the following method to extract low-level features from hyperspectral images and LiDAR data, generate cross-modal fusion features, and extract high-level semantic features:

[0065] Low-level feature extraction: Principal component analysis (PCA) is used to perform spectral dimensionality reduction preprocessing on the hyperspectral image, while normalization and registration of the lidar data are performed. The feature extraction module is used to extract the low-level features of the hyperspectral image after spectral dimensionality reduction preprocessing and the lidar data after normalization and registration.

[0066] Fusion feature generation: A cross-modal cross-attention mechanism is used to generate two sets of fusion features corresponding to the hyperspectral image and the corresponding lidar data, respectively, to realize information interaction between the hyperspectral image and lidar data, enhance the spatial and semantic association between the two modalities, and improve the expressive power and discriminative power of the fusion features;

[0067] High-level semantic feature extraction: The two sets of fused features generated are respectively input into the semantic extraction module used to extract high-level semantic representations with strong discriminative power, and high-level semantic features are extracted.

[0068] Example 3:

[0069] This embodiment is a further design based on Embodiment 2, in which the feature extraction module consists of three sets of convolutional structures, each set of convolutional structures including a two-dimensional convolutional layer (Conv2d), a batch normalization layer (BatchNorm2d), and a non-linear activation function (ReLU);

[0070] In the data processing stream of the corresponding hyperspectral image, the feature extraction module extracts low-level features by progressively compressing the channel dimensions and enhances spatial local perception capabilities.

[0071] In the data processing flow corresponding to LiDAR data, the feature extraction module adapts to the output dimension of the hyperspectral branch and adopts a symmetrical structure, and extracts the low-level features by gradually expanding the channels.

[0072] Example 4:

[0073] This embodiment, based on Embodiment 2, further incorporates the following design: The specific method for generating fused features using a cross-modal cross-attention mechanism includes:

[0074] The underlying features in one data processing stream are used as query vector Q (also called "Query"), and the importance of the underlying features in another data stream as key vector (K-V) (also called "(Key,Value)") is learned. Attention weights are calculated to re-express the information in the data stream corresponding to the query vector, thereby dynamically enhancing the representation ability of the relevant regions between the two modalities in the feature space. The fused features are obtained by the dot product similarity between the query vector Q and the key vector K.

[0075]

[0076] In the formula, F fusion Features of fusion;

[0077] W q W k W v These are the learnable linear mapping matrices corresponding to the query vector Q, the key vector K, and the value vector V in the key-value vector, respectively;

[0078] QW q KW k VW v These are vector representations of the attention weights of the same dimension obtained through projection onto a linear layer; The square root of the dimension of the key vector K is used as a scaling factor in the formula to prevent the dot product value from being too large and causing gradient instability.

[0079] QW q (KW k ) TThe dot product similarity between the query vector Q and the key vector K is used. The larger the dot product, the more relevant the reference feature is to the current query feature.

[0080] σ(·) is the Softmax function, which converts similarity into attention weights, and the sum of all weights is 1;

[0081] Conv(·) is a 1×1 convolution operation that compresses channels of the fused features while maintaining resolution, reducing computational cost and adjusting dimensions to prepare for residual connections; + represents a residual connection operation. +Q performs a residual connection on the query vector to avoid information loss.

[0082] In the data processing stream corresponding to the hyperspectral image, the low-level features of the hyperspectral image are used as the query vector, and the low-level features of the lidar data are used as the key vector.

[0083] Conversely, in the corresponding LiDAR data processing stream, the low-level features of the LiDAR data are used as query vectors, and the low-level features of the hyperspectral image are used as key vectors to achieve symmetrical and complementary information.

[0084] The interaction patterns between modalities are automatically learned through training. This bidirectional cross-modal mechanism enables mutual guidance and enhancement between modalities, allowing the features of each modality to be reconstructed or supplemented by another modality through an attention mechanism during the fusion process. The final output is a fused representation that encodes cross-modal semantic relationships, which is helpful for downstream classification or comparison tasks.

[0085] Example 5:

[0086] This embodiment is a further design based on embodiment two, in that the semantic extraction module in this example includes several fully connected layers, as well as activation functions and normalization layers connected sequentially after each fully connected layer;

[0087] The fusion features first undergo a projection transformation of the spatial-semantic distribution through several fully connected layers (Linear) to improve its discriminative power. To enhance nonlinear modeling capabilities, activation functions (such as ReLU) and normalization layers (such as BatchNorm1d) are applied after each fully connected layer to maintain training stability and improve generalization ability. The output features after multiple mappings are regarded as the final semantic representation result, which has a higher level of abstraction and is used to support subsequent classification prediction and contrastive learning processes.

[0088] The following example illustrates the data processing process of the hierarchical cross-attention fusion module. In this example, the original hyperspectral image and the corresponding lidar elevation map of a publicly available remote sensing dataset are used.

[0089] In the raw data preparation stage, the original hyperspectral images and corresponding lidar elevation maps from publicly available remote sensing datasets are preprocessed. Specifically, the original spectral channel data with hundreds of dimensions per pixel is compressed to 32 principal components, significantly reducing the computational burden on the subsequent feature extraction stage of the model. Spatially, to ensure alignment between the two modalities, the lidar elevation map is resampled to maintain the same spatial resolution as the hyperspectral image. After alignment, the data from both modalities are sliced ​​according to a uniform patch size of 7×7 pixels to construct training samples. Regarding sample partitioning, to ensure good generalization ability and stability of the model, the dataset is divided into training and test sets according to the standard pre-defined partitioning scheme of publicly available datasets. The training set contains a certain number of samples from each class (e.g., approximately 20 samples per class), and the remaining samples are assigned to the test set. During the training phase, the model processes input data in batches of 128. To prevent overfitting and improve robustness, data augmentation operations such as random rotation and mirroring are optionally performed on the input data during training.

[0090] After preprocessing the raw data, the hyperspectral image branch and the LiDAR branch are respectively input into their corresponding convolutional neural network modules to extract the low-level spatial structure features within the modality. Specifically, for the hyperspectral image branch, the input size of the patch after PCA dimensionality reduction is 7×7×32 (height×width×number of channels). The CNN module of this branch consists of three basic convolutional blocks, each containing a two-dimensional convolutional layer (Conv2d), a batch normalization layer (BatchNorm2d), and a non-linear activation function (ReLU). The first layer uses a 3×3 kernel size, reducing the number of channels from 32 to 64; the second layer maintains the same kernel size, reducing the number of channels from 64 to 32; the third layer further compresses the output channels to 16. For the LiDAR branch, since its original data is a single-channel elevation map, the initial patch size is 7×7×1. Before entering the CNN structure, a linear convolution layer is used to expand its number of channels to 16 to ensure fusion compatibility with HSI features. Subsequently, feature extraction is performed using a three-layer convolutional structure. Each layer is similar to the HSI branch, expanding the channels from 16 to 32, then to 64, and finally compressing them back to 16, forming an output representation aligned with the HSI feature space. Unlike the hyperspectral branch, the LiDAR branch is designed to increase the channel dimension, thereby enhancing the ability to express spatial structural information to compensate for its lack of spectral dimension.

[0091] After extracting the low-level features from the hyperspectral and lidar data, the low-level spatial structure features of each modality were obtained. These features were then input into the hierarchical cross-attention fusion module (HCIF) proposed in this invention, such as... Figure 3 As shown, the basic idea of ​​this module is to use features from one modality as a query vector to learn the importance of key-value pairs from another modality, thereby dynamically enhancing the representation of related regions between the two modalities in the feature space. Specifically, when hyperspectral features are used as the query and LiDAR features as the key and value, the model calculates attention weights to re-represent the information in the hyperspectral data, emphasizing the semantic guidance of the LiDAR on the hyperspectral data. Conversely, when LiDAR features are used as the query and hyperspectral features as the key and value, symmetrical complementarity of information is achieved. This bidirectional cross-attention operation can strengthen the information flow between the two modalities and improve the expressive power of the fused features.

[0092] After cross-modal fusion, the fused features are further fed into a semantic extraction module for advanced semantic feature extraction. This module consists of a multilayer perceptron (MLP), primarily composed of a series of linear transformation layers, nonlinear activation functions (such as ReLU), and normalization layers. Its purpose is to map the fused features into a more abstract semantic space, thereby obtaining a high-order semantic representation for classification and contrastive learning. This semantic representation integrates spatial structure information and spectral content from different modalities and possesses advantages such as strong discriminative power and robustness to heterogeneous modal differences.

[0093] Example 6:

[0094] This embodiment is further designed based on Embodiment 1, as follows: Figure 4 As shown in the example, the specific method for constructing multiple positive sample pairs of the same land cover across different modalities and levels for contrastive learning using a multi-positive sample matching strategy includes:

[0095] Positive sample pairs are constructed, including high-level-fusion layer positive sample pairs and high-level-low-level positive sample pairs. This strategy enhances inter-modal alignment and training stability by meaningfully pairing feature representations from different semantic levels in different modalities to form multiple sets of positive pairs without negative samples. Specifically, the model first extracts the high-level semantic features of the object in both hyperspectral and lidar modalities, as well as intermediate fusion features and low-level features, through the HCIF module.

[0096] Feature similarity measure, calculating the feature similarity for each positive sample pair;

[0097] Based on the feature similarity of each positive sample pair, a contrastive learning loss function using multiple positive sample pairs is constructed for contrastive learning.

[0098] Example 7:

[0099] This embodiment, based on Embodiment Six, further designs the following: In this example, the high-level-fusion layer positive sample pairs include positive sample pairs formed by pairing high-level semantic features corresponding to hyperspectral images with fusion features corresponding to LiDAR data, and positive sample pairs formed by pairing high-level semantic features corresponding to LiDAR data with fusion features corresponding to hyperspectral images. This bidirectional pairing method ensures symmetrical modeling of intermodal fusion information, which helps improve the modeling capability of the mid-level spatial-spectral structure. The high-level-low-level positive sample pairs include positive sample pairs formed by pairing high-level semantic features corresponding to hyperspectral images with low-level features corresponding to LiDAR data, and positive sample pairs formed by pairing high-level semantic features corresponding to LiDAR data with low-level features corresponding to hyperspectral images. This type of matching enables the model to effectively transfer top-level semantic information to the bottom-level perceptual space, strengthening the correspondence between different modalities at multiple scales.

[0100] Example 8:

[0101] This embodiment, based on Embodiment Six, further designs the use of cosine similarity as a measure of feature similarity between positive sample pairs. This measure effectively captures the directional consistency of feature vectors across different modalities or semantic levels. The feature similarity calculation method is as follows:

[0102] For positive sample pairs (u,v) corresponding to feature representations of two modalities and different semantic layers, the formula for calculating feature similarity is as follows:

[0103]

[0104] In the formula, u and v represent the feature representations of two modalities and different semantic layers; · represents the vector inner product operation, and ||·|| represents the L2 norm of the feature vector; (the above calculation results are real numbers in the interval [-1, 1]. The closer the value is to 1, the more consistent the directions of the two feature vectors are, that is, the more similar the feature expressions of the two modalities are, which means that they may correspond to the same land cover category or have a strong correlation.)

[0105] Example 9:

[0106] This embodiment, based on Embodiment Six, is further designed in that the expression for the multi-positive sample pair contrastive learning loss function mPMSloss (used to encourage the model to minimize the distance between high-level semantic features and corresponding low-level and fused features) is as follows:

[0107] mPMS loss = loss1 + loss2

[0108] loss1 = (-(sim_cos(h)) high ,l fusion )+sim_cos(l high ,h fusion ))+2)

[0109] loss2=(-(sim_cos(h high ,l low )+sim_cos(l high ,h low ))+2)

[0110] Where loss1 and loss2 are the first and second loss functions, respectively; h high h fusion and h low These represent the high-level semantic features, fusion features, and low-level features corresponding to the hyperspectral image, respectively. high , l fusion and l low These are the high-level semantic features, fusion features, and low-level features corresponding to the lidar data.

[0111] Example 10:

[0112] This embodiment, based on Embodiment Nine, further designs the end-to-end training objective function of the multimodal contrastive learning remote sensing classification model in this example, which consists of a cross-entropy classification loss function and a positive sample pair contrastive learning loss function (also known as a "contrarian learning loss function without negative samples"), as shown in the following expression:

[0113] Loss=CE loss +λ×mPMS loss

[0114]

[0115] In the formula, CE loss For cross-entropy loss, mPMS loss λ is the loss function for contrastive learning of multiple positive samples, and λ is the loss weight coefficient used to balance the influence of classification tasks and contrastive learning tasks. This represents the label of sample i in the c-th land cover category. express Corresponding probability.

[0116] Example 11:

[0117] An electronic device includes a memory and a processor. The memory stores a computer program, and the processor is used to invoke and run the computer program stored in the memory to perform the methods of any of the above embodiments.

[0118] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above embodiments.

[0119] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

[0120] Application Examples:

[0121] This embodiment uses the Houston dataset as the experimental object to verify the proposed multimodal contrastive learning remote sensing classification method based on multiple positive sample matching. This dataset contains hyperspectral images and corresponding LiDAR data, with 15 classes and sufficient data samples in each class to fully verify the model's classification performance. The experiment was implemented using the PyTorch framework, with a hardware environment including an Intel Xeon Platinum 8336C CPU (64 cores, 256GB RAM) and an NVIDIA RTX 4090 GPU (24GB VRAM). In this embodiment, the batch size was set to 128, the Adam optimizer was used, the learning rate was set to 0.001, a total of 200 training epochs were performed, and overall accuracy (OA) was used as the primary evaluation metric.

[0122] Experimental results show that the method of this invention achieves good overall classification accuracy and possesses stable classification ability in distinguishing certain categories, effectively improving the classification effect of hyperspectral images in complex scenes. Therefore, the method of this invention is feasible and practically valuable in applications such as automatic classification and recognition of remote sensing images.

Claims

1. A multimodal contrastive learning remote sensing classification method based on multiple positive sample matching, used for land cover classification of hyperspectral images and their corresponding lidar data, characterized in that, include: A multimodal contrastive learning remote sensing classification model is constructed. The model extracts low-level features, generates cross-modal fusion features, and extracts high-level semantic features from hyperspectral images and lidar data through two data processing streams in the hierarchical cross-attention fusion module. It also constructs multiple positive sample pairs of the same ground feature in different modalities and at different levels for contrastive learning through a multi-positive sample matching strategy. By jointly optimizing the cross-entropy classification loss and the multi-positive sample pair contrastive learning loss, the model focuses on cross-modal feature consistency, thereby enabling end-to-end training of the multimodal contrastive learning remote sensing classification model. Based on the trained model, the hyperspectral image and its corresponding lidar data are fused and classified to output the final land cover classification result. The specific method for constructing multiple positive samples of the same land cover across different modalities and levels for comparative learning using a multi-positive-sample matching strategy includes: Construct positive sample pairs, which include high-level-fusion layer positive sample pairs and high-level-low-level positive sample pairs; Feature similarity measure, calculating the feature similarity for each positive sample pair; Based on the feature similarity of each positive sample pair, a contrastive learning loss function using multiple positive sample pairs is constructed for contrastive learning.

2. The multimodal contrastive learning remote sensing classification method based on multi-positive sample matching according to claim 1, characterized in that, The hierarchical cross-attention fusion module employs the following methods for low-level feature extraction, cross-modal fusion feature generation, and high-level semantic feature extraction from hyperspectral images and LiDAR data: Principal component analysis was used to perform spectral dimensionality reduction preprocessing on the hyperspectral image, while normalization and registration of the lidar data were performed. The feature extraction module was used to extract the low-level features of the hyperspectral image after spectral dimensionality reduction preprocessing and the lidar data after normalization and registration. A cross-modal cross-attention mechanism is used to generate two sets of fused features corresponding to the hyperspectral image and the corresponding lidar data, respectively. The two sets of fused features generated are respectively input into the semantic extraction module used to extract high-order semantic representations with strong discriminative power, and high-level semantic features are extracted.

3. The multimodal contrastive learning remote sensing classification method based on multi-positive sample matching according to claim 2, characterized in that, The feature extraction module consists of three sets of convolutional structures, each set of convolutional structures containing a two-dimensional convolutional layer, a batch normalization layer and a non-linear activation function. In the data processing stream corresponding to the hyperspectral image, the feature extraction module extracts low-level features and enhances spatial local perception capabilities by progressively compressing the channel dimensions. In the data processing stream corresponding to the lidar data, the feature extraction module is adapted to the output dimension of the hyperspectral branch and adopts a symmetrical structure to extract the low-level features by gradually expanding the channels.

4. The multimodal contrastive learning remote sensing classification method based on multi-positive sample matching according to claim 2, characterized in that, The specific method for generating fused features using a cross-modal cross-attention mechanism includes: Use the low-level features in a data processing stream as query vectors Learn from another data stream as a key-value vector The importance of the underlying features is determined, and attention weights are calculated to re-express the information in the data stream corresponding to the query vector, in order to query the vector. and key vector The dot product similarity is used to obtain the fusion feature, which is represented as: ； In the formula, Features of fusion; , , They are query vectors Key vector in key value vector Sum value vector The corresponding learnable linear mapping matrix; , , These are vector representations of the attention weights of the same dimension obtained through projection onto a linear layer; This is the scaling factor; query vector and key vector Dot product similarity; for function; for Convolution operations; This is for residual join operations.

5. The multimodal contrastive learning remote sensing classification method based on multi-positive sample matching according to claim 2, characterized in that, The semantic extraction module includes several fully connected layers, and activation functions and normalization layers connected sequentially after each fully connected layer; The fusion features are first projected onto the spatial-semantic distribution through several fully connected layers; then they are processed by activation functions and normalization layers after each fully connected layer.

6. The multimodal contrastive learning remote sensing classification method based on multi-positive sample matching according to claim 1, characterized in that, The high-level-fusion layer positive sample pairs include positive sample pairs formed by pairing high-level semantic features corresponding to hyperspectral images with fusion features corresponding to lidar data, and positive sample pairs formed by pairing high-level semantic features corresponding to lidar data with fusion features corresponding to hyperspectral images; the high-level-low-level positive sample pairs include positive sample pairs formed by pairing high-level semantic features corresponding to hyperspectral images with low-level features corresponding to lidar data, and positive sample pairs formed by pairing high-level semantic features corresponding to lidar data with low-level features corresponding to hyperspectral images.

7. The multimodal contrastive learning remote sensing classification method based on multi-positive sample matching according to claim 1, characterized in that, The method for calculating feature similarity is as follows: For positive sample pairs corresponding to feature representations of two modalities and different semantic layers The formula for calculating their feature similarity is as follows: ； In the formula, and Feature representations for two modalities and different semantic layers; This represents the vector dot product operation. This represents the L2 norm of the eigenvector.

8. The multimodal contrastive learning remote sensing classification method based on multi-positive sample matching according to claim 1, characterized in that, The multi-positive-sample-pair contrastive learning loss function The expression is: ； ； ； in, and These are the first and second loss functions, respectively; , and These represent the high-level semantic features, fusion features, and low-level features corresponding to the hyperspectral image. , and These are the high-level semantic features, fusion features, and low-level features corresponding to the lidar data.

9. The multimodal contrastive learning remote sensing classification method based on multi-positive sample matching according to claim 8, characterized in that, The objective function for end-to-end training of the multimodal contrastive learning remote sensing classification model consists of a cross-entropy classification loss function and a positive sample pair contrastive learning loss function, expressed as follows: ； ； In the formula, For cross-entropy loss, For the contrastive learning loss function of multiple positive sample pairs, These are the loss weighting coefficients; Indicates sample In the Labels for land cover categories, express Corresponding probability.

Images