A multi-modal collaborative ground object classification method and system based on mask contrast learning

Abstract

This invention discloses a multimodal collaborative ground cover classification method and system based on mask contrastive learning, belonging to the field of remote sensing image classification technology. The method includes: acquiring hyperspectral data and LiDAR point cloud data; obtaining hyperspectral image patches and LiDAR image patches based on the above data; extracting features from these patches to obtain hyperspectral image feature blocks and LiDAR image feature blocks; fusing these features to obtain hyperspectral image features, LiDAR image features, a first cross-modal attention map, and a second cross-modal attention map; obtaining a first mask fusion feature and a second mask fusion feature based on the above data; calculating the contrastive loss based on the hyperspectral image features, LiDAR image features, the first mask fusion feature, and the second mask fusion feature, and updating the model parameters; and classifying the multimodal ground cover data to be classified based on the updated model. This invention improves the accuracy and stability of cross-modal learning tasks.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image classification technology, specifically relating to a multimodal collaborative ground feature classification method and system based on mask contrast learning. Background Technology

[0002] In recent years, remote sensing technology has developed rapidly, and remote sensing imagery plays an irreplaceable role in various applications such as disaster detection, agricultural management, and urban development planning. With the emergence of various remote sensing data, multimodal learning has gradually attracted researchers' attention. Among these multimodal data, HSI (Hyperspectral Imaging) data provides detailed spectral information for identifying specific objects on the ground, while LiDAR (Light Detection and Ranging) data provides elevation information for the area. However, HSI faces challenges in identifying ground objects with the same spectral characteristics but different heights, and LiDAR data struggles to distinguish between ground objects of different materials and the same height. Simultaneously, joint classification using HSI and LiDAR data can fully utilize their complementary information to improve classification accuracy. Therefore, feature fusion of cross-modal data has attracted widespread attention and has been widely applied in the classification of ground features in multi-source remote sensing images.

[0003] In recent years, many machine learning techniques have been applied to the joint classification of HSI and LiDAR data, including Support Vector Machines (SVM), Random Forests, and Rotating Forests (RoF). In traditional machine learning frameworks, feature extraction and selection are first performed separately for the two different modalities of data. These features are then fused to form a comprehensive feature set, which is finally used to classify ground features. However, traditional methods relying on the quality of handcrafted features lack sufficient depth and cannot adequately fit the complex nonlinear relationships of ground feature features in HSI and LiDAR data, thus limiting their application scenarios.

[0004] Deep learning-based methods have garnered significant attention due to their superior capabilities in automatic feature extraction. However, building an effective HSI and LiDAR classification model is no easy feat. One key reason is that deep learning-based models typically require a large number of labeled samples to achieve satisfactory accuracy, which is both expensive and limited in land cover classification. Furthermore, these methods often employ attention mechanisms to bridge the semantic gap between modalities, focusing on coordinating feature similarities between different modalities by dynamically assigning weights to key information. The associations between modalities often depend on shared class labels in both modalities, making it difficult for attention mechanisms to explore deeper relationships between them.

[0005] Contrastive learning is a self-supervised learning technique aimed at extracting meaningful representations from unlabeled data. It typically involves using a proxy task to distinguish between positive and negative samples, automatically acquiring sample features for model training. Extensive research has led to the discovery of many well-known contrastive learning frameworks capable of training encoders with powerful representation extraction capabilities, such as SimCLR, MoCo, and BYOL. These studies have been extended to create numerous contrastive learning frameworks for multimodal learning, such as CMC and FactorCL. Contrastive learning methods generally consist of two stages: a pre-training stage and a fine-tuning stage. In the pre-training stage, the input data is augmented to construct positive and negative samples as supervisory information to learn features. The high-quality feature extraction module learned in the pre-training stage is then transferred to downstream tasks for fine-tuning. With the development of contrastive learning, researchers have proposed many methods for contrastive learning on multimodal remote sensing data, primarily focusing on increasing sample diversity through different data augmentation strategies.

[0006] Masked Image Modeling (MIM), an advanced self-supervised learning method, learns effective visual representations by randomly occluding regions of an input image and training the model to recover these masked parts based on information from the unoccluded areas. In practice, the input image is first segmented into a series of fixed-size non-overlapping blocks. Then, some of these blocks are masked to simulate real-world scenarios with incomplete or noisy data. A deep neural network is then used to encode the masked image, enabling the model to infer the content of the occluded regions based on contextual information. This allows the model to understand not only the overall structure and local details of the image but also the complex relationships between different parts.

[0007] In computer vision, the combination of contrastive learning and masked image models offers a new direction for constructing representations with high instance discrimination and local perception capabilities. Contrastive learning aims to optimize the model by reducing the distance between similar inputs (positive pairs) and increasing the distance between dissimilar inputs (negative pairs) in the feature space. Traditionally, positive pairs are typically obtained by augmenting the original input into two distinct views, while negative pairs are acquired using random sampling or memory bank techniques. While effective, this approach often relies on a fixed range of augmented samples to generate positive pairs, limiting its flexibility.

[0008] Based on the above methods, Masked Contrastive Learning (MCL) cleverly combines the advantages of contrastive learning and masked image models. Through careful design, MCL not only utilizes masked image models to enable the model to recover the masked content based on contextual information, thereby learning rich knowledge about image structure, texture, and shape, but also employs a contrastive learning mechanism to narrow the gap between similar input representations while pushing away dissimilar input representations. This combined approach allows the model to effectively capture the internal details of the image and the complex relationships between its parts, resulting in more compact and discriminative feature representations.

[0009] Furthermore, to further improve model performance, some studies have explored adaptive masking strategies, which dynamically adjust the masking method based on the characteristics of the input samples and the model's learning state. This differs from previous methods that relied solely on contrastive learning during the pre-training phase. MCL can be implemented simultaneously during supervised training, avoiding additional pre-training costs and improving computational efficiency. As research continues, more and more effective improvements have been discovered, collectively driving progress in the field of visual representation learning. Therefore, MCL-based methods have demonstrated outstanding performance in various downstream tasks, such as image classification, object detection, and semantic segmentation, showcasing their powerful generalization and expressive capabilities. The development of this approach provides a solid foundation and broad scope for future visual analysis techniques and applications. Summary of the Invention

[0010] This invention proposes a framework for multimodal remote sensing data classification that reduces the dependence of deep learning models on labeled remote sensing samples, enhances their ability to extract complementary features between two modalities, and improves the accuracy and robustness of land cover classification. The method includes the following steps:

[0011] S1. Acquire hyperspectral data and lidar point cloud data, and preprocess the hyperspectral data and lidar point cloud data to obtain hyperspectral image blocks and LiDAR image blocks.

[0012] S2. Use a multi-scale feature extraction module to extract features from the hyperspectral image block and the LiDAR image block respectively to obtain hyperspectral image feature blocks and LiDAR image feature blocks;

[0013] S3. The hyperspectral image feature block and the LiDAR image feature block are fused using the CGAFT module to obtain hyperspectral image features, LiDAR image features, a first cross-modal attention map, and a second cross-modal attention map.

[0014] S4. Based on the hyperspectral image feature block, the LiDAR image feature block, the first cross-modal attention map, and the second cross-modal attention map, obtain the first mask fusion feature and the second mask fusion feature;

[0015] S5. Calculate the contrast loss based on the hyperspectral image features, the LiDAR image features, the first mask fusion features, and the second mask fusion features, and update the model parameters by combining the cross-entropy function;

[0016] S6. Based on the updated model, classify the multimodal data of the ground features to be classified and obtain the classification results.

[0017] More preferably, the preprocessing method includes: spatial alignment, format unification and resampling, enhancement and normalization processing, and slicing operation.

[0018] More preferably, the method for obtaining the hyperspectral image feature block includes:

[0019] The hyperspectral image patch is convolved using a 3D convolution with a kernel size of 9×3×3. Then, a set of two-dimensional convolutions with kernel sizes of 1×1, 3×3, and 5×5 are used to extract features simultaneously. Finally, element-wise addition is used to fuse the outputs of the two-dimensional convolutions to obtain the hyperspectral image feature patch.

[0020] More preferably, the method for obtaining the LiDAR image feature block includes:

[0021] The LiDAR image patch is convolved using a 3×3 kernel two-dimensional convolution. Then, a set of 1×1, 3×3, and 5×5 kernel two-dimensional convolutions are used simultaneously for feature extraction. Finally, element-wise addition is used to fuse the outputs of the two-dimensional convolutions to obtain the LiDAR image feature patch.

[0022] More preferably, the method for obtaining the hyperspectral image features and the LiDAR image features includes:

[0023] The hyperspectral image feature block and the LiDAR image feature block are flattened and concatenated with learnable classification labels to generate a first sequence feature and a second feature sequence;

[0024] Learnable location embeddings are added to the first sequence features and the second feature sequence to obtain the hyperspectral image features and the LiDAR image features.

[0025] More preferably, the method for obtaining the first mask fusion feature and the second mask fusion feature includes:

[0026] Based on the first cross-modal attention map and the second cross-modal attention map, the comprehensive activation intensity at each feature location is calculated;

[0027] The CGAFT encoder is applied N times to both the hyperspectral image feature block and the LiDAR image feature block, and then input into the FC network to obtain feature representations for c channels. F ;

[0028] From feature representation F Calculate query Q, key K, and value V, and calculate the attention map score based on query Q, key K, and value V;

[0029] The attention map score is converted into a probability distribution, which reflects the probability that features with high overall activation intensity are masked; the Gumble max technique is combined to guide the masking strategy in a probabilistic manner.

[0030] The hyperspectral image feature block and the LiDAR image feature block are masked based on the masking strategy to obtain the first mask fusion feature and the second mask fusion feature.

[0031] More preferably, the method for updating model parameters based on the contrastive loss includes:

[0032] ,

[0033] in,

[0034] ,

[0035] ,

[0036] ,

[0037] ,

[0038] In the formula, ltotal Represents the total loss function; lCE Represents the cross-entropy function; α Indicates the weights of the comparative loss; lContrastive Indicates the total comparison loss; lHSI-liDAR Indicates the contrast loss of HSI-LiDAR; lHSI-MaskedHSI This indicates the HSI-MaskedHSI contrast loss; lLiDAR-MaskedLiDAR This indicates the contrast loss between LiDAR and MaskedLiDAR. For temperature parameters; Nb This indicates the number of images in a batch during the training process; , This indicates that in the corresponding batch, the first...k High-dimensional representations generated from each sample , This indicates the first [number]th [item] in the corresponding batch. i High-dimensional representations generated from each sample Indicates in k In the traversal of the first k The high-dimensional representation of the sample and the current number i The value is 1 when the high-dimensional representation of a sample is a high-dimensional representation generated from different samples, and 0 otherwise. This indicates the first [number]th [item] in the corresponding batch. k LiDAR high-dimensional features generated from masked samples; This indicates the first [number]th [item] in the corresponding batch. i LiDAR high-dimensional features generated from masked samples; This indicates the first [number]th [item] in the corresponding batch. i High-dimensional LiDAR features generated from unmasked samples.

[0039] This invention also provides a multimodal collaborative ground feature classification system based on mask contrast learning, comprising: a pre-training module and a classification module;

[0040] The pre-training module includes: a data processing unit, a feature extraction unit, a fusion processing unit, a mask fusion unit, and a loss calculation unit;

[0041] The data processing unit is used to acquire hyperspectral data and lidar point cloud data, and to preprocess the hyperspectral data and lidar point cloud data to obtain hyperspectral image blocks and LiDAR image blocks.

[0042] The feature extraction unit is used to extract features from the hyperspectral image block and the LiDAR image block respectively using a multi-scale feature extraction module to obtain hyperspectral image feature blocks and LiDAR image feature blocks;

[0043] The fusion processing unit is used to perform fusion processing on the hyperspectral image feature block and the LiDAR image feature block respectively using the CGAFT module to obtain hyperspectral image features and LiDAR image features, as well as a first cross-modal attention map and a second cross-modal attention map;

[0044] The mask fusion unit is used to obtain a first mask fusion feature and a second mask fusion feature based on the hyperspectral image feature block, the LiDAR image feature block, the first cross-modal attention map, and the second cross-modal attention map;

[0045] The loss calculation unit is used to calculate the contrast loss based on the hyperspectral image features, the LiDAR image features, the first mask fusion features, and the second mask fusion features, and to update the model parameters in combination with the cross-entropy function;

[0046] The classification module is used to classify the multimodal data of ground features to be classified based on the updated model, and obtain the classification results.

[0047] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0048] By introducing a feature fusion module based on an attention-guided probabilistic masking strategy, this invention enables dynamic masking of multimodal inputs (such as hyperspectral images and LiDAR data). This strategy enhances the model's generalization ability and robustness by comparing masked features with original features, allowing it to more accurately capture semantic information between modalities with a limited number of labeled samples, thereby improving the model's performance and accuracy in complex tasks.

[0049] Furthermore, the cross-attention fusion module and hierarchical contrastive learning paradigm proposed in this invention effectively enhance feature alignment between HSI and LiDAR data, ensuring consistency between the original data and masked data at different levels. This scheme optimizes the fusion effect of cross-modal data, not only improving the overall performance of the model but also enhancing its adaptability and robustness in different environments and application scenarios. This method effectively mitigates the problems caused by inconsistencies in information between modalities, thereby improving the accuracy and stability of cross-modal learning tasks. We conducted extensive experiments on four benchmark datasets to verify the effectiveness of this invention. The results show that the proposed method can effectively solve the classification problem with limited data and labeled samples, and outperforms other state-of-the-art methods. Attached Figure Description

[0050] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A flowchart illustrating the overall framework of the method provided in this embodiment of the invention;

[0052] Figure 2 This is a schematic diagram of the preprocessing method according to an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of the multi-scale feature extraction module structure according to an embodiment of the present invention;

[0054] Figure 4 This is a diagram of the CGAFT module architecture according to an embodiment of the present invention;

[0055] Figure 5 This is a flowchart illustrating the framework for classification based on a model with updated parameters in an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] Example 1:

[0059] like Figure 1 As shown, this embodiment provides a multimodal cooperative land cover classification method based on mask contrast learning, including the following steps:

[0060] S1. Acquire hyperspectral data and lidar point cloud data, and preprocess the hyperspectral data and lidar point cloud data to obtain hyperspectral image blocks and LiDAR image blocks.

[0061] Further implementation lies in, such as Figure 2 As shown, the preprocessing methods include: spatial alignment, format unification and resampling, enhancement and normalization, and slicing.

[0062] Specifically, raw hyperspectral data acquired by a hyperspectral camera typically appears as a regular three-dimensional data cube with a dimensional structure of "height × width × number of bands". Height and width correspond to the spatial dimensions of the imaging area, while the number of bands represents the reflectance information collected within different narrow wavelength ranges. Unlike ordinary color images that only record the brightness of the red, green, and blue channels, hyperspectral data records a complete and continuous spectral curve at each pixel location, covering a rich spectral response from visible light (e.g., 400 nm) to near-infrared, and even extending to short-wave infrared (e.g., 2500 nm). This fine-grained spectral information can accurately characterize the physical composition, chemical structure, and biological characteristics of ground features, giving hyperspectral data extremely high discrimination capabilities in tasks such as ground feature classification and material identification. Furthermore, due to its regular spatial arrangement, hyperspectral images are also convenient for registration and fusion with other types of remote sensing data. Raw data acquired by a LiDAR system is typically presented as a point cloud, which, unlike hyperspectral imagery, is an irregularly distributed three-dimensional data set. Each LiDAR point records its three-dimensional coordinates (X, Y, Z) in space, reflecting the spatial geometric position of the object's surface. It also includes reflection intensity, i.e., the energy of the returned signal, providing additional clues about the surface material or roughness. Furthermore, depending on the configuration and application requirements of the LiDAR system, each point may also contain additional attributes such as echo count, scan timestamp, echo sequence, and calculated normal vector. LiDAR point clouds can meticulously depict spatial information such as terrain features, building outlines, and vegetation structures. Although typically sparse in spatial distribution, they possess extremely strong three-dimensional perception capabilities. Therefore, LiDAR data plays an irreplaceable role in 3D modeling, scene reconstruction, object detection, and ground feature segmentation, forming a natural complement to hyperspectral data, which excels in spectral detail, at the information level, providing rich and powerful data support for multimodal collaborative classification. Tables 1 and 2 illustrate the original data structures of the hyperspectral camera and LiDAR, respectively.

[0063] Table 1

[0064]

[0065] Table 2

[0066]

[0067] Therefore, before performing multimodal collaborative classification of hyperspectral data and lidar point cloud data, systematic preprocessing of both types of data is necessary to eliminate differences in spatial, format, and feature levels between the original data. The first step is spatial alignment. Hyperspectral data is typically organized in a two-dimensional regular grid, with each pixel corresponding to a fixed area on the Earth's surface and having a clear positional relationship in the geographic coordinate system. Lidar point cloud data, on the other hand, is a sparse, irregularly distributed three-dimensional point cloud, with each point having its own independent spatial coordinates. Therefore, before multimodal fusion, it is essential to ensure a strict spatial correspondence between the two, meaning that the same ground feature can be matched in both data sources. This embodiment uses registration based on Ground Control Points (GCPs). Ground control points are a set of feature points that can be clearly identified and whose locations are known in both hyperspectral and lidar point cloud data, such as road intersections, isolated buildings, or obvious terrain features. By labeling these control points in both types of data and calculating a spatial transformation model based on the control point coordinates, the hyperspectral image data or lidar point cloud data can be accurately reprojected onto a unified geographic coordinate system (such as UTM projection). The final registration result was evaluated using root mean square error (RMSE) to ensure that the spatial deviation of the control points was within an acceptable range, thus laying a reliable foundation for subsequent fusion.

[0068] Following format unification and resampling, even after spatial registration, differences in data organization between hyperspectral data and lidar point cloud data persist: hyperspectral data features a regular two-dimensional pixel arrangement, while lidar point cloud data is an irregular, sparse, three-dimensional discrete point set. To achieve data structure unification, the lidar point cloud is projected onto the ground plane (XY plane) and rasterized. Specifically, the study area is divided into regular grid cells, and the point cloud attribute features within each cell are statistically analyzed, such as average height, maximum height, point density, and average reflectance intensity, thereby generating a regular two-dimensional feature map. This processing method ensures that the lidar data maintains structural consistency with the hyperspectral image, facilitating pixel-by-pixel feature fusion and classification analysis.

[0069] Thirdly, at the feature level, both types of data require enhancement and normalization. Gamma correction and atmospheric correction are applied to hyperspectral data to improve data quality and accuracy. Gamma correction is primarily used to correct brightness distortion caused by the sensor's nonlinear response, adjusting the brightness values ​​of the data to better match human visual perception. This is achieved through a formula... This is used to adjust the image, suitable for low-light environments or uneven brightness; where, This indicates the spectral value of a specific pixel in the output hyperspectral image; IoriginalThis represents the spectral value of a pixel in the input hyperspectral image, typically normalized to the range [0,1]. γ represents the gamma coefficient, used to control the intensity of the correction. Atmospheric correction restores ground reflectivity by removing atmospheric scattering and absorption effects. Correction is performed using the 6S model to ensure the data reflects the true radiation of the ground, rather than interference from atmospheric factors. Furthermore, normalization is a crucial step in ensuring data quality and improving analytical accuracy in hyperspectral and lidar point cloud data processing. This invention employs minimum-maximum normalization for hyperspectral and lidar point cloud data. By compressing the data to a specified range (typically [0,1]), dimensional differences between different bands are eliminated. Specifically, this is achieved through the following formula:

[0070] ,

[0071] In the formula, This represents the pixel values ​​of a specific channel in the output image. x This represents all pixel values ​​of a specific channel in the input image; min ( x )and max ( x ) represent the minimum and maximum values ​​of all pixels in a certain channel of the input image, respectively.

[0072] Finally, after the above processing, the hyperspectral data and LiDAR point cloud data are segmented and labeled. Typically, the hyperspectral data and LiDAR point cloud data are cut into fixed-size patches (e.g., 11×11 pixels). Each patch is labeled with the category of its center pixel. Based on the model input requirements, the processed hyperspectral patches and point cloud patches are packaged into a unified format, such as [Batch, height, width, number of channels], for use in neural network model training.

[0073] S2. The hyperspectral image block and the LiDAR image block are extracted using a multi-scale feature extraction module to obtain hyperspectral image feature blocks and LiDAR image feature blocks respectively.

[0074] Hyperspectral image patches and LiDAR image blocks Feature extraction is performed in the multi-scale feature extraction module, where... s × s Indicates block size; c 1. c 2 represents the raw number of channels in the hyperspectral (HSI) data and lidar point cloud data (LiDAR), such as... Figure 3The architecture diagram is shown. For the hyperspectral data branch, a 3D convolution operation with a kernel size of 9×3×3 is performed on the hyperspectral image patch. Then, a set of two-dimensional convolutions with kernel sizes of 1×1, 3×3, and 5×5 are used simultaneously for feature extraction. Finally, element-wise addition is used to fuse the outputs of the two-dimensional convolutions to obtain the hyperspectral image feature patch. For LiDAR point cloud data, since it is a single-channel image, the 3D convolution used for hyperspectral data is replaced with a single 3×3 two-dimensional convolution to extract elevation information from the LiDAR image patch. Here, the number of filters in the two convolutional layers in the hyperspectral data branch are 8 and 64, respectively, while in the LiDAR point cloud data branch, the number is 64. Furthermore, to accelerate model convergence, batch normalization (BN) and LeakyReLU functions are applied sequentially after each convolutional layer. After this step, the blocks from the two modalities are mapped to the hyperspectral image feature patches, respectively. and LiDAR image feature blocks Among them c =64.

[0075] S3. The hyperspectral image feature block and the LiDAR image feature block are fused using the CGAFT module to obtain hyperspectral image features, LiDAR image features, a first cross-modal attention map, and a second cross-modal attention map.

[0076] Subsequently, using two dual-branch CGAFTs from the obtained and The system obtains global context content and further enhances the complementary relationship between modalities through a cross-guided attention fusion mechanism. These features are processed by a cleverly designed cross-attention fusion module to generate corresponding attention maps. Taking the HSI branch CGAFT as an example, specifically, the system first... and Flatten and concatenate with learnable classification labels to generate first-sequence features Second feature sequence ,in N = s 2+1. Then, learnable positional embeddings are added. and To generate hyperspectral image features and LiDAR image features Next, and Enter in order M In a dual-branch CGAFT, global contextual knowledge is mined and feature fusion between the two modalities is enhanced. Simultaneously, attention maps between the two modalities are obtained to augment the original input. This embodiment sets...N =2. Since all CGAFTs have the same structure, this embodiment uses the CGAFT of the first HSI branch as an example for explanation. The specific process is detailed in the appendix. Figure 4 Tables 3 and 4 are provided. Table 3 shows the specific algorithm process for obtaining hyperspectral features and LiDAR image features, and Table 4 shows the specific algorithm process for obtaining the first cross-modal attention map and the second cross-modal attention map.

[0077] Table 3

[0078]

[0079] Table 4

[0080]

[0081] S4. Based on the hyperspectral image feature block, the LiDAR image feature block, the first cross-modal attention map, and the second cross-modal attention map, the first mask fusion feature and the second mask fusion feature are obtained.

[0082] The combined activation levels of the first and second cross-modal attention maps generated by the CGAFT module are obtained to dynamically enhance the input features through masking, thereby improving the model's robustness and ensuring the effectiveness of information from each branch. Subsequently, hierarchical contrastive learning and cross-entropy are used to fully mine the complementary information between different modalities and the original and masked inputs. Through semantic exchange between HSI and LiDAR and the original and masked inputs, the complementary features of the two branches can be fully explored and fused.

[0083] Taking the radar point cloud data branch as an example, specifically, firstly, based on the generated first and second cross-modal attention maps, the comprehensive activation intensity of each feature location is calculated to measure its importance in cross-modal interaction. In the cross-modal attention map (CGAttention), the dot product operation between the query vector of the LiDAR branch and the key vector of the HSI branch inherently achieves the fusion and correlation modeling of different modal features in the spatial dimension. The attention score obtained by the dot product not only reflects the similarity between features but also implicitly captures the complementary and synergistic relationship between the two modalities. Therefore, the response value in the cross-modal attention map can serve as an indicator of the contribution of each feature location in the fusion process. Subsequently, a mask is dynamically generated based on the activation intensity of each location, and the features in low-response regions are perturbed, specifically by randomly setting them to zero or using additive Gaussian noise. In this way, the model pays more attention to high-value regions in the fused features during training, while improving robustness to the loss of local information. Furthermore, to better adapt to the characteristics of different samples or batches, the mask ratio can also be adjusted based on the statistical characteristics of the attention distribution within the current batch, thereby achieving more flexible and targeted feature enhancement.

[0084] Using a CGAFT-based multi-head cross-guided self-attention backbone to calculate radar spectral feature attention weights (i.e., radar integrated activation intensity) in the spatial dimension, respectively... and CGAFT encoder applied gθ M times, then output to an FC network to obtain feature representations for c channels. Each MHCGA block consists of H headers, defined for the HSI branch as follows:

[0085] ,

[0086] In the formula, i ∈1,..., h This represents the attention head. Self-attention is used to extract information from feature representations. F Calculate the query Q, key K, and value V. d k express k The dimension;

[0087] Calculate the average of the attention maps on the self-attention head to obtain the attention score A for the feature:

[0088] .

[0089] We use only the last CGAFT block to compute the most active pixels, as it inherits information learned from the previous blocks. The computed attention score A serves as an empirical semantic richness prior to guide the multimodal feature fusion masking strategy. Our goal is to mask the spectra with the highest activation intensity to reduce reliance on them, thereby enhancing model robustness and generalization ability. Therefore, we transform the attention score into a probability distribution that reflects the probability of each spectral feature being masked:

[0090] ,

[0091] In the formula, This is a temperature hyperparameter used to control the sharpness of the output probability. Lower temperatures (less than 1) will sharpen the distribution, making it more concentrated and focused on the most active part of the spectrum. Therefore, setting... And use the Gumble max technique to guide the masking strategy probabilistically:

[0092] ,

[0093] ,

[0094] ,

[0095] In the formula, It is a uniform distribution, and Top means sorting the input from largest to smallest and then selecting the K largest ones; r `maskinds` represents a Gumbel-distributed random variable; `indices` represents the indexes of the features. `maskinds` are the indexes of the features to be masked, which are then replaced with learnable masking labels to obtain the first mask fusion features. Second mask fusion features As shown in Table 5.

[0096] Table 5

[0097]

[0098] Similarly, adopting the same method as described above... and get and A similar process, based on and Obtain hyperspectral mask features ZMasked_H and LiDAR mask features ZMasked_L .

[0099] S5. Calculate the contrast loss based on the hyperspectral image features, LiDAR image features, first mask fusion features, and second mask fusion features, and update the model parameters by combining the cross-entropy function.

[0100] By comparing the masked fusion features with the original data fusion features, the model is forced to perform efficient feature learning under incomplete information conditions, thereby significantly improving its robustness in noisy, missing, or partially occluded data conditions.

[0101] The learning objective of our network is to learn the similarity between different modalities of the same sample and the difference between masked and unmasked samples within the same modality. Therefore, this embodiment constructs three contrast objectives: first, cross-modal contrast between HSI and LiDAR features to promote feature consistency at the same location across different modalities; second, contrast between the original HSI and the masked HSI to ensure semantic consistency of features before and after mask perturbation; and third, contrast between the original LiDAR and the masked LiDAR to improve the stability of the LiDAR branch. Specifically, this involves one cross-modal contrast loss and two intramodal contrast losses: HSI-LiDAR contrast loss, HSI-Masked HSI contrast loss, and LiDAR-Masked LiDAR contrast loss. All the above contrasts are performed within a high-dimensional feature space. Features are projected through a small MLP network and used for computation, optimized using InfoNCE loss based on cosine similarity. This process promotes deeper semantic fusion between modalities, strengthening complementary information sharing between the two modalities and further enhancing the semantic representation capability of the features. Finally, by fusing original features and masked features, the model effectively achieves full mining and integration of multimodal complementarity, significantly improving its classification performance and generalization ability in complex scenarios.

[0102] The three comparative losses are as follows:

[0103] ,

[0104] ,

[0105] ,

[0106] In the formula, lHSI-liDAR Indicates the contrast loss of HSI-LiDAR; lHSI-MaskedHSI This indicates the HSI-MaskedHSI contrast loss; lLiDAR-MaskedLiDAR This indicates the contrast loss between LiDAR and MaskedLiDAR. For temperature parameters; Nb This indicates the number of images in a batch during the training process; , This indicates that in the corresponding batch, the first... k High-dimensional representations generated from each sample , This indicates the first [number]th [item] in the corresponding batch. i High-dimensional representations generated from each sample Indicates in k In the traversal of the first k The high-dimensional representation of the sample and the current number i The value is 1 when the high-dimensional representation of a sample is a high-dimensional representation generated from different samples, and 0 otherwise. This indicates the first [number]th [item] in the corresponding batch. k LiDAR high-dimensional features generated from masked samples; This indicates the first [number]th [item] in the corresponding batch. i LiDAR high-dimensional features generated from masked samples; This indicates the first [number]th [item] in the corresponding batch. i High-dimensional LiDAR features generated from unmasked samples.

[0107] Finally, the model parameters are updated using both the cross-entropy function and the multimodal contrastive loss. In this embodiment, the model refers to the network module model used in the above steps, so no additional pre-training cost is required. The formula is shown below:

[0108] ,

[0109] ,

[0110] In the formula, l total Represents the total loss function; l CE Represents the cross-entropy function; l Contrastive Indicates the total comparison loss; α This indicates the weight of the comparative loss.

[0111] S6. Freeze the network parameters and remove the contrast and mask operations; input the image to be classified into the trained multi-scale feature extraction module and CGAFT module to extract its high-dimensional semantic features; then, map the features to the target class space through a fully connected layer, and combine the Softmax activation function to achieve class prediction for each pixel, thereby completing the pixel-level classification task of multimodal remote sensing images. Specifically, as follows... Figure 5 As shown.

[0112] Example 2:

[0113] This embodiment provides a multimodal collaborative ground cover classification system based on mask contrast learning, including a pre-training module and a classification module. The pre-training module includes a data processing unit, a feature extraction unit, a fusion processing unit, a mask fusion unit, and a loss calculation unit. The data processing unit acquires hyperspectral data and LiDAR point cloud data, and preprocesses the hyperspectral data and the LiDAR point cloud data to obtain hyperspectral image patches and LiDAR image patches. The feature extraction unit uses a multi-scale feature extraction module to extract features from the hyperspectral image patches and the LiDAR image patches respectively, obtaining hyperspectral image feature blocks and LiDAR image feature blocks. The fusion processing unit uses a CGAFT module to process the hyperspectral image feature blocks... The feature blocks and the LiDAR image feature blocks are fused to obtain hyperspectral image features, LiDAR image features, a first cross-modal attention map, and a second cross-modal attention map. The mask fusion unit is used to obtain a first mask fusion feature and a second mask fusion feature based on the hyperspectral image feature blocks, the LiDAR image feature blocks, the first cross-modal attention map, and the second cross-modal attention map. The loss calculation unit is used to calculate the contrast loss based on the hyperspectral image features, the LiDAR image features, the first mask fusion feature, and the second mask fusion feature, and update the model parameters using the cross-entropy function. The classification module is used to classify the multimodal data of ground features to be classified based on the model with updated parameters to obtain the classification result.

[0114] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multimodal cooperative land cover classification method based on mask contrastive learning, characterized in that, Includes the following steps: S1. Acquire hyperspectral data and lidar point cloud data, and preprocess the hyperspectral data and lidar point cloud data to obtain hyperspectral image blocks and LiDAR image blocks. S2. Use a multi-scale feature extraction module to extract features from the hyperspectral image block and the LiDAR image block respectively to obtain hyperspectral image feature blocks and LiDAR image feature blocks; S3. The hyperspectral image feature block and the LiDAR image feature block are fused using the CGAFT module to obtain hyperspectral image features, LiDAR image features, a first cross-modal attention map, and a second cross-modal attention map. S4. Based on the hyperspectral image feature block, the LiDAR image feature block, the first cross-modal attention map, and the second cross-modal attention map, obtain the first mask fusion feature and the second mask fusion feature; S5. Calculate the contrast loss based on the hyperspectral image features, the LiDAR image features, the first mask fusion feature, and the second mask fusion feature, and update the model parameters using the cross-entropy function; the model includes: a multi-scale feature extraction module and a CGAFT module; S6. Based on the updated model, classify the multimodal data of the ground features to be classified and obtain the classification results; The methods for obtaining the first mask fusion feature and the second mask fusion feature include: Based on the first cross-modal attention map and the second cross-modal attention map, the comprehensive activation intensity at each feature location is calculated; The CGAFT encoder is applied M times to both the hyperspectral image feature block and the LiDAR image feature block, and then input into the FC network to obtain feature representations for c channels. F ; From feature representation F Calculate query Q, key K, and value V, and calculate the attention map score based on query Q, key K, and value V; The attention map score is converted into a probability distribution, which reflects the probability that features with high overall activation intensity are masked; the Gumble max technique is combined to guide the masking strategy in a probabilistic manner. The hyperspectral image feature block and the LiDAR image feature block are masked based on the masking strategy to obtain the first mask fusion feature and the second mask fusion feature.

2. The multimodal cooperative land cover classification method based on mask contrast learning according to claim 1, characterized in that, The preprocessing methods include: spatial alignment, format unification and resampling, enhancement and normalization, and slicing.

3. The multimodal cooperative land cover classification method based on mask contrast learning according to claim 1, characterized in that, The method for obtaining the hyperspectral image feature blocks includes: The hyperspectral image patch is convolved using a 3D convolution with a kernel size of 9×3×3. Then, a set of two-dimensional convolutions with kernel sizes of 1×1, 3×3, and 5×5 are used to extract features simultaneously. Finally, element-wise addition is used to fuse the outputs of the two-dimensional convolutions to obtain the hyperspectral image feature patch.

4. The multimodal cooperative land cover classification method based on mask contrast learning according to claim 1, characterized in that, The method for obtaining the LiDAR image feature blocks includes: The LiDAR image patch is convolved using a 3×3 kernel two-dimensional convolution. Then, a set of 1×1, 3×3, and 5×5 kernel two-dimensional convolutions are used simultaneously for feature extraction. Finally, element-wise addition is used to fuse the outputs of the two-dimensional convolutions to obtain the LiDAR image feature patch.

5. The multimodal cooperative land cover classification method based on mask contrast learning according to claim 1, characterized in that, The methods for obtaining the hyperspectral image features and the LiDAR image features include: The hyperspectral image feature block and the LiDAR image feature block are flattened and concatenated with learnable classification labels to generate a first sequence feature and a second feature sequence; Learnable location embeddings are added to the first sequence features and the second feature sequence to obtain the hyperspectral image features and the LiDAR image features.

6. The multimodal cooperative land cover classification method based on mask contrast learning according to claim 1, characterized in that, The method for updating model parameters based on the contrastive loss includes: ， in, ， ， ， ， In the formula, ltotal Represents the total loss function; lCE Represents the cross-entropy function; α Indicates the weights of the comparative loss; lContrastive Indicates the total comparison loss; lHSI-liDAR Indicates the contrast loss of HSI-LiDAR; lHSI-MaskedHSI This indicates the HSI-MaskedHSI contrast loss; lLiDAR-MaskedLiDAR This indicates the contrast loss between LiDAR and MaskedLiDAR. For temperature parameters; Nb This indicates the number of images in a batch during the training process; , This indicates that in the corresponding batch, the first... k High-dimensional representations generated from each sample , This indicates the first [number]th [item] in the corresponding batch. i High-dimensional representations generated from each sample Indicates in k In the traversal of the first k The high-dimensional representation of the sample and the current number i The value is 1 when the high-dimensional representation of a sample is a high-dimensional representation generated from different samples, and 0 otherwise. This indicates the first [number]th ... k LiDAR high-dimensional features generated from masked samples; This indicates the first [number]th ... i LiDAR high-dimensional features generated from masked samples; This indicates the first [number]th [item] in the corresponding batch. i High-dimensional LiDAR features generated from unmasked samples.

7. A multimodal collaborative land cover classification system based on mask contrastive learning, the system being used to implement the method as described in any one of claims 1-6, characterized in that, include: Pre-training module and classification module; The pre-training module includes: a data processing unit, a feature extraction unit, a fusion processing unit, a mask fusion unit, and a loss calculation unit; The data processing unit is used to acquire hyperspectral data and lidar point cloud data, and to preprocess the hyperspectral data and lidar point cloud data to obtain hyperspectral image blocks and LiDAR image blocks. The feature extraction unit is used to extract features from the hyperspectral image block and the LiDAR image block respectively using a multi-scale feature extraction module to obtain hyperspectral image feature blocks and LiDAR image feature blocks; The fusion processing unit is used to perform fusion processing on the hyperspectral image feature block and the LiDAR image feature block respectively using the CGAFT module to obtain hyperspectral image features and LiDAR image features, as well as a first cross-modal attention map and a second cross-modal attention map; The mask fusion unit is used to obtain a first mask fusion feature and a second mask fusion feature based on the hyperspectral image feature block, the LiDAR image feature block, the first cross-modal attention map, and the second cross-modal attention map; The loss calculation unit is used to calculate the contrast loss based on the hyperspectral image features, the LiDAR image features, the first mask fusion features, and the second mask fusion features, and to update the model parameters in combination with the cross-entropy function; The classification module is used to classify the multimodal data of ground features to be classified based on the updated model, and obtain the classification results.

Images