Remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention

Abstract

This application discloses a remote sensing data fusion method based on Locality Sensitive Hashing (LSH) and cross-modal attention, comprising: acquiring optical data and radar backscattering data and preprocessing them; inputting the preprocessed data into a two-branch neural network to extract optical features and SAR features; projecting the two types of features onto a shared attention space to obtain a query feature set and a key feature set; encoding the feature vectors using a LSH function and constructing a hash table; using each query feature as a retrieval term, performing an approximate nearest neighbor search in the hash table to retrieve key features that are adjacent in the feature space but of different categories as a hard negative sample set; for each query feature, performing cross-modal attention calculation only on its positive samples and hard negative samples to obtain attention weights, and then weighted and aggregated to obtain fused features; and performing land cover classification based on the fused features. The method provided in this application achieves effective alignment of heterogeneous modal features and reduces computational complexity.

Description

Technical Field

[0001] This application relates to the field of remote sensing data fusion technology, specifically to a remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention. Background Technology

[0002] Multimodal remote sensing data fusion technology aims to integrate heterogeneous remote sensing data from different sensors (e.g., optical cameras, synthetic aperture radar, hyperspectral imagers, and lidar) to generate more comprehensive and accurate representations of ground features by extracting and fusing complementary information from each modality. This type of technology overcomes the limitations of single-modal data in terms of resolution, viewing angle, and information type, and is widely used in environmental monitoring, agricultural surveys, urban planning, and disaster assessment. With the rapid development of Earth observation technology, existing technologies have acquired massive amounts of multimodal remote sensing images from spaceborne, airborne, and ground platforms. The significant improvements in spatial, temporal, and spectral resolution of these images provide a rich source of information for accurately perceiving the Earth's surface.

[0003] The core challenge of multimodal remote sensing data fusion lies in effectively addressing the heterogeneity between modalities. Different sensors, due to their varying imaging mechanisms, produce data with significant differences in radiometric characteristics, geometry, temporal phase, and perspective, posing substantial difficulties for high-precision registration and fusion of multimodal remote sensing images. Furthermore, complex nonlinear relationships exist between multimodal data, and traditional methods often struggle to capture these deep-seated correlations. To address these challenges, deep learning-based methods have gradually become mainstream, especially cross-modal attention mechanisms that can adaptively learn intermodal correlations, providing new technical approaches for achieving efficient and accurate multimodal fusion. For example:

[0004] 1. Hybrid architecture based on convolutional neural networks and Transformers:

[0005] Among existing multimodal remote sensing data fusion methods, the most common approach is a hybrid architecture that combines the local feature extraction capabilities of Convolutional Neural Networks (CNNs) with the global dependency modeling advantages of Transformers. These methods typically utilize CNNs to capture detailed information and local spatial features in images, while leveraging the self-attention mechanism of Transformers to establish long-range dependencies, thus addressing the complex relationships between multimodal data.

[0006] (1) Complementary Information Guided Interactive Fusion Network (CIGIF-Net): This representative technique was proposed by a research team from Hohai University and is specifically designed for the joint classification of hyperspectral image (HSI) and lidar (LiDAR) data. The network comprises three core modules: a multi-scale feature learning module (using serial small kernel convolution to reduce computational complexity), a feature interactive fusion module (achieving cross-modal local spatial information complementarity based on channel attention technology), and a complementary information guided attention mechanism (dynamically utilizing cross-modal complementary information to guide global feature construction). Experiments show that CIGIF-Net achieves a significant improvement in classification accuracy on three mainstream multimodal remote sensing datasets (overall accuracy improvement of 3.6–5.2%).

[0007] (2) LiteSeger Network: Another lightweight hybrid architecture proposed by the research team of Beijing Normal University. By combining the advantages of CNN and Transformer, it significantly reduces computational complexity while maintaining high accuracy. The network is designed with a distance-aware group attention mechanism, which effectively preserves spatial details and global dependencies while reducing computational complexity, thereby enhancing the model's expressive power and deployability.

[0008] The advantage of this type of hybrid architecture method is that it can balance local and global features, but its limitation is that the computational complexity is still high. In particular, the self-attention mechanism of the Transformer part increases quadratically with the sequence length, which poses a significant computational efficiency challenge when processing large-scale remote sensing images.

[0009] 2. Fusion method based on state-space model:

[0010] To overcome the limitations of traditional CNN and Transformer models in terms of long-range modeling capabilities and computational complexity, researchers have begun to explore multimodal fusion methods based on State Space Models (SSMs). These methods attempt to model the dynamic relationships between sequential data through state space equations, maintaining strong modeling capabilities while reducing computational complexity.

[0011] AFMamba Network: This is a high-efficiency multimodal fusion network based on SSM (Semantic Mode Array) for semantic segmentation of remote sensing images. The network incorporates an efficient dynamic visual state space (EDVSS) module, which dynamically improves local features and reduces channel redundancy to enhance the efficiency of the standard Mamba model. Simultaneously, it introduces a cross-attention aligned fusion (CAAFM) module, combining cross-image attention fusion and channel interaction alignment to improve the accuracy and efficiency of cross-modal feature fusion. Experimental results show that this model reduces computational complexity (GFLOPs) by at least 61% in multimodal hyperspectral image semantic segmentation tasks while maintaining a low parameter count, achieving an overall accuracy of approximately 92%.

[0012] While the SSM method excels in reducing computational complexity, it still faces the problem of feature redundancy (due to the use of multiple scan paths) and has room for improvement in handling intermodal inconsistencies when dealing with highly heterogeneous multimodal data.

[0013] 3. Dedicated cross-modal attention mechanism:

[0014] To address the heterogeneity and complexity of multimodal data, researchers have developed various dedicated cross-modal attention mechanisms aimed at better capturing and fusing complementary information between different modalities. These methods typically optimize intermodal information interaction by designing specific attention weight calculation methods.

[0015] (1) Cross-Attention Mechanism Multimodal Matching Network: A research team from the National University of Defense Technology proposed a network for multimodal remote sensing image matching. This network utilizes phase consistency to suppress large differences between remote sensing images, while learning descriptors for multimodal image matching through a cross-attention mechanism, achieving generalized training of the neural network on small datasets. This method demonstrates excellent performance on publicly available multimodal remote sensing datasets and remains effective on multimodal data in other domains.

[0016] (2) Modular Duplex Attention (MODA): An attention paradigm jointly proposed by Kuaishou and Nankai University, specifically designed to address the attention imbalance problem (i.e., the unbalanced distribution of attention scores across different modalities) in large multimodal models. This method uses two components, duplex attention alignment and modular attention masking, to adjust defective attention scores through "alignment correction," effectively alleviating the attention imbalance problem caused by modal bias. Experiments show that this method significantly reduces the cross-modal attention difference rate of each layer of the model from the original 56-62% to 41-50%.

[0017] (3) Multimodal Graph Attention and Cross-modal Fusion Network: A method for human emotion recognition with a technical approach that can be learned from. This network innovatively integrates a hybrid model of Graph Attention Network (GAT), Wav2Vec 2.0TM and CNN, as well as the BERT-BiGRU text analysis module, achieving a recognition accuracy of 98.7% through a cross-modal attention mechanism. Although applied to emotion recognition, its technical framework has important reference value for remote sensing multimodal fusion.

[0018] While these dedicated attention mechanisms have demonstrated excellent performance in different scenarios, they often need to be customized for specific tasks, have limited generalization capabilities, and often require high computing resources in actual deployment.

[0019] In addition to the main technical approaches mentioned above, researchers have also proposed a variety of optimization strategies and innovative solutions to address specific challenges in multimodal remote sensing data fusion:

[0020] Lightweight Design: To reduce the computational complexity and number of parameters of the model while maintaining performance, various lightweight design schemes have been proposed. These schemes achieve a balance between accuracy and efficiency by optimizing the network structure, introducing efficient attention mechanisms, or sparse activation. For example, LiteSeger significantly reduces computational requirements while maintaining accuracy through lightweight networks and distance-aware grouped attention; SkySense V2 reduces the number of parameters and improves the model's expressive power through expert fusion (MoE) technology.

[0021] Applications of Multimodal Large Language Models (MLLM): In recent years, researchers have begun to explore the application of multimodal large language models to remote sensing tasks. For example, VectorLLM was the first to use MLLM for remote sensing vector contour extraction, skipping the traditional complex process of "segmentation-vectorization-refinement" through end-to-end point regression and directly simulating the manual annotation process. This method demonstrates strong zero-shot generalization ability, capable of vectorizing unseen targets such as aircraft and water bodies without additional training.

[0022] Joint Spatiotemporal and Semantic Alignment: To address the spatiotemporal and semantic inconsistencies among multimodal data, researchers have proposed joint alignment methods. These methods simultaneously consider the spatial correspondence and semantic consistency between data, improving fusion performance through joint optimization. While these methods typically involve complex optimization processes, they can more comprehensively address the heterogeneity problem among multimodal data.

[0023] In summary, existing cross-modal attention mechanisms have the following problems when applied to multimodal remote sensing data fusion:

[0024] 1. High computational complexity: Traditional cross-modal attention mechanisms require calculating the similarity between all cross-modal feature pairs. The computational complexity increases quadratically with the feature map space size (O(N²)), resulting in huge computational and memory overhead when processing large-scale, high-resolution remote sensing images, which is difficult to meet the needs of practical applications.

[0025] 2. Significant modal differences lead to difficulties in semantic alignment: Optical data (which provides spectral information) and SAR data (which provides structural information) have significantly different feature distributions due to their different imaging mechanisms. Existing methods struggle to achieve efficient cross-modal feature alignment and fusion while maintaining their respective modal characteristics. Summary of the Invention

[0026] To address this, this application provides a remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention, in order to solve the problems of high computational complexity and difficulty in aligning cross-modal features when applying cross-modal attention mechanisms to multimodal remote sensing data fusion in existing technologies.

[0027] To achieve the above objectives, this application provides the following technical solution:

[0028] Firstly, a remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention includes:

[0029] Step 1: Acquire multimodal remote sensing data; the multimodal remote sensing data includes optical data and radar backscattering data;

[0030] Step 2: Perform data preprocessing on the optical data and the radar backscattering data; the data preprocessing includes: performing radiometric correction and normalization on the optical data; and performing geometric correction and registration on the radar backscattering data;

[0031] Step 3: Input the preprocessed optical data and radar backscattering data into a dual-branch deep neural network architecture to extract preliminary features, obtaining optical features and SAR features; the dual-branch deep neural network architecture includes a Swing Transformer network and a ResNet-50 network; the Swing Transformer network is used to extract preliminary features from the optical data; the ResNet-50 network is used to extract preliminary features from the radar backscattering data;

[0032] Step 4: Project the optical features and the SAR features onto a shared attention space to obtain a query feature set and a key feature set; encode each feature vector in the query feature set and the key feature set using a locality-sensitive hash function to generate a corresponding hash code, and construct a first hash table and a second hash table based on the hash code; using each query feature in the query feature set as a retrieval item, perform an approximate nearest neighbor search in the second hash table based on the hash code corresponding to the query feature to retrieve key features that are adjacent to the query feature in the feature space but have different categories, as a hard negative sample set;

[0033] Step 5: For each query feature, perform cross-modal attention calculation only on the corresponding positive samples and the key features in the hard negative sample set to obtain attention weights, and perform weighted aggregation on the information in the key feature set based on the attention weights to obtain fused features; perform land cover classification based on the fused features.

[0034] Preferably, in step 3, the ResNet-50 network is a ResNet-50 variant with deformable convolutions.

[0035] Preferably, in step 4, the optical features and the SAR features are projected into a shared attention space, specifically through a learnable linear projection matrix:

[0036] ,

[0037] in, Indicates the query feature set. Indicates optical characteristics, and Represents the projection matrix. Represents the key feature set, Indicates SAR characteristics.

[0038] Preferably, in step 4, the locality-sensitive hash function is:

[0039]

[0040] in, Describe the hash code, Represents a symbolic function. Represents a random matrix. Represents the projected feature vector. This represents the bias vector.

[0041] Preferably, in step 4, the approximate nearest neighbor search specifically involves: calculating the similarity between the hash code of the query feature and the hash codes of each key feature in the second hash table based on Hamming distance; selecting key features with similarity exceeding a preset threshold as candidate samples; and filtering out key features from the candidate samples that are different from the category of the query feature to form the hard negative sample set.

[0042] Preferably, in step 4, the first hash table and the second hash table adopt a momentum update mechanism, and are synchronously updated once every preset training batch to track changes in feature distribution.

[0043] Preferably, in step 5, the cross-modal attention calculation specifically involves:

[0044]

[0045] in, This represents the similarity between all position pairs. Indicates query characteristics, Indicates SAR bond characteristics, Let T denote the normalization factor, T denote the matrix transpose, and d denote the eigenvector dimension. This indicates a hard negative sample.

[0046] Preferably, in step 5, the loss function for the cross-modal attention is:

[0047]

[0048] in, This represents the standard cross-entropy classification loss. This represents the contrast loss designed for hard negative samples. Indicates the balance factor;

[0049] .

[0050] Secondly, a remote sensing data fusion system based on locality-sensitive hashing and cross-modal attention includes:

[0051] The data acquisition module is used to acquire multimodal remote sensing data, including optical data and radar backscattering data.

[0052] A data preprocessing module is used to preprocess the optical data and the radar backscattering data; the data preprocessing includes: performing radiometric correction and normalization on the optical data; and performing geometric correction and registration on the radar backscattering data;

[0053] The feature extraction module is used to input the preprocessed optical data and radar backscattering data into a dual-branch deep neural network architecture to extract preliminary features, thereby obtaining optical features and SAR features. The dual-branch deep neural network architecture includes a Swing Transformer network and a ResNet-50 network. The Swing Transformer network is used to extract preliminary features from the optical data, and the ResNet-50 network is used to extract preliminary features from the radar backscattering data.

[0054] The hash index module projects the optical features and the SAR features onto a shared attention space to obtain a query feature set and a key feature set. It then encodes each feature vector in the query feature set and the key feature set using a locality-sensitive hash function to generate a corresponding hash code, and constructs a first hash table and a second hash table based on the hash code. Using each query feature in the query feature set as a retrieval item, it performs an approximate nearest neighbor search in the second hash table based on the hash code corresponding to the query feature to retrieve key features that are adjacent to the query feature in the feature space but have a different category, which are then used as a hard negative sample set.

[0055] The cross-modal attention module is used to perform cross-modal attention calculation only on the corresponding positive samples and key features in the hard negative sample set for each query feature, to obtain attention weights, and to perform weighted aggregation on the information in the key feature set based on the attention weights to obtain fused features; and to perform land cover classification based on the fused features.

[0056] Thirdly, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention.

[0057] Compared with the prior art, this application has at least the following beneficial effects:

[0058] 1. This application provides a remote sensing data fusion method based on Locality Sensitive Hashing (LSH) and cross-modal attention. It extracts features from optical data and radar backscatter data through a dual-branch deep neural network architecture and maps them to a unified attention space. While preserving the unique information of each modality, it reduces the feature distribution differences between heterogeneous modalities and achieves effective alignment of heterogeneous modal features. Then, LSH technology is applied to the remote sensing cross-modal attention mechanism and combined with a specialized hash function and retrieval strategy to solve the problem of high computational complexity of traditional methods.

[0059] 2. The first and second hash tables adopt a momentum update mechanism, which is updated synchronously once every preset training batch to track changes in feature distribution, thus overcoming the defect that static indexes cannot adapt to changes in feature distribution.

[0060] 3. A multi-task learning framework that combines classification loss with hard negative sample contrast loss enhances the discriminative power of features while maintaining classification accuracy. Attached Figure Description

[0061] To more intuitively illustrate the prior art and this application, exemplary drawings are provided below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary drawings, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).

[0062] Figure 1 A flowchart of a remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention is provided in Embodiment 1 of this application;

[0063] Figure 2 This is a schematic diagram of the structure of a remote sensing data fusion system based on locality-sensitive hashing and cross-modal attention, provided in Embodiment 2 of this application.

[0064] Figure 3 This is a schematic diagram of the structure of the hash index module and the cross-modal attention module provided in Embodiment 2 of this application. Detailed Implementation

[0065] The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] In the description of this application: unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," "third," etc., in this application are intended to distinguish the objects referred to and do not have any special meaning in terms of technical connotation (e.g., they should not be construed as an emphasis on importance or order). Expressions such as "including," "comprising," and "having" also mean "not limited to" (certain units, components, materials, steps, etc.).

[0067] The terms used in this application, such as "upper," "lower," "left," "right," and "middle," are generally used to indicate the general relative positional relationship for the purpose of intuitive understanding by referring to the accompanying drawings, and are not absolute limitations on the positional relationship in the actual product.

[0068] Example 1

[0069] Please see Figure 1 This embodiment provides a remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention, including:

[0070] S1: Acquire multimodal remote sensing data; multimodal remote sensing data includes optical data and radar backscattering data;

[0071] Specifically, in this embodiment, optical data is acquired through an optical sensor (such as Sentinel-2), which includes RGB or multispectral data; and radar backscattering data (i.e. SAR data) is acquired through a synthetic aperture radar sensor (such as Sentinel-1).

[0072] S2: Perform data preprocessing on optical data and radar backscattering data; data preprocessing includes: radiometric correction and normalization of optical data; geometric correction and registration of radar backscattering data;

[0073] S3: Input the preprocessed optical data and radar backscattering data into a dual-branch deep neural network architecture to extract preliminary features, obtaining optical features and SAR features; the dual-branch deep neural network architecture includes a SwinTransformer network and a ResNet-50 network; the SwinTransformer network is used to extract preliminary features from the optical data; the ResNet-50 network is used to extract preliminary features from the radar backscattering data.

[0074] Specifically, this embodiment employs a dual-branch deep neural network architecture when extracting preliminary features from optical data and radar backscattering data. This architecture includes a Swing Transformer network (i.e., the optical branch) and a ResNet-50 network (i.e., the SAR branch). The ResNet-50 network uses a variant of ResNet-50 with deformable convolutions.

[0075] The optical branch uses a Swing Transformer network to process RGB or hyperspectral data, while the SAR branch employs an improved ResNet-50 network (containing deformable convolutions) to process radar backscattering coefficients. The feature maps output by both branches maintain the same spatial resolution. The number of channels is C=256.

[0076] Assuming the given input optical data is SAR data is Preliminary features are obtained through a feature extraction network, and are represented as follows:

[0077] ,

[0078] in, , These are the output characteristics of the optical and SAR branches, respectively. and It is the spatial dimension after downsampling. It is the number of feature channels. This represents the Swing Transformer network. This indicates a ResNet-50 variant with deformable convolutions.

[0079] S4: Project the optical features and SAR features onto the shared attention space to obtain the query feature set and the key feature set; use the locality-sensitive hash function to encode each feature vector in the query feature set and the key feature set to generate the corresponding hash code, and construct the first hash table and the second hash table based on the hash code; use each query feature in the query feature set as the retrieval item, and perform an approximate nearest neighbor search in the second hash table based on the hash code corresponding to the query feature to retrieve the key features that are adjacent to the query feature in the feature space but have different categories, as the hard negative sample set;

[0080] Specifically, step S4 includes:

[0081] S401, Feature Projection: Project the optical features and SAR features onto the shared attention space to obtain the query feature set and the key feature set;

[0082] More specifically, this step projects optical and SAR features into a shared attention space using a learnable linear projection matrix:

[0083] ,

[0084] in, Indicates the query feature set. Indicates optical characteristics, and Represents the projection matrix. , d=64 is the dimension of the attention space. Represents the key feature set, This represents SAR features. This projection operation has two purposes: first, to align features from different modes to a unified space; and second, to reduce dimensionality to reduce subsequent computational overhead.

[0085] S402, LSH hash encoding: The locality-sensitive hash function is used to encode each feature vector in the query feature set and the key feature set to generate the corresponding hash code, and the first hash table and the second hash table are constructed based on the hash code;

[0086] More specifically, the locality-sensitive hash function is:

[0087]

[0088] in, Describe the hash code, Represents a symbolic function. This represents a random matrix (elements sampled from a standard normal distribution). It is the hash code length. This represents the projected feature vector (i.e., the query feature). (Right now In the (characteristics of a spatial location) This represents the bias vector. The same hashing process is also applied to the features of SAR modes. (i.e., SAR key features).

[0089] In this step, the first hash table stores the hash codes of optical features, and the second hash table stores the hash codes of SAR features. Both hash tables employ a momentum update mechanism, updating synchronously once every preset training batch to track changes in feature distribution.

[0090] S403, Hard Negative Sample Retrieval: Using each query feature in the query feature set as a retrieval item, an approximate nearest neighbor search is performed in the second hash table based on the hash code corresponding to the query feature to retrieve key features that are adjacent to the query feature in the feature space but have different categories, which are then used as a set of hard negative samples.

[0091] More specifically, this step in the approximate nearest neighbor search is as follows: Calculate the similarity between the hash code of the query feature and the hash codes of each key feature in the second hash table based on Hamming distance; select key features with similarity exceeding a preset threshold as candidate samples; and filter key features from the candidate samples that are different from the query feature category to form a hard negative sample set, expressed by the formula:

[0092]

[0093] in, This represents calculating the Hamming similarity between two hash codes. It is a similarity threshold. and These are the labels for the query sample and the candidate sample, respectively. In actual implementation, since remote sensing data usually only has image-level labels, this embodiment uses spatial location as pseudo-labels—that is, it assumes that features at different locations belong to different categories.

[0094] This embodiment ensures locality sensitivity through the design of the hash function: features that are similar in the original space have a high probability of being mapped to the same hash bucket. This property is crucial for hard negative sample mining, because hard negative samples are precisely those samples that are close in the feature space but do not belong to the same category.

[0095] S5: For each query feature, cross-modal attention is calculated only for the key features in the corresponding positive sample and hard negative sample sets to obtain attention weights. Based on the attention weights, the information in the key feature set is weighted and aggregated to obtain fused features. Land cover classification is performed based on the fused features.

[0096] Specifically, traditional cross-modal attention requires calculating the similarity of all position pairs, i.e.:

[0097]

[0098] In this embodiment, attention calculation is limited to two types of samples: first, SAR features (positive samples) located in the same spatial position as the query features; and second, hard negative samples obtained through hash retrieval. Therefore, the improved attention calculation is as follows:

[0099]

[0100] in, This represents the similarity between all position pairs. Indicates query characteristics, Indicates SAR bond characteristics, Let T denote the normalization factor, T denote the matrix transpose, and d denote the eigenvector dimension. This represents a hard negative sample. This sparse attention computation reduces complexity from... Reduce to ,in It represents the number of spatial locations in the feature map.

[0101] Therefore, the final loss function of cross-modal attention consists of two parts:

[0102]

[0103] in, This represents the standard cross-entropy classification loss. This represents the contrast loss designed for hard negative samples. It is a balancing factor;

[0104] ,

[0105] in, This is the marginal threshold. This loss function forces the model to reduce its attention weight on hard negative samples, thereby enhancing the discriminative power of the features.

[0106] This implementation of cross-modal attention operates on top of a hash index: for each query feature (e.g., a spatial location feature from an optical modality), it first quickly retrieves potential hard negative samples using a hash table, and then calculates precise attention weights. Unlike traditional methods, the attention calculation here is performed on only a small number of candidate samples, significantly reducing computational cost.

[0107] Finally, the classification decision receives the fused feature representation and outputs the class prediction for each spatial location through a two-layer MLP (containing a GELU activation function). The entire process is end-to-end, and the loss function includes standard cross-entropy loss and a specially designed hard negative sample contrast loss.

[0108] The method provided in this embodiment will be described in detail below with several examples:

[0109] (1) Examples based on multi-scale features

[0110] In remote sensing image processing, different land cover categories often exhibit significant multi-scale characteristics. This example extends the original invention by introducing a multi-scale pyramid structure during the feature extraction stage, enhancing the system's adaptability to multi-scale targets. Specific implementation includes:

[0111] ① In the optical branch of Swin Transformer, a hierarchical feature fusion strategy is adopted to integrate the four scale features output from Stage 1 to Stage 4 through upsampling and stitching operations;

[0112] ② In the SAR branch of ResNet-50, the traditional pooling layer is replaced with a deformable pyramid pooling module (Deeplabv3+ architecture), which outputs feature maps of 5 different receptive fields;

[0113] ③ An LSH hash table is independently constructed for each scale feature, and hard negative samples are retrieved by scale group during cross-modal attention calculation. This implementation is particularly suitable for complex scenarios containing mixed land cover types such as buildings, roads, and vegetation.

[0114] (2) Incremental learning example

[0115] To address the continuously updated nature of remote sensing data, this example adds incremental learning capabilities to the original architecture:

[0116] ① The hash table uses a circular buffer design to retain the latest hash table. Feature codes of each batch ( Old features are gradually eliminated during the training process;

[0117] ② Projection matrix and The Elastic Weight Fixation (EWC) strategy is adopted to constrain the changes in important parameters during model fine-tuning;

[0118] ③ The classification head adds a new category extension interface. When a new land cover category is added, only the output node needs to be added to the last layer of the MLP and the original parameters frozen. This scheme is suitable for continuous model optimization based on regularly revisited satellite data.

[0119] (3) Edge computing example

[0120] To adapt to edge computing scenarios such as drones, this example features a lightweight modification to the model:

[0121] ① Replace the Swing Transformer with the MobileViT hybrid architecture, and switch the SAR branch to ShuffleNetV2, reducing the number of parameters by 70%;

[0122] ② The LSH hash function has been changed to an 8-bit quantized version, and the hash code length has been compressed to L=64;

[0123] ③ An asynchronous update strategy is adopted, and the hash table is only updated synchronously once every K batches (K=10);

[0124] ④ Only the top-5 hard negative samples are retained during cross-modal attention computation. This example achieves real-time processing at 15fps on a Jetson Xavier NX device.

[0125] (4) Example of multi-task learning

[0126] Extend the original classification task to a joint learning framework, simultaneously performing land cover classification and change detection:

[0127] ① After feature extraction, a shared encoder is added to output 256-dimensional shared features and 128-dimensional task-specific features;

[0128] ② Construct a dual-stream hash table system: a shared table is used for cross-modal retrieval, and a task-specific table is used for single-modal hard negative sample mining;

[0129] ③ The loss function is increased by adding a triplet loss term to the change detection task, which forces the features of the changed region to be far away from the unchanged region;

[0130] ④ The classification head is connected in parallel to the change detection branch (1x1 convolution + sigmoid). This implementation is suitable for scenarios such as disaster emergency response where multiple analysis tasks need to be completed simultaneously.

[0131] (5) Example of Federated Learning

[0132] To protect data privacy and utilize distributed data, this example designs a federated learning scheme:

[0133] ① Each participant trains its own feature extractor locally, while the central server aggregates the hash table and classification header parameters;

[0134] ② The LSH hash function uses homomorphic encryption technology to ensure that the feature encoding is irreversible during transmission;

[0135] ③ During hard negative sample retrieval, each client only accesses the local hash table, and the exchange of negative samples across nodes is coordinated by the server;

[0136] ④ The projection matrix uses a differential privacy update mechanism and adds Gaussian noise ( This solution aims to prevent data leakage. It is suitable for scenarios involving cross-regional collaborative modeling where data cannot be processed centrally.

[0137] (6) Semi-supervised example

[0138] To address the problem of scarce labeled data, this implementation method introduces a semi-supervised learning mechanism:

[0139] ① Construct a teacher-student model architecture, and generate pseudo-labels for unlabeled data using the teacher model;

[0140] ② During student model training, temporary hash entries are constructed for pseudo-samples with high confidence (>0.9);

[0141] ③ Hard negative sample contrast loss is only calculated on labeled data;

[0142] ④ The projection matrix employs consistency regularization, forcing different enhanced versions of the same image to output similar hash codes. This scheme can maintain over 85% classification accuracy even with only 10% labeled data.

[0143] The remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention provided in this embodiment has the following advantages:

[0144] 1. LSH-accelerated hard negative sample mining: For the first time, locality-sensitive hashing technology is applied to remote sensing cross-modal attention mechanism. A special hash function and retrieval strategy are designed to solve the problem of high computational complexity of traditional methods.

[0145] 2. Dynamic hash table update mechanism: A momentum update hash table construction method is proposed, which enables the hard negative sample library to evolve dynamically with the model training process, overcoming the defect that static indexes cannot adapt to changes in feature distribution.

[0146] 3. Cross-modal projection alignment: A learnable bimodal projection matrix was designed to map features to a comparable shared space while preserving the characteristics of each modality, thus solving the problem of large modal differences in remote sensing data.

[0147] 4. Sparse Attention Computation: Based on hash retrieval results, it innovatively transforms fully connected attention into sparse attention, which significantly reduces memory consumption and computation time;

[0148] 5. Joint Loss Function: A multi-task learning framework is proposed that combines classification loss with hard negative sample contrast loss, which enhances the discriminative power of features while maintaining classification accuracy.

[0149] In summary, this embodiment optimizes the hard negative sample mining process by introducing approximate nearest neighbor retrieval and locality-sensitive hashing techniques. While maintaining the interpretability of the cross-modal attention mechanism, this method significantly reduces computational complexity and improves the discriminative ability of feature learning. Experiments demonstrate that this method reduces memory usage by 76% while maintaining over 90% classification accuracy. It exhibits significant advancement and practicality in the field of remote sensing multimodal data fusion.

[0150] In remote sensing applications, optical and SAR data are naturally complementary: optical data provides rich spectral information, while SAR data is sensitive to surface structure and dielectric properties. However, this complementarity also brings challenges such as large differences in feature distribution and difficulties in semantic alignment. The innovation of this embodiment lies in the design of a dynamic and learnable cross-modal retrieval method, which can automatically identify and suppress sample pairs that are similar in feature space but semantically mismatched (i.e., hard negative samples), thereby effectively improving the final classification performance.

[0151] Example 2

[0152] Please see Figure 2 and Figure 3 This embodiment provides a remote sensing data fusion system based on locality-sensitive hashing and cross-modal attention, including:

[0153] The data acquisition module is used to acquire multimodal remote sensing data, including optical data and radar backscattering data.

[0154] A data preprocessing module is used to preprocess the optical data and the radar backscattering data; the data preprocessing includes: performing radiometric correction and normalization on the optical data; and performing geometric correction and registration on the radar backscattering data;

[0155] The feature extraction module is used to input the preprocessed optical data and radar backscattering data into a dual-branch deep neural network architecture to extract preliminary features, thereby obtaining optical features and SAR features. The dual-branch deep neural network architecture includes a Swing Transformer network and a ResNet-50 network. The Swing Transformer network is used to extract preliminary features from the optical data, and the ResNet-50 network is used to extract preliminary features from the radar backscattering data.

[0156] The hash index module projects the optical features and the SAR features onto a shared attention space to obtain a query feature set and a key feature set. It then encodes each feature vector in the query feature set and the key feature set using a locality-sensitive hash function to generate a corresponding hash code, and constructs a first hash table and a second hash table based on the hash code. Using each query feature in the query feature set as a retrieval item, it performs an approximate nearest neighbor search in the second hash table based on the hash code corresponding to the query feature to retrieve key features that are adjacent to the query feature in the feature space but have a different category, which are then used as a hard negative sample set.

[0157] The cross-modal attention module is used to perform cross-modal attention calculation only on the corresponding positive samples and key features in the hard negative sample set for each query feature, to obtain attention weights, and to perform weighted aggregation on the information in the key feature set based on the attention weights to obtain fused features; and to perform land cover classification based on the fused features.

[0158] For details on the implementation of each module in a remote sensing data fusion system based on locality-sensitive hashing and cross-modal attention, please refer to the above description of the limitations of a remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention; these details will not be repeated here.

[0159] Example 3

[0160] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention.

[0161] The technical features of the above embodiments can be combined in any way (as long as there is no contradiction in the combination of these technical features). For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; these embodiments not explicitly written should also be considered to be within the scope of this specification.

Claims

1. A remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention, characterized in that, include: Step 1: Acquire multimodal remote sensing data; The multimodal remote sensing data includes optical data and radar backscattering data; Step 2: Perform data preprocessing on the optical data and the radar backscattering data; The data preprocessing includes: performing radiometric correction and normalization on the optical data; and performing geometric correction and registration on the radar backscattering data. Step 3: Input the preprocessed optical data and radar backscattering data into a dual-branch deep neural network architecture to extract preliminary features, obtaining optical features and SAR features; the dual-branch deep neural network architecture includes a Swing Transformer network and a ResNet-50 network; the Swing Transformer network is used to extract preliminary features from the optical data; the ResNet-50 network is used to extract preliminary features from the radar backscattering data; Step 4: Project the optical features and the SAR features onto a shared attention space to obtain a query feature set and a key feature set; encode each feature vector in the query feature set and the key feature set using a locality-sensitive hash function to generate a corresponding hash code, and construct a first hash table and a second hash table based on the hash code; using each query feature in the query feature set as a retrieval item, perform an approximate nearest neighbor search in the second hash table based on the hash code corresponding to the query feature to retrieve key features that are adjacent to the query feature in the feature space but have different categories, as a hard negative sample set; Step 5: For each query feature, perform cross-modal attention calculation only on the corresponding positive samples and the key features in the hard negative sample set to obtain attention weights, and perform weighted aggregation on the information in the key feature set based on the attention weights to obtain fused features; perform land cover classification based on the fused features.

2. The remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention as described in claim 1, characterized in that, In step 3, the ResNet-50 network is a ResNet-50 variant with deformable convolutions.

3. The remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention according to claim 1, characterized in that, In step 4, the optical features and the SAR features are projected into a shared attention space, specifically through a learnable linear projection matrix: ， in, Indicates the query feature set. Indicates optical characteristics, and Represents the projection matrix. Represents the key feature set, Indicates SAR characteristics.

4. The remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention according to claim 1, characterized in that, In step 4, the locality-sensitive hash function is: ， in, Describe the hash code, Represents a symbolic function. Represents a random matrix. Represents the projected feature vector. This represents the bias vector.

5. The remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention according to claim 1, characterized in that, In step 4, the approximate nearest neighbor search specifically involves: calculating the similarity between the hash code of the query feature and the hash codes of each key feature in the second hash table based on Hamming distance; selecting key features with similarity exceeding a preset threshold as candidate samples; and filtering out key features from the candidate samples that are different from the category of the query feature to form the hard negative sample set.

6. The remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention according to claim 1, characterized in that, In step 4, the first hash table and the second hash table adopt a momentum update mechanism, and are updated synchronously once every preset training batch to track changes in feature distribution.

7. The remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention according to claim 1, characterized in that, In step 5, the cross-modal attention calculation specifically involves: ， in, This represents the similarity between all position pairs. Indicates query characteristics, Indicates SAR bond characteristics, Let T denote the normalization factor, T denote the matrix transpose, and d denote the eigenvector dimension. This indicates a hard negative sample.

8. The remote sensing data fusion method based on locality-sensitive hashing and cross-modal attention according to claim 7, characterized in that, In step 5, the loss function for cross-modal attention is: ， in, This represents the standard cross-entropy classification loss. This represents the contrast loss designed for hard negative samples. Indicates the balance factor; ， in, This represents the marginal threshold.

9. A remote sensing data fusion system based on locality-sensitive hashing and cross-modal attention, characterized in that, include: The data acquisition module is used to acquire multimodal remote sensing data; The multimodal remote sensing data includes optical data and radar backscattering data; The data preprocessing module is used to preprocess the optical data and the radar backscattering data; The data preprocessing includes: performing radiometric correction and normalization on the optical data; and performing geometric correction and registration on the radar backscattering data. The feature extraction module is used to input the preprocessed optical data and radar backscattering data into a dual-branch deep neural network architecture to extract preliminary features, thereby obtaining optical features and SAR features. The dual-branch deep neural network architecture includes a Swing Transformer network and a ResNet-50 network. The Swing Transformer network is used to extract preliminary features from the optical data, and the ResNet-50 network is used to extract preliminary features from the radar backscattering data. The hash index module projects the optical features and the SAR features onto a shared attention space to obtain a query feature set and a key feature set. It then encodes each feature vector in the query feature set and the key feature set using a locality-sensitive hash function to generate a corresponding hash code, and constructs a first hash table and a second hash table based on the hash code. Using each query feature in the query feature set as a retrieval item, it performs an approximate nearest neighbor search in the second hash table based on the hash code corresponding to the query feature to retrieve key features that are adjacent to the query feature in the feature space but have a different category, which are then used as a hard negative sample set. The cross-modal attention module is used to perform cross-modal attention calculation only on the corresponding positive samples and key features in the hard negative sample set for each query feature, to obtain attention weights, and to perform weighted aggregation on the information in the key feature set based on the attention weights to obtain fused features; and to perform land cover classification based on the fused features.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images