A multi-source remote sensing image fusion method and device based on cross-modal interaction enhancement

Abstract

This invention belongs to the field of image fusion technology and discloses a multi-source remote sensing image fusion method and apparatus based on cross-modal interactive enhancement. It acquires OPT images, SAR images, and DEM images; extracts basic feature maps of the OPT, SAR, and DEM images at different scales based on dynamic feature extraction blocks, and then enhances and fuses them sequentially to obtain a unified fused feature map; wherein, the dynamic feature extraction block includes a cascaded normalization layer, an SS2D module, an adaptive channel redundancy removal sub-block, a learnable descriptive convolution sub-block, and a bidirectional attention gate; this invention can effectively adapt to the feature differences of multimodal images, achieve collaborative and accurate extraction of global and local features, and remove feature redundancy, thereby improving the overall quality of multimodal remote sensing image fusion.

Description

Technical Field

[0001] This invention belongs to the field of image fusion technology, and in particular relates to a method and apparatus for multi-source remote sensing image fusion based on cross-modal interaction enhancement. Background Technology

[0002] Multimodal fusion aims to combine the complementary advantages of different sensing modalities to enhance the system's ability to perceive complex environments by integrating multi-source information. In the field of all-weather remote sensing, the fusion of optical images (OPT images), synthetic aperture radar (SAR images), and digital elevation models (DEM images) demonstrates significant complementary technological value. Among them, OPT images can provide detailed spatial resolution and rich surface texture representation; SAR images have the ability to penetrate clouds and are not limited by lighting conditions, providing all-weather perception capabilities; while DEM images can provide key terrain undulation and height information.

[0003] Feature extraction is the core of remote sensing image fusion. Existing methods use common extractors such as CNN basic convolutional structures and SIFT / HOG operators. These methods are difficult to comprehensively extract global and local features, and the extracted features have high redundancy and low effective information ratio, which increases computational complexity and cannot meet the accuracy requirements of high-resolution remote sensing image fusion, thus limiting their application.

[0004] Especially for multimodal remote sensing image fusion, the feature differences between different modal images are large, and existing extractors are even more difficult to adapt to the collaborative extraction of multi-source features, which further exacerbates the problems of incomplete feature extraction and excessive redundancy. Summary of the Invention

[0005] The purpose of this invention is to provide a method and apparatus for multi-source remote sensing image fusion based on cross-modal interaction enhancement, which can effectively adapt to the feature differences of multi-modal images, achieve collaborative and accurate extraction of global and local features, remove feature redundancy, and improve the overall quality of multi-modal remote sensing image fusion.

[0006] This invention adopts the following technical solution: a multi-source remote sensing image fusion method based on cross-modal interaction enhancement, comprising the following steps:

[0007] Acquire OPT images, SAR images, and DEM images;

[0008] Based on dynamic feature extraction blocks, basic feature maps of OPT images, SAR images and DEM images at different scales are extracted and then enhanced and fused sequentially to obtain a unified fused feature map;

[0009] The dynamic feature extraction block includes a cascaded normalization layer, an SS2D module, an adaptive channel redundancy removal sub-block, a learnable descriptive convolutional sub-block, and a bidirectional attention gate.

[0010] The output of the adaptive channel deduplication sub-block and the output of the SS2D module are connected using a residual connection.

[0011] The outputs of the learnable description convolutional sub-blocks and the SS2D module are fused using a bidirectional attention gate;

[0012] The output of the bidirectional attention gate and the input of the dynamic feature extraction block are residually connected, and then nonlinearly projected through a multilayer perceptron to obtain the basic feature map of the preset channel dimension.

[0013] Another technical solution of the present invention: a multi-source remote sensing image fusion device based on cross-modal interaction enhancement, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the above-mentioned multi-source remote sensing image fusion method based on cross-modal interaction enhancement.

[0014] The beneficial effects of this invention are as follows: By using adaptive channel redundancy removal sub-blocks and residual connections with the SS2D module, this invention can efficiently remove channel redundancy while retaining the effective features extracted by the SS2D module, thereby improving feature purity; the learnable description of the output of the convolutional sub-blocks and the SS2D module is fused through bidirectional attention gates, which can accurately focus on key features and enhance the discriminative and representational capabilities of features; the combination of bidirectional attention gates can effectively alleviate feature degradation, retain the original input information, and ultimately achieve accurate collaborative extraction of global and local features, reduce computational complexity, and significantly improve the quality and efficiency of multimodal remote sensing image fusion. Attached Figure Description

[0015] Figure 1 This is a schematic diagram illustrating the principle of a multi-source remote sensing image fusion method based on cross-modal interaction enhancement according to an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the feature extraction submodule for SAR images in an embodiment of the present invention;

[0017] Figure 3 This is a schematic diagram of the feature extraction submodule for OPT images in an embodiment of the present invention;

[0018] Figure 4 This is a schematic diagram of the feature extraction submodule for DEM images in an embodiment of the present invention;

[0019] Figure 5 This is a schematic diagram of the DFEM in an embodiment of the present invention;

[0020] Figure 6 This is a schematic diagram of the TFMM in an embodiment of the present invention;

[0021] Figure 7This is a schematic diagram of the step-by-step dynamic upsampling reconstruction mechanism in an embodiment of the present invention;

[0022] Figure 8 This is a comparison chart of the recall rates of the method of the present invention and existing methods in the task of remote sensing natural scene fusion and recognition;

[0023] Figure 9 This is a comparison chart of the accuracy of the method of the present invention and existing methods in the task of remote sensing natural scene fusion and recognition.

[0024] Figure 10 This is a schematic diagram illustrating the detection results of the method of the present invention and existing methods under the condition of complete three modes in an embodiment of the present invention;

[0025] Figure 11 This is a schematic diagram illustrating the detection results of the method of the present invention and existing methods in the case of missing SAR images in an embodiment of the present invention;

[0026] Figure 12 This is a schematic diagram illustrating the detection results of the method of the present invention and existing methods in the case of missing DEM images in an embodiment of the present invention;

[0027] Figure 13 This is a schematic diagram showing the image segmentation results of the method of the present invention and the existing method 1 under the condition of complete three-modal operation in an embodiment of the present invention;

[0028] Figure 14 This is a schematic diagram showing the image segmentation results of the method of the present invention and the existing method 1 in the case of missing OPT images in an embodiment of the present invention;

[0029] Figure 15 This is a schematic diagram of the image segmentation results of the method of the present invention and the existing method 1 in the case of missing SAR images in an embodiment of the present invention. Detailed Implementation

[0030] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0031] In practical applications of multi-source remote sensing image fusion, the risk of data quality issues increases as the number of sensor modes acquired simultaneously increases. In complex and variable weather or hardware environments, situations involving missing or low-quality data are frequently encountered, such as optical slices covered by clouds, incomplete information due to sensor malfunctions, or complete loss of modes. These challenges place higher demands on the robustness and flexibility of the fusion model.

[0032] Existing fusion methods mainly focus on bimodal fusion architectures, with research typically limited to mitigating the occlusion effects of a single modality or addressing specific bimodal missing data issues. Moreover, these models lack the ability to directly process trimodal and higher-level heterogeneous data in their architectural design, making it difficult to achieve reliable and flexible scene understanding in dynamically changing and complex environments.

[0033] With the development of deep learning, significant progress has been made in multimodal remote sensing image fusion techniques based on convolutional neural networks (CNNs), generative adversarial networks, autoencoders, and Transformers. In recent years, Mamba-based architectures have shown great potential in the field of remote sensing due to their ability to model long-range dependencies in linear time complexity. However, analysis of existing technologies reveals that current solutions still suffer from limitations in modal scalability and insufficient robustness to modality loss problems.

[0034] Regarding the limited modal scalability, most existing fusion models (including early CNNs and more recent Transformer / Mamba variants) are primarily designed for bimodal settings (such as visible light and infrared). When processing trimodal (such as OPT images, SAR images, and DEM images) or more modalities, existing bimodal architectures are difficult to directly extend, resulting in inefficiency and insufficient information redundancy handling capabilities in interactive modeling of multi-source heterogeneous sensor data.

[0035] Regarding insufficient robustness, in real-world remote sensing scenarios, modal data loss frequently occurs due to sensor malfunctions, adverse weather conditions, or transmission limitations. Existing technologies typically assume that all modal data is complete and available during the inference phase. If data from a single sensor becomes missing, model performance often degrades significantly, potentially leading to fusion task failure. In other words, existing models generally lack dynamic fault tolerance and adaptive compensation mechanisms when processing multimodal images.

[0036] This invention proposes a multi-source remote sensing image fusion method based on cross-modal interaction enhancement, which has significant advantages over existing technologies. First, this invention overcomes the bottleneck of traditional remote sensing image fusion methods being limited to dual-modal settings, achieving efficient collaborative processing of heterogeneous information from OPT, SAR, and DEM images. Furthermore, by utilizing a linear complexity architecture, this invention significantly expands the sensing dimension while fully preserving the texture details of OPT images, the penetration of SAR images, and the topographic features of DEM images, providing more comprehensive and multi-perspective scene understanding support for Earth observation missions.

[0037] Secondly, this invention significantly enhances the robustness of the model under complex and harsh conditions through an innovative modal perception masking mechanism. When faced with scenarios where modalities are missing due to sensor failure, cloud cover, or data transmission interruption, this invention can achieve adaptive information compensation and numerical masking, ensuring that the model can still output stable and consistent perception results even when the input modalities are incomplete.

[0038] Furthermore, the unified feature space constructed in this invention takes into account both global semantic dependence and local structural protection, enabling a single backbone network to efficiently support various heterogeneous downstream tasks such as scene classification, image reconstruction, re-identification, and cross-modal generation, significantly improving the versatility and industrial application value of fused features.

[0039] Specifically, this invention discloses a multi-source remote sensing image fusion method based on cross-modal interactive enhancement, comprising the following steps: acquiring OPT images, SAR images, and DEM images; extracting basic feature maps of the OPT images, SAR images, and DEM images at different scales based on dynamic feature extraction blocks, and enhancing and fusing them sequentially to obtain a unified fused feature map; wherein, the dynamic feature extraction block includes a cascaded normalization layer, an SS2D module, an adaptive channel redundancy removal sub-block, a learnable descriptive convolutional sub-block, and a bidirectional attention gate; the output of the adaptive channel redundancy removal sub-block and the output of the SS2D module are connected by a residual connection; the output of the learnable descriptive convolutional block and the output of the SS2D module are fused by a bidirectional attention gate; the output of the bidirectional attention gate and the input of the dynamic feature extraction block are connected by a residual connection, and then nonlinearly projected through a multilayer perceptron to obtain a basic feature map of a preset channel dimension.

[0040] like Figure 1 As shown, the multi-source remote sensing image fusion method proposed in this invention is constructed by serially connecting a dynamic feature extraction module, a modality-aware differential enhancement module (MDEM), and a three-modality fusion module (TFMM). The dynamic feature extraction module consists of three parallel feature extraction sub-modules. Each feature extraction sub-module extracts a feature map of one modality of the image and then sends it to the MDEM for feature enhancement. The feature extraction sub-modules take image features of different modalities as input and, through multi-scale feature processing, finally generate a basic feature map for downstream tasks. It should be noted that the basic feature map is only named to distinguish it from the subsequently enhanced and fused feature maps; its essence is still a feature map.

[0041] In the feature extraction submodule, a hierarchical multi-scale feature extraction method is designed. This method is based on a selective state-space model, which can capture cross-regional global contextual information in remote sensing images and suppress the interference of local noise without significantly increasing computational overhead. However, a single selective state-space model will lose some key details, losing some deep feature information while capturing global information.

[0042] Therefore, the present invention extracts basic feature maps from five different scales: (1) the basic feature map at scale 0, which is obtained directly by downsampling the image without feature extraction; (2) the basic feature map at scale 1, which contains rich spatial texture information details; (3) the basic feature maps at scale 2 and scale 3, which contain mid-level target semantic information; and (4) the basic feature map at scale 4, which contains global feature information. The basic feature map at scale 4 is not sent to MDEM for processing, but directly enters the upsampling process.

[0043] Finally, through aggregation and normalization operations, the basic feature maps from the first four different scales (i.e., scales 0 to 3) are fused to obtain the single-modal fused basic feature map. As input for subsequent processing, m Indices representing modalities , For the mathematical representation of the OPT image, For the mathematical representation of SAR images, This is the mathematical representation of a DEM image.

[0044] The mathematical formula is as follows:

[0045] (1)

[0046] in, Indicates the normalization layer. This represents the parameters of the normalization layer. This represents a matrix concatenation operation. Indicates the first m The 0th-scale fundamental feature map of each modality. Indicates the first m The first-scale basic feature map of each modality Indicates the first m The second-scale basic feature map of each modality. Indicates the first m The third-scale basic feature map of each modality. , This indicates that feature extraction is performed using Dynamic Feature Extraction Blocks (DFEM). This indicates a block downsampling operation. Indicates the first m The fundamental feature map of the i-th scale of a modality. Indicates the first m The first mode The scale-based feature map, in this embodiment, i=1,2,3,4.

[0047] Regarding downsampling operations, methods such as mean pooling downsampling, max pooling downsampling, and block-based fixed-position downsampling can also be used, depending on the specific application scenario.

[0048] Specifically, the number of DFEMs at different scales is designed according to the specific application scenario. In this embodiment, such as... Figure 2 As shown, in the SAR image feature extraction submodule, the SAR image... (Image size is 256×256×1) After downsampling by the block downsampling module, the image then passes through a random dropout layer (Pos_drop) to obtain the 0th-scale basic feature map of the SAR image. Then The first-scale basic feature map is obtained by concatenating two DFEMs (i.e., DFEM×2). Then, the data is downsampled by a block downsampling module, and then processed by two concatenated DFEMs to obtain the second-scale basic feature map. Then, the data is downsampled by a block downsampling module, and then processed by nine cascaded DFEMs (i.e., DFEM×9) to obtain the third-scale basic feature map. Then, the data is downsampled by a block downsampling module, and then processed by two concatenated DFEMs to obtain the fourth-scale basic feature map. In this process, obtain , , and Aggregate them It is then fed into MDEM after passing through a normalization layer.

[0049] like Figure 3 As shown, in the OPT image feature extraction submodule, the OPT image... (Image size is 256×256×3) After downsampling by the block downsampling module, the OPT image is then passed through a random dropout layer to obtain the 0th-scale basic feature map. Then The first-scale basic feature map is obtained by concatenating two DFEMs. Then, the data is downsampled by a block downsampling module, and then processed by two concatenated DFEMs to obtain the second-scale basic feature map. Then, the data is downsampled by a block downsampling module, and then processed by nine concatenated DFEMs to obtain the third-scale basic feature map. Then, the data is downsampled by a block downsampling module, and then processed by two concatenated DFEMs to obtain the fourth-scale basic feature map. In this process, obtain , , and Aggregate them It is then fed into MDEM after passing through a normalization layer.

[0050] like Figure 4 As shown, in the DEM image feature extraction submodule, the DEM image... (Image size is 256×256×1) After downsampling by the block downsampling module, the base feature map of the DEM image at scale 0 is obtained by passing it through a random dropout layer. Then The first-scale basic feature map is obtained by concatenating two DFEMs. Then, the data is downsampled by a block downsampling module, and then processed by two concatenated DFEMs to obtain the second-scale basic feature map. Then, the data is downsampled by a block downsampling module, and then processed by nine concatenated DFEMs to obtain the third-scale basic feature map. Then, the data is downsampled by a block downsampling module, and then processed by two concatenated DFEMs to obtain the fourth-scale basic feature map. In this process, obtain , , and Aggregate them It is then fed into MDEM after passing through a normalization layer.

[0051] It should be noted that the features output by the three feature extraction submodules are all fed into the same MDEM.

[0052] In DFEM, this invention constructs an improved state-space model architecture by integrating an SS2D module, an adaptive channel deduplication sub-block (i.e., an efficient channel attention module), and a learnable descriptive convolutional sub-block, and employing a bidirectional attention gate (BIATT) for feature fusion. By dynamically recovering the pixel-level short-range spatial structure using the learnable descriptive convolutional sub-block, and combining it with the adaptive channel deduplication sub-block to suppress redundant features, this achieves global dependency modeling within a large receptive field while preserving fine local feature details, significantly improving the quality of single-modal feature encoding.

[0053] like Figure 5 As shown, the input to DFEM is the basic feature map of each modality. By using parallel branches of global dependency modeling and local structure recovery, a new basic feature map with spatial consistency is generated. And output. Specifically, it includes sub-blocks for global feature extraction based on state-space model, sub-blocks for adaptive channel redundancy removal, sub-blocks for local detail feature extraction, and sub-blocks for global / local feature fusion.

[0054] Regarding the global feature extraction sub-block, the input base feature map is first processed through a normalization layer. Layer normalization is performed to map the model to a space with a mean of 0 and a variance of 1, thereby accelerating model convergence and mitigating the distribution offset problem between different channels and spatial locations. Then, the SS2D module (i.e., SS2D in the figure) is used to perform sequence modeling on the normalized base feature map.

[0055] The SS2D module, or state-space model, excels at modeling long sequences and effectively captures dependencies between distant pixels. However, its native structure primarily focuses on global modeling and has limited ability to perceive local structures. Therefore, in this invention, a two-dimensional selective scan operator is used to flatten the normalized base feature map along four scanning directions (top left to bottom right, bottom right to top left, etc.) into a 1D labeled sequence. Selective scan operations are then performed on the flattened 1D labeled sequence, as shown in the following formula:

[0056] (2)

[0057] in, Indicates the first m The first mode Scale-based global features This represents a two-dimensional feature-selective scanning operator. Representation layer normalization operator, express Parameters express The parameters.

[0058] While two-dimensional selective scanning operators excel at capturing global relationships, channel redundancy is significant in multimodal high-dimensional features, and semantic overlap weakens discriminative representations. Therefore, this invention adds adaptive channel de-redundancy sub-blocks to... The input adaptive channel de-redundancy sub-block (i.e., ECA in the image) is subjected to global average pooling, followed by a one-dimensional convolution with a kernel size of kernel, adaptively calculating the inter-channel dependency weights. A weight vector is generated using the Sigmoid activation function and multiplied pixel-by-pixel. Implement feature reweighting. The weighted features are then compared with... Perform residual summation to obtain the features after redundancy removal. The formula is as follows:

[0059] (3)

[0060] in, The mathematical representation of the adaptive channel redundancy removal sub-block. This represents the parameters for adaptive channel deduplication of sub-blocks. It should be noted that since DFEM processes the base feature map at each scale, therefore... and subscript Similarly, it represents scale index and superscript. m This also represents a modal index.

[0061] The adaptive channel deduplication sub-block model the local interaction relationships between channels, adaptively allocates weights, highlights discriminative channels and suppresses redundant channels, thereby alleviating the feature redundancy problem introduced by the two-dimensional selective scanning operator in the global modeling process, and making the model more focused on key semantic channels.

[0062] Global feature extraction sub-blocks based on state-space models tend to overlook fine-grained structural information such as edges and textures. Therefore, learnable descriptive convolutional sub-blocks (i.e., LDCs in the figure) are utilized. The invention dynamically extracts local neighborhood features, enabling the learning of convolutional sub-blocks to emphasize local textures by calculating the differences between pixels and their neighbors. Simultaneously, it designs a global / local feature fusion mechanism, generating two complementary weight maps through a bidirectional attention gate (BiAttn in the figure). and ,Will Fusion with local features. The formula is as follows:

[0063] (4)

[0064] in, Indicates the first m The base feature map after global-local fusion of the i-th scale of each modality. This represents a bidirectional attention operator. To provide a learnable mathematical representation of convolutional sub-blocks, This indicates that the parameters describing the convolutional sub-blocks can be learned.

[0065] Finally, and Element-wise addition is performed, and residual connections are built to prevent gradient vanishing. A multilayer perceptron is then used. A non-linear projection is performed to map the feature map back to the preset channel dimension. The formula is as follows:

[0066] (5)

[0067] in, Represents a multilayer perceptron Trainable parameters.

[0068] In one embodiment, as an optional approach, the input and output of the multilayer perceptron can be residually concatenated to produce a new base feature map output. Additionally, it should be noted that the new base feature map... It can be used as input for the next DFEM. It can also be used as input for the extraction of the basic feature map at the next scale. Figure 5 In this context, `drop_path` represents the random path dropping operation. It is a structural regularization that randomly discards the entire residual branch / module path (not a single neuron / channel) with a set probability during training; and retains all paths during testing. This is used to prevent overfitting in deep networks, alleviate gradient vanishing, and improve generalization and robustness.

[0069] In the feature enhancement stage, this invention designs MDEM, which processes the basic feature maps. Specifically, MDEM performs texture detail enhancement and cross-modal contrast enhancement on the basic feature maps output by different feature sub-modules in parallel, thereby obtaining enhanced texture feature maps and contrast-enhanced feature maps. Then, the enhanced texture feature maps and contrast-enhanced feature maps are aggregated to obtain the enhanced modality-aware feature maps. MDEM aims to achieve deep complementarity of multimodal information through a global attention mechanism, while combining local texture and global semantic information, and using entropy to adaptively adjust the weights of the basic feature map quality for each modality.

[0070] In one embodiment, cross-modal contrast enhancement includes: calculating adaptive weights based on Shannon entropy of the base feature map; generating a contrast enhancement feature map of the base feature map of the current image using a query vector of the base feature map of the current image, key vectors, value vectors of the base feature maps of other images, and adaptive weights; wherein the current image refers to one of OPT images, SAR images, and DEM images, and the other images refer to the other two of OPT images, SAR images, and DEM images.

[0071] Specifically, for the base feature map of each obtained modality image, local feature resampling is performed using learnable descriptive convolutional sub-blocks. By adaptively adjusting the convolutional kernel weights, the local edges, fine-grained structures, and texture information of the image are emphasized. The formula is as follows:

[0072] (6)

[0073] in, Indicates the first m Enhanced texture feature maps for each modality, for example Represents the enhanced texture feature map of the OPT image. Represents the enhanced texture feature map of a SAR image. This represents an enhanced texture feature map of a DEM image.

[0074] Actually, here It can be a base feature map at a certain scale or a base feature map fused from a single modality.

[0075] In one embodiment, when When generating the base feature maps for single-modal fusion, after obtaining the base feature maps of OPT, SAR, and DEM images at different scales, the process includes: stitching the base feature maps of OPT images at different scales and passing them through a normalization layer; stitching the base feature maps of SAR images at different scales and passing them through a normalization layer; and stitching the base feature maps of DEM images at different scales and passing them through a normalization layer. This yields the base feature maps for OPT image fusion, SAR image fusion, and DEM image fusion, which are then subjected to enhancement operations.

[0076] In this invention, an entropy-adaptive sensing computation module is designed to address potential sensor failures or incomplete data in practical applications. This module utilizes a Transformer-based multi-head attention mechanism to achieve differentiated compensation for cross-modal information. For the basic feature maps of each modality, query vectors, key vectors, and value vectors are constructed using learnable linear projection matrices.

[0077] To address the potential for noisy or low-information modal images in practical applications, an entropy-driven weighting mechanism is introduced to calculate the Shannon entropy of the basic feature map for each modality:

[0078] (7)

[0079] in, Indicates the first m Shannon entropy of the base feature map for modality fusion. for The Middle j The probability distribution of pixel values ​​after normalization.

[0080] Then, based on the preset entropy threshold H th Generate by ReLU function Adaptive weights :

[0081] (8)

[0082] Among them, when Shannon entropy is lower than the entropy threshold, It almost completely suppresses the mode; conversely, This modal information is preserved. This weight can be directly used in cross-modal attention calculations to achieve continuous and differentiable noise suppression.

[0083] This mechanism not only allows the model to detect the absence of a certain modality image, but also reduces the weight of some low-quality images (such as those with severe cloud cover or insufficient information) to avoid fusing their information.

[0084] Then, cross-modal contrast calculation is performed. In this embodiment of the invention, the contrast enhancement feature map of the OPT image is used. Let's take an example to illustrate. The basic feature map of the OPT image is enhanced using information from the basic feature maps of the SAR and DEM images. Simultaneously, adaptive weights from both the SAR and DEM image's basic feature maps are used, adjusted using the following formula:

[0085] (9)

[0086] in, This is a feature map of the OPT image after cross-modal enhancement. express The query vector, This represents the basic feature map for OPT image fusion. express The key vector, This represents the basic feature map for SAR image fusion. express Adaptive weights, express The key vector, This represents the basic feature map for DEM image fusion. express Adaptive weights, This represents the scaling factor in the attention mechanism. express The value vector, express The value vector.

[0087] Next, After passing through a fully connected layer and performing residual operations with the input feature map, the contrast enhancement feature map of the OPT image is obtained. Similarly, the same method can be used to obtain the contrast enhancement feature maps of SAR and DEM images.

[0088] After obtaining the number m Enhanced texture feature maps of each modality and contrast-enhanced feature maps Then, the two are aggregated along the channel dimension to obtain the final modality-aware feature map of the m-th modality. The formula is as follows:

[0089] (10)

[0090] in, This represents a pixel-by-pixel addition operation, used to fuse global contrast information with local structural details.

[0091] Regarding image fusion, existing multimodal image fusion methods typically employ simple feature concatenation or element-by-element addition when processing data from different modalities, which has the following drawbacks:

[0092] (1) Difficulty in heterogeneous semantic alignment: The physical imaging mechanisms of the three modalities are different (such as optical reflectivity, radar backscattering, and elevation geometry). Simple splicing can easily lead to feature conflicts and generate fusion noise. (2) Excessive redundant information and computational overhead: If a self-attention mechanism is used for global interaction of the three modalities, the computational complexity will increase exponentially. (3) Lack of modal adaptive selection mechanism: It is impossible to dynamically assign different weights to each modal feature according to different scenarios, which leads to a significant reduction in fusion effect when some modalities are of poor quality.

[0093] This invention employs TFMM to solve these problems. In TFMM, such as... Figure 6 As shown, deep feature extraction and adaptive weight perception calculation are first performed based on the modality-aware adaptive mapping unit.

[0094] Specifically, the fusion process includes: processing the modality-aware feature map sequentially through layer normalization and linear mapping layers to obtain gated weight features and local spatial features, respectively; aggregating the local spatial features of the OPT image, SAR image, and DEM image and passing them through the SS2D module to obtain multimodal fusion features; reconstructing the multimodal fusion features based on the gated weight features to obtain reconstructed features; passing the reconstructed features sequentially through a fully connected layer, a random dropout layer, and an adaptive channel redundancy removal sub-block, and then performing residual connection with the modality-aware fusion feature map to obtain a deep fusion feature map; wherein, the modality-aware fusion feature map is obtained by aggregating the modality-aware feature maps of the OPT image, SAR image, and DEM image.

[0095] In other words, modal-aware feature maps for the three modalities are first acquired and then input into the corresponding modal-aware adaptive mapping units. Within each modal-aware adaptive mapping unit, the modal-aware feature maps, after being processed by layer normalization and linear mapping layers, are divided into two parallel processing branches:

[0096] (1) The gating signal generation branch based on the gating mechanism is processed by the activation function to generate the gating weight feature representing the spatial reliability of the m-th mode. This feature is in matrix form and is reserved for adaptive modulation of the multimodal fusion feature in subsequent steps.

[0097] (2) The local feature extraction branch sequentially passes through a two-dimensional convolutional layer and an activation function to extract local context information and generate the local spatial features of the m-th modality. This feature is in matrix form.

[0098] The three local spatial features output by the local feature extraction branch are aggregated element-wise. Then, the aggregated features are input into the SS2D module to capture global long-range dependencies in the spatial dimension with linear computational complexity, outputting a multimodal fusion feature with a global perspective, as shown in the following formula:

[0099] (11)

[0100] in, Indicates multimodal fusion features, Representing the local spatial features of SAR images, Represents the local spatial features of a DEM image. This represents the local spatial features of the OPT image. It should be noted that the SS2D module here has the same structure as the SS2D module mentioned earlier, but their parameters are independent.

[0101] Then, the three gate weight features generated by the gate signal generation branch (i.e., the gate weight features of the SAR image modal sensing feature map) are called. Gating weight features of OPT image modality-aware feature maps Gated weight features of modality-aware feature maps of DEM images ), respectively with Element-wise multiplication and then element-wise addition are performed to obtain the reconstructed features. Specifically, the reconstruction of multimodal fusion features based on gated weight features includes: performing element-wise multiplication of the multimodal fusion features based on the gated weight features of OPT images, SAR images, and DEM images respectively to obtain three reconstructed sub-features; and then performing element-wise addition of the three reconstructed sub-features to obtain the reconstructed features.

[0102] The reconstructed features are sequentially passed through a fully connected layer, a random dropout layer, and an adaptive channel deduplication sub-block (with the same structure and independent parameters as the adaptive channel deduplication sub-block mentioned earlier). Finally, global residual fusion is performed, which involves adding the output of the adaptive channel deduplication sub-block to the sum of the modality-aware feature maps of the three modalities to obtain the final deep fusion feature map. Specifically, it is expressed as:

[0103] (12)

[0104] in, Indicates a fully connected layer. This indicates a randomly discarded layer.

[0105] In one embodiment, the enhancement and fusion process includes: splitting the deep fusion feature map to obtain deep fusion feature sub-maps at different scales; and performing progressive dynamic upsampling and reconstruction on the deep fusion feature sub-maps at different scales to obtain a unified fusion feature map.

[0106] As a specific implementation method, the basic feature maps at the minimum scale of OPT images, SAR images, and DEM images are not enhanced or fused; the basic feature maps at the minimum scale of OPT images, SAR images, and DEM images are added element by element to obtain the deep fusion feature sub-map at the minimum scale; the deep fusion feature sub-map at the minimum scale is separated from the deep fusion feature map to obtain deep fusion feature sub-maps at different scales, and then dynamically upsampled and reconstructed step by step to obtain a unified fusion feature map.

[0107] This invention employs a scale-wise dynamic upsampling reconstruction mechanism based on cross-layer feature aggregation. This mechanism receives deep fusion feature sub-maps from different modalities at each scale and gradually restores spatial resolution through multiple cascaded reconstruction stages. In each scale reconstruction stage, the input feature map first undergoes a block-wise upsampling module to expand its spatial resolution. Then, local context information is extracted using DFEM, and the resulting sub-maps are aggregated element-wise with the corresponding scale's deep fusion feature sub-maps from the skip connection list to obtain the reconstructed feature map output for each scale, as shown in the following equation:

[0108] (13)

[0109] in, Indicates the first Reconstructed feature maps at various scales This indicates the block upsampling module. Indicates the first Reconstructed feature maps at various scales. Similarly, the DFEM here has the same structure as the DFEM in the previous text, and the parameters are independent of each other. Indicates the first Each scale represents a deep fusion feature sub-map. It should be noted that when... At that time, the deep fusion feature sub-image at the fourth scale is... As .

[0110] In the end alignment stage at the highest resolution, to preserve the original high-frequency texture details to the greatest extent and reduce computational redundancy, the features are no longer dynamically extracted. Instead, they are directly added to the initial cross-layer features. After block upsampling, they are integrated across channels through a 1×1 convolutional layer to obtain a unified fused feature map. .

[0111] like Figure 7 As shown, The deep fusion feature map at the smallest scale (scale 4) is processed by a block upsampling module, then by two concatenated DFEMs, and finally combined with the deep fusion feature map from the skip connection list at scale 3. Element-wise aggregation is performed to obtain the reconstructed feature map at the third scale. ;Will After passing through a block upsampling module, then through nine cascaded DFEMs, and finally with a depth-fused feature submap from the second scale of the skip connection list. Element-wise aggregation is performed to obtain the reconstructed feature map at the second scale. ;Will After passing through a block upsampling module, then two cascaded DFEMs, and finally a depth-fused feature submap from the first scale of the skip connection list. Element-wise aggregation is performed to obtain the reconstructed feature map at the first scale. ;Will After passing through the block upsampling module, then through two cascaded DFEMs, and finally fused with the depth-fused feature submap from the 0th scale of the skip connection list. Element-wise aggregation is performed to obtain the reconstructed feature map at scale 0. This means restoring the feature map to the highest resolution. To preserve the original high-frequency texture details to the greatest extent and reduce computational redundancy, this feature map is directly upsampled in blocks and then no longer undergoes dynamic feature extraction. Instead, it is then integrated across channels through a 1×1 convolutional layer to obtain a unified fused feature map. .

[0112] This invention also relates to network optimization of a multi-task loss function used to train the aforementioned modules. Through this optimization, the backbone network can be guided to extract robust deep fusion features. These features simultaneously consider low-level visual details and high-level semantic information, thereby supporting multi-objective tasks such as image reconstruction (fusion) and classification (recognition).

[0113] In one embodiment, for a fusion recognition task, a fully connected layer is applied after the unified fusion feature map. The hidden layer depth and the number of neurons in the fully connected layer are determined by the number of channels in the unified fusion feature map, and the number of neurons in the output layer is equal to the number of target categories C. Loss function. The design is as follows:

[0114] (14)

[0115] in, N Indicates the number of training samples. C Indicates the number of target categories. Indicates sample Category k Unique hot-coded real tags, Represents model samples Category k The softmax probability value is calculated based on the above formula. After calculating the loss value of the training set, the model parameters are updated based on the error backpropagation algorithm.

[0116] In another embodiment, for the image fusion task, a complementary loss function consisting of brightness / contrast, texture, and structure was designed to constrain the fused image. (That is, the image obtained by reconstructing the unified fusion feature maps) from multiple source images ( , and Extract the most significant information from the data.

[0117] High-frequency texture loss: utilizing spatial gradient operator Extracting image edges and forcing the fused image gradient to follow the maximum gradient in the source image can guide the model to retain high-frequency textures and sharp edges, preventing image blurring.

[0118] (15)

[0119] in, H represents the high-frequency texture loss, and H and W represent the height and width of the fused image, respectively.

[0120] Brightness and contrast loss: This penalty penalizes the deviation between the maximum intensity of pixels in the fused image and the source image. This loss guides the model to maintain the overall brightness and contrast of the image.

[0121] (16)

[0122] in, Indicates a loss of brightness and contrast. express brightness, express brightness, express brightness, express The brightness.

[0123] Structural similarity loss: Based on structural similarity, this measures the degree to which the structure of the fused image is preserved between the source image and the source image.

[0124] (17)

[0125] in, SSIM represents the structural similarity loss.

[0126] Based on the three losses mentioned above, the overall reconstruction loss can be obtained. :

[0127] (18)

[0128] in, express Weighting factors express Weighting factors express Weighting factors.

[0129] After calculating the loss value of the training set based on the above formula, the model parameters are updated based on the backpropagation algorithm.

[0130] To further illustrate the effectiveness of the method of the present invention, a comparison was made between the method of the present invention and existing methods in the task of remote sensing natural scene fusion recognition. Existing method 1 is Chen Y, Zhao M, Bruzzone L. A novel approach to incomplete multimodal learning for remote sensing data fusion[J].IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 1-14; Existing method 2 is Yang Y, Zhu D, Qu T, et al. Single-stream CNN with learnable architecture for multisource remote sensing data[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-18, and the dataset for detection and segmentation comes from Huang X, Ren L, Liu C, et al. Urban building classification (ubc)-a dataset for individual building detection and classification from satellite imagery[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022: 1413-1421. Both existing methods 1 and 2 are three-modal image fusion methods. In the experiment, different task heads (such as detection head and segmentation head) are changed according to different task requirements, and the task heads of the three methods are kept consistent.

[0131] like Figure 8 As shown, the recall of the three methods on the multimodal fusion recognition dataset changes with the increase of the missing rate of visible light images.

[0132] As shown in the figure, the recall rate of the method of this invention remains stable at 64%–66% across a full range of missing values ​​(10%–90%), exhibiting only a very slight linear decrease with increasing missing value, without any sharp drop, demonstrating extremely strong robustness against missing values. Existing method 1 performs similarly to this method at low missing values ​​(≤20%), but its recall rate drops significantly and continuously when the missing value exceeds 30%, reaching only 23.37% at a 90% missing value, making it completely incapable of handling high missing value scenarios. Existing method 2 consistently shows a significantly lower full-range recall rate than the method of this invention, and its recall rate decreases slowly with increasing missing value, resulting in overall performance far weaker than the method of this invention.

[0133] like Figure 9 As shown, the recognition accuracy of the three methods changes with the increase of the missing rate of visible light images on a multimodal fusion recognition dataset.

[0134] As shown in the figure, the accuracy of the method of this invention remains stable at 66%–68% across the entire missing data range, with only a slight decrease at missing data rates between 10% and 20%. Subsequently, it remains almost stable with no significant performance degradation (from 20% to 90%), demonstrating extremely strong resistance to missing data. Existing Method 1 performs reasonably well at low missing data rates, but its accuracy drops drastically when the missing data rate exceeds 30%, reaching only 23.37% at a 90% missing data rate, making it completely unsuitable for high-missing-data scenarios. Existing Method 2 consistently achieves a much lower accuracy across the entire range than the method of this invention, and its accuracy decreases slowly with increasing missing data rate, resulting in consistently inferior overall performance.

[0135] In addition, the method of this invention and existing methods were compared in the task of remote sensing natural scene fusion detection. Figure 10 , Figure 11 , Figure 12 The results of the three methods under different modalities on the remote sensing target detection dataset are shown respectively. In each figure, the left column is the method of the present invention, the middle column is the existing method 1, and the right column is the existing method 2. Figure 10 The detection results are given under the condition that all three modes are present. Figure 11 The detection results are under the condition of missing SAR images. Figure 12 The image shows the detection results under the condition of missing DEM images; in the image, green boxes represent correctly detected targets, yellow boxes represent missed targets, and red boxes represent incorrectly detected targets.

[0136] from Figure 10As can be seen, the method of this invention accurately detects the vast majority of targets with green boxes, with only a very small number of red false positive boxes and no obvious yellow missed detection boxes. It exhibits high detection accuracy and an extremely low missed / false detection rate, achieving optimal target detection performance in the complete modality. Existing method 1 has a similar detection performance to the method of this invention, also primarily using green correct boxes with only a small number of false positives, and its performance in the complete modality is acceptable, but its overall missed / false detection rate is higher than that of the method of this invention. Existing method 2 produces a large number of yellow missed detection boxes and red false positive boxes, failing to detect a large number of real targets, and also exhibits false alarms in areas without targets. Its detection performance in the complete modality is significantly inferior to that of the method of this invention.

[0137] from Figure 11 As can be seen from this, the method of the present invention is similar to... Figure 10 The results for complete modalities were almost consistent, with green correct boxes dominating. There was no significant increase in missed detections / false positives, and it maintained extremely high detection stability and accuracy even with missing SAR images, demonstrating strong resistance to modal loss. Existing Method 1 showed obvious yellow missed detection boxes, with some real targets not being detected. False positives also increased slightly, and its performance declined significantly after SAR image loss. Its adaptability to modal loss was weaker than the method of this invention. Existing Method 2 further exacerbated the missed detection and false positive problems, with a large number of targets missed and a significant increase in the number of false alarms. Its detection capability essentially failed after SAR image loss, making it completely unsuitable for the mission requirements.

[0138] from Figure 12 As can be seen, the method of this invention only produces a small number of yellow missed detection boxes, while the vast majority of targets are still accurately detected by green boxes. False detections do not increase significantly, and performance only slightly decreases after the DEM image is missing, still maintaining a detection effect far superior to existing methods. Existing method 1 experiences a significant increase in missed and false detections, with a large number of real targets not being detected, and multiple false alarms occurring. The missing DEM image severely impacts its performance, and its robustness is far lower than that of the method of this invention. Existing method 2 is almost completely ineffective, with only a very small number of correct green boxes, the vast majority of targets being missed, and a large number of meaningless yellow boxes. Detection performance completely collapses after the DEM image is missing.

[0139] Therefore, it can be seen that when all three modalities are complete, the method of this invention has the highest detection accuracy and the fewest false negatives and missed detections. Even in incomplete modal scenarios with missing SAR or DEM images, the method of this invention still maintains stable and accurate detection results, with only a slight increase in false negatives and missed detections. In contrast, the detection performance of the two existing methods declines significantly, with the problems of false negatives and missed detections becoming significantly aggravated; existing method 2 even essentially loses its detection capability. The experimental results fully verify the high accuracy advantage of the method of this invention under complete modalities and its strong robustness under modality-missing scenarios, effectively adapting to the complex and varied input conditions in remote sensing target detection.

[0140] Figure 13 , Figure 14 , Figure 15 The following figures illustrate the segmentation results of the two methods (the left column represents existing method 1, the middle column represents the method of this invention, and the right column represents the ground truth labels) on a remote sensing target segmentation dataset under different modalities. Figure 13 This is the segmentation result under the condition that all three modes exist. Figure 14 The segmentation results are given under the condition of missing OPT images. Figure 15 This is the segmentation result under the condition of missing SAR images.

[0141] In a complete trimodal scene, existing method 1 achieves an overall segmentation contour close to the ground truth label, but with coarse edge details and segmentation breaks / misclassifications in some areas (such as color block boundaries and fragmented targets). The color consistency between these areas and the ground truth label is also generally poor. The method of this invention achieves segmentation results that highly overlap with the ground truth label, with clear and complete edges. The contours, sizes, and color distributions of all target types are completely consistent with the ground truth label, with no obvious misclassification / omission areas. It achieves a segmentation effect nearly identical to the ground truth in the complete modality. The ground truth label serves as a benchmark, providing accurate target boundaries and category distributions, intuitively verifying the optimal segmentation accuracy of the method of this invention.

[0142] In scenarios where OPT images are missing, existing method 1 exhibits significantly reduced segmentation performance, with noticeable edge breaks and missegmentation, loss of some small, fragmented targets, blurred boundaries of large target areas, and significant deviations in color distribution from the true labels, resulting in insufficient robustness. The method of this invention... Figure 13 The complete modality results are almost identical, with segmentation edges and region distribution still highly matching the real labels, showing no obvious distortion, omissions, or misclassifications, only extremely slight differences in details, perfectly adapting to the case of missing OPT images. The segmentation results of the method of this invention are far superior to those of the existing method 1 in terms of consistency with the real labels, verifying its strong anti-interference ability for missing OPT images.

[0143] In SAR image loss scenarios, existing method 1 exhibits further deterioration in segmentation performance, with large areas of misclassification, almost complete loss of fine targets, and significant deviation of the overall segmentation contour from the true label, resulting in blurred edges and numerous holes / mismatches, making it difficult to adapt to SAR image loss. The method of this invention exhibits only a very small amount of detail-level deviation, with the overall segmentation result still highly consistent with the true label, complete edges, and accurate region division. Only a very few areas with fragmented edges show slight differences, significantly outperforming existing method 1 and demonstrating strong robustness to SAR image loss. The method of this invention still shows significantly superior segmentation performance under SAR image loss conditions, proving the effectiveness of multimodal feature fusion.

[0144] Therefore, when all three modalities are input, the segmentation results of the method of this invention are highly consistent with the real labels, with complete edges and accurate details, significantly outperforming the existing method 1 and achieving optimal remote sensing target segmentation performance. Strong robustness under modality loss: In extreme scenarios where OPT or SAR images are missing, the segmentation results of the existing method 1 show significant degradation, with severe edge breakage and region misclassification; while the method of this invention only shows very slight performance fluctuations, and the segmentation results still maintain a high degree of consistency with the real labels, demonstrating extremely strong resistance to modality loss.

[0145] In summary, this invention discloses a feature completion logic based on a global attention mechanism in MDEM. After training, the model parameters solve the problem of modality loss or low-quality data interference in fusion performance caused by sensor failure or environmental interference (such as cloud cover) in multi-sensor all-weather perception tasks. When a modality is missing, the model can still adaptively extract effective features from the remaining complete modalities without changing the network structure, significantly improving the robustness of trimodal fusion under extreme conditions.

[0146] In TFMM, a progressive fusion method is designed, which proceeds from spatial mixing preprocessing to long-range correlation modeling and then to global residual integration. This method ensures a high degree of collaborative interaction among the three modalities while guaranteeing the complete preservation of the unique saliency features of the three modalities through triple residual connections, thereby generating a more expressive unified feature space.

[0147] The present invention also discloses a multi-source remote sensing image fusion device based on cross-modal interaction enhancement, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the above-mentioned multi-source remote sensing image fusion method based on cross-modal interaction enhancement.

[0148] The present invention also discloses an embodiment that provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0149] The present invention also provides a computer program product that, when run on a data storage device, enables the data storage device to implement the steps in the above-described method embodiments.

[0150] If the integrated unit module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a storage device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0151] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0152] Those skilled in the art will recognize that the algorithmic steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0153] It should be noted that the data used in the implementation of this invention were all collected or gathered through legal and compliant channels, and the collection and gathering activities fully comply with the requirements of relevant laws, regulations and industry standards; the existing technical methods involved in this invention were also obtained and used through legal and compliant means.

Claims

1. A multi-source remote sensing image fusion method based on cross-modal interaction enhancement, characterized in that, Includes the following steps: Acquire OPT images, SAR images, and DEM images; Based on dynamic feature extraction blocks, basic feature maps of OPT images, SAR images and DEM images at different scales are extracted and then enhanced and fused sequentially to obtain a unified fused feature map; The dynamic feature extraction block includes a series of normalization layers, an SS2D module, an adaptive channel redundancy removal sub-block, a learnable descriptive convolutional sub-block, and a bidirectional attention gate. The output of the adaptive channel redundancy removal sub-block and the output of the SS2D module are connected using a residual connection. The outputs of the learnable description convolutional sub-blocks and the SS2D module are fused using the bidirectional attention gate; two sets of complementary weight maps are generated through the bidirectional attention gate. and The global features of the m-th modality at the i-th scale The formula for fusing with local features is as follows: ， in, This represents the base feature map after global-local fusion of the m-th modality at the i-th scale. This represents a bidirectional attention operator. To provide a learnable mathematical representation of convolutional sub-blocks, express Features after adaptive channel redundancy removal sub-blocks and Performing a residual summation operation yields the redundancy-free features. This indicates that the parameters describing the convolutional sub-blocks can be learned; The output of the bidirectional attention gate and the input of the dynamic feature extraction block are residually connected, and then nonlinearly projected through a multilayer perceptron to obtain the basic feature map of the preset channel dimension. Will and Perform element-level addition, construct residuals, and then use a multilayer perceptron. A non-linear projection is performed to map the feature map back to the preset channel dimension, as shown in the following formula: ， in, Indicates the first m The first mode i Scale-based feature map Represents a multilayer perceptron Trainable parameters.

2. The multi-source remote sensing image fusion method based on cross-modal interaction enhancement as described in claim 1, characterized in that, After obtaining the basic feature map of the preset channel dimensions, the process further includes: The input and output of the multilayer perceptron are residually connected to form the basic feature map.

3. The multi-source remote sensing image fusion method based on cross-modal interaction enhancement as described in claim 2, characterized in that, The enhancements include: The base feature maps are subjected to texture detail enhancement and cross-modal contrast enhancement respectively to obtain enhanced texture feature maps and contrast enhancement feature maps; The enhanced texture feature map and the contrast enhancement feature map are aggregated to obtain the enhanced modality-aware feature map.

4. The multi-source remote sensing image fusion method based on cross-modal interaction enhancement as described in claim 3, characterized in that, The cross-modal contrast enhancement includes: Adaptive weights are calculated based on the Shannon entropy of the aforementioned basic feature map; A contrast-enhanced feature map of the current image's base feature map is generated using the query vector of the base feature map of the current image, the key vector and value vector of the base feature maps of other images, and adaptive weights. The current image refers to one of the OPT image, SAR image, and DEM image, while the other images refer to the other two of the OPT image, SAR image, and DEM image.

5. A multi-source remote sensing image fusion method based on cross-modal interaction enhancement as described in claim 3 or 4, characterized in that, The fusion includes: The modality-aware feature map is processed sequentially through layer normalization and linear mapping layers to obtain gating weight features and local spatial features, respectively. The local spatial features of OPT images, SAR images and DEM images are aggregated and then processed by the SS2D module to obtain multimodal fusion features; The multimodal fusion features are reconstructed based on the gated weight features to obtain the reconstructed features; The reconstructed features are sequentially passed through a fully connected layer, a random dropout layer, and an adaptive channel deduplication sub-block, and then residually connected with the modality-aware fusion feature map to obtain a deep fusion feature map. The modality-aware fusion feature map is obtained by aggregating the modality-aware feature maps of OPT image, SAR image and DEM image.

6. The multi-source remote sensing image fusion method based on cross-modal interaction enhancement as described in claim 5, characterized in that, Reconstructing the multimodal fusion features based on the gated weight features includes: The gated weight features based on the OPT image, SAR image, and DEM image are used to perform element-wise multiplication of the multimodal fusion features to obtain three reconstructed sub-features. The reconstructed features are obtained by adding the three reconstructed sub-features element by element.

7. The multi-source remote sensing image fusion method based on cross-modal interaction enhancement as described in claim 6, characterized in that, After obtaining the basic feature maps of OPT images, SAR images, and DEM images at different scales, the following is included: The basic feature maps of the OPT image at different scales are stitched together and then passed through a normalization layer; The basic feature maps of SAR images at different scales are stitched together and then passed through a normalization layer. The basic feature maps of the DEM image at different scales are stitched together and then passed through a normalization layer.

8. The multi-source remote sensing image fusion method based on cross-modal interaction enhancement as described in claim 7, characterized in that, After sequential enhancement and fusion, the process includes: The deep fusion feature map is split into deep fusion feature sub-maps at different scales; The deep fusion feature maps at different scales are dynamically upsampled and reconstructed step by step to obtain a unified fusion feature map.

9. The multi-source remote sensing image fusion method based on cross-modal interaction enhancement as described in claim 8, characterized in that, Also includes: The basic feature maps at the minimum scale of the OPT image, SAR image, and DEM image are not enhanced or fused; The minimum-scale basic feature maps of the OPT image, SAR image, and DEM image are added element by element to obtain the minimum-scale deep fusion feature sub-map. The deep fusion feature map at the smallest scale is split into deep fusion feature maps at different scales and then dynamically upsampled and reconstructed to obtain a unified fusion feature map.

10. A multi-source remote sensing image fusion device based on cross-modal interaction enhancement, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements a multi-source remote sensing image fusion method based on cross-modal interaction enhancement as described in any one of claims 1-9.

Images