Multi-modal image fusion method based on super-pixel and multi-scale image fusion

Abstract

The invention discloses a multi-modal image fusion method based on super-pixel and multi-scale image fusion, which belongs to the technical field of multi-modal image fusion, and comprises the following steps: decoupling the features of a source image pair into basic features and detail features through a double-branch encoder, inputting the basic features into a multi-scale image fusion module, and carrying out multi-scale image fusion; according to the module, a semantic graph node is constructed by adopting differentiable superpixel sampling to replace a traditional fixed grid, so that the integrity of an irregular target is kept; performing cross-modal interaction through a multi-scale image attention network, adaptively fusing features of different granularities by using an expert hybrid gating mechanism, compensating texture details through a brightness adaptive high-frequency injection strategy, and reconstructing the fused features into a final image through a decoder; according to the multi-modal image fusion method based on super-pixel and multi-scale image fusion, the structural integrity and detail fidelity of the fused image are effectively improved, and better performance is shown in a downstream target detection task.

Description

Technical Field

[0001] This invention relates to the field of multimodal image fusion technology, and in particular to a multimodal image fusion method based on superpixel and multiscale graph fusion. Background Technology

[0002] The core objective of multimodal image fusion is to integrate complementary information from different imaging modalities—such as the thermal radiation target representation capabilities of infrared images and the fine texture information of visible light images, as well as the anatomical and functional metabolic information of different modalities in medical images—to form a fused image with more complete information and superior visual quality. With the development of computer vision technology, this technology has been widely applied in fields such as pedestrian segmentation, salient target detection, medical diagnosis, and intelligent transportation, becoming a key prerequisite for improving the performance of downstream perception tasks.

[0003] Existing multimodal image fusion techniques are mainly divided into two categories: traditional methods and deep learning methods. Traditional methods rely on manually designed image transformation and fusion rules, which require modal information integration in a specific transform domain or spatial domain. However, these methods often suffer from modal contribution imbalance and are difficult to adapt to the fusion requirements of complex scenarios. Deep learning methods significantly improve fusion performance by automatically learning hierarchical feature representations. Their architectures encompass Convolutional Neural Networks (CNNs), Transformers, Generative Adversarial Networks (GANs), Autoencoders (AEs), and Diffusion Models. Among these, CNN-based methods rely on local convolution operators, limiting their effective receptive field and making it difficult to transmit long-range complementary cues. Transformer-based methods introduce self-attention mechanisms to model global dependencies, but they typically segment images based on rigid, fixed grids, inevitably fragmenting the semantic boundaries of irregular objects and leading to structural information fragmentation. While GANs and Diffusion Models can enhance texture fidelity, they still fail to address the semantic inconsistency issues caused by fixed-block processing.

[0004] To address the issue of diverse target scales, existing technologies attempt to introduce multi-scale pyramids, residual connections, or dynamic routing mechanisms (such as expert hybridization). However, most of these still operate on regular feature maps and lack the ability to model object-level geometric structures. Furthermore, existing fusion methods have several limitations in practical applications: first, they fail to achieve synergy between object-aware node construction and dynamic scale adaptation, making it difficult to balance the structural integrity of irregular objects with multi-scale target adaptability; second, graph aggregation and projection processes are prone to low-pass filtering effects, leading to attenuation of high-frequency texture details; and third, local high-brightness areas in modalities such as infrared images can easily cause overexposure artifacts in the fusion results, affecting visual quality and the reliability of downstream tasks.

[0005] In summary, existing multimodal image fusion technologies still have room for improvement in terms of semantic boundary preservation, multi-scale dynamic adaptation, high-frequency detail restoration, and artifact suppression. There is an urgent need for a fusion scheme that can overcome the limitations of fixed grid processing and take into account both structural integrity and detail fidelity. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal image fusion method based on superpixel and multi-scale graph fusion to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, this invention provides a multimodal image fusion method based on superpixel and multi-scale map fusion, comprising the following steps: S1. Obtain the registered multimodal source image pair. The multimodal source image pair consists of two image pairs with different modalities. The features of each source image are decoupled into basic features and detail features by a dual-branch encoder. The basic features contain low-frequency semantic structure information, and the detail features contain high-frequency texture edge information. S2. Input the basic features of the multimodal source image pairs into the multi-scale graph fusion module for semantic fusion to obtain the fused basic features. The multi-scale graph fusion module first presets cluster centers, constructs semantic graph nodes through a differentiable superpixel sampling algorithm, then uses graph attention network to perform feature interaction at multiple scales, and finally outputs the weighted aggregation through a gating network. S3. The detail features of the multimodal source image pair are fused to obtain the initial detail fusion features. At the same time, high-frequency residual information is extracted from the source image and injected into the fused basic features or the initial fused detail features after brightness adaptive adjustment to obtain enhanced fused detail features. S4. The fused basic features obtained in S2 and the fused detailed features obtained in S3 are concatenated and input into the decoder for reconstruction to obtain the final fused image.

[0008] Preferably, the multimodal source image pair is an infrared (IR)-visible light (VI) image pair, an MRI-CT image pair, an MRI-PET image pair, or an MRI-spect image pair.

[0009] Preferably, the dual-branch decoder in S1 is a Restromer-based decoder, and the expression for the feature decomposition process is as follows: in, For a Restromer-based decoder, Basic features, For detailed features, This is one of the source images in a multimodal source image pair. Infrared image, This is a visible light image.

[0010] Preferably, the specific steps of S2 include: S21. Through the differentiable superpixel sampling algorithm, the pixel-level dual-modal basic features are dynamically allocated to the preset cluster centers, establishing a soft association relationship between pixels and cluster centers, and aggregating to form semantically consistent superpixel graph nodes. S22. Construct at least two graph scales with different numbers of superpixel nodes. At each graph scale, use a graph attention network to model the association between superpixel graph nodes of different modalities to achieve cross-modal complementary information interaction. S23. Adaptive weight allocation is performed on the interactive features of each graph scale through a gating network, and the weighted aggregation is used to obtain the fused basic features.

[0011] Preferably, the differentiable superpixel sampling algorithm in S21 is a superpixel sampling network; the defined set of cluster centers is... ; Soft association is defined as: ; in, The number of channels representing the basic features. Indicates the number of superpixels. Represents pixels With cluster center Association weights, Represents the normalization function and soft interconnection graph. The dimension is Dual-modal basic characteristics The dimension is , Indicates the height of the image. Indicates the width of the image. Represents pixels eigenvectors, Indicates the first The feature vectors of the cluster centers The value range is 1 to , The feature Euclidean distance between a pixel and the cluster center is represented. This represents a temperature parameter used to adjust the smoothness of the distribution of associated weights. Superpixel map node features through Aggregation in, Indicates by the first The first cluster centers are aggregated to form the... The feature vector of each superpixel graph node.

[0012] Preferably, in S22, there are three image scales, corresponding to coarse-grained, medium-grained, and fine-grained, with the number of superpixels in that order being... , , ; Graph attention networks achieve bidirectional interaction between dual-modal superpixel nodes through a cross-attention mechanism. The fused node features after interaction are projected back into the pixel space through a transposed soft relational graph, satisfying the following formula: ; in, To use graph attention networks Infrared modal superpixel node features With visible light modal superpixel node features The fused node features obtained after processing. Represents a soft relational graph The transpose of the matrix, This represents the blending feature projected back into pixel space.

[0013] Preferably, the gated network in S23 consists of a 3×3 convolutional layer, a LeakyReLU activation function, and a 1×1 convolutional layer; The spatial weighted graph is generated using the following formula: ; in, This represents the spatial weighted graph corresponding to the three graph scales. Indicates the height of the image. Indicates the width of the image. The mapping function representing the gated network, This indicates a feature concatenation operation. , , These represent the interactive features at the coarse-grained, medium-grained, and fine-grained scales, respectively. This represents the normalization function, used to map weight values ​​to the [0,1] interval; The formula for weighted aggregation is as follows: ; in, Represents element-wise product. Indicates the first Spatial weight map corresponding to each scale Indicates the first Features after interaction at each scale, This represents the basic characteristics after fusion.

[0014] Preferably, in S3, detailed features are fused through a lightweight CNN layer, high-frequency residual information is extracted using a pooling difference operator, a maximum selection strategy is adopted to preserve significant texture features, and the features are injected after normalization by the tanh function, satisfying the following formula: ; in, This indicates the enhanced fundamental features after injecting high-frequency residuals. Indicates the basic characteristics after fusion. This represents a nonlinear activation function used to normalize the injected residuals to the interval [-1, 1]. This represents the maximum selection function. Represents the high-frequency residuals of infrared images. Represents the high-frequency residuals of a visible light image; Brightness adaptive adjustment is achieved through spatial attention masking, satisfying the following formula: ; in, This indicates the basic characteristics after brightness adjustment. This represents a spatial attention mask with a value range of (0,1), used to adaptively suppress highlight areas while preserving dark area information.

[0015] Preferably, the decoder in S4 is a Restromer-based decoder, and the image reconstruction satisfies the following formula: ; in, This indicates a Restromer-based decoder. Indicates the basic characteristics after fusion. This indicates the detailed features after fusion. This represents the final fused image.

[0016] Preferably, the model training step is also included, wherein the model consists of a dual-branch encoder, a multi-scale graph fusion module, and a decoder. The first stage trains the feature decomposition process, enabling the encoder to decouple basic features from detailed features. The second stage trains the fusion and reconstruction process based on the total loss function, which is a weighted combination of structural similarity loss, decomposition constraint loss, gradient loss, and intensity consistency loss, as shown in the following formula: ; in, Represents the total loss function. The structural similarity loss is used to constrain the structural consistency between the fused image and the source image. This represents the decomposition constraint loss, used to ensure the effectiveness of feature decomposition. This represents gradient loss, used to enhance edge detail recovery. This indicates intensity consistency loss, used to ensure overall brightness consistency and prevent energy loss. , , These represent the weight coefficients for the decomposition constraint loss, gradient loss, and strength consistency loss, respectively. ; ; in, This represents the Sobel gradient operator. Represents the absolute value of the gradient in an infrared image. This represents the absolute value of the gradient in a visible light image. This represents the L1 norm.

[0017] Therefore, the multimodal image fusion method based on superpixel and multi-scale map fusion described above, as used in this invention, has the following beneficial effects: (1) Semantic graph nodes are constructed by differentiable superpixel sampling, avoiding the fragmentation of semantic boundaries of irregular objects by traditional fixed grid segmentation. At the same time, combined with multi-scale gating mechanism, the scale difference from small targets to large environmental structures is dynamically adapted to ensure the integrity of target structure and multi-scale fusion compatibility.

[0018] (2) The high-frequency detail attenuation during the image aggregation-projection process is compensated by the high-frequency residual injection mechanism, and the local overexposure artifacts are suppressed by the brightness adaptive adjustment strategy. While preserving fine information such as texture and edge, the brightness consistency and visual coordination of the fused image are maintained.

[0019] (3) It can be applied to various multimodal image fusion scenarios such as infrared-visible light, MRI-CT / PET / spect without fine-tuning for specific modal pairs. The fused image has better semantic consistency and can provide high-quality feature representation for downstream computer vision tasks such as target detection, thereby improving the detection accuracy and reliability of downstream tasks.

[0020] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0021] Figure 1 This is a flowchart of a multimodal image fusion method based on superpixel and multi-scale graph fusion according to the present invention; Figure 2 This is an overall architecture diagram of the SMG fusion framework in an embodiment of the present invention; Figure 3 This is a schematic diagram of the semantically preserved graph interaction process in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the structure and working principle of the multi-scale gating network in an embodiment of the present invention; Figure 5 This is a visual comparison of SMG fusion with other existing methods on the TNO dataset in this embodiment of the invention; Figure 6 This is a visual comparison of SMG fusion with other existing methods on the RoadScene dataset in this embodiment of the invention; Figure 7 This is a visual comparison of SMG fusion with other existing methods on the MSRS dataset in this embodiment of the invention; Figure 8 This is a comprehensive quantitative comparison chart of radar charts based on the TNO, RoadScene, and MSRS datasets in this embodiment of the invention; Figure 9 This is a visual comparison image of SMG fusion on a medical image dataset in an embodiment of the present invention; Figure 10 This is a comparison chart of the detection performance of SMG fusion in various categories of downstream target detection tasks in embodiments of the present invention; Figure 11 This is a qualitative comparison diagram of SMG fusion and other methods in downstream target detection tasks in this embodiment of the invention; Figure 12 This is a visualization of the MoE gating weights in an embodiment of the present invention; Figure 13 This is a visual ablation matrix diagram on the TNO dataset in this embodiment of the invention; Figure 14This is a comparison diagram of the topological structures of fixed-size block segmentation and SSN superpixel segmentation in an embodiment of the present invention. Detailed Implementation

[0022] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0023] Example like Figure 1 As shown, this invention provides a multimodal image fusion method based on superpixel and multi-scale map fusion, comprising the following steps: This embodiment is applicable to multimodal image fusion scenarios such as infrared-visible light image fusion, MRI-CT image fusion, MRI-PET image fusion, and MRI-spect image fusion. The core objective is to achieve a balance between structural integrity and detail fidelity through a superpixel-guided multi-scale image fusion mechanism, thereby improving the quality of the fused image and the performance of downstream target detection.

[0024] This embodiment proposes an SMG-Fusion (Superpixel-based Multiscale Graph Fusion) model, the model architecture of which is as follows: Figure 2 As shown, this model is an end-to-end neural network model. Its core architecture includes a Restormer-based dual-branch encoder, an innovative Multiscale Graph Fusion Module (MSG), and a Restormer-based decoder. Figure 2 clearly illustrates the overall workflow of the model: the dual-branch Restormer encoder decomposes the source image into basic features and detail features. The MSG module achieves basic feature fusion through superpixel sampling aggregation across three scales, cross-modal graph inference, and a Mixture of Experts (MoE) gating mechanism. Detail features are enhanced by a high-frequency injection module after CNN branch fusion, and finally, the decoder completes image reconstruction.

[0025] The model training employs a two-stage strategy, using the PyTorch framework and implemented on a single NVIDIA Tesla V100 GPU. In this embodiment, the model's training cycle is 120 epochs. The first stage involves training the feature decomposition process over 40 epochs with a batch size of 8. The Adam optimizer is used, and the initial learning rate is 1×10⁻⁶. -4The learning rate is decayed by 0.5 times every 20 epochs to train the encoder, enabling it to decouple basic features from detailed features.

[0026] The second stage trains the fusion and reconstruction process based on the total loss function, with a training period of 80 epochs. The optimizer and learning rate strategy are the same as in the first stage. The total loss function is a weighted combination of structural similarity loss, decomposition constraint loss, gradient loss, and intensity consistency loss, as shown in the following formula: ; in, Represents the total loss function. The structural similarity loss is represented by a similarity calculation method based on three elements—brightness, contrast, and structure—known in the art, used to constrain the structural consistency between the fused image and the source image. The loss represents the decomposition constraint loss, which adopts a combination of feature reconstruction and orthogonal constraints. It inherits the bi-branch feature decomposition constraint logic from CDDFuse to ensure the effectiveness of feature decomposition. This represents gradient loss, used to enhance edge detail recovery. This indicates intensity consistency loss, used to ensure overall brightness consistency and prevent energy loss. , , These represent the weight coefficients of the decomposition constraint loss, gradient loss, and strength consistency loss, respectively; in this embodiment... , , .

[0027] ; ; in, This represents the Sobel gradient operator. This is a function for finding the maximum value; the operation is performed to find the maximum value. Represents the absolute value of the gradient in an infrared image. This represents the absolute value of the gradient in a visible light image. Describing the L1 norm, Infrared image, This is a visible light image.

[0028] S1. Obtain the registered multimodal source image pairs. Each multimodal source image pair consists of two image pairs with different modalities, such as... Figure 2As shown in the left half, a dual-branch encoder decouples the features of each source image into basic features and detail features. The basic features contain low-frequency semantic structural information, such as global illumination, the obvious shape of objects, and common structures. Since they represent the semantic "skeleton" of the scene, they require complex global reasoning capabilities to maintain structural integrity. The detail features contain high-frequency texture and edge information, including high-frequency textures, edge details, and noise patterns. Unlike the basic features, these features are highly localized and sensitive to smoothing, thus requiring a fusion strategy that prioritizes fidelity over global abstraction.

[0029] The multimodal source image pairs are infrared-visible light image pairs, MRI-CT image pairs, MRI-PET image pairs, or MRI-spect image pairs. This embodiment takes the most representative infrared-visible light image pair as an example to elaborate on the specific implementation of the SMG-Fusion model. The input image resolution is uniformly adjusted to 256×256 pixels, and the pixel values ​​are normalized to the [0,1] range.

[0030] The dual-branch decoder employs a Restormer-based shared encoder, comprising six Restormer blocks. Each Restormer block consists of four multi-scale convolutional heads (kernel sizes of 3×3, 5×5, 7×7, and 9×9, respectively) and a transposed attention mechanism, combining local feature capture and long-range dependency modeling capabilities. The expression for its feature decomposition process is as follows: in, For a Restromer-based decoder, The basic feature is 64 channels. For detailed features, the number of channels is 32. This is one of the source images in a multimodal source image pair.

[0031] S2. Input the basic features of the multimodal source image pairs into the MSG module for semantic fusion to obtain the fused basic features. The MSG module is used to maintain semantic consistency across different object scales; for example... Figure 2 As shown in the dashed box, the MSG module first presets cluster centers, constructs semantic graph nodes through a differentiable superpixel sampling algorithm, then uses graph attention networks to perform feature interactions at multiple scales, and finally outputs the weighted aggregation through a gating network. This module is the core component of this embodiment that breaks through the limitations of traditional fixed grid segmentation.

[0032] S21. Differentiable superpixel sampling algorithm: The differentiable superpixel sampling algorithm dynamically allocates pixel-level bimodal basic features to preset cluster centers for the Superpixel Sampling Network (SSN), establishes a soft association relationship between pixels and cluster centers, and aggregates to form semantically consistent superpixel graph nodes. This process transforms the processing unit from rectangular blocks to semantic regions, ensuring that object boundaries are respected during the aggregation process and avoiding the fragmentation of semantic boundaries of irregular objects by traditional fixed grid segmentation. SSN superpixel segmentation can preserve continuous structures such as tree branches, while fixed-size block segmentation will produce jagged artifacts.

[0033] SSN transforms traditional non-differentiable superpixel segmentation into differentiable soft association assignment, adapting to end-to-end training; the defined set of cluster centers is... ; Soft association is defined as: ; in, The number of channels representing the basic features is 64. Indicates the number of superpixels. Represents pixels With cluster center Association weights, Represents the normalization function and soft interconnection graph. The dimension is Dual-modal basic characteristics The dimension is , Indicates the height of the image. Indicates the width of the image. Represents pixels eigenvectors, Indicates the first The feature vectors of the cluster centers The value range is 1 to , The feature Euclidean distance between a pixel and the cluster center is represented. This represents the temperature parameter, with an empirical value of 0.1, used to adjust the smoothness of the distribution of the association weights; Superpixel map node features through Aggregation in, Indicates by the first The first cluster centers are aggregated to form the... The feature vectors of each superpixel graph node are aggregated in this way, which allows pixels of the same object to be grouped into a single node, ensuring semantic consistency.

[0034] S22. Construct at least two graph scales with different numbers of superpixel nodes. At each graph scale, use a Graph Attention Network (GAT) to model the relationships between superpixel graph nodes of different modalities, achieving cross-modal complementary information interaction. This step adapts to scale differences from small targets to large environmental structures (e.g., ...) through a multi-scale graph topology pyramid. Figure 3 As shown, the infrared node acts as the query node to obtain complementary texture information from the visible node, and the residual connection can maintain the target intensity.

[0035] The image scale has three levels, corresponding to coarse-grained, medium-grained, and fine-grained, with the number of superpixels decreasing accordingly. , , Each is adapted to capture target features of different sizes; Graph attention networks achieve bidirectional interaction between dual-modal superpixel nodes through a cross-attention mechanism. This means each infrared node can obtain relevant contextual information from all visible nodes, and vice versa. The fused node features after interaction are projected back into the pixel space through a transposed soft relational graph, satisfying the following formula: ; in, To use graph attention networks Infrared modal superpixel node features With visible light modal superpixel node features The fused node features obtained after processing. It contains two attention heads, each with a hidden layer dimension of 64. Represents a soft relational graph The transpose of the matrix, This represents the blending feature projected back into pixel space.

[0036] S23. Adaptive weight allocation is performed on the interactive features of each graph scale through a gating network, and the weighted aggregation is used to obtain the fused basic features.

[0037] The gated network consists of 3×3 convolutional layers, LeakyReLU activation function, and 1×1 convolutional layers, with the specific architecture as follows: Figure 4 As shown; The spatial weighted graph is generated using the following formula: ; in, This represents the spatial weighted graph corresponding to the three graph scales. Indicates the height of the image. This represents the width of the image; both the width and height are 256. The mapping function representing the gated network, This indicates a feature concatenation operation. , , These represent the interactive features at the coarse-grained, medium-grained, and fine-grained scales, respectively. This represents the normalization function, used to map weight values ​​to the [0,1] interval; The formula for weighted aggregation is as follows: ; in, Represents element-wise product. Indicates the first Spatial weight map corresponding to each scale Indicates the first Features after interaction at each scale, This represents the basic characteristics after fusion.

[0038] S3. The detailed features of the multimodal source image pair are fused to obtain the initial detailed fused features. At the same time, high-frequency residual information is extracted from the source image and injected into the fused basic features or the initial fused detailed features after brightness adaptive adjustment to obtain enhanced fused detailed features. This step solves the low-pass filtering effect (high-frequency detail attenuation) in the image aggregation-projection process and suppresses overexposure artifacts in the infrared highlight region.

[0039] Detailed features are fused using two lightweight CNN layers. The CNN layers have 3×3 kernels and 32 output channels, with ReLU activation to prioritize local texture fidelity. High-frequency residual information is extracted using a 3×3 pooling difference operator, employing a maximum selection strategy to preserve significant texture features and avoid introducing background noise. After normalization using the tanh function, the residual information is injected, satisfying the following formula: ; in, This indicates the enhanced fundamental features after injecting high-frequency residuals. Indicates the basic characteristics after fusion. This represents a nonlinear activation function used to normalize the injected residuals to the interval [-1, 1]. This represents the maximum selection function. Represents the high-frequency residuals of infrared images. Represents the high-frequency residuals of a visible light image; Brightness adaptive adjustment is achieved through spatial attention masking, satisfying the following formula: ; in, This indicates the basic characteristics after brightness adjustment. This represents the spatial attention mask, with a value range of (0,1), used to adaptively suppress highlight areas, preserve dark area information, and avoid overexposure in the fusion result.

[0040] S4. The fused basic features obtained in S2 and the fused detailed features obtained in S3 are concatenated and input into the decoder for reconstruction to obtain the final fused image.

[0041] The decoder is a Restormer-based decoder, symmetrical in structure to the encoder, containing 6 Restormer blocks, and possesses efficient feature reconstruction capabilities. The number of feature channels after concatenation is 64 + 32 = 96, and the image reconstruction satisfies the following formula: ; in, This indicates a Restromer-based decoder. Indicates the basic characteristics after fusion. This indicates the detailed features after fusion. This represents the final fused image, a reconstruction process that preserves both semantic structural integrity and fine texture details.

[0042] To fully verify the superiority, generalization ability, and improvement effect on downstream tasks of the SMG-Fusion method proposed in this embodiment in maintaining structural integrity and enhancing texture details, a systematic simulation experiment was conducted. The following describes the experimental setup, comparison results, downstream applications, and ablation research in detail with reference to the attached figures and tables.

[0043] The experiments were conducted using the PyTorch framework on a single NVIDIA Tesla V100 GPU. The experimental datasets covered three general infrared-visible image datasets (MSRS, RoadScene, and TNO) and three medical image modalities (MRI-CT, MRI-PET, and MRI-spect). The MSRS dataset contained 1444 image pairs (1083 training pairs and 361 testing pairs), RoadScene used 50 pairs of images with complex lighting scenes, and TNO used 25 commonly paired images. The medical datasets were used to validate the generalization ability of the method. Six industry-standard metrics were used for evaluation: entropy (EN), standard deviation (SD), spatial frequency (SF), visual information fidelity (VIF), average gradient (AG), and edge retention rate (…). The higher the value, the better the fusion quality.

[0044] Comparative experiments selected eight state-of-the-art methods (U2Fusion, TGF, SwinFusion, MetaFusion, Diff-IF, Conti-Fuse, GIFNet, and CDDFuse) as benchmarks. Quantitative results show that the method in this embodiment performs best on key metrics across all datasets. On the MSRS dataset, specific data are shown in Table 1. MetaFusion has the highest AG (6.76) and SF (19.33) values, but its first moment (MI=0.45), VIF (0.05), and... The (0.08) index is significantly lower than other methods, indicating poor feature fusion balance and a problem of modal information imbalance; while SMG-Fusion has a VIF (1.00). It is optimal at (0.73) and MI (3.69), and AG (4.25) and SF (13.09) remain at a high level, which confirms that it can maximize the transmission of complementary information while taking into account structural integrity and texture richness.

[0045] Table 1 Quantitative comparison results of MSRS datasets

[0046] MetaFusion had the highest AG (7.30) and SF (18.28) values, but its first moment (MI=0.91), VIF (0.05), and... The (0.14) index is much lower than other methods, indicating a serious problem of modal information imbalance; while SMG-Fusion has a VIF of (0.77). (0.62) ranked first, while AG (6.00) and SF (16.39) maintained the best level, effectively suppressing glare artifacts and preserving fine filament details, with better overall performance.

[0047] Table 2 Quantitative comparison results of the RoadScene dataset

[0048] On the TNO dataset, the specific data is shown in Table 3. MetaFusion has the highest AG (8.20) and SF (19.64) values, but its first moment (MI=0.50), VIF (0.04), and... (0.13) The index is significantly lagging behind, and the modal fusion balance is extremely poor; while SMG-Fusion is at VIF (0.83). It performs best in 0.61 and MI (2.68), and maintains a high level in AG (5.01). It can still stably retain the thermal target structure in low light environment and has better overall performance.

[0049] Table 3 Quantitative comparison results of the TNO dataset

[0050] Qualitative comparison further verifies (e.g.) Figure 5 Visual comparison of the TNO dataset shown. Figure 6 Visual comparison of the RoadScene dataset shown. Figure 7 (As shown in the visual comparison of the MSRS dataset), the method in this embodiment effectively avoids the boundary fragmentation and texture blurring problems of traditional methods, highlighting thermal targets while maintaining clear background textures in infrared-visible light fusion. Furthermore, based on comprehensive evaluation using radar images (such as...), Figure 8 As shown in the figure, the method of this embodiment forms the largest and most regular closed region in six indicators: SSIM, MS-SSIM, CC, SD, SCD, and FMI, confirming a robust balance between detail injection and structural fidelity.

[0051] To verify the strong generalization ability of SMG-Fusion, we directly applied the model trained on the infrared-visible light dataset to a medical image fusion task (without any fine-tuning). For this scenario, we selected representative benchmark methods in the field of medical image fusion (TarDAL, RFNet, DeFusion, ReCoNet) for comparison, while retaining the cross-scenario universal baseline method CDDFuse to ensure fairness and industry consistency in the comparison. Experiments were conducted on three modalities: MRI-CT, MRI-PET, and MRI-spect. The visual comparison results are shown below. Figure 9 As shown, SMG-Fusion can accurately integrate anatomical structure and functional metabolic information; quantitative results show that it ranks first in all core indicators, as detailed below: In MRI-CT fusion, SF=37.11 =0.64, and the specific data are shown in Table 4. This is significantly better than other methods, confirming that the detail injection mechanism can effectively transmit high-frequency bone edges and tissue textures.

[0052] Table 4 Quantitative results of MRI-CT

[0053] In MRI-PET fusion, SMG-Fusion had an MI of 2.09 and a VIF of 0.75. The VIF score was 13% higher than that of CDDFuse, and it was closest to human visual perception. Specific data are shown in Table 5.

[0054] Table 5 Quantitative results of MRI-PET

[0055] In MRI-spectral fusion, MI=2.03. =0.76, and the specific data are shown in Table 6. All of them are optimal, realizing the precise integration of anatomical structure and functional metabolic information.

[0056] Table 6 Quantitative results of MRI-spect

[0057] Downstream target detection application experiments used the YOLOv9-c detector. The performance improvement of fused images on the M3FD dataset was evaluated. The comparison method continued the core benchmark methods for general infrared-visible light fusion scenarios (Diff-IF, GIFNet, CDDFuse, Conti-Fuse). Quantitative results are shown in Table 7. Diff-IF had the highest mAP@0.5=0.8932, but its accuracy was only 0.8947. CDDFuse showed outstanding performance with an accuracy of 0.9121 and mAP@0.5:0.95=0.6257. SMG-Fusion achieved a detection accuracy of 91.45%, an improvement of 1.98% over Diff-IF and 0.24% over CDDFuse, with mAP@0.5:0.95=0.6248, placing it at the same excellent level as CDDFuse and Diff-IF. Detection performance for each category (e.g.) Figure 10 As shown in the figure, it performs well in categories of rigid objects such as automobiles and lamps, and small targets; qualitative comparisons (such as...) show that it performs well in categories of rigid objects such as automobiles and lamps, and small targets; Figure 11 As shown in the figure, in complex scenes such as smoke and night, Diff-IF has the problem of missing detection, Conti-Fuse is prone to false detection, and GIFNet is not good at recognizing distant targets. SMG-Fusion can effectively avoid these problems, successfully detecting people only in smoke scenes and accurately recognizing all vehicles in night scenes. This proves that the semantic consistency of fused images significantly improves the reliability of downstream perception tasks.

[0058] Table 7. Object detection performance on the YOLOv9-C dataset.

[0059] To validate the contribution of the core modules, three model variants were designed for ablation studies: removal of the MSG module (replacing it with a standard convolutional layer), removal of the SSN module (replacing it with a 16×16 fixed patch), and removal of the injection of details branch. Quantitative results of the ablation experiments are shown in Table 8. This study focuses on validating the effectiveness of the internal modules of the model and does not involve external comparison methods. Compared with the complete model, the "SSN-free" variant... The MI dropped from 2.68 to 1.95, confirming the crucial role of superpixel segmentation in preserving semantic boundaries; the AG of the "no injection of details" variant plummeted from 5.01 to 4.23, and the SF dropped from 13.57 to 11.89, verifying that the high-frequency injection mechanism can effectively offset the low-pass filtering effect of graph networks; the VIF of the "no MSG" variant dropped from 0.83 to 0.65, indicating the importance of multi-scale graph reasoning in capturing global complementary information.

[0060] Table 8. Quantitative ablation analysis results of the TNO dataset.

[0061] Qualitative analysis and visualization further validated the effectiveness of the core modules: such as Figure 12 The visualization of MoE gating weights, with colors ranging from 0 (blue) to 1 (red) representing confidence weights, clearly demonstrates the semantic perception division of labor learned by the gating network: coarse-grained scale (Scale 1) focuses on homogeneous background areas such as the sky and roads, medium-grained scale (Scale 2) focuses on the main structures, and fine-grained scale (Scale 3) specifically highlights high-frequency details, verifying that the MoE strategy can dynamically select the optimal receptive field for different image regions; as shown in the image. Figure 13 The TNO dataset visual ablation matrix shown demonstrates that the complete model produces sharper edges for hot targets in smoke scenes, and better preserves heat source boundaries and grating structures; as shown... Figure 14 The comparison of superpixel topology advantages shows that SSN superpixel segmentation can preserve continuous structures such as tree branches, while fixed-size block segmentation will produce jagged artifacts.

[0062] In summary, simulation experiments fully validated the technical advantages of the SMG-Fusion method in this embodiment from five dimensions: quantitative indicators, visual effects, generalization ability, downstream tasks, and module effectiveness. Although MetaFusion achieves high scores on individual indicators such as AG and SF in some datasets, it suffers from severe modal information imbalance, and its fusion quality does not meet the requirements of practical applications; while SMG-Fusion achieves high scores on core indicators (VIF, SF, SF) in other aspects. It maintains the best performance in both structural integrity preservation and texture detail enhancement, and has a balanced overall performance. It outperforms existing methods in terms of structural integrity preservation, texture detail enhancement, and multi-scene adaptation, and has practical application value.

[0063] Therefore, the present invention adopts the above-mentioned multimodal image fusion method based on superpixel and multi-scale image fusion, which effectively improves the structural integrity and detail fidelity of the fused image and shows better performance in downstream target detection tasks.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A multimodal image fusion method based on superpixel and multi-scale map fusion, characterized in that, Includes the following steps: S1. Obtain the registered multimodal source image pair. The multimodal source image pair consists of two image pairs with different modalities. The features of each source image are decoupled into basic features and detail features by a dual-branch encoder. The basic features contain low-frequency semantic structure information, and the detail features contain high-frequency texture edge information. S2. Input the basic features of the multimodal source image pairs into the multi-scale graph fusion module for semantic fusion to obtain the fused basic features. The multi-scale graph fusion module first presets cluster centers, constructs semantic graph nodes through a differentiable superpixel sampling algorithm, then uses graph attention network to perform feature interaction at multiple scales, and finally outputs the weighted aggregation through a gating network. S3. The detail features of the multimodal source image pair are fused to obtain the initial detail fusion features. At the same time, high-frequency residual information is extracted from the source image and injected into the fused basic features or the initial fused detail features after brightness adaptive adjustment to obtain enhanced fused detail features. S4. The fused basic features obtained in S2 and the fused detailed features obtained in S3 are concatenated and input into the decoder for reconstruction to obtain the final fused image.

2. The multimodal image fusion method based on superpixel and multi-scale map fusion according to claim 1, characterized in that: The multimodal source image pairs are infrared-visible light image pairs, MRI-CT image pairs, MRI-PET image pairs, or MRI-spect image pairs.

3. The multimodal image fusion method based on superpixel and multi-scale map fusion according to claim 1, characterized in that, The dual-branch decoder in S1 is a Restromer-based decoder, and the expression for the feature decomposition process is as follows: in, For a Restromer-based decoder, Basic features, For detailed features, This is one of the source images in a multimodal source image pair. Infrared image, This is a visible light image.

4. The multimodal image fusion method based on superpixel and multi-scale map fusion according to claim 1, characterized in that, The specific steps of S2 include: S21. Through a differentiable superpixel sampling algorithm, pixel-level dual-modal basic features are dynamically allocated to preset cluster centers, establishing a soft association relationship between pixels and cluster centers, and aggregating to form semantically consistent superpixel graph nodes. S22. Construct at least two graph scales with different numbers of superpixel nodes. At each graph scale, use a graph attention network to model the association between superpixel graph nodes of different modalities to achieve cross-modal complementary information interaction. S23. Adaptive weight allocation is performed on the interactive features of each graph scale through a gating network, and the weighted aggregation is used to obtain the fused basic features.

5. The multimodal image fusion method based on superpixel and multi-scale map fusion according to claim 4, characterized in that: The differentiable superpixel sampling algorithm in S21 is a superpixel sampling network; the defined set of cluster centers is... ; Soft association is defined as: ; in, The number of channels representing the basic features. Indicates the number of superpixels. Represents pixels With cluster center Association weights Represents the normalization function and soft interconnection graph. The dimension is Dual-modal basic characteristics The dimension is , Indicates the height of the image. Indicates the width of the image. Represents pixels eigenvectors, Indicates the first Feature vectors of cluster centers The feature Euclidean distance between a pixel and the cluster center is represented. Indicates temperature parameter; Superpixel map node features through Aggregation in, Indicates by the first The first cluster centers are aggregated to form the... The feature vector of each superpixel graph node.

6. The multimodal image fusion method based on superpixel and multi-scale map fusion according to claim 4, characterized in that: The S22 image scale has three levels, corresponding to coarse-grained, medium-grained, and fine-grained, with the number of superpixels decreasing accordingly. , , ; Graph attention networks achieve bidirectional interaction between dual-modal superpixel nodes through a cross-attention mechanism. The fused node features after interaction are projected back into the pixel space through a transposed soft relational graph, satisfying the following formula: ； in, To use graph attention networks Infrared modal superpixel node features With visible light modal superpixel node features The fused node features obtained after processing. Represents a soft relational graph The transpose of the matrix, This represents the blending feature projected back into pixel space.

7. The multimodal image fusion method based on superpixel and multi-scale map fusion according to claim 4, characterized in that: The gated network in S23 consists of 3×3 convolutional layers, LeakyReLU activation function and 1×1 convolutional layers; The spatial weighted graph is generated using the following formula: ； in, This represents the spatial weighted graph corresponding to the three graph scales. Indicates the height of the image. Indicates the width of the image. The mapping function representing the gated network, This indicates a feature concatenation operation. , , These represent the interactive features at the coarse-grained, medium-grained, and fine-grained scales, respectively. Represents the normalization function; The formula for weighted aggregation is as follows: ； in, Represents element-wise product. Indicates the first Spatial weight map corresponding to each scale Indicates the first Features after interaction at each scale, This represents the basic characteristics after fusion.

8. The multimodal image fusion method based on superpixel and multi-scale map fusion according to claim 1, characterized in that: In S3, detailed features are fused through a lightweight CNN layer, high-frequency residual information is extracted using a pooling difference operator, a maximum selection strategy is used to preserve significant texture features, and the features are injected after normalization by the tanh function, satisfying the following formula: ； in, This indicates the enhanced fundamental features after injecting high-frequency residuals. Indicates the basic characteristics after fusion. Represents a non-linear activation function. This represents the maximum selection function. Represents the high-frequency residuals of infrared images. Represents the high-frequency residuals of a visible light image; Brightness adaptive adjustment is achieved through spatial attention masking, satisfying the following formula: ； in, This indicates the basic characteristics after brightness adjustment. This represents a spatial attention mask.

9. The multimodal image fusion method based on superpixel and multi-scale map fusion according to claim 1, characterized in that, The decoder in S4 is a Restromer-based decoder, and the image reconstruction satisfies the following formula: ； in, This indicates a Restromer-based decoder. Indicates the basic characteristics after fusion. This indicates the detailed features after fusion. This indicates a feature concatenation operation. This represents the final fused image.

10. The multimodal image fusion method based on superpixel and multi-scale map fusion according to claim 1, characterized in that, It also includes a model training step, wherein the model consists of a dual-branch encoder, a multi-scale graph fusion module, and a decoder: The first stage trains the feature decomposition process, enabling the encoder to decouple basic features from detailed features. The second stage trains the fusion and reconstruction process based on the total loss function, which is a weighted combination of structural similarity loss, decomposition constraint loss, gradient loss, and intensity consistency loss, as shown in the following formula: ； in, Represents the total loss function. Represents structural similarity loss. Represents the loss due to the decomposition of constraints. Represents gradient loss, Indicates the loss of strength consistency. , , These represent the weight coefficients for the decomposition constraint loss, gradient loss, and strength consistency loss, respectively. ； ； in, This represents the Sobel gradient operator. This represents the final fused image. Represents the absolute value of the gradient in an infrared image. This represents the absolute value of the gradient in a visible light image. This represents the maximum selection function. Describing the L1 norm, Represents an infrared image. Represents a visible light image.

Images