Multi-modal image fusion method, device and equipment based on double-branch heterogeneous network

Abstract

The invention discloses a multi-modal image fusion method, device and equipment based on a double-branch heterogeneous network, and relates to the technical field of multi-modal image fusion. The multi-modal image fusion method comprises the following steps: acquiring a first modal image and a second modal image to be fused; and extracting a global energy feature of the first modal image through an intensity encoder. The intensity encoder includes a multi-layer convolution module and a channel attention module. And extracting local structure features of the second modal image through a structure encoder. The structure encoder comprises a pyramid convolution module and a space attention module. And inputting the global energy features and the local structure features into a fusion module to obtain fused features. And the fusion module is a Softmax gating feature fusion module, dynamically generates an adaptive fusion weight for each channel and spatial position of each mode through a lightweight score network, and performs weighted fusion. And decoding the fused features through a decoder to obtain a fused image.

Description

Technical Field

[0001] This invention relates to the field of multimodal image fusion technology, and more specifically, to a multimodal image fusion method, apparatus, and device based on a dual-branch heterogeneous network. Background Technology

[0002] Multimodal image fusion technology aims to integrate image information from different sources, such as complementary information from infrared and visible light images, or magnetic resonance imaging (MRI) and positron emission tomography (PET) images. Its goal is to generate a single fused image that is richer in information and easier for human observation or subsequent machine processing. This technology has wide applications in fields such as object detection, semantic segmentation, and medical diagnosis.

[0003] With the development of deep learning technology, image fusion methods based on convolutional neural networks (CNNs), generative adversarial networks (GANs), and Transformers have made significant progress. However, existing fusion methods generally have some technical limitations when processing images of different modalities, resulting in room for improvement in the quality and information retention of the fused images.

[0004] Most existing technical solutions employ a uniform network structure to process images of different modalities, neglecting the fundamental differences in feature distribution between modalities (such as the energy characteristics of infrared images and the texture characteristics of visible light images). This leads to insufficient feature extraction and poor fusion quality. Furthermore, traditional end-to-end training strategies struggle to simultaneously ensure faithful reconstruction of source image features and effective complementarity of cross-modal information. Moreover, existing fusion strategies (such as simple feature addition or concatenation) lack adaptive feature selection and weighting capabilities, failing to achieve fine-grained information integration at the pixel and channel levels. Additionally, generic loss function designs lack specific consideration for the characteristics of different modalities, making it difficult to optimally preserve both texture details and energy saliency in the fusion result, thus limiting the upper limit of the final fusion performance. Summary of the Invention

[0005] The present invention provides a multimodal image fusion method, apparatus and device based on a dual-branch heterogeneous network to improve at least one of the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a multimodal image fusion method based on a dual-branch heterogeneous network, which includes steps S1 to S5.

[0007] S1. Obtain the first modality image and the second modality image to be fused.

[0008] S2. Extract global energy features from the first modality image using an intensity encoder. The intensity encoder includes a multi-layer convolutional module and a channel attention module.

[0009] S3. Extract local structural features from the second modality image using a structural encoder. The structural encoder includes a pyramid convolution module and a spatial attention module.

[0010] S4. Input the global energy features and the local structural features into the fusion module to obtain the fused features. The fusion module is a Softmax-gated feature fusion module, which dynamically generates adaptive fusion weights for each channel and spatial location of each modality through a lightweight scoring network, and performs weighted fusion.

[0011] S5. Decode the fused features using a decoder to obtain the fused image.

[0012] The neural network of the above-mentioned multimodal image fusion method is optimized through a two-stage training strategy. In the first stage, the intensity encoder and the structure encoder are trained independently to faithfully reconstruct the images of their corresponding modalities. In the second stage, the fusion module and the decoder are jointly optimized to generate high-quality fused images.

[0013] As a further aspect of the present invention, the intensity encoder is used to process a first-modality image dominated by energy or radiation information. The core of the intensity encoder consists of three consecutive 3x3 convolutional layers, each followed by a ReLU activation function to form a multi-layer convolutional module for progressively abstracting global energy features. Then, a channel attention module is connected, which generates channel weights through global average pooling and adaptively recalibrates the feature channels, strengthening key channels related to thermal radiation or metabolic activity.

[0014] As a further aspect of the present invention, the structural encoder is used to process a second modality image dominated by texture and detail information. The core of the structural encoder is a pyramidal convolutional module, containing three parallel convolutional branches with kernel sizes of 1x1, 3x3, and 5x5, used to simultaneously capture local texture and global structure at different scales. Then, a spatial attention module is connected, which dynamically highlights detailed regions and structural boundaries in the image by calculating an importance weight map of spatial locations.

[0015] As a further aspect of the present invention, the specific operation of the Softmax gated feature fusion module includes: S41. The global energy features and local structural features are spliced ​​together along the channel dimension.

[0016] S42. The concatenated features are passed through a scoring network, which outputs two corresponding fusion weights (logits) for each spatial location and feature channel. The scoring network contains a 3x3 convolutional layer, a ReLU activation function, and a 1x1 convolutional layer.

[0017] S43. Normalize logits in the modal dimension using Softmax to obtain the spatial and channel adaptive weight maps of global energy features and local structural features.

[0018] S44. Use the obtained weight map to perform element-wise weighting on the original global energy features and local structural features respectively, and add the weighted results to obtain the fused features.

[0019] As a further aspect of the present invention, the decoder consists of multiple convolutional layers and upsampling layers, responsible for reconstructing the image from the encoded features (or fused features). The output layer of the decoder uses the sigmoid activation function to ensure that the output value range is consistent with the input image.

[0020] As a further aspect of the present invention, the intensity encoder and the structure encoder are independently trained to faithfully reconstruct the images of their corresponding modes. Specifically, the first mode image and the second mode image are input into the intensity encoder and the structure encoder, respectively. The extracted features are reconstructed back to the original image by the decoder, and each is independently optimized by a targeted reconstruction loss function to ensure that each encoder can capture and retain the unique characteristics of its corresponding mode.

[0021] Loss function for the first stage of training as follows.

[0022] .

[0023] .

[0024] .

[0025] In the formula The reconstruction loss of the first modality image, For the reconstruction loss of the second modality image, For L1 loss, for Weighting coefficients For total variation loss, for Weighting coefficients For information entropy loss, for Weighting coefficients For structural similarity loss, for Weighting coefficients For mean square error loss, for Weighting coefficients For gradient loss, for The weighting coefficients.

[0026] As a further aspect of the present invention, the fusion module and the decoder are jointly optimized to generate a high-quality fused image. Specifically, the parameters of the pre-trained dual-branch encoder are frozen, the features extracted by the encoder are input into the Softmax gated feature fusion module for fusion, the fused features are then processed by the decoder to generate a fused image, and the fusion module and the decoder are jointly optimized by fusion loss and cross-modal consistency loss.

[0027] Loss function for the second stage of training as follows.

[0028] .

[0029] In the formula This is a loss based on strength consistency.

[0030] As a further embodiment of the present invention, the first modal image is an infrared image, and the second modal image is a visible light image. Alternatively, the first modal image is a PET image or a CT image, and the second modal image is an MRI image.

[0031] Secondly, the present invention provides a multimodal image fusion device based on a dual-branch heterogeneous network, which includes an original image acquisition module, a first encoding module, a second encoding module, a fusion module, and a decoding module.

[0032] The original image acquisition module is used to acquire the first modality image and the second modality image to be fused.

[0033] The first encoding module is used to extract global energy features of the first modality image through an intensity encoder. The intensity encoder includes a multi-layer convolutional module and a channel attention module.

[0034] The second encoding module is used to extract local structural features of the second modality image through a structural encoder. The structural encoder includes a pyramid convolution module and a spatial attention module.

[0035] The fusion module is used to input the global energy features and the local structural features into the fusion module to obtain the fused features. The fusion module is a Softmax-gated feature fusion module, which dynamically generates adaptive fusion weights for each channel and spatial location of each modality through a lightweight scoring network, and performs weighted fusion.

[0036] The decoding module is used to decode the fused features using a decoder to obtain the fused image.

[0037] The neural network of the above-mentioned multimodal image fusion method is optimized through a two-stage training strategy. In the first stage, the intensity encoder and the structure encoder are trained independently to faithfully reconstruct the images of their corresponding modalities. In the second stage, the fusion module and the decoder are jointly optimized to generate high-quality fused images.

[0038] Thirdly, the present invention provides a multimodal image fusion device based on a dual-branch heterogeneous network, comprising a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement a multimodal image fusion method based on a dual-branch heterogeneous network as described in any paragraph of the first aspect.

[0039] Fourthly, the present invention provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform a multimodal image fusion method based on a dual-branch heterogeneous network as described in any paragraph of the first aspect.

[0040] By adopting the above technical solution, the present invention can achieve the following technical effects: The present invention provides a multimodal image fusion method based on a dual-branch heterogeneous network. Through a heterogeneous dual-branch encoder, a two-stage training strategy, and a modality-aware loss function, it achieves differentiated extraction and efficient fusion of features from different modalities, significantly improving the quality of the fused image and the performance of downstream tasks. Attached Figure Description

[0041] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the specific embodiments of the present invention will be briefly introduced below. It should be understood that the following drawings only show some specific embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is the logic diagram for the first stage training of the multimodal image fusion method.

[0043] Figure 2 This is a logic diagram for the second stage training of the multimodal image fusion method.

[0044] Figure 3 This is the qualitative result of the present invention on the RoadScene dataset.

[0045] Figure 4 This is the qualitative result of the present invention on the MSRS dataset.

[0046] Figure 5This is the qualitative result of the present invention on the MRI-PET dataset.

[0047] Figure 6 This is a visualization of the quantitative results of the computational efficiency of this invention.

[0048] Figure 7 This is a visualization of the quantitative results of the present invention in terms of model complexity and inference speed. Detailed Implementation

[0049] The technical solutions of the present invention will now be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention.

[0050] Example 1, please refer to Figures 1 to 7 The first embodiment of the present invention provides a multimodal image fusion method based on a dual-branch heterogeneous network, which can be executed by a multimodal image fusion device (hereinafter referred to as: multimodal image fusion device). In particular, it is executed by one or more processors in the multimodal image fusion device to implement steps S1 to S5.

[0051] S1. Obtain the first modality image and the second modality image to be fused.

[0052] Preferably, the first modal image is an infrared image, and the second modal image is a visible light image. Alternatively, the first modal image is a PET image, CT image, or SPECT image, and the second modal image is an MRI image. Wherein, PET stands for Positron Emission Tomography; CT stands for Computed Tomography; SPECT stands for Single Photon Emission Tomography; and MRI stands for Magnetic Resonance Imaging.

[0053] The neural network construction of the multimodal image fusion method of the present invention consists of a dual-branch heterogeneous encoder composed of an intensity encoder and a structure encoder.

[0054] S2. Extract global energy features from the first modality image using an intensity encoder. The intensity encoder includes a multi-layer convolutional module and a channel attention module.

[0055] Specifically, the intensity encoder is used to process first-modality images (such as infrared images, PET images, and CT images) that are dominated by energy or radiation information. The intensity encoder includes multi-layer convolutional modules and channel attention modules to extract and enhance energy-related global features.

[0056] like Figure 1As shown, the core of the intensity encoder consists of three consecutive 3x3 convolutional layers, each followed by a ReLU activation function to form a multi-layer convolutional module for progressively abstracting global energy features. Then, a channel attention module is connected. This module generates channel weights through global average pooling and adaptively recalibrates the feature channels, strengthening key channels related to thermal radiation or metabolic activity. Specifically, the channel attention mechanism uses global average pooling and sigmoid activation to dynamically adjust the feature channel weights.

[0057] S3. Extract local structural features from the second modality image using a structural encoder. The structural encoder includes a pyramid convolution module and a spatial attention module.

[0058] like Figure 1 As shown, the structural encoder is used to process second-modal images (such as visible light images and MRI images) that are dominated by texture and detail information. The structural encoder includes a pyramid convolution module and a spatial attention module to extract and focus on structure-related local detail features at multiple scales.

[0059] like Figure 1 As shown, the core of the structural encoder is a pyramid convolution module, containing three parallel convolutional branches with kernel sizes of 1x1, 3x3, and 5x5, used to simultaneously capture local textures and global structures at different scales. This is then connected to a spatial attention module, which dynamically highlights detailed regions and structural boundaries in the image by calculating importance weight maps of spatial locations. Specifically, the spatial attention mechanism generates a spatial weight matrix through convolutional mapping to highlight structural regions.

[0060] S4. Input the global energy features and the local structural features into the fusion module to obtain the fused features. The fusion module is a Softmax-gated feature fusion module, which dynamically generates adaptive fusion weights for each channel and spatial location of each modality through a lightweight scoring network, and performs weighted fusion.

[0061] The specific operations of the Softmax gated feature fusion module include: S41. The global energy features and local structural features are spliced ​​together along the channel dimension.

[0062] S42. The concatenated features are passed through a scoring network, which outputs two corresponding fusion weights (logits) for each spatial location and feature channel. The scoring network contains a 3x3 convolutional layer, a ReLU activation function, and a 1x1 convolutional layer.

[0063] S43. Normalize logits in the modal dimension using Softmax to obtain the spatial and channel adaptive weight maps of global energy features and local structural features.

[0064] S44. Use the obtained weight map to perform element-wise weighting on the original global energy features and local structural features respectively, and add the weighted results to obtain the fused features.

[0065] Specifically, such as Figure 2 As shown, the fusion module receives features from the intensity encoder and the structure encoder. First, the bimodal features are concatenated along the channel dimension. Then, the concatenated features are fed into a lightweight scoring network, which consists of a 3x3 convolutional layer, a ReLU activation function, and a 1x1 convolutional layer connected in sequence. The scoring network outputs two corresponding fusion weight logits for each spatial location and feature channel. These logits are softmax normalized along the modal dimension (i.e., between the two modalities), thereby dynamically generating a pair of spatially and channel-adaptive fusion weight maps. Finally, these two weight maps are used to element-wise weight the input original global energy features and local structural features, respectively, and the weighted results are summed to output the fused features. This module achieves pixel-level and channel-level fine-grained fusion.

[0066] S5. Output the fused image result through the decoder.

[0067] The decoder consists of multiple convolutional layers and upsampling layers, responsible for reconstructing the image from the encoded features (or fused features). The decoder's output layer uses the sigmoid activation function to ensure that the output value range is consistent with the input image.

[0068] Based on the above embodiments, in an optional embodiment of the present invention, the neural network of the above multimodal image fusion method is optimized through a two-stage training strategy. In the first stage, the intensity encoder and the structure encoder are trained independently to faithfully reconstruct the images of their corresponding modalities. In the second stage, the fusion module and the decoder are jointly optimized to generate high-quality fused images.

[0069] The present invention provides a multimodal image fusion method based on a dual-branch heterogeneous network. Through a heterogeneous dual-branch encoder, a two-stage training strategy, and a modality-aware loss function, it achieves differentiated extraction and efficient fusion of features from different modalities, significantly improving the quality of the fused image and the performance of downstream tasks.

[0070] The intensity encoder and structure encoder are trained independently to faithfully reconstruct images of their corresponding modalities. Specifically, the first modal image and the second modal image are input into the intensity encoder and the structure encoder, respectively. The extracted features are reconstructed back into the original image by the decoder, and each is independently optimized by a targeted reconstruction loss function to ensure that each encoder can capture and retain the unique characteristics of its corresponding modality.

[0071] like Figure 1 As shown, a large number of paired source images (such as { , Training is then performed to extract features. The feature extraction process is described below.

[0072] .

[0073] .

[0074] In the formula For infrared light images, For visible light images, For multi-layer convolution, For pyramid convolution, For channel attention, For spatial attention, for Corresponding extracted features, for Corresponding extracted features .

[0075] Loss function for the first stage of training as follows.

[0076] .

[0077] .

[0078] .

[0079] In the formula The reconstruction loss of the first modality image, For the reconstruction loss of the second modality image, For L1 loss, for Weighting coefficients For total variation loss, for Weighting coefficients For information entropy loss, for Weighting coefficients For structural similarity loss, for Weighting coefficients For mean square error loss, for Weighting coefficients For gradient loss, for The weighting coefficients.

[0080] The reconstruction loss for the first modality image combines L1 loss, total variation loss, and information entropy loss to ensure the accuracy and smoothness of intensity information while avoiding over-smoothing. The reconstruction loss for the second modality image combines structural similarity loss, mean squared error loss, and gradient loss to ensure faithful preservation of texture and structural details.

[0081] The fusion module and the decoder are jointly optimized to generate a high-quality fused image. Specifically, the parameters of the pre-trained dual-branch encoder are frozen, the features extracted by the encoder are input into the Softmax gated feature fusion module for fusion, the fused features are then processed by the decoder to generate a fused image, and the fusion module and the decoder are jointly optimized by fusion loss and cross-modal consistency loss.

[0082] like Figure 2 As shown, load the encoder weights from the first stage of pre-training and freeze them. { , The input encoder yields the features. , The input is fed into the Softmax gated feature fusion module for fusion to obtain the fused features. Finally, a fused image is generated using a decoder. .

[0083] .

[0084] In the formula This indicates the Softmax gated feature fusion module.

[0085] Loss function for the second stage of training as follows.

[0086] .

[0087] In the formula For loss based on strength consistency, For gradient loss, for The weighting coefficients.

[0088] The second stage employs a fusion loss function, which guides the fusion process by comparing the salient intensity patterns and gradient information between the fused image and the source image.

[0089] The following example demonstrates the effectiveness of the multimodal image fusion method based on a dual-branch heterogeneous network according to the present invention.

[0090] Taking infrared-visible image fusion as an example, experiments were conducted on three publicly available datasets that can be downloaded and used for free online: MSRS, RoadScene, and TNO. The MSRS dataset (a multispectral road scene infrared-visible paired dataset built based on MFNet) is available for download on GitHub. The RoadScene dataset, provided by Ma Jiayi et al.'s team on GitHub, consists of road-scene infrared-visible images used for fusion, registration, and other tasks. The TNO benchmark dataset, released by the Dutch research institution TNO, can be downloaded from Figshare or the relevant paper page.

[0091] The model training parameters are set as follows: image patch size 128x128, total number of training rounds 120 (40 in the first stage and 80 in the second stage), batch size 8, using the Adam optimizer, initial learning rate 1e-4 and decaying by 0.5 times every 20 rounds.

[0092] Figure 3 and Figure 4 Qualitative comparisons on the RoadScene and MSRS datasets are presented respectively. Clearly, this invention better integrates thermal radiation information from infrared images with detailed texture from visible images. Objects in dark areas are clearly highlighted, allowing for easy differentiation between foreground targets and background. Furthermore, background details that are difficult to discern due to low illumination exhibit sharp edges and rich contour information, which helps us better understand the scene.

[0093] Table 1. Quantitative comparisons on the MSRS infrared-visible fusion dataset.

[0094]

[0095] Table 2 Quantitative comparisons on the TNO infrared-visible fusion dataset.

[0096]

[0097] Table 3 Quantitative comparisons on the RoadScene infrared-visible fusion dataset.

[0098]

[0099] Tables 1 through 3 present quantitative comparisons on the MSRS, TNO, and RoadScene datasets, respectively. Our invention achieves the highest or near-highest scores on almost all metrics, consistently outperforming most recent methods. It is noteworthy that some competing models may optimize for specific metrics (e.g., SSIM or Qabf) but often sacrifice complementary properties such as entropy (EN) or mutual information (MI), resulting in reduced information content in the fused output. In contrast, DBHFuse demonstrates a good balance across all evaluation metrics. On each dataset, it ranks first in more than five metrics and remains competitive on the remaining ones. This consistency highlights the robustness and generality of our method, effectively preserving structural detail, semantic consistency, and perceptual quality simultaneously.

[0100] Table 4. Quantitative analysis of MRI-CT medical image fusion.

[0101]

[0102] Table 5. Quantitative analysis of MRI-PET medical image fusion.

[0103]

[0104] Table 6 - Quantitative analysis of SPECT medical image fusion.

[0105]

[0106] Figure 5 Qualitative analysis on the Harvard Medical Image Dataset is presented. Tables 4 through 6 show the quantitative analysis on the Harvard Medical Image Dataset. This invention preserves detailed texture and highlights structural information, and achieves good performance on almost all metrics.

[0107] The Harvard Medical Image Dataset refers to publicly available brain medical image data such as Whole BrainAtlas / AANLIB provided by Harvard Medical School; it is a publicly accessible dataset.

[0108] Figure 6 and Figure 7 A visualization of the quantitative analysis of operational efficiency is presented. It can be seen that this invention achieves an excellent balance between computational cost and inference speed.

[0109] Specifically, this invention requires only 4.12 G FLOPs and 0.25 M parameters, the lowest among all comparison models, while achieving 598 FPS on a single RTX 3090 GPU. In contrast, methods such as RFNet and DeFusion consume billions of FLOPs due to their complex multi-stage architecture, resulting in significantly reduced inference speed. Although U2Fusion and DIDFuse exhibit lightweight architectures, their inference speeds are still far lower than this invention, indicating that the proposed dual-branch heterogeneous design maintains efficiency without sacrificing fusion quality. These results demonstrate that this invention achieves a better trade-off between accuracy and computational efficiency, making it more suitable for real-time infrared-visible imaging or medical fusion applications.

[0110] The image fusion method used for comparison in a specific case is as follows.

[0111] A deep image decomposition method for fusion of infrared and visible light images (DIDFuse).

[0112] U2Fusion (U2Fusion: A Unified Unsupervised Image Fusion Network).

[0113] SDNet: A Versatile Squeeze-and-Decomposition Network for Real-Time Image Fusion.

[0114] RFNet: Unsupervised Network for Mutually Reinforcing Multi-Modal Image Registration and Fusion.

[0115] A correlation-driven dual-branch feature decomposition network for multi-modality image fusion (CDDFuse).

[0116] A self-supervised decomposition-based image fusion method (DeFusion).

[0117] Target-aware Dual Adversarial Learning (TarD) is a method for image fusion based on target awareness.

[0118] Edge-preserving Multi-scale Fusion (EMF) method.

[0119] The table below shows the quantitative evaluation criteria used to measure the quality and performance of image fusion algorithms.

[0120] EN: Entropy - A measure of the richness of information in a fused image.

[0121] SD: Standard Deviation - measures the degree of dispersion in the contrast and information distribution of a fused image.

[0122] SF: Spatial Frequency - Measures the texture detail and sharpness of a fused image.

[0123] MI: Mutual Information - measures the amount of information shared between the fused image and the source image.

[0124] SCD: Sum of the Correlations of Differences - measures the ability of a fused image to retain structural information from the source images.

[0125] VIF: Visual Information Fidelity - measures the fidelity of fused images to the human visual system.

[0126] Qbaf: Quality index based on an artifact measure - measures the degree of distortion or artifacts introduced into the fusion result.

[0127] SSIM: Structural Similarity Index Measure - measures the similarity between the fused image and the source image in terms of structure, brightness, and contrast.

[0128] Example 2: The present invention provides a multimodal image fusion device based on a dual-branch heterogeneous network, which includes an original image acquisition module, a first encoding module, a second encoding module, a fusion module and a decoding module.

[0129] The original image acquisition module is used to acquire the first modality image and the second modality image to be fused.

[0130] The first encoding module is used to extract global energy features of the first modality image through an intensity encoder. The intensity encoder includes a multi-layer convolutional module and a channel attention module.

[0131] The second encoding module is used to extract local structural features of the second modality image through a structural encoder. The structural encoder includes a pyramid convolution module and a spatial attention module.

[0132] The fusion module is used to input the global energy features and the local structural features into the fusion module to obtain the fused features. The fusion module is a Softmax-gated feature fusion module, which dynamically generates adaptive fusion weights for each channel and spatial location of each modality through a lightweight scoring network, and performs weighted fusion.

[0133] The decoding module is used to decode the fused features using a decoder to obtain the fused image.

[0134] The neural network of the above-mentioned multimodal image fusion method is optimized through a two-stage training strategy. In the first stage, the intensity encoder and the structure encoder are trained independently to faithfully reconstruct the images of their corresponding modalities. In the second stage, the fusion module and the decoder are jointly optimized to generate high-quality fused images.

[0135] Example 3: This invention provides a multimodal image fusion device based on a dual-branch heterogeneous network, comprising a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement a multimodal image fusion method based on a dual-branch heterogeneous network as described in any paragraph of Example 1.

[0136] It is understood that the multimodal image fusion device can be an electronic device with computing power, such as a portable laptop computer, desktop computer, server, smartphone or tablet computer.

[0137] Example 4: This invention provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform a multimodal image fusion method based on a dual-branch heterogeneous network as described in any paragraph of Example 1.

[0138] Obviously, the embodiments described above are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0139] In the several embodiments provided in this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0140] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0141] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0142] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0143] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0144] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0145] The terms "first" and "second" used in the embodiments are merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permitted. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0146] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-modal image fusion method based on a dual-branch heterogeneous network, characterized in that, The method comprises the steps of: S1, obtaining a first modality image dominated by energy or radiation information and a second modality image dominated by texture and detail information to be fused; S2, extracting global energy features of the first modality image by an intensity encoder; The intensity encoder comprises a multi-layer convolution module and a channel attention module; S3, extracting local structure features of the second modality image by a structure encoder; the structure encoder comprises a pyramid convolution module and a spatial attention module; S4, inputting the global energy features and the local structure features into a fusion module to obtain fused features; The fusion module is a Softmax gated feature fusion module, which dynamically generates adaptive fusion weights for each channel and spatial position of each modality through a lightweight scoring network, and performs weighted fusion; S5, decoding the fused features by a decoder to obtain a fused image; The neural network of the above multi-modal image fusion method is optimized by a two-stage training strategy, the intensity encoder and the structure encoder are independently trained in the first stage to faithfully reconstruct the images of the corresponding modalities, and the fusion module and the decoder are jointly optimized in the second stage to generate high-quality fused images.

2. The multi-modal image fusion method based on a dual-branch heterogeneous network according to claim 1, characterized in that, The core of the intensity encoder is three consecutive 3x3 convolution layers, each convolution layer is followed by a ReLU activation function to form a multi-layer convolution module for progressively abstracting global energy features; Then, a channel attention module is connected, the channel attention module generates channel weights through global average pooling, adaptively recalibrates the feature channels, and strengthens the key channels related to thermal radiation or metabolic activity. 3.The multi-modal image fusion method based on a dual-branch heterogeneous network according to claim 1, characterized in that, The core of the structure encoder is a pyramid convolution module, which comprises three parallel convolution branches with convolution kernel sizes of 1x1, 3x3 and 5x5, for simultaneously capturing local textures and global structures of different scales; then, a spatial attention module is connected, the spatial attention module dynamically highlights the detail-rich areas and structure boundaries in the image by calculating the importance weight map of the spatial position.

4. The multi-modal image fusion method based on a dual-branch heterogeneous network according to claim 1, characterized in that, The specific operation of the Softmax gated feature fusion module comprises: S41, concatenating the global energy features and the local structure features in the channel dimension; S42, passing the concatenated features through a scoring network, the scoring network outputs two corresponding fusion weights logits for each spatial position and feature channel; the scoring network comprises a 3x3 convolution layer, a ReLU activation function and a 1x1 convolution layer; S43, performing Softmax normalization on the logits in the modality dimension to obtain spatial and channel adaptive weight maps of the global energy features and the local structure features respectively; S44, using the obtained weight maps to element-wise weight the original global energy features and local structure features respectively, and adding the weighted results to obtain the fused features.

5. The multi-modal image fusion method based on a dual-branch heterogeneous network according to any one of claims 1 to 4, characterized in that, The decoder is composed of multiple convolution layers and up-sampling layers, and is responsible for reconstructing the encoded features (or fused features) into images; the output layer of the decoder uses a Sigmoid activation function to ensure that the output value range is consistent with the input image.

6. The multi-modal image fusion method based on a dual-branch heterogeneous network according to any one of claims 1 to 4, characterized in that, The intensity encoder and the structure encoder are independently trained to faithfully reconstruct images of their corresponding modalities, specifically, the first modality image and the second modality image are input into the intensity encoder and the structure encoder respectively, the extracted features are reconstructed back to the original images by the decoder, and independent optimization is performed through a targeted reconstruction loss function to ensure that each encoder can capture and retain the unique characteristics of its corresponding modality; Loss function for first stage training As follows; ； ； ； In the formula The reconstruction loss of the first modality image, For the reconstruction loss of the second modality image, For L1 loss, for Weighting coefficients For total variation loss, for Weighting coefficients For information entropy loss, for Weighting coefficients For structural similarity loss, for Weighting coefficients For mean square error loss, for Weighting coefficients For gradient loss, for Weighting coefficients; The fusion module and the decoder are jointly optimized to generate a high-quality fused image, specifically, the pre-trained dual-branch encoder parameters are frozen, the features extracted by the encoder are input into the Softmax gated feature fusion module for fusion, the fused features are then input into the decoder to generate a fused image, and the fusion module and the decoder are jointly optimized through a fusion loss and a cross-modality consistency loss; Loss function for second stage training As follows; ； In the formula is a loss based on consistency of intensity.

7. The multi-modal image fusion method based on a dual-branch heterogeneous network according to any one of claims 1 to 4, characterized in that, The first modality image is an infrared image, and the second modality image is a visible light image; or the first modality image is a PET image or a CT image, and the second modality image is an MRI image.

8. A multi-modal image fusion device based on a dual-branch heterogeneous network, characterized in that, Comprise: An original image acquisition module configured to acquire a first modality image dominated by energy or radiation information and a second modality image dominated by texture and detail information to be fused; A first encoding module configured to extract global energy features of the first modality image by an intensity encoder; The intensity encoder comprises a plurality of convolution modules and a channel attention module; A second encoding module configured to extract local structure features of the second modality image by a structure encoder; the structure encoder comprises a pyramid convolution module and a spatial attention module; A fusion module configured to input the global energy features and the local structure features into the fusion module to obtain fused features; The fusion module is a Softmax gated feature fusion module, which dynamically generates adaptive fusion weights for each channel and spatial position of each modality through a lightweight scoring network, and performs weighted fusion; A decoding module configured to decode the fused features by a decoder to obtain a fused image; The neural network of the above multi-modal image fusion method is optimized through a two-stage training strategy, the intensity encoder and the structure encoder are independently trained in the first stage to faithfully reconstruct images of their corresponding modalities, and the fusion module and the decoder are jointly optimized in the second stage to generate a high-quality fused image.

9. A multi-modal image fusion device based on a dual-branch heterogeneous network, characterized by, The computer program can be executed by the processor to implement the multi-modal image fusion method based on the dual-branch heterogeneous network according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored computer program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to execute the multi-modal image fusion method based on the dual-branch heterogeneous network according to any one of claims 1 to 7 when the computer program runs.

Images