A multi-modal image fusion method, device and equipment based on a dual-branch heterogeneous network

Abstract

The application discloses a multi-modal image fusion method, device and equipment based on a double-branch heterogeneous network, and relates to the technical field of multi-modal image fusion. The multi-modal image fusion method comprises the following steps: obtaining a first modal image and a second modal image to be fused. A global energy feature of the first modal image is extracted through an intensity encoder. The intensity encoder comprises a multi-layer convolution module and a channel attention module. A local structure feature of the second modal image is extracted through a structure encoder. The structure encoder comprises a pyramid convolution module and a spatial attention module. The global energy feature and the local structure feature are input into a fusion module to obtain a fused feature. The fusion module is a Softmax gate feature fusion module, a lightweight score network is used to dynamically generate adaptive fusion weights for each channel and spatial position of each modal, and weighted fusion is performed. The fused feature is decoded through a decoder to obtain a fused image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal image fusion, in particular to a multi-modal image fusion method, device and equipment based on a dual-branch heterogeneous network. BACKGROUND

[0002] Multi-modal image fusion technology aims to integrate image information from different sources, such as complementary information between infrared images and visible light images, or magnetic resonance imaging (MRI) and positron emission tomography (PET) images. The purpose is to generate a single fusion image with more abundant information, which is more conducive to human observation or subsequent machine processing. This technology has wide application value in target detection, semantic segmentation and medical diagnosis.

[0003] With the development of deep learning technology, image fusion methods based on convolutional neural networks (CNN), generative adversarial networks (GAN) and Transformers have made significant progress. However, existing fusion methods have some technical limitations when processing different modal images, resulting in the need to improve the quality and information retention of the fusion image.

[0004] Most existing technical solutions use a unified network structure to process images of different modalities, ignoring the essential differences in feature distribution between modalities (such as the energy characteristics of infrared images and the texture characteristics of visible light images), resulting in insufficient feature extraction and poor fusion quality. At the same time, traditional end-to-end training strategies are difficult to balance the faithful reconstruction of source image features and the effective complementarity of cross-modal information. In addition, existing fusion strategies (such as simple feature addition or splicing) lack the ability to adaptively select and weight features, and cannot achieve fine-grained information integration at the pixel and channel level. Furthermore, the design of general loss functions lacks targeted consideration of different modal characteristics, making it difficult to optimally retain texture details and energy saliency in the fusion results, limiting the upper limit of the final fusion performance. SUMMARY

[0005] The present application provides a multi-modal image fusion method, device and equipment based on a dual-branch heterogeneous network to improve at least one of the above technical problems.

[0006] In a first aspect, the present application provides a multi-modal image fusion method based on a dual-branch heterogeneous network, which comprises steps S1 to S5.

[0007] S1, obtaining a first modal image and a second modal image to be fused.

[0008] S2, extracting global energy features of the first modal image through an intensity encoder. The intensity encoder comprises a multi-layer convolution module and a channel attention module.

[0009] S3, extracting local structure features of the second modality image by a structure encoder. The structure encoder comprises a pyramid convolution module and a spatial attention module.

[0010] S4, inputting the global energy features and the local structure features into a fusion module to obtain fused features. The fusion module is a Softmax gated feature fusion module, which dynamically generates adaptive fusion weights for each channel and spatial position of each modality by a lightweight scoring network, and performs weighted fusion.

[0011] S5, decoding the fused features by a decoder to obtain a fused image.

[0012] The neural network of the above multi-modal image fusion method is optimized by a two-stage training strategy. In the first stage, the intensity encoder and the structure encoder are independently trained to faithfully reconstruct the images of the corresponding modalities. In the second stage, the fusion module and the decoder are jointly optimized to generate high-quality fused images.

[0013] As a further scheme of the present application, the intensity encoder is used to process the first modality image dominated by energy or radiation information. The core of the intensity encoder is three consecutive 3x3 convolution layers, each followed by a ReLU activation function to form a multi-layer convolution module for progressively abstracting global energy features. Then a channel attention module is connected, which generates channel weights by global average pooling to adaptively recalibrate the feature channels and strengthen the key channels related to thermal radiation or metabolic activity.

[0014] As a further scheme of the present application, the structure encoder is used to process the second modality image dominated by texture and detail information. The core of the structure encoder is a pyramid convolution module, which comprises three parallel convolution branches with kernel sizes of 1x1, 3x3 and 5x5, respectively, for simultaneously capturing local textures and global structures at different scales. Then a spatial attention module is connected, which dynamically highlights the detail-rich areas and structure boundaries in the image by calculating the importance weight map of the spatial position.

[0015] As a further scheme of the present application, the specific operation of the Softmax gated feature fusion module includes:

[0016] S41, concatenating the global energy features and the local structure features in the channel dimension.

[0017] S42, passing the concatenated features through a scoring network, which outputs two corresponding fusion weights logits for each spatial position and feature channel. The scoring network comprises a 3x3 convolution layer, a ReLU activation function and a 1x1 convolution layer.

[0018] S43, Softmax normalization is performed on the logits in the modal dimension to obtain spatial and channel adaptive weight maps of the global energy features and the local structure features respectively.

[0019] S44, the obtained weight maps are used to respectively perform element-wise weighting on the original global energy features and the local structure features, and the weighted results are added to obtain the fusion features.

[0020] As a further scheme of the present application, the decoder is composed of multiple convolutional layers and up-sampling layers, and is responsible for reconstructing the encoded features (or fusion features) into images. The output layer of the decoder uses a Sigmoid activation function to ensure that the output value range is consistent with the input image.

[0021] As a further scheme of the present application, the intensity encoder and the structure encoder are independently trained to faithfully reconstruct the images of their corresponding modalities. Specifically, the first modality image and the second modality image are input into the intensity encoder and the structure encoder respectively, the extracted features are reconstructed back to the original image through the decoder, and independent optimization is performed through the corresponding reconstruction loss function to ensure that each encoder can capture and retain the unique characteristics of its corresponding modality.

[0022] Loss function of the first stage training As follows.

[0023] .

[0024] .

[0025] .

[0026] wherein is the reconstruction loss of the first modality image, is the reconstruction loss of the second modality image, is the L1 loss, is the weight coefficient of is the total variation loss, is the weight coefficient of is the information entropy loss, is the weight coefficient of is the structural similarity loss, is the weight coefficient of is the mean square error loss, is the weight coefficient of is the gradient loss, is The weighting coefficients.

[0027] As a further aspect of the present invention, the fusion module and the decoder are jointly optimized to generate a high-quality fused image. Specifically, the parameters of the pre-trained dual-branch encoder are frozen, the features extracted by the encoder are input into the Softmax gated feature fusion module for fusion, the fused features are then processed by the decoder to generate a fused image, and the fusion module and the decoder are jointly optimized by fusion loss and cross-modal consistency loss.

[0028] Loss function for the second stage of training as follows.

[0029] .

[0030] In the formula This is a loss based on strength consistency.

[0031] As a further embodiment of the present invention, the first modal image is an infrared image, and the second modal image is a visible light image. Alternatively, the first modal image is a PET image or a CT image, and the second modal image is an MRI image.

[0032] Secondly, the present invention provides a multimodal image fusion device based on a dual-branch heterogeneous network, which includes an original image acquisition module, a first encoding module, a second encoding module, a fusion module, and a decoding module.

[0033] The original image acquisition module is used to acquire the first modality image and the second modality image to be fused.

[0034] The first encoding module is used to extract global energy features of the first modality image through an intensity encoder. The intensity encoder includes a multi-layer convolutional module and a channel attention module.

[0035] The second encoding module is used to extract local structural features of the second modality image through a structural encoder. The structural encoder includes a pyramid convolution module and a spatial attention module.

[0036] The fusion module is used to input the global energy features and the local structural features into the fusion module to obtain the fused features. The fusion module is a Softmax-gated feature fusion module, which dynamically generates adaptive fusion weights for each channel and spatial location of each modality through a lightweight scoring network, and performs weighted fusion.

[0037] The decoding module is used to decode the fused features using a decoder to obtain the fused image.

[0038] The neural network of the multi-modal image fusion method is optimized by a two-stage training strategy, the first stage independently trains the intensity encoder and the structure encoder to faithfully reconstruct the images of the corresponding modal, and the second stage jointly optimizes the fusion module and the decoder to generate high-quality fused images.

[0039] In a third aspect, the present application provides a multi-modal image fusion device based on a dual-branch heterogeneous network, comprising a processor, a memory, and a computer program stored in the memory. The computer program can be executed by the processor to implement the multi-modal image fusion method based on a dual-branch heterogeneous network as described in any one of the first aspect.

[0040] In a fourth aspect, the present application provides a computer-readable storage medium. The computer-readable storage medium comprises a stored computer program, wherein when the computer program is running, the computer-readable storage medium controls the device where the computer-readable storage medium is located to execute the multi-modal image fusion method based on a dual-branch heterogeneous network as described in any one of the first aspect.

[0041] By adopting the above technical solutions, the present application can achieve the following technical effects:

[0042] The multi-modal image fusion method based on a dual-branch heterogeneous network of the present application realizes differentiated extraction and efficient fusion of features of different modal images through a heterogeneous dual-branch encoder, a two-stage training strategy, and a modal-aware loss function, significantly improving the quality of the fused image and the performance of downstream tasks. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the specific embodiments of the present application. It should be understood that the following drawings only show some specific embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0044] Figure 1 The logic diagram for the first stage training of the multi-modal image fusion method.

[0045] Figure 2 The logic diagram for the second stage training of the multi-modal image fusion method.

[0046] Figure 3 The qualitative results of the present application on the RoadScene dataset.

[0047] Figure 4 The qualitative results of the present application on the MSRS dataset.

[0048] Figure 5 for the qualitative results of the present invention on MRI-PET data sets.

[0049] Figure 6 for the visualization of the quantitative results of the present invention on computational efficiency.

[0050] Figure 7 for the visualization of the quantitative results of the present invention on model complexity and inference speed. DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

[0052] Embodiment one, please refer to Figures 1 to 7 The first embodiment of the present application provides a multi-modal image fusion method based on a dual-branch heterogeneous network, which can be executed by a multi-modal image fusion device (hereinafter referred to as: multi-modal image fusion device). In particular, it is executed by one or more processors in the multi-modal image fusion device to implement steps S1 to S5.

[0053] S1, obtaining a first modality image and a second modality image to be fused.

[0054] Preferably, the first modality image is an infrared image, and the second modality image is a visible light image. Alternatively, the first modality image is a PET image, a CT image or a SPECT image, and the second modality image is an MRI image. Wherein, PET: positron emission tomography. CT: computed tomography. SPECT: single photon emission computed tomography. MRI: magnetic resonance imaging.

[0055] The neural network construction of the multi-modal image fusion method of the present application consists of a dual-branch heterogeneous encoder composed of an intensity encoder and a structure encoder.

[0056] S2, extracting global energy features of the first modality image by the intensity encoder. The intensity encoder contains a multi-layer convolution module and a channel attention module.

[0057] Specifically, the intensity encoder is used to process the first modality image (such as infrared image, PET image, CT image) dominated by energy or radiation information. The intensity encoder contains a multi-layer convolution module and a channel attention module to extract and enhance global features related to energy.

[0058] As Figure 1As shown, the core of the intensity encoder is three layers of consecutive 3x3 convolutional layers, each followed by a ReLU activation function to form a multi-layer convolutional module for progressive abstraction of global energy features. Then a channel attention module is connected, which generates channel weights through global average pooling to adaptively recalibrate the feature channels and strengthen the key channels related to thermal radiation or metabolic activity. Specifically, the channel attention mechanism uses global average pooling and Sigmoid activation to dynamically adjust the feature channel weights.

[0059] S3, extracting local structure features of the second modality image through a structure encoder. The structure encoder includes a pyramid convolutional module and a spatial attention module.

[0060] As shown in Figure 1 The structure encoder is used to process the second modality image (such as visible light image, MRI image) dominated by texture and detail information. The structure encoder includes a pyramid convolutional module and a spatial attention module to extract and focus on local detail features related to structure at multiple scales.

[0061] As shown in Figure 1 The core of the structure encoder is a pyramid convolutional module, which includes three parallel convolutional branches with kernel sizes of 1x1, 3x3, and 5x5, respectively, for simultaneously capturing local textures and global structures at different scales. Then a spatial attention module is connected, which dynamically highlights the detail-rich areas and structure boundaries in the image by calculating the importance weight map of the spatial position. Specifically, the spatial attention mechanism generates a spatial weight matrix through convolution mapping to highlight the structure area.

[0062] S4, inputting the global energy features and the local structure features into a fusion module to obtain fused features. The fusion module is a Softmax gated feature fusion module, which dynamically generates adaptive fusion weights for each channel and spatial position of each modality through a lightweight scoring network and performs weighted fusion.

[0063] The specific operation of the Softmax gated feature fusion module includes:

[0064] S41, concatenating the global energy features and the local structure features in the channel dimension.

[0065] S42, passing the concatenated features through a scoring network, which outputs two corresponding fusion weights logits for each spatial position and feature channel. The scoring network includes a 3x3 convolutional layer, a ReLU activation function, and a 1x1 convolutional layer.

[0066] S43, the logits are Softmax normalized in the modality dimension to obtain spatial and channel adaptive weight maps for the global energy feature and the local structure feature respectively.

[0067] S44, the obtained weight maps are used to respectively element-wise weight the original global energy feature and the local structure feature, and the weighted results are added to obtain the fused feature.

[0068] Specifically, as shown in the figure, Figure 2 The fusion module receives the features from the intensity encoder and the structure encoder. First, the dual-modal features are concatenated in the channel dimension. Then, the concatenated features are sent to a lightweight scoring network composed of a 3x3 convolution layer, a ReLU activation function and a 1x1 convolution layer connected in turn. The scoring network outputs two corresponding fusion weight logits for each spatial position and feature channel. These logits are Softmax normalized in the modality dimension (i.e. between the two modalities), thereby dynamically generating a pair of spatial and channel adaptive fusion weight maps. Finally, the two weight maps are used to respectively element-wise weight the input original global energy feature and local structure feature, and the weighted results are added to output the fused feature. This module realizes pixel-level and channel-level fine-grained fusion.

[0069] S5, the fused image result is output by the decoder.

[0070] The decoder is composed of multiple convolution layers and up-sampling layers, and is responsible for reconstructing the encoded features (or fused features) into images. The output layer of the decoder uses a Sigmoid activation function to ensure that the output value range is consistent with the input image.

[0071] On the basis of the above embodiment, in an optional embodiment of the present application, the neural network of the multi-modal image fusion method is optimized by a two-stage training strategy. In the first stage, the intensity encoder and the structure encoder are independently trained to faithfully reconstruct the images of their corresponding modalities. In the second stage, the fusion module and the decoder are jointly optimized to generate high-quality fused images.

[0072] The multi-modal image fusion method based on the dual-branch heterogeneous network of the present application realizes differentiated extraction and efficient fusion of features of different modal images through the heterogeneous dual-branch encoder, the two-stage training strategy and the modality-aware loss function, significantly improving the quality of the fused image and the performance of the downstream task.

[0073] The intensity encoder and structure encoder are trained independently to faithfully reconstruct images of their corresponding modalities. Specifically, the first modal image and the second modal image are input into the intensity encoder and the structure encoder, respectively. The extracted features are reconstructed back into the original image by the decoder, and each is independently optimized by a targeted reconstruction loss function to ensure that each encoder can capture and retain the unique characteristics of its corresponding modality.

[0074] like Figure 1 As shown, a large number of paired source images (such as { , Training is then performed to extract features. The feature extraction process is described below.

[0075] .

[0076] .

[0077] In the formula For infrared light images, For visible light images, For multi-layer convolution, For pyramid convolution, For channel attention, For spatial attention, for Corresponding extracted features, for Corresponding extracted features .

[0078] Loss function for the first stage of training as follows.

[0079] .

[0080] .

[0081] .

[0082] In the formula The reconstruction loss of the first modality image, For the reconstruction loss of the second modality image, For L1 loss, for Weighting coefficients For total variation loss, for Weighting coefficients For information entropy loss, for Weighting coefficients For structural similarity loss, for Weighting coefficients For mean square error loss, for Weighting coefficients For gradient loss, for The weighting coefficients.

[0083] The reconstruction loss for the first modality image combines L1 loss, total variation loss, and information entropy loss to ensure the accuracy and smoothness of intensity information while avoiding over-smoothing. The reconstruction loss for the second modality image combines structural similarity loss, mean squared error loss, and gradient loss to ensure faithful preservation of texture and structural details.

[0084] The fusion module and the decoder are jointly optimized to generate a high-quality fused image. Specifically, the parameters of the pre-trained dual-branch encoder are frozen, the features extracted by the encoder are input into the Softmax gated feature fusion module for fusion, the fused features are then processed by the decoder to generate a fused image, and the fusion module and the decoder are jointly optimized by fusion loss and cross-modal consistency loss.

[0085] like Figure 2 As shown, load the encoder weights from the first stage of pre-training and freeze them. { , The input encoder yields the features. , The input is fed into the Softmax gated feature fusion module for fusion to obtain the fused features. Finally, a fused image is generated using a decoder. .

[0086] .

[0087] In the formula This indicates the Softmax gated feature fusion module.

[0088] Loss function for the second stage of training as follows.

[0089] .

[0090] In the formula For loss based on strength consistency, For gradient loss, for The weighting coefficients.

[0091] The second stage employs a fusion loss function, which guides the fusion process by comparing the salient intensity patterns and gradient information between the fused image and the source image.

[0092] The effect of a multi-modal image fusion method based on a dual-branch heterogeneous network according to the present application is demonstrated and explained below with a specific case.

[0093] Take infrared-visible image fusion as an example, experiments are conducted on three public datasets, MSRS, RoadScene and TNO.

[0094] The model training parameters are set as follows: image block size 128x128, total training rounds 120 (40 for the first stage and 80 for the second stage), batch size 8, Adam optimizer, initial learning rate 1e-4 and decay by 0.5 times every 20 rounds.

[0095] Figure 3 and Figure 4 Qualitative comparisons on the RoadScene and MSRS datasets are shown respectively. Obviously, the present application better integrates the thermal radiation information in the infrared image and the detailed texture in the visible image. The objects in the dark area are clearly highlighted so that the foreground targets and the background can be easily distinguished. In addition, the background details that are difficult to identify due to low illumination have clear edges and rich contour information, which helps us better understand the scene.

[0096] Table 1 Quantitative comparison on the MSRS infrared-visible image fusion dataset.

[0097]

[0098] Table 2 Quantitative comparison on the TNO infrared-visible image fusion dataset.

[0099]

[0100] Table 3 Quantitative comparison on the RoadScene infrared-visible image fusion dataset.

[0101]

[0102] Tables 1-3 show quantitative comparisons on MSRS, TNO, and RoadScene datasets, respectively. The present invention achieves the highest or near-highest scores on almost all metrics, consistently outperforming most recent methods. It is worth noting that some competing models can be optimized for specific metrics (e.g., SSIM or Qabf), but often at the expense of complementary properties such as entropy (EN) or mutual information (MI), leading to reduced information content in the fusion output. In contrast, DBHFuse exhibits good balanced performance on all evaluation criteria. On each dataset, it ranks first in more than five metrics and remains competitive on the rest. This consistency highlights the robustness and versatility of our method, which can effectively preserve structural details, semantic consistency, and perceptual quality simultaneously.

[0103] Table 4 - Quantitative analysis on MRI-CT medical image fusion.

[0104]

[0105] Table 5 - Quantitative analysis on MRI-PET medical image fusion.

[0106]

[0107] Table 6 - Quantitative analysis on -SPECT medical image fusion.

[0108]

[0109] Figure 5 Qualitative results on the Harvard medical image dataset are shown. Tables 4-6 show quantitative analysis on the Harvard medical image dataset. The present invention can preserve detailed textures and highlight structural information, and achieves decent performance on almost all metrics.

[0110] The Harvard medical image dataset refers to the public brain medical image data provided by Harvard Medical School, which is a publicly available dataset.

[0111] Figure 6 and Figure 7 Visualization of the quantitative analysis of running efficiency is shown. It can be seen that the present invention achieves excellent balance between computational cost and inference speed.

[0112] Specifically, the present invention only requires 4.12 GFLOPs and 0.25 M parameters, which are the lowest among all the compared models, while achieving 598 FPS on a single RTX 3090 GPU. In contrast, methods such as RFNet and DeFusion consume billions of FLOPs due to their complex multi-stage architecture, resulting in a significant reduction in inference speed. Although U2Fusion and DIDFuse exhibit lightweight structures, their inference speed is still much lower than the present invention, which indicates that the proposed dual-branch heterogeneous design maintains efficiency without sacrificing fusion quality. These results show that the present invention achieves a better trade-off between accuracy and computational efficiency, making it more suitable for real-time infrared-visible image or medical fusion applications.

[0113] The image fusion methods used for comparison in specific cases are as follows.

[0114] Deep image decomposition method for infrared and visible image fusion (DIDFuse, DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion).

[0115] Unified unsupervised image fusion network (U2Fusion, U2Fusion: A Unified Unsupervised Image Fusion Network).

[0116] Multifunctional compression-decomposition network for real-time image fusion (SDNet, SDNet: A Versatile Squeeze-and-Decomposition Network for Real-Time Image Fusion).

[0117] Mutually reinforcing unsupervised network for multi-modal image registration and fusion (RFNet, RFNet: Unsupervised Network for Mutually Reinforcing Multi-Modal Image Registration and Fusion).

[0118] Correlation-driven dual-branch feature decomposition network for multi-modal image fusion (CDDFuse, CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion).

[0119] Self-supervised decomposition-based image fusion method (DeFusion, Decomposition for Fusion).

[0120] Target-aware dual adversarial learning image fusion method (TarD, Target-aware Dual Adversarial Learning).

[0121] Edge-preserving multi-scale image fusion method (EMF, Edge-preserving Multi-scale Fusion).

[0122] The quantitative evaluation criteria used to measure the quality and performance of image fusion algorithms are as follows.

[0123] EN: Entropy (information entropy) - measures the richness of information in the fused image.

[0124] SD: Standard Deviation (standard deviation) - measures the degree of dispersion of the contrast and information distribution of the fused image.

[0125] SF: Spatial Frequency (spatial frequency) - measures the texture details and clarity of the fused image.

[0126] MI: Mutual Information (mutual information) - measures the amount of shared information between the fused image and the source image.

[0127] SCD: Sum of the Correlations of Differences (sum of the correlation coefficients of differences) - measures the ability of the fused image to retain structural information from the source image.

[0128] VIF: Visual Information Fidelity (visual information fidelity) - measures the fidelity of the fused image to the human visual system.

[0129] Qbaf: Quality index based on an artifact measure (quality index based on an artifact measure) - measures the degree of distortion or artifacts introduced in the fusion result.

[0130] SSIM: Structural Similarity Index Measure (structural similarity index measure) - measures the degree of similarity in structure, brightness, and contrast between the fused image and the source image.

[0131] In the second embodiment, the application provides a multi-modal image fusion device based on a dual-branch heterogeneous network, which comprises an original image acquisition module, a first encoding module, a second encoding module, a fusion module and a decoding module.

[0132] The original image acquisition module is configured to acquire a first modal image and a second modal image to be fused.

[0133] The first encoding module is configured to extract global energy features of the first modal image by using an intensity encoder.

[0134] The second encoding module is configured to extract local structure features of the second modal image by using a structure encoder.

[0135] The fusion module is configured to input the global energy features and the local structure features into the fusion module to obtain fused features.

[0136] The decoding module is configured to decode the fused features by using a decoder to obtain a fused image.

[0137] The neural network of the multi-modal image fusion method is optimized by using a two-stage training strategy.

[0138] In the third embodiment, the application provides a multi-modal image fusion device based on a dual-branch heterogeneous network, which comprises a processor, a memory and a computer program stored in the memory.

[0139] It can be understood that the multi-modal image fusion device can be a portable notebook computer, a desktop computer, a server, a smart phone or a tablet computer, or an electronic device with computing performance.

[0140] In the fourth embodiment, the application provides a computer readable storage medium.

[0141] Obviously, the above-described embodiments are only some of the embodiments of the present application, but not all of them. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts should fall within the scope of the present application.

[0142] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed apparatus and method can also be implemented by other ways. The embodiments of the apparatus and method described above are only schematic, and for example, the flowcharts and block diagrams in the drawings show the possible implementation modes of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that, in some alternative implementation modes, the functions noted in the blocks can occur in different order from that noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for implementing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0143] In addition, each functional module in the various embodiments of the present application can be integrated together to form a separate part, or each module can exist independently, or two or more modules can be integrated to form a separate part.

[0144] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts of the prior art that make contributions or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various media that can store program codes. It should be noted that in this document, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements that are not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0145] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "an" and "the" used in the embodiments of the present application are also intended to include plural forms unless the context clearly indicates otherwise.

[0146] It should be understood that the term "and / or" used herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this document generally represents an "or" relationship between the associated objects before and after it.

[0147] Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if it is determined" or "if (a stated condition or event) is detected" can be interpreted as "when it is determined" or "in response to determining" or "when (a stated condition or event) is detected" or "in response to detecting (a stated condition or event)".

[0148] The "first / second" mentioned in the embodiments only distinguishes similar objects, and does not represent a specific order for the objects. Understandably, the "first / second" can be interchanged in a specific order or sequence as appropriate. It should be understood that the objects distinguished by "first / second" can be interchanged as appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0149] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Various modifications and changes can be made to the present application by those skilled in the art. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-modal image fusion method based on a dual-branch heterogeneous network, characterized in that, The method comprises the steps of: S1, obtaining a first modality image dominated by energy or radiation information and a second modality image dominated by texture and detail information to be fused; S2, extracting global energy features of the first modality image by an intensity encoder; The intensity encoder comprises a multi-layer convolution module and a channel attention module; S3, extracting local structure features of the second modality image by a structure encoder; the structure encoder comprises a pyramid convolution module and a spatial attention module; S4, inputting the global energy features and the local structure features into a fusion module to obtain fused features; The fusion module is a Softmax gated feature fusion module, which dynamically generates adaptive fusion weights for each channel and spatial position of each modality through a lightweight scoring network, and performs weighted fusion; S5, decoding the fused features by a decoder to obtain a fused image; The neural network of the above multi-modal image fusion method is optimized by a two-stage training strategy, the intensity encoder and the structure encoder are independently trained in the first stage to faithfully reconstruct the images of the corresponding modalities, and the fusion module and the decoder are jointly optimized in the second stage to generate high-quality fused images; The core of the intensity encoder is three consecutive 3x3 convolution layers, each of which is followed by a ReLU activation function to form a multi-layer convolution module for progressively abstracting global energy features; Then, a channel attention module is connected, which generates channel weights through global average pooling to adaptively recalibrate feature channels and strengthen key channels related to thermal radiation or metabolic activity. The core of the structure encoder is a pyramid convolution module, which comprises three parallel convolution branches with convolution kernel sizes of 1x1, 3x3 and 5x5, respectively, for simultaneously capturing local textures and global structures of different scales; then, a spatial attention module is connected, which dynamically highlights the detail-rich areas and structure boundaries in the image by calculating the importance weight map of the spatial position; The specific operation of the Softmax gated feature fusion module comprises: S41, concatenating the global energy features and the local structure features in the channel dimension; S42, passing the concatenated features through a scoring network, which outputs two corresponding fusion weights logits for each spatial position and feature channel; the scoring network comprises a 3x3 convolution layer, a ReLU activation function and a 1x1 convolution layer; S43, performing Softmax normalization on logits in the modality dimension to obtain spatial and channel adaptive weight maps of the global energy features and the local structure features, respectively; S44, using the obtained weight maps to weight the original global energy features and local structure features element by element, respectively, and adding the weighted results to obtain the fused features.

2. The multi-modal image fusion method based on the dual-branch heterogeneous network according to claim 1, characterized in that, The decoder is composed of multiple convolution layers and up-sampling layers, and is responsible for reconstructing the encoded features into images; The output layer of the decoder uses a Sigmoid activation function to ensure that the output value range is consistent with the input image. 3.The multi-modal image fusion method based on a dual-branch heterogeneous network according to claim 1, characterized in that, The intensity encoder and the structure encoder are independently trained to faithfully reconstruct images of their corresponding modalities, specifically, the first modality image and the second modality image are input into the intensity encoder and the structure encoder respectively, the extracted features are reconstructed back to the original images by the decoder, and independent optimization is performed through a targeted reconstruction loss function to ensure that each encoder can capture and retain the unique characteristics of its corresponding modality; Loss function for first stage training As follows; ； ； ； In the formula The reconstruction loss of the first modality image, For the reconstruction loss of the second modality image, For L1 loss, for Weighting coefficients For total variation loss, for Weighting coefficients For information entropy loss, for Weighting coefficients For structural similarity loss, for Weighting coefficients For mean square error loss, for Weighting coefficients For gradient loss, for Weighting coefficients; The fusion module and the decoder are jointly optimized to generate a high-quality fused image, specifically, the pre-trained dual-branch encoder parameters are frozen, the features extracted by the encoder are input into the Softmax gated feature fusion module for fusion, the fused features are then input into the decoder to generate a fused image, and the fusion module and the decoder are jointly optimized through a fusion loss and a cross-modality consistency loss; Loss function for second stage training As follows; ； In the formula is a loss based on consistency of intensity. 4.The method of claim 1, wherein, The first modality image is an infrared image, and the second modality image is a visible light image; or the first modality image is a PET image or a CT image, and the second modality image is an MRI image.

5. A multi-modal image fusion device based on a dual-branch heterogeneous network, characterized in that, A multi-modal image fusion method based on a dual-branch heterogeneous network is used to perform any one of claims 1-4. A multi-modal image fusion device comprises: An original image acquisition module is configured to acquire a first modality image dominated by energy or radiation information and a second modality image dominated by texture and detail information to be fused; A first encoding module is configured to extract global energy features of the first modality image through an intensity encoder; The intensity encoder comprises a multi-layer convolution module and a channel attention module; A second encoding module is configured to extract local structure features of the second modality image through a structure encoder; the structure encoder comprises a pyramid convolution module and a spatial attention module; A fusion module is configured to input the global energy features and the local structure features into the fusion module to obtain fused features; The fusion module is a Softmax gated feature fusion module, which dynamically generates adaptive fusion weights for each channel and spatial position of each modality through a lightweight scoring network and performs weighted fusion; A decoding module is configured to decode the fused features through a decoder to obtain a fused image. The neural network of the multi-modal image fusion method is optimized through a two-stage training strategy, the first stage independently trains the intensity encoder and the structure encoder to faithfully reconstruct images of their corresponding modalities, and the second stage jointly optimizes the fusion module and the decoder to generate a high-quality fused image.

6. A multi-modal image fusion device based on a dual-branch heterogeneous network, characterized by, The computer program can be executed by the processor to implement a multi-modal image fusion method based on a dual-branch heterogeneous network as claimed in any one of claims 1-4.

7. A computer readable storage medium characterized in that, The computer readable storage medium comprises a stored computer program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to execute a multi-modal image fusion method based on a dual-branch heterogeneous network as claimed in any one of claims 1-4 when the computer program is running.

Images