A dual-brain heterogeneous network-based multi-modal medical image fusion device and method

Abstract

The application discloses a kind of based on the fusion model and method of multi-modal medical image of double-brain heterodirectional network.It is a kind of based on the fusion model of multi-modal medical image of double-brain heterodirectional network, comprising: double-brain heterodirectional network module and loss function module;The double-brain heterodirectional network module includes: Lefthemisphere1, Lefthemispher2 and Righthemisphere;Wherein, the Lefthemisphere1, Left hemispher2 as main network respectively extract global feature;The Righthemisphere is a dynamic guide network, for extracting local feature;The loss function module drives network to pay attention to edge details and structure information.The fusion model and method of multi-modal medical image of double-brain heterodirectional network described in the application, adopt the mode that global feature dynamically guides local feature, to achieve long-term dynamic guidance relationship modeling, so that the network can fully realize the function of cross-east-west hemisphere integrated complementary feature information, solve the problem in multi-modal medical image fusion.

Description

Technical Field

[0001] This invention belongs to the technical field of medical imaging, specifically relating to a fusion device and method for multimodal medical images based on binaural brain networks. Background Technology

[0002] Multimodal medical image fusion technology aims to overcome the problem of insufficient information transmission under a single imaging mechanism. It combines salient information from source images to generate an information-rich fused image, thereby promoting its subsequent applications. During diagnosis, due to the complexity of disease types, clinicians often need to repeatedly read images of the same area from different modalities, which places a heavy workload on clinical practice. Therefore, we focus on organically combining the specific information encompassed by single-photon emission computed tomography (SPECT) and magnetic resonance imaging (MRI) to advance applications in clinical diagnosis, treatment planning, and preoperative navigation.

[0003] After decades of development, the problem of multimodal medical image fusion has been studied in depth. It can be summarized into two categories: traditional fusion methods and deep learning-based fusion methods. Traditional fusion methods are designed and optimized to achieve the desired fusion results by manually analyzing the information content in the transformation domain or spatial domain based on mathematical transformation, and designing and optimizing decomposition strategies and fusion rules to meet different needs. In order to avoid the limitations of traditional fusion methods, deep learning-based medical image fusion methods have been widely developed, but at present, the following shortcomings still exist: (1) Existing deep learning methods mostly capture local and global features of different modalities in a non-interactive manner, but cannot avoid the problem of feature similarity. However, research based on human visual perception shows that global features of images have a guiding role in the learning of local features. (2) Existing deep learning methods usually feed the image directly into the feature extraction network and only use addition or splicing to fuse features at high levels, without considering the heterogeneity of modal features at different levels, which may lead to the loss of some key information. (3) The loss function used by existing deep learning-based methods still needs to be improved in terms of its attention to noise and edge details. To some extent, the quality of the loss function design directly affects the final fusion effect of the model.

[0004] In view of this, the present invention proposes a new device and method for fusing multimodal medical images based on bibrain heterogeneous networks, which effectively overcomes the inadequacy of expressing information in single-modal images. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal medical image fusion device and method based on a bipolar hemisphere network. It comprehensively considers the specific information of functional and anatomical images at different levels, and adopts a global feature dynamic guidance method to local feature, thereby achieving long-term dynamic guidance relationship modeling. This enables the network to fully realize the function of integrating complementary feature information across the Eastern and Western Hemispheres, thus solving the problems in multimodal medical image fusion.

[0006] To achieve the above objectives, the technical solution adopted is as follows:

[0007] A fusion device for multimodal medical images based on a binaural anisotropic network includes: a binaural anisotropic network module and a loss function module;

[0008] The aforementioned bipolar network module includes: Left hemisphere1, Left hemispher2, and Righthemisphere;

[0009] The Left hemisphere1 and Left hemisphere2 serve as backbone networks to extract global features respectively; the Left hemisphere1 includes: OD module, Transformer model1 module and Transformer model 2 module; the Left hemisphere2 includes: OD1 module, OD2 module and Transformerblock module.

[0010] The Right hemisphere is a dynamically guided network used to extract local features, including: DAFM_1 module, DAFM_2 module and DAFM_3 module;

[0011] The loss function module is used to train the bipolar network, driving the network to focus on edge details and structural information.

[0012] Furthermore, the loss function module adopts the following formula:

[0013] L all =L SSIM +L RMI +L Texture ,

[0014] Among them, L SSIM L RMI and L intensity These represent structural loss, region loss, and texture loss, respectively.

[0015] Furthermore, the structural-level loss is expressed using the following formula:

[0016] ssim(·) represents the structural similarity operation; the region-level loss is calculated using the following formula: In the formula, γ and δ are used as hyperparameters for balancing;

[0017] RMI is defined as follows:

[0018]

[0019] In the formula, ε∈[0,1] is the weight parameter. Represents the source image I and the fused image The normal cross-entropy loss between them, where C represents the batch size. Represents the lower limit of mutual information;

[0020] The texture loss is expressed using the following formula:

[0021] In the formula, For the Prewitte gradient operator, |·| denotes the absolute operation, and |||1 denotes the l1 norm.

[0022] Another objective of this invention is to provide a method for fusing multimodal medical images based on bipolar heterogeneous networks, which effectively overcomes the inadequacy of expressing information in single-modal images.

[0023] To achieve the above objectives, the technical solution adopted is as follows:

[0024] A method for fusing multimodal medical images based on binaural heterotropic networks, employing the aforementioned fusion device, includes the following steps:

[0025] (1) Use RGB-YCbCr color conversion to make the SPECT image I1∈R H×W×Cin1 Perform conversion to obtain and Three channel components;

[0026] (2) and MRI images I2∈R H×W×Cin2 The images are used as inputs to the Left hemisphere1 and Left themispher2, respectively, and then fused by the binaural anisotropic network module and loss function module.

[0027] (3) Through and Implement YCbCr to RGB color conversion to generate the final fused image I. f ∈R H ×W×Cout .

[0028] Furthermore, in step (2), Left hemisphere1 generates the feature map using the following formula:

[0029]

[0030] The Left hemisphere2 feature map is generated using the following formula:

[0031]

[0032] The aforementioned and The generation process uses the following formula:

[0033]

[0034] In the formula, Representing an image The feature map output by the OD module. express The output feature map after passing through the Transformer Model 1 module, express The output feature map after passing through the Transformer Model 2 module. express express The output feature map after passing through the Transformer Block module.

[0035] Furthermore, the Transformer model 1 module, Transformer model 2 module, and Transformerblock module are filter modules, using the following formula:

[0036]

[0037] In the formula, and These represent the input of the Transformer Model, the result of the first addition operation, and the result of the second addition operation, respectively. LN represents layer normalization, MSA represents multi-head self-attention mechanism, and MLP represents multi-layer perception mechanism.

[0038] Furthermore, in step (2), multiple stage feature maps of Left hemisphere1 and Left hemisphere2 are used as network inputs of Right hemisphere to extract local features.

[0039] Furthermore, the DAFM_1 module only accepts two adjacent feature maps from different modalities each time, fusing semantic information from high-level functional images and detailed information from low-level anatomical images;

[0040] The DAFM_2 and DAFM_3 modules accept three adjacent feature maps;

[0041] The DAFM_2 module uses the GCAB and LCAB modules for depth extraction, thereby improving the image representation capabilities.

[0042] Furthermore, the generation diagrams for each stage of the DAFM_1, DAFM_2, and DAFM_3 modules are generated using the following formula:

[0043]

[0044] In the formula, τ1 is the DAFM_1 module, τ2 is the DAFM_2 module, and τ3 is the DAFM_3 module. and They have the same shape.

[0045] Furthermore, the fusion method described above can be used for the fusion of SPECT and MRI, PET and MRI, and CT and MRI images.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0047] This invention's multimodal medical image fusion technology effectively overcomes the shortcomings of single-modal image representation, helping doctors improve diagnostic efficiency and promote precise preoperative navigation. Previous fusion studies often employ non-interactive methods to simultaneously focus on local and global features of different modalities, and only use addition or splicing methods to fuse features at high levels, easily losing modal differentiation information. Based on this, this invention proposes a dual-brain anisotropic network (TBHNet) for multimodal medical image fusion. Specifically, this invention designs a dual-brain mechanism to extract global features separately, attempting to construct a dynamic interactive module to achieve autonomous guidance of local features from global features in a dual-modal context. Secondly, the anisotropic input mechanism designed in this invention can simultaneously capture high-level semantic features of functional images and low-level detail features of anatomical images, avoiding the loss of structural detail information. Furthermore, the loss function composed of SSIM loss, region loss, and texture loss can better drive the network to focus on edge details and structural information. Experimental results show that, in multimodal medical image fusion scenarios, TBHNet outperforms existing representative techniques in both subjective and objective evaluations. Importantly, this invention also allows for direct, non-adjustable application to PET-MRI and CT-MRI fusion problems, with satisfactory fusion results demonstrating TBHNet's good generalization ability. Specifically:

[0048] 1. The present invention provides a multimodal medical image fusion device and method based on a bibrain heterogeneous network. From the perspective of human visual perception, it proposes a bibrain heterogeneous network that simultaneously focuses on the heterogeneity of local and global features under different modalities in an interactive manner, thereby improving image fusion performance.

[0049] 2. The multimodal medical image fusion device and method based on binaural heterogeneous network described in this invention differs from existing fusion methods. The heterogeneous mechanism in TBHNet focuses on the complementarity between the semantic information of functional images at a high level and the detailed information of anatomical images at a low level.

[0050] 3. The multimodal medical image fusion device and method based on bibrain heterogeneous network described in this invention defines a unified loss function to preserve salient information and edge information from three aspects: structural detail preservation, regional mutual information loss, and texture loss.

[0051] 4. The multimodal medical image fusion device and method based on bipolar hemispheric networks described in this invention have been extensively tested on mainstream datasets, demonstrating that the proposed method can, to some extent, overcome the modal limitations of SPECT and MRI, acquiring single composite images containing significant information simultaneously. This invention extends it without fine-tuning to PET and MRI, and CT and MRI image fusion tasks. Experimental results show that it outperforms other representative and state-of-the-art methods in qualitative and quantitative evaluation, exhibiting good generalization ability. Attached Figure Description

[0052] Figure 1 Presentation of SPECT and MRI image fusion results;

[0053] Figure 2 This forms the overall architecture of the model;

[0054] Figure 3 Here are the model structure diagrams for LCAB and GCAB;

[0055] Figure 4 The results of SPECT and MRI are shown below; where a is MRI, b is SPECT, c is MSMG_PCNN_EA, d is CSMCA, e is TL_ST, f is MLEPF, g is DATFuse, h is YDTR, i is Swinfusion, j is Crossfusion, k is MATR, and l is Ours.

[0056] Figure 5 Display of quantitative indicators for SPECT and MRI;

[0057] Figure 6 The results of PET and MRI are shown below; where a is MRI, b is SPECT, c is MSMG_PCNN_EA, d is CSMCA, e is TL_ST, f is MLEPF, g is DATFuse, h is YDTR, i is Swinfusion, j is Crossfusion, k is MATR, and l is Ours.

[0058] Figure 7 Presenting CT and MRI results;

[0059] Figure 8 Display of quantitative indicators for PET and MRI;

[0060] Figure 9 Display of quantitative indicators for CT and MRI;

[0061] Figure 10 This is an ablation diagram of the model structure; column a is MRI, column b is SPECT, column c is TBHNet (with loss function coefficients of α=1.5, β=1, γ=1, δ=1, and ε=1), and column d is the modified TBHNet model structure diagram when they are in the same direction (see [link to modified network structure diagram]). Figure 11 ), column e shows the TBHNet model structure diagram without DFAM_1, DFAM_2, and DFAM_3 (see the modified network structure diagram). Figure 12 ), column f shows the TBHNet model structure without DFAM_1 (see the modified network structure diagram). Figure 13 ), column g shows the TBHNet model structure diagram after swapping the input images of Left hemispher1 and Left hemispher2 (see the modified network structure diagram). Figure 15 The h column shows the TBHNet model structure diagram without dynamic mechanisms (see the modified network structure diagram). Figure 14 (i) represents the loss function L. all1 Column j represents the loss function L. all2 The k column represents the loss function L. all3 The column 'l' represents the loss function L. all ;

[0062] Figure 11 When presenting a unidirectional structure in the modified TBHNet model structure diagram;

[0063] Figure 12 The structure diagram of the TBHNet model without DFAM_1, DFAM_2, and DFAM_3 is shown.

[0064] Figure 13 The structure diagram of the TBHNet model without DFAM_1 is shown.

[0065] Figure 14 The diagram shows the structure of the TBHNet model without a dynamic mechanism.

[0066] Figure 15 This is a diagram of the TBHNet model structure after the input image has been swapped. Detailed Implementation

[0067] To further illustrate the present invention's device and method for fusing multimodal medical images based on binaural heterogeneous networks, and to achieve the intended objectives, the following detailed description, in conjunction with preferred embodiments, details the specific implementation, structure, features, and effects of the device and method for fusing multimodal medical images based on binaural heterogeneous networks proposed in this invention. In the following description, different "embodiments" or "embodiments" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable manner.

[0068] Before detailing the present invention’s device and method for fusing multimodal medical images based on bipolar heterogeneous networks, it is necessary to further explain the relevant background mentioned in the present invention in order to achieve better results.

[0069] 1. Image fusion

[0070] Early methods for image fusion are collectively referred to as traditional methods. Among these, the development of filters in spatial domain algorithms is the most classic. Initially, Tomasi proposed the Bilateral Filter (BF), and Jian proposed a combined use of bilateral filtering and rolling guided filtering. While these spatial domain-based algorithms are computationally efficient and simple to implement, they are relatively poor at preserving the contrast and brightness information of the source image. Therefore, to compensate for the shortcomings of spatial domain algorithms, transform domain-based methods have emerged. Examples include Pajares' Discrete Wavelet Transform (DWT), Wang et al.'s research on wavelet transform (WT), Dual Tree-Complex Wavelet Transform (DT-CWT), Contourlet Transform (CT), Shearlet Transform (ST), Non-Subsampled Contourlet Transform (NSCT), and Non-Subsampled Shearlet Transform (ST). The above methods include the Transform (NSST), and medical image fusion based on NSST and parameter-adaptive pulse-coupled neural networks published in recent years; the hybrid image decomposition model for MRI and CT fusion proposed by Zhu et al.; an image decomposition optimization model based on pixel intensity skewness (SPI) and a novel adaptive co-occurrence filter (ACOF); a multi-level image decomposition method based on latent low-rank representation (LatLRR) for infrared and visible light image fusion; a multimodal medical image fusion method based on structural patch decomposition (SPD); and a wavelet transform based on Haar filtering to implement a multifocal image fusion algorithm. Most of these methods follow the "decomposition-fusion-reconstruction" steps, but they have two main shortcomings: firstly, to ensure the steady progress of subsequent feature fusion, traditional methods generally use the same transformation for different source image inputs, neglecting the complementarity of features between different modalities during the decomposition process; secondly, manually designed fusion strategies have significant limitations, resulting in limited image fusion performance.

[0071] At present, deep learning is playing an increasingly important role in the field of computer vision due to its powerful feature representation capabilities, and a number of multimodal medical image fusion methods based on deep learning have emerged accordingly.

[0072] Although existing deep learning-based medical image fusion methods have been widely developed, they still have the following shortcomings: they neglect the guiding role of global features on local features, often employ non-interactive methods to simultaneously capture local and global features from different modalities, which can easily lead to feature similarity issues and thus affect the quality of the fused image. Most studies have not explored the heterogeneity and complementarity of modal features at different levels.

[0073] 2. Vision Transformers for multi-task image fusion

[0074] As the Transformer architecture has expanded from Natural Language Processing (NLP) to Computer Vision (CV), more and more researchers have focused on its unique advantages in extracting global semantic information, leading to its widespread use. Fu et al. proposed a Patch Pyramid Transformer (PPT) architecture that better focuses on the local relevant information of neighboring pixels. Specifically, the Patch Transformer first transforms the image into a sequence of patches; then, with the help of the pyramid transformer, it extracts the non-local information of the image; finally, it constructs an effective image reconstruction network, resulting in excellent fusion performance. Ma et al. focused on the complementarity of long-distance dependencies within and across domains, achieving outstanding fusion model performance by constructing attention-guided cross-domain modules and designing an efficient loss function. Tang et al. abandoned traditional convolution and introduced highly effective adaptive convolution, employing an adaptive transformer to enhance global semantic extraction capabilities, fully extracting source image information from different scales. Tang et al. designed a Y-shaped dynamic transformer (YDTR) for infrared and visible light image fusion, considering both global and local features. Li et al. recognized the necessity of fusing global and local features. Therefore, their research designed a convolution feature extraction module (CFEM) and a transformer feature extraction module (TFEM) to extract local and global features respectively, and attempted to capture the intrinsic relationship between local features and long-range dependencies by alternating between CFEM and the transformer module. Qu et al. designed an encoder-decoder-based image fusion framework, using CNN and Transformer to extract local and global features respectively, and then fused them. This framework achieved satisfactory results on multiple tasks. Tang et al. designed a local feature extraction branch (LFEB) and a global feature extraction branch (GFEB) to extract local and global information respectively, and designed a cross-correlation loss to train the model for infrared and visible light image fusion. Chen et al. proposed a transformer and hybrid feature extractor-based fusion framework. First, a dual-branch CNN module was used to extract local features, then a visual transformer module was used to capture global features, and finally, the fused result was obtained through an image reconstruction module.

[0075] Transformer is favored by many researchers for its ability to capture long-range dependencies. However, current research only uses simple fusion operations to fuse global and local features under multimodal conditions, and has not yet explored the role of global features in guiding local features in improving the quality of fused images under interactive methods.

[0076] Therefore, based on human visual perception, this invention meticulously designs an end-to-end bibrain anisotropic network TBHNet for multimodal medical image fusion. Building upon the OCDConv module, it employs an adaptive transformer with a shift window mechanism to construct long-term relationships, fully extracting specific features from SPECT and MRI images to improve the quality of the fused image. Furthermore, this invention focuses on the dynamic interaction between global and local features across the two modalities, attempting to increase the differentiated representation of features between modalities. Secondly, this invention proposes an anisotropic mechanism to focus on the complementary features encompassed by functional and anatomical images at different levels. In addition, a complex loss function composed of SSIM loss, region loss, and texture loss enables the fused image to exhibit optimal apparent strength.

[0077] Having understood the relevant background mentioned in this invention, the following will provide a more detailed description of the fusion device and method for multimodal medical images based on binaural brain networks, in conjunction with specific embodiments:

[0078] This invention discloses a multimodal medical image fusion device and method based on a bipolar anisotropic network. It proposes a bipolar mechanism to extract global features from two different modalities, and uses an autonomous guidance module to promote the dynamic interaction between global and local features, thereby increasing the difference between feature representations. Secondly, it proposes an anisotropic mechanism to focus on the semantic information of functional images at a higher level and the detailed information of anatomical images at a lower level. Furthermore, the loss function designed in this invention consists of SSIM loss, region loss, and texture loss, which can better drive the network to focus on texture details and structural information.

[0079] To demonstrate the effectiveness of the fusion method of the present invention, Figure 1 An example pair of SPECT and MRI images is shown, along with the fusion results based on advanced fusion methods and the method proposed in this invention. Local textures are outlined with rectangles and magnified below the corresponding images to better compare the detail information of the fused images. It can be observed that other fusion algorithms inevitably ignore the specific information of the source images. More specifically, such as... Figure 1As can be observed within the red box, in some SPECT images, regions lacking functional information often suffer from structural information distortion or even black artifacts due to the lack of dynamic guidance from global information to local information, as other contrast algorithms typically destroy these details. However, TBHNet focuses more on the structural details and edge texture information of MRI images, resulting in better fusion performance.

[0080] Example 1.

[0081] The specific operating steps are as follows:

[0082] The invention will be described in three parts. First, the overall framework TBHNet (A) is described. Second, the bipolar network structure is introduced (B). Finally, the optimal objective function proposed for construction (C) and the experimental results analysis are presented (D).

[0083] A. Overall Framework:

[0084] Research indicates that the left and right hemispheres of the brain tend to capture global and local features more readily. Therefore, this invention, considering both the semantic information of functional images and the detailed information of anatomical images, employs a dual-brain mechanism to extract global features separately. It attempts to construct an interactive module to guide local features under dual-modal conditions, thereby achieving long-term dynamic guidance for relationship modeling. Furthermore, the anisotropic input mechanism designed in this invention can focus on the high-level semantic features of SPECT images and the low-level detailed features of MRI images, thus addressing the problems in multimodal medical image fusion.

[0085] The network structure of TBHNet is as follows: Figure 2 As shown in the solid rectangle, Left hemisphere1 and Left hemispher2 use the ODConv module and two adaptive transformer modules as backbone networks to extract respectively. And the global features of I2. The Right hemispheric is a dynamically guided network used to extract local features. Specifically, the embodiments of the present invention first use RGB-YCbCr color conversion to solve the color medical image fusion problem, so that the SPECT image I1∈R H×W×Cin1 Perform conversion to obtain and Three channel components. Then, and MRI images I2∈R H×W×Cin2 The images are used as inputs to Leftthemisphere1 and Lefthemispher2 respectively, and the fused images are output by TBHNet. Ultimately, through and To generate the final fused image, YCbCr to RGB color conversion is implemented. f ∈R H×W×Cout .

[0086] Detailed information about B TBHNet:

[0087] 1)Left hemisphere1 and Left hemispher2

[0088] ①Left hemisphere structure:

[0089] like Figure 2 As shown, Left hemisphere1 and Left hemispher2 are two feature extraction networks. and It consists of three phases: the OD module, the Transformer model 1 module, and the Transformer model 2 module. It consists of three phases: the OD1 module, the OD2 module, and the Transformerblock module. We will... I2 and I2 are input to respectively and The module will generate three important feature maps during network operation. After each stage, the width and height of the feature maps will remain the same. and The specific generated feature maps are shown in Formula 1 and Formula 2. and The generation process is shown in Equations 3 and 4.

[0090]

[0091] in, Representing an image The feature map output by the OD module. express The output feature map after passing through the Transformer Model 1 module, express The output feature map after passing through the Transformer Model 2 module. express express The output feature map after passing through the Transformer Block module. OD and T2 represent... The indices for the start and end of the network phase, represented by OD1 and T3. Indexes to the start and end of network phases.

[0092] Specifically, the ability to fully preserve important complementary information from multimodal source images has become a crucial factor affecting the performance of fused images. Conventional convolutional layers have significant advantages in extracting local image features, but their single static kernel and independence from the input samples make it easy to lose complex texture information contained in anatomical images when using only conventional convolutional layers for fusion tasks. Therefore, considering the importance of rich contextual semantic information, this invention attempts to introduce full-dimensional dynamic convolution to adaptively adjust the weights of the convolutional kernel in four dimensions, thereby significantly improving the feature representation capability of the convolutional kernel. In addition, more complementary information can be adaptively maintained to obtain more comprehensive fusion results. The Transformer Model structure used in this invention is as follows: Figure 2 As shown in the solid line rounded corner matrix, this filter block can be completed by two addition operations, corresponding to Formula 5 and Formula 6 respectively.

[0093]

[0094] in, and These represent the input to the Transformer Model, the result of the first addition operation, and the result of the second addition operation, respectively. LN represents layer normalization, MSA represents multi-head self-attention mechanism, and MLP represents multilayer perceptron mechanism.

[0095] Effectively extracting functional metabolic information and structural soft tissue details is crucial for improving the quality of fused images. Compared with existing technologies, this invention proposes a method more suitable for extracting global features from medical images rich in structural details. Specifically, since traditional convolution operations have limited ability to preserve global contextual information, this invention introduces OCDonv to replace traditional convolution for feature extraction. The introduction of the OCDonv module allows for adaptive adjustment of the convolution kernel weights across four dimensions. Furthermore, this invention employs a dual Transformer Model for deep extraction of global contextual information, which is beneficial for generating fusion results with greater information content.

[0096] ②Right hemisphere structure:

[0097] like Figure 2As shown, the Right hemisphere Block is a dynamically guided network τ used to extract local features, consisting of three DAFM modules. In this invention, multiple stage feature maps of Left hemisphere1 and Left hemisphere2 are used as network inputs of τ, and the generation maps of each stage of the DAFM module are shown in formulas (7)-(9).

[0098]

[0099] Where τ1 is the DAFM_1 module, τ2 is the DAFM_2 module, and τ3 is the DAFM_3 module. and They have the same shape. Clearly, DAFM_1 accepts only two adjacent feature maps from different modalities at a time, while the DAFM_2 and DAFM_3 modules accept three adjacent feature maps. The goal of this invention is to use a dynamic, progressive approach to advance feature flow from high to low levels, attempting to alleviate the semantic gap between high-level and low-level features.

[0100] This invention designs a method more suitable for multimodal medical image fusion. For example... Figure 2 As shown in the dashed rectangle, DAFM_1 accepts high-level features from the functional image and low-level features from the anatomical image, and applies a multiplicative guiding factor to apply texture, edge, and other information from the anatomical image to the semantic features of the functional image. This dynamically guides semantic information with detail information, and under the influence of an additive guiding factor, the fused feature map is finally obtained. Similarly, the DAFM_2 module, after obtaining specific features for both modalities, uses a dynamic guidance method to apply to local features respectively, establishing guidance relationships between differentiated global and local features for the two modalities to achieve better fusion results. The model structure diagrams for LCAB and GCAB are shown below. Figure 3 As shown.

[0101] Compared with existing technologies, this invention proposes a method that employs intermodal feature differential representation learning. First, based on the heterogeneous input, the DAFM_1 module fuses semantic information from high-level functional images and detailed information from low-level anatomical images. Second, for the different modal features of functional and anatomical images, this invention designs GCAB and LCAB modules respectively to effectively extract depth, attempting to improve the image representation capability. This invention emphasizes the differential feature input of modules, constructing three DAFM sub-modules, which can progressively narrow the differences in semantic features between high and low levels to obtain better fusion results. Furthermore, this invention uses multiplicative and additive factors to enable the network to automatically focus on key regions in different modalities and guide the supplementary learning of local features.

[0102] C loss function construction

[0103] In unsupervised learning, the construction of the loss function plays a guiding role in the fusion results of multimodal medical image fusion. This invention proposes a new loss function to train the TBHNet network, and the loss function formula is shown in Equation 10:

[0104] L all =L SSIM +L RMI +L Texture (10)

[0105] Among them, L SSIM L RMI and L intensity These represent structural loss, region loss, and texture loss, respectively.

[0106] Structural loss: The structural similarity (SSIM) index can effectively reflect image distortion from three aspects. Therefore, in order to further preserve the useful features of SPECT and MRI images, this invention adopts a structural loss based on the definition of the structural similarity index (SSIM), as shown in Equation 11:

[0107]

[0108] Here, ssim(·) represents the structural similarity operation.

[0109] Regional-level loss: This invention introduces the concept of Regional Mutual Information (RMI), which can accurately preserve image information under multimodal conditions. RMI The formula is shown in Formula 12:

[0110]

[0111] Where γ and δ are used as hyperparameters for balancing, and RMI is defined as shown in Equation 13:

[0112]

[0113] Where ε∈[0,1] is a weight parameter. Represents the source image I and the fused image The normal cross-entropy loss between them. C represents the batch size. This represents the lower limit of mutual information.

[0114] Texture loss: Preserving more texture details is one of the important factors in improving image fusion results. This invention attempts to improve the texture details of the fusion result with the help of the Prewittee edge detection operator. This invention uses texture loss L Texture The construction details are shown in formula (14).

[0115]

[0116] in, This is the Prewitte gradient operator, which can effectively measure the texture details of an image. |·| represents the absolute operation, and |||1 represents the l1 norm.

[0117] Analysis of Experimental Results (D):

[0118] This invention describes the dataset used and training details. Secondly, it presents qualitative and quantitative comparisons and analyses of the TBHNet method with other state-of-the-art methods on SPECT-MRI fusion tasks. Thirdly, this invention extends the model's applicability to PET-MRI and CT-MRI image fusion tasks. Finally, this invention conducts extensive ablation experiments on the model structure, the setting of tradeoff parameters in the loss function, and the composition of the objective function, verifying the effectiveness of the TBHNet module's specific design.

[0119] (1) Dataset preparation and training details

[0120] The Harvard University dataset is a widely used medical image fusion dataset that can be used for fair comparisons. This study downloaded 607 pairs of medical images from the mainstream medical image database published by Harvard University (http: / / www.med.harvard.edu / AANLIB / home.html), including 390 pairs of SPECT and MRI images, 97 pairs of PET and MRI images, and 120 pairs of CT and MRI images. Since image registration is a prerequisite for image fusion, it was assumed that all image pairs used in this study could be accurately aligned. In this embodiment, the 390 image pairs were subjected to the following five experiments: First, the dataset was automatically divided into 350 image pairs as the training set, 20 SPECT and MRI image pairs as validation samples, and the remaining 20 image pairs as the test set. The validation set was used to determine the model structure, loss function, and hyperparameter settings within the loss function. In this embodiment, the final hyperparameters in the loss function were set to α = 1.5, β = 1, γ = 1, δ = 1, and ε = 1. Secondly, in this embodiment, 20 pairs of validation samples were manually removed from the remaining four experiments, and the remaining 370 pairs were automatically divided into 350 image pairs as the training set and 20 image pairs as the test set. To obtain sufficient medical image training samples, this embodiment employed a widely used random cropping strategy to enhance the training image data, collecting 17,500 training pairs of 120×120 pixels in each of the five experiments. Since the cropping strategy was used for data augmentation, the cropping was not used for the validation and test sets. Next, this embodiment used automatic sampling to sequentially extract 20 image pairs from each of the 97 pairs of PET and MRI images and the 120 pairs of CT and MRI images as test samples. This embodiment was deployed on an NVIDIA GeForce GTX 3090 GPU, implemented using the PyTorch framework, with the Adam optimizer set to a learning rate of 0.001, a batch size of 32, and an epoch of 10.

[0121] (2) Comparison of algorithms and evaluation metrics

[0122] Comparison Methods: Nine representative and advanced methods were compared with the method of this invention: CSMCA, MSMG-PCNN-EA, TL-ST, MLEPF, SwinFusion, MATR, YDTR, DATFuse, and CrossFuse. All comparison methods used were performed using the authors' publicly available code.

[0123] Evaluation metrics: This study used eight evaluation metrics to assess the performance of fused images from multiple perspectives: normalized mutual information (Q... MI), nonlinear correlation information entropy(Q NCIE ), image feature-based metric using phase congruency Q P The metrics include gradient-based metric (QG), visual information fidelity (VIF), multi-scale structural similarity index (MS-SSIM), FMI_dctand, and FMI_w. For all of these metrics, a higher value generally indicates a better medical image fusion result.

[0124] (3) Results and discussion of SPECT and MRI fusion

[0125] Figure 4 The image demonstrates the fusion of four sample pairs using different contrast methods. The two magnified local areas at the bottom of the image show varying degrees of information; these magnifications are provided for a more detailed comparison. Specifically, while the methods described above achieve relatively satisfactory fusion performance, they inevitably weaken the fundamental information of the source images compared to the method proposed in this invention. Figure 4 The demonstration showed that other competing methods have limited ability to preserve edge details and are prone to artifacts, affecting subjective visual perception. For example... Figure 4 The magnified images in Sample 3 show unwanted black artifacts appearing under the CSMCA, YDTR, Crossfusion, and MATR methods. The images fused from MSMG_PCNN_EA and MLEPF contain shadows and noise. The images fused from TL_ST, DATFuse, and Swinfusion lack detail clarity. It is worth noting that, due to the dynamic guidance of global information on local information and appropriate intensity control, the TBHNet model can preserve the rich texture details in MRI images to the greatest extent possible while fully representing the functional information in PET images.

[0126] Figure 5 Quantitative metrics for SPECT and MRI are presented. Specifically, the proposed TBHNet is quantitatively compared with nine representative and state-of-the-art methods in SPECT and MRI image fusion across eight objective evaluation metrics. The legend illustrates the mean score ± standard deviation of different methods on 100 pairs of samples under five-fold cross-validation. It is worth noting that in the MATR method legend, the '&' represents the mean of the 20 test samples presented in the original paper.

[0127] Figure 5The results show a quantitative comparison of 20 automatically selected SPECT and MRI image test sets in each experiment across eight widely used evaluation metrics. For each metric, the mean and standard deviation of 100 test samples are plotted in the legend, and the sample metric evaluation scores are presented as a line graph. The metric graphs show that QMI, QNCIE, Qp, QG, VIF, and FMI_w achieved the highest average scores across all 100 images. For MS_SSIM and FMI_dct, the method scored relatively lower, but the differences from other methods were small.

[0128] In summary, the TBHNet proposed in this invention has better fusion characteristics in preserving complementary information of the source images.

[0129] (4) Display of PET and MRI results

[0130] To explore the generalization performance of the TBHNet method proposed in this invention, we applied TBHNet directly to positron emission tomography (PET)-MRI and CT-MRI image fusion tasks without fine-tuning. A total of 100 test samples were used to comprehensively test the model performance using typical comparison methods and evaluation metrics.

[0131] Figure 6 Five typical PET and MRI image pairs are presented in detail. From the magnified bounding boxes in Sample 1 and Sample 3, it can be seen that, due to the lack of attention to global information and intensity in other methods, corresponding regions in the PET image that do not contain functional information are inevitably ignored, resulting in a low degree of preservation of soft tissue information from the MRI image in the fused image. In the results of other schemes, the texture details in the MRI image are weakened or ignored to varying degrees.

[0132] Figure 7 Five typical CT and MRI image pairs are presented in detail. Edge detail information in the full detail image of Sample 3 and the magnified region bounding image of Sample 5 in the RI image is reduced by DATFuse, YDTR, Swinfusion, and MATR. Furthermore, other schemes reduce the source image information to varying degrees. Conversely, as shown in Samples 1, 2, and 4, our TBHNet loses less functional metabolic information from CT images and retains texture and other information from MRI images. Overall, this provides physicians with more comprehensive auxiliary information to formulate appropriate treatment plans.

[0133] Figure 8Quantitative metrics for PET and MRI are presented. Note: The proposed TBHNet is quantitatively compared with nine representative and state-of-the-art methods in PET and MRI image fusion across eight objective evaluation metrics. The legend illustrates the mean score ± standard deviation of different methods on 100 pairs of samples under five-fold cross-validation. Notably, in the MATR method legend, the value after the '&' represents the mean of the 20 pairs of test samples presented in the original paper.

[0134] Figure 9 Quantitative metrics for CT and MRI are presented. Note: The proposed TBHNet is quantitatively compared with nine representative and state-of-the-art methods in CT and MRI image fusion across eight objective evaluation metrics. The legend illustrates the mean score ± standard deviation of different methods on 100 pairs of samples under five-fold cross-validation. Notably, in the MATR method legend, the value after the '&' represents the mean of the 20 pairs of test samples presented in the original paper.

[0135] from Figure 8 Objective evaluation metrics for PET and MRI showed that QMI, QNCIE, Qp, QG, VIF, and FMI_w scored highest across all 100 test samples, with averages reaching optimal levels. However, the average value for MS_SSIM was relatively low. Figure 9 Objective evaluation metrics from CT and MRI showed that the average scores for QMI, QNCIE, and Qp on the test samples all reached optimal levels. Based on a comparison of subjective quality and objective evaluation, the proposed method outperforms other comparative methods to a certain extent, indicating that TBHNet of this invention has good generalization ability.

[0136] (5) Ablation Research

[0137] The fusion performance of TBHNet in this invention relies on a carefully designed network architecture and loss function. On one hand, the proposed dual-brain mechanism enables dynamic interaction between global and local features across two modalities, effectively increasing the difference between feature representations. On the other hand, the proposed anisotropic mechanism effectively focuses on complementary features encompassed by functional and anatomical images from different levels. Furthermore, the complex loss function, composed of SSIM loss, region loss, and texture loss, better drives the network to focus on texture details and structural information. This invention conducted a series of ablation studies to verify the effectiveness and necessity of specific module designs. The visual results of the multimodal image fusion ablation experiments are shown below. Figure 10 As shown.

[0138] Heterogeneous Mechanism Analysis: Low-level deep learning models are better at capturing detailed features such as image edges, while high-level models tend to retain rich semantic features. SPECT and MRI images convey complementary specific information; therefore, constructing appropriate heterogeneous mechanisms to extract salient information from both images separately is crucial. For example... Figure 10 As shown in (d), when the input direction of Left hemispher2 is changed to be the same as that of Left hemispher1 (see the modified TBHNet model for details), Figure 11 However, the fusion result did not show a suitable visual improvement. Specifically, the existence of the anisotropic mechanism makes it easier to perceive complementary information in the two modal source images during the fusion process.

[0139] DAFM module analysis: The existence of dynamic progressive modules allows features to flow from high-level to low-level features, effectively mitigating the semantic gap between high-level and low-level features. For example... Figure 10 As shown in (e), we replaced DAFM_1, DAFM_2, and DAFM_3 with simple element-wise addition operations (see [link to modified TBHNet model] for details). Figure 12 Furthermore, fusion models cannot effectively control the texture structure details of the fused image, and may even exhibit artifacts. For example... Figure 10 As shown in (f), we reduced the structure of Left hemispher1 and Left hemispher2 and discarded them respectively. and This resulted in the dynamic progressive module retaining only DAFM_2 and DAFM_3, which reduced the amount of structural detail and intensity information contained in the fused image (see the modified TBHNet model for details). Figure 13 ).like Figure 10 As shown in (h), we will enable the dynamic progressive module to remove the two-sided multiplicative guiding factor of global features on local features. The presence of the multiplicative guiding factor in the fused image is beneficial for preserving the structural and intensity information in the fused image (see the modified TBHNet model for details). Figure 14 Overall, the presence of the dynamic progressive module makes it easier to capture the texture details contained in the MRI images during the fusion process, which helps to improve the quality of the fused images.

[0140] Exchange analysis under dual-brain mechanism: The initial Left hemispher1 and Left hemispher2 focus on the semantic features of SPECT images and the texture details of MRI images, respectively. To explore the heterogeneity of extracting multimodal features at different levels, we exchange the input images of Left hemispher1 and Left hemispher2, so that they focus on MRI images and SPECT images, respectively. Figure 10 (g) It can be seen that the fusion model obtained after swapping the input feature maps cannot effectively preserve the structural details in the source MRI images (see the modified TBHNet model for details). Figure 15 ).

[0141] Loss function component factor analysis: such as Figure 10 As shown in (i), L all1 The fusion result exhibits color distortion and lacks saliency information between the two modalities. For example... Figure 10 As shown in (j), L all2 The fusion results exhibit low contrast and blurriness. For example... Figure 10 As shown in (k), L all3 The fusion results are relatively high, but compared to Figure 10 (l) Some texture details, such as edges, are still missing. In summary, our TBHNet is equipped with SSIM loss, region loss, and texture loss, which can better drive the network to focus on texture details and structural information, resulting in more accurate fusion performance.

[0142] L all1 =L RMI +L Texture (8)

[0143] L all2 =L SSIM +L Texture (9)

[0144] L all3 =L SSIM +L RMI (10)

[0145] The quantitative results of the ablation study are shown in Table 1. The results show that each component has its necessity, and removing any one of them will reduce the fusion performance of TBHNet to varying degrees.

[0146] Table 1 Objective Indicators of Model Structure Ablation

[0147]

[0148] Note: In the table, (c) TBHNet (with loss function coefficients of α = 1.5, β = 1, γ = 1, δ = 1, and ε = 1), (d) co-directional network, (e) no DFAM_1, DFAM_2, and DFAM_3, (f) no DFAM_1, (g) input images of Left hemispher1 and Left hemispher2 swapped, (h) no dynamic mechanism, (i) L all1 (j)L all2 (k)L all3 (l)L all .

[0149] Loss function tradeoff parameter analysis: Quantitative results are shown in Table 2, where all scores represent the average scores of all samples in the validation set. It is evident that the fusion performance of the validation set is optimal when α = 1.5, β = 1, γ = 1, δ = 1, and ε = 1. Therefore, we set the default values ​​for α, β, γ, δ, and ε to 1.5, 1, 1, 1, and 1, respectively.

[0150] Table 2 Comparison of Loss Function Coefficients

[0151]

[0152]

[0153] This invention proposes an end-to-end bi-brain heterogeneous network, TBHNet, for multimodal medical image fusion. The method incorporates ODConv and an adaptive transformer as dual left-hemispheric branches to heterogeneously extract global complementary contextual information. To enable global features to dynamically guide local features for accurate fusion, a right-hemispheric branch composed of three DAFM modules is designed, effectively fusing high-level semantic features from functional images and low-level structural features from anatomical images. Furthermore, a loss function is designed from the perspectives of structural loss, region loss, and edge loss to train TBHNet. Extensive experiments demonstrate that our method outperforms other representative and state-of-the-art methods in both objective evaluation and subjective visual assessment. We also extend this method without fine-tuning to PET and MRI, and CT and MRI image fusion problems, showing its good generalization ability.

[0154] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A fusion device for multimodal medical images based on binaural brain networks, characterized in that, The fusion device includes: a bipolar anisotropic network module and a loss function module; The aforementioned bipolar network module includes: Left hemisphere1, Left hemispher2, and Righthemisphere; The Left hemisphere1 and Left hemisphere2 serve as backbone networks to extract global features respectively; the Left hemisphere1 includes: OD module, Transformer model1 module and Transformer model 2 module; the Left hemisphere2 includes: OD1 module, OD2 module and Transformerblock module. The Right hemisphere is a dynamically guided network used to extract local features, including: DAFM_1 module, DAFM_2 module and DAFM_3 module; The loss function module is used to train the bibrain heterotropic network, driving the network to focus on edge details and structural information; The loss function module uses the following formula: L all =L SSIM +L RMI +L Texture , Among them, L SSIM L RMI and L intensity These represent structural loss, region-level loss, and texture loss, respectively. The structural level loss is expressed by the following formula: ssim(·) represents the structural similarity operation; The regional-level loss is calculated using the following formula: In the formula, γ and δ are used as hyperparameters for balancing; RMI is defined as follows: In the formula, ε∈[0,1] is the weight parameter. Represents the source image I and the fused image The normal cross-entropy loss between them, where C represents the batch size. Represents the lower limit of mutual information; The texture loss is expressed using the following formula: In the formula, For the Prewitte gradient operator, |·| denotes the absolute operation, and |||1 denotes the l1 norm.

2. A method for fusing multimodal medical images based on bibrain heterotropic networks, characterized in that, The fusion apparatus according to claim 1 includes the following steps: (1) Use RGB-YCbCr color conversion to make the SPECT image I1∈R H×W×Cin1 Perform conversion to obtain and Three channel components; (2) and MRI images I2∈R H×W×Cin2 The images are used as inputs to Left hemisphere1 and Left hemisphere2, respectively, and then fused by the binaural anisotropic network module and loss function module. (3) Through and Implement YCbCr to RGB color conversion to generate the final fused image I. f ∈R H×W×Cout .

3. The fusion method according to claim 2, characterized in that, In step (2), Left hemisphere1 generates the feature map using the following formula: The Left hemisphere2 feature map is generated using the following formula: The aforementioned and The generation process uses the following formula: In the formula, Representing an image The feature map output by the OD module. express The output feature map after passing through the TransformerModel 1 module, express The output feature map after passing through the Transformer Model 2 module. express express The output feature map after passing through the Transformer Block module.

4. The fusion method according to claim 2, characterized in that, The Transformer model 1 module, Transformer model 2 module, and Transformerblock module are filter modules, using the following formula: In the formula, and These represent the input of the Transformer Model, the result of the first addition operation, and the result of the second addition operation, respectively. LN represents layer normalization, MSA represents multi-head self-attention mechanism, and MLP represents multi-layer perception mechanism.

5. The fusion method according to claim 2, characterized in that, In step (2), multiple stage feature maps of Left hemisphere1 and Left hemisphere2 are used as network inputs of Right hemisphere to extract local features.

6. The fusion method according to claim 5, characterized in that, The DAFM_1 module only accepts two adjacent feature maps from different modalities each time, fusing semantic information from high-level functional images and detailed information from low-level anatomical images; The DAFM_2 and DAFM_3 modules accept three adjacent feature maps; The DAFM_2 module uses the GCAB and LCAB modules for depth extraction, thereby improving the image representation capabilities.

7. The fusion method according to claim 6, characterized in that, The generation diagrams for each stage of the DAFM_1, DAFM_2, and DAFM_3 modules are generated using the following formulas: In the formula, τ1 is the DAFM_1 module, τ2 is the DAFM_2 module, and τ3 is the DAFM_3 module. and They have the same shape.

8. The fusion method according to claim 2, characterized in that, The fusion method described above can be used to fuse SPECT and MRI, PET and MRI, and CT and MRI images.

Images