A multi-modal medical image fusion method, system and device based on a Transformer and a dual-branch feature decoupling and a medium

Abstract

The application belongs to the technical field of medical image processing, and discloses a multi-modal medical image fusion method, system, device and medium based on a Transformer and double-branch feature decoupling, comprising: acquiring multi-modal medical images, performing normalization processing on the multi-modal medical images to obtain preprocessed multi-modal medical images; inputting the preprocessed multi-modal medical images into a medical image fusion model for image fusion to obtain a fusion image; wherein the medical image fusion model comprises shallow feature extraction modules, a double-modal cross attention module, a double-branch feature decoupling module and a splicing fusion module connected in sequence. The technical scheme disclosed by the application comprehensively fuses the features of double-modal medical images, significantly improves the image fusion effect, can better capture and combine the details and structural information between different modalities, and thus provides more accurate and comprehensive support for medical image analysis.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, and in particular relates to a multimodal medical image fusion method, system, device and medium based on Transformer and dual-branch feature decoupling. Background Technology

[0002] The development of medical imaging technology has provided doctors with a variety of means to observe the internal structure and function of the human body. From the initial X-ray imaging to modern computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and single-photon emission computed tomography (SPECT), medical images obtained by different imaging devices present different modalities. However, the information contained in a single modal image is limited, and it is usually difficult to comprehensively describe the imaging target, failing to meet the large amount of information required for clinical diagnosis. Therefore, in order to observe all the features of a part in a single image, it is necessary to extract effective information from multimodal medical images and fuse complementary information from multiple original medical images so that the fused image can provide a more comprehensive and reliable description of the lesion. By fusing source images of the same target obtained from different devices, a more information-rich and comprehensive fused image can be generated, which can obtain information about human organs and tissues that cannot be observed from a single modal image. This information can be used for the discovery, classification, and precise localization of abnormal lesions in human tissues, and has important scientific significance and practical application value for medical research, clinical diagnosis, surgical navigation, and prognosis evaluation.

[0003] Traditional multimodal medical image fusion methods typically rely on convolutional neural networks (CNNs) or transform domain methods for feature extraction and fusion. However, due to the inherent limitations of their local receptive fields, CNNs struggle to effectively capture long-range dependencies in medical images, resulting in insufficient global information modeling capabilities. In recent years, the application of Transformer architectures in computer vision tasks has made significant progress. Their self-attention mechanism can effectively model global feature relationships and achieve deeper modal complementarity during feature fusion. Existing multimodal image fusion methods often directly mix image features from different modalities, leading to information redundancy and optimization difficulties. Information flow between modalities is crucial in multimodal image fusion. However, existing methods often rely on simple channel stitching or attention weighting mechanisms, failing to adequately model the complementary relationships between modalities. This results in some modal information being overemphasized or ignored, thus affecting the quality of the fused image. Summary of the Invention

[0004] The purpose of this invention is to provide a multimodal medical image fusion method, system, device, and medium based on Transformer and dual-branch feature decoupling, so as to solve the problems existing in the prior art.

[0005] To achieve the above objectives, this invention provides a multimodal medical image fusion method based on Transformer and dual-branch feature decoupling, comprising:

[0006] Acquire multimodal medical images, and perform normalization processing on the multimodal medical images to obtain preprocessed multimodal medical images;

[0007] The preprocessed multimodal medical images are input into a medical image fusion model for image fusion to obtain a fused image; wherein, the medical image fusion model includes a shallow feature extraction module, a dual-modal cross-attention module, a dual-branch feature decoupling module, and a stitching fusion module connected in sequence.

[0008] Optionally, the normalization processing of the multimodal medical images specifically includes:

[0009] The multimodal medical images are format-converted, and the format-converted multimodal medical images are normalized to obtain preprocessed multimodal medical images.

[0010] Optionally, the training process of the medical image fusion model specifically includes:

[0011] Acquire training data, which includes multimodal medical training images and corresponding fusion results;

[0012] An initial medical image fusion model is constructed. The training data is input into the initial medical image fusion model to perform image fusion. The model is trained with the goal of minimizing the loss between the initial training result after image fusion and the fusion result corresponding to the multimodal medical training image, and a trained medical image fusion model is obtained.

[0013] Optionally, the processing procedure of the medical image fusion model specifically includes:

[0014] The preprocessed multimodal medical image is input into the shallow feature extraction module to enhance edge features and extract shallow features of the image through a general knowledge encoder to obtain a multi-scale initial feature representation.

[0015] The multi-scale initial feature representation is input into the dual-modal cross-attention module. The cross-modal feature interaction is enhanced by gradient extraction block and cross-attention block to obtain gradient enhancement features and cross-attention features. The obtained gradient enhancement features and cross-attention features are fused by adding them element by element to obtain weighted optimized fusion features.

[0016] The weighted optimized fusion features are input into the dual-branch feature decoupling module to extract global and local features;

[0017] The global and local features are input into the stitching and fusion module for feature stitching and reconstruction to generate a fused image.

[0018] Optionally, the processing procedure of the shallow feature extraction module specifically includes:

[0019] The initial features of the input multimodal medical image X are X0. Under the dense connection mechanism, features X are generated. l :

[0020] X l =H d ([X0,X1,...,X l-1 ])

[0021] In the formula, H d (·) represents a combination function of 3×3 convolution and Leaky ReLU activation;

[0022] Anisotropic Sobel operators are applied to a gradient enhancement unit to extract gradient information, thereby enhancing the structural contours and edge details of the image; let the Sobel filter K... x K y Gradient calculations for the X and Y directions respectively:

[0023] G x =W(X)*K x G y =W(X)*K y

[0024] Where W(X) represents the discrete wavelet transform of the input image, and G X G represents the gradient calculation result in the X direction. Y This is the gradient calculation result in the Y direction;

[0025] A residual fusion strategy is employed to combine the original features with enhanced gradient information to obtain the output features. Then, a general knowledge encoder is used to generate multi-scale initial feature representations.

[0026] Optionally, the processing procedure of the bimodal cross-attention module specifically includes:

[0027] A cross-modal complementary network is constructed, which includes a cross attention block and a gradient extraction block. The cross attention block is composed of a first cross attention block and a second cross attention block that interact with each other.

[0028] The initial feature representation is subjected to 3×3 convolution and ReLU activation function, and the input features of mode A and mode B are respectively used as the first gradient extraction block, the first cross attention block, the second cross attention block, and the element-wise summation component. The final summation result is the weighted optimized fusion feature.

[0029] Optionally, the processing procedure of the dual-branch feature decoupling module specifically includes:

[0030] The weighted optimized fusion features are input into the dual-branch feature decoupling module. The low-frequency common features of the input data are extracted by the global representation encoder to obtain the global features; the high-frequency specific features of the input data are extracted by the local texture encoder to obtain the local features.

[0031] Optionally, the processing procedure of the splicing and fusion module specifically includes:

[0032] Global and local features of different modalities are concatenated along the channel dimension to obtain fused global and fused local features.

[0033] The combined global features and local features are used to obtain the fused features;

[0034] The fused features are input into the decoder for feature reconstruction to generate the fused image.

[0035] A multimodal medical image fusion system based on Transformer and dual-branch feature decoupling includes:

[0036] The data acquisition module is used to acquire multimodal medical images and perform normalization processing on the multimodal medical images to obtain preprocessed multimodal medical images.

[0037] The image fusion module is used to input the preprocessed multimodal medical images into the medical image fusion model for image fusion to obtain a fused image; wherein, the medical image fusion model includes a shallow feature extraction module, a dual-modal cross-attention module, a dual-branch feature decoupling module, and a stitching fusion module connected in sequence.

[0038] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor running the computer program to cause the electronic device to perform a multimodal medical image fusion method based on Transformer and dual-branch feature decoupling as described above.

[0039] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned multimodal medical image fusion method based on Transformer and dual-branch feature decoupling.

[0040] The technical effects of this invention are as follows:

[0041] To address the problems of existing technologies, this invention proposes a multimodal medical image fusion method based on Transformer and dual-branch feature decoupling. First, an edge enhancement strategy is employed to process the initial input multimodal features, improving image detail. Then, the processed features are input into a general knowledge encoder to extract shallow features. Next, these shallow features are fed into a dual-modal cross-attention network containing gradient blocks and attention blocks to further extract and fuse intermodal correlation information. Based on this, this invention designs a dual-branch feature decoupling architecture, decoupling the global and local features of the two modalities respectively. Finally, the decoupled global and local features are fed into their respective fusion layers, and the final fused image is generated through feature reconstruction. This method significantly improves image fusion performance by comprehensively fusing features from dual-modal medical images, better capturing and combining detailed and structural information between different modalities, thus providing more accurate and comprehensive support for medical image analysis. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0044] Figure 1 This is a flowchart of the multimodal medical image fusion method based on Transformer and dual-branch feature decoupling in an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram of the overall multimodal medical image fusion method based on Transformer and dual-branch feature decoupling in an embodiment of the present invention;

[0046] Figure 3 This is a detailed schematic diagram of the edge enhancement module in the overall schematic diagram of the multimodal medical image fusion method based on Transformer and dual-branch feature decoupling in an embodiment of the present invention.

[0047] Figure 4This is a detailed schematic diagram of the bimodal cross-attention module in the overall schematic diagram of the multimodal medical image fusion method based on Transformer and bi-branch feature decoupling in an embodiment of the present invention.

[0048] Figure 5 This is a comparison of the visualization results between this embodiment and other algorithms on the Harvard multimodal medical image fusion dataset MRI-PET in this embodiment of the invention;

[0049] Figure 6 This is a comparison of the visualization results between this embodiment and other algorithms on the Harvard multimodal medical image fusion dataset MRI-CT in this embodiment of the invention;

[0050] Figure 7 This is a comparison of the visualization results between this embodiment and other algorithms on the MRI-SPECT dataset of the Harvard multimodal medical image fusion dataset. Detailed Implementation

[0051] Various exemplary embodiments of the present invention will now be described in detail. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and embodiments of the present invention.

[0052] It should be understood that the terminology used in this invention is merely for describing particular embodiments and is not intended to limit the invention. Furthermore, with respect to numerical ranges in this invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Every smaller range between any stated value or intermediate value within a stated range, and any other stated value or intermediate value within said range, is also included in this invention. The upper and lower limits of these smaller ranges may be independently included or excluded from the range.

[0053] Various modifications and variations can be made to the specific embodiments described in this specification without departing from the scope or spirit of the invention, as will be apparent to those skilled in the art. Other embodiments derived from this specification will also be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0054] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0055] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0056] like Figure 1 - Figure 7As shown, this embodiment provides a multimodal medical image fusion method based on Transformer and dual-branch feature decoupling, including: acquiring multimodal medical images; normalizing the multimodal medical images to obtain preprocessed multimodal medical images; inputting the preprocessed multimodal medical images into a medical image fusion model for image fusion to obtain a fused image; wherein, the medical image fusion model includes a shallow feature extraction module, a dual-modal cross-attention module, a dual-branch feature decoupling module, and a stitching fusion module connected in sequence.

[0057] To address the problems of existing technologies, a fusion method combining feature decoupling, CNN-Transformer interaction, and bimodal cross-attention modules is needed to ensure that the model can fully mine and fuse complementary information from multimodal medical images, thereby improving the global consistency and local detail representation of the fused images and achieving higher quality medical image fusion.

[0058] To achieve the above objectives, this embodiment proposes a multimodal medical image fusion method based on Transformer and dual-branch feature decoupling. First, an edge enhancement strategy is employed to process the initial input multimodal features, improving image detail. Then, the processed features are input into a general knowledge encoder to extract shallow features. Next, these shallow features are fed into a dual-modal cross-attention network containing gradient blocks and attention blocks to further extract and fuse intermodal correlation information. Based on this, this embodiment designs a dual-branch feature decoupling architecture, decoupling the global and local features of the two modalities respectively. Finally, the decoupled global and local features are fed into their respective fusion layers, and the final fused image is generated through feature reconstruction. This method significantly improves image fusion performance by comprehensively fusing features from dual-modal medical images, better capturing and combining detailed and structural information between different modalities, thereby providing more accurate and comprehensive support for medical image analysis.

[0059] This embodiment proposes a high- and low-frequency feature decoupling strategy, achieving independent modeling of global and local features through a dual-branch architecture, thereby enhancing the feature extraction capability of the fusion model while improving its flexibility and scalability. This embodiment also designs an efficient multimodal medical image fusion architecture combining CNN and Transformer to fully leverage the local feature extraction capabilities of CNN and the global modeling capabilities of Transformer, effectively improving the clarity and information integrity of the fused image. Finally, this embodiment proposes a Dual-Modality Cross-Attention (DMCA) module to enhance complementary information interaction between modalities, improve the quality of fused features, maintain the integrity of key information, and thus enhance the generalization ability of the fusion model.

[0060] The main steps of this embodiment are as follows:

[0061] Step 1: Preprocess the input multimodal medical images by converting the JPG format medical image data into h5 format and normalizing the data to improve the stability and effectiveness of subsequent feature extraction.

[0062] Step 2: Edge feature enhancement is performed on the normalized multimodal images, and then the images are input into a Transformer-based Universal Knowledge Encoder (UKE) to extract shallow features. The encoder uses convolutional mapping layers to transform the input features and generate multi-scale initial feature representations.

[0063] A feature enhancement network including an Edge Enhancement Module (EEM) is constructed. The Edge Enhancement Module consists of multi-scale feature extraction units and gradient enhancement units to fully preserve image edge features and optimize local contrast. Specifically, the multi-scale feature extraction units are composed of dense blocks. The initial features of the input image X are X0, and under a dense connection mechanism, features X are generated. l The generation process is as follows:

[0064] X l =H d ([X0,X1,...,X) l-1 ])

[0065] Among them, H d (·) represents a combination function of 3×3 convolution and Leaky ReLU activation, ensuring the stability and non-linear expressive power of feature extraction. The gradient enhancement unit applies the anisotropic Sobel operator to extract gradient information to enhance the structural contours and edge details of the image. Let the Sobel filter K... x K y Gradient calculations for the X and Y directions respectively:

[0066] G x =W(X)*K x G y =W(X)*K y

[0067] Here, W(X) represents the Discrete Wavelet Transform (DWT) of the input image to improve the robustness of the gradient features. Finally, the feature enhancement module employs a Residual Fusion Strategy to combine the original features with the enhanced gradient information to obtain the output features. Then, using UKE, multi-scale initial feature representations are generated.

[0068] Step 3, represent the initial features obtained in Step 2. The input is fed into the Dual-Modality CrossAttention (DMCA) module, which includes a gradient extraction block (GEB) and a cross attention block (CAB) to enhance cross-modal feature interaction.

[0069] A cross-modal complementary network (CMB) is constructed, comprising multiple cross-attention blocks. The features output from step 2 are processed through 3×3 convolutions and ReLU activation functions. The input features of modalities A and B are used as components of GEB, CAB1, CAB2, and element-wise summation, respectively, and the final summation result is the output. Specifically, the cross-attention blocks employ a key-query-value (KQV) mechanism to improve feature fusion through cross-modal information interaction. Each CAB consists of two interacting sub-modules (CAB1 and CAB2), processing features from different modalities respectively, and achieving information sharing and alignment through cross-modal attention computation. The computation method is as follows:

[0070]

[0071] The Attention mechanism is defined as follows:

[0072]

[0073] After CAB1 and CAB2 calculations, information from mode B is incorporated into the features of mode A, and information from mode A is incorporated into the features of mode B, thus achieving deep cross-modal interaction.

[0074] Finally, the gradient enhancement features and cross-attention features are fused through element-wise addition to obtain the optimized cross-modal feature representation:

[0075]

[0076] Step 4: Input the weighted optimized fusion features obtained in Step 3 into the Dual-Branch Feature Disentanglement module. The Global Representation Encoder (GRE) is used to extract low-frequency common features, and the Local Texture Encoder (LTE) is used to extract high-frequency specific features.

[0077] A dual-branch feature decoupling and adaptive representation modeling framework is constructed. A Transformer-based Global Representation Encoder (GRE) is used to extract common features, while an invertible transformation to a Local Texture Encoder (LTE) is introduced to accurately preserve high-frequency information. Furthermore, an adversarial feature decoupling loss ensures feature independence. Specifically, the features output from step 3 are... High-frequency and low-frequency features of the corresponding modality are extracted through two branches respectively. The global representation encoder adopts a self-attention mechanism, which enhances the cross-regional information correlation through global context modeling, thereby extracting low-frequency structural information. This embodiment introduces a Transformer-based feature encoder to perform global semantic interaction in multi-layer self-attention computation to preserve the overall scene structure while avoiding interference from local redundant information. Specifically, the input features... and The data is transformed into keys, queries, and values ​​through linear projection, and then a global feature representation is calculated using a multi-head self-attention (MHSA) mechanism.

[0078]

[0079] Where d is the feature dimension. After stacking multiple Transformer layers, the global feature representation... It is extracted. GRE captures long-range dependencies in the feature space and enhances the ability to model low-frequency structural information.

[0080] However, self-attention-based global representation methods are prone to losing local details during feature fusion. To address this, this embodiment introduces a Local Texture Encoder (LTE), constructed based on a neural network, to decouple and reconstruct high-frequency detail information with high fidelity. This embodiment employs a Conditional Reversible Transformation (CRT) to reconstruct the input features without introducing information loss. and Bidirectional mapping is performed. Due to the reversibility that ensures lossless information transmission, this embodiment can maintain the original high-frequency texture after decoupling and accurately restore details during reconstruction.

[0081] This embodiment introduces an adaptive feature disentanglement mechanism between GRE and LTE, ensuring the complementarity of local and global features through a dynamic feature transformation strategy. Specifically, this embodiment constructs a feature disentanglement adversarial loss (FDA-Loss) to encourage the minimization of mutual information between the two feature representations:

[0082]

[0083] Among them, D KL (·||·) represents the Kullback-Leibler divergence, used to measure the statistical distance between two feature distributions. This is achieved by minimizing D... KL (P(Φ GRE )||P(Φ LTE This embodiment makes the extracted features more independent in a statistical sense, avoids information omissions between modalities, thereby improving the discriminativeness and reconstructability of the final fused features, and ultimately achieves efficient decoupling of multimodal dual-branch features.

[0084] Step 5: Input the global and local features obtained from the dual-branch decoupling in Step 4 into the fusion layer for feature stitching and reconstruction, and finally generate the fused image.

[0085] For the two pairs of high- and low-frequency features obtained from the decoupling of the two modes in step 4, in order to fully explore the complementarity of these four features, this embodiment adopts a step-by-step fusion strategy, namely: fusion the high-frequency features of modes A and B. and Low-frequency features are sent to the local fusion layer. and Send to the global fusion layer.

[0086] In the local fusion layer, specifically, this embodiment splices the high-frequency features of mode A and mode B along the channel dimension:

[0087]

[0088] Since high-frequency features are often affected by noise and local variations, this embodiment introduces an Adaptive Feature Weighting (AFW) mechanism to calculate the weights of high-frequency features for each modality:

[0089]

[0090] Where σ(·) is the Sigmoid activation function, W LTE and b LTEThese are learnable parameters. To further enhance information representation capabilities, this embodiment employs a gated feature fusion mechanism to weighted combine high-frequency features:

[0091]

[0092] Similar to high-frequency feature fusion, this embodiment first stitches together low-frequency features along the channel dimension:

[0093]

[0094] Then, the adaptive weights for the low-frequency features are calculated:

[0095]

[0096] The fused low-frequency features are represented as follows:

[0097]

[0098] After fusing the high-frequency and low-frequency features, this embodiment needs to combine them to form the final fused feature:

[0099]

[0100] The final fused features are fed into the decoder for feature reconstruction to generate the fused image.

[0101] This embodiment addresses the shortcomings of existing multimodal medical image fusion methods in global and local information modeling, intermodal feature interaction, and edge detail preservation. It proposes a multimodal medical image fusion method based on Transformer and bi-branch feature decoupling. Finally, an adaptive fusion strategy enables the model to dynamically adjust the contribution of different modal information, thereby generating more accurate, clearer, and more stable fused images in medical image fusion tasks.

[0102] like Figure 1 The diagram shown is a flowchart of the multimodal medical image fusion method based on Transformer and dual-branch feature decoupling in this embodiment. Figure 2 , Figure 3 , Figure 4 The diagram shown is a schematic of the multimodal medical image fusion method based on Transformer and dual-branch feature decoupling in this embodiment, and a detailed schematic of the edge enhancement module EEM and the dual-modal cross-attention module DMCA included therein. The specific description is as follows:

[0103] (1) The input multimodal medical images are preprocessed by converting the JPG format medical image data into h5 format and normalizing the data to improve the stability and effectiveness of subsequent feature extraction.

[0104] (2) The multimodal image is input to the edge feature enhancement module in two paths. One branch of image features is processed through dense blocks, while the gradient enhancement unit of the other branch uses the anisotropic Sobel operator to extract gradient information to enhance the structural contours and edge details of the image. Finally, the feature enhancement module uses a residual fusion strategy to combine the original features with the enhanced gradient information to obtain the output features. The inputs are then fed into a Transformer-based general knowledge encoder to extract shallow features. The encoder uses convolutional mapping layers to transform the input features, generating multi-scale initial feature representations.

[0105] (3) Initial features are input into a cross-modal complementary network (CMB) comprising multiple cross-attention blocks. The bimodal features are processed by 3×3 convolution and ReLU activation functions, respectively. The input features of modal A and B are used as components of GEB, CAB1, CAB2, and element-wise addition, respectively, and the final sum of features is the output. Specifically, in the cross-attention blocks, a key-query-value (KQV) mechanism is used to improve the feature fusion effect through cross-modal information interaction. CAB consists of two interactive sub-modules (CAB1 and CAB2), which process features of different modalities respectively and achieve information sharing and alignment through cross-modal attention calculation. After calculation by CAB1 and CAB2, the features of modal A incorporate the information of modal B, and the features of modal B incorporate the information of modal A, realizing deep cross-modal interaction. Finally, the gradient enhancement features and cross-attention features are fused through element-wise addition to obtain the optimized cross-modal feature representation.

[0106] (4) The optimized features are input into the dual-branch feature decoupling module. The global representation encoder is used to extract low-frequency common features, and the local texture encoder is used to extract high-frequency specific features. The global representation encoder is based on the Transformer's self-attention mechanism, which can effectively extract global features. The local texture encoder uses conditional invertible transformation to refine the input features. and A bidirectional mapping is performed. Due to reversibility ensuring lossless information transmission, this embodiment can maintain the original high-frequency texture after decoupling and accurately recover details during reconstruction. This embodiment introduces an adaptive feature decoupling mechanism between GRE and LTE, ensuring the complementarity of local and global features through a dynamic feature transformation strategy.

[0107] The global and local features obtained through the two-branch decoupling are input into the fusion layer for feature stitching and reconstruction, ultimately generating a fused image.

[0108] The following specific examples will further illustrate this embodiment:

[0109] This embodiment uses the MSRS, TNO, and RoadScene datasets for training and testing, and performs method testing and generalization on the Harvard WholeBrainAtlas medical dataset. The specific dataset processing flow is as follows:

[0110] In this embodiment, 1083 image pairs were used for training and 361 image pairs were used for testing on the MSRS dataset. For the RoadScene dataset, 30 out of 50 image pairs were used for training and the remaining 20 pairs were used for testing. For the TNO dataset, 300 out of 361 image pairs were used for training and 61 pairs were used for testing. This embodiment preprocessed the images in these datasets, uniformly adjusting the image size to 256×256 pixels and normalizing the pixel intensity within the range [0,1].

[0111] The experimental results of this embodiment were qualitatively compared with other methods. The multimodal medical image fusion method based on Transformer and dual-branch feature decoupling achieved high-quality fusion performance on the Harvard Whole BrainAtlas medical dataset. Specifically, on the MRI-PET dataset, it is compared with the following networks: A Unified Unsupervised Image Fusion Network (U2Fusion), Cross-domain Long-range Learning for General Image Fusion via Swin Transformer (SwinFusion), Aversatile Squeeze-and-Decomposition Network for Real-Time Image Fusion (SDNet), A General Semantic-Guided Network with Couple Mask Ensemble for Medical Image Fusion (GeSeNet), a Generative Infrared and Visible Image Fusion Adversarial Network (FusionGAN), a Dual-Discriminator Condition Generative Adversarial Network for Multi-Resolution Image Fusion (DDcGAN), and a Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion. Compared to CDDFuse, the fusion quality is significantly improved (see Tables 1, 2, and 3). This embodiment's multimodal medical image fusion method based on Transformer and dual-branch feature decoupling effectively helps the model achieve good fusion performance.

[0112] Table 1. Comparison results with other methods using the Harvard Whole BrainAtlas medical dataset for MRI-PET.

[0113]

[0114]

[0115] Table 2 shows the comparison results between MRI-CT using the Harvard Whole BrainAtlas medical dataset and other methods.

[0116]

[0117] Table 3 shows the comparison results with other methods using the Harvard Whole BrainAtlas medical dataset for MRI-SPECT.

[0118]

[0119]

[0120] Specific visualization results are as follows: Figure 5 , Figure 6 , Figure 7 As shown in the results, the fusion results of this embodiment exhibit better intermodal information complementarity compared to other algorithms. Therefore, this embodiment employs the aforementioned multimodal medical image fusion method based on Transformer and dual-branch feature decoupling, achieving improved fusion quality by combining cross-modal image feature enhancement and cross-modal consistency learning.

[0121] This embodiment also provides a multimodal medical image fusion system based on Transformer and dual-branch feature decoupling, comprising:

[0122] The data acquisition module is used to acquire multimodal medical images and perform normalization processing on the multimodal medical images to obtain preprocessed multimodal medical images.

[0123] The image fusion module is used to input the preprocessed multimodal medical images into the medical image fusion model for image fusion to obtain a fused image; wherein, the medical image fusion model includes a shallow feature extraction module, a dual-modal cross-attention module, a dual-branch feature decoupling module, and a stitching fusion module connected in sequence.

[0124] In practice, this embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform a multimodal medical image fusion method based on Transformer and dual-branch feature decoupling as described above.

[0125] In practice, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned multimodal medical image fusion method based on Transformer and dual-branch feature decoupling.

[0126] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multimodal medical image fusion method based on Transformer and dual-branch feature decoupling, characterized in that, include: Acquire multimodal medical images, and perform normalization processing on the multimodal medical images to obtain preprocessed multimodal medical images; The preprocessed multimodal medical images are input into a medical image fusion model for image fusion to obtain a fused image; wherein, the medical image fusion model includes a shallow feature extraction module, a dual-modal cross-attention module, a dual-branch feature decoupling module, and a stitching and fusion module connected in sequence; The processing procedure of the medical image fusion model specifically includes: The preprocessed multimodal medical image is input into the shallow feature extraction module to enhance edge features and extract shallow features of the image through a general knowledge encoder to obtain a multi-scale initial feature representation. The multi-scale initial feature representation is input into the dual-modal cross-attention module. The cross-modal feature interaction is enhanced by gradient extraction block and cross-attention block to obtain gradient enhancement features and cross-attention features. The obtained gradient enhancement features and cross-attention features are fused by adding them element by element to obtain weighted optimized fusion features. The weighted optimized fusion features are input into the dual-branch feature decoupling module to extract global and local features; The global and local features are input into the stitching and fusion module for feature stitching and reconstruction to generate a fused image.

2. The method according to claim 1, characterized in that, The training process of the medical image fusion model specifically includes: Acquire training data, which includes multimodal medical training images and corresponding fusion results; An initial medical image fusion model is constructed. The training data is input into the initial medical image fusion model to perform image fusion. The model is trained with the goal of minimizing the loss between the initial training result after image fusion and the fusion result corresponding to the multimodal medical training image, and a trained medical image fusion model is obtained.

3. The method according to claim 1, characterized in that, The processing procedure of the shallow feature extraction module specifically includes: Input multimodal medical images The initial features are Under the dense connection mechanism, features are generated. : ， In the formula, This represents a combination function of 3×3 convolution and Leaky ReLU activation; Anisotropic Sobel operators are applied to a gradient enhancement unit to extract gradient information, thereby enhancing the structural contours and edge details of the image; a Sobel filter is then used. , Gradient calculations for the X and Y directions respectively: ， in, G represents the discrete wavelet transform of the input image. X G represents the gradient calculation result in the X direction. Y This is the gradient calculation result in the Y direction; A residual fusion strategy is employed to combine the original features with enhanced gradient information to obtain the output features. , Then, a general knowledge encoder is used to generate multi-scale initial feature representations. , .

4. The method according to claim 1, characterized in that, The processing procedure of the bimodal cross-attention module specifically includes: A cross-modal complementary network is constructed, which includes a cross attention block and a gradient extraction block. The cross attention block is composed of a first cross attention block and a second cross attention block that interact with each other. The initial feature representation is subjected to 3×3 convolution and ReLU activation function, and the input features of mode A and mode B are respectively used as the first gradient extraction block, the first cross attention block, the second cross attention block, and the element-wise summation component. The final summation result is the weighted optimized fusion feature.

5. The method according to claim 1, characterized in that, The processing procedure of the dual-branch feature decoupling module specifically includes: The weighted optimized fusion features are input into the dual-branch feature decoupling module. The low-frequency common features of the input data are extracted by the global representation encoder to obtain the global features; the high-frequency specific features of the input data are extracted by the local texture encoder to obtain the local features.

6. The method according to claim 1, characterized in that, The processing steps of the splicing and fusion module specifically include: Global and local features of different modalities are concatenated along the channel dimension to obtain fused global and fused local features. The combined global features and local features are used to obtain the fused features; The fused features are input into the decoder for feature reconstruction to generate the fused image.

7. A multimodal medical image fusion system based on Transformer and dual-branch feature decoupling, employing the method as described in any one of claims 1-6, characterized in that, include: The data acquisition module is used to acquire multimodal medical images and perform normalization processing on the multimodal medical images to obtain preprocessed multimodal medical images. The image fusion module is used to input the preprocessed multimodal medical images into the medical image fusion model for image fusion to obtain a fused image; wherein, the medical image fusion model includes a shallow feature extraction module, a dual-modal cross-attention module, a dual-branch feature decoupling module, and a stitching fusion module connected in sequence.

8. An electronic device, characterized in that, The device includes a memory and a processor, wherein the memory stores a computer program and the processor runs the computer program to enable the electronic device to perform a multimodal medical image fusion method based on Transformer and dual-branch feature decoupling according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements a multimodal medical image fusion method based on Transformer and dual-branch feature decoupling as described in any one of claims 1-6.

Images