A multi-branch medical image fusion method based on contrast constraint and IDFAM

Abstract

The application discloses a kind of multi-branch medical image fusion methods based on contrast constraint and IDFAM, it is related to image fusion technical field, including in coding layer stage, the source image of pair is respectively sent into private feature extraction branch, in private branch, multi-scale residual module and gradient dense connection module are designed, and it is alternately utilized, to effectively extract the unique feature of each modality;While source image is put into public feature extraction branch and extracts complementary information, a mechanism of information interaction is designed to ensure that complementary information is fully extracted;Secondly, attention mechanism for fusing deep features is constructed for public branch and private branch respectively, in fusion layer, the features extracted by the three paths are fused using Concatenation and convolution strategy.In decoding layer, we use a residual decoder network to reconstruct public and private features through the first three convolution layers of the decoder, and the final fused image is reconstructed by the last convolution layer of the decoder.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to a multi-branch medical image fusion method based on contrast constraints and IDFAM. Background Technology

[0002] Multimodal image fusion combines information from different imaging modalities to generate a single, information-rich, and high-quality image. Medical image fusion, as an important subfield of multimodal image fusion, is of great significance. Because the fused image can comprehensively describe a scene by integrating complementary information, medical image fusion technology has been widely applied in fields such as security, remote sensing, and clinical treatment.

[0003] Currently, medical imaging primarily includes Magnetic Resonance Imaging (MRI), Computed Tomography (CT), Single Photon Emission Computed Tomography (SPECT), and Positron Emission Tomography (PET). Different modalities of medical images contain different information and have their own advantages and disadvantages. MRI images clearly display soft tissue information but have limitations in detecting bone information; CT images effectively reflect bone information but cannot clearly display soft tissue information; SPECT images reflect the body's metabolic status but lack functional and structural information. During diagnosis, patients require multiple medical imaging sessions to obtain multimodal fused images, which may pose radiation hazards to their health. Furthermore, medical imaging is expensive, resulting in high costs. Therefore, there is an urgent need to introduce a multimodal medical image fusion method that can reduce the harm of medical imaging to patients and lower costs. By fully utilizing information from multiple modalities to achieve multimodal medical image fusion, we can not only enhance the complementarity between different modalities and eliminate redundancy between modalities, but also provide more comprehensive information for clinical diagnosis and treatment.

[0004] This invention provides a multi-branch medical image fusion method based on contrast constraints and IDFAM, which fuses medical images from different modalities to provide more feature information, helping doctors to better distinguish different tissues or lesions and improve diagnostic accuracy. Summary of the Invention

[0005] The purpose of this invention is to solve the above-mentioned problems by providing a multi-branch medical image fusion method based on contrast constraints and IDFAM.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] According to one aspect of the present invention, a multi-branch medical image fusion method based on contrast constraints and IDFAM is provided for image fusion, comprising the following steps:

[0008] (1) The registered images are processed by a feature extractor with a private branch, which includes two parallel MRI and CT / PET / SPECT feature extraction streams. Each feature extraction stream consists of two multi-scale residual modules and two gradient-dense residual modules. MSRB is constructed using ordinary convolutional layers to extract shallow features. Then, GRDB extracts fine-grained features from these shallow features, as follows:

[0009] The feature extraction module is used to extract deep features with rich fine-grained details from MRI and SPECT, as shown in the following formula:

[0010] {F M ,F S}={E F (I M ),E S (I S )} (1)

[0011] Among them, I M and I S These represent MRI features and SPECT features, respectively.

[0012] Furthermore, GRDB embedded in private branch feature extraction not only has the ability to extract high-level semantic features, but also enhances the description of fine-grained details. Given the input of GRDB, its output formula is as follows:

[0013]

[0014] In the formula, Conv(·) represents a convolutional layer, and Conv n (·) represents n cascaded convolutional layers. Gradient operator; This indicates element-wise summation.

[0015] The gradient operator convolves the input features with high-frequency convolution kernels to extract fine-grained details, and uses the Sobel and Laplace operators to calculate the gradient magnitude. It represents element-wise summation; GRDB aggregates learnable convolutional features with gradient magnitude information.

[0016] (2) Focus salient information on the feature information extracted from private and public branches using IDFAM;

[0017] The multimodal deep features extracted from the private branch are first fed into a channel attention module along the channel dimension. This module generates attention weights through convolution and pooling operations. These weights are then applied to the original features through element-wise multiplication and weighted together with the original features. Subsequently, the weighted features are added to the original features from another branch to enhance their corresponding features. This feature enhancement process is shown in equations (3)-(4).

[0018]

[0019] in This refers to summing element by element. Pω-Conv represents element-wise multiplication. n (·) represents n concatenated pointwise convolutional layers, C(·) represents the concatenation operation in the channel dimension, δ(·) and GAP(·) represent the sigmoid function and global average pooling, respectively. The enhanced features are then concatenated in the channel dimension and transported to the parallel channel attention and spatial attention modules to generate the final fusion weights. The fusion weight generation process is shown in equations (5)-(7):

[0020]

[0021] Since MRI and SPECT features are complementary, the generated weights are used for one modality, while the fusion weights for the other modality can be expressed as 1-W. i The surface feature fusion process is shown in equation (8):

[0022]

[0023] A cross-attention-based deep fusion module (CBCAM) is employed to integrate the deep features of common branches and rich contextual information for comprehensive understanding. CBCAM first uses dense layers to enhance the features extracted from the backbone and outputs enhanced deep features "F". M "and "F S Then, by deploying a projection function that includes convolution and shaping operations, the enhanced features are converted into keys and values, as shown in (9)-(10):

[0024]

[0025] Where x∈{M,S} represents a mode. Indicates key, The values ​​are represented by H. Conv(·) and Reshape(·) correspond to a 3×3 kernel-sized convolutional layer and a reshaping operation, respectively. i W i and C i These refer to the input features respectively. Height, width and passage,

[0026] Both MRI and SPECT features are merged to generate modality-invariant features, utilizing the complementary properties of multimodal features, as shown in Equation (11):

[0027]

[0028] in,

[0029] Then, the modality-specific attention map for each modality x is calculated according to the following formula.

[0030]

[0031] Subsequently, this value is multiplied by the attention to obtain features with global context. Similar to PBCDAM, the global features are added to the original features of another branch and along the channel dimension of the connected features. Finally, the cascaded features are fed into the convolutional layer to obtain fused features, as described in Equation (13):

[0032]

[0033] (3) The images extracted by the IDFAM attention mechanism are merged using the Concation fusion strategy to ensure the preservation of complementary information and obtain high-quality fused images;

[0034] (4) In the decoding layer, the common features extracted from it are compared with the source image I in the last layer of the decoding part. S and source image I P The private features are combined to reconstruct the source image I separately. S and I P Residual-based decoder networks are used in image reconstruction tasks to improve reconstruction results and generate output images that are as similar as possible to the input images.

[0035] (5) Design of loss function

[0036] The loss function consists of three types of loss terms: content loss, intensity loss, and gradient loss, as shown in formula (14):

[0037] L = L int +α1L grad +α2L text (14)

[0038] Among them, L grad L represents the gradient loss. int L represents the strength loss. textThis represents the content loss, where α1 and α2 are tuning parameters.

[0039] (6) Content loss

[0040] Content loss includes structural similarity loss and relevance consistency loss. Based on these two losses, a relevance consistency loss function is proposed to further enhance the sharing of information between modalities, making the basic features across modalities related, while weakening the correlation between detailed high-frequency features in different modalities. The content loss is shown in formula (15):

[0041] L text =L SSIM +βL CC (15)

[0042] Among them, L SSIM L represents the structural similarity loss. CC Represents the relevance consistency loss, with β used to balance structural similarity and relevance consistency terms:

[0043] Specifically, the structural similarity loss is shown in formula (16):

[0044] L SSIM =1-SSIM (16)

[0045] SSIM is the structural similarity index:

[0046] The relevant consistency loss is shown in Equation (17):

[0047]

[0048] Here, CC is the correlation coefficient operator, and ε is set to 1.01 to ensure that the term is always positive;

[0049] feature This includes common modal information encompassing background and large-scale environment;

[0050] (7) Gradient loss

[0051] An adaptive weight block is introduced into the gradient loss function. The adaptive weight block first uses a Gaussian low-pass filter for linear smoothing to minimize the impact of noise on the weight block. Then, the optimization objective of the corresponding pixels in the source image is evaluated based on the gradient richness.

[0052] The gradient loss formula is as follows:

[0053]

[0054] In the formula, i and j represent the weight map of the pixel in the i-th row and j-th column, H and W represent the height and width of the image, I1 and I2 are the source images, and Ifused Represents the fused image, |·| represents the absolute value function. Let L be the Laplace operator, L(·) denote the Gaussian low-pass filter function, min(·) denote the minimum function, and sign(·) denote the sign function;

[0055] (8) Strength loss

[0056] The formula for strength loss is as follows:

[0057]

[0058] In the formula, i and j represent the weight map of the pixel in the i-th row and j-th column, H and W represent the height and width of the image, I1 and I2 represent the source image, and I fused Represents a fused image.

[0059] Furthermore, MSRB consists of ordinary convolutional layers with a kernel size of 3×3 and an activation function of leakage correction linear units. Specifically, it is a lightweight module with a stride of 1 and no edge padding. It uses two convolutional layers with a kernel size of 3×3 to replace a single convolutional layer with a kernel size of 5×5, and three convolutional layers with a kernel size of 3×3 to replace a single convolutional layer with a kernel size of 7×7. Ordinary convolutional layers with an activation function of LReLU are used to extract shallow features.

[0060] Furthermore, GRDB is a variant of densely connected modules and residual blocks, employing dense connections while integrating gradient operations;

[0061] The main part of GRDB uses two convolutional layers with a kernel size of 3×3 and an activation function of LReLU, and a shared convolutional layer with a kernel size of 1×1 and Tanh as the activation function.

[0062] The mainstream approach introduces dense connections to fully utilize convolutional layers for feature extraction. The residual flow uses gradient operations to calculate the gradient magnitude of features and employs ordinary convolutional layers with 1×1 kernels to eliminate channel dimension differences. Then, the outputs of the main dense flow and the residual gradient flow are combined through element-wise addition to integrate deep features and fine-grained detail features.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] (1) A contrast-constrained multi-branch encoder network is proposed to learn the common and private features of paired images. A three-branch autoencoder is designed, with two branches for learning the private features of different modalities and one branch for learning the common features. At the same time, a multi-scale residual module and a gradient-dense module are designed to extract the unique features of each source image in the private branch, thereby obtaining richer detail information.

[0065] (2) Design an information interaction module to extract complementary information between different modal images in the public branch, so as to better capture the common functional and structural information in the source image. This invention utilizes the separated public and private features to improve the interpretability of the model and help to obtain more accurate fused images.

[0066] (3) Attention mechanisms for fusing deep features were constructed for private and public branches respectively. Considering that the unique features of the source image contain rich details and structural information, this invention proposes PACDAM to extract the unique features of each modality's deep layers; since the extraction of public branch features usually requires rich contextual information for a comprehensive understanding, CDCAM was designed to integrate the deep features of the public branch.

[0067] (4) A correlation consistency loss is introduced to ensure the effective separation of modality-shared information and private features. Comodal information is highly correlated, while complementary information is less correlated. This loss function is designed to ensure that public and private features contain more modality-shared information, effectively suppress redundant information, improve the quality of extracted features, and thus enhance overall performance. Attached Figure Description

[0068] Figure 1 Here is a flowchart of a multi-branch medical image fusion method based on contrast constraints and IDFAM;

[0069] Figure 2 This is a schematic diagram of the private feature extraction branch of the present invention;

[0070] Figure 3 This is the multi-scale residual module of the present invention;

[0071] Figure 4 This is the gradient residual dense block of the present invention;

[0072] Figure 5 This invention relates to a cross-dimensional interactive attention mechanism based on private branch fusion of deep features;

[0073] Figure 6 This invention relates to a cross-attention mechanism based on the fusion of deep features from common branches.

[0074] Figure 7 This invention relates to a decoder network based on a residual network.

[0075] Figure 8 Qualitative comparison of fused images obtained using 10 methods (CT-MRI) according to the present invention;

[0076] Figure 9 Qualitative comparison of fused images obtained using 10 methods (PET-MRI) according to the present invention;

[0077] Figure 10 This is a qualitative comparison of fused images obtained using 10 methods (SPECT-MRI) according to the present invention. Detailed Implementation

[0078] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0079] This invention provides a multi-branch medical image fusion method based on contrast constraints and IDFAM. The process of the multi-branch medical image fusion method based on contrast constraints and IDFAM is as follows: Figure 1 As shown. In the encoding layer stage, paired source images are fed into private feature extraction branches. In these branches, multi-scale residual modules and gradient-dense connection modules are designed and used alternately to effectively extract unique features from each modality. Simultaneously, source images are fed into the public feature extraction branch to extract complementary information. An information interaction mechanism is designed to ensure sufficient extraction of complementary information. Secondly, attention mechanisms for combining deep features are constructed for both the public and private branches. In the fusion layer, features extracted from the three paths are fused using concatenation and convolution strategies. In the decoding layer, a residual decoder network is used. Public and private features are reconstructed through the first three convolutional layers of the decoder, and the final fused image is reconstructed through the last convolutional layer of the decoder. The specific implementation steps are as follows:

[0080] (1) The registered images are processed through a private branch feature extractor, which includes two parallel MRI and CT / PET / SPECT feature extraction streams. Each feature extraction stream consists of two Multi-Scale Residual Modules (MSRBs) and two Gradient Residual Dense Blocks (GRDBs). MSRBs are constructed using ordinary convolutional layers with a kernel size of 3×3 and an activation function of Leaky Rectified Linear Units (LReLU) to extract shallow features. Then, GRDBs extract fine-grained features from these shallow features. By alternating between MSRBs and GRDBs, local features are extracted more fully. The structure of the private branch feature extraction is as follows: Figure 2 As shown.

[0081] Specifically, the MSRB design is a lightweight module with a stride of 1 and no edge padding. It uses two 3×3 convolutional layers instead of a single 5×5 convolutional layer, and three 3×3 convolutional layers instead of a single 7×7 convolutional layer. This invention uses ordinary convolutional layers with a kernel size of 3×3 and an LReLU activation function to extract shallow features. The MSRB structure is as follows: Figure 3 As shown.

[0082] GRDB is a variant of densely connected modules and residual blocks, primarily employing dense connections while integrating gradient operations. The main part of GRDB uses two convolutional layers with 3×3 kernels and LReLU activation function, along with a shared convolutional layer with a 1×1 kernel and Tanh activation function. Dense connections are introduced to fully utilize the convolutional layers for feature extraction. The residual stream uses gradient operations to calculate the gradient magnitude of features and employs ordinary convolutional layers with 1×1 kernels to eliminate channel dimension differences. Then, the outputs of the main dense stream and the residual gradient stream are combined using element-wise addition, thus integrating deep features and fine-grained detail features. GRDB, as shown... Figure 4 As shown.

[0083] Specifically, a feature extraction module is used to extract deep features with rich fine-grained detail from MRI and SPECT, which can be represented as:

[0084] {F M ,F S}={E F (I M ),E S (I S )} (1)

[0085] Among them, I M and I S These represent MRI features and SPECT features, respectively.

[0086] Furthermore, GRDB embedded in private branch feature extraction not only possesses the ability to extract high-level semantic features but also enhances the description of fine-grained details. Given the input of GRDB, its output can be represented as:

[0087]

[0088] Where Conv(·) represents a convolutional layer, Conv n (·) represents n cascaded convolutional layers. The gradient operator refers to a special type of convolution operation whose kernel is manually designed. The gradient operator convolves input features with high-frequency kernels to extract fine-grained details. This invention uses the Sobel and Laplace operators to calculate the gradient magnitude. Furthermore, This indicates element-wise summation. GRDB combines learnable convolutional features with gradient magnitude information.

[0089] (2) The feature information extracted from the private and public branches is focused on salient information using IDFAM. IDFAM can adaptively select and integrate hierarchical features, focusing on key parts of the input image, suppressing the influence on redundant information and noise, and more effectively preserving key information of multimodal medical images. Since the channel attention mechanism mainly focuses on salient information by increasing weights to improve the generalization ability of the network, it cannot reflect spatial dimension information. The spatial attention mechanism can enhance important information in spatial location but fails to fully extract channel information. Therefore, this invention proposes a channel-space cross-dimensional interactive attention mechanism module (PBCDAM) to fuse deep features extracted from the private branch. PBCDAM is as follows: Figure 5 As shown.

[0090] The multimodal deep features extracted from the private branch are first fed into a channel attention module along the channel dimension. This module generates attention weights through convolution and pooling operations. These weights are then applied to the original features through element-wise multiplication and weighted together with the original features. Subsequently, the weighted features are added to the original features from another branch to enhance their corresponding features. This feature enhancement process is shown in equations (3)-(4):

[0091]

[0092]

[0093] in This refers to summing element by element. Pω-Conv represents element-wise multiplication. n (·) represents n concatenated pointwise convolutional layers, C(·) represents the concatenation operation in the channel dimension, and δ(·) and GAP(·) represent the sigmoid function and global average pooling, respectively. The enhanced features are then concatenated in the channel dimension and fed into parallel channel attention and spatial attention modules to generate the final fusion weights. The fusion weight generation process is shown in equations (5)-(7):

[0094]

[0095] Since MRI and SPECT features are complementary, the generated weights are used for one modality, while the fusion weights for the other modality can be expressed as 1-W. i The surface feature fusion process is shown in equation (8):

[0096]

[0097] Since common branch feature extraction typically requires rich contextual information for comprehensive understanding, a deep fusion module based on cross-attention (CBCAM) was designed to integrate the deep features of common branches, such as... Figure 6 As shown, CBCAM first employs dense layers to enhance the features extracted from the backbone and outputs enhanced depth features "F". M "and "F S Then, by deploying a projection function that includes convolution and shaping operations, the enhanced features are converted into keys and values, as shown in (9)-(10):

[0098]

[0099] Where x∈{M,S} represents a mode. Indicates key, The values ​​are represented by `Conv(·)` and `Reshape(·)`. `Conv(·)` and `Reshape(·)` correspond to a 3×3 kernel-sized convolutional layer and a reshaping operation, respectively. H i W i and C i These refer to the input features respectively. The height, width, and channels. Notably, both MRI and SPECT features are combined to generate our modality-invariant features, which allows us to fully utilize the complementary properties of the multimodal features, as shown in Equation (11).

[0100]

[0101] in,

[0102] Then, we calculate the modality-specific attention map for each modality x according to the following formula. As shown in equation (12).

[0103]

[0104] Subsequently, this value is multiplied by the attention to obtain features with global context. Similar to PBCDAM, we add the global features to the original features of another branch and along the channel dimension to obtain the features obtained from the connections. Finally, we feed the cascaded features into the convolutional layer to obtain fused features. This process can be expressed as (13):

[0105]

[0106] (3) The images extracted by the IDFAM attention mechanism are merged using the Concation fusion strategy to ensure the preservation of complementary information and obtain high-quality fused images.

[0107] (4) In the decoding layer, the common features extracted from it are compared with the source image I in the last layer of the decoding part. S and source image I P The private features are combined to reconstruct the source image I separately. S and I P Residual-based decoder networks are used in image reconstruction tasks to improve reconstruction results and generate output images that are as similar as possible to the input image. The decoder network consists of ordinary convolutional layers with a kernel size of 3×3 and the LReLU activation function. For example... Figure 7 As shown.

[0108] (5) Design of loss function

[0109] The encoder network generates a single fused image by extracting intensity and gradient information, while the decoder network reconstructs the intensity and gradient information to generate an image that is as consistent as possible with the source image. Intensity loss constrains the fused image to maintain an intensity distribution similar to the source image, gradient loss forces the fused image to contain rich texture details, and content loss ensures that the generated fused image retains the key features of the input image, making the fusion result contain richer scene content, thereby achieving a better fusion effect. Therefore, the loss function of this invention consists of three types of loss terms: content loss, intensity loss, and gradient loss, as shown in formula (14).

[0110] L = L int +α1L grad +α2L text (14)

[0111] Among them, L grad L represents the gradient loss. int L represents the strength loss. text This represents the content loss, where α1 and α2 are tuning parameters.

[0112] (6) Content loss

[0113] Content loss includes structural similarity loss and relevance consistency loss. This invention proposes a relevance consistency loss function, which aims to further enhance the sharing of information between modalities, making the basic features across modalities related, while simultaneously weakening the correlation between detailed high-frequency features in different modalities. The content loss is shown in Equation (15).

[0114] Ltext =L SSIM +βL CC (15)

[0115] Among them, L SSIM L represents the structural similarity loss. CC This represents the relevant consistency loss, and β is used to balance structural similarity and relevant consistency terms.

[0116] Specifically, the structural similarity loss is shown in Equation (16).

[0117] L SSIM =1- SSIM (16)

[0118] SSIM is the structural similarity index.

[0119] The relevant consistency loss is shown in Equation (17).

[0120]

[0121] Here, CC is the correlation coefficient operator, and ε is set to 1.01 to ensure that the term is always positive.

[0122] feature They contain more common-modal information, such as context and large-scale environment, and are therefore typically highly correlated. In contrast, Features in PET / SPECT images primarily represent high brightness or color information, while features in MRI images mainly represent soft tissue and well-defined edges; these are modality-specific. Therefore, the correlation between these feature maps is low. Based on past experience, during gradient descent... Gradually approaching 0, It becomes larger, which aligns with our intuition about feature reconstruction.

[0123] (7) Gradient loss

[0124] This invention employs an adaptive gradient loss L grad The aim is to ensure that the fused image contains rich texture features. An adaptive weight block is introduced into the gradient loss. The adaptive weight block is first linearly smoothed using a Gaussian low-pass filter to minimize the impact of noise on the weight block, and then evaluated based on the optimization objective of the corresponding pixels in the source image according to the gradient richness. The goal of this process is to guide the texture of the fused image to be as consistent as possible with the strongest texture of the source image at the corresponding location.

[0125] The gradient loss formula is as follows:

[0126]

[0127] In the formula, i and j represent the weight map of the pixel in the i-th row and j-th column, H and W represent the height and width of the image, I1 and I2 are the source images, and I fused Represents the fused image, |·| represents the absolute value function. Let L be the Laplace operator, L(·) denote the Gaussian low-pass filter function, min(·) denote the minimum function, and sign(·) denote the sign function.

[0128] (8) Strength loss

[0129] Strength loss L int The goal is to constrain the fused image to maintain a similar intensity distribution to the source image, which helps the fused image retain useful information such as the contrast represented by the pixel intensity, while making the overall scene style of the fused image more natural and realistic.

[0130] The intensity loss can be formalized as follows:

[0131]

[0132] In the formula, i and j represent the weight map of the pixel in the i-th row and j-th column, H and W represent the height and width of the image, I1 and I2 represent the source image, and I fused Represents a fused image.

[0133] The key algorithms of this invention are a three-branch autoencoder network, an attention mechanism fusing deep features, and a residual-based decoder network. Compared with currently popular multimodal medical image fusion algorithms, this algorithm shows a significant performance improvement. In the encoding stage, private feature extraction branches and public feature extraction branches are constructed to extract features from paired source images, respectively. To fully extract detailed information, MSRB and GRDB are designed in the private branch and used alternately to effectively extract unique features of each modality. For the public branch, an information interaction module is designed to ensure the full extraction of complementary information. Finally, attention mechanisms based on the fusion of deep features from the private and public branches are designed respectively to effectively filter redundant information by focusing on important features. In the decoding stage, the residual-based decoder network is used to improve the reconstruction effect in the image reconstruction task, generating an output image that is as similar as possible to the input image. The combined application of a three-branch autoencoder network, an attention mechanism that integrates deep features, and a residual-based decoder network enables multimodal medical image fusion. Furthermore, it demonstrates the unique advantages of this model in image processing, resulting in more refined texture detail and contrast processing, lower algorithm complexity, and optimization of existing methods and upgrades to existing technologies.

[0134] Comparing experimental data and evaluation results

[0135] To demonstrate the superior performance of this invention and the practicality of the proposed fusion model, 200 pairs of 256×256 medical images from the publicly available Harvard Medical Database (http: / / www.med.harvard.edu / aanlib / home.html) were selected and cropped into 39,200 120×120 image patches for training. Since this invention is a fully convolutional network, the source images did not need to be cropped to the same size as the training data during the testing phase. Experimental validation was conducted using 166 sets of images, including 42 sets of CT-MRI medical images, 42 sets of PET-MRI medical images, 52 sets of SPECT-MRI medical images, and 30 sets of infrared and visible light images. The medical image slices were all from the publicly available Harvard Medical Database and included real-world case examinations of: mild Alzheimer's disease, multiple embolic infarcts, Lyme disease encephalopathy, and cavernous hemangioma.

[0136] The experimental setup included 30 epochs, a batch size of 32, and a learning rate of 1e-4. For the optimization algorithm, we chose the adaptive optimization algorithm Adam, which is highly adaptable and has good convergence. In Equation 15, β is set to 2, minimizing the content loss between the fused image and the original image to generate a more realistic fused image that is closer to the original, reducing distortion and redundancy. In Equation 21, λ is set to 0.5, extracting the main intensity information from the input image to maintain significant contrast. Furthermore, the proportions of content loss, gradient loss, and intensity loss in the total loss were set to 2, 10, and 1 respectively after repeated experiments.

[0137] Subjective visual evaluation and objective data comparison were conducted with nine high-level fusion methods: SDNet-based fusion method, EMFusion-based fusion method, U2Fusion-based fusion method, MATR-based fusion method, CFL-based fusion method, SwinFusion-based fusion method, DFENet-based fusion method, MCAFusion-based fusion method, and GeSeNet-based fusion method.

[0138] This invention provides a multimodal medical image fusion performance evaluation method that includes both subjective visual evaluation and objective index evaluation. To fully demonstrate the superior fusion performance of the proposed algorithm framework, information entropy (EN), mutual information (MI), and normalized mutual information (Q) were selected as objective indexes for evaluation. MI ), Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), Average Gradient (AG), Spatial Frequency (SF), Fusion Quality (Q) AB / FObjective metrics are used to evaluate the results of the fused images, further illustrating the effectiveness of our method. EN measures the amount of information contained in the image; a higher value indicates richer information in the fused result. MI evaluates the change in information content before and after image fusion; a higher value indicates a greater amount of information in the fused image. Q... MI A higher value indicates more information extracted from the source image, resulting in better fusion. SSIM measures the structural consistency between image patches; a higher value indicates a more consistent structure between the fused image and the source image. PSNR measures the degree of image quality loss; a higher value indicates less image quality loss. AG reflects the sharpness of the fused image; a higher value indicates a sharper image. SF evaluates detail preservation and image sharpness; a higher value indicates a sharper image. AB / F This is an estimate of the degree of information representation in the fused image; a larger value indicates a greater amount of information and better image quality. Experimental data on disease features from three different source images are shown in Tables 1, 2, and 3.

[0139] Table 1. Mean values ​​of evaluation indicators for 42 groups of CT-MRI images

[0140]

[0141]

[0142] Table 2 Mean values ​​of evaluation indicators for 42 groups of PET-MRI images

[0143]

[0144] Table 3. Mean values ​​of SPECT-MRI evaluation indicators for 52 groups

[0145]

[0146] The results of the eight index values ​​in Tables 1, 2, and 3 show that the AG, EN, SF, MI, and Q values ​​of the multimodal medical image fusion method proposed in this invention are... AB / F and Q MIWhile achieving good performance across various metrics, PSNR fell short, primarily because it's calculated using signal-to-noise ratio (SNR) and doesn't account for the non-linear characteristics of human vision. This invention improves image naturalness while reducing SNR through adjustments to color, contrast, and brightness, making the image more consistent with human visual perception. This invention is more sensitive to color information, whereas PSNR, which primarily considers grayscale accuracy, suffers from reduced quality due to grayscale distortion. Therefore, compared to other comparison algorithms, this invention prioritizes color richness in PET and SPECT images, edge contour and soft tissue information in MRI images, and skeletal information in CT images during the fusion process. The fused image exhibits clearer edges, more natural colors, and better conforms to human visual characteristics.

[0147] Figures 8 to 10 The images show a comparison of different methods for fusing images of different diseases. Sub-images (a) and (b) are the source images; sub-image (c) shows a medical image fusion algorithm based on SDNet; (d) shows a medical image fusion algorithm based on EMFusion; (e) shows a medical image fusion algorithm based on U2fusion; (f) shows a medical image fusion algorithm based on MATR; (g) shows a medical image fusion algorithm based on CFL; (h) shows a medical image fusion algorithm based on SwimFusion; (i) shows a medical image fusion algorithm based on DFENet; (j) shows a medical image fusion algorithm based on MCAFusion; (k) shows a medical image fusion algorithm based on GeSeNet; and (l) shows the result of the multimodal medical image fusion algorithm of this invention. In summary, the fused image obtained by the method of this invention is significantly better than other fusion algorithms in terms of both subjective perception and objective evaluation, indicating that the experimental results of the method of this invention are better.

[0148] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0149] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A multi-branch medical image fusion method based on contrast constraints and IDFAM, used for image fusion, characterized in that: Includes the following: (1) The registered images are processed by a feature extractor with a private branch, which includes two parallel MRI and CT / PET / SPECT feature extraction streams. Each feature extraction stream consists of two multi-scale residual modules and two gradient-dense residual modules. MSRB is constructed using ordinary convolutional layers to extract shallow features. Then, GRDB extracts fine-grained features from these shallow features, as follows: The feature extraction module is used to extract deep features with rich fine-grained details from MRI and SPECT, as shown in the following formula: (1) in, and These represent MRI features and SPECT features, respectively. Furthermore, GRDB embedded in private branch feature extraction not only has the ability to extract high-level semantic features, but also enhances the description of fine-grained details. Given the input of GRDB, its output formula is as follows: (2) In the formula, Indicates a convolutional layer. Represents n cascaded convolutional layers. Gradient operator; This indicates element-wise summation. The gradient operator convolves the input features with high-frequency convolution kernels to extract fine-grained details, and uses the Sobel and Laplace operators to calculate the gradient magnitude. It represents element-wise summation; GRDB aggregates learnable convolutional features with gradient magnitude information. (2) Focus salient information on the feature information extracted from private and public branches using IDFAM; The multimodal deep features extracted from the private branch are first fed into a channel attention module in the channel dimension. This module generates attention weights through convolution and pooling operations. Then, the attention weights are applied to the original features through element-wise multiplication and weighted with the original features. Subsequently, the weighted features are added to the original features from another branch to enhance their corresponding features. The feature enhancement process is shown in equations (3)-(4): (3) (4) in This refers to summing element by element. This indicates element-wise multiplication. Represents n cascaded pointwise convolutional layers. This indicates a cascading operation within the channel dimension. and Let represent the sigmoid function and global average pooling, respectively. Then, the enhanced features are concatenated in the channel dimension and transported to the parallel channel attention and spatial attention modules to generate the final fusion weights. The fusion weight generation process is shown in equations (5)-(7): (5) (6) (7) Since MRI and SPECT features are complementary, the generated weights are used for one modality, while the fusion weights for the other modality are expressed as follows: The surface feature fusion process is shown in equation (8): (8) A cross-attention-based deep fusion module (CBCAM) is employed to integrate the deep features of common branches and rich contextual information for comprehensive understanding. CBCAM first uses dense layers to enhance the features extracted from the backbone and outputs enhanced deep features. "and" Then, by deploying a projection function that includes convolution and shaping operations, the enhanced features are converted into keys and values, as shown in (9)-(10): (9) (10) in, Represents mode, Indicates key, Represents value, and These correspond to a 3×3 kernel-sized convolutional layer and a shaping operation, respectively. , and These refer to the input features respectively. Height, width and passage, Both MRI and SPECT features are merged to generate modality-invariant features, utilizing the complementary properties of multimodal features, as shown in Equation (11): (11) in, ; Then, calculate each mode according to the following formula. Modality-specific attention graph , (12) Then Multiply by attention to obtain features with global context. Similar to PBCDAM, add the global features to the original features of another branch and along the channel dimension to obtain the features obtained from the connection. Finally, feed the cascaded features into the convolutional layer to obtain fused features, as described in Equation (13): (13) (3) The images extracted by the IDFAM attention mechanism are merged using the Concation fusion strategy to ensure the preservation of complementary information and obtain high-quality fused images; (4) In the decoding layer, the common features extracted from it are compared with the source image I in the last layer of the decoding part. S and source image I P The private features are combined to reconstruct the source image I separately. S and I P Residual-based decoder networks are used in image reconstruction tasks to improve reconstruction results and generate output images that are as similar as possible to the input images. (5) Design of loss function The loss function consists of three types of loss terms: content loss, intensity loss, and gradient loss, as shown in formula (14): (14) in, Represents gradient loss, Indicates strength loss. Indicates content loss. and These are the tuning parameters; (6) Content loss Content loss includes structural similarity loss and relevance consistency loss. The relevance consistency loss function further enhances the shared information between modalities, making the basic features across modalities related, while weakening the correlation between detailed high-frequency features in different modalities. The content loss is shown in formula (15): (15) in, Represents structural similarity loss. Indicates the relevant consistency loss, Used to balance structural similarity and related consistency terms: Specifically, the structural similarity loss is shown in formula (16): (16) in, It is a structural similarity index: The relevant consistency loss is shown in Equation (17): (17) in, This is the correlation coefficient operator, and here ε is set to 1.01 to ensure that the term is always positive; feature This includes common modal information encompassing background and large-scale environment; Features in PET / SPECT images represent high brightness or color information; (7) Gradient loss An adaptive weight block is introduced into the gradient loss function. The adaptive weight block first uses a Gaussian low-pass filter for linear smoothing to minimize the impact of noise on the weight block. Then, the optimization objective of the corresponding pixels in the source image is evaluated based on the gradient richness. The gradient loss formula is as follows: (18) (19) (20) In the formula, and Indicates the pixel at the 1st row and number Column weight chart, and Represents the height and width of the image. and Source image, Indicates a fused image. Represents the absolute value function. For the Laplace operator, This represents the Gaussian low-pass filter function. Represents the minimum function. It is a symbolic function; (8) Strength loss The formula for strength loss is as follows: (21) In the formula, and Indicates the pixel at the 1st row and number Column weight chart, and Represents the height and width of the image. and Represents the source image. Represents a fused image.

2. The multi-branch medical image fusion method based on contrast constraints and IDFAM as described in claim 1, characterized in that: MSRB consists of ordinary convolutional layers with a kernel size of 3×3 and an activation function of leakage correction linear units. Specifically, it is a lightweight module with a stride of 1 and no edge padding. It uses two convolutional layers with a kernel size of 3×3 to replace a single convolutional layer with a kernel size of 5×5, and three convolutional layers with a kernel size of 3×3 to replace a single convolutional layer with a kernel size of 7×7. Ordinary convolutional layers with an activation function of LReLU are used to extract shallow features.

3. The multi-branch medical image fusion method based on contrast constraints and IDFAM as described in claim 1, characterized in that: GRDB is a variant of densely connected modules and residual blocks, employing dense connections while integrating gradient operations; The main part of GRDB uses two convolutional layers with a kernel size of 3×3 and an activation function of LReLU, and a shared convolutional layer with a kernel size of 1×1 and Tanh as the activation function. The mainstream approach introduces dense connections to fully utilize convolutional layers for feature extraction. The residual flow uses gradient operations to calculate the gradient magnitude of features and employs ordinary convolutional layers with 1×1 kernels to eliminate channel dimension differences. Then, the outputs of the main dense flow and the residual gradient flow are combined through element-wise addition to integrate deep features and fine-grained detail features.

Images