Medical image fusion method based on semantic information guidance and multi-modal feature decomposition

Abstract

The application discloses a medical image fusion method based on semantic information guidance and multi-modal feature decomposition, which comprises the following steps: 1, input two images of different modalities, and obtain first modality specific features F M1 , second modality specific features F M2 and two modality common structure features F str through a decomposition branch; 2, input the two images of different modalities into a semantic branch, and obtain semantic features F sem ; 3, input the common structure features F str and the semantic features F sem into a selective fusion module, and obtain comprehensive fusion features F out ; 4, add the first modality specific features F M1 , the second modality specific features F M2 and the comprehensive fusion features F out to obtain a reconstructed fusion image. The application can capture common information of different modalities, improve the fusion effect by combining the guidance of semantic information, and thus assist clinical analysis.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and designs a medical image fusion method based on semantic information guidance and multimodal feature decomposition. Background Technology

[0002] Magnetic resonance imaging (MRI) is a versatile, non-invasive medical imaging technique widely used in brain research. Its modalities, such as T1-weighted (T1), T1-contrast-enhanced (T1ce), T2-weighted (T2), and T2-fluid attenuation inversion recovery (T2Flair), are acquired under standardized scanning conditions, inherently sharing similar structural information and preserving modality-specific features. Each single-modal image has its unique advantages but may not provide a comprehensive understanding of tissue and pathological conditions. Multimodal medical image fusion techniques can integrate specific features from different imaging modalities to improve image quality and detail, further facilitating downstream analysis.

[0003] With the development of deep learning, researchers have attempted to apply deep learning techniques to image fusion, designing end-to-end models without fusion rules. For example, image fusion methods based on convolutional neural networks (CNNs) calculate Laplacian coefficients to obtain fused images. Furthermore, the proposed general image fusion framework IFCNN selects appropriate fusion rules based on the input image type to fuse feature maps, and finally uses two convolutional layers to reconstruct the feature maps to obtain the output image.

[0004] While significant progress has been made in end-to-end models, mainstream state-of-the-art image fusion methods have not explicitly addressed the effective processing of complementary information between modalities. This limitation leads to information redundancy and the loss of modality-specific features during fusion, thus affecting the overall effectiveness of the fusion. Furthermore, most current multimodal medical image fusion methods face challenges when combined with real-world tasks, lacking guidance from semantic information, which hinders the generation of high-quality images for accurate lesion identification. Summary of the Invention

[0005] This invention addresses the shortcomings of existing methods by proposing a medical image fusion method based on semantic information guidance and multimodal feature decomposition. The aim is to reflect the features of different modal MR images in the fused image, thereby improving fusion accuracy and facilitating subsequent segmentation tasks, thus providing a new approach for medical image fusion.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0007] The medical image fusion method of this invention, based on semantic information guidance and multimodal feature decomposition, is characterized by the following steps:

[0008] Step 1: Preprocess the two MRI images of different modalities to obtain the preprocessed medical images I1 and I2 of the two modalities. Where C, H, and W represent the number of channels, height, and width of the image, respectively;

[0009] Step 2: Construct a semantic information-guided modal decomposition medical image fusion network, including: decomposition branch, semantic branch, and selective fusion module;

[0010] Step 2.1: The decomposition branch consists of m encoders based on central difference convolution, n decoders based on central difference convolution, and 1 mutual information calculation module. Each encoder and decoder shares a common structure.

[0011] When i=1, the i-th encoder processes the medical images I1 and I2 of the two modalities respectively to obtain the i-th feature of the first modality. Second mode i-th feature

[0012] When i = 2, 3, ..., m, the (i-1)th feature of the first mode. and Input into the i-th encoder and obtain the i-th feature of the first mode accordingly. Second mode i-th feature Thus, after processing by the m-th encoder, the m-th feature of the first mode is obtained. Second mode m-th feature

[0013] The first modal features output by m encoders Second mode features The input is processed in the mutual information calculation module to obtain the low-dimensional feature vector of the first mode. Second mode low-dimensional eigenvectors in, This represents the i-th low-dimensional feature of the first mode. Represents the i-th low-dimensional feature of the second mode;

[0014] and After being concatenated along the channel dimension, the data is input into the first decoder and processed by n decoders in sequence. The nth decoder then outputs the common structural feature F of the two modalities. str Then and Subtract F respectively strThen, the first mode decoupling feature F is obtained. M1 Second mode decoupling feature F M2 ;

[0015] Step 2.2: The semantic branch consists of W U-Net-based encoders, W U-Net-based decoders, and 1 activation function layer. Each encoder and decoder contains an n1×n1 convolutional layer, a normalization layer (BN), and a ReLU layer.

[0016] I1 and I2 are concatenated along the channel dimension and then input into the first U-Net-based encoder. After being processed by W U-Net-based encoders in sequence, the deep semantic features F are obtained. s ;

[0017] Deep semantic features F s The input is fed into the first U-Net-based decoder, and after being processed by W U-Net-based decoders in sequence, the semantic decoding features F are obtained. sd ;

[0018] Semantic decoding feature F sd The input is given to the activation function layer, and the semantic feature F is output. sem ;

[0019] Step 2.3: The selective fusion module consists of two self-attention units, two context-aware units, and one integration unit;

[0020] Common structural feature F str The inputs are processed separately into the first self-attention unit and the first context-aware unit in parallel, respectively, to obtain the shared self-attention feature representations. Shared features with context

[0021] Semantic feature F sem The inputs are processed in parallel by the second self-attention unit and the second context-aware unit to obtain the self-attention semantic feature representations. and contextual semantic feature representation

[0022] Will and After multiplication, the structural feature representation O is obtained. str ;

[0023] Will and After multiplication, we obtain the semantic feature representation O. sem ;

[0024] O str Osem and F sem The input is processed in the integration unit to obtain the aggregated feature F. out ;

[0025] Step 2.4: F M1 F M2 and F out After addition, the reconstructed MR image I is obtained. f ;

[0026] Step 3: Based on I f And I1 and I2 are used to establish the overall loss function L. total The semantic information-guided modality decomposition medical image fusion network was trained using gradient descent, and the overall loss function L was calculated. total When the number of training iterations reaches the set number, training stops, thus obtaining the optimal modal medical image fusion network for the fusion of medical images.

[0027] The semantic information-guided modal decomposition medical image fusion method of the present invention is also characterized in that the i-th encoder in step 2.1 is composed of one basic convolutional layer and two edge convolutional layers;

[0028] The edge convolutional layer consists of two convolutional layers with n1×n1 kernels, one ReLU layer, one horizontal / vertical convolutional layer, and one diagonal convolutional layer.

[0029] After being processed in the base convolutional layer, the data then sequentially passes through the first convolutional layer, the ReLU layer, the second convolutional layer, one horizontal / vertical convolutional layer, and one diagonal convolutional layer in the first edge convolutional layer to obtain the intermediate features.

[0030] The intermediate features After processing through the second edge convolutional layer, we obtain

[0031] The mutual information calculation module in step 2.1 consists of one convolutional layer with an n1×n1 kernel and one fully connected layer;

[0032] Input into the mutual information calculation module, and with After addition, the mixture is processed sequentially through convolutional layers and fully connected layers to obtain the final product.

[0033] The self-attention unit in step 2.3 consists of three convolutional layers with n1×n1 kernels, four convolutional layers with n2×n2 kernels, and a Softmax layer.

[0034] The input to the first self-attention unit is processed sequentially by a convolutional layer with kernel size n2×n2 and a convolutional layer with kernel size n1×n1 to obtain the query variable.

[0035] F str After processing through a second convolutional layer with a kernel size of n2×n2 and then a second convolutional layer with a kernel size of n1×n1, the key variables are obtained.

[0036] F str After processing through a third convolutional layer with an n2×n2 kernel and a third convolutional layer with an n1×n1 kernel, the value variable is obtained.

[0037] The query variable Q str Key variable K str Sum value variable V str After matrix dimension transformation, the transformed query variable is obtained. Transformed key variables and the transformed value variable

[0038] The transformed query variables and the transformed key variable After multiplication, the result is input into the Sofmax layer for processing, yielding the transposed attention matrix.

[0039] and After multiplication, we get the attention variable. After processing through the fourth n2×n2 convolutional layer, the attention structure features are obtained.

[0040] For F att Perform a matrix dimension transformation operation to obtain the attention structure feature F' after dimension transformation. att and Add them together to obtain the self-attention structure features.

[0041] In step 2.3, the context-aware unit consists of one n2×n2 convolutional layer, two n1×n1 convolutional layers, and one Gelu layer;

[0042] The structural features The input to the first context-aware unit first passes through an n2×n2 convolutional layer, and then through the first n1×n1 convolutional layer to obtain the contextual dimensionality reduction feature F. con , will Fcon After bisecting along the channel, the intermediate feature x of the first structure is obtained. str,1 Second structure intermediate feature x str,2 , will x str,1 The input is processed in the Gelu layer, and the result is then compared with x. str,2 After multiplication, the previous dimensions are restored through a second n1×n1 convolutional layer, thus obtaining the contextual structure features.

[0043] The integration unit in step 2.3 consists of one Sigmoid layer;

[0044] The Sigmoid layer is for O sem After processing, the weight matrix A is obtained. sem , will A sem With the semantic feature F sem After multiplication, the selected semantic feature F is obtained. sem ⊙A sem ;

[0045] Connect the all-1 matrix J to A sem Subtract from O, then add to O str Multiply and obtain the supplementary structural feature O. str ⊙(JA sem );

[0046] F sem ⊙A sem and O str ⊙(JA sem After adding them together, we get the fusion feature F. out .

[0047] In step 3, the total loss function L is constructed using equation (2). total :

[0048] L total =αL text +βL grad +γL corr (2)

[0049] In equation (2), L text L grad and L corr Let represent the image brightness difference loss, image gradient consistency loss, and correlation mutual information loss, respectively. α, β, and γ are the weights for balancing these losses, and are obtained from equations (3), (4), and (5):

[0050]

[0051]

[0052]

[0053] In equations (3) and (4), For Sobel operators, Max(·) is the maximum value operation;

[0054] In equation (5), cc(·) is the operation for calculating correlation, and MI(·) is the operation for calculating mutual information.

[0055] The present invention provides an electronic device, including a memory and a processor, characterized in that the memory is used to store a program that supports the processor in executing the multimodal medical image fusion method, and the processor is configured to execute the program stored in the memory.

[0056] The present invention provides a computer-readable storage medium storing a computer program, characterized in that the computer program, when run by a processor, executes the steps of the multimodal medical image fusion method.

[0057] Compared with existing image fusion methods, the advantages of this invention are as follows:

[0058] 1. This invention proposes a fusion model for multimodal MR images, the SIMFusion network. It uses decomposition branches to capture common and specific features from different modalities of MR images for fusion, reducing information redundancy through relevant mutual information loss. Subsequently, a pre-trained segmentation network is used to obtain their semantic features through semantic branches, and finally, a specialized fusion strategy is employed to achieve an adaptive balance between the two sets of features, resulting in a fused MR image. This assists doctors in further screening and judgment.

[0059] 2. This invention introduces a groundbreaking correlation mutual information loss into the decomposition branches, which helps the model distinguish common and unique features between modalities. It merges semantic information from the segmentation model into the semantic branches, enhances the accurate identification of lesion regions in the fused images, makes up for the shortcomings of traditional end-to-end fusion network-based methods, improves the performance of image fusion, and is of great significance for the study of multimodal medical images. Attached Figure Description

[0060] Figure 1 This is a flowchart of the multimodal medical image fusion method SIMFusion provided in an embodiment of the present invention;

[0061] Figure 2 This is a schematic diagram of the principle framework of SIMFusion, a multimodal medical image fusion method provided in an embodiment of the present invention.

[0062] Figure 3 This is a schematic diagram of the encoder / decoder structure based on central difference convolution provided in an embodiment of the present invention;

[0063] Figure 4 This is a schematic diagram of the structure of the mutual information calculation module provided in the embodiment of the present invention;

[0064] Figure 5 The selective fusion module provided in the embodiments of the present invention;

[0065] Figure 6 The self-attention unit provided in the embodiments of the present invention;

[0066] Figure 7 The context-aware unit provided in the embodiments of the present invention;

[0067] Figure 8 This is a visual comparison diagram on the BraTS2019 dataset provided in an embodiment of the present invention. Detailed Implementation

[0068] In this embodiment, a medical image fusion method based on semantic information guidance and multimodal feature decomposition is named SIMFusion, and its overall flowchart is as follows. Figure 1 As shown, the proposed multimodal medical image fusion method explicitly distinguishes between common features and modality-specific features through decomposition branches to prevent information redundancy and preserve modality-specific features. On the other hand, it captures the semantic information shared between the two modalities through semantic branches. Then, a selective fusion module fuses the obtained common features with semantically corresponding features, and finally merges its output with modality-specific features to generate a fused image. This comprehensive approach aims to effectively preserve relevant information, optimize the fusion process, and leverage advanced semantic understanding to improve the quality of the fused image. Extensive experiments validate the effectiveness and generalization of the proposed method.

[0069] This invention explicitly utilizes an introduced relevance mutual information loss to describe common and modality-specific features, which are further used as structural properties for subsequent decomposition. Furthermore, this method incorporates semantic features extracted from high-level tasks for selective fusion. Comprehensive experiments demonstrate its ability to generate more informative medical images while enhancing the performance of downstream tasks. Finally, the method provided by this invention enables end-to-end training and achieves optimal performance, offering a new and practical approach to multimodal medical image fusion. In this embodiment, the MICCAI BraTS2019 multimodal medical dataset is used as an example, and the specific implementation method is illustrated with reference to the accompanying drawings. The method mainly includes:

[0070] Step 1: Preprocess the two MRI images of different modalities to obtain the preprocessed medical images I1 and I2 of the two modalities. Wherein, C, H, and W represent the number of channels, height, and width of the image, respectively. In this embodiment, the publicly available dataset from the 2019 MICCAI BraTS Brain Tumor Segmentation Challenge is used. The data includes T1, T2, T1ce, T2Flair, and their corresponding brain tumor segmentation maps. For each data set, a standard selection of segmentation maps with a resolution higher than a certain pixel is adopted. The dataset contains 3557 images. Two image pairs of different modalities are read, cropped, and normalized to obtain the corresponding preprocessed data, which is 160×160×1, where 160 corresponds to the image size and 1 is the number of channels.

[0071] Step 2: Construct a semantically guided modality decomposition medical image fusion network, such as... Figure 2 As shown. It includes: decomposition branches, semantic branches, and a selective fusion module;

[0072] Step 2.1: The decomposition branch consists of m encoders based on central difference convolution, n decoders based on central difference convolution, and 1 mutual information calculation module. Each encoder / decoder shares a common structure and is used to process the two modalities of medical images I1 and I2. The images are input into the encoders based on central difference convolution to obtain the first modality features at different stages. Second mode features As shown in equation (1), the above feature input mutual information calculation module obtains the corresponding low-dimensional feature vector. and Used for subsequent loss function calculation; different modal features in the final stage and After concatenating along the channel dimension, the input to the decoder based on central difference convolution yields the shared structural features F of the two modalities. str Then, the common structural features are subtracted from the images of different input modalities to obtain the first modality-specific features F. M1 Second mode specific feature F M2 As shown in equation (2);

[0073]

[0074]

[0075] Step 2.1.1: Construct m encoders based on central difference convolution and n decoders based on central difference convolution, which share a common structure, such as... Figure 3As shown, it consists of one basic n1×n1 convolutional layer and two edge convolutional layers. The edge convolutional layers consist of two n1×n1 convolutional layers, one ReLU layer, one n1×n1 horizontal / vertical convolutional layer, and one n1×n1 diagonal convolutional layer. In this embodiment, m=2, n=1, and n1=3, that is, the convolutional kernel of each layer is 3×3, the convolutional stride is 1, and zero padding is used. The number of channels in the output feature map is 32.

[0076] The initial image I1 is input into the basic n1×n1 convolutional layer, and then processed by the first n1×n1 convolutional layer, the ReLU layer, and the second n1×n1 convolutional layer in the first edge convolutional layer to obtain the intermediate features. intermediate features The output of the i-th encoder based on central difference convolution is obtained after passing through the second edge convolution layer.

[0077] After the initial image I2 undergoes the same processing, the output of the i-th encoder based on center difference convolution is generated.

[0078] Step 2.1.2: Constructing the mutual information calculation module consists of one n²×n² convolutional layer and one fully connected layer, as follows: Figure 4 As shown;

[0079] The output of the i-th encoder based on central difference convolution The input mutual information is processed in the mutual information calculation module and correspondingly compared with the output of the (n-1)th encoder based on central difference convolution. After addition, the corresponding low-dimensional feature representation is obtained through an n3×n3 convolutional layer and a fully connected layer. The output of the i-th encoder based on central difference convolution corresponding to the initial image I2 After the same processing, the corresponding low-dimensional feature representation is obtained. As shown in equation (3); in this embodiment, n3 = 3, that is, the convolution kernel of each convolution layer is 3×3, the convolution stride is 1, and zero padding is used. The number of channels of the output feature map is 32.

[0080]

[0081] Step 2.2: The semantic branch consists of four U-Net-based encoders, four decoders, and one activation function layer. Each U-Net-based encoder and decoder contains one n4×n4 convolutional layer, one normalization layer (BN), and one ReLU layer. This is used to process the two modalities of the medical images I1 and I2 from Step 1 to obtain semantic features F. semAs shown in equation (4); in this embodiment, n4 = 3, that is, the convolution kernel of each convolution layer is 3×3, the convolution stride is 1, and zero padding is used. The number of channels of the output feature map is 32.

[0082] F sem =U([I1,I2]) (4)

[0083] Step 2.3: The selective fusion module consists of two self-attention units, two context-aware units, and one fusion unit; for example... Figure 5 As shown.

[0084] Modal common structural feature F str Inputting one self-attention unit and one context-aware unit yields a self-attention feature representation. and contextual feature representation Semantic feature F sem Inputting one self-attention unit and one context-aware unit yields a self-attention feature representation. and contextual feature representation The final structural feature representation O is obtained by multiplying the self-attention feature representation and the context feature representation respectively. str and semantic feature representation O sem Then, the structural feature representation O str Semantic feature representation O sem and the original semantic features F sem The input is integrated into the unit to obtain the final output feature F. out .

[0085] Step 2.3.1: The self-attention unit consists of 4 n3×n3 convolutional layers, 3 n4×n4 convolutional layers, and one softmax layer; as shown... Figure 6 As shown.

[0086] Structural features or semantic features The input to the self-attention unit first passes through the first n3×n3 convolutional layer and the first n4×n4 convolutional layer to obtain the query variable. The same operation is performed, using the second n3×n3 convolutional layer and the second n4×n4 convolutional layer to obtain the key variables. The value variable is obtained through the third n3×n3 convolutional layer and the second n4×n4 convolutional layer. Query variable O str Key variable F str Sum value variable V str The transformed query variables are obtained through matrix dimension transformation operations. Transformed key variables and the transformed value variable By transforming the query variables and the transformed key variable After multiplication, the result is input into the Sofmax layer to obtain the transposed attention matrix. With the transformed value variable Multiply to obtain the attention variable As shown in Equation (5). Then, the attention structure features are obtained through the fourth n3×n3 convolutional layer. For F att Perform a matrix dimension transformation operation to obtain the attention structure feature F' after dimension transformation. att and Add them together to obtain the self-attention structure features. As shown in equation (6).

[0087]

[0088]

[0089] Step 2.3.2: The context-aware unit consists of one n3×n3 convolutional layer, two n4×n4 convolutional layers, and one Gelu layer; as shown... Figure 7 As shown.

[0090] Structural features or semantic features In the context-aware unit, the contextual dimensionality reduction feature F is obtained first through an n3×n3 convolutional layer, and then through the first n4×n4 convolutional layer. con , reduce the dimensionality of the context features F con The first structural intermediate feature x is obtained by bisecting along the channel. str,1 Second structure intermediate feature x str,2 , will x str,1 Input to Gelu layer and then x str,2 After multiplication, the layers are restored to their previous dimensions using a second n4×n4 convolutional layer to obtain the context structure features. or contextual semantic features

[0091] The self-attention structure features in step 2.3.1 and the context structure features in step 2.3.2 Multiplying them together yields the final structural feature O. str The same operation is performed on the self-attention semantic features in step 2.3.1. and the contextual semantic features in step 2.3.2 Multiplying them yields the final semantic feature O. sem As shown in equation (7).

[0092]

[0093] Step 2.3.3: The integration unit consists of one Sigmoid layer, such as... Figure 8 As shown;

[0094] The final structural feature O str The final semantic feature O sem and the initial semantic feature F sem The input ensemble unit first inputs the final semantic features into the Sigmoid layer to obtain the weight matrix A. sem The weight matrix A and the semantic features F are combined. sem Multiplication yields the selected semantic feature F sem ⊙A sem Then subtract the all-1 matrix J from the weight matrix and then subtract it from the final structural feature O. str Multiplication yields supplementary structural features O str ⊙(JA sem Finally, the selected semantic features F sem ⊙A sem and supplementary structural features O str ⊙(JA sem The final fusion feature F of the integrated unit is obtained by adding the two components together. out As shown in equation (8).

[0095] F out =F sem ⊙A sem +O str ⊙(JA sem (8)

[0096] Step 2.4: The first mode-specific feature F from Step 2.1... M1 Second mode specific feature F M2 The final fusion feature F in step 2.3.3 out The final fused image I is obtained by adding them together. f As shown in equation (9).

[0097] I f =F out +F M1 +F M2 (9)

[0098] Step 3: Construct the overall loss function L total :

[0099] L total =αL text +βL grad +γL corr (10)

[0100] In equation (10), L text Lgrad and L corr Let represent the image brightness difference loss, image gradient consistency loss, and correlation mutual information loss, respectively. α, β, and γ are the weights used to balance these losses. In this embodiment, α = 1, β = 10, and γ = 0.1, and the results are obtained from equations (11), (12), and (13):

[0101]

[0102]

[0103]

[0104] In equations (11) and (12), For the Sobel operator, Max(·) is the maximum value operation; in equation (13), cc(·) is the correlation calculation operation, and MI(·) is the mutual information calculation operation.

[0105] Step 4: Train the multimodal medical image input semantic information-guided modality decomposition medical image fusion network and calculate the overall loss function L. total Training stops when the set number of iterations is reached, thus obtaining the optimal multimodal medical image fusion network. In this embodiment, the Adam optimizer is used, and the learning rate decreases with the number of iterations, with 100 training iterations. The final results are compared with those of different methods. Figure 8 As shown, this fusion method can better combine the clear structure inside the lesion in the Flair image with the overall brain structure in the T2 image compared to other methods, thus enabling its application in subsequent medical tasks.

[0106] To quantify the effectiveness of this invention and verify its validity, the method of this invention is compared with nine algorithms, including IFCNN. Peak Signal-to-Noise Ratio (PSNR) and Mutual Information (MI), common metrics in image fusion, are selected as unsupervised evaluation criteria. In addition, the fused image is input into U-Net, a common segmentation network model in the medical field, to obtain a segmentation map. Dice Similarity Coefficient (DSC) and Hausdorff Distance (HD, HD95, i.e., 95% HD) are selected as two performance metrics for supervised evaluation.

[0107] This embodiment lists nine comparison methods and the multimodal medical image fusion method proposed in this invention. The nine comparison methods are IFCNN, U2Fusion, EMFusion, MATR, PMGI, MsgFusion, FusionGAN, SwinFusion, and CDDFuse. The corresponding paper titles of these nine comparison methods are: IFCNN: A General Image Fusion Framework based on Convolutional Neural Network, U2Fusion: A Unified Unsupervised Image Fusion Network, EMFusion: An UnsupervisedEnhanced Medical Image Fusion Network, MATR: Multimodal Medical Image Fusionvia Multiscale Adaptive Transformer, Rethinking the Image Fusion: A FastUnified Image Fusion Network based on Proportional Maintenance of Gradientand Intensity, MsgFusion: Medical Semantic Guided Two-branch Network for Multimodal Brain Image Fusion, FusionGAN: A Generative Adversarial Network for Infrared and Visible Image Fusion, SwinFusion: Cross-domai nLong-range Learning for General Image Fusion via Swin Transformer and CDDFuse: Correlation-drivenDual-branch Feature Decomposition for Multi-modality Image Fusion.

[0108] In this embodiment, the metrics used to evaluate the fused image include information theory-based metrics PSNR and MI, and image feature-based metrics AG and Q. abf Q p And SD, Q, a metric based on structural similarity yand Q w Q, an indicator based on human perception cb And VIFF.

[0109] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor in executing the multimodal medical image fusion method described above. The processor is configured to execute the program stored in the memory.

[0110] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the multimodal medical image fusion method described above.

Claims

1. A medical image fusion method based on semantic information guidance and multimodal feature decomposition, characterized in that, The procedure is as follows: Step 1: Preprocess the two MRI images of different modalities to obtain the preprocessed medical images I1 and I2 of the two modalities. Where C, H, and W represent the number of channels, height, and width of the image, respectively; Step 2: Construct a semantic information-guided modal decomposition medical image fusion network, including: decomposition branch, semantic branch, and selective fusion module; Step 2.1: The decomposition branch consists of m encoders based on central difference convolution, n decoders based on central difference convolution, and 1 mutual information calculation module. Each encoder and decoder shares a common structure. When i=1, the i-th encoder processes the medical images I1 and I2 of the two modalities respectively to obtain the i-th feature of the first modality. Second mode i-th feature When i = 2, 3, ..., m, the (i-1)th feature of the first mode. and Input into the i-th encoder and obtain the i-th feature of the first mode accordingly. Second mode i-th feature Thus, after processing by the m-th encoder, the m-th feature of the first mode is obtained. Second mode m-th feature The first modal features output by m encoders Second mode features The input is processed in the mutual information calculation module to obtain the low-dimensional feature vector of the first mode. Second mode low-dimensional eigenvectors in, This represents the i-th low-dimensional feature of the first mode. Represents the i-th low-dimensional feature of the second mode; and After being concatenated along the channel dimension, the data is input into the first decoder and processed by n decoders in sequence. The nth decoder then outputs the common structural feature F of the two modalities. str Then and Subtract F respectively str Then, the first mode decoupling feature F is obtained. M1 Second mode decoupling feature F M2 ; Step 2.2: The semantic branch consists of W U-Net-based encoders, W U-Net-based decoders, and 1 activation function layer. Each encoder and decoder contains an n1×n1 convolutional layer, a normalization layer (BN), and a ReLU layer. After I1 and I2 are spliced according to the channel dimension, they are input into a first U-Net-based encoder and sequentially processed by W U-Net-based encoders to obtain deep semantic features F s ; Deep semantic features F s The input is fed into the first U-Net-based decoder, and after being processed by W U-Net-based decoders in sequence, the semantic decoding feature F is obtained. sd ; Semantic decoding feature F sd The input is given to the activation function layer, and the semantic feature F is output. sem ; Step 2.3: The selective fusion module consists of two self-attention units, two context-aware units, and one integration unit; Common structural feature F str The inputs are processed separately into the first self-attention unit and the first context-aware unit in parallel, respectively, to obtain the shared self-attention feature representations. Shared features with context Semantic feature F sem The inputs are processed in parallel by the second self-attention unit and the second context-aware unit to obtain the self-attention semantic feature representations. and contextual semantic feature representation Will and After multiplication, the structural feature representation O is obtained. str ; Will and After multiplication, we obtain the semantic feature representation O. sem ; O str O sem and F sem The input is processed in the integration unit to obtain the aggregated feature F. out ; Step 2.4: F M1 F M2 and F out After addition, the reconstructed MR image I is obtained. f ; Step 3: Based on I f And I1 and I2 are used to establish the overall loss function L. total The semantic information-guided modality decomposition medical image fusion network was trained using gradient descent, and the overall loss function L was calculated. total When the number of training iterations reaches the set number, training stops, thus obtaining the optimal modal medical image fusion network for the fusion of medical images.

2. The semantic information-guided modal decomposition medical image fusion method according to claim 1, characterized in that, In step 2.1, the i-th encoder consists of one basic convolutional layer and two edge convolutional layers; The edge convolutional layer consists of two convolutional layers with n1×n1 kernels, one ReLU layer, one horizontal / vertical convolutional layer, and one diagonal convolutional layer. After being processed in the base convolutional layer, the data then sequentially passes through the first convolutional layer, the ReLU layer, the second convolutional layer, one horizontal / vertical convolutional layer, and one diagonal convolutional layer in the first edge convolutional layer to obtain the intermediate features. The intermediate features After processing through the second edge convolutional layer, we obtain 3. The semantic information-guided modal decomposition medical image fusion method according to claim 2, characterized in that, The mutual information calculation module in step 2.1 consists of one convolutional layer with an n1×n1 kernel and one fully connected layer; Input into the mutual information calculation module, and with After addition, the mixture is processed sequentially through convolutional layers and fully connected layers to obtain the final product.

4. The semantic information-guided modal decomposition medical image fusion method according to claim 3, characterized in that, The self-attention unit in step 2.3 consists of three convolutional layers with n1×n1 kernels, four convolutional layers with n2×n2 kernels, and a Softmax layer. The input to the first self-attention unit is processed sequentially by a convolutional layer with kernel size n2×n2 and a convolutional layer with kernel size n1×n1 to obtain the query variable. F str After processing through a second convolutional layer with a kernel size of n2×n2 and then a second convolutional layer with a kernel size of n1×n1, the key variables are obtained. F str After processing through a third convolutional layer with an n2×n2 kernel and a third convolutional layer with an n1×n1 kernel, the value variable is obtained. The query variable Q str Key variable K str Sum value variable V str After matrix dimension transformation, the transformed query variable is obtained. Transformed key variables and the transformed value variable The transformed query variables and the transformed key variable After multiplication, the result is input into the Sofmax layer for processing, yielding the transposed attention matrix. and After multiplication, we get the attention variable. After processing through the fourth n2×n2 convolutional layer, the attention structure features are obtained. For F att Perform a matrix dimension transformation operation to obtain the attention structure feature F' after dimension transformation. att and Add them together to obtain the self-attention structure features.

5. The semantic information-guided modal decomposition medical image fusion method according to claim 4, characterized in that, In step 2.3, the context-aware unit consists of one n2×n2 convolutional layer, two n1×n1 convolutional layers, and one Gelu layer; The structural features The input to the first context-aware unit first passes through an n2×n2 convolutional layer, and then through the first n1×n1 convolutional layer to obtain the contextual dimensionality reduction feature F. con , will F con After bisecting along the channel, the intermediate feature x of the first structure is obtained. str,1 Second structure intermediate feature x str,2 , will x str,1 The input is processed in the Gelu layer, and the result is then compared with x. str,2 After multiplication, the dimensions are restored to the previous dimensions through a second n1×n1 convolutional layer, thus obtaining the contextual structure features.

6. The semantic information-guided modal decomposition medical image fusion method according to claim 5, characterized in that, The integration unit in step 2.3 consists of one Sigmoid layer; The Sigmoid layer is for O sem After processing, the weight matrix A is obtained. sem , will A sem With the semantic feature F sem After multiplication, the selected semantic feature F is obtained. sem ⊙A sem ; Connect the all-1 matrix J to A sem Subtract from O, then divide by O str Multiply and obtain the supplementary structural feature O. str ⊙(JA sem ); F sem ⊙A sem and O str ⊙(JA sem After adding them together, we get the fusion feature F. out .

7. The semantic information-guided modal decomposition medical image fusion method according to claim 1, characterized in that, In step 3, the total loss function L is constructed using equation (2). total : L total =αL text +βL grad +γL corr (2) In equation (2), L text L grad and L corr Let represent the image brightness difference loss, image gradient consistency loss, and correlation mutual information loss, respectively. α, β, and γ are the weights for balancing these losses, and are obtained from equations (3), (4), and (5): In equations (3) and (4), For Sobel operators, Max(·) is the maximum value operation; In equation (5), cc(·) is the operation for calculating correlation, and MI(·) is the operation for calculating mutual information.

8. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the multimodal medical image fusion method of any one of claims 1-7, the processor being configured to execute the program stored in the memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program, when run by a processor, performs the steps of any of the multimodal medical image fusion methods described in claims 1-7.

Images