A multi-modal medical image fusion method

Abstract

The application belongs to the technical field of medical image fusion, and discloses a kind of based on knowledge distillation and explainable multimodal medical image fusion model and method.Based on knowledge distillation and explainable multimodal medical image fusion model, including a generator G and two discriminators Dc and Dm, generator is responsible for generating fusion image, discriminator is used to distinguish real image and fusion image, the model solves the problem that the fusion image clarity of multimodal medical image is reduced due to the limited pair image data set available for fusion.From experimental results, the model has good performance in subjective visual observation and objective index evaluation, and can assist doctors in making accurate diagnosis and treatment of lesion sites when applied to multimodal medical image fusion.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image fusion, and particularly relates to a multi-modal medical image fusion method. BACKGROUND

[0002] With the continuous development of medical imaging technology, different modalities of imaging methods such as magnetic resonance (MR), computed tomography (CT), positron emission tomography (PET) and X-ray have appeared. A large number of deep features reflecting the lesion conditions are contained in images of different modalities, and there is limitation in representing these features by using only single-modality medical images. In this regard, in clinical practice, key information in multiple modalities of medical images is selected to be fused, so that the original image features can be retained while the defects of single-modality medical images are made up. However, the information integrity of the fused image is easily lost by first extracting deep features from single-modality images and then fusing, and therefore, it is a new idea for multi-modal medical image fusion to extract deep features required for fusion by using the relationship between the features of the current modality and other modalities. However, the deep network is often required to accurately extract deep features in each modality image, which means that a large training data set is required. Medical images involve patient privacy, and it is difficult to obtain a training data set from a hospital, and the publicly available medical image data set provides very limited paired images for a specific disease. Therefore, it is very important to train a fused image with deep features by using a limited data set for the medical field which is difficult to collect sufficient data sets.

[0003] At present, image fusion models are usually based on traditional methods, convolutional neural networks (CNN) and generative adversarial networks (GAN). However, the fused images obtained based on traditional methods have spectral distortion and spatial distortion, which is not conducive to the observation of the patient's condition by doctors; only CNN is used to extract features of medical images, which is prone to problems such as single scale of extracted features and gradient explosion when the network depth increases, thereby reducing the quality of the fused image; GAN is particularly widely used in the field of image fusion, but GAN only uses one discriminator to distinguish two source images, which can lose part of the deep features due to uneven attention allocation of the discriminator, and does not solve the problem that a large data set is required to train the model of an image fusion model with a deep network level.

[0004] To solve the above problems, the current methods to solve the difficulty of training model for small-scale data set include model fine-tuning, transfer learning and data enhancement. Model fine-tuning is to pre-train the model with a large data set, and then fine-tune the model parameters with a small amount of target data, but the model can only be used in the case that the source large data set data and the target data are consistent in distribution. Transfer learning also needs pre-training, but it transfers the learned knowledge to a new task. As a special transfer learning, knowledge distillation uses the knowledge of a complex teacher network to guide the training of a simple student network, so that the student network has similar ability to the teacher network while reducing the structure. However, the traditional knowledge distillation uses the difference between the inputs of the softmax layers of the teacher and student networks as the loss function to optimize the network, which loses the spatial and semantic information and is not suitable for image fusion. Some new knowledge distillation methods only improve the distillation method, rarely innovate the way of constructing the student network, and the distillation method improves the distillation efficiency while increasing the demand for data size.

[0005] In recent years, with the continuous development of image fusion technology, the multi-modal medical image model only stays in modifying the network structure or parameters, which is difficult to meet the actual needs. Only by solving specific problems according to the needs can it be the development direction of future medical image fusion model. SUMMARY

[0006] The present application provides a multi-modal medical image fusion method for the above problems.

[0007] In order to achieve the above purpose, the present application adopts the following technical scheme:

[0008] The present application provides a multi-modal medical image fusion model based on knowledge distillation and explainability, which includes a generator G and two discriminators Dc and Dm. The generator is responsible for generating a fusion image, and the discriminators are used to distinguish real images and fusion images. The generator includes two parts: feature extraction and feature fusion. The feature extraction part uses a student network obtained by removing redundant channels through knowledge distillation, and the feature fusion part consists of a Concat layer and five convolution layers. The discriminators include a discrimination module and an explainable module. The discrimination module is used to discriminate real images and fusion images, and the explainable module is used to generate an explanation image of the image identified as "Real" by the discrimination module of the discriminator.

[0009] Further, the student network includes: 4 encoders, the filter numbers of which are 56, 112, 224 and 448 respectively; a module x, which is both an encoder and a decoder, is built as X, and the number of channels is 896; four decoders, and the input channels are all set to 280.

[0010] Further, the convolutional layer of the feature fusion part uses a 3*3 filter to compress the number of channels, the number of filters of each layer is 128, 64, 32, 16, 1 respectively, and the batch normalization layer is also used to alleviate gradient explosion, and the activation function LeakyReLU adaptively learns parameters to improve accuracy and speed up training.

[0011] Further, the discrimination module of the discriminator includes five convolutional layers and one fully connected layer, each convolutional layer has a 3*3 filter, the number of filters is 64, 128, 128, 256, 256 respectively, the convolution step of the first layer is set to 1, and the step of the remaining layers is set to 2, and the dropout of the fully connected layer is set to 0.4.

[0012] The application also provides a multi-modal medical image fusion method based on the above model, which comprises the following steps:

[0013] Step 1, in the feature fusion part, each fusion path is respectively distilled with a U-Net3+ with a cross-modal attention block as a teacher model to obtain a student model for extracting image features, and then the student model is used to replace the teacher model to extract deep features of the image to be fused;

[0014] Step 2, the feature fusion part fuses the extracted deep features, and after channel dimension reduction and convolution operation, a fused image is generated;

[0015] Step 3, the discriminator adds an interpretable module to generate an explanation image of the image discriminated as "Real" by the discrimination module of the discriminator, according to the accuracy of the key features covered by the explanation image, the discriminator is stopped training in advance when the discrimination performance is optimal, without training all the data, and in the training process, the generator and the discriminator of the model are optimized together, and the trained generator can generate a clear fused image.

[0016] Further, the detailed steps of obtaining the student model by knowledge distillation in step 1 and applying it to the feature extraction part of the model are as follows:

[0017] (1) Constructing a student network structure

[0018] The teacher network includes four encoders ENC 1 , ENC 2 , ENC 3 and ENC 4 and four decoders DEC1 、 DEC 2 、 DEC 3 and DEC 4 ; the filter numbers of the encoders ENC 1 、 ENC 2 、 ENC 3 and ENC 4 are 64, 128, 256 and 512 respectively; the filter numbers of the decoders DEC 1 、 DEC 2 、 DEC 3 and DEC 4 are 320, i.e. 64 filters with size 3x3 are used to compress or expand the channels of the five input feature maps of the upper encoder and the lower decoder to 64 respectively, and then the 64 channels of the five input feature maps are stacked into 320 channels; each encoder of each layer has a large number of channels, but the contribution of some channels is small, which is equivalent to invalid redundant channels; by setting a threshold γ, we binarize the attention feature maps output by the cross-modal attention block to obtain a specific number of redundant channels, assuming that the cross-modal attention block outputs T-channel attention feature maps, and the activation values of t' channels are not all 0, after ensuring that the input of each decoder is equal to the output of the encoder, as many channels as possible with activation value of 0 are removed to obtain binary attention feature maps, and the number thereof is set to t, where t≥t';

[0019] According to the above settings, the filter numbers of the four encoders enc 1 、 enc 2 、 enc 3 and enc 4 of the student network are 56, 112, 224 and 448 respectively; the module x , which is both an encoder and a decoder, is built as X, and the number of channels is 896; each decoder dec 1 、 dec 2 、 dec 3 and dec 4The input channels are set to 280 to ensure that the number of output channels of the student network encoder matches the number of input channels of the decoder. At this point, the student network that replaces the teacher network to complete the feature extraction task is completed.

[0020] (2) Knowledge transfer

[0021] Knowledge transfer is divided into encoder learning and decoder learning, taking the student network as an example. enc 2 and dec 3 Taking the learning process of the encoder as an example, the learning process of other encoders and decoders is similar; during encoder learning, the image... I They were first input into the first layer. ENC 1 and enc 1 Then, it is trained by encoders and decoders located in each layer. ENC N The output features are Each floor enc N The output features are In order to make ENC N Output features Closer enc N Output features The difference between the two is used as the encoder loss function for optimization. enc N encoder loss function As shown in equation (1); during decoder learning, the decoder loss function Intended to make dec N Output features Closer enc N Input features ;make dec 1 Output feature extraction result image I rec Closer I , As shown in equation (2):

[0022] (1);

[0023] (2);

[0024] (3);

[0025] In formula (2) represent the output of each layer dec N The third term of formula (2) is the perceptual loss, which updates the image by back propagation and promotes feature extraction. It is worth noting that when N =1, the first term of loss is not calculated, and when N =5, the first term of loss is 0; in formula (3), represents the output feature of each layer enc N enc N The output of each layer feature map of is the superimposed feature map obtained before inputting the decoder, and respectively represent the up-sampling and down-sampling operations; represents the operation of superimposing all channels.

[0026] From N =1 to N =4, four pairs of encoders of the teacher-student network are trained, and from N =4 to N =1, four pairs of decoders are trained, and when N =5, is trained, which is both an encoder and a decoder. Since the encoder learning aims to minimize X ; the decoder learning aims to minimize ; and in summary, when training the paired enc N and dec N , the problem to be solved is shown in formula (4): (4).

[0027] Further, the specific process of step 2 is that the feature extraction part obtains a feature map with a size of 128×128×280, which is sequentially spliced through the Concat layer, and then is subjected to channel dimension reduction through five convolutional layers, and finally is compressed into a size of 128×128×1, a fusion image with rich deep details and fully representing two modal key features.

[0028] Further, the specific process of step 3 is that the discriminator adds an interpretable module, dynamically monitors the performance of the discriminator, and stops training when the performance is optimal.

[0029] ​​​​​(1) The real image and the fusion image G(z) obtained by the generator G are input into the discrimination module for image classification, D(G(z)), D(real) respectively represent the classification results of the discrimination module on G(z) and the real image, taking the brain image as an example, Real represents the image classified as a brain CT, and Fake represents the image classified as a non-brain CT image; the interpretable module visually interprets half of the images classified as Real, and the discrimination result is used to update the discriminator loss and the generator loss , as shown in equations (5)-(6):

[0030] (5);

[0031] (6);

[0032] (2) Dynamic threshold adjustment

[0033] To dynamically control the training of the discriminator, according to the interpretable image of the first half of the images classified as Real output by the interpretable module, the training of the remaining second half of the images is dynamically adjusted; the evaluation of the interpretation effect is divided into subjective evaluation method and objective evaluation;

[0034] The subjective evaluation method is to judge whether the discriminator performance reaches the best by observing the proportion of the key feature activated area covered in the interpretable image;

[0035] The objective evaluation is to measure whether the discriminator performance meets the standard after the first half of the images are trained, with an interpretable threshold (0 <1) as the standard; The calculation method of the initial value of is as follows:

[0036] (7);

[0037] Wherein, is analogous to the interpretable threshold, represents the probability distribution of the real image, represents the probability distribution of the generator generating the fusion image; when tends to 1, it means that too few fusion images generated by the generator are discriminated as real, and the generator performance is poor; when tends to 0, it means that too few real images are discriminated as real, and the discriminator discrimination ability is insufficient; therefore, when tends to , the discriminator threshold reaches the best ( );

[0038] According to the subjective evaluation method to obtain the interpretation image, the objective evaluation method is adjusted gradually If the proportion of key features covered by the activated region is too small, it indicates that the discriminator's ability to distinguish key features is insufficient, and the threshold needs to be lowered; conversely, if the proportion is large, the threshold needs to be raised. The average of the discrimination loss of the real image and G(z) is used as the loss variable T. D ,when When the error rate is too high, it indicates that the discriminator's accuracy is low, and it needs to be used again. Training the discriminator; when When this occurs, it indicates that the discriminator's discrimination loss has converged to a safe range, and discriminator training can be skipped. It is worth noting that regardless of whether the discriminator is trained, it is necessary to utilize... Training the generator.

[0039] Furthermore, the specific process of jointly optimizing the generator and discriminator in step 3 is as follows: First, the generator G is fixed and the discriminators Dc and Dm are trained. Then, the discriminators Dc and Dm are fixed and the generator G is trained. This process is repeated to improve the generator G's ability to generate fused images and the discriminators Dc and Dm's ability to distinguish the difference between the generated images and the real images, until the images generated by the generator G are sufficient to pass the discrimination of the discriminators Dc and Dm. After the generator G is trained, the test set data is input into the trained generator G to obtain the final fusion result.

[0040] Compared with the prior art, the present invention has the following advantages:

[0041] This invention proposes a knowledge distillation and interpretable multimodal medical image fusion model (KDE-GAN), which solves the problem of decreased image clarity in multimodal medical image fusion due to the limited availability of paired image datasets for fusion. Experimental results show that the KDE-GAN model performs well in both subjective visual observation and objective evaluation metrics. Applying KDE-GAN to multimodal medical image fusion can assist doctors in making accurate diagnoses and treatments for lesions. Furthermore, the KDE-GAN model has the following characteristics: 1) It utilizes knowledge distillation to obtain a simplified SU-Net3+ structure, extracting deep image features with only a small dataset while maintaining good feature extraction performance; 2) It employs an interpretable module to improve the accuracy of the discriminator in identifying images, thereby maximizing the utilization of the limited dataset. Attached Figure Description

[0042] Figure 1 This is a diagram of the KDE-GAN network structure in the embodiments of this application.

[0043] Figure 2 The process of constructing the SC-U-Net3+ network structure.

[0044] Figure 3Fig. 2 is a process diagram of the binarization process of feature maps from the cross-modal attention block in the pre-training of the TC-U-Net3+.

[0045] Figure 4 Fig. 3 is the network structure of the SC-U-Net3+. enc 2 and dec 2 learning process.

[0046] Figure 5 Fig. 4 is the network structure of the feature fusion part.

[0047] Figure 6 Fig. 5 is the training process of the discriminator Dc.

[0048] Figure 7 Fig. 6 is the structure of the discriminant module for image classification.

[0049] Figure 8 Fig. 7 is a detailed network structure diagram of the model of the present application.

[0050] Figure 9 Fig. 8 is the fusion result of the 17th slice of the CT and MR-T1 images of the sarcoma fused by different methods.

[0051] Figure 10 Fig. 9 is the fusion result of the 13th slice of the CT and MR-T2 images of the brain toxoplasmosis fused by different methods.

[0052] Figure 11 Fig. 10 is the fusion result of the 17th slice of the CT and MR-PD images of the brain meningioma fused by different methods. Figure 12 Fig. 11 is the fusion result of the 10th slice of the CT and MR-Gad images of the metastatic bronchial tumor fused by different methods.

[0053] Figures 9-12 Fig. 13 is a zoomed-in area extracted from (a)-(j) of (a) NestFuse, (b) IFCNN, (c) DenseFuse, (d) DTCWT, (e) SGF, (f) FPRSGF, (g) PCL E, (h) FusionGAN, (i) DDcGAN, (j) UC-DDGAN. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present application will be specifically and concretely described below in combination with the embodiments of the present application and the drawings. It should be pointed out that, for those skilled in the art, without departing from the principles of the present application, a number of variations and improvements can be made, which should be regarded as falling within the protection scope of the present application.

[0055] KDE-GAN uses a generator G and two discriminators (Dc, Dm) to complete image fusion, and the network structure is as shown in Figure 1 The generator is responsible for generating the fusion image, and the discriminator is used to distinguish the real image and the fusion image. The generator includes two parts of feature extraction and feature fusion. Taking the fusion of CT and MR as an example, in the feature extraction part, the CT and MR paths are respectively used to join the U-Net3+ with the cross-modal attention block as the teacher model (CT path: TC-U-Net3+, MR path: TM-U-Net3+) for knowledge distillation, to obtain the student model (CT path: SC-U-Net3+, MR path: SM-U-Net3+) for extracting CT and MR features. Then SC-U-Net3+ and SM-U-Net3+ are used to replace TC-U-Net3+ and TM-U-Net3+ to extract deep features of CT and MR images. The feature fusion part fuses the extracted deep features, and after channel dimension reduction and convolution operation, the fusion image is generated. The discriminator adds the interpretable module LIME, and the explanation image of the image identified as "Real" by the discrimination module of the generation discriminator is generated, and according to the accuracy of the key features covered by the explanation image, the discriminator stops training in advance when the discrimination performance is optimal, without training all the data. In the training process, the generator and the discriminator of KDE-GAN are optimized together, and the trained generator can generate clear fusion images.

[0056] The feature extraction part of the generator

[0057] The present application takes the process of obtaining the student model SC-U-Net3+ as an example (see Figure 2 ), and introduces the detailed steps of obtaining the student model through knowledge distillation and applying it to the feature extraction part of KDE-GAN. It can be roughly divided into two stages: constructing the network structure of SC-U-Net3+ and knowledge transfer. It should be noted that the training of SC-U-Net3+ is guided by TC-U-Net3+, and the training of SM-U-Net3+ is guided by TM-U-Net3+, and the two are performed synchronously. Therefore, the cross-modal attention block of each layer will calculate the feature distribution of the two models and pass it to the next layer.

[0058] 1) Construct SC-U-Net3+

[0059] First, the four encoders of TC-U-Net3+ are named ENC 1 , ENC 2 , ENC 3 and ENC 4 , and the four decoders are DEC 1、 DEC 2 、 DEC 3 and DEC 4 In addition, the four encoders of SC-U-Net3+ are named as enc 1 、 enc 2 、 enc 3 and enc 4 The four decoders are respectively dec 1 、 dec 2 、 dec 3 and dec 4 . ENC 1 、 ENC 2 、 ENC 3 and ENC 4 The number of filters are 64, 128, 256 and 512 respectively. DEC 1 、 DEC 2 、 DEC 3 and DEC 4 There are 320 filters, that is, 64 filters with a size of 3x3 are used to compress or expand the channels of the five input feature maps of the upper encoder and the lower decoder to 64 respectively, and then the 64 channels of the five input feature maps are stacked into 320 channels.

[0060] Each layer of the TC-U-Net3+ encoder has a large number of channels, but some channels have little contribution, which is equivalent to invalid redundant channels. By setting a threshold γ, we binarize the attention feature map output by the cross-modal attention block to obtain a specific number of redundant channels (see Figure 3 ). Assuming that the cross-modal attention block outputs an attention feature map with T channels, and the activation values of t' channels are not all 0. After ensuring that the input of each decoder is equal to the output of the encoder, as many channels as possible with activation values of 0 are removed to obtain a binary attention feature map, and the number of which is set to t (t≥t').

[0061] The conclusion obtained by experiment according to the above method is that the encoder of SC-U-Net3+ enc 1 、 enc 2, enc 3 and enc 4 The number of filters after removing redundant channels is 56, 112, 224, and 448 (modules). x The SC-U-Net3+ encoder (designated as X, with 896 channels) acts as both an encoder and decoder. In this experiment, the input channels of each decoder are set to 280 to ensure that the number of output channels of the encoder matches the number of input channels of the decoder. Furthermore, the cross-modal attention blocks of SC-U-Net3+ do not require binarization. At this point, SC-U-Net3+, replacing TC-U-Net3+ for extracting CT features, is complete and can be tested. Table 1 shows the parameters of SC-U-Net3+ when extracting CT images of size 256×256×1; the parameters of SC-U-Net3+ when extracting MR images are similar.

[0062] Table 1. Parameters of each encoder and decoder in SC-U-Net3+

[0063]

[0064] (2) Knowledge transfer

[0065] Knowledge transfer is divided into encoder learning and decoder learning, taking SC-U-Net3+ as an example. enc 2 and dec 2 Taking learning as an example (see) Figure 4 The learning process for other encoders and decoders is similar; during encoder learning, the image... I They were first input into the first layer. ENC 1 and enc 1 Then, it is trained by encoders and decoders located in each layer. ENC N The output features are Each floor enc N The output features are In order to make ENC N Output characteristics Closer enc N Output characteristics The difference between the two is used as the encoder loss function for optimization. enc N encoder loss function As shown in equation (1); during decoder learning, the decoder loss function aiming to make dec N output feature closer enc N input feature ; make dec 1 output feature extraction result map I rec closer I , As shown in equation (2):

[0066] (1);

[0067] (2);

[0068] (3);

[0069] In equation (2) represents the output of each layer dec N , the third term loss of equation (2) is the perceptual loss, which updates the image by back propagation, promotes feature extraction, and it should be noted that when N =1, the first term loss is not calculated, and when N =5, the first term loss is 0; in equation (3) represents the output feature of enc N in each layer, represents the feature map output of each layer of enc N , the superimposed feature map obtained before inputting the decoder, and represent up-sampling and down-sampling operations respectively; represents the operation of superimposing all channels.

[0070] Train S-U-Net3+ four pairs of encoders from N =1 to N =4, and train S-U-Net3+ four pairs of decoders from N =4 to N =1, and when N =5, train X which is both an encoder and a decoder, since the encoder learns to minimize ; the decoder learns to minimize ; in summary, when training the paired enc N and dec N , the problem to be solved is shown in equation (4): (4).

[0071] The feature fusion part is composed of one Concat layer and five convolution layers. These convolution layers use 3x3 filters to compress the number of channels, and the number of filters in each layer is 128, 64, 32, 16, and 1, respectively. Batch normalization layers (BN) are also used to alleviate gradient explosion, and the activation function LeakyReLU adaptively learns parameters to improve accuracy and speed up training. The feature extraction part obtains CT and MR feature maps with a size of 128x128x280, which are sequentially spliced by the Concat layer and then undergo channel dimension reduction by five convolution layers. Finally, the fused image (Fused image) is compressed to a size of 128x128x1, which is rich in deep details and fully represents the key features of the two modalities.

[0072] Discriminator

[0073] The discriminator of the KDE-GAN can use LIME to generate an interpretable image for classifying images as real or fake. Based on the interpretable image, the performance of the discriminator can be monitored dynamically. When the performance of the discriminator reaches the optimal state, the training of the discriminator is stopped to reduce overfitting caused by overtraining (see Figure 6 ).

[0074] 1) Interpretable part

[0075] Real images and the fused image G(z) obtained by the generator G are input into the discriminant module for image classification. D(G(z)) and D(real) represent the classification results of the discriminant module for G(z) and real images, respectively. Taking brain images as an example, Real represents images classified as brain CT, and Fake represents images classified as non-brain CT images. This discriminant module has a total of five convolution layers with 3x3 filters, and the number of filters is 64, 128, 128, 256, and 256, respectively. The convolution step size of the first layer is set to 1, and the step size of the remaining layers is set to 2. The dropout of the fully connected layer is set to 0.4 (see Figure 7 ). The input image is normalized by five convolution layers and a fully connected layer to obtain the classification result. LIME visualizes the explanation of half of the images classified as Real. The discriminant loss and the generator loss are updated using the discriminant result, as shown in equations (5) and (6):

[0076] (5) ;

[0077] (6) ;

[0078] 2) Dynamic threshold adjustment

[0079] To dynamically control the training of the discriminator, the training of the remaining half of the images is dynamically adjusted according to the interpretable images output by the interpretable module for the first half of the images classified as Real; the evaluation of the explanation effect is divided into subjective evaluation and objective evaluation;

[0080] The subjective evaluation method is to judge whether the performance of the discriminator reaches the best by observing the proportion of the key feature activated area covered in the interpretable image; for example, brain tumor images must ensure that the brain detail area in the image is covered. The image covered by invalid areas such as background is an invalid image.

[0081] The objective evaluation is to measure whether the performance of the discriminator meets the standard after the first half of the images is trained by taking an interpretable threshold (0 <1) as the standard; The calculation method of the initial value of

[0082] (7);

[0083] Wherein, is analogous to the interpretable threshold, represents the probability distribution of the real image, represents the probability distribution of the generator generating the fusion image; when tends to 1, it means that the fusion image generated by the generator is too few to be identified as real, and the performance of the generator is poor; when tends to 0, it means that the real image is too few to be identified as real, and the discrimination ability of the discriminator is insufficient; therefore, when tends to , the threshold of the discriminator reaches the best ( );

[0084] According to the subjective evaluation method to obtain the explanation image, the objective evaluation method is adjusted step by step; when the proportion of the key feature activated area covered is too small, it means that the discrimination of the discriminator on the key feature is not enough, and the threshold needs to be reduced, otherwise, the threshold needs to be raised; the average value of the discrimination loss of the real image and G(z) is taken as the loss variable T D , when , it means that the loss is too large, and the correct rate of the discriminator is low, and the discriminator needs to be trained using ; when , it means that the discrimination loss of the discriminator has converged to a safe range, and the training of the discriminator can be skipped; it is worth noting that whether the discriminator is trained or not, the generator needs to be trained using .

[0085] The overall training of the model aims to train G and Dc, Dm with LIME, respectively, using the training set data. First, fix G to train Dc, Dm, then fix Dc, Dm to train G, and then cycle training to improve the ability of G to generate fused images and the ability of Dc, Dm to distinguish the difference between generated images and real images, until the generated images of G are sufficient to pass the discrimination of Dc, Dm, and the training of G is completed. The test set data is input into the trained G to obtain the final fusion result. In the test phase, the trained generator is used to generate the fusion image, and the overall training algorithm of KDE-GAN is shown in Algorithm 1 (Algorithm 1), where Iteration refers to the total number of epochs to be learned. The loss variable T D The process of dynamically training the discriminator is shown in Algorithm 2 (Algorithm 2).

[0086]

[0087]

[0088]

[0089] To verify the fusion performance of the above-mentioned model KDE-GAN of the application, the method of the application is experimentally performed on four pairs of test images, and nine methods are selected as comparative experiments, including NestFuse, IFCNN, DenseFuse based on neural network; DTCWT, SGF, FPRSGF based on filter; FusionGAN, DDcGAN, UC-DDGAN based on GAN. The application analyzes the progress of KDE-GAN from both qualitative and quantitative aspects. The test images include the 17th slice of CT and MR-T1 images of sarcoma, the 13th slice of CT and MR-T2 images of cerebral toxoplasmosis, the 17th slice of CT and MR-PD images of meningioma, and the 10th slice of CT and MR-Gad images of metastatic bronchial tumor. The size of all test images is 256x256 pixels, and the gray level is 256 levels. The subjective visual effect is shown in Figures 9-12 The objective evaluation index value is shown in Table 2.

[0090] Table 2 Objective evaluation index value

[0091]

[0092] Bold data is the optimal value

[0093] Figures 9 to 12 Display: the fusion result obtained by the NestFuse method (see Figures 9-12 (a) The edge is blurred, which is not conducive to the observation of the lesion contour by the doctor; the fusion result obtained by the IFCNN method (seeFigures 9-12 (b) The brightness is too low, resulting in the loss of a large amount of detailed information; the fusion result obtained by the DenseFuse method (see...) Figures 9-12 (c) Edge information was not well preserved; the fusion results obtained by the DTCWT method (see...) Figures 9-12 (d) Excessive brightness causes distortion of bone detail information; the fusion results obtained by the SGF method (see...) Figures 9-12 (e) Low contrast easily leads to edge block artifacts; the fusion results obtained by the FPRSGF method (see...) Figures 9-12 (f) Poor shaving contrast can easily cause partial blurring of the texture structure, resulting in the loss of a large amount of detail information; the fusion results obtained by the FusionGAN method (see...) Figures 9-12 (g) Insufficient deep detail representation, making it impossible to observe internal details; the fusion results obtained by the DDcGAN method (see...) Figures 9-12 (h) The edges are not smooth enough; the fusion results obtained by the UC-DDGAN method (see...) Figures 9-12 (i) The internal tissue boundaries are blurred, and the disease fusion results obtained by KDE-GAN (see...) ​ (j) The skeletal details are clear, and the soft tissue color layers are distinct. In summary, the fusion effect of the knowledge distillation and interpretable multimodal medical image fusion model KDE-GAN in this application is superior to the other nine fusion methods used for comparison. Applying it to multimodal medical image fusion can assist doctors in making accurate diagnoses and treatments for lesions.

Claims

1. A multi-modal medical image fusion method, characterized by, The application relates to a knowledge distillation and explainable multi-modal medical image fusion model, which comprises a generator G and two discriminators Dc and Dm, the generator is responsible for generating a fusion image, and the discriminators are used for discriminating between a real image and the fusion image; wherein the generator comprises a feature extraction part and a feature fusion part, the feature extraction part uses a student network obtained by removing redundant channels through knowledge distillation, and the feature fusion part comprises a Concat layer and five convolution layers; the discriminators comprise a discrimination module and an explainable module, the discrimination module is used for discriminating between a real image and a fusion image, and the explainable module is used for generating an explanation image of an image discriminated as "Real" by the discrimination module of the discriminator. The method comprises the following steps: Step 1: in the feature fusion part, each fusion path is subjected to knowledge distillation by using a U-Net3+ with a cross-modal attention block as a teacher model to obtain a student model for extracting image features, and then the student model is used to replace the teacher model to extract deep features of the image to be fused; Step 2: the deep features extracted are fused in the feature fusion part, and a fusion image is generated after channel dimension reduction and convolution operation; Step 3: the discriminator is provided with an explainable module to generate an explanation image of an image discriminated as "Real" by the discrimination module of the discriminator, the accuracy of the key features covered by the explanation image is determined, the discriminator is stopped from training in advance when the discrimination performance is optimal, and the training does not need to be completed on the whole data; in the training process, the generator and the discriminator of the model are optimized together, and the trained generator can generate a clear fusion image; The detailed steps of obtaining the student model through knowledge distillation and applying the student model to the feature extraction part of the model in step 1 are as follows: (1) constructing a student network structure The teacher network comprises four encoders ENC1, ENC2, ENC3 and ENC4 and four decoders DEC1, DEC2, DEC3 and DEC4; the filter numbers of the encoders ENC1, ENC2, ENC3 and ENC4 are 64, 128, 256 and 512 respectively; the decoder DEC1, DEC2, DEC3 and DEC4 have 320 filters, that is, 64 filters with a size of 3*3 are used to compress or expand the channels of the five input feature maps of the upper layer encoder and the lower layer decoder to 64 respectively, and then the 64 channels of the five input feature maps are stacked into 320 channels; each layer of the encoder has a large number of channels, but the contribution of some channels is very small, which is equivalent to invalid redundant channels; by setting a threshold value gamma, the attention feature map output by the cross-modal attention block is binarized to obtain a specific number of redundant channels, assuming that the cross-modal attention block outputs an attention feature map with T channels, and the activation values of t' channels are not all 0, after ensuring that the input of each decoder is equal to the output of the encoder, as many channels as possible with the activation value of 0 are removed to obtain a binary attention feature map, and the number thereof is set as t, wherein t>=t'. The filter numbers of the four encoders enc1, enc2, enc3 and enc4 of the student network are 56, 112, 224 and 448 respectively; the module x is both an encoder and a decoder, and is built as X, and the channel number is 896; the input channel of each decoder dec1, dec2, dec3 and dec4 is set to 280, so as to ensure that the output channel number of the student network encoder matches the input channel number of the decoder, and thus the student network for replacing the teacher network to complete the feature extraction task is constructed. (2) Knowledge transfer The knowledge transfer is divided into encoder learning and decoder learning. Taking the learning of enc2 and dec2 of the student network as an example, the learning processes of other encoders and decoders are similar to them; during the encoder learning, the image I is input into the first layer ENC1 and enc1 respectively, and then trained by the encoders and decoders located at each layer. The output features of the encoders and decoders located at each layer are N , and the output features of the encoders and decoders located at each layer are N To make the output features of ENC N closer to the output features of enc N , the difference between the two is used as an encoder loss function to optimize enc N , and the encoder loss function is as shown in formula (1); during the decoder learning, the decoder loss function is used to make the output features of dec N closer to the input features of enc N , and make the feature extraction result of the image I output by dec1 closer to I rec as shown in formula (2)​​​​​​​ ； (3) In formula (2) represent the output of each layer dec N , the third term loss of formula (2) is the perceptual loss, which updates the image by back propagation, promotes feature extraction, and it is worth noting that when N = 1, the first term loss is not calculated, and when N = 5, the first term loss is 0; In formula (3) represent the output feature of each layer enc N , represent the output feature map of each layer of enc N , the superimposed feature map obtained before inputting the decoder, and respectively represent the up-sampling and down-sampling operations, represent the operation of superimposing all channels; Training four pairs of encoders of the teacher-student network from N=1 to N=4, and training four pairs of decoders from N=4 to N=1, when N=5, training X which is both an encoder and a decoder, since the encoder learning aims to minimize ; the decoder learning aims to minimize ; in summary, when training the pairs of enc N and dec N , the problem to be solved is shown in equation (4): (4)。 2. The multi-modal medical image fusion method of claim 1, characterized in that, The specific process of step 2 is that the feature extraction part obtains a feature map with a size of 128x128x280, which is sequentially spliced through a Concat layer, and then is subjected to channel dimension reduction through five convolution layers, and finally is compressed into a size of 128x128x1, so as to obtain a fusion image with rich deep details and fully representing two modal key features.

3. The multi-modal medical image fusion method of claim 1, wherein: The specific process of step 3 is that the discriminator adds an interpretable module to dynamically monitor the performance of the discriminator, so that the training of the discriminator is stopped when the performance is optimal. (1) The real image and the fusion image G(z) obtained by the generator G are input into the discrimination module for image classification, D(G(z)), D(real) respectively represent the classification results of the discrimination module on G(z) and the real image, taking the brain image as an example, Real represents the image classified as brain CT, Fake represents the image classified as non-brain CT image; the interpretable module visually interprets half of the images classified as Real, and the discrimination result is used to update the discriminator loss and the generator loss as shown in equations (5)-(6): (5) (6) (2) Dynamic threshold adjustment In order to dynamically control the training of the discriminator, the training of the remaining half of the images is dynamically adjusted according to the interpretable images of the first half of the images classified as Real output by the interpretable module; the evaluation of the interpretation effect is divided into subjective evaluation and objective evaluation. The subjective evaluation method is to judge whether the performance of the discriminator reaches the best by observing the proportion of the key feature activated area covered in the interpretable image. Objective evaluation is to measure the performance of the discriminator after the first half of the images are trained with an interpretable threshold <1 as a standard, whether the performance of the discriminator meets the standard after the first half of the images are trained; The initial value of the initial value is calculated as follows:​ (7) wherein, is analogous to an explainable threshold, represents a probability distribution of real images, represents a probability distribution of the generator generating the fusion image; when tends to 1, it means that the fusion image generated by the generator is too few to be identified as real, and the performance of the generator is poor; when tends to 0, it means that the real image is too few to be identified as real, and the discrimination ability of the discriminator is insufficient; therefore, when tends to infinity, the discriminator threshold reaches the optimal . Based on the interpretation of the image obtained from the subjective evaluation method, the objective evaluation method was gradually adjusted. If the proportion of key features covered by the activated region is too small, it indicates that the discriminator's ability to distinguish key features is insufficient, and the threshold needs to be lowered; conversely, if the proportion is large, the threshold needs to be raised. The average of the discrimination loss of the real image and G(z) is used as the loss variable T. D ,when When the error rate is too high, it indicates that the discriminator's accuracy is low, and it needs to be used again. Training the discriminator; when When this occurs, it indicates that the discriminator's discrimination loss has converged to a safe range, and discriminator training can be skipped. It is worth noting that regardless of whether the discriminator is trained, it is necessary to utilize... Training the generator.

4. The multi-modal medical image fusion method of claim 1, characterized in that: The specific process of step 3 is that the generator and the discriminator of the model are jointly optimized, that is, the generator G is first fixed to train the discriminators Dc and Dm, then the discriminators Dc and Dm are fixed to train the generator G, and then the training is cycled to improve the ability of the generator G to generate fusion images and the ability of the discriminators Dc and Dm to distinguish the difference between the generated images and the real images, until the generated images generated by the generator G are sufficient to pass the discrimination of the discriminators Dc and Dm, and the training of the generator G is completed; after that, the test set data is input into the trained generator G to obtain the final fusion result.

5. The multi-modal medical image fusion method of claim 1, wherein: The student network includes four encoders, and the filter numbers are 56, 112, 224 and 448 respectively; the module x is both an encoder and a decoder, and is built as X, and the channel number is 896; the input channels of the four decoders are all set to 280.

6. The multi-modal medical image fusion method of claim 1, wherein: The convolution layers of the feature fusion part use 3x3 filters to compress the channel number, and the filter numbers of the layers are 128, 64, 32, 16 and 1 respectively; batch normalization layers are also used to alleviate gradient explosion; the activation function LeakyReLU adaptively learns parameters to improve accuracy and speed up training.

7. The multi-modal medical image fusion method of claim 1, wherein: The discrimination module of the discriminator includes five convolution layers and a fully connected layer, each convolution layer has a 3x3 filter, and the filter numbers are 64, 128, 128, 256 and 256 respectively; the convolution step length of the first layer is set to 1, and the step lengths of the other layers are set to 2; the dropout of the fully connected layer is set to 0.4.

Images