Weakly supervised medical image fusion method with modal synthesis and enhancement

Abstract

The present application belongs to the technical field of image fusion, and particularly relates to a weakly supervised medical image fusion method for modality synthesis and enhancement. In view of the problems of human body injury and high cost caused by multiple imaging of the patient for multi-modal image fusion, long time and high difficulty of image fusion requiring registration of a large number of paired data sets, and unclear texture and fine structure of the fused image, the present application provides a weakly supervised medical image fusion method for modality synthesis and enhancement based on CycleGAN and OctopusNet, which is composed of three parts of modality synthesis, image enhancement and image fusion.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image fusion, and particularly relates to a weakly supervised medical image fusion method for modality synthesis and enhancement. BACKGROUND

[0002] With the continuous development of medical imaging, medical images have become an effective tool for disease diagnosis. There are various modalities of medical images today, such as magnetic resonance (MR) images, computed tomography (CT) images, positron emission tomography (PET) images, etc., and different modalities have their own advantages and limitations. For example, CT images can well show bone information, but cannot clearly represent structure information such as soft tissue; MR images can fully show soft tissue information, but have great defects in representing bone information; PET images can provide rich metabolic information for the clinic, but have low resolution. Therefore, integrating image information from CT, MR and PET three modalities into one image to complete multi-modal image fusion can achieve complementary advantages. Multi-modal fusion images not only retain the features of the original images, but also make up for the defects of single-modal medical images, and show rich detailed information, providing an effective basis for clinical diagnosis and treatment.

[0003] In recent years, deep learning has been widely applied in the field of image processing such as modality synthesis, image fusion and image segmentation due to its strong feature extraction capability, and good results have been achieved. Among them, convolutional neural network (CNN) as a representative method of deep learning plays an increasingly important role in the field of image fusion. As a kind of deep learning method, generative adversarial network (GAN) generates output through the mutual game of generation module and discrimination module, and this mutual game process improves the accuracy of image feature extraction, so that the fusion image retains more source image information. However, the above methods are all supervised training methods, and a large number of paired data sets with registration are needed for model training in the network training process, but a lot of time and labor cost is needed in the image registration process.

[0004] In addition, in the actual medical diagnosis process, multiple medical imaging is needed to obtain the multi-modal fusion image. However, the high radiation in medical imaging can cause harm to the human body, and the medical imaging is expensive, which requires a large amount of spending. Therefore, from the actual point of view, it is necessary to introduce a multi-modal medical image fusion method which can reduce the harm of medical imaging to the human body and reduce the cost. As a kind of image processing technology, modal synthesis can complete the synthesis of single modal image into another modal image. CT and MR images are common modalities, and contain rich feature information. The synthesis of MR images with more texture detail information into CT images with less detail information can retain more image information than the synthesis of CT images into MR images. Therefore, modal synthesis is introduced as the basis of image fusion to reduce the modal of input image in the image fusion process, which can reduce the imaging times of patients, reduce the harm of medical images to the human body, and save the medical expenses of patients.

[0005] In the existing deep learning-based image fusion methods, the DenseFuse fusion method based on CNN proposed by Li et al. and the IFCNN fusion method proposed by Zhang et al. have good feature fusion ability, but the above two methods have low accuracy in feature extraction process and lack of specificity for medical image texture detail information. Zhou et al. proposed a Hi-Net hybrid fusion network based on GAN network, which effectively improves the fusion performance of the image, but the representation of fine structure is still not clear enough.

[0006] With the rise of generative adversarial networks in recent years, it has achieved remarkable results in modal synthesis. Nie et al. proposed a modal synthesis method based on a single generator and discriminator of GAN network, which generates realistic synthetic images through adversarial learning. Although the synthetic image has good structural features compared to the target prediction image, the image details are blurred and the image texture details are not well represented. Wang et al. proposed an automatic context-based GAN network synthesis method to make the synthetic image retain more detail features, which extracts more image features from the source image using an automatic context method to strengthen the key information in the synthetic image. However, the clarity of the synthetic image in representing texture detail information and fine structure still needs to be improved. Zhu et al. proposed a CycleGAN based on unsupervised cyclic consistency using double generators and double discriminators, which significantly improves the clarity of the synthetic image in representing detail information, but the representation of fine structure is still not clear enough. Jiang et al. proposed a dual-generator structure (global-local generator) modal synthesis network, which uses two generators to learn and present global information (image structure information) and to enhance details (image texture and fine structure), respectively, so that the synthetic image has complete global structure representation while enhancing the image details. However, the representation of texture details and fine structure is not stable.

[0007] Therefore, in view of the problems of human body injury and high cost caused by multiple imaging of patients for multi-modal image fusion, long time and great difficulty of image fusion requiring registration of a large number of paired data sets, and unclear texture and fine structure of the fused image, the application provides a weakly supervised medical image fusion method based on modal synthesis and enhancement of CycleGAN and OctopusNet. SUMMARY

[0008] In view of the above problems, the application provides a weakly supervised medical image fusion method based on modal synthesis and enhancement of CycleGAN and OctopusNet.

[0009] In order to achieve the above purpose, the application adopts the following technical solutions:

[0010] A weakly supervised medical image fusion method based on modal synthesis and enhancement, which is composed of modal synthesis, image enhancement and image fusion, and includes the following steps:

[0011] Step 1, in the modal synthesis part, the MR image is input into the generator obtained by training the multi-generator modal synthesis network to generate a deep structure synthesis image s CT_stru and a shallow detail synthesis image s CT_det .

[0012] Step 2, in the image enhancement part, the MR image is input into the generator obtained by training the double-generator image enhancement network to generate a deep structure enhancement image s MR_stru and a shallow detail enhancement image s MR_det .

[0013] Step 3, in the image fusion part, s CT_stru , s CT_det , s MR_stru , s MR_det and the original PET image are taken as inputs to generate a CT, MR and PET three-modal fusion image through a multi-modal image fusion generative adversarial network.

[0014] Further, the modal synthesis part includes two stages: modal synthesis training and modal synthesis prediction; in the modal synthesis training stage, the multi-generator modal synthesis network takes CycleGAN as the basis, and divides the generator into a deep structure generator G CT_deep for synthesizing MR images into CT images and a shallow detail generator G CT_shallow , a deep structure generator G MR_deep for synthesizing CT images into MR images and a shallow detail generator G MR_shallow , four generators and modal synthesis discriminators D CT and D MR .the mutual adversarial cycle training network, constantly extracting and improving the deep structure and shallow detail feature extraction of the image, and the image synthesis performance of the generator; in the modal synthesis prediction stage, the MR image is input as the input, and the deep structure generator G CT_deep and the shallow detail generator G CT_shallow generate a deep structure synthesis image s CT_stru and a shallow detail synthesis image s CT_det .

[0015] Further, the modal synthesis training stage uses a weakly supervised method to train the network, using a small amount of paired data set and a large amount of non-paired data set as training data, and improving the specificity of image structure and texture details and fine structure through a small amount of paired CT and MR images; a large number of non-paired CT and MR images reduce the difficulty and time of registration. As the basis of image fusion, the modal synthesis solves the problem that image fusion requires registered paired data, and it is time-consuming and difficult to obtain a large amount of paired data set.

[0016] The specific training process is as follows:

[0017] The MR image is input into G CT_deep and G CT_shallow to generate a CT deep structure synthesis image S CT_structure and a CT shallow detail synthesis image S CT_details , and the CT image is also input into G MR_deep and G MR_shallow to generate an MR deep structure synthesis image S MR_structure and an MR shallow detail synthesis image S MR_details ; then S CT_structure and S CT_details continue to enter G MR_deep and G MR_shallow , S MR_structure and S MR_details continue to input G CT_deep and G CT_shallow cycle training network, through G CT_deep , G CT_shallow and D CT , G MR_deep , G MR_shallow and D MR , the modal synthesis performance of the generator is improved through mutual adversarial.

[0018] Further, the deep structure generator G CT_deep of the MR synthesized CT image and the deep structure generator G MR_deepFor highlighting high-level information such as overall image structure, all are composed of three parts of encoding, conversion and decoding; the encoding part is composed of two convolution layers Conv6, Conv7, wherein the convolution layer Conv6 represents a 7x7, k filter and step length of 1 Convolution-InstanceNorm-ReLU layer; the convolution layer Conv7 is a 3x3, k filter and step length of 2 Convolution-InstanceNorm-ReLU layer; the conversion part is composed of 9 Residual blocks, and the decoding part is composed of one deconvolution layer DeConv4 and one convolution layer Conv8, DeConv4 is a 3x3, k filter and step length of 1 / 2 fractional-strided-Convolution-InstanceNorm-ReLU layer; Conv8 represents a 7x7, k filter and step length of 1 Convolution-InstanceNorm-ReLU layer;

[0019] The shallow detail generator G of the MR synthesized CT image CT_shallow And the shallow detail generator G of the CT synthesized MR image MR_shallow For image detail enhancement, highlighting low-level information such as image texture and fine structure, also composed of three parts of encoding, conversion and decoding; the encoding part is composed of two convolution layers Conv6, Conv7, wherein Conv6 represents a 7x7, k filter and step length of 1 Convolution-InstanceNorm-ReLU layer, Conv7 is a 3x3, k filter and step length of 2 Convolution-InstanceNorm-ReLU layer; the conversion part completes the synthesis from one modality to another modality, and the conversion part is composed of 9 Residual blocks; the decoding part is composed of one deconvolution layer DeConv4 and one convolution layer Conv8, wherein the deconvolution layer DeConv4 is a 3x3, k filter and step length of 1 / 2 fractional-strided-Convolution-InstanceNorm-ReLU layer, and the convolution layer Conv8 represents a 7x7, k filter and step length of 1 Convolution-InstanceNorm-ReLU layer.

[0020] Further, the G CT_deep , G CT_shallow and D CT , G MR_deep , G MR_shallow and D MR The mutual confrontation of G is specifically:

[0021] In the modal synthesis training process, D CT , D MR discriminates the authenticity of the synthesized image and the original training image to improve the synthesis effect of the deep structure and shallow detail generator. In the process of synthesizing CT images from MR images, D CT discriminates the authenticity between the synthesized image S CT_structure , S CT_details and the original CT training image; in the process of synthesizing MR images from CT images, D MR discriminates the authenticity between S MR_structure , S MR_details and the original MR training image. In the training process, (1) through the adversarial loss and the cycle consistency loss, X MR and Y CT domains of deep structure information and X′ MR and Y′ CT domains of shallow detail information are learned to be enhanced;

[0022] (2) the network includes four mapping functions: G:X MR →Y CT , F:Y CT →X MR , M:X′ MR →Y′ CT and N:Y′ CT →X′ MR ;

[0023] (3) given a training sample where x i ∈X MR , X′ MR , y j ∈Y CT , Y′ CT , N is a positive integer;

[0024] (4) the data distribution is represented as x~p data (x), y~p data (y), and the adversarial loss is applied to the four mapping functions;

[0025] (5) the adversarial loss of MR synthesizing CT image G:X MR →Y CT , M:X′ MR →Y′ CT and D CT is shown in formulas (1) and (2):

[0026]

[0027]

[0028] CT synthetic MR image F:Y CT → X MR , N:Y' → X' CT → X' MR and D MR The adversarial loss of F:Y

[0029]

[0030]

[0031] (6) For G:X MR → Y CT and F:Y CT → X MR mapping and M:X' MR → Y' CT and N:Y' CT → X' MR mapping, the cycle consistency loss is introduced, as shown in formulas (5) and (6):

[0032]

[0033]

[0034] (7) The total objective function highlighting deep structure information is shown in formula (7):

[0035]

[0036] The total objective function highlighting shallow detail information is shown in formula (8).

[0037]

[0038] Where λ1, λ2 are the relative importance of the control target;

[0039] (8) The total objective function is shown in formula (9):

[0040]

[0041] Further, the image enhancement part includes two stages of image enhancement training and image enhancement prediction. In the image enhancement training stage, the dual generator image enhancement network is based on GAN, which contains a structure generator G structure , a detail generator G detail and an enhancement discriminator D en inside. Through the structure generator G structure , the detail generator G detail and the enhancement discriminator Den The mutual competition between the two generators improves the feature enhancement performance;

[0042] The structure generator G structure Detail Generator G detail Both consist of two parts: encoding and decoding. The structure generator G... structure The internal structure is as follows: The encoding part consists of two convolutional layers, Conv6 and Conv7. Conv6 is a 7×7 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1; Conv7 is a 3×3 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 2. The decoding part consists of one deconvolutional layer, DeConv4, and one convolutional layer, Conv8. DeConv4 is a 3×3 fractional-strided Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1 / 2; Conv8 is a 7×7 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1.

[0043] The detail generator G detail The internal structure is as follows: the encoding part consists of two convolutional layers, Conv6 and Conv7. Conv6 is a 7×7 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1, and Conv7 is a 3×3 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 2. The decoding part consists of one deconvolutional layer, DeConv4, and one convolutional layer, Conv8. DeConv4 is a 3×3 fractional-strided Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1 / 2, and Conv8 is a 7×7 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1.

[0044] The enhanced discriminator D en Internal structure identification CycleGAN discriminator.

[0045] Further, the multi-modal image fusion generative adversarial network is based on OctopusNet and consists of three parts of encoding, feature fusion and decoding; the encoding part sets a separate encoder for each input image, the encoder consists of 3 encoding blocks EB1, EB2 and EB3 and 1 decoding block DB, each encoding block consists of 1 convolutional layer and 1 Dense block, which are used for deep feature extraction, each Dense block has 3 convolutional layers, and the output of each layer is cascaded as the input of the next layer, the decoding block DB consists of 5 convolutional layers, the input image is extracted by the encoding block and the decoding block from shallow to deep layer by layer, and the corresponding image feature blocks OUT1, OUT2, OUT3 and OUT4 are output;

[0046] The feature fusion part fuses the image feature blocks obtained by the encoding part in the same layer to obtain fusion blocks FOUT1, FOUT2, FOUT3 and FOUT4, and the features of each layer extracted by the encoder of the original PET image are connected in series with the fusion blocks FOUT1, FOUT2, FOUT3 and FOUT4 to obtain output blocks FFOUT1, FFOUT2, FFOUT3 and FFOUT4;

[0047] The decoding part inputs the output blocks FFOUT1, FFOUT2, FFOUT3 and FFOUT4 into a max-pooling layer for down-sampling, so that the channel number of each output block is 1, and generated images F1, F2, F3 and F4 are obtained, and the images F1, F2, F3 and F4 are input into an image fusion generator G F to output a final fusion image, and an image fusion discriminator D F is used to respectively identify the generated images F1, F2, F3 and F4 and the image fusion generator G F outputs the final fusion image; wherein the image fusion generator G F learns the mapping function W: P→Q between the P-domain generated image and the Q-domain final fusion image, and the adversarial loss of the image fusion discriminator D F is shown in formula (10):

[0048]

[0049] wherein, is the adversarial loss, L densefuse is the internal structure loss function of the image fusion generator G F , L densefuse contains pixel loss and SSIM loss, and a is 0.1.

[0050] Compared with the prior art, the present application has the following advantages:

[0051] (1) proposed based on modal synthesis, image enhancement image fusion method, which consists of modal synthesis, image enhancement and image fusion three parts. In the modal synthesis part, a multi-generator modal synthesis network based on cycle consistency generative adversarial network (CycleGAN) is proposed, which divides the generator into deep structure and shallow detail generator. The deep structure generator is used to highlight the overall image structure and other high-level information, and the shallow detail generator highlights the image texture and fine structure and other low-level information. In the image enhancement part, the MR image is input to the structure and detail generator trained by the dual-generator image enhancement network based on generative adversarial network (GAN), to generate an enhanced MR image with more prominent structure and texture detail information. In the image fusion part, the CT and MR images generated by modal synthesis and image enhancement and the original PET image are input to the multi-modal image fusion generative adversarial network based on the deep learning segmentation network (OctopusNet) of multi-modal medical images. In the encoding part, the image features are extracted layer by layer from shallow to deep. In the feature fusion part, the obtained feature maps of each layer are fused and input to the decoding part. In the decoding part, the image fusion generator and the image fusion discriminator are used to improve the image fusion performance, and the multi-modal fusion image is obtained.

[0052] (2) proposed a weakly supervised training method based on modal synthesis. The modal synthesis training data consists of a small amount of paired data set and a large amount of unpaired data set. The small amount of paired data set improves the specificity of image structure and texture detail, fine structure; a large number of unpaired data sets reduces the registration difficulty and time. As the basis of image fusion, modal synthesis retains more structure and detail information of the source image, and solves the problem of obtaining a large amount of paired data set which is time-consuming and difficult.

[0053] (3) The invention realizes the fusion of CT, MR and PET three modal images. The fusion image retains the rich detail information of each modality. Compared with 10 kinds of current advanced modal synthesis and image fusion methods on the public brain image data set of Harvard Medical School, the synthesis and fusion images are improved in 5 evaluation indexes, which can assist doctors to make more accurate diagnosis and treatment of cancer, brain tumor, etc.

[0054] (4) Through experimental verification, it is known that the invention can generate a fusion image with complete structure representation, clearer texture and fine structure detail information, which has good performance in subjective visual observation and objective index evaluation. The application of the invention in multi-modal medical image fusion can assist doctors to make accurate diagnosis and treatment of the lesion site. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1This is a diagram illustrating the overall framework of a weakly supervised medical image fusion method that combines modality synthesis and enhancement.

[0056] Figure 2 This is a diagram illustrating the overall framework of a multi-generator modal synthesis network.

[0057] Figure 3 This is a training graph for modal synthesis.

[0058] Figure 4(a) shows the internal structure of the structure generator, and Figure 4(b) shows the internal structure of the detail generator.

[0059] Figure 5 This is a prediction graph for modal synthesis.

[0060] Figure 6 Diagram of the dual-generator image enhancement network structure;

[0061] Figure 7 Generate adversarial network structure diagram for multimodal image fusion;

[0062] Figure 8 This is a diagram of the internal structure of an image encoder.

[0063] Figure 9 Qualitative comparison results of MSE-Fusion and six image fusion comparison methods on CT / MR / PET images;

[0064] Figure 10 Qualitative comparison results of MSE-Fusion and four modality synthesis methods on CT images;

[0065] Figure 11 Quantitative results for MSE-Fusion and six image fusion comparison methods;

[0066] Figure 12 For qualitative comparison results of ablation experiments;

[0067] Figure 13 This is the qualitative comparison result of the ablation experiment 2. Detailed Implementation

[0068] Example 1

[0069] like Figure 1 As shown, a weakly supervised medical image fusion method for modality synthesis and enhancement consists of three parts: modality synthesis, image enhancement, and image fusion, including the following steps:

[0070] Step 1: In the modality synthesis part, the MR image is used to generate a deep structure composite image s through a generator trained by a multi-generator modality synthesis network. CT_stru Composite image with shallow detail s CT_det ,(like Figure 5 (as shown);

[0071] As shown in Figure 2 , the modal synthesis part includes two stages: modal synthesis training and modal synthesis prediction; in the modal synthesis training stage, the multi-generator modal synthesis network is based on CycleGAN, and the generator is divided into a deep structure generator G CT_deep and a shallow detail generator G CT_shallow for synthesizing MR images into CT images, and a deep structure generator G MR_deep and a shallow detail generator G MR_shallow for synthesizing CT images into MR images; through the mutual antagonistic cycle training network of the four generators and the modal synthesis discriminators D CT , D MR , the image deep structure and shallow detail features are continuously extracted and the generator image synthesis performance is improved; in the modal synthesis prediction stage, the MR image is input, and through the trained deep structure generator G CT_deep and shallow detail generator G CT_shallow for synthesizing MR images into CT images, the deep structure synthesis image s CT_stru and the shallow detail synthesis image s CT_det are generated.

[0072] As shown in Figure 3 , the modal synthesis training stage adopts a weakly supervised manner to train the network, and the specific training process is as follows:

[0073] The MR image is input into G CT_deep and G CT_shallow , and the CT deep structure synthesis image S CT_structure and the CT shallow detail synthesis image S CT_details are generated; the CT image is also input into G MR_deep and G MR_shallow , and the MR deep structure synthesis image S MR_structure and the MR shallow detail synthesis image S MR_details are generated; then S CT_structure and S CT_details continue to enter G MR_deep and G MR_shallow , S MR_structure and S MR_details continue to input G CT_deep and G CT_shallow , and the cycle training network is trained; through the mutual antagonism of G CT_deep , G CT_shallow and D CT , G MR_deep , G MR_shallow and D MR , the modal synthesis performance of the generator is improved.

[0074] As shown in Fig. 4(a), the deep structure generator G of the MR synthesized CT image CT_deep and the deep structure generator G of the CT synthesized MR image MR_deep are all composed of three parts of encoding, conversion and decoding; the encoding part is composed of two convolution layers Conv6, Conv7, wherein the convolution layer Conv6 represents a Convolution-InstanceNorm-ReLU layer with 7x7, k filters and a step of 1; the convolution layer Conv7 is a Convolution-InstanceNorm-ReLU layer with 3x3, k filters and a step of 2; the conversion part is composed of 9 Residualblocks; the decoding part is composed of one deconvolution layer DeConv4 and one convolution layer Conv8, wherein the deconvolution layer DeConv4 is a fractional-strided-Convolution-InstanceNorm-ReLU layer with 3x3, k filters and a step of 1 / 2; the convolution layer Conv8 represents a Convolution-InstanceNorm-ReLU layer with 7x7, k filters and a step of 1.

[0075] As shown in Fig. 4(b), the shallow detail generator G of the MR synthesized CT image CT_shallow and the shallow detail generator G of the CT synthesized MR image MR_shallow are all composed of three parts of encoding, conversion and decoding; the encoding part is composed of two convolution layers Conv6, Conv7, wherein the convolution layer Conv6 represents a Convolution-InstanceNorm-ReLU layer with 7x7, k filters and a step of 1; the convolution layer Conv7 is a Convolution-InstanceNorm-ReLU layer with 3x3, k filters and a step of 2; the conversion part is composed of 9 Residualblocks; the decoding part is composed of one deconvolution layer DeConv4 and one convolution layer Conv8, wherein the deconvolution layer DeConv4 is a fractional-strided-Convolution-InstanceNorm-ReLU layer with 3x3, k filters and a step of 1 / 2; the convolution layer Conv8 represents a Convolution-InstanceNorm-ReLU layer with 7x7, k filters and a step of 1.

[0076] The G CT_deep , G CT_shallow and D CT , G MR_deep , G MR_shallow and D MRThe specific forms of their mutual confrontation are:

[0077] (1) By using adversarial loss and cycle consistency loss, we learn X to enhance deep structural information. MR With Y CT Domain, shallow detail information X′ MR With Y′ CT domain;

[0078] (2) The network includes four mapping functions: G:X MR →Y CT F:Y CT →X MR M:X′ MR →Y′ CT and N:Y′ CT →X′ MR ;

[0079] (3) Given training samples Where x i ∈X MR ,X′ MR y j ∈Y CT ,Y′ CT N is a positive integer;

[0080] (4) The data distribution is represented as x~p data (x), y~p data (y), applying the adversarial loss to four mapping functions;

[0081] (5) MR-synthesized CT images G:X MR →Y CT M:X′ MR →Y′ CT and D CT The adversarial loss is shown in formulas (1) and (2):

[0082]

[0083]

[0084] CT composite MR image F:Y CT →X MR N:Y′ CT →X′ MR and D MR The adversarial losses are shown in equations (3) and (4):

[0085]

[0086]

[0087] (6) for G:X MR → Y CT and F:Y CT → X MR mapping and M:X' MR → Y' CT and N:Y' CT → X' MR The cyclic consistency loss is introduced as shown in equations (5) and (6):

[0088]

[0089]

[0090] (7) The total objective function highlighting deep structural information is shown in equation (7):

[0091]

[0092] The total objective function highlighting shallow detail information is shown in equation (8).

[0093]

[0094] where λ1 and λ2 are the relative importance of the control objectives;

[0095] (8) The total objective function is shown in equation (9):

[0096]

[0097] Step 2, in the image enhancement part, the MR image is trained by the generator of the dual generator image enhancement network to generate a deep structure enhancement image s MR_stru and a shallow detail enhancement image s MR_det .

[0098] As shown in Figure 6 , the image enhancement part includes two stages of image enhancement training and image enhancement prediction. In the image enhancement training stage, the dual generator image enhancement network is based on GAN, which contains a structure generator G structure , a detail generator G detail and an enhancement discriminator D en for feature enhancement. The structure generator G structure , the detail generator G detail and the enhancement discriminator D en are mutually opposed to improve the feature enhancement performance of the two generators.

[0099] The structure generator G structure , the detail generator G detailBoth consist of encoding and decoding parts, the structure generator G structure The internal structure is: the encoding part consists of two convolution layers Conv6 and Conv7, wherein the convolution layer Conv6 represents a 7x7, k filter and step 1 Convolution-InstanceNorm-ReLU layer; the convolution layer Conv7 is a 3x3, k filter and step 2 Convolution-InstanceNorm-ReLU layer; the decoding part consists of one deconvolution layer DeConv4 and one convolution layer Conv8, the deconvolution layer DeConv4 is a 3x3, k filter and step 1 / 2 fractional-strided-Convolution-InstanceNorm-ReLU layer; the convolution layer Conv8 represents a 7x7, k filter and step 1 Convolution-InstanceNorm-ReLU layer;

[0100] The internal structure of the detail generator G detail The internal structure is: the encoding part consists of two convolution layers Conv6 and Conv7, wherein the convolution layer Conv6 represents a 7x7, k filter and step 1 Convolution-InstanceNorm-ReLU layer, and the convolution layer Conv7 is a 3x3, k filter and step 2 Convolution-InstanceNorm-ReLU layer; the decoding part consists of one deconvolution layer DeConv4 and one convolution layer Conv8, wherein the deconvolution layer DeConv4 is a 3x3, k filter and step 1 / 2 fractional-strided-Convolution-InstanceNorm-ReLU layer, and the convolution layer Conv8 represents a 7x7, k filter and step 1 Convolution-InstanceNorm-ReLU layer;

[0101] The internal structure of the enhanced discriminator D en The internal structure of the CycleGAN discriminator.

[0102] Step 3, in the image fusion part, taking s CT_stru , s CT_det , s MR_stru , s MR_det and the original PET image as input, a multi-modal image fusion generative adversarial network is generated, and in the network training process, the game between the image fusion generator (G F ) and the image fusion discriminator (D F ) is used to improve G FThe image fusion performance is improved, three-mode image fusion is realized, and CT, MR and PET three-mode fusion images are obtained.

[0103] As shown in Figure 7 , the multi-modal image fusion generative adversarial network is based on OctopusNet and consists of three parts of encoding, feature fusion and decoding; the encoding part is used to more comprehensively extract deep structure and shallow detail features of the image, so that the fusion image retains more texture details and fine structures while being complete in structure, a separate encoder is set for each input image, the encoder consists of three encoding blocks EB1, EB2 and EB3 and one decoding block DB, as shown in Figure 8 , each encoding block consists of one convolutional layer and one Dense block, which are used for deep feature extraction, each Dense block has three convolutional layers, and the output of each layer is cascaded as the input of the next layer, the decoding block DB consists of five convolutional layers, the input image is extracted layer by layer from shallow to deep through the encoding block and the decoding block, and the corresponding image feature blocks OUT1, OUT2, OUT3 and OUT4 are output, and the encoder network structure is set as shown in Table 1:

[0104] Table 1 Encoder network structure setting

[0105]

[0106] The feature fusion part is used to better retain the detail information contained in different modalities, and the image feature blocks obtained by the encoding part are fused in the same layer to obtain fusion blocks FOUT1, FOUT2, FOUT3 and FOUT4, and the features of each layer extracted by the encoder from the original PET image are mapped and concatenated with the fusion blocks FOUT1, FOUT2, FOUT3 and FOUT4 to obtain output blocks FFOUT1, FFOUT2, FFOUT3 and FFOUT4.

[0107] The decoding part inputs the output blocks FFOUT1, FFOUT2, FFOUT3 and FFOUT4 into a max-pooling layer for down-sampling, so that the number of channels of each output block is 1, and generated images F1, F2, F3 and F4 are obtained, and the images F1, F2, F3 and F4 are input into an image fusion generator G F to output a final fusion image, and an image fusion discriminator D F discriminates the generated images F1, F2, F3 and F4 and the final fusion image output by the image fusion generator G F ; wherein the image fusion generator G F learns a mapping function W: P→Q between the generated images in the P domain and the final fusion images in the Q domain, and the image fusion discriminator D FThe adversarial loss is shown as formula (10):

[0108]

[0109] wherein, is the adversarial loss, L densefuse is the image fusion generator G F is the internal structure loss function, L densefuse contains the pixel loss and the SSIM loss, and a is 0.1.

[0110] The experimental environment and training details of the embodiment: the hardware platform of the experiment is Intel Core i7-11700 with 16G memory, and the GPU is GeForce RTX3060Ti with 8G video memory; the software platform uses Windows 1064-bit operating system, and builds a Pytorch deep learning framework. Limited by the size of the video memory, the experiment is divided into three parts of modal synthesis, image enhancement and image fusion training. In the multi-generator modal synthesis network, λ1=λ2=10 is set, the Adam optimizer with a batch-size of 1 is used for training, and the weights are initialized by a Gaussian distribution with a mean of 0 and a standard deviation of 0.02. During the training, the model starts from 0, the learning rate is kept at 0.0002, and a total of 200 epochs are trained, of which the first 100 epochs keep the same learning speed, and in the next 100 epochs, linearly decay to 0. In the multi-generator image enhancement network, the Adam optimizer with a batch-size of 1 is also used for training, and the network training parameters are the same as those of the multi-generator modal synthesis network. In the multi-modal image fusion generative adversarial network, a is set to 0.1, the stochastic gradient descent (SGD) optimization algorithm is used, the total number of iterations is set to 200, the initial learning rate is set to 0.001, and the batch-size is set to 1. During the training process, the parameters of the discriminator and the generator are updated alternately in each epoch, and as the number of training increases, the loss value gradually decreases, and when the number of training reaches 200, the loss value is close to 0. Then the training is completed, and the model parameters of the generator are obtained.

[0111] Embodiment 2

[0112] Dataset: The experiment selected high-quality MR, CT and PET images with high clarity, rich texture and complex details from the Harvard Medical School, MedPix public medical database, and the normal brain and brain tumor disease open source by the Shanxi Key Laboratory of Biomedical Imaging and Big Data. The selected data set is divided into training set and test set two parts, the training set contains a small part of 40 pairs of registered MR, CT images, and a large number of 2500 non-paired MR, CT, PET images, and the test set contains 20 groups of brain MR, CT and PET test images. The data set is adjusted to 256x256. In order to fully train the model and avoid overfitting, four data enhancement methods of brightness enhancement, contrast enhancement, horizontal flip and random direction flip are used for preprocessing, and four times of data are obtained, that is, 160 pairs of paired MR, CT images, and 10000 non-paired MR, CT, PET images are trained.

[0113] Evaluation index: In order to objectively evaluate the performance of different image fusion methods, five commonly used indexes, mutual information (MI), spatial frequency (SF), peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and edge information retention (Q AB / F ) are used to evaluate the quantitative performance of the proposed method on multi-modal medical image data set. MI is used to measure the similarity of source image and synthesized image, fusion image in gray distribution, which reflects the correlation between them. The more information extracted, the higher the MI value, and the richer the information obtained from the source image; SF reflects the overall activity of the image in the spatial domain, the higher the SF index, the clearer the image, and the higher the image resolution; PSNR is based on the error between corresponding pixel points, that is, the error-sensitive image quality evaluation, the larger the PSNR value, the higher the image quality; SSIM measures the similarity of images from brightness, contrast and structure, the larger the SSIMI value, the better the modal synthesis and fusion effect; Q AB / F is a gradient-based measure that links important visual information with edge information present in each pixel of the image to measure the edge loss of the fusion image, the larger the value of Q AB / F , the smaller the edge information loss of the fusion image.

[0114] Comparative method: In order to verify the performance of the method of the application, 10 methods are selected for comparison, which are: 1) modal synthesis method based on random forest (based-RF); 2) modal synthesis method based on convolutional neural network (based-CNN); 3) modal synthesis method based on generative adversarial network (based-GAN); 4) modal synthesis method based on cycle-consistent generative adversarial network (based-CycleGAN). And the image fusion methods based on convolutional neural network are: 5) DenseFuse; 6) DMC-Fusion; 7) EMFusion, the image fusion method based on generative adversarial network is: 8) DDcGAN; 9) GANFuse; 10) DSAGAN; the method proposed in the application is recorded as MSE-Fusion.

[0115] Effect: In this experiment, the different methods of modal synthesis and image fusion under the same data set are compared from the aspects of qualitative comparison and quantitative comparison.

[0116] I. Qualitative comparison

[0117] In this experiment, the effects of modal synthesis and overall image fusion methods are evaluated from the aspects of overall vision and details of the image.

[0118] The four modal synthesis methods and the MSE-Fusion proposed in the application are compared for the modal synthesis results, as shown in Figure 9 The based-RF method obtains a relatively blurred synthesis result, only highlights the image boundary, but the internal fine structure of the image cannot be observed; the based-CNN method obtains a synthesis result image whose texture and boundary are not clear enough, which is not conducive to observing the image details; the based-GAN method obtains a synthesis result with local dark areas, resulting in a large number of missing details; the based-CycleGAN method obtains a synthesis result whose texture details and fine structure representation are not clear enough, and some areas still have missing details. The CT synthesis image synthesis result obtained by the MSE-Fusion has moderate brightness, and the image structure, texture details are highlighted. The CT, MR and PET fusion results obtained by the MSE-Fusion have vivid brain grooves, clear image edge, texture details and fine structure representation.

[0119] II. Quantitative comparison

[0120] In the application, five evaluation indexes (MI, SF, PSNR, SSIM and Q AB / F Twenty pairs of brain CT, MR and PET images are tested to objectively evaluate the performance of the proposed method and the comparative method. The above evaluation indexes are used to compare the modal synthesis results and image fusion results under different methods and the MSE-Fusion proposed in the application.

[0121] The test results of the MSE-Fusion and four comparative modal synthesis methods in MI, SF, PSNR and SSIM are shown in Table 1, and the values in the table are average experimental results. As can be seen from Table 1, the deep structure synthesis image generated by the MSE-Fusion is excellent in the evaluation index SSIM, and retains more image structure information; the shallow detail synthesis image is more prominent in the evaluation index MI, and retains very rich image texture details and fine structures. The deep structure and shallow detail synthesis images obtained by the MSE-Fusion are superior to the other four comparative modal synthesis methods in all evaluation indexes, indicating the obvious advantages of the method.

[0122] Table 1 Average values of objective evaluation indexes of modal synthesis results

[0123]

[0124] The test results of the MSE-Fusion and six comparative fusion methods in MI, SF, PSNR, SSIM and Q AB / F are shown in the line graphs of Figure 11 , respectively, Figure 11 where (a), (b), (c), (d) and (e) correspond to MI, SF, PSNR, SSIM and Q AB / F , respectively, and the average evaluation results are shown in Table 2. Among them, the evaluation index MI is excellent, and the information obtained from the source image is better than other comparative image fusion methods

[0125] Table 2 Average values of objective evaluation indexes of image fusion results

[0126]

[0127] III. Ablation experiment

[0128] In order to analyze the effectiveness of the modal synthesis and image fusion parts in the MSE-Fusion, two groups of ablation experiments are performed to analyze the correlation between the proposed modal synthesis and image fusion.

[0129] In Experiment 1, the image enhancement and image fusion method of the MSE-Fusion are controlled unchanged, and the modal synthesis is replaced by: 1) a modal synthesis method based on random forest (based-RF); 2) a modal synthesis method based on convolutional neural network (based-CNN); 3) a modal synthesis method based on generative adversarial network (based-GAN); and 4) a modal synthesis method based on cycle-consistent generative adversarial network (based-CycleGAN), and the final fusion results are compared, and the qualitative comparison results are as followsFigure 12 Compared with the image fusion method based on the synthesis of four different modalities, the CT, MR and PET three-modality fusion image structure and texture details and fine structure representation obtained by the MSE-Fusion are clearer. The MSE-Fusion performs better than the other four image fusion methods based on the synthesis of different modalities in terms of MI, SF, PSNR, SSIM and Q AB / F The performance of the five objective evaluation indexes is better than that of the other four image fusion methods based on the synthesis of different modalities, and the quantitative comparison results are shown in Table 3. It is proved that the modal synthesis part of the image fusion method based on modal synthesis and enhancement proposed in the application, the MSE-Fusion, is effective, and the modal synthesis effect is good.

[0130] Table 3 Average values of objective evaluation indexes in ablation experiment one

[0131]

[0132] In the second experiment, the modal synthesis and image enhancement method of the MSE-Fusion is unchanged, and the image fusion is replaced by 1) DenseFuse, 2) DMC-Fusion, 3) EMFusion, 8) DDcGAN, 9) GANFuse and 10) DSAGAN respectively, and the final fusion results are compared, and the quantitative comparison results are shown in Table 4, and the qualitative results are shown in Table 4. Figure 13 Compared with the six different image fusion methods based on modal synthesis, the CT, MR and PET three-modality fusion image obtained by the MSE-Fusion can retain more image texture and fine structure information, and the MSE-Fusion performs better than the other six different image fusion methods based on modal synthesis and enhancement in terms of MI, SF, PSNR, SSIM and Q AB / F The performance of the five objective evaluation indexes is better than that of the other four image fusion methods based on the synthesis of different modalities, and the quantitative comparison results are shown in Table 3. It is proved that the modal synthesis part of the image fusion method based on modal synthesis and enhancement proposed in the application, the MSE-Fusion, is effective, and the modal synthesis effect is good.

[0133] Table 4 Average values of objective evaluation indexes in ablation experiment two

[0134]

Claims

1. A weakly supervised medical image fusion method for modality synthesis and enhancement, characterized in that, It consists of three parts: modality synthesis, image enhancement, and image fusion, and includes the following steps: Step 1: In the modality synthesis part, the MR image is used to generate a deep structure composite image through a generator trained by a multi-generator modality synthesis network. Composite image with shallow details ; Step 2: In the image enhancement part, the MR image is used to generate a deep structure enhancement map through a generator trained by a dual-generator image enhancement network. With shallow detail enhancement map ; Step 3, in the image fusion section, with , , , Using the original PET image as input, a multimodal image fusion generative adversarial network is generated to obtain a fused CT, MR, and PET three-modal image: The multimodal image fusion generative adversarial network is based on OctopusNet and consists of three parts: encoding, feature fusion, and decoding. The encoding part sets up a separate encoder for each input image. The encoder consists of two parts: three encoding blocks EB1, EB2, and EB3 and one decoding block DB. Each encoding block consists of one convolutional layer and one dense block for deep feature extraction. Each dense block has three convolutional layers, and the output of each layer is concatenated to become the input of the next layer. The decoding block DB consists of five convolutional layers. The input image is processed by the encoding and decoding blocks to extract features layer by layer from shallow to deep, and the corresponding image feature blocks OUT1, OUT2, OUT3, and OUT4 are output. The feature fusion section performs in-layer feature fusion on the image feature blocks obtained by the encoding section to obtain fusion blocks FOUT1, FOUT2, FOUT3, and FOUT4. The feature maps of each layer extracted by the encoder from the original PET image are concatenated with the fusion blocks FOUT1, FOUT2, FOUT3, and FOUT4 to obtain output blocks FFOUT1, FFOUT2, FFOUT3, and FFOUT4. The decoding section inputs the output blocks FFOUT1, FFOUT2, FFOUT3, and FFOUT4 into a max-pooling layer for downsampling, making each output block have 1 channel, resulting in generated images F1, F2, F3, and F4. Images F1, F2, F3, and F4 are then processed by an image fusion generator. Output the final fused image; image fusion discriminator The generated images F1, F2, F3, and F4 were respectively compared with the image fusion generator. The final fused image output is then individually identified; the image fusion generator... It is a mapping function that learns the relationship between the image generated in the P domain and the final fused image in the Q domain. , and image fusion discriminator The adversarial loss is shown in Equation (10): （10） in, For counter-loss, Image fusion generator Internal structure loss function Includes pixel loss and SSIM loss. It is 0.

1.

2. The weakly supervised medical image fusion method for modality synthesis and enhancement according to claim 1, characterized in that, The modality synthesis component comprises two stages: modality synthesis training and modality synthesis prediction. During the modality synthesis training phase, the multi-generator modality synthesis network, based on CycleGAN, divides the generators into deep structure generators that synthesize CT images from MR images. and shallow detail generator And a deep structure generator for synthesizing MR images from CT scans. and shallow detail generator Through 4 generators and modal synthesis discriminator , The network is trained through a repetitive adversarial training process, continuously extracting deep structural and shallow detail features from images and improving the generator's image synthesis performance. In the modality synthesis prediction stage, MR images are used as input, and the trained MR-to-CT image deep structure generator is used to synthesize the CT image. and shallow detail generator Generate deep structure composite map Composite image with shallow details .

3. The weakly supervised medical image fusion method for modality synthesis and enhancement according to claim 2, characterized in that, The modality synthesis training phase employs a weakly supervised approach to train the network. The specific training process is as follows: MR images are used as training data and input into and Generate a composite image of deep CT structures. Composite image with CT shallow detail CT images were also used as training data input. and Generate MR deep structure composite map Composite image with MR shallow detail ;then and Continue to enter and , and Continue typing and The network is trained in a loop, through , and , , and The mutual antagonism between them improves the modal synthesis performance of the generator.

4. The weakly supervised medical image fusion method for modality synthesis and enhancement according to claim 2, characterized in that, The deep structure generator of the MR-synthesized CT image Deep structure generator for synthesizing MR images with CT Each part consists of three sections: encoding, conversion, and decoding. The encoding section consists of two convolutional layers, Conv6 and Conv7. Conv6 is a 7×7 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1. Conv7 is a 3×3 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 2. The conversion section consists of nine residual blocks. The decoding section consists of one deconvolutional layer, DeConv4, and one convolutional layer, Conv8. DeConv4 is a 3×3 fractional-strided Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1 / 2. Conv8 is a 7×7 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1. The shallow detail generator of the MR-synthesized CT image Shallow detail generator for synthesizing MR images with CT The system consists of three parts: encoding, conversion, and decoding. The encoding part consists of two convolutional layers, Conv6 and Conv7. Conv6 is a 7×7 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1, and Conv7 is a 3×3 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 2. The conversion part consists of nine residual blocks. The decoding part consists of one deconvolutional layer, DeConv4, and one convolutional layer, Conv8. DeConv4 is a 3×3 fractional-strided Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1 / 2, and Conv8 is a 7×7 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1.

5. The weakly supervised medical image fusion method for modality synthesis and enhancement according to claim 3, characterized in that, The , and , , and The specific forms of their mutual confrontation are: (1) Learning to enhance deep structural information through adversarial loss and cyclic consistency loss. and Domain, shallow details and domain; (2) The network includes four mapping functions: , , and ; (3) Given training samples , ,in , N is a positive integer; (4) The data distribution is represented as , The adversarial loss is applied to four mapping functions; (5) MR-synthesized CT images , as well as The adversarial losses are shown in formulas (1) and (2): （1） （2） CT composite MR images , as well as The adversarial losses are shown in formulas (3) and (4): （3） （4） (6) Targeting and Mapping and and The mapping introduces a cycle consistency loss, as shown in equations (5) and (6): （5） （6） (7) The overall objective function that highlights deep structural information is shown in formula (7): （7） The overall objective function for highlighting shallow detail information is shown in formula (8): （8） in , To control the relative importance of the objectives; (8) The overall objective function is shown in formula (9): （9）。 6. The weakly supervised medical image fusion method for modality synthesis and enhancement according to claim 1, characterized in that, The image enhancement component comprises two stages: image enhancement training and image enhancement prediction. During the image enhancement training stage, the dual-generator image enhancement network is based on a Generative Adversarial Network (GAN), which internally includes a structure generator for feature enhancement. Detail generator and an enhanced discriminator Through the structure generator Detail generator With enhanced discriminator The mutual competition between the two generators improves the feature enhancement performance of the two generators; The structure generator Detail generator Both consist of encoding and decoding parts, and the structure generator... The internal structure is as follows: The encoding part consists of two convolutional layers, Conv6 and Conv7. Conv6 is a 7×7 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1; Conv7 is a 3×3 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 2. The decoding part consists of one deconvolutional layer, DeConv4, and one convolutional layer, Conv8. DeConv4 is a 3×3 fractional-strided Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1 / 2; Conv8 is a 7×7 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1. The detail generator The internal structure is as follows: the encoding part consists of two convolutional layers, Conv6 and Conv7. Conv6 is a 7×7 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1, and Conv7 is a 3×3 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 2. The decoding part consists of one deconvolutional layer, DeConv4, and one convolutional layer, Conv8. DeConv4 is a 3×3 fractional-strided Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1 / 2, and Conv8 is a 7×7 Convolution-InstanceNorm-ReLU layer with k filters and a stride of 1. The enhanced discriminator Internal structure identification CycleGAN discriminator.

Images