A Brain PET Image Synthesis Method Based on Generative Adversarial Networks

Abstract

For the PET image generation task, the present invention designs a method for synthesizing brain PET images based on a generative adversarial network. Fully exploring the internal connection between MRI and PET, synthesizing high-quality PET at the pixel and semantic levels is the key to this task. The present invention designs a generator containing multiple convolutional series-parallel modules, which can improve the expression ability of the network without significantly increasing the number of parameters and time complexity, realize complex and learnable cross-channel information interaction, and effectively synthesize real and diverse PET from MRI; on the basis of traditional adversarial loss and pixel-level loss, 3D GP loss and SSIM loss are introduced to better retain texture information and contrast information. Based on the characteristics and advantages of deep learning and combined with the characteristics and requirements of medical images, the present invention designs an end-to-end PET image generative adversarial network, which has broad application prospects in multi-modal medical data augmentation, auxiliary diagnosis, etc.

Description

Technical Field

[0001] The present invention designs a method for synthesizing brain PET images based on a generative adversarial network to achieve the mapping from MRI images to PET images, and relates to the fields of deep learning technology, medical image processing technology, and image synthesis technology. Background Art

[0002] Medical images, as the cornerstone of precision medicine, have become an essential part of public health research. With the development of modern medical imaging devices, various forms of medical imaging have emerged. Research has shown that morphological or functional abnormalities caused by diseases in the human body are often manifested in various aspects, and single-modal medical images often cannot comprehensively reflect the complex characteristics of diseases. Multi-modal medical images can provide complementary and effective information from different aspects to help radiologists and clinicians detect or treat diseases more effectively. Many clinical applications, such as tumor detection and brain disease diagnosis, require high-quality multi-modal data to obtain good diagnostic results. Among them, Magnetic Resonance Imaging (MRI) and Positron Emission Computed Tomography (PET) are two commonly used medical images. MRI images have no radiation damage to the human body, clearly show soft tissue structures, and can provide richer imaging information for clarifying the nature of lesions. However, the examination takes a long time and is prone to artifacts. PET images can make an early diagnosis of diseases through the functional changes of tissues in the lesion area; however, due to reasons such as high cost, lack of imaging equipment, radiation exposure, and increased lifetime cancer risk, it is difficult to collect MRI images and PET images for each patient simultaneously. Therefore, how to accurately synthesize PET images using existing MRI images with computer vision technology has been a research hotspot in recent years. Summary of the Invention

[0003] Based on the characteristics and advantages of deep learning, the present invention provides a research method for synthesizing brain PET images based on a generative adversarial network. A multi-convolution series-parallel module is constructed to extract more abstract high-level semantic features and spatial features. At the same time, a hybrid loss combining 3D Gradient profile (3D GP) loss, Structural Similarity Index Measure (SSIM) loss, adversarial loss, KL loss, and L1 loss is proposed to supervise the training process at multiple levels. The generated PET images, as augmented medical data, alleviate problems such as insufficient medical image data and are an important basis for realizing multi-modal medical image analysis.

[0004] The ADNI dataset is a currently publicly available large-scale dataset for Alzheimer's disease, including T1-weighted structural MRI, FDG PET images, and other imaging modalities. A total of 1732 pairs of images from 873 subjects in the ADNI database were used in our experiments. The images of the subjects were divided into three categories: Cognitively Unimpaired (CN), Mild Cognitive Impairment (MCI), and Alzheimer's Disease (AD). Each subject had two imaging modalities, MRI and PET. Our model achieved good test results on the ADNI sub-dataset, and through ablation experiments, the rationality of this method was verified.

[0005] The present invention realizes the above object through the following technical solutions:

[0006] 1. In step one, a dataset is constructed. Based on the publicly available ADNI dataset, subjects with both high-quality MRI and PET images are selected to form a dataset for generating PET images from MRI images.

[0007] 2. In step two, the dataset for generating PET images from the MRI images constructed in step one is preprocessed to obtain registered, sampled, and normalized MRI and PET image pairs.

[0008] 3. In step three, the present invention proposes an adversarial generation network including a multi-convolution series-parallel module. Its network structure includes a generator, a discriminator, and an encoder. Among them, the generator is a U-Net network with a multi-convolution series-parallel module added. The multi-convolution series-parallel module is inserted at the bottom of the U-Net network. Without significantly increasing the network complexity and time complexity, high-level semantic feature extraction is achieved using multiple series and parallel 1×1×1 convolutions. In the series mode, due to the gradual increase in the number of convolutional layers, the depth of the network can be increased, effectively improving the learning ability of the network. In the parallel mode, the outputs of the three series branches are added to obtain more diverse and multi-level feature information from the image. It also allows complex, learnable cross-channel information interaction, improving the generalization ability and expression ability of the model. At the same time, residual connections are used to prevent gradient disappearance.

[0009] 4. In step four, the present invention proposes a hybrid loss function including a weighted sum of adversarial loss, KL loss, L1 loss, 3D GP loss, and SSIM loss to guide the training of the generative adversarial network. The entire training process is the result of continuous gaming between the generator and the discriminator and continuous learning of the encoder. The total loss function of this network is shown in formula (1).

[0010]

[0011] Among them and are the loss functions of the GAN, is the loss function of the image pixel distribution, is the 3D GP loss function, is the KL loss function.

[0012] The main content of the present invention lies in proposing a research method for brain PET image synthesis based on a generative adversarial network. The designed deep network for generating PET images from MRI images can generate PET images closer to the original PET images, which has great significance for multi-modal medical image completion, multi-modal medical image analysis, etc. Brief Description of the Drawings

[0013] Figure 1 is the preprocessing flow chart of MRI and PET images.

[0014] Figure 2 is the algorithm structure diagram of generating PET based on MRI proposed.

[0015] Figure 3 is the structure diagram of the proposed multi-convolution series-parallel module. Detailed Embodiments

[0016] The present invention will be further described below with reference to the accompanying drawings:

[0017] Figure 1 is the preprocessing flow chart of MRI and PET images. First, preprocess the MRI images and PET images respectively, then align them spatially and map them to the MNI template, then remove non-brain tissues to avoid the interference of redundant information, and finally perform normalization and sample to a fixed size.

[0018] Figure 2 is the brain PET image synthesis method based on a generative adversarial network proposed by the present invention. In order to extract high-level spatial features and semantic features, a U-Net with a multi-convolution series-parallel module is used as the generator. To verify the rationality of the multi-convolution module as shown in Figure 3 experiments were carried out on the original model and the model without the multi-convolution module, and the results are shown in Table 1. At the same time, in order to stabilize the training of the generative adversarial network model, optimize the quality of the generated PET images, and better retain the texture information and contrast information of the generated PET images, the weighted sum of multiple losses is used to guide the training of the network. To verify the effectiveness of the proposed hybrid loss, ablation experiments were carried out, and the model was trained using three different hybrid losses in the same experimental environment, and the test results are shown in Table 2:

[0019] Table 1

[0020]

[0021] Table 2

[0022]

[0023] To verify the effectiveness of the method proposed in the present invention, multiple natural image generation models and multiple PET image generation models are selected for comparison with the method proposed in the present invention. The symbol * indicates that the experimental data and data preprocessing process selected by the method are different from those of the present invention. The test results are shown in Table 3 as follows:

[0024] Table 3

[0025]

[0026] It can be seen from Table 3 that the method proposed in the present invention has great advantages over other models in generating PET images from MRI images. Therefore, the effectiveness of the method proposed in the present invention is verified.

Claims

1. A method for synthesizing brain PET images based on a generative adversarial network, characterized in that The steps include the following: Step 1: Based on the publicly available ADNI dataset, select subjects with both high-quality MRI and PET images to form a dataset for generating PET images from MRI images. Step 2: Preprocess the MRI and PET image pairs, that is, register the two modal images to achieve rigid alignment. Also, to avoid the influence of redundant information, remove the non-brain tissues from the MRI and PET images. In addition, we sample the MRI and PET scans to a size of 128×128×128 to reduce the computational amount and perform normalization processing at the same time. Step 3: Train the generative adversarial network model. Send the medical images in the dataset constructed in Step 1 through the preprocessing process in Step 2 into the adversarial generative network containing multiple convolutional series-parallel modules for training, and complete the model training through multiple rounds of parameter adjustment. Its network structure includes a generator, a discriminator, and an encoder. Among them, the generator is a U-Net network with multiple convolutional series-parallel modules added. Insert multiple convolutional series-parallel modules at the bottom of the U-Net network to achieve high-level semantic feature extraction using multiple 1×1×1 convolutional layers in series and parallel without significantly increasing the network complexity and time complexity. Step 4: To stabilize the training of the generative adversarial network model and optimize the quality of the generated PET images, construct a hybrid loss including 3D gradient profile (3D GP) loss, structural similarity (SSIM) loss, adversarial loss, KL loss, and L1 loss to supervise the training process at multiple levels. The total loss function of this network is shown in Equation (1). Among them and are the loss functions of GAN, is the loss function of the image pixel distribution, is the 3D GP loss function, is the KL loss function.

2. The method for synthesizing brain PET images based on a generative adversarial network according to claim 1, wherein The process of making the dataset in Step 1 is as follows: The authoritative publicly available dataset for AD diagnosis is ADNI. Our data subset uses a total of 1732 pairs of MRI and PET images from 873 subjects in the ADNI database. Select the T1-weighted structural MRI preprocessed by Gradwarp, B1 non-uniformity, and N3 bias field correction and the PET images preprocessed by Co-registered and Averaged. The subjects are divided into three categories: cognitively unimpaired, mild cognitive impairment, and Alzheimer's disease. Each subject has images in two modalities, T1-weighted structural MRI and FDG-PET. Consider randomly dividing the scan pairs into a training set and a test set.

3. A method for synthesizing brain PET images based on a generative adversarial network according to claim 1, wherein In step 2, before model training, the data is preprocessed using the Clinica software platform. First, the scans of the two modalities, MRI and PET, are mapped to a Dartel template. Then, these two modalities are spatially aligned to the standardized Montreal Neurological Institute (MNI) coordinate space. When input, the scans are converted to the BIDS format. To avoid the influence of redundant information, the non-brain tissues between the MRI and PET scans are stripped. In addition, we resize the MRI and PET scans to 128×128×128 to reduce the computational cost, and further normalize the pixel value range of the images to [-1, 1] to avoid the problem of gradient explosion.

4. A method for synthesizing brain PET images based on a generative adversarial network according to claim 1, characterized in that In step 4, the loss function includes the weighted sum of adversarial loss, KL loss, L1 loss, 3D GP loss, and SSIM loss, which is used to guide the training of the generative adversarial network, further improve the quality of the generated PET scans, and better retain the texture information and contrast information of the PET images.

Images