Artificial intelligence method for mutual conversion of PET and CT images of whole body

Abstract

The invention provides an artificial intelligence method for mutual conversion of a whole body PET image and a CT image. The mode conversion between the whole body CT image and the PET image is realized by using an enhanced conditional generative adversarial network (cGAN) model. According to the method, a residual error convolution block and a full convolution Transform block are combined, so that the capability of capturing local features and global context information of a model is improved. According to the method, a customized loss function is designed, and the overall quality of the generated image can be improved in combination with the structural consistency loss. At CT-gt; pET (polyethylene terephthalate) and PET-gt; a large number of experiments are carried out on two modal conversion tasks of CT, and results show that the model can generate high-quality images, and the images are close to original images in the aspects of clinical correlation and diagnostic value. The transformer-combined cGAN framework provided by the invention can reduce the requirement of multi-modal image examination, so that the radiation exposure dose and the overall medical cost of a patient are reduced, and a solid foundation is laid for clinical application of PET-CT multi-modal image conversion.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image modality conversion, and in particular relates to an artificial intelligence method for converting whole-body PET and CT images. Background Art

[0002] In the medical field, CT (computed tomography) and PET (positron emission tomography) are relatively mature and common imaging technologies, which provide complementary information on human anatomical structure and metabolic activity, respectively. However, PET-CT multimodal scanning has problems such as high cost, high radiation dose, and long scanning time. Therefore, how to reduce patients' radiation exposure, reduce medical costs, and improve diagnostic accuracy has become a research hotspot in the current field of medical imaging. In recent years, with the rapid development of artificial intelligence and deep learning technologies, deep learning-based medical image modality conversion methods have provided innovative solutions to this problem. Through artificial intelligence methods based on deep learning models, CT and PET images can be efficiently converted, so that it is possible to obtain multimodal images with only a single-modality scan. This can reduce the radiation exposure and related costs of patients while reducing the need for PET-CT multimodal scanning. In addition, using this method, it is possible to generate more information-rich multimodal images in the lesion area, while seeing anatomical details and metabolic conditions, and improving the reliability of diagnosis. However, existing deep learning generation models still face some technical bottlenecks in CT and PET image conversion tasks, especially in terms of image generation quality and accuracy in high uptake areas (such as tumors). Due to the significant differences in the grayscale value range and distribution between PET and CT images, especially the wide range and uneven distribution of grayscale values ​​in PET images, it is difficult for many models to generate PET images that accurately reflect the image details of high uptake areas. In addition, traditional convolutional neural networks (CNNs) are mainly good at local feature extraction and cannot fully capture global context information, making models based solely on convolutional networks perform poorly in multimodal conversion tasks.

[0003] In order to solve these problems, generative adversarial networks (GANs) have gradually become the mainstream technology in the field of cross-modal image generation. GAN can generate realistic target domain images from the source domain through adversarial training of the generator and the discriminator. Among them, Pix2Pix, CycleGAN and cGAN are classic GAN models widely used in medical image generation. Pix2Pix is ​​suitable for the case of paired data, and uses the L1 norm loss function to enhance detail generation; CycleGAN performs well under the condition of no paired data, relying on the cycle consistency loss function to ensure the consistency of the generated image with the target image; cGAN introduces the target image as label information, and optimizes the entire GAN model under this a priori condition, which can make better use of the training data. However, these classic GAN models still have certain limitations in processing multimodal conversion of whole-body CT and PET images, especially when capturing global context information and generating high uptake areas, the quality of the generated images is not ideal. With the introduction of the Transformer architecture into the task of medical image generation, it has become the key to improving the effect of global feature extraction. Through the self-attention mechanism, the Transformer can effectively capture the global dependencies in the image, making the model more flexible and accurate in generating complex images. This innovative architecture inspired the FCTR (Fully Connected Transformer Residual) block designed in the present invention, which is embedded in the bottleneck module of the generator, so that the model proposed in the present invention not only retains the ability of convolutional neural networks to capture local features, but also improves the understanding of global contextual information through the self-attention mechanism.

[0004] Based on this, the present invention combines the conditional generative adversarial network (cGAN) and the Transformer module to propose a bidirectional generation method for whole-body CT and PET images. This method can not only generate high-quality PET images through CT images, but also generate high-quality CT images from PET images, thereby realizing efficient conversion of multimodal images. The generator module is enhanced by the Transformer, which can not only capture local details, but also better understand the global anatomical structure, making the generated image more realistic and reliable. The discriminator is used to distinguish between the target image and the generated image to ensure the fidelity of the generated image in structure and texture. At the same time, the introduction of the structural consistency loss function ensures the detailed expression of the generated image in key anatomical areas. Through this bidirectional generation method, the present invention can effectively solve the limitations of CT and PET images in their respective applications, reduce the demand for PET-CT multimodal scanning, reduce the radiation exposure of patients, and also improve the diagnostic accuracy and clinical applicability of single-modality images. Summary of the invention

[0005] Purpose of the invention: In view of the problems of high scanning cost, high radiation dose and long imaging time in PET-CT multimodal images, the present invention provides an artificial intelligence method for converting whole-body PET and CT images. Through the bidirectional generation of PET and CT images, efficient conversion between multimodal images is achieved, which can improve the diagnostic quality and accuracy of single-modal images. Specifically, by generating PET images from CT images, the present invention can obtain human metabolic information without using radioactive drugs; by generating CT images from PET images, the present invention can reconstruct human anatomical structures without X-ray irradiation.

[0006] Technical solution: The present invention proposes an artificial intelligence method for converting whole-body PET and CT images, which includes the following 6 steps:

[0007] a) Literature research: We conducted extensive research on literature in related fields, including but not limited to the current status of research on medical image generation, the current status of research on medical image modality conversion, and the current status of research on generative adversarial network models.

[0008] b) Data collection: In collaboration with doctors from Nanjing Drum Tower Hospital, we collected 102 whole-body PET-CT multimodal image data, and obtained more than 21,000 pairs of two-dimensional images. We randomly divided the collected data sets into training, validation, and test sets in a ratio of 8:1:1, and preprocessed the image spatial resolution to 1 mm.

[0009] ×1mm×1mm, and the number of cross-sectional pixels of the image is uniformly preprocessed to 256×256.

[0010] c) Model building: Based on the Transformer residual block and conditional generative adversarial network (cGAN), PET and

[0011] The framework and deep network model for bidirectional conversion of CT images, namely the PET-CT image bidirectional conversion model.

[0012] d) Design loss function: Adversarial loss, pixel loss, and structural consistency loss are added to the loss function to ensure the similarity between the generated image and the target image in terms of overall appearance, pixel value distribution, and key anatomical areas, that is, the detailed expression of the generated image in key anatomical areas.

[0013] e) Training model: The PET-CT image bidirectional conversion model constructed in c) is trained and optimized on the training set using a two-stage training method, and then verified on the validation set.

[0014] f) After training, the network model obtained in e) is tested on the test set, and the PET generation results obtained are compared with other models, and quantitative and visual evaluations are performed to analyze the quality and clinical practicality of the generated PET and CT images.

[0015] Specifically, the CT image in step b) is a conventional whole-body CT scan with a spatial resolution of 1.172×1.172×4.0 mm. 3 Each 2D cross-sectional image is 512×512 in size; the spatial resolution of whole-body PET scans is 4.0×4.0×4.0mm 3 , each 2D cross-sectional image is 144 × 144 in size. Next, the collected CT and PET image data are randomly divided into training, validation, and test sets in a ratio of 8:1:1; the processing method includes using the 99.5% percentile of the pixel value of each image as the threshold cutoff of the intensity value, normalizing the pixel value of the image to [-1,1], randomly flipping horizontally and vertically, and randomly cropping to eliminate the influence of extreme values ​​and ensure that the image data maintains a consistent ratio for training and testing.

[0016] Specifically, the PET-CT image bidirectional conversion model in step c) is composed of a generator based on convolutional residual and Transformer and a discriminator of PatchGAN structure. The generator is composed of an encoder and a decoder. The encoder part is composed of a downsampling feature extraction module and a bottleneck module, wherein the downsampling module is composed of three layers of convolutional residual blocks, each layer extracts the features of the image through two convolution operations, and at the same time, the image size is halved and the number of channels is increased each time the downsampling is performed to better capture the multi-scale features of the image. The bottleneck module uses nine FCTR blocks, each of which contains a full convolution Wide-Focus module, a convolutional attention module and a residual block, through which multi-scale feature information is captured, and local details and global context information are extracted. The decoder part of the generator first uses two layers of deconvolution layers, each of which doubles the size of the input feature map and reduces the number of channels to restore it to a size close to the original source image, and then connects a 1×1 convolution layer, and uses a hyperbolic tangent activation function to output the final generated image. The output of the discriminator is a two-dimensional matrix, each value of which corresponds to the authenticity score of a 70×70 local area in the input image, so as to more accurately evaluate the authenticity of the local anatomical area in the generated image. Through this conversion framework, CT images can generate PET images that are very close to real PET images, and vice versa.

[0017] Specifically, the adversarial loss, pixel loss and structural consistency loss in step d) are used to evaluate the difference between the generated image (the output of the model) and the target image (the expected result of the model). The adversarial loss is a commonly used loss function in the GAN model, and the pixel loss is taken as the L1 norm of the pixel difference between the target image and the generated image. The structural consistency loss is designed as:

[0018]

[0019] Among them, S x and S y The structural similarity graphs (SC graphs) representing the source image and the generated image respectively, the source image is the input of the model and the generated image is the output of the model. The SC graph in this patent is calculated by extracting features from the source image and the generated image using the VGG16 model at the feature map level. i It captures the correlation between the query point i and other points γ in the 2D image patch centered on it. Given a 2D image patch of size N = H × W, the SC mapping of x relative to the query point i is Defined as:

[0020]

[0021] Where Z x C channel feature map extracted for the VGG16 model. This patent uses cosine loss to measure the spatial similarity between the source image and the generated image, and captures and preserves the local and global structural information between CT and PET images through contrastive learning.

[0022] Specifically, the step e) adopts a two-stage training method, the model is trained through a generative adversarial network (GAN) framework, the source image is used to generate the corresponding target image, and the generator and discriminator are optimized. This application uses the LambdaLR strategy to dynamically adjust the learning rate to reduce overfitting, and linearly decays to 0 starting from the second stage. During the training process, the model is verified using a validation set to ensure that the generated images also have good performance on unseen data. By calculating the loss value and evaluation index of the validation set, the model hyperparameters are adjusted to prevent overfitting.

[0023] Specifically, the testing process described in step f) is specifically implemented as follows: the trained network model is comprehensively tested on the test set, and the generated image results are compared with the results generated by other models. During the testing phase, quantitative evaluation and visual comparison methods are used to analyze the quality and clinical practicality of the generated images. Among them, the quantitative evaluation uses three commonly used image quality indicators, normalized root mean square error (NRMSE), peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) to quantify the similarity between the generated image and the target image. The visual comparison method evaluates the clinical practicality of the generated image in terms of anatomical structure, lesion identification, etc. through subjective scoring by clinical medical experts. The method proposed in the present invention has been compared in many aspects with other cutting-edge medical image modality conversion models, reflecting the superiority of the present invention.

[0024] Beneficial effects:

[0025] The present invention redesigns the bidirectional modality conversion task between PET and CT images, and proposes an artificial intelligence method for the mutual conversion between whole-body PET and CT images, which can significantly improve the quality of conversion between the two images. Especially in key anatomical areas, the generated PET and CT images are more in line with clinical needs, ensuring diagnostic accuracy in complex cases. On the one hand, by converting CT images into PET images, the need for PET scans is reduced, thereby reducing the patient's radiation exposure, while also shortening the imaging time and reducing medical costs. On the other hand, generating CT images from PET images can obtain high-quality CT images at a low radiation dose, optimizing the collaborative use of multimodal images. The innovative model design of the present invention ensures the accuracy of image structure and detail expression, provides reliable technical support for the efficient bidirectional conversion of whole-body PET-CT multimodality in medical images, and has high clinical application potential.

[0026] Figure content

[0027] Figure 1 It is the overall framework diagram of the method proposed in the present invention;

[0028] Figure 2 The schematic diagram of the discriminator module in the method framework proposed by the present invention;

[0029] Figure 3 The loss function architecture in the method framework proposed in the present invention;

[0030] Figure 4 Qualitative comparison between PET images generated for the proposed model and those generated by other models;

[0031] Figure 5 A frequency domain K-space image of a PET image generated by the method framework proposed in the present invention;

[0032] Figure 6 Qualitative comparison between CT images generated for the proposed model and those generated by other models;

[0033] Figure 7 A frequency domain K-space image of a CT image generated by the method framework proposed in the present invention;

[0034] Figure 8 The whole-body PET-CT image of the patient and the visualization results of the coronal and sagittal slices of the generated PET image in the framework of the method proposed in the present invention;

[0035] Fig. 9 The intensity value results of the real PET-CT image and the generated PET image in the framework of the method proposed in the present invention;

[0036] Fig.10 The whole body PET-CT image of the patient and the visualization results of the coronal and sagittal slices of the generated CT image in the framework of the method proposed in the present invention;

[0037] Fig.11 The intensity value results of the real PET-CT image and the generated CT image in the framework of the method proposed in the present invention; DETAILED DESCRIPTION

[0038] The objectives, technical solutions and advantages of the present invention are described in detail below in conjunction with the accompanying drawings and specific embodiments of the present invention.

[0039] The present invention redesigns the modality conversion task of whole-body PET-CT images and proposes an artificial intelligence method for converting whole-body PET and CT images, which can be used for PET-CT multimodality image conversion of any part of the body.

[0040] The overall flow chart of the present invention is as follows: Figure 1 As shown, the specific steps include the following steps.

[0041] a) Data collection: In collaboration with doctors from Nanjing Drum Tower Hospital, we collected 102 pairs of PET-CT whole-body image data, which resulted in more than 21,000 pairs of two-dimensional image pairs. We divided the collected data sets into 8:1:

[0042] The data set was randomly divided into training set, validation set and test set with a ratio of 1. The specific implementation of the preprocessing of the data set is as follows: first, the pixel values ​​of all images are normalized to [-1, 1]; second, data augmentation is performed by random horizontal flipping and random vertical flipping operations; finally, the cross-sectional two-dimensional images of PET and CT are resampled to 256×

[0043] 256 resolution.

[0044] b) Model building: Based on the Transformer residual block and the generative adversarial network, a model framework for bidirectional conversion between PET and CT images is constructed. Figure 1-Figure 2 As shown;

[0045] g) Design loss function: Add adversarial loss, pixel loss and structural consistency loss to the loss function to ensure the similarity between the generated image and the target image in terms of overall, pixel value distribution and key anatomical regions. The structural consistency loss focuses on preserving the key anatomical structures in the source image in the generated image, especially in important areas of medical images. Figure 3 As shown, the expression is as follows:

[0046]

[0047] c) Training model: The two-stage training method is used to train and optimize the bidirectional conversion model of PET and CT images constructed in b) on the training set. Starting from the 50th epoch, the learning rate is linearly decayed to 0. If the training loss of the model does not decrease by less than 1% in 20 consecutive rounds of training, or the PSNR and SSIM indicators on the validation set do not increase by more than 1% in 20 consecutive rounds of training, the training of the model is stopped. The present invention randomly initializes the network parameters θ; and uses the Adam optimizer to optimize the network parameters. The hyperparameters in the training process are shown in Table 1.

[0048] Table 1 Model training hyperparameters

[0049]

[0050]

[0051] d) After the training is completed, the network model obtained in d) is tested on the test set, and the obtained medical modality conversion results are compared with the results obtained by other models, and quantified and visually evaluated. For the modality conversion evaluation of CT images and PET images, the present invention uses NRMSE, PSNR, and SSIM indicators for evaluation. In order to more clearly illustrate the feasibility and superiority of the present invention, Table 2 shows its performance in the test set and compares it with other modality conversion methods.

[0052] Table 2 Comparison of evaluation results of modal conversion model test set

[0053]

[0054] This application conducts ablation experiments on the training process of the model in the following ways: (1) using the proposed model; (2) removing the FCT module containing Transformer from the bottleneck layer (denoted as Model 1); (3) deleting the residual block in the bottleneck layer (denoted as Model 2); (4) replacing the proposed residual feature extraction module with a simple downsampling convolutional layer (denoted as Model 3); (5) deleting the structural consistency loss in the loss function (denoted as Model 4). The corresponding experimental results are shown in Table 3.

[0055] Table 3 Ablation experimental results of modal conversion model

[0056]

[0057] In order to illustrate the feasibility and effectiveness of the method proposed in the present invention in the PET-CT multi-modal conversion task, the present application further performs spatial domain visualization and frequency domain visualization on the generated results, such as Figures 4 to 7 As shown. Compared with other models, the present invention can obtain a clearer image with less artifacts. This advantage shows that in describing tissue details, the present invention can more accurately generate human anatomical structures and metabolic activity areas, thereby helping doctors to make better diagnoses. Figure 5 and Figure 7 The corresponding frequency domain K-space image is also attached. It can be seen that the image generated by the model proposed in the present invention covers more frequency domain details, which is better than the results of Pix2pix and AttUnet models.

[0058] In order to illustrate the superiority of the proposed method in the whole body image conversion task, Figures 8 to 11 In the present application, the generation effects of the other two cross sections are also compared. The image generated by the present invention has a high similarity with the target image in the axial view, which also verifies the effectiveness of the method proposed in the present invention. It can be seen that the present invention generates a result that is almost the same as the target image and is better than the result of the pGAN model. The present application takes all voxels along the dotted line position of the real PET-CT image, the result generated by the method proposed in the present invention, and the result generated by the pGAN model, and draws their pixel value histograms, as shown in FIG. Fig. 9 and Fig.11 It can be seen that the histogram corresponding to the result generated by the method proposed in the present invention is very close to the pixel value histogram of the target image at the dotted line position, which once again verifies the effectiveness of the present invention.

Claims

1. An artificial intelligence method for converting whole-body PET and CT images, characterized in that: The steps include: a) Data collection: Whole-body PET-CT image data were collected; the collected data sets were randomly divided into training set, validation set, and test set in a ratio of 8:1:1; the image spatial resolution was uniformly preprocessed to 1mm*1mm*1mm, and the number of cross-sectional pixels of the image was uniformly preprocessed to 256*256; b) Model building: Based on the residual convolution block and the Transformer block, a multimodal conversion model of whole-body CT and whole-body PET images, namely the PET-CT image bidirectional conversion model, is constructed under the conditional generative adversarial network (cGAN) framework; c) Design loss function: In addition to adversarial loss and pixel loss, we continue to introduce structural consistency loss to ensure the detailed representation of key anatomical areas in the generated images; d) Training model: The PET-CT image bidirectional conversion model constructed in b) is trained and optimized on the training set using a two-stage training method, and then verified on the validation set; e) After training, the deep network model obtained in d) is tested on the test set, and the generated results are compared with other models to analyze the clinical practicality of the generated PET and CT images.

2. The artificial intelligence method for converting whole-body PET and CT images according to claim 1, characterized in that: The established PET-CT image bidirectional conversion model can be used for the mutual conversion between whole-body CT images and whole-body PET images.

3. The artificial intelligence method for converting whole-body PET and CT images according to claim 1, characterized in that: The PET-CT image bidirectional conversion model is composed of a generator based on residual convolution blocks and Transformer blocks and a discriminator; wherein the generator is composed of an encoder and a decoder; the encoder part is composed of a downsampling feature extraction module and a bottleneck module, the downsampling module is composed of three layers of convolution residual blocks, each layer extracts image features through two convolution operations, and at the same time, the image size is halved and the number of channels is increased during each downsampling to better capture the multi-scale features of the image; the bottleneck module uses nine FCTR blocks, each FCTR block contains a global focus module, a convolution attention module and a residual block, through which multi-scale feature information can be captured, and local details and global context information can be extracted at the same time; the decoder part first uses two deconvolution layers, each layer doubles the size of the input feature map, reduces the number of channels, and restores it to a size close to the original source image, and then uses a 1×1 convolution layer connected to a hyperbolic tangent activation function to output the final generated image; the output of the discriminator is a two-dimensional matrix, each value corresponds to the authenticity score of a 70×70 local area in the input image, so as to more accurately evaluate the generation effect of each local anatomical area in the generated image.

4. The artificial intelligence method for converting whole-body PET and CT images according to claim 1, characterized in that: The loss function is divided into adversarial loss, pixel loss and structural consistency loss. The quality and accuracy of the generated image are ensured by calculating the overall similarity between the generated image and the target image, the difference at the pixel level, and the structural consistency in key anatomical areas; the structural consistency loss is designed as: Among them, S x and S y Represents the structural similarity SC map of the source image and the generated image respectively; the SC map is calculated by extracting features from the source image and the generated image using the VGG16 model at the feature map level; the SC map S i Captured query point i The correlation between x and all other points γ in the image patch centered on it; given a two-dimensional image patch of size N = H × W, the SC mapping of x relative to the query point i Defined as: Where Z x C channel feature map extracted for the VGG16 model; cosine loss is used to measure the spatial similarity between the source image and the generated image, and contrastive learning is used to capture and preserve the local and global structural information between CT and PET images.

Images