An artificial intelligence method for mutual conversion of whole-body pet and ct images

Abstract

The application provides an artificial intelligence method for mutual conversion of whole-body PET and CT images, and uses an enhanced conditional generative adversarial network (cGAN) model to realize the modal conversion between whole-body CT and PET images. The method combines residual convolution blocks and full convolution Transformer blocks to improve the ability of the model to capture local features and global context information. A customized loss function is designed in the application, which combines structural consistency loss to improve the overall quality of the generated images. A large number of experiments are conducted on the CT->PET and PET->CT two modal conversion tasks, and the results show that the model of the application can generate high-quality images, and is close to the original images in terms of clinical relevance and diagnostic value. The cGAN framework combined with the Transformer provided by the application can reduce the need for multi-modal image examination, thereby reducing the radiation exposure dose of patients and the overall medical cost, and lays a solid foundation for the clinical application of PET-CT multi-modal image conversion.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image modal conversion, and particularly relates to an artificial intelligence method for mutual conversion of whole-body PET and CT images. BACKGROUND

[0002] In the medical field, CT (Computed Tomography) and PET (Positron Emission Tomography) are relatively mature and common imaging techniques, which provide complementary information of human anatomical structure and metabolic activity, respectively. However, PET-CT multimodal scanning has problems such as high cost, large radiation dose, long scanning time, etc. Therefore, how to reduce the radiation exposure of patients, reduce medical costs and improve diagnostic accuracy has become a research hotspot in the field of medical images. In recent years, with the rapid development of artificial intelligence and deep learning technology, the medical image modal conversion method based on deep learning provides an innovative solution to this problem. Through the artificial intelligence method based on the deep learning model, CT and PET images can be efficiently converted, so it is possible to obtain multimodal images only by single modality scanning. In this way, while reducing the demand for PET-CT multimodal scanning, the radiation exposure of patients and related costs can be reduced. In addition, using this method, it is possible to generate multimodal images with more information in the lesion area, see anatomical details and metabolic conditions at the same time, and improve the reliability of diagnosis. However, the existing deep learning generation model still faces some technical bottlenecks in the CT and PET image conversion task, especially in the quality of image generation and the accuracy of high uptake areas (such as tumors). Due to the significant difference in the gray value range and distribution between PET and CT images, especially the large span and uneven distribution of the gray value range of PET images, it is difficult for many models to accurately reflect the image details of high uptake areas. In addition, the traditional convolutional neural network (CNN) is mainly good at local feature extraction and cannot fully capture global context information, so the model based solely on convolutional network performs poorly in the multimodal conversion task.

[0003] To address these issues, generative adversarial networks (GANs) have gradually become the mainstream technology in the field of cross-modal image generation. GANs can generate realistic target domain images from the source domain through the adversarial training of generators and discriminators. Among them, Pix2Pix, CycleGAN, and cGAN are classic GAN models widely used in medical image generation. Pix2Pix is suitable for paired data and uses the L1 norm loss function to enhance detail generation; CycleGAN performs well without paired data and relies on the cycle consistency loss function to ensure the consistency of generated images with target images; cGAN introduces target images as label information to optimize the entire GAN model under this posterior condition, which can make full use of training data. However, these classic GAN models still have certain limitations in processing multi-modal conversion of whole-body CT and PET images, especially in capturing global context information and generating high uptake regions, the generated image quality is not ideal. With the introduction of the Transformer architecture into the medical image generation task, it has become a key to improving global feature extraction. The Transformer can effectively capture global dependencies in images through self-attention mechanisms, making the model more flexible and accurate when generating complex images. This innovative architecture inspired the design of the FCTR (Fully Connected Transformer Residual) block, which is embedded in the bottleneck module of the generator, allowing the model proposed in the present invention to retain the ability of convolutional neural networks to capture local features while improving the understanding of global context information through self-attention mechanisms.

[0004] Based on this, the present invention combines conditional generative adversarial networks (cGAN) and Transformer modules and proposes a bidirectional generation method for whole-body CT and PET images. This method can not only generate high-quality PET images from CT images, but also generate high-quality CT images from PET images, thereby achieving efficient conversion of multi-modal images. The generator module is enhanced by the Transformer, which can not only capture local details but also better understand global anatomical structures, making the generated images more realistic and reliable. The discriminator is used to distinguish between target images and generated images to ensure the realism of the generated images in structure and texture. At the same time, the introduction of the structural consistency loss function ensures the detail performance of the generated images in key anatomical regions. Through this bidirectional generation method, the present invention can effectively address the limitations of CT and PET images in their respective applications, reducing the need for PET-CT multi-modal scanning, reducing patient radiation exposure, and improving the diagnostic accuracy and clinical applicability of single-modal images. SUMMARY

[0005] Invention purpose: In view of the problems of high scanning cost, large radiation dose and long imaging time existing in PET-CT multi-modal images, the application provides an artificial intelligence method for mutual conversion of whole body PET and CT images, which realizes efficient conversion between multi-modal images through bidirectional generation of PET and CT images, and can improve the diagnosis quality and accuracy of single modal images. Specifically, the application can obtain human metabolic information without using radioactive drugs by generating PET images from CT images; and can reconstruct human anatomical structure without X-ray irradiation by generating CT images from PET images.

[0006] Technical scheme: The application provides an artificial intelligence method for mutual conversion of whole body PET and CT images, which includes the following 6 steps:

[0007] a) Literature research: Extensive research on literature materials in related fields, including but not limited to, research status of medical image generation; research status of medical image modal conversion; research status of generative adversarial network model.

[0008] b) Data collection: Cooperate with doctors in Nanjing Gulou Hospital, collect 102 cases of whole body PET-CT multi-modal image data, and accordingly obtain more than 21000 pairs of two-dimensional image pairs; divide the collected data set into training set, validation set and test set according to the ratio of 8:1:1, and uniformly preprocess the image space resolution to 1mm

[0009] ×1mm×1mm, and uniformly preprocess the cross-sectional pixel number of the image to 256×256.

[0010] c) Model establishment: based on the Transformer residual block and the conditional generative adversarial network (cGAN), a framework and a deep network model for bidirectional conversion of PET and

[0011] CT images, i.e. PET-CT image bidirectional conversion model.

[0012] d) Design loss function: add adversarial loss, pixel loss and structural consistency loss to the loss function, which are respectively used to ensure the similarity of the generated image and the target image in the whole, the pixel value distribution and the key anatomical region, i.e. the detail performance of the generated image in the key anatomical region.

[0013] e) Model training: adopt two-stage training method to train and optimize the PET-CT image bidirectional conversion model constructed in c) on the training set, and verify it on the validation set.

[0014] f) After the training is completed, the network model obtained in e) is tested on the test set, the generated PET results are compared with other models, and quantitative and visual evaluation is performed to analyze the quality and clinical utility of the generated PET and CT images.

[0015] Specifically, the CT image in step b) is a conventional whole-body CT plain scan, and the spatial resolution is 1.172x1.172x4.0mm 3 , and the size of each two-dimensional cross-sectional image is 512x512; the spatial resolution of the whole-body PET scan is 4.0x4.0x4.0mm 3 , and the size of each two-dimensional cross-sectional image is 144x144. Next, the collected CT and PET image data is randomly divided into a training set, a validation set, and a test set in a ratio of 8:1:1; the processing method includes using the 99.5% percentile of the pixel value of each image as the threshold cutoff of the intensity value, normalizing the pixel value of the image to [-1, 1], randomly flipping horizontally and vertically, and randomly cropping to eliminate the influence of extreme values and ensure that the image data is consistent in proportion for training and testing.

[0016] Specifically, the PET-CT image bidirectional conversion model in step c) is composed of a generator based on convolution residual and Transformer and a discriminator of PatchGAN structure. The generator is composed of an encoder and a decoder. The encoder part is composed of a down-sampling feature extraction module and a bottleneck module. The down-sampling module is composed of three layers of convolution residual blocks, and each layer extracts the features of the image through two convolution operations, while the image size is halved and the number of channels is increased each time the image is down-sampled, in order to better capture the multi-scale features of the image. The bottleneck module uses nine FCTR blocks, each of which contains a full convolution Wide-Focus module, a convolution attention module, and a residual block. Through these modules, multi-scale feature information is captured, and local details and global context information are extracted. The decoder part of the generator first uses two deconvolution layers, each of which expands the size of the input feature map by a factor of two while reducing the number of channels, restoring the size close to the original source image, then connects a 1x1 convolution layer and uses the hyperbolic tangent activation function, and outputs the final generated image. The output of the discriminator is a two-dimensional matrix, and each value corresponds to the authenticity score of a 70x70 local region in the input image, in order to more accurately evaluate the authenticity of the local anatomical region in the generated image. Through this conversion framework, CT images can generate PET images very close to real PET images, and vice versa.

[0017] Specifically, the adversarial loss, pixel loss and structural consistency loss in step d) are used to evaluate the difference between the generated image (output of the model) and the target image (expected result of the model). The adversarial loss is a commonly used loss function in GAN models, and the pixel loss is the L1 norm of the pixel difference between the target image and the generated image. The structural consistency loss is designed as:

[0018]

[0019] where S x and S y represent the structural similarity map (SC map) of the source image and the generated image, respectively. The source image is the input of the model, and the generated image is the output of the model. The SC map in this patent is calculated by extracting features from the source image and the generated image at the feature map level using the VGG16 model. The SC map S i captures the correlation between the query point i and other points γ in the 2D image block centered on it. Given a two-dimensional image block of size N = H × W, the SC map of x relative to the query point i is defined as:

[0020]

[0021] where Z x is the C-channel feature map extracted by the VGG16 model. This patent uses cosine loss to measure the spatial similarity between the source image and the generated image, and captures and preserves the local and global structural information between the CT and PET images through contrastive learning.

[0022] Specifically, step e) uses a two-stage training method. The model is trained through a generative adversarial network (GAN) framework, which generates corresponding target images from source images and optimizes the generator and discriminator. This application uses the LambdaLR strategy to dynamically adjust the learning rate to alleviate overfitting, and linearly decays to 0 from the second stage. During training, the model is validated using a validation set to ensure that the generated images also have good performance on unseen data. By calculating the loss value and evaluation metrics of the validation set, the model hyperparameters are adjusted to prevent overfitting.

[0023] ​Specifically, the test process in step f) is implemented as follows: the network model trained is comprehensively tested on the test set, and the generated image results are compared with the results generated by other models. In the test phase, quantitative evaluation and visual comparison methods are used to analyze the quality and clinical utility of the generated images. Among them, the quantitative evaluation uses three commonly used image quality indicators, normalized mean square error (NRMSE), peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) to quantify the similarity between the generated image and the target image. The visual comparison method evaluates the clinical utility of the generated image in terms of anatomical structure, lesion identification, etc. through the subjective score of clinical medical experts. The method proposed in the present application is compared with other advanced medical image modal conversion models in multiple aspects, which embodies the superiority of the present application.

[0024] Beneficial effects:

[0025] The present application redesigns the bidirectional modal conversion task of PET and CT images and proposes an artificial intelligence method for mutual conversion of whole-body PET and CT images, which can significantly improve the conversion quality between the two images. Especially in the key anatomical region, the generated PET and CT images are more in line with clinical needs, ensuring the diagnostic accuracy in complex cases. On the one hand, by converting CT images into PET images, the need for PET scanning is reduced, thereby reducing the radiation exposure of patients, shortening the imaging time and reducing the medical cost. On the other hand, generating CT images from PET images can obtain high-quality CT images with lower radiation dose, optimizing the collaborative use of multi-modal images. The innovative model design of the present application ensures the accuracy of image structure and detail expression, providing reliable technical support for whole-body PET-CT multi-modal bidirectional efficient conversion in medical images, and has high clinical application potential.

[0026] Content of the drawings

[0027] Figure 1 The overall framework diagram of the method proposed in the present application;

[0028] Figure 2 The architecture diagram of the discriminator module in the method framework proposed in the present application;

[0029] Figure 3 The loss function architecture in the method framework proposed in the present application;

[0030] Figure 4 Qualitative comparison between the PET images generated by the proposed model and the PET images generated by other models;

[0031] Figure 5 The frequency domain K-space image of the PET image generated in the method framework proposed in the present application;

[0032] Figure 6 Qualitative comparison between CT images generated for the proposed model and CT images generated for other models;

[0033] Figure 7 Frequency domain K-space images of CT images generated for the proposed method framework of the application;

[0034] Figure 8 Patient whole-body PET-CT images in the proposed method framework of the application and generated PET images coronal and sagittal slice visualization results;

[0035] Figure 9 Intensity value results of real PET-CT images and generated PET images in the proposed method framework of the application;

[0036] Figure 10 Patient whole-body PET-CT images in the proposed method framework of the application and generated CT images coronal and sagittal slice visualization results;

[0037] Figure 11 Intensity value results of real PET-CT images and generated CT images in the proposed method framework of the application; DETAILED DESCRIPTION

[0038] The purpose, technical solution and advantages of the application will be described in detail below in combination with the drawings and specific embodiments of the application.

[0039] The application redesigns the modality conversion task of whole-body PET-CT images and proposes an artificial intelligence method for mutual conversion of whole-body PET and CT images, which can be used for PET-CT multi-modal image conversion of any part of the whole body.

[0040] The overall flowchart of the application is shown in Figure 1 , and specifically includes the following steps.

[0041] a) Collect data: cooperate with doctors in Nanjing Gulou Hospital to collect paired whole-body image data including 102 pairs of PET-CT, and accordingly obtain more than 21000 pairs of two-dimensional image pairs; divide the collected data set into training set, validation set and test set according to the ratio of 8:1:1;

[0042] 1. The specific implementation of the preprocessing of the data set is as follows: first, normalize the pixel values of all images to [-1, 1]; second, perform data augmentation by random horizontal flipping and random vertical flipping; and finally, resample the cross-sectional two-dimensional images of PET and CT to a resolution of 256x

[0043] 256.

[0044] b) Building a model: based on the Transformer residual block and the generative adversarial network, a model framework for bidirectional conversion of PET and CT images is constructed, as shown in Figures 1-2

[0045] g) Designing a loss function: add adversarial loss, pixel loss and structural consistency loss to the loss function, which are used to ensure the similarity of the generated image and the target image in the overall, pixel value distribution and key anatomical region. Among them, the structural consistency loss focuses on preserving the key anatomical structure in the generated image, especially in the important area of medical images. As shown in Figure 3

[0046]

[0047] c) Training the model: the bidirectional conversion model of PET and CT images constructed in b) is trained and optimized on the training set by using a two-stage training method. From the 50th epoch, the learning rate is linearly decayed to 0. If the training loss of the model does not decrease by less than 1% in 20 consecutive training rounds, or the PSNR and SSIM indicators on the validation set increase by no more than 1% in 20 consecutive training rounds, the training of the model is stopped. The network parameters θ of the present application are randomly initialized; the Adam optimizer is used to optimize the network parameters. The hyperparameters in the training process are shown in Table 1.

[0048] Table 1 Model training hyperparameters

[0049]

[0050]

[0051] d) After the training is completed, the network model obtained in d) is tested on the test set, and the obtained medical modal conversion results are compared with the results obtained by other models, and quantitative and visual evaluation is performed. For the modal conversion evaluation of CT images and PET images, the present application uses NRMSE, PSNR and SSIM indicators for evaluation. In order to more clearly illustrate the feasibility and superiority of the present application, Table 2 shows its performance in the test set, and is compared with other modal conversion methods.

[0052] Table 2 Comparison of modal conversion model test set evaluation results

[0053]

[0054] ​​The training process of the model is ablated by the following ways: (1) using the proposed model; (2) removing the FCT module containing the Transformer from the bottleneck layer (denoted as model 1); (3) deleting the remaining blocks in the bottleneck layer (denoted as model 2); (4) replacing the proposed residual feature extraction module with a simple down-sampling convolutional layer (denoted as model 3); (5) deleting the structure consistency loss in the loss function (denoted as model 4), and the corresponding experimental results are shown in Table 3.

[0055] Table 3 Ablation experiment results of the modal conversion model

[0056]

[0057] To illustrate the feasibility and effectiveness of the proposed method in the PET-CT multi-modal conversion task, the application further performs spatial domain visualization and frequency domain visualization on the generated results, as shown in Figures 4-7 Compared with other models, the application can obtain clearer and smaller artifact generated images. This advantage shows that in describing the details of the organization, the application can more accurately generate human anatomical structures and metabolic activity regions, thereby helping doctors to better diagnose. Figure 5 and Figure 7 The corresponding frequency domain K-space images are also attached. It can be seen that the images generated by the proposed model cover more frequency domain details, which are superior to the results of Pix2pix and AttUnet models.

[0058] To illustrate the superiority of the proposed method in the whole body image conversion task, in Figures 8-11 the application also compares the generation effects of the other two cross sections. The image generated by the application has a high similarity with the target image in the axial view, which also verifies the effectiveness of the proposed method. It can be seen that the application generates almost the same result as the target image, and is superior to the result of the pGAN model. The application takes all the voxels along the dashed line position of the real PET-CT image, the result generated by the method proposed in the application and the result generated by the pGAN model, respectively, and draws their pixel value histograms, as shown in Figure 9 and Figure 11 It can be seen that the histogram corresponding to the result generated by the proposed method is very close to the pixel value histogram of the target image at the dashed line position, which again verifies the effectiveness of the application.

Claims

1. An artificial intelligence method for whole-body PET-CT image interconversion, characterized by, Comprising the following steps: a) Collecting data: collecting whole-body PET-CT image data; randomly dividing the collected data set into a training set, a validation set, and a test set in a ratio of 8:1:1; uniformly preprocessing the image spatial resolution to 1mm*1mm*1mm, and the number of image cross-sectional pixels to 256*256; b) Building a model: based on residual convolution blocks and Transformer blocks, a multi-modal conversion model of whole-body CT and whole-body PET images, i.e., a PET-CT image bidirectional conversion model, is constructed under the framework of conditional generative adversarial network cGAN; c) Designing a loss function: in addition to adversarial loss and pixel loss, structural consistency loss is introduced to ensure the details of the generated image in key anatomical regions; d) Training the model: the PET-CT image bidirectional conversion model constructed in b) is trained and optimized on the training set and validated on the validation set using a two-stage training method; e) After the training is completed, the PET-CT image bidirectional conversion model obtained in d) is tested on the test set, the generated results are compared with other models, and the clinical practicability of the generated PET and CT images is analyzed; The PET-CT image bidirectional conversion model is composed of a generator based on residual convolution blocks and Transformer blocks and a discriminator; wherein the generator is composed of an encoder and a decoder; the encoder part is composed of a down-sampling feature extraction module and a bottleneck module; the down-sampling module is composed of three convolution residual blocks, each of which extracts the features of the image through two convolution operations, while the image size is halved and the number of channels is increased each time the image is down-sampled, so as to better capture the multi-scale features of the image; the bottleneck module uses nine FCTR blocks, each of which contains a global focusing module, a convolution attention module, and a residual block, which can capture multi-scale feature information while extracting local details and global context information; the decoder part first uses two deconvolution layers, each of which expands the size of the input feature map by one and reduces the number of channels, restoring it to a size close to the original source image, then uses a 1*1 convolution layer followed by a hyperbolic tangent activation function to output the final generated image; the output of the discriminator is a two-dimensional matrix, each value corresponding to the true or false score of a 70*70 local region in the input image, so as to more accurately evaluate the generation effect of each local anatomical region in the generated image; The loss function is divided into adversarial loss, pixel loss, and structural consistency loss, which ensures the quality and accuracy of the generated image by calculating the overall similarity of the generated image and the target image, the difference at the pixel level, and the structural consistency in the key anatomical regions; the structural consistency loss is designed as: ； where, and respectively represent the structural similarity SC map of the source image and the generated image; the SC map is computed using the VGG16 model at the feature map level from the source image and the generated image; the SC map captures the correlation between the query point i and all other points γ in the image block centered at it; given a two-dimensional image block of size N = H x W, the SC map of x with respect to the query point i is defined as: where, ； wherein C-channel feature maps extracted for the VGG16 model; the cosine loss is used to measure the spatial similarity between the source image and the generated image, and the local and global structural information between the CT and PET images is captured and preserved through contrastive learning.

2. The artificial intelligence method for mutual conversion of whole-body PET and CT images according to claim 1, characterized in that: The established PET-CT image bidirectional conversion model can be used for mutual conversion of whole-body CT images and whole-body PET images.

Images