A CycleGAN-based unpaired PET image quality enhancement method
By combining the CycleGAN model with an improved loss function, and using unsupervised training with unpaired images acquired by novel and conventional PET scanners, the problem of improving the quality of PET images in ordinary hospitals is solved, resulting in significantly improved image quality and reduced geometric distortion.
Patent Information
- Application Number
- CN202310724652.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-19
- Publication Date
- 2025-12-12
- Estimated Expiration
- 2043-06-19
AI Technical Summary
Existing technologies struggle to upgrade low-quality PET images from general hospitals to high-quality PET images comparable to those from newer scanners using software methods, primarily due to the lack of paired low- and high-quality image data from the same patient.
The CycleGAN model is used for unpaired PET image quality enhancement. By establishing a CycleGAN-based generative and discriminative network and combining it with an improved loss function, high-quality and low-quality images are acquired using novel and conventional PET scanners, respectively, and unsupervised training is performed to generate high-quality PET images.
This technology enables the improvement of PET image quality without requiring images to be paired with the same patient, solving the problem of image quality improvement. At the same time, the improved CycleGAN model reduces the geometric distortion of the generated images, resulting in images with a higher contrast-to-noise ratio.
Smart Images
Figure CN116664542B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of PET image processing, and particularly relates to a non-paired PET image quality enhancement method based on CycleGAN. BACKGROUND
[0002] Positron Emission Tomography (PET) is a non-invasive functional imaging method that can help identify and characterize malignant lesions that cannot be seen or are ambiguous on anatomical imaging, reveal information related to patient diagnosis, staging and treatment outcome, and play an important role in medical diagnosis, especially in the early diagnosis of cancer.
[0003] With the improvement of hardware technology, the PET image quality obtained by new scanners (such as the uEXPLORER scanner of United Imaging) is significantly higher than that of conventional scanners, but it has not yet been popularized; and due to the limitations of scanning time and radioactive tracer dose, the PET image quality obtained by most hospitals is poor, which has a certain impact on medical quantitative analysis and diagnosis. Deep learning methods have achieved certain success in improving PET image quality, but they mostly need paired low-high quality data for supervised training, however, in practice, hospitals usually only scan one patient at a time, and it is usually difficult to obtain paired low-high quality PET images of the same patient, and it is even more difficult to obtain paired low-high quality PET image sets collected from new scanners and conventional scanners of the same patient. Therefore, it is an urgent problem in the field to realize PET image quality enhancement through software technology, so that ordinary hospitals can obtain PET images with data quality comparable to new scanners.
[0004] Considering the difficulty of collecting paired low-high quality PET images, non-paired learning methods for processing PET images have been developed, mainly including unsupervised methods and CycleGAN-based methods, wherein:
[0005] Deep Image Prior (DIP) proposed in the document [Ulyanov D, Vedaldi A, Lempitsky V. Deep Image Prior [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 9446-9454] is an important technology for unsupervised image denoising, which inputs random noise into the network, takes the noise image itself as the label, and stops iteration before the network discovers the noise, that is, the denoised image can be output; this method denoises a single image, does not require additional training data set, and does not require any high-quality PET image as a training label, and its main drawback lies in the need for manual setting of the iteration stopping condition, and the specificity between different patients, which has limitations in processing a large number of PET images.
[0006] Cycle-consistent Generative Adversarial Networks (CycleGAN) proposed in the document [Lei Y, Dong X, Wang T, et al. Whole-Body PET Estimation from Low Count Statistics Using Cycle-Consistent Generative Adversarial Networks [J]. Physics in Medicine & Biology, 2019, 64(21): 215017] is an advanced image conversion method, which can capture the features of a data set and learn how to convert these features to other data sets, rather than pixel-by-pixel conversion, so it does not require paired data as labels, which makes it a general method that can be applied to various fields. At present, the use of CycleGAN method for PET image quality enhancement is still limited to training using paired low-high quality images, and it is still worth further exploring to apply CycleGAN to unpaired PET images. SUMMARY
[0007] In view of the above, the present application provides a non-paired PET image quality enhancement method based on CycleGAN, which can convert low-quality PET images obtained by ordinary hospital scanning into high-quality PET images with data quality comparable to new scanners without collecting low-high quality image pairs of the same patient.
[0008] A CycleGAN-based method for unpaired PET image quality enhancement includes the following steps:
[0009] (1) Use a new type of PET scanner to scan different patients to acquire a large number of high-quality PET images, and fix these PET images as a high-quality template dataset;
[0010] (2) Using a conventional PET scanner to scan different patients to acquire a large number of low-quality PET images;
[0011] (3) The low-quality PET images and high-quality PET images obtained above are combined into an unpaired training set, and the low-quality PET images in the training set are preprocessed to obtain the corresponding prior images.
[0012] (4) Establish the CycleGAN model and improve its loss function design;
[0013] (5) Input the unpaired high-quality PET images and low-quality PET images in the training set, as well as the prior images corresponding to the low-quality PET images, into the CycleGAN model for training.
[0014] (6) For the trained CycleGAN model, take the generative network G used to convert low-quality PET images into high-quality PET images. H The low-quality PET images that require quality improvement are then input into the generator network G. H The corresponding high-quality PET image can be directly output from the source.
[0015] Furthermore, in step (3), the BM3D (Block-Matching 3D) denoising algorithm is used to preprocess the low-quality PET image to obtain the corresponding prior image.
[0016] Furthermore, the CycleGAN model includes two generative networks G. H and F L And two discriminant networks D H and D L The generator network G H The generator network F is used to convert low-quality PET images into high-quality PET images. L The discriminant network D is used to convert high-quality PET images into low-quality PET images. H The discriminant network D is used to determine whether the high-quality PET images generated by the conversion conform to the domain distribution of high-quality PET images in the training set. L This is used to determine whether the low-quality PET images generated by the conversion conform to the low-quality PET image domain distribution in the training set.
[0017] Further, the generation network G H and F L Adopt 2D U-net network structure, the discriminant network D H and D L Adopt PatchGAN structure.
[0018] Further, the step (4) is improved design to CycleGAN model's loss function, its loss function By the expression of the following four parts: Adversarial loss Cycle consistency loss Correlation coefficient loss And image prior loss
[0019]
[0020] Wherein: λ cycle , λ cor-coe And λ prior For the corresponding weight parameters.
[0021] Further, the expression of the adversarial loss As follows:
[0022]
[0023]
[0024]
[0025] Wherein: H and L respectively indicate the input high-quality PET image and low-quality PET image, D H () indicates the transfer function of discriminant network D H , D L () indicates the transfer function of discriminant network D L , G H () indicates the transfer function of generation network G H , F L () indicates the transfer function of generation network F L , Indicates that H is subject to the expectation function of high-quality PET image domain P data(H) Distribution in training set, Indicates that L is subject to the expectation function of low-quality PET image domain P data(L) Distribution in training set.
[0026] Further, the expression of the cycle consistency loss As follows:
[0027]
[0028] where H and L represent the input high-quality PET image and low-quality PET image respectively, G H denotes the transfer function of the generator network G H , F L denotes the transfer function of the generator network F L , denotes the expectation function of H subject to the distribution of the high-quality PET image domain P data(H) in the training set, denotes the expectation function of L subject to the distribution of the low-quality PET image domain P data(L) in the training set, and ‖‖1 denotes the L1 norm.
[0029] Further, the expression of the correlation coefficient loss is as follows:
[0030]
[0031] where H and L represent the input high-quality PET image and low-quality PET image respectively, G H denotes the transfer function of the generator network G H , F L denotes the transfer function of the generator network F L , Cov() denotes the covariance operator, and σ() denotes the variance operator.
[0032] Further, the expression of the image prior loss is as follows:
[0033]
[0034] where L represents the input low-quality PET image, G H denotes the transfer function of the generator network G H , I prior_img is the prior image corresponding to L, and ‖‖1 denotes the L1 norm.
[0035] The CycleGAN-based non-paired PET image quality enhancement method of the present application can convert the low-quality PET images obtained by ordinary hospital scanning into high-quality PET images with data quality comparable to that of new scanners without collecting low-high quality image pairs of the same patient, thereby solving the problem that the low-high quality PET image pairs of the same patient are difficult to obtain, so that the supervised deep learning method cannot be used to improve the quality. At the same time, the present application adds a correlation coefficient loss and an image prior loss on the basis of the original CycleGAN model, directly constraining the structural consistency between the generated image and the input image, and solving the problem that the generated image of CycleGAN often has geometric distortion. BRIEF DESCRIPTION OF DRAWINGS
[0036] Figure 1 Fig. 1 is a flowchart of the PET image quality enhancement method of the present application.
[0037] Figure 2 Fig. 2 is a schematic diagram of the CycleGAN model structure used in the present application.
[0038] Figure 3 (a) is the original low-quality PET image.
[0039] Figure 3 (b) is the high-quality PET image obtained by using the method of the present application.
[0040] Figure 3 (c) is the PET image obtained by using the BM3D method.
[0041] Figure 3 (d) is the PET image obtained by using the deep image prior (DIP) method. DETAILED DESCRIPTION
[0042] In order to more specifically describe the present application, the technical solutions of the present application will be described in detail below in combination with the drawings and specific embodiments.
[0043] As shown in Figure 1 , the CycleGAN-based non-paired PET image quality enhancement method of the present application comprises the following steps:
[0044] (1) Scanning different patients using a new PET scanner to collect a large number of high-quality PET images, which are fixed as a high-quality template data set.
[0045] (2) Scanning different patients using a conventional PET scanner to collect a large number of low-quality PET images.
[0046] (3) All the PET images obtained above are grouped into a training set, and the low-quality PET images in the training set are preprocessed to obtain the corresponding prior images.
[0047] (4) An improved CycleGAN model is obtained by establishing a CycleGAN model and improving the design of its loss function, as shown in FIG. 4, and the specific process is as follows: Figure 2
[0048] 4.1 Construction of the generation network G H and F L , using a 2D U-net network structure;
[0049] 4.2 Construction of the discriminator network D H and D L , using a PatchGAN structure;
[0050] 4.3 Definition of the loss function, after improving the design of the loss function of the CycleGAN model, the loss function of the improved CycleGAN model is composed of four parts, i.e., an adversarial loss a cycle consistency loss a correlation coefficient loss and an image prior loss , and the specific expressions are as follows:
[0051]
[0052] wherein λ cycle , λ cor-coe and λ prior are the corresponding weight parameters.
[0053] The expression of the adversarial loss is as follows:
[0054]
[0055]
[0056]
[0057] wherein H and L represent the input high-quality PET image and the low-quality PET image respectively, D H () represents the transfer function of the discriminator D H , D L () represents the transfer function of the discriminator D L , C H () represents the transfer function of the generation network G H , and F L () represents the transfer function of the generation network F L E (H | P) represents the expectation function of H subject to the distribution of high-quality PET image domain P in the training set data(H) E (L | P) represents the expectation function of L subject to the distribution of low-quality PET image domain P in the training set E (H | P) represents the expectation function of H subject to the distribution of high-quality PET image domain P in the training set data(L) E (L | P) represents the expectation function of L subject to the distribution of low-quality PET image domain P in the training set
[0058] Cycle consistency loss The expression is as follows:
[0059]
[0060] wherein H and L represent the input high-quality PET image and low-quality PET image respectively, G H () represents the transfer function of the generation network G H , F L () represents the transfer function of the generation network F L , Cov() represents the covariance operator, and σ() represents the variance operator. E (H | P) represents the expectation function of H subject to the distribution of high-quality PET image domain P in the training set data(H) E (L | P) represents the expectation function of L subject to the distribution of low-quality PET image domain P in the training set E (H | P) represents the expectation function of H subject to the distribution of high-quality PET image domain P in the training set data(L) E (L | P) represents the expectation function of L subject to the distribution of low-quality PET image domain P in the training set, and ‖‖1 represents the L1 norm.
[0061] Correlation coefficient loss The expression is as follows:
[0062]
[0063] wherein H and L represent the input high-quality PET image and low-quality PET image respectively, G H () represents the transfer function of the generation network G H , F L () represents the transfer function of the generation network F L , Cov() represents the covariance operator, and σ() represents the variance operator.
[0064] Image prior loss The expression is as follows:
[0065]
[0066] wherein L represents the input low-quality PET image, G H () represents the transfer function of the generation network G H , I prior_img is the prior image corresponding to L, and ‖‖1 represents the L1 norm.
[0067] (5) Input the unpaired high-quality PET image and low-quality PET image in the training set and the prior image corresponding to the low-quality PET image into the CycleGAN model for training.
[0068] (6) For the trained CycleGAN model, take the generation network G in the model for converting low-quality PET images into high-quality PET images H , and then input the low-quality PET image that needs to be quality improved into the generation network G H , and the corresponding high-quality PET image can be directly output.
[0069] Next, we use the NVIDIA 1080Ti graphics card on the Python 3.7, Pytorch 1.12 platform to implement the present application, and use the PET image scanned by the new scanner Total-body PET-CT uEXPLORER system (United Imaging Healthcare, Shanghai, China) as the high-quality data set, the scanning time is 360s, and the image size is 256x256; and use the PET image scanned by the conventional scanner Biograph 64 Vision 600 PET / CT system (Siemens Medical Solutions, Erlangen, Germany) as the low-quality data set, the scanning time is 60s, and the image size is 256x256.
[0070] As shown in Figure 3 (a) to Figure 3 (d), compared with the BM3D method and the DIP method, the method of the present application can realize the quality enhancement of the PET image, and can better retain the tumor uptake, and the generated image has a higher contrast noise ratio.
[0071] The above description of the embodiments is for the purpose of facilitating the understanding and application of the present application by those of ordinary skill in the art, and those skilled in the art can easily make various modifications to the above embodiments, and apply the general principles described herein to other embodiments without inventive labor. Therefore, the present application is not limited to the above embodiments, and the improvements and modifications of the present application made by those skilled in the art according to the disclosure of the present application should be within the scope of protection of the present application.
Claims
1. A CycleGAN-based unpaired PET image quality enhancement method, comprising the following steps: (1) Scanning different patients using a new PET scanner to obtain a large number of high-quality PET images, and fixing the PET images as a high-quality template data set; (2) Scanning different patients using a conventional PET scanner to obtain a large number of low-quality PET images; (3) Forming an unpaired training set by combining the obtained low-quality PET images and high-quality PET images, and pre-processing the low-quality PET images in the training set to obtain corresponding prior images; (4) Establish CycleGAN model and improve the design of its loss function, the improved loss function consists of four parts: adversarial loss cycle consistency loss correlation coefficient loss and image prior loss , the specific expression is as follows: wherein: λ cycle , λ cor-coe , and λ prior are the corresponding weight parameters, H and L represent the input high-quality PET image and low-quality PET image, respectively, G H ( ) denotes the transfer function of the generative network G H , F L ( ) denotes the transfer function of the generative network F L , Cov( ) denotes the covariance operator, σ( ) denotes the variance operator, I prior_img is the prior image corresponding to L, and || ||1 denotes the L1 norm; (5) Inputting the unpaired high-quality PET images and low-quality PET images in the training set and the corresponding prior images of the low-quality PET images into a CycleGAN model for training. (6) For the trained CycleGAN model, taking the generation network G in the model for converting low-quality PET images into high-quality PET images H , and then inputting the low-quality PET images that need to be quality-improved into the generation network G H , the corresponding high-quality PET images can be directly outputted.
2. The unpaired PET image quality enhancement method of claim 1, wherein: In the step (3), the BM3D denoising algorithm is used to pre-process the low-quality PET images to obtain the corresponding prior images.
3. The unpaired PET image quality enhancement method of claim 1, wherein: The CycleGAN model comprises two generation networks G H and F L and two discrimination networks D H and D L , wherein the generation network G H is used to convert low-quality PET images into high-quality PET images, the generation network F L is used to convert high-quality PET images into low-quality PET images, the discrimination network D H is used to judge whether the converted generated high-quality PET images conform to the high-quality PET image domain distribution in the training set, and the discrimination network D L is used to judge whether the converted generated low-quality PET images conform to the low-quality PET image domain distribution in the training set.
4. The unpaired PET image quality enhancement method of claim 3, wherein: The generation network G H and F L The discriminator network D H and D L PatchGAN structure is adopted.
5. The unpaired PET image quality enhancement method of claim 1, wherein: The adversarial loss The expression of the adversarial loss is as follows: where H and L represent the input high-quality PET image and low-quality PET image, respectively, D H represents the transfer function of the discriminator network D H L represents the transfer function of the discriminator network D L H represents the transfer function of the generator network G H L represents the transfer function of the generator network F L represents the expected function of H subject to the distribution of the high-quality PET image domain P data(H) in the training set, represents the expected function of L subject to the distribution of the low-quality PET image domain P data(L) in the training set. 6. The unpaired PET image quality enhancement method of claim 1, wherein: the cycle consistency loss The expression is as follows: where H and L represent the input high-quality PET image and low-quality PET image, respectively, G H represents the transfer function of the generator network G H L represents the transfer function of the generator network F L represents the expected function of H subject to the distribution of the high-quality PET image domain P data(H) in the training set, represents the expected function of L subject to the distribution of the low-quality PET image domain P data(L) in the training set, and || ||1represents the L1norm.