Systems and methods for synthesizing image data
The system generates synthetic medical data using a variational autoencoder and denoising diffusion model to address privacy concerns in medical imaging, producing realistic data that supports effective deep learning model training while ensuring patient privacy, thus overcoming the limitations of limited datasets.
Patent Information
- Application Number
- PCT/US2024/031962
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-05-30
- Filing Date
- 2024-05-31
- Publication Date
- 2024-12-05
AI Technical Summary
The challenge in medical imaging is the limited availability of large and diverse public medical datasets due to stringent patient privacy regulations, such as HIPAA and GDPR, which hinders the development of effective deep learning models for tasks like low-dose CT denoising and MRI/PET super-resolution, as existing methods struggle to balance data sharing with privacy protection.
The system employs a variational autoencoder to generate synthetic medical image data by transforming an original dataset into a latent space, which is then used to train a denoising diffusion model to produce synthetic images. Enhanced privacy protection is achieved by converting images into vectors and removing synthetic images with an L2 distance below a threshold to prevent membership inference attacks.
This approach allows for the generation of realistic synthetic medical data that maintains patient privacy, enabling effective training of deep learning models for downstream medical imaging tasks with performance comparable to models trained on original data, while minimizing the risk of privacy leakage.
Smart Images

Figure US2024031962_05122024_PF_FP_ABST
Abstract
Description
[0001]104848-201 SYSTEMS AND METHODS FOR SYNTHESIZING IMAGE DATA CROSS REFERENCE TO RELATED APPLICATION(S) This application claims the benefit of U.S. Provisional Application No.63 / 470,001, filed May 31, 2023, and U.S. Provisional Application No.63 / 653,441, filed May 30, 2024, both of which are incorporated by reference as if disclosed herein in their entireties. GOVERNMENT LICENSE RIGHTS This invention was made with government support under award number EB016977 awarded by the National Institutes of Health (NIH). The government has certain rights in the invention. FIELD The present disclosure relates to deep learning methods in medical imaging, in particular to generating synthetic training data for deep learning models. BACKGROUND Artificial intelligence (“AI”) methods, especially deep learning technologies, have demonstrated positive results in healthcare imaging by harnessing big datasets (such as ImageNet for natural image processing) and employing large models (such as ChatGPT and diffusion models). However, the imperative to preserve patient privacy has been a challenge for data sharing and hindered the development of big data-empowered AI models in the medical field. Governed by privacy legislation, including the United States Health Insurance Portability and Accountability Act (HIPAA) and the European General Data Protection Regulation (GDPR), patient privacy is legally protected, posing a major challenge to medical data sharing and deep learning-based medical imaging. Adherence to ethical, moral, legal and scientific guidelines is crucial to advancing science while respecting privacy, complicating the collection of medical data, which is both intricate and costly. Consequently, public medical datasets are limited in size and diversity, typically containing only a few thousand cases at most and being relevant only to a limited number of clinical problems. What is desired, therefore, are systems and methods for reliably and effectively generating training data that both provides effective model training and preserves patient privacy. 1 29151945.1 104848-201 SUMMARY According to some embodiments of the present technology, systems, methods, and software for generating synthetic image data based on an original image dataset are provided. Some embodiments utilize a variational autoencoder trained on an original dataset to generate a latent space dataset to train a denoising diffusion model to generate synthetic latent space data which is then converted to a synthetic image by the trained decoder circuitry of the variational autoencoder. Some embodiments include enhanced privacy protection in which the original images and the synthetic images are converted to vectors, the L2 distance between a synthetic image vector and each original image vector is compared, and synthetic images for which the L2 distance is below a selected threshold are removed from the synthetic dataset. Various embodiments of the technology will now be described with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS The drawings show embodiments of the disclosed subject matter for the purpose of illustrating features and advantages of the disclosed subject matter. However, it should be understood that the present application is not limited to the precise arrangements and instrumentalities shown in the drawings, wherein: FIG.1a shows the architecture of a latent diffusion model according to an embodiment of the present technology. FIG.1b shows the architecture of a variational autoencoder according to the embodiment of FIG.1a. FIG.1c shows additional details about the components shown in FIG.1a and 1b. FIG.2a is a flow chart showing a privacy protection feature of the first embodiment of the present technology. FIG.2b shows the histograms for root-mean-square error between synthetic medical images generated by an embodiment of the present technology without certain privacy features (LDM) and between synthetic medical images generated by an embodiment of the present technology with certain privacy features (PE-LDM). FIG.3 shows a functional block diagram of a system according to an embodiment of the present technology. FIG.4a shows chest CT denoising images. For the FBP image, left and right halves are from NDCT and LDCT respectively; for other methods, left and right halves are from 2 29151945.1 104848-201 original and synthetic datasets respectively, and the zoom-in views of the ROIs for each image correspond to the box in the FBP image to show the results denoised based on the original and synthetic data respectively. FIG.4b shows abdomen CT denoising images presented in the same fashion as FIG. 4a. FIG.4c shows MRI super-resolution images. For the analytic reconstructed image, left and right halves are from high-resolution and low-resolution respectively; for other methods, left and right halves are from original and synthetic datasets respectively, and the zoom-in views of the ROIs for each image correspond to the box in the analytic image to show the results denoised based on the original and synthetic data respectively. FIG.4d shows PET super-resolution images presented in the same fashion as FIG.4c. Although the following Detailed Description will proceed with reference being made to illustrative embodiments, many alternatives, modifications, and variations thereof will be apparent to those skilled in the art. DETAILED DESCRIPTION Score-based generative models, a category that includes denoising diffusion probabilistic models (“DDPM”) and latent diffusion models (“LDM”), have demonstrated advantages in the synthesis of natural images, including as compared to generative adversarial networks (“GAN”). The diffusion model transforms input data into noisy data by gradually adding Gaussian noise and then iteratively regenerates the data distribution from pure noise, effectively recovering subtle image details and structures of various types. Although diffusion models exhibit potential for medical image synthesis, privacy features are desired for use of such models in privacy-sensitive applications, such as those in which HIPAA applies. Accordingly, some embodiments of the present technology utilize diffusion models for privacy protection in medical image synthesis for downstream medical imaging tasks, such as low-dose computed tomography (LDCT) denoising, MRI super-resolution and positron emission tomography (PET) super-resolution, as well as the development of a healthcare metaverse. As used herein, medical images comprise CT, MRI, PET, and other medical image types consistent with this disclosure. Downstream medical imaging tasks have been a vibrant area of research, spurred by the concerns over patient exposure to ionizing radiation of CT and the needs for enhanced image resolution / quality in MRI and PET. Deep learning techniques have been employed in denoising and super-resolution using neural networks to learn the relationship between low- 3 29151945.1 104848-201 quality images and their paired high-quality counterparts. Deep learning-based denoising methods for LDCT have included projection domain pre-processing, analytic unrolling, dual domain, deep prior-based iterative reconstruction, model-based unrolling, and image post- processing methods. Deep learning-based super-resolution methods in MRI and PET have primarily concentrated on image post-processing techniques. For the development of these downstream medical imaging tasks, a sufficiently large dataset is imperative for training the respective systems. Protecting patient privacy associated with any medical images is also important. Thus, some embodiments of this technology include systems and methods for synthesizing realistic data under the original distribution of clinical medical data while protecting patient privacy. That is, in some embodiments, an LDM generates medical images that obey the same distribution of the original clinical medical imaging data. In some scenarios, such data will be shareable without compromising patient data privacy. In one embodiment, an LDM is constructed and trained to synthesize realistic medical images from original clinical medical data. In some embodiments, privacy protection is effected inside the LDM by blocking any synthetic images that sufficiently resemble the original images. Some such embodiments provide an effective defense against membership inference attacks. Experimental results show that a number of deep learning-based models trained on synthetic data created using an embodiment of the present technology provide acceptable and even equivalent performance in downstream medical imaging tasks (including LDCT denoising and MRI and PET super-resolution) as compared to the same models trained on original medical imaging data. FIGs 1a, 1b, and 1c show components of a system 100 for generating synthetic medical data according to some embodiments of the present technology. In some embodiments, the data is in the form of images. The system comprises a score-based generative model circuitry 101 and a decoder circuitry 102. In some embodiments, including the embodiment shown in FIG.1a, the score-based generative model is in the form of a denoising diffusion model, but it should be understood that other types of score-based generative or score matching models are used in other embodiments. In this embodiment, the denoising diffusion model circuitry 101 has been trained using a latent space dataset 103 encoded from a set of captured medical data 104 and configured to generate a synthetic latent space data distribution 103’ by denoising a Gaussian noise distribution 105. In this embodiment, the decoder circuitry 102 decodes the synthetic latent space 103’ to generate a 4 29151945.1 104848-201 synthetic medical data 107. The decoder circuitry 102 was trained to decode the latent space dataset 103 corresponding to the set of captured medical data 104. As discussed further below, the system includes in some embodiments one or more computer processors and one or more computer-readable storage media storing circuitry comprising software instructions as described herein. In some embodiments, the latent space dataset 103 was encoded using a variational autoencoder circuitry 108. In some embodiments, the variational autoencoder (VAE) circuitry is employed to compress the image data into a latent space, which can reduce the computational cost for training the diffusion model. FIG.1b shows the VAE circuitry architecture according to some embodiments. The encoder circuitry 109, in some embodiments, comprises a series of ResBlocks 110 and downsampling blocks 111, followed by a convolutional layer 112. The decoder circuitry 102 comprises a convolutional layer 113, followed by a series of ResBlocks 110 and upsampling blocks 115. FIG.1c provides additional details about the structure of the ResBlocks 110 / 114, downsampling blocks 111, and upsampling blocks 115 in some embodiments. In some embodiments, the ResBlocks 110 comprise ConvBlocks 116 and a convolutional layer 117, in which the output of a layer is added to a layer deeper in the block. In some embodiments, each ConvBlock 116 comprises a convolutional layer, a normalization layer, and an activation layer. The downsampling blocks 111 comprise a convolutional layer with stride = 2, in some embodiments. The upsampling blocks 115 comprise, in some embodiments, a nearest neighbor unpooling layer and a convolutional layer. Training the VAE involves two processes: encoding and decoding. In the encoding process, a distribution of real medical images ^^ (104) is mapped to a posterior distribution ^^( ^^| ^^) that is approximated as a normal distribution of mean μ and standard deviation σ where the latent variable ^^ = ^^ + ^^ ^^, ^^~ ^^(0,1). The encoder can be parametrized by a neural network as ^^^^( ^^| ^^). In the decoding process, VAE generates a new medical image ^^̂ from a latent variable ^^ though the generative model ^^( ^^| ^^). The decoder can be parametrized as another neural network ^^^^( ^^| ^^). The goal of the VAE training is to reconstruct the image ^^̂ close to the original image ^^. The VAE loss consists of two terms. The first term is the negative Kullback-Leibler (KL) divergence between ^^^^(^^|^^)and ^^(^^), where ^^(^^)is a Gaussian distribution. The second term corresponds to the expected log-likelihood of the observation ^^: 5 29151945.1 104848-201 Once the VAE is trained, the encoder ^^^^( ^^| ^^) is capable of encoding a medical image x into a latent representation z, and the decoder ^^^^( ^^| ^^) can then reconstruct an image ^^̂ from the latent ^^. The set of reconstructed images is depicted as 104’ in FIG.1b. Thus, in some embodiments, training the decoder circuitry comprises mapping the set of captured medical data into the latent space dataset using the encoder circuitry; reconstructing, on an iterative basis, data from the set of captured medical data in the decoder circuitry from the latent space dataset; calculating a loss function based on a comparison of each reconstructed data to its corresponding data in the set of captured medical data; and updating one or more parameters of the decoder circuitry based on the calculation of the loss function. The following table titled Algorithm 1 shows pseudo-code for training the VAE circuitry 108 according to some embodiments: After the set of captured medical images are compressed into the latent space dataset using the VAE, the denoising diffusion model circuitry 101 can be initiated to generate a synthetic dataset. Referring again to FIG.1a, the system 100 (which can be considered a latent diffusion model or LDM in this embodiment) includes two sequentially trained models: the VAE circuitry 108 and the denoising diffusion model circuitry 101. In the first training phase described above, the VAE circuitry 108 is trained to encode the image space into a latent space. During this VAE training phase, the latent space is directly decoded back into the image space. In the second training phase, upon completion of the training of the VAE circuitry including the decoder circuitry, the denoising diffusion model circuitry 101 is trained in the latent space defined by the pre-trained VAE. In some embodiments, the denoising diffusion model circuitry 101 is in the form of a DDPM. In some embodiments, the weights of the VAE are frozen in the DDPM training phase. The compressed data ^^ is 6 29151945.1 104848-201 denoted as ^^0= ^^ for the DDPM training. The forward process 118 of DDPM gradually adds Gaussian noise upon a compressed sample ^^0∼ ^^(^^0)over the course of ^^ timesteps according to a variance schedule ^^1, ⋯ , ^^^^. The latent variables ^^1, ⋯ , ^^^^have the same dimensionality as the starting point ^^0∼ ^^(^^0). By the nature of the Gaussian distribution, sampling ^^^^at an arbitrary timestep ^^ can be written as ^^^^= where ^^^^= 1 − ^^^^and ^̅^^^=∏ ^^^^=1^^^^. After the forward process, ^^^^follows a standard normal distribution when ^^ is large enough. Thus, if the joint distribution ^^( ^^^^−1| ^^^^) is known, the reverse process can be used to get a sample from ^^( ^^0) and ^^^^~ ^^(0,1). However, ^^( ^^^^−1| ^^^^) depends on the entire data distribution, making it difficult to calculate. Hence, a neural network was designed to learn a latent data distribution by gradually denoising a Gaussian random variable, which is to learn the reverse process of a fixed Markov chain with length ^^. Specifically, a neural network model ^^^^(^^^^, ^^)was used to predict the noise ^^. Consequently, the corresponding objective can be written as with t being uniformly sampled from {1, ⋯ , ^^}. In this study, the latent space is diffused into Gaussian noise using ^^ = 1,000 steps. The following table titled Algorithm 2 shows pseudo- code for training the denoising diffusion model circuitry in some embodiments: In some embodiments, a U-Net model ^^^^( ^^^^, ^^) (120 in FIG.1) is trained to predict the noise ^ in the latent space. A U-Net is a convolutional neural network that includes only convolutional layers, and is a relatively popular architecture for medical image segmentation. Each U-Net is an asymmetrical network that includes one or more encoder and decoder units. In other embodiments, other artificial neural network (ANN) architectures are implemented 7 29151945.1 104848-201 consistent with the present disclosure. In some embodiments, samples (i.e., synthetic medical images) are then generated with the denoising diffusion circuitry 101 and the decoder circuitry 102 of the VAE circuitry 108 in a single pass. In some embodiments, the denoising diffusion model circuitry 101 comprises a denoising diffusion implicit model (DDIM) and ^^ = 150 steps for generating the synthetic medical images in conjunction with the decoder of the VAE in a single pass. In some embodiments, using the LDM to generate a synthetic dataset for privacy protection, offers a fundamental advantage over other approaches to the problem (such as use of federated learning): there is no need to share original data or the weights trained using the original data. However, using an LDM includes a small risk that individual training examples will be memorized (known as the membership inference), thereby still potentially compromising privacy in a small number of instances. To address this concern, some embodiments include privacy protection features that directly eliminate synthetic images that closely resemble original images, serving as a defense against membership inference attacks. The privacy protection features of some embodiments are shown schematically in FIG.2. In some embodiments, the system 100 further comprises a privacy protection circuitry 200. The privacy protection circuitry 200 includes instructions for comparing the L2 distance between a compressed synthetic vector corresponding to the synthetic medical data generated by the LDM, and each of a set of compressed captured vectors corresponding to each item of captured medical data; and instructions for identifying synthetic medical data whose L2 distance is less than a selected threshold. In some embodiments, the privacy protection circuitry includes visual geometry group (“VGG”) circuitry 201, comprising instructions to convert each item of captured medical data (e.g., the contents of the original medical image dataset 104) to a compressed captured vector of length N and to convert each synthetic medical data 107 to a compressed synthetic vector of length N. In some embodiments, the VGG circuitry 201 converts the images in the original dataset into vectors of length 1,000 and records all these vectors in an original dictionary 202. Similarly, in some embodiments the output of the LDM system 100, i.e., the synthetic medical data, are converted to vectors of length 1,000 and stored in a synthetic dictionary 203. In some embodiments, the system further comprises L2 calculation circuitry 204 comprising instructions to calculate the L2 distance between each compressed synthetic vector and all of the compressed captured vectors and to compare the calculated L2 distance to a selected threshold in an L2 threshold circuitry 205. L2 distance refers to the Euclidean 8 29151945.1 104848-201 distance. By comparing the output vectors with the vectors in the original dictionary and setting a threshold based on the L2 distance, the synthetic images whose L2 distance is smaller than the threshold can be identified. In some embodiments, synthetic medical data identified as having an L2 distance less than the selected threshold is deleted, removed, blocked, or quarantined in the privacy protection circuitry 200. The result is a “privacy enhanced” dataset 206. The following table titled Algorithm 3 shows pseudo-code for the process of privacy enhancement using the an LDM in some embodiments (i.e., “PE-LDM”): In some embodiments, the selected threshold is set empirically based on the level of privacy protection desired by a user, e.g., a radiologist. In some embodiments, the threshold is set to avoid or reduce the vulnerability of the output to a membership inference attack. The threshold can be set at different levels depending on the datasets at issue. In general, synthetic images created using some embodiments of the present technology incorporating the above privacy features present very low risk of privacy leakage. FIG.2b shows the histograms of the root-mean-square error (“RMSE”) between synthetic images made in an LDM according to an embodiment without privacy protection features and closely similar original images and between synthetic images made in a PE-LDM according to an embodiment and closely similar original images. The histograms demonstrate that the minimal RMSE of PE-LDM is larger than that of LDM, indicating that synthetic images generated by PE-LDM are significantly dissimilar to the original images. FIG.3 shows a functional block diagram of a system 300 according to some embodiments of the present technology. Computing device 301 may include, but is not limited to, a computing system (e.g., a server, a workstation computer, a desktop computer, a laptop computer, a tablet computer, an ultraportable computer, an ultramobile computer, a netbook computer and / or a subnotebook computer, etc.), and / or a smart phone. Computing device 301 includes processor circuitry 340, a memory circuitry 342, input / output (I / O) 9 29151945.1 104848-201 circuitry 344, a user interface (UI) 346, and data store 348. Processor circuitry 340 includes a plurality of processing units 350-1, …, 350-p. One or more of the processing units 350-1, …, 350-p may include and / or may correspond to a graphics processing unit (GPU). Processor circuitry 340 is configured to perform operations of system 300, including, for example, system 100, Privacy protection circuitry 200, VGG circuitry 201, L2 calculation circuitry 204, L2 threshold circuitry 205, and training circuitry 303. Memory circuitry 342 (i.e., computer-readable storage media) may include a plurality of memory units 342-1, …, 342-p, similar to the processing units 350-1, …, 350-p of processor circuitry 340. Thus, each processing unit, e.g., processing unit 350-1, may have an associated memory unit, e.g., memory unit 342-1, configured to store a respective subset of data, as described herein. In one nonlimiting example, each pair of processing unit 350-1 and memory unit 342-1, may be distributed. In some embodiments, therefore, the system 300 comprises a plurality of processors and a first set of the plurality of processors is located at a first node of a distributed computing system and a second set of the plurality of processors is located at a second node of the distributed computing system. Memory circuitry 342 may be configured to store data associated with the system 300 and its various components. I / O circuitry 344 may be configured to provide wired and / or wireless communication functionality the system. For example, I / O circuitry 344 may be configured to receive training input data, and / or input data and parameters 317. UI 346 may include a user input device (e.g., keyboard, mouse, microphone, touch sensitive display, etc.) and / or a user output device, e.g., a display. Data store 348 may be configured to store one or more of training input data, input data and parameters 317, training data, network parameters, and / or other data associated with the system. In some embodiments, the system 300 further comprises training circuitry 303 which is configured to is configured to receive training input data, and to determine network parameters associated with the system 11. In some embodiments, the training circuitry 303 is configured to train, via instructions on the storage media of the system, the VAE circuitry 108 and the denoising diffusion model circuitry 101. According to another embodiment of the present technology, a method for generating synthetic medical data is provided, comprising the steps of: generating, by a score-based generative model circuitry 101 trained using a latent space dataset 103 encoded from a set of captured medical data 104, a synthetic data latent space 103’ by denoising a Gaussian noise 10 29151945.1 104848-201 distribution; and decoding, by a decoder circuitry 102, the synthetic data latent space 103’ to generate a synthetic medical data 107, wherein the decoder circuitry was trained to decode the latent space dataset corresponding to the set of captured medical data 104. In some embodiments, the method further comprises the steps of comparing, in a privacy protection circuitry 200, the L2 distance between a compressed synthetic vector corresponding to the synthetic medical data and each of a set of compressed captured vectors corresponding to each item of captured medical data; and identifying, in the privacy protection circuitry 200, synthetic medical data whose L2 distance is less than a selected threshold. In some embodiments, the method further comprises the step of training the decoder circuitry, in a variational autoencoder circuitry 108, by: mapping the set of captured medical data into the latent space dataset using an encoder circuitry 109; reconstructing, on an iterative basis, data from the set of captured medical data in the decoder circuitry 102 from the latent space dataset; calculating a loss function based on a comparison of each reconstructed data to its corresponding data in the set of captured medical data; and updating one or more parameters of the decoder circuitry based on the calculation of the loss function. In some embodiments, the method further comprises the step of training the denoising diffusion model circuitry using the latent space dataset by a training circuitry 303 upon completion of the training of the decoder circuitry. In some embodiments, the method further comprises the steps of: converting, in a visual geometry group circuitry 201, each captured medical data to a compressed captured vector of length N; converting, in the visual geometry group circuitry 201, each synthetic medical data to a compressed synthetic vector of length N; and deleting, in the privacy protection circuitry, synthetic medical data identified as having an L2 distance less than the selected threshold. EXAMPLES In exemplary embodiments, LDM systems, built and trained as described above, were used to generate synthetic datasets from the Mayo low-dose CT challenge dataset, IXI brain MRI dataset, and brain PET image analysis and disease prediction challenge dataset. The Mayo LDCT challenge dataset, IXI brain MRI dataset and brain PET image analysis and disease prediction challenge dataset were used to individually train each LDM and validate the clinical performance of the LDCT denoising and MRI, PET super-resolution networks, trained on either the original or synthetic data. In the VAE training phase, the high-quality images from the original datasets were used to train the networks. Then, these high-quality 11 29151945.1 104848-201 images were compressed into a latent space to train the DDPM model. The trained LDMs were used to randomly generate 2,377 CT, 7000 MRI, and 7000 PET synthetic images, each with dimensions of 512×512, 256×256, and 256×256 respectively. Simultaneously, 2,377 CT, 7000 MRI, and 7000 PET images were selected from their respective original dataset, with 211 CT, 1000 MRI, and 1000 PET images forming the test dataset. For both the original and synthetic datasets, the corresponding LDCT images were produced by adding Poisson noise into the sinograms simulated from the normal-dose CT images. The noise level can be controlled by the number of photons in the reference air scan, which was uniformly set to 104 photons. The FBP algorithm was employed for benchmark image reconstruction. Additionally, the corresponding low-resolution MRI and PET images were produced by 4× down-sampling. To compress 512×512 or 256×256 images into a latent space of 64×64 or 32×32 dimensions, the VAE was first trained with 4 channels. Following the VAE training, its weights were frozen and the DDPM was trained using an UNet architecture for denoising. Both VAE and DDPM were trained at least 100 epochs, and the training process was stopped when the loss did not decrease by 1% relative to the average loss for the previous 30 epochs. A fixed batch size of 4 was used. The trained LDMs were used to generate synthetic datasets, and then the selected LDCT denoising and MRI, PET super-resolution networks were evaluated on the original and / or synthetic datasets. In reference to the corresponding original references, the parameters of all the methods were optimized and the same parameters were maintained for each algorithm in the cases of the original and / or synthetic datasets. In these exemplary embodiments, the synthetic medical image datasets achieved Fréchet Inception Distance (FID) scores of 31, 59 and 66, for the LDCT, MIR, and PET datasets, respectively. Representative images from the involved imaging modalities, including chest CT, brain MRI, and brain PET, were selected for assessment from both the original and synthetic datasets. For each synthetic image, its nearest-matched image in the original dataset was identified for comparison. The synthetic images exhibited global consistency with their corresponding original images and local structural differences. For example, the lung textural features in the original and synthetic CT images are both realistic and yet not identical. The synthetic MRI and PET images exhibited a higher degree of consistency with their respective original images and showed anatomically reasonable variations. These variations enhance the diversity of the synthetic dataset, while the global 12 29151945.1 104848-201 consistency suggests high image quality. To assess the potential of synthetic medical image data in the performance of downstream medical imaging tasks, several representative deep learning models were trained using the public medical dataset and the synthetic counterpart for comparison. The results indicate that these models exhibit similar performance after the networks are trained on either original or synthetic datasets, which supports the feasibility of using synthetic data for the downstream medical imaging tasks. In these exemplary embodiments, the approach for training deep denoising and super- resolution networks involves two essential steps: First, an LDM as described above was used to generate a synthetic dataset. Subsequently, a first set of denoising / super-resolution networks were trained on the original datasets, while a second set were trained on the synthetic datasets. Previously reported studies have demonstrated promising results with these deep learning-based methods when applied to the original dataset, as they effectively learn the relationship between low-quality and high-quality images. The primary objective of this experiment was to assess the feasibility of training these deep learning-based methods on the synthetic dataset. To achieve this, the denoising / super-resolution performance of the models trained on the synthetic dataset was compared with that of the same models trained on the original dataset. The selected representative LDCT denoising methods include RED- CNN, FISTA-Net, SUNet. Additionally, the selected representative MRI and PET super- resolution methods, including SRCNN, SWIN, and DDPM, focus on the image domain. All denoising results obtained with these methods were evaluated through both visual inspection and quantitative assessments. The image quality metrics employed encompassed PSNR and SSIM. The statistical analysis employed the independent samples t-test to assess PSNR. A significance threshold of p < 0.05 (two-tailed) was applied for all tests. All calculations were performed with a standard personal computer using SPSS software for Windows, version 29.0.2.0. To illustrate the denoising effect of the representative networks for LDCT reconstruction, slices from the chest and abdomen regions were selected, as depicted in FIGs. 5a and 5b respectively, where the magnified regions-of-interest (ROIs) show details in the lungs and abdomen as marked by the rectangles. The filtered back-projection (FBP) LDCT images in Figs.5a and 5b exhibit severe noise, with a higher noise level in Fig.5b due to the stronger attenuation of the abdomen than the lungs. This difference is more pronounced in the magnified ROIs: while the lung texture is affected by noise but still identifiable, the details in 13 29151945.1 104848-201 the abdomen are difficult to discern. Unlike natural images, CT images have non-uniform noise levels in different regions, posing a unique challenge. It is observed that all evaluated networks demonstrated their denoising capabilities to different degrees. The streak artifacts in the chest images were effectively suppressed, and the texture details in the lungs were well preserved. In contrast, the image quality in the abdominal images significantly improved, but the details could not be recovered well due to the severe noise, as shown in the corresponding magnified views. Although all the LDCT reconstruction models significantly improve image quality, particularly with respect to textural details in the lung regions, small differences can still be observed between them. For example, the RED-CNN method suppresses the noise and preserves texture details well, but over-smooths small structures. In contrast, the SUNet59 method enhances these small structures but amplifies some artifacts in the lungs. When comparing these methods trained on the original and synthetic datasets respectively, the denoising results do not show any significant difference. In Fig.5b, all the methods lose some fine details due to the elevated noise level. The RED-CNN method over-smooths fine details, while the SUNet method distorts some details. When comparing these methods trained on the original and synthetic datasets, the denoising performance is again essentially identical. It appears that the results obtained using the same network trained on either the original or synthetic data are virtually the same without any noticeable difference. This visual inspection demonstrates that training the network on original and synthetic datasets makes little difference. To showcase the super-resolution performance in the MRI and PET scenarios, a comparative analysis of various super-resolution networks using brain images was conducted, as depicted in Figs.5c and 5d. To highlight specific details, the ROIs were magnified indicated by the rectangles. All the selected methods, including SRCNN60, SWIN61, and DDPM53, exhibited a consistent performance trend in both MRI and PET 4× super- resolution tasks. Notably, low-resolution images in Figs.5c and 5d introduced jagged artifacts. The SRCNN method mitigated these artifacts, although the overall result remained somewhat blurry. The SWIN method enhanced performance by introducing sharper edges, yet some fine details remained unrecovered. In visual evaluation, the DDPM method generated sharp edges, closely approximating the ground truth; however, it produced tiny details that differed from the ground truth. While different methods displayed varying performance on MRI and PET super-resolution tasks, visual inspections of these results 14 29151945.1 104848-201 yielded no significant differences from training the models on either original or synthetic data. A systematic quantitative analysis was also conducted by calculating the peak signal- to-noise ratio (PSNR) and structure similarity (SSIM) values for each image in the whole test dataset and compared the averages in Table I. The results demonstrate excellent alignment with the visual inspection described above. Although all the denoising methods show significant improvement relative to the FBP LDCT reconstruction, different methods demonstrate varied quantitative metrics. FISTA-Net47 achieved relatively higher PSNR and SSIM by incorporating sinogram information into the deep denoising process. Using an advanced transformer architecture, The SUNet method produced the best metrics. Most importantly, the denoising results using methods trained on either the original or synthetic data are quantitatively equivalent, as indicated by all p value exceeding 0.05. This statistical summary establishes the feasibility of using a synthetic dataset for LDCT training. Table I. Measurements of PSNR and SSIM. Modality MethodsPNSR SSIMp ValueOriginal Synthetic Original Synthetic LD-FBP 36.10±2.34 NA 0.778±0.055 NA NA CT RED-CNN 42.46±1.80 42.45±1.78 0.879±0.029 0.879±0.030 0.936 FISTA-Net 43.47±2.02 43.26±2.14 0.896±0.242 0.894±0.263 0.303 SUNet 44.92±1.89 44.77±1.84 0.922±0.026 0.920±0.027 0.416 Analytic 21.39±1.06 NA 0.844±0.028 NA NA MRI SRCNN 26.83±1.25 26.82±1.28 0.936±0.014 0.938±0.015 0.937 SWIN 28.17±1.45 28.06±1.49 0.952±0.014 0.952±0.014 0.100 DDPM 26.19±1.60 26.28±1.59 0.913±0.015 0.912±0.014 0.245 Analytic 26.85±0.83 NA 0.856±0.273 NA NA PET SRCNN 38.75±3.94 38.69±4.12 0.972±0.013 0.973±0.014 0.729 SWIN 39.00±3.60 38.78±4.17 0.976±0.012 0.974±0.014 0.208 DDPM 37.05±3.32 36.80±3.28 0.942±0.018 0.946±0.016 0.079 Note. – Unless otherwise specified, data are means ± standard deviations. The results of super-resolution also aligned well with the visual observations. Although all super-resolution methods demonstrated significant improvement over the low- resolution baseline, they exhibited diverse quantitative metrics. The SWIN method achieved higher PSNR and SSIM than SRCNN. Despite DDPM generated visually appealing results, its quantitative metrics were inferior to SRCNN and SWIN due to discrepancies in relevant details. Most importantly, super-resolution methods trained on either original or synthetic data showed quantitatively very similar results, with all p value exceeding 0.05. This statistical summary affirms the feasibility of using a synthetic dataset for MRI and PET super-resolution training. Thus, the statistical distribution of original data is substantially the same as that of 15 29151945.1 104848-201 synthetic data produced by adapted diffusion model in some embodiments with a dedicated safeguard in the latent space to improve privacy protection. Further, the use of either original or synthetic medical images made by some embodiments of this technology in training deep networks for LDCT denoising or MRI, PET super-resolution has not been found not produce any significant difference in performance. One embodiment for sharing big data according to this technology is to simply share privacy-proof synthetic data. By the design of some embodiments of the adapted diffusion model, all the synthetic data samples are ‘new’ vis-à-vis the original data and realistic under the same original data distribution. Another embodiment is to share an executable code of a locally trained diffusion model. This approach avoids the need to transfer synthetic data. A user who receives the trained model can generate as many data samples as needed. In some embodiments, a secure computing technique can be applied to ensure that the executable code cannot be reverse engineered. Some embodiments of the present technology thus include embedding a privacy safeguard algorithm in the diffusion model so that image synthesis, data augmentation, and privacy protection can be simultaneously achieved. Other embodiments of the present technology include non-transitory computer- readable storage media and / or devices having stored thereon instructions that when executed by one or more processors perform the methods and processes described herein. The processor may include, for example, a processing unit and / or programmable circuitry. The storage device may include a machine readable storage device including any type of tangible, non-transitory storage device, for example, any type of disk including floppy disks, optical disks, compact disk read-only memories (CD-ROMs), compact disk rewritables (CD-RWs), and magneto-optical disks, semiconductor devices such as read-only memories (ROMs), random access memories (RAMs) such as dynamic and static RAMs, erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), flash memories, magnetic or optical cards, or any type of storage devices suitable for storing electronic instructions. As used in any embodiment herein, “network”, “model”, “ANN”, and “neural network” (NN) may be used interchangeably, and all refer to an artificial neural network that has an appropriate network architecture. Network architectures may include one or more layers that may be sparse, dense, linear, convolutional, and / or fully connected. It may be appreciated that deep learning includes training an ANN. Each ANN may include, but is not 16 29151945.1 104848-201 limited to, a deep NN (DNN), a convolutional neural network (CNN), a deep CNN (DCNN), a multilayer perceptron (MLP), etc. Training generally corresponds to “optimizing” the ANN, according to a defined metric, e.g., minimizing a cost (e.g., loss) function. As used in any embodiment herein, the term “logic” may refer to an app, software, firmware and / or circuitry configured to perform any of the aforementioned operations. Software may be embodied as a software package, code, instructions, instruction sets and / or data recorded on non-transitory computer readable storage media. Firmware may be embodied as code, instructions or instruction sets and / or data that are hard-coded (e.g., nonvolatile) in memory devices. “Circuitry”, as used in any embodiment herein, may include, for example, singly or in any combination, hardwired circuitry, programmable circuitry such as computer processors comprising one or more individual instruction processing cores (including, but not limited to, graphics processing units), state machine circuitry, software instructions, and / or firmware that stores instructions executed by programmable circuitry. The logic may, collectively or individually, be embodied as circuitry that forms part of a larger system, for example, an integrated circuit (IC), an application-specific integrated circuit (ASIC), a system on-chip (SoC), desktop computers, laptop computers, tablet computers, servers, smart phones, etc. Memory circuitry 342 may include one or more of the following types of memory: semiconductor firmware memory, programmable memory, non-volatile memory, read only memory, electrically programmable memory, random access memory, flash memory, magnetic disk memory, solid state memory, and / or optical disk memory. Either additionally or alternatively system memory may include other and / or later-developed types of computer- readable memory. The terms and expressions which have been employed herein are used as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding any equivalents of the features shown and described (or portions thereof), and it is recognized that various modifications are possible within the scope of the claims. Accordingly, the claims are intended to cover all such equivalents. Various features, aspects, and embodiments have been described herein. The features, aspects, and embodiments are susceptible to combination with one another as well as to variation and modification, as will be understood by those having skill in the art. The present disclosure should, therefore, be considered to encompass such combinations, variations, and modifications. 17 29151945.1
Claims
104848-201 CLAIMS What is claimed is:
1. A system for generating synthetic medical data, comprising: one or more processors; and one or more computer-readable storage media storing: a score-based generative model circuitry trained using a latent space dataset encoded from a set of captured medical data and configured to generate a synthetic data latent space by denoising a Gaussian noise distribution; and a decoder circuitry for decoding the synthetic data latent space to generate a synthetic medical data, wherein the decoder circuitry was trained to decode the latent space dataset corresponding to the set of captured medical data. a privacy protection circuitry for: comparing the L2 distance between a compressed synthetic vector corresponding to the synthetic medical data and each of a set of compressed captured vectors corresponding to each item of captured medical data; and identifying synthetic medical data whose L2 distance is less than a selected threshold.
2. The system of claim 1, wherein the latent space dataset was encoded using a variational autoencoder circuitry.
3. The system of claim 2, wherein the score-based generative model circuitry comprises a U-Net model.
4. The system according to claim 3, wherein the score-based generative model circuitry comprises a denoising diffusion implicit model.
5. The system of claim 1, further comprising that the privacy protection circuitry includes a visual geometry group circuitry, comprising instructions to convert each captured medical data to a compressed captured vector of length N and to convert each synthetic medical data to a compressed synthetic vector of length N; and 18 29151945.1104848-201 instructions to delete synthetic medical data identified as having an L2 distance less than the selected threshold.
6. A method for generating synthetic medical data, the method comprising the steps of: generating, in a score-based generative model circuitry trained using a latent space dataset encoded from a set of captured medical data, a synthetic data latent space by denoising a Gaussian noise distribution; decoding, in a decoder circuitry, the synthetic data latent space to generate a synthetic medical data, wherein the decoder circuitry was trained to decode the latent space dataset corresponding to the set of captured medical data; comparing, in a privacy protection circuitry, the L2 distance between a compressed synthetic vector corresponding to the synthetic medical data and each of a set of compressed captured vectors corresponding to each item of captured medical data; and identifying, in the privacy protection circuitry, synthetic medical data whose L2 distance is less than a selected threshold.
7. The method of claim 6, further comprising the step of training the decoder circuitry, in a variational autoencoder circuitry, by: mapping the set of captured medical data into the latent space dataset using an encoder circuitry; reconstructing, on an iterative basis, data from the set of captured medical data in the decoder circuitry from the latent space dataset; calculating a loss function based on a comparison of each reconstructed data to its corresponding data in the set of captured medical data; and updating one or more parameters of the decoder circuitry based on the calculation of the loss function.
8. The method of claim 7, further comprising the step of training the score-based generative model circuitry using the latent space dataset by a training circuitry upon completion of the training of the decoder circuitry.
9. The method of claim 6, wherein the score-based generative model circuitry comprises a U-Net model. 19 29151945.1104848-201 10. The method of claim 6, wherein the score-based generative model circuitry comprises a denoising diffusion implicit model.
11. The method of claim 6, further comprising the steps of: converting, in a visual geometry group circuitry, each captured medical data to a compressed captured vector of length N; converting, in the visual geometry group circuitry, each synthetic medical data to a compressed synthetic vector of length N; and deleting, in the privacy protection circuitry, synthetic medical data identified as having an L2 distance less than the selected threshold.
12. The method of claim 11, wherein N = 1000.
13. One or more non-transitory computer-readable storage media comprising instructions executable by the one or more processors for generating synthetic medical data by: generating, in a score-based generative model circuitry trained using a latent space dataset encoded from a set of captured medical data, a synthetic data latent space by denoising a Gaussian noise distribution; decoding, in a decoder circuitry, the synthetic data latent space to generate a synthetic medical data, wherein the decoder circuitry was trained to decode the latent space dataset corresponding to the set of captured medical data; comparing, in a privacy protection circuitry, the L2 distance between a compressed synthetic vector corresponding to the synthetic medical data and each of a set of compressed captured vectors corresponding to each item of captured medical data; and identifying, in the privacy protection circuitry, synthetic medical data whose L2 distance is less than a selected threshold.
14. The media of claim 13, wherein the instructions further comprise instructions for training the decoder circuitry, in a variational autoencoder circuitry, by: mapping the set of captured medical data into the latent space dataset in an encoder circuitry; reconstructing, on an iterative basis, data from the set of captured medical data in the decoder circuitry from the latent space dataset; 20 29151945.1104848-201 calculating a loss function based on a comparison of each reconstructed data to its corresponding data in the set of captured medical data; and updating one or more parameters of the decoder circuitry based on the calculation of the loss function.
15. The media of claim 14, further comprising instructions for training the denoising diffusion model circuitry using the latent space dataset by a training circuitry upon completion of the training of the decoder circuitry.
16. The media of claim 13, wherein the score-based generative model circuitry comprises a U-Net model.
17. The media of claim 13, wherein the score-based generative model circuitry comprises a denoising diffusion implicit model.
18. The media of claim 13, further comprising instructions for: converting, in a visual geometry group circuitry, each captured medical data to a compressed captured vector of length N; converting, in the visual geometry group circuitry, each synthetic medical data to a compressed synthetic vector of length N; and 19. deleting, in the privacy protection circuitry, synthetic medical data identified as having an L2 distance less than the selected threshold. The media of claim 13, wherein N = 1000. 21 29151945.1
Citation Information
Cited By
Rapid magnetic resonance image reconstruction method based on potential variable diffusion model
CN120182422A
Federal learning auxiliary edge caching method based on AE-DDPM model
CN120475448A
