Deep transfer learning for fully automated tumor quantification and prognosis of cancer on pet / ct

Abstract

Machine learning system for estimating a patient prognosis based on a patient radiotracer image are presented. The techniques include: obtaining patient imaging data that includes a patient radiotracer image; providing the patient imaging data to a first trained machine learning system trained using a plurality of training images, which include: images captured using a plurality of radiotracer types, images of a plurality of cancer types, or images captured at a plurality of institutions; obtaining a first output (e.g., a segmentation or quantification) from the first trained machine learning system; generating patient feature quantifications from the first output; providing the patient feature quantifications to a second trained machine learning system trained using training feature quantifications and training prognostic data; obtaining a second output (e.g., patient prognostic data) from the second trained machine learning system; and providing a patient prognosis based on the patient prognostic data.

Description

DEEP TRANSFER LEARNING FOR FULLY AUTOMATED TUMOR QUANTIFICATION AND PROGNOSIS OF CANCER ON PET / CTField

[0001] This disclosure relates generally to machine learning in the medical field.Background

[0002] Advances in artificial intelligence (Al) and machine learning (ML) have been driven by advances in high-performance computing and the availability of large- scale digital data. The number of medical AI / ML publications and AI / ML-enabled medical devices approved by the United States Food and Drug Administration (FDA) has rapidly increased, and 70% of the reviewed devices involved radiology. Al and ML approaches in medical imaging have significant promise in assisting physicians in the detection, diagnosis, and prognosis of cancer.

[0003] Radiomics performs high-throughput extraction of quantitative features from radiologic data and can provide information about a cancerous tumor, such as tumor morphology, spatial heterogeneity, and intensity distribution. Such features can be used for subtype identification and biomarker discovery for precision medicine and therapy. However, traditional radiomics approaches suffer from limited reproducibility due to small dataset sizes as well as variability in the scanner protocols, image reconstruction methods, and manually drawn delineations around regions of interest (ROIs).

[0004] Deep learning (DL) is a subset of Al based on deep neural networks that have had exceptional success in computer vision and medical imaging in recent years.Unlike radiomics workflows that utilize domain knowledge and engineered features,DL approaches automatically extract features from input images to directly model medical endpoints. Much of the success of DL is due to the ability of DL methods to automatically extract abstract visual features from data allowing inference in a lowdimensional space. Central to this is the manifold hypothesis, which states that highdimensional data (e.g., the number of voxels in an imaged volume) can be described with a number of features, known as the intrinsic dimension, that is much smaller than the original data dimensions.

[0005] DL approaches for radiological applications face significant challenges. DL methods require sufficient training on datasets with thousands, if not, millions of annotated images, which is difficult to obtain for radiological applications. DL methods are also more difficult to train on medical images than natural images for moderate to low dataset sizes due to the steep negative relationship between the intrinsic dimension of radiological data and generalization ability. Medical images are visually evaluated by radiologists for diagnosis and monitoring of diseases. The current training and evaluation paradigm present in Al and DL applications requires the generation of a large number of physician-defined annotations and ROIs on medical images to build a sufficient training set that is representative of the target patient population. However, this process is often a time-consuming and burdensome endeavor that is subject to sources of inter-operator and intra-operator variability, which may adversely affect the reliability of Al-based predictions. Differences in image processing and reconstruction protocols also contribute to variability across datasets and limit generalizability.

[0006] Recently, FDA-approved AI / ML medical devices have been largely developed for computed tomography (CT), magnetic resonance imaging (MRI), X-ray, or ultrasound, whereas positron emission tomography (PET) was the leastrepresented modality. The paucity of Al-based tools developed for PET reflects the relative lack of available annotated imaging and clinical data as nuclear medicine imaging is a relatively low-volume clinical modality when compared to other anatomical imaging modalities, such as CT and MRI.

[0007] Cancer is a major health concern worldwide and the second leading cause of death after heart disease accounting for 18% of all deaths for men and women in the United States in 2020. Cancer is also the leading cause of death for men and women aged 60-79 years and 40-79 years, respectively, where lung, prostate, breast, colorectal, and pancreas cancers were in the top five leading causes of cancer death in 2020. The American Cancer Society projected approximately 2 million new cancer cases and 610 thousand cancer deaths in the United States in 2023. Prostate, breast, lung, colorectal, melanoma, and lymphoma were among the top ten leading cancer types for estimated new cancer cases in 2023. Prostate and breast cancer alone accounted for 29% and 31 % of the proportion of cancer in men and women, respectively. Delays in diagnosis and treatment of cancer, which were especially pronounced during the coronavirus disease 2019 (COVID-19) pandemic, may lead to increased mortality and advanced-stage disease.Summary

[0008] According to various embodiments, a machine learning method of estimating a patient prognosis based on a patient radiotracer image is presented. The method includes: obtaining patient imaging data including a patient radiotracer image; providing the patient imaging data as an input to a first trained machine learning system, where the first trained machine learning system has been trained using a plurality of training images, where the plurality of training images include at least oneof: a plurality of images captured using a plurality of radiotracer types, a plurality of images of a plurality of cancer types, or a plurality of images captured at a plurality of institutions; obtaining a first output from the first trained machine learning system, where the first output includes at least one of a patient tumor segmentation or a patient tumor quantification; generating a plurality of patient feature quantifications from the first output; providing the plurality of patient feature quantifications as an input to a second trained machine learning system, where the second trained machine learning system has been trained using a plurality of training feature quantifications and a plurality of training prognostic data; obtaining a second output from the second trained machine learning system, where the second output includes patient prognostic data; and providing a patient prognosis based on the patient prognostic data.

[0009] According to various embodiments, a machine learning system for estimating a patient prognosis based on a patient radiotracer image is presented. The system includes: a non-transitory computer readable medium including instructions; and at least one electronic processor that executes the instructions to perform actions including: obtaining patient imaging data including a patient radiotracer image; providing the patient imaging data as an input to a first trained machine learning system, where the first trained machine learning system has been trained using a plurality of training images, where the plurality of training images include at least one of: a plurality of images captured using a plurality of radiotracer types, a plurality of images of a plurality of cancer types, or a plurality of images captured at a plurality of institutions; obtaining a first output from the first trained machine learning system, where the first output includes at least one of a patient tumor segmentation or a patient tumor quantification; generating a plurality of patient feature quantifications from the first output; providing the plurality of patient feature quantifications as an input to asecond trained machine learning system, where the second trained machine learning system has been trained using a plurality of training feature quantifications and a plurality of training prognostic data; obtaining a second output from the second trained machine learning system, where the second output includes patient prognostic data; and providing a patient prognosis based on the patient prognostic data.

[0010] Various optional features of the above system and method embodiments include the following. The plurality of training images may include images generated by: generating, by a third trained machine learning system, a plurality of training anatomical images depicting normal anatomy; generating, by a fourth trained machine learning system, a plurality of training radiotracer images depicting radiotracer lesion activity; and generating a plurality of combination images, where the plurality of training images includes the plurality of combination images, where a respective combination image includes an image of the plurality of training anatomical images and an image of the plurality of training radiotracer images. The generating the plurality of combination images may include: transforming an image of the plurality of training anatomical images into a projection space, where a projected training anatomical image is produced; transforming an image of the plurality of training radiotracer images into the projection space, where a projected training radiotracer image is produced; combining the projected training anatomical image with the projected training radiotracer image in the projection space, where a projected combination image is produced; and transforming the projected combination image into a 3D image space, where a combination image of the plurality of combination images is produced. The method (or actions) may include: simulating, in the projection space, at least one of: detector blurring, noise, scatter, scatter correction, randoms, randoms correction, attenuation, attenuation map, or attenuation correction. Thepatient imaging data may include both a positron emission tomography (PET) image and a computer tomography (CT) image. The plurality of training images may include a plurality of images of a plurality of cancer types, and the plurality of cancer types may include two or more of: breast cancer, prostate cancer, head and neck cancer, lung cancer, melanoma, or lymphoma. The plurality of training images may include a plurality of images captured using a plurality of radiotracer types, and the plurality of radiotracer types may include: fluorodeoxyglucose (FDG) and a prostate-specific membrane antigen (PSMA)-targeted radiotracer. The plurality of patient feature quantifications may include a plurality of radiomics. The second trained machine learning system may include at least one of: a logistic regression model, a multilayer perceptron, a decision tree, a random forest, a linear discriminant analysis, a generalized additive model, a k-nearest neighbors algorithm, or a support vector machine. The patient prognosis may include at least one of: a patient risk stratification, a patient survival analysis, or a patient treatment response prediction. The generating the plurality of patient feature quantifications from the first output may include providing the first output to a trained feature quantification system, where the trained feature quantification system outputs the plurality of feature quantifications.

[0011] Combinations, (including multiple dependent combinations) of the above-described elements and those within the specification have been contemplated by the inventors and may be made, except where otherwise indicated or where contradictory.Brief Description of the Drawings

[0012] Various features of the examples can be more fully appreciated, as the same become better understood with reference to the following detailed description of the examples when considered in connection with the accompanying figures, in which:

[0013] Fig. 1 is a schematic diagram of a process obtaining, preprocessing, and storing annotated clinical 3D image data according to various embodiments;

[0014] Fig. 2 is a schematic diagram of a generative adversarial training (GAN) deep learning architecture, according to various embodiments;

[0015] Fig. 3 is a schematic diagram of a CycleGAN deep learning architecture, according to various embodiments;

[0016] Fig. 4 is a schematic diagram of a deep learning architecture for generating synthetic PET images by combining, in a projection space, anatomical images, radiotracer images of lesions, and image artifacts, according to various embodiments;

[0017] Fig. 5 is a flow chart for a method of generating synthetic PET images by combining, in a projection space, anatomical images, radiotracer images of lesions, and image artifacts, according to various embodiments; and

[0018] Fig. 6 is a flow chart for a method of estimating a patient prognosis based on a patient radiotracer image according to various embodiments.Description of the Examples

[0019] Reference will now be made in detail to example implementations, illustrated in the accompanying drawings. Wherever convenient, the same reference numbers will be used throughout the drawings to refer to the same or like parts. In the following description, reference is made to the accompanying drawings that form apart thereof, and in which is shown by way of illustration specific exemplary examples in which the invention may be practiced. These examples are described in sufficient detail to enable those skilled in the art to practice the invention and it is to be understood that other examples may be utilized and that changes may be made without departing from the scope of the invention. The following description is, therefore, merely exemplary.

[0020] Developing generalizable Al and DL approaches for the automatic characterization of malignant disease can facilitate the early detection, diagnosis, prognosis, and treatment of cancer. This is an important clinical need that may help bridge the gap in reducing delays in cancer diagnosis and treatment and could significantly impact the public health of millions of individuals globally. Nuclear medicine imaging modalities such as PET offer rich functional information about in vivo biological, biochemical, and pathophysiological processes that can provide important insights into the distribution of radiotracer uptake in benign and malignant pathologies to help inform diagnosis and treatment, but it is underutilized in the context of Al and DL. There is a significant unmet need to develop clinical tools that leverage Al and DL methods for nuclear medicine imaging and therapy.

[0021] Some embodiments provide large-scale Al using radiomics, DL, and physics-based generative modeling in nuclear medicine to improve the clinical outcomes of patients with cancer while solving existing problems with the prior art.

[0022] First, some embodiments use clinical PET / CT 3D image training data with physician-annotated ground truth to train large-scale Al-based systems. Training data may be obtained from multiple sources, e.g., different medical institutions. Unlike most radiomics and DL studies in PET, which focus on moderate to small datasets consisting of a single radiotracer targeting a specific malignancy, the training data ofsome embodiments may include 3D images representing a wide range of cancers and radiotracers. Further, according to some embodiments, the PET and CT image volumes may be co-registered at high resolution and resampled to have isotropic voxel spacing. Physician-annotated segmentations and clinical data, such as age, sex, and outcomes, may be included in the training data. The use of such diverse training data solves an existing problem with the prior art, in which Al systems trained with data from a single source, representing a single radiotracer, and for a single cancer type produce inaccurate results when analyzing patient images from different sources, with different radiotracers, and reflecting different cancer types.

[0023] Second, some embodiments utilize highly-realistic artificially generated training data. For example, some embodiments generate a large, simulated dataset of PET / CT images that reflect the diverse population-level biological variations in anatomy and radiotracer uptake using deep generative modeling, while also modeling PET imaging physics in the simulation process. Most existing simulation approaches for medical images use digital anthropomorphic phantoms, such as the Zubal orXCAT phantoms, to model human anatomy. However, those phantoms are based on only two human subjects and therefore do not fully capture the diverse range of anatomic variability present in the patient and normal populations. Moreover, existing approaches that use deep generative modeling techniques to generate synthetic medical images do not incorporate important aspects of imaging physics, and therefore do not produce realistic training data. By contrast, some embodiments utilize a simulation-based approach, e.g., using a generative adversarial network (GAN), that generates a population of digital phantoms that not only reflect realistic biological and anatomical variations, but also incorporate PET imaging physics and physical variations in the imaging reconstruction process. According to some embodiments,the artificially generated training data includes realistic population-level biological variations in anatomy and radiotracer uptake of both normal controls and patients with cancer. Further, some embodiments incorporate the PET imaging physics and physical variations in imaging reconstruction protocols in the simulation process, e.g., in the projection space. According to various embodiments, the artificially generated training data of some embodiments, which is generated by combining a projected training anatomical image, a projected training radiotracer image, and modeled artifacts, in a projection space, solves the prior art problem of lack of sufficient annotated PET images, and the incorporation of artifacts in the projection space solves the prior art problem of inaccurate and unrealistic artificial 3D training images.

[0024] Third, some embodiments utilize IBSI-standardized radiomic features extracted from the training data. Some embodiments utilize radiomics features for which robustness and reproducibility have been systemically quantified and evaluated based on physician-defined ground truth ROIs (for clinical training data) and known ground truth ROIs (for simulated training data). Some embodiment delineate regions of interest (ROIs) using semi-automated and automated DL-based segmentation approaches based on the radiomics features. The use of standardized radiomic features solves the prior art problem of interpretability of machine learning systems trained exclusively with machine-defined features.

[0025] Fourth, some embodiments include a generalized Al framework that incorporates radiomics analysis, DL, and meta-learning ensemble approaches to automated detection, characterization, and prognostication tasks. According to some embodiments, the Al framework is trained using both simulated and clinical PET / CT datasets with different radiotracers reflecting both normal controls and patients with a wide range of cancer types. According to some embodiments, the Al framework istrained with the simulated training PET / CT images by applying transfer learning techniques to generalize DL systems that have already been trained on a small number of clinical images. According to some embodiments, the trained Al framework utilizes both the visual features extracted from the DL approach developed using transfer learning and the extracted IBSI-standardized radiomic features in parallel to build baseline prognostic ML models, such as logistic regression, multilayer perceptron, decision trees, random forest, and support vector machine. According to some embodiments, the trained Al framework may be used for automated tumor detection, segmentation, and classification tasks across several clinical PET / CT datasets of cancer patient populations imaged with different tracers. According to some embodiments, the trained Al framework may be used to predict clinical outcomes for patients, including overall survival, progression-free survival, and treatment response, of patients with cancer. Some embodiments combine one or more baseline models a meta-learning ensemble approach to perform prognosis of clinical outcomes.

[0026] These and other features and advantages are shown and described herein in reference to the accompanying figures.

[0027] Fig. 1 is a schematic diagram of a process 100 obtaining, preprocessing 104, and storing annotated clinical 3D image data according to various embodiments. The process 100 aims to create a large database 106 of clinical PET / CT images with physician-annotated ground truth. In general, Al systems require large amounts of annotated data to achieve acceptable levels of performance as well as generalizability. Thus, some embodiments utilize a database of clinical PET / CT images of patients with cancer. Retrospective, deidentified data may be extracted from openly available andinstitutional sources 102 (15). Clinical data, such as age, sex, clinical outcomes, and physician-defined manual annotations and segmentations may be included.

[0028] An example implementation included 32 datasets from a variety of sources that have a combined 10,392 studies of 5,289 subjects representing a wide range of cancers, including bladder, brain, breast, endometrial, head and neck, kidney, liver, lung, lymphoma, melanoma, neuroendocrine, prostate, soft-tissue sarcoma, and thyroid cancer. Collectively across all 32 datasets, there were 3,703 subjects with a PET, 5,059 with a CT, and 416 with an MRI. Datasets characteristics are summarized in Table 1 , where each row represents a separate data source.Tab e 1 . Characteristics of the origination data sets

[0029] For the example implementation, PET images from 29 TCIA datasets were imaged with radiotracers including, [18F]FDG, [18F]FLT, [18F]FMISO, [18F]NaF, and68Ga aerosol. Of the 1 ,014 whole-body [18F]FDG PET / CT studies available in Dataset 9, there were 513 PET / CT scans of negative controls. Three PET / CT retrospective, deidentified datasets were collected at two different insttitutions. Dataset 30 included 267 [18F]DCFPyL PSMA PET / CT scans of patients with prostate cancer, where lesions were manually segmented by nuclear medicine physicians and assigned to one of the 9 PSMA-RADS categories. Dataset 31 included 140 [18F]DCFPyL PSMA scans of patients with prostate cancer as well as clinical data, including Gleason scores and prostate-specific antigen (PSA) levels. Dataset 32 included [68Ga]DOTATATE PET / CT scans of 93 patients (53 men and 40 women) with gastroenteropancreatic neuroendocrine tumors who underwent LUTATHERA- DOTATATE therapy, where 16 out of 93 patients were non-responders to therapy.

[0030] Image preprocessing 304 may include any, or a combination, of the following actions, by way of non-limiting example. PET and CT image volumes from the various data sources may be co-registered at the higher resolution of the CT image and resampled to 1 mm3isotropic voxel size. Manually delineated ROIs may be similarly processed. PET images may be converted to have standardized uptake value (SUV) units. CT images may be windowed with a range of -1024 to 3071 Hounsfield units (HU) to cover the intensity ranges for lung, soft tissue, and bone (29).

[0031] Regarding clinical data, patient demographics and available clinical outcomes data, including overall survival, progression-free survival, grade, stage, recurrence, pathological biomarkers, and treatment response, may be extracted from the various data sources and included. Additionally, clinical, genetic, and pathological data may be extracted, e.g., from The Cancer Genome Atlas (TCGA) for corresponding datasets. Clinical data may be processed and normalized across all datasets.

[0032] In the example implementation, segmentations or radiotherapy contours were available for 2,442 out of 4,789 subjects (~51 %) from the 29 TCIA datasets. In general, data from subjects without segmentations may be used to train generative modeling approaches in an unsupervised fashion, as shown and described herein in reference to Figs. 2-4. Data with segmentations may be used to train and evaluate task-based radiomics and DL approaches. Additionally, for subjects that do not have a PET / CT image pair, an MRI may be used, if possible, when a corresponding CT image is not available.

[0033] Regarding data partitioning for training an Al framework, data from 20% of the patients from all datasets may be randomly partitioned into an external test set, and the remaining 80% will be used for training and k-fold cross-validation during model development, by way of non-limiting example.

[0034] In general, large amounts of data with physician-defined annotations may be used to train medical Al and DL-based systems. Because physician-defined manual annotations are difficult to obtain, some embodiments utilize physics-based deep generative modeling to produce a large dataset of simulated PET / CT images with known ground truth. The simulated 3D images reflect the realistic populationlevel biological variations in anatomy and radiotracer uptake. The physics of PETimaging and / or physical variations in imaging reconstruction protocols may be incorporated into the simulation process according to various embodiments.

[0035] A summary overview of a non-limiting example physics-guided simulated image generation process follows. Some embodiments include one or more deep convolutional generative adversarial networks (GANs) to generate synthetic activity distributions and CT attenuation maps for realistic simulations. A first GAN (an architecture of which is shown and described in reference to Fig. 2) may be trained on negative (i.e. , cancer-free) PET control images to generate synthetic PET images that represent activity distributions of normal patient backgrounds. A second GAN (with architecture as shown and described in reference to Fig. 2) may be trained on physician-defined tumor segmentations from the clinical PET / CT data to generate synthetic lesions that represent activity distributions of malignant tumors. The synthetic lesions may serve as the known ground truth for tumor delineations on the simulated background images. A CycleGAN (an architecture of which is shown and described in reference to Fig. 3) may be trained with a cycle consistency loss to generate the corresponding CT image from the input synthetic activity distribution, and the generated CT may be converted into an attenuation map for CT-based attenuation correction. Finally, a complex deep learning architecture (as shown and described in reference to Fig. 4) generates synthetic combined PET images by combining, in a projection space, anatomical images from the first GAN (e.g., as corrected using a CT image from the CycleGAN), radiotracer images of lesions from the second GAN (e.g., as corrected using a CT image from the CycleGAN), and image artifacts. The synthetic combined PET images may then be used to train various Al systems for tumor detection, tumor segmentation, tumor classification, and / or clinical outcomes for patients, as disclosed herein.

[0036] Details of the non-limiting example physics-guided image simulation process summarized above are shown and described presently in reference to Figs. 2-4.

[0037] Fig. 2 is a schematic diagram of a GAN deep learning architecture 200, according to various embodiments. As shown in Fig. 2, the GAN architecture 200 includes a generator DL network and a discriminator deep learning network. During a training phase, the generator DL network interacts with the discriminator DL network in an adversarial manner, with the generator DL network using randomness to generate images, and the discriminator DL network guessing whether a given image is real or generated by the generator DL network. Through iteratively training in this manner, the generator DL network is trained to produce artificial images that are practically indistinguishable from real images. According to various embodiments, a first GAN may be used to generate synthetic normal patient PET images, and a second GAN may be used to generate synthetic radiotracer images depicting radiotracer lesion activity.

[0038] Fig. 3 is a schematic diagram of a CycleGAN deep learning architecture 300, according to various embodiments. In general, the CycleGAN architecture 300 solves the problem of translating synthetic (or clinical) PET images into corresponding synthetic CT images. Formally, the CycleGAN learns to map from a domain PET of PET images to a domain CT of CT images in an unsupervised manner. To do so, the CycleGAN learns a mapping Gi : PET — CT such that the distribution of images from G- (PET) is indistinguishable from the distribution CT. The training is based on the general GAN architecture (see Fig. 2 and description) in that it employs an adversarial process. However, for the CycleGAN, two discriminators are used. First, a discriminator Di attempts to distinguish between real CT images and synthetic CTimages generated from PET images by the translation Gi. Second, a discriminator D2 attempts to distinguish between real PET images and synthetic PET images generated from CT images by a translation G2 : CT — PET, where G2 is intended to be an inverse of G1. The CycleGAN is iteratively trained in an adversarial manner using a loss function, with the goal that G2(GI (PET)) « CT and GI (G2(CT)) « PET. According to various embodiments, a CycleGAN may be used to transform a synthetic normal patient PET image into a synthetic CT image, which may be converted into an attenuation map that is used to attenuation-correct the synthetic normal patient PET image. According to various embodiments, a CycleGAN may be used to transform a synthetic radiotracer image depicting radiotracer lesion activity into a synthetic CT image, which may be converted into an attenuation map that is used to attenuation- correct the synthetic radiotracer image.

[0039] Fig. 4 is a schematic diagram of a deep learning architecture 400 for generating synthetic PET images by combining, in a projection space, normal anatomical PET images, PET radiotracer images of lesions, and modeled PET image artifacts, according to various embodiments. The normal anatomical PET images may be obtained from the first GAN, as shown and described herein in reference to Fig. 2, and may be attenuation-corrected using a synthetic CT image generated as shown and described herein in reference to Fig. 3. The PET radiotracer images of lesions may be obtained from the second GAN, as shown and described herein in reference to Fig. 2, and may be attenuation-corrected using a synthetic CT image generated as shown and described herein in reference to Fig. 3. The synthetic anatomical backgrounds and lesions may be forward projected by a system matrix that simulates a PET system, incorporating various PET image artifacts.

[0040] Some embodiments model various artifacts in projection space and incorporate such artifacts in the combination image construction process. Examples of such artifacts include image-degrading processes such as detector blur, which may be modeled by full-width at half-maximum (FWHM) Gaussian blur and noise. More generally, examples of such artifacts may include any, or any combination, of: detector blurring, noise, scatter, scatter correction, randoms, randoms correction, attenuation, attenuation map, and / or attenuation correction. Scatter may include a coincidence event in PET where one of the detected photons has undergone Compton scattering prior to detection. Randoms may include random coincidence events when two photons not arising from the same annihilation event are incident (detected) on the detectors. Attenuation may include the loss of detection of true coincidence events due to their absorption in the body or due to scattering. Attenuation map may include the tissue densities containing information about the distribution of linear attenuation coefficients throughout the body and may be derived from x-rays of the simulated CT image or a clinical CT image. Attenuation correction may include that the attenuation map is used to correct for the absorption of emitted photons to correct for image artifacts due to attenuation.

[0041] Note that the simulated background images and lesions may be summed, and / or one or more of the artifacts may be introduced, all within the tomography projection space, e.g., the space corresponding to the raw data captured by the PET scan. In general, the projection space corresponds to raw images captured by a rotating sensor gantry during the scanning process, and appears as flat ribbon images that are not readily amenable to human interpretation. Through the computed tomography process, the projection space data is converted into 3D volumes, which are suitable for viewing and human interpretation. Introducing artifacts in theprojection space generates highly realistic simulated PET images, solving the prior art problem of lack of sufficient PET training images for machine learning, where prior art simulated images lacked realism and therefore, when used to train a machine learning system, resulted in a trained machine learning system that performed inadequately when processing actual patient images.

[0042] According to some embodiments, the combined image is reconstructed from the projection space image data to incorporate the effect on tumor appearance and noise texture due to the iterative reconstruction process. The simulated images may be reconstructed via an ordered subsets expectation maximization (OSEM) algorithm with a varying number of subsets and iterations to capture the variability in noise texture due to differences in reconstruction protocols. The result of the reconstruction is a highly realistic simulated 3D volume image in everyday 3D space.

[0043] Various embodiments may utilize the following example 3D PET simulation and reconstruction software. PET simulations may be performed in 3D to capture the full anatomic variability in the imaged volume. A fast analytic simulator for PET scanners provided by the ASIM software package may be used to perform 3D PET simulations while also modeling noise, scatter, randoms, and attenuation correction. The STIR (Software for Tomographic Image Reconstruction) software package may be used to perform iterative reconstruction of the simulated images.

[0044] To ensure the quality of the simulated images, they be assessed with evaluation metrics, such as the Frechet Inception Distance (FID) and multi-scale structural similarity index measure (MS-SSIM). The FID may measure the distance between the distribution of simulated and clinical images, where lower values indicate higher quality images. The MS-SSIM may evaluate the structural similarity betweenpairs of generated and clinical images. The MS-SSIM may range from 0 to 1 , where a higher value indicates higher structural similarity.

[0045] Note that Figs. 1 -4 are described in reference to an example non-limiting embodiment. Other implementations are possible and within the scope of this disclosure. For example, although Fig. 2 describes deep convolutional GANs trained to generate 3D activity distributions and attenuation maps, other embodiments may instead train progressively growing (PG)-GANs to generate phantoms with progressively increasing levels of resolution. Some embodiments may employ patchwise training to combat memory limitations. Further, some embodiments may train Transformer-based GANs, which can directly model long-range dependencies in the data. Yet further, by using the ASIM analytic simulator (by way of non-limiting example) to perform fast PET simulations, some embodiments may use the GATE open-source software to perform advanced Monte Carlo-based PET simulations to more accurately model the imaging physics. Yet further, some embodiments may use a CycleGAN to generate pairs of patient background and synthetic lesions that reflect a highly realistic lesion distribution that reflects true clinical and biological variability. Yet further, while some embodiments may train a GAN on corresponding MRI images to generate attenuation maps when a CT is not available, other embodiments may use alternative deep generative modeling approaches, such as variational autoencoders (VAEs) or diffusion models. Yet further, some embodiments may train conditional GANs to generate samples conditioned on biological sex to ensure both male and female anatomies are evenly represented in the population of synthetic phantoms.

[0046] As previously indicated, some embodiments incorporate standardized radiomics in training data. However, many sources of variability may hinder the reproducibility of radiomic features, including variations in scanners, image analysismethods, and software implementations. Sources of inter-operator and intra-operator variability when performing manual or semi-automated segmentation also contribute to reproducibility issues in radiomic analysis. Therefore, some embodiments may restrict to the usage of radiomics features that are robustly reproducible. To that end, some embodiments may utilize a quantification of the robustness and reproducibility of radiomic features on simulated PET / CT data using the known ground truth and clinical PET / CT data using physician-defined ROIs. To the extent that robustness data may not be readily available, a brief description of obtaining such a quantification follows.

[0047] The Image Biomarker Standardization Initiative (IBSI) standardized 169 radiomic features, including morphology, intensity, and textural features, with excellent reproducibility in CT, PET, and MRI. Intensity features included local intensity, intensity-based statistics, intensity histogram, and intensity-volume histogram. Textural features included a grey-level co-occurrence matrix, grey-level distance-zone matrix, grey-level run-length matrix, grey-level size-zone matrix, neighboring greylevel dependence matrix, and neighborhood grey tone difference matrix. The 169 standardized features may be extracted from the simulated PET / CT images with known ground truth and clinical PET / CT images with physician-defined segmentations. Radiomic features may be extracted from the ROIs and computed following IBSI guidelines.

[0048] Radiomic features may also be extracted from ROIs determined by semiautomated and automated DL-based segmentation methods on both a 2D and 3D basis. Semi-automated segmentation methods will include 30%, 40%, and 50% of SUVmax thresholding methods, active contours, and clustering-based methods. DL- based segmentation methods will include those based on convolutional neuralnetworks (CNNs) and transformers, such as ll-net, nnll-net, Swin-UNET, and nnFormer.

[0049] The reproducibility of the radiomic features may be evaluated across simulated images generated with different reconstruction parameters (e.g., the number of subsets and iterations used in OSEM reconstruction) and across simulated images with resampled voxel sizes of 1 mm3, 2 mm3, and 4 mm3. Features may be extracted from ROIs delineated with different semi-automated and automated DL- based segmentation methods. The accuracy of those extracted features may be evaluated and compared to features extracted from the simulated lesion ground truth activity distribution.

[0050] The reproducibility of each radiomic feature may thus be quantified, e.g., by using Pearson’s correlations, coefficient of variation (COV), and / or Bland-Altman analysis of each radiomic feature across all image reconstruction settings, voxel samplings, and / or ROI segmentation methods. By way of non-limiting example, a COV < 5% may be considered robust, and COV > 20% may be considered non-robust. The Medical Image Radiomics Processor, an IBSI-compliant radiomics processor with high reproducibility, may be used to extract radiomic features.

[0051] Note that because the ground truth for clinical images may not be known, physician-defined ROIs may be used as a surrogate ground truth, and the robustness of radiomic features may be evaluated on the clinical images across various semiautomated and automated DL-based segmentation techniques. Features that are evaluated as robust on simulated images may be externally validated on the test set of clinical images.

[0052] A description of using transfer learning to train a machine learning system, such as is shown and described herein in reference to Figs. 1 -4, follows.Simulated data with known ground truth may be used to augment training for small to moderately-sized clinical datasets with limited physician-defined annotations. The simulation-based training approach may be validated on automated image analysis tasks across multiple PET / CT datasets of patients with cancer, as follows.

[0053] Transfer learning may be employed using a large dataset of simulated images to train DL approaches to automatically detect, segment, and classify tumors. According to some embodiments, DL approaches may be pre-trained on simulated images and fine-tuned on clinical images.

[0054] According to some embodiments, for task-based training, multiple DL systems may be trained for each automated lesion characterization task. For the lesion detection task, for example, DL approaches may include single-stage object detectors, such as SSD, YOLO, and Faster R-CNN. Multi-stage detectors that perform both the lesion detection and classification tasks may be trained, including Mask R- CNN, Cascade Mask R-CNN, and nnDetection. According to some embodiments, DL approaches for voxel-wise segmentation, including U-net, nnU-net, Swin-UNET, and nnFormer, may also be trained.

[0055] The lesion detection task may be assessed by the mean average precision (mAP) evaluated at several bounding box intersection-over-union (loU) thresholds and by precision-recall analysis. The segmentation task may be evaluated on the basis of overlap and distance metrics, including the Dice similarity coefficient (DSC), loU, and Hausdorff distance. The classification task may be evaluated on the basis of overall accuracy, precision, recall, F1 score, and receiver operator characteristic (ROC) analysis.

[0056] The simulation-based transfer learning approach may be evaluated with a per-dataset 5-fold cross-validation, by way of non-limiting example. The clinicalPET / CT datasets may be randomly partitioned on a per-dataset basis into five folds, where four folds are used for simulation-based training, and the hold-out fold used for validation.

[0057] Significant differences may be determined using a two-tailed t-test with a P < 0.05. ROC curve 95% confidence and tolerance intervals may be computed with 1 ,000 bootstrap samples, by way of non-limiting example.

[0058] As described in detail presently, some embodiments may include a multipronged radiomics, DL, and ensemble approach to predict clinical outcomes. The robust radiomic features and the visual features extracted by the trained DL systems may be used to build baseline prognostic models to predict clinical outcomes. The baseline models may be trained in parallel and then combined in an ensemble using a meta-learning approach to perform the prognostic task.

[0059] For feature extraction and dataset partitioning, radiomic features (e.g., as evaluated as robust) may be extracted from ROIs delineated by the automated DL- based segmentation methods on the clinical PET / CT training datasets. Learned visual image features may also be extracted from intermediate layers of deep neural networks of the trained DL approaches for the automated lesion characterization tasks. Clinical data, such as age and sex, and clinical outcomes, including overall survival, progression-free survival, and response to therapy, may be extracted.

[0060] Baseline prognostic models (e.g., logistic regression, multilayer perceptron, decision trees, random forest, support vector machines, etc.) may be trained using the radiomic features and / or the DL-learned features as inputs to predict clinical outcomes. The baseline prognostic models may be trained via a 5-fold cross- validation on the training set, where model predictions on all hold-out folds may be obtained.

[0061] An ensemble meta-learning approach may be used based on stacked generalization, for example. Meta-learner models, including logistic regression, multilayer perceptron, decision trees, random forest, and support vector machines, may be trained on the predictions of the baseline models to predict the clinical outcomes via a 5-fold cross-validation on the training set. The meta-learner with the best performance on the cross-validation as evaluated on the external test set may be for the clinical outcome prediction task.

[0062] Fig. 5 is a flow chart for a method 500 of generating synthetic PET images by combining, in a projection space, anatomical images, radiotracer images of lesions, and image artifacts, according to various embodiments. The synthetic PET images may be generated using a system such as is shown and described herein in reference to Figs. 1 -4, for example. The method 500 is described as generating one image; however, the method 500 may be repeated sequentially or in parallel to generate many images. The images generated by the method 500 may be used to train a machine learning system as shown and described herein.

[0063] At 502, the method 500 includes generating a training radiotracer image. The training radiotracer image may be generated as shown and described herein in reference to Figs. 2, 3, and 4, for example. The training radiotracer image may depict a portion of a simulated PET scan, for example. The training radiotracer image may depict radiotracer lesion activity, for example.

[0064] At 504, the method 500 includes generating a training anatomical image. The training anatomical image may be generated as shown and described herein in reference to Figs. 1 -4, for example. The training anatomical image may depict all or a portion of a simulated PET scan, for example. The training anatomical image may depict a normal anatomy, e.g., without any lesions, for example.

[0065] At 506, the method 500 includes transforming the training radiotracer image into projection space. The transformation may be performed as shown and described herein in reference to Figs. 1 -4, for example. The projection space may correspond to raw images captured by a sensor in a medical imaging system, such as a PET scanner, e.g., the projection space may represent a domain for raw radiological imaging data prior to having been converted into human-readable 3D volumes using computed tomography.

[0066] At 508, the method 500 includes transforming the training anatomical image into projection space. The transformation may be performed as shown and described herein in reference to Figs. 1 -4, for example. The projection space may correspond to raw images captured by a sensor in a medical imaging system, such as a PET scanner, e.g., the projection space may represent a domain for raw radiological imaging data prior to having been converted into human-readable 3D volumes using computed tomography.

[0067] At 510, the method 500 includes simulating at least one artifact in projection space. The simulating may be performed as shown and described herein in reference to Figs. 1 -4, for example. Any of a variety of artifacts may be simulated according to various embodiments, including any, or a combination, of: detector blurring, noise, scatter, scatter correction, randoms, randoms correction, attenuation, attenuation map, and / or attenuation correction.

[0068] At 512, the method 500 includes combining the projected anatomical image, the projected radiotracer image, and the at least one artifact in projection space, into a projected combination image. The combination may be performed as shown and described herein in reference to Figs. 1 -4, for example. For example, the combination may including summing the images in the projection space.

[0069] At 514, the method 500 includes transforming the projected combination image into image space. The transforming may be performed as shown and described herein in reference to Figs. 1 -4, for example. For example, the transforming may be performed using computed tomography. The image space may include regular 3D volume space, in which a readily human-interpretable image may be presented. The transformed projected combination image may be used to train a machine learning system, e.g., for estimating patient prognosis, as shown and described herein.

[0070] Fig. 6 is a flow chart for a method 600 of estimating a patient prognosis based on a patient radiotracer image according to various embodiments. The method 600 may be implemented using a machine learning system trained using synthetic PET images, such as are shown and described herein in reference to Figs. 1 -5, and as is described further below in reference to an example implementation used in a study conducted by the inventors.

[0071] At 602, the method 600 includes obtaining patient imaging data. The patient imaging data may include a PET scan, and may also include one or more of a CT scan, and / or an MRI scan.

[0072] At 604, the method 600 includes providing patient imaging data to a first trained machine learning system. The first trained machine learning system may include any of a variety of machine learning systems, e.g., as described in reference to the study below and as shown and described in reference to Figs. 1 -4, for example. For example, the first trained machine learning system may include one or more of: a single-stage object detector (e.g., SSD, YOLO, Faster R-CNN), a multi-stage detector (e.g., Mask R-CNN, Cascade Mask R-CNN, nnDetection), and / or a DL voxel-wise segmenter (e.g., Il-net, nnll-net, Swin-UNET, nnFormer). The first trained machinelearning system may be implemented in a computer executing non-transitory instructions to perform the actions described herein.

[0073] At 606, the method 600 includes obtaining a first output from the first trained machine learning system. The first output may be as shown and described in reference to Figs. 1 -4 and as described in reference to the study below, and may include any, or a combination, of tumor quantification (e.g., DL image features) and / or tumor segmentation. The first output may be obtained from a terminal layer or portion of the first trained machine learning system, or from an intermediate layer or layers thereof.

[0074] At 608, the method 600 includes generating patient feature quantifications. The first output may be as shown and described in reference to Figs. 1 -4 and as described in reference to the study below, and may include radiomics, by way of non-limiting example. The patient feature quantifications may be generated by providing the first output to a trained feature quantification model, where the trained feature quantification model outputs the plurality of feature quantifications. The patient feature quantifications may include patient clinical data in combination with the first output and / or other data such as patient radiomics.

[0075] At 610, the method 600 includes providing patient feature quantifications to a second trained machine learning system. The second trained machine learning system may include any of a variety of machine learning systems, e.g., as described in reference to the study below and as trained as shown and described in reference to Figs. 1 -4, for example. For example, the second trained machine learning system may include any, or a combination, of a logistic regression model, a multilayer perceptron, a decision tree, a random forest, a linear discriminant analysis, a generalized additive model, a k-nearest neighbors algorithm, or a support vector machine. The secondtrained machine learning system may be implemented in a computer executing non- transitory instructions to perform the actions described herein.

[0076] At 612, the method 600 includes obtaining a second output. The second output may include an indication of a patient prognosis, such as a patient risk stratification, a patient survival analysis, or a patient treatment response prediction.

[0077] At 614, the method 600 includes providing a patient prognosis based on the second output. The patient prognosis may include any, or a combination, of a patient risk stratification, a patient survival analysis, or a patient treatment response prediction. The patient prognosis may be provided by display on a computer monitor, by transmission over a network, or by storage in a medical records system, by way of non-limiting examples.

[0078] The inventors performed a study that implemented deep semisupervised transfer learning systems for fully automated whole-body tumor segmentation and prognosis on positron emission tomography (PET) / computed tomography (CT) images. A description of the study, the implementations, and their evaluations follows.

[0079] Briefly, the retrospective study included of 1 ,019 fluorodeoxyglucose (FDG) and prostate-specific membrane antigen (PSMA) PET / CT scans of patients with prostate, lung, melanoma, lymphoma, head and neck, and breast cancers. The semi-supervised transfer learning approach with a nn-Unet backbone learned the segmentation task on FDG and PSMA PET / CT using limited annotations and radiomics analysis. True positive rate (TPR) and Dice similarity coefficient (DSC) were assessed. Imaging measures were extracted from predicted segmentations for prognostic model development. A risk model for prostate cancer was developed and was evaluated on follow-up prostate-specific antigen (PSA) levels. A risk score forhead and neck cancer was developed and evaluated by Cox regression analysis. A prognostic model predicted pathological complete response (pCR) for breast cancer patients undergoing neoadjuvant chemotherapy using pre- and post-therapy imaging measures. Overall accuracy and area under the receiver operating characteristic (AUROC) curve were assessed.

[0080] To summarize the results, the study yielded median TPR values of 0.75, 0.85, 0.87, and 0.75 and median DSC values of 0.81 , 0.76, 0.83, and 0.73 for patients with lung cancer, melanoma, lymphoma, and prostate cancer, respectively. The risk model for prostate cancer yielded an overall accuracy of 0.83 and an AUROC of 0.86. Patients classified as low-intermediate and high risk had mean follow-up PSA levels of 18.61 and 727.46 ng / mL, respectively (P < 0.05). The risk score for head and neck cancer was significantly associated with overall survival by univariable and multivariable Cox regression analyses (P < 0.05). Prognostic models using only pretherapy imaging measures and both pre- and post-therapy measures predicted pCR in breast cancer with accuracies of 0.72 and 0.84 and AUROC values of 0.72 and 0.76, respectively.

[0081] A detailed description of the study follows.

[0082] The study implemented a semi-supervised transfer learning framework to learn the whole-body tumor segmentation task using limited manual tumor annotations. The approach jointly optimized a nnU-net backbone, an automatically self-configuring DL framework, across source and target domains consisting of FDG and PSMA PET / CT images with complete and incomplete manual annotations, respectively. Radiomics analysis improved predictions by enriching true positive detections while removing false positives. The implementation performed domainadaptation on FDG and PSMA PET / CT while iteratively improving segmentation performance.

[0083] The study used the following data. Data from 1 ,019 cancer patients with PET / CT scans across five datasets and six cancer types were used. Five datasets were included. Datasets 1 and 2 had18F-DCFPyL PSMA PET / CT scans of prostate cancer patients. Dataset 1 had 270 patients with incomplete manual segmentations where PSMA reporting and data system (PSMA-RADS) scores of 1-5 were assigned to segmented lesions and overall scans indicating the likelihood of prostate cancer. Dataset 2 had Gleason scores, initial serum prostate-specific antigen (PSA) levels, follow-up PSA levels, PSA doubling times, and post-PSMA PET therapies, for 138 patients with no tumor annotations. Datasets 3, 4, and 5 included FDG PET / CT scans from TCIA. Dataset 3 had complete manual tumor annotations for 168, 188, and 145 patients with lung cancer, melanoma, and lymphoma, respectively. Dataset 4 had clinical information on overall staging by the American Joint Committee on Cancer (AJCC) 7th edition, surgery, chemotherapy, radiotherapy duration, and overall survival for 74 head and neck cancer patients with no tumor annotations. Dataset 5 had longitudinal scans of 36 breast cancer patients undergoing neoadjuvant chemotherapy. Pre- and post-therapy scans were acquired for 36 and 25 patients, respectively, with no tumor annotations. Pathological complete response (pCR) was defined as the absence of invasive cancer in the breast or lymph nodes at definitive surgery. Non-responders (non-pCR) had residual invasive cancer or disease progression. Datasets 1 and 3 were used to cross-validate the segmentation task via 5-fold cross-validation. Datasets 2, 4, and 5 were used for external testing and prognostic model development.

[0084] Tumor detection and segmentation were evaluated on a lesion-wise and voxel-wise basis. True positive rate (TPR), positive predictive value (PPV), Dice similarity coefficient (DSC), false discovery rate (FDR), true negative rate (TNR), and negative predictive value (NPV) were assessed. Tumor detection performance was compared to that of models trained only on FDG or PSMA PET / CT images. Imaging measures, including molecular tumor volume (MTV), total lesion activity (TLA), number of lesions, mean standardized uptake value (SUVmean), and maximum standardized uptake value (SUVmax) were quantified from the predicted segmentations.

[0085] The Standardized Environment for Radiomics Analysis based on the Image Biomarker Standardization Initiative was used for radiomics analysis. First- order statistical and higher-order textural features, including morphology, intensity, intensity histogram, intensity volume histogram, co-occurrence matrix, run length matrix, size zone matrix, distance zone matrix, and neighbor grey tone difference matrix, were extracted. Noninformative features were removed. A total of 397 radiomic features were calculated from PET / CT volumes of interest (VOIs). Radiomics classifiers using random forest detected true positive VOIs via 10-fold cross-validation. Overall accuracy and receiver operating characteristic (ROC) analysis were assessed.

[0086] A risk model for prostate cancer was developed using the extracted whole-body imaging measures. Initial PSA levels <10 ng / mL, 10-20 ng / mL, or >20 ng / mL were assigned to low, intermediate, or high risk groups, respectively. The risk model used random forest to classify low-intermediate vs high risk via 10-fold cross- validation. Risk predictions and Gleason scores were combined where patients predicted as high risk with Gleason scores >8 were considered high risk. Patients predicted as low risk with Gleason scores of 7 and patients with Gleason scores <6 were considered low risk. Other cases were intermediate risk. Overall accuracy, areaunder the ROC curve (AUROC), follow-up PSA levels, and PSA doubling times were assessed where elevated PSA levels and shorter PSA doubling times were indicators of disease progression.

[0087] A risk score for head and neck cancer incorporated imaging measures and AJCC staging. Imaging measures in the lower quartile, within the interquartile range, or the upper quartile were assigned 0, 1 , or 2 points, respectively. AJCC stages I, II, or III— IV were assigned 0, 1 , or 2 points, respectively. Points were summed to yield a risk score ranging from 0-12. Patients with a risk score of 0, 1-9, and 10-12 were considered low, intermediate, and high risk, respectively. Overall survival was estimated by the Kaplan-Meier method where groups were compared by the log-rank test. Univariable and multivariable Cox regression models were applied.

[0088] For a treatment response prediction, imaging measures were extracted and assessed for both pre- and post-therapy scans of breast cancer patients undergoing neoadjuvant chemotherapy. Decision tree classifiers predicted pCR via leave-one-out cross-validation using pre- and post-therapy imaging measures. Overall accuracy, AUROC, TPR, PPV, TNR, and NPV were assessed.

[0089] A statistical assessment was performed as follows. Normality was assessed by the Shapiro-Wilk test. Statistical significance was assessed using the Wilcoxon signed-rank test, Wilcoxon rank-sum test, and McNemar's test when comparing paired, unpaired, and binary observations, respectively. A P < 0.05 was used to infer significant differences. The Benjamini-Hochberg method was used for multiple comparisons. Spearman’s rank correlation coefficients (p) were quantified.ROC curves with 95% confidence intervals (Cl) were computed with 1 ,000 bootstrap samples. Optimal thresholds were determined by ROC analysis using Youden's index.Analyses were conducted with MATLAB and Python. The implementation used PyTorch with an NVIDIA A6000 GPU.

[0090] Results from the study are set forth presently.

[0091] For tumor segmentation, the implementation yielded median TPR values of 0.75, 0.85, 0.87, and 0.75, median PPV values of 0.92, 0.76, 0.87, and 0.76, median DSC values of 0.81 , 0.76, 0.83, and 0.73, and median FDR values of 0.08, 0.24, 0.13, and 0.24 for patients with lung cancer, melanoma, lymphoma, and prostate cancer, respectively, on voxel-wise segmentation. The implementation yielded median TNR and NPV values of 1 .00 across all patients. For tumor detection, radiomics classifiers yielded overall accuracies of 0.85, 0.81 , 0.74, and 0.93 and AUROC values of 0.87, 0.83, 0.79, and 0.87 for patients with lung cancer, melanoma, lymphoma, and prostate cancer, respectively.

[0092] The implementation yielded a TPR of 0.77 on PSMA-RADS-4 and -5 lesions with improved detection rates throughout training (P < 0.001 ). The implementation had a higher TPR of 0.76 on prostate cancer lesions compared to baseline models trained on only FDG or PSMA PET / CT images with TPR values of 0.38 and 0.63, respectively (P < 0.001 ). While a system trained on FDG and PSMA PET / CT images had a higher TPR compared to baseline models, the system trained on both sets of images had a higher FDR of 0.33 compared to the model trained only on PSMA PET / CT images which had an FDR of 0.28. The implementation maintained a high TPR and had the lowest FDR of 0.25 compared to all other models (P < 0.05).

[0093] The implementation was also evaluated for risk stratification. Specifically, a prognostic risk model stratified prostate cancer patients by low- intermediate vs high risk and yielded an overall accuracy of 0.83 and an AUROC of 0.86. Risk scores derived from the model had positive correlations with overall PSMA-RADS scores (p = 0.44, P < 0.001 ) and post-PSMA PET therapies (p = 0.37, P < 0.001 ). Risk stratifications by imaging measures, initial PSA levels, Gleason scores, risk model predictions, and the risk model predictions combined with Gleason scores were evaluated. Optimal thresholds determined by ROC analysis for MTV, TLA, lesion number, SUVmean, SUVmax, and Gleason score were 22.00 mL, 174.56 SUV mL, 10, 9.38 SUV, 38.87 SUV, and 8, respectively.

[0094] High risk patients classified by MTV, TLA, the risk model, and the risk model combined with Gleason scores had higher follow-up PSA levels compared to low-intermediate risk patients (P < 0.05). High risk patients classified by Gleason scores and the risk model with Gleason scores had shorter PSA doubling times compared to low-intermediate risk patients (P < 0.05). Low, intermediate, and low risk patients classified by the risk model combined with Gleason scores had mean followup PSA levels of 9.18, 26.92, and 727.46 ng / mL and mean PSA doubling times of 8.67, 8.18, and 4.81 months, respectively.

[0095] The implementation was also evaluated for its survival analysis. Imaging measures were extracted from FDG PET / CT scans of head and neck cancer patients. MTV (measured in milliliters), TLA, SUVmean, and SUVmax were used as continuous variables in the survival analysis. MTV, TLA, lesion number, SUVmean, SUVmax, age, AJCC stage, and the risk score were significantly associated with overall survival by univariable Cox regression analysis. Patients were stratified based on median values for all imaging measures and plotted using Kaplan-Meier estimators. For MTV, TLA, and SUVmax, patients in the upper half had a shorter median overall survival (P < 0.05).

[0096] Patients were stratified based on the risk score and plotted using Kaplan-Meier estimators. Patients with high risk had a shorter median overall survivalcompared to those with low or intermediate risk (1 .64 years vs median not reached, P < 0.001 ). Patients with intermediate risk had a shorter median overall survival compared to those with low risk (P < 0.05). Age and the risk score were independent prognosticators of overall survival by multivariable Cox regression analysis.

[0097] The implementation was also evaluated for treatment response prediction. Imaging measures were extracted from pre- and post-therapy FDG PET / CT scans of breast cancer patients undergoing neoadjuvant chemotherapy. Post-therapy measures were all lower for pCR compared to non-pCR, indicating higher post-therapy tumor burden for non-responders (P < 0.05). Optimal thresholds of pre-therapy measures determined by ROC analysis for MTV, TLA, lesion number, SUVmean, and SUVmax were 188.96 mL, 96.46 SUV mL, 2, 1 .85 SUV, and 7.84 SUV, respectively. Classifiers trained to predict pCR using only pre-therapy measures (Decision Tree 1 ) and both pre- and post-therapy measures (Decision Tree 2) had overall accuracies of 0.72 and 0.84 and AUROC values of 0.72 and 0.76, respectively.

[0098] Thus, the study implemented a semi-supervised transfer learning approach that performed fully automated whole-body tumor detection and segmentation on FDG and PSMA PET / CT of 1 ,019 patients with six different cancer types using limited annotations. The results demonstrated accurate quantification of molecular tumor burden and prognostic prediction of risk, survivability, and treatment response.

[0099] The implementation of the study performed domain adaptation where source domain FDG PET / CT images of patients with lung cancer, melanoma, and lymphoma with complete manual annotations were used for training and cross- validation. Transfer learning was applied to target domain PSMA PET / CT images of prostate cancer patients with incomplete annotations. Despite limited annotations, theapproach yielded accurate tumor segmentation on PSMA PET / CT and achieved a high detection rate of PSMA-RADS-4 and -5 lesions, for which prostate cancer was highly likely, while limiting false discoveries by leveraging radiomics analysis. The approach generalized across FDG and PSMA-targeted radiotracers and performed well on external test datasets with out-of-distribution cancers, including head and neck and breast cancers.

[0100] A risk model for prostate cancer used automatically extracted PSMA PET / CT imaging measures and achieved an accuracy of 0.83 in classifying low- intermediate vs high risk. The prognostic model predicted higher risk scores for patients who received higher PSMA-RADS scores and systemic therapies, corroborating the risk model against two separate indications. Compared to SUV measures and initial PSA levels individually in terms of predictors of risk based on follow-up PSA levels and PSA doubling times, respectively, the risk model of the implementation yielded improved performance by incorporating all molecular imaging measures. Risk model predictions combined with Gleason scores were validated against follow-up PSA levels and PSA doubling times and yielded significant differences between risk groups.

[0101] A risk score for head and neck cancer incorporated extracted FDG PET / CT imaging measures and overall AJCC staging based on tumor-node- metastasis classification. The risk score and all imaging measures were significant prognosticators of overall survival by univariable Cox regression. The risk score was a negative prognosticator of overall survival with a hazard ratio of 1 .69 by multivariable Cox regression where patients with higher risk levels had shorter median overall survival by Kaplan-Meier analysis. Risk stratification was improved when imaging measures were combined with Gleason scores and AJCC staging for patients withprostate cancer and head and neck cancer, respectively, indicating synergy between molecular imaging measures and clinical pathologic and anatomic factors.

[0102] A decision tree classifier using FDG PET / CT imaging measures from pre- and post-therapy scans predicted pCR in breast cancer patients undergoing neoadjuvant chemotherapy with an accuracy of 0.84. While the classifier achieved a PPV of 1.00, TNR of 1.00, and NPV of 0.82, the classifier had a TPR of 0.43. A decision tree classifier using only pre-therapy measures predicted pCR with an accuracy of 0.72, demonstrating the feasibility of predicting pCR to inform treatment management prior to neoadjuvant therapy and surgery using a small cohort of 36 patients.

[0103] Given the wide array of available radiotracers, it is advantageous for Al approaches to generalize across molecular imaging agents to support automated analysis. Indeed, the implementation performed whole-body tumor quantification on FDG and PSMA PET / CT images of multiple cancers, despite the differences in biodistribution between FDG and PSMA-targeted uptake patterns.

[0104] Unlike approaches focusing on specific VOIs or cross-sectional slices, the implementation performed tumor quantification on whole-body PET / CT. Additionally, the implementation provided lesion-wise radiomics analysis to support staging, disease tracking, and treatment response monitoring. The state-of-the-art nn- Unet architecture was used as a backbone, which achieved robust whole-body tumor quantification. Some embodiments may also incorporate multi-organ segmentation for systemic analysis and radiation dosimetry applications.

[0105] DL models often require training data with extensive expert annotations.This limitation is ameliorated by the semi-supervised transfer learning approach of theimplementation, which learns the segmentation task on the target domain using incomplete annotations.

[0106] In sum, a deep semi-supervised transfer learning implementation performed fully automated whole-body tumor segmentation on PET / CT using limited tumor annotations and generalized across patients with six different cancers imaged with FDG and PSMA-targeted radiotracers. Molecular imaging measures were automatically quantified and demonstrated prognostic value for risk stratification, survival analysis, and treatment response prediction. The implementation demonstrated accurate tumor segmentation and prognosis of patients across six cancers on FDG and PSMA PET / CT.

[0107] Certain examples can be performed using a computer program or set of programs. The computer programs can exist in a variety of forms both active and inactive. For example, the computer programs can exist as software program (s) comprised of program instructions in source code, object code, executable code or other formats; firmware program(s), or hardware description language (HDL) files. Any of the above can be embodied on a transitory or non-transitory computer readable medium, which include storage devices and signals, in compressed or uncompressed form. Exemplary computer readable storage devices include conventional computer system RAM (random access memory), ROM (read-only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), flash memory, and magnetic or optical disks or tapes.

[0108] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / orblock diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented using computer readable program instructions that are executed by an electronic processor.

[0109] These computer readable program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the electronic processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0110] In embodiments, the computer readable program instructions may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, statesetting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the C programming language or similar programming languages. The computer readable program instructions may execute entirely on a user's computer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.

[0111] As used herein, the terms “A or B” and “A and / or B” are intended to encompass A, B, or {A and B}. Further, the terms “A, B, or C” and “A, B, and / or C” are intended to encompass single items, pairs of items, or all items, that is, all of: A, B, C, {A and B}, {A and C}, {B and C}, and {A and B and C}. The term “or” as used herein means “and / or.”

[0112] As used herein, language such as “at least one of X, Y, and Z,” “at least one of X, Y, or Z,” “at least one or more of X, Y, and Z,” “at least one or more of X, Y, or Z,” “at least one or more of X, Y, and / or Z,” or “at least one of X, Y, and / or Z,” is intended to be inclusive of both a single item (e.g., just X, or just Y, or just Z) and multiple items (e.g., {X and Y}, {X and Z}, {Y and Z}, or {X, Y, and Z}). The phrase “at least one of” and similar phrases are not intended to convey a requirement that each possible item must be present, although each possible item may be present.

[0113] The techniques presented and claimed herein are referenced and applied to material objects and concrete examples of a practical nature that demonstrably improve the present technical field and, as such, are not abstract, intangible or purely theoretical. Further, if any claims appended to the end of this specification contain one or more elements designated as “means for [perform]ing [a function]...” or “step for [performing [a function]...”, it is intended that such elements are to be interpreted under 35 U.S.C. § 112(f). However, for any claims containing elements designated in any other manner, it is intended that such elements are not to be interpreted under 35 U.S.C. § 112(f).

[0114] While the invention has been described with reference to the exemplary examples thereof, those skilled in the art will be able to make various modifications tothe described examples without departing from the true spirit and scope. The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. In particular, although the method has been described by examples, the steps of the method can be performed in a different order than illustrated or simultaneously. Those skilled in the art will recognize that these and other variations are possible within the spirit and scope as defined in the following claims and their equivalents.

Claims

What is claimed is:1 . A machine learning method of estimating a patient prognosis based on a patient radiotracer image, the method comprising: obtaining patient imaging data comprising a patient radiotracer image; providing the patient imaging data as an input to a first trained machine learning system, wherein the first trained machine learning system has been trained using a plurality of training images, wherein the plurality of training images comprise at least one of: a plurality of images captured using a plurality of radiotracer types, a plurality of images of a plurality of cancer types, or a plurality of images captured at a plurality of institutions; obtaining a first output from the first trained machine learning system, wherein the first output comprises at least one of a patient tumor segmentation or a patient tumor quantification; generating a plurality of patient feature quantifications from the first output; providing the plurality of patient feature quantifications as an input to a second trained machine learning system, wherein the second trained machine learning system has been trained using a plurality of training feature quantifications and a plurality of training prognostic data; obtaining a second output from the second trained machine learning system, wherein the second output comprises patient prognostic data; and providing a patient prognosis based on the patient prognostic data.

2. The method of claim 1 , wherein the plurality of training images comprise images generated by: generating, by a third trained machine learning system, a plurality of training anatomical images depicting normal anatomy; generating, by a fourth trained machine learning system, a plurality of training radiotracer images depicting radiotracer lesion activity; and generating a plurality of combination images, wherein the plurality of training images comprises the plurality of combination images, wherein a respective combination image comprises an image of the plurality of training anatomical images and an image of the plurality of training radiotracer images.

3. The method of claim 2, wherein the generating the plurality of combination images comprises: transforming an image of the plurality of training anatomical images into a projection space, wherein a projected training anatomical image is produced; transforming an image of the plurality of training radiotracer images into the projection space, wherein a projected training radiotracer image is produced; combining the projected training anatomical image with the projected training radiotracer image in the projection space, wherein a projected combination image is produced; and transforming the projected combination image into a 3D image space, wherein a combination image of the plurality of combination images is produced.

4. The method of claim 3, further comprising:simulating, in the projection space, at least one of: detector blurring, noise, scatter, scatter correction, randoms, randoms correction, attenuation, attenuation map, or attenuation correction.

5. The method of claim 1 , wherein the patient imaging data comprises a positron emission tomography (PET) image and a computer tomography (CT) image.

6. The method of claim 1 , wherein the plurality of training images comprise a plurality of images of a plurality of cancer types, and wherein the plurality of cancer types comprise two or more of: breast cancer, prostate cancer, head and neck cancer, lung cancer, melanoma, or lymphoma.

7. The method of claim 1 , wherein the plurality of training images comprise a plurality of images captured using a plurality of radiotracer types, and wherein the plurality of radiotracer types comprises: fluorodeoxyglucose (FDG) and a prostate-specific membrane antigen (PSMA)-targeted radiotracer.

8. The method of claim 1 , wherein the plurality of patient feature quantifications comprise a plurality of radiomics.

9. The method of claim 1 , wherein the second trained machine learning system comprises at least one of: a logistic regression model, a multilayer perceptron, a decision tree, a random forest, a linear discriminant analysis, ageneralized additive model, a k-nearest neighbors algorithm, or a support vector machine.

10. The method of claim 1 , wherein the patient prognosis comprises at least one of: a patient risk stratification, a patient survival analysis, or a patient treatment response prediction.11 . The method of claim 1 , wherein the generating the plurality of patient feature quantifications from the first output comprises providing the first output to a trained feature quantification system, wherein the trained feature quantification system outputs the plurality of feature quantifications.

12. A machine learning system for estimating a patient prognosis based on a patient radiotracer image, the system comprising: a non-transitory computer readable medium comprising instructions; and at least one electronic processor that executes the instructions to perform actions comprising: obtaining patient imaging data comprising a patient radiotracer image; providing the patient imaging data as an input to a first trained machine learning system, wherein the first trained machine learning system has been trained using a plurality of training images, wherein the plurality of training images comprise at least one of: a plurality of images captured using a plurality of radiotracer types, a plurality of images of a plurality of cancer types, or a plurality of images captured at a plurality of institutions;obtaining a first output from the first trained machine learning system, wherein the first output comprises at least one of a patient tumor segmentation or a patient tumor quantification; generating a plurality of patient feature quantifications from the first output; providing the plurality of patient feature quantifications as an input to a second trained machine learning system, wherein the second trained machine learning system has been trained using a plurality of training feature quantifications and a plurality of training prognostic data; obtaining a second output from the second trained machine learning system, wherein the second output comprises patient prognostic data; and providing a patient prognosis based on the patient prognostic data.

13. The system of claim 12, wherein the plurality of training images comprise images generated by: generating, by a third trained machine learning system, a plurality of training anatomical images depicting normal anatomy; generating, by a fourth trained machine learning system, a plurality of training radiotracer images depicting radiotracer lesion activity; and generating a plurality of combination images, wherein the plurality of training images comprises the plurality of combination images, wherein a respective combination image comprises an image of the plurality of training anatomical images and an image of the plurality of training radiotracer images.

14. The system of claim 13, wherein the generating the plurality of combination images comprises:transforming an image of the plurality of training anatomical images into a projection space, wherein a projected training anatomical image is produced; transforming an image of the plurality of training radiotracer images into the projection space, wherein a projected training radiotracer image is produced; combining the projected training anatomical image with the projected training radiotracer image in the projection space, wherein a projected combination image is produced; and transforming the projected combination image into a 3D image space, wherein a combination image of the plurality of combination images is produced.

15. The system of claim 14, wherein the actions further comprise: simulating, in the projection space, at least one of: detector blurring, noise, scatter, scatter correction, randoms, randoms correction, attenuation, attenuation map, or attenuation correction.

16. The system of claim 12, wherein the patient imaging data comprises a positron emission tomography (PET) image and a computer tomography (CT) image.

17. The system of claim 12, wherein the plurality of training images comprise a plurality of images of a plurality of cancer types, and wherein the plurality of cancer types comprise two or more of: breast cancer, prostate cancer, head and neck cancer, lung cancer, melanoma, or lymphoma.

18. The system of claim 12,wherein the plurality of training images comprise a plurality of images captured using a plurality of radiotracer types, and wherein the plurality of radiotracer types comprises: fluorodeoxyglucose (FDG) and a prostate-specific membrane antigen (PSMA)-targeted radiotracer.

19. The system of claim 12, wherein the plurality of patient feature quantifications comprise a plurality of radiomics.

20. The system of claim 12, wherein the second trained machine learning system comprises at least one of: a logistic regression model, a multilayer perceptron, a decision tree, a random forest, a linear discriminant analysis, a generalized additive model, a k-nearest neighbors algorithm, or a support vector machine.21 . The system of claim 12, wherein the patient prognosis comprises at least one of: a patient risk stratification, a patient survival analysis, or a patient treatment response prediction.

22. The system of claim 12, wherein the generating the plurality of patient feature quantifications from the first output comprises providing the first output to a trained feature quantification system, wherein the trained feature quantification system outputs the plurality of feature quantifications.

Images