Machine learning methods and related aspects for generating synthetic tumors on computed tomography scans

Abstract

Examples may provide a method of producing a synthetic computed tomography (CT) image. The method includes generating, by a computer, a synthetic tumor image using a simulated tumor development model. The method also includes integrating, by the computer, the synthetic tumor image with a reference CT image. Examples may also provide electronic neural networks trained using sets of synthetic CT images. Additional methods as well as related systems and computer readable media are also provided.

Description

Attorney Docket No.0184.0313-PCT (P18130-02) MACHINE LEARNING METHODS AND RELATED ASPECTS FOR GENERATING SYNTHETIC TUMORS ON COMPUTED TOMOGRAPHY SCANS Cross-Reference to Related Applications

[0001] This application claims priority to, and the benefit of, U.S. Provisional Patent Application Ser. No.63 / 662,836, filed June 21, 2024, the disclosure of which is incorporated herein by reference. Field

[0002] This disclosure relates generally to machine learning, e.g., in the context of medical applications, such as diagnostics. Background

[0003] Training AI models for image segmentation requires extensive manual per-pixel annotations by human experts. To reduce their annotation efforts, data synthesis is an exciting approach to generate enormous synthetic per-pixel annotations for both training and testing AI models and particularly for cancerous tumors whose manual per-pixel annotations are hard to acquire.

[0004] Preliminary studies have focused on synthesizing specific disease conditions in medical images, such as synthesizing pulmonary inflammatory lesion in COVID-19, retinal diabetic lesions, lung nodules, abdominal tumors, and brain tumors. Despite these advances, the creation of highly realistic tumors, the generalization of synthetic tumors across multiple organs, and the effective utilization of such synthetic tumors in AI training present ongoing challenges. Three important aspects warrant careful consideration in this context.

[0005] (i) Using no manual annotation. Learning-based approaches, such as GAN and Diffusion, excel in learning tumor representations but need abundant paired tumor data for effective CT image generation. Moreover, the generation process needs extra manual efforts, including the creation of masks to indicate tumor locations and shapes

[0006] (ii) Applicable to multiple organs. Modeling-based approaches leverage specialized design and domain expertise to simulate tumor appearances.Although the utilization of modeling-based synthetic tumors eliminates the need for manual annotation, they require significant effort for designing proper tumor characteristics customized to a specific organ, therefore limiting generalization, especially across different organs.

[0007] (iii) Generating realistic tumors. None of the existing synthetic approaches can simulate tumor development, especially in the intricate processes of proliferation and invasion. The complexity of these processes, influenced by the surrounding environment, makes it difficult to generate synthetic tumors that interact with organ structures, particularly when applied to different organs.

[0008] Accordingly, there is a need for approaches to generate synthetic or fake tumors that closely simulate real tumors for use in training AI models for medical image segmentation, among other applications. Summary

[0009] The present disclosure demonstrates, in some aspects, that tumor segmentation can be done by artificial intelligence (AI) models without manual per- voxel tumor annotation developed by humans. In some embodiments, this is achieved by tumor synthesis in CT scans. The advantages of these synthetic tumors include being: (I) realistic, for example, in shape and texture, which even medical professionals can confuse with real tumors and (II) effective for training AI models, which can perform tumor segmentation similar to models trained on real tumors. That is, in some embodiments, AI models trained solely using the synthetic tumors of the present disclosure perform as well as or even better than models trained using annotated CT scans of real tumors. These results also imply that manual efforts for developing per-voxel annotation of tumors (which often took years to create) can now be considerably reduced or eliminated for training AI models. Moreover, the methods disclosed herein can automatically generate many examples of small (or even tiny) synthetic tumors that can improve the success rate of detecting small tumors in patients, which is important for detecting the early stages of cancer. In addition to enriching the available training data, the tumor synthesis methods of the present disclosure can also be used to validate electronic neural networks that have been to detect and segment tumors in test subject CT images. Furthermore, the synthetictumor data sets created using the methods disclosed herein allow for the detection of failure modes in current AI models and accordingly, can be used to make the algorithms more robust for detecting tumors in the real world. These and other aspects will be apparent upon a complete review of the present disclosure, including the accompanying figures.

[0010] According to various embodiments, a method of producing a synthetic computed tomography (CT) image is presented. The method includes generating, by a computer, a synthetic tumor image using a simulated tumor development model, wherein each pixel of the synthetic tumor image is assigned an initial state value that together represent a population of cells of the synthetic tumor image and wherein the simulated tumor development model applies an accumulation rule in which cells having a first state value accumulate with a first probability, a growth rule in which cells having a second state value incentivize at least one neighboring cell to grow with a second probability, and a death rule in which cells having a third state value and surrounded by neighboring cells having the third state value die with a third probability. The method also includes integrating, by the computer, the synthetic tumor image with a reference CT image, thereby producing the synthetic CT image.

[0011] Various optional features of the above embodiments include the following. The synthetic CT image produced by the methods disclosed herein. The method further comprises producing a set of synthetic CT images. The method further comprises training an electronic neural network that detects and segments tumors in test subject CT images using at least a portion of the set of synthetic CT images to produce a trained electronic neural network. The method further comprises passing a test subject CT image through the trained electronic neural network to detect a tumor in the test subject and to segment the tumor in the test subject CT image to produce a detected and segmented tumor in the test subject. The method further comprises administering one or more therapies to the test subject to treat the detected and segmented tumor in the test subject. The trained electronic neural network produced by the methods disclosed herein. The trained electronic neural network achieves detection and segmentation performance levels that are comparable to or exceed a detection and segmentation performance level achieved by an electronic neural network trained only using real tumor CT images. The method comprises continuallytraining the electronic neural network that detects and segments tumors in the test subject CT images using additional synthetic CT images as the additional synthetic CT images are generated. The method further comprises validating a trained electronic neural network that detects and segments tumors in test subject CT images using at least a portion of the set of synthetic CT images to produce a validated electronic neural network. The reference CT image is obtained from at least a portion of a reference subject. The simulated tumor development model comprises a cellular automata. Each of the pixels represents one cell of the population of cells of the synthetic tumor image. The initial state value is selected from a finite number of states having a value of 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10. The accumulation rule is applied to cells in which the first state value is greater than zero and less than 10. The growth rule is applied to cells in which the second state value is greater than zero. The death rule is applied to cells in which the third state value is 10 and the third state values of the neighboring cells are 10. The accumulation, growth, and death rules simulate a shape, size, texture, location, intensity, and / or invasive behavior of the tumor. The method comprises using the simulated tumor development model for a selected time, t, to generate the synthetic tumor image. The method comprises defining a structure of the synthetic tumor of the synthetic tumor image that comprises an inner region representing dead cells, an intermediate region representing living but non- proliferative cells, and an outer region representing proliferative cells. The synthetic tumor of the synthetic tumor image is generalizable across multiple organs of a given subject. The synthetic tumor of the synthetic tumor image comprises a radius of about 25 mm or less, about 20 mm or less, about 15 mm or less, about 10 mm or less, or about 5 mm or less.

[0012] According to various embodiments, a system is presented. The system includes a controller that comprises, or is capable of accessing, computer readable media comprising non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least: passing a test subject computed tomography (CT) image through a trained electronic neural network to detect a tumor in the test subject and to segment the tumor in the test subject CT image to produce a detected and segmented tumor in the test subject, wherein the trained electronic neural network is at least partially trained using a set of synthetic CTimages that are produced by generating synthetic tumor images using a simulated tumor development model, wherein each pixel of a given synthetic tumor image is assigned an initial state value that together represents a population of cells of the given synthetic tumor image and wherein the simulated tumor development model applies an accumulation rule in which cells having a first state value accumulate with a first probability, a growth rule in which cells having a second state value incentivize at least one neighboring cell to grow with a second probability, and a death rule in which cells having a third state value and surrounded by neighboring cells having the third state value die with a third probability, and integrating the synthetic tumor images with reference CT images to produce the set of synthetic CT images; and, generating a report of the detected and segmented tumor in the test subject.

[0013] Various optional features of the above embodiments include the following. The system further comprises a CT scanner configured to generate test subject CT images, wherein the controller is operably connected to the CT scanner, and wherein the non-transitory computer executable instructions which, when executed by the electronic processor, further perform: generating the test subject CT image. The non-transitory computer executable instructions which, when executed by the electronic processor, further perform: generating at least one therapy recommendation to treat the detected and segmented tumor in the test subject. The simulated tumor development model comprises a cellular automata. The initial state value is selected from a finite number of states having a value of 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10. The accumulation rule is applied to cells in which the first state value is greater than zero and less than 10. The growth rule is applied to cells in which the second state value is greater than zero. The death rule is applied to cells in which the third state value is 10 and the third state values of the neighboring cells are 10. The accumulation, growth, and death rules simulate a shape, size, texture, location, intensity, and / or invasive behavior of the tumor. The synthetic tumor of the synthetic tumor image comprises a radius of about 25 mm or less, about 20 mm or less, about 15 mm or less, about 10 mm or less, or about 5 mm or less.

[0014] According to various embodiments, a computer readable media is presented. The computer readable media comprises non-transitory computer executable instructions which, when executed by at least electronic processor,perform at least: passing a test subject computed tomography (CT) image through a trained electronic neural network to detect a tumor in the test subject and to segment the tumor in the test subject CT image to produce a detected and segmented tumor in the test subject, wherein the trained electronic neural network is at least partially trained using a set of synthetic CT images that are produced by generating synthetic tumor images using a simulated tumor development model, wherein each pixel of a given synthetic tumor image is assigned an initial state value that together represents a population of cells of the given synthetic tumor image and wherein the simulated tumor development model applies an accumulation rule in which cells having a first state value accumulate with a first probability, a growth rule in which cells having a second state value incentivize at least one neighboring cell to grow with a second probability, and a death rule in which cells having a third state value and surrounded by neighboring cells having the third state value die with a third probability, and integrating the synthetic tumor images with reference CT images to produce the set of synthetic CT images; and, generating a report of the detected and segmented tumor in the test subject. Drawings

[0015] The above and / or other aspects and advantages will become more apparent and more readily appreciated from the following detailed description of examples, taken in conjunction with the accompanying drawings, in which:

[0016] Fig.1 is a flow chart that schematically shows exemplary method steps of producing a synthetic computed tomography (CT) image according to some aspects disclosed herein;

[0017] Fig.2 is a schematic diagram of an exemplary system suitable for use with certain aspects disclosed herein;

[0018] Figs. 3A-3C. Simulation of tumor development. (a) The state of the tumor can be represented by a 3-dimensional array, with each cell corresponding to a pixel in the CT image. (b) Each cell contains a value of states from zero to ten, which represents the population of tumor cells in one pixel. (c) The synthetic tumor develops from one pixel to a small tumor, medium tumor, and even large tumor. This enablesthe tumor synthesis approach to provide different synthetic tumors of various sizes, shapes, and textures;

[0019] Fig.4. Idealized tumors. The gray region contains dead tumor cells; the inactive region contains living, quiescent cells (non- proliferative). The outer shell isthe active region with proliferativecells, and the tumor radius is denoted as Rt. Idealizedtumors canbe used to formulate generic rules for tumor development;

[0020] Figs. 5A and 5B. Cellular automata initialization. (a) Two common definitions of the neighborhood: (1) considering the orthogonally adjacent cells as the neighborhood; (2) considering the orthogonally and diagonally adjacent cells as the neighborhood. The latter aligns better with our synthesis approach. (b) The basic cellular automata, with a uniform growth rule, can generate sphere tumors;

[0021] Fig.6. Organ / tumor quantification. The quantified organ map (left) translates CT images into several discrete levels from low-intensity to high-intensity. Different levels can affect the speed of tumor development. In the quantified tumor map (right), each pixel has a state value from zero to ten, which represents the tumor cell population;

[0022] Figs. 7A and 7B. (a) Tumor interacting with organ boundary and vessel. The upper interaction occurs at the tumor-organ boundary. As the tumor continues to grow, it will push against the organ’s boundary. The lower interaction takes place at the boundary between the tumor and the vessels. The tumor’s surface becomes deformed when it interacts with the vessels. (b) Mapping the synthetic tumor back to the CT image. To generate synthetic CT data, we also require a mapping function to correlate the synthetic tumor with CT values. The mapping function integrates the tumor population’s state with the original CT intensity to generate the synthetic tumor in CT images;

[0023] Fig.8. Effectiveness in early tumor detection. Segmentation results for small tumors are obtained using models trained on real tumors, state-of-the-art synthetic tumors from prior work, and our synthetic tumors. The panels, arranged from top to bottom, display results for the liver, pancreas, and kidney. Our data synthesis approach enhances model performance in detecting challenging small tumors across various organs, as evidenced by segmentation and sensitivity comparisons;

[0024] Fig. 9. Generalizable tumor synthesis across organs. Early-stage tumors present similar imaging characteristics in computed tomography (CT), whether they are located in the liver, pancreas, or kidneys. Leveraging this observation, we develop a generative AI model on a few examples of annotated tumors in a specific organ, e.g., the liver (upper right images). This AI model (upper right images), trained exclusively on liver tumors, can directly create synthetic tumors in those organs where CT volumes of annotated tumors are relatively scarce, e.g., the pancreas (upper left images) and kidneys (lower right and left images). By integrating synthetic tumors into extensive CT volumes of healthy organs—routinely collected in clinical settings—we can substantially augment the training set for tumor segmentation. This enhancement can also significantly improve the AI generalizability across CT volumes sourced from diverse hospitals and patient demographics;

[0025] Fig.10A and 10B. Reader studies and feature analysis. We assess the performance of a support vector machine (SVM) classifier, using Radiomics features, and three expert radiologists in identifying the originating organs of cropped tumors. The SVM classifier is tasked with a three-way classification to ascertain whether a tumor originates from the liver, pancreas, or kidneys. In a similar test, radiologists examine the original CT images of these tumors. Reader study results on the left panel indicate significant challenges for both the SVM classifier and the radiologists in accurately identifying the origin of early-stage tumors. The precision and recall scores for both methods closely resemble those of random guessing. Additionally, on the right panel, we present a t-SNE visualization of Radiomics features for tumors from the liver, pancreas, and kidneys. These results highlight the considerable similarity in features and images of early-stage tumors;

[0026] Fig. 11. Overview of the DiffTumor framework. Towards generalizable tumor synthesis, developing our DiffTumor involves three stages. ① Training an Antoencoder Model—consisting of an encoder and decoder—to learn comprehensive latent features. The learning task here is image reconstruction performed on 9,262 unlabeled three-dimensional CT volumes. Both the trained encoder and decoder will be used in subsequent stages. ② Training a Diffusion Model—a specific type of generative models—using latent features and tumor masks as conditions. Once trained, this model can generate latent features necessary forreconstructing CT volumes with tumors based on arbitrary masks. ③ Training a Segmentation Model using CT volumes of synthetic tumors, which are reconstructed by the decoder. With a large repository of healthy CT volumes, our DiffTumor framework can produce a vast array of synthetic tumors, varying in location, size, shape, texture, and intensity, therefore fostering high-performing AI models for tumor detection / segmentation;

[0027] Fig.12. Generalizable to various demographics. Tumor detection and segmentation enhancement for individuals across various age groups and genders. DiffTumor can consistently boost tumor detection and segmentation performance by a significant margin in each patient group;

[0028] Fig. 13. Reduced annotations for Diffusion Model. Diffusion Model, trained on annotated tumors in Stage ②, can generate syn- thetic tumors for the subsequent training of Segmentation Model in Stage ③. We investigate the relationship between the number of annotated real tumors required for Diffusion Model and the resultant performance of Segmentation Model. Results with varying numbers of annotated tumors reveal a surprising finding: extensive annotations are not necessary for tumor synthesis, contrary to the experience in computer vision. Notably, training Diffusion Model with only one annotated tumor seems to be sufficient. This efficiency is connected to our earlier observation that tumors, particularly in their early stages, tend to present similar appearances across different organs, thus facilitating the learning process of Diffusion Model with fewer annotated examples;

[0029] Fig. 14. Accelerated tumor synthesis. The speed of generating synthetic tumors in Diffusion Model is significantly influenced by the timestep hyper- parameter (see figure). A faster tumor synthesis is preferred in Stage ③ when training Segmentation Model, so we examine the impact of timestep on the performance of Segmentation Model. Our findings indicate that a timestep of 4, generating a tumor in 0.2 seconds, provides the most favorable results among various tested timesteps. Considering the trade-off between performance and efficiency, we chose a timestep of 4 for this example;

[0030] Fig.15. Enhanced early tumor detection. We analyzed failure cases of AI models trained on real tumors, identifying that these models often overlook early- stage tumors characterized by blurry boundaries, small sizes, and peripheral organlocations. Our DiffTumor enhances the detection and segmentation of these challenging tumors by extensively generating small tumors for AI training (evidenced in Table 2);

[0031] Figs. 16A-16D. Illustration of continual learning framework. (a) The setting for the static training, where the real-tumor dataset is partitioned into training and validation sets. The AI model is then developed using these datasets and subsequently tested with unseen data. (b) Dynamic training setting integrated with synthetic data. (c) Tumor generator pipeline. By leveraging an advanced tumor generator, we can create a dynamic training and validation set. Models developed using the continual learning framework exhibit superior performance compared to static training settings. (d) Visualization of synthetic data;

[0032] Figs. 17A and 17B. The overfitting is due to a small-scale, biased real-tumor validation set. Model checkpoints are saved at each training epoch, and their performance trajectories are evaluated. The black curve plots the test set performance of each checkpoint. It serves as the gold standard for a specific dataset, though the test set performance is often apriori unknown. The grey curve plots the validation set performance of each checkpoint. This performance is accessible during the AI training and instrumental in selecting the best model checkpoint. We train a model on the LiTS training set and subsequently test it on the LiTS test set as in- domain evaluation (a) and the FLARE’23 dataset as out-domain evaluation (b). At the initial epochs, the model performs increasingly well, but its test set performance declines when trained for more epochs, highlighted by the arrow. This decline is attributed to overfitting, where the model becomes too specialized on the training set and loses its ability to generalize effectively to the test set. The purpose of a validation set is to select the best model checkpoint that is expected to perform well on unseen data. However, in practice, the size and diversity of the validation set may be limited, leading to potential inaccuracies in checkpoint selection. This is evidenced by both in- domain (a) and out-domain (b) evaluations. The dots on the curve represent the best checkpoint selected by the test set (black) or the real-tumor validation (grey). A comparative analysis reveals that the checkpoints selected based on the real-tumor validation set might not be the most suitable for test sets;

[0033] Figs. 18A and 18B. The overfitting is alleviated by a large-scale, synthetic-tumor validation set. Similar to Figure 2, (a) and (b) denote in-domain and out-domain evaluations, and the black curves are the test set performance on these two datasets—serving as the gold standard for checkpoint selection. The grey curves are the validation performance using synthetic tumors (cohort 5). In theory, we can generate an unlimited number of tumors in varied conditions, e.g., size, location, shape, and texture, as needed, using the tumor generator described in §3.2. Such extensive coverage enhances the ability of the validation set to estimate how well the model can be generalized to previously unseen data distributions. As shown in both in- domain and out-domain evaluation, synthetic data as validation can accurately select the best model checkpoint that is almost identical to that selected by test sets;

[0034] Figs. 19A and 19B. The overfitting can be addressed by continual learning on synthetic data. We set up the continual learning framework for liver tumor segmentation, described in §3.1. Its learning curves are presented in solid black, referred to as dynamic training. In comparison, dashed curves are the conventional framework training with a limited number of real data, referred to as static training. Both in-domain (a) and out-domain (b) evaluations show that the AI model continuously trained on synthetic data outperforms the one trained on real data;

[0035] Figs.20A and 20B. Synthetic data can benefit early cancer detection. We evaluate the efficacy of synthetic data in detecting tiny liver tumors (radius < 5mm). Specifically, the AI model developed with our continual learning framework – trained and evaluated with synthetic data – is evaluated on tumor detection. We compare it with AI model developed on static real data. The “trained@real“ and “trained@synt“ hold the same meaning as described in Table 2. The “best@real“ and “best@synt“ denote the selection of best checkpoints based on real- and synthetic- tumor validation sets, respectively. We report the sensitivities (%) achieved on the test set. As shown in both in-domain (a) and out-domain (b) evaluations, our continual learning framework with synthetic data proves to be effective in detecting tiny liver tumors (radius < 5mm), thereby benefiting early cancer detection;

[0036] Fig.21. The continual learning framework is enhanced by in-domain synthetic-tumor validation. Our framework enables the utilization of healthy cases from various domains. Specifically, an interesting situation arises where we candirectly create synthetic-tumor validation using healthy cases from the same domain as the test set. This in-domain synthetic-tumor validation set can benefit our continual learning framework. To illustrate this advantage, we utilize the FLARE’23 dataset, which contains both disease cases and healthy cases. We trained the model using dynamic synthetic data and saved model checkpoints at each training epoch. Subsequently, we evaluated them on three datasets, i.e., out-domain synthetic-tumor validation set from assembly dataset (cohort 5, dashed curve), in-domain synthetic- tumor validation set from FLARE’23 dataset (cohort 6, grey curve), and FLARE’23 test set served as the gold standard (cohort 7, black curve). As shown, the in-domain synthetic-tumor validation set accurately identifies the best model, which aligns with the model selected by the FLARE’23 test set;

[0037] Fig. 22. Intensity distribution of liver tumors and their healthy counterparts. Our tumor generator, which is based on modeling and medical knowledge, incorporates the distributional characteristics of real tumors. We present the intensity distributions of real tumors obtained from the LiTS dataset;

[0038] Fig.23. Size distribution of liver tumors. We have calculated the size distribution of liver tumors from the LiTS dataset. This tumor size distribution will serve as a guide for determining the sizes of the synthetic tumors we generate;

[0039] Fig. 24. Examples of the generated shape. The tumor generator pipeline enables us to control the size and deformation of the generated tumors. Here, we present some examples of generated shapes under different conditions;

[0040] Fig. 25. Examples of the generated texture. Our data synthesis strategy also enables us to generate different textures, as illustrated here for visualization; and

[0041] Fig.26. Visualization of synthetic data. By combining all the pipelines together, we can obtain a wide range of diverse synthetic data for validation and training. Definitions

[0042] In order for the present disclosure to be more readily understood, certain terms are first defined below. Additional definitions for the following terms and other terms may be set forth throughout the specification. If a definition of a term set forthbelow is inconsistent with a definition in an application or patent that is incorporated by reference, the definition set forth in this application should be used to understand the meaning of the term.

[0043] As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Thus, for example, a reference to “a method” includes one or more methods, and / or steps of the type described herein and / or which will become apparent to those persons skilled in the art upon reading this disclosure and so forth.

[0044] It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. Further, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. In describing and claiming the methods, systems, and computer readable media, the following terminology, and grammatical variants thereof, will be used in accordance with the definitions set forth below.

[0045] Cancer Type: As used herein, “cancer type” refers to a type or subtype of cancer defined, e.g., by histopathology. Cancer type can be defined by any conventional criterion, such as on the basis of occurrence in a given tissue (e.g., blood cancers, central nervous system (CNS), brain cancers, lung cancers (small cell and non-small cell), skin cancers, throat cancers, nose cancers, liver cancers, bone cancers, lymphomas, pancreatic cancers, thyroid cancers, bladder cancers, kidney cancers, mouth cancers, stomach cancers, breast cancers, prostate cancers, bowel cancers, rectal cancers, ovarian cancers, intestinal cancers, soft tissue cancers, neuroendocrine cancers, lung cancers, gastroesophageal cancers, urothelial cancers, solid state cancers, heterogeneous cancers, homogenous cancers), head and neck cancers, gynecological cancers, colorectal cancers, unknown primary origin and the like, and / or of the same cell lineage (e.g., carcinoma, sarcoma, lymphoma, cholangiocarcinoma, leukemia, mesothelioma, melanoma, or glioblastoma) and / or cancers exhibiting cancer markers, such as Her2, CA15-3, CA19-9, CA-125, CEA, AFP, PSA, HCG, hormone receptor and NMP-22. Cancers can also be classified by stage (e.g., stage 1, 2, 3, or 4) and whether of primary or secondary origin.

[0046] Classifier: As used herein, “classifier” generally refers to algorithm computer code that receives, as input, test data and produces, as output, a classification of the input data as belonging to one or another class.

[0047] Data set: As used herein, “data set” refers to a group or collection of information, values, or data points related to or associated with one or more objects, records, and / or variables. In some embodiments, a given data set is organized as, or included as part of, a matrix or tabular data structure. In some embodiments, a data set is encoded as a feature vector corresponding to a given object, record, and / or variable, such as a given test or reference subject. For example, a medical data set for a given subject can include one or more observed values of one or more variables associated with that subject.

[0048] Electronic neural network: As used herein, “electronic neural network” refers to a machine learning algorithm or model that includes layers of at least partially interconnected artificial neurons (e.g., perceptrons or nodes) organized as input and output layers with one or more intervening hidden layers that together form a network that is or can be trained to classify data, such as test subject medical data sets (e.g., medical images or the like).

[0049] Labeled: As used herein, “labeled” or “annotated” in the context of data sets or points refers to data that is classified as, or otherwise associated with, having or lacking a given characteristic or property.

[0050] Machine Learning Algorithm: As used herein, "machine learning algorithm" generally refers to an algorithm, executed by computer, that automates analytical model building, e.g., for clustering, classification or pattern recognition. Machine learning algorithms may be supervised or unsupervised. Learning algorithms include, for example, artificial neural networks (e.g., back propagation networks, transformer networks, etc.), discriminant analyses (e.g., Bayesian classifier or Fisher’s analysis), multiple-instance learning (MIL), support vector machines, decision trees (e.g., recursive partitioning processes such as CART-classification and regression trees, or random forests), linear classifiers (e.g., multiple linear regression (MLR), partial least squares (PLS) regression, and principal components regression), hierarchical clustering, and cluster analysis. A dataset on which a machine learning algorithm learns can be referred to as "training data." A model produced using amachine learning algorithm is generally referred to herein as a “machine learning model.”

[0051] Subject: As used herein, “subject” or “test subject” refers to an animal, such as a mammalian species (e.g., human) or avian (e.g., bird) species. More specifically, a subject can be a vertebrate, e.g., a mammal such as a mouse, a primate, a simian or a human. Animals include farm animals (e.g., production cattle, dairy cattle, poultry, horses, pigs, and the like), sport animals, and companion animals (e.g., pets or support animals). A subject can be a healthy individual, an individual that has or is suspected of having a disease or pathology or a predisposition to the disease or pathology, or an individual that is in need of therapy or suspected of needing therapy. The terms “individual” or “patient” are intended to be interchangeable with “subject.” A “reference subject” refers to a subject known to have or lack specific properties (e.g., a known pathology, such as melanoma and / or the like).

[0052] Value: As used herein, “value” generally refers to an entry in a dataset that can be anything that characterizes the feature to which the value refers. This includes, without limitation, numbers, words or phrases, symbols (e.g., + or -) or degrees. Description of the Embodiments

[0053] Reference will now be made in detail to example implementations. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the invention. The following description is, therefore, merely exemplary.

[0054] I. Introduction

[0055] AI for cancer detection encounters the bottleneck of data scarcity, annotation difficulty, and low prevalence of early tumors. Tumor synthesis seeks to create artificial tumors in medical images, which can greatly diversify the data and annotations for AI training. However, current tumor synthesis approaches are not applicable across different organs due to their need for specific expertise and design. Accordingly, in some aspects, the present disclosure establishes a set of generic rules to simulate tumor development. In some embodiments, each cell (pixel) is initiallyassigned a state between zero and ten to represent the tumor population, and a tumor can be developed based on three rules. R1. Accumulation: cells with a state greater than zero but less than ten can accumulate with a probability (self-state +1). R2. Growth: cells with a state greater than zero can incentivize one of its neighbors to grow (neighbor-state +1). R3. Death: cells with a state of ten surrounded by neighbors with ten can die with a probability (self-state = -1). In some embodiments, these three generic rules are applied to simulate tumor development—from pixel to cancer—using Cellular Automata. In some embodiments, the tumor state is then integrated with the original CT images to generate synthetic tumors across different organs. This tumor synthesis approach, for example, allows for sampling tumors at multiple stages and analyzing tumor-organ interaction. Clinically, a reader study involving three expert radiologists reveals that the synthetic tumors and their developing trajectories are convincingly realistic. Technically, we generate tumors at varied stages in 9,262 raw, unlabeled CT images sourced from 68 hospitals worldwide. The performance in segmenting tumors in the liver, pancreas, and kidneys exceeds prevailing literature benchmarks, underlining the immense potential of tumor synthesis, especially for earlier cancer detection.

[0056] Further, tumor synthesis enables the creation of artificial tumor examples in medical images, facilitating the training of AI models for tumor detection and segmentation. However, success in tumor synthesis hinges on creating visually realistic tumors that are generalizable across multiple organs and, furthermore, the resulting AI models being capable of detecting real tumors in images sourced from different medical domains (e.g., hospitals). Accordingly, in some aspects, the present disclosure provices for generalizable tumor synthesis by leveraging an important observation: early-stage tumors (< 2cm) tend to have similar imaging characteristics in computed tomography (CT), whether they originate in the liver, pancreas, or kidneys. We have ascertained that generative AI models, e.g., Diffusion Models, can create realistic tumors generalized to a range of organs even when trained on a limited number of tumor examples from just one organ. Moreover, we have shown that AI models trained on these synthetic tumors can be generalized to detect and segment real tumors from CT volumes, encompassing a broad spectrum of patient demographics, imaging protocols, and healthcare facilities.

[0057] In some aspects, the present disclosure leverages synthetic data as a validation set to reduce overfitting and ease the selection of the best model in AI development. While synthetic data have been used for augmenting the training set, we find that synthetic data can also significantly diversify the validation set, offering marked advantages in domains like healthcare, where data are typically limited, sensitive, and from out-domain sources (i.e., hospitals). In the present disclosure, we illustrate the effectiveness of synthetic data for early cancer detection in computed tomography (CT) volumes, where synthetic tumors are generated and superimposed onto healthy organs, thereby creating an extensive dataset for rigorous validation. Using synthetic data as validation can improve AI robustness in both in-domain and out-domain test sets. Furthermore, we establish a new continual learning framework that continuously trains AI models on a stream of out-domain data with synthetic tumors. The AI model trained and validated in dynamically expanding synthetic data can consistently outperform models trained and validated exclusively on real-world data. Specifically, as illustrated herein, the DSC score for liver tumor segmentation improves from 26.7% (95% CI: 22.6%–30.9%) to 34.5% (30.8%–38.2%) when evaluated on an in-domain dataset and from 31.1% (26.0%–36.2%) to 35.4% (32.1%– 38.7%) on an out-domain dataset. Importantly, the performance gain is particularly significant in identifying very tiny liver tumors (radius < 5mm) in CT volumes, with Sensitivity improving from 33.1% to 55.4% on an in-domain dataset and 33.9% to 52.3% on an out-domain dataset, justifying the efficacy in early detection of cancer. The application of synthetic data, from both training and validation perspectives, underlines a promising avenue to enhance AI robustness when dealing with data from varying domains. Additional applications and advantages of the methods and other aspects of the present disclosure will be apparent upon a complete review of this specification, including the accompanying figures.

[0058] In some aspects, the present disclosure provides a strategy to synthesize tumors in computed tomography (CT) scans. To illustrate, Fig.1 is a flow chart that schematically shows exemplary method steps of producing a synthetic CT image according to some aspects disclosed herein. As shown, method 100, which is typically computer-implemented, includes generating a synthetic tumor image using a simulated tumor development model (step 102) and integrating the synthetic tumorimage with a reference CT image to produce the synthetic CT images (step 104). In some embodiments, each pixel of the synthetic tumor image is assigned an initial state value that together represent a population of cells of the synthetic tumor image. In some embodiments, the simulated tumor development model applies an accumulation rule in which cells having a first state value accumulate with a first probability, a growth rule in which cells having a second state value incentivize at least one neighboring cell to grow with a second probability, and a death rule in which cells having a third state value and surrounded by neighboring cells having the third state value die with a third probability. In some embodiments, the reference CT image is obtained from at least a portion of a reference subject. In some embodiments, the simulated tumor development model comprises a cellular automata. Typically, each of the pixels represents one cell of the population of cells of the synthetic tumor image. In some embodiments, the synthetic tumor of the synthetic tumor image is generalizable across multiple organs of a given subject. In some embodiments, the synthetic tumor of the synthetic tumor image comprises a radius of about 25 mm or less, about 20 mm or less, about 15 mm or less, about 10 mm or less, or about 5 mm or less.

[0059] In some embodiments, the initial state value is selected from a finite number of states having a value of 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10. In some embodiments, the accumulation rule is applied to cells in which the first state value is greater than zero and less than 10. In some embodiments, the growth rule is applied to cells in which the second state value is greater than zero. In some embodiments, the death rule is applied to cells in which the third state value is 10 and the third state values of the neighboring cells are 10. Typically, the accumulation, growth, and death rules simulate a shape, size, texture, location, intensity, and / or invasive behavior of the tumor.

[0060] In some embodiments, method 100 includes producing a set of synthetic CT images. Typically, the set comprises at least about 100, at least about 1000, at least about 10000, at least about 100000, at least about 1000000, at least about 10000000, or more synthetic CT images. In some embodiments, method 100 further includes training an electronic neural network that detects and segments tumors in test subject CT images using at least a portion of the set of synthetic CT images to produce a trained electronic neural network. In some embodiments, method 100further includes passing a test subject CT image through the trained electronic neural network to detect a tumor in the test subject and to segment the tumor in the test subject CT image to produce a detected and segmented tumor in the test subject. In some embodiments, method 100 further includes administering one or more therapies to the test subject to treat the detected and segmented tumor in the test subject. In some embodiments, the trained electronic neural network achieves detection and segmentation performance levels that are comparable to or exceed a detection and segmentation performance level achieved by an electronic neural network trained only using real tumor CT images. In some embodiments, method 100 includes continually or dynamically training the electronic neural network that detects and segments tumors in the test subject CT images using additional synthetic CT images as the additional synthetic CT images are generated. In some embodiments, method 100 further includes validating a trained electronic neural network that detects and segments tumors in test subject CT images using at least a portion of the set of synthetic CT images to produce a validated electronic neural network. In some embodiments, method 100 includes using the simulated tumor development model for a selected time, t, to generate the synthetic tumor image. In some embodiments, method 100 includes defining a structure of the synthetic tumor of the synthetic tumor image that comprises an inner region representing dead cells, an intermediate region representing living but non-proliferative cells, and an outer region representing proliferative cells.

[0061] Fig. 2 is a schematic diagram of a hardware computer system 200 suitable for implementing various embodiments. For example, Fig.2 illustrates various hardware, software, and other resources that can be used in implementations of any of methods disclosed herein, including, e.g., method 100 and / or one or more instances of an electronic neural network. System 200 includes training corpus source 202 and computer 201. Training corpus source 202 and computer 201 may be communicatively coupled by way of one or more networks 204, e.g., the internet.

[0062] Training corpus source 202 may include an electronic clinical records system, such as an LIS, a database, a compendium of clinical data, or any other source of images suitable for use as a training corpus as disclosed herein. Due to hardware volatile memory storage limitations, each constituent image of an image maybe broken down into a number of tiles, which may be, e.g., 128 pixels by 128 pixels. Such tiles are examples of “components” as that term is used herein. According to some embodiments, each component is implemented as a vector, such as a feature vector, that represents a respective tile. Thus, the term “component” refers to both a tile and a feature vector representing a tile.

[0063] Computer 201 may be implemented as any of a desktop computer, a laptop computer, can be incorporated in one or more servers, clusters, or other computers or hardware resources, or can be implemented using cloud-based resources. Computer 201 includes volatile memory 214 and persistent memory 212, the latter of which can store computer-readable instructions, that, when executed by electronic processor 210, configure computer 201 to perform any of the methods disclosed herein, including method 100, 110, or 120, and / or form or store any electronic neural network, and / or perform any classification technique as described herein. Computer 201 further includes network interface 208, which communicatively couples computer 201 to training corpus source 202 via network 204. Other configurations of system 200, associated network connections, and other hardware, software, and service resources are possible.

[0064] Certain embodiments can be performed using a computer program or set of programs. The computer programs can exist in a variety of forms both active and inactive. For example, the computer programs can exist as software program(s) comprised of program instructions in source code, object code, executable code or other formats; firmware program(s), or hardware description language (HDL) files. Any of the above can be embodied on a transitory or non-transitory computer readable medium, which include storage devices and signals, in compressed or uncompressed form. Exemplary computer readable storage devices include conventional computer system RAM (random access memory), ROM (read-only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), and magnetic or optical disks or tapes.

[0065] II. Description of Example Embodiments

[0066] EXAMPLE 1: From Pixel to Cancer: Cellular Automata in Computed Tomography

[0067] 1. Introduction

[0068] This example establishes a set of generic rules to simulate tumor development. Each cell (pixel) is initially assigned a state between zero and ten to represent the tumor population, and a tumor can be developed based on three rules. In these rules, the basic element is referred to as a cell, representing a single pixel in CT images. Initially, we define the population of the tumor: each cell is assigned a state value between zero and ten to represent the tumor population. Consequently, we design three rules1. R1. Accumulation: cells with a state greater than zero but less than ten can accumulate with a probability (self-state +1). R2. Growth: cells with a state greater than zero can incentivize one of its neighbors to grow (neighbor-state +1). R3. Death: cells with a state of ten surrounded by neighbors with ten can die with a probability (self-state = -1). We then apply these three generic rules to simulate tumor development using cellular automata. After simulating the state of tumors, we apply a mapping function to map tumor states back to CT images. The mapping function integrates the tumor state with the original CT intensity to generate synthetic tumors in CT images. Our tumor synthesis approach can generate cancer from a pixel, so we name it Pixel2Cancer. It has the capacity to generate diverse stages of tumor development (as illustrated in Figure 3), simulate the interaction between tumors and their surrounding environments, exhibit applicability across various organs, and potentially contribute to the estimation of tumor prognosis.

[0069] We developed the rules (R1-R3) by considering various aspects of tumor behavior, including the processes of tumor growth and death, tumor invasion into normal tissue, and interactions between tumors and growth environments. These tumor behaviors guide us to not only synthesize the appearance of tumors in CT images but also incorporate the underlying clinical knowledge of tumor growth

[0070] We have conducted experiments with our Pixel2Cancer in two aspects. Clinically, a reader study involving three expert radiologists has been conducted, and the results have been so convincing that even medical professionals with over five years of experience can mistake them for real tumors (Table 1). Technically, we applied our Pixel2Cancer to generate synthetic tumors at unlabeled CT images. The performance in segmenting tumors in the liver, pancreas, and kidneys exceeds prevailing literature benchmarks, underlining the immense potential of tumorsynthesis, especially for earlier cancer detection / diagnosis (Figure 8). AI models trained on our synthetic tumors achieve the Dice Similarity Coefficient (DSC) of 55.4%, 31.5%, and 28.6% for the segmentation of real liver, pancreas, and kidney tumors, while AI models trained on real tumors achieve DSC of 53.5%, 28.3%, and 29.4% (Table 2). These results underscore the enormous potential of tumor synthesis basedon Cellular Automata. A key contribution of this example is a novel tumor synthesisapproach, Pixel2Cancer, which provides at least five exemplary advantages summarized below: 1. Pixel2Cancer simulates the tumor development from a single pixel to a whole tumor, enabling us to sample tumors at various stages, which can provide a variety of tumors for model training, including those of various sizes, from small to large (Figure 3). 2. Pixel2Cancer borrows the concept of the game of life— using Cellular Automata—to simulate tumor development in CT images, generating the idealized tumor based on tumor proliferation, invasion, and interaction with the environment (Figure 4, Figure 7). 3. Pixel2Cancer does not require manual annotation, and AI models trained on synthetic tumors can achieve comparable performance to those trained on real tumors with manual annotations for tumor segmentation (Table 2). 4. Pixel2Cancer devises generic rules based on clinical knowledge, which is generalizable to other organs without extra expertise and design (Table 2, Figure 8). 5. Pixel2Cancer promises to simulate tumor development on healthy CT images using real-world data, eliminating the need for finite and static datasets, and thus serving as a valuable data resource for lifelong learning (Table 3).

[0071] 2. Related Work

[0072] Tumor development is intricately regulated by biological mechanisms at various scales. Computational models have been employed for intra- and inter-cellular, tissue- and organ-specific scales. A model was proposed for prostate tumordevelopment, considering androgen concentration. A model was described for capturing the dynamics of tumor growth and angiogenesis. Tumor growth models also address invasion and necrosis. A stochastic computer model was presented for incorporating spatial and temporal components, including tumor oxygenation. A rapidperitoneal metastases prediction model was validated in breast tumor-bearing mice. Despite their structural simplicity, these mathematical models of tumor development involve complex biological factors. Moreover, compared to the numerous models for specific tumor types, relatively few models depicting general tumor growth patterns have been proposed. In this example, we introduce a universal approach to tumor development modeling using a visualized cellular automaton algorithm. The goal is to describe and predict diverse biological behaviors of tumor cells, including growth, invasion, and necrosis.

[0073] Cellular Automata represent a mathematical concept in computational modeling. It was well-known by Con- way’s Game of Life, which consists of a grid of cells, each of which can be in one of two states: alive or dead. The game starts with an initial configuration of live and dead cells. After each generation, the state of thecells is updated according to the rules, creating a new generation.The evolution of thegame produces intricate and often unpredictable patterns. We utilize the cellular automata to implement our generic rules for tumor simulation.

[0074] Tumor synthesis. Sufficient data image with class balance and variability is crucial for training successful machine learning algorithms in medical image analysis. Therefore, an effective and universally applicable tumor synthesis strategy is urgently needed to address these challenges. Various successful approaches to tumor synthesis in different medical modalities have been explored, including colon polyps in colonoscopy videos, brain tumors in MR images, lung nodules in CT images, and skin tumor lesions. In recent work, tumors in the liver and pancreas were synthesized using a model-based approach guided by radiologists. The synthetic tumors demonstrate a realistic appearance compared with actual liver and pancreatic tumors. Importantly, the AI trained with synthetic tumors achieves comparable segmentation and detection performance to the AI trained with real tumors. However, current model-based methods require significant effort and expertise to discern tumor imaging characteristics. Synthetic tumors generated by these methods may not accurately represent the dynamic tumor growth process. Therefore, we are making efforts to explore a novel pixel-level approach to address these challenges.

[0075] 3. Pixel2Cancer

[0076] 3.1 Preliminary

[0077] Cellular automata are computational models used to simulate complex systems through simple rules and interactions. It employs a grid of cells (pixels), each in one of a finite number of statesThe state of each cell evolves over discrete time steps t based on rules, often influenced by neighboring cells whereis the neighborhood of pixel. In tumor development simulation, cellular automata allows us to model the entire process, starting from a single pixel. Through this process, we apply specific rules to simulate the shape, texture, and invasive behavior of the tumor accurately. These tumor growth rules draw from medical knowledge, guided by the idealized tumor reflecting real-world characteristics.

[0078] Idealized tumors are shown in Figure 4. This model establishes the structure of the tumor, dividing the tumor into three distinct regions. The inner gray region represents dead cells, the cross-hatched layer represents living but non- proliferative cells, and the outer shell represents proliferative cells. This idealized tumor instructs us to develop dis- tinct growth rules for each region of the tumor. With the idealized tumor defined, we then proceed to simulate the development of tumors through several steps.

[0079] 3.2 Game of Tumor Life

[0080] We start by choosing a seed location in the organ (or tissue) of interest. This will be our starting point or the “initiation” site of the tumor.

[0081] 1. Initializing Growth. Considering the characteristics of CT images, we utilize the pixel in the CT image as the element (cell) for the cellular automata to simulate tumor growth. Within the cellular automata, there are two methods for identifying the cell’s neighbors. As shown in Figure 5, Von Neumann’s method only considers direct horizontal and vertical neighborswhile Moore’s method includes diagonal neighbors as well. To simulate tumor growth and interaction effects in 3D CT images more realistic, we employ Moore’s method for neighbor searching. Subsequently, we employ this to initialize a basic cellular automata with the uniform growth rule. The basic cellular automata with a uniform growth rule involves each cellrandomly selecting its growth direction. Figure 5 shows that this basic tumor growth simulation yields a spherical shape with identical values assigned to each cell. However, for a more precise simulation, as illustrated in Figure 6, we allocate distinct states to individual cells within the cellular automata to rep- resent varying populations of tumor cells within a single pixel. This approach enables us to simulate the interaction between tumors and organ tissues, especially including the blurry boundaries of the synthetic tumor. To simulate the interaction between tumors and organ tissues, a quantified map of organ tissue is naturally required to quantify and simplify the interaction process.

[0082] 2. Quantifying Organ. The CT image values typically range from −1000 to 1000 Hounsfield Units (HU), serving as the measure of human body density. However, this extensive spectrum of CT intensities introduces complexity when simulating the interaction between the synthetic tumor and surrounding organ tissues. To address this issue, we employ a quantification process to transform the CT image into a quantified map, as illustrated in Figure 6. The quantified map divides organ tissue into four levels from low-intensity to high-intensity:where denotes the intensity (HU) of the pixel at in the CT image, donates the quantified value of the pixel at in organ region, is the average value of the CT image in the organ region, max(H) is the max value of the CT image in the organ region, and the ∆ is the interval between each quantified level:. This simplifies the interaction process, and we formulate interaction rules based on this quantified organ map.

[0083] 3. Interacting with Environment. In the field of medicine, the interaction between a tumor and the surrounding organ tissue is a primary factor that influences tumor development. This complex process can be simplified by utilizing the quantified map of organs, which is essential for tumor synthesis. Normal tissues classified into levels with can be directly invaded by tumors. Cells with values greater than zero and less than ten can grow into normal tissues by adding one:To invade high-intensity tissues with tumors are required to apply higher pressure on the tissue boundary. Tumor cells withwill attempt to invade tissues witheach step, and the pressure is represented by the frequency of invasion attempts made by tumor cells with. The vessels and organ boundaries are tissues with the highest intensity. These regions will significantly impede the growth of the tumor. Similarly, tumor cells with a value of ten will also attempt to invade the vessels and organ boundaries at each step. When the pressure reaches the threshold, tumors can cause the extrusion and bending of vessels and organ boundaries, as illustrated by Figure 7. This simulates the mass effect resulting from the interaction between tumors and surrounding tissues. Our tumor synthesis approach incorporates the simulation of tumor cell death. The degeneration of tumor cells is determined by their size and the vascular feeding environment. When Cellular Automata have evolved for more than 50 steps (different across organs), it means that the tumor has grown large enough. In this case, the cells withcan die

[0084] 4. Mapping to CT Images. After simulating tumor growth in the quantified map through these steps, the synthetic tumor must be mapped back to CT images, as illustrated in Figure 7. The mapping approach is based on a fixed mapping function, which has no training process:where β is a preset bias value, ^ is a random noise following Gaussian distribution. denotes the synthetic CT image value of the pixel at positiondenotes the original CT image value of the pixel at position , and bias represents a range of values indicating the difference between the typical tumor CT value and the organ CT value, and noise is applied to avoid over-fitting. The map- ping function can integratethe tumor population’s state (the state value of each pixel) with the original CT intensity to calculate the synthetic tumor intensity in CT images.

[0085] 3.3 Rules of Tumor Growth

[0086] Setting: Each element (cell / pixel) in the cellular automaton is initially assigned a value between zero and ten, representing the population of tumor cells in a single pixel. The development of the tumor is guided by three summarized rules: ● Accumulation: Cells with a state value greater than zero but less than ten (10 > sx,y,z> 0) can accumulate with a certain probability. This accumulation leads to an increment of one in their own population state valuesimulating the proliferative behavior of the tumor cell. ● Growth: Cells with a state value greater than zero (sx,y,z> 0) can incentivize one of their neighbors to grow. This incentive leads to an increment of one in the neighbor’s population state valuesimulating the invasive behavior of the tumor. ● Death: Cells with a state value of ten (sx,y,z = 10), surrounded by neighbors with a state value of ten may undergo cell death with a certain probability. Following cell death, the state value is decreased to minus onesimulating the effects of crowding in the tumor, which leads to the death of tumor cells.

[0087] The rules are applied iteratively, simulating the continuous growth and development of the tumor. The tumor grows in complexity and size, occupying more space in the organ. Once the tumor has grown to a desired size or time point, the synthetic tumor can be studied for its shape, interaction with surrounding tissue, and its resemblance to real tumors. Such tumor synthesis holds immense potential in training AI models. By generating tumors of various sizes and stages in a vast number of unlabeled CT images, we can equip AI systems with a rich training set, improving their accuracy and potential for early cancer detection.

[0088] 3.4 Tumor Development: Clinical Perspectives

[0089] Tumors, genetic disorders arising from DNA mutations in a single cell, undergo complex growth processes. Mutations inherited during cell division lead to uncontrolled proliferation, forming neoplastic lesions that can be benign or malignant. Key distinguishing features between benign and malignant tumors include growth rate and invasiveness. Malignancy is often associated with rapid growth. For example, pancreatic branch duct intraductal papillary mucinous neoplasm (IPMN) malignantlesions tend to exhibit larger final sizes, greater growth per- centages, and faster growth rates compared to benign lesions. Slow growth rates in renal tumors have been linked to low malignant potential, a pattern also observed in glioma and hepatocellular carcinoma. Invasiveness distinguishes malignant tumors, involving gradual infiltration, invasion, and destruction of surrounding tissues. In contrast, benign tumors grow as cohesive, expansile masses, typically confined to their original sites. Even slowly growing malignant tumors may penetrate margins and infiltrate neighboring structures, contributing to the indistinct margins seen in CT images. Therefore, we have designed an Accumulation and Growth rule to simulate these two features. Tumor necrosis, a form of cell death, indicates a worse prognosis. Histologically, necrosis results from hypoxia caused by rapid cell proliferation outpacing vascular supply, manifesting as non- enhancing irregular areas in CT images. The Death rule is designed to represent this biological phenomenon. Considering these features, we propose a hybrid cellular automaton model to simulate tumor development from in- dividual cells to invasive tumors, capturing the continuous progression within their microenvironment.

[0090] 4. Experiments

[0091] 4.1 Dataset

[0092] Datasets for Evaluation: Per-pixel annotations for liver tumors are provided in LiTS. We employ 5-fold cross-validation, with each fold using 101 CT images for the training set and 22 CT images for the testing set. Annotated pancreas tumors are available in MSD (Task07 Pancreas). The training set includes 96 CT images, and the testing set includes 24 CT images. Kidney annotations are provided by KiTS. We utilize 96 CT images for the training set and 24 CT images for the testing set. Importantly, these data show no significant accompanying changes in organs caused by tumors, suggesting that the structure of cancerous organs remains similar to that of healthy organs.

[0093] Datasets for Healthy CT: We apply Pixel2Cancer tumor synthesis to healthy CT images. The healthy liver dataset comprises 116 CT images with healthy livers assembled from CHAOS (20 CT images), BTCV (47 CT images), Pancreas-CT

[0072] (38 CT images), and healthy subjects in LiTS (11 CT images). The healthy pancreas dataset includes 104 CT images assembled from Pancreas- CT (80 CT images) and BTCV (24 CT images). The healthy kidney dataset consists of 120 CTimages assembled from BTCV (11 CT images), WORD (16 CT images), and Abdomenct-1k (93 CT images). Table 1. Results of Radiologist Study. The test was conducted with three medical professionals having 7, 9, and 14 years of experience, respectively. Each professional evaluated 50 CT images for each organ, consisting of both real and synthetic tumors. They were tasked with categorizing each CT image as either real, syn- thetic, or unsure. “Synt” denotes synthetic tumors, while P and N are used to denote positive and negative classes for calculating Sensitivity and Specificity. Table 2. Comparison with state-of-the-art methods across organs. We compare our methods with the state-of-the-art synthesis approach and real data. Our method significantly outperforms the SOTA synthesis approach and even surpasses the real tumor data with detailed pixel-wise annotation. The higher the DSC and NSD values, the more accurate the segmentation. Conversely, lower SD and HD values indicate more accurate segmentation of the boundary. The evaluation data exhibit no significant changes in tumor-affected organs, indicating similarity to healthy organs.

[0094] 4.2 Evaluation Metrics

[0095] Tumor segmentation performance was evaluated using the Dice Similarity Coefficient (DSC), Normalized Surface Dice (NSD), Surface Distance (SD), and robust Hausdorff Distance (HD). Tumor detection performance was assessed through Sensitivity and Specificity.

[0096] 4.3. Implementation

[0097] Cellular Automata: To optimize tumor development simulation, we implemented a 3D Cellular Automata package using CUDA (Compute Unified Device Architecture). In the 3D space, the kernel size of the cellular automata is 3×3×3, and the growth frequency of cells is set to 1 in each iteration. When conducting multi- processing, we set the number of blocks to 1024 and threads to 64. Compared to standard Cellular Automata implementations on the CPU, our CUDA-based implementation achieves a speedup of 500×. This significantly improves the efficiency of tumor synthesis and model training.

[0098] Segmentation: We implement segmentation code based on the MONAI framework with U-Net and Swin UN- ETR. Input images are clipped within the window range of [-21, 189] and then normalized to have a zero mean and unit standard deviation. During training, random patches of size 96 × 96 × 96 are cropped from 3D CT images. All models are trained for 2,000 epochs with the base learning rate of 0.0002. The batch size is two per GPU. We employ the linear warm-up strategy and the cosine annealing learning rate schedule. For inference, we use the sliding window strategy with an overlapping area ratio of 0.75.

[0099] 5. Results & Discussion

[0100] Summary. By utilizing healthy CT images for tumor synthesis, Pixel2Cancer achieves comparable performance with real tumors across the liver, pancreas, and kidney. Moreover, Pixel2Cancer indicates its potential for data augmentation, lifelong learning, small tumor detection, and accurate boundary segmentation.

[0101] 5.1. Clinical Validation

[0102] We conducted Visual Turing Test on 50 CT images. In the liver, 25 images are with real tumors from LiTS, and the remaining 25 images are healthy livers from WORD with synthetic tumors. In the pancreas, 25 images are real tumors from MSD, and the remaining 25 images are healthy pancreases from Pancreas-CT withsynthetic tumors. In the kidney, 25 images are with real tumors from KiTS, and the remaining 25 images are healthy kidneys from Abdomenct-1k with synthetic tumors. The results in Table 1 reveal performance metrics for different radiologists. For the junior radiologist #1 (7 years of experience), accuracy, sensitivity, and specificity all fall below 40%. Particularly, a specificity of 35.5% im- plies that 64.5% of synthetic tumors are wrongly identified as real. Intermediate radiologist #2 (9 years of experience) shows similar metrics around 40%, with 59.2% of synthetic tumors causing confusion. Even the senior radiologist #3 (14 years of experience) misidentifies 44.1% of synthetic tumors as real, highlighting the challenge even for experienced professionals. This underscores the realistic simulation of tumor development achieved by Pixel2Cancer.

[0103] 5.2. Performance of Pixel2Cancer

[0104] Label-free Tumor Segmentation Potential: We bench- mark Pixel2Cancer against the state-of-the-art modeling- based method and real tumor data. Table 2 demonstrates Pixel2Cancer’s superiority in liver segmentation with a DSC of 57.2%, NSD of 62.9%, SD reduction of 14.7%, and HD reduction of 22.8%. We also achieve comparable performance with real tumors in the pancreas and kidney, with DSCs of 36.5% and 28.6%, respectively. Compared to models trained on real liver tumors, our Pixel2Cancer outperforms by 4.9% in NSD. We also sur- pass Hu et al.’s method by 2.7% in DSC and 5.3% in NSD.

[0105] Superiority in Boundary Segmentation: Pixel2Cancer generates synthetic tumors with absolutely precise tumor masks, whereas real data annotations are often inaccurate at the boundaries, leading to label noise and challenges in boundary segmentation accuracy. In Table 2, we utilize distance metrics, including NSD, SD, and HD. Our Pixel2Cancer synthesis approach surpasses real liver tu- mors, achieving a 4.9% improvement in NSD and notable reductions of 22.9% in SD and 14.7% in HD. These results highlight the precision of our Pixel2Cancer in boundary segmentation, indicating its potential for surgical guidance, particularly in tumor excision procedures. Table 3. Effectiveness as data augmentation. We apply our synthesis approach Pixel2Cancer to healthy CT images and integrate the generated synthetic tumors with real tumors as data augmentation. As data augmentation, our synthesisapproach significantly enhances performance across organs. “Pixel2Cancer” represents using solely synthetic tumors, and “Pixel2Cancer & real” represents the combi- nation of real and synthetic tumors. Table 4. Ablation study on our generic rules. We evaluate the performance at each stage of our simulation process, starting from basic uniform growth rules, then considering the interaction be- tween the organ and tumor, and finally incorporating death rules for the tumor as it develops larger.

[0106] Data Augmentation and Lifelong Learning: We combine real tumors from real cancerous CT images and synthetic tumors from our Pixel2Cancer to evaluate the performance of using synthetic tumors as data augmentation. In Table 3, our Pixel2Cancer performs 58.9% in DSC, 63.7% in NSD, and achieves a reduction of 11.6% in SD and 8.1% in HD. The significant improvements in NSD by 5.7%, 2.2%, and 7.8% across organs demonstrate the effectiveness of using synthetic tumors as data augmentation. The second scenario involves handling a continuous stream of data within the context of lifelong learning, where the model encounters an ongoing influx of new data. We propose a novel framework for continual learning that integrates a continuous stream of synthetic data, characterized by diverse data distributions. This suggests the potential application of life- long continued learning.

[0107] Ability in Small Tumor Detection: Early detection of small tumors is essential for timely cancer diagnosis, but real datasets often lack sufficient instances due to the asymptomatic nature of patients in the early stages. Figure 8 illustrates the performance of small tumor detection, presenting three cases each of small liver, pancreas, and kidney tumors. Training solely on our synthetic data outperformsmodels trained on real tumors and Hu et al.’s synthetic data approach. This highlights that training on synthetic small tumors enhances the efficacy of the model in detecting real small tumors, which is crucial for early cancer detection.

[0108] Ablation Study on Rules Application: We conducted experiments to evaluate the influence of our generic rules and design on liver tumors. As illustrated in Table 4, the basic cellular automata with uniform growth rules only achieve 43.0% in DSC and 52.1% in NSD, indicating an ineffective simulation of tumor appearance in the organ. Subsequently, we introduce rules that account for the interaction between the tumor and the organ, resulting in our synthetic data achieving comparable performance to real data. Lastly, as the tumor develops further, we implement a death rule for the tumor. Our Pixel2Cancer finally outperforms the models trained on real tumor data by 4.9% in NSD, and achieves a reduction of 14.7% in SD. This demonstrates the effectiveness of each part of our generic rules.

[0109] 6. Conclusion & Discussion

[0110] In this example, we designed generic rules based on medical knowledge to simulate tumor behaviors using cellular automata. Our results demonstrated Pixel2Cancer’s potential for data augmentation, lifelong learning, small tumor detection, and accurate boundary segmentation. Despite simulating tumor behaviors and interactions with organ tissues, including the bending of vessels and the deformation of organ boundaries (Figure 7), there are still various mechanisms of accompanying changes in organs caused by tumors that we have not yet considered. Examples include main pancreatic duct dilatation, focal pancreas parenchymal atrophy, hepatic capsular retraction, and splenomegaly. In future work, we plan to devise organ rules within our approach to accurately simulate changes induced by tumors when generating synthetic tumors.

[0111] EXAMPLE 2: Towards Generalizable Tumor Synthesis

[0112] 1. Introduction

[0113] Tumor synthesis enables the creation of artificial tumor examples in medical images, facilitating the training of AI models for tumor detection and segmentation. Synthetic tumors are particularly valuable when there is a scarcity of per-voxel annotated real tumors for effective AI training. Typically, to train AI models for tumor detection in multiple (N) organs, annotated real tumor examples from each ofthese organs are necessary, and ideally, in substantial numbers. Furthermore, AI models of- ten fail to generalize across images from different hospitals, which may vary due to variations in imaging protocols, patient demographics, and scanner manufacturers. The challenge amplifies with the need for extensive manual annotations, a task that could demand up to 25 human years for annotating just one tumor type. The task of collecting and annotating a comprehensive dataset encompassing tumors from multiple organs (N ) and images from numerous hospitals (M ) is daunting, considering both annotation cost and complexity (N × M ). We hypothesize that tumor synthesis could solve this challenge by creating various tumor types across non-tumor images from multiple hospitals, even when only one tumor type is available, thereby simplifying the complexity from N × M to 1 × M .

[0114] Success in tumor synthesis hinges on creating visually realistic tumors that are generalizable across multiple organs and, furthermore, the resulting AI models being generalizable in detecting real tumors in images sourced from different hospitals. Previous studies have introduced generative models to create synthetic medical data (not limited to tumors) such as polyp detection from colonoscopy videos, COVID-19 detection from Chest X-ray, and diabetic lesion detection from retinal images. However, these studies have primarily focused on enhancing the detection and segmentation of specific tumors without fully exploring the wider generalizability of these models across different organs and patient demographics.

[0115] This example made a progressive stride toward generalizable tumor synthesis by leveraging a critical observation: early-stage tumors (< 2cm) tend to have similar imaging characteristics in computed tomography (CT). Note that, owing to the dataset constraints, we have only verified the similarity across early hepatocellular carcinoma and intrahepatic cholangiocarcinoma from the liver, pancreatic ductal adenocarcinoma from the pancreas, renal cell carcinoma with three histological subtypes (clear cell, papillary and chromophobe renal cell carcinoma) from kidneys. Early-stage tumors typically present small, round, or oval shapes with minimal deformation and exhibit relatively simple and uniform textures in CT volumes. Hence, early tumors in parenchymal organs (e.g., liver, spleen, pancreas, adrenal glands, and kidneys) should appear similarly, as shown in Figure 9. The major difference is the contrast between the tumors and background organs or other anatomical structuresrather than the tumors themselves. Using four public datasets and our proprietary datasets, this example also verifies the similarity of early-stage tumors across various organs.

[0116] Leveraging this observation, we introduce a novel frame- work, termed DiffTumor, that can learn the common imaging characteristics of tumors across various organs, and the generated synthetic tumors are useful for training AI models to detect and segment real tumors from CT volumes of varying patient demographics. The development of DiffTumor is composed of three stages. ① Training an Autoencoder Model—consisting of an encoder and decoder—on 9,262 unlabeled three- dimensional CT volumes. The use of large, diverse datasets can enhance the model’s ability to generalize across CT volumes of different patient demographics and reduce the need for annotated tumor volumes for training Diffusion Models in the subsequent stages. The proxy task is image reconstruction, which facilitates the model in learning comprehensive latent features. ② Training a Diffusion Model—a specific type of generative models—using latent features and tumor masks as conditions. Once trained, this model can generate latent features necessary for reconstructing CT volumes with tumors based on arbitrary masks. ③ Training a Segmentation Model using synthetic tumors, which are reconstructed by the decoder, and their corresponding masks. With a large repository of healthy CT volumes, our DiffTumor framework can pro- duce a vast array of synthetic tumors, varying in location, size, shape, texture, and intensity, therefore fostering high- performing AI models for tumor detection / segmentation.

[0117] Exemplary key contributions of this example are two-fold. Firstly, we have verified with feature analysis, reader studies, and clinical knowledge that early-stage tumors (< 2cm) mani- fest with similar imaging characteristics across various organs in CT volumes, establishing the foundation for the development of generalizable tumor synthesis. Secondly, we have developed a three-stage tumor synthesis framework, DiffTumor, that trains generative models with minimal an- notations (Figure 13; one annotated CT volume), creates synthetic tumors in real-time (Figure 14; 100 ms / tumor), and improves early-stage tumor detection (Figure 15; improved sensitivity up to +28.6%). In summary, compared with training AI on extensively annotated CT volumes of real tumors, our DiffTumor is generalizable from two critical perspectives.1. DiffTumor can create visually realistic tumors generalizable to a range of organs even when the diffusion model was trained on a limited number of tumor examples from a specific organ (§4.2; +10.7% DSC). 2. DiffTumor can develop an AI model to detect and segment real tumors generalizable to a variety of CT volumes of varied patient demographics, imaging protocols, and healthcare facilities (§4.3; +9.1% DSC).

[0118] 2. Preliminary

[0119] We observe that early-stage tumors (< 2cm) often share similar imaging characteristics in CT volumes, whether they originate in the liver, pancreas, or kidneys. Based on the TNM system, the most widely used staging system for classifying a malignancy tumor, we recognize a primary malignant tumor with a diameter less than 2 cm and no evidence of nearby lymph node involvement or metastasis as an early- stage tumor. We select three abdominal organs (liver, pancreas, and kidney) for demonstration using publicly available datasets. Specifically, LiTS and MSD-Hepatic Vessel provide liver tumors (hepatocellular carcinoma, intrahepatic cholangiocarcinoma, and metastatic tumors from colorectal, breast, and lung primary cancers); KiTS provides kidney tumors (renal cell carcinoma and oncocytoma); and MSD-Pancreas provides pancreatic tumors (pancreatic ductal adenocarcinoma). We also created a large-scale private dataset to augment the number of pancreatic tumors. This observation, when confirmed, could have profound implications for generative AI in medical imaging. It suggests that generative AI might be trained on one tumor type, for which data and annotations are more easily obtained, and then applied to create various tumor types in different organs, where acquiring sufficient data can be challenging. Using synthetic tumors can substantially improve AI performance in tumor detection and segmentation in practice. In light of this, we have rigorously pursued the validation through three approaches as follows.

[0120] (1) Radiologist reader study. The objective of the reader study is to assess the ability of radiologists to recognize the organ class of early cancer. We uniformly crop 360 CT images of the tumor region from three abdominal organs, as per the annotations. In order to exclude the in- fluence of surrounding organ textures on recognition, we only retain a small amount of organ textures in the tumor boundary region. Examples of CT crops used for the reader study are provided in Figure 10.Three expert radiologists, qualified under the Quality Standards Act, participate in the reader study. The recognition results are shown in Figure 10(a). The nearly random probability of the precision and recall scores indicates that the appearance of early- stage tumors is so similar that even experienced radiologists have difficulty distinguishing the organ types of these tumors.

[0121] (2) Radiomics feature analysis. We now analyze the similarity in Radiomics features of early-stage tumors. We utilize the official radiomics feature repository to extract the appearance features, which include 3D shape-based features, gray level co-occurrence matrix, gray level run length matrix, gray level size zone matrix, neighboring gray-tone difference matrix, and gray level dependence matrix. Quantitatively, we train a support vector machine (SVM) classifier to identify the organ types of early-stage tumors. To draw a general conclusion, we conducted ten repeated experiments and calculated the precision and recall of the SVM classifier in both the training and test sets. The final results show that the precision and recall for the training set are close to 1, indicating that the SVM is well-trained and capable of learning a decent decision boundary for the training set. However, the precision scores for the test set are nearly equivalent to random probability, as shown in Figure 10(a). This suggests that even a well-trained SVM classifier struggles to recognize the organ types of unseen early-stage tumors. Qualitatively, Figure 10(b) visualizes the feature mapping in a two-dimensional space using t-SNE. The appearance features of early- stage tumors are distributed in a joint feature space, and there is no separation for different organ types.

[0122] (3) Clinical evidence and justification. Tumorigenesis is a gradual, multi-step transformation process occurring at both the cellular and histological levels. This process in- volves the successive progression of advanced precancerous, early cancerous, and overtly malignant lesions. Histologically, early-stage tumors typically consist of well- to-moderately differentiated neoplastic cells exhibiting mild atypia, with hemorrhage and necrosis being rare occurrences. Therefore, early-stage tumors from different parenchymal organs, such as the liver, pancreas, spleen, adrenal gland, and kidney, tend to display similar imaging features. These features are consistent across different populations, ages, and genders and typically include a relatively homogeneous, nodular appearance with indistinct margins and small diameter in CTvolumes. This similarity suggests the feasibility of learning shared imaging characteristics of tumors across different organs and achieving generalizable tumor synthesis. ● Liver tumors: Early hepatocellular carcinoma (HCC) is characterized by a vaguely nodular small, well- differentiated HCC with a good prognosis and minimal metastatic potential. In small lesions of HCCs, active neoangiogenesis progresses from more unpaired arteries and reduces portal triads, leading to isoattenuation or hypoattenuation compared to the surrounding liver parenchyma (wash-out) during the venous phase. ● Pancreatic tumors: Multiphase CT with intravenous contrast medium is a preferred diagnostic test for suspected pancreatic lesions. The majority of pancreatic ductal adenocarcinomas (PDACs) typically exhibit poor enhancement, appearing hypoattenuating relative to the surrounding pancreatic parenchyma. This hypoenhancement is attributed to the development of a dense fibroblastic stroma component in PDACs. ● Kidney tumors: For suspicious renal masses, CT is considered the gold standard in evaluating renal cell carcinoma (RCC). The most prevalent subtype, clear cell RCC, constitutes 80% of all RCCs. It typically presents as a small hypoattenuating renal lesion surrounded by homogenously enhancing renal parenchyma in the nephrographic phase.

[0123] 3. DiffTumor

[0124] 3.1 Autoencoder Model

[0125] Directly applying generative diffusion models on high- dimensional data like 3D CT volumes can require significant computational costs. To reduce the cost, Latent Diffusion Models (LDMs) train their diffusion models in a compressed lower- dimensional latent space. Inspired by LDM, we also build our diffusion model in the latent space of 3D CT volumes. The first step is, therefore, to train a 3D Autoencoder that encodes 3D CT volumes into a meaningful and compressed latent representation, where we mainly adopt the vector quantized Generative Adversarial Networks (VQGAN) architecture but replace its 2D convolution operations with 3D convolutions.

[0126] Formally, we denote a CT sub-volume aswhere H denotes the height, W the width, and D the depth. The CT sub-volume The CT sub-volumeis first converted to latent features by an encoder and a quantization operation i.e., where denotes the feature height, the feature width, and the feature depth. In the vector quantization step, the latent features are quantizedby replacing each one with its closest corresponding codebook vector in the learned codebook is the codebook size. Finally, a decoder reconstructs the latent fromThe loss is a summation of three terms:denotes the stop-gradient operation and the coefficient.

[0127] In addition to these three loss terms, we also adopt a perceptual loss and a discriminator to improve the reconstruction quality. For 3D CT reconstruction, we adopt a 3D volume discriminatorto penalize implausible artifacts for the 3D reconstruction ofand a 2D slice discriminatorto encourage per-slice quality. To stabilize the GAN training, we add the feature matching losses . Moreover, due to the CT volumes being preprocessed to isotropic volume, we constrain the high- frequency texture for all three planes of by using perceptual loss for projected reconstruction slicesoverall objective of the Autoencoder is:

[0128] 3.2. Diffusion Model

[0129] We aim to synthesize realistic and diverse CT volumes with tumors to facilitate the training of the tumor segmentation model. Given the fact that healthy CTvolumes are much more accessible than CT volumes with tumors, we focus only on tumor synthesis, and we do not intend to model organ textures outside of the tumors, which can be easily obtained from healthy CT volumes. To be specific, our dif- fusion model is conditioned on both a tumor mask that indicates the shape and location of tumors in the latent feature and the healthy region of CT volumes.

[0130] Formally, given a pair of tumor-present CT volume and the mask of its tumor region , the diffusion model is conditioned on both the tumor mask and the healthy region. The diffusion model approximates the distribution of the latent features of tumor-present CT volumes. In the forward process, the latent feature is gradually converted to white Gaussian noiseby recursively adding a small amount of Gaussian noise T times following the Markov process below:the timestep and is the variance schedule of noise.

[0131] In the inference, we synthesize the latent feature of CT volumes by sampling from which is approximated by recursively sampling from. The training objective of our diffusion model is as follows:where is a 3D U-Net with interleaved self-attention layers and convolutional layers that predicts the noise given the input. To reduce the heavy computational cost for 3D CT volumes, we factorize the self-attention over the entire 3D data to first only attend to each 2D slide and then attend to the depth dimension, inspired by 3D video Transformers. This design largely reduces the computation cost of the self- attention layers in the 3D U-Net.

[0132] 3.3. Segmentation Model

[0133] To synthesize generalizable CT volumes with tumors, we take advantage of various healthy CT volumes and synthesize tumor masks as conditions for our diffusion model. Healthy CT volumes are abundant and easily available in public datasets. We construct a large-scale healthy CT database by filtering out healthy organs in public datasets. The database consists of 1246 CT volumes with healthy livers, 1901 CT volumes with healthy pancreas, and 1005 CT volumes withhealthy kidneys, covering diverse subjects of different ages, genders, and countries. It also includes CT volumes acquired from various scanners, hospitals, and imaging protocols. Inspired by prior work, we generate tumor-like shapes with ellipsoids and adjust the elastic deformation according to feedback from professional radiologists to guarantee the tumor shape is clinically reasonable. Therefore, by combining the generated tumor masks and diverse healthy CT as condition (Figure 11), we can generate various tumors from different domains to achieve generalizable tumor synthesis.

[0134] 4. Experiments & Results

[0135] Real-tumor datasets. LiTS, MSD-Pancreas, and KiTS were used for training and testing Segmentation Models on the liver, pancreas, and kidneys, respectively. We performed 5-fold cross-validation on 118 tumor CT volumes for LiTS and 120 tumor CT volumes for MSD- Pancreas and KiTS.

[0136] Healthy CT datasets. We collect a large repository of healthy CT volumes for liver, pancreas and kidney. Due to the computational cost and memory limitation for training, we only random selected 120 healthy CT volumes for kidney and pancreas, respectively. For liver, we adopt the same healthy CT volumes as in previous work.

[0137] 4.1 Visual Turing Test

[0138] We conduct the Visual Turing Test on 60 CT volumes for three organs, respectively, where 30 volumes are with real tumors, and the remaining 30 volumes with synthesized tumors by our method. Two professionals are involved in this test, with 4 and 11 years of experience, respectively. Following previous work, each sample is inspected in a 3D view to be classified as either real or synthetic, allowing for the observation of a continuous slice sequences. The testing results are shown in Table 1. Radiologists R1 and R2 can both identify real tumors with a high sensitivity score (above 80%). This indicates their familiarity with the characteristics of real tumors. However, R1’s near-zero specificity scores indicate that the synthetic data strongly resembles real tumors, leading to most synthetic tumors being misidentified as real ones. This results in R1’s accuracy scores hovering around 50%. As for R2, who has more experience, the specificity scores are higher than that of R1, approximating 50%. This suggests that nearly 50% of synthetic samples are still incorrectly identified asreal samples. These results confirm the efficacy of DiffTumor in generating visually realistic tumors. Table 1. Visual Turing Test over three organs has been conducted with two radiologists (R1 and R2). Both radiologists are provided with 60 three-dimensional CT volumes of each organ, including 30 scans with real tumors and the remaining 30 with synthetic ones. Radiologists are tasked to label each CT volume as real or synthetic. A lower specificity score indicates a higher number of synthetic tumors being identified as real.

[0139] 4.2. Generalizable to Different Organs

[0140] DiffTumor can generate visually realistic tumors generalizable to a range of organs although Diffusion Model only trained on a specific organ tumor. In order to verify the effectiveness of our method’s generalization capacity across different organs, we conducted comparative experiments across three different abdominal organs. This involved training all Segmentation Models on tumor data from a single organ, and then applying that training to the other two organs. For the results of our method, we train DiffTumor on source organ data, then utilize healthy CT volumes to synthesize tumors in the target organ, which are used for further training of Segmentation Models. To showcase the broad applicability of our synthetic data, we compare the early-stage tumor detection capabilities across three commonly used backbones. The generalization result, shown in Table 2, suggests that it is difficult for Segmentation Models trained on real data to generalize across different organs, leading to poor performance in early-stage tumor detection. Previous work introduces a modeling-based method, which can maintain consistent sensitivity scores for the same target, regardless of the source domain. The generalization ability of DiffTumor across organs surpasses that of most models, except in the setting that generalizing tumors from kidney to liver. Moreover, we demonstrate the strength of DiffTumor used as an augmentation method for real tumors in the same organ, as shown in Table 2.In particular, there is a notable improvement of 10.7% in the Dice Similarity Coefficient (DSC) for kidney tumors when using nnU-Net backbone. Furthermore, a decrease in the standard deviations of the DSC scores suggests that the Segmentation Models become more stable. The significant improvement in DSC scores across all three organs proves that DiffTumor is an effective data augmentation method to enhance performance of Segmentation Model. Table 2. Generalizable to different organs: comparison of generalization for early- stage tumor detection under different source organs. The scores in bold represent the best performance in each domain. DiffTumor achieves the best performance in almost all domains. Furthermore, DiffTumor serves as an effective data augmentation method for real tumors in three abdominal organs, yielding substantial improvements in all-stage tumor segmentation.

[0141] 4.3. Generalizable to Different Demographics

[0142] The ability of Segmentation Model to be generalizable to different demographics is critically important. It indicates that the model can effectively process CT scans from a diverse population, including various ages, genders, and ethnicities. To affirm the enhancement of DiffTumor for Segmentation Model to detect and segment real tumors across different individuals, we evaluate the generalization ability of Segmentation Model using a private dataset, JHH. This dataset includes various real pancreatic tumors (PDAC and Cyst) from diverse patient demographics. We utilizeDiffTumor with Diffusion Model trained on MSD-Pancreas to enhance Segmentation Model. Figure 12 shows that our synthetic data can yield an average improvement of 6.9% in DSC and 16.4% in sensitivity with the U-Net backbone. In particular, the improvement for people aged 50–60 is significant, with an enhancement of 18.9% in sensitivity and 9.1% in DSC. For both males and females, there are noticeable performance improvements for tumor detection and segmentation. These results demonstrate that our synthetic data can provide valuable assistance in clinical tumor analysis for individuals across various age groups and genders.

[0143] 4.4. Advantages of DiffTumor

[0144] (1) Reduced annotations for Diffusion Model. The quality of synthetic data produced by a generative model is typically heavily reliant on the quantity and diversity of the paired training data used during the training phase. We study the relationship between the number of annotated real tumors needed for the Diffusion Model and the performance of the Segmentation Model. We find that the relationship between the amount of paired training data and the quality of synthetic data isn’t always linear, as shown in Figure 13. In particular, DiffTumor only requires just one annotated tumor to train the Diffusion Model and generate synthetic tumors for the subsequent training of Segmentation Model. This contradicts the typical experience in computer vision, which generally requires large-scale data for training. The results indicate that for training the Diffusion Model, particularly for early tumors, we can rely on a smaller number of real tumors. This finding could have important implications for the efficiency and cost-effectiveness of training DiffTumor.

[0145] (2) Accelerated tumor synthesis. The speed of tumor synthesis plays a crucial role in the practical application of synthetic data. Real-time synthes can significantly speed up the training process of Segmentation Model. The speed of generating synthetic tumors in Diffusion Model is significantly influenced by the timestep T. We examine the impact of the timestep on the performance of Segmentation Model. The synthetic quality using DDPM sampling with different timestep is illustrated in Figure 14. As can be seen, when T = 1, the model collapses and fails to synthesize realistic textures for both the organ and tumor textures. Consequently, using these synthetic data to train the Segmentation Model results in poor performance. However, when T is increased to more than 1, the corresponding texture can be well-generated,leading to good performance in the Segmentation Model. In consideration of the trade- off between performance and efficiency, we default to a timestep of T = 4 for early tumor synthesis. This balance allows for the generation of high-quality synthetic data while maintaining a reasonable efficiency level.

[0146] (3) Improved early tumor detection. Detecting tumors in their early stages can greatly increase the chances of successful treatment and survival. However, obtaining early- stage cancer data is challenging in practice and such cases in real datasets remain scarce. This limits the AI model’s ability to detect early tumors. As shown in Figure 15, there are several failure cases for Segmentation Model trained on real data. However, with the incorporation of our synthetic data, the Segmentation Models’ capability to detect early-stage tumors improves significantly. This is one of the primary reasons why DiffTumor can achieve the best performance as displayed in Table 2. This demonstrates the value and efficacy of synthetic data in enhancing early tumor detection.

[0147] 5. Related Work

[0148] Generative models such as Energy-Based Models, Variational Autoencoders (VAE), Generative Adversarial Networks (GAN), and normalizing flows have shown significant potential in creating realistic images. Among these, Diffusion Models and their variants have recently emerged as particularly advanced in image generation. In the medical field, generative models have been effectively utilized for tasks like image-to-image translation, reconstruction, segmentation, image denoising, and anomaly detection. In this work, we focus on generating tumors in abdominal organs based on the textures of the surrounding organs, which significantly reduces the annotated data required for training.

[0149] Tumor synthesis that is widely effective for a variety of organs is an attractive topic. Successful works related to tumor synthesis based on various medical modalities include colon polyp synthesis in colonoscopy videos, tumor cell synthesis in fluorescence microscopy images, synthesized brain tumors in MRI for pre-training, and lung nodule synthesis in CT images. Additionally, many works on synthesizing non-cancerous lesions have been explored, such as COVID-19 lesion synthesis in chest CT, and diabetic lesion synthesis in retinal images. Recent studies have improved the realism of synthetic tumors in the liver and pancreas. AI trained on thesesynthetic tumors performs similarly well as those trained with real tumors. However, these methods need to be redesigned for tumors in other organs, which severely limits the generalization capabilities. In this example, we learn the tumor distribution based on Diffusion Models to realize generalizable tumor synthesis.

[0150] 6. Conclusion

[0151] This example introduces DiffTumor for achieving generalizable tumor synthesis. Early-stage tumors exhibit similar imaging characteristics in CT scans, regardless of whether they are located in the liver, pancreas, or kidneys. Taking advantage of this observation, we develop DiffTumor based on a few annotated tumors in a specific organ, such as the liver. This AI model, solely trained on liver tumors, can directly generate synthetic tumors in organs where CT volumes of annotated tumors are relatively rare, such as the pancreas and kidneys. By incorporating these synthetic tumors into extensive CT volumes of healthy organs— commonly collected in clinical settings—we can substantially expand the training set for tumor segmentation. This augmentation greatly enhances the AI’s generalizability across CT volumes obtained from diverse hospitals and patient demographics.

[0152] EXAMPLE 3: Synthetic Data as Validation

[0153] 1. Introduction

[0154] Standard AI development divides the dataset into a training set and a test set; the former is used for model training and the latter for evaluation. The AI model is updated every training epochs, resulting in a number of intermediate models during the training trajectory. The performance of these models tends to improve on the training set, but this does not mean that the performance on the test set also improves due to the over-fitting problem. A question then arises: How do we identify the best model that performs well on the test set, especially when it is evaluated on test sets taken from different domains? A prevalent strategy is to delineate a validation set from the training set. This validation set neither contributes to training nor to evaluating the AI performance. Instead, it functions as an independent set to fix the training hyper- parameters and, more importantly, to estimate the performance of each model on different datasets, thus enabling the selection of the best model from the many intermediate models during the training trajectory.

[0155] The validation set is often kept small. Naturally, we would like to maximize the use of the training data. Annotating data for AI training is time-consuming and expensive, requiring specialized expertise, so the annotated datasets are limited in size in many fields. Allocating too many annotated data for validation would inevitably diminish the training set size and compromise the AI training. On the other hand, the validation set should be sufficiently representative to provide a reliable performance estimate on unseen data. An overly small validation set might risk the reliability of performance estimation and checkpoint selection. As a result, the calibration of the validation set remains largely empirical and lacks systematic investigation for better alternatives to select the best checkpoint. Fulfilling this knowledge gap is particularly important in scenarios where real-world data are scarce, sensitive, or costly to collect and annotate, as seen in the field of AI for healthcare. Therefore, this example uses early detection of cancerous tumors in computed tomography (CT) volumes as a demonstration. While early detection of cancer holds immense clinical potential, it faces profound constraints like disease prevalence and annotation difficulty to collect examples of early-stage tumors. The scarcity of annotated early cancer not only constrains the data available for validation but also amplifies the overfitting problem inherent in a small, biased validation set, potentially causing underdiagnosis and overdiagnosis.

[0156] We propose using synthetic data as validation, a strategy that guarantees the full utilization of the training set while ensuring ample data diversity for validation. Data synthesis has held longstanding interest and presents numerous intriguing merits for augmenting training and test data as reviewed in 2, but its use in validation has seldom been explored. We find that synthetic data can facilitate a more reliable performance estimate on unseen data and effectively address the constraints commonly associated with small, biased validation sets. Specifically, we synthesize tumors in the healthy liver, which gives us orders of magnitude larger datasets for training. To ensure the realism of the synthetic tumors, we employ a modeling-based strategy (Hu et al., 2023) to simulate cancerous tumors with controlled shape, size, texture, location, and intensity. The use of diverse, healthy CT volumes, supplemented with synthetic tumors, as validation has demonstrated efficacy in mitigating model overfitting and enhancing the selection of checkpoints. Furthermore, we relieve thepressing demand for human annotations to train AI models by utilizing CT volumes with synthetic tumors as the training set. We then assess the model’s performance using a substantial number of publicly available, fully-annotated CT volumes with real- world cancerous tumors, showing that our models generalize well to these volumes from different hospitals and accurately segment the tumors at their early stage. Our findings can be summarized as follows: 1. The best model checkpoint, selected by standard AI development with an in-domain real-tumor validation set, may not necessarily be generalized to unseen data, especially for an out-domain test set. This limitation arises from the validation set failing to adequately represent corner cases. 2. The best model checkpoint, selected by our strategy with a diverse synthetic-tumor validation set, tends to be generalized well to unseen data. This is because the validation set can cover theoretically infinite examples of possible cancerous tumors across diverse conditions. 3. We introduce a novel continual learning framework. This framework integrates a continuous stream of synthetic data, characterized by diverse data distribution, for both training and validation. Traditional validation sets, constrained by static and limited in-domain real tumors, fall short in such a setting, whereas our synthetic tumors can be dynamically tailored to align with emerging distributions. Importantly, our framework can continuously generate tumors spanning a spectrum of sizes—from small to large—enhancing the detection rate of tumors at their early stages.

[0157] Although this example focuses on AI in healthcare, the insight should be pertinent to various imaging applications within the field of computer vision. However, at the time this example is written, very few studies in computer vision have provided evidence that training exclusively on generated synthetic data can match or surpass the performance achieved when trained on real data. In specific applications, integrating synthetic data with real data—essentially acting as data augmentation— has been found empirically to boost AI performance. In this regard, data synthesis— cancerous tumor synthesis in particular—in medical imaging is relatively more successful1 with specific applications benefiting more from training exclusively on synthetic data than real data. The greater success of data synthesis in medical imaging(reviewed in §2), compared with computer vision, can be attributed to two factors from our perspective. Firstly, the focus is primarily on synthesizing tumors rather than other components of the human anatomy. Secondly, the synthesis of tumors in 3D medical images is less complex as it does not require considerations for intricate variables such as lighting conditions, pose, and occlusion, which are typical in computer vision tasks.

[0158] 2. Related Work

[0159] The dilemma of validation.

[0160] In the field of machine learning, it is customary to use finite, static datasets with a pre-defined data split. While this standard offers a fair benchmark for comparing different AI models, it does not accurately represent real-world learning conditions. Two more realistic scenarios often arise in practice. ● The first scenario is the small data regime, commonly observed in medical applications due to constraints like disease prevalence and annotation difficulty. In such cases, curating an appropriate validation set poses a conundrum. A large validation set would compromise the size of the training set, whereas a small one may not sufficiently estimate the model’s performance. Despite its critical importance, this issue has yet to receive adequate attention in the field. ● The second scenario involves dealing with a stream of data, in a context of continual learning where the model encounters a continuous flow of new data. A finite, static validation set proves unsuitable as it cannot accurately assess the model’s capability in processing an extensive and diverse data range. We argue that a validation set— made up of real-world data—might not be needed during the training stage in such situations. Given the vastness of the training data, overfitting can be naturally avoided. Consequently, selecting the last-epoch model checkpoint could be a judicious choice.

[0161] Progresses in data synthesis. Real-world data often encounters challenges such as poor quality, limited quantity, and inaccessibility. To tackle these obstacles, the notion of synthetic data has emerged as a practical alternative, allowing for the generation of samples as needed. This approach has proven valuable in addressing data limitations and facilitating machine learning processes, including computer vision, natural language processing, voice, and many other fields. In the medical domain, the practice of data synthesis—tumor synthesis in particular—endeavors to produce artificial tumors in the image, which can significantly diversify the data and annotations for AI training and, arguably, can strengthen the AI robustness evaluation using corner cases generated by data synthesis. Successful works related to tumor synthesis include polyp detection from colonoscopy videos, COVID-19 detection from Chest CT and X-ray, diabetic lesion detection from retinal images, cancer detection from fluorescence microscopy images, brain tumor detection from MRI, early pancreatic cancer localization from CT, and pneumonia detection from ultrasound. A recent study indicated that AI trained exclusively on synthetic tumors can segment liver tumors with accuracy comparable to that on real tumors. To the best of our knowledge, data synthesis has been widely recognized for its contribution to enhancing training and test datasets, but its capacity for improving the validation set remains largely untapped. In this example, we extend the application of synthetic data to the validation set, enabling the full use of the annotated data for AI training while ensuring diverse and comprehensive validation data in the framework of continual learning.

[0162] 3. Method & Material

[0163] 3.1 Continual Learning for Tumor Segmentation

[0164] According to prior work, continual learning can be categorized into three settings: class-incremental learning, task-incremental learning, and domain- incremental learning. In the domain-incremental setting, which is relevant to our situation, the task remains the same while the data distribution changes. More specifically, the model sequentially encounters data from a continuum of domains (datasets):The objective is to train a model F : X → Y that can be effectively queried at any given time, regardless of the data distribution. In liver tumor segmentation tasks, X is the CT volume and Y is the tumor mask. The continuum of domains refers to CT volumes taken from different medical centers.

[0165] In the setting of static training, shown in Figure 16(a), the AI model is trained and validated on fixed subsets of a dataset. This setting presents three limitations: Firstly, the limited scales and acquisition sources of the data, coupled with unchanged data distribution, pose challenges in generalizing to out-domain data. Secondly, the task of data annotation, specifically tumor annotation, is exceptionallychallenging as it often requires the use of corroborative pathology reports. This requirement adds to the difficulty of extending the dataset. Thirdly, there are specific cases, such as extremely small tumors, where obtaining real data becomes significantly challenging. As a result, static training will likely result in biased, sub- optimal performance on unseen data, especially for out-domain test sets.

[0166] In contrast, the setting of dynamic training achieved by synthetic data, shown in Figure 16(b), can overcome the aforementioned limitations. In this setting, the AI model is trained and validated on a dynamically changing dataset. In this example, this dynamic dataset is a stream of normal CT volumes—over 40 million CT volumes in the United States each year. Generating synthetic tumors is advantageous because, firstly, acquiring healthy CT volumes is much easier than obtaining those with cancerous tumors. As a result, our continual learning framework can start from a diverse dataset comprising CT volumes of healthy subjects from multiple domains. Secondly, by controlling the parameters within our framework, we have the ability to generate synthetic data that fulfills specific requirements, including those of a tiny radius (<5 mm, shown in Figure 24). Consequently, our framework achieves a noteworthy level of diversity, encompassing a wide array of variations. Therefore, the AI model developed using this framework of synthetic data has the potential to improve its performance on out-domain data.

[0167] 3.2 Modeling-Based Synthetic Tumor Generation

[0168] Following the standardized clinical guidance and statistical distribution of real tumors, as detailed in Figures 22 and 23, we develop a modeling-based strategy to generate synthetic tumors. For example, according to the Liver Imaging Reporting and Data System (LI-RADS), the malignancy of hepatocellular carcinomas is determined by shape, size, location, and texture, enhancing capsule appearance. We use a sequence of morphological image-processing operations to model real tumors, as shown in Figure 1(c). The tumor generator consists of four steps: (1) location selection, (2) shape generation, (3) texture generation, and (4) post-processing. 1. Location selection. Liver tumors generally do not allow the passage of preexisting blood vessels from the host tissue through them. To address this concern, we initially perform voxel value thresholding for vessel segmentation. Utilizing thevessel mask acquired from this step enables us to identify if a particular location can cause the tumor-blood collision. 2. Shape generation. Based on clinical knowledge, a tumor is initiated from a malignant cell and gradually proliferates and expands, resulting in a nearly spherical shape for small tumors (≤5mm). On the other hand, statistical distributions of real liver tumors indicate that larger tumors tend to exhibit an elliptical shape. This observation has inspired us to generate a tumor-like shape using an ellipsoid ellip(a, b, c), where a, b, c are the lengths of the semi-axes. Additionally, we utilize elastic deformation to enhance the authenticity of the generated tumor shapes D(ellip(a, b, c), σd), where σdcontrol the magnitude of displacements. We show examples of the generated tumor shapes in Figure 24. 3. Texture generation. The generation of textures is a significant challenge due to the varied patterns found in tumors. Our current understanding of tumor textures is derived solely from clinical expertise, which considers factors such as the attenuation value and the distribution characteristics. To achieve the desired texture, we introduce Gaussian noise N (µ, σg) with a predetermined mean attenuation value, matching the standard deviation of liver tumors. Subsequently, we use cubic interpolation to smooth the texture. Furthermore, to better replicate textures obtained from CT imaging, we use a final step of texture blurring. Examples of the generated texture can be found in Figure 15. 3. Post-processing. The post-processing involves evaluating image characteristics through visual inspection and feedback from medical professionals. The purpose of these steps is to replicate the phenomena of mass effect and the appearance of a capsule. Mass effect refers to the phenomenon wherein the tumor undergoes growth, resulting in the displacement and deformation of surrounding tissues. We utilize local scaling warping to replicate this effect. Additionally, we brighten the edges of the tumor, thereby simulating the capsule appearance. Consequently, CT volumes with synthetic tumors can be used for the continual learning framework, where examples of the generated liver tumors can be found in Figure 26.

[0169] 4. Experiment

[0170] 4.1 Dataset & Benchmark

[0171] Table 1 summarizes a total of five publicly available datasets used in this study. We group them into three classes. ● Real-tumor dataset. We select the LiTS dataset for training and testing AI models. LiTS provides detailed per-voxel annotations of liver tumors. The tumor types include HCC and secondary liver tumors and metastasis derived from colorectal, breast, and lung cancer. The size of liver tumors ranges from 38mm3 to 349 cm3, and the radius of tumors is approximately in the range of [2, 44] mm. LiTS is partitioned into a training set (cohort 1; 25 CT volumes), validation set (cohort 2; 5 CT volumes), and test set (cohort 3; 70 CT volumes). ● Healthy CT assembly. We have collected a dataset of 75 CT volumes with healthy liver assembled from CHAOS, Pancreas-CT and BTCV. This assembled dataset is partitioned into a training set (cohort 4; 25 CT volumes) and a validation set (cohort 5; 50 CT volumes). As illustrated in Figure 16(b) For the training set, tumors were dynamically generated within these volumes during training, resulting in a sequential collection of image-label pairs comprising synthetic tumors. For the validation set, we generated three different tumor sizes (small, medium, and large) for each healthy CT volume offline, giving a total of 150 CT volumes. ● External benchmark. FLARE’23 is used for an external benchmark because it provides out-domain CT volumes from the LiTS dataset. This dataset was specifically chosen due to its extensive coverage, encompassing over 4000 3D CT volumes obtained from more than 30 medical centers. The inclusion of such a diverse dataset ensures the generalizability of the benchmark. The FLARE’23 dataset contains partially labeled annotations. To ensure the suitability of the test set, specific criteria are applied to the annotations. These criteria require that the annotations include per- voxel labeling for both the liver and tumors, with the additional constraint that the connected component of the tumor must intersect with the liver. Adhering to these conditions, we chose the external test set (cohort 7; 120 CT volumes). Additionally, same as the assembly dataset, we can use the healthy cases within the FLARE’23 to generate synthetic data to serve as in-domain validation set (cohort 6; 50 CT volumes), which will be used in §5.5. Table 1: Datasets description. The LiTS dataset was used to train, validate, and evaluate AI models in segmenting liver tumors. The FLARE’23 dataset was used forexternal validation. An assembly of the CHAOS, BTCV, and Pancreas-CT datasets were used for generating synthetic training and validation sets, in which the liver in these datasets is confirmed to be healthy. dataset notation split annotation # of CTs tumor cohort 1 training✓25 realLiTS cohort 2 validation ✓ 5 real cohort 3 testing✓70 realAssemblycohort 4training✗25 syntheticcohort 5 validation✗50 syntheticFLARE’23cohort 6validation ✗ 50 syntheticcohort 7 testing✓120 real

[0172] 4.2 Implementation

[0173] We have implemented our codes utilizing the MONAI framework for the U-Net architecture, a well-established network commonly employed in medical image segmentation tasks. During the pre- processing stage, input images undergo clipping with a window range of [-21,189]. Following this, they are normalized to achieve a zero mean and unit standard deviation. For training purposes, random patches with dimensions of 96 × 96 × 96 are cropped from the 3D image volumes. A base learning rate of 0.0002 is utilized in the training process, accompanied by a batch size of two per GPU. To further enhance the training process, we employ both the linear warmup strategy and the cosine annealing learning rate schedule. Our model is trained for 6,000 epochs, with a model checkpoint being saved every 100 epochs, and a total of 60 model checkpoints are saved throughout the entire training process. During the inference phase, a sliding window strategy with an overlapping area ratio of 0.75 is adopted. To ensure robustness and comprehensiveness in obtaining results, the experiment is conducted ten times each to perform statistical analysis. By averaging all runs, we obtain reliable results. The segmentation performance is evaluated using the Dice Similarity Coefficient (DSC) score, while Sensitivity is used to evaluate the performance of detecting very tiny liver tumors (radius < 5mm).

[0174] 5. Result

[0175] Summary. Using synthetic data as validation can select the best model checkpoint and alleviate the overfitting problem. Furthermore, the AI model developed using our continual learning framework outperforms models trained and validated on a static dataset. The performance is particularly high for detecting small / tiny tumorsbecause we can generate a vast number of examples of small / tiny tumors for both training and validation.

[0176] 5.1 Overfitting is Attributed to Small-Scale, Real-Tumor Validation

[0177] To demonstrate the potential limitations of selecting the best model checkpoint based on a small-scale and biased real-tumor validation set, we evaluate all the model checkpoints in real-tumor validation set (cohort 2), in-domain LiTS test set (cohort 3) and out-domain FLARE’23 test set (cohort 7). In-domain test set (cohort 3) assesses the performance of each checkpoint and aids in determining the effectiveness of the selected best checkpoints using the validation set (cohort 2). Out- domain test set (cohort 7) serves as a robust benchmark, providing an enhanced evaluation of the performance of the model checkpoints on out-domain unseen data.

[0178] As shown in Figure 17, two significant observations can be made. Firstly, the best checkpoint identified by the small-scale real-tumor validation set exhibits considerable instability, with notable variations observed when different validation samples are chosen. This result indicates that the small-scale real-tumor validation is inherently biased and lacks the ability to adequately represent the broader range of cases. Secondly, the performance of the best checkpoint determined by the real validation set does not effectively generalize to unseen test data, particularly when confronted with out-domain data. These observations indicate that overfitting can be attributed to a small-scale, biased real-tumor validation set.

[0179] 5.2 Overfitting is Alleviated by Large-Scale, Synthetic-Tumor Validation

[0180] The overfitting can be alleviated by a diverse, large-scale synthetic- tumor validation set. We conducted a similar experiment to § 5.1. This experiment involved evaluating all the model checkpoints in synthetic-tumor validation set (cohort 5), in-domain test set (cohort 3) and out-domain test set (cohort 7).

[0181] The evaluation trajectory can be observed in Figure 18, with the synthetic-tumor validation set represented by the upper, lighter greyscale line. It is clear that the best checkpoint selected using the synthetic-tumor validation set performs much better than the best checkpoint chosen using the real-tumor validation set when tested with unseen data. Notably, this improvement is especially remarkablewhen dealing with out-domain data (cohort 7), as the selected model checkpoint is identical to the one chosen by the out-domain test set. These findings emphasize the effectiveness of synthetic-tumor validation set, which serves as a superior alternative to mitigate overfitting issues.

[0182] 5.3 Overfitting Can Be Addressed by Continual Learning on Synthetic Data

[0183] We have shown the effectiveness of large-scale synthetic-tumor validation set. Now, we will shift our attention to synthetic data in handling the overfitting problem from a training perspective. For this purpose, we introduce a continual learning framework on synthetic data, detailed in § 3.1.

[0184] The training trajectory of static training on real data and dynamic training on synthetic data is shown in Figure 19, and the liver tumor segmentation results are presented in Table 2. Specifically, the AI model trained on static real data demonstrates a DSC score of 26.7% for the in-domain test set (cohort 3) and 31.1% for the out-domain test set (cohort 7). In comparison, the AI model developed using our continual learning framework with synthetic data achieves notably higher DSC scores, reaching 34.5% on cohort 3 and 35.4% on cohort 7, respectively. These results indicate a notable improvement in the synthetic data. Based on these findings, we can confidently assert that incorporating our continual learning framework with synthetic data allows us to effectively address the issue of overfitting, encompassing both the training and validation perspectives. Table 2: Synthetic data for both training and validation. As shown in Figure 19, AI model can be trained on either static real data or dynamic synthetic data. The terms “train @ real” and “trained @ synt” denote static training with real data and dynamic training with synthetic data, respectively. We save the model checkpoints at each training epoch and then use a validation set to select the best model. These selected model checkpoints are tested on the in-domain LiTS test set (cohort 3) and out- domain FLARE’23 test set (cohort 7). We report the DSC score (%) and 95% confidence interval achieved on the test set. The result reveals that training and validating AI models with our continual learning framework can significantly improve liver tumor segmentation.evaluated on real data evaluated on synthetic data train @ real train @ synt train @ real train @ synt cohort 3 26.7 (22.6-30.9) 33.4 (28.7-38.0) 27.0 (23.7-30.3) 34.5 (30.8-38.2) cohort 7 31.1 (26.0-36.2) 33.3 (30.6-36.0) 32.0 (28.5-35.5) 35.4 (32.1-38.7)

[0185] 5.4 Synthetic Data Can Benefit Early Cancer Detection

[0186] Early tumor (radius < 5mm) detection plays a critical role in clinical applications, providing valuable information for early cancer diagnosis. Acquiring real data of such a small size is challenging, often posing difficulties or even making it impossible to acquire them. However, our strategy can dynamically generate numerous tiny tumors as required. As a result, the AI model developed within the continual learning framework yields a significant improvement in detecting tiny liver tumors. The improvement can be found in Figure 20. We assessed the sensitivity of the AI model under different settings. The performance of the AI model trained and validated on the static real data is 33.1% for the in-domain test set (cohort 3) and 33.9% for the out-domain test set (cohort 7). Comparatively, the AI model developed using our continual learning framework on synthetic data gives a sensitivity of 55.4% for cohort 3 and 52.3% for cohort 7. These results prove the effectiveness of our framework in early detection of cancer.

[0187] 5.5 Continual Learning Framework is Enhanced by In-domain Synthetic-Tumor Validation

[0188] We have demonstrated the effectiveness of our continual learning framework. Moving forward, let’s consider the framework itself. Synthetic data offers a significant advantage as we can utilize healthy CT volumes from various domains. A notable scenario arises wherein we can directly generate synthetic-tumor validation using healthy cases from the same domain as the test set, providing valuable insights for our continual learning framework. As shown in Figure 21, the in-domain synthetic- tumor validation set showcases its capability to accurately identify the best model, which aligns with the model selected by the test set. This result highlights that the continual learning framework yields more favorable outcomes when we can generate in-domain synthetic-tumor validation.

[0189] 6. Conclusion

[0190] Data synthesis strategies continue to pique the interest of researchers and practitioners, propelling ongoing investigations within this field. This example justifies the potential and stresses the necessity of leveraging synthetic data as validation to select the best model checkpoint along the training trajectory. Moreover, by employing a continual learning framework on synthetic data, we realize a marked improvement in liver tumor segmentation as well as in the early detection of cancerous tumors compared with the static training on real data, where procuring ample annotated examples can be cost-prohibitive. It is particularly valuable in scenarios characterized by limited annotated data. In the future, we plan to improve the generation of synthetic tumors and verify our findings across different organs, such as the pancreas, kidneys, and stomach.

[0191] Some further aspects are defined in the following clauses:

[0192] Clause 1: A method of producing a synthetic computed tomography (CT) image, generating, by a computer, a synthetic tumor image using a simulated tumor development model, wherein each pixel of the synthetic tumor image is assigned an initial state value that together represent a population of cells of the synthetic tumor image and wherein the simulated tumor development model applies an accumulation rule in which cells having a first state value accumulate with a first probability, a growth rule in which cells having a second state value incentivize at least one neighboring cell to grow with a second probability, and a death rule in which cells having a third state value and surrounded by neighboring cells having the third state value die with a third probability; and, integrating, by the computer, the synthetic tumor image with a reference CT image, thereby producing the synthetic CT image.

[0193] Clause 2: The synthetic CT image produced by the method of Clause 1.

[0194] Clause 3: The method of Clause 1 or Clause 2, further comprising producing a set of synthetic CT images.

[0195] Clause 4: The method of any one of the preceding Clauses 1-3, further comprising training an electronic neural network that detects and segments tumors in test subject CT images using at least a portion of the set of synthetic CT images to produce a trained electronic neural network.

[0196] Clause 5: The method of any one of the preceding Clauses 1-4, further comprising passing a test subject CT image through the trained electronic neuralnetwork to detect a tumor in the test subject and to segment the tumor in the test subject CT image to produce a detected and segmented tumor in the test subject.

[0197] Clause 6: The method of any one of the preceding Clauses 1-5, further comprising administering one or more therapies to the test subject to treat the detected and segmented tumor in the test subject.

[0198] Clause 7: The trained electronic neural network produced by the method of any one of the preceding Clauses 1-6.

[0199] Clause 8: The method of any one of the preceding Clauses 1-7, wherein the trained electronic neural network achieves detection and segmentation performance levels that are comparable to or exceed a detection and segmentation performance level achieved by an electronic neural network trained only using real tumor CT images.

[0200] Clause 9: The method of any one of the preceding Clauses 1-8, comprising continually training the electronic neural network that detects and segments tumors in the test subject CT images using additional synthetic CT images as the additional synthetic CT images are generated.

[0201] Clause 10: The method of any one of the preceding Clauses 1-9, further comprising validating a trained electronic neural network that detects and segments tumors in test subject CT images using at least a portion of the set of synthetic CT images to produce a validated electronic neural network.

[0202] Clause 11: The method of any one of the preceding Clauses 1-10, wherein the reference CT image is obtained from at least a portion of a reference subject.

[0203] Clause 12: The method of any one of the preceding Clauses 1-11, wherein the simulated tumor development model comprises a cellular automata.

[0204] Clause 13: The method of any one of the preceding Clauses 1-12, wherein each of the pixels represents one cell of the population of cells of the synthetic tumor image.

[0205] Clause 14: The method of any one of the preceding Clauses 1-13, wherein the initial state value is selected from a finite number of states having a value of 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10.

[0206] Clause 15: The method of any one of the preceding Clauses 1-14, wherein the accumulation rule is applied to cells in which the first state value is greater than zero and less than 10.

[0207] Clause 16: The method of any one of the preceding Clauses 1-15, wherein the growth rule is applied to cells in which the second state value is greater than zero.

[0208] Clause 17: The method of any one of the preceding Clauses 1-16, wherein the death rule is applied to cells in which the third state value is 10 and the third state values of the neighboring cells are 10.

[0209] Clause 18: The method of any one of the preceding Clauses 1-17, wherein the accumulation, growth, and death rules simulate a shape, size, texture, location, intensity, and / or invasive behavior of the tumor.

[0210] Clause 19: The method of any one of the preceding Clauses 1-18, comprising using the simulated tumor development model for a selected time, t, to generate the synthetic tumor image.

[0211] Clause 20: The method of any one of the preceding Clauses 1-19, comprising defining a structure of the synthetic tumor of the synthetic tumor image that comprises an inner region representing dead cells, an intermediate region representing living but non-proliferative cells, and an outer region representing proliferative cells.

[0212] Clause 21: The method of any one of the preceding Clauses 1-20, wherein the synthetic tumor of the synthetic tumor image is generalizable across multiple organs of a given subject.

[0213] Clause 22: The method of any one of the preceding Clauses 1-21, wherein the synthetic tumor of the synthetic tumor image comprises a radius of about 25 mm or less, about 20 mm or less, about 15 mm or less, about 10 mm or less, or about 5 mm or less.

[0214] Clause 23: A system, comprising: a controller that comprises, or is capable of accessing, computer readable media comprising non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least: passing a test subject computed tomography (CT) image through a trained electronic neural network to detect a tumor in the test subject and to segmentthe tumor in the test subject CT image to produce a detected and segmented tumor in the test subject, wherein the trained electronic neural network is at least partially trained using a set of synthetic CT images that are produced by generating synthetic tumor images using a simulated tumor development model, wherein each pixel of a given synthetic tumor image is assigned an initial state value that together represents a population of cells of the given synthetic tumor image and wherein the simulated tumor development model applies an accumulation rule in which cells having a first state value accumulate with a first probability, a growth rule in which cells having a second state value incentivize at least one neighboring cell to grow with a second probability, and a death rule in which cells having a third state value and surrounded by neighboring cells having the third state value die with a third probability, and integrating the synthetic tumor images with reference CT images to produce the set of synthetic CT images; and, generating a report of the detected and segmented tumor in the test subject.

[0215] Clause 24: The system of Clause 23, further comprising a CT scanner configured to generate test subject CT images, wherein the controller is operably connected to the CT scanner, and wherein the non-transitory computer executable instructions which, when executed by the electronic processor, further perform: generating the test subject CT image.

[0216] Clause 25: The system of Clause 23 or Clause 24, wherein the non- transitory computer executable instructions which, when executed by the electronic processor, further perform: generating at least one therapy recommendation to treat the detected and segmented tumor in the test subject.

[0217] Clause 26: The system of any one of the preceding Clauses 23-25, wherein the simulated tumor development model comprises a cellular automata.

[0218] Clause 27: The system of any one of the preceding Clauses 23-26, wherein the initial state value is selected from a finite number of states having a value of 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10.

[0219] Clause 28: The system of any one of the preceding Clauses 23-27, wherein the accumulation rule is applied to cells in which the first state value is greater than zero and less than 10.

[0220] Clause 29: The system of any one of the preceding Clauses 23-28, wherein the growth rule is applied to cells in which the second state value is greater than zero.

[0221] Clause 30: The system of any one of the preceding Clauses 23-29, wherein the death rule is applied to cells in which the third state value is 10 and the third state values of the neighboring cells are 10.

[0222] Clause 31: The system of any one of the preceding Clauses 23-30, wherein the accumulation, growth, and death rules simulate a shape, size, texture, location, intensity, and / or invasive behavior of the tumor.

[0223] Clause 32: The system of any one of the preceding Clauses 23-31, wherein the synthetic tumor of the synthetic tumor image comprises a radius of about 25 mm or less, about 20 mm or less, about 15 mm or less, about 10 mm or less, or about 5 mm or less.

[0224] Clause 33: A computer readable media comprising non-transitory computer executable instructions which, when executed by at least electronic processor, perform at least: passing a test subject computed tomography (CT) image through a trained electronic neural network to detect a tumor in the test subject and to segment the tumor in the test subject CT image to produce a detected and segmented tumor in the test subject, wherein the trained electronic neural network is at least partially trained using a set of synthetic CT images that are produced by generating synthetic tumor images using a simulated tumor development model, wherein each pixel of a given synthetic tumor image is assigned an initial state value that together represents a population of cells of the given synthetic tumor image and wherein the simulated tumor development model applies an accumulation rule in which cells having a first state value accumulate with a first probability, a growth rule in which cells having a second state value incentivize at least one neighboring cell to grow with a second probability, and a death rule in which cells having a third state value and surrounded by neighboring cells having the third state value die with a third probability, and integrating the synthetic tumor images with reference CT images to produce the set of synthetic CT images; and, generating a report of the detected and segmented tumor in the test subject.

[0225] While the invention has been described with reference to the exemplary embodiments thereof, those skilled in the art will be able to make various modifications to the described embodiments without departing from the true spirit and scope. The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. In particular, although the method has been described by examples, the steps of the method can be performed in a different order than illustrated or simultaneously. Those skilled in the art will recognize that these and other variations are possible within the spirit and scope as defined in the following claims and their equivalents. All patents, patent applications, other publications or documents, and the like cited herein are incorporated by reference in their entirety for all purposes to the same extent as if each individual item were specifically and individually indicated to be so incorporated by reference.

Claims

What is claimed is:

1. A method of producing a synthetic computed tomography (CT) image, the method comprising: generating, by a computer, a synthetic tumor image using a simulated tumor development model, wherein each pixel of the synthetic tumor image is assigned an initial state value that together represent a population of cells of the synthetic tumor image and wherein the simulated tumor development model applies an accumulation rule in which cells having a first state value accumulate with a first probability, a growth rule in which cells having a second state value incentivize at least one neighboring cell to grow with a second probability, and a death rule in which cells having a third state value and surrounded by neighboring cells having the third state value die with a third probability; and, integrating, by the computer, the synthetic tumor image with a reference CT image, thereby producing the synthetic CT image.

2. The synthetic CT image produced by the method of claim 1.

3. The method of claim 1, further comprising producing a set of synthetic CT images.

4. The method of claim 3, further comprising training an electronic neural network that detects and segments tumors in test subject CT images using at least a portion of the set of synthetic CT images to produce a trained electronic neural network.

5. The method of claim 4, further comprising passing a test subject CT image through the trained electronic neural network to detect a tumor in the test subject and to segment the tumor in the test subject CT image to produce a detected and segmented tumor in the test subject.

6. The method of claim 5, further comprising administering one or more therapies to the test subject to treat the detected and segmented tumor in the test subject.

7. The trained electronic neural network produced by the method of claim 4.

8. The method of claim 4, wherein the trained electronic neural network achieves detection and segmentation performance levels that are comparable to or exceed a detection and segmentation performance level achieved by an electronic neural network trained only using real tumor CT images.

9. The method of claim 4, comprising continually training the electronic neural network that detects and segments tumors in the test subject CT images using additional synthetic CT images as the additional synthetic CT images are generated.

10. The method of claim 3, further comprising validating a trained electronic neural network that detects and segments tumors in test subject CT images using at least a portion of the set of synthetic CT images to produce a validated electronic neural network.

11. The method of claim 1, wherein the reference CT image is obtained from at least a portion of a reference subject.

12. The method of claim 1, wherein the simulated tumor development model comprises a cellular automata.

13. The method of claim 1, wherein each of the pixels represents one cell of the population of cells of the synthetic tumor image.

14. The method of claim 1, wherein the initial state value is selected from a finite number of states having a value of 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10.

15. The method of claim 1, wherein the accumulation rule is applied to cells in which the first state value is greater than zero and less than 10.

16. The method of claim 1, wherein the growth rule is applied to cells in which the second state value is greater than zero.

17. The method of claim 1, wherein the death rule is applied to cells in which the third state value is 10 and the third state values of the neighboring cells are 10.

18. The method of claim 1, wherein the accumulation, growth, and death rules simulate a shape, size, texture, location, intensity, and / or invasive behavior of the tumor.

19. The method of claim 1, comprising using the simulated tumor development model for a selected time, t, to generate the synthetic tumor image.

20. The method of claim 1, comprising defining a structure of the synthetic tumor of the synthetic tumor image that comprises an inner region representing dead cells, an intermediate region representing living but non-proliferative cells, and an outer region representing proliferative cells.

21. The method of claim 1, wherein the synthetic tumor of the synthetic tumor image is generalizable across multiple organs of a given subject.

22. The method of claim 1, wherein the synthetic tumor of the synthetic tumor image comprises a radius of about 25 mm or less, about 20 mm or less, about 15 mm or less, about 10 mm or less, or about 5 mm or less.

23. A system, comprising: a controller that comprises, or is capable of accessing, computer readable media comprising non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least: passing a test subject computed tomography (CT) image through a trained electronic neural network to detect a tumor in the test subject and to segment the tumor in the test subject CT image to produce a detected and segmented tumor in the test subject, wherein the trained electronic neural network is at least partially trained using a set of synthetic CT images that are produced by generating synthetic tumor images using a simulated tumor development model, wherein each pixel of a given synthetic tumor image is assigned an initial state value that together represents a population of cells of the given synthetic tumor image and wherein the simulated tumor development model applies an accumulation rule in which cells having a first state value accumulate with a first probability, a growth rule in which cells having a second state value incentivize at least one neighboring cell to grow with a second probability, and a death rule in which cells having a third state value and surrounded by neighboring cells having the third state value die with a third probability, and integrating the synthetic tumor images with reference CT images to produce the set of synthetic CT images; and, generating a report of the detected and segmented tumor in the test subject.

24. The system of claim 23, further comprising a CT scanner configured to generate test subject CT images, wherein the controller is operably connected to the CT scanner, and wherein the non-transitory computer executable instructions which, when executed by the electronic processor, further perform: generating the test subject CT image.

25. The system of claim 23, wherein the non-transitory computer executable instructions which, when executed by the electronic processor, further perform: generating at least one therapy recommendation to treat the detected and segmented tumor in the test subject.

26. The system of claim 23, wherein the simulated tumor development model comprises a cellular automata.

27. The system of claim 23, wherein the initial state value is selected from a finite number of states having a value of 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10.

28. The system of claim 23, wherein the accumulation rule is applied to cells in which the first state value is greater than zero and less than 10.

29. The system of claim 23, wherein the growth rule is applied to cells in which the second state value is greater than zero.

30. The system of claim 23, wherein the death rule is applied to cells in which the third state value is 10 and the third state values of the neighboring cells are 10.

31. The system of claim 23, wherein the accumulation, growth, and death rules simulate a shape, size, texture, location, intensity, and / or invasive behavior of the tumor.

32. The system of claim 23, wherein the synthetic tumor of the synthetic tumor image comprises a radius of about 25 mm or less, about 20 mm or less, about 15 mm or less, about 10 mm or less, or about 5 mm or less.

33. A computer readable media comprising non-transitory computer executable instructions which, when executed by at least electronic processor, perform at least: passing a test subject computed tomography (CT) image through a trained electronic neural network to detect a tumor in the test subject and to segment the tumor in the test subject CT image to produce a detected and segmented tumor in the test subject, wherein the trained electronic neural network is at least partiallytrained using a set of synthetic CT images that are produced by generating synthetic tumor images using a simulated tumor development model, wherein each pixel of a given synthetic tumor image is assigned an initial state value that together represents a population of cells of the given synthetic tumor image and wherein the simulated tumor development model applies an accumulation rule in which cells having a first state value accumulate with a first probability, a growth rule in which cells having a second state value incentivize at least one neighboring cell to grow with a second probability, and a death rule in which cells having a third state value and surrounded by neighboring cells having the third state value die with a third probability, and integrating the synthetic tumor images with reference CT images to produce the set of synthetic CT images; and, generating a report of the detected and segmented tumor in the test subject.

Images