Machine learning methods and related aspects for generating synthetic tumors on computed tomography scans

Abstract

Examples may provide a computer-implemented method of producing a computed tomography (CT) scan that comprises a synthetic tumor. The method includes identifying a target tissue on the CT scan to produce an identified target tissue and selecting a location at least proximal to the identified target tissue on the CT scan to position a synthetic tumor to produce a selected location. The method also includes generating a simulated shape of the synthetic tumor to produce a synthetic tumor shape, producing a simulated texture of the synthetic tumor to produce a synthetic tumor texture, and combining the synthetic tumor shape and the synthetic tumor texture at the selected location on the CT scan. Examples may also provide a computer-implemented method of determining whether a tumor is present in a test subject. Additional methods as well as related systems and computer readable media are also provided.

Description

MACHINE LEARNING METHODS AND RELATED ASPECTS FOR GENERATING SYNTHETIC TUMORS ON COMPUTED TOMOGRAPHY SCANSCross-Reference to Related Applications

[0001] This application claims priority to U.S. Provisional Patent Application Ser. No. 63 / 538,373, filed September 14, 2023, the disclosure of which is incorporated herein by reference.Field

[0002] This disclosure relates generally to machine learning, e.g., in the context of medical applications, such as diagnostics.Background

[0003] Artificial intelligence (Al) has dominated medical image segmentation, but training an Al model (e.g., U-Net) often requires a large number of detailed per- voxel annotations. Annotating medical images is not only expensive and timeconsuming, but also requires extensive medical expertise, and sometimes needs the assistance of radiology reports and biopsy results to precisely annotate a tumor. Due to its high annotation cost, only a total of roughly 100 computed tomography (CT) scans with annotated liver tumors are publicly available (provided by LiTS) for training and testing models.

[0004] Generating synthetic tumors is a recent research topic. There are some early attempts at generating COVID-19 infections on Chest CT scans, abdominal tumors in CT scans, diabetic lesions on retinal images, brain tumors on MRI images, and cancers in fluorescence microscopy images. However, the synthetic tumors in those existing studies appear very different from the real tumors, and Al models trained using synthetic tumors perform significantly worse than those trained using real tumors due to the pronounced domain gap between real and synthetic tumors. What makes synthesizing tumors so hard? There are several important factors: shape, intensity, size, location, and most importantly, texture.

[0005] Accordingly, there is a need for approaches to generate synthetic or fake tumors that closely simulate real tumors for use in training Al models for medical image segmentation, among other applications.Summary

[0006] The present disclosure demonstrates, in some aspects, that tumor segmentation can be done by artificial intelligence (Al) models without manual per- voxel tumor annotation developed by humans. In some embodiments, this is achieved by tumor synthesis in CT scans. The advantages of these synthetic tumors include being: (I) realistic, for example, in shape and texture, which even medical professionals can confuse with real tumors and (II) effective for training Al models, which can perform tumor segmentation similar to models trained on real tumors. That is, in some embodiments, Al models trained solely using the synthetic tumors of the present disclosure perform as well as or even better than models trained using annotated CT scans of real tumors. These results also imply that manual efforts for developing per-voxel annotation of tumors (which often took years to create) can now be considerably reduced or eliminated for training Al models. Moreover, the methods disclosed herein can automatically generate many examples of small (or even tiny) synthetic tumors that can improve the success rate of detecting small tumors in patients, which is important for detecting the early stages of cancer. In addition to enriching the available training data, the tumor synthesis methods of the present disclosure also enable the performance and robustness of Al models to be rigorously assessed under at least five parameters, namely, tumor shape, size, texture, intensity, and location. In some embodiments, each of these parameters can take at least five different values, yielding 5x5 synthetic tumor examples in each CT scan. Furthermore, the synthetic tumor data sets created using the methods disclosed herein allow for the detection of failure modes in current Al models and accordingly, can be used to make the algorithms more robust for detecting tumors in the real world. These and other aspects will be apparent upon a complete review of the present disclosure, including the accompanying figures.

[0007] According to various embodiments, a computer-implemented method of producing a computed tomography (CT) scan that comprises a synthetic tumor ispresented. The computer-implemented method includes identifying a target tissue on the CT scan to produce an identified target tissue and selecting a location at least proximal to the identified target tissue on the CT scan to position a synthetic tumor to produce a selected location. The computer-implemented method also includes generating a simulated shape of the synthetic tumor to produce a synthetic tumor shape, producing a simulated texture of the synthetic tumor to produce a synthetic tumor texture, and combining the synthetic tumor shape and the synthetic tumor texture at the selected location on the CT scan, thereby producing the CT scan that comprises the synthetic tumor.

[0008] Various optional features of the above embodiments include the following. The computer-implemented method comprises producing multiple synthetic tumors such that the CT scan comprises the multiple synthetic tumors. The CT scan comprises an image of at least one anatomical region of a subject. An organ comprises the identified target tissue. The organ comprises a liver. The organ comprises a pancreas. A mammalian organism comprises the identified target tissue. The mammalian organism comprises a human organism. The identifying step comprises passing the CT scan through an electronic neural network that is trained to identify the target tissue in CT scans. The identified target tissue comprises the selected location. The selected location substantially avoids vasculature imaged in the CT scan. The computer-implemented method comprises conducting vessel segmentation using a voxel value thresholding technique to substantially avoid the vasculature imaged in the CT scan. The computer-implemented method comprises applying at least one segmented vessel mask to substantially avoid the vasculature imaged in the CT scan. The generating step comprises applying an ellipsoid shaped mask at the selected location. The generating step comprises performing ellipsoid mask generation, elastic deformation of the ellipsoid mask, and / or ellipsoid mask blurring using a Gaussian filter to produce the synthetic tumor shape. The synthetic tumor shape comprises a deformed ellipsoidal shape. A tumor mask array comprises the synthetic tumor shape. The producing step comprises performing salt and / or pepper noising and / or denoising, Gaussian filtering, scaling, and / or clipping to produce the synthetic tumor texture. A texture array comprises the synthetic tumor texture. The computer-implemented method comprises performing the generating step concurrent with or after performingthe producing step. The synthetic tumor comprises a radius selected from the group consisting of: less than about 5 mm, between about 5 and about 10 mm, and greater than about 10 mm.

[0009] Various additional optional features of the above embodiments include the following. The combining step comprises placing the synthetic tumor shape and the synthetic tumor texture on a scanning volume and a corresponding target tissue mask volume. The scanning volume and / or the target tissue mask volume is unlabeled. The computer-implemented method comprises using the scanning volume and / or the target tissue mask volume to train an electronic neural network that comprises non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least detecting one or more tumors in the target tissue in a CT scan obtained from a test subject. The electronic neural network comprises a Dice Similarity Coefficient (DSC) of more than about 50% for segmenting real tumors in CT scans obtained from test subjects. The computer-implemented method comprises producing multiple CT scans that each comprise one or more synthetic tumors placed at one or more locations at least proximal to the identified target tissue, and wherein the method comprises using scanning volumes and / or target tissue mask volumes of the multiple CT scans to train the electronic neural network. The computer-implemented method further comprises using additional CT scans that comprise the identified target tissue to train the electronic neural network, wherein the additional CT scans are obtained from reference subjects. At least some of the additional CT scans comprise images of real tumors that comprise per-voxel annotation. At least some of the additional CT scans are labeled. The computer- implemented method further comprises modifying a shape of one or more edges of the synthetic tumor to simulate a mass effect. The computer-implemented method comprises modifying the edges of the synthetic tumor using a local scaling warping technique to simulate the mass effect. The computer-implemented method further comprises modifying an intensity of one or more edges of the synthetic tumor to simulate a capsule appearance of the synthetic tumor. The computer-implemented method comprises brightening one or more portions of the edges of the synthetic tumor to simulate the capsule appearance of the synthetic tumor.

[0010] According to various embodiments, a computer-implemented method of determining whether a tumor is present in a test subject is presented. The computer- implemented method includes passing a test data set that comprises a test computed tomography (CT) scan obtained from the test subject through an electronic neural network, wherein the test CT scan comprises an image of a target tissue of the test subject, wherein the electronic neural network has been trained on a set of training data that comprises multiple reference CT scans that each comprises one or more synthetic tumors placed at one or more locations at least proximal to the target tissue in the multiple reference CT scans, and wherein a given synthetic tumor in the set of training data comprises a synthetic tumor shape and a synthetic tumor texture, and outputting from the electronic neural network a detection score for the test data set, wherein a detection score value that is above a predetermined threshold value indicates that the tumor is present in the test subject, and wherein a detection score value that is below the predetermined threshold value indicates that the tumor is not present in the test subject, thereby determining whether the tumor is present in the test subject.

[0011] Various optional features of the above embodiments include the following. The electronic neural network comprises a Dice Similarity Coefficient (DSC) of more than about 50% for segmenting real tumors in test data sets. The method of determining whether the tumor is present in the test subject comprises a sensitivity of at least about 85%, at least about 90%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, or at least about 99%. The set of training data comprises at least about 100, at least about 1000, at least about 10000, at least about 100000, at least about 1000000, at least about 10000000, or more reference CT scans. The given synthetic tumor in the set of training data further comprises a simulated mass effect and / or a simulated capsule appearance. The set of training data is unlabeled. The set of training data comprises paired CT scans and tumor masks. The set of training data that further comprises additional CT scans that comprise the target tissue, wherein the additional CT scans are obtained from reference subjects, and wherein at least some of the additional CT scans comprise images of real tumors that comprise per-voxel annotation.

[0012] According to various embodiments, a system for producing a computed tomography (CT) scan that comprises a synthetic tumor is presented. The system includes a processor; and a memory communicatively coupled to the processor, the memory storing non-transitory computer executable instructions which, when executed on the processor, perform operations comprising: identifying a target tissue on the CT scan to produce an identified target tissue; selecting a location at least proximal to the identified target tissue on the CT scan to position a synthetic tumor to produce a selected location; generating a simulated shape of the synthetic tumor to produce a synthetic tumor shape; producing a simulated texture of the synthetic tumor to produce a synthetic tumor texture; and combining the synthetic tumor shape and the synthetic tumor texture at the selected location on the CT scan. In some embodiments, the non-transitory computer executable instructions which, when executed on the processor, further perform operations comprising: modifying a shape of one or more edges of the synthetic tumor to simulate a mass effect. In some embodiments, the non-transitory computer executable instructions which, when executed on the processor, further perform operations comprising: modifying an intensity of one or more edges of the synthetic tumor to simulate a capsule appearance of the synthetic tumor

[0013] According to various embodiments, a system for producing a computed tomography (CT) scan that comprises a synthetic tumor is presented. The system includes a processor; and a memory communicatively coupled to the processor, the memory storing non-transitory computer executable instructions which, when executed on the processor, perform operations comprising: passing a test data set that comprises a test computed tomography (CT) scan obtained from the test subject through an electronic neural network, wherein the test CT scan comprises an image of a target tissue of the test subject, wherein the electronic neural network has been trained on a set of training data that comprises multiple reference CT scans that each comprises one or more synthetic tumors placed at one or more locations at least proximal to the target tissue in the multiple reference CT scans, and wherein a given synthetic tumor in the set of training data comprises a synthetic tumor shape and a synthetic tumor texture; and outputting from the electronic neural network a detection score for the test data set, wherein a detection score value that is above apredetermined threshold value indicates that the tumor is present in the test subject, and wherein a detection score value that is below the predetermined threshold value indicates that the tumor is not present in the test subject.

[0014] According to various embodiments, a computer readable media is presented. The computer readable media comprises non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least: identifying a target tissue on the CT scan to produce an identified target tissue; selecting a location at least proximal to the identified target tissue on the CT scan to position a synthetic tumor to produce a selected location; generating a simulated shape of the synthetic tumor to produce a synthetic tumor shape; producing a simulated texture of the synthetic tumor to produce a synthetic tumor texture; and combining the synthetic tumor shape and the synthetic tumor texture at the selected location on the CT scan.

[0015] According to various embodiments, a computer readable media is presented. The computer readable media comprises non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least: passing a test data set that comprises a test computed tomography (CT) scan obtained from the test subject through an electronic neural network, wherein the test CT scan comprises an image of a target tissue of the test subject, wherein the electronic neural network has been trained on a set of training data that comprises multiple reference CT scans that each comprises one or more synthetic tumors placed at one or more locations at least proximal to the target tissue in the multiple reference CT scans, and wherein a given synthetic tumor in the set of training data comprises a synthetic tumor shape and a synthetic tumor texture; and outputting from the electronic neural network a detection score for the test data set, wherein a detection score value that is above a predetermined threshold value indicates that the tumor is present in the test subject, and wherein a detection score value that is below the predetermined threshold value indicates that the tumor is not present in the test subject.Drawings

[0016] The above and / or other aspects and advantages will become more apparent and more readily appreciated from the following detailed description of examples, taken in conjunction with the accompanying drawings, in which:

[0017] Fig. 1 A is a flow chart that schematically shows exemplary method steps of producing a computed tomography (CT) scan that comprises a synthetic tumor according to some aspects disclosed herein;

[0018] Fig. 1 B is a flow chart that schematically shows exemplary method steps of determining whether a tumor is present in a test subject according to some aspects disclosed herein;

[0019] Fig. 2 is a schematic diagram of an exemplary system suitable for use with certain aspects disclosed herein;

[0020] Fig. 3 shows exemplary CT scans that include synthetic or fake liver tumors generated using the methods disclosed herein. We have recruited two medical professionals with at least six years of experience to distinguish fake tumors, generated by the methods of the present disclosure, from the real ones (namely, a Visual Turing Test). Our synthetic tumors have passed the Visual Turing Test on both two medical professionals (<50% fake tumors were picked out). More importantly, using our label-free synthetic tumors, Al models can segment real tumors with performance similar to the Al models trained on real tumors with expensive, detailed, per-voxel annotation;

[0021] Fig. 4 depicts the paradigm shift from label-hungry to label-free tumor segmentation as disclosed herein. Al trained on synthetic tumors can segment, for example, liver tumors as accurately as Al trained on real tumors. In some embodiments, the performance is measured on real tumors with Dice Similarity Coefficient (DSC) and Normalized Surface Distance (NSD);

[0022] Fig. 5 schematically depicts aspects of an exemplary synthetic liver tumor generation method according to some embodiments. As shown, after randomly selecting a location that avoids the vessels, we generate a Gaussian texture and deformed ellipsoidal shape for a tumor. Then, the texture and shape are combined and placed on the selected location. In addition, we take another two post-processing steps to make a generated tumor more realistic: (1 ) tumor edge expansion by local scaling warping; (2) capsular generation by brightening the tumor edge. It isnoteworthy that, in some embodiments, several steps (i.e., vessel segmentation, collision detection, mass effect, and capsule appearance) are only used to generate examples for use in Visual Turing Testing. They are generally removed in training procedures to avoid excessive computational cost;

[0023] Fig. 6 are plots showing generalization to different models and data. Training ll-Net on synthetic liver tumors outperforms as well as training it on real tumors with per-voxel annotation (see Table 3 of Example 1 ). We further examine this observation using Swin UNETR, including its variance of Tiny (Param = 4.0M), Small (Param = 15.7M), and Base (Param = 62.1 M). The DSC, NSD, and Sensitivity scores are evaluated on the LiTS datasets. Moreover, the model trained on synthetic tumors can also be generalized to CT scans with the healthy liver across datasets (e.g. MSD- Pancreas, MSD-Spleen, and MSD-Colon), generating fewer false positives and yielding a higher Specificity compared with the model trained on real tumors;

[0024] Fig. 7 shows exemplary aspects of small tumor detection. The upper panel presents two examples of small tumors and the segmentation results. For both models trained on real and synthetic tumors, the false negatives are mostly smaller than 10mm. The lower panel presents the tumor detection rate. The model trained on synthetic tumors could detect tumors as small as 2mm;

[0025] Fig. 8 illustrate controllable robustness benchmarks. UNet++, nnll-Net, and Swin UNETR are very competitive segmentation models in the medical domain. However, their limitations of tumor segmentation are not fully revealed due to the lack of sufficient testing images (only 70 CT scans are available for testing in MSD Task03). On the contrary, synthetic tumors enable us to perform an extensive evaluation of these models in segmenting liver tumors that vary from different conditions, i.e. shape, size, texture, intensity and location. We downloaded the checkpoints of these models trained on LiTS, and evaluated them on our synthetic tumors. Synthetic tumors were generated using different parameters, where p and o denote mean and standard deviation, respectively. Our findings indicate that these public models exhibit robustness to variations in tumor shape, location, and texture, but they are sensitive to tumor size and intensity. Specifically, these models are prone to errors when encountering tumors that are smaller or larger than those in the training set, or whenfaced with differing Hounsfield Unit (HU) values, which may be attributable to contrast enhancement;

[0026] Fig. 9 are exemplary visualizations of synthetic and real pancreatic tumors and real tumor statistics. Left panel: Could you tell which tumors are real? Right panel: Selected normalized statistics of PDAC and Cyst, including size ratio (tumor / pancreas), normalized intensity residual (pancreas-tumor), and Z-offset of centers of the tumor and pancreas;

[0027] Fig. 10A-10C show exemplary aspects of tumor parameters with generation workflow, (a) Cyst intensity regression; (b) PDAC intensity regression; and (c) tumor generation with all three steps;

[0028] Fig. 11 shows ROC curves that were plotted to evaluate the performance of an Al binary classifier and two radiologists (a junior with six months of experience and a senior with five years of experience). To conduct the study, we generated 50 synthetic tumors and mixed them with 50 real tumors. The resulting set of scans was presented to the radiologists for testing without disclosing the ratio of real to synthetic tumors in the dataset. The synthetic tumors generated by our method are realistic and could confound human readers. Some examples of the Visual Turing Test are in Fig. 9; and

[0029] Fig. 12 is a plot that shows that Al trained on synthetic tumors smaller than 20mm outperforms that trained on real tumors due to difficulty in collecting and annotating CT scans with small tumors. The lack of training examples limits Al performance in small tumor detection. In contrast, synthetic tumors provide diverse and ample examples of small tumors, therefore improving Al performance.Definitions

[0030] In order for the present disclosure to be more readily understood, certain terms are first defined below. Additional definitions for the following terms and other terms may be set forth throughout the specification. If a definition of a term set forth below is inconsistent with a definition in an application or patent that is incorporated by reference, the definition set forth in this application should be used to understand the meaning of the term.

[0031] As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Thus, for example, a reference to “a method” includes one or more methods, and / or steps of the type described herein and / or which will become apparent to those persons skilled in the art upon reading this disclosure and so forth.

[0032] It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. Further, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. In describing and claiming the methods, systems, and computer readable media, the following terminology, and grammatical variants thereof, will be used in accordance with the definitions set forth below.

[0033] Cancer Type: As used herein, “cancer type” refers to a type or subtype of cancer defined, e.g., by histopathology. Cancer type can be defined by any conventional criterion, such as on the basis of occurrence in a given tissue (e.g., blood cancers, central nervous system (CNS), brain cancers, lung cancers (small cell and non-small cell), skin cancers, throat cancers, nose cancers, liver cancers, bone cancers, lymphomas, pancreatic cancers, thyroid cancers, bladder cancers, kidney cancers, mouth cancers, stomach cancers, breast cancers, prostate cancers, bowel cancers, rectal cancers, ovarian cancers, intestinal cancers, soft tissue cancers, neuroendocrine cancers, lung cancers, gastroesophageal cancers, urothelial cancers, solid state cancers, heterogeneous cancers, homogenous cancers), head and neck cancers, gynecological cancers, colorectal cancers, unknown primary origin and the like, and / or of the same cell lineage (e.g., carcinoma, sarcoma, lymphoma, cholangiocarcinoma, leukemia, mesothelioma, melanoma, or glioblastoma) and / or cancers exhibiting cancer markers, such as Her2, CA15-3, CA19-9, CA-125, CEA, AFP, PSA, HCG, hormone receptor and NMP-22. Cancers can also be classified by stage (e.g., stage 1 , 2, 3, or 4) and whether of primary or secondary origin.

[0034] Classifier. As used herein, “classifier” generally refers to algorithm computer code that receives, as input, test data and produces, as output, a classification of the input data as belonging to one or another class.

[0035] Data set. As used herein, “data set” refers to a group or collection of information, values, or data points related to or associated with one or more objects, records, and / or variables. In some embodiments, a given data set is organized as, or included as part of, a matrix or tabular data structure. In some embodiments, a data set is encoded as a feature vector corresponding to a given object, record, and / or variable, such as a given test or reference subject. For example, a medical data set for a given subject can include one or more observed values of one or more variables associated with that subject.

[0036] Electronic neural network. As used herein, “electronic neural network” refers to a machine learning algorithm or model that includes layers of at least partially interconnected artificial neurons (e.g., perceptrons or nodes) organized as input and output layers with one or more intervening hidden layers that together form a network that is or can be trained to classify data, such as test subject medical data sets (e.g., medical images or the like).

[0037] Labeled: As used herein, “labeled” or “annotated” in the context of data sets or points refers to data that is classified as, or otherwise associated with, having or lacking a given characteristic or property.

[0038] Machine Learning Algorithm: As used herein, "machine learning algorithm" generally refers to an algorithm, executed by computer, that automates analytical model building, e.g., for clustering, classification or pattern recognition. Machine learning algorithms may be supervised or unsupervised. Learning algorithms include, for example, artificial neural networks (e.g., back propagation networks, transformer networks, etc.), discriminant analyses (e.g., Bayesian classifier or Fisher’s analysis), multiple-instance learning (MIL), support vector machines, decision trees (e.g., recursive partitioning processes such as CART -classification and regression trees, or random forests), linear classifiers (e.g., multiple linear regression (MLR), partial least squares (PLS) regression, and principal components regression), hierarchical clustering, and cluster analysis. A dataset on which a machine learning algorithm learns can be referred to as "training data." A model produced using a machine learning algorithm is generally referred to herein as a “machine learning model.”

[0039] Subject. As used herein, “subject” or “test subject” refers to an animal, such as a mammalian species (e.g., human) or avian (e.g., bird) species. More specifically, a subject can be a vertebrate, e.g., a mammal such as a mouse, a primate, a simian or a human. Animals include farm animals (e.g., production cattle, dairy cattle, poultry, horses, pigs, and the like), sport animals, and companion animals (e.g., pets or support animals). A subject can be a healthy individual, an individual that has or is suspected of having a disease or pathology or a predisposition to the disease or pathology, or an individual that is in need of therapy or suspected of needing therapy. The terms “individual” or “patient” are intended to be interchangeable with “subject.” A “reference subject” refers to a subject known to have or lack specific properties (e.g., a known pathology, such as melanoma and / or the like).

[0040] Value: As used herein, “value” generally refers to an entry in a dataset that can be anything that characterizes the feature to which the value refers. This includes, without limitation, numbers, words or phrases, symbols (e.g., + or -) or degrees.Description of the Embodiments

[0041] Reference will now be made in detail to example implementations. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the invention. The following description is, therefore, merely exemplary.

[0042] I. Introduction

[0043] In some aspects, the present disclosure provides a heuristic strategy to synthesize tumors in computed tomography (CT) scans. The synthetic tumors produced according to the methods disclosed herein are realistic — even medical professionals can confuse them with real tumors in Visual Turing Tests. In addition, Al models trained on the synthetic tumors disclosed herein can detect and segment real tumors with performance similar to those trained on real tumors with expensive, detailed per-voxel annotation. In some embodiments, for example, a model trained on the label-free synthetic tumors of the present disclosure achieves a Dice Similarity Coefficient (DSC) of at least 59.81 % for segmenting real liver tumors, whereas Almodels trained on real tumors obtain a DSC of 57.63% (no statistical difference). These exemplary results are unprecedented because no existing approach, using only synthetic tumors, has reached a similar performance to the model trained on real tumors.

[0044] In addition, synthesizing tumors using the methods of the present disclosure also enables one to exhaustively generate tumors with desired locations, sizes, shapes, textures, and intensity, which are not limited to a fixed finite-size training set. For example, it is typically difficult to gather sufficient training examples with annotated small tumors since they often occur at the early stage of cancer, and present subtle abnormal textures to human eyes. As a consequence, it is typically challenging for experts to specify the boundary of the tumor manually. In contrast, the synthesis strategies disclosed herein can readily generate enormous numbers of examples with small or minute tumors, thereby enabling the detection of early-stage tumors more effectively. Aside from being able to train Al models for tumor segmentation without using any manual annotation, other applications and advantages of the methods and other aspects of the present disclosure will be apparent upon a complete review of this specification, including the accompanying figures.

[0045] To illustrate, Fig. 1A is a flow chart that schematically shows exemplary method steps of producing a computed tomography (CT) scan that comprises a synthetic or fake tumor according to some aspects disclosed herein. As shown, method 100, which is typically computer-implemented, includes identifying a target tissue on the CT scan to produce an identified target tissue (step 102) and selecting a location at least proximal to the identified target tissue on the CT scan to position a synthetic tumor to produce a selected location (step 104). Method 100 also includes generating a simulated shape of the synthetic tumor to produce a synthetic tumor shape (step 106), producing a simulated texture of the synthetic tumor to produce a synthetic tumor texture (step 108), and combining the synthetic tumor shape and the synthetic tumor texture at the selected location on the CT scan (step 110).

[0046] In some embodiments, method 100 includes producing multiple synthetic tumors such that a given CT scan comprises multiple synthetic tumors. The CT scan typically comprises an image of an anatomical region of a subject (e.g., a test subject). In some embodiments, an organ of the subject comprises the identified targettissue. Essentially any organ is optionally utilized, including, for example, a brain, eyes, lungs, skin, throat, nose, bone, thyroid gland, bladder, kidneys, mouth, stomach, breast, prostate, testicles, ovaries, uterus, intestines, and soft tissues, among other organs. In some embodiments, the organ comprises a liver and / or a pancreas of a given subject. Typically, a mammalian organism (e.g., a human) comprises the identified target tissue.

[0047] In some embodiments, the identifying step comprises passing the CT scan through an electronic neural network that is trained to identify the target tissue in CT scans. The identified target tissue typically comprises the selected location. Further, the selected location generally substantially avoids vasculature imaged in the CT scan. In some embodiments, method 100 comprises conducting vessel segmentation using a voxel value thresholding technique to substantially avoid the vasculature imaged in the CT scan. In some embodiments, method 100 comprises applying at least one segmented vessel mask to substantially avoid the vasculature imaged in the CT scan.

[0048] In some embodiments, the producing step comprises performing salt and / or pepper noising and / or denoising, Gaussian filtering, scaling, and / or clipping to produce the synthetic tumor texture. In some embodiments, the generating step comprises applying an ellipsoid shaped mask at the selected location. In some embodiments, the generating step comprises performing ellipsoid mask generation, elastic deformation of the ellipsoid mask, and / or ellipsoid mask blurring using a Gaussian filter to produce the synthetic tumor shape. In some embodiments, the synthetic tumor shape comprises a deformed ellipsoidal shape. In some embodiments, a tumor mask array comprises the synthetic tumor shape. In some embodiments, a texture array comprises the synthetic tumor texture. In some embodiments, method 100 comprises performing the generating step concurrent with or after performing the producing step. In some embodiments, the synthetic tumor comprises a radius of less than about 5 mm, between about 5 and about 10 mm, or greater than about 10 mm. In some embodiments, the combining step comprises placing the synthetic tumor shape and the synthetic tumor texture on a scanning volume and a corresponding target tissue mask volume. In some embodiments, the scanning volume and / or the target tissue mask volume is unlabeled.

[0049] In some embodiments, method 100 comprises using the scanning volume and / or the target tissue mask volume to train an electronic neural network that comprises non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least detecting one or more tumors in the target tissue in a CT scan obtained from a test subject. Typically, the electronic neural network comprises a Dice Similarity Coefficient (DSC) of more than about 50% for segmenting real tumors in CT scans obtained from test subjects. In some embodiments, method 100 comprises producing multiple CT scans that each comprise one or more synthetic tumors placed at one or more locations at least proximal to the identified target tissue. In these embodiments, method 100 typically comprises using scanning volumes and / or target tissue mask volumes of the multiple CT scans to train the electronic neural network. In some embodiments, method 100 further comprises using additional CT scans that comprise the identified target tissue to train the electronic neural network in which the additional CT scans are obtained from reference subjects. At least some of the additional CT scans typically comprise images of real tumors that comprise per-voxel annotation. In some embodiments, method 100 further comprises modifying a shape of one or more edges of the synthetic tumor to simulate a mass effect. In some of these embodiments, for example, method 100 comprises modifying the edges of the synthetic tumor using a local scaling warping technique to simulate the mass effect. In some embodiments, method 100 further comprises modifying an intensity of one or more edges of the synthetic tumor to simulate a capsule appearance of the synthetic tumor. In some of these embodiments, for example, method 100 comprises brightening one or more portions of the edges of the synthetic tumor to simulate the capsule appearance of the synthetic tumor.

[0050] To further illustrate, Fig. 1 B is a flow chart that schematically shows exemplary method steps of determining whether a tumor is present in a test subject. As shown, method 112 includes passing a test data set that comprises a test computed tomography (CT) scan obtained from the test subject through an electronic neural network (step 114). The test CT scan comprises an image of a target tissue of the test subject. In addition, the electronic neural network has been trained on a set of training data that comprises multiple reference CT scans that each comprises one or more synthetic tumors placed at one or more locations at least proximal to the target tissuein the multiple reference CT scans, and a given synthetic tumor in the set of training data comprises a synthetic tumor shape and a synthetic tumor texture. Method 112 also includes outputting from the electronic neural network a detection score for the test data set in which a detection score value that is above a predetermined threshold value indicates that the tumor is present in the test subject and in which a detection score value that is below the predetermined threshold value indicates that the tumor is not present in the test subject.

[0051] In some embodiments, the electronic neural network comprises a Dice Similarity Coefficient (DSC) of more than about 50% for segmenting real tumors in test data sets. In some embodiments, the method of determining whether the tumor is present in the test subject comprises a sensitivity of at least about 85%, at least about 90%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, or at least about 99%. Typically, the set of training data comprises at least about 100, at least about 1000, at least about 10000, at least about 100000, at least about 1000000, at least about 10000000, or more reference CT scans. In some embodiments, the given synthetic tumor in the set of training data further comprises a simulated mass effect and / or a simulated capsule appearance. Typically, the set of training data is unlabeled. In some embodiments, the set of training data comprises paired CT scans and tumor masks. In some embodiments, the set of training data that further comprises additional CT scans that comprise the target tissue in which the additional CT scans are obtained from reference subjects, and in which at least some of the additional CT scans comprise images of real tumors that comprise per-voxel annotation.

[0052] Fig. 2 is a schematic diagram of a hardware computer system 200 suitable for implementing various embodiments. For example, Fig. 2 illustrates various hardware, software, and other resources that can be used in implementations of any of methods disclosed herein, including, e.g., method 100 and / or one or more instances of an electronic neural network. System 200 includes training corpus source 202 and computer 201 . Training corpus source 202 and computer 201 may be communicatively coupled by way of one or more networks 204, e.g., the internet.

[0053] Training corpus source 202 may include an electronic clinical records system, such as an LIS, a database, a compendium of clinical data, or any othersource of images suitable for use as a training corpus as disclosed herein. Due to hardware volatile memory storage limitations, each constituent image of an image may be broken down into a number of tiles, which may be, e.g., 128 pixels by 128 pixels. Such tiles are examples of “components” as that term is used herein. According to some embodiments, each component is implemented as a vector, such as a feature vector, that represents a respective tile. Thus, the term “component” refers to both a tile and a feature vector representing a tile.

[0054] Computer 201 may be implemented as any of a desktop computer, a laptop computer, can be incorporated in one or more servers, clusters, or other computers or hardware resources, or can be implemented using cloud-based resources. Computer 201 includes volatile memory 214 and persistent memory 212, the latter of which can store computer-readable instructions, that, when executed by electronic processor 210, configure computer 201 to perform any of the methods disclosed herein, including method 100, 110, or 120, and / or form or store any electronic neural network, and / or perform any classification technique as described herein. Computer 201 further includes network interface 208, which communicatively couples computer 201 to training corpus source 202 via network 204. Other configurations of system 200, associated network connections, and other hardware, software, and service resources are possible.

[0055] Certain embodiments can be performed using a computer program or set of programs. The computer programs can exist in a variety of forms both active and inactive. For example, the computer programs can exist as software program(s) comprised of program instructions in source code, object code, executable code or other formats; firmware program(s), or hardware description language (HDL) files. Any of the above can be embodied on a transitory or non-transitory computer readable medium, which include storage devices and signals, in compressed or uncompressed form. Exemplary computer readable storage devices include conventional computer system RAM (random access memory), ROM (read-only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), and magnetic or optical disks or tapes.

[0056] II. Description of Example Embodiments

[0057] EXAMPLE 1 : Label-Free Liver Tumor Segmentation

[0058] 1. Overview

[0059] Artificial intelligence (Al) has dominated medical image segmentation, but training an Al model (e.g., Il-Net) often requires a large number of annotations. Annotating medical images is not only expensive and time-consuming, but also requires extensive medical expertise, and sometimes needs the assistance of radiology reports and biopsy results to achieve annotation accuracy. Due to its high annotation cost, only roughly 200 CT scans with annotated liver tumors are publicly available (provided by LiTS) for training and testing models.

[0060] To minimize annotation expenses, generating synthetic tumors is an emerging research topic. Early attempts include, but only limited to, synthesizing COVID-19 infections, lung nodules abdominal tumors, diabetic lesions, and brain tumors. However, the synthetic tumors in those studies appear very different from the real tumors; due to this, Al models trained using synthetic tumors perform significantly worse than those trained using real tumors. What makes synthesizing tumors so hard? There are several important factors: shape, intensity, size, location, and texture. In this example, we handcraft a strategy to synthesize liver tumors in abdominal CT scans. Our key novelties include (i) location without collision with vessels, (ii) texture with scaled-up Gaussian noise, and (iii) shape generated from distorted ellipsoids. These three aspects are proposed according to the clinical knowledge of liver tumors (detailed in §3.2). The resulting synthetic tumors are realistic — even medical professionals usually confuse them with real tumors in the visual examination (Fig. 3; Table 2). In addition, the model trained on our synthetic tumors achieves a Dice Similarity Coefficient (DSC) of 59.81 % for segmenting real liver tumors, whereas Al trained on real tumors obtains a DSC of 57.63% (Fig. 4), showing that synthetic tumors have the potential to be used as an alternative to real tumors in training Al models.

[0061] These results are exciting because using synthetic tumors only, no previous work has thus far reached a similar (or even close) performance to the model trained on real tumors. Moreover, our synthesizing strategy can exhaustively generate tumors with desired locations, sizes, shapes, textures, and intensities, which are not limited to a fixed finite-size training set (the well-known limitation of the conventional training paradigm). For example, it is hard to collect sufficient training examples withsmall tumors. It is because early-stage tumors may not cause symptoms, which can delay detection, and these tumors are relatively small and exhibit subtle abnormal textures that make it difficult for radiologists to manually delineate the tumor boundaries. In contrast, our synthesis strategy can generate a large number of examples featuring small tumors.

[0062] One contribution of the present example is a synthetic tumor generator, which offers at least five advantages as summarized below.1 . The synthesis strategy embeds medical knowledge into an executable program, enabling the generation of realistic tumors through the collaboration of radiologists and computer scientists (§5.1 ; Table 2; Fig. 5).2. The entire training stage requires no annotation cost, and the resulting model significantly outperforms previous unsupervised anomaly segmentation approaches and tumor synthesis strategies (§5.2; Table 3).3. The Al model trained on synthetic tumors can achieve similar performance as Al models trained on real tumors with per-voxel annotation in real tumors segmentation, and can be generalized to CT scans with healthy liver and scans from other hospitals (§5.3; Fig. 6).4. The synthesizing strategy can generate a variety of tumors for model training, including those at small, medium, and large scales, and therefore have the potential to detect small tumors and facilitate the early detection of liver cancer (§5.4; Fig. 7).5. The synthesis strategy allows for straightforward manipulation of parameters such as tumor location, size, texture, shape, and intensity, providing a comprehensive test-bed for evaluating Al models under out-of-distribution scenarios (§5.5; Fig. 8).

[0063] These results have the potential to stimulate a shift in the tumor segmentation training paradigm, as illustrated in Fig. 4, from label-intensive to label- free Al development for tumor segmentation. Our ultimate goal is to train Al models for tumor segmentation without using manual annotation — this example makes a significant step towards it.

[0064] 2. Related Work

[0065] Unsupervised anomaly segmentation. Anomaly segmentation is a challenging application area, especially for medical diagnosis and industrial defect detection. Compared with the supervised counterparts, unsupervised methods raise more attention for their low cost and scalability. The general unsupervised anomaly detection setting is to train with normal samples only, without any anomalous data, and no image-level annotation or pixel-level annotation is provided. Under the unsupervised setting, some previous works use self-organizing maps for unsupervised anomaly detection and have introduced gradient magnitude similarity and structured similarity index losses in addition to mean square error to compute the loss of image reconstruction. Evidenced in Table 3, our label-free synthetic tumors achieve a significantly better performance in unsupervised tumor segmentation than some of the most recent work in this area.

[0066] Tumor synthesis. Successful works about tumor synthesis include polyp detection from colonoscopy videos, COVID-19 detection from Chest CT, diabetic lesion detection from retinal images, cancer detection from fluorescence microscopy images, and brain tumor detection from MRI. However, these works are restricted to the types of tumors, and other diseases, that are fairly easy to visually identify in CT scans. Most recently, synthesized liver and brain tumors have been used for pretraining and the model has been adapted to tumor segmentation within the same organ under a low-annotation regime. The manually crafted “counterfeit” tumors in the related work appear very differently from real tumors. As a result, Al algorithms, trained on synthetic tumors, may work well in detecting synthetic tumors in the test set but fail to recognize the actual tumors (evidenced in Table 3). We tackle these limitations by integrating radiologists in the tumor synthesis study for feedback (§3.2). This enables us to understand deeply about tumors, and in turn, benefit in developing Al algorithms to segment them more accurately.

[0067] Generalization from synthetic to real domains. The problem of domain generalization was introduced for zero-shot adaptation to data with a domain gap. Specifically, the goal is to make a model using data from a single or multiple related source domain(s) while achieving great generalization ability well to any target domain(s). In this example, evaluating the model generalization ability on real data is of great importance to justify whether our tumor generator is powerful enough. Domaingeneralization has been widely studied in multiple computer vision tasks like object recognition, semantic segmentation and medical imaging. As deep learning models are data hungry and annotated data are very expensive, how to train a model with synthetic data but generalize well to real data has been targeted in some previous works and some datasets are created for benchmark and further exploring. While previous works focus on preserving the transferable knowledge learned from synthetic data, this example proves that our tumor generator is powerful enough to generate tumors with reasonable domain gap and our model has outstanding generalization ability to detect real tumors (detailed in Section 5.3).

[0068] 3. Method

[0069] We have developed a hand-craft heuristic strategy to generate synthetic liver tumors. Visual inspection and prior clinical knowledge are the primary cues for improving the quality of synthetic tumors associated with the clinical characteristics of different tumors. In addition to considering the location, geometry, and texture of the tumor to be synthesized, we also consider simulating mass effect and capsule appearance due to the existence of tumors.

[0070] 3.1. Tumor Generation

[0071] To localize the liver, we first apply the pre-trained nnllNet to the CT scans. With a coarse location of the liver available, we then develop a sequence of morphological image-processing operations to synthesize realistic tumors within the liver (see Fig. 5). In some embodiments, the tumor generation consists of four steps: (1 ) location selection, (2) texture generation, (3) shape generation, and (4) postprocessing.

[0072] Location selection. The first step is to select a proper location for the tumor. This step is crucial because liver tumors usually do not allow any vessels (e.g., hepatic vein, portal vein, and inferior vena cava) to pass through them. To avoid the blood vessels, we first conduct vessel segmentation through the voxel value thresholding. The segmented vessel mask is given by the following equation:where f'(x, y, z) is the smoothed CT scan, f'(x, y, z) = f(x, y, z) 0 g(x, y, z; oa), by applying a Gaussian filter g(x, y, z; oa) with standard deviation oato the original CT scans f(x, y, z); 0 is the standard image filtering operator. Smoothing can effectively eliminate noise caused by CT reconstruction. The threshold T is set to a value slightly greater than the mean Hounsfield Unit (HU) of the liver.where l(x, y, z) is the liver mask (backgrounds, I iver= 1 ), O is point-wise multiplication, and b is a hyperparameter.

[0073] With the vessel mask, one can detect whether a chosen location is at risk of making a tumor collide with vessels. After proposing a random location (X, Y,Z) e {x, y, z | l(x, y, z) = 1 }, we conduct the collision detection by judging whether there are blood vessels within the range of tumor radius r. If 3 v(x, y, z) = 1 , V x e [X- r,X+r], y e [Y -r, Y +r], z e [Z— r,Z+r], there is a risk of collision, so the location needs to be reselected. This process iterates until a tumor location (xt, yt, zt) without collision is found. With the desirable tumor location, we are able to generate the tumor texture and shape.

[0074] Texture generation. The HU values of liver and tumor textures follow the Gaussian distributions. To obtain realistic tumor textures, we first generate a 3D Gaussian noise with the predefined mean HU intensity pt and the same standard deviation oPas the hepatic parenchyma (the liver area excluding vessels), T(x, y, z) ~ N(pt, Op). Since the random Gaussian noise is usually too sharp as the texture for tumor, we soften the texture by scaling it up with spline interpolation of the order 3 (cubic interpolation) on x, y, z directions. The scaled-up texture is denoted as T'(x, y, z) in this example, we want it to exhibit graininess close to the hepatic parenchyma. The scaling factor q e [1 ,°°) determines how rough the generated grain feels, q = 1 means the Gaussian texture is not scaled, resulting in large value fluctuation between adjacent voxels. Larger q brings greater graininess, which may be similar to the real tumor texture. Finally, considering the tomography imaging quality, we further blur the texture with Gaussian filter g(x, y, z; Ob)where Ob is the standard deviation. After blurring, the texture resembles those generated by real imaging.

[0075] Shape generation. Most tumors grow from the centers and gradually swell, making small tumors (i.e., r < 20mm) nearly spherical. This motivates us to generate tumor-like shapes with ellipsoids. We randomly sample the half-axis lengths of the ellipsoid for x, y, z directions from a uniform distribution U(0.75r, 1.25r), and place the generated ellipsoid mask centered at (xt, yt, zt). For a generated ellipsoid tumor mask t(x, y, z) (backgrounds, tumor=1 ), and with the same shape as the scanning volume f(x, y, z)), elastic deformations, controlled by oe, are applied to enrich its diversity. The deformed tumor mask is more similar to the naturally grown tumors in appearance than simple ellipsoids. In addition, it can also improve the model’s robustness by learning the shape-semantic invariance. The deformed tumor mask is denoted as t'(x, y, z). In order to make the transition between the generated tumor and the surrounding liver parenchyma more natural, we finally blur the mask by applying a Gaussian filter g(x, y, z; oc) with the standard deviation oc. To be more specifically, we obtain blur shape t"(x, y, z) = t'(x, y, z) 0 g(x, y, z; oc).

[0076] Post-processing. The first step of post-processing is placing the tumor on the scanning volume f(x, y, z) and corresponding liver mask l(x, y, z). Assuming the tumor mask array t"(x, y, z) and the texture array T"(x, y, z) have the same shape as f(x, y, z) and l(x, y, z). We can obtain new scanning volume with tumor through equation

[0077] For the new mask with the tumor (bg=0, liver=1 ,tumor=2), it can be synthesized with l'(x, y, z) = l(x, y, z)+t"(x, y, z). After placing the tumor, we adopt another two steps to make the generated tumor more realistic to medical professionals. They aim to simulate mass effect and the capsule appearance, respectively. Mass effect means the expanding tumor pushes its surrounding tissue apart. If the tumor grows large enough, it will compress the surrounding blood vessels to make them bend, or even cause the edge of the nearby liver to bulge. Local scaling warping is chosen in this work to implement mass effect. It remaps pixels in a circle tobe nearer to the circumference. For a pixel with a distance y to the circle center, the remapped pixel distance y' iswhere ymax is the radius of the expanded circular area, I e [0, 100] is a hyperparameter for controlling the expansion intensity. Larger I leads to stronger warping. Note that when I = 0, the remapping reduces to the identity function y' = y. The remapping procedure is conducted on both scanning and mask volumes. After warping, they are named f"(x, y, z) and l"(x, y, z). The latter one (liver / tumor segmentation label) is now ready for the subsequent training. Finally, we simulate the capsule appearance by brightening the tumor edge. The edge area can be obtained bywhere lb and ub are the lower bound and upper bound for filtering the edge from tumor mask. Then we increase HU intensity of the blurred edge area to simulate the capsulewhere d is the pre-defined HU intensity difference between a tumor and its capsule. The new scanning volume f'"(x, y, z) is now ready for training or Turing test. The parameters we use are shown in Table 1 .Table 1. Hyper-parameters. pP, oPare the mean and standard deviation of the hepatic parenchyma. The values are adjusted by (1 ) feedback from clinicians based on the clinical prior knowledge about liver tumors (§3.2) and (2) visual assessment between the real and synthetic tumors (§5.1 ).

[0078] 3.2. Clinical Knowledge about Liver Tumors

[0079] This example focused on generating hepatocellular carcinomas (tumor grown from liver cells). After the contrast injection, the clinical examination process of liver is divided into three phases, arterial phase (30s after injection), portal venous phase (60-70s after injection), and the delay phase (3min after injection). Typically, only the first two phases are used for detecting hepatocellular carcinomas, and the tumor HU intensity value in different stages distributes differently. The mean attenuation measurement of the lesions in the hepatic arterial phase was 111 HU (range, 32-207 HU), and it decreased in the portal venous phase to a mean of 106 HU (range, 36-162 HU). There was a mean difference of 26 HU (range, -44 to 146 HU) between the lesion and liver in the arterial phase. On average, the hepatocellular carcinomas measured 11 HU (range, -98 to 61 HU) less than the adjacent liver parenchyma in the portal venous phase. The distributional characteristics help us determine the generated mean tumor HU values.

[0080] The location, shape, and number of tumors depend on how severe the hepatocellular carcinomas are according to the standardized guidance of the Liver Imaging Reporting and Data System (LI-RADS). Milder carcinomas usually lead to smaller, fewer spherical lesions. Only one small tumor emerges in most cases. While multi-focal lesions, which means scattered small tumors, only appear in seldom cases. Severe carcinomas usually present a satellite lesion, a large lesion surrounded by a cluster of small lesions. The large central lesion also takes on a more irregular shape than small lesions. And also, larger tumors usually display evident mass effects,accompanied by capsule appearances that separate the tumor from the liver parenchyma.

[0081] 4. Experiments

[0082] Datasets. Detailed per-voxel annotations for liver tumors are provided in LiTS. The volume of liver tumors ranges from 38mm3 to 349 cm3, and the radius of tumors is in the range of [2, 44]mm. We perform 5-fold cross-validation. An Al model (e.g. Il-Net) is trained on 101 CT scans with annotated liver and liver tumors. For comparison, a dataset of 116 CT scans with healthy livers is assembled from CHAOS (20 CT scans), BTCV (47 CT scans), Pancreas-CT (38 CT scans) and health subjects in LiTS (11 CT scans). We then generate tumors in these scans on the fly, resulting in enormous image-label pairs of synthetic tumors for training the Al model. We generate five levels of tumor sizes for model training.

[0083] Evaluation metrics. Tumor segmentation performance was evaluated by Dice similarity coefficient (DSC) and Normalized Surface Dice (NSD) with 2mm tolerance; tumor detection performance was evaluated by Sensitivity and Specificity. For all the metrics above, 95% Cis were calculated and the p-value cutoff of less than 0.05 was used for defining statistical significance.

[0084] Implementation. Our codes are implemented based on the MONAI framework for both ll-Net and Swin UN-ETR. Input images are clipped with the window range of [-21 ,189] and then normalized to have zero mean and unit standard deviation. Random patches of 96 x 96 x 96 were cropped from 3D image volumes during training. All models are trained for 4,000 epochs, and the base learning rate is 0.0002. The batch size is two per GPU. We adopt the linear warmup strategy and the cosine annealing learning rate schedule. For inference, we use the sliding window strategy by setting the overlapping area ratio to 0.75.1The junior professional achieves an Accuracy, Sensitivity, and Specificity of 26.5%, 27.6%, and 25,0%, One CT scan is marked unsure."The senior professional achieves an Accuracy, Sensitivity, and Specificity of 71.0%, 63.2%, and 83.3%. 19 CT scans are marked unsure.Table 2. Results of Visual Turing Test. The test has been performed on two medical professionals with 6-year and 15-year experience. Each professional is given 50 CT scans, some of which contain real tumors and the others contain synthetic ones. The professional can mark each CT scan as real, synthetic, or unsure. “Synt” denotes synthetic tumors, P and N indicate positive and negative classes for computing Sensitivity and Specificity.

[0085] 5. Results & Discussion

[0086] Using 116 CT scans from Pancreas-CT, CHAOS, and BTCV with our label-free tumor generator, we outperformed all those methods on the LiTS benchmark, wherein previous methods used 101 CT scans and annotations from LiTS.

[0087] 5.1. Clinical Validation using Visual Turing Test

[0088] We conduct the Visual Turing Test on 50 CT scans, where 20 scans are with real tumors from LiTS, and remaining 30 scans are healthy livers from WORD with synthetic tumors. Two professionals with different experience levels take part in this test. They can inspect each sample in 3D view, which means continuous variation of slice sequence can be observed by scrolling the mouse. This is an important setting for the test because some important tumor characteristics are not obvious in 2D view (e.g., vessel collision). In the test, professionals can label each sample as real, synthetic or unsure. When calculating the performance metrics, only the samples with definite results are counted.

[0089] The testing results are shown in Table 2. For junior professionals with 6- year experience, definite judgements of 49 out of 50 samples are given. All of accuracy, sensitivity, and specificity are below 30%, which means the generated samples succeed in confusing the junior professional. In particular, the sensitivity of 27.6% means that the rest 72.4% synthetic samples are mistakenly regarded to real samples. The result verifies that our synthesis method can generate realistic tumors. According to the results given by the senior professional with 15-year experience, 36.8% synthetic samples are seemed to be real, indicating that nearly half of the generated samples can tease senior professionals. Noteworthy, the senior professional only gives 19 judgements among all 30 synthetic samples. Adding up misjudged samples and uncertain samples, a total of 18 out of 30 generated samples have confused him / her.

[0090] 5.2. Comparison with State-of-the-art Methods

[0091] We compare our label-free tumor synthesis strategy with several prominent unsupervised tumor segmentation methods designed for both natural and medical images, such as PatchCore, f-AnoGAN, VAE, and the method proposed by Yao et al. [Yao et al., “Label-free segmentation of covid-19 lesions in lung ct,” IEEE Transactions on Medical Imaging, 2021 ], To enhance the performance of these baseline methods, we focus solely on the liver region for training and testing to minimize noise caused by extraneous information. Table 3 shows that all the previous unsupervised methods exhibit suboptimal performance in segmenting real liver tumors. In contrast, our label-free tumor synthesis — a novel approach to unsupervised tumor segmentation — significantly outperforms all these methods, achieving a DSC of 59.77% and an NSD of 61 .29%. On the other hand, the model trained on real tumors using fully supervised learning achieves a DSC of 57.51 % and an NSD of 58.04%. These results highlight the potential of a paradigm shift from label-intensive to label- free tumor segmentation.

[0092] 5.3. Generalization from Different Models and Data

[0093] We verify the generalizability of synthetic tumors using Swin UNETR, including its Tiny, Small, and Base variants. Fig. 6 shows that the model trained on real tumors performs slightly better than that on synthetic tumors, but there is no statistical difference between the two results as the p-value is greater than 0.05. In addition to evaluating the models on the LiTS dataset, we assess their domain generalization ability using data from other datasets (i.e., MSD-Pancreas, MSD- Spleen, MSD-Colon). As shown in the right panel of Fig. 6, our model trained with healthy data collected from 3 different datasets shows better robustness than the model trained on real data only from LiTS, while achieving much higher Specificity on the three external datasets. It is noteworthy that higher Specificity (fewer false positives) is crucial in clinical applications as it reduces the number of patients subjected to invasive diagnostic procedures and their associated costs.

[0094] 5.4. Small Tumor Detection

[0095] Early detection of small tumors is essential for prompt cancer diagnosis, but such cases are scarce in real datasets because most patients remain asymptomatic during the early stages. Al models trained on these datasets exhibitreduced detection sensitivity for small tumors (radius < 5mm) compared with larger tumors (radius > 5mm), displaying sensitivities of 52.0% and 91.6%, respectively. Thus, an advanced tumor generator could create synthetic data containing various tumor sizes for training and testing models, addressing the size imbalance issue found in real data. The lower panel of Fig. 7 presents quantitative tumor detection performance stratified by tumor size, and the upper panel presents two cases with small tumors for qualitative comparison. Evidently, Al models (trained solely on synthetic data) outperform those trained on real tumors in detecting and segmenting small tumors in the liver. The results indicate that the generation of copious amounts of synthetic small tumors can improve the efficacy of models in detecting real small tumors, thereby playing a crucial role in the early detection of cancer.Table 3. Comparison with state-of-the-art methods. We compare our methods with other unsupervised anomaly segmentation baselines, tumor synthesis strategies and fully-supervised methods. Our method significantly outperforms all other state-of-the- art unsupervised baseline methods and even surpasses the fully-supervised method with detailed pixel-wise annotation.

[0096] 5.5. Controllable Robustness Benchmark

[0097] Standard evaluation in medical imaging is limited to determining the effectiveness of Al in detecting tumors. This is because the number of annotated tumors in the existing test datasets is not big enough to be representative of the tumors that occur in real organs and, in particular, contains only a limited number of very small tumors. We show that synthetic tumors can serve as an accessible and comprehensive source for rigorously evaluating Al’s performance in detecting tumors at a variety of different sizes and locations with the organs. To be specific, our tumor generator can synthesize liver tumors varying in five dimensions, i.e. location, size, shape, intensity, and texture, by tuning hyperparameters in the tumor generation pipeline. Taking five different options in each dimension, our tumor generator could create 25 (5x5) variants for each single CT scan. Generating a large number of synthetic tumors during testing enables us to find failure scenarios of current Almodels. After locating the worst cases of Al, we can synthesize and include worstcase tumors in the training set and fine-tune Al algorithms. Fig. 8 illustrates the out-ofdistribution (o.o.d.) benchmark created by synthetic tumors, wherein we evaluate several state-of-the-art models (trained on public datasets). These models show good robustness in the shape, location, and texture dimensions but are sensitive to tumors of extreme sizes and intensities.

[0098] 5.6. Ablation Study on Shape Generation

[0099] To show the importance of each step in the tumor generation, we design ablation studies focusing on the shape generation and synthesizing small tumors. We evaluate the models trained with different incomplete settings of synthetic strategies on two aspects: all tumor segmentation and small tumor detection. As shown in Table 4, the performance would be much poorer if without synthesizing small tumors, edge blurring or elastic deformation in the shape generation. The reasons are simple: (1 ) without elastic deformation and edge blurring steps for shape generation (shown in Fig. 5), the synthetic tumors can be extremely unrealistic (i.e. the edge is sharp and shape can only be ellipsoid). (2) The model doesn’t have the generalization ability to small tumors (radius < 5mm) when the training set does not have them.Table 4. Ablation study on shape generation. The quality of synthesized tumors influences model performance to a certain degree, emphasizing the importance of each component in our method (§3; Fig. 5). Moreover, generating tiny synthetic tumors positively impacts the sensitivity of small tumors.

[0100] 6. Conclusion

[0101] In this example, we have developed an effective strategy to synthesize liver tumors. With zero manual annotation, we verify that the Al model trained on synthetic tumors can perform similarly to the ones trained on real tumors in the LiTSdataset (which took months to create). This reveals the great potential for the use of synthesis tumors to train Al models on larger-scale healthy CT datasets (which are much easier to obtain than CT scans with liver tumors). Furthermore, synthetic tumors allow us to assess Al’s capability of detecting tumors of varying locations, sizes, shapes, intensities, textures, and stages in the CT scans. In the future, we will consider more advanced strategies such as generative adversarial nets (GANs), Diffusion Models and possibly improved with 3D geometry models like NeRF to generate better tumor texture.

[0102] EXAMPLE 2: Early Detection and Localization of Pancreatic Cancer by Label-Free Tumor Synthesis

[0103] 1. Overview

[0104] Early detection and localization of cancerous tumors are crucial for disease diagnosis, prognosis, radiotherapy planning, and surgical intervention. Pancreatic cancer, one of the deadliest forms of cancer, can be detected by visual inspection of computed tomography (CT) scans, but early signs can be subtle and easily overlooked by radiologists unless the cancer is already suspected. For the vast majority of the 40 million abdominal CT scans taken annually in the United States, there are no grounds for suspicion; hence, abdominal cancer may not be detected promptly. Artificial intelligence (Al) has the potential to assist radiologists in the early detection of cancer. Although Al technology is constantly innovating, Al is data-hungry and requires large-scale, high-quality annotated datasets. Annotating early-stage pancreatic tumors is expensive and time-consuming and requires extensive medical expertise, radiology reports, and biopsy results to achieve high annotation accuracy.

[0105] To address this challenge, we create synthetic tumors in the pancreas, including small tumors which are critical for early detection and diagnosis, to train Al to detect and localize tumors. Synthetic tumors refer to artificially generated tumors that are added to CT scans using computer algorithms. Synthetic tumors can be used to generate large datasets of paired CT scans and tumor masks without the need for human annotation. The synthetic tumors are generated using a model-based approach that mimics the appearance of real tumors. The resulting dataset of synthetic pancreatic tumors can be used to train Al models for the early detection of pancreatic tumors. Two public datasets and an in-house dataset are used for evaluation, MSD-Pancreas, PCL, and AS (Anonymous Site, censored for blind review). The results show that Al trained on synthetic tumors is comparable in the overall detection rate of pancreatic tumors to those trained on real tumors. Importantly, the use of synthetic tumors led to a significantly higher detection rate for small tumors, which are particularly challenging to detect using CT scans of real tumors (due to the lack of sufficient examples). Moreover, in conjunction with enormous synthetic tumors, training Al using CT scans with annotated tumors at an advanced stage (i.e., large tumors) leads to more accurate segmentation of pancreatic tumors than Al trained solely on real tumors. Finally, synthetic tumors can help overcome the problem of dataset bias, improving Al generalizability to CT scans collected from a variety of hospitals. We summarize exemplary contributions of this example as follows:1 . We develop an effective data synthesis method for pancreatic tumors, which overcomes several technical challenges such as tumor location, intensity distribution, and texture characteristics (Fig. 9).2. We quantify the fidelity of synthetic tumors by conducting radiologist reviews and training a binary classifier. In the discrimination of real and synthetic tumors, radiologists and binary classifiers yield an accuracy of 70.0% and 62.1 %, respectively. (Fig. 11 ).3. Al trained on synthetic tumors achieves comparable detection and localization performance to Al trained on real tumors: with FP / Subject <1 , Cyst detection could achieve an average Sensitivity of 95.8% with synthetic training data, and PDAC detection could achieve an average Sensitivity of 95.6% with synthetic training data (Table 4).4. We show the ability of early tumor detection for our synthetic data strategy on our in-house dataset. For early-stage pancreatic tumors (radius<20mm), the model trained with synthetic data performed slightly better than the model trained with real data.5. We show using hybrid training data (real together with synthetic data) could boost the performance of segmentation. Al trained on a small set of real tumors (about 100 cases from either MSD or PCL) and enormous synthetic tumors performs tumor segmentation significantly better than those trained solely on real tumors,highlighting the potential of synthetic tumors as a compelling data augmentation strategy (Table 2).6. We demonstrate the effectiveness of synthetic tumors for unsupervised domain adaptation, wherein the Al model was trained on public datasets and can be generalized to our in-house dataset without additional annotation (Table 3).

[0106] Related work. By synthesizing images of diseased regions, researchers can generate large datasets that can be used to train Al algorithms for disease detection and segmentation in many organs, including colon, lung, brain, and others. However, the performance of Al trained on these synthetic data performs significantly worse than that trained on real data. As exemplified herein, a holistic tumor synthesis method has been devised that suggests Al trained on synthetic data can perform similarly to real data. But also disclosed herein, this conclusion can be further generalized to more sophisticated organs other than the liver. This example focuses on pancreatic tumors, which are much harder to synthesize due to several factors. (I) The pancreas is a relatively small organ surrounded by other structures, such as the liver, stomach, and intestines, making it difficult to visualize and isolate tumors in CT scans. (II) Pancreatic tumors have complex and heterogeneous morphology, making it difficult to capture the full range of tumor features using synthetic data. (Ill) Early- stage tumors are rare, small, and difficult to detect in CT scans, making it challenging to collect a sufficiently large number of examples for reference.

[0107] 2. Method

[0108] To synthesize pancreatic tumors, three steps were followed: Shape generation, position generation, and texture generation. All mentioned steps were built based on the statistics of sampled pancreatic tumors. In this section, we do not distinguish PDACs from Cysts because they all follow the same tumor generation workflow; thus, they are all referred to as ’tumors’. The complete tumor generation workflow can be found in Fig. 10(c).

[0109] Shape generation. During the statistics phase, we calculated the bounding boxes of the pancreatic tumors and analyzed their shapes. Based on our analysis, ellipsoids could best fit pancreatic tumors, although we do need to do modifications to the ellipsoids. First, the dimensions of said ellipsoid were sampledfrom the distribution of tumor sizes. As Fig. 9 tumor size distribution shows, the distribution of tumor sizes follows a positive skewed normal distribution. To generate more small tumors, we manually stratified the sizes and added more small tumors. Then, an elastic deformation was applied to the target ellipsoid, so the shape could be more natural, see Fig. 9 left.

[0110] Position generation. After we have the geometry of the tumor, the position needs to be selected. As Fig. 9 right part shows, pancreatic tumors could present everywhere in the pancreas (from head to tail), so it is reasonable to randomly select one point in the pancreas as the center of the synthetic tumor. Such position generation strategy could lead to a synthetic tumor T with most of its content outside the pancreas P. It could be possible in a real human body. However, we must pay additional attention to such extreme cases in the texture generation step.

[0111] Texture generation. For each generated tumor T with a fixed position x, y, z, a small neighborhood of pancreas N(Tx,y,z , r) around the tumor was sampled to calculate the median intensity difference of the tumor Al, where r is the sampling radius:a, [3 are linear regression coefficients of intensity (see Fig. 10), and e is a small random term to add more randomness to intensity. The texture of the tumor follows a normal distribution. And finally, a Gaussian blur mask is applied to the generated texture to smooth the edge of the tumor. For tumors with extreme positions, only pancreas intensity (instead of all surrounding organs) should be calculated for Al.Table 1. Al trained by synthetic tumors achieves an average Sensitivity improvement from 81.3% to 97.8% at 0.7, 0.8, 0.9, and 1.0 false positives per subject compared with Al trained by real tumors; however, synthetic tumors are outperformed by real tumors for segmenting pancreatic cysts. In general, label-free synthetic tumors enable Al to segment pancreatic tumors as effectively as label-intensive real tumors.

[0112] 3. Experiment & Result

[0113] Implementation and Datasets. We tested our algorithm on three datasets. Our in-house dataset AS is a contrast-enhanced pancreatic CT dataset. For training, AS has 662 healthy cases, 443 cases with Cyst, and 1015 cases with PDAC. Fortesting purposes, there are 323 cases which is a hybrid of healthy, Cyst, and PDAC cases. 2 public datasets were included during the study: MSD and PCL. Because the testing labels were not available to us, we only used images from the training set. We randomly split MSD to 197 in the training set and 84 in the test set. We randomly split PCL to 152 in the training set and 52 in the test set. All cases in MSD are diseased, but whether the patients had Cyst or PDAC was unidentified. All cases in PCL was mentioned to have Cysts. To calculate the statistics of pancreatic tumors, 100 cases with PDAC and 100 cases with Cyst were randomly sampled from AS. For the Turing test, we randomly selected 50 cases with Cyst, and 50 healthy cases with synthetic Cyst from AS. For each case, one slice with the tumor was extracted, together with the tumor masks. During the training stage, synthetic tumors were generated on the fly. During each epoch, each healthy image yields two images with synthetic tumors. The tumor’s size and position were randomly sampled on the tumor statistics, and the median intensity of the tumor was calculated with the intensity regression.Table 2. Al trained by a combination of real and synthetic tumors performs better than that trained solely on real tumors. This has been validated on two publicly available datasets, suggesting that synthetic tumors have the potential to serve as a promising data augmentation strategy as complementary to real tumors. Therefore, numerous public CT scans with a healthy pancreas — relatively easier to collect with the help of radiology reports — can also be augmented and used for training Al models for tumor segmentation.

[0114] Results of Visual Turing Test. First, we need to understand whether the synthetic tumors generated look realistic or not. The capability of synthesizing realistic tumors is important because it enables us to generate a lot of training data for Al algorithms, particularly to get examples of small tumors that may appear only occasionally in the annotated datasets. To visually assess tumor synthesis quality, werecruited two radiologists to distinguish “fake” tumors generated by our method from real tumors. Each radiologist was provided with 100 images, and half of them were synthesized. As Fig. 11 shows, for tumors of all sizes, the senior radiologist could correctly classify 70% of the images.

[0115] However, the junior radiologist could only correctly identify 29% of them. Such results mean that about 60% of our synthetic data could even fool a clinician with domain expertise. We further stratified the results by the radius of the tumor (the threshold was 20mm as recommended by our radiologists), and the classification accuracy of the senior radiologist dropped, which coincides with our assumption: small tumors are harder to observe and are more difficult to diagnose.Table 3. Synthetic tumors enable more robust domain adaptation because they are be generated to CT scans collected from novel domains (hospitals). We evaluate this by using our large-scale in-house dataset, namely AS, where PDAC and Cyst are annotated. The Al model is trained on CT scans from public datasets with real tumors and CT scans from the in-house dataset with a healthy pancreas. The Sensitivity (Sen) is computed at 0.7 false positives per subject.

[0116] Al binary classifier. Following the idea of adversarial generative models, we trained an Al classifier to check whether the synthetic tumors could confuse deep learning algorithms. A ResNet-50 was trained to classify extracted 2D tumors together with the annotation masks. As shown in Fig. 11 , the trained ResNet- 50 showed a similar classification ability to the senior radiologist: it could classify all sizes of tumors better than small tumors. This is evidence that our synthetic tumor algorithm works better on small tumors, which makes it a promising method to use to enhance better the prevalence of small tumors in pancreatic CT datasets.

[0117] Results on a large-scale in-house dataset. This is a set of experiments that we conducted with our in-house dataset to show synthetic tumors could be able to train a segmentation deep neural network with reasonable performance. We followed the state-of-the-art work on pancreatic tumor segmentation,and trained a single-phase segmentation neural network with the nnll-Net framework. This network training strategy was applied to all the following experiments. The models trained with synthetic tumors for Cyst and PDAC could achieve a similar tumor detection performance. For PDAC, Al trained on synthetic tumors outperformed the model trained by real data significantly. Actually, models trained with synthetic data could achieve a lower FP / subject at 0.4 and still perform a reasonable sensitivity (around 95%). To test whether our data could help boost the performance of small tumor detection, we stratified the detection results based on the radius of the tumors. Tumors with a radius < 20mm are considered small tumors. Because in the training stage, we synthesized more small tumors than their actual prevalence, model trained with synthetic data has a better ability to detect small tumors. However, with extremely small ones (radius < 10mm), further research is still needed. But it will be a challenging task since a small pancreatic tumor looks might exactly like its neighbor and could be hard to detect with only one phase of CT.

[0118] Results on two public datasets. On these two public datasets, we showed that a hybrid data training strategy (to train the model with both real data and synthetic data) could provide a much stronger performance. All cases in these two public datasets are diseased, and for MSD, we do not know which kind of tumor is in the CT. We adopted our tumor generation method directly and ignored the fact that our synthetic tumor could overlap with the real tumor. Table 2 shows that segmentation tasks could benefit from involving synthetic data in the training loop, as the Dice scores both improved from training with real data only.

[0119] Domain adaptation from public to in-house datasets. Finally, we tested models trained with public datasets on our in-house data AS. Table 3 shows, with synthetic data, models could perform better on an unforeseen dataset. However, we need to notice that the model generation ability is still limited by the similarities between the source dataset and the target dataset. For example, PCL has no PDAC, so even if we augmented the dataset with synthetic PDAC, its ability to detect PDAC on AS was still below expectation.

[0120] 4. Discussion & Conclusion

[0121] In this example, we presented a method to generate pancreatic tumors with label-free tumor synthesis, and these synthetic tumors could help boost theperformance of detection and localization of pancreatic cancer. Compared with synthetic liver tumors, synthesizing pancreatic tumors is a more difficult task: the size of the organ is small, and the contrast of the tumor with its surrounding tissues is subtle. And thus, the generation of synthetic tumors could help the training of deep neural networks more, since the networks could be trained with a larger variety of data. We provided evidence that training with synthetic data and hybrid training could boost the segmentation performance of pancreatic tumors. We still observed the weakness of all current methods: the detection rate is still low for small pancreatic tumors. This version of the synthetic method depends on some key parameters, for example, differences in intensities and size. With related clinical knowledge, we could generate these kinds of parameters for other kinds of tumors in different organs. On the other hand, it would be better if such parameters could also be learned in a zero-shot way. Future potential research for synthetic pancreatic tumors could include (I) synthesizing Pancreatic neuroendocrine tumors (pNET), since pNET is a more difficult tumor to identify and diagnose and (II) automatic parameter learning with a generative adversarial network or diffusion model.

[0122] In conclusion, synthetic tumors provide a promising direction to generate large-scale, label-free synthetic data and annotations for training Al in organs, such as the pancreas, that lack detailed, per-voxel tumor annotations. The use of synthetic tumors is attractive as it requires no manual annotation and enables us to specify the location, size, shape, intensity, and texture of the tumor to be synthesized. Moreover, synthetic tumors can also be used as supplementary data when training Al with real tumors. Specifically, a variety of small tumors automatically generated by our method has demonstrated the potential to improve the early detection of pancreatic cancer.

[0123] Some further aspects are defined in the following clauses:

[0124] Clause 1 : A computer-implemented method of producing a computed tomography (CT) scan that comprises a synthetic tumor, the method comprising: identifying a target tissue on the CT scan to produce an identified target tissue; selecting a location at least proximal to the identified target tissue on the CT scan to position a synthetic tumor to produce a selected location; generating a simulated shape of the synthetic tumor to produce a synthetic tumor shape; producing a simulated texture of the synthetic tumor to produce a synthetic tumor texture; and,combining the synthetic tumor shape and the synthetic tumor texture at the selected location on the CT scan, thereby producing the CT scan that comprises the synthetic tumor.

[0125] Clause 2: The computer-implemented method of Clause 1 , comprising producing multiple synthetic tumors such that the CT scan comprises the multiple synthetic tumors.

[0126] Clause 3: The computer-implemented method of Clause 1 or Clause 2, wherein the CT scan comprises an image of at least one anatomical region of a subject.

[0127] Clause 4: The computer-implemented method of any one of the preceding Clauses 1-3, wherein an organ comprises the identified target tissue.

[0128] Clause 5: The computer-implemented method of any one of the preceding Clauses 1-4, wherein the organ comprises a liver.

[0129] Clause 6: The computer-implemented method of any one of the preceding Clauses 1-5, wherein the organ comprises a pancreas.

[0130] Clause 7: The computer-implemented method of any one of the preceding Clauses 1-6, wherein a mammalian organism comprises the identified target tissue.

[0131] Clause 8: The computer-implemented method of any one of the preceding Clauses 1-7, wherein the mammalian organism comprises a human organism.

[0132] Clause 9: The computer-implemented method of any one of the preceding Clauses 1-8, wherein the identifying step comprises passing the CT scan through an electronic neural network that is trained to identify the target tissue in CT scans.

[0133] Clause 10: The computer-implemented method of any one of the preceding Clauses 1-9, wherein the identified target tissue comprises the selected location.

[0134] Clause 11 : The computer-implemented method of any one of the preceding Clauses 1-10, wherein the selected location substantially avoids vasculature imaged in the CT scan.

[0135] Clause 12: The computer-implemented method of any one of the preceding Clauses 1 -11 , comprising conducting vessel segmentation using a voxel value thresholding technique to substantially avoid the vasculature imaged in the CT scan.

[0136] Clause 13: The computer-implemented method of any one of the preceding Clauses 1 -12, comprising applying at least one segmented vessel mask to substantially avoid the vasculature imaged in the CT scan.

[0137] Clause 14: The computer-implemented method of any one of the preceding Clauses 1 -13, wherein the generating step comprises applying an ellipsoid shaped mask at the selected location.

[0138] Clause 15: The computer-implemented method of any one of the preceding Clauses 1 -14, wherein the generating step comprises performing ellipsoid mask generation, elastic deformation of the ellipsoid mask, and / or ellipsoid mask blurring using a Gaussian filter to produce the synthetic tumor shape.

[0139] Clause 16: The computer-implemented method of any one of the preceding Clauses 1 -15, wherein the synthetic tumor shape comprises a deformed ellipsoidal shape.

[0140] Clause 17: The computer-implemented method of any one of the preceding Clauses 1 -16, wherein a tumor mask array comprises the synthetic tumor shape.

[0141] Clause 18: The computer-implemented method of any one of the preceding Clauses 1 -17, wherein the producing step comprises performing salt and / or pepper noising and / or denoising, Gaussian filtering, scaling, and / or clipping to produce the synthetic tumor texture.

[0142] Clause 19: The computer-implemented method of any one of the preceding Clauses 1 -18, wherein a texture array comprises the synthetic tumor texture.

[0143] Clause 20: The computer-implemented method of any one of the preceding Clauses 1 -19, comprising performing the generating step concurrent with or after performing the producing step.

[0144] Clause 21 : The computer-implemented method of any one of the preceding Clauses 1 -20, wherein the synthetic tumor comprises a radius selected fromthe group consisting of: less than about 5 mm, between about 5 and about 10 mm, and greater than about 10 mm.

[0145] Clause 22: The computer-implemented method of any one of the preceding Clauses 1 -21 , wherein the combining step comprises placing the synthetic tumor shape and the synthetic tumor texture on a scanning volume and a corresponding target tissue mask volume.

[0146] Clause 23: The computer-implemented method of any one of the preceding Clauses 1 -22, wherein the scanning volume and / or the target tissue mask volume is unlabeled.

[0147] Clause 24: The computer-implemented method of any one of the preceding Clauses 1 -23, comprising using the scanning volume and / or the target tissue mask volume to train an electronic neural network that comprises non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least detecting one or more tumors in the target tissue in a CT scan obtained from a test subject.

[0148] Clause 25: The computer-implemented method of any one of the preceding Clauses 1 -24, wherein the electronic neural network comprises a Dice Similarity Coefficient (DSC) of more than about 50% for segmenting real tumors in CT scans obtained from test subjects.

[0149] Clause 26: The computer-implemented method of any one of the preceding Clauses 1 -25, comprising producing multiple CT scans that each comprise one or more synthetic tumors placed at one or more locations at least proximal to the identified target tissue, and wherein the method comprises using scanning volumes and / or target tissue mask volumes of the multiple CT scans to train the electronic neural network.

[0150] Clause 27: The computer-implemented method of any one of the preceding Clauses 1 -26, further comprising using additional CT scans that comprise the identified target tissue to train the electronic neural network, wherein the additional CT scans are obtained from reference subjects.

[0151] Clause 28: The computer-implemented method of any one of the preceding Clauses 1 -27, wherein at least some of the additional CT scans comprise images of real tumors that comprise per-voxel annotation.

[0152] Clause 29: The computer-implemented method of any one of the preceding Clauses 1 -28, wherein at least some of the additional CT scans are labeled.

[0153] Clause 30: The trained electronic neural network produced by the method of any one of the preceding Clauses 1 -29.

[0154] Clause 31 : The CT scan that comprises the synthetic tumor produced by the method of any one of the preceding Clauses 1 -29.

[0155] Clause 32: The computer-implemented method of any one of the preceding Clauses 1 -29, further comprising modifying a shape of one or more edges of the synthetic tumor to simulate a mass effect.

[0156] Clause 33: The computer-implemented method of any one of the preceding Clauses 1 -29 or Clause 32, comprising modifying the edges of the synthetic tumor using a local scaling warping technique to simulate the mass effect.

[0157] Clause 34: The computer-implemented method of any one of the preceding Clauses 1 -29, Clause 32, or Clause 33, further comprising modifying an intensity of one or more edges of the synthetic tumor to simulate a capsule appearance of the synthetic tumor.

[0158] Clause 35: The computer-implemented method of any one of the preceding Clauses 1 -29 or Clauses 32-34, comprising brightening one or more portions of the edges of the synthetic tumor to simulate the capsule appearance of the synthetic tumor.

[0159] Clause 36: A computer-implemented method of determining whether a tumor is present in a test subject, the method comprising: passing a test data set that comprises a test computed tomography (CT) scan obtained from the test subject through an electronic neural network, wherein the test CT scan comprises an image of a target tissue of the test subject, wherein the electronic neural network has been trained on a set of training data that comprises multiple reference CT scans that each comprises one or more synthetic tumors placed at one or more locations at least proximal to the target tissue in the multiple reference CT scans, and wherein a given synthetic tumor in the set of training data comprises a synthetic tumor shape and a synthetic tumor texture; and, outputting from the electronic neural network a detection score for the test data set, wherein a detection score value that is above a predetermined threshold value indicates that the tumor is present in the test subject,and wherein a detection score value that is below the predetermined threshold value indicates that the tumor is not present in the test subject, thereby determining whether the tumor is present in the test subject.

[0160] Clause 37: The computer-implemented method of Clause 36, wherein the electronic neural network comprises a Dice Similarity Coefficient (DSC) of more than about 50% for segmenting real tumors in test data sets.

[0161] Clause 38: The computer-implemented method of Clause 36 or Clause 37, wherein the method of determining whether the tumor is present in the test subject comprises a sensitivity of at least about 85%, at least about 90%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, or at least about 99%.

[0162] Clause 39: The computer-implemented method of any one of the preceding Clauses 36-38, wherein the set of training data comprises at least about 100, at least about 1000, at least about 10000, at least about 100000, at least about 1000000, at least about 10000000, or more reference CT scans.

[0163] Clause 40: The computer-implemented method of any one of the preceding Clauses 36-39, wherein the given synthetic tumor in the set of training data further comprises a simulated mass effect and / or a simulated capsule appearance.

[0164] Clause 41 : The computer-implemented method of any one of the preceding Clauses 36-40, wherein the set of training data is unlabeled.

[0165] Clause 42: The computer-implemented method of any one of the preceding Clauses 36-41 , wherein the set of training data comprises paired CT scans and tumor masks.

[0166] Clause 43: The computer-implemented method of any one of the preceding Clauses 36-42, wherein the set of training data that further comprises additional CT scans that comprise the target tissue, wherein the additional CT scans are obtained from reference subjects, and wherein at least some of the additional CT scans comprise images of real tumors that comprise per-voxel annotation.

[0167] Clause 44: A system for producing a computed tomography (CT) scan that comprises a synthetic tumor, the system comprising: a processor; and, a memory communicatively coupled to the processor, the memory storing non-transitory computer executable instructions which, when executed on the processor, perform operations comprising: identifying a target tissue on the CT scan to produce anidentified target tissue; selecting a location at least proximal to the identified target tissue on the CT scan to position a synthetic tumor to produce a selected location; generating a simulated shape of the synthetic tumor to produce a synthetic tumor shape; producing a simulated texture of the synthetic tumor to produce a synthetic tumor texture; and combining the synthetic tumor shape and the synthetic tumor texture at the selected location on the CT scan.

[0168] Clause 45: The system of Clause 44, wherein the non-transitory computer executable instructions which, when executed on the processor, further perform operations comprising: modifying a shape of one or more edges of the synthetic tumor to simulate a mass effect.

[0169] Clause 46: The system of Clause 44 or Clause 45, wherein the non- transitory computer executable instructions which, when executed on the processor, further perform operations comprising: modifying an intensity of one or more edges of the synthetic tumor to simulate a capsule appearance of the synthetic tumor.

[0170] Clause 47: A system for producing a computed tomography (CT) scan that comprises a synthetic tumor, the system comprising: a processor; and, a memory communicatively coupled to the processor, the memory storing non-transitory computer executable instructions which, when executed on the processor, perform operations comprising: passing a test data set that comprises a test computed tomography (CT) scan obtained from the test subject through an electronic neural network, wherein the test CT scan comprises an image of a target tissue of the test subject, wherein the electronic neural network has been trained on a set of training data that comprises multiple reference CT scans that each comprises one or more synthetic tumors placed at one or more locations at least proximal to the target tissue in the multiple reference CT scans, and wherein a given synthetic tumor in the set of training data comprises a synthetic tumor shape and a synthetic tumor texture; and outputting from the electronic neural network a detection score for the test data set, wherein a detection score value that is above a predetermined threshold value indicates that the tumor is present in the test subject, and wherein a detection score value that is below the predetermined threshold value indicates that the tumor is not present in the test subject.

[0171] Clause 48: A computer readable media comprising non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least: identifying a target tissue on the CT scan to produce an identified target tissue; selecting a location at least proximal to the identified target tissue on the CT scan to position a synthetic tumor to produce a selected location; generating a simulated shape of the synthetic tumor to produce a synthetic tumor shape; producing a simulated texture of the synthetic tumor to produce a synthetic tumor texture; and combining the synthetic tumor shape and the synthetic tumor texture at the selected location on the CT scan.

[0172] Clause 49: A computer readable media comprising non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least: passing a test data set that comprises a test computed tomography (CT) scan obtained from the test subject through an electronic neural network, wherein the test CT scan comprises an image of a target tissue of the test subject, wherein the electronic neural network has been trained on a set of training data that comprises multiple reference CT scans that each comprises one or more synthetic tumors placed at one or more locations at least proximal to the target tissue in the multiple reference CT scans, and wherein a given synthetic tumor in the set of training data comprises a synthetic tumor shape and a synthetic tumor texture; and outputting from the electronic neural network a detection score for the test data set, wherein a detection score value that is above a predetermined threshold value indicates that the tumor is present in the test subject, and wherein a detection score value that is below the predetermined threshold value indicates that the tumor is not present in the test subject.

[0173] While the invention has been described with reference to the exemplary embodiments thereof, those skilled in the art will be able to make various modifications to the described embodiments without departing from the true spirit and scope. The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. In particular, although the method has been described by examples, the steps of the method can be performed in a different order than illustrated or simultaneously. Those skilled in the art will recognize that these and other variations are possible within the spirit and scope as defined in the following claims and theirequivalents. All patents, patent applications, other publications or documents, and the like cited herein are incorporated by reference in their entirety for all purposes to the same extent as if each individual item were specifically and individually indicated to be so incorporated by reference.

Claims

What is claimed is:1 . A computer-implemented method of producing a computed tomography (CT) scan that comprises a synthetic tumor, the method comprising: identifying a target tissue on the CT scan to produce an identified target tissue; selecting a location at least proximal to the identified target tissue on the CT scan to position a synthetic tumor to produce a selected location; generating a simulated shape of the synthetic tumor to produce a synthetic tumor shape; producing a simulated texture of the synthetic tumor to produce a synthetic tumor texture; and, combining the synthetic tumor shape and the synthetic tumor texture at the selected location on the CT scan, thereby producing the CT scan that comprises the synthetic tumor.

2. The computer-implemented method of any one of the preceding claims, comprising producing multiple synthetic tumors such that the CT scan comprises the multiple synthetic tumors.

3. The computer-implemented method of any one of the preceding claims, wherein the CT scan comprises an image of at least one anatomical region of a subject.

4. The computer-implemented method of any one of the preceding claims, wherein an organ comprises the identified target tissue.

5. The computer-implemented method of any one of the preceding claims, wherein the organ comprises a liver.

6. The computer-implemented method of any one of the preceding claims, wherein the organ comprises a pancreas.

7. The computer-implemented method of any one of the preceding claims, wherein a mammalian organism comprises the identified target tissue.

8. The computer-implemented method of any one of the preceding claims, wherein the mammalian organism comprises a human organism.

9. The computer-implemented method of any one of the preceding claims, wherein the identifying step comprises passing the CT scan through an electronic neural network that is trained to identify the target tissue in CT scans.

10. The computer-implemented method of any one of the preceding claims, wherein the identified target tissue comprises the selected location.11 . The computer-implemented method of any one of the preceding claims, wherein the selected location substantially avoids vasculature imaged in the CT scan.

12. The computer-implemented method of any one of the preceding claims, comprising conducting vessel segmentation using a voxel value thresholding technique to substantially avoid the vasculature imaged in the CT scan.

13. The computer-implemented method of any one of the preceding claims, comprising applying at least one segmented vessel mask to substantially avoid the vasculature imaged in the CT scan.

14. The computer-implemented method of any one of the preceding claims, wherein the generating step comprises applying an ellipsoid shaped mask at the selected location.

15. The computer-implemented method of any one of the preceding claims, wherein the generating step comprises performing ellipsoid mask generation,elastic deformation of the ellipsoid mask, and / or ellipsoid mask blurring using a Gaussian filter to produce the synthetic tumor shape.

16. The computer-implemented method of any one of the preceding claims, wherein the synthetic tumor shape comprises a deformed ellipsoidal shape.

17. The computer-implemented method of any one of the preceding claims, wherein a tumor mask array comprises the synthetic tumor shape.

18. The computer-implemented method of any one of the preceding claims, wherein the producing step comprises performing salt and / or pepper noising and / or denoising, Gaussian filtering, scaling, and / or clipping to produce the synthetic tumor texture.

19. The computer-implemented method of any one of the preceding claims, wherein a texture array comprises the synthetic tumor texture.

20. The computer-implemented method of any one of the preceding claims, comprising performing the generating step concurrent with or after performing the producing step.21 . The computer-implemented method of any one of the preceding claims, wherein the synthetic tumor comprises a radius selected from the group consisting of: less than about 5 mm, between about 5 and about 10 mm, and greater than about 10 mm.

22. The computer-implemented method of any one of the preceding claims, wherein the combining step comprises placing the synthetic tumor shape and the synthetic tumor texture on a scanning volume and a corresponding target tissue mask volume.

23. The computer-implemented method of any one of the preceding claims, wherein the scanning volume and / or the target tissue mask volume is unlabeled.

24. The computer-implemented method of any one of the preceding claims, comprising using the scanning volume and / or the target tissue mask volume to train an electronic neural network that comprises non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least detecting one or more tumors in the target tissue in a CT scan obtained from a test subject.

25. The computer-implemented method of any one of the preceding claims, wherein the electronic neural network comprises a Dice Similarity Coefficient (DSC) of more than about 50% for segmenting real tumors in CT scans obtained from test subjects.

26. The computer-implemented method of any one of the preceding claims, comprising producing multiple CT scans that each comprise one or more synthetic tumors placed at one or more locations at least proximal to the identified target tissue, and wherein the method comprises using scanning volumes and / or target tissue mask volumes of the multiple CT scans to train the electronic neural network.

27. The computer-implemented method of any one of the preceding claims, further comprising using additional CT scans that comprise the identified target tissue to train the electronic neural network, wherein the additional CT scans are obtained from reference subjects.

28. The computer-implemented method of any one of the preceding claims, wherein at least some of the additional CT scans comprise images of real tumors that comprise per-voxel annotation.

29. The computer-implemented method of any one of the preceding claims, wherein at least some of the additional CT scans are labeled.

30. The trained electronic neural network produced by the method of any one of the preceding claims.31 . The CT scan that comprises the synthetic tumor produced by the method of any one of the preceding claims.

32. The computer-implemented method of any one of the preceding claims, further comprising modifying a shape of one or more edges of the synthetic tumor to simulate a mass effect.

33. The computer-implemented method of any one of the preceding claims, comprising modifying the edges of the synthetic tumor using a local scaling warping technique to simulate the mass effect.

34. The computer-implemented method of any one of the preceding claims, further comprising modifying an intensity of one or more edges of the synthetic tumor to simulate a capsule appearance of the synthetic tumor.

35. The computer-implemented method of any one of the preceding claims, comprising brightening one or more portions of the edges of the synthetic tumor to simulate the capsule appearance of the synthetic tumor.

36. A computer-implemented method of determining whether a tumor is present in a test subject, the method comprising: passing a test data set that comprises a test computed tomography (CT) scan obtained from the test subject through an electronic neural network, wherein the test CT scan comprises an image of a target tissue of the test subject, wherein the electronic neural network has been trained on a set of training data that comprises multiple reference CT scans that each comprises one or more synthetic tumorsplaced at one or more locations at least proximal to the target tissue in the multiple reference CT scans, and wherein a given synthetic tumor in the set of training data comprises a synthetic tumor shape and a synthetic tumor texture; and, outputting from the electronic neural network a detection score for the test data set, wherein a detection score value that is above a predetermined threshold value indicates that the tumor is present in the test subject, and wherein a detection score value that is below the predetermined threshold value indicates that the tumor is not present in the test subject, thereby determining whether the tumor is present in the test subject.

37. The computer-implemented method of any one of the preceding claims, wherein the electronic neural network comprises a Dice Similarity Coefficient (DSC) of more than about 50% for segmenting real tumors in test data sets.

38. The computer-implemented method of any one of the preceding claims, wherein the method of determining whether the tumor is present in the test subject comprises a sensitivity of at least about 85%, at least about 90%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, or at least about 99%.

39. The computer-implemented method of any one of the preceding claims, wherein the set of training data comprises at least about 100, at least about 1000, at least about 10000, at least about 100000, at least about 1000000, at least about 10000000, or more reference CT scans.

40. The computer-implemented method of any one of the preceding claims, wherein the given synthetic tumor in the set of training data further comprises a simulated mass effect and / or a simulated capsule appearance.41 . The computer-implemented method of any one of the preceding claims, wherein the set of training data is unlabeled.

42. The computer-implemented method of any one of the preceding claims, wherein the set of training data comprises paired CT scans and tumor masks.

43. The computer-implemented method of any one of the preceding claims, wherein the set of training data that further comprises additional CT scans that comprise the target tissue, wherein the additional CT scans are obtained from reference subjects, and wherein at least some of the additional CT scans comprise images of real tumors that comprise per-voxel annotation.

44. A system for producing a computed tomography (CT) scan that comprises a synthetic tumor, the system comprising: a processor; and, a memory communicatively coupled to the processor, the memory storing non-transitory computer executable instructions which, when executed on the processor, perform operations comprising: identifying a target tissue on the CT scan to produce an identified target tissue; selecting a location at least proximal to the identified target tissue on the CT scan to position a synthetic tumor to produce a selected location; generating a simulated shape of the synthetic tumor to produce a synthetic tumor shape; producing a simulated texture of the synthetic tumor to produce a synthetic tumor texture; and combining the synthetic tumor shape and the synthetic tumor texture at the selected location on the CT scan.

45. The system of any one of the preceding claims, wherein the non- transitory computer executable instructions which, when executed on the processor, further perform operations comprising: modifying a shape of one or more edges of the synthetic tumor to simulate a mass effect.

46. The system of any one of the preceding claims, wherein the non- transitory computer executable instructions which, when executed on the processor, further perform operations comprising: modifying an intensity of one or more edges of the synthetic tumor to simulate a capsule appearance of the synthetic tumor.

47. A system for determining whether a tumor is present in a test subject, the system comprising: a processor; and, a memory communicatively coupled to the processor, the memory storing non-transitory computer executable instructions which, when executed on the processor, perform operations comprising: passing a test data set that comprises a test computed tomography (CT) scan obtained from the test subject through an electronic neural network, wherein the test CT scan comprises an image of a target tissue of the test subject, wherein the electronic neural network has been trained on a set of training data that comprises multiple reference CT scans that each comprises one or more synthetic tumors placed at one or more locations at least proximal to the target tissue in the multiple reference CT scans, and wherein a given synthetic tumor in the set of training data comprises a synthetic tumor shape and a synthetic tumor texture; and outputting from the electronic neural network a detection score for the test data set, wherein a detection score value that is above a predetermined threshold value indicates that the tumor is present in the test subject, and wherein a detection score value that is below the predetermined threshold value indicates that the tumor is not present in the test subject.

48. A computer readable media comprising non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least: identifying a target tissue on the CT scan to produce an identified target tissue;selecting a location at least proximal to the identified target tissue on the CT scan to position a synthetic tumor to produce a selected location; generating a simulated shape of the synthetic tumor to produce a synthetic tumor shape; producing a simulated texture of the synthetic tumor to produce a synthetic tumor texture; and combining the synthetic tumor shape and the synthetic tumor texture at the selected location on the CT scan.

49. A computer readable media comprising non-transitory computer executable instructions which, when executed by at least one electronic processor, perform at least: passing a test data set that comprises a test computed tomography (CT) scan obtained from the test subject through an electronic neural network, wherein the test CT scan comprises an image of a target tissue of the test subject, wherein the electronic neural network has been trained on a set of training data that comprises multiple reference CT scans that each comprises one or more synthetic tumors placed at one or more locations at least proximal to the target tissue in the multiple reference CT scans, and wherein a given synthetic tumor in the set of training data comprises a synthetic tumor shape and a synthetic tumor texture; and outputting from the electronic neural network a detection score for the test data set, wherein a detection score value that is above a predetermined threshold value indicates that the tumor is present in the test subject, and wherein a detection score value that is below the predetermined threshold value indicates that the tumor is not present in the test subject.

Images