Weakly supervised lesion segmentation

Abstract

To identify an automated technique for processing images to detect and predict sizes of biological anomalies.SOLUTION: A Generative Adversarial Network (GAN) can be trained, where the GAN includes an anomaly-removing Generator network configured to modify a medical image to remove a depiction of a biological anomaly and one or more Discriminator networks (each configured to discriminate between real and fake images). The anomaly-removing Generator network can then receive a medical image that depicts a particular biological anomaly (or pre-processed version thereof) and generate a modified image predicted to lack any depiction of the particular biological anomaly. The size of the particular biological anomaly may be estimated based on the modified image and the received image (or pre-processed version thereof).SELECTED DRAWING: Figure 1

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to PCT / US2021 / 014611, U.S. Provisional Patent Application No. 62 / 965,515, filed January 24, 2020, and U.S. Provisional Patent Application No. 62 / 966,084, filed January 27, 2020. Each of these applications is incorporated herein by reference in its entirety for all purposes. [Technical Field]

[0002] Field In general, the disclosed technology relates to estimating the size of a biological anomaly depicted in a medical image by generating a fake version of the image lacking the anomaly by using a neural network (e.g., a generator network) and subtracting the fake version from the medical image. The generator network can be trained by training a generative adversarial network (GAN) (e.g., a recycle GAN or cycle GAN) that includes the generator network. [Background technology]

[0003] background Medical imaging is often used to detect and / or monitor biological abnormalities (e.g., lesions or tumors). Quantifying the size of a biological abnormality often involves an annotator marking the contours of the abnormality on one or more images (e.g., corresponding to one or more slices). This is time-consuming and prone to error as a result of variability across annotations due to subjectivity in boundary location.

[0004] Therefore, it would be advantageous to identify automated techniques for processing images to detect and predict the size of biological abnormalities. Summary of the Invention

[0005] overview The anomaly removal generator network is used to process real images depicting a given type of biological anomaly (e.g., a tumor or lesion). The generator network may include one or more three-dimensional kernels. The processing may include generating a false image that corresponds to the real image but lacks a depiction of the given type of biological anomaly. The real and false images may then be used to segment the biological anomaly, thereby identifying the boundary, area, or volume of the biological anomaly. Additionally or alternatively, the real and false images may be used to estimate the size and / or location of the given type of biological anomaly. Estimating the size of the biological anomaly may include subtracting the false image from the corresponding real image. The size of the biological anomaly may be estimated based on the total number of pixels or voxels having an intensity above a predetermined threshold. In some cases, filtering or other processing is performed before estimating the size (e.g., by applying one or more thresholds and / or applying one or more spatial smoothing functions).

[0006] The anomaly removal generator network can be configured in response to training a larger generative adversarial network (GAN). The GAN can include a cycle GAN, which can include an anomaly removal generator network (configured to generate images without depictions of a given type of biological anomaly) as well as a separate anomaly addition generator network (configured to generate images depicting a given type of biological anomaly). The cycle GAN can further include multiple classifier networks. Each classifier network can be configured to receive both real and fake images (corresponding to either instances with anomalies or instances without anomalies) and determine whether each of the images is real. Feedback generated based on the accuracy of the results generated by each classifier network can be fed back to the corresponding one of the generator networks. The cycle GAN can be configured such that the anomaly addition generator network and / or the anomaly removal generator network receive and generate three-dimensional images (using a set of three-dimensional kernels). Similarly, the cycle GAN can be configured such that each classifier network receives a three-dimensional image (and predicts whether the image is real or fake).

[0007] Alternatively, the GAN may include a recycle GAN, which includes the generator network and discriminator network of a cycle GAN and further includes one or more predictor networks. Each predictor network can be configured and trained to generate fake images corresponding to a different viewpoint, imaging modality, position, zoom, and / or slice compared to the viewpoint, imaging modality, position, zoom, and / or slice depicted in the image generated by the generator network that feeds the predictor network. Each discriminator network can be configured and trained to receive both real and fake images (corresponding to either instances with or without anomalies) and determine whether each of the images is real. Results generated by a given discriminator network can be fed back to the generator network that feeds the predictor network that feeds the given discriminator network to potentially trigger parameter learning.

[0008] In a first embodiment, a computer-implemented method is provided. A medical image corresponding to a subject and depicting a portion of a biological anomaly is accessed, the biological anomaly being a particular type of biological anomaly. A corrected image is generated based on the medical image and using an anomaly removal generator network. The anomaly removal generator network can be configured with parameters learned during training using a training dataset that does not have annotations of the particular type of biological anomaly. A size of the biological anomaly is estimated based on the medical image and the corrected image. The estimated size of the biological anomaly is output.

[0009] In a second embodiment, the method may include the method of the first embodiment, further including training an anomaly removal generator network by training a generative adversarial network, wherein the generative adversarial network includes an anomaly removal generator network and one or more classifier networks, each of the one or more classifier networks configured and trained to distinguish between a real image and an image generated by the generator network, wherein the generator network includes one or more classifier networks, including an anomaly removal generator network or an anomaly addition generator network.

[0010] In a third embodiment, the method may include the method of the first embodiment, wherein the set of parameters of the anomaly removal generator network is defined by training a generative adversarial network including the anomaly removal generator network and one or more classifier networks, each of the one or more classifier networks configured and trained to distinguish between a real image and an image generated by the generator network, and the generator network includes an anomaly removal generator network or an anomaly addition generator network.

[0011] In a fourth embodiment, the method may include the method of the first embodiment and may further include training the anomaly removal generator network by training a generative adversarial network including an anomaly removal generator network, an anomaly addition generator network, a first classifier network, and a second classifier network. The first classifier network may be configured to distinguish between real images labeled as depicting at least a portion of biological anomalies of a particular type of biological anomaly and false images generated by the anomaly addition generator network. The anomaly addition generator network may receive feedback during training based on a first classification result generated by the first classifier network. The second classifier network may be configured to distinguish between real images labeled as not depicting any biological anomalies of a particular type of biological anomaly and false images generated by the anomaly removal generator network. The anomaly removal generator network may receive feedback based on the first classification result generated by the first classifier network.

[0012] In a fifth embodiment, the method may include the method of the first embodiment, wherein the set of anomaly removal generator networks is defined by training a generative adversarial network including an anomaly removal generator network, an anomaly addition generator network, a first classifier network, and a second classifier network. The first classifier network is configured to distinguish between real images labeled as depicting at least a portion of another anomaly of a particular type of biological anomaly and false images generated by at least the anomaly addition generator network. The anomaly addition generator network may receive feedback during training based on a first classification result generated by the first classifier network. The second classifier network may be configured to distinguish between real images labeled as not depicting any biological anomaly of the particular type of biological anomaly and false images generated by the anomaly removal generator network. The anomaly removal generator network may receive feedback during training based on a first classification result generated by the first classifier network.

[0013] In a sixth embodiment, the method may include the method of the first embodiment, and may further include training the anomaly removal generation network by training a generative adversarial network (GAN) by inputting real anomaly-present images to the anomaly removal generation network, the real anomaly-present images depicting at least a portion of the subject and at least a portion of another biological anomaly of a particular type of biological anomaly; generating false anomaly-free images using at least the anomaly removal generator network and the real anomaly-present images; performing classification using a classifier network of the GAN to predict whether the false anomaly-free images correspond to true images of the actual sample or false images; and adjusting one or more weights of the anomaly removal generator network based on the classification performed by the classifier network. In a seventh embodiment, the method may include the method of the sixth embodiment and may further include inputting the false anomaly-free images to the anomaly removal generator network of the GAN; generating periodic false anomaly-present images using the anomaly addition generator network and the false anomaly-free images; comparing the periodic false anomaly-present images with real anomaly-present images; and determining a cycle loss based on the comparison of the cycled false anomaly-present images and the real anomaly-present images.

[0014] In an eighth embodiment, the method may include any of the methods of the first to seventh embodiments and may further include preprocessing the medical image to adjust the distribution of each of the one or more color channels, and a corrected image is generated based on the preprocessed medical image.

[0015] In a ninth embodiment, the method may include any of the methods of the first to eighth embodiments and may further include pre-processing the medical image to perform segmentation of a specific organ, and a modified image is generated based on the pre-processed medical image.

[0016] In a tenth embodiment, the method may include the method of any of the first to ninth embodiments, wherein estimating the size of the biological abnormality includes subtracting the modified image from the medical image.

[0017] In an eleventh embodiment, the method may include the method of any of the first to tenth embodiments, wherein the particular type of biological abnormality is a lesion or tumor.

[0018] In a twelfth embodiment, the method may include the method of any of the first to eleventh embodiments, wherein the medical image includes a CT image, an X-ray image, or an MRI image.

[0019] In a thirteenth embodiment, the method may include the method of any of the first to twelfth embodiments, wherein the medical image includes a three-dimensional image.

[0020] In a fourteenth embodiment, the method may include the method of any of the first to thirteenth embodiments, wherein the anomaly rejection generator network includes a convolutional neural network.

[0021] In a fifteenth embodiment, the method may include any of the methods of the first to fourteenth embodiments, wherein the training data set lacked identification of the boundary, area or volume of any delineated abnormality of the particular type of biological anomaly.

[0022] In a sixteenth embodiment, a method is provided that includes: receiving, by a user device and to a computing system, an estimated size of the biological anomaly, using a medical image corresponding to a subject and delineating a portion of the biological anomaly, where the biological anomaly is a specific type of biological anomaly; receiving, by the user device and from the computing system, an estimated size of the biological anomaly, where the computing system generates a corrected image based on the medical image and using an anomaly removal generator network, where the anomaly removal generator network is trained using a training dataset that lacks annotations of the specific type of biological anomaly; and determining the estimated size of the biological anomaly based on the medical image and the corrected image.

[0023] In a seventeenth embodiment, the method may include the method of the sixteenth embodiment and may further include selecting a diagnostic or treatment recommendation for the subject based on the estimated size.

[0024] In an eighteenth embodiment, the method may include the method of the seventeenth embodiment and may further include communicating the selected diagnostic or treatment recommendation to the subject.

[0025] In a nineteenth embodiment, the method may include any of the methods of the sixteenth to eighteenth embodiments and may further include acquiring medical images using a medical imaging system.

[0026] A twentieth embodiment includes the use of an estimated size of a biological anomaly depicted in a medical image in the treatment of a subject, wherein the estimated size is provided by a computing device that performs a series of operations including: generating, by a computing system, a corrected image based on the medical image, using an anomaly removal generator network that has been trained using a training dataset identified as lacking annotations for a particular type of biological anomaly; and estimating, by the computing system, the size of the biological anomaly based on the medical image and the corrected image.

[0027] A twenty-first embodiment includes a system including one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein (e.g., a method of any of the first through nineteenth embodiments).

[0028] A twenty-second embodiment includes a computer program product tangibly embodied in a non-transitory machine-readable storage medium that includes instructions configured to cause one or more data processors to perform some or all of one or more methods disclosed herein (e.g., the method of any of the first through nineteenth embodiments).

[0029] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium containing instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0030] The terms and expressions which have been employed are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims. [Brief explanation of the drawings]

[0031] The present disclosure is described in conjunction with the accompanying drawings, in which:

[0032] [Figure 1] 1 illustrates an exemplary biological anomaly characterization network, according to some embodiments.

[0033] [Figure 2] 1 illustrates a representation of a network and network connections in a cycle generative adversarial network (GAN) according to some embodiments of the present invention.

[0034] [Figure 3] 1 illustrates a flow chart for generating fake images using cycle GAN and discriminating between real and fake images according to some embodiments of the present invention.

[0035] [Figure 4] 1 illustrates various networks and network connections in a recycled GAN, according to some embodiments of the present invention.

[0036] [Figure 5] 10 illustrates how various fake images can be generated and evaluated in a neural network configuration, according to some embodiments of the present invention.

[0037] [Figures 6A-6B] 10 illustrates how various types of losses can occur in neural network configurations, according to some embodiments of the present invention.

[0038] [Figure 7] 10 illustrates how spatial characteristics of biological anomalies can be estimated based on a comparison between real and fake images, according to some embodiments of the present invention.

[0039] [Figure 8] 1 shows a flowchart of an exemplary process for estimating the size of a biological anomaly.

[0040] [Figure 9] Shows example images used and generated using Cycle GAN to detect lesions.

[0041] [Figure 10] 1 shows exemplary lesion detection performed by using the Cycle-GAN algorithm or by four human readers.

[0042] [Figure 11] We show exemplary results of a trained recycled GAN detecting lesions compared to radiologist lesion detection.

[0043] In the accompanying drawings, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes between the similar components. When only a first reference label is used in this specification, the description is applicable to any of the similar components having the same first reference label, regardless of the second reference label. DETAILED DESCRIPTION OF THE INVENTION

[0044] I. Overview Detailed Description The systems, methods, and software disclosed herein facilitate estimating the size of biological anomalies, such as tumors. More specifically, an anomaly removal generator neural network is trained to receive real images associated with a particular context (e.g., a particular slice level for a particular subject and / or a particular biological region for a particular subject) and generate corresponding false images associated with the same or different contexts (e.g., a different slice level for a particular subject, the same slice level for a particular subject, or the same particular biological region for a particular subject). For some aspects (e.g., a particular subject), the corresponding false images can reflect the real image, but the neural network can be configured to lack some or all of the depiction of a given type of biological anomaly depicted in the real image. For example, while both the real image and the false image can include a three-dimensional image of a lung, the real image can depict a tumor, while the false image does not. Thus, the size of the biological anomaly is estimated by subtracting the false image from the real image (e.g., and determining how many pixels or voxels in the difference image exceed a threshold).

[0045] Generative adversarial networks used to train IA anomaly removal generative networks The anomaly removal generator network can include parameters learned during training of a generative adversarial network (GAN). The GAN further includes a classifier network configured to predict whether an input image is fake (generated by the anomaly removal generator network) or (depicts a real image collected from a subject). Feedback based on the accuracy of these predictions can be provided to the anomaly removal generator network.

[0046] In some cases, the GAN used to train the anomaly removal generator network includes a cycle GAN. The cycle GAN includes multiple generator networks and multiple classifier networks. In addition to the anomaly removal generator network, the cycle GAN further includes an anomaly addition generator network configured and trained to receive real anomaly-free images that do not depict a specific type of biological anomaly and generate false anomaly-present images that depict a specific type of biological anomaly. The cycle GAN also includes a first classifier network that predicts whether an image (an image that does not truly depict a specific type of biological anomaly or has been modified to lack a depiction of a specific type of biological anomaly) is real or false. The accuracy of the prediction can be used to provide feedback to the anomaly removal generator network. The cycle GAN can also include a second classifier network that predicts whether an image (an image that truly depicts a specific type of biological anomaly or has been modified to include a specific type of biological anomaly) is real or false. The accuracy of the prediction can be used to provide feedback to the anomaly addition generator network.

[0047] In some cases, the GAN used to train the anomaly-removal generator network includes a recycled GAN. Similar to a cycle GAN, a recycled GAN includes multiple generator networks (e.g., an anomaly-removal generator network and an anomaly-addition generator network) and multiple classifier networks (e.g., a classifier network configured to distinguish between real and false anomaly-present images and a classifier network configured to distinguish between real and false anomaly-present images). Additionally, the recycled GAN includes one or more predictor networks that generate images associated with different contexts relative to an image input to the predictor network.

[0048] A first predictor network within the recycling GAN can receive one or more images depicting a particular type of biological anomaly (e.g., a tumor, lesion, or plaque) and associated with a particular subject and one or more slice levels, and the first predictor can generate outputs corresponding to different predicted context images associated with the particular subject. Each modified image can correspond to a different viewpoint, imaging modality, position, zoom, and / or slice compared to the viewpoint, imaging modality, position, zoom, and / or slice for the image received by the predictor network. In particular, the one or more images depicting the biological anomaly and received by the first predictor network can be real or fake. For example, the one or more images can be fake images generated by an anomaly addition generator network configured to modify one or more real images (e.g., lacking a depiction of a given type of biological anomaly) to add a depiction of the given type of biological anomaly.

[0049] Conversely, a second predictor network within a recycling GAN can receive one or more images that lack a representation of a particular type of biological anomaly (e.g., not depicting a tumor) and generate an output corresponding to a predicted different context image (e.g., associated with a different viewpoint, imaging modality, position, zoom, and / or slice) that also lacks a representation of the particular type of biological anomaly. In particular, the one or more images that do not depict a biological anomaly and are received by the second predictor network can be real or fake. For example, the one or more images can be fake images generated by an anomaly removal generator network configured to modify one or more real images (e.g., including any depiction of a given type of biological anomaly) to remove the depiction of the given type of biological anomaly.

[0050] Anomaly size prediction using IB false anomaly-free images In some cases, the anomaly removal generation network is configured to receive a three-dimensional image (depicting at least a portion of a biological anomaly) and generate a false three-dimensional image (lacking a depiction of at least a portion of the biological anomaly). The difference between the real image and the false image can then indicate which voxels are predicted to be part of the biological anomaly. Thus, the size of the biological anomaly can be predicted based on the amount of voxels for which the difference between the real image and the false image exceeds a threshold.

[0051] Alternatively, the anomaly removal generation network may be configured to receive a two-dimensional image (depicting at least a portion of the biological anomaly) and generate a false two-dimensional image (lacking a depiction of at least a portion of the biological anomaly). The difference between the real image and the false image may represent a predicted region of the anomaly. However, multiple real images (e.g., corresponding to different slice levels) may be available for a given subject, and multiple false images may be generated. The regions across the slice levels may then be processed together to predict the size and / or volume of the anomaly.

[0052] IC Advantages The techniques disclosed herein, involving the use of GANs, offer significant advantages in reducing the need for manual annotation and improving the objectivity and accuracy of delineating abnormalities in medical images. For example, manually annotating medical images to estimate tumor volume can require extensive time to identify tumor boundaries in each of multiple consecutive slices. For example, a whole-body CT scan of a subject with advanced cancer may contain 250 or more tumors. Detecting anomalies using manual annotation can require thousands of labor hours per image and millions of dollars per tumor.

[0053] This time commitment can result in a training set that is relatively small in size and / or has fairly little diversity. This manual approach is also prone to error, as it may be difficult for human annotators to recall details from adjacent slides that could be informative as to where the tumor boundary is on the current slide.

[0054] On the other hand, the GAN-based techniques disclosed herein can use machine learning models trained using data with high-level labels indicating whether each image depicts at least a portion of a biological anomaly (such that the training data includes binary labels). The training data does not need to be labeled to indicate the location of a given depiction of a biological anomaly, the size of the depicted biological anomaly, the boundary of the biological anomaly, or any other spatial characteristics of the biological anomaly. That is, the training data set can be collected without manual segmentation. Therefore, it is easier to obtain training data, resulting in a larger training data set and higher model accuracy. Alternatively or additionally, binary labeling can reduce the time or monetary investment required to collect the training set and / or predict the size of the biological anomaly.

[0055] Furthermore, the techniques disclosed herein do not rely on a training dataset containing paired data. That is, the generator network does not need to be trained on a dataset containing a set of anomaly-free images and a corresponding "paired" set of anomaly-present images (e.g., associated with the same set of subjects). Collecting paired images can include collecting images from multiple imaging sessions for each subject. For example, one or more images from a first imaging session can depict a biological anomaly, while images from a second imaging session can lack the biological anomaly (or vice versa). However, predicting whether an anomaly will cease to appear (e.g., in response to treatment) or whether a new anomaly will appear can be very difficult. Therefore, obtaining paired images can be very challenging. Without paired data, it would be impossible to use many existing training techniques (e.g., those using L1 or L2 regularization) to train a neural network to generate predicted anomaly-free images based on actual anomaly-present images (e.g., to subsequently predict the size of the anomaly). On the other hand, the techniques disclosed herein, which do not require paired data, can collect larger training datasets (e.g., unpaired training datasets) and can provide accurate predictions of the size of biological abnormalities.

[0056] II. Exemplary Biological Anomaly Characterization Networks FIG. 1 illustrates an exemplary biological anomaly characterization network 100 according to some embodiments. The biological anomaly characterization network 100 includes an image generation system 105 configured to collect images of a portion of a subject's body. Each image can depict at least a portion of one or more biological structures (e.g., at least a portion of one or more tumors, at least a portion of one or more lesions, at least a portion of one or more plaques, and / or at least a portion of one or more organs). The subject can include a person who has been diagnosed with a particular disease or has a probable diagnosis of a particular disease. The particular disease can include cancer or a particular type of cancer. The one or more images can depict all or a portion of (for example) a lung, a brain, or a liver.

[0057] II.A. Image Generation System An image includes one or more two-dimensional images and / or one or more three-dimensional images. A two-dimensional image depicts a cross-sectional slice (e.g., a horizontal slice) of a subject or a portion of a cross-sectional slice of a subject. A three-dimensional image depicts a three-dimensional region of a subject. A three-dimensional image can be generated by stacking or combining multiple two-dimensional images (corresponding to slices of a subject imaged at multiple slice levels). Thus, as used herein, a "region" of a subject depicted in an image refers to a three-dimensional volume within the subject, and a "slice" of a subject depicted in an image refers to a two-dimensional cross-section of the subject.

[0058] The imaging system 105 may include (for example) a computed tomography (CT) scanner, an X-ray machine, or a magnetic resonance imaging (MRI) machine. The images may include radiological images, CT images, X-ray images, or MRI images. The images may be acquired without a contrast agent being administered to the subject, or after a contrast agent has been administered to the subject. In some cases, the imaging system 105 may first acquire a set of two-dimensional images and use the two-dimensional images to generate a three-dimensional image.

[0059] The images acquired by the imaging system 105 may be acquired without a contrast agent being administered to the subject or after a contrast agent has been administered to the subject. The subject being imaged may include a subject who has been diagnosed with cancer, has a possible or preliminary diagnosis of cancer, and / or has symptoms consistent with cancer or a tumor.

[0060] The image generation system 105 may store the collected images in an image data store 110, which may include (for example) a cloud data store. Each image may be stored in association with one or more identifiers, such as an identifier for the subject and / or an identifier for a caregiver associated with the subject. Each image may also be stored in association with the date the image was collected.

[0061] II.B. Image Processing System The one or more images are utilized by an image labeling system 115 that facilitates identification of a label for each of the one or more images. The label indicates whether the image depicts a biological anomaly. It will be appreciated that a label indicating that an image depicts a biological anomaly can indicate that the image depicts a portion of the biological anomaly (e.g., a slice of the biological anomaly).

[0062] The image labeling system 115 can identify labels based on input received by a human user. For example, the image labeling system 115 can present each of one or more images on a display and receive input (e.g., clicking a given button, selecting a pull-down option, entering text, etc.) indicating whether each image depicts a biological abnormality (e.g., depicting at least a portion of a tumor).

[0063] Alternatively, the image labeling system 115 may identify labels using automated image processing techniques. For example, the image labeling system 115 may predict that an image depicts a biological abnormality if at least a threshold percentage or at least a threshold number of voxels or pixels have an intensity above a predetermined threshold. The threshold may be defined to distinguish between parts of the body that have absorbed the contrast agent (e.g., tumors) and other parts of the body. As another example, the image labeling system 115 may identify labels based on the source of the images. For example, a first source of images may include a database, library, or healthcare provider system associated with confirmed tumor cases, and a second source of images may include a database, library, or healthcare provider system associated with healthy subjects.

[0064] The image labeling system 115 can update the image data store 110 to include one or more labels that indicate whether each of the one or more images includes a depiction of a biological anomaly.

[0065] II.C. Image Processing System The image processing system 125 (which may include, for example, a remote and / or cloud-based computing system) is configured to predict, for each of one or more biological anomalies of a given type, the size of the depicted anomaly.

[0066] II.C.1. Pre-processing Controller More specifically, the image processing system 125 can be configured to train a GAN, and then use the trained anomaly removal generator network from the GAN to process input images (representing a particular type of biological anomaly) to generate modified images (not representing the particular type of biological anomaly), and to predict the size of the biological anomaly using the input images and modified images.

[0067] More specifically, image processing system 125 includes a preprocessing controller 130 that initiates and / or controls preprocessing of images. Preprocessing can include (for example) converting an image to a predetermined format, resampling an image to a predetermined sampling size, resampling an image to a predetermined size (e.g., no more than a specified number of pixels or voxels in each of one, multiple, or all dimensions), cropping an image to a predetermined size, generating a three-dimensional image based on multiple two-dimensional images, generating one or more images with different (e.g., target) viewpoints, adjusting intensity values ​​(e.g., standardizing or normalizing), and / or adjusting color values.

[0068] Preprocessing can include transformations that adjust one or more color channels, for example, a grayscale image can be transformed to include one or more color channels.

[0069] In some cases, for each image, the preprocessing controller 130 segments the image by detecting regions that depict at least a portion of a particular type of biological organ (e.g., lungs). The segmentation can be performed using (for example) a neural network (e.g., a convolutional neural network) trained using supervised learning. The image can be modified to include only portions of the image identified as depicting a particular type of biological organ. For example, the intensity values ​​of all other pixels or voxels may be set to 0 or a non-numeric value.

[0070] II.C.2. GAN Training Controller The image processing system 125 includes a GAN training controller 135 that trains the GAN using labeled images. The GAN can be trained using a training set that includes a first set of images, each depicting at least one biological anomaly (which can include depicting at least a portion of a biological anomaly), where the at least one biological anomaly is a particular type of anomaly. For example, the images can include a three-dimensional image depicting an entire tumor, a two-dimensional image depicting an entire cross-section of the tumor, a three-dimensional image depicting a portion of the tumor, or a two-dimensional image depicting a portion of a cross-section of the tumor. The training set also includes a second set of images, each of which does not depict a biological anomaly of a particular type of anomaly. Each image in the dataset can be associated with a label indicating whether it depicts a particular type of biological anomaly, but the training set need not include segmentation or annotation data indicating where the anomaly is depicted in the image. The images in the training set can all depict at least a portion (or all) of a particular organ, or the training set can include images depicting different parts of the body.

[0071] The GAN may include a cycle GAN or a recycling GAN. The GAN includes an anomaly removal generator neural network trained to receive images depicting a specific type of biological anomaly (e.g., a tumor) and output modified (fake) images that do not depict the specific type of biological anomaly. The anomaly removal generator network may include one or more convolutional layers, a U-net, or a V-net. In some cases, the anomaly removal generator network includes a feature extraction encoder (including one or more convolutional layers), a transformer (including one or more convolutional layers), and a decoder (including one or more convolutional layers). The modified images and the received images may share various contexts (e.g., depicted area / volume, size, depicted organ, etc.). A classifier network receives the fake images from the anomaly removal generator network and also receives real images that do not depict the specific type of biological anomaly. The classifier network may include one or more convolutional layers and an activation layer. For each image, the classifier network predicts whether the image is real or fake. Feedback may be sent to the anomaly removal generator network based on the accuracy of the classifier network's prediction.

[0072] Training a GAN can include using one or more loss functions. Training can include introducing a penalty when the GAN's classifier network incorrectly predicts that a real image is a fake image (or vice versa). Training can additionally or alternatively include introducing a cycle loss. The cycle loss can be calculated by double-processing an original image that includes (or alternatively lacks) representations of a particular type of anomaly with one generator network of the GAN to remove (or alternatively add) representations of such anomalies to generate a modified image predicted to lack (or alternatively include) such representations, then processing the modified image with another generator network of the GAN to add (or alternatively remove) representations of the anomaly to generate a cycle image. The loss can be scaled by the difference between the original image and the cycled image.

[0073] II.C.2.a. Cycle GAN The GAN trained by the GAN training controller 135 may include a cycle GAN, which includes multiple generator networks and multiple discriminator networks. Figure 2 shows a representation of the network and network connections in a cycle GAN, and Figure 3 shows a flow for generating fake images and discriminating between real and fake images using a cycle-GAN. In this depiction, each rectangular box represents an image (real or fake), and each diamond represents a discrimination.

[0074] The X and Y domains relate to different types of images, where the Y domain corresponds to images depicting a particular type of biological abnormality (e.g., a lesion) and the X domain corresponds to images lacking depiction of a particular type of biological abnormality.

[0075] In addition to including the anomaly removal generator network, the cycle GAN also includes an anomaly addition generator network configured and trained to modify images to add depictions of specific types of biological anomalies. The anomaly addition generator network may include one or more convolutional layers, a U-net, or a V-net. In some cases, the anomaly addition generator network includes a feature extraction encoder (including one or more convolutional layers), a transformer (including one or more convolutional layers), and a decoder (including one or more convolutional layers). The architecture of the anomaly addition generator network may be the same as the architecture of the anomaly removal generator network.

[0076] The anomaly-addition generator network and / or the anomaly-removal generator network can be collectively trained using cycle consistency loss or cycle loss. In this case, the original image is compared to an image generated by processing first by one generator network and then by the other generator network; the loss can be larger if the image difference is more dramatic. For example, the original image 305 may depict a patient's lungs but not a tumor. This original image can be processed by the anomaly-addition generator network to generate a false tumor-present image 310, and then by the anomaly-removal generator network to generate a false tumor-absent image 310'. The loss may be calculated to scale according to the difference between the original image and the false tumor-absent image. Cycle consistency loss can facilitate reducing the occurrence or extent of mode collapse (in which the generator begins to generate stereotypes of images that may not correspond to the original image). However, minimizing cycle loss in cycle GANs does not guarantee that, when mapped from one domain to another, the image will remain stuck in a single "mode" where only a few pixels change from one image to another. For example, an anomaly-adding generator network can generate false anomaly-present images that look nearly identical to input (e.g., real) anomaly-present images. Similarly, an anomaly-removing generator network can generate false anomaly-absent images that look nearly identical to input (e.g., false) anomaly-present images. Thus, in some cases, the loss can be calculated based on a combination of periodic and aperiodic losses (e.g., the aperiodic loss is determined based on the degree to which the classifier network can distinguish between real and false images).

[0077] In addition to including a classifier network that identifies (at block 315′) real and fake images that do not depict a particular type of abnormality (e.g., real tumor-free image 305 and fake tumor-free image 310′), the cycle GAN also includes another classifier network that identifies (at block 315) fake and real images that depict a particular type of abnormality (e.g., real tumor-present image 305′ and fake tumor-present image 310). Each of the classifier network and the other classifier network can include one or more convolutional layers and activation layers. The architecture of the classifier network may be the same as the architecture of the other classifier network.

[0078] Each generator network (G X and G Y ) is configured to receive an image from one domain and generate a corresponding image in another domain (the other domain is indicated by a subscript). G Y is configured to receive a real image 305 that does not contain any representation of a particular type of biological anomaly (X domain) and generate a corresponding false image 310 that does contain a representation of a particular type of biological anomaly (Y domain). X is configured to receive a real image 305' that includes a representation of a particular type of biological anomaly (Y domain) and generate a corresponding image 310' that lacks a representation of the particular type of biological anomaly (X domain).

[0079] Each classifier network processes fake images (generated by the generator network) to predict whether the image is real or not. Each classifier network similarly processes real images to predict whether the image is real or not. Each classifier network is domain-specific. Thus, for example, classifier network D Y is a network of anomaly generators G that are known (e.g., via metadata associated with the image) to depict a particular type of biological anomaly. Y or a predictor network P Y, and generate (at block 315) a prediction of the real or fake image being generated (by X is known (e.g., via metadata associated with the image) as lacking a representation of a given type of biological anomaly or (by the anomaly removal generator network G X or a predictor network P X 3. Generate (at block 315') a prediction of the real or fake image being generated (by

[0080] A classification result corresponding to chance indicates that the classifier network is unable to distinguish between real and fake images, and therefore the fake images are likely to be of high quality. The results of classification performed by a classifier network operating in a given domain can be used to adjust the parameters of the corresponding generator network in the training process.

[0081] A cycle GAN can be used to process three-dimensional images. The generator and discriminator networks can then include three-dimensional convolutional layers. The network can then be configured to learn how features extending across slices predict whether an image depicts a particular type of biological anomaly and where the image depicts a particular type of biological anomaly. Alternatively or additionally, the three-dimensional image can be reshaped into a one-dimensional vector and processed using two-dimensional convolutional layers.

[0082] It will be appreciated that using neural networks to process three-dimensional images can use a significant amount of memory. Therefore, if the biological region of interest (e.g., an organ) is large, preprocessing techniques can be performed to segment the organ to reduce its size. Alternatively, or additionally, the image may be preprocessed to reduce the spatial resolution of the image. For example, if the organ of interest is the lung (where the lung is relatively large compared to other organs), it may be useful to segment the organ and resample it to reduce its resolution.

[0083] II.C.2.b. Recycled GAN The GAN trained by the GAN training controller 135 may include a recycled GAN, which includes multiple generator networks, multiple discriminator networks, and multiple predictor networks. Figure 4 shows a representation of the networks and network connections in a recycled GAN. In addition to the network depicted in the cycle GAN of Figure 2, the recycled-GAN shown in Figure 4 includes multiple predictor networks, P X and P Y Includes.

[0084] Each predictor network may include one or more convolutional layers and / or U-nets. Each predictor network may predict an image acquired in a different context than one or more images input to the predictor network. For example, a predictor network may include a convolutional neural network and / or may generate an image corresponding to a different viewpoint, imaging modality, position, zoom, and / or slice compared to the viewpoint, imaging modality, position, zoom, and / or slice of an image received by the predictor network. For example, a predictor network may include a convolutional neural network and / or a U-net. X is the adjacent slice x t Slice x based on the image t+1 can predict the image of P Y is the adjacent slice y s Slice y based on the images+1 It is possible to predict the image of

[0085] 5 illustrates how various fake images can be generated and evaluated in a neural network configuration according to some embodiments of the present invention. In this depiction, each rectangular box represents an image (real or fake) and each diamond represents a classification.

[0086] Each of the first and second real anomaly-free images 505 shows a first slice and a second slice (respectively) of the sample, lacking depictions of a particular type of biological anomaly. Y uses the first and second real anomaly-free images 505 to generate first and second false anomaly-present images 510 of the first and second slices by adding depictions of specific types of biological anomalies. Y uses the first and second false anomaly images of the first slice 510 to generate a third false anomaly image 515 of the third slice. Y predicts whether the third false anomaly presence image 515 is real or false. Y further predicts whether the first and second real anomaly presence images 505' of other first and second slices of another sample are real or false. Y can further predict whether the first and second false anomaly presence images 505 of the first and second slices are real or false. Feedback from the identification (e.g., based on recall statistics) can be used to predict whether the first and second false anomaly presence images 505 of the first and second slices are real or false. Y can provide feedback to.

[0087] The third false anomaly image 515 of the third slice is G X , thereby generating a periodic false-abnormality-free image 525 of the third slice. X uses the first and second real anomaly-free images 505 to generate another false anomaly-free image 530 of the third slide. In block 535, the false anomaly-free images 525 and 530 (corresponding to the third slice) are compared to each other to generate a PX Further, at 540, the periodic false-abnormality-free image of the third slide 525 can be compared to the third true-abnormality-free image of the third slice 535 to determine cycle loss.

[0088] 5 illustrates symmetrical operation. More specifically, first and second true anomaly images 505' of other first and second slices of another sample generate first and second false anomaly-free images 510' of other first and second slices of another sample. X supplied to P X can use the first and second false-abnormality-free images 510' to generate a third false-abnormality-free image 515' of a third slice of the other sample 515'. X predicts whether the third false anomaly-free image 515' is real or false. X further predicts whether the first and second real anomaly-free images 505 are real or false. X can further predict whether the first and second false anomaly-free images 510′ of the other first and second slices are real or false. Feedback from the discrimination (e.g., based on recall statistics) can be used to determine whether G X can provide feedback to.

[0089] The third false anomaly-free image 515' of the other third slice generates a periodic false anomaly-present image 525' of the third slice. Y supplied to P Y uses the first and second real anomaly images 505' to generate another false anomaly image 530' of another third slide. In block 535', the false anomaly images 525' and 530' (corresponding to another third slice) are compared to each other to generate P Y Further, at 540′, a periodic false anomaly image 525′ of another third slide can be compared with a third real anomaly image 535′ of a third slice to determine cycle loss.

[0090] 6A-6B show how various types of losses can occur in a recycled GAN. As illustrated in FIG. 6A, even though the discriminator network and the predictor network participate in global training and backpropagation, their weights do not need to be updated in response to the discrimination results. Rather, the discrimination results can selectively trigger the update of only the weights of the corresponding generator network. However, due to the backpropagation flowing through the discriminator network, its weights are taken into account in the generator training, thereby giving the generator training some advantage over the discriminator. (FIG. 6A)

[0091] Even though a generator network is used to generate fake images, the generator network precedes the corresponding classifier in the architecture. Therefore, the generator weights may not be accessible for classifier training (Figure 6B).

[0092] A recycled GAN can be used to process a set of two-dimensional images. The generator and discriminator networks can then include two-dimensional convolutional layers. The predictor network can facilitate introducing weak coherence between domains (e.g., slices) so that the network can learn anatomical structures (e.g., across the volume encompassed by multiple slices), and then generate more realistic fake images (lacking that anatomical structure because they are only 2D slices), distinguish between real and fake images, and then facilitate further improvement of the fake image generation.

[0093] II.C.3. Generator Network Controller The GAN training controller 135 can train a full GAN ​​network, after which the generator network controller 140 can use a single generator network from the GAN to evaluate medical images. More specifically, the single generator network can include one that receives an input image depicting a particular type of biological anomaly (e.g., a tumor or lesion) and generates a modified image that lacks the depiction of the particular type of biological anomaly.

[0094] II.C.4. Size Detector The size detector 145 can then be configured to receive each of the one or more modified images (i.e., lacking a depiction of a particular type of biological anomaly) and the input image (e.g., depicting a particular type of biological anomaly) generated by the anomaly removal generator network controller 140 and predict the size of the biological anomaly. For example, the size detector 145 can subtract the modified image from the input image. In some cases, the size detector 145 can first process the difference image (e.g., to apply one or more spatial filters and / or thresholds).

[0095] The predicted size may be the size of a region (e.g., if the input image is a two-dimensional image) identified in the difference image, or the size of a volume (e.g., if the input image is a three-dimensional image). For example, the size detector 145 may determine, for each pixel or voxel, whether an intensity condition (e.g., a condition configured to be met when the intensity of the pixel or voxel is not equal to 0, or a condition configured to be met when the intensity of the pixel or voxel is within a predetermined open or closed range) is met. The predicted size may be defined based on the amount of pixels or voxels for which the condition is met. Alternatively, a binary image may be generated for each pixel or voxel indicating whether the condition is met. A low-pass spatial filter may then be applied to pass pixels or voxels for which the condition is met if they are adjacent to other pixels (e.g., of sufficient quantity and / or shape). The filtered image may include another binary image. The predicted size of the biological anomaly may then be determined by processing the filtered image (e.g., by thresholding, summing, or averaging over the pixels or voxels in the filtered image).

[0096] In some cases, the size detector 145 processes multiple original and modified two-dimensional images, and the size detector 145 predicts the size based on the processing of the multiple images. As an example, a biological anomaly area can be estimated for each two-dimensional image, and the areas can be aggregated to estimate the volume of the biological anomaly. As another example, multiple input images (e.g., depicting a particular type of biological anomaly) are processed to generate a three-dimensional anomaly-present image, and multiple modified images (e.g., lacking depictions of the particular type of biological anomaly) are processed to generate a three-dimensional anomaly-absent image, and the size detector 145 can estimate the size of the biological anomaly based on the three-dimensional anomaly-present image and the three-dimensional anomaly-absent image. For example, the size detector 145 can subtract the three-dimensional anomaly-absent image from the three-dimensional anomaly-present image. The resulting image can be, but need not be, processed (e.g., via thresholding, filtering, and / or smoothing). The size (e.g., area or volume) of the biological anomaly can be predicted. In some cases, the predicted size can be based on and / or dependent on the resolution and / or size of the original and / or modified images. As yet another example, for each of multiple orthogonal dimensions, real tumor-present images can be accessed, pseudo-tumor-absent images can be generated, and predicted tumor delineations can be generated by subtracting the pseudo-tumor-absent images from the real tumor-present images. The size detector 145 can then determine the longest diameter of the predicted tumor delineation, which can be defined to be the maximum of the three longest diameters.

[0097] For example, FIG. 7 illustrates how spatial characteristics of a biological abnormality can be estimated based on a comparison between real and fake images, according to some embodiments of the present invention. Specifically, the left image shows an initial image including a depiction of the lesion and surrounding tissue. The center image shows a fake image generated by a trained anomaly removal generator network configured to remove the lesion. The right image shows the result of subtracting the fake image from the corresponding initial image. These images can be processed to characterize the lesion. For example, pixels having a threshold value above a predetermined threshold can be counted. The count may be normalized or scaled based on the image scale and / or image resolution, and the normalized or scaled count can indicate the total lesion size for the imaged slice. In some cases, this process can be repeated across images corresponding to different slices, and tumor volume can be estimated by performing a sum across the images.

[0098] In addition to, or instead of, predicting the size of a biological anomaly based on the real and fake images, the real and fake images can be used to segment the biological anomaly depicted in the real image. The segmentation result can identify which portions of the image depict a particular type of biological anomaly. For example, the segmentation result can include an overlay on the original image that indicates which portions of the original image were predicted to depict a particular type of biological anomaly.

[0099] The image processing system 125 can return the size (and / or segmentation results) to the user device 150. The user device 150 can include a device that requests an estimated size corresponding to a biological anomaly or a device that provides one or more images depicting the biological anomaly. The user device 150 can associate the imaged subject with a medical professional and / or care provider who is treating and / or evaluating the imaged subject. In some cases, the image processing system 125 can return an estimate of the size of the biological anomaly image generating system 105 (e.g., the estimated volume can then be transmitted to the user device). In some cases, rather than or in addition to outputting the estimated size, the image processing system 125 outputs a treatment recommendation (e.g., to the user device 150). For example, the image processing system 125 can use one or more rules to identify a treatment recommendation based at least in part on the estimated size (e.g., potentially one or more previous estimated sizes of the biological anomaly). By way of example, the rules may include a threshold value that indicates that if the estimated size of a given subject's biological abnormality is not at least X% smaller than the previous estimated size of the given subject's biological abnormality (e.g., associated with a defined period of time), a recommendation to change the treatment strategy should be considered.

[0100] It will be appreciated that in some cases, the biological anomaly characterization network 100 can be used to estimate the size of each of multiple biological anomalies. For example, the anomaly removal generator network may be trained to generate modified images that do not depict any brain lesions, even if the input image depicts multiple images. The difference between the input image and the modified images can then be used to estimate the cumulative size of the brain lesions.

[0101] III. Exemplary Biological Anomaly Characterization Networks 8 shows a flowchart of an example process 800 for estimating the size of a biological anomaly. In block 805, the image processing system 125 accesses a medical image. The medical image depicts at least a portion of the biological anomaly (e.g., a tumor or a lesion). The medical image may be a two-dimensional image or a three-dimensional image. The medical image may be a CT image, an MRI image, an X-ray image, etc. The medical image may have been acquired and / or accessed by the image generation system 105.

[0102] At block 810, the generator network controller 140 generates a modified image of the medical image using an anomaly removal generator network. The modified image can be generated or predicted (by the anomaly removal generator network) to lack at least some depictions of biological anomalies. The anomaly removal generator network may be trained by a GAN training controller 135, which trains a GAN. The modified image can be generated based on processing (by the generator network controller 140) of the preprocessed image. The GAN may include at least an anomaly removal generator network and one or more classifier networks. The GAN may also include an anomaly addition generator network and / or one or more predictor networks.

[0103] At block 815, the size of the biological abnormality is estimated (e.g., by size detector 145) based on the medical image and the modified image. For example, the original image (or a preprocessed version thereof) is subtracted from the modified image to generate a residual image. The residual image may then be processed by (for example) summing or averaging voxels or pixels in the residual image (e.g., potentially after applying filtering and / or thresholding).

[0104] At block 820, the estimated size of the biological anomaly is output (e.g., by image processing system 125 and / or to user device 150). The output may include transmitting the estimated size and / or presenting the estimated size. In some cases, the output includes transmitting and / or presenting a result (e.g., a treatment recommendation) based on the estimated size.

[0105] V. Working Examples VA Example 1 Figure 9 shows an example of an image used and generated using Cycle GAN to detect lesions. The top image in Figure 9 shows three actual orthogonal CT slices of a subject's right lung. The slice visualization shows the slices intersecting in 3D space to represent the spatial relationship between the slices. Each of the actual orthogonal CT slices contained a depiction of a lesion.

[0106] A cycle GAN model with the architecture shown in Figure 2 was trained using the training data. The trained anomaly removal generator network G X processed each of the three real CT slices to generate corresponding pseudo-CT images that are predicted to lack lesion depiction. These pseudo-tumor-absent images are shown as interesting in 3D space in the center image of Figure 9.

[0107] For each real image, the corresponding fake image was subtracted from the real image to generate a predicted two-dimensional representation of the tumor. The bottom image in Figure 9 shows the tumor prediction images intersecting in three-dimensional space. Predicting the size of the tumor in each of the three dimensions can facilitate estimation of the tumor's longest diameter. For example, the longest diameter of the tumor representation can be estimated for each of the three tumor prediction images, and the longest diameter can be defined as the maximum of the three longest diameters.

[0108] VB Example 2 Figure 10 shows exemplary lesion detection performed using the Cycle-GAN algorithm or by four human readers. The unmasked image shown in Figure 10 is the same as the image shown in the top image of Figure 9. However, here, the images are placed side by side rather than intersecting. The remaining images show the same unmasked image, but with predicted lesion depictions overlaid. The predicted lesion depictions shown in the algorithm image are the same as those shown in the bottom image of Figure 9 and were calculated by comparing the unmasked image with false anomaly-free images (generated by an anomaly removal generator network trained using Cycle-GAN).

[0109] The remaining images show the lesion annotations made by each of the four human readers. Notably, the predicted lesion areas identified by using the anomaly removal generator network were similar to the lesion areas identified by the human readers.

[0110] VC Example 3 The training dataset was defined to include 1,300 whole-body CT images of breast volumes with cancerous lesions and 300 whole-body CT images of breast volumes without cancer. Each image was labeled to indicate whether the image corresponded to a breast volume with cancer (one or more lesions). The data was split 80:20 for training and testing. A model with the architecture shown in Figure 4 was trained using the training data.

[0111] Figure 11 shows the results from processing the test data by comparing the lesion detection output by the trained model with lesion detection from an experienced radiologist. Notably, the depicted results are quite consistent. The model had an ROI-level sensitivity of 0.94 and a lesion-level sensitivity of 0.89. The segmentation achieved a Dice similarity coefficient of 0.2.

[0112] VI. Further Considerations Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium containing instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0113] The terms and expressions which have been employed are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0114] The description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the description of preferred exemplary embodiments provides those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0115] In the description, specific details are set forth to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments. Examples of potentially patentable subject matter include, but are not limited to, the following:

Claims

1. 1. A computer-implemented method comprising: accessing a medical image corresponding to a subject and depicting at least a portion of a biological anomaly, wherein the biological anomaly is a particular type of biological anomaly; generating a corrected image based on the medical image and using an anomaly removal generator network, the anomaly removal generator network being trained using a training dataset lacking annotations of the particular type of biological anomaly; estimating a size of the biological abnormality based on the medical image and the corrected image; outputting the estimated size of the biological anomaly; and 11. A computer-implemented method comprising:

2. further comprising training the anomaly removal generator network by training a generative adversarial network, the generative adversarial network comprising: the anomaly rejection generator network; one or more classifier networks, each of which is configured and trained to distinguish between a real image and an image generated by a generator network, the generator networks including the anomaly removal generator network or the anomaly addition generator network; The computer-implemented method of claim 1 , comprising:

3. The set of parameters of the anomaly removal generator network is defined by training a generative adversarial network, the generative adversarial network being: the cancellation generator network; one or more classifier networks, each of which is configured and trained to distinguish between a real image and an image generated by a generator network, the generator networks including the anomaly removal generator network or the anomaly addition generator network; The computer-implemented method of claim 1 , comprising:

4. training the anomaly removal generator network by training a generative adversarial network, the generative adversarial network comprising: the anomaly rejection generator network; an anomalous summation generator network; a first classifier network, an actual image labeled as depicting at least a portion of a biological anomaly of the particular type of biological anomaly; and a false image generated by the anomaly summation generator network; a first classifier network, the anomaly summation generator network receiving feedback based on a first classification result generated by the first classifier network; a second classifier network, an actual image that is labeled as not depicting any biological anomaly of the particular type of biological anomaly; and a false image generated by the anomaly removal generator network; a second classifier network, the anomaly rejection generator network receiving feedback based on the first classification result generated by the first classifier network; The computer-implemented method of claim 1 , further comprising training the anomaly rejection generator network, the anomaly rejection generator network comprising:

5. The set of parameters of the anomaly removal generator network is defined by training a generative adversarial network, the generative adversarial network being: the anomaly rejection generator network; an anomalous summation generator network; a first classifier network, an actual image labeled as depicting at least a portion of another of the particular type of biological anomaly; and a false image generated by at least the anomaly summation generator network; a first classifier network, wherein the abnormal summation generator network receives feedback based on a first classification result generated by the first classifier network; a second classifier network, an actual image labeled as not depicting any biological anomaly of the particular type of biological anomaly; and a false image generated by the anomaly removal generator network; a second classifier network, the anomaly removal generator network receiving feedback based on the first classification result generated by the first classifier network; and The computer-implemented method of claim 1 , comprising:

6. training the anomaly removal generator network by training a generative adversarial network (GAN), inputting an actual anomaly presence image depicting at least a portion of the subject and depicting at least a portion of another biological anomaly of the particular type of biological anomaly into an anomaly removal generator network; generating a false anomaly-free image using at least the anomaly removal generator network and the real anomaly-present image; performing classification using the GAN classifier network to predict whether the false anomaly-free image corresponds to a true image of a real sample or a false image; adjusting one or more weights of the anomaly rejection generator network based on the classification performed by the classifier network; The computer-implemented method of claim 1 , further comprising training the anomaly rejection generator network by:

7. inputting the false anomaly-free image into the anomaly removal generator network of the GAN; generating periodic false anomaly-present images using an anomaly summation generator network and the false anomaly-absent images; comparing the periodic false anomaly presence images with the real anomaly presence images; determining a cycle loss based on the comparison of the periodic false anomaly images and the real anomaly images; The computer-implemented method of claim 6 further comprising:

8. 8. The computer-implemented method of claim 1, further comprising preprocessing the medical image to adjust the distribution of each of one or more color channels, and wherein the corrected image is generated based on the preprocessed medical image.

9. The computer-implemented method of claim 1 , further comprising pre-processing the medical image to perform segmentation of specific organs, wherein the modified image is generated based on the pre-processed medical image.

10. The computer-implemented method of claim 1 , wherein estimating the size of the biological anomaly comprises subtracting the modified image from the medical image.

11. The computer-implemented method of claim 1 , wherein the particular type of biological abnormality is a lesion or a tumor.

12. The computer-implemented method of claim 1 , wherein the medical image comprises a CT image, an X-ray image, or an MRI image.

13. The computer-implemented method of claim 1 , wherein the medical images include three-dimensional images.

14. The computer-implemented method of claim 1 , wherein the anomaly-rejection generator network comprises a convolutional neural network.

15. 10. The computer-implemented method of claim 1, wherein the training data set lacked any identification of the boundary, area, or volume of any delineated abnormality of the particular type of biological abnormality.

16. utilizing, by a user device and to a computing system, a medical image corresponding to a subject and a delineation of a portion of a biological anomaly, wherein the biological anomaly is a particular type of biological anomaly; receiving, at the user device and from the computing system, an estimated size of the biological anomaly, the computing system comprising: generating a corrected image based on the medical image and using an anomaly removal generator network, the anomaly removal generator network being trained using a training dataset lacking annotations of the particular type of biological anomaly; determining the estimated size of the biological anomaly based on the medical image and the corrected image; receiving an estimated size of the biological anomaly, wherein the estimated size is determined by A method comprising:

17. 17. The method of claim 16, further comprising selecting a diagnostic or treatment recommendation for the subject based on the estimated size.

18. 18. The method of claim 17, further comprising communicating the selected diagnostic or treatment recommendation to the subject.

19. The method of claim 16 , further comprising acquiring the medical image using a medical imaging system.

20. 1. Use of an estimated size of a biological abnormality depicted in a medical image in the treatment of a subject, said estimated size comprising: generating, by a computing system, a corrected image based on the medical image and using an anomaly removal generator network, the anomaly removal generator network being trained using a training dataset identified as lacking annotations for a particular type of biological anomaly; estimating, by the computing system, the size of the biological anomaly based on the medical image and the corrected image; A use provided by a computing device that performs a set of operations including:

21. 1. A system comprising: one or more data processors; a non-transitory computer-readable storage medium containing instructions that, when executed on said one or more data processors, cause said one or more data processors to perform the method of any one of claims 1 to 19; and A system comprising:

22. 20. A computer program product tangibly embodied in a non-transitory machine-readable storage medium comprising instructions configured to cause one or more data processors to perform the method of any one of claims 1 to 19.

Images