Weak-teacher lesion segmentation

Abstract

A generative adversarial network (GAN) can be trained, including an anomaly removal generator network configured to modify medical images to remove depictions of biological anomalies, and one or more classifier networks, each configured to distinguish between real and fake images. The anomaly removal generator network can then receive medical images depicting specific biological anomalies (or preprocessed versions thereof) and generate modified images predicted to lack depictions of the specific biological anomalies. The size of the specific biological anomalies can be estimated based on the modified images and the received images (or preprocessed versions thereof).

Description

Cross - reference to related applications

[0001] This application claims the benefit and priority of PCT / US2021 / 014611, U.S. Provisional Patent Application No. 62 / 965,515, filed on January 24, 2020, and U.S. Provisional Patent Application No. 62 / 966,084, filed on January 27, 2020. Each of these applications is hereby incorporated by reference in its entirety for all purposes.

Technical Field

[0002] Field Generally, the disclosed technology relates to generating a fake version of an image without anomalies by using a neural network (e.g., a generator network), and estimating the size of biological anomalies depicted in a medical image by subtracting the fake image from the medical image. The generator network can be trained by training an adversarial generative network (GAN) that includes the generator network (e.g., a recycled GAN or a cycle GAN).

Background Art

[0003] Background Medical imaging is often used to detect and / or monitor biological anomalies (e.g., lesions or tumors). Quantifying the size of a biological anomaly often involves an annotator marking the contour of the anomaly on one or more images (e.g., corresponding to one or more slices). This is time - consuming and error - prone as a result of variations in the overall annotation due to the subjectivity of the boundary positions.

[0004] Therefore, it would be advantageous to identify automated techniques for processing images to detect and predict the size of biological anomalies.

Summary of the Invention

[0005] Summary The anomaly removal generator network is used to process real images depicting a given type of biological anomaly (e.g., tumor or lesion). The generator network can include one or more three-dimensional kernels. The processing can include generating a fake image that corresponds to the real image but lacks the depiction of the given type of biological anomaly. The real and fake images are then used to segment the biological anomaly, thereby identifying the boundaries, area, or volume of the biological anomaly. Additionally or alternatively, the real and fake images can be used to estimate the size and / or location of a given type of biological anomaly. Estimating the size of the biological anomaly can include subtracting the fake image from the corresponding real image. The size of the biological anomaly can be estimated based on the total number of pixels or voxels having an intensity exceeding a predetermined threshold. In some cases, filtering or other processing is performed prior to estimating the size (e.g., by applying one or more thresholds and / or one or more spatial smoothing functions).

[0006] The anomaly removal generator network can be configured in response to the training of a larger adversarial generator network (GAN). The GAN can include a cycle GAN that, in addition to the anomaly removal generator network (configured to generate images without depicting a given type of biological anomaly), can include another anomaly addition generator network (configured to generate images depicting a given type of biological anomaly). The cycle GAN can further include a plurality of discriminator networks. Each discriminator network can be configured to receive both real and fake images (corresponding to either an instance with an anomaly or an instance without an anomaly) and determine whether each of the images is real. Feedback generated based on the accuracy of the results generated by each discriminator network can be fed back to the corresponding one of the generator networks. The cycle GAN can be configured such that the anomaly addition generator network and / or the anomaly removal generator network receive and generate three-dimensional images (using a set of three-dimensional kernels). Similarly, the cycle GAN can be configured such that each discriminator network receives three-dimensional images (and predicts whether the images are real or fake).

[0007] Alternatively, the GAN may include a CycleGAN, including a generator network and a discriminator network of the CycleGAN, and further including one or more predictor networks. Each predictor network can be configured and trained to generate a fake image corresponding to a different viewpoint, imaging modality, position, zoom, and / or slice as compared to the viewpoint, imaging modality, position, zoom, and / or slice depicted in the image generated by the generator network supplied to the predictor network. Each discriminator network can be configured and trained to receive both a real image and a fake image (corresponding to either an abnormal instance or a non-abnormal instance) and determine whether each of the images is real. The result generated by a given discriminator network can be fed back to the generator network supplied to the predictor network that supplies the given discriminator network to potentially trigger parameter learning.

[0008] In a first embodiment, a computer-implemented method is provided. A medical image corresponding to a subject and depicting a part of a biological abnormality is accessed, the biological abnormality being a particular type of biological abnormality. A corrected image is generated based on the medical image and using an abnormality removal generator network. The abnormality removal generator network can be configured by parameters learned during training using a training dataset without annotations of a particular type of biological abnormality. Based on the medical image and the corrected image, the size of the biological abnormality is estimated. The estimated size of the biological abnormality is output.

[0009] In the second embodiment, the method can include the method of the first embodiment, and further includes training the anomaly removal generator network by training an adversarial generation network, where the adversarial generation network includes the anomaly removal generator network and one or more discriminator networks, each of the one or more discriminator networks being configured and trained to distinguish between a real image and an image generated by the generator network, and the generator network includes one or more discriminator networks including the anomaly removal generator network or the anomaly addition generator network.

[0010] In the third embodiment, the method can include the method of the first embodiment, and a set of parameters of the anomaly removal generator network is defined by training an adversarial generation network including the anomaly removal generator network and one or more discriminator networks, each of the one or more discriminator networks being configured and trained to distinguish between a real image and an image generated by the generator network, and the generator network includes the anomaly removal generator network or the anomaly addition generator network.

[0011] In the fourth embodiment, the method can include the method of the first embodiment, and can further include training the anomaly removal generator network by training an adversarial generation network including an anomaly removal generator network, an anomaly addition generator network, a first discriminator network, and a second discriminator network. The first discriminator network can be configured to discriminate between real images labeled as depicting at least a part of a biological anomaly of a particular type of biological anomaly and fake images generated by the anomaly addition generator network. The anomaly addition generator network can receive feedback during training based on a first discrimination result generated by the first discriminator network. The second discriminator network can be configured to discriminate between real images labeled as not depicting any biological anomaly of a particular type of biological anomaly and fake images generated by the anomaly removal generator network. The anomaly removal generator network can receive feedback based on a first discrimination result generated by the first discriminator network.

[0012] In a fifth embodiment, the method can include the method of the first embodiment, and a set of anomaly removal generator networks is defined by training an adversarial generation network that includes an anomaly removal generator network, an anomaly addition generator network, a first discriminator network, and a second discriminator network. The first discriminator network is configured to discriminate between real images labeled as depicting at least a portion of another anomaly of a particular type of biological anomaly and fake images generated by at least the anomaly addition generator network. The anomaly addition generator network can receive feedback during training based on a first discrimination result generated by the first discriminator network. The second discriminator network can be configured to discriminate between real images labeled as depicting no biological anomaly of a particular type of biological anomaly and fake images generated by the anomaly removal generator network. The anomaly removal generator network can receive feedback during training based on a first discrimination result generated by the first discriminator network.

[0013] In the sixth embodiment, the method can include the method of the first embodiment, and input an actual abnormal presence image that depicts at least a part of a subject and at least a part of another biological abnormality of a specific type of biological abnormality into the abnormal removal generation network; use at least the abnormal removal generator network and the actual abnormal presence image to generate a pseudo-abnormal absence image; use the discriminator network of the GAN to perform discrimination to predict whether the pseudo-abnormal absence image corresponds to a true image of an actual sample or a pseudo-image; and adjust one or more weights of the abnormal removal generator network based on the discrimination performed by the discriminator network, thereby further including training the abnormal removal generation network by training the adversarial generation network (GAN). In the seventh embodiment, the method can include the method of the sixth embodiment, input a pseudo-abnormal absence image into the abnormal removal generator network of the GAN; use the abnormal addition generator network and the pseudo-abnormal absence image to generate a periodic pseudo-abnormal presence image; compare the periodic pseudo-abnormal presence image with the actual abnormal presence image; and determine a cycle loss based on the comparison between the cycled pseudo-abnormal presence image and the actual abnormal presence image.

[0014] In the eighth embodiment, the method can include any method of the first to seventh embodiments, and can further include preprocessing a medical image to adjust the distribution of each of one or more color channels, and the corrected image is generated based on the preprocessed medical image.

[0015] In the ninth embodiment, the method can include any method of the first to eighth embodiments, and can further include preprocessing a medical image to perform segmentation of a specific organ, and the corrected image is generated based on the preprocessed medical image.

[0016] In the tenth embodiment, the method can include the method of any of the first to ninth embodiments, and estimating the size of a biological abnormality includes subtracting a corrected image from a medical image.

[0017] In the eleventh embodiment, the method can include the method of any of the first to tenth embodiments, and a specific type of biological abnormality is a lesion or a tumor.

[0018] In the twelfth embodiment, the method can include the method of any of the first to eleventh embodiments, and the medical image includes a CT image, an X-ray image, or an MRI image.

[0019] In the thirteenth embodiment, the method can include the method of any of the first to twelfth embodiments, and the medical image includes a three-dimensional image.

[0020] In the fourteenth embodiment, the method can include the method of any of the first to thirteenth embodiments, and the abnormality removal generator network includes a convolutional neural network.

[0021] In the fifteenth embodiment, the method can include the method of any of the first to fourteenth embodiments, and the training data set lacked the identification of the boundary, area, or volume of any depicted abnormality of a specific type of biological abnormality.

[0022] In the 16th embodiment, a user device uses a medical image corresponding to a subject and depicts a part of a biological abnormality for a computing system, where the biological abnormality is a specific type of biological abnormality, uses a medical image corresponding to the subject and depicts a part of the biological abnormality, and in the user device and from the computing system, receives an estimated size of the biological abnormality, where the computing system generates a corrected image based on the medical image and using an abnormality removal generator network, and the abnormality removal generator network is trained using a training dataset lacking annotations of a specific type of biological abnormality, generates a corrected image, and based on the medical image and the corrected image, determines the estimated size of the biological abnormality, and thus provides a method including receiving the estimated size of the biological abnormality determined by determining the estimated size.

[0023] In the 17th embodiment, the method can include the method of the 16th embodiment and can further include selecting a diagnostic recommendation or a treatment recommendation for the subject based on the estimated size.

[0024] In the 18th embodiment, the method can include the method of the 17th embodiment and can further include communicating the selected diagnostic recommendation or treatment recommendation to the subject.

[0025] In the 19th embodiment, the method can include any of the methods of the 16th to 18th embodiments and can further include collecting a medical image using a medical imaging system.

[0026] The 20th embodiment includes the use of an estimated size of a biological abnormality depicted in a medical image in the treatment of a subject, the estimated size being determined by a computing system using an abnormality removal generator network trained using a training data set identified as lacking annotations of a particular type of biological abnormality to generate a corrected image based on the medical image, and estimating, by the computing system, the size of the biological abnormality based on the medical image and the corrected image, and is provided by a computing device that performs a series of operations including these.

[0027] The 21st embodiment includes a system including one or more data processors and a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein (e.g., any of the methods of the 1st to 19th embodiments).

[0028] The 22nd embodiment includes a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause one or more data processors to perform some or all of one or more methods disclosed herein (e.g., any of the methods of the 1st to 19th embodiments).

[0029] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system is a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to execute some or all of one or more of the methods disclosed herein and / or some or all of one or more of the processes. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause one or more data processors to execute some or all of one or more of the methods disclosed herein and / or some or all of one or more of the processes.

[0030] The terms and expressions used are used as terms of description and not of limitation, and in the use of such terms and expressions, there is no intention to exclude equivalents or portions thereof of the features shown and described, but it is recognized that various modifications are possible within the scope of the invention as set forth in the claims. Accordingly, while the invention as set forth in the claims is specifically disclosed by embodiments and any features, modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and it should be understood that such modifications and variations are considered to be within the scope of the invention as defined by the appended claims.

Brief Description of the Drawings

[0031] The present disclosure is described in conjunction with the following accompanying drawings:

[0032]

Figure 1

[0033]

Figure 2

[0034]

Figure 3

[0035]

Figure 4

[0036]

Figure 5

[0037]

Figures 6A-6B

[0038]

Figure 7

[0039]

Figure 8

[0040]

Figure 9

[0041]

Figure 10

[0042]

Figure 11

[0043] In the accompanying drawings, like components and / or features can have the same reference labels. Further, various components of the same type can be distinguished by following the reference label with a dash and a second label that distinguishes the like components. When only the first reference label is used herein, the description is applicable to any of the like components having the same first reference label, regardless of the second reference label.

Best Mode for Carrying Out the Invention

[0044] I. Overview Detailed Description The systems, methods, and software disclosed herein facilitate the estimation of the size of biological abnormalities such as tumors. More specifically, an abnormality removal generator neural network receives a real image associated with a particular context (e.g., a particular slice level of a particular subject and / or a particular biological region of a particular subject) and is trained to generate a corresponding fake image associated with the same or a different context (e.g., a different slice level for a particular subject, the same slice level for a particular subject, or the same particular biological region for a particular subject). For some aspects (e.g., a particular subject), the corresponding fake image can reflect the real image, but the neural network can be configured such that the corresponding fake image lacks some or all of the depiction of a given type of biological abnormality depicted in the real image. For example, while both the real image and the fake image can include 3D images of the lungs, the real image can depict a tumor while the fake image does not. Thus, the size of the biological abnormality is estimated by subtracting the fake image from the real image (e.g., and determining how many pixels or voxels in the difference image exceed a threshold).

[0045] I.A. Adversarial generation network used to train the anomaly removal generation network The anomaly removal generator network can include parameters learned while training an adversarial generation network (GAN). The GAN further includes a discriminator network configured to predict whether an input image is fake (generated by the anomaly removal generator network) or real (depicting a real image collected from a subject). Feedback based on the accuracy of these predictions can be provided to the anomaly removal generator network.

[0046] In some cases, the GAN used to train the anomaly removal generator network includes a CycleGAN. The CycleGAN includes a plurality of generator networks and a plurality of discriminator networks. In addition to the anomaly removal generator network, the CycleGAN further includes an anomaly addition generator network configured and trained to receive real anomaly - free images that do not depict a particular type of biological anomaly and generate fake anomaly - present images that depict a particular type of biological anomaly. The CycleGAN also includes a first discriminator network that predicts whether an image (an image that does not truly depict a particular type of biological anomaly or has been modified to lack a depiction of a particular type of biological anomaly) is real or fake. The accuracy of the prediction can be used to provide feedback to the anomaly removal generator network. The CycleGAN can also include a second discriminator network that predicts whether an image (an image that truly depicts a particular type of biological anomaly or has been modified to include a particular type of biological anomaly) is real or fake. The accuracy of the prediction can be used to provide feedback to the anomaly addition generator network.

[0047] In some cases, the GAN used to train the anomaly removal generator network includes a CycleGAN. Similar to the CycleGAN, the CycleGAN includes a plurality of generator networks (e.g., an anomaly removal generator network and an anomaly addition generator network) and a plurality of discriminator networks (e.g., a discriminator network configured to distinguish between real anomaly-present images and fake anomaly-present images, and a discriminator network configured to distinguish between real anomaly-present images and fake anomaly-present images). Further, the CycleGAN includes one or more predictor networks that generate images associated with different contexts for an image input to the predictor network.

[0048] The first predictor network within the CycleGAN can receive one or more images depicting a biological anomaly of a particular type (e.g., a tumor, a lesion, or a plaque) and associated with a particular subject and one or more slice levels, and the first predictor can generate an output corresponding to a predicted different context image associated with the particular subject. Each corrected image can correspond to a different viewpoint, imaging modality, position, zoom, and / or slice as compared to the image received by the predictor network. In particular, the one or more images depicting the biological anomaly and received by the first predictor network can be real or fake. For example, the one or more images can be fake images generated by an anomaly addition generator network configured to modify one or more real images (e.g., lacking a depiction of a given type of biological anomaly) to add a depiction of the given type of biological anomaly.

[0049] Conversely, the second predictor network within the Recycling GAN can receive one or more images lacking the depiction of a particular type of biological abnormality (e.g., not depicting a tumor) and can generate an output corresponding to a predicted different context image also lacking the depiction of a particular type of biological abnormality (e.g., associated with a different viewpoint, imaging modality, location, zoom, and / or slice). In particular, the one or more images received by the second predictor network that do not depict a biological abnormality can be real or fake. For example, the one or more images can be fake images generated by an abnormality removal generator network configured to modify one or more real images (e.g., including any depiction of a given type of biological abnormality) to remove the depiction of a given type of biological abnormality.

[0050] I.B. Abnormality Size Prediction Using Fake Abnormality Absent Images In some cases, the abnormality removal generation network is configured to receive a 3D image (depicting at least a portion of a biological abnormality) and generate a fake 3D image lacking the depiction of at least a portion of the biological abnormality. Then, the difference between the real and fake images can indicate which voxels are predicted to be part of the biological abnormality. Thus, the size of the biological abnormality can be predicted based on the amount of voxels for which the difference between the real and fake images exceeds a threshold.

[0051] Alternatively, the abnormality removal generation network can be configured to receive a 2D image (depicting at least a portion of a biological abnormality) and generate a fake 2D image lacking the depiction of at least a portion of the biological abnormality. The difference between the real and fake images can represent the predicted region of the abnormality. However, multiple real images (e.g., corresponding to different slice levels) can be made available for a given subject, and multiple fake images can be generated. Then, the regions across the slice levels can be processed together to predict the size and / or volume of the abnormality.

[0052] I.C. Advantages The techniques disclosed herein related to the use of GANs have strong advantages in reducing the need for manual annotation and improving the objectivity and accuracy of depicting anomalies on medical images. For example, manually annotating medical images to estimate the volume of a tumor can require an enormous amount of time to identify the boundaries of the tumor in each of a plurality of consecutive slices. For example, a whole-body CT scan of a subject with advanced cancer can potentially include over 250 tumors. Detecting anomalies using manual annotation can require working hours per thousands of images and potentially millions of dollars per tumor.

[0053] This time commitment can result in a training set with a relatively small size and / or fairly low diversity. This manual approach can also be error-prone because it can be difficult for a human annotator to recall details from adjacent slides that could be helpful regarding where the boundary of the tumor is on the current slide.

[0054] On the other hand, the GAN-based techniques disclosed herein can use a machine learning model trained using data having high-level labels that indicate (such as) whether each image depicts at least a portion of a biological anomaly (such that the training data includes binary labels). The training data need not be labeled to indicate the location of a given depiction of a biological anomaly, the size of the depicted biological anomaly, the boundary of the biological anomaly, or any other spatial characteristic of the biological anomaly. That is, a training data set can be collected without performing manual segmentation. Thus, it becomes easier to obtain training data, and as a result, the training data set can be made larger and the accuracy of the model can be made higher. Alternatively or additionally, binary labeling can reduce the time or monetary investment in collecting the training set and / or predicting the size of biological anomalies.

[0055] Furthermore, the techniques disclosed herein do not rely on a training dataset that includes paired data. That is, the generator network need not be trained on a dataset that includes a set of anomaly - absent images and a corresponding "paired" set of anomaly - present images (e.g., associated with the same set of subjects). Collecting paired images can include collecting images collected from multiple imaging sessions for each subject. For example, one or more images from a first imaging session can depict a biological anomaly, and images from a second imaging session can be absent of a biological anomaly (or vice versa). However, it is very difficult to predict whether an anomaly will disappear (e.g., in response to treatment) or whether a new anomaly will appear. Thus, it can be very difficult to obtain paired images. Without paired data, it would be impossible to use many existing training techniques (e.g., those using L1 or L2 regularization) to train a neural network to generate a predicted anomaly - absent image based on a real anomaly - present image (e.g., to subsequently predict the size of the anomaly). On the other hand, the techniques disclosed herein that do not require paired data can collect a larger training dataset (e.g., an unpaired training dataset) and can result in an accurate prediction of the size of a biological anomaly.

[0056] II. Exemplary Biological Anomaly Characterization Network Figure 1 shows an exemplary biological abnormality characterization network 100 according to some embodiments. The biological abnormality characterization network 100 includes an image generation system 105 configured to collect images of a part of a subject's body. Each image can depict at least a portion of one or more biological structures (e.g., at least a portion of one or more tumors, at least a portion of one or more lesions, at least a portion of one or more plaques, and / or at least a portion of one or more organs). The subject can include a person diagnosed with a specific disease or having a diagnosis of a possible specific disease. The specific disease can include cancer or a specific type of cancer. One or more images can depict all or part of, for example, the lung, brain, or liver.

[0057] II.A. Image Generation System The images include one or more two-dimensional images and / or one or more three-dimensional images. The two-dimensional images depict a cross-sectional slice (e.g., a horizontal slice) of the subject or a part of a cross-sectional slice of the subject. The three-dimensional images depict a three-dimensional region of the subject. The three-dimensional images can be generated by stacking or combining a plurality of two-dimensional images (corresponding to slices of the subject imaged at multiple slice levels). Thus, as used herein, the "region" of the subject depicted in an image refers to a three-dimensional volume within the subject, and the "slice" of the subject depicted in an image refers to a two-dimensional cross-section of the subject.

[0058] The image generation system 105 can include, for example, a computed tomography (CT) scanner, an X-ray machine, or a magnetic resonance imaging (MRI) machine. The images can include radiographic images, CT images, X-ray images, or MRI images. The images may be collected without a contrast agent being administered to the subject or after a contrast agent has been administered to the subject. In some cases, the image generation system 105 can first collect a set of two-dimensional images and use the two-dimensional images to generate a three-dimensional image.

[0059] The images collected by the image generation system 105 may be collected without administering a contrast agent to the subject or after administering a contrast agent to the subject. The subject being imaged can include a subject diagnosed with cancer, having a diagnosis or preliminary diagnosis of possible cancer, and / or having symptoms consistent with cancer or a tumor.

[0060] The image generation system 105 can store the collected images in an image data store 110 that can include, for example, a cloud data store. Each image can be stored in relation to one or more identifiers such as an identifier of the subject and / or an identifier of a caregiver associated with the subject. Each image may be further stored in association with the date on which the image was collected.

[0061] II.B. Image Processing System One or more images are utilized by an image labeling system 115 that facilitates identification of labels for each of the one or more images. The label indicates whether the image depicts a biological abnormality. It will be understood that a label indicating that an image depicts a biological abnormality can indicate that the image depicts a part of the biological abnormality (e.g., a slice of the biological abnormality).

[0062] The image labeling system 115 can identify a label based on input received by a human user. For example, the image labeling system 115 can present each of the one or more images on a display and receive an input (e.g., a click of a given button, a selection from a pull - down option, an input of text, etc.) indicating whether each image depicts a biological abnormality (e.g., depicts at least a part of a tumor).

[0063] Alternatively, the image labeling system 115 may identify labels using automated image processing techniques. For example, the image labeling system 115 can predict that an image depicts a biological abnormality if at least a threshold percentage or at least a threshold number of voxels or pixels have an intensity that exceeds a predetermined threshold. The threshold can be defined to distinguish a body part that has absorbed a contrast agent (e.g., a tumor) from other parts of the body. As another example, the image labeling system 115 can identify labels based on the source of the image. For example, the first source of the image can include a database, library, or medical provider system associated with confirmed tumor cases, and the second source of the image can include a database, library, or medical provider system associated with healthy subjects.

[0064] The image labeling system 115 can update the image data store 110 to include one or more labels indicating whether each of the one or more images includes a depiction of a biological abnormality.

[0065] II.C. Image Processing System The image processing system 125 (which can include, for example, a remote and / or cloud-based computing system) is configured to predict the size of the depicted abnormality for each of one or more biological abnormalities of a given type.

[0066] II.C.1. Pretreatment Controller More specifically, the image processing system 125 can train a GAN and subsequently use the trained anomaly removal generator network from the GAN to process an input image (depicting a particular type of biological abnormality) to generate a corrected image (not depicting a particular type of biological abnormality) and be configured to predict the size of the biological abnormality using the input image and the corrected image.

[0067] More specifically, the image processing system 125 includes a preprocessing controller 130 that initiates and / or controls the preprocessing of an image. The preprocessing can include, for example, converting the image to a predetermined format, resampling the image to a predetermined sampling size, resampling the image to a predetermined size (e.g., fewer than a specified number of pixels or voxels in each of one, more, or all dimensions), trimming the image to a predetermined size, generating a three-dimensional image based on a plurality of two-dimensional images, generating one or more images having different (e.g., target) viewpoints, adjusting intensity values (e.g., normalizing or standardizing), and / or adjusting color values.

[0068] The preprocessing can include a transformation that adjusts one or more color channels. For example, a grayscale image can be transformed to include one or more color channels.

[0069] In some cases, for each image, the preprocessing controller 130 segments the image by detecting a region that depicts at least a portion of a particular type of biological organ (e.g., the lungs). Segmentation can be performed using, for example, a neural network (e.g., a convolutional neural network) trained using supervised learning. The image can be modified to include only the portion of the image identified as depicting a particular type of biological organ. For example, the intensity values of all other pixels or voxels can be set to 0 or a value other than a number.

[0070] II.C.2. GAN Training Controller The image processing system 125 includes a GAN training controller 135 that trains a GAN using labeled images. The GAN can be trained using a training set that includes a first set of images, each of which depicts (which can include depicting at least a portion of) at least one biological abnormality, where the at least one biological abnormality is a particular type of abnormality. For example, the images can include 3D images that depict an entire tumor, 2D images that depict a full cross-section of a tumor, 3D images that depict a portion of a tumor, or 2D images that depict a portion of a cross-section of a tumor. The training set also includes a second set of images, where each image in the second set does not depict a biological abnormality of the particular type of abnormality. Each image in the dataset can be associated with a label indicating whether it depicts a particular type of biological abnormality, but the training set need not include segmentation or annotation data indicating where in the image an abnormality is depicted. The images in the training set can all depict at least a portion (or all) of a particular organ, or the training set can include images that depict different parts of the body.

[0071] The GAN can include a Cycle GAN or a Recycle GAN. The GAN includes an anomaly removal generator neural network trained to receive an image depicting a particular type of biological anomaly (e.g., a tumor) and output a modified (fake) image that does not depict the particular type of biological anomaly. The anomaly removal generator network can include one or more convolutional layers, a U-net, or a V-net. In some cases, the anomaly removal generator network includes a feature extraction encoder (including one or more convolutional layers), a transformer (including one or more convolutional layers), and a decoder (including one or more convolutional layers). The modified image and the received image can share various contexts (e.g., the depicted area / volume, size, depicted organ, etc.). The discriminator network receives the fake image from the anomaly removal generator network and also receives a real image that does not depict the particular type of biological anomaly. The discriminator network can include one or more convolutional layers and activation layers. The discriminator network predicts, for each image, whether the image is real or fake. Based on the accuracy of the discriminator network's prediction, feedback can be sent to the anomaly removal generator network.

[0072] Training the GAN can include using one or more loss functions. Training can include introducing a penalty when the discriminator network of the GAN incorrectly predicts that a real image is a fake image (or vice versa). Training may additionally or alternatively include introducing a cycle loss. The cycle loss can be calculated by double-processing the original image including (or alternatively lacking) a depiction of a particular type of anomaly using one of the GAN's generator networks to remove (or alternatively add) such a depiction in order to generate a modified image predicted to lack (or alternatively include) a depiction of the particular type of anomaly, and then processing the modified image using another generator network of the GAN to add (or alternatively remove) a depiction of the anomaly in order to generate a cycle image. The loss can be scaled by the difference between the original image and the cycled image.

[0073] II.C.2.a. Cycle GAN GAN trained by the GAN training controller 135 can include a Cycle GAN that includes a plurality of generator networks and a plurality of discriminator networks. FIG. 2 shows a representation of the networks and network connections in the Cycle GAN, and FIG. 3 shows a flow of generating a fake image using the Cycle-GAN and discriminating between a real image and the fake image. In this depiction, each rectangular box represents an (actual or fake) image, and each diamond represents a discrimination.

[0074] The X and Y domains are related to different types of images. In this case, the Y domain corresponds to images depicting a particular type of biological abnormality (e.g., a lesion), and the X domain corresponds to images lacking the depiction of a particular type of biological abnormality.

[0075] In addition to including an abnormality removal generator network, the Cycle GAN also includes an abnormality addition generator network configured and trained to modify an image to add a depiction of a particular type of biological abnormality. The abnormality addition generator network can include one or more convolutional layers, a U-net, or a V-net. In some cases, the abnormality addition generator network includes a feature extraction encoder (including one or more convolutional layers), a transformer (including one or more convolutional layers), and a decoder (including one or more convolutional layers). The architecture of the abnormality addition generator network may be the same as the architecture of the abnormality removal generator network.

[0076] The anomaly addition generator network and / or the anomaly removal generator network can be collectively trained using a cycle consistency loss or a cycle loss. In this case, the original image is compared with the image generated by first processing it by one generator network and then by the other generator network, and the loss can be made larger if the difference in the images is more dramatic. For example, the original image 305 may depict a patient's lung and not include a depiction of a tumor. This original image can be processed by the anomaly addition generator network to generate a fake tumor presence image 310, which can then be processed by the anomaly removal generator network to generate a fake tumor absence image 310'. The loss may be calculated to scale according to the difference between the original image and the fake tumor absence image. The cycle consistency loss can facilitate reducing the occurrence or degree of mode collapse (in which case the generator may start generating stereotypes of images that may not correspond to the original image). However, minimizing the cycle loss in CycleGAN does not guarantee that when mapping from one domain to another, the image does not move in a single "mode" where the image changes by only a few pixels from one image to another. For example, the anomaly addition generator network can generate a fake anomaly presence image that looks almost identical to the input (e.g., real) anomaly presence image. Similarly, the anomaly removal generator network can generate a fake anomaly absence image that looks almost identical to the input (e.g., fake) anomaly presence image. Thus, in some cases, the loss can be calculated based on a combination of a periodic loss and an aperiodic loss (e.g., the aperiodic loss is determined based on the degree to which the discriminator network can distinguish between real and fake images).

[0077] In addition to including a discriminator network that discriminates real images and fake images that do not depict a particular type of abnormality (e.g., real tumor - absent image 305 and fake tumor - absent image 310’) (at block 315’), CycleGAN also includes another discriminator network that discriminates fake images and real images that depict a particular type of abnormality (e.g., real tumor - present image 305’ and fake tumor - present image 310) (at block 315). Each of the discriminator network and the other discriminator network can include one or more convolutional layers and activation layers. The architecture of the discriminator network may be the same as the architecture of the other discriminator network.

[0078] Each generator network (G X and G Y ) is configured to receive an image from one domain and generate a corresponding image in another domain (the other domain is indicated by a subscript). G Y is configured to receive a real image 305 that does not include any depiction of a particular type of biological abnormality (X domain) and generate a corresponding fake image 310 that includes a depiction of a particular type of biological abnormality (Y domain). On the other hand, G X is configured to receive a real image 305’ that includes a depiction of a particular type of biological abnormality (Y domain) and generate a corresponding image 310’ that lacks a depiction of a particular type of biological abnormality (X domain).

[0079] Each discriminator network processes the fake image (generated by the generator network) to predict whether the image is real. Each discriminator network similarly processes real images to predict whether the image is real. Each discriminator network is domain - specific. Thus, for example, discriminator network D Y is for depicting a particular type of biological abnormality, and the biological abnormality is known (e.g., via metadata associated with the image) or (abnormality - addition generator network G Y or predictor network P YGenerate predictions of real or fake images (in block 315) generated by). Similarly, the discriminator network D X is known to lack a depiction of a given type of biological abnormality (e.g., via metadata associated with the image) or (the abnormality removal generator network G X or the predictor network P X to generate predictions of real or fake images (in block 315') generated by).

[0080] Discrimination results corresponding by chance indicate that the discriminator network is unable to distinguish between real and fake images, and thus that the fake images are likely to be of high quality. The results of discrimination performed by a discriminator network operating in a given domain can be used to adjust the parameters of the corresponding generator network during the learning process.

[0081] Cycle GAN can be used to process 3D images. The generator and discriminator networks can then include 3D convolutional layers. The network can then be configured to learn how features extending across slices predict whether an image depicts a particular type of biological abnormality and where in the image the particular type of biological abnormality is depicted. Alternatively or additionally, the 3D image can be reshaped into a 1D vector and processed using 2D convolutional layers.

[0082] Processing 3D images using a neural network can be understood to use a significant amount of memory. Therefore, if the biological region of interest (e.g., an organ) is large, preprocessing techniques can be performed to segment the organ to reduce its size. Alternatively or additionally, the image may be preprocessed to reduce the spatial resolution of the image. For example, if the organ of interest is the lung (when the size of the lung is relatively large compared to other organs), it can be useful to segment the organ and resample it to reduce the resolution.

[0083] II.C.2.b. Recycling GAN The GAN trained by the GAN training controller 135 can include a recycling GAN that includes a plurality of generator networks, a plurality of discriminator networks, and a plurality of predictor networks. FIG. 4 shows a representation of the networks and network connections in the recycling GAN. In addition to the networks depicted in the cycle GAN of FIG. 2, the recycling-GAN shown in FIG. 4 includes a plurality of predictor networks, P X and P Y and includes.

[0084] Each predictor network can include one or more convolutional layers and / or a U-net. Each predictor network predicts an image obtained in a context different from one or more images input to the predictor network. For example, the predictor network can include a convolutional neural network and / or generate an image corresponding to a different viewpoint, imaging modality, position, zoom, and / or slice compared to the viewpoint, imaging modality, position, zoom, and / or slice of the image received by the predictor network. For example, the predictor P X can predict the image of slice x t based on the image of adjacent slice x t+1 , and P Y can predict the image of slice y s based on the image of adjacent slice ys+1 can predict the image of.

[0085] Figure 5 shows how various fake images can be generated and evaluated in a neural network configuration according to some embodiments of the present invention. In this depiction, each rectangular box represents an (actual or fake) image, and each diamond represents an identification.

[0086] Each of the first and second actual anomaly - free images 505 shows the first and second slices (respectively) of the sample and lacks the depiction of a particular type of biological anomaly. G Y G adds the depiction of a particular type of biological anomaly and uses the first and second actual anomaly - free images 505 to generate the first and second fake anomaly - present images 510 of the first and second slices. P Y D uses the first and second fake anomaly - present images of the first slice 510 to generate the third fake anomaly - present image 515 of the third slice. In block 520, D Y predicts whether the third fake anomaly - present image 515 is real or fake. D Y D further predicts whether the first and second actual anomaly - present images 505' of the other first and second slices of another sample are real or fake. Although not shown, D Y can further predict whether the first and second fake anomaly - present images 505 of the first and second slices are real or fake. Feedback from the identification (e.g., based on recall statistics) can be fed back to G Y can be fed back.

[0087] The third fake anomaly - present image 515 of the third slice is fed to G X which thereby generates the periodic fake anomaly - free image 525 of the third slice. P X P uses the first and second actual anomaly - free images 505 to generate another fake anomaly - free image 530 of the third slide. In block 535, the fake anomaly - free images 525 and 530 (corresponding to the third slice) are compared with each other, and PX It is possible to identify the loss. Further, at 540, the periodic false anomaly non-existence image of the third slide 525 is compared with the third actual anomaly non-existence image 535 of the third slice to determine the cycle loss.

[0088] Figure 5 represents symmetric operation. More specifically, the first and second actual anomaly existence images 505' of the first and second slices of other samples generate the first and second false anomaly non-existence images 510' of the first and second slices of other samples to G X which is supplied. P X can use the first and second false anomaly non-existence images 510' to generate the third false anomaly non-existence image 515' of the third slice of the other sample 515'. In block 520', D X predicts whether the third false anomaly non-existence image 515' is real or false. D X further predicts whether the first and second actual anomaly non-existence images 505 are real or false. Although not shown, D X can further predict whether the first and second false anomaly non-existence images 510' of the other first and second slices are real or false. The feedback from the identification (e.g., based on the recall statistic) can be fed back to G X .

[0089] The third false anomaly non-existence image 515' of the other third slice generates the periodic false anomaly existence image 525' of the third slice to G Y which is supplied. P Y uses the first and second actual anomaly existence images 505' to generate another false anomaly existence image 530' of another third slide. In block 535', the false anomaly existence images 525' and 530' (corresponding to other third slices) are compared with each other to identify the loss of P Y . Further, at 540', the periodic false anomaly existence image 525' of the other third slide is compared with the third actual anomaly existence image 535' of the third slice to determine the cycle loss.

[0090] Figures 6A - 6B show how various types of losses can occur in a Recycled GAN. As illustrated in Figure 6A, even though the discriminator network and the predictor network are involved in the overall training and backpropagation, their weights do not need to be updated in response to the discrimination result. Rather, the discrimination result can selectively trigger the update of only the weights of the corresponding generator network. However, the backpropagation flowing through the discriminator network causes its weights to be considered in generator training, thereby giving some advantage beyond the discriminator to generator training. (Figure 6A)

[0091] Even if the generator network is used to generate a fake image, the generator network is in front of the corresponding discriminator within the architecture. Thus, the generator weights may not have access to discriminator training. (Figure 6B)

[0092] The Recycled GAN can be used to process a set of 2D images. Then, the generator and discriminator networks can include 2D convolutional layers. The predictor network facilitates introducing weak coherence (e.g., slices) between domains, such that the network can learn anatomical tissues (e.g., over a volume encompassed by multiple slices), and then generate more realistic fake images (lacking that anatomical tissue as they are only 2D slices), distinguish real images from fake images, and then can easily further improve the generation of fake images.

[0093] II.C.3. Generator Network Controller The GAN training controller 135 can train the complete GAN network, and then the generator network controller 140 can use a single generator network from the GAN to evaluate medical images. More specifically, the single generator network can include those that receive an input image depicting a particular type of biological abnormality (e.g., a tumor or a lesion) and generate a corrected image lacking the depiction of the particular type of biological abnormality.

[0094] II.C.4. Size Detector Next, the size detector 145 can be configured to receive each of one or more corrected images (i.e., lacking the depiction of a particular type of biological abnormality) and the input image (e.g., depicting a particular type of biological abnormality) generated by the abnormality removal generator network controller 140 and predict the size of the biological abnormality. For example, the size detector 145 can subtract the corrected image from the input image. In some cases, the size detector 145 can first process the difference image (e.g., to apply one or more spatial filters and / or thresholds).

[0095] The predicted size may be the size of a region identified in the difference image (e.g., when the input image is a two-dimensional image) or the size of a volume (e.g., when the input image is a three-dimensional image). For example, the size detector 145 can determine, for each pixel or voxel, whether an intensity condition (e.g., a condition configured to be satisfied when the intensity of the pixel or voxel is not equal to 0, or a condition configured to be satisfied when the intensity of the pixel or voxel is within a predetermined open or closed range) is satisfied. The predicted size may be defined based on the amount of pixels or voxels for which the condition is satisfied. Alternatively, for each pixel or voxel, a binary image indicating whether the condition is satisfied may be generated. Then, a low-pass spatial filter can be applied so as to pass when the pixels or voxels for which the condition is satisfied are adjacent to other pixels (e.g., of a sufficient amount and / or shape). The filtered image can include another binary image. Then, the predicted size of the biological abnormality can be determined by processing the filtered image (e.g., by thresholding, summing, or averaging across the pixels or voxels in the filtered image).

[0096] In some cases, the size detector 145 processes a plurality of original images and modified two-dimensional images, and the size detector 145 predicts the size based on the processing of the plurality of images. As an example, the biological abnormality area can be estimated for each two-dimensional image, and the areas can be aggregated to estimate the volume of the biological abnormality. As another example, a plurality of input images (e.g., depicting a particular type of biological abnormality) are processed to generate a three-dimensional abnormality presence image, and a plurality of modified images (e.g., lacking the depiction of a particular type of biological abnormality) are processed to generate a three-dimensional abnormality absence image, and the size detector 145 can estimate the size of the biological abnormality based on the three-dimensional abnormality presence image and the three-dimensional abnormality absence image. For example, the size detector 145 can subtract the three-dimensional abnormality absence image from the three-dimensional abnormality presence image. The resulting image can be processed (e.g., via thresholding, filtering, and / or smoothing), but it is not necessary to process it. The size (e.g., area or volume) of the biological abnormality can be predicted. In some cases, the predicted size may be based on and / or dependent on the resolution and / or size of the original image and / or the modified image. As yet another example, for each of a plurality of orthogonal dimensions, an actual tumor presence image can be accessed, a pseudo-tumor absence image can be generated, and a predicted tumor depiction can be generated by subtracting the pseudo-tumor absence image from the actual tumor presence image. The size detector 145 can then determine the longest diameter of the predicted tumor depiction, and the longest diameter can be defined as the maximum of the three longest diameters.

[0097] For example, FIG. 7 shows how the spatial characteristics of biological abnormalities can be estimated based on a comparison between a real image and a fake image according to some embodiments of the present invention. Specifically, the left image shows an initial image including a depiction of a lesion and surrounding tissue. The central image shows a fake image generated by a trained anomaly removal generator network configured to remove the damage. The right image shows the result of subtracting the fake image from the corresponding initial image. These images can be processed to characterize the lesion. For example, pixels having a threshold value above a predetermined threshold can be counted. The count may be normalized or scaled based on the scale of the image and / or the resolution of the image, and the normalized or scaled count can indicate the total lesion size of the imaged slice. In some cases, this process is repeated over images corresponding to different slices, and the tumor volume can be estimated by performing a sum over the images.

[0098] In addition to, or instead of, predicting the size of biological abnormalities based on real and fake images, the real and fake images can be used to segment the biological abnormalities depicted in the real image. The segmentation result can identify which parts of the image depict a particular type of biological abnormality. For example, the segmentation result can include an overlay on the original image, which indicates which parts of the original image are predicted to depict a particular type of biological abnormality.

[0099] The image processing system 125 can return the size (and / or segmentation result) to the user device 150. The user device 150 can include a device that requests an estimated size corresponding to a biological abnormality, or a device that provides one or more images depicting a biological abnormality. The user device 150 can be associated with a medical professional and / or a caregiver who is treating and / or evaluating the imaged subject. In some cases, the image processing system 125 can return an estimated value of the size of the biological abnormality image generation system 105 (for example, and then the estimated volume can be transmitted to the user device). In some cases, instead of or in addition to outputting the estimated size, the image processing system 125 outputs treatment recommendations (for example, to the user device 150). For example, the image processing system 125 can use one or more rules to identify treatment recommendations based at least in part on the estimated size (for example, potentially one or more previous estimated sizes of the biological abnormality). By way of illustration, the rules can include thresholds, and indicate that treatment strategy change recommendations should be considered if the estimated size of a biological abnormality of a given subject is not at least X% smaller than the previous estimated size of the biological abnormality of the given subject (for example, associated with a defined period).

[0100] It will be appreciated that in some cases, the biological abnormality characterization network 100 can be used to estimate the size of each of a plurality of biological abnormalities. For example, the abnormality removal generator network may be trained to generate a corrected image that does not depict any brain lesions even if the input image depicts a plurality of images. Then, the difference between the input image and the corrected image can be used to estimate the cumulative size of the brain lesions.

[0101] III. Exemplary Biological Abnormality Characterization Network FIG. 8 shows a flowchart of an exemplary process 800 for estimating the size of a biological abnormality. At block 805, the image processing system 125 accesses a medical image. The medical image depicts at least a portion of a biological abnormality (e.g., a tumor or a lesion). The medical image may be a two-dimensional image or a three-dimensional image. The medical image can be a CT image, an MRI image, an X-ray image, etc. The medical image may be collected and / or accessed by the image generation system 105.

[0102] At block 810, the generator network controller 140 uses the abnormality removal generator network to generate a corrected image of the medical image. The corrected image can be generated or predicted (by the abnormality removal generator network) to lack the depiction of at least a portion of the biological abnormality. The abnormality removal generator network may be trained by the GAN training controller 135 that trains the GAN. The corrected image can be generated based on the processing (by the generator network controller 140) of the preprocessed image. The GAN may include at least the abnormality removal generator network and one or more discriminator networks. The GAN may also include an abnormality addition generator network and / or one or more prediction networks.

[0103] At block 815, based on the medical image and the corrected image, the size of the biological abnormality is estimated (e.g., by the size detector 145). For example, the original image (or its preprocessed version) is subtracted from the corrected image to generate a remaining image. The remaining image can then be processed (e.g., after potentially applying filtering and / or thresholding) by summing or averaging the voxels or pixels within the remaining image.

[0104] In block 820, an estimated size of the biological abnormality is output (e.g., by image processing system 125 and / or to user device 150). The output can include transmitting the estimated size and / or presenting the estimated size. In some cases, the output includes transmitting and / or presenting a result (e.g., treatment recommendation) based on the estimated size.

[0105] V. Examples V.A. Example 1 FIG. 9 shows examples of images used and generated using CycleGAN to detect lesions. The upper images in FIG. 9 show three actual orthogonal CT slices of a subject's right lung. The visualization of the slices shows slices intersecting in three-dimensional space to represent the spatial relationship between the slices. Each of the actual orthogonal CT slices included a depiction of a lesion.

[0106] A CycleGAN model having an architecture as represented in FIG. 2 was trained using training data. Then, the trained anomaly removal generator network G X processed each of the three actual CT slices to generate corresponding fake CT images predicted to lack a depiction of the lesion. These fake tumor absent images are shown as interesting in the three-dimensional space of the middle images in FIG. 9.

[0107] For each of the real images, the corresponding fake image was subtracted from the real image to generate a predicted two-dimensional depiction of the tumor. The lower images in FIG. 9 show tumor prediction images intersecting in three-dimensional space. Predicting the size of the tumor in each of the three dimensions can facilitate estimating the longest diameter of the tumor. For example, the longest diameter of the tumor depiction can be estimated for each of the three tumor prediction images, and the longest diameter can be defined as the maximum value of the three longest diameters.

[0108] V.B. Example 2 Figure 10 shows exemplary lesion detection performed by using the Cycle-GAN algorithm or by four human readers. The unmasked images shown in Figure 10 are the same as the images shown in the upper image of Figure 9. However, here the images are arranged side by side rather than overlapping. The remaining images show the same unmasked image with the predicted lesion delineations overlaid. The predicted lesion delineations shown in the algorithm image are the same as those shown in the lower image of Figure 9 and were calculated by comparing the unmasked image with a pseudo-abnormality absent image (generated by an abnormality removal generator network trained using CycleGAN).

[0109] The remaining images show the lesion annotations made by each of four human readers. In particular, the predicted lesion regions identified by using the abnormality removal generator network were similar to the lesion regions identified by the human readers.

[0110] V.C. Example 3 The training dataset was defined to include 1,300 whole-body CT images of chest volumes with cancer lesions and 300 whole-body CT images of chest volumes without cancer. Each image was labeled to indicate whether the image corresponded to a chest volume with cancer (having one or more lesions). The data was split 80:20 for training and testing. A model having an architecture as shown in Figure 4 was trained using the training data.

[0111] Figure 11 shows the results from the processing of the test data by comparing the lesion detection output by the trained model with the lesion detection by a trained radiologist. In particular, the delineated results are quite consistent. This model had a sensitivity of 0.94 at the ROI level and a sensitivity of 0.89 at the lesion level. The segmentation achieved a Dice similarity coefficient of 0.2.

[0112] VI. Further Considerations Some embodiments of the present disclosure include a system that includes one or more data processors. In some embodiments, the system is a non-transitory computer-readable storage medium that includes instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods and / or some or all of one or more of the processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium that includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods and / or some or all of one or more of the processes disclosed herein.

[0113] The terms and expressions used are used as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, to exclude equivalents or portions thereof of the features shown and described, but it is recognized that various modifications are possible within the scope of the invention as set forth in the claims. Accordingly, the invention as set forth in the claims is specifically disclosed by embodiments and any features, but modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and it is to be understood that such modifications and variations are considered to be within the scope of the invention as defined by the appended claims.

[0114] The description provides only exemplary embodiments that are preferred and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the description of the preferred exemplary embodiments provides those skilled in the art with a possible description for implementing various embodiments. It is understood that various changes can be made to the functions and arrangements of the elements without departing from the spirit and scope recited in the appended claims.

[0115] In the description, specific details are set forth in order to provide a thorough understanding of the embodiments. It will be understood, however, that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the embodiments with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments. Examples of potentially patentable subject matter include, but are not limited to, the following.

Claims

1. A computer-implemented method, comprising: accessing a medical image corresponding to a subject and depicting at least a portion of a biological abnormality, wherein the biological abnormality is a specific type of biological abnormality; generating a corrected image based on the medical image and using an abnormality removal generator network, wherein the abnormality removal generator network is trained using a training dataset lacking annotations of the specific type of biological abnormality; estimating the size of the biological abnormality based on the medical image and the corrected image; outputting the estimated size of the biological abnormality; wherein the computer-implemented method further comprises: training the abnormality removal generator network by training an adversarial generative network (GAN), inputting a real abnormality presence image depicting at least a portion of the subject and at least a portion of another biological abnormality of the specific type of biological abnormality into the abnormality removal generator network; generating a fake abnormality absence image using at least the abnormality removal generator network and the real abnormality presence image; performing discrimination using the discriminator network of the GAN to predict whether the fake abnormality absence image corresponds to a true image of an actual sample or a fake image; adjusting one or more weights of the abnormality removal generator network based on the discrimination performed by the discriminator network; thereby training the abnormality removal generator network; inputting the fake abnormality absence image into the abnormality removal generator network of the GAN; generating a periodic fake abnormality presence image using an abnormality addition generator network and the fake abnormality absence image; comparing the periodic fake abnormality presence image with the real abnormality presence image; determining a cycle loss based on the comparison between the periodic fake abnormality presence image and the real abnormality presence image. A computer-implemented method further comprising the above.

2. The adversarial generative network comprises: the abnormality removal generator network; One or more discriminator networks, each of the one or more discriminator networks being configured and trained to discriminate between a real image and an image generated by a generator network, the generator network including the anomaly removal generator network or the anomaly addition generator network, one or more discriminator networks The computer-implemented method according to claim 1, comprising:

3. A set of parameters of the anomaly removal generator network is defined by training an adversarial generation network, the adversarial generation network the anomaly removal generator network, and One or more discriminator networks, each of the one or more discriminator networks being configured and trained to discriminate between a real image and an image generated by a generator network, the generator network including the anomaly removal generator network or the anomaly addition generator network, one or more discriminator networks The computer-implemented method according to claim 1, comprising:

4. The adversarial generation network the anomaly removal generator network, and anomaly addition generator network, and A first discriminator network, A real image labeled as depicting at least a portion of the biological anomaly of the specific type of biological anomaly, configured to discriminate between a fake image generated by the anomaly addition generator network, The anomaly addition generator network receives feedback based on a first discrimination result generated by the first discriminator network, a first discriminator network A second discriminator network, A real image labeled as depicting no biological anomaly of the specific type of biological anomaly, configured to discriminate between a fake image generated by the anomaly removal generator network, The anomaly removal generator network receives feedback based on a first discrimination result generated by the first discriminator network, a second discriminator network The computer-implemented method according to claim 1, comprising:

5. A set of parameters of the anomaly removal generator network is defined by training an adversarial generation network, the adversarial generation network the anomaly removal generator network, and An abnormal addition generator network, and A first discriminator network, Real images labeled as depicting at least a part of another abnormality of the specific type of biological abnormality, It is configured to distinguish between at least the fake images generated by the abnormal addition generator network, A first discriminator network, wherein the abnormal addition generator network receives feedback based on the first discrimination result generated by the first discriminator network, A second discriminator network, Real images labeled as not depicting any biological abnormality of the specific type of biological abnormality, It is configured to distinguish between the fake images generated by the abnormal removal generator network, A second discriminator network, wherein the abnormal removal generator network receives feedback based on the first discrimination result generated by the first discriminator network The computer-implemented method according to claim 1, comprising

6. The computer-implemented method according to any one of claims 1 to 5, further comprising preprocessing the medical image to adjust the distribution of each of one or more color channels, and the corrected image being generated based on the preprocessed medical image.

7. The computer-implemented method according to claim 1, further comprising preprocessing the medical image to perform segmentation of a specific organ, and the corrected image being generated based on the preprocessed medical image.

8. The computer-implemented method according to claim 1, wherein estimating the size of the biological abnormality includes subtracting the corrected image from the medical image.

9. The computer-implemented method according to claim 1, wherein the specific type of biological abnormality is a lesion or a tumor.

10. The computer-implemented method according to claim 1, wherein the medical image includes a CT image, an X-ray image, or an MRI image.

11. The computer-implemented method according to claim 1, wherein the medical image includes a three-dimensional image.

12. The computer-implemented method according to claim 1, wherein the abnormal removal generator network includes a convolutional neural network.

13. The computer-implemented method of claim 1, wherein the training data set lacks any identification of the boundaries, areas, or volumes of any of the depicted abnormalities of the specific type of biological abnormality.

14. Using, by a user device and for a computing system, a medical image corresponding to a subject and depicting a part of a biological abnormality, wherein the biological abnormality is a specific type of biological abnormality, and depicting a part of the biological abnormality; Receiving, by the user device and from the computing system, an estimated size of the biological abnormality, wherein the computing system Generating, based on the medical image and using an abnormality removal generator network, a corrected image, wherein the abnormality removal generator network is trained using a training data set lacking annotations of the specific type of biological abnormality; Determining the estimated size of the biological abnormality based on the medical image and the corrected image; Receiving the estimated size of the biological abnormality, which is determined by determining the estimated size; comprising; wherein the computing system Trains the abnormality removal generator network by training an adversarial generative network (GAN), Inputting, into the abnormality removal generator network, a real abnormality presence image depicting at least a part of the subject and at least a part of another biological abnormality of the specific type of biological abnormality; Generating a fake abnormality absence image using at least the abnormality removal generator network and the real abnormality presence image; Performing an identification using the discriminator network of the GAN to predict whether the fake abnormality absence image corresponds to a true image of an actual sample or a fake image; Adjusting one or more weights of the abnormality removal generator network based on the identification performed by the discriminator network; Training the abnormality removal generator network; Inputting the fake abnormality absence image into the abnormality removal generator network of the GAN; Using the abnormal addition generator network and the pseudo-abnormal non-existent image to generate a periodic pseudo-abnormal existent image; Comparing the periodic pseudo-abnormal existent image with the actual abnormal existent image; Determining a cycle loss based on the comparison between the periodic pseudo-abnormal existent image and the actual abnormal existent image; A method further configured to perform the above.

15. The method according to claim 14, further comprising selecting a diagnostic recommendation or a treatment recommendation for the subject based on the estimated size.

16. The method according to claim 15, further comprising communicating the selected diagnostic recommendation or treatment recommendation to the subject.

17. The method according to claim 14, further comprising collecting the medical image using a medical imaging system.

18. Use of the estimated size of a biological abnormality depicted in a medical image in the treatment of a subject, wherein the estimated size is Generating a corrected image by a computing system based on the medical image and using an abnormal removal generator network, wherein the abnormal removal generator network is trained using a training data set identified as lacking annotations of a specific type of biological abnormality; Estimating the size of the biological abnormality by the computing system based on the medical image and the corrected image Provided by a computing device that performs a set of operations including The set of operations includes Training the abnormal removal generator network by training an adversarial generative network (GAN), Inputting an actual abnormal existent image that depicts at least a part of the subject and at least a part of another biological abnormality of the specific type of biological abnormality into the abnormal removal generator network; Generating a pseudo-abnormal non-existent image using at least the abnormal removal generator network and the actual abnormal existent image; Performing an identification using the discriminator network of the GAN to predict whether the pseudo-abnormal non-existent image corresponds to a true image of an actual sample or a false image; Adjusting one or more weights of the abnormal removal generator network based on the identification performed by the discriminator network training the anomaly removal generator network; inputting the fake anomaly-free image into the anomaly removal generator network of the GAN; using the anomaly addition generator network and the fake anomaly-free image to generate a periodic fake anomaly-present image; comparing the periodic fake anomaly-present image with the real anomaly-present image; further comprising determining a cycle loss based on the comparison between the periodic fake anomaly-present image and the real anomaly-present image. **Claim 19** A system comprising: one or more data processors; a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform the method according to any one of claims 1 to 17. A system comprising the above. **Claim 20** A computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause one or more data processors to perform some or all of the one or more methods according to any one of claims 1 to 17.

Images