Machine Learning Based Segmentation of Biological Objects in Medical Images

A neural network system addresses the limitations of RECIST by automating tumor and organ segmentation in medical imaging, providing precise and efficient lesion tracking across multiple organs, thus improving diagnostic accuracy and reducing manual effort.

JP2024542233APending Publication Date: 2024-11-13GENENTECH INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
JP2024529856
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-22
Filing Date
2022-11-21
Publication Date
2024-11-13

AI Technical Summary

Technical Problem

Current methods for tumor detection and segmentation in medical imaging, such as RECIST, are limited by their reliance on manual annotation, inconsistency in lesion selection, and inability to accurately assess tumor burden in multi-lesion cases, leading to variability and inefficiency in disease staging.

Method used

A neural network processing system that employs a multi-stage approach for automated tumor and organ segmentation, using coarse and fine segmentation models to accurately identify and track lesions across multiple organs, reducing computational load and enhancing precision.

Benefits of technology

The system provides precise, automated tumor and organ segmentation, improving lesion tracking and disease assessment by reducing computational requirements and minimizing manual intervention, thereby enhancing diagnostic accuracy and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542233000001_ABST
    Figure 2024542233000001_ABST
Patent Text Reader

Abstract

In one embodiment, a method includes accessing a first scan image from a set of computed tomography (CT) scan images, each CT scan image being at a first resolution; generating a first downscaled image of the first scan image by resampling the first scan image to a second resolution lower than the first resolution; determining a coarse segmentation corresponding to an organ depicted in the first scan image by a first machine learning model based on the first downscaled image; extracting segments of the first scan image based on the coarse segmentation, each extracted segment being at the first resolution; determining a fine segmentation corresponding to each organ depicted in the extracted segments by a second machine learning model based on the extracted segments; and generating a segmented image of the first scan image based on the fine segmentation, wherein the segmented image includes the confirmed segmentation corresponding to the organ.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 282,038, filed November 22, 2021, the entire contents of which are incorporated herein by reference in their entirety.

[0002] The present disclosure relates to systems and methods for identification and segmentation of biological objects using medical images. [Background technology]

[0003] Medical imaging (e.g., CT scans, X-rays, or MRI scans) are widely used for tumor detection to aid in the diagnosis and treatment of cancer (e.g., lung cancer, breast cancer, etc.). In many cases, health care professionals evaluate the effectiveness of drugs and / or treatment plans through measuring changes in tumor size or volume. Response Evaluation Criteria in Solid Tumors (RECIST) is a standardized method for evaluating treatment response in cancer subjects and is part of the regulatory criteria for new oncology drug approval. RECIST requires a significant amount of trained expert (e.g., radiologist) time. In detail, the annotator manually identifies (e.g., by a radiologist) up to five target lesions and up to ten non-target lesions. The annotator identifies the circumference of each target lesion in each scan where a cross-section of the target lesion is drawn, and records the cross-sectional diameter for each target lesion. A quantitative metric (e.g., sum of the longest diameters) is then determined for all target lesions. The non-target lesions are qualitatively evaluated to indicate whether the non-target lesions were observed during the scan and whether there was an obvious change. Scans can be collected at multiple time points and metrics for target and non-target lesions can be determined for each time point. Changes in the metrics over a period of time can then be used to assess the extent to which a disease is progressing and / or being effectively treated.

[0004] However, RECIST includes several limitations. In other words, the method does not consider the overall "burden" of disease, since RECIST very often only measures a small subset of tumors (e.g., less than 5-10) for each subject. The technique cannot accurately assess disease progression and / or efficacy of treatment for subjects with cancer that has metastasized to include multiple lesions (e.g., 6 or more lesions) given that the size of only up to 5 tumors is tracked. Moreover, there is also inconsistency in the selection of target lesions due to variability in lesion selection, which causes significant intra- and inter-reader variability that leads to differing assessments of tumor burden even within the same subject. For example, different sets of lesions may be identified (e.g., inadvertently) across different time points. Additionally, many tumors often have a heterogeneous appearance on CT and can vary by location, size, and shape. For example, lung lesions may be of the cavitary or calcified type, bone metastases may be (for example) lytic (destroying skeletal tissue) or blastic (abnormal bone growth) morphology, where each lesion type is associated with a different structural and visual appearance, and thus, due to the high variability of lesions, it is difficult to assess the stage of disease and / or each lesion of that lesion type without obtaining a complete reading. This would be advantageous to identify automated techniques that use more comprehensive data sets and more objective techniques to assess tumor growth and / or metastases. Summary of the Invention

[0005] SUMMARY OF THE DISCLOSURE Provided herein are systems and methods for biological object identification and segmentation using medical images.

[0006] CT-RECIST defines at most two target (measured) lesions per organ. Therefore, it may be necessary to learn an organ mask to know the organ borders. There may be two types of lesions. One type of lesion is a target lesion that indicates a specific disease. The other type of lesion is non-target. The target lesion may be more important for diagnostic purposes and the difference between the target lesions may have to be measured, for example, by a radiologist. It may be assumed that if the lesion appears as the only inhomogeneity in an otherwise nearly homogeneous image texture, a machine learning model, for example a neural network, may be significantly more accurate in lesion segmentation. This may be achieved by masking out everything except one organ from the input image, since the inside of the organ may be more homogeneous than the CT scan image as a whole. On the other hand, the CT scan image may be three-dimensional and include a large collection of slices. From a computational point of view, there may not be enough memory to load and process such a large image at full resolution. Therefore, the CT scan images may be passed through the neural network for each organ, which not only allows full resolution images to be used, but also allows the neural network to focus on the range of CT values ​​unique to each organ. Considering the requirements and computational issues of RECIST, the neural network processing system may use an automated approach based on a segmentation model for effective organ segmentation. In certain embodiments, the neural network processing system may search for heterogeneous areas in homogeneous organ masses, segment the organs using a segmentation model, transform the segmented organs to a CT reference frame, and resolve organ intersections. Although the present disclosure describes segmenting a particular object by a particular system in a particular manner, the present disclosure contemplates segmenting any suitable object in any suitable manner and using any suitable policy.

[0007] In certain embodiments, the neural network processing system may access a first scan image from a set of computed tomography (CT) scan images. Each CT scan image may be at a first resolution. The neural network processing system may then generate a first downscaled image of the first scan image by resampling the first scan image to a second resolution. The second resolution may be lower than the first resolution. Downsampling the scan image may result in a technical advantage of computational efficiency, as the downsampled scan image may occupy less memory and may also be loaded and processed faster than a scan image with full resolution. In certain embodiments, the neural network processing system may determine one or more coarse segmentations, each corresponding to one or more organs depicted in the first scan image, by one or more first machine learning models based on the first downscaled image. The neural network processing system may then extract one or more segments of the first scan image, each based on the one or more coarse segmentations. Each extracted segment may be at the first resolution. In certain embodiments, the neural network processing system may determine, based on the one or more extracted segments of the first scan image by one or more second machine learning models, one or more fine segmentations corresponding to the one or more organs depicted in the extracted segments. Using a first machine learning model for the coarse segmentation and a second machine learning model for the fine segmentation may yield one or more technical advantages. One technical advantage may include automating the segmentation process, which may be manual in most existing approaches. Another technical advantage may include assisting radiologists' workflow by implementing automation of tumor / lesion detection and segmentation based on machine learning models.Another technical advantage may include improved accuracy of segmentation, since machine learning models may be significantly more accurate in lesion segmentation when a lesion appears as the only inhomogeneity in an otherwise nearly homogeneous image texture. The neural network processing system may further generate a segmented image of the first scan image based on the one or more fine segmentations. The segmented image may include one or more confirmed segmentations corresponding to one or more organs. The automated segmentation of each organ may yield one or more technical advantages. One technical advantage may include improved tracking of lesions over time, since each automatically segmented organ may provide a focused view that captures lesion progression, which may further aid medical analysis of certain diseases. Another technical advantage may include reducing the amount of nonlinear distortion required and speeding up computations in lesion registration, since the neural network processing system 120 may register lesions on an organ-by-organ basis instead of on a scan-by-scan basis after the organs are segmented.

[0008] One or more of the drawings included herein are in color in accordance with 37 CFR § 1.84. The color drawings are necessary to illustrate the invention. [Brief description of the drawings]

[0009] [Figure 1A] FIG. 1 illustrates an exemplary dialogue system for using, acquiring, and processing medical images using a multi-stage neural network platform. [Figure 1B] FIG. 1 illustrates an example image stack including a set of patches and bounding boxes for detected biological objects. [Diagram 2] FIG. 1 illustrates an exemplary flow chart describing the general process of synthetic CT generation. [Diagram 3]FIG. 1 illustrates an exemplary synthetic CT generated by pairing a chest scan and an abdominal-pelvic scan. [Figure 4A] FIG. 1 illustrates an exemplary CT scan image. [Figure 4B] FIG. 2 illustrates an exemplary segmentation of an organ in a CT scan image. [Figure 4C] FIG. 4B illustrates an enlarged version of the exemplary CT scan image of FIG. 4A. [Figure 4D] FIG. 4C illustrates an enlarged version of an exemplary segmentation of an organ in the CT scan image of FIG. 4B. [Diagram 5] FIG. 1 illustrates an exemplary process for organ segmentation. [Figure 6A] FIG. 1 illustrates an exemplary CT scan image. [Figure 6B] FIG. 2 illustrates an exemplary segmentation of an organ in a CT scan image. [Figure 7] FIG. 1 illustrates an exemplary segmentation process for bronchi. [Figure 8] FIG. 1 illustrates an exemplary 3D segmentation result. [Figure 9] FIG. 1 illustrates an exemplary comparison between coarse segmentation only, coarse plus fine segmentation, and the literature. [Figure 10] FIG. 1 illustrates an exemplary segmentation of the intestine, mesentery, and aorta. [Figure 11] FIG. 1 illustrates an illustration based on deep neural network ensembles of different sizes. [Figure 12A] FIG. 1 illustrates an exemplary effect of ensemble size and P threshold on spleen segmentation. [Figure 12B] FIG. 1 illustrates an exemplary effect of ensemble size and P threshold on pancreas segmentation. [Figure 12C] FIG. 1 illustrates an exemplary effect of ensemble size and P threshold on right adrenal segmentation. [Figure 13A] FIG. 1 illustrates example ground truth labels of an organ. [Figure 13B] FIG. 1 illustrates an example ensemble output. [Figure 13C] FIG. 13B illustrates an expanded version of the example ground truth labels of the organ of FIG. [Figure 13D] FIG. 13C illustrates an expanded version of the example ensemble output of FIG. 13B. [Figure 14] FIG. 1 illustrates an example segmentation performance of a distillation model. [Figure 15] FIG. 1 illustrates an exemplary lesion registration based on organ segmentation. [Figure 16] FIG. 1 illustrates an exemplary method for organ segmentation. [Figure 17] FIG. 1 illustrates an example of a computing system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] 1A illustrates an exemplary dialogue system for using, acquiring, and processing medical images using a multi-stage neural network platform. In this particular example, the dialogue system is specifically configured to locate and segment depictions of tumor biological structures and organs within the medical images.

[0011] One or more imaging systems 101 (e.g., CT machines, MRI machines, and / or X-ray machines) may be used to generate one or more sets of medical images 102 (e.g., CT, MRI, and / or X-ray images). The imaging system 101 may be configured to iteratively adjust focus and / or position as multiple images are collected such that each image in the set of images is associated with a different depth, position, and / or viewpoint relative to other images in the set. The imaging system 201 may include a light source (e.g., a motorized and / or X-ray source), a photodetector (e.g., a camera), lenses, objective lenses, filters, magnets, shim coils (e.g., for correcting magnetic field inhomogeneities), a gradient system (e.g., for localizing magnetic resonance signals), and / or an RF system (e.g., for exciting a sample and detecting the resulting nuclear magnetic resonance signals).

[0012] Each set of images 102 can correspond to an imaging session, a session date, and a subject. The subjects can include human or animal subjects. The subjects can be diagnosed with a particular disease (e.g., cancer) and / or have one or more tumors.

[0013] Each set of images 102 may depict the interior of a corresponding object, and in some cases, each image depicts at least one region of interest of the object (e.g., one or more organs, a thoracic region, an abdominal region, and / or a pelvic region).

[0014] Each image in the set of images 102 may additionally have the same viewing angle such that each depicts a plane that is parallel to other planes depicted in other images in the set. In some cases, each of the set of images may correspond to a different distance along an axis that is non-parallel (e.g., perpendicular) to the plane. For example, the set of images 102 may correspond to a set of horizontal virtual slices that correspond to different positions along the anterior-posterior axis of the subject. The set of images 102 may be pre-processed (e.g., collectively or individually). For example, pre-processing may include normalizing pixel intensity, aligning the images to each other or to another reference point / image, cropping the images to a uniform size, and / or adjusting contrast to distinguish between bright and dark pixels. In some cases, the set of images 102 may be processed to generate a three-dimensional (3D) image structure. The 3D image structure may then be used to generate another set of images that correspond to different angles about the virtual slice.

[0015] Some medical images collected by at least one of the imaging systems 101 may include training images that are to be included in a training data set for training one or more neural networks (e.g., a bounding box detection network and a segmentation network). The training images may be associated with other objects compared to the object for which the trained network is used.

[0016] Each training image may have one or more characteristics of a medical image 102 described herein and may be associated with annotation data indicating whether and / or where the image depicts a tumor and / or organ. To identify this annotation data, images collected by imaging system 101 may be made available to (e.g., transmitted to) annotator device 103.

[0017] The images may be presented at annotator device 103, and an annotator user (e.g., a radiologist) may provide input using (e.g.) a mouse, trackpad, stylus, and / or keyboard indicating (e.g.) whether the image depicts a tumor (or one or more particular types of organs), the number of tumors depicted in the image, the number of tumors that have been annotated (e.g., outlined) by the annotator, and the outer perimeter of each of the one or more tumors and / or one or more particular types of organs.

[0018] The annotator device 103 may translate the input into (for example) label data 104. Each label dataset may be associated with a corresponding image dataset. The label data 104 may indicate whether the image includes a tumor and / or one or more particular types of organs. The label data 104 may further indicate where the tumor and / or organ are located within the image by identifying spatial features (e.g., perimeter and / or area) of the tumor and / or organ. For example, the label data 104 may include a set of coordinates that identify coordinates associated with the perimeter of each of a set of depicted tumors. As another example, the label data 104 may include an indication as to which pixels (or voxels) in the training images correspond to the perimeter and / or area of ​​the depicted tumor.

[0019] Additionally, spatial features may be identified for multiple objects. In some cases, the label data 104 may (but need not) identify spatial features of all tumors, organs, and / or other biological objects depicted in the training images. For example, if the training images depict ten tumors, the label data 104 may identify perimeters for each of the ten tumors, or for only two of the depicted tumors. In such cases, an incomplete subset of objects may (but need not) be selected based on predefined selection criteria. For example, the annotator user may be instructed to mark only depictions of tumors that meet a threshold tumor length and / or a threshold tumor volume and / or are within a region of interest (e.g., within one or more particular organs).

[0020] The label data 104 may further identify a tumor classification, which may represent the type, location, and / or size of the tumor as identified based on input from an annotator. For example, a particular label may indicate that a depicted tumor is within a region of the image 102 as corresponding to a particular organ (e.g., liver). The label data 104 may further include a probability that a particular label actually corresponds to a tumor or organ of interest. The probability value may be calculated based on tumor length, tumor volume, location with interest, and / or the number of annotation users who identify a particular label as corresponding to a tumor or organ. The label data 104 may be used to train one or more neural networks to detect each region that includes a depiction of a tumor or organ for each image in the image scan. The trained neural network may be configured to define each region identified as including a depicted tumor or organ by processing the image scans with individual focuses using image stacks corresponding to each of the respective scans (e.g., to define a particular region for each individual image).

[0021] The neural network processing system 120 may be configured to receive one or more sets of images 102 and corresponding label data 104. Initially, each image of the one or more sets of images may be preprocessed by the preprocessing controller 105. For example, one or more images depicting different regions of an object may be stitched together to generate an aggregate image depicting all of the different regions. In some cases, the aggregate image depicts a "full-body" view of the object. As another example, one or more images may be scaled and / or cropped to a predefined size. In yet another example, one or more images may be aligned to another image included in the set (e.g., using alignment markings in the image, correlation-based techniques, or entropy-based techniques) or to a reference image. In another example, pixel intensities of one or more images may be adjusted via normalization or standardization methods. In some cases, the set of images 102 does not undergo preprocessing techniques.

[0022] The pre-processed images may be made available to a bounding box detection controller 106, which may control and / or perform all of the functions and operations of the bounding box detection network as described herein. The bounding box detection network may be a convolutional neural network, a deconvolutional neural network, or a three-dimensional neural network configured to identify regions (e.g., bounding boxes) within the set of images 102 that contain depictions of tumors. The regions identified by the bounding box detection neural network may include one or more rectangular or hyper-rectangular regions.

[0023] The bounding box detection controller 106 can use the training images and corresponding annotations to train the bounding box detection network to learn a set of detection parameters 107. The detection parameters 107 can include weights between nodes in the convolutional network. A penalty function can be set to introduce a penalty when a portion of the detected bounding box fails to completely contain the tumor depiction and / or when the padding between the additional horizontal and / or vertical points is below a lower threshold and / or exceeds an upper threshold. In some cases, the penalty function is configured to penalize for bounding boxes that are larger or smaller than a predefined zoom range. The penalty function can include a focus loss. Focal loss (defined in Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollar, P., "Focal loss for dense object detection." ICCV 2017, pp. 2980-2988 (2017), which is incorporated by reference in its entirety for all purposes) can be used to address class imbalance as well as to "refocus" training of the detection task towards difficult-to-predict cases resulting from tag-perceptual variability in tumors.

[0024] The bounding box detection network may be trained and / or defined using one or more fixed hyper-parameters. For example, the hyper-parameters may include a learning rate, a number of nodes per layer, a number of layers, etc.

[0025] The bounding box detection network can detect one or more bounding boxes 108 corresponding to potential tumor depictions in each of the images 102. Detecting the bounding boxes can include using an image stack for each image to identify the location of the bounding boxes. For example, if 100 images (numbered consecutively according to imaging depth) were collected during a particular imaging session, an image stack can be defined to include the sixth image, the seventh image, and the eighth image when detecting a bounding box in the seventh image. An image stack can include two or more adjacent images in one or more orientations (e.g., to include the third through eleventh images when detecting a bounding box in the seventh image).

[0026] When determining whether and / or where one or more regions include a tumor and / or organ, features of the image stack are used to provide context information. Features can include three-dimensional features that extend across images in the image stack. For example, if a feature (e.g., a learned feature) is present in similar locations across the entire image stack (e.g., a combination of the top virtual slice, the bottom virtual slice, and the center virtual slice), the bounding box detection network can determine that the image region corresponding to (e.g., including) the feature represents a bounding box for the tumor. As an alternative example, if a feature in the center slice of the image stack is not present in either the top or bottom slices of the image stack, the bounding box detection network can determine that the image region corresponding to the feature corresponds to the image background (i.e., some biological structure other than the tumor) and does not point to a bounding box. In some cases, the bounding box detection network can additionally assign a probability value to each detected bounding box. If the probability value for a bounding box does not exceed a threshold, the bounding box may be discarded as background.

[0027] The bounding box detection network may further process each detected bounding box 108 such that the margins of the bounding box include at least a certain amount of padding (e.g., 10px, 15px, or another suitable amount) from each edge of the region corresponding to the tumor. In some cases, the amount of padding is predefined (e.g., by generating an initial box that intersects the pixels furthest to the left, top, right, and bottom of the detected object depiction, and extending the box using the predefined padding or until an image boundary is encountered). In other cases, varying degrees of padding are added to maintain a uniform bounding box size.

[0028] The bounding box data associated with each bounding box 108 may include a definition of each bounding box (e.g., as two or more corner coordinates, coordinates of one or more edges, etc.) and / or one or more identifiers of the corresponding image or set of images (e.g., an identifier of the image, subject, date captured, etc.).

[0029] It will be appreciated that the location of a bounding box in one image may be related to the location of a bounding box in another image. An image stack may be used to convey this dependency, although other processing may also or alternatively be used. For example, the input to a bounding box detection neural network may include an identification of each of one or more bounding boxes detected from a previously processed image (corresponding to the same imaging session and same subject). As another example, the bounding box output may be post-processed to modify (e.g., translate, resize, remove or add) a bounding box detection corresponding to an image based on the bounding box detection from one or more other adjacent images.

[0030] 1B shows an example image stack depicting a set of bounding boxes for a single biological object 125. The image stack may include at least image 121, image 122, and image 123, with each image in the image stack depicting a different axial perspective of the region of interest. In some cases, the image stack may include additional images not shown in the figure. Each image in the image stack may further include a bounding box that defines the likely location of the biological object 125 in the particular image, such that each bounding box identifies the presence of the same biological object 125 and may be related to corresponding bounding boxes included in other images in the image stack. For example, image 121 includes a bounding box 121A covering at least a portion of image 121, and image 122 includes a bounding box 122A covering at least a corresponding portion of image 122, such that bounding box 121A and bounding box 122A are relevant bounding boxes and include areas predicted to depict a first and second possible location of biological object 125 from a first and second axial perspective, respectively. In other cases, biological object 125 may not be detected in at least a subset of images in the image stack (e.g., one or more of the images), and therefore the subset of images in the image stack may not include a relevant bounding box for biological object 125.

[0031] Further, there may be differences in the exact location (e.g., as represented by a set of coordinates), surface area, and / or shape of the relevant bounding boxes within the image stack. In this example, the surface area of ​​bounding box 121A may be smaller than the surface area of ​​bounding box 122A because it is estimated that most of the biological object 125 is located within image 122. Additionally, the location of each of the relevant bounding boxes may include one or more variations (e.g., in the x-plane, v-plane, or both) that describe the corresponding location of the same biological object 125 from one or more different axial viewpoints of images in the image stack.

[0032] In some cases, in response to identifying a set of relevant bounding boxes for the image stack, a detection area is determined for each of the relevant bounding boxes. For example, image 121 may include detection area 121B that surrounds bounding box 121A. The detection area may be the same size and in the same location for each image in the image stack. In some embodiments, the size and location of the detection area may be determined from the location of the bounding box in a central slice of the image stack (e.g., in this case, image 122). The detection area may be configured to include the entirety of each of the identified bounding boxes, along with additional padding. In some cases, the detection area may be determined by a separate neural network separate from the bounding box detection network.

[0033] Referring again to FIG. 1A, the bounding box data associated with the bounding boxes 108 may be transmitted to a tumor segmentation controller 109, which may control and / or perform all of the functions or operations of the tumor segmentation network as described herein. The tumor segmentation network may be trained using a training dataset of at least the predicted bounding box data determined during training of the bounding box detection network. A set of segmentation parameters 115 (e.g., weights) may be learned during training. In the depicted case, the tumor segmentation network may be (for example) a neural convolutional neural network or a three-dimensional neural network configured to detect and segment tumor depictions. In some cases, the tumor segmentation network does not include a neural network and may instead use (for example) clustering techniques (e.g., K-means techniques), histogram-based techniques, edge detection techniques, region growing techniques, and / or graph partitioning techniques. The tumor segmentation network may be configured to segment the tumor within each of the detected bounding boxes 108.

[0034] For each medical image in the set of images 102, the bounding box 108 includes one or more portions of the image corresponding to the bounding box, or the entire image, along with (for example) an identity of the bounding box (e.g., vertex coordinates and / or edge coordinates) associated with the respective image. In some embodiments, interim processing (not shown) may be performed to generate a cropped set of images (e.g., referred to herein as detection areas) corresponding to only the regions of the images 102 enclosed by the bounding boxes 108. In instances where multiple bounding boxes are defined for a given image, the tumor segmentation network may receive as input each corresponding detection area and process the detection areas separately.

[0035] The detection area may provide a focused view of the target tumor depicted in FIG. 1B. In some cases, the detection area may be of a predefined size. In such cases, the detection area may include another set of regions adjacent to the region corresponding to the bounding box as additional padding to maintain the predefined size of the detection area. In other cases, if the bounding box is larger than a predefined size (e.g., 400 pixels or 200 pixels by 200 pixels), the region corresponding to the bounding box is divided into two or more windows (e.g., of a predefined size and / or not larger than a predefined size), with each window corresponding to a separate detection area. In such cases, the detection area corresponding to a single bounding box may include overlapping portions of the image.

[0036] If the bounding box extends across the entire image stack (as shown in FIG. 1B), a separate detection area may be defined for each image in the image stack. In some embodiments, the processing of the detection area is performed by a bounding box detection network prior to sending the bounding box data associated with the bounding box 108 to the tumor segmentation controller 109.

[0037] The tumor segmentation controller 109 implements a tumor segmentation network configured to further identify and evaluate features (e.g., pixel intensity variation) for each detection area to identify a perimeter, a set of edges, and / or a contour corresponding to the tumor. The features identified by the tumor segmentation network may have similarities to and / or may differ from the features identified by the bounding box detection network. Both networks may be trained to identify regions of an image that correspond to a tumor, but different features may be useful for detecting relatively small structures compared to relatively large structures. In some instances, the tumor segmentation network may learn to detect the location of an object by analyzing (for example) pixel intensity, pixel color, and / or any other suitable image feature. As an example, the tumor segmentation network may identify edges of objects by analyzing an image to detect regions with high contrast, large intensity range, and / or high intensity variation (e.g., as determined by comparing a region-specific metric to a predefined threshold). The tumor segmentation network may include nodes corresponding to different receptive fields (thereby analyzing the representation of different sets of pixels). This allows the network to learn to detect and use at least several different types of features.

[0038] In some cases, the tumor segmentation network may utilize spatial context provided by other images in an image stack to identify a set of edges and / or contours that correspond to the tumor. The image stack may include (for example) three images, with the central image being the image in which the tumor is detected.

[0039] The tumor segmentation network may further generate a two-dimensional (e.g., binary) tumor mask 110 that corresponds to the entire surface area of ​​the tumor within a given detection area using the identified edges and / or contours. The tumor mask 110 may be defined to have a value of zero over pixels that are not identified as depicting part of a tumor. Pixels identified as depicting part of a tumor may be assigned a value of one (e.g., in the case of a binary mask), or another value.

[0040] In some cases, a binary tumor mask 110 is generated for each image in the image stack, with each binary tumor mask 110 corresponding to a different axial perspective of the tumor. In such cases, the post-processing controller 111 can aggregate the set of binary tumor masks 110 to construct a 3D tumor mask 110 that represents the three-dimensional geometry and shape of the tumor.

[0041] In some cases, the neural network processing system 120 may include an organ segmentation controller 111 configured to implement an organ-specific segmentation network. The organ-specific segmentation network may include (for example) a convolutional neural network and / or a 3D neural network. Exemplary convolutional neural networks may include VGG16, U-Net, and / or ResNet18 networks. The organ-specific segmentation network may be configured to analyze a medical image corresponding to a subject and to segment one or more organs depicted in the image. In such cases, each of the one or more organ-specific segmentation networks may be configured (e.g., via parameters learned during training) to segment a particular type of organ. Exemplary organs of interest may be (for example) the liver, or lungs, or kidneys, or pancreas, etc.

[0042] In some cases, the organ-specific segmentation network may be configured to perform a series of convolutions, such as depth-wise and point-wise convolutions, as part of the segmentation process. In such cases, one or more dilations along a particular dimension may be further performed. The particular dimension may be a third dimension, a fourth dimension, etc. In some cases, the tumor segmentation network may also apply one or more filters, such as a replica filter.

[0043] In the depicted case, the organ segmentation controller 111 may control an organ-specific segmentation network configured to detect a particular type of organ. The organ-specific segmentation network may be trained using a training dataset including training images and annotations indicating which portions within each of at least some of the training images depict a particular type of organ. The training dataset may be separate from the training dataset used by the bounding box detection network and the tumor segmentation network. The training dataset may include a plurality of medical images and corresponding annotations (e.g., generated by the annotator device 103) and / or segmentation boundaries for a particular organ of interest. A set of organ segmentation parameters 112 (e.g., weights) may be learned during training. In some cases, the pre-processing controller 105 may send the same set of medical images 102 to both the bounding box detection controller 106 and the organ segmentation controller 111.

[0044] The trained organ-specific segmentation network may be used to process each of the images and / or the set of preprocessed images to detect organs. The images used to detect a particular type of organ may be the same as (or different from) the set of images 102 provided to the bounding box detection controller 106, and thus the images are provided to the organ segmentation controller 111 at the same time. The set of images may be divided into multiple (e.g., overlapping) subsets containing one, two, or three images. For example, the subsets may be defined to have three images per subset and a shift of one image per subset. In some cases, the images may undergo preprocessing to align the images to a 3D image depicting a "full-body" view of the subject.

[0045] Within each image, the organ-specific segmentation network can indicate whether a given image depicts a particular type of organ, and further identifies the perimeter of the organ depiction. The output of the organ-specific segmentation network can include an organ mask 113 that (for example) has a value of zero for pixels that do not depict a particular type of organ, and a non-zero value for pixels that do depict a particular type of organ. In some instances, multiple two-dimensional organ masks can be generated that correspond to different virtual slices (e.g., viewpoints) of the organ of interest. These two-dimensional organ masks can be aggregated to generate a 3D organ mask for each organ.

[0046] The post-processing controller 114 can process the tumor mask 110 and the organ mask 113 individually and / or collectively to generate statistics and / or descriptors. For example, for each tumor, the post-processing controller 114 can identify the tumor volume, and further identify whether the tumor is within any organ (and if so, which type of organ). The post-processing controller 114 can further process (the 2D or 3D tumor mask) to calculate subject-level tumor statistics, such as total tumor volume and / or density for the subject and / or sum of the longest dimension. In some cases, the sum of the longest dimension can be the sum of the longest diameters, and thus the longest diameters are calculated for each tumor and summed to form a grand total of the longest diameters. In some cases, the post-processing controller 114 can identify the percentage of the tumor mass compared to the mass of the corresponding organ of interest, as another example statistic.

[0047] The neural network processing system 120 can output the descriptors and / or statistics to a user device. Additionally, a representation of one or more tumor masks and / or one or more organ masks can be transmitted. For example, an image can be generated that includes a representation of the original image with an overlay that identifies the perimeter of each detected tumor and / or organ of interest. In some cases, the post-processing controller 114 can further process (e.g., or send to another model and / or controller for processing) the subject-level tumor statistics to generate a score for the probability of survival using one or more treatment methods.

[0048] Although the dialogue system depicted in FIG. 1A is concerned with detecting tumors and determining whether various tumors are in different organs, alternative embodiments may be concerned with detecting other types of biological objects. For example, a first network may be trained to detect brain lesions, and other networks may be trained to detect various brain regions, such that it may be determined in which brain region the lesion is located. Thus, alternative embodiments may replace at least the tumor segmentation network with a different segmentation neural network trained to segment other biological structures in medical images.

[0049] Recent image analysis efforts have focused on developing automated algorithms that can aid radiologists' workflow by performing tumor detection and segmentation. Recent methods focus on detecting and / or segmenting RECIST lesions in a single axial CT (computed tomography) section, when the target lesion may indicate a specific disease that may need to be measured over time. These recent efforts may be limited by segmenting tumors only on a single slice or in a single organ (e.g., in the lung) for tumor screening, as opposed to advanced stage subjects that may suffer from a higher and more variable tumor burden.

[0050] CT-RECIST relies on whole-body CT scans in tumor detection / segmentation. However, patients sometimes do not get whole-body coverage, i.e., chest, abdomen, and pelvis, in a single scanning. Individual scans of chest, abdomen, and pelvis may be acquired separately with time intervals of a few seconds to minutes. Therefore, prior to organ segmentation, the neural network processing system 120 may use a synthetic CT module to generate a whole-body CT scan covering chest, abdomen, and pelvis for each visit of the patient. Image series of different anatomical structures may have different contrast, resolution, and some parts may be present in more than one series. For example, an abdominal scan may cover a part of the lower chest region. These overlapping regions of different series should be aligned in physical coordinates, and the synthetic CT module may use this condition to concatenate the optimal series.

[0051] FIG. 2 illustrates an exemplary flow chart describing the general process of synthetic CT generation. In step 205, the synthetic CT module may query the CT scans of each visit of the patient. In step 210, the synthetic CT module may find candidate scans of different anatomical structures with contrast. If there is an existing chest-abdomen-pelvis (i.e., CAP) scan, the synthetic CT module may simply use it. In step 215, the synthetic CT module may copy the original chest-abdomen-pelvis (CAP) as the synthetic CT and perform downsampling if the z-spacing is less than 2.5 mm. If there is no complete chest-abdomen-pelvis scan, the synthetic CT module may check for image pairs that may cover the chest-abdomen-pelvis in step 220. More specifically, the synthetic CT module may look for candidate pairs of series, for example, chest-abdomen and pelvis scans, chest scans and abdomen-pelvis scans, chest scans, abdomen scans, and pelvis scans. If there are no image pairs that could potentially cover the chest-abdomen-pelvis, the synthetic CT module may determine in step 225 that a synthetic chest-abdomen-pelvis (CAP) cannot be generated.

[0052] In step 230, the synthetic CT module may further check for pairs that may cover chest-abdomen. If there are still no pairs that may cover chest-abdomen, the synthetic CT module may determine in step 235 that a synthetic CT cannot be generated. If there are image pairs that may cover chest-abdomen-pelvis or chest-abdomen, the synthetic CT module may check in step 240 whether the acquisition time difference is less than 500 seconds. If the acquisition time difference is not less than 500 seconds, the synthetic CT module may determine in step 245 that a synthetic CT cannot be generated. Once an ideal pair of series is found (the acquisition time interval is less than 500 seconds), the synthetic CT module may align them based on image origin, orientation, and resolution. More specifically, the synthetic CT module may align them in plan origin, spacing, and size. The synthetic CT module may additionally resample the z-spacing to the maximum if the z-spacing is greater than 2.5 mm, otherwise resample to 2.5 mm in step 250. Therefore, cropping and resampling may be required to achieve the same size and resolution. In step 255, the synthetic CT module may generate a synthetic CT and associated metadata. Such metadata may be generated to guide subsequent analysis modules in the pipeline. For example, the metadata may include one or more of slice locations, voxel sizes, or dimensions for concatenating multiple series. In certain embodiments, a quality control (QC) module based on neural networks may also be used to verify that the generated synthetic CT has complete anatomical coverage.

[0053] FIG. 3 illustrates an exemplary synthetic CT generated by a pair of a chest scan and an abdominal-pelvic scan. In FIG. 3, sub-figure 310 depicts the chest scan, and sub-figure 320 depicts the abdominal-pelvic scan. Sub-figure 310 and sub-figure 320 have different resolutions, contrasts, and noise levels. Based on the chest scan and the abdominal-pelvic scan, the synthetic CT module may generate a synthetic CT 330 according to the process described in FIG. 2. In a particular embodiment, the synthetic CT scan may be included in the image 102 described in FIG. 1A. Therefore, the synthetic CT scan may be input to the pre-processing controller 105, after which segmentation may be performed by the neural network processing system 120.

[0054] CT-RECIST defines at most two target (measured) lesions per organ. Therefore, it may be necessary to learn an organ mask to know the borders of the organ. There may be two types of lesions. One type of lesion is a target lesion that indicates a specific disease. The other type of lesion is a non-target. The target lesion may be more important for diagnostic purposes and the difference between the target lesions may have to be measured, for example, by a radiologist. It may be assumed that if the lesion appears as the only inhomogeneity in an otherwise nearly homogeneous image texture, the machine learning model, for example, a neural network, may be significantly more accurate in lesion segmentation. This may be achieved by masking out everything except one organ from the input image, since the inside of the organ may be more homogeneous than the CT scan image as a whole. FIGS. 4A-4D illustrate an exemplary CT scan image and an exemplary segmentation of an organ in the CT scan image. In particular, FIG. 4C illustrates a magnified version of the exemplary CT scan image of FIG. 4A, and FIG. 4D illustrates a magnified version of an exemplary segmentation of an organ in the CT scan image of FIG. 4B.

[0055] FIG. 4A illustrates an exemplary CT scan image 410. The CT scan image 410 may include portions depicting various organs, for example, an organ 420. The organ 420 may include a lesion 430. FIG. 4B illustrates an exemplary segmentation of an organ in a CT scan image. FIG. 4C illustrates an enlarged version of the exemplary CT scan image of FIG. 4A. In FIG. 4B, the organ 420 is segmented by masking out everything from the CT scan image 410. On the other hand, a CT scan image may be three-dimensional and include a large collection of slices. From a computational perspective, there may not be enough memory to load and process such a large-scale image at full resolution. Therefore, a CT image may be passed through a neural network for each organ, which not only allows the full resolution image to be used, but also allows the neural network to focus on a range of CT values ​​unique to each organ. Considering the requirements and computational issues according to RECIST, the neural network processing system 120 may use an automated approach based on a segmentation model for effective organ segmentation. In certain embodiments, the neural network processing system 120 may search for inhomogeneous areas in a homogeneous organ mass, segment the organ using a segmentation model, transform the segmented organ into a CT reference frame, and resolve organ intersections.

[0056] In certain embodiments, the neural network processing system 120 may access a first scan image from a set of computed tomography (CT) scan images. Each CT scan image may be at a first resolution. The neural network processing system 120 may then detect one or more portions of the first scan image that depict one or more of the air or a portion of the CT scanner associated with the set of CT scan images. The neural network processing system 120 may then remove the detected one or more portions from the first scan image. In certain embodiments, the neural network processing system 120 may generate a first downscaled image of the first scan image by resampling the first scan image to a second resolution. The second resolution may be lower than the first resolution. Downsampling the scan image may result in technical advantages of computational efficiency, as the downsampled scan image may occupy less memory and may be loaded and processed faster than the full resolution scan image and the large image at full resolution.

[0057] The neural network processing system 120 may determine one or more coarse segmentations, each corresponding to one or more organs depicted in the first scan image, by one or more first machine learning models based on the first downscaled image. In certain embodiments, at least one of the first machine learning models may be based on a neural network model, at least one of the first machine learning models may be based on an ensemble model of two or more neural network models, and at least one of the first machine learning models may be based on a distillation model of the ensemble model. The neural network processing system 120 may then determine, for each of the one or more coarse segmentations, a region of interest associated with the corresponding organ. The neural network processing system 120 may then extract one or more segments of the first scan image based on the one or more coarse segmentations and their respective regions of interest. Each extracted segment may be at a first resolution. In certain embodiments, the neural network processing system 120 may determine, based on the one or more extracted segments of the first scanned image, by one or more second machine learning models, one or more fine segmentations corresponding to a respective one or more organs depicted in the extracted segments, at least one of the second machine learning models may be based on a neural network model.

[0058] At least one of the second machine learning models may be based on an ensemble model of two or more neural network models. At least one of the second machine learning models may be based on a distillation model of the ensemble models. Using a first machine learning model for coarse segmentation and a second machine learning model for fine segmentation may result in one or more technical advantages. One technical advantage may include automating the segmentation process, which may be manual in most existing approaches. Another technical advantage may include aiding the radiologist's workflow by implementing automation of tumor / lesion detection and segmentation based on the machine learning model. Another technical advantage may include improved accuracy of segmentation, since the machine learning model may be significantly more accurate in lesion segmentation when the lesion appears as the only heterogeneity in an otherwise nearly homogeneous image texture.

[0059] In certain embodiments, the neural network processing system 120 may map one or more fine segmentations to the first scan image as follows: The neural network processing system 120 may identify one or more organ intersections. Each of the one or more organ intersections may include a plurality of voxels. Each of the plurality of voxels may be associated with two or more labels, each of which may point to two or more organs. Each of the two or more labels may be associated with a respective probability score. The neural network processing system 120 may then resolve each of the one or more organ intersections by assigning, to each of the plurality of voxels within the organ intersection, a label that points to one of the two or more organs associated with the highest probability score. After the mapping, the neural network processing system may further generate a segmented image of the first scan image based on the one or more fine segmentations and the mapping.

[0060] The segmented image may include one or more confirmed segmentations corresponding to one or more organs. The automated segmentation of each organ may result in one or more technical advantages. One technical advantage may include improved tracking of lesions over time, since each automatically segmented organ may provide a focused view that captures lesion progression, which may further aid medical analysis of certain diseases. Another technical advantage may include reducing the amount of nonlinear distortion required and speeding up computations in lesion registration, since the neural network processing system 120 may register lesions on an organ-by-organ basis instead of on a scan-by-scan basis after the organs are segmented.

[0061] FIG. 5 illustrates an exemplary process 500 for organ segmentation. In FIG. 5, an original CT scan image 510 may depict a complete torso with multiple organs, for example, organ 515a, organ 515b, organ 515c, organ 515d, organ 515e, and organ 515f. The original CT scan image 510 may be a raw whole-body CT scan. Alternatively, the original CT scan image 510 may be a synthetic CT scan generated by a synthetic CT module. The CT scan image 510 may include portions corresponding to air and portions of a CT scanner. In certain embodiments, the neural network processing system 120 may first remove image portions that are not relevant to the segmentation task. The neural network processing system 120 may then downscale the CT scan image 510 by resampling the complete torso CT scan image 510 to a coarse resolution to generate a downscaled CT scan image 520. By way of example and not by way of limitation, the coarse resolution may be 144 (front-back) x 256 (left-right) x 192 (top-bottom). In a particular embodiment, the neural network processing system 120 may load the downscaled full torso CT scan image 520 into the neural network, which may determine the location of the organs in the downscaled CT scan image 520. The neural network processing system 120 may additionally set a threshold for the segmentation of each organ to define a sharp segmentation boundary, which is then used to define the ROI of the organ. The neural network processing system 120 may then run separate models, each trained to segment the organs based on their respective ROIs in the original full torso CT scan. The segmentation may leave the largest volumetric components for each organ. By way of example and not by way of limitation, in FIG. 5, the coarsely segmented organs may be depicted in image 530. In certain embodiments, the neural network processing system 120 may use a coarse organ segmentation to define an ROI centered on a given organ.

[0062] In certain embodiments, the neural network processing system 120 may resample the corresponding CT scan image fragments (coarse organ segmentation). By way of example and not by way of limitation, the fragments may be resampled to 192x192x192. FIG. 5 shows that the CT scan image fragment 540 corresponding to the organ 515f is resampled to the original resolution. The neural network processing system 120 may then perform fine segmentation for each coarsely segmented organ based on its corresponding full resolution CT image. More specifically, the neural network processing system 120 may focus on each coarsely segmented organ and segment the organ with full resolution to accurately identify the boundaries. In certain embodiments, the neural network processing system 120 may run the refinement model trained to segment the particular organ in the ROI, which may leave the largest volumetric components.

[0063] By way of example and not by way of limitation, FIG. 5 illustrates a fine segmentation 550 of an organ 515f, generated from a resampled image fragment 540. In certain embodiments, the neural network processing system 120 may convert the fine segmentation results (e.g., fine segmentation 550 and fine segmentations 560a-c) to the original CT reference frame at full resolution. The organ segmentations may sometimes intersect with each other or sometimes conflict with each other. Therefore, the neural network processing system 120 may merge these segmentations by decomposing the organ intersections by keeping the labels of the voxels with the highest probability scores.

[0064] In a particular embodiment, the neural network processing system 120 may apply image pre-processing before performing segmentation due to the following issues: One issue may be that very grainy CT images may result in poor segmentation. To remedy such issues, the neural network processing system 120 may smooth all CT scan images before segmentation. Another issue may be that organs may have contrast discontinuities when stitching two images at their boundaries. For example, a complete CT scan is sometimes done partially by several scans on the same day after contrast agent is injected into the patient. There is usually a time interval of several minutes between these scans. Due to acquisition time differences, the uptake of the injected contrast agent may vary for the concatenated scans, which may result in a step in contrast in some organs across the stitching boundary.

[0065] In certain embodiments, the neural network processing system 120 may determine the contrast mismatch between the first and second scans for a given organ on different sides of the apposition line. To reconcile this issue, the neural network processing system 120 may equalize the contrast mismatch based on the CT numbers of the two scans. In certain embodiments, the equalization of the contrast mismatch may be completed by applying a two-pass segmentation for each organ. In particular, in the first pass, an approximate organ segmentation may be used to calculate the contrast step across the organ. In the second pass, the contrast may be equalized by scaling the CT number in one portion of the image, and then the organ segmentation may be repeated. In certain embodiments, the neural network processing system 120 may calculate the average intensity of the organ on both sides of the apposition line and equalize the intensity by using a multiplier. By way of example and not by way of limitation, the neural network processing system may multiply the less visible portion by 1.2. In an alternative embodiment, the neural network processing system may train on images of a synthetic CT scan with apposition instead of the whole torso to solve such issues.

[0066] 6A-6B illustrate an example juxtaposition and corresponding reconciliation of two scans cutting an organ. FIG. 6A illustrates an example contrast step in an organ when stitching two CT scan images together. An upper CT scan image 610 and a lower CT scan image 620 may be stitched together. For illustrative purposes, a dashed line 630 is added to indicate that the full CT scan image 600 was generated by stitching two CT scan images together. As can be seen, the organ 640 may have a juxtaposition problem because the contrast agent was injected closer in time in the CT scan image 610 than in the CT scan image 620. FIG. 6B illustrates an example reconciliation of scan juxtaposition. By applying a two-pass segmentation, the neural network processing system 120 may equalize the contrast mismatch to generate a full stitched CT scan image 650. By way of example and not by way of limitation, the upper CT scan image 660 may be generated by multiplying the corresponding scan values ​​with a multiplier. For example, the organ 640 may appear more homogenous above and below the dashed line 630.

[0067] In certain embodiments, the neural network processing system 120 may perform specific processing for specific organs. By way of example and not by way of limitation, since the skull was not segmented in the training data set, the head may need to be removed prior to bone segmentation. Once the lungs have been segmented, this may be done by cutting the torso a few centimeters (e.g., 5 cm) above the top of the lungs. Similarly, the legs below the pelvis may be removed prior to segmentation by finding the lower pelvis.

[0068] After certain organs have been segmented, the neural network processing system 120 may use computer vision techniques to construct their convex hulls using standard algorithms and identify areas of interest inside the convex hull based on image contrast. By way of example and not by way of limitation, the bronchi may be segmented by finding the air volumes within the convex hull of the mediastinum and lungs and keeping the largest such volumes. In this manner, the neural network processing system 120 may perform organ segmentation using the finest resolution. FIG. 7 illustrates an exemplary segmentation process of the bronchi. The top image 710 of FIG. 7 shows several organs segmented. The middle image 720 shows the identified convex hull. The bottom image 730 shows the identified bronchi. In a similar fashion to the mesentery, once the intestines and other abdominal organs have been segmented, the neural network processing system 120 may take the convex hull of the intestines and identify the mesentery as the volume with a sufficiently high intensity (above fat) to exclude the intestines themselves or any other segmented organs inside the hull.

[0069] FIG. 8 illustrates an example 3D segmentation result. Difficult organs such as the pancreas, gallbladder, etc. may be erroneously segmented into multiple separated components. For such difficult organs, the neural network processing system 120 may perform an additional clean-up step to remove all but the largest volumetric components. In certain embodiments, the segmentation model may be applied to various organs of interest, including reproductive organs (e.g., breast, ovaries, uterus, and prostate), or intestines, mesentery, and large blood vessels, where metastatic cancer activity may occur, and where many adjacent lymph nodes may develop cancer lesions.

[0070] Fig. 9 illustrates an exemplary comparison between coarse segmentation alone and fine segmentation plus coarse segmentation with the literature. The evaluation metric used is the Dice score (% overlap of the discovered segmentation with the true segmentation), where higher is better. As can be seen from Fig. 9, overall the performance of fine segmentation plus coarse segmentation is comparable to the best results in the literature.

[0071] FIG. 10 illustrates an exemplary segmentation of the intestine, mesentery, and aorta. As can be seen, the large and small intestine, mesentery, and aorta may be segmented. In the abdomen, the aorta is important because it is covered with multiple lymph nodes. The mesentery (a thin membrane connected to the intestine) also contains many abdominal lymph nodes, but is extremely difficult to segment directly. Instead, the neural network processing system 120 may segment the mesentery in the same manner as the bronchi. In certain embodiments, the neural network processing system 120 may first calculate a convex hull above the segmented intestine. The neural network processing system 120 may then exclude the intestine and all other previously segmented organs from the hull volume. The neural network processing system 120 may further exclude visceral fat and blood vessels and threshold the remaining volume against a range of CT numbers characteristic of the mesentery.

[0072] If the same neural network model is trained multiple times, starting from random weights each time, the resulting models may not yield the same results. This indicates that the solutions found may not be global optima, but rather various local optima. Each model learns a set of features, but may randomly miss some features. As a result, during training, the confidence score associated with the segmentation may reach close to 100% due to overfitting (e.g., noise fitting). However, during testing, the confidence score for each model may be much lower due to missing features. An ensemble of multiple neural network models may learn more features by having its models end up in different local minima. Therefore, it may be expected that combining these multiple solutions may capture more features than each individual model can provide. In certain embodiments, the neural network processing system 120 may use a deep neural network ensemble (i.e., called an ensemble model of neural networks) for organ segmentation, especially for smaller or less regularly shaped organs.

[0073] By way of example and not by way of limitation, such organs may include the pancreas, which is very small, or the intestine, which is very convoluted. The neural network processing system 120 may train multiple neural networks based on the same training set, each based on different starting weights. In certain embodiments, the deep neural network ensemble may simply average the predictions of the individual models. FIG. 11 illustrates exemplary segmentation results based on deep neural network ensembles with various sizes. The results in FIG. 11 are based on sizes 1 to 21, indicating that 1 to 21 neural network-based segmentation models were ensembled, respectively. As can be seen, the performance is generally better when 5, 11, or 15 neural network-based segmentation models were ensembled. This result may be data dependent.

[0074] 12A-12C illustrate an exemplary effect of ensemble size and P threshold on spleen segmentation, pancreas segmentation, and right adrenal gland segmentation. FIG. 12A illustrates an exemplary effect of ensemble size and P threshold on spleen segmentation. FIG. 12B illustrates an exemplary effect of ensemble size and P threshold on pancreas segmentation. FIG. 12C illustrates an exemplary effect of ensemble size and P threshold on right adrenal gland segmentation. In a particular embodiment, the neural network processing system 120 tested ensembles of sizes 1, 4, 8, 12, 16, and 20, which refer to 1, 4, 8, 12, 16, and 20 neural network-based segmentation models, respectively, are ensembled. As can be seen from FIG. 12A-12C, the performance gain was most significant for the 4-10 model ensembles. On the other hand, the optimal probability thresholds for the binary organ masks were 0.1 for the spleen, 0.15 for the pancreas, and 0.25 for the adrenal glands. In certain embodiments, deep neural network ensembles may be more advantageous when applied to lower performing models.

[0075] In certain embodiments, the neural network processing system 120 may use a distillation model of a deep neural network ensemble for segmentation. The distillation model may enable the system to capture the increased performance of the ensemble in a single model. In certain embodiments, the neural network processing system 120 may generate the distillation model by training a model on the output of the ensemble (i.e., soft labels) rather than on the actual ground truth labels. Figures 13A-13D illustrate an exemplary comparison between the ground truth labels and the ensemble output. In particular, Figure 13C illustrates an enlarged version of the exemplary ground truth labels of the organs of Figure 13A, and Figure 13D illustrates an enlarged version of the exemplary ensemble output of Figure 13B. Figure 13A illustrates the exemplary ground truth labels of the organs. In certain embodiments, each of the neural network model and the ensemble model may be trained based on the ground truth segmentation labels associated with one or more organs. Figure 13B illustrates an exemplary ensemble output. In certain embodiments, the distillation model may be trained based on the output from the ensemble model. As can be seen, the ensemble output (soft labels) may be more comprehensive than the ground truth labels (hard labels). As a result, ensemble distillation may further improve performance due to the soft labels that may incorporate all ensemble features.

[0076] FIG. 14 illustrates an exemplary segmentation performance of the distillation model. The results were evaluated by Dice score based on the segmentation of the right adrenal gland. For the ensemble of neural network-based segmentation models, 10 models were ensembled. As can be seen, the distillation model performs better than a single model, but worse than the full ensemble. By way of example and not by way of limitation, the distillation model can obtain up to 5% improved segmentation accuracy for organs with lower accuracy. Besides the performance gain, the distillation model can also provide some information about the segmentation uncertainty, allowing different versions of the ground truth to be easily incorporated into the solution. However, further improvements can be seen when the ensemble is applied to the distillation model. As can be seen in FIG. 14, the distillation ensemble model is advantageous over all other three models.

[0077] In certain embodiments, during training of the segmentation model, the deep neural network ensemble model, and the distillation model, the neural network processing system 120 may use different techniques to instantiate starting random weights for the neural networks. In addition, the neural network processing system may use different subsets of training images to train different models and select the best performing model to use for organ segmentation.

[0078] In certain embodiments, the neural network processing system 120 may use segmentation to aid in lesion registration to match tumors that grow over time. In certain embodiments, lesion registration may include shifting, scaling, shearing, or skewing images to match different images of the same organ taken over time. To reduce the amount of nonlinear distortion required (which also distorts the lesion volume) and to speed up the calculations, the neural network processing system 120 may register lesions per organ instead of per scan. In certain embodiments, the neural network processing system may first segment all organs of interest using the segmentation model disclosed herein. The neural network processing system may then segment lesions per organ. To avoid registration bias due to possible large lesion changes between time points, the neural network processing system 120 may mask out (i.e., ignore during registration calculations) the diseased tissue for both time points. In certain embodiments, the neural network processing system 120 may then register the remaining healthy tissue of the organ using affine registration and a coarse (e.g., 5 cm support grid) non-linear registration. The neural network processing system 120 may use the discovered mapping to register the lesions in the organ between the two time points. In certain embodiments, the neural network processing system 120 may repeat for all time points using screening as the reference time point.

[0079] In certain embodiments, the first scan image may be associated with a first time point. A first fine segmentation of the one or more fine segmentations corresponding to a first organ of the one or more organs may depict one or more first lesions in the first organ. The neural network processing system 120 may access a second scan image from the set of CT scan images. The second scan image may be associated with a second time point. The second scan image may include a portion depicting the first organ. In certain embodiments, the neural network processing system 120 may determine a second fine segmentation corresponding to the first organ depicted in the second scan image. The second fine segmentation may depict one or more second lesions in the first organ. The neural network processing system 120 may further determine a lesion registration based on a comparison between the one or more first lesions and the one or more second lesions.

[0080] FIG. 15 illustrates an exemplary lesion registration based on organ segmentation. The first column includes segmented images of organs from CT scan images from the initial screening. For example, image 1510a corresponds to the first organ, image 1520a corresponds to the second organ, image 1530a corresponds to the third organ, and image 1540a corresponds to the fourth organ. The second column includes segmented images of organs from CT scan images at 6 weeks for lesion registration. The third column includes segmented images of organs from CT scan images at 12 weeks for lesion registration. As can be seen from the comparison between image 1510b and image 1510a, two of the tumors from the initial screening (i.e., tumor 1512 and tumor 1514) may have disappeared, where there is also a new tumor 1515 that has developed. Tumor 1518 may have become smaller. As can be seen from the comparison between image 1510c and image 1510b, tumor 1515 may have disappeared. As can be seen from a comparison between image 1520c and images 1520b and 1520a, tumor 1522 may have continuously shrunk from the initial screening to week 6 and week 12. As can be seen from a comparison between image 1530c and images 1530b and 1530a, tumor 1532 may have disappeared at week 6 and week 12. As can be seen from a comparison between image 1540b and images 1540a, tumor 1542 may have shrunk from the initial screening to week 6. However, comparison between image 1540c and image 1540b, tumor 1542 may have grown again.

[0081] 16 illustrates an exemplary method 1600 for organ segmentation. The method may begin at step 1610, where the neural network processing system 120 may access a first scan image from a set of computed tomography (CT) scan images, each CT scan image being at a first resolution. At step 1620, the neural network processing system 120 may generate a first downscaled image of the first scan image by resampling the first scan image to a second resolution, where the second resolution is lower than the first resolution. In step 1630, the neural network processing system 120 may determine one or more coarse segmentations corresponding to one or more organs depicted in the first scan image by one or more first machine learning models based on the first downscaled image, where at least one of the first machine learning models is based on a neural network model, at least one of the first machine learning models is based on an ensemble model of two or more neural network models, and at least one of the first machine learning models is based on a distillation model of the ensemble model.

[0082] In step 1640, the neural network processing system 120 may determine, for each of the one or more coarse segmentations, a region of interest associated with a corresponding organ. In step 1650, the neural network processing system 120 may extract one or more segments of the first scanned image based on the one or more coarse segmentations and their respective regions of interest, each extracted segment being at a first resolution. In step 1660, the neural network processing system 120 may determine, by one or more second machine learning models based on the one or more extracted segments of the first scanned image, one or more fine segmentations corresponding to each one or more organs depicted in the extracted segments, at least one of the second machine learning models is based on a neural network model, at least one of the second machine learning models is based on an ensemble model of two or more neural network models, and at least one of the second machine learning models is based on a distillation model of the ensemble model.

[0083] At step 1670, the neural network processing system 120 may map one or more fine segmentations to the first scan image, the mapping including identifying one or more organ intersections, each of the one or more organ intersections including a plurality of voxels, each of the plurality of voxels associated with two or more labels that respectively refer to two or more organs, each of the two or more labels being associated with a respective probability score, and decomposing each of the one or more organ intersections by assigning to each of the plurality of voxels within the organ intersection a label that refers to one of the two or more organs associated with the highest probability score. At step 1680, the neural network processing system 120 may generate a segmented image of the first scan image based on the one or more fine segmentations and the mapping, the segmented image including one or more confirmed segmentations corresponding to the one or more organs.

[0084] Certain embodiments may repeat one or more steps of the method of FIG. 16 where appropriate. Although the present disclosure describes and illustrates certain steps of the method of FIG. 16 as occurring in a particular order, the present disclosure contemplates any suitable steps of the method of FIG. 16 occurring in any suitable order. Moreover, while the present disclosure describes and illustrates an exemplary method for organ segmentation that includes certain steps of the method of FIG. 16, the present disclosure contemplates any suitable method for organ segmentation that includes any suitable steps, which may include all, some, or none of the steps of the method of FIG. 16, where appropriate. Moreover, while the present disclosure describes and illustrates certain components, devices, or systems that perform certain steps of the method of FIG. 16, the present disclosure contemplates any suitable combination of any suitable components, devices, or systems that perform any suitable steps of the method of FIG. 16.

[0085] FIG. 17 illustrates an exemplary computer system 1700. In certain embodiments, one or more computer systems 1700 perform one or more steps of one or more methods described or illustrated herein. In certain embodiments, one or more computer systems 1700 provide functionality described or illustrated herein. In certain embodiments, software running on one or more computer systems 1700 performs one or more steps of one or more methods described or illustrated herein or provides functionality described or illustrated herein. Certain embodiments include one or more portions of one or more computer systems 1700. In this specification, reference to a computer system may encompass a computing device, and vice versa, where appropriate. Moreover, reference to a computer system may encompass one or more computer systems, where appropriate.

[0086] The present disclosure contemplates any suitable number of computer systems 1700. The present disclosure contemplates computer system 1700 taking any suitable physical form. By way of example and not limitation, computer system 1700 may be an embedded computer system, a system on a chip (SOC), a single board computer system (SBC) (e.g., a computer on module (COM) or system on module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a mesh of computer systems, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, or a combination of two or more of these. Where appropriate, computer system 1700 may be monolithic or distributed, spanning multiple locations, spanning multiple machines, spanning multiple data centers, or in a cloud that may include one or more cloud components in one or more networks, including one or more computer systems 1700. Where appropriate, one or more computer systems 1700 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. By way of example, and not by way of limitation, the one or more computer systems 1700 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch mode. The one or more computer systems 1700 may, where appropriate, perform one or more steps of one or more methods described or illustrated herein at different times or in different locations.

[0087] In a particular embodiment, computer system 1700 includes a processor 1702, memory 1704, storage 1706, an input / output (I / O) interface 1708, a communications interface 1710, and a bus 1712. Although this disclosure describes and illustrates a particular computer system having a particular number of particular components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of any suitable components in any suitable arrangement.

[0088] In particular embodiments, the processor 1702 includes hardware for executing instructions, such as those that make up a computer program. By way of example and not by way of limitation, to execute an instruction, the processor 1702 may retrieve (or fetch) an instruction from an internal register, an internal cache, memory 1704, or storage 1706, decode and execute the instruction, and then write one or more results to an internal register, an internal cache, memory 1704, or storage 1706. In particular embodiments, the processor 1702 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates the processor 1702 including any suitable number of any suitable internal caches, where appropriate. By way of example and not by way of limitation, the processor 1702 may include one or more instruction caches, one or more data caches, and one or more translation lookaside buffers (TLBs). Instructions in an instruction cache may be copies of instructions in memory 1704 or storage 1706, and the instruction cache may speed up retrieval of those instructions by the processor 1702.

[0089] The data in the data cache may be a copy of data in memory 1704 or storage 1706 on which instructions executing in the processor 1702 are to operate, results of previous instructions executed in the processor 1702 for access by subsequent instructions executing in the processor 1702 or for writing to memory 1704 or storage 1706, or other suitable data. The data cache may speed up read or write operations by the processor 1702. The TLB may speed up virtual address translation for the processor 1702. In particular embodiments, the processor 1702 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates a processor 1702 including any suitable number of any suitable internal registers, where appropriate. Where appropriate, the processor 1702 may include one or more arithmetic logic units (ALUs), be a multi-core processor, or include one or more processors 1702. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

[0090] In a particular embodiment, the memory 1704 includes a main memory for storing instructions to be executed by the processor 1702 or data on which the processor 1702 is to operate. By way of example and not by way of limitation, the computer system 1700 may load instructions from the storage 1706 or another source (such as, for example, another computer system 1700) into the memory 1704. The processor 1702 may then load the instructions from the memory 1704 into an internal register or cache. To execute an instruction, the processor 1702 may retrieve the instruction from the internal register or cache and decode the instruction. During or after the execution of the instruction, the processor 1702 may write one or more results (which may be intermediate or final results) to an internal register or cache. The processor 1702 may then write one or more of those results to the memory 1704.

[0091] In particular embodiments, the processor 1702 executes only instructions in one or more internal registers or internal caches (as opposed to storage 1706 or elsewhere) or in memory 1704, and operates only on data in one or more internal registers or internal caches (as opposed to storage 1706 or elsewhere) or in memory 1704. One or more memory buses (each of which may include an address bus and a data bus) may couple the processor 1702 to the memory 1704. The bus 1712 may include one or more memory buses, as described below. In particular embodiments, one or more memory management units (MMUs) are between the processor 1702 and the memory 1704 and facilitate accesses to the memory 1704 requested by the processor 1702. In particular embodiments, the memory 1704 includes random access memory (RAM). This RAM may be volatile memory, where appropriate. This RAM may be dynamic RAM (DRAM) or static RAM (SRAM), where appropriate. Moreover, this RAM may be single-ported or multi-ported RAM, where appropriate. This disclosure contemplates any suitable RAM. The memory 1704 may include one or more memories 1704, where appropriate. Although this disclosure describes and illustrates particular memory, this disclosure contemplates any suitable memory.

[0092] In particular embodiments, storage 1706 includes mass storage for data or instructions. By way of example and not limitation, storage 1706 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Storage 1706 may include removable or non-removable (or fixed) media, where appropriate. Storage 1706 may be internal or external to computer system 1700, where appropriate. In particular embodiments, storage 1706 is a non-volatile solid-state memory. In particular embodiments, storage 1706 includes read-only memory (ROM). Where appropriate, this ROM may be a mask program ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these. This disclosure contemplates mass storage 1706 taking any suitable physical form. Storage 1706 may include one or more storage control units that facilitate communications between processor 1702 and storage 1706, as appropriate. Storage 1706 may include one or more storages 1706, as appropriate. Although this disclosure describes and illustrates particular storage, this disclosure contemplates any suitable storage.

[0093] In particular embodiments, I / O interface 1708 includes hardware, software, or both, providing one or more interfaces for communication between computer system 1700 and one or more I / O devices. Computer system 1700 may include one or more of these I / O devices, as appropriate. One or more of these I / O devices may enable communication between a person and computer system 1700. By way of example and not by way of limitation, the I / O devices may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, stylus, tablet, touch screen, trackball, video camera, another suitable I / O device, or a combination of two or more of these. The I / O devices may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interfaces 1708 therefor. As appropriate, I / O interface 1708 may include one or more device or software drivers that enable processor 1702 to drive one or more of these I / O devices. I / O interface 1708 may include one or more I / O interfaces 1708, where appropriate. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0094] In particular embodiments, the communication interface 1710 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication, etc.) between the computer system 1700 and one or more other computer systems 1700 or one or more networks. By way of example and not by way of limitation, the communication interface 1710 may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network, or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI network. This disclosure contemplates any suitable network and any suitable communication interface 1710 therefor. By way of example and not by way of limitation, the computer system 1700 may communicate with one or more portions of an ad-hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or the Internet, or a combination of two or more of these.

[0095] One or more portions of one or more of these networks may be wired or wireless. By way of example, computer system 1700 may communicate with a wireless PAN (WPAN) (e.g., a BLUETOOTH WPAN), a WI-FI network, a WI-MAX network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network), or other suitable wireless networks or a combination of two or more of these. Computer system 1700 may include any suitable communication interface 1710 for any of these networks, where appropriate. Communication interface 1710 may include one or more communication interfaces 1710, where appropriate. Although this disclosure describes and illustrates a particular communication interface, this disclosure contemplates any suitable communication interface.

[0096] In particular embodiments, the bus 1712 includes hardware, software, or both that couples together the components of the computer system 1700. By way of example, and not by way of limitation, the bus 1712 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HYPERTRANSPORT (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus or a combination of two or more of these. The bus 1712 may include one or more buses 1712, where appropriate. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0097] As used herein, one or more computer-readable non-transitory storage media may include, where appropriate, one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, field programmable gate arrays (FPGAs) or application specific ICs (ASICs)), hard disk drives (HDDs), hybrid hard drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy diskettes, floppy disk drives (FDDs), magnetic tapes, solid state drives (SSDs), RAM drives, SECURE DIGITAL cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. A computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate.

[0098] In this specification, "or" is inclusive and not exclusive, unless otherwise clearly indicated or indicated by the context. Thus, in this specification, "A or / or B" means "A, B, or / and both" unless otherwise clearly indicated or indicated by the context. Moreover, "and" is both jointly and severally, unless otherwise clearly indicated or indicated by the context. Thus, in this specification, "A and / and B" means "A and / and B, jointly or severally," unless otherwise clearly indicated or indicated by the context.

[0099] The scope of the present disclosure encompasses all changes, substitutions, variations, modifications, and alterations to the exemplary embodiments described or illustrated herein that would be understood by a person skilled in the art. The scope of the present disclosure is not limited to the exemplary embodiments described or illustrated herein. Moreover, although the present disclosure has described and illustrated each embodiment herein as including certain components, elements, features, functions, operations, or steps, any of these embodiments may include any combination or permutation of any of the components, elements, features, functions, operations, or steps described or illustrated anywhere herein that would be understood by a person skilled in the art. Moreover, references in the appended claims to an apparatus or system, or an apparatus or system component, being adapted, arranged, capable, configured, enabled, operable, or operative to perform a particular function encompass that apparatus, system, component, or particular function thereof, so long as the apparatus, system, or component is so adapted, arranged, capable, configured, enabled, operable, or operative, regardless of whether the apparatus, system, component, or particular function thereof is activated, turned on, or unlocked. Additionally, although the present disclosure describes or illustrates particular embodiments as providing certain advantages, a particular embodiment may provide none of these advantages, some of these advantages, or all of these advantages.

[0100] Embodiment Among the embodiments provided are the following: 1. by one or more computing systems, Accessing a first scan image from a set of computed tomography (CT) scan images, each CT scan image being at a first resolution; generating a first downscaled image of the first scanned image by resampling the first scanned image to a second resolution, the second resolution being lower than the first resolution; determining, by one or more first machine learning models based on the first downscaled image, one or more coarse segmentations corresponding to one or more organs depicted in the first scanned image, respectively; Extracting one or more segments of the first scanned image based on the one or more coarse segmentations, respectively, where each extracted segment is at a first resolution; determining, by one or more second machine learning models based on the one or more extracted segments of the first scanned image, one or more fine segmentations corresponding to a respective one or more organs depicted in the extracted segments; generating a segmented image of the first scanned image based on the one or more fine segmentations, the segmented image including one or more identified segmentations corresponding to the one or more organs; The method includes: 2. Detecting one or more portions of the first scan image depicting one or more of air associated with the CT scan image or a portion of a CT scanner; removing the detected portion or portions from the first scanned image; 2. The method of embodiment 1, further comprising: 3. determining, for each of the one or more coarse segmentations, a region of interest associated with a corresponding organ, wherein extracting one or more segments of the first scan image further determines a region of interest based on the respective region of interest associated with the one or more coarse segmentations. 3. The method of any of embodiments 1 to 2, further comprising: 4. Mapping one or more fine segmentations onto the first scanned image, where generating a segmented image of the first scanned image further comprises mapping the one or more fine segmentations based on the mapping. 4. The method of any of embodiments 1 to 3, further comprising: 5. Mapping one or more fine segmentations onto the first scanned image includes: identifying one or more organ intersections, each of the one or more organ intersections including a plurality of voxels, each of the plurality of voxels associated with two or more labels indicating two or more organs, respectively, each of the two or more labels associated with a respective probability score; decomposing each of the one or more organ intersections by assigning to each of a plurality of voxels within the organ intersection a label indicating one of the two or more organs associated with a highest probability score; 5. The method of any of embodiments 1 to 4, comprising: 6. A first scanned image depicts a first portion of a torso, a segmented image of the first scanned image includes a first confirmed segmentation of the one or more confirmed segmentations corresponding to a first organ of the one or more organs, and the first scanned image is associated with a line of apposition, and the method includes: accessing a second scan image from the set of CT scan images, the second scan image depicting a second portion of the torso, the second scan image being associated with the line of apposition; generating a segmented image of the second scan image, the segmented image of the second scan image including a second confirmed segmentation corresponding to the first organ; determining a contrast mismatch between a first confirmed segmentation and a second confirmed segmentation, the first confirmed segmentation and the second confirmed segmentation being on different sides of a line of juxtaposition; equalizing the contrast mismatch based on a CT number associated with the first identified segmentation and a CT number associated with the second identified segmentation; stitching the segmented image of the first scanned image and the segmented image of the second scanned image based on the equalized contrast mismatch along the juxtaposition line; 6. The method of any of the preceding embodiments, further comprising: 7. A method comprising: a first organ of the one or more organs comprising: Identifying a convex hull associated with the first organ in the segmented image of the first scan; identifying one or more air volumes within the convex hull; determining a first air passage based on a largest air volume of the one or more air volumes; 7. The method of any of the preceding embodiments, further comprising: 8. Accessing one or more second scan images of the set of computed tomography (CT) scan images; generating one or more segmented images of the one or more second scan images, respectively, where each of the one or more segmented images of the one or more second scan images includes one or more identified segmentations corresponding to one or more organs; generating a three-dimensional (3D) segmented image based on the segmented image of the first scan image and the one or more segmented images of the respective one or more second scan images, the 3D segmented image including one or more 3D segmentations corresponding to one or more organs; 8. The method of any of the preceding embodiments, further comprising: 9. The method of any of embodiments 1 to 8, wherein at least one of the first or second machine learning models is based on a neural network model, at least one of the first or second machine learning models is based on an ensemble model of two or more neural network models, and at least one of the first or second machine learning models is based on a distillation model of the ensemble model. 10. The method of any of embodiments 1 to 9, wherein each of the neural network model and the ensemble model is trained based on ground truth segmentation labels associated with one or more organs, and the distillation model is trained based on output from the ensemble model. 11. A first scanned image is associated with a first time point, and a first fine segmentation of the one or more fine segmentations corresponding to a first organ of the one or more organs depicts one or more first lesions in the first organ, and the method includes: accessing a second scan image from the set of CT scan images, the second scan image being associated with a second time point, the second scan image including a portion depicting the first organ; determining a second fine segmentation corresponding to the first organ depicted in the second scan image, the second fine segmentation depicting one or more second lesions within the first organ; determining a lesion registration based on a comparison between the one or more first lesions and the one or more second lesions; 11. The method of any of the preceding embodiments, further comprising: 12. One or more computer-readable non-transitory storage media embodying software that, when executed by one or more processors, is operable to perform the steps of any of embodiments 1 to 11. 13. A system comprising one or more processors and a non-transitory memory coupled to the processors, the non-transitory memory including instructions executable by the one or more processors, the system being operable to perform any of the steps of embodiments 1 to 11 when the one or more processors execute the instructions.

Claims

1. by one or more computing systems, Accessing a first scan image from a set of computed tomography (CT) scan images, each CT scan image being at a first resolution; detecting one or more portions of the first scan image depicting one or more of air or a portion of a CT scanner associated with the set of CT scan images; removing the one or more detected portions from the first scanned image; and generating a first downscaled image of the first scanned image by resampling the first scanned image to a second resolution, the second resolution being lower than the first resolution; determining, based on the first downscaled image, by one or more first machine learning models, one or more coarse segmentations respectively corresponding to one or more organs depicted in the first scan image, where at least one of the first machine learning models is based on a neural network model, where at least one of the first machine learning models is based on an ensemble model of two or more neural network models, and where at least one of the first machine learning models is based on a distillation model of the ensemble model; determining, for each of the one or more coarse segmentations, a region of interest associated with a corresponding organ; extracting one or more segments of the first scanned image based on the one or more coarse segmentations and their respective regions of interest, each extracted segment being at the first resolution; determining, by one or more second machine learning models based on the one or more extracted segments of the first scanned image, one or more fine segmentations corresponding to each of the one or more organs depicted in the extracted segments, wherein at least one of the second machine learning models is based on a neural network model, at least one of the second machine learning models is based on an ensemble model of two or more neural network models, and at least one of the second machine learning models is based on a distillation model of the ensemble model; Mapping the one or more fine segmentations onto the first scanned image, the mapping comprising: identifying one or more organ intersections, each of the one or more organ intersections including a plurality of voxels, each of the plurality of voxels associated with two or more labels indicating two or more organs, respectively, each of the two or more labels associated with a respective probability score; decomposing each of the one or more organ intersections by assigning to each of the plurality of voxels within the organ intersection a label indicating one of the two or more organs associated with a highest probability score; Mapping the one or more fine segmentations, comprising: generating a segmented image of the first scan image based on the one or more fine segmentations and the mapping, the segmented image including one or more confirmed segmentations corresponding to the one or more organs; The method includes:

2. by one or more computing systems, Accessing a first scan image from a set of computed tomography (CT) scan images, each CT scan image being at a first resolution; generating a first downscaled image of the first scanned image by resampling the first scanned image to a second resolution, the second resolution being lower than the first resolution; determining, by one or more first machine learning models based on the first downscaled image, one or more coarse segmentations respectively corresponding to one or more organs depicted in the first scan image; extracting one or more segments of the first scanned image based on the one or more coarse segmentations, each extracted segment being at the first resolution; determining, by one or more second machine learning models based on the one or more extracted segments of the first scan image, one or more fine segmentations corresponding to each of the one or more organs depicted in the extracted segments; generating a segmented image of the first scan image based on the one or more fine segmentations, the segmented image including one or more confirmed segmentations corresponding to the one or more organs; The method includes:

3. detecting one or more portions of the first scan image depicting one or more of air associated with the CT scan image or a portion of a CT scanner; removing the detected one or more portions from the first scanned image; and The method of claim 2 , further comprising:

4. determining, for each of the one or more coarse segmentations, a region of interest associated with a corresponding organ, wherein extracting the one or more segments of the first scan image further includes determining a region of interest based on the respective region of interest associated with the one or more coarse segmentations. The method of claim 2 , further comprising:

5. Mapping the one or more fine segmentations onto the first scanned image, wherein generating the segmented image of the first scanned image further comprises mapping the one or more fine segmentations based on the mapping. The method of claim 2 , further comprising:

6. Mapping the one or more fine segmentations onto the first scanned image includes: identifying one or more organ intersections, each of the one or more organ intersections including a plurality of voxels, each of the plurality of voxels associated with two or more labels indicating two or more organs, respectively, each of the two or more labels associated with a respective probability score; decomposing each of the one or more organ intersections by assigning to each of the plurality of voxels within the organ intersection a label indicating one of the two or more organs associated with a highest probability score; The method of claim 5 , comprising:

7. The first scanned image depicts a first portion of a torso, the segmented image of the first scanned image includes a first confirmed segmentation of the one or more confirmed segmentations corresponding to a first organ of the one or more organs, and the first scanned image is associated with a line of apposition, the method comprising: accessing a second scan image from the set of CT scan images, the second scan image depicting a second portion of the torso, the second scan image being associated with the line of apposition; generating a segmented image of the second scan image, the segmented image of the second scan image including a second confirmed segmentation corresponding to the first organ; determining a contrast mismatch between the first confirmed segmentation and the second confirmed segmentation, the first confirmed segmentation and the second confirmed segmentation being on different sides of the line of juxtaposition; equalizing the contrast mismatch based on a CT number associated with the first identified segmentation and a CT number associated with the second identified segmentation; stitching the segmented image of the first scan image and the segmented image of the second scan image along the juxtaposition line based on the equalized contrast mismatch; The method of claim 2 , further comprising:

8. a first organ of the one or more organs comprises air, the first organ comprises a first air passageway, and the method further comprises: identifying a convex hull associated with the first organ in the segmented image of the first scan; identifying one or more air volumes within the convex hull; determining the first air passage based on a largest air volume of the one or more air volumes; The method of claim 2 , further comprising:

9. accessing one or more second scan images of the set of computed tomography (CT) scan images; generating one or more segmented images of the one or more second scan images, respectively, where each of the one or more segmented images of the one or more second scan images includes one or more confirmed segmentations corresponding to the one or more organs; generating a three-dimensional (3D) segmented image based on the segmented image of the first scan image and the one or more segmented images of each of the one or more second scan images, the 3D segmented image including one or more 3D segmentations corresponding to the one or more organs; The method of claim 2 , further comprising:

10. 3. The method of claim 2, wherein at least one of the first or second machine learning models is based on a neural network model, wherein at least one of the first or second machine learning models is based on an ensemble model of two or more neural network models, and wherein at least one of the first or second machine learning models is based on a distillation model of the ensemble model.

11. 11. The method of claim 10, wherein each of the neural network model and the ensemble model is trained based on ground truth segmentation labels associated with the one or more organs, and the distillation model is trained based on output from the ensemble model.

12. The first scanned image is associated with a first time point, and a first fine segmentation of the one or more fine segmentations corresponding to a first organ of the one or more organs depicts one or more first lesions in the first organ, and the method further comprises: accessing a second scan image from the set of CT scan images, the second scan image being associated with a second time point, the second scan image including a portion depicting the first organ; determining a second fine segmentation corresponding to the first organ depicted in the second scan image, the second fine segmentation depicting one or more second lesions within the first organ; determining a lesion registration based on a comparison between the one or more first lesions and the one or more second lesions; The method of claim 2 , further comprising:

13. One or more computer-readable non-transitory storage media embodying software that, when executed, Accessing a first scan image from a set of computed tomography (CT) scan images, each CT scan image being at a first resolution; generating a first downscaled image of the first scanned image by resampling the first scanned image to a second resolution, the second resolution being lower than the first resolution; determining, by one or more first machine learning models based on the first downscaled image, one or more coarse segmentations respectively corresponding to one or more organs depicted in the first scan image; extracting one or more segments of the first scanned image based on the one or more coarse segmentations, each extracted segment being at the first resolution; determining, by one or more second machine learning models based on the one or more extracted segments of the first scan image, one or more fine segmentations corresponding to each of the one or more organs depicted in the extracted segments; generating a segmented image of the first scan image based on the one or more fine segmentations, the segmented image including one or more confirmed segmentations corresponding to the one or more organs; One or more computer-readable non-transitory storage media operable to perform the steps of:

14. The software, when executed, detecting one or more portions of the first scan image depicting one or more of air or a portion of a CT scanner associated with the set of CT scan images; removing the one or more detected portions from the first scanned image; and The medium of claim 13 , further operable to:

15. The software, when executed, determining, for each of the one or more coarse segmentations, a region of interest associated with a corresponding organ, wherein extracting the one or more segments of the first scan image further includes determining a region of interest based on the respective region of interest associated with the one or more coarse segmentations. The medium of claim 13 , further operable to:

16. The software, when executed, Mapping the one or more fine segmentations onto the first scanned image, wherein generating the segmented image of the first scanned image further comprises mapping the one or more fine segmentations based on the mapping. The medium of claim 13 , further operable to:

17. 1. A system comprising one or more processors and a non-transitory memory coupled to the processors, the non-transitory memory including instructions executable by the processors, the processor, when executing the instructions, Accessing a first scan image from a set of computed tomography (CT) scan images, each CT scan image being at a first resolution; generating a first downscaled image of the first scanned image by resampling the first scanned image to a second resolution, the second resolution being lower than the first resolution; determining, by one or more first machine learning models based on the first downscaled image, one or more coarse segmentations respectively corresponding to one or more organs depicted in the first scan image; extracting one or more segments of the first scanned image based on the one or more coarse segmentations, each extracted segment being at the first resolution; determining, by one or more second machine learning models based on the one or more extracted segments of the first scan image, one or more fine segmentations corresponding to each of the one or more organs depicted in the extracted segments; generating a segmented image of the first scan image based on the one or more fine segmentations, the segmented image including one or more confirmed segmentations corresponding to the one or more organs; The system is operable to:

18. The processor, when executing the instructions, detecting one or more portions of the first scan image depicting one or more of air or a portion of a CT scanner associated with the set of CT scan images; removing the detected one or more portions from the first scanned image; and 20. The system of claim 17, further operable to:

19. The processor, when executing the instructions, determining, for each of the one or more coarse segmentations, a region of interest associated with a corresponding organ, wherein extracting the one or more segments of the first scan image further includes determining a region of interest based on the respective region of interest associated with the one or more coarse segmentations.

20. The system of claim 17, further operable to:

20. The processor, when executing the instructions, Mapping the one or more fine segmentations onto the first scanned image, wherein generating the segmented image of the first scanned image further comprises mapping the one or more fine segmentations based on the mapping.

20. The system of claim 17, further operable to:

Citation Information

Patent Citations

  • Systems and methods for fast neural network-based image segmentation and radiopharmaceutical uptake determination

    JP2021509721A

  • Radiation imaging apparatus

    WO2009022408A1