Automatic detection of tumors based on image processing

Abstract

To provide a method and system for processing an image to estimate whether at least a portion of a tumor is represented in the image.SOLUTION: The method includes accessing an image of at least a portion of a biological structure of a particular subject, processing the image using a segmentation algorithm to extract a plurality of image objects depicted in the image, determining one or more structural characteristics associated with an image object of the plurality of image objects, processing the one or more structural metrics using a trained machine learning model to generate a classification metric corresponding to an estimation of whether the image object corresponds to a lesion or tumor associated with the biological structure, and outputting the classification metric for the particular subject.SELECTED DRAWING: Figure 21

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Patent Application No. 62 / 989,261, filed March 13, 2020, which is incorporated herein by reference in its entirety for all purposes.

[0002] Field The methods and systems disclosed herein generally relate to methods and systems for processing images to estimate whether at least a portion of a tumor is represented in the image. [Background technology]

[0003] background Recent advances in medical imaging technology have resulted in image output with much improved image quality. The higher quality images allow medical professionals to more accurately identify symptoms associated with subjects and diagnose various types of diseases. Furthermore, image processing systems are beginning to use machine learning techniques to gain better insight into these images. For example, traditional approaches include using trained machine learning models to determine tissue type based on features associated with the image.

[0004] While using trained machine learning models can help segment images to identify image objects corresponding to tissues and / or tumors, training these machine learning models can be difficult. This is because training a machine learning model typically involves the time-consuming process of manually labeling individual images. Furthermore, manual labeling of images may require numerous experts to ensure accuracy. Training a machine learning model can also be difficult if a particular set of features in the training images can confound the labeling process. For example, a training image showing a large, unknown mass may obscure other features in the training image that may be relevant to diagnosing a disease. This can subsequently hinder the training process of the machine learning model by increasing the time required to perform the analysis or increasing the error rate of the machine learning classification. Summary of the Invention

[0005] overview In some embodiments, the computer-implemented method includes accessing an image of at least a portion of a biological structure of a particular subject. In some cases, the image is a three-dimensional image depicting a portion of the biological structure. The computer-implemented method may also include processing the image using a segmentation algorithm to extract a plurality of image objects depicted in the image. The computer-implemented method may also include determining one or more structural characteristics associated with an image object of the plurality of image objects. The computer-implemented method may also include processing the one or more structural characteristics using a trained machine learning model to generate inferred data corresponding to an estimation of whether the image object corresponds to a lesion or tumor associated with the biological structure. The trained machine learning model may be trained by a three-dimensional model constructed from a set of training images. The computer-implemented method may also include outputting the inferred data for the particular subject.

[0006] In some cases, at least a portion of the biological structure includes at least a portion of a lung. The image can depict a skeletal structure surrounding at least a portion of the biological structure. The image can depict a cross-section of at least a portion of the biological structure. The image can include or be derived from image data captured using a computed tomography scanner, which can be a micro-computed tomography scanner. The one or more structural characteristics can include a shape, a position, a surface area, and / or a longest diameter of the image object.

[0007] In some cases, processing the image using the segmentation algorithm includes determining one or more structural characteristics associated with the other image objects. Processing the image using the segmentation algorithm can also include processing the one or more structural characteristics of the other image objects using a trained machine learning model to generate inferred data corresponding to an estimation of whether the other image objects correspond to a type of biological structure. In some cases, the trained machine learning model processes the one or more structural characteristics to identify a level of tumor burden shown in the image. Processing the image using the segmentation algorithm can also include outputting inferred data associated with the other image objects. The type of biological structure can include a blood vessel, a lung, a heart, and / or a liver.

[0008] In some cases, the segmentation algorithm is a watershed transform algorithm. The trained machine learning model can be a trained support vector machine (SVM).

[0009] In some cases, the computer-implemented method includes preprocessing the image using a trained image preprocessing machine learning model to generate a filter. The filter can be applied to generate a filtered image by separating one or more image regions from the image. The separated image regions can represent biological structures. In some cases, the trained image preprocessing machine learning model includes a U-Net or V-Net model. The computer-implemented method can also include processing the separated image regions of the filtered image using a segmentation algorithm to extract multiple image objects depicted in the image. The trained image preprocessing machine learning model can be a trained convolutional neural network machine learning model.

[0010] In some cases, the computer-implemented method includes processing the image using an image filter to generate a filtered image that excludes one or more skeletal structures surrounding at least a portion of the biological structure. The filtered image can be used in place of the image to separate the image object from the plurality of image objects.

[0011] In some cases, the computer-implemented method includes processing the image using co-registration to align the image with a reference image, where the aligned image can be used in place of the image to extract the image object from the plurality of image objects.

[0012] In some cases, the computer-implemented method includes converting the image to a binary image. The computer-implemented method may also include converting one or more pixels of the binary image by using a flood-fill operation. The computer-implemented method may also include performing erosion and dilation operations to reduce image noise from the binary image.

[0013] In some cases, processing the image using the segmentation algorithm includes applying a negative distance transform function to the image to identify boundaries between two or more overlapping image objects of the plurality of image objects.

[0014] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0015] In some embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0016] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium containing instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0017] The terms and expressions which have been employed are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims. [Brief explanation of the drawings]

[0018] The present disclosure is described in conjunction with the accompanying drawings, in which:

[0019] [Figure 1] 1 illustrates an exemplary computing environment for processing an image to estimate whether at least a portion of a tumor is represented in the image, according to some embodiments.

[0020] [Figure 2] 1 shows an exemplary schematic diagram of an image processing system for processing an image to estimate whether at least a portion of a tumor is represented in the image, according to some embodiments.

[0021] [Figure 3] 1 illustrates an exemplary U-Net according to some embodiments.

[0022] [Figure 4A] 1 illustrates a residual block according to some embodiments.

[0023] [Figure 4B] 1 illustrates pyramidal layers according to some embodiments.

[0024] [Figure 5]FIG. 1 shows an exemplary schematic diagram of a V-Net architecture used for image pre-processing, according to some embodiments.

[0025] [Figure 6] 1 shows an example set of graphs illustrating the performance of a trained image preprocessing machine learning model.

[0026] [Figures 7A-7B] 1 shows an exemplary schematic diagram of an image filter subsystem for identifying biological structures from an input image, according to some embodiments.

[0027] [Figure 8] FIG. 1 illustrates an exemplary schematic diagram of a binary converter subsystem that converts a filtered image into a binary image, according to some embodiments.

[0028] [Figure 9A-9B] 1 illustrates an exemplary schematic diagram of an image registration subsystem for registering binary images to a single spatial coordinate system, according to some embodiments.

[0029] [Figure 10] FIG. 1 shows an exemplary schematic diagram of an image segmentation subsystem for extracting a set of image objects from aligned images, according to some embodiments.

[0030] [Figure 11] 1 illustrates an exemplary schematic diagram of a feature extractor subsystem that identifies a set of structural characteristics associated with an image object, according to some embodiments.

[0031] [Figure 12] 10 illustrates an exemplary set of box plots corresponding to the distribution of centroid locations of image objects detected by the image segmentation subsystem.

[0032] [Figure 13] 10 illustrates an exemplary set of box plots corresponding to the distribution of orientations of image objects detected by the image segmentation subsystem.

[0033] [Figure 14] 10 shows an exemplary set of box plots corresponding to the distribution of major axis lengths of image objects detected by the image segmentation subsystem.

[0034] [Figure 15] 10 shows an exemplary set of box plots corresponding to the distribution of sizes of image objects detected by the image segmentation subsystem.

[0035] [Figure 16] 10 shows an exemplary set of box plots corresponding to the distribution of shapes of image objects detected by the image segmentation subsystem.

[0036] [Figure 17] 10 shows an exemplary set of box plots corresponding to the distribution of voxel intensities of image objects detected by the image segmentation subsystem.

[0037] [Figure 18] 1 shows an exemplary schematic diagram of a support vector machine for estimating whether an image object corresponds to a tumor, according to some embodiments;

[0038] [Figure 19] 1 shows an example of a graph identifying the relationship between lung volume and total tumor volume detected from an image.

[0039] [Figure 20] 1 shows an exemplary set of graphs identifying the relationship between blood vessel volume and tumor volume.

[0040] [Figures 20A-20C] 10 shows an example screenshot of a user interface for labeling training image objects corresponding to three-dimensional image data, according to some embodiments.

[0041] [Figure 21] 10 illustrates a process for processing an image to estimate whether at least a portion of a tumor is represented in the image, according to some embodiments.

[0042] [Figure 22] 1 illustrates a process for processing an image to extract a set of image objects according to some embodiments.

[0043] In the accompanying drawings, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes between the similar components. When only a first reference label is used in this specification, the description is applicable to any of the similar components having the same first reference label, regardless of the second reference label. DETAILED DESCRIPTION OF THE INVENTION

[0044] Detailed Description I. Overview The technique can involve processing an image to estimate whether at least a portion of a tumor is represented in the image. In particular, the technique can identify individual tumors in such an image, even when multiple tumors are present in the three-dimensional image. An image depicting a biological structure (e.g., a three-dimensional micro-computed tomography (microCT) image) can be accessed by an image processing system. The image processing system can process the image through a series of image transformation operations, including filtering, binarization, image registration, and segmentation. As a result, the image processing system can define boundaries within the image that can be used to distinguish regions of the image. Based on the identified boundaries, a set of image objects can be identified from the image. A classification system can process each of the extracted image objects to determine a set of structural characteristics. The set of structural characteristics can be processed using a trained machine learning model (e.g., a support vector machine). Based on the output from the trained machine learning model, the classification system can identify, for each of the extracted image objects, information indicating whether the image object corresponds to a tumor, a lesion, or normal tissue. In some cases, the classification system processes a set of structural features to identify a level of tumor burden corresponding to the biological structure depicted in the image.

[0045] The trained machine learning model can be trained based on training data including a set of training images. Each training image of the set can include multiple training image objects. Each training image object of the multiple training image objects can be associated with a label that identifies a biological structure corresponding to the training image object. To generate labels corresponding to the training image objects, the training system can render the three-dimensional training images into a set of two-dimensional CT images representing slices of the biological structure so that the image objects corresponding to the three-dimensional training images can be labeled. The images can also be rendered as interactive three-dimensional image data that can include a set of three-dimensional image objects corresponding to respective portions of the biological structure. A label can be assigned to each object of the set of three-dimensional image objects including (for example) a tumor, heart, liver, lungs, mediastinum, and / or blood vessels. The training data (including the training images and labels) can be used to generate a set of structural features that can be used to train the machine learning model.

[0046] The classification system can receive a trained machine learning model from a training system. The classification system can receive unlabeled images from which a set of image objects can be extracted by an image processing system. For each unlabeled image object, the classification system can identify its structural characteristics. The classification system can use the trained machine learning model to identify whether structural characteristics of a particular image object correspond to structural characteristics associated with a particular biological structure (e.g., tumor, lung, blood vessel). Based on the identification, the classification system can estimate whether a particular image corresponds to a tumor, a lesion, or normal tissue. In addition to classification, the trained machine learning model can identify the level of tumor burden from an image. For example, the trained machine learning model can predict that a given image of a particular subject contains a tumor and further predict a moderate tumor burden in a given image. The trained machine learning model can include a discriminant analysis model, a kernel classification model, a k-nearest neighbor model, a linear classification model, a naive Bayes classifier, a support vector machine, a gradient boosting ensemble classification algorithm, and / or one or more classification trees.

[0047] Thus, embodiments of the present disclosure provide technical advantages over conventional systems that analyze images to detect the presence of tumors in tissue. Conventional systems can simply identify whether a tumor is present in an image, but cannot identify individual tumor types based on the area depicted in the image. Furthermore, tumor identification by conventional systems requires defining rules regarding image intensity, morphology, and relationships with other anatomical features, which must then be implemented programmatically, which is often difficult. Techniques for preprocessing images through filtering, registration, and segmentation operations can further improve the training (e.g., unsupervised training) and performance of trained classifier systems so that they can identify and classify individual tumors. Furthermore, the automatic generation of three-dimensional data constructed from training data significantly improves the speed at which training images can be processed and labeled to train machine learning models.

[0048] II. Automatic tumor detection technology based on image processing II.A. Exemplary Computing Environment FIG. 1 illustrates an exemplary computing environment 100 for processing images to estimate whether at least a portion of a tumor is represented in the image, according to some embodiments. The computing environment 100 may include a tissue classification system 105, which may further include an image processing subsystem 110, a classification subsystem 112, and a training subsystem 115. The image processing subsystem 110 may extract image objects from the images corresponding to individual subjects by processing the images through different types of image processing operations (e.g., automatic registration, image segmentation, and / or feature extraction). The classification subsystem 112 may process each of the extracted image objects to generate a set of structural features, which may be processed using a trained machine learning model (e.g., a support vector machine). Based on output from the trained machine learning model, the classification subsystem 112 may generate information corresponding to each image object, the information indicating whether the image object corresponds to a tumor, a lesion, or normal tissue.

[0049] The image processing subsystem 110 can access images (e.g., 3D micro-CT images) showing one or more biological structures. A biological structure can refer to one or more types of tissue represented in the image. For example, the biological structure can include individual organs such as the lung, heart, or liver, various types of tissue (e.g., bone, blood vessel, nerve, tumor), and / or any structure (e.g., lesion) that exhibits an alteration of at least a portion of a biological structure. The image processing subsystem 110 can process the image using a series of image transforms to extract multiple image objects and derive structural properties from each of the multiple image objects.

[0050] The training subsystem 115 can train the machine learning model and send it to the classification subsystem 112. The training subsystem 115 can use training data (e.g., structural features corresponding to labeled training image objects) to learn parameter values ​​for the machine learning model. Various machine learning techniques can be used to train the machine learning model. The machine learning techniques can include decision tree learning, association rule learning, artificial neural networks, deep learning, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity and metric learning, sparse dictionary learning, genetic algorithms, and / or rule-based machine learning. The machine learning techniques can include ensemble techniques that learn inter-ensemble weights to apply to results generated from various underlying techniques (such as two or more of the foregoing). The inter-ensemble weights can be identified based on (for example) accuracy, speed, and / or resource usage associated with the underlying techniques.

[0051] The training data may include a set of structural characteristics corresponding to image objects in the training images and a label corresponding to each of the image objects. The training subsystem 115 may receive a label corresponding to each image object in the training images. The labels associated with the training data may be received or derived from data received from the provider system 120, each of which may be associated with (for example) a doctor, lab technician, nurse, hospital, CT scan technician, etc. The training data may be generated based on communications from a healthcare provider device associated with the provider system 120. The communications from the healthcare provider device may include medical records corresponding to a particular subject. The medical records may include (for example) an expert's diagnosis or characterization indicating whether at least a portion of a tumor is represented in the CT image.

[0052] The training subsystem 115 can also collect labels based on the 3D training images. The training subsystem 115 can perform a segmentation operation (e.g., watershed segmentation, k-means) to define boundaries within the 3D training images and then render 2D slice images. Based on the segmentation operation, the training subsystem 115 can define each of a set of 3D image objects within the images, each image object corresponding to a respective portion of a biological structure. A label can be assigned to each object in the set of 3D image objects. For example, each image object can be labeled as (for example) a tumor, heart, liver, lung, mediastinum, and / or blood vessel.

[0053] Additionally or alternatively, the training subsystem 115 can present two-dimensional slice images rendered from the three-dimensional training images to facilitate collection of labels based on the three-dimensional training images. The two-dimensional slice images correspond to successive and / or adjacent slices of the biological structures depicted in the training images. Each of the two-dimensional training images can include regions identified based on boundaries generated from the segmentation operation. Visualization of the two-dimensional images can be used to facilitate labeling of the regions.

[0054] The labeled image objects can be processed to generate their corresponding structural properties. Such data can be used to train a machine learning model. Specifically, a set of structural properties corresponding to each labeled training image object can be generated based on morphological features identified from the labeled training image object. For example, the set of structural properties can include a diameter, a surface area, a shape, and a median convex volume corresponding to each labeled training image object. For each labeled training image object, the set of structural properties can be stored in a data structure including a two-dimensional array. The data structure storing the set of structural properties of the image objects can be input to train a machine learning model, and the machine learning model can be trained to predict whether the image object corresponds to a tumor based on the determined structural properties.

[0055] The user device 130 can be used to assign labels to the set of three-dimensional image objects. For example, the user device 130 can include a user interface. The user interface can be used to label each of the set of three-dimensional image objects. The training subsystem 115 can utilize the user interface of the user device 130 (e.g., to locally present or transmit configuration data) to represent training images with corresponding sets of three-dimensional image objects. Thus, three-dimensional images can be loaded into a user interface (e.g., the user interface 1000 of FIGS. 10A-10C) such that the user interface can enable access of image objects of the set of image objects corresponding to each of the loaded three-dimensional images. In this manner, a user using the user device 130 can assign tissue labels (e.g., tumor, liver, heart) to each image object, thereby generating training data used to train a machine learning model.

[0056] The classification subsystem 112 can receive a trained machine learning model from the training subsystem 115. The classification subsystem 112 can receive an input image from which a set of unlabeled image objects can be extracted by the image processing subsystem 110. The input image can be preprocessed using the trained image preprocessing machine learning model to identify regions of interest (ROIs) including organs and tumors. The ROIs can be processed using binarization, registration, and segmentation (for example) to extract a set of unlabeled image objects. For each unlabeled image object, the classification subsystem 112 can identify or otherwise access its structural properties. Structural properties can refer to morphological features associated with the image object that can be measured using any technique known to those skilled in the art. For example, structural properties can include diameter, surface area, shape, median convex volume, equivalent diameter, orientation, solidity, and / or volume associated with the image object. The classification subsystem 112 can use a trained machine learning model to identify whether structural characteristics of a particular image object correspond to pre-identified structural characteristics associated with a particular biological structure (e.g., tumor, lung, blood vessel). Based on the identification, the classification subsystem 112 can estimate whether a particular image corresponds to a tumor or a lesion.

[0057] At least a portion of the images processed by the trained machine learning model may be collected using and may include or be derived from data received from the imaging system 120. The imaging system 120 may include a CT system that may include a tomography imager and / or a micro-CT component (or micro-tomosynthesis component). The tomography imager may provide global information for determining the region / volume of interest, extracting the surface of the head as a reference, and assisting in local micro-tomography / local micro-tomosynthesis. The micro-CT (or micro-tomosynthesis) component may be integrated within or separate from the tomography imager to obtain high-resolution data of the region / volume of interest. Based on the determined region / volume of interest, the CT system may capture an image and transmit the captured image to the tissue classification system 105.

[0058] II.B. Exemplary Schematic of an Image Processing System FIG. 2 illustrates an exemplary schematic diagram 200 of an image processing system for processing an image to estimate whether at least a portion of a tumor is present in the image, according to some embodiments. Input images 205 can be accessed by an image processing subsystem 210. The input images 205 can be three-dimensional images derived from data captured using a CT system, including a micro-CT system. Each of the input images 205 can depict a three-dimensional representation corresponding to a biological structure. Each input image 205 can be captured at a specific time point. For example, an input image of the input images 205 can be captured at a time point when no lesion or tumor is detected. Another input image of the input images 205 can be captured at a different time point when a lesion or tumor is visible in the image. The input images 205 may or may not be captured using a respiratory gating operation. For example, an imaging system (e.g., imaging system 130 of FIG. 1) can use a respiratory gating operation to identify a specific phase (e.g., inspiration, expiration) of a subject's respiratory cycle. The imaging system can then capture each of the input images 205 during a specific phase corresponding to the pre-identified phase. In some cases, the input image 205 is an image depicting a ROI, where image regions depicting biological structures including the heart, lungs, and tumor are cropped from the original input image.

[0059] To generate a filtered image, a filter can be applied to each of the input images 205. A first filter can be applied to each of the input images 205 to isolate a biological structure of interest from the image. In some cases, the first filter includes an image mask (e.g., a lung mask) specifically used to isolate a particular biological structure (e.g., lungs). Additionally or alternatively, the first filter can include another type of image mask (e.g., a thoracic cavity mask) that can be used to isolate multiple biological structures of interest (e.g., lungs, heart, liver, mediastinum, tumor, blood vessels). A second filter can be further applied to each of the input images to identify specific skeletal structures appearing in the input images. Image portions corresponding to the specific skeletal structures can be removed from each of the input images 205. The identified skeletal structures can alternatively be used as a boundary surrounding the biological structure, where the boundary can be used to adjust the size of the region corresponding to the biological structure of interest.

[0060] In some cases, a trained image preprocessing machine learning model is applied to the input image to generate the filter 215. For example, a trained V-Net may be used to process the input image to identify a biological structure of interest (e.g., the heart). In another example, a trained U-Net may be used to process the input image to identify a biological structure of interest. Thus, the trained image preprocessing model may be used to generate one or more filters (e.g., a lung mask) to separate the biological structure (e.g., the lungs) from other regions of the input image (e.g., the rib cage, blood vessels).

[0061] The binary converter 220 can convert the filtered image into a binary image. The binary converter 220 can convert each pixel in at least a portion of the filtered image into a pixel value of either 0 or 1. For example, the filtered image can be a grayscale image, which can be converted into a binary image by the binary converter 220. The binary converter 220 can further convert binary values ​​corresponding to a subset of pixels in the binary image (e.g., 0 pixels surrounded by 1 pixels) based on a flood-fill operation. Thus, the binary converter 220 can remove background signals and image objects in the filtered image. In fact, converting the filtered image into a binary image can sharpen the boundaries surrounding each image object, thus generating an output optimized for subsequent image processing operations. For example, an image registration operation can better align images as a result of the conversion. This is because the binary image with sharpened boundaries can be properly aligned with that of the reference image. In another example, generating a binary image can enable an image segmentation algorithm to generate a more accurate set of image objects because the image objects are more clearly defined based on their sharp boundaries. To further reduce image noise in the binary image, an erosion-dilation operation can be applied. Optionally, a binary converter 220 converts the filtered image into a binary image after the alignment operation has been performed.

[0062] The image registration subsystem 225 can process the binary images using image registration operations to generate aligned images. The image registration operations (e.g., rotate, scale, translate, crop) can include associating the binary images with a reference image to align the binary images to a single spatial coordinate system. Because the aligned images are aligned to a single spatial coordinate system, subsequent operations (e.g., image segmentation algorithms) can use similar position parameters (e.g., x, y, and z coordinates) for the set. Furthermore, training machine learning models with aligned images can improve accuracy during deployment. Otherwise, unless all images are aligned in the same coordinate system, the position of each image object, represented by x, y, and z coordinates, can correspond differently from image to image. The image registration subsystem 225 can align the binary images using a reference filter to derive transformation parameters from the filtered images. For example, a bone mask filter of the reference image can be used as a reference filter that can be applied to the bone mask filter corresponding to the set of filtered images. As a result, a first set of transformation parameters (e.g., a transform) can be identified. For each of the binary images, a first set of transformation parameters can be applied by the image registration subsystem 225 so that the binary image can be warped toward the position coordinates corresponding to the bone mask filter. The filtered image can also be warped using the first set of transformation parameters so that it can be used to generate additional sets of transformation parameters.

[0063] The image registration subsystem 225 can further apply a second registration operation to the distorted image (e.g., the distorted image based on the first set of transformation parameters) using the reference image. Each of the distorted filtered images can be registered to a corresponding region of the reference image by the image registration subsystem 225. Once the registration is complete, a second set of transformation parameters can be identified (e.g., tform_grayscale). The distorted image generated based on the reference filter can be re-warped using the second set of transformation parameters to generate an aligned image. The second registration operation using the reference image can allow the aligned images to be further calibrated toward a single coordinate system so that training of machine learning models as well as other image processing operations can be more efficient.

[0064] The image segmentation subsystem 230 can process the aligned images using a segmentation algorithm to extract image objects corresponding to each aligned image. The segmentation algorithm can enable identification of morphological features in each of the aligned images and can define boundaries of the image objects by using the morphological features. The image segmentation subsystem 230 can extract each image object from the aligned images using the identified boundaries. Various techniques can be used to extract image objects from the aligned images, including watershed segmentation algorithms, graph segmentation algorithms, and model-based segmentation algorithms.

[0065] For each of the extracted image objects, the feature extractor 235 can identify its structural characteristics, and other types of characteristics from each of the image objects, including, but not limited to, the position coordinates of the image object's center of mass, eigenvalues ​​associated with the voxels that form the image object, and Euler angles associated with the image object.

[0066] II.C. Exemplary Schematic of Applying an Image Filter to an Input Image As described herein, a filter can be applied to each of the input images to generate a filtered image. A first filter can be applied to each of the input images to isolate a biological structure of interest from the image. A trained U-Net (for example) can be used to generate the first filter, which can identify and isolate a specific organ, such as the lungs. In some cases, the first filter includes an image mask (e.g., a lung mask) specifically used to isolate a specific biological structure (e.g., the lungs). Additionally or alternatively, the first filter can include another type of image mask (e.g., a thoracic cavity mask) that can be used to isolate multiple biological structures of interest (e.g., the lungs, heart, liver, mediastinum, tumor, blood vessels). A second filter can be further applied to each of the input images to identify specific skeletal structures appearing in the input images. Image portions corresponding to the specific skeletal structures can be removed from each input image. The identified skeletal structures can alternatively be used as boundaries surrounding the biological structures, where the boundaries can be used to adjust the size of the region corresponding to the biological structure of interest.

[0067] II.C.1 Image Preprocessing: Generating Image Filters Using Machine Learning Models In some embodiments, a trained image preprocessing machine learning model is used to process each input image to generate one or more filters that can be used to identify organs, tissues, tumors, and blood vessels depicted in the input image. The trained machine learning model can additionally or alternatively be used for organ-specific identification of one or more organs, such as the spleen, liver, lungs, and kidneys. Based on the identification, the generated filters can be applied to each of the input images, thereby focusing subsequent image processing on image regions corresponding to the ROIs identified by the filters.

[0068] II.C.1.a Example training dataset for training image preprocessing machine learning models The image preprocessing machine learning model can be trained using a training dataset containing multiple training images. The training images correspond to images in which ROIs corresponding to biological structures have been identified. In some cases, the ROIs in the training images are identified as follows: (i) a technician marks multiple 2D regions corresponding to biological structures (e.g., lungs). (ii) the 2D regions are propagated into a contiguous 3D ROI; (iii) the 3D ROIs are segmented from the training images; and (iv) tissue volumes are calculated from the 3D ROIs. An exemplary training dataset can include 3520 CT scans with lung ROIs; a subset of the training images can be discarded, and the ROIs are not accurately identified (e.g., missing ROIs, corrupted files, incorrect identification of biological structures). II.C.1.b Generating Image Filters Using U-Net

[0069] The 3D training images can be used to train a machine learning model (e.g., U-Net), which can generate filters to identify biological structures of interest. As shown in FIG. 3, U-Net 300 can include a reduction path 305 and an expansion path 310, giving it a U-shaped architecture. The reduction path 305 is a CNN network that includes repeated application of convolutions (e.g., 3x3 convolutions (unpadded convolutions)), each followed by a rectified linear unit (ReLU) and max-pooling operation for downsampling (e.g., 2x2 max-pooling with stride 2). At each downsampling step or pooling operation, the number of feature channels can be doubled. During the reduction, the spatial information of the image data is reduced, and the feature information is increased. The expansion path 310 is a CNN network that combines the feature and spatial information from the reduction path 305 (upsampling the feature maps from the reduction path 305). The upsampling of the feature map is followed by a series of upconvolutions (upsampling operators) that halve the number of channels, concatenation with the corresponding cropped feature map from the reduction path 305, repeated application of each convolution (e.g., two 3x3 convolutions) followed by a rectified linear unit (ReLU), and a final convolution (e.g., a 1x1 convolution) to generate a 2D tumor mask. For localization, the high-resolution features from the reduction path 305 are combined with the upsampled output from the augmentation path 310. The U-Net 300 uses the effective portion of each convolution without a fully connected layer; i.e., the segmentation map contains only pixels for which the full context is available in the input image, and uses skip connections to link the context features learned during the reduction block with the localization features learned in the augmentation block.

[0070] In a conventional U-Net architecture, a convolutional block consists of convolutional layers (e.g., typically two or three layers) for performing convolutions. However, according to various embodiments, the convolutional block and convolutional layers are replaced by a residual block 315 with separable convolutions performed in pyramid layers 320 at one or more augmentation levels (e.g., a single convolutional layer can be replaced by two or more pyramid layers 320). (e.g., stacked filtered images). Figure 4A shows one layer structure of the residual block 315 shown in Figure 3. As shown, the residual block 400 can include multiple pyramid layers 405. In a network (e.g., a ResNet) including the residual block 400, each pyramid layer 405 feeds the next layer (A, B, C, etc.) and then directly feeds layers (D, E, etc.) approximately two to three layers away. The use of the residual block 400 in the network helps overcome the degradation problem that arises from increasing the number of pyramid layers (as the number of layers continues to increase, accuracy initially improves, but begins to saturate at a point and eventually degrades). The residual block 400 uses skip or residual connections to skip over some of these additional pyramid layers, eventually propagating large gradients to the initial pyramid layers. Skipping effectively simplifies the network by using fewer pyramid layers in the initial training phase. This speeds up learning by reducing the impact of vanishing gradients because there are fewer layers to propagate (i.e., multi-speed residual learning). The network then gradually restores the skipped layers as it learns the feature space.

[0071] FIG. 4B illustrates a single pyramid layer 405 of FIG. 4A according to various embodiments. As shown in FIG. 4B, the pyramid layer 405 can use atrous separable convolutions ("atrous convolution blocks") at multiple different scales (four levels in this example). The pyramid layer 405 includes the same image at multiple different scales to improve the accuracy of object (e.g., tumor) detection. Atrous convolution refers to a filter with a "diffusive" receptive field that increases the size of the receptive field relative to the kernel size. In some embodiments, the one or more atrous convolution levels are four atrous convolution levels. In other embodiments, more or fewer levels of atrous convolution, such as six levels, can be used. The convolution layer output 415 is the output of atrous convolution blocks 420 (here labeled as atrous convolutions 1, 2, 4, and 8). The illustrated example of FIG. 4B assumes four atrous convolution blocks, each outputting two channels (of the same color), for a total of eight channels. The number of channels output by each atrous convolution block can vary depending on the residual block in question. The example of Figure 4B shows the upper left or upper right residual block 315 of Figure 3. In some embodiments, the number of channels output by each enhancement block 415 in the pyramid layer 410 of the residual block 405 is equal to the number of k filters on the residual block 405 divided by four.

[0072] Empirical evidence shows that the residual block allows for improved accuracy and easier optimization. Separable convolution, depthwise convolution followed by pointwise convolution, also shows a significant increase in convergence speed and a significant reduction in model size. Dilated convolution expands the receptive field without losing resolution, thus allowing multiscale contextual information to be aggregated with downsampling. Redesigning the convolution block allows for the extraction of highly localized and rare information within an image.

[0073] II.C.1.c Generating Image Filters using V-Net A model (e.g., a 3D convolutional neural network such as a V-Net for 3D segmentation) can include downsampling and upsampling subnetworks with skip connections to propagate high-resolution information to the final segmentation. In some cases, the downsampling subnetwork can be a sequence of multiple dense feature stacks connected by downsampling convolutions, where each skip connection can be a single convolution of the corresponding dense feature stack output, and the upsampling network includes bilinear upsampling to the final segmentation resolution.

[0074] The trained image preprocessing machine learning model can be used to extract features from input images (e.g., microCT images) using a convolutional neural network (CNN) system that includes multiple distinct submodels to identify regions of interest (ROIs) within the image. As shown in Figure 5, a trained V-Net 500 can be used to refine the input image. The trained V-Net 500 can perform a series of operations to form a single convolutional layer for several types of CNN architectural elements: (1) convolution, (2) nonlinear transformation (e.g., ReLU), (3) pooling or subsampling, and (4) classification (fully connected layer). In some cases, the convolution operations of the trained V-Net 500 preserve spatial relationships between pixels or voxels within or across two-dimensional or three-dimensional scans by learning image features using small squares of input data. For example, the input image can be thought of as a matrix of pixel and voxel values, with a value assigned to each pixel and voxel region in the matrix. Furthermore, the input image may include a black and white image having pixel or voxel values ​​ranging from 0 to 1. Alternatively or additionally, the input image may include a color image having three assigned RGB pixel or voxel values ​​ranging from 0 to 255.

[0075] After the input image is accessed, the trained V-Net 500 can perform convolutions on the input image to extract features corresponding to the anatomical regions depicted in the image. The left side of the trained V-Net 500 can include a compression path 510 for downsampling, and the right side can include a decompression path 515 for upsampling, which decompresses the signal until it reaches its original size. The compression path 510 can be divided into different stages operating at different resolutions. Each stage can include one or more convolutional layers. The convolutions within each layer can be applied with appropriate padding. Each stage can be configured to learn a residual function via residual connections, where the input of each stage is (i) used in a convolutional layer, processed through a nonlinearity, and (ii) added to the output of the last convolutional layer in that stage to enable learning of the residual function. The convolutions performed at each stage use a volumetric kernel with a predetermined size, such as 5x5x5 voxels. As data progresses through different stages along the compression path 510, its resolution can decrease. Each stage along the compression path 510 can be performed by convolution with a kernel of a predetermined size, such as a 2x2x2 voxel-wide kernel with an appropriate stride (e.g., a slide of 2). The second operation considers only non-overlapping volume patches to extract features, so the size of the resulting feature map can be halved (subsampled). This strategy can serve a similar purpose to a pooling layer. Replacing the pooling operation with a convolution operation can reduce the memory footprint of the network because backpropagation does not require a switch to map the output of the pooling layer to the input. Each stage in the compression path 510 can compute a number of features that is multiple times higher than the number of features from the previous layer.

[0076] The decompression path 515 can extract features and expand the spatial support of the low-resolution feature maps to collect and assemble the information necessary to output a two-channel volumetric segmentation corresponding to the image ROI. After each stage of the decompression path 515, a deconvolution operation can be used to increase the size of the input, followed by one or more convolution layers containing half the number of kernels, such as a 5x5x5 kernel, used in the previous layer. Similar to the compression path 510, a residual function can be learned in the convolution stage of the decompression path 515. Additionally, features extracted from earlier stages of the compression path 510 can be transferred to the decompression path 515, as indicated by horizontal connection 520. The two feature maps (two volumes with the same resolution as the original input data) computed by the final convolution layer, which has an appropriate kernel size, such as a 1x1x1 kernel size, and produces an output the same size as the input volume, can be processed through a softmax layer, which outputs the probability that each voxel belongs to the foreground and background maximal voxel units. Based on the probabilities output by the softmax layer, the trained image preprocessing machine learning model can indicate a probability as to whether a particular voxel corresponds to an image region of the ROI. Thus, the output generated by the trained V-Net 500 for all voxels in the input image can indicate multiple image regions corresponding to the ROI, which may include tissue, blood vessels, tumor, and mediastinum.

[0077] After the ROI is identified, it can be used to generate a filter to isolate the biological structure from the input image. The use of a trained image preprocessing machine learning model can improve the accuracy of image classification because only the relevant portions of the image are classified.

[0078] II.C.1.d Illustrative Results 6 shows an example set of graphs 600 illustrating the performance of a trained image preprocessing machine learning model. In the linear regression graph 602, each plot point represents, for the corresponding image, the amount of tissue volume manually identified by tracking the image (x-axis) and the amount of tissue volume detected by the trained V-Net (y-axis). In graph 602, the coefficient of determination (R2) is 0.96, demonstrating the accuracy of V-Net in detecting tissue volumes across images containing different amounts of tissue volume.

[0079] In the linear regression graph 604, each plot point represents, for the corresponding image, the amount of tissue volume manually identified by tracing the image (x-axis) and the amount of tissue volume detected by rule-based analysis (y-axis). In this example, rule-based analysis is performed using the technique discussed in Barck, Kai H, et al., "Quantification of Tumor Burden in a Genetically Engineered Mouse Model of Lung Cancer by Micro-CT and Automated Analysis." Translational Oncology Vol. 8, 2 (2015): 126-35. In graph 604, the coefficient of determination (R2) is 0.72. Using the manually traced tissue volume as a reference point, the trained V-Net detected tissue volumes more accurately than the tissue volumes detected by rule-based analysis. Such accuracy may indicate improved performance of the V-Net in generating filters for separating tissues (e.g., lung) from corresponding images.

[0080] II.C.2 Applying Image Filters 7A-7B illustrate an exemplary schematic diagram 700 of an image filter subsystem for identifying biological structures from input images, according to some embodiments. An image filter 715 of an image processing system (e.g., image processing system 310 of FIG. 3 ) can access a set of input images 705 depicting a subject's biological structures (e.g., organs, blood vessels, tumors). For each of the set of input images 705, the image filter 715 can apply an image mask to generate a set of filtered images. Each of the generated filtered images depicts the biological structure of interest (e.g., lungs) differentiated from background portions of the input image.

[0081] In FIG. 7A , a first image filter 720 may generate a first set of image masks corresponding to a set of input images 705. In each image mask of the first set of image masks, the first image filter 720 may designate one or more biological structures identifiable in the image, while image regions that do not correspond to the designated biological structures (e.g., background portions) may be obscured or removed. The first image filter 720 may apply each image mask of the generated first set of image masks to each input image 705. By applying the first set of image masks to the set of input images 705, the first image filter 720 may generate a first set of filtered images 730. For each filtered image in the set of filtered images, at least a portion of the filtered image (e.g., lungs, thoracic cavity) may be distinguished from other portions of the filtered image (e.g., rib cage, air). Further details regarding the use of image masking operations to identify biological structures as disclosed herein are described in Barck KH et al., "Quantification of Tumor Burden in a Genetically Engineered Mouse Model of Lung Cancer by Micro-CT and Automated Analysis." Transl Oncol. 2015;8:126-135 and Wyatt SK et al., "Fully-automated, high-throughput micro-computed tomography analysis of body composition enables therapeutic efficacy monitoring in preclinical models." Int J Obes (Lond). June 2015, both of which are incorporated by reference in their entirety for all purposes.

[0082] In FIG. 7B , the second image filter 725 may also process each of the set of input images 705 to generate a second set of image masks. Using the second image masks, the second image filter 725 can generate a second set of filtered images 740. For each of the second set of filtered images 740, different portions of the filtered image can be identified for removal. For example, the filtered images of the second set of filtered images 740 can indicate image regions corresponding to specific skeletal structures that can be removed, while other portions of the filtered image (e.g., lungs) remain. The first image filter 720 can also use the second set of image masks to draw a boundary defining a biological structure of interest (e.g., lungs) and generate the first set of filtered images 735 such that image regions outside the boundary can be removed. The second set of filtered images can be sent to the binary converter subsystem 735.

[0083] II.E. Exemplary Schematic of a Binary Converter for an Image Processing System FIG. 8 shows an example schematic 800 of a binary converter subsystem that converts filtered images into binary images, according to some embodiments. A binary converter subsystem 810 (e.g., binary converter subsystem 835 in FIG. 8 ) can access a set of filtered images 830 output from an image filter (e.g., image filter 815 in FIG. 8 ). The binary converter subsystem 810 can binarize each pixel of each filtered image in the filtered images 830 based on the pixel's corresponding radiological density value. For example, a pixel associated with an HU above a determined threshold can be configured as a 1 pixel value, and another pixel associated with an HU below the threshold can be configured as a 0 pixel value. The binary converter subsystem 810 can use a binarization operation in which a threshold range can be determined to minimize the variance of black and white pixels. For example, the binary converter subsystem 810 can set a threshold range between −200 HU and 400 HU, and any pixel associated with an HU outside the threshold range can be converted to a 0 pixel value. As a result, a set of binary images 835 can be output, each with a pixel value of either 1 or 0.

[0084] The binary converter subsystem 810 may further perform a flood-fill operation on each of the set of binary images 835 to remove holes in regions of the image. For example, a binary image may include a group of pixels with a zero value (e.g., a hole) surrounded by a larger group of pixels with a value of one (e.g., a region surrounding the hole). The binary converter system may perform a flood-fill operation on the group of zero-value pixels to output a new binary image in which the two groups of pixels are merged into a new group of pixels with a value of one (e.g., a region without a hole).

[0085] Each of the set of binary images 835 can be further processed by an erosion operation 815 and / or a dilation operation 820. By performing the erosion operation 815 and the dilation operation 820, small image objects that cannot be accurately classified can be removed from each binary image of the set of binary images 835. Removing smaller image objects from classification can improve the performance of subsequent segmentation operations. The erosion operation 815 can access a binary image of the set of binary images and initialize an image processing kernel (e.g., a 5×5 kernel) at the first pixel in the binary image. As the image processing kernel traverses each pixel of the binary image, a pixel with a value of 1 (e.g., white) can be converted to 0 (e.g., black) if at least one neighboring pixel covered by the kernel has a value of 0. In effect, the erosion operation 815 erodes the boundaries of image objects appearing in the binary image. Furthermore, the erosion operation 815 can remove any salt noise depicted in the binary image, specifically, any sparsely occurring white pixels that are not part of the image object.

[0086] The dilation operation 820 can access the binary image processed by the erosion operation 815. Similar to the erosion operation 815, an image processing kernel is set to the first pixel of the binary image. As the image processing kernel traverses each pixel of the processed binary image, if at least one adjacent pixel covered by the kernel has a value of 1, the pixel having a value of 0 (e.g., black) can be converted to 1 (e.g., white). In effect, the dilation operation 820 can gradually expand the boundary corresponding to the biological structure depicted in the processed binary image. Furthermore, similar to the fill operation, the dilation operation 820 can remove pepper noise depicted in the binary image, specifically, sparsely occurring black pixels within the image object.

[0087] Performing an erosion operation 815 followed by a dilation operation 820 can remove salt noise from each of the set of binary images 835. Such a sequence of operations can be referred to as an opening operation. Conversely, performing a dilation operation 820 followed by an erosion operation can remove pepper noise from each of the set of binary images 835. A sequence of dilation followed by erosion can be referred to as a closing operation. By performing the erosion operation 815 and / or the dilation operation 820 in different orders, the binary converter subsystem 810 can remove abrupt disturbances in each of the set of binary images 835. The set of binary images can be provided to an image registration subsystem.

[0088] II.F. Exemplary Schematic of an Image Registration Subsystem of an Image Processing System 9A-9B show an exemplary schematic diagram 900 of an image registration subsystem that aligns binary images to a single spatial coordinate system, according to some embodiments. Specifically, the image registration subsystem 910 can process these image sets using a series of image registration operations. An image registration operation can refer to an optimization process in which a “moving” image is warped to minimize pairwise distances between all voxels in the “moving” image relative to voxels in the “fixed” or reference image. Image registration operations can be performed to align processed images onto corresponding regions (e.g., skeletal structures, lungs) of the reference image. Each image registration operation can result in a set of output images that are better scaled, aligned, and rotated to fit a single coordinate system. Subsequent image registration operations can calibrate and consequently improve the alignment of the set of output images. By using the aligned images to train a machine learning model, the image registration operations can improve the accuracy and performance of the trained machine learning model.

[0089] In FIG. 9A , the image registration subsystem 910 can access a set of filtered images 905 (e.g., the first and second sets of filtered images of FIG. 6 ) and a set of binary images 915 (e.g., binary image 835 of FIG. 8 ). During a first registration operation 920, the image registration subsystem 910 can first select a reference image from the set of filtered images 905. The reference image can refer to a filtered image that includes location parameters such that the set of binary images 915 can be registered based on the location parameters. From the reference image, a reference filter can be generated that corresponds to an image region of the reference image that depicts a skeletal structure (e.g., a reference bone mask). A filter (e.g., a bone mask filter) of the set of filtered images 905 that corresponds to the reference filter can be registered such that a first subset of filtered images can be registered, as shown in a first set of registered images 925. By aligning the filters, a first set of transformation parameters (e.g., a transform) can be determined. The first set of transformation parameters may indicate an estimated set of numerical values ​​corresponding to an affine or linear transformation (e.g., rotation, scale, translation, crop) to be applied to each filtered image of the set of filtered images 905 relative to the reference image.

[0090] The image registration subsystem 910 may apply a first set of transformation parameters to each of the set of filtered images 905 and the set of binary images 915 using an image distortion operation 930. For each of the set of binary images 915, the image distortion operation 930 may warp the binary image using the first set of transformation parameters to adjust its position and orientation. Each filtered image of the set of filtered images 905 may also be warped using the first transformation parameters, whereby the warped filtered image may be used to generate transformation parameters for a subsequent registration operation. The image registration subsystem 910 may output the warped set of filtered images 905 and the set of binary images 915 as a set of warped images 935.

[0091] 9B , the image registration subsystem 910 can access and perform a second registration operation 940 for each of the set of warped images 935. During the second registration operation 940, the image registration subsystem 910 can use the same reference image used by the first registration operation 920 of FIG. 9A . The reference image can include a reference filtered grayscale image. The reference image can be used to register each filtered image (e.g., a filtered grayscale image) of the set of filtered images 905, such that the filtered images of the set of filtered images 905 can be registered toward the reference image, as shown in the second set of registered images 950. By registering the filtered images of the set of filtered images 905, a second set of transformation parameters (e.g., tform_grayscale) can be determined. The second set of transformation parameters can indicate to the reference image an estimated set of numerical values ​​corresponding to an affine or linear transformation (e.g., rotation, scale, translation, crop) to be applied to each filtered image of the set of warped images 935.

[0092] The image registration subsystem 910 can use an image distortion operation 930 to apply a second set of transformation parameters to each of the set of warped images 935. For each of the set of warped images 935, the image distortion operation 930 can re-warp the warped image using the second set of transformation parameters to adjust its position and orientation. The image registration subsystem 910 can output the warped set of filtered images 905 and the set of binary images 915 as a set of aligned images 955. In some cases, only the first alignment operation 920 is performed to generate the set of aligned images 955. For example, the set of warped images 935 is the set of aligned images 955 that can then be processed by the image segmentation subsystem 1005.

[0093] II.G. Exemplary Schematic of an Image Segmentation Subsystem of an Image Processing System FIG. 10 shows an example schematic 1000 of an image segmentation subsystem for extracting a set of image objects from aligned images, according to some embodiments. The image segmentation subsystem 1005 can access a set of aligned images 1010. The set of aligned images 1010 can include one or more binary images (e.g., the set of binary images 835 of FIG. 8) applied by one or more alignment operations (e.g., the first alignment operation 920 of FIG. 9A , the second operation 940 of FIG. 9B ). For each of the sets of aligned images 1010, the image segmentation subsystem 1005 can perform watershed segmentation to extract the set of image objects. To perform watershed segmentation, the image segmentation system 1005 can shrink each aligned image in the set of aligned images 1010 and identify new image objects in the aligned images as they separate and disappear. The image segmentation system 1005 can then create seed points for each of the new image objects, and each seed point can be conditionally expanded (a layer added) until the aligned image is filled in. By repeating these operations, a set of image objects can be extracted from the aligned image.

[0094] Additionally or alternatively, the image segmentation subsystem 1005 may invert pixels of each aligned image in the set of aligned images 1010 to transform the aligned image into its complementary image. For example, black pixels corresponding to one portion of the aligned image may be transformed into white pixels, and white pixels corresponding to another portion of the aligned image may be transformed into black pixels. As the watershed segmentation continues, the image segmentation subsystem 1005 may process each complementary image corresponding to the set of aligned images 1010 using a negative distance transform operation. For each pixel corresponding to each complementary image, a negative distance transform operation may be performed to identify a distance value from the pixel to the nearest non-zero-valued pixel, calculate the negative of the identified distance value, and transform the pixel based on the calculated negative value. As a result of transforming the pixels corresponding to the complementary images, the image segmentation subsystem 1005 may generate a set of transformed images and process the set of transformed images through a segmentation algorithm (e.g., a watershed algorithm). The segmentation algorithm may generate a set of segmented images 1015, and a set of lines may be drawn on each of the set of segmented images 1015. For each segmented image in the set of segmented images 1015, the image segmentation subsystem 1005 may extract a set of image objects using the set of lines as boundaries corresponding to each image object represented in the segmented image.

[0095] II.H. Exemplary Schematic of a Feature Extractor Subsystem of an Image Processing System II.H.1 Identifying structural characteristics of image objects FIG. 11 shows an example schematic 1100 of a feature extractor subsystem that identifies a set of structural properties associated with image objects, according to some embodiments. A set of segmented images 1105 (e.g., set of segmented images 1015) can be accessed. For each segmented image in the set of segmented images 1105, a set of image objects can be identified based on a set of lines depicted in the segmented image. A feature extractor 1110 can access the set of image objects and determine a set of structural properties corresponding to each image object in the set of image objects. Each structural property in the set of structural properties includes a value corresponding to a particular structural property category (e.g., diameter, surface area, shape, median convex volume, voxel intensity). For each image object, the feature extractor 1110 can store the set of structural properties corresponding to the image object in a data structure 1115 (e.g., an array), and an identifier can be assigned to identify the image object. The data structure 1115 storing the set of structural properties of the image object can be transmitted to a trained machine learning model, which can predict whether the image object corresponds to a tumor based on the determined structural properties.

[0096] A set of various types of structural features can be determined by the feature extractor 1110 that can be processed and considered by a trained machine learning model. The trained machine learning model can include various types of machine learning models, such as a discriminant analysis model, a kernel classification model, a k-nearest neighbor model, a linear classification model, a naive Bayes classifier, a support vector machine, a gradient boosting ensemble classification algorithm, and / or one or more classification trees. For example, the set of structural features can include, but is not limited to, structural features corresponding to the following structural feature categories: "bounding box" - The smallest cuboid containing the area, returned as a 1x6 vector of the form [ulf_x ulf_y ulf_z width_z width_y width_z]. ulf_x, ulf_y, and ulf_z specify the upper-left front corner of the cuboid. width_z, width_y, and width_z specify the width of the cuboid along each dimension. "centroid" - The centroid or area, returned as a 1x3 vector of the form [centroid_x centroid_y and centroid_z]. The first element, centroid_x, is the horizontal coordinate (or x-coordinate) of the center of mass. The second element, centroid_y, is the vertical coordinate (or y-coordinate). The third element, centroid_z, is the plane coordinate (or z-coordinate). "Convex Hull" - The smallest convex polygon that can contain the area, returned as a p-by-3 matrix. Each row of the matrix contains the x-, y-, and z-coordinates of one vertex of the polygon. "Convex Image" - The image of the convex hull, returned as a volumetric binary image (logical) with all voxels inside the hull filled (set "on"). The image is the size of the region's bounding box. "Convex Volume" - The number of voxels in the "Convex Image", returned as a scalar. "eigenvalues" - the eigenvalues ​​of the voxels representing the region, returned as a 3x1 vector. For example, the regionprops3 operation can use the eigenvalues ​​to calculate the major axis lengths. "eigenvectors" - The eigenvectors or voxels representing the region, returned as a 3x3 vector. For example, the regionprops3 operation can use the eigenvectors to compute the orientation of an ellipsoid with the same normalized second-order central moment as the region. "equivalent diameter"—The diameter of a sphere with the same volume as the area, returned as a scalar. Calculated as (6*volume / pi)^(1 / 3). "Area" - The ratio of voxels in the region to voxels in the total bounding box, returned as a scalar. Calculated as volume divided by the volume of the bounding box: [Volume / (bounding box width * bounding box height * bounding box depth)]. "image" - The bounding box of the region, returned as a volumetric binary image (logical) that is the same size as the bounding box of the region. "On" voxels correspond to the region, and all other voxels are "off". "Orientation" (x, y, and z values) - Euler angles, returned as a 1x3 vector. Angles are based on the right-hand rule. For example, the regionprops3 operation can interpret angles by looking at the origin along the x, y, and z axes, which represent roll, pitch, and yaw, respectively. Positive angles represent counterclockwise rotation. Rotation operations are not commutative, so they must be applied in the correct order to achieve the intended effect. "Major Axis Length" (x, y, and z values) - The length (in voxels) of the major axis of an ellipsoid with the same normalized second-order central moments as the region, returned as a 1x3 vector. For example, the regionprops3 operation can sort values ​​from largest to smallest. "solidity"—The fraction of voxels inside the convex hull that are also inside the domain, returned as a scalar. Calculated as volume / convective volume. "Subarray Idx" - is the index used to extract elements within the object bounding box, returned as a cell array such that L(idx{:}) extracts the elements of L within the object bounding box. "surface area"—The distance around the boundary of the region, returned as a scalar. "Volume" - A count of the actual number of "on" voxels in the region, returned as a scalar. Volume represents a metric or measurement of the number of voxels in a region in the volumetric binary image BW. "VoxelIdxList" - the linear indices of the voxels in the region, returned as a p-element vector; and "voxellist" - The locations of the voxels within the region, returned as a p-by-3 matrix. Each row of the matrix has the form [xyz] and specifies the coordinates of one voxel within the region.

[0097] II.H.2 Examples of structural properties of image objects 12-17 show an example set of box plots identifying structural characteristics of image objects detected by the image segmentation subsystem. The box plots in FIGS. 12-17 show data corresponding to a feature array (e.g., the feature array generated by the feature extractor 1110 in FIG. 11). The feature array represents training images used to train a machine learning model for automatically detecting at least portions of tumors from images. For example, the plots show object features of objects in training images that were manually labeled using a labeling application (e.g., FIGS. 20A-20C). As the difference between the tumor and non-tumor box plots increases across FIGS. 12-17, the machine learning model (e.g., SVM) becomes increasingly likely to be able to distinguish between tumor and non-tumor objects after being trained. In each box plot in FIGS. 12-17, the horizontal line represents the median value (e.g., object volume) representing each class (e.g., tumor, non-tumor), and the edges of the "box" represent values ​​falling within the 25% and 75% quantiles. The vertical dashed lines (whiskers) indicate the total range of values ​​corresponding to each class.

[0098] FIG. 12 illustrates an exemplary set of box plots 1200 corresponding to the distribution of centroid locations of image objects detected by the image segmentation subsystem. Box plots 1202, 1204, and 1206 each illustrate a first distribution of centroid locations for a first set of image objects classified as tumor regions and a second distribution of centroid locations for a second set of image objects classified as non-tumor regions. Furthermore, box plot 1202 illustrates the distribution of centroid locations of image objects along the x-axis, box plot 1204 illustrates the distribution of centroid locations of image objects along the y-axis, and box plot 1206 illustrates the distribution of centroid locations of image objects along the z-axis. As shown in box plots 1202, 1204, and 1206, there is a similar distribution of centroid locations between image objects classified as tumor regions and non-tumor regions, which may indicate that centroid location features are less likely to predict the detection of tumor regions from an image compared to other features.

[0099] FIG. 13 illustrates an example set of box plots 1300 corresponding to the distribution of orientations of image objects detected by the image segmentation subsystem. Box plots 1302, 1304, and 1306 each illustrate a first distribution of orientations for a first set of image objects classified as tumor regions and a second distribution of orientations for a second set of image objects classified as non-tumor regions. The magnitude of the orientations is expressed in Euler angle units. Furthermore, box plot 1302 illustrates the distribution of image object orientations along the z-axis, box plot 1304 illustrates the distribution of image object orientations along the y-axis, and box plot 1306 illustrates the distribution of image object orientations along the z-axis. As shown in box plots 1302, 1304, and 1306, there is a similar distribution of orientations between image objects classified as tumor regions and non-tumor regions, which may indicate that object orientation features are less likely to predict the detection of tumor regions from images compared to other features. For example, box plot 1302 showed that object orientation along the x-axis (plot 1302 in FIG. 13) appeared similar for tumor and non-tumor classes. Thus, box plot 1032 indicates that object orientation is likely less predictive of tissue class (although it can still be informative) compared to other features (e.g., object volume).

[0100] FIG. 14 illustrates an exemplary set of box plots 1400 corresponding to the distribution of major axis lengths of image objects detected by the image segmentation subsystem. Box plots 1402, 1404, and 1406 each illustrate a first distribution of major axis lengths for a first set of image objects classified as tumor regions and a second distribution of major axis lengths for a second set of image objects classified as non-tumor regions. Furthermore, box plot 1402 illustrates the distribution of major axis lengths of image objects along the x-axis, box plot 1404 illustrates the distribution of major axis lengths of image objects along the y-axis, and box plot 1406 illustrates the distribution of major axis lengths of image objects along the z-axis. As illustrated in box plots 1402, 1404, and 1406, the major axis lengths corresponding to image objects classified as tumor regions are, on average, larger than the major axis lengths corresponding to image objects classified as non-tumor regions. This observation may indicate that major axis length is more likely to predict the detection of tumor regions from an image compared to other features (e.g., the centroid position shown in FIG. 12).

[0101] 15 shows an example set of box plots 1500 corresponding to the distribution of sizes of image objects detected by the image segmentation subsystem. Each of box plots 1502, 1504, 1506, and 1508 shows a first distribution of sizes for a first set of image objects classified as tumor regions and a second distribution of sizes for a second set of image objects classified as non-tumor regions. Additionally, box plot 1502 shows the distribution of tissue volume of the image objects expressed in voxels; box plot 1504 shows the distribution of surface area of ​​the image objects expressed as distance around the boundary of the corresponding image objects; box plot 1506 shows the distribution of convex volume of the image objects expressed as the volume of the smallest convex polygon containing the corresponding image objects; and box plot 1508 shows the distribution of equivalent diameter of the image objects expressed as the diameter of a sphere with the same number of voxels measured for the corresponding image objects. As shown in box plots 1502, 1504, 1506, and 1508, size characteristics (e.g., volume, surface area, equivalent diameter) corresponding to image objects classified as tumor regions are, on average, larger than size characteristics corresponding to image objects classified as non-tumor regions. Such observations may indicate that size characteristics, compared to other features, are more likely to be predictive for detecting tumor regions from images. For example, box plot 1502 in FIG. 15 shows that objects labeled "tumor" have an average volume that is approximately 10 times higher than non-tumor objects. Thus, box plot 1502 indicates that object volume is a meaningful feature and is likely to be highly predictive of which class an object belongs to when using a trained machine learning model to predict the object class of new images that are not part of the training set.

[0102] FIG. 16 shows an example set of box plots 1600 corresponding to the distribution of shapes of image objects detected by the image segmentation subsystem. Box plots 1602, 1604, and 1606 each show a first distribution of shapes for a first set of image objects classified as tumor regions and a second distribution of shapes for a second set of image objects classified as non-tumor regions. Additionally, box plot 1602 shows the distribution of fractional anisotropy for the image objects. While the median corresponding to fractional anisotropy was similar for image objects in both tumor and non-tumor regions, the interquartile range of the distribution was larger for image objects classified as non-tumor regions. Box plot 1604 shows the distribution of the extent of image objects represented by tissue volume divided by the volume of their bounding boxes. The range across tissue classes was larger for image objects classified as tumor regions. Box plot 1606 shows the distribution of solidity for image objects represented by tissue volume divided by the volume of their convex hulls. The distribution of solidity was larger for image objects classified as non-tumor regions. As shown in box plots 1602, 1604, and 1606, there is a significant difference between the shapes (e.g., extent, solidity) of image objects classified as tumor regions and those classified as non-tumor regions. Such an observation may indicate that object shape features are more likely to be predictive for detecting tumor regions from images compared to other features.

[0103] FIG. 17 illustrates an exemplary set of box plots 1700 corresponding to the distribution of voxel intensities of image objects detected by the image segmentation subsystem. Box plots 1702 and 1704 each illustrate a first distribution of voxel intensities for a first set of image objects classified as tumor regions and a second distribution of voxel intensities for a second set of image objects classified as non-tumor regions. Additionally, box plot 1702 illustrates the distribution of average voxel intensities for the image objects expressed in Hounsfield units, and box plot 1704 illustrates the distribution of maximum voxel intensities for the image objects expressed in Hounsfield units. As illustrated in box plots 1702 and 1704, voxel intensities corresponding to image objects classified as tumor regions are greater than voxel intensities corresponding to image objects classified as non-tumor regions, indicating that voxel intensity features are more likely to predict the detection of tumor regions from an image than other features (e.g., centroid location, as illustrated in FIG. 12 ).

[0104] II.I. Exemplary Schematic of a Classification Subsystem with a Trained Support Vector Machine FIG. 18 shows an example schematic 1800 of a support vector machine for estimating whether an image object corresponds to a tumor, according to some embodiments. A training subsystem 1805 can train a support vector machine 1810 into a trained support vector machine 1815. To train the support vector machine 1810, the training subsystem 1805 can use training data 1820, which can include a set of structural features corresponding to each training image object. Each set of structural features can be assigned a corresponding label from a set of labels 1825a-n. For example, a first set of structural features corresponding to a first training image object in the training data 1820 can correspond to label 1825a, and a second set of structural features corresponding to a second training image object in the training data 1820 can correspond to label 1825b. Each label from the set of labels 1825a-n can indicate whether its corresponding set of structural features is indicative of a tumor. Thus, a set of structural features can be used to characterize the training image for each tumor in the training image. In practice, trained machine learning models can be used to detect individual tumor regions based on their structural characteristics (e.g., tissue volume), and in some cases, these individual tumor regions are aggregated to identify a measurement across the training images, such as the level of tumor burden.

[0105] II.I.1 Example Training Data for Training Support Vector Machines As described herein, the training data 1820 may correspond to structural characteristics corresponding to each training image object in the training images. The training data 1820 may represent a plurality of training images corresponding to scans of a subject. Each of the training images may be labeled as having no tumor, a low tumor burden, a medium tumor burden, or a high tumor burden. For example, the training data 1820 included 71 scans, consisting of 21 images showing no tumor, 20 images showing low tumor burden, 15 images showing medium tumor burden, and 15 images showing high tumor burden. In this example, the level of tumor burden may be labeled for each training image based on the following criteria: [Table 1]

[0106] From the 71 scanned images, 12 images were identified as a holdout set for testing. Furthermore, the training data 1820 may contain 1941 training image objects, of which 364 training image objects were identified as a holdout set for testing.

[0107] II.I.2 Feature Selection A support vector machine 1810 (or other machine learning model, such as a gradient boosting algorithm) can be trained using a subset of the set of structural features associated with each training image object. Such feature selection techniques reduce the number of input variables for training the machine learning model, allowing for efficient use of computational resources and potentially improving the performance of the trained machine learning model by reducing noise.

[0108] For example, the subset of structural properties can be selected by processing the set of structural properties of the training data 1820 and generating, for a structural property category (e.g., orientation, centroid location), a first distribution of structural property values ​​corresponding to tumor regions and a second distribution of structural property values ​​corresponding to non-tumor regions. A statistical difference between the two distributions can be identified. For example, the statistical difference corresponds to a difference in median or mean between the first distribution and the second distribution. The statistical difference between the two distributions can be analyzed to determine whether structural property values ​​of the corresponding category can be added as elements of the subset of structural properties. If so, the structural property of the image object associated with the corresponding category can be added as elements of the subset of structural properties of the image object. In some cases, the subset of structural properties includes values ​​corresponding to structural property categories such as the volume, surface area, equivalent diameter, and voxel intensity of the image object.

[0109] Additionally or alternatively, the subset of structural features can be selected by processing the set of structural features of the training data 1820 to generate a correlation metric (e.g., a Pearson correlation coefficient) for each structural feature category. The correlation metric can be compared to a predetermined threshold (e.g., 0.7). In response to determining that the correlation metric exceeds the predetermined threshold, structural feature values ​​of image objects associated with the corresponding structural feature category can be added as elements of the subset. In some cases, the subset of structural features can be selected by generating a SHapley Additive Explanations plot that identifies the contribution of each structural feature category to arriving at a tumor classification.

[0110] II.I.3 Support Vector Machines For each set of structural properties in the training data 1820, the training subsystem 1805 can use values ​​(e.g., shape, volume) from the set of structural properties to plot the set on a multidimensional graph representing the support vector machine 1810. As a result, training image objects corresponding to the set of structural properties (with associated labels) can be identified in the multidimensional graph corresponding to the support vector machine 1810. The training subsystem 1805 can then train the support vector machine 1810 by generating an optimal hyperplane that separates the set of structural properties assigned a tumor label from the set of structural properties assigned a non-tumor label. In some cases, a K-fold cross-validation technique is used when optimizing one or more hyperparameters of the support vector machine. For example, a Bayesian hyperparameter optimization technique can be used when optimizing one or more hyperparameters. Examples of hyperparameters can include: [Table 2]

[0111] The trained support vector machine with the optimal hyperplane can then be tested using a holdout data set (e.g., 21 images from the 71 total scanned images). As a result of generating an optimal hyperplane, the training subsystem 1805 can determine that the support vector machine 1810 is adequately trained for classification. The trained support vector machine 1815 with the optimal hyperplane can then be used by the classification subsystem 1830.

[0112] The classification subsystem 1830 can process the information stored in the data structure 1835 by using the trained support vector machine 1815 to estimate whether the image objects represented in the data structure 1835 correspond to a tumor. For the data structure 1835, the classification subsystem 1830 can access a set of structural features corresponding to the image objects and plot values ​​corresponding to the structural features in a multidimensional graph having a hyperplane that best fits the trained support vector machine 1815. In some cases, a feature vector is generated to plot the corresponding set of structural features in the multidimensional graph. Once the set of structural features is plotted, the trained support vector machine 1815 can generate inferred data indicating whether the image corresponds to a tumor based on the graph position of the data structure 1835 relative to the hyperplane that best fits the trained support vector machine 1815. The classification subsystem 1830 can collect the inferred data corresponding to the image objects represented in the data structure 1835 and generate an output 1840 including information indicating whether each image object corresponds to a tumor, a lesion, or normal tissue. In some cases, output 1840 identifies the image object as either a tumor or a non-tumor region. Additionally or alternatively, output 1840 can identify the image object as a tumor, a blood vessel, or another object type.

[0113] The classification system 1830 can generate another output (not shown) that identifies the level of tumor burden corresponding to the biological structure depicted in the input image. The level of tumor burden can be determined by aggregating image objects classified as having tumor and generating a structural feature value (e.g., tumor volume) that represents the aggregated image object. The generated structural feature value can then be compared to a reference structural feature value that represents all image objects (e.g., lung volume) segmented and identified from the image. Such a comparison can identify the proportion and / or ratio of the aggregated image object to all image objects in the image. In some cases, the level of tumor burden identifies a low tumor burden (e.g., less than 20% of the total lung volume), a moderate tumor burden (e.g., 20-40% of the total lung volume), or a high tumor burden (e.g., greater than 40% of the total lung volume). Additionally or alternatively, the classification system 1830 can identify a numerical value that represents the level of tumor burden in the image, such as a percentage of lung volume. Other outputs can be used to diagnose the level of cancer in a patient, identify the type of treatment for the subject corresponding to the image, and / or determine the subject's diagnostic or prognostic factors.

[0114] II.I.4 Illustrative Results Figure 19 shows an example graph 1900 identifying the relationship between lung volume detected from an image and total tumor volume. As shown in graph 1900, each plot point represents the amount of measured tumor volume (x-axis) and the amount of lung volume detected by the trained 3D U-Net (y-axis) for the corresponding image. The y-intercept is shown as 0.24 mL, the slope is 0.92, and the R2 is 0.79. Each image is classified as having no tumor burden, low tumor burden (<20% of lung volume), moderate tumor burden (20-40% of lung volume), or high tumor burden (>40% of lung volume). Graph 1900 shows that the amount of lung volume detected by the trained 3D U-Net increases approximately proportionally with increasing tumor burden. Such an increase may indicate significant volumetric expansion as the lungs adapt to tumor growth. Graph 1900 also demonstrates that the trained machine learning model can be used to consistently capture changes in total lung volume under different physiological / disease conditions. Thus, the consistent and accurate performance of machine learning models can be reliably used to track the progression of (for example) cancer and other non-oncological diseases.

[0115] II.J. Exemplary Schematic of a User Interface for Labeling Training Data 20A-20C show example screenshots of a user interface 2000 for labeling training image objects corresponding to three-dimensional image data, according to some embodiments. As described herein, a training subsystem (e.g., training subsystem 115 of FIG. 1) can identify boundaries associated with each region represented in a training image of the training data. The regions can correspond to three-dimensional image objects in the training images. Each three-dimensional image object of the set can be labeled via the user interface 2000 and used as training data for training a machine learning model.

[0116] In the user interface 2000, each object in a set of three-dimensional image objects corresponding to a three-dimensional training image can be represented by a different color. Two-dimensional portions (e.g., slices) rendered from the three-dimensional training image can be presented simultaneously with the three-dimensional training image. Additionally or alternatively, the user interface 2000 can be used to scroll through a stack of two-dimensional portions rendered from the three-dimensional training image. For example, the three-dimensional training image can be displayed in a first portion of the user interface 2000, and the two-dimensional portions can be simultaneously displayed in a second portion of the user interface 2000. The two-dimensional portions can show regions corresponding to one or more image objects in the three-dimensional training image. For example, regions corresponding to image objects can be identified (e.g., via highlighting and / or border overlay) over a portion of the two-dimensional portion. FIG. 20A shows a first screenshot of the user interface 2000 showing three-dimensional training images corresponding to subjects with low tumor burden. FIG. 20B shows a second screenshot of the user interface 2000 showing three-dimensional training images corresponding to subjects with medium to high tumor burden. FIG. 20C shows a third screenshot of the user interface 2000 showing three-dimensional training images corresponding to subjects with very high tumor burden.

[0117] II.K. Exemplary Process for Automated Tumor Detection Based on Image Processing 21 illustrates a process 2100 for processing an image to estimate whether at least a portion of a tumor is represented in the image, according to some embodiments. Process 2100 begins at block 2105, where an image showing at least a portion of a biological structure is accessed. The biological structure may refer to one or more types of tissue represented in the image. For example, the biological structure may include individual organs such as the lung, heart, or liver, various types of tissue (e.g., bone, blood vessel, tumor), and / or any structure (e.g., lesion) that shows an alteration of at least a portion of the biological structure.

[0118] The accessed images may be acquired using an imaging system and may include or be derived from data received from the imaging system. The imaging system may include a CT system, which may include a tomographic imager and / or a micro-CT component (or a micro-tomosynthesis component).

[0119] In block 2110, the images are processed using a segmentation algorithm to extract image objects from a set of image objects depicted in the images. Each of the set of image objects may depict a particular type of biological structure (e.g., tissue, tumor, blood vessel). The images may be preprocessed (e.g., a negative distance transform operation) to identify boundaries within the images and use the boundaries to extract a set of image objects from the images. Various segmentation algorithms may be used to extract image objects from the aligned images, including watershed segmentation algorithms, graph segmentation algorithms, and model-based segmentation algorithms.

[0120] In block 2115, structural properties associated with each image object are determined. Structural properties may refer to morphological features associated with the image object that can be measured using any technique known to those skilled in the art. For example, structural properties may include diameter, surface area, shape, median convex volume, equivalent diameter, orientation, solidity, and / or volume associated with the image object. The structural properties may be stored in a data structure to which an identifier can be assigned to identify the image object.

[0121] In block 2120, the structural characteristics are processed using a trained machine learning model to generate a classification metric corresponding to an estimation of whether the image object corresponds to a lesion or a tumor. The trained machine learning model can identify whether the structural characteristics of the image object correspond to pre-identified structural characteristics associated with a particular biological structure (e.g., tumor, lung, blood vessel). Based on the identification, a classification metric can be generated to estimate whether a particular image corresponds to a tumor or a lesion.

[0122] At block 2125, the classification metric is output. For example, the classification metric can be presented locally or transmitted to another device. The classification metric can be output along with an identifier for the image. The classification metric can be output along with a probability metric corresponding to a confidence level associated with the classification metric.

[0123] 22 illustrates a process 2200 for processing an image to extract a set of image objects according to some embodiments. Process 2200 begins at block 2205, where an image showing at least a portion of a biological structure is accessed. The biological structure may refer to one or more types of tissue represented in the image. For example, the biological structure may include individual organs such as the lung, heart, or liver, various types of tissue (e.g., bone, blood vessel, tumor), and / or any structure (e.g., lesion) that shows an alteration of at least a portion of the biological structure.

[0124] The accessed images may be acquired using an imaging system and may include or be derived from data received from the imaging system. The imaging system may include a CT system, which may include a tomographic imager and / or a micro-CT component (or a micro-tomosynthesis component).

[0125] At block 2210, a filter is applied to the image to separate the biological structure of interest from the background of the image. For example, a first filter can be a lung mask that can be applied to separate the biological structure (e.g., lung) from the image. The background can include a region depicted in the image that excludes the biological structure of interest (e.g., bone). To apply the filter, at least a portion of the background can be used as a boundary surrounding the biological structure, and the boundary can be used to adjust the size of the region corresponding to the biological structure of interest. In some cases, a trained image preprocessing machine learning model is used to process the accessed image to identify or isolate an ROI, where the ROI can include organs, tissues, tumors, and blood vessels depicted in the accessed image.

[0126] In block 2215, the filtered image is converted into a binary image. Each pixel of at least a portion of the filtered image can be converted to a pixel value of either 0 or 1. For example, the filtered image can be a grayscale image that can be converted into a binary image. Binary values ​​corresponding to a subset of pixels (e.g., 0 pixels surrounded by 1 pixels) can be converted based on a fill operation. To further reduce image noise in the binary image, erosion-dilation and / or hole-filling operations can be applied.

[0127] At block 2220, a registration operation is performed to align the binary image to the reference image. The filtered image can be registered based on a reference filter corresponding to the reference image, such that a set of transformation parameters is generated. An image distortion operation can then be performed by applying the set of transformation parameters to the binary image, such that its position and orientation are adjusted. Subsequent registration operations can be performed on the warped binary image based on different sets of transformation parameters generated based on registering the filtered image to the reference image.

[0128] A segmentation algorithm is used to extract a set of image objects from the aligned images at block 2225. To extract the set of image objects, the aligned images can be eroded to identify the set of image objects, and seed points can be created for each image object and conditionally expanded until the aligned images can be filled.

[0129] The transformed aligned images can be processed by a segmentation algorithm to generate a segmented image that identifies a set of lines that can be used as boundaries of a set of image objects depicted in the aligned images. Based on the boundaries, the set of image objects can be extracted from the segmented image.

[0130] At block 2230, the set of image objects is output. For example, the set of image objects may be presented locally or transmitted to another device. The set of image objects may be output along with an identifier for the image. The set of image objects may also be processed by a trained machine learning model to generate a classification metric corresponding to an estimation of whether the image object corresponds to a lesion or tumor.

[0131] III. Further Considerations Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium containing instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0132] The terms and expressions which have been employed are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0133] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of preferred exemplary embodiments provides those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0134] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

Claims

1. 1. A computer-implemented method comprising: accessing an image of at least a portion of a biological structure of a particular subject; processing the image using a segmentation algorithm to extract a plurality of image objects depicted in the image; determining one or more structural characteristics associated with a graphical object of the plurality of graphical objects; processing the one or more structural features using a trained machine learning model to generate inferred data corresponding to an estimation of whether the image object corresponds to a lesion or tumor associated with the biological structure, wherein the trained machine learning model is trained with a three-dimensional model constructed from a set of training images; and outputting the estimated data for the particular subject; 11. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , wherein the at least a portion of the biological structure comprises at least a portion of a lung.

3. The computer-implemented method of claim 1 or 2, wherein the image depicts a skeletal structure surrounding the at least a portion of the biological structure.

4. The computer-implemented method of claim 1 , wherein the image depicts a cross-section of the at least a portion of the biological structure.

5. The computer-implemented method of claim 1 , wherein the image comprises or is derived from image data captured using a computed tomography scanner.

6. The computer-implemented method of claim 5 , wherein the computed tomography scanner is a micro-computed tomography scanner.

7. The computer-implemented method of claim 1 , wherein the one or more structural characteristics identify the shape, position, surface area, and / or longest diameter of the image object.

8. processing the image using the segmentation algorithm; determining one or more structural characteristics associated with the other image objects; processing the one or more structural properties of the other image objects using the trained machine learning model to generate inferred data corresponding to an inference of whether the other image objects correspond to the type of biological structure; and outputting the estimated data associated with the other image object; The computer-implemented method of claim 1 , further comprising:

9. The computer-implemented method of claim 8 , wherein the types of biological structures include blood vessels, lungs, hearts, and / or livers.

10. The computer-implemented method of claim 1 , wherein the segmentation algorithm is a watershed transformation algorithm.

11. The computer-implemented method of claim 1 , wherein the trained machine learning model is a trained support vector machine (SVM).

12. 1. A computer-implemented method comprising: moreover, preprocessing the image using a trained image preprocessing machine learning model to generate a filter for identifying one or more image regions corresponding to the biological structure; applying the filter to the image to isolate the one or more image regions; and processing the separated image regions using the segmentation algorithm to extract the plurality of image objects depicted in the image; 12. The computer-implemented method of claim 1, comprising:

13. 13. The computer-implemented method of claim 12, wherein the trained image preprocessing machine learning model is a trained convolutional neural network machine learning model.

14. 14. The computer-implemented method of claim 1, further comprising processing the image using an image filter to generate a filtered image that excludes one or more skeletal structures surrounding the at least a portion of the biological structure, the filtered image being used in place of the image to separate the image object from the plurality of image objects.

15. 15. The computer-implemented method of claim 1, further comprising processing the image using co-registration to align the image with a reference image, the aligned image being used in place of the image to extract the image object from the plurality of image objects.

16. 1. A computer-implemented method comprising: moreover, converting the image to a binary image; Transforming one or more pixels of the binary image by using a fill operation; and performing erosion and dilation operations to reduce image noise from said binary image; 16. The computer-implemented method of claim 1, comprising:

17. 17. The computer-implemented method of claim 1, wherein processing the image using a segmentation algorithm further comprises applying a negative distance transform function to the image to identify boundaries between two or more overlapping image objects of the plurality of image objects.

18. 18. The computer-implemented method of claim 1, further comprising generating estimated data for the plurality of image objects by performing determining, processing, and outputting steps for each remaining image object of the plurality of image objects.

19. 20. The computer-implemented method of claim 18, further comprising determining a level of tumor burden associated with the at least a portion of the biological structure based on the estimated data for the plurality of image objects.

20. 20. The computer-implemented method of claim 19, wherein the level of tumor burden corresponds to a ratio of a first structural property value derived from one or more image objects of the plurality of image objects classified as having the tumor to a second structural property value derived from all of the plurality of image objects.

21. the one or more structural properties correspond to a set of structural properties of the graphical object; Processing the one or more structural features using the trained machine learning model further includes selecting a subset of structural features from the set of structural features, the subset of structural features comprising: generating, for a structural property category associated with a particular structural property of the set of structural properties, a first distribution of structural property values ​​corresponding to a first set of image objects identified as having the tumor and a second distribution of structural property values ​​corresponding to a second set of image objects identified as not having the tumor, wherein the plurality of image objects includes the first set and the second set of image objects; identifying a statistical difference between the first distribution and the second distribution; determining, based on the statistical difference, that the structural property category should be added as a category of the subset of structural properties; and and in response to determining that the structural property category should be added as the category of the subset of structural properties, adding the particular structural property as an element of the subset of structural properties.

22. 22. The computer-implemented method of claim 21, wherein the subset of structural properties identifies the volume, surface area, equivalent diameter, and voxel intensity of the image object.

23. 1. A system comprising: one or more data processors; a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein.

24. A computer program product tangibly embodied in a non-transitory machine-readable storage medium comprising instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

Images