Three-dimensional object segmentation of medical images localized with object detection

Abstract

To provide techniques for segmenting objects within medical images using a deep learning network that is localized with object detection based on a derived contrast mechanism.SOLUTION: Aspects of the present invention are directed to localizing an object of interest within a first medical image having a first characteristic, projecting a bounding box or segmentation mask of the object of interest onto a second medical image having a second characteristic to define a portion of the second medical image, and inputting the portion of the second medical image into a deep learning model that is constructed as a detector using a weighted loss function capable of segmenting the portion of the second medical image and generating a segmentation boundary around the object of interest. The segmentation boundary may be used to calculate a volume of the object of interest for determining a diagnosis and / or a prognosis of a subject.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 886,844, filed August 14, 2019, the contents of which are incorporated herein by reference in their entirety for all purposes.

[0002] The present disclosure relates to automated object segmentation of medical images, and in particular to techniques for segmenting objects in medical images that have been localized by object detection based on derived contrast mechanisms using deep learning networks. [Background technology]

[0003] Computer vision involves working with digital images and videos to infer some understanding of the content within these images and videos. Object recognition is associated with computer vision and refers to a set of related computer vision tasks that involve identifying objects present within an image frame. The tasks include image classification, object localization, object detection, and object segmentation. Image classification involves predicting the class of one or more objects in an image frame. Object localization refers to identifying the location of one or more objects in an image frame and drawing a boundary around their extent. Object detection combines these two tasks and locates and classifies one or more objects in an image frame. Object segmentation involves highlighting specific pixels of a located or detected object (generating a mask) instead of a coarse bounding box. Techniques for object recognition are generally categorized as either machine learning-based or deep learning-based techniques. In machine learning-based approaches to object localization and detection, features in an image are first defined using feature descriptors such as Haar-like features, scale-invariant feature transforms, or histograms of oriented gradients (HOG), and then objects of interest are detected based on the feature descriptors using techniques such as support vector machines (SVMs). On the other hand, deep learning techniques can perform end-to-end object detection and segmentation without specifically defining features, and are typically based on convolutional neural networks (CNNs) such as region-based networks (R-CNN, Faster R-CNN, Faster R-CNN, and Cascade R-CNN). Summary of the Invention

[0004] In some embodiments, a computer-implemented method for segmenting objects in a medical image is provided, the method including acquiring medical images of a subject, the medical images including a first image having a first characteristic and a second image having a second characteristic, the medical images being generated using one or more medical imaging modalities, locating and classifying objects in the first image into a plurality of object classes using a localization model, the classifying including assigning a set of pixels or voxels of the first image to one or more of the plurality of object classes, and localizing and classifying objects in the first image based on the set of pixels or voxels assigned to the object classes of the plurality of object classes using the localization model. the first image includes a first region of the first image and a second region of the second image including the object of interest; a second region of the first image including the object of interest; a third region of the first image including the object of interest; a fourth region of the first image including the object of interest; a fifth region of the first image including the object of interest; a fifth region of the first image including the object of interest;

[0005] In some embodiments, the one or more medical imaging modalities include a first medical imaging modality and a second medical imaging modality different from the first medical imaging modality, and the first image is generated from the first medical imaging modality and the second image is generated from the second medical imaging modality.

[0006] In some embodiments, the one or more medical imaging modalities include a first medical imaging modality and a second medical imaging modality that is the same as the first medical imaging modality, and the first image is generated from the first medical imaging modality and the second image is generated from the second medical imaging modality.

[0007] In some embodiments, the first image is a first type of image and the second image is a second type of image, and the first type of image is different from the second type of image.

[0008] In some embodiments, the first image is a first type of image and the second image is a second type of image, and the first type of image is the same as the second type of image.

[0009] In some embodiments, the first characteristic is different from the second characteristic.

[0010] In some embodiments, the first characteristic is the same as the second characteristic.

[0011] In some embodiments, the first medical imaging modality is magnetic resonance imaging, diffusion tensor imaging, computed tomography, positron emission tomography, photoacoustic tomography, x-ray, sonography, or a combination thereof, and the second medical imaging modality is magnetic resonance imaging, diffusion tensor imaging, computed tomography, positron emission tomography, photoacoustic tomography, x-ray, sonography, or a combination thereof.

[0012] In some embodiments, the first type of image is a magnetic resonance image, a diffusion tensor image or map, a computed tomography image, a positron emission tomography image, a photoacoustic tomography image, an X-ray image, a sonography image, or a combination thereof, and the second type of image is a magnetic resonance image, a diffusion tensor image or map, a computed tomography image, a positron emission tomography image, a photoacoustic tomography image, an X-ray image, a sonography image, or a combination thereof.

[0013] In some embodiments, the first characteristic is anisotropy ratio contrast, average diffusivity contrast, axial diffusivity contrast, radial diffusivity contrast, proton density contrast, T1 relaxation time contrast, T2 relaxation time contrast, diffusion coefficient contrast, low resolution, high resolution, drug contrast, radioactive tracer contrast, optical absorption contrast, echo distance contrast, or a combination thereof, and the second characteristic is anisotropy ratio contrast, average diffusivity contrast, axial diffusivity contrast, radial diffusivity contrast, proton density contrast, T1 relaxation time contrast, T2 relaxation time contrast, diffusion coefficient contrast, low resolution, high resolution, drug contrast, radioactive tracer contrast, optical absorption contrast, echo distance contrast, or a combination thereof.

[0014] In some embodiments, the one or more medical imaging modalities is diffusion tensor imaging, the first image is a fractional anisotropy (FA) map, the second image is a mean diffusivity (MD) map, the first property is anisotropy ratio contrast, the second property is mean diffusivity contrast, and the object of interest is a kidney of interest.

[0015] In some embodiments, locating and classifying the object in the first image includes applying one or more clustering algorithms to a plurality of pixels or voxels of the first image.

[0016] In some embodiments, the one or more clustering algorithms include a k-means algorithm that assigns observations to clusters associated with multiple object classes.

[0017] In some embodiments, the one or more clustering algorithms further include an Expectation Maximization algorithm that calculates probabilities of cluster membership based on one or more probability distributions, and the k-means algorithm initializes the Expectation Maximization algorithm by estimating initial parameters for each object class of the multiple object classes.

[0018] In some embodiments, a segmentation mask is determined, where determining the segmentation mask includes identifying seed locations for the object of interest using a set of pixels or voxels assigned to an object class, growing the seed locations by projecting the seed locations toward a z-axis representing a depth of the segmentation mask, and determining the segmentation mask based on the projected seed locations.

[0019] In some embodiments, determining the segmentation mask further comprises performing morphological closing and filling on the segmentation mask.

[0020] In some embodiments, the method further includes cropping the second image based on the object mask and the margin to generate a portion of the second image prior to inputting the portion of the second image into the three-dimensional neural network model.

[0021] In some embodiments, the method further includes inputting the second image into a deep super-resolution neural network to increase resolution of the portion of the second image prior to inputting the portion of the second image into the three-dimensional neural network model.

[0022] In some embodiments, the three-dimensional neural network model includes a plurality of model parameters identified using a set of training data, the set of training data including a plurality of medical images having annotations associated with a segmentation boundary around the object of interest, and a plurality of additional medical images having annotations associated with a segmentation boundary around the object of interest, the plurality of additional medical images being artificially generated by matching image histograms from the plurality of medical images to image histograms from a plurality of reference maps, and the plurality of model parameters identified using the set of training data based on minimizing a weighted loss function.

[0023] In some embodiments, the weighted loss function is a weighted Dice loss function.

[0024] In some embodiments, the three-dimensional neural network model is a modified 3D U-Net model.

[0025] In some embodiments, the modified 3D U-Net model includes a total of 5,000,000 to 12,000,000 learnable parameters.

[0026] In some embodiments, the modified 3D U-Net model includes a total of 800 to 1,700 kernels.

[0027] In some embodiments, the method further includes determining a size, surface area, and / or volume of the object of interest based on an estimated boundary around the object of interest, and providing (i) a portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) the size, surface area, and / or volume of the object of interest.

[0028] In some embodiments, the method further includes determining, by the user, a diagnosis of the subject based on (i) a portion of the second image having an estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest.

[0029] In some embodiments, the method further includes acquiring, by a user using an imaging system, a medical image of the subject, where the imaging system generates the medical image using one or more medical imaging modalities; determining a size, surface area, and / or volume of the object of interest based on the estimated segmentation boundary around the object of interest; providing (i) a portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) the size, surface area, and / or volume of the object of interest; receiving, by the user, (i) a portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) the size, surface area, and / or volume of the object of interest; and determining, by the user, a diagnosis of the subject based on (i) the portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) the size, surface area, and / or volume of the object of interest.

[0030] In some embodiments, the method further includes administering, by a user, a treatment with a compound based on (i) a portion of the second image having an estimated segmentation boundary around the object of interest, (ii) a size, surface area, and / or volume of the object of interest, and / or (iii) a diagnosis of the subject.

[0031] In some embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable storage medium that includes instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0032] In some embodiments, a computer program product is provided that is tangibly embodied in a non-transitory machine-readable storage medium and includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0033] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause the one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein.

[0034] The terms and expressions used are used as terms of description and not of limitation, and in the use of such terms and expressions there is no intention to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the invention as claimed has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims. [Brief description of the drawings]

[0035] The present disclosure is described in conjunction with the accompanying drawings, in which:

[0036] [Figure 1] 1 illustrates an exemplary computing environment for segmenting instances of an object of interest in accordance with various embodiments.

[0037] [Diagram 2] 13 illustrates histogram matching used to simulate other contrasts and increase the variance of a training dataset, according to various embodiments.

[0038] [Diagram 3] 1 illustrates an exemplary U-Net in accordance with various embodiments.

[0039] [Figure 4] 1 illustrates a process for segmenting an instance of an object of interest according to various embodiments.

[0040] [Figure 5A] 1 illustrates a diffusion tensor element according to various embodiments.

[0041] [Figure 5B] 1 illustrates partial anisotropic images used for expectation-maximization (EM) segmentation (12 classes) and object detection steps according to various embodiments.

[0042] [Figure 5C] 1 illustrates super-resolution images in the slice direction according to various embodiments.

[0043] [Figure 6A] 1 shows segmentation results using different strategies. [Figure 6B] 1 shows segmentation results using different strategies. [Figure 6C] 1 shows segmentation results using different strategies. [Figure 6D] 1 shows segmentation results using different strategies. [Figure 6E] Figure 6 shows segmentation results using various strategies. Figure 6A: 3D U-Net. Figure 6B: Detecting foreground by connected component preprocessing. Figure 6C: EM segmentation. Figure 6D: Kidney detection by EM segmentation. Figure 6E: Kidney detection by EM segmentation on super-resolution image. The first row shows the ground truth manual labels overlaid on the magnetic resonance imaging (MRI). The second row shows the transparent surface rendering of the ground truth and segmentation masks. Coronal and axial views are shown pairwise. The third row shows the Dice Similarity Coefficient (DSC) shown as a violin plot. An example dataset was selected based on the average DSC of each segmentation strategy. All segmentation results are 3D U-Net based except for C, which is only EM segmentation. The dashed boxes indicate the regions for object detection. Scale bar = 4m.

[0044] In the accompanying drawings, similar components and / or features may have the same reference label. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes the similar components. If only a first reference label is used in this specification, the description is applicable to any of the similar components having the same first reference label, regardless of the second reference label. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] I. Overview This disclosure describes techniques for automatic object segmentation of medical images. More specifically, embodiments of the disclosure provide techniques for segmenting objects in medical images using deep learning networks localized by object detection based on derived contrast mechanisms.

[0046] Medical image segmentation, which identifies pixels of objects (e.g., organs, lesions or tumors) from background medical images such as computed tomography (CT) or MRI images, is a fundamental task in medical image analysis to provide information about the shape, size, and volume of the object. Changes in the size or volume of an organ can be a major feature of a disease process or a manifestation of pathology elsewhere in the subject. Furthermore, the size or volume of a tumor or lesion can be an important independent indicator in subjects with carcinoma (e.g., repeated size measurements during primary systemic therapy generate detailed information about the response that can be used to select the most effective treatment regimen and estimate the subject's prognosis). Various radiological and clinical techniques have been attempted in the past to estimate the size and volume of tumors or organs, but most have had limited usefulness due to unacceptable accuracy, poor reproducibility or difficulties in obtaining images suitable for size and volume measurements. More recently, several quantitative protocols have been developed and have demonstrated promising results for measuring the size and volume of various organs, lesions, or tumors from various imaging modalities such as CT and MRI. Size and volume data for this purpose are typically obtained using time-consuming manual segmentation methods. However, the promising capabilities of deep learning techniques have made these techniques the primary choice for image segmentation, especially medical image segmentation, greatly improving the practicality of obtaining size and volume data for quantitative protocols.

[0047] Although the use of deep learning techniques is beneficial for object segmentation and achieves significant improvements compared to traditional manual and machine learning-based techniques, object segmentation remains challenging when it comes to low-contrast and low-resolution images, which are especially common in medical images such as MRI and CT images. The main reasons for this challenge are due to: (i) the definition of objects such as organs, lesions, or tumors is strongly affected by the blurred visual identity and excessive noise in low-contrast and low-resolution images, which misguids deep learning models to predict the true contours of the objects; (ii) due to the iterative pooling operations in deep learning architectures, it is inevitable to lose the semantic and image structure information of objects, which is severely lacking in low-contrast and low-resolution images; therefore, the results of deep learning models typically suffer from inaccurate shapes and poor localization of objects; and (iii) the pixels around the object boundary are often too close to the center of a similar receptive field. Keep in mind that deep learning models only distinguish between binary labels of image pixels, so it is difficult for deep learning algorithms to learn boundary knowledge; (iv) many medical images have large fields of view or backgrounds that prevent aliasing effects, but if the background represents a significant part of the image, deep learning may not be optimally trained to segment the foreground object of interest (the "background effect"); (v) there may be objects of similar appearance in the background (e.g., the liver, heart, and kidneys may look like tumors), and deep learning and simpler machine learning algorithms may not be optimally trained to distinguish between these structures.

[0048] To address these limitations and issues, the technique for automated object segmentation of the present embodiment uses various imaging modalities and / or medical image types with different characteristics (such as contrast or resolution, characteristics that make the object (or its representation in an image or display) distinguishable) as a derived contrast mechanism to locate the object of interest using a deep learning model, isolate the object of interest, and subsequently segment the object of interest. For example, a first image of an object obtained by a first imaging modality may have a first characteristic (e.g., good contrast) that works well to provide a general outline of the object so that the first image can be used for object detection (providing a coarse grain around the object and classification). However, this first image may be so unclear or blurred that the deep learning network cannot accurately determine where exactly the edges of the object are for accurate object segmentation. In contrast, a second image of an object acquired using a second imaging modality, or a second image of image features / contrast features of the same modality, may have a second characteristic (e.g., high resolution) that works well to provide a well-defined boundary of the object so that the second image can be used for edge detection and fine object segmentation. Once an object is detected in the first image, the coarse grain boundary of the object is projected onto the second image to localize the object in the second image. The coarse grain boundary of the object on the second image is then used to crop the second image prior to object segmentation. The localization and cropping of the second image mitigates background effects and allows the deep learning model to focus on the edges of the object to learn boundary knowledge for fine-grained object segmentation.

[0049] An exemplary embodiment of the present disclosure is a method that includes first locating an object of interest, such as an organ, tumor, or lesion, in a first medical image having a first characteristic (e.g., using an algorithm such as expectation maximization), projecting a bounding box or segmentation mask of the object of interest onto a second medical image having a second characteristic to define a portion of the second medical image that includes the object of interest, and subsequently inputting the portion of the second medical image into a deep learning model, such as a convolutional neural network model, configured as a detector using a weighted loss function that can segment the portion of the second medical image and generate a segmentation boundary around the object of interest. The segmentation boundary can be used to calculate a volume of the object of interest to determine a diagnosis and / or prognosis. In some examples, the calculated volume may be further associated with a time point. The volume of the object from a time point can be compared to the volume of the object from a previous time point to determine the effectiveness of a treatment. The time point analysis provides a picture of the changes of the organ or tumor over time. Additionally, certain contents within the object defined by the segmentation boundary may have changes, e.g., more necrotic contents or aggressive tumor types. In some examples, the segmentation boundary and corresponding segmented region or volume can be used to quantify image metrics such as image intensity. For example, in PET there is standardized uptake value (SUV), or in MRI there is diffusivity, T2, T1, etc. that correlate with certain image metrics such as image intensity, and thus quantification of image metrics can be used to determine values / metrics such as SUV that are specific to the object of interest. In other words, the volume within the segmentation boundary itself is a useful measurement, and so are the values ​​or measurements within the segmentation boundary and corresponding segmented region.

[0050] Advantageously, these techniques utilize multiple medical images with various characteristics and object detection techniques to detect the general region of the object of interest before attempting to segment the object of interest using a deep learning model. This reduces background effects, reduces the complexity of the input data, and allows the deep learning model to focus on the edges of the object to learn boundary knowledge for object segmentation. Furthermore, since the complexity of the input data is reduced, the complexity of the deep learning model can be reduced (e.g., by reducing the number of kernels per convolutional layer). In some examples, the deep learning model is constructed with a weighted loss function that minimizes segmentation error, improves optimization of training performance, and further reduces background effects that may still be evident in some general regions determined by the object detection technique. II. Definition

[0051] As used herein, when an action is "based on" something, this means that the action is based at least in part on at least a part of the something.

[0052] As used herein, the terms "substantially," "approximately," and "about" are defined as being largely, but not necessarily entirely, specified (and including that which is entirely specified), as would be understood by one of ordinary skill in the art. In any disclosed embodiment, the terms "substantially," "approximately," or "about" can be replaced with "within [a percentage]" of that which is specified, where the percentage includes 0.1, 1, 5, and 10%.

[0053] As used herein, a "mask" refers to an image that represents the surface area of ​​a detected object. A mask may include pixels of non-zero intensity that indicate one or more regions of interest (e.g., one or more detected objects) and pixels of zero intensity that indicate the background.

[0054] As used herein, a "binary mask" refers to a mask in which each pixel value is set to one of two values, e.g., 0 or 1. An intensity value of zero can indicate that the corresponding pixel is part of the background, and a non-zero intensity value (e.g., a value of 1) can indicate that the corresponding pixel is part of the region of interest.

[0055] "Classification," as used herein, refers to the process of receiving an input (e.g., an image or portion of an image) and outputting a class (e.g., "organ" or "tumor") or the probability that the input is a particular class. This can include binary classification (member of a class or not), multi-class classification (assignment to one or more classes), providing a probability of membership in each class (e.g., there is a 90% probability that this input is an organ), and similar classification schemes.

[0056] As used herein, "object localization" or "object detection" refers to the process of detecting instances of a particular class of object within an image.

[0057] As used herein, a "bounding box" refers to a rectangular box that represents the general location of a particular class of object in an image. A bounding box may be defined by the x and y axis coordinates of the upper left and / or upper right corners of the rectangle, and the x and y axis coordinates of the lower right and / or lower left corners.

[0058] As used herein, a "segmentation boundary" refers to an estimated perimeter of an object in an image. The segmentation boundary may be generated during a segmentation process in which image features are analyzed to determine the location of the edges of the object. The segmentation boundary may further be represented by a mask, such as a binary mask.

[0059] As used herein, "segmentation" refers to determining the location and shape of an object in an image. Segmentation can include determining a set of pixels that describe the area or perimeter of an object in an image. Segmentation can include generating a mask, such as a binary mask, of the object. Segmentation can further include processing multiple masks corresponding to the object to generate a 3D mask of the object. III. Derived contrast mechanisms

[0060] The goal of imaging procedures of a subject's anatomical structures (e.g., organs or other human or mammalian tissues) and physiological processes is the generation of image contrast with good spatial resolution. Early developments in medical imaging focused on tissue (proton) density functions and tissue relaxation properties for signal contrast generation, which is the main principle behind conventional MRI. MRI detects signals from protons in water molecules, but can only provide grayscale images in which each pixel contains one integer value. Unless two anatomical regions A and B contain water molecules with different physical or chemical properties, these two regions cannot be distinguished from each other by MRI. Otherwise, no matter how high the image resolution, region A will be indistinguishable from region B. To generate MR contrast based on the physical properties of water molecules, proton density (PD), T1 and T2 relaxation times, and diffusion coefficient (D) are widely used. PD represents the water concentration. T1 and T2 are the signal relaxation (decay) times after excitation and are related to environmental factors such as viscosity and the presence of nearby macromolecules. The diffusion term D represents the thermal (or Brownian) motion of water molecules.

[0061] After initial focus on tissue (proton) density function and tissue relaxation properties, researchers investigated other ways to exploit other properties of the water molecule to generate contrast. Diffusion imaging (DI) was a result of these research efforts. In DI, supplemental MR gradients are applied during image acquisition. The movement of protons during the application of these gradients affects the signal in the image, thereby providing information about molecular diffusion. DI can be performed using several techniques, including diffusion spectral imaging (DSI) and diffusion weighted imaging (DWI).

[0062] DWI is a non-invasive imaging method that uses existing MRI techniques in combination with specialized software and has sensitivity to water diffusion within tissue structures without the need for additional hardware equipment, contrast agents, or chemical tracers. To measure diffusion using MRI, supplemental MR gradients are used to create images that are sensitive to diffusion in specific directions. In DWI, the intensity of each image element (voxel) reflects the best estimate of the water diffusion rate in a particular direction. However, biological tissues are highly anisotropic, which means that their diffusion rates are not the same in all directions. For mundane DWI, tissue anisotropy is often ignored and diffusion is reduced to a single average value, the apparent diffusion coefficient (ADC), which is overly simplistic for many use cases. An alternative method is to model diffusion in a composite material using a diffusion tensor, a [3x3] array of numbers corresponding to the diffusion rate in each combination of directions. The three diagonal elements (Dxx,Dyy,Dzz) represent the diffusion coefficients measured along each of the major (x-, y-, and z-) laboratory axes. The six off-diagonal terms (Dxy, Dyx, etc.) reflect the correlation of random motion between each pair of principal directions.

[0063] The introduction of diffusion tensor models allows the indirect measurement of the degree of anisotropy and structural orientation that characterize diffusion tensor imaging (DTI). The basic concept behind DTI is that water molecules diffuse differently along tissues depending on their type, integrity, structure, and the presence of barriers, giving information about their orientation and quantitative anisotropy. With DTI analysis, it is possible to infer, at each voxel, properties such as the molecular diffusion rate (mean diffusivity (MD) or apparent diffusion coefficient (ADC)), the directional preference of diffusion (fractional anisotropy (FA)), the axial diffusivity (AD) (the diffusion rate along the main axis of diffusion), and the radial diffusivity (RD) (the diffusion rate in the transverse direction). DTI is usually displayed by condensing the information contained in the tensor into one number (a scalar) or four numbers (giving the R, G, B colors and a brightness value known as the color fractional anisotropy). Diffusion tensors can also be viewed using glyphs, which are small three-dimensional (3D) representations of the principal eigenvectors or the entire tensor.

[0064] Similar to MRI and DTI, other modalities of medical imaging such as CT, X-ray, positron emission tomography (PET), photoacoustic tomography (PAT), sonography, and combinations thereof such as PET-CT, PET-MR, etc., rely on various measurements, algorithms, and agents to generate image contrast and spatial resolution. For example, CT and X-ray use X-ray absorption to distinguish between dense structures such as air, soft tissue, and bone. While dense structures in the body stop X-rays and are therefore easily imaged and visualized, soft tissues vary in their ability to stop X-rays and therefore may be faint or difficult to image and visualize. One technique to increase image contrast in X-ray or CT scans is to utilize contrast agents that contain substances that better stop X-rays and make them more visible on the X-ray or CT image, and thus can be used to better visualize soft tissues such as blood vessels. PET uses small amounts of radioactive substances called radiotracers that can be detected and measured in the scan. The measured difference between areas that have accumulated or been labeled with a radioactive tracer and areas that have not accumulated or been labeled is used to generate contrast to visualize structures and functions within a subject. PAT is an imaging modality based on the photoacoustic (PA) effect. A short-pulsed light source is typically used to irradiate tissue, resulting in broadband PA waves. Following absorption of light, an initial temperature rise induces a pressure rise that propagates as a photoacoustic wave and is detected by an ultrasound transducer to image the light absorption contrast. Ultrasound is a non-invasive diagnostic technique used to image inside the body. The transducer sends a beam of sound waves into the body. The sound waves are reflected back to the transducer by boundaries between tissues in the path of the beam (e.g., boundaries between fluid and soft tissue or tissue and bone). When these echoes hit the transducer, they generate an electrical signal that is sent to the ultrasound scanner. Using the speed of sound and the time for each echo to return, the scanner calculates the distance from the transducer to the tissue boundaries. These distances are then used to generate contrast to visualize tissues and organs.

[0065] All of these imaging modalities produce image contrast with sufficient spatial resolution to visualize internal body representations for clinical analysis, medical intervention, and / or medical diagnosis, as well as visual representations of the functions of some organs or tissues. However, as described herein, the image contrast and spatial resolution provided by each of these imaging modalities individually is not sufficient for accurate object segmentation performed by deep learning networks, particularly object segmentation used to obtain size and volume data. To overcome this and other limitations, the techniques described herein use a combination of imaging modalities, image types, and / or various characteristics to locate objects of interest using deep learning models, isolate objects of interest, and subsequently segment the objects of interest. Specifically, it has been discovered that some imaging modalities, image types, and / or characteristics perform better with object detection compared to object segmentation, while other imaging modalities, image types, and / or characteristics perform better with object segmentation. By identifying which imaging modalities, image types, and / or characteristics are better suited for which computer vision tasks (e.g., object detection or object segmentation), it is possible to exploit these differences as derived contrast mechanisms.

[0066] Image characteristics that can be exploited by the derived contrast mechanism include brightness, contrast, and spatial resolution. Brightness (or luminance) is a measure of the relative intensity values ​​across a pixel array after the image is captured by a digital camera or digitized by an analog-to-digital converter. Higher relative intensity values ​​result in brighter pixels and generally appear whiter in the image. On the other hand, lower relative intensity values ​​result in darker pixels and generally appear blacker in the image. Contrast refers to the distinction that exists between various image features in both analog and digital images. The distinction within an image can be in the form of different shades of gray, light intensity, or color. Images with higher contrast levels generally display a greater degree of grayscale, color, or intensity variation than images with lower contrast. Spatial resolution refers to the number of pixels utilized to construct a digital image. An image with higher spatial resolution is constructed by more pixels than an image with lower spatial resolution.

[0067] The derived contrast mechanism comprises: (i) a first imaging modality capable of acquiring images having characteristics used to detect objects of interest (e.g., DTI-FA); and (ii) a second imaging modality capable of acquiring images having characteristics used to segment objects of interest (e.g., DTI-MD). Various imaging modalities, image types, and / or characteristics can be combined to improve each computer vision task (e.g., object detection or object segmentation). In various embodiments, the imaging modality of the derived contrast mechanism is the same, such as MRI or DTI. In some embodiments, an imaging modality is used to acquire a first image having a first characteristic and a second image having a second characteristic, where the first image is different from the second image. For example, an MRI can be used to acquire a diffusion tensor parametric map of the subject. The diffusion tensor parametric map can include a first measurement map, such as a FA map, and a second measurement map, such as a MD map. In some embodiments, an imaging modality is used to acquire a first image having a first characteristic and a second image having a second characteristic, where the first characteristic is different from the second characteristic. For example, CT may be used to obtain multiple CT scans on the subject. The CT scans may include a first CT scan, such as a low-resolution CT scan, and a second CT scan, such as a high-resolution CT (HRCT) scan. Alternatively, MRI may be used to obtain a diffusion tensor parametric map of the subject. The diffusion tensor parametric map may include a first MD map, such as a low-resolution MD map, and a second MD map, such as a high-resolution MD map. In other embodiments, the imaging modalities of the derived contrast mechanisms are different, such as PAT and ultrasound. PAT may be used to obtain a first type of image having a first characteristic, and ultrasound may be used to obtain a second type of image having a second characteristic, where the first type of image and the first characteristic are different from the second type of image and the second characteristic.

[0068] Examples of contrast mechanisms derived using different types of imaging modalities, image types, and characteristics include the following: (A)MRI Kidney segmentation: (i) FA measurement maps (contrast generated by fractional anisotropy) for object detection; (ii) MD measurement maps (contrast generated by mean diffusivity) or T2-weighted anatomical images (contrast generated from signal relaxation time after excitation) for object segmentation. Multiple sclerosis brain lesion segmentation: (i) single-echo T2 images for object detection (contrast generated from single-shot echo-planar imaging of the signal relaxation time after excitation); (ii) echo-weighted or T2-weighted anatomical images for object segmentation (contrast generated from low flip angles, long echo times, and long repetition times used to emphasize the signal relaxation time after excitation). Liver segmentation: (i) MD measurement maps (contrast generated by average diffusivity) for object detection, and (ii) high-resolution MD measurement maps (contrast generated by high-resolution and average diffusivity), T2-weighted anatomical images (contrast generated from signal relaxation time after excitation) or PD (contrast generated from water concentration) for object segmentation. (B)CT Lung and liver tumor segmentation: (i) CT scan (low resolution) for lung or liver object detection, and (ii) CT scan (HRCT) for tumor object segmentation. Trabecular bone: (i) CT scan (low resolution) for object detection of the trabecular cavity (non-cortical bone), and (ii) CT scan (HRCT) for object segmentation of the trabeculae. (C)PET Tumor or organ detection: (i) PET high contrast / low resolution for object detection (contrast generated by radiotracer measurements), and (ii) PET-CT or PET-MR high contrast / high resolution for object segmentation (contrast generated by radiotracer measurements). (D) Photoacoustic tomography (photoacoustic technology) Tumor or organ detection: (i) PAT (contrast generated by light absorption) for object detection, and (ii) ultrasound (contrast generated from the echo return distance between the transducer and tissue boundaries) for object segmentation.

[0069] It should be understood that examples and embodiments relating to MRI, CT, PAT, PET, etc. are described herein for illustrative purposes only, and that alternative imaging modalities (e.g., fluoroscopy, magnetic resonance angiography (MRA), and mammography) will be suggested to one of skill in the art for implementing various derived contrast mechanisms according to aspects of the present disclosure. Additionally, parameters of any of these imaging modalities can be altered (e.g., different tracers, angular configurations, wavelengths, etc.) to capture different structures or regions of the body, and one or more of these types of altered imaging techniques can be combined with one or more other imaging techniques to implement various derived contrast mechanisms according to aspects of the present disclosure. IV. Techniques for Segmenting Medical Images

[0070] Segmentation of MRI images is divided into two parts. The first part of segmentation concerns a first visual model built to perform localization (object detection) of classes in a first image (e.g., a diffusion tensor parametric map or a CT image). These classes are "semantically interpretable" and correspond to real-world categories such as liver, kidney, heart, etc. Localization is performed using EM, using Look-Once-Looking (YOLO) or (YOLOv2) or (YOLOv3) or similar object detection algorithms that are heuristically initialized with standard clustering techniques (e.g., k-means clustering technique, Otsu's method, Density-Based Spatial Clustering of Applications with Noise (DBSCAN) technique, mini-batch K-means technique, etc.). Initialization is used to provide initial estimates of the parameters of the likelihood model for each class. Expectation-maximization is an iterative process to find the (local) maximum likelihood or maximum a posteriori (MAP) estimates of parameters in one or more statistical models. The EM iteration alternates between an expectation (E) step, which forms a function of the expected value of the log-likelihood evaluated using the current estimates of the parameters, and a maximization (M) step, which calculates the parameters that maximize the expected log-likelihood found in the E step. These parameter estimates are then used to determine the distribution of the latent variables in the next E step. The result of the localization is a bounding box or segmentation mask around each object with a probability per class. The general localization of the object of interest (e.g., kidney) is separated using one or more of the classes associated with the object of interest.

[0071] To reduce background effects in medical images, a bounding box or segmentation mask of the object of interest is projected onto a second image (e.g., a diffusion tensor parametric map or a CT image) in the slice directions (axial, coronal, and sagittal). When localization is used to determine the segmentation mask, the boundaries of the projected segmentation mask are used to define a bounding box (a rectangular box completely delineated around the pixel-wise mask associated with the object of interest) in the second image around the general location of the object of interest (e.g., a kidney). In some examples, the bounding box (determined by localization or defined based on the boundaries of the projected segmentation mask) is expanded by a predetermined number of pixels on all sides to ensure coverage of the object of interest. The area within the bounding box is then cropped from the second image to obtain a portion of the second image having the object of interest, and the portion of the second image is used as an input to a second visual model to segment the object of interest. The second part of the segmentation involves a second visual model (deep learning neural network) constructed by a weighted loss function (e.g., Dice loss) to overcome the disparity between the object of interest and the background and thus focus on evaluating the segmentation of the object of interest. Furthermore, the second visual model can be trained using an augmented dataset so that the deep learning neural network can be trained on a limited set of medical images. The trained second visual model takes as input the cropped portion of the second image and outputs a portion of the second image with an estimated segmentation boundary around the object of interest. The estimated segmentation boundary can be used to calculate the volume, surface area, axial dimension, maximum axial dimension, or other size-related metric of the object of interest. Any one or more of these metrics may then be used alone or in combination with other factors to determine the diagnosis and / or prognosis of the subject. IV.A. Exemplary Computing Environment

[0072] 1 illustrates an exemplary computing environment 100 (i.e., a data processing system) for segmenting instances of an object of interest in an image using a multi-stage segmentation network, according to various embodiments. As shown in FIG. 1, the segmentation performed by the computing environment 100 in this example includes several stages: an image acquisition stage 105, a model training stage 110, an object detection stage 115, a segmentation stage 120, and an analysis stage 125.

[0073] The image acquisition stage 110 includes one or more imaging systems 130 (e.g., MRI imaging systems) for acquiring images 135 (e.g., MR images) of various parts of the subject. The imaging systems 130 are configured to use one or more radiological imaging techniques, such as X-ray radiography, fluoroscopy, MRI, ultrasound, nuclear medicine functional imaging (e.g., PET), thermography, CT, mammography, etc., to acquire the images 135. The imaging systems 130 can determine differences between various structures and functions within the subject based on characteristics (e.g., brightness, contrast, and spatial resolution) associated with each of the imaging systems 130 and generate a series of two-dimensional images. Once the series of two-dimensional images are collected by the scanner's computer, the two-dimensional images can be digitally "stacked" together by computer analysis to reconstruct a three-dimensional image of the subject or part of the subject. The two-dimensional images and / or the reconstructed three-dimensional images 135 allow easier identification and location of underlying structures (e.g., organs) as well as possible tumors or abnormalities. Each two-dimensional image and / or reconstructed three-dimensional image 135 corresponds to a session time and a subject and may depict an internal region of the subject. Each two-dimensional image and / or reconstructed three-dimensional image 135 may further be of a standardized size, resolution, and / or magnification.

[0074] In some embodiments, the one or more imaging systems 130 include a DI system (e.g., an MRI system with special software) configured to apply supplemental MR gradients during image acquisition. The movement of protons during the application of these gradients affects the signal in the image, thereby providing information about molecular diffusion. The DTI matrix is ​​obtained from a series of diffusion-weighted images of various gradient orientations. Three diffusivity parameters or eigenvalues ​​(λ1, λ2, λ3) are generated by matrix diagonalization. Diffusivity is a scalar index that represents the diffusion of water in a particular voxel (the smallest volume element in the image) related to the geometry of the tissue. Various diffusion imaging techniques can be used to calculate diffusion tensor parametric maps and additional image contrast from diffusivity. The DTI characteristics or indices represented by these maps can include (but are not limited to) molecular diffusion rates (MD map or ADC map), directional preference of diffusion (FA map), AD map (diffusion rate along the main axis of diffusion), and RD map (transverse diffusion rate). The diffusivities (λ1, λ2, λ3) obtained by DTI matrix diagonalization can be partitioned into components parallel (λ1) and perpendicular (λ2 and λ3) to the tissue. The sum of the diffusivities (λ1, λ2, λ3) is called the trace, and its average (=trace / 3) is called the MD or ADC. Fractional anisotropy (FA) is a measure of the amount of diffusion asymmetry in a voxel and is defined in terms of its diffusivities (λ1, λ2, λ3). The value of FA varies between 0 and 1. For perfect isotropic diffusion, λ1=λ2=λ3, the diffusion ellipsoid is a sphere, and FA=0. As diffusion anisotropy progresses, the eigenvalues ​​become unequal, the ellipsoid becomes elongated, and FA→1. The axial diffusivity (AD), λ|≡λ1>λ2, λ3, represents the average diffusion coefficient of water molecules diffusing parallel to the tracts in the voxel of interest. Similarly, the radial diffusivity (RD) λ┴≡(λ2+λ3) / 2 can be defined as the magnitude of water diffusion perpendicular to the principal eigenvector.

[0075] The images 135 depict one or more objects of interest. The object of interest can be any "thing" within a subject, such as a region (e.g., abdomen), an organ (e.g., kidney), a lesion / tumor (e.g., malignant liver tumor or brain lesion), a metabolic function (e.g., synthesis of plasma proteins in the liver), etc. In some examples, the multiple images 135 depict the object of interest such that each of the multiple images 135 can correspond to a virtual "slice" of the object of interest. Each of the multiple images 135 can have the same viewing angle such that each image 135 describes a plane parallel to other planes depicted in the other images 135 corresponding to the subject and the object of interest. Each of the multiple images 135 can further correspond to a different distance along a normal axis relative to the plane. In some examples, the multiple images 135 depicting the object of interest undergo a pre-processing step to align each image and generate a three-dimensional image structure of the object of interest.

[0076] In some embodiments, the image 135 includes a diffusion tensor parametric map depicting one or more objects of interest of the subject. In a particular example, at least two diffusion tensor parametric maps (e.g., a first diffusion tensor parametric map and a second diffusion tensor parametric map) are generated for the object of interest. The diffusion tensor parametric maps are generated by a DTI system and can describe the rate and / or direction of diffusion of water molecules to provide further context to the object of interest. Two or more diffusion tensor parametric maps can be generated such that each diffusion tensor parametric map corresponds to a different direction. For example, the diffusion tensor parametric maps can include an image representing FA, an image representing MD, an image representing AD, and / or an image representing RD. Each of the diffusion tensor parametric maps can further have a viewing angle depicting the same plane and the same distance along the normal axis of the plane as the corresponding MR image, such that each MR image depicting a virtual "slice" of the object of interest has a corresponding diffusion tensor image depicting the same virtual "slice" of the object of interest.

[0077] The model training stage 110 builds and trains one or more models 140a-140n (where "n" represents any natural number) (which may be referred to herein individually as models 140 or collectively as models 140) to be used by other stages. The models 140 may be machine learning ("ML") models such as, for example, a convolutional neural network ("CNN"), such as an initial neural network, a residual neural network ("Resnet"), a U-Net, a V-Net, a single-shot multi-box detector ("SSD") network, or a recurrent neural network ("RNN"), such as, for example, a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model, or any combination thereof. Model 140 may also be any other suitable ML model trained in object detection and / or segmentation from images, such as three-dimensional CNN ("3DCNN"), dynamic time warping ("DTW") techniques, hidden Markov models ("HMM"), or the like, or a combination of one or more of such techniques, e.g., CNN-HMM or MCNN (multi-scale convolutional neural network). Computing environment 100 may use the same or different types of models to segment instances of objects of interest. In some examples, model 140 is configured with a weighted loss function that compensates for the imbalanced nature in each image between a large field of view or background and a small foreground object of interest, as described in further detail herein.

[0078] To train the model 140 in this example, the samples 145 are generated by acquiring digital images, splitting the images into a subset of images 145a (e.g., 90%) for training and a subset of images 145b (e.g., 10%) for validation, preprocessing the subset of images 145a and the subset of images 145b, augmenting the subset of images 145a, and in some examples annotating the subset of images 145a with labels 150. The subset of images 145a are acquired from one or more imaging modalities (e.g., MRI and CT). In some examples, the subset of images 145a are acquired from a data storage structure, such as a database, an imaging system (e.g., one or more imaging systems 130), associated with one or more imaging modalities. Each image shows one or more objects of interest, such as a radial region, a thoracic region, an abdominal region, a pelvic region, a spleen, a liver, a kidney, a brain, a tumor, a lesion, etc.

[0079] The splitting may be performed randomly (e.g., 90 / 10% or 70 / 30%), or the splitting may be performed according to more complex validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, nested cross-validation, etc., to minimize sampling bias and overfitting. Pre-processing may include cropping the images so that each image contains only a single object of interest. In some examples, pre-processing may further include standardization or normalization to put all features on the same scale (e.g., the same size scale or the same color or saturation scale). In certain examples, the images are resized with a minimum size (width or height) of a given number of pixels (e.g., 2500 pixels) or a maximum size (width or height) of a given number of pixels (e.g., 3000 pixels) to maintain the original aspect ratio.

[0080] Augmentation can be used to artificially increase the size of the subset of images 145a by creating modified versions of images in the dataset. Image data augmentation can be performed by creating transformed versions of images in the dataset that belong to the same class as the original image. Transformations include a range of operations from the field of image manipulation such as shift, flip, zoom, etc. In some examples, operations include random erasure, shift, brightness, rotation, Gaussian blur, and / or elastic transformations to ensure that the model 140 can perform under circumstances outside those available from the subset of images 145a.

[0081] Additionally or alternatively, augmentation can be used to artificially enhance a number of images in a dataset that the model 140 can take as input during training. In some examples, at least a portion of the training dataset (i.e., the subset of images 145a) can include a first set of images corresponding to a region of one or more subjects and a second set of images corresponding to a different region of the same or different subjects. For example, if at least a first subset of images corresponding to an abdominal region is used to detect one or more objects of interest in the abdominal region, a second subset of images corresponding to a head region can also be included in the training dataset. In such a case, the first set of images corresponding to a region of one or more subjects is histogram matched to the second set of images corresponding to a different region of the same or different subjects in the training dataset. With respect to the previous example, the histogram of the images corresponding to the radial region can be treated as a reference histogram, and then the histogram of the images corresponding to the abdominal region is matched to the reference histogram. Histogram matching can be based on pixel intensity, pixel color, and / or pixel brightness such that processing of the histogram involves matching pixel intensities, pixel colors, and / or pixel brightness of a histogram to pixel intensities, pixel colors, and / or pixel brightness of a reference histogram.

[0082] Annotation may be performed manually by one or more humans (annotators, such as radiologists or pathologists) who confirm the presence of one or more objects of interest in each image of the subset of images 145a and provide labels 150 for the one or more objects of interest, such as, for example, using annotation software to draw a bounding box (ground truth) or segmentation boundary around the area confirmed by the human to contain one or more objects of interest. In certain examples, the bounding box or segmentation boundary may be drawn only for cases where the probability of being an object of interest is greater than 50%. For images annotated by multiple annotators, the bounding boxes or segmentation boundaries from all annotators may be used. In some examples, the annotation data may further indicate a type of object of interest. For example, if the object of interest is a tumor or lesion, the annotation data may indicate a type of tumor or lesion, such as a liver, lung, pancreas, and / or kidney tumor or lesion.

[0083] In some examples, the subset of images 145 may be sent to the annotator device 155 for inclusion in the training dataset (i.e., subset of images 145a). Inputs may be provided to the annotator device 155 (e.g., by a radiologist) using a mouse, trackpad, stylus, and / or keyboard (e.g.,) indicating (e.g.,) whether the image depicts an object of interest (e.g., a lesion, an organ, etc.); the number of objects of interest depicted in the image; and the perimeter (bounding box or segmentation boundary) of each depicted object of interest in the image. The annotator device 155 may be configured to generate a label 150 for each image using the provided inputs. For example, the label 150 may include the number of objects depicted in the image; a type classification for each depicted object of interest; the number of each depicted object of a particular type; and the perimeter and / or mask of one or more identified objects of interest in the image. In some examples, the label 150 may further include the perimeter and / or mask of one or more identified objects of interest superimposed on the first type of image and the second type of image.

[0084] The training process of the model 140 includes selecting hyperparameters for the model 140 and performing an iterative operation of inputting images from the subset of images 145a to the model 140 to find a set of model parameters (e.g., weights and / or biases) that minimizes the loss or error function of the model 140. Hyperparameters are settings that can be tuned or optimized to control the behavior of the model 140. Most models explicitly define hyperparameters that control different aspects of the model, such as memory or execution cost. However, additional hyperparameters can be defined to adapt the model to specific scenarios. For example, hyperparameters can include the number of hidden units of the model, the learning rate of the model, the convolution kernel width, or the number of kernels of the model. In some examples, as described in detail herein, the number of model parameters is reduced for each convolutional and deconvolutional layer, and / or the number of kernels is reduced by half for each convolutional and deconvolutional layer compared to a typical CNN. Each iteration of training may include finding a set of model parameters of model 140 (configured by a defined set of hyper-parameters) such that the value of the loss or error function using the set of model parameters is less than the value of the loss or error function using a different set of model parameters in the previous iteration. The loss or error function may be constructed to measure the difference between an output inferred using model 140 (in some examples, a segmentation boundary around one or more instances of the object of interest is measured using a Dice similarity coefficient) and a ground truth segmentation boundary annotated to the image using labels 150.

[0085] Once a set of model parameters has been identified, the model 140 can be trained and validated using a subset 145a of images (a test or validation data set). The validation process involves iterative operations of inputting images from the subset 145b of images into the model 140 using validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, nested cross-validation, etc., to tune the hyper-parameters and ultimately find an optimal set of hyper-parameters. Once an optimal set of hyper-parameters is obtained, a reserved test set of images from the subset 145b of images is input into the model 145 to obtain an output (in this example, a segmentation boundary around one or more objects of interest), which is then evaluated against the ground truth segmentation boundary using correlation techniques such as the Bland-Altman method and Spearman's rank correlation coefficient, and calculating performance metrics such as error, precision, precision, recall, and receiver operating characteristic curves (ROC).

[0086] As should be understood, other training / validation mechanisms are envisioned and may be implemented within the computing environment 100. For example, a model may be trained and hyperparameters may be tuned on images from the subset of images 145a, and images from the subset of images 145b may be used only to test and evaluate the performance of the model. Furthermore, the training mechanisms described herein focus on training new models 140. These training mechanisms can also be utilized to fine-tune existing models 140 trained from other datasets. For example, in some instances, the model 140 may have been pre-trained using images of other objects or biological structures, or from sections from other subjects or studies (e.g., human or mouse studies). In those cases, the model 140 can be used for transfer learning and retrained / validated using images 135.

[0087] The model training stage 110 outputs trained models including one or more trained object detection models 160 and one or more trained segmentation models 165. A first image 135 is acquired by a localization controller 170 in the object detection stage 115. The first image 135 depicts an object of interest. In some examples, the first image is a diffusion tensor parametric map having a first characteristic, such as FA or MD contrast. In other examples, the first image is an MR image having a first characteristic, such as single echo T2 contrast or T2 weighted anatomical contrast. In other examples, the first image 135 is a CT image having a first characteristic, such as low resolution or high resolution. In other examples, the first image 135 is a CT image having a first characteristic, such as a contrast agent. In other examples, the first image 135 is a PET image having a first characteristic, such as a radioactive tracer contrast or low resolution. In other examples, the first image 135 is a PET-CT image having a first characteristic, such as a radioactive tracer contrast or high resolution. In another example, the first image 135 is a PET-MR image having a first characteristic, such as radioactive tracer contrast or high resolution. In another example, the first image 135 is a PAT image having a first characteristic, such as optical absorption contrast. In another example, the first image 135 is an ultrasound image having a first characteristic, such as echo or transducer to tissue boundary distance.

[0088] The localization controller 170 includes a process for locating an object of interest in the image 135 using one or more object detection models 160. The localization includes: (i) locating and classifying an object in the first image having a first characteristic into a plurality of object classes using the object detection model 160, where the classifying assigns a set of pixels or voxels of the first image to one or more of the plurality of object classes, and (ii) determining a bounding box or segmentation mask of the object of interest in the first image using the object detection model 160 based on the set of pixels or voxels assigned to the object classes of the plurality of object classes. The object detection model 160 utilizes one or more object detection algorithms to extract statistical features used to locate and label objects in the first image and predict a bounding box or segmentation mask of the object of interest.

[0089] In some examples, localization is performed using EM, YOLO, YOLOv2, YOLOv3, or similar object detection algorithms that are heuristically initialized with standard clustering techniques (e.g., K-means or Otsu's method). The initialization is used to provide an initial estimate of the parameters of the likelihood model for each class. For example, when using EM with K-means clustering technique, given a fixed number of k clusters, observations are assigned to the k clusters such that the means across the clusters (for all variables) are as different from each other as possible. The EM clustering technique then calculates the posterior probabilities of cluster membership and cluster boundaries based on one or more prior probability distributions parameterized with the initial estimates of the parameters of each cluster (class). The objective of the EM clustering technique is to maximize the overall probability or likelihood of the data given the (final) clusters. The results of the EM clustering technique differ from those calculated by the K-means clustering technique. The K-means clustering technique assigns observations (pixels or voxels, pixel or voxel intensities, etc.) to clusters to maximize the distance between the clusters. The EM clustering technique does not calculate the actual assignment of observations to clusters, but rather calculates classification probabilities. In other words, each observation belongs to each cluster with a certain probability. The observations can then be assigned to clusters by the localization controller 170 based on the (maximum) classification probability. The result of the localization is a bounding box or segmentation mask with per-class probabilities. The general location of the object of interest (e.g., a kidney) is separated based on the set of pixels or voxels assigned to the object class associated with the object of interest.

[0090] The bounding box or segmentation mask of the object of interest is utilized by the map processing controller 175 in the object detection stage 115. The second image 135 is acquired by the map processing controller 175 in the object detection stage 115. The second image 135 depicts the same object of interest depicted in the first image 135. In some examples, the second image is a diffusion tensor parametric map having a second characteristic, such as FA or MD contrast. In other examples, the second image is an MR image having a second characteristic, such as single echo T2 contrast or T2 weighted anatomical contrast. In other examples, the second image 135 is a CT image having a second characteristic, such as low resolution or high resolution. In other examples, the second image 135 is a CT image having a second characteristic, such as contrast agent. In other examples, the second image 135 is a PET image having a second characteristic, such as radioactive tracer contrast or low resolution. In other examples, the second image 135 is a PET-CT image having a second characteristic, such as radioactive tracer contrast or high resolution. In another example, the second image 135 is a PET-MR image having a second characteristic, such as radiotracer contrast or high resolution. In another example, the second image 135 is a PAT image having a second characteristic, such as optical absorption contrast. In another example, the second image 135 is an ultrasound image having a second characteristic, such as echo or transducer to tissue boundary distance.

[0091] The map processing controller 170 includes a process for overlaying a bounding box or segmentation mask corresponding to the detected object of interest from the first image on top of the same of the object of interest depicted in the second image. When the segmentation mask is determined, the segmentation mask is projected onto the second image (e.g., a two-dimensional slice of the second image) such that the boundaries of the segmentation mask can be used to define a rectangular bounding box that encloses a region of interest corresponding to the object of interest in the second image. In some examples, the bounding box includes additional padding (e.g., padding of 5 pixels, 10 pixels, 15 pixels, etc.) on each edge around the segmentation mask to ensure that the entirety of the region of interest is enclosed. The map processing controller 175 further includes a process configured to crop the second image such that only a crop portion 180 corresponding to the bounding box is shown. If more than one bounding box is defined (e.g., if multiple objects of interest are detected in the image), a cropped portion of the second image is generated for each bounding box. In some instances, each cropped portion is further resized (e.g., with additional padding) to maintain a uniform size.

[0092] The cropped portion 180 of the second image is sent to a segmentation controller 185 in the segmentation stage 120. The segmentation controller 185 includes a process for segmenting an object of interest in the cropped portion 180 of the second image using one or more segmentation models 165. The segmenting includes generating an estimated segmentation boundary around the object of interest using one or more segmentation models 165, and outputting the cropped portion of the second image having an estimated segmentation boundary 190 around the object of interest using one or more segmentation models 165. The segmenting may include evaluating the variation in pixel or voxel intensity of each cropped portion to identify a set of edges and / or contours that correspond to the object of interest. Upon identifying the set of edges and / or contours, the one or more segmentation models 165 generate an estimated segmentation boundary 190 of the object of interest. In some embodiments, the estimated segmentation boundary 190 corresponds to a three-dimensional representation of the object of interest. In some examples, the segmenting further includes determining a probability score of the object of interest being present at the estimated segmentation boundary 190 and outputting the probability score having the estimated segmentation boundary 190.

[0093] The cropped portion of the second image having the estimated segmentation boundary 190 (and any probability score) around the object of interest can be sent to an analysis controller 195 in the analysis stage 125. The analysis controller 195 includes a process for obtaining or receiving the cropped portion of the second image having the estimated segmentation boundary 190 (and any probability score) around the object of interest and determining an analysis result 197 based on the estimated segmentation boundary 190 around the object of interest (and any probability score). The analysis controller 195 can further include a process for determining a size, axial dimension, surface area, and / or volume of the object of interest based on the estimated segmentation boundary 190 around the object of interest. In some examples, the estimated segmentation boundary 190 for the object of interest or a derivation thereof (e.g., the size, axial dimension, volume, etc. of the object of interest) is further used to determine a diagnosis and / or prognosis of the subject. In a further example, the estimated segmentation boundary 190 of the object of interest is compared to the estimated segmentation boundary 190 of the same object of interest imaged at a previous time point to determine treatment efficacy for the subject. For example, if the target object is a lesion, the estimated segmentation boundary 190 of the subject's lesion can provide information regarding the type of cancer (e.g., location of the lesion), metastatic progression (e.g., if the number of lesions has increased for the subject and / or the number of locations of the lesion has increased), and drug efficacy (e.g., whether the number, size, and / or volume of lesions has increased or decreased).

[0094] Although not explicitly shown, it will be understood that computing environment 100 can further include a developer device associated with the developer. Communications from the developer device to components of computing environment 100 can indicate the type of input images used for the models, the number and type of models used, the hyperparameters of each model, e.g., learning rate and number of hidden layers, how to format data requests, the training data used (e.g., and how to access the training data) and validation techniques used, and / or how to configure the controller process. IV.B. Exemplary Data Augmentation for Model Training

[0095] The second part of the segmentation involves a second visual model (e.g., a deep learning neural network) configured with a weighted loss function (e.g., Dice loss). The deep learning neural network is trained on images of one or more objects of interest from a subject. The images are generated from one or more medical imaging modalities. However, the datasets of images generated from some medical imaging modalities can be sparse. To address the sparseness of these images, the images of the training data are augmented to artificially increase the number and variety of images in the dataset. More specifically, the augmentation can be performed by performing histogram matching to simulate other contrasts in other regions of the same or different subjects (e.g., regions of the subject where the object of interest may not be found) and increase the variance of the training dataset.

[0096] As an example, each image in the training set or a subset of images from the training set can be histogram matched with one or more reference images to generate a new set of images, artificially increasing the training dataset size in a so-called data augmentation process that reduces overfitting. As shown in FIG. 2, each image in the training set (left image) can be histogram matched with a reference image (center image) to generate a new set of images (right image). Thus, the original training set or subset of images is essentially increased in number and diversity by a factor of two through histogram matching. Histogram matching is the transformation of an original image such that its histogram matches a reference histogram. Histogram matching is performed by first equalizing both the original and reference histograms using histogram equalization (e.g., expanding the histogram to fill the dynamic range while simultaneously trying to keep the histogram uniform), and then mapping from the original histogram to the reference histogram based on the equalized image and a transformation function. For example, assuming a pixel intensity value of 20 in the original image maps to 35 in the equalized image and a pixel intensity value of 55 in the reference image maps to 35 in the equalized image, it can be determined that a pixel intensity value of 20 in the original image should be mapped to a pixel intensity value of 55 in the reference image. The original image can then be transformed into a new image using the mapping from the original image to the reference image that is equalized. In some examples, one or both datasets (i.e., the set of original images and the set of new images) can be further augmented using standard techniques such as rotation and flipping (e.g., rotating each image 90 degrees, flipping left to right, flipping top to bottom, etc.) to further increase the number and variety of images available for training.

[0097] The advantages of using data augmentation based on histogram matching are: (i) the technique utilizes another dataset from the same species / instrument / image intensity etc; (ii) the mask (label) corresponding to the histogram-matched image is exactly the same as the original image in the training set; (iii) the number of images in the training set is multiplied by the number of images used as reference; and (iv) the variance of the training set is increased, so the structure of the images in the training set is preserved while the pixel intensities are changing, which makes the segmentation framework independent of pixel intensities and dependent on the image and structure of the object of interest. IV.C. Exemplary 3D Deep Neural Networks

[0098] In the exemplary embodiment shown in FIG. 3, the modified 3D U-Net 300 extracts features from an input image (e.g., a cropped portion of a second image) separately, detects an object of interest in the input image, generates a three-dimensional segmentation mask around the shape of the object of interest, and outputs the input image with the three-dimensional segmentation mask around the shape of the object of interest. The 3D U-Net 300 includes a reduction path 305 and an expansion path 310, which gives it a U-shaped architecture. The reduction path 305 is a CNN network that includes repeated application of convolutions (e.g., 3×3×3 convolutions (unpadded convolutions)), each of which is followed by a rectified linear unit (ReLU) and a max-pooling operation for downsampling (e.g., 2×2×2 max-pooling with stride 2 in each direction). The input for the convolution operation is a three-dimensional volume (i.e., an input image of size n×n× channels, where n is the number of input features), and a set of "k" filters (also called kernels or feature extractors), each of size (f×f×f channels, where f is any number, e.g., 3 or 5). The output of the convolution operation is also a three-dimensional volume (also called an output image or feature map) of size (m×m×k, where M is the number of output features and k is the convolution kernel size).

[0099] Each block 315 of the reduction path 315 contains one or more convolution layers (indicated by grey horizontal arrows), where the number of feature channels varies, for example, from 1 → 64 (for example, in the first process depending on the starting number of channels) as the convolution process increases the depth of the input image. The downward grey arrows between each block 315 are max pooling processes that halve the size of the input image. At each downsampling step or pooling operation, the number of feature channels can be doubled. During reduction, the spatial information of the image data is reduced and the feature information is increased. Thus, what was present in, for example, a 572x572 image before pooling, after pooling (almost) the same information is present in, for example, a 284x284 image. Now, when the convolution operation is applied again in a subsequent process or layer, the filters of the subsequent process or layer can see a larger context, i.e., as the input image goes deeper into the network, the size of the input image decreases, but the receptive field increases (the receptive field (context) is the area of ​​the input image that the kernel or filter covers at any given time). Once block 315 is executed, two more convolutions are performed in block 320, but without max pooling. The image after block 320 is resized to, for example, 28x28x1024 (this size is merely illustrative, the size at the end of process 320 can be different depending on the starting size of the input image - size nxnxchannels).

[0100] The augmentation path 310 is a CNN network that combines the features and spatial information from the reduction path 305 (upsampling the feature maps from the reduction path 305). As described herein, the output of 3D segmentation is not simply class labels or bounding box parameters. Instead, the output (3D segmentation mask) is the complete image (e.g., high-resolution image) with all voxels classified. If a regular convolutional network with pooling and dense layers were used, the CNN network would lose the "where" information and only retain the "what" information that is not acceptable for image segmentation. For image segmentation, both the "what" and "where" information are used. Thus, the image is upsampled to convert the low-resolution image to a high-resolution image and restore the "where" information. Transposed convolution, represented by the upward white arrow, is an exemplary upsampling technique that can be used in the augmentation path 310 for upsampling the feature maps and expanding the size of the image.

[0101] After the transposed convolution in block 325, the image is expanded by two step sizes, for example from 28x28x1024 → 56x56x512 via a 2x2x2 upconvolution (upsampling operator) in each dimension, and then the image is concatenated with the corresponding image from the reduction path (see horizontal gray bar 330 from the reduction path 305), together forming an image of size, for example, 56x56x1024. The reason for the concatenation is to combine information from previous layers to get a more accurate prediction (i.e., high-resolution features from the reduction path 305 are combined with the upsampled output from the upsampling path 310). This process continues as a series of up-convolutions that halve the number of channels to generate a multi-channel segmentation as a three-dimensional segmentation mask, concatenation with the correspondingly cropped feature map from the reduction path 305, repeated application of convolutions (e.g., two 3x3x3 convolutions), each followed by a rectified linear unit (ReLU), and a final convolution in block 335 (e.g., a 1x1x1 convolution). For localization, U-Net 300 uses the useful parts of each convolution without a fully connected layer, i.e., the segmentation map contains only voxels for which the full context is available in the input image, and uses skip connections to link the context features learned during the reduction block with the localization features learned in the expansion block.

[0102] In traditional neural networks with 3D U-Net architecture, the output and ground truth labels are typically compared using a softmax function with cross-entropy loss. Although these networks show improved segmentation performance over traditional CNNs, they do not readily translate to small foreground objects, small sample sizes, and anisotropic resolution in medical imaging datasets. To address these and other issues, 3D U-Net300 is constructed to include a reduced number of parameters and / or kernels compared to traditional 3D U-Net (a total of 2,784 kernels and 19,069,955 learnable parameters - see, e.g., O. Cicek, A. Abdulkadir, S. Lienkamp, ​​T. Brox, and O. Ronneberger, "3D U-Net: learning dense volumetric segmentation from sparse annotation," in International conference on medical image computing and computer-assisted intervention, 2016: Springer, pp. 424-432). Specifically, the number of weights, the number of layers, and / or the overall width of the network are reduced to reduce the network complexity and avoid the over-parameterization problem. In some examples, the number of kernels for each convolutional and deconvolutional layer is halved compared to the conventional 3D U-Net. This then reduces the number of learnable parameters for each convolutional and deconvolutional layer of the 3D U-Net300, and thus the 3D U-Net300 has 9,534,978 learnable parameters compared to the conventional 3D U-Net having 19,069,955 learnable parameters. In terms of kernels, the total number of kernels is reduced from 2,784 in the conventional 3D U-Net to 1,392 kernels in the 3D U-Net300. In some examples, the total number of learnable parameters of the 3D U-Net300 is reduced to between 5,000,000 and 12,000,000 learnable parameters.In some examples, the total number of kernels in the 3D U-Net300 is reduced to between 800 and 1,700 kernels. Parameter and / or kernel reduction is advantageous because it allows the model to more efficiently process smaller sample sizes (i.e., the cropped portion of the second DTI parametric map).

[0103] Furthermore, the 3D U-Net300 is constructed for volumetric segmentation using a weighted loss function. Specifically, the metric used to evaluate the segmentation performance was the Dice Similarity Coefficient (DSC, Equation 1). Thus, to train the 3D U-Net300 with the objective of maximizing the DSC, the DSC was minimized for all images (Equation 2). Also, due to the imbalanced distribution of background and objects of interest in the foreground in stereoscopic images, a weighted loss function was used, which is referred to herein as the Dice loss (Equation 3), in which the weight of the frequently observed background is reduced and the weight of objects of interest in the foreground is increased to reach a balanced influence of foreground and background voxels on the loss. JPEG2025016513000002.jpg45170, where N is the number of images and p i represents the prediction mask, and q i represents the ground truth mask corresponding to the target object.

[0104] As will be appreciated by those skilled in the art, the 3D U-Net 300 does not need to be incorporated into the entire computing environment 100 described with respect to Figure 1 to implement object segmentation according to aspects of the present disclosure. Instead, various types of models can be used for object segmentation (e.g., CNN, Resnet, classic U-Net, V-Net, SSD network, or recurrent neural network RNN, etc.) as long as the type of model can be learned for object segmentation of medical images. V. Volumetric Segmentation Techniques

[0105] 4 shows a flowchart of an example process 400 for using the described multi-stage segmentation network to segment instances of an object of interest. Process 400 can be performed using one or more computing systems, models, and networks as described in Section IV with respect to FIGS.

[0106] Process 400 begins at block 405, where a medical image of a subject is acquired. The medical image may depict regions corresponding to the subject's head region, chest region, abdominal region, pelvic region, and / or limbs. The medical image is generated using one or more medical imaging modalities. For example, a user may operate one or more imaging systems that generate medical images using one or more medical imaging modalities, as described in Section IV with respect to FIG. 1.

[0107] In block 410, a medical image of the subject is acquired. For example, the medical image acquired in step 405 can be acquired from a data storage device or one or more medical imaging systems. The medical image includes a first image having a first characteristic and a second image having a second characteristic. In some embodiments, the image is a DTI parametric map including a first measurement map (first image having a first characteristic) and a second measurement map (second image having a second characteristic). The first measurement map is different from the second measurement map. The DTI parametric map is generated by applying a supplemental MR gradient during acquisition of the MR image. For example, a user can input parameters of one or more diffusion gradients into the imaging system, and the DTI parametric map is generated by applying a supplemental MR gradient during acquisition of the MR image based on the parameters of the one or more diffusion gradients (movement of protons during application of the gradient affects the signal in the image). In some examples, a diffusion gradient is applied in two or more directions during acquisition of the MR image. In some examples, the first measurement map is a partial anisotropy map and the second measurement map is a mean diffusivity map.

[0108] At block 415, the object in the first image is localized and classified using a localization model. The classifying assigns a set of pixels or voxels of the first image to one or more of a plurality of object classes. The object classes may include a class corresponding to the object of interest (e.g., depending on the type of object of interest), one or more classes corresponding to different biological structures, one or more classes corresponding to different organs, and / or one or more classes corresponding to different tissues. For example, if the object of interest is a lesion, object classes may be defined to identify the lesion, the blood vessel, and / or the organ. The localization and classification may be performed by the localization model using one or more clustering algorithms that assign the set of pixels or voxels to one or more of the plurality of object classes. In some examples, the one or more clustering algorithms include a k-means algorithm that assigns observations to clusters associated with the plurality of object classes. In some examples, the one or more clustering algorithms further include an expectation maximization algorithm that calculates probabilities of cluster membership based on one or more probability distributions. The k-means algorithm may be used to initialize the expectation maximization algorithm by estimating initial parameters for each object class of the plurality of object classes.

[0109] At block 420, a bounding box or segmentation mask is determined for the object of interest in the first image using the localization model. The bounding box or segmentation mask is determined for the object of interest based on a set of pixels or voxels assigned to an object class of a plurality of object classes. To determine the segmentation mask, a seed location for the object of interest is identified using the set of pixels assigned to the object class corresponding to the object of interest. The identified seed location is projected toward the z-axis to grow the seed location and determine the segmentation mask. The z-axis represents depth, and the seed location is grown to fill the entire volume of the object mask in a third or final dimension. In some examples, morphological closing and filling is further performed on the segmentation mask.

[0110] At block 425, the bounding box or segmentation mask is transferred onto the second image to define a portion of the second image that includes the object of interest. Transferring the object mask includes projecting the bounding box or segmentation mask in the slice direction onto a corresponding region of the image (the portion of the second image that includes the object of interest) and / or overlaying the bounding box or segmentation mask onto a corresponding region of the second image (the portion of the second image that includes the object of interest). In some examples, the segmentation mask is projected onto a two-dimensional slice of the second image such that the boundary of the segmentation mask in two-dimensional space can be used to define a rectangular bounding box that encloses a region of interest that corresponds to the detected object of interest. In some examples, the bounding box includes additional padding (e.g., padding of 5 pixels, 10 pixels, 15 pixels, etc.) on each edge around the perimeter of the segmentation mask to ensure that the entirety of the region of interest is enclosed. In some examples, the second image is cropped based on the bounding box or segmentation mask plus an optional margin to generate a portion of the second image. Each cropped portion may be further resized (e.g., with additional padding) to maintain a uniform size.

[0111] In some embodiments, a portion of the second image is sent to a deep super-resolution neural network for pre-processing. The deep super-resolution neural network can be (for example) a convolutional neural network, a residual neural network, an attention-based neural network, and / or a recurrent convolutional neural network. The deep super-resolution neural network processes the sent portion of the second image to improve the image spatial resolution (e.g., enlargement and / or refinement of image details) of the portion of the second image.

[0112] At block 430, the portion of the second image is input to a three-dimensional neural network model constructed for volumetric segmentation using a weighted loss function (e.g., a modified 3D U-Net model). In some examples, the weighted loss function is a weighted Dice loss function. The three-dimensional neural network model includes a plurality of parameters trained using a set of training data. The number of the plurality of model parameters can be reduced compared to a standard three-dimensional U-Net architecture. The set of training data can include a plurality of images having annotations associated with segmentation boundaries around the object of interest and a plurality of additional images having annotations associated with segmentation boundaries around the object of interest. In some examples, the plurality of additional images are artificially generated by matching image histograms from the plurality of images to image histograms from a plurality of reference maps (e.g., maps obtained from other regions of the subject). The plurality of model parameters are identified using the set of training data based on minimizing the weighted loss function. The three-dimensional neural network model further includes a plurality of kernels, and the number of kernels can be reduced compared to a standard three-dimensional U-Net architecture.

[0113] At block 435, the three-dimensional neural network model segments a portion of the second image. The segmenting includes using the identified features to generate an estimated segmentation boundary of the object of interest. For example, the segmenting may include evaluating features, such as variations in pixel intensity of each cut-out portion, to identify a set of edges and / or contours corresponding to the object of interest, and using the set of identified edges and / or contours to generate an estimated segmentation boundary of the object of interest. The estimated segmentation boundary may represent a three-dimensional perimeter of the object of interest. In some examples, the three-dimensional neural network model may also determine a classification of the object of interest. For example, an object corresponding to a lesion may be classified based on its type or location within the subject, such as a lung lesion, a liver lesion, and / or a pancreatic lesion. As another example, an object corresponding to an organ and / or tissue may be classified as healthy, inflamed, fibrotic, necrotic, and / or cast-filled.

[0114] At block 440, a portion of the second image having an estimated segmentation boundary around the object of interest is output. In some examples, the portion of the second image is provided. For example, the portion of the second image may be stored in a storage device and / or displayed on a user device.

[0115] At optional block 445, an operation is performed based on the estimated segmentation boundary around the object of interest. In some examples, the operation includes determining a size, surface area, and / or volume of the object of interest based on the estimated segmentation boundary around the object of interest. In some examples, (i) a portion of the second image having an estimated segmentation boundary around the object of interest, and / or (ii) the size, surface area, and / or volume of the object of interest is provided. For example, (i) the portion of the second image having an estimated segmentation boundary around the object of interest, and / or (ii) the size, surface area, and / or volume of the object of interest may be stored in a storage device and / or displayed on a user device. A user may receive or obtain (i) the portion of the second image having an estimated segmentation boundary around the object of interest, and / or (ii) the size, surface area, and / or volume of the object of interest. In other examples, (i) a portion of the second image having an estimated segmentation boundary around the object of interest, and / or (ii) the size, surface area, and / or volume of the object of interest are used to quantify an image metric such as image intensity. For example, in PET there is the standardized uptake value (SUV), or in MRI there is diffusivity, T2, T1, etc. that correlate with certain image metrics such as image intensity, and thus quantification of the image metric can be used to determine a value / metric such as SUV that is specific to the object of interest.

[0116] In some examples, the operations include determining a diagnosis of the subject using: (i) a portion of the second image having an estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest. In some examples, the operations include administering, by a user, a treatment with a compound (e.g., to the subject) based on (i) the portion of the second image having an estimated segmentation boundary around the object of interest, (ii) the size, surface area, and / or volume of the object of interest, and / or (iii) the subject's diagnosis. In other examples, the operations include determining a treatment plan based on (i) the portion of the second image having an estimated segmentation boundary around the object of interest, (ii) the size, surface area, and / or volume of the object of interest, and / or (iii) the subject's diagnosis, such that a dosage of a drug can be calculated based on the size, surface area, and / or volume of the object of interest. In some examples, the operations include determining whether treatment is effective or whether a dosage of a drug needs to be adjusted based on a comparison of a size, surface area, and / or volume corresponding to the object of interest at a first time point to a size, surface area, and / or volume corresponding to the object of interest at a second time point. VI. Working Examples

[0117] The systems and methods implemented in various embodiments may be better understood with reference to the following examples. VI.A. Example 1. - Kidney Segmentation

[0118] Kidney segmentation using 3D U-Net localized by expectation maximization. VI.Ai background

[0119] Renal function and activity are highly dependent on kidney volume in various diseases such as polycystic kidney disease, lupus nephritis, renal parenchymal disease, and renal transplant rejection. Automated assessment of the kidney by imaging can be used to determine a subject's diagnosis, prognosis, and / or treatment plan. In vivo imaging modalities offer inherent strengths and limitations. In particular, MRI has no ionizing radiation, is operator independent, and has good tissue contrast that allows kidney segmentation and volume-related information. Traditional methods have been used to assess the kidney more locally, such as manual tracking, stereology, or general image processing. These methods can be labor intensive or inconsistent. To address these issues, an integrated deep learning model was utilized to segment the kidney.

[0120] Deep learning segmentation networks have been used for semantic segmentation of large biomedical image datasets. These networks offer state-of-the-art performance but suffer from high computational cost and memory consumption that limits their field of view and depth. Thus, these networks can be particularly problematic to segment small objects in the limited images typically found in MRI examinations. MRIs tend to include a large field of view or background to prevent aliasing artifacts. When the background represents a significant part, the network may not be optimally trained to segment the foreground object of interest. Therefore, alternative strategies are needed to reduce the parameters of large 3D segmentation networks, avoid overfitting, and improve network performance.

[0121] First, to address the issue of background effects, a derived MRI contrast mechanism (using DTI) was incorporated into the localization step prior to learned segmentation. Second, the 3D U-Net was modified to reduce the number of parameters and incorporate a Dice loss function for segmentation. Third, augmentation and MRI histogram matching were incorporated to increase the number of training datasets. Furthermore, these techniques were, in some instances, applied to super-resolution images of the datasets to determine whether the augmented images could improve segmentation performance. These techniques were implemented with preclinical MRI using an animal model of lupus nephritis. VI.A.ii. Animal models and data acquisition

[0122] Fifteen Friend virus B female mice were used in this study, eight for the lupus nephritis (LN) disease group and seven for the control group. Animals were imaged every 2 weeks for four time points starting at 13 weeks of age. At each time point, multiple MRI data sets were acquired for each animal. A total of 196 3D MR images were acquired for this study. All images were manually segmented by a single user. The kidneys were outlined slice-by-slice for the entire image volume using Amira (Thermo Fisher Scientific, Hillsboro, OR). During MR imaging, animals were anesthetized under isoflurane, breathing freely, and maintained at 37 °C. MRI was performed on a Bruker 7T (Billerica, MA) equipped with volume transmit and cryogenic surface receive coils. A custom in vivo holder was constructed by 3D printing (Stratasys Dimension) to provide reliable positioning of the brain and spine. MRI diffusion tensor imaging was performed with individual regional shims (single shot EPI) using the following parameters: TR=4s, TE=42ms, BW=250kHz, diffusion direction=12. FOV=22×22mm 2 ,Encoding matrix = 110 × 110, slices = 15, image resolution = 200 × 200 μm 2, slice thickness = 1 mm, acquisition time = 13 min. Diffusion tensor parametric maps were calculated including: FA, MD, AD, and RD. FA and MD images were used for an integrated semantic segmentation algorithm. VI.A.iii. Phase 1: Localization by EM

[0123] FA images were used for the localization step. FA images were segmented using EM heuristically initialized with K-means (12 classes). One of the tissue classes was used to isolate a general kidney neighborhood and used as the detected object. These parameters were used for the algorithm: number of iterations for convergence = 7 and Markov Random Field smoothing coefficient = 0.05. VI.A.iv. Data Augmentation

[0124] A new dataset was generated by histogram matching of the MD images with the mouse brain dataset (Figure 2). Both datasets were rotated 90°, flipped left-right, and flipped up-down. To ensure that the network was validated on completely unseen data, data augmentation was performed only on the training set. The total number of datasets obtained was n=196. With augmentation, the training dataset increased from n=180 to n=1800, and the testing dataset was n=16. A training and testing split was performed for each animal, leaving one animal for testing and using the rest for training each time. VI.Av Stage 2: Deep Semantic Segmentation

[0125] The metric used to evaluate the segmentation performance was the Dice Similarity Coefficient (DSC, Equation 1). Thus, to train the 3D U-Net with the objective of maximizing the DSC, the DSC was minimized for all images (Equation 2). Also, due to the unbalanced distribution of the background and kidney in the stereoscopic images, a weighted loss function was used (Dice loss, Equation 3). To mitigate the background effect, the EM segmentation masks were projected in the slice direction. The boundary of each projected EM segmentation mask was used to define a rectangular box for object detection. The defined box was expanded by 5 pixels on all sides to ensure coverage of the kidney. The 3D U-Net was trained and tested on the MD images within the detected region. The same detection region was used for the super-resolution images. Because the cropped objects had arbitrary sizes in the first two dimensions based on the 2D projection mask, all cropped images were resized to 64 × 64 × 16 for the original resolution images and 64 × 64 × 64 for the super-resolution images. VI.A.vi. Super-resolution

[0126] The MD images were super-resolved in the through-plane direction to improve spatial resolution. The original 110 × 110 × 15 matrix was decomposed by 5× to obtain a 110 × 110 × 75 matrix. The images were augmented using a deep super-resolution neural network. VI.A.vii. Results

[0127] FIG. 5A shows the six elements of the diffusion tensor. The varying diffusion contrast is most prominent in the inner and outer medulla regions. The varying contrast is seen in the diagonal elements (D xx ,D yy ,D zz ) and off-diagonal elements (D xy ,D xz ,D yz) is noticeable. As a result, the contrast does not change in the cortex, thus resulting in a very low FA (Figure 5B). This low FA allowed the kidney to be segmented from the background. The MR images were super-resolved in the through-plane direction, as shown in Figure 5C. These improvements are most evident in the sagittal and coronal directions. The in-plane resolution is only minimally affected, as shown in the axial slice (Figure 5C). Figure 6A shows the results of training a 3D U-Net on the MD images without preprocessing. The DSC plot shows a uniform distribution with a mean of 0.49. In Figure 6B, the abdominal region is detected as foreground in the connected component analysis and cropped using the MD images. The DSC plot shows a normal distribution with a mean of 0.52. Figure 6C shows the results using only EM segmentation. An average DSC of 0.65 was achieved. Figure 6D represents the results of the integration strategy: first the kidney was detected using EM segmentation on the FA images, and then the 3D U-Net was trained on the kidney region detected from the MD images. The average DSC for this approach was 0.88. The DSC plot of the semantic segmentation with super-resolution MD images (Figure 6E) is fairly similar to the semantic segmentation at the original resolution (Figure 3D). Here, the average DSC was 0.86. The results are summarized in Table 1 along with additional comparison metrics such as volumetric difference (VD) and positive predictive value (PPV). Table 1. Means and standard deviations of segmentation results using DSC, VD and PPV. The best values ​​for each method are shown in bold. JPEG2025016513000003.jpg52170VI.A.viii. Discussion and Conclusion

[0128] This example demonstrates the integration of EM-based localization and 3D U-Net for kidney segmentation. The localization step significantly improved the results of the deep learning method. It was also demonstrated that EM segmentation improves the performance of deep learning, while the EM segmentation method alone performed poorly. EM segmentation isolated the kidney in the central slice but did not preserve the joint representation of the kidney volume. Therefore, the central slice was used for all slices throughout the volume as the detected rectangular object. The weighted Dice loss can be important for error minimization and object and background balance. However, it was found that without the localization step, the performance was not significantly increased by including the weighted Dice loss. As a result, the background contained objects and organs that looked similar to the kidney that could not be distinguished by the 3D U-Net alone.

[0129] The technique presented in this example reduced background effects and reduced data complexity. Thus, the network complexity can be reduced by reducing the number of kernels per convolutional layer by at least half. In this study, a DSC of 0.88 was achieved on a limited MRI dataset of n=196. VII. Further Considerations

[0130] Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause the one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein.

[0131] The terms and expressions used are used as terms of description and not of limitation, and in the use of such terms and expressions there is no intention to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the invention as claimed has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and such modifications and variations are considered to be within the scope of the invention as defined by the appended claims.

[0132] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of preferred exemplary embodiments provides those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0133] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other examples, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

Claims

1. 1. A method for segmenting an object in a medical image, comprising: acquiring medical images of a subject, the medical images including a first image having a first characteristic and a second image having a second characteristic, the medical images being generated using one or more medical imaging modalities; using a localization model to locate and classify objects in the first image into a plurality of object classes, the classifying assigning a set of pixels or voxels of the first image to one or more of the plurality of object classes; determining a bounding box or segmentation mask for an object of interest in the first image based on a set of pixels or voxels assigned to an object class among the plurality of object classes using the localization model; transferring the bounding box or the segmentation mask onto the second image to define a portion of the second image that includes the object of interest; inputting the portion of the second image into a three-dimensional neural network model constructed for volumetric segmentation using a weighted loss function; generating an estimated segmentation boundary around the object of interest using the 3D neural network model; and and using the 3D neural network to output the portion of the second image having the estimated segmentation boundary around the object of interest.

2. 2. The method of claim 1, wherein the one or more medical imaging modalities include a first medical imaging modality and a second medical imaging modality different from the first medical imaging modality, and the first image is generated from the first medical imaging modality and the second image is generated from the second medical imaging modality.

3. 2. The method of claim 1, wherein the one or more medical imaging modalities include a first medical imaging modality and a second medical imaging modality that is the same as the first medical imaging modality, and the first image is generated from the first medical imaging modality and the second image is generated from the second medical imaging modality.

4. The method of claim 1 or 2, wherein the first image is a first type of image and the second image is a second type of image, and the first type of image is different from the second type of image.

5. The method of claim 1 or 3, wherein the first image is a first type of image and the second image is a second type of image, and the first type of image is the same as the second type of image.

6. The method of any one of claims 1 to 5, wherein the first characteristic is different from the second characteristic.

7. The method of any one of claims 1 to 5, wherein the first characteristic is the same as the second characteristic.

8. 4. The method of claim 2 or 3, wherein the first medical imaging modality is magnetic resonance imaging, diffusion tensor imaging, computed tomography, positron emission tomography, photoacoustic tomography, x-ray, ultrasound imaging, or a combination thereof, and the second medical imaging modality is magnetic resonance imaging, diffusion tensor imaging, computed tomography, positron emission tomography, photoacoustic tomography, x-ray, ultrasound imaging, or a combination thereof.

9. 6. The method of claim 4 or 5, wherein the first type of image is a magnetic resonance image, a diffusion tensor image or map, a computed tomography image, a positron emission tomography image, a photoacoustic tomography image, an X-ray image, a sonography image, or a combination thereof, and the second type of image is a magnetic resonance image, a diffusion tensor image or map, a computed tomography image, a positron emission tomography image, a photoacoustic tomography image, an X-ray image, a sonography image, or a combination thereof.

10. 8. The method of claim 6 or 7, wherein the first characteristic is anisotropy ratio contrast, average diffusivity contrast, axial diffusivity contrast, radial diffusivity contrast, proton density contrast, T1 relaxation time contrast, T2 relaxation time contrast, diffusion coefficient contrast, low resolution, high resolution, drug contrast, radioactive tracer contrast, optical absorption contrast, echo distance contrast, or a combination thereof, and the second characteristic is anisotropy ratio contrast, average diffusivity contrast, the axial diffusivity contrast, radial diffusivity contrast, proton density contrast, T1 relaxation time contrast, T2 relaxation time contrast, diffusion coefficient contrast, low resolution, high resolution, drug contrast, radioactive tracer contrast, optical absorption contrast, echo distance contrast, or a combination thereof.

11. 2. The method of claim 1, wherein the one or more medical imaging modalities are diffusion tensor imaging, the first image is a fractional anisotropy (FA) map, the second image is a mean diffusivity (MD) map, the first property is a ratio anisotropy contrast, the second property is a mean diffusivity contrast, and the object of interest is a kidney of the subject.

12. 12. The method of claim 1, wherein locating and classifying the object in the first image comprises applying one or more clustering algorithms to a plurality of pixels or voxels of the first image.

13. The method of claim 12 , wherein the one or more clustering algorithms comprise a k-means algorithm that assigns observations to clusters associated with the multiple object classes.

14. 14. The method of claim 13, wherein the one or more clustering algorithms further include an Expectation Maximization algorithm that calculates probabilities of cluster membership based on one or more probability distributions, and wherein the k-means algorithm initializes the Expectation Maximization algorithm by estimating initial parameters for each object class of the plurality of object classes.

15. The segmentation mask is determined, determining the segmentation mask comprises: identifying a seed location for the object of interest using the set of pixels or voxels assigned to the object class; growing the seed locations by projecting the seed locations toward a z-axis representing a depth of the segmentation mask; and determining the segmentation mask based on the projected seed positions.

16. The method of claim 15 , wherein determining the segmentation mask further comprises performing morphological closing and filling on the segmentation mask.

17. 17. The method of claim 15 or 16, further comprising cropping the second image based on an object mask plus a margin to generate the portion of the second image prior to inputting the portion of the second image into the 3D neural network model.

18. 18. The method of claim 1, further comprising, prior to inputting the portion of the second image into the three-dimensional neural network model, inputting the second image into a deep super-resolution neural network to increase resolution of the portion of the second image.

19. The three-dimensional neural network model includes a plurality of model parameters identified using a set of training data, the set of training data comprising: A plurality of medical images having annotations associated with segmentation boundaries around objects of interest; a plurality of additional medical images having annotations associated with segmentation boundaries around the object of interest, the plurality of additional medical images being artificially generated by matching image histograms from the plurality of medical images to image histograms from a plurality of reference maps; The method of any one of claims 1 to 18, wherein the plurality of model parameters are identified using the set of training data based on minimizing the weighted loss function.

20. 20. The method of claim 19, wherein the weighted loss function is a weighted Dice loss function.

21. The method according to any one of claims 1 to 20, wherein the three-dimensional neural network model is a modified 3D U-Net model.

22. The method of claim 21 , wherein the modified 3D U-Net model comprises a total of 5,000,000 to 12,000,000 learnable parameters.

23. The method of claim 21 or 22, wherein the modified 3D U-Net model comprises a total of 800 to 1,700 kernels.

24. determining a size, surface area, and / or volume of the object of interest based on an estimated boundary around the object of interest; (i) providing the portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest; The method of any one of claims 1 to 23, further comprising:

25. 25. The method of claim 24, further comprising determining, by a user, a diagnosis of the subject based on (i) the portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest.

26. acquiring, by a user using an imaging system, the medical image of the subject, the imaging system generating the medical image using the one or more medical imaging modalities; determining a size, surface area, and / or volume of the object of interest based on the estimated segmentation boundary around the object of interest; (i) providing the portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest; receiving, by the user, (i) the portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest; determining, by the user, a diagnosis of the subject based on (i) the portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest; The method of any one of claims 1 to 23, further comprising:

27. The method of claim 25 or 26, further comprising administering, by the user, a treatment with a compound based on: (i) the portion of the second image having the estimated segmentation boundary around the object of interest; (ii) the size, surface area, and / or volume of the object of interest; and / or (iii) the subject's diagnosis.

28. 1. A system for segmenting objects in a medical image, comprising: one or more data processors; and a non-transitory computer-readable storage medium comprising instructions that, when executed on the one or more data processors, cause the one or more data processors to perform operations, the operations including: acquiring medical images of a subject, the medical images including a first image having a first characteristic and a second image having a second characteristic, the medical images being generated using one or more medical imaging modalities; using a localization model to locate and classify objects in the first image into a plurality of object classes, the classifying assigning a set of pixels or voxels of the first image to one or more of the plurality of object classes; determining a bounding box or segmentation mask for an object of interest in the first image based on a set of pixels or voxels assigned to an object class among the plurality of object classes using the localization model; transferring the bounding box or the segmentation mask onto the second image to define a portion of the second image that includes the object of interest; inputting the portion of the second image into a three-dimensional neural network model constructed for volumetric segmentation using a weighted loss function; generating an estimated segmentation boundary around the object of interest using the 3D neural network model; and and outputting the portion of the second image having the estimated segmentation boundary around the object of interest using the three-dimensional neural network.

29. 30. The system of claim 28, wherein the one or more medical imaging modalities include a first medical imaging modality and a second medical imaging modality different from the first medical imaging modality, and the first image is generated from the first medical imaging modality and the second image is generated from the second medical imaging modality.

30. 30. The system of claim 28, wherein the one or more medical imaging modalities include a first medical imaging modality and a second medical imaging modality that is the same as the first medical imaging modality, and the first image is generated from the first medical imaging modality and the second image is generated from the second medical imaging modality.

31. 30. The system of claim 28 or 29, wherein the first image is a first type of image and the second image is a second type of image, and the first type of image is different from the second type of image.

32. 32. The system of claim 28 or 31, wherein the first image is a first type of image, the second image is a second type of image, and the first type of image is the same as the second type of image.

33. The system of any one of claims 28 to 32, wherein the first characteristic is different from the second characteristic.

34. The system of any one of claims 28 to 32, wherein the first characteristic is the same as the second characteristic.

35. 31. The system of claim 29 or 30, wherein the first medical imaging modality is magnetic resonance imaging, diffusion tensor imaging, computed tomography, positron emission tomography, photoacoustic tomography, x-ray, ultrasound imaging, or a combination thereof, and the second medical imaging modality is magnetic resonance imaging, diffusion tensor imaging, computed tomography, positron emission tomography, photoacoustic tomography, x-ray, ultrasound imaging, or a combination thereof.

36. 33. The system of claim 31 or 32, wherein the first type of image is a magnetic resonance image, a diffusion tensor image or map, a computed tomography image, a positron emission tomography image, a photoacoustic tomography image, an X-ray image, a sonography image, or a combination thereof, and the second type of image is a magnetic resonance image, a diffusion tensor image or map, a computed tomography image, a positron emission tomography image, a photoacoustic tomography image, an X-ray image, a sonography image, or a combination thereof.

37. 35. The system of claim 33 or 34, wherein the first characteristic is anisotropy ratio contrast, average diffusivity contrast, axial diffusivity contrast, radial diffusivity contrast, proton density contrast, T1 relaxation time contrast, T2 relaxation time contrast, diffusion coefficient contrast, low resolution, high resolution, drug contrast, radioactive tracer contrast, optical absorption contrast, echo distance contrast, or a combination thereof, and the second characteristic is anisotropy ratio contrast, average diffusivity contrast, axial diffusivity contrast, radial diffusivity contrast, proton density contrast, T1 relaxation time contrast, T2 relaxation time contrast, diffusion coefficient contrast, low resolution, high resolution, drug contrast, radioactive tracer contrast, optical absorption contrast, echo distance contrast, or a combination thereof.

38. 29. The system of claim 28, wherein the one or more medical imaging modalities are diffusion tensor imaging, the first image is a fractional anisotropy (FA) map, the second image is a mean diffusivity (MD) map, the first property is anisotropy ratio contrast, the second property is mean diffusivity contrast, and the object of interest is a kidney of the subject.

39. The system of any one of claims 28 to 38, wherein locating and classifying the object in the first image comprises applying one or more clustering algorithms to a plurality of pixels or voxels of the first image.

40. 40. The system of claim 39, wherein the one or more clustering algorithms comprise a k-means algorithm that assigns observations to clusters associated with the multiple object classes.

41. 41. The system of claim 40, wherein the one or more clustering algorithms further comprise an Expectation Maximization algorithm that calculates probabilities of cluster membership based on one or more probability distributions, and wherein the k-means algorithm initializes the Expectation Maximization algorithm by estimating initial parameters for each object class of the plurality of object classes.

42. The segmentation mask is determined, determining the segmentation mask comprises: identifying a seed location for the object of interest using the set of pixels or voxels assigned to the object class; growing the seed locations by projecting the seed locations toward a z-axis representing a depth of the segmentation mask; and determining the segmentation mask based on the projected seed positions.

43. 43. The system of claim 42, wherein determining the segmentation mask further comprises performing morphological closing and filling on the segmentation mask.

44. 44. The system of claim 42 or 43, wherein the operations further comprise cropping the second image based on an object mask plus a margin to generate the portion of the second image prior to inputting the portion of the second image into the 3D neural network model.

45. 45. The system of claim 28, wherein the operations further comprise inputting the second image to a deep super-resolution neural network to increase resolution of the portion of the second image prior to inputting the portion of the second image to the three-dimensional neural network model.

46. The three-dimensional neural network model includes a plurality of model parameters identified using a set of training data, the set of training data comprising: A plurality of medical images having annotations associated with segmentation boundaries around objects of interest; a plurality of additional medical images having annotations associated with segmentation boundaries around the object of interest, the plurality of additional medical images being artificially generated by matching image histograms from the plurality of medical images to image histograms from a plurality of reference maps; The system of any one of claims 28 to 45, wherein the plurality of model parameters are identified using the set of training data based on minimizing the weighted loss function.

47. 47. The system of claim 46, wherein the weighted loss function is a weighted Dice loss function.

48. The system of any one of claims 28 to 37, wherein the three-dimensional neural network model is a modified 3D U-Net model.

49. 49. The system of claim 48, wherein the modified 3D U-Net model includes a total of 5,000,000 to 12,000,000 learnable parameters.

50. The system of claim 48 or 49, wherein the modified 3D U-Net model includes a total of 800 to 1,700 kernels.

51. The operation, determining a size, surface area, and / or volume of the object of interest based on an estimated boundary around the object of interest; 51. The system of claim 28, further comprising: (i) providing the portion of the second image having the estimated segmentation boundary around the object of interest; and / or (ii) a size, surface area, and / or volume of the object of interest.

52. 52. The system of claim 51 , wherein the operations further include determining, by a user, a diagnosis of the subject based on (i) the portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest.

53. The operation, acquiring, by a user using an imaging system, the medical image of the subject, the imaging system generating the medical image using the one or more medical imaging modalities; determining a size, surface area, and / or volume of the object of interest based on the estimated segmentation boundary around the object of interest; (i) providing the portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest; receiving, by a user, (i) the portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest; determining, by the user, a diagnosis of the subject based on (i) the portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest; The system of any one of claims 28 to 52, further comprising:

54. The system of claim 52 or 53, wherein the actions further include administering, by the user, a treatment with a compound based on: (i) the portion of the second image having the estimated segmentation boundary around the object of interest, (ii) the size, surface area, and / or volume of the object of interest, and / or (iii) the subject's diagnosis.

55. A computer program product tangibly embodied in a non-transitory machine-readable storage medium that includes instructions configured to cause one or more data processors to perform operations, said operations including: acquiring medical images of a subject, the medical images including a first image having a first characteristic and a second image having a second characteristic, the medical images being generated using one or more medical imaging modalities; using a localization model to locate and classify objects in the first image into a plurality of object classes, the classifying assigning a set of pixels or voxels of the first image to one or more of the plurality of object classes; determining a bounding box or segmentation mask for an object of interest in the first image based on a set of pixels or voxels assigned to an object class among the plurality of object classes using the localization model; transferring the bounding box or the segmentation mask onto the second image to define a portion of the second image that includes the object of interest; inputting the portion of the second image into a three-dimensional neural network model constructed for volumetric segmentation using a weighted loss function; generating an estimated segmentation boundary around the object of interest using the 3D neural network model; and and using the three-dimensional neural network to output the portion of the second image having the estimated segmentation boundary around the object of interest.

56. 56. The computer program product of claim 55, wherein the one or more medical imaging modalities include a first medical imaging modality and a second medical imaging modality different from the first medical imaging modality, and the first image is generated from the first medical imaging modality and the second image is generated from the second medical imaging modality.

57. 56. The computer program product of claim 55, wherein the one or more medical imaging modalities include a first medical imaging modality and a second medical imaging modality that is the same as the first medical imaging modality, and the first image is generated from the first medical imaging modality and the second image is generated from the second medical imaging modality.

58. 57. A computer program product according to claim 55 or 56, wherein the first image is a first type of image and the second image is a second type of image, and the first type of image is different from the second type of image.

59. 58. The computer program product of claim 55 or 57, wherein the first image is a first type of image, the second image is a second type of image, and the first type of image is the same as the second type of image.

60. 60. A computer program product according to any one of claims 55 to 59, wherein the first characteristic is different from the second characteristic.

61. 61. A computer program product according to any one of claims 55 to 60, wherein the first characteristic is the same as the second characteristic.

62. 58. The computer program product of claim 56 or 57, wherein the first medical imaging modality is magnetic resonance imaging, diffusion tensor imaging, computed tomography, positron emission tomography, photoacoustic tomography, x-ray, ultrasound imaging, or a combination thereof, and the second medical imaging modality is magnetic resonance imaging, diffusion tensor imaging, computed tomography, positron emission tomography, photoacoustic tomography, x-ray, ultrasound imaging, or a combination thereof.

63. 60. The computer program product of claim 58 or 59, wherein the first type of image is a magnetic resonance image, a diffusion tensor image or map, a computed tomography image, a positron emission tomography image, a photoacoustic tomography image, an X-ray image, a sonography image, or a combination thereof, and the second type of image is a magnetic resonance image, a diffusion tensor image or map, a computed tomography image, a positron emission tomography image, a photoacoustic tomography image, an X-ray image, a sonography image, or a combination thereof.

64. 62. The computer program product of claim 60 or 61, wherein the first characteristic is anisotropy ratio contrast, average diffusivity contrast, axial diffusivity contrast, radial diffusivity contrast, proton density contrast, T1 relaxation time contrast, T2 relaxation time contrast, diffusion coefficient contrast, low resolution, high resolution, drug contrast, radioactive tracer contrast, optical absorption contrast, echo distance contrast, or a combination thereof, and the second characteristic is anisotropy ratio contrast, average diffusivity contrast, the axial diffusivity contrast, radial diffusivity contrast, proton density contrast, T1 relaxation time contrast, T2 relaxation time contrast, diffusion coefficient contrast, low resolution, high resolution, drug contrast, radioactive tracer contrast, optical absorption contrast, echo distance contrast, or a combination thereof.

65. 56. The computer program product of claim 55, wherein the one or more medical imaging modalities are diffusion tensor imaging, the first image is a fractional anisotropy (FA) map, the second image is a mean diffusivity (MD) map, the first property is a ratio anisotropy contrast, the second property is a mean diffusivity contrast, and the object of interest is a kidney of the subject.

66. 66. The computer program product of any one of claims 55 to 65, wherein locating and classifying the object in the first image comprises applying one or more clustering algorithms to a plurality of pixels or voxels of the first image.

67. 67. The computer program product of claim 66, wherein the one or more clustering algorithms comprise a k-means algorithm that assigns observations to clusters associated with the multiple object classes.

68. 68. The computer program product of claim 67, wherein the one or more clustering algorithms further comprise an Expectation Maximization algorithm that calculates probabilities of cluster membership based on one or more probability distributions, and wherein the k-means algorithm initializes the Expectation Maximization algorithm by estimating initial parameters for each object class of the plurality of object classes.

69. The segmentation mask is determined, determining the segmentation mask comprises: identifying a seed location for the object of interest using the set of pixels or voxels assigned to the object class; growing the seed locations by projecting the seed locations toward a z-axis representing a depth of the segmentation mask; and determining the segmentation mask based on the projected seed positions.

70. 70. The computer program product of claim 69, wherein determining the segmentation mask further comprises performing morphological closing and filling on the segmentation mask.

71. 71. The computer program product of claim 69 or 70, wherein the operations further comprise cropping the second image based on an object mask plus a margin to generate the portion of the second image prior to inputting the portion of the second image into the three-dimensional neural network model.

72. 72. The computer program product of any one of claims 55 to 71, wherein the operations further comprise inputting the second image to a deep super-resolution neural network to increase resolution of the portion of the second image prior to inputting the portion of the second image to the three-dimensional neural network model.

73. The three-dimensional neural network model includes a plurality of model parameters identified using a set of training data, the set of training data comprising: A plurality of medical images having annotations associated with segmentation boundaries around objects of interest; a plurality of additional medical images having annotations associated with segmentation boundaries around the object of interest, the plurality of additional medical images being artificially generated by matching image histograms from the plurality of medical images to image histograms from a plurality of reference maps; 73. The computer program product of any one of claims 55 to 72, wherein the plurality of model parameters are identified using the set of training data based on minimizing the weighted loss function.

74. 74. The computer program product of claim 73, wherein the weighted loss function is a weighted Dice loss function.

75. 75. The computer program product of any one of claims 55 to 74, wherein the three-dimensional neural network model is a modified 3D U-Net model.

76. 76. The computer program product of claim 75, wherein the modified 3D U-Net model comprises a total of 5,000,000 to 12,000,000 learnable parameters.

77. 77. The computer program product of claim 75 or 76, wherein the modified 3D U-Net model comprises a total of 800 to 1,700 kernels.

78. The operation, determining a size, surface area, and / or volume of the object of interest based on the estimated boundary around the object of interest; 78. The computer program product of any one of claims 55 to 77, further comprising: (i) providing the portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest.

79. 80. The computer program product of claim 78, wherein the operations further include determining, by a user, a diagnosis of the subject based on (i) the portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest.

80. The operation, acquiring, by a user using an imaging system, the medical image of the subject, the imaging system generating the medical image using the one or more medical imaging modalities; determining a size, surface area, and / or volume of the object of interest based on the estimated segmentation boundary around the object of interest; (i) providing the portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest; receiving, by a user, (i) the portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest; 80. The computer program product of claim 55, further comprising: determining, by the user, a diagnosis of the subject based on (i) the portion of the second image having the estimated segmentation boundary around the object of interest, and / or (ii) a size, surface area, and / or volume of the object of interest.

81. 81. The computer program product of claim 79 or 80, wherein the operations further comprise administering, by the user, a treatment with a compound based on: (i) the portion of the second image having the estimated segmentation boundary around the object of interest; (ii) the size, surface area, and / or volume of the object of interest; and / or (iii) the subject's diagnosis.

Images