Connected machine learning models with joint training for lesion detection

Abstract

[0001] Embodiments disclosed herein generally relate to a connected machine learning model using joint training for lesion detection. In particular, aspects of the present disclosure relate to accessing three-dimensional magnetic resonance imaging (MRI) images, where the three-dimensional MRI images display regions of a subject's brain, the brain regions including at least a first type of lesion and a second type of lesion; inputting the three-dimensional MRI images into a machine learning model including a first convolutional neural network and a second convolutional neural network; generating a first segmentation mask of the first type of lesion using the first convolutional neural network that takes the three-dimensional MRI images as input; generating a second segmentation mask of the second type of lesion using the second convolutional neural network that takes the three-dimensional MRI images as input; and outputting the first segmentation mask and the second segmentation mask.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 073,686 (filed September 2, 2020), which is incorporated herein by reference in its entirety for all purposes. [Background technology]

[0002] background Multiple sclerosis (MS) is a disease that affects the central nervous system. As a result of the disease, lesions form in the brains of subjects with MS. In most cases, MS initially manifests as relapsing-remitting MS, characterized by spatial and temporal separation. For example, a subject may experience multiple symptomatic episodes affecting different body regions and / or different functional systems. As another example, a subject may experience lesions in different parts of the central nervous system. As yet another example, a subject may experience symptoms and have lesions in brain regions that do not correspond to a system. Over time, the size and number of lesions in MS subjects frequently increase, reflecting disease worsening. Relapsing-remitting MS typically eventually progresses to secondary progressive MS, during which gradual deterioration of symptoms and brain health is observed rather than discrete episodes of symptomatic relapse and recovery. Primary progressive MS is similarly characterized by gradual symptom worsening and brain degradation, but this diagnosis is limited to subjects who have not previously experienced relapsing-remitting MS.

[0003] Magnetic resonance imaging (MRI) can provide visualization of new (contrast-enhanced) lesions, old lesions, and brain atrophy (via brain size changes). These metrics are the foundation for supporting MS diagnosis, facilitating MS prognosis, selecting MS treatments, and evaluating MS treatments. MRI images are generated using MRI sequences, which are specific settings of radiofrequency pulses and gradients that result in specific image appearances. The three main MRI sequence types include T1 sequences, T2 sequences, and fluid-attenuated inversion recovery (FLAIR) sequences.

[0004] T1 MRI is acquired relatively quickly and can provide structural information. Black holes, which are plaques indicating axonal destruction and neuronal death, appear dark on T1 scans. T2 MRI is frequently used to show total disease burden. New and old lesions appear as hyperintense areas on T2 scans. FLAIR MRI is similar to T2 MRI, except that the ventricles and cerebrospinal fluid appear dark on FLAIR MRI (bright on T2 MRI). Therefore, lesions that appear near the ventricles (e.g., in the corpus callosum) are more visible on FLAIR MRI.

[0005] In some cases, a contrast agent (e.g., gadolinium) is administered to the subject (e.g., intravenously) before acquiring one or more MRI scans to improve the visibility of active lesions. If the blood-brain barrier is intact, the contrast agent does not enter the central nervous system. If the blood-brain barrier is disrupted, the contrast agent can migrate into the central nervous system and localize to areas of inflammation. The contrast agent appears as hyperintense on T1 MRI.

[0006] Traditionally, radiologists manually annotate MRI scans. However, given that scans are collected at various depths and perspectives, as well as different sequence types, there are many scans to annotate. Furthermore, there is significant variability in annotation across radiologists. Therefore, it would be advantageous to identify automated techniques that can process MRI images to improve efficiency and consistency. Summary of the Invention

[0007] overview In some embodiments, a computer-implemented method is provided. A three-dimensional magnetic resonance imaging (MRI) image is accessed. The three-dimensional MRI image displays a region of the subject's brain including at least a first type of lesion (e.g., a T1 non-enhancing lesion) and a second type of lesion (e.g., a T1 Gd-enhancing lesion). The three-dimensional MRI image is input to a machine learning model including a first convolutional neural network and a second convolutional neural network. The first convolutional neural network is connected to the second convolutional network using one or more cross-network connections between one or more layers of the first convolutional neural network and one or more layers of the second convolutional neural network. A first segmentation mask of the first type of lesion is generated using the first convolutional neural network that takes the three-dimensional MRI image as input. The first segmentation mask includes a first estimated segmentation boundary around a representation of the first type of lesion. A second segmentation mask of the second type of lesion is generated using a second convolutional neural network that takes the three-dimensional MRI image as input. The second segmentation mask includes a second estimated segmentation boundary around the representation of the second type of lesion. The first segmentation mask and the second segmentation mask are generated in parallel, and the set of extracted features is shared across one or more cross-network connections while the first segmentation mask and the second segmentation mask are generated. The first segmentation mask and the second segmentation mask are output.

[0008] In some embodiments, the first convolutional neural network is connected to the second convolutional neural network using one or more cross-network connections from the coding block of the first convolutional neural network to the coding block of the second convolutional neural network.

[0009] In some embodiments, the first convolutional neural network is connected to the second convolutional neural network using one or more cross-network connections from the decoding block of the first convolutional neural network to the decoding block of the second convolutional neural network.

[0010] In some embodiments, each of the cross-network connections includes a feature transformation block and a compression block.

[0011] In some embodiments, the computer-implemented method can include generating a final image mask by combining information from the first segmentation mask and the second segmentation mask. The final image mask includes an estimated segmentation boundary around a representation of a new set of lesions representing a combination of the first type of lesion and the second type of lesion. The final image mask is output.

[0012] In some embodiments, the first convolutional neural network and the second convolutional network were trained using a loss function comprising a combination of Tversky loss and weighted binary cross-entropy loss.

[0013] In some embodiments, the first convolutional neural network includes a plurality of model parameters identified using a set of training data including a plurality of medical images having annotations associated with segmentation boundaries around representations of various types of lesions, including a first type of lesion and a second type of lesion, and the plurality of model parameters are identified using the set of training data based on minimizing a loss function.

[0014] In some embodiments, the second convolutional neural network includes a plurality of model parameters identified using the set of training data, and the second convolutional neural network is co-trained with the first convolutional neural network such that the plurality of model parameters of the second convolutional neural network are collaboratively identified using the set of training data based on minimizing a loss function.

[0015] In some embodiments, the loss function further includes penalty terms for any overlap between the first type of lesion and the second type of lesion and for false positive contributions from the first convolutional neural network and the second convolutional neural network.

[0016] In some embodiments, the first convolutional neural network and the second convolutional network are U-Nets.

[0017] In some embodiments, the first convolutional neural network includes three layers of encoding and decoding blocks.

[0018] In some embodiments, the second convolutional neural network includes three layers of encoding and decoding blocks.

[0019] In some embodiments, the three-dimensional MRI image includes multiple consecutive slices stacked along the channel dimension, including a first three-dimensional MRI slice generated using a first type of MRI sequence (e.g., post-T1 contrast), a second three-dimensional MRI slice generated using a second type of MRI sequence (e.g., pre-T1 contrast), and a third three-dimensional MRI slice generated using a third type of MRI sequence (e.g., fluid-attenuated inversion recovery).

[0020] In some embodiments, the first three-dimensional MRI slice and the second three-dimensional slice are normalized to capture contrast agent enhancement, where the normalization includes using the mean and standard deviation of the second three-dimensional slice to z-score both the first three-dimensional MRI slice volume and the second three-dimensional slice volume.

[0021] In some embodiments, the computer-implemented method may include determining a count of the number of lesions of a first type using a first segmentation mask, and determining a count of the number of lesions of a second type using a second segmentation mask.

[0022] In some embodiments, the computer-implemented method may include determining a count of the number of lesions of a first type using a first segmentation mask, determining a count of the number of lesions of a second type using a second segmentation mask, and / or determining a count of the number of lesions in a new set of lesions using a final image mask.

[0023] In some embodiments, the computer-implemented method may include determining one or more lesion sizes or lesion burdens using the first segmentation mask, the second segmentation mask, and / or the final image mask.

[0024] In some embodiments, the computer-implemented method can include accessing data corresponding to a previous MRI, determining a change in volume, size, or cumulative size of one or more lesions using the first segmentation mask, the second segmentation mask, and / or the final image mask and the data, and generating an output representative of the change.

[0025] In some embodiments, the computer-implemented method can include recommending a change in treatment strategy based on the first segmentation mask, the second segmentation mask, and / or the final image mask.

[0026] In some embodiments, the computer-implemented method can include providing an output corresponding to a possible or confirmed diagnosis of the subject with multiple sclerosis based at least in part on the first segmentation mask, the second segmentation mask, and / or the final image mask.

[0027] In some embodiments, the computer-implemented method may include diagnosing the subject as having multiple sclerosis based at least in part on the first segmentation mask, the second segmentation mask, and / or the final image mask.

[0028] In some embodiments, the computer-implemented method may include assessing and / or predicting a treatment response based at least in part on the first segmentation mask, the second segmentation mask, and / or the final image mask.

[0029] In some embodiments, a computer-implemented method is provided. A three-dimensional MRI image is accessed. The three-dimensional MRI image displays a region of the subject's brain including at least a first type of lesion and a second type of lesion. The three-dimensional MRI image is input to a machine learning model including a first convolutional neural network connected to a second convolutional neural network. A first segmentation mask of the first type of lesion is generated using the first convolutional neural network that takes the three-dimensional MRI image as input. A second segmentation mask of the second type of lesion is generated using the second convolutional neural network that takes the first segmentation mask on the three-dimensional MRI image as input. A final image mask is generated by combining information from the first segmentation mask and the second segmentation mask. The final image mask includes a first estimated segmentation boundary around a representation of the first type of lesion and / or a second estimated segmentation boundary around a representation of the second type of lesion. The final image mask is output.

[0030] Some embodiments of the present disclosure include a system including one or more data processors. The system may further include a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0031] In some embodiments, a computer program product tangibly embodied in a non-transitory machine-readable storage medium is provided, which may include instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein. [Brief explanation of the drawings]

[0032] [Figure 1]1 illustrates an exemplary computing environment for segmenting representations of different types of lesions in images using a model having multiple connected networks, according to various embodiments.

[0033] [Figure 2] 1 illustrates an exemplary model including a first modified U-Net used to segment representations of non-enhancing lesions and a second modified U-Net used to segment representations of Gd-enhancing lesions, according to various embodiments.

[0034] [Figure 3] 1 illustrates an exemplary processing environment for segmenting representations of different types of lesions in an image using a model having multiple connected networks, according to various embodiments.

[0035] [Figure 4] 1 shows exemplary histograms of total T1 non-enhancing lesion volume and total T1 gadolinium (Gd)-enhancing lesion volume for a dataset.

[0036] [Figure 5] We show example predicted lesion segmentations from the model compared to ground truth magnetic resonance imaging (MRI) images of portions of the brains of three subjects.

[0037] [Figure 6A] 1 shows exemplary results for T1 Gd-enhancing lesion counts for two treatment groups at each of three time points.

[0038] [Figure 6B] 10 shows exemplary results of T1 non-enhancing lesion counts from ground truth masks using an empirical approach and model predicted masks using the same empirical approach. DETAILED DESCRIPTION OF THE INVENTION

[0039] Detailed Description I. Overview This disclosure describes techniques for automated lesion segmentation in medical images. More specifically, embodiments of the disclosure provide techniques for segmenting representations of various types of lesions (e.g., non-enhancing T1 lesions and gadolinium (Gd)-enhancing T1 lesions) in medical images (e.g., pre- and post-contrast T1w images) using co-trained deep learning networks with connections between the deep learning networks.

[0040] In MS, various deep learning models have been developed to segment T2 hyperintense lesions, and recently, a separate deep learning model was developed to segment T1 Gd-enhancing lesions. However, the use of two independent deep learning models does not adequately capture the discontinuity / non-overlapping nature of T1 nonenhancing and Gd-enhancing lesion types, especially for lesions with low contrast enhancement, resulting in an increased number of false positives (FPs). Due to the large disparity in lesion prevalence and size, using a single deep learning model to segment both lesion types in a multi-class setting is challenging because the deep learning model will only be trained to segment the lesion type with a high prevalence. Furthermore, these two types of T1 lesions have significant differences in their shape characteristics and may not occur adjacent to each other in MS, so there may be little overlap in the features required for segmenting these two types of T1 lesions.

[0041] To address these limitations and problems, the present embodiment of a technique for automated object segmentation utilizes multiple (e.g., two or more) deep learning networks jointly trained to segment representations of various types of lesions, such as non-enhancing T1 lesions and Gd-enhancing T1 lesions. For example, a data processing system accesses a three-dimensional MRI image. The three-dimensional MRI image displays a region of a subject's brain, the brain region including at least a first type of lesion (e.g., a non-enhancing lesion) and a second type of lesion (e.g., a Gd-enhancing lesion). The three-dimensional MRI image is input to a machine learning model including multiple connected networks, such as a first convolutional neural network connected to a second convolutional neural network. The first convolutional neural network may be connected to the second convolutional network using one or more cross-network connections between one or more layers of the first convolutional neural network and one or more layers of the second convolutional neural network. The first convolutional neural network is configured to generate a first segmentation mask of the first type of lesion using the three-dimensional MRI image as input. The first segmentation mask includes a first estimated segmentation boundary around a representation of the first type of lesion. The second convolutional neural network is configured to generate a second segmentation mask of a second type of lesion using the three-dimensional MRI image as input. The second segmentation mask includes a second estimated segmentation boundary around a representation of the second type of lesion. The first segmentation mask and the second segmentation mask are generated in parallel, and the set of extracted features is shared across a cross-network connection while the first segmentation mask and the second segmentation mask are generated. The first segmentation mask and the second segmentation mask are output and can be further analyzed downstream (e.g., used to determine the number of lesions of each type using the first and second segmentation masks).

[0042] II. Techniques for Segmenting Medical Images Image segmentation is the procedure of separating an image into parts that exhibit similarities in various features, such as shape, size, and color. Segmentation of lesion representations allows for visualization of the size and location of lesions within a body region (e.g., the brain) and can also provide a basis for analyzing treatments. The gold standard for lesion segmentation has long been manual segmentation, which is time-consuming, labor-intensive, and therefore unsuitable for large-scale studies. Considerable research has attempted to fully or partially automate the process of lesion segmentation. For example, image segmentation techniques such as thresholding, region growing, fuzzy clustering, and the use of watershed algorithms have been used to separate representations of abnormal tissue (e.g., lesions) from representations of normal tissue, such as brain white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF). Nevertheless, the segmentation process remains challenging due to the diversity in the shape, location, and size of lesion representations.

[0043] This paper describes an end-to-end method that incorporates a model that uses multiple three-dimensional convolutional neural networks (CNNs) to segment lesion representations and extract treatment response predictions based on lesion segmentation (e.g., T1 Gd-enhanced lesion segmentation). The developed model is computationally lightweight (e.g., it can run on an average desktop computing device and return predictions on-demand, e.g., within minutes) and is devised to accommodate the complexity of three-dimensional scans, the extreme imbalance between various lesion types, and the heterogeneity of input images (e.g., variable density and object size). As used herein, a "scan" is a graphical representation of the signal on a single plane through a subject's body. The model has lesion segmentation performance comparable to conventional algorithms that rely on manual intervention (e.g., manual selection of seeds or manual identification of bounding boxes), such as thresholding, edge-based, or region-based segmentation methods, and demonstrates high correlation with ground-truth manual readings.

[0044] The prediction or lesion information obtained by lesion segmentation can then be used alone or in combination with other factors to determine the number of lesions (e.g., some T1 non-enhancing lesions and / or T1 Gd-enhancing lesions) and / or determine the size or lesion burden (sometimes referred to as lesion burden) of one or more lesions. The method can also be used to evaluate the clinical effectiveness and / or prognosis of a subject's treatment. The method can also be used to determine a change in the amount, size, or cumulative size of one or more lesions using the prediction and data and generate an output representing that change. The model output can further be used to recommend a change in treatment strategy based on the prediction. The method can also be used to provide a diagnosis for the subject, for example, to provide an output corresponding to a possible or confirmed diagnosis of a subject with multiple sclerosis based at least in part on the prediction, or to diagnose a subject with multiple sclerosis based at least in part on the prediction.

[0045] II.A. Exemplary Computing Environment 1 illustrates an exemplary computing environment 100 (i.e., a data processing system) for segmenting representations of various types of lesions in images using a model having multiple connected networks, according to various embodiments. As shown in FIG. 1, the segmentation performed by the computing environment 100 in this example includes several stages: an image acquisition stage 105, a model training stage 110, a segmentation stage 115, and an analysis stage 120.

[0046] The image acquisition stage 105 includes one or more imaging systems 130 (e.g., MRI imaging systems) for acquiring input images 135 (e.g., MRI images) of various portions of the subject. The imaging systems 130 are configured to acquire the input images 135 using radiological imaging techniques such as MRI. The imaging systems 130 can determine differences between various structures and functions within the subject based on characteristics associated with each of the imaging systems 130 (e.g., brightness, contrast, and spatial resolution) and generate a series of two-dimensional images. Once the series of two-dimensional images are collected by the scanner's computer, the two-dimensional images can be digitally "stacked" together through computer analysis to reconstruct a three-dimensional image of the subject or portion of the subject. The two-dimensional images and / or reconstructed three-dimensional input images 135 enable easier identification and location of underlying structures (e.g., organs) and possible lesions or abnormalities. Each two-dimensional image and / or reconstructed three-dimensional input image 135 corresponds to a session time and subject and can display an internal region of the subject. Each two-dimensional image and / or reconstructed three-dimensional input image 135 may further be of a standardized size, resolution, and / or magnification.

[0047] The model training stage 110 builds and trains one or more models 140a-140n (where "n" represents the total number of models) (which may be referred to herein individually as models 140 or collectively as models 140) to be used by other stages. The models 140 may be machine learning ("ML") models including multiple networks, such as, for example, an initial neural network, a residual neural network ("Resnet"), a U-Net, a V-Net, a single-shot multi-box detector ("SSD") network, or a convolutional neural network ("CNN"), such as, for example, a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model, or any combination thereof. Model 140 may also be any other suitable ML model trained in object detection and / or segmentation from images, such as a three-dimensional CNN ("3DCNN"), a dynamic time warping and warping ("DTW") technique, a hidden Markov model ("HMM"), or a combination of one or more of such techniques, e.g., a CNN-HMM or an MCNN (multiscale convolutional neural network). The computing environment 100 may use the same or different types of models to segment the representation of different types of lesions. In a particular example, model 140 is constructed by a combined asymmetric loss function, e.g., a combination of a Tversky loss and a weighted binary cross entropy (wBCE) loss, to train each network. In the case of joint training of networks within model 140, a penalty term may be added for overlap between lesion types and false positive contributions from the networks in addition to the network losses.

[0048] To train model 140 in this example, training images 145 are generated by acquiring digital images, dividing the images into a subset of training images 145a for training (e.g., 90%) and a subset of training images 145b for validation (e.g., 10%), preprocessing the subset of training images 145a and the subset of training images 145b, expanding the subset of training images 145a, and optionally annotating the subset of training images 145a with labels 150. The subset of training images 145a are acquired from one or more imaging modalities (e.g., MRI T1, T2, proton density (PD), or FLAIR). Optionally, the subset of training images 145a are acquired from a data storage structure, such as a database, imaging system (e.g., one or more imaging systems 130), associated with one or more imaging modalities. Each image displays one or more lesions.

[0049] The partitioning may be performed randomly (e.g., 90 / 10% or 70 / 30%), or according to more complex validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, and nested cross-validation to minimize sampling bias and overfitting. Preprocessing may include bias field correction to correct for low-frequency intensity heterogeneity present in the MRI image data, known as the bias or gain field. In some cases, preprocessing may include identifying the MRI baseline image transformation to an MNI template image (e.g., MNI space) using rigid registration. In some cases, preprocessing may include rigorous registration of baseline and follow-up visits to skull stripping. In some cases, preprocessing may further include standardization or normalization to place all features on the same scale (e.g., the same size scale or the same color or saturation scale). In a particular example, the image is resized to a minimum size (width or height) of a predetermined number of pixels (e.g., 2500 pixels) or a maximum size (width or height) of a predetermined number of pixels (e.g., 3000 pixels), while maintaining the original aspect ratio.

[0050] Augmentation can be used to artificially increase the size of a subset of training images 145a by creating modified versions of images in the dataset. Image data augmentation can be performed by creating transformed versions of images in the dataset that belong to the same class as the original image. Transformations include a range of operations from the field of image manipulation, such as shift, flip, and zoom. In some cases, operations include random erasure, shift, brightness, rotation, Gaussian blur, and / or elastic transformation to ensure that the model 140 can perform under conditions outside those available from the subset of training images 145a.

[0051] Annotation can be performed manually by one or more humans (annotators, such as radiologists or pathologists) who identify the presence of one or more lesion indications in each image of the subset of training images 145a and provide labels 150 for the one or more lesions, e.g., using annotation software to draw a bounding box (ground truth) or segmentation boundary around an area identified by the human as containing one or more lesion indications. In certain examples, bounding boxes or segmentation boundaries can be drawn only for cases where the probability of a lesion being present is greater than 50%. For images annotated by multiple annotators, the bounding boxes or segmentation boundaries from all annotators may be used. In some cases, the annotation data can further indicate the type of lesion. For example, for various types of lesions, the annotation data can indicate the type, such as T2 hyperintense lesion or T1 hypointense lesion.

[0052] In some cases, a subset of training images 145 may be sent to the annotator device for inclusion in the training dataset (i.e., a subset of training images 145a). Input can be provided to the annotator device (e.g., by a radiologist) using a mouse, trackpad, stylus, and / or keyboard indicating (for example) whether the image displays an object of interest (e.g., a lesion, organ, etc.); the number and type of lesions shown in the image; and the perimeter (bounding box or segmentation boundary) of each displayed lesion in the image. The annotator device can be configured to generate a label 150 for each image using the provided input. For example, label 150 can include the number and / or type of lesions shown in the image; a type classification of each displayed lesion; the number of each displayed lesion of a particular type; and the perimeter and / or mask of one or more identified lesions in the image. In some cases, label 150 can further include the perimeter and / or mask of one or more identified lesions overlaid on the medical image.

[0053] In some cases, the model 140 is trained and used to process digital images of three-dimensional (3D) patches of the brain. The training data includes a first subset of training MRI images, each displaying at least a portion of one or more lesions, and a second subset of training MRI images, each lacking any lesions. The training data can include images generated using various MRI sequence types. For example, a training dataset can include three-dimensional MRI images generated using a T1 sequence, a T2 sequence, and a FLAIR sequence. The training data can include labels for MRI images that display lesions and MRI images that do not display lesions. The labels can be received, identified, or derived from data received from a database or provider system. The label data can include, for example, an indication for a single image, whether the image displays at least a portion of a lesion, the amount of lesion at least partially displayed in the image, the location within the image (e.g., a particular voxel) that displays at least a portion of a lesion, the type of lesion (e.g., an enhancing or non-enhancing lesion) that is at least partially displayed in the image or at a particular location within the image, etc.

[0054] The training data used to train model 140 can include MRI images acquired after a contrast agent, such as gadolinium (Gd), was administered to a subject, MRI images showing at least a portion of one or more contrast-enhancing lesions (referred to as Gd-enhancing lesions), MRI images acquired without a contrast agent being administered to the subject, MRI images acquired before a contrast agent was administered to the subject, and / or MRI images that do not display any contrast-enhancing lesions. For example, model 140 can be trained on only images acquired without a contrast agent being recently administered to the subject, only images acquired after a contrast agent was recently administered to the subject, or some of both types of images. It will be appreciated that images acquired after a contrast agent has been administered may not show any enhancing lesions, in some cases in situations where the blood-brain barrier is intact and / or if a given scan does not show brain regions through which the contrast agent has traveled.

[0055] In a particular case, the model 140 is trained using a weighted binary cross entropy (wBCE) loss or a Tversky loss. The Tversky loss can reduce false positives in predictions, and the wBCE can help identify smaller lesions. Additionally, a combination of the Tversky loss and the wBCE loss can be used to calculate the loss. For the Tversky loss, the following equation (1) can be used: JPEG0007766678000001.jpg23170

[0056] In the above formula (1), JPEG0007766678000002.jpg8170 is the model-predicted probability that voxel i belongs to a lesion class (e.g., T1-nonenhanced, Gd-enhanced), JPEG0007766678000003.jpg8170 is the probability that a voxel does not belong to the lesion class. JPEG0007766678000004.jpg9170 indicates whether a voxel belongs to the lesion class or not with ground truth annotation. β weights the contribution from false positives (FP), and (1-β) weights the contribution from false negatives (FN). For wBCE, the following equation (2) can be used: JPEG0007766678000005.jpg15170

[0057] In the above equation (2), g and p represent the ground truth and model-predicted lesion masks, respectively. To improve the sensitivity for detecting small lesions, a wBCE loss can be added. However, wBCE can result in an increase in FP. To account for this, FP can be weighted more heavily than FN in the Tversky loss term. The loss is calculated using equation (3): JPEG0007766678000006.jpg12170

[0058] For joint training, in addition to the losses of the multiple networks, terms can be added for the overlap of the two lesion types and the false positive contribution from the networks using the following equation (4): JPEG0007766678000007.jpg12170

[0059] In the above formula (4), JPEG0007766678000008.jpg8170T1 Corresponding to the loss of the top and bottom models for segmenting non-enhancing and Gd-enhancing lesions, JPEG0007766678000009.jpg8170 Corresponding to the prediction model and ground truth (GT) mask for de-enhancing lesions, JPEG0007766678000010.jpg8170Corresponds to the prediction model and GT mask for Gd-enhancing lesions. JPEG0007766678000011.jpg9170To explain the lower prevalence of Gd-enhancing lesions JPEG0007766678000012.jpg10170 can be weighted.

[0060] The training process for model 140 involves selecting hyperparameters for model 140 and performing iterations of inputting images from a subset of images 145a into model 140 to find a set of model parameters (e.g., weights and / or biases) that minimize a loss or error function for model 140. Hyperparameters are settings that can be tuned or optimized to control the behavior of model 140. Most models explicitly define hyperparameters that control various aspects of the model, such as memory or execution cost. However, additional hyperparameters can be defined to adapt the model to specific scenarios. For example, hyperparameters can include the number of hidden units in the model, the learning rate of the model, the convolution kernel width, or the number of kernels in the model. Each iteration of training involves finding a set of model parameters for model 140 (configured by the defined set of hyperparameters) such that the value of the loss or error function using the set of model parameters is smaller than the value of the loss or error function using a different set of model parameters in the previous iteration. A loss or error function can be constructed to measure the difference between the output inferred using the model 140 and the ground truth segmentation boundaries annotated to the image using the labels 150.

[0061] Once a set of model parameters has been identified, the model 140 can be trained and validated using a subset of the training images 145b (a test or validation dataset). The validation process involves iteratively inputting images from the subset of training images 145b into the model 140 using validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, and nested cross-validation to adjust the hyperparameters and ultimately find an optimal set of hyperparameters. Once an optimal set of hyperparameters is obtained, a reserved test set of images from the subset of training images 145b is input into the model 140 to obtain an output (in this example, a segmentation boundary around one or more lesion representations), which is then evaluated against the ground truth segmentation boundary using correlation techniques such as the Bland-Altman method and Spearman's rank correlation coefficient, and calculating performance metrics such as error, precision, precision, recall, and receiver operating characteristic curves (ROC).

[0062] As should be understood, other training / validation mechanisms are envisioned and may be implemented within computing environment 100. For example, a model may be trained and hyperparameters may be tuned on images from a subset of training images 145a, and images from a subset of training images 145b may be used only to test and evaluate the model's performance. Furthermore, while the training mechanisms described herein focus on training new models 140, these training mechanisms may also be utilized to fine-tune existing models 140 trained from other datasets. For example, in some cases, model 140 may have been pre-trained using images of other objects or biological structures, or from sections from other subjects or studies (e.g., human trials or mouse experiments). In those cases, model 140 may be used for transfer learning and retrained / validated using input images 135.

[0063] The model training stage 110 outputs a trained model, including one or more trained segmentation models 165, including multiple networks (e.g., a first U-Net connected to a second U-Net). One or more medical input images 135 are acquired by a segmentation controller 170 within the segmentation stage 115. In various examples, the medical input images 135 are three-dimensional MRI images. Each of the three-dimensional MRI images displays the same region of the subject's brain. The brain region includes at least a first type of lesion (e.g., a T1 non-enhancing lesion) and a second type of lesion (e.g., a T1 Gd-enhancing lesion). Each three-dimensional MRI image includes a three-dimensional patch of the complete MRI image. The three-dimensional patch can be a subset of voxels of the complete three-dimensional MRI image. In some cases, the first three-dimensional MRI image can be generated using a first type of MRI sequence that is different from the second type of MRI sequence used to generate the second three-dimensional MRI image. In certain cases, three-dimensional MRI images obtained using different MRI sequences can be stacked and used simultaneously as input to segmentation model 165. For example, multiple (e.g., three) consecutive slices generated using different MRI sequences, such as FLAIR, T1 pre-enhanced, and T1 post-enhanced, can be stacked along the channel dimension and used simultaneously as input to segmentation model 165.

[0064] In some cases, the three-dimensional MRI images are preprocessed before being input to the segmentation model 165. For example, the FLAIR MRI images can be intensity rescaled and z-scored. Furthermore, to capture the increased intensity of lesions in the T1 post-contrast images relative to the T1 pre-contrast images, the volumes can be normalized together (e.g., rescaled while maintaining the intensity relationship between the volumes). Both volumes can be z-scored using the mean and standard deviation of the brain in the pre-contrast scan. Alternatively, if the relative contrast agents within the sequences are sufficient for a given output task, the T1 post-contrast and T1 pre-contrast images can be independently normalized by z-score, similar to the FLAIR MRI images.

[0065] The segmentation controller 170 includes a process for processing each three-dimensional MRI image using a trained segmentation model 165 including multiple connected networks. The trained segmentation model 165 can be deployed on the scanner or as software in a cloud environment. In some cases, the multiple connected networks include a first convolutional neural network connected to a second convolutional neural network using one or more cross-network connections between one or more layers of the first convolutional neural network and one or more layers of the second convolutional neural network. The cross-network connections can promote lesion characteristics of a first type of lesion (e.g., non-enhancing lesions) to influence feature selection for segmenting a second type of lesion (e.g., Gd-enhancing lesions). In an example, because features for segmenting various types of lesions are only loosely coupled, primarily for large or rim-enhancing lesions, cross-network connections may be added only to the contracted path of the multiple connected networks and not to the expanded path of the multiple connected networks. Feature transformation (e.g., 3x3x3) and compression (e.g., 1x1x1) blocks may be added to adapt the lesion features of the first type of lesion so that they are relevant for segmenting the second type of lesion. The compression block may include a set of convolutional layers, batch norm layers, and activation layers. The output of the compression block may be concatenated with the output of an encoding block at a similar level to the first convolutional neural network of the second convolutional neural network. The concatenated features may then be provided as input to the encoding block at the next level of the second convolutional neural network to segment the second type of lesion.

[0066] In some embodiments, multiple connected networks process each image in parallel. When each image is processed in parallel, the multiple connected networks process each image simultaneously or substantially simultaneously, and the extracted feature sets are shared across one or more cross-network connections during generation of the first and second segmentation masks. This design configuration, in which a first convolutional neural network for segmenting a first type of lesion (e.g., T1 non-enhancing lesions) is connected to a second convolutional neural network for segmenting a second type of lesion (e.g., T1 Gd-enhancing lesions) using cross-network connections, is advantageous instead of the reverse or using shared layers. Because the prevalence of the second type of lesion may be lower, little information may be available if the second type of lesion is not present in other possible configurations. The processing includes generating a first segmentation mask of the first type of lesion using a first convolutional neural network that takes a three-dimensional MRI image as input. The first segmentation mask includes a first estimated segmentation boundary around a representation of the first type of lesion. The process further includes generating a second segmentation mask of the second type of lesion using a second convolutional neural network that takes the three-dimensional MRI image as input. The second segmentation mask includes a second estimated segmentation boundary around the representation of the second type of lesion. The segmentation controller 170 can generate a final image mask 175 by combining information from the first segmentation mask and the second segmentation mask. The final image mask 175 includes an estimated segmentation boundary around a representation of a new set of regions that represent a combination of the first type of lesion and the second type of lesion. The segmentation controller 170 further includes processes for outputting the first segmentation mask, the second segmentation mask, and / or the final image mask 175.

[0067] In another embodiment, multiple connected networks process each image in a cascaded arrangement (e.g., serially or substantially serially), with the extracted feature set from each preceding network being shared with subsequent networks. The processing includes: (i) generating a first segmentation mask of a first type of lesion using a first convolutional neural network that takes the three-dimensional MRI image as input; (ii) generating a second segmentation mask of a second type of lesion using a second convolutional neural network that takes the first segmentation mask on the three-dimensional MRI image as input; and (iii) generating a final image mask 175 by combining information from the first and second segmentation masks. The final image mask 175 includes a first estimated segmentation boundary around a representation of the first type of lesion and / or a second estimated segmentation boundary around the second type of lesion. The segmentation controller 170 further includes a process for outputting the final image mask 175.

[0068] The first segmentation mask, the second segmentation mask, and / or the final image mask 175 may be transmitted to an analysis controller 180 within the analysis stage 120. The analysis controller 180 includes a process for acquiring or receiving the first segmentation mask, the second segmentation mask, and / or the final image mask 175 and for determining an analysis result 185 based on the first segmentation mask, the second segmentation mask, and / or the final image mask 175. The analysis controller 180 may further include a process for determining the number of lesions of a first type using the first segmentation mask, the number of lesions of a second type using the second segmentation mask, and / or the number of lesions in the new set of lesions using the final image mask 175. The analysis controller 180 may further include a process for determining one or more lesion sizes or lesion burdens using the first segmentation mask, the second segmentation mask, and / or the final image mask 175. Analysis controller 180 may further include a process for accessing data corresponding to a previous MRI and using the first segmentation mask, the second segmentation mask, and / or the final image mask 175 and the data to determine a change in the amount, size, or cumulative size of one or more lesions and generate an output representing the change. Analysis controller 180 may further include a process for recommending a change in treatment strategy based on the first segmentation mask, the second segmentation mask, and / or the final image mask 175. Analysis controller 180 may further include a process for providing an output corresponding to a possible or confirmed diagnosis of the subject with multiple sclerosis based at least in part on the first segmentation mask, the second segmentation mask, and / or the final image mask 175. Analysis controller 180 may further include a process for diagnosing the subject with multiple sclerosis based at least in part on the first segmentation mask, the second segmentation mask, and / or the final image mask 175.The analysis controller 180 may further include processes for assessing and / or predicting treatment response based at least in part on the first segmentation mask, the second segmentation mask, and / or the final image mask 175.

[0069] It will be appreciated that the machine learning models disclosed herein can be trained and used to detect indications of various types of lesions. For example, non-enhancing T2 lesions can be detected, enhancing T1 lesions can be detected, etc. In some cases, the machine learning models disclosed herein may be trained and used to detect black holes.

[0070] In some embodiments, false positive predictions can be reduced using vessel tracking. Blood vessels can contribute to a significant proportion of false positives for Gd-enhancing lesions. Region growing is performed to identify the top 3 percentile (>=97) of voxels in the brain on T1 post-contrast images. 番目 ) can be used for voxels with an intensity of 0.1. The largest vascular tree can be retained and the brain boundary can be enhanced. Lesions that overlap the vascular tree beyond a certain threshold (e.g., 30%) can be removed to reduce the number of false positives.

[0071] Although not explicitly shown, it will be understood that computing environment 100 may further include a developer device associated with the developer. Communications from the developer device to components of computing environment 100 may indicate the type of input images to be used for the models, the number and type of models to be used, the hyperparameters of each model, e.g., learning rate and number of hidden layers, how to format data requests, the training data to be used (e.g., and how to access the training data) and validation techniques to be used, and / or how to configure the controller process.

[0072] II.B. Exemplary Model Involving Multiple Connection Networks In the exemplary embodiment shown in FIG. 2, multiple networks within a machine learning model are used collectively to detect various types of lesions displayed in three-dimensional MRI images. FIG. 2 illustrates an exemplary model 200 including a first modified U-Net 205 used to segment non-enhancing lesions and a second modified U-Net 210 used to segment Gd-enhancing lesions. The modified U-Nets 205 and 210 are relatively shallow, with three levels. The modified U-Nets 205 and 210 each include a reduction path 215 (encoder) and an expansion path 220 (decoder), which give the network a U-shaped architecture. Each encoding block 225 of the reduction path 215 consists of two sets of convolutional (3×3×3 kernel) layers, batch norm layers, and rectified linear unit (ReLU) activation layers. The modified U-Nets 205 and 210 use dropout layers during training to avoid overfitting and residual connections for easily propagating gradients during backpropagation. Each network extracts features in the reduction path 215, with the number of feature maps at each scale gradually increasing. The feature maps are downsampled using strided convolution with an anisotropic kernel 230 (1 × 3 × 3) to account for the limited coverage along the slice dimension. Correspondingly, in the reduction path 215, the feature maps are upsampled (1 × 2 × 2) using an interpolation layer 235. Skip features from the reduction path 215, which contain fine-grained details, are concatenated with the upsampled semantic features, and the combined features are refined using a residual convolution block 240 similar to that used in the reduction path 215. The top-level decoding block features are concatenated with the upsampled refined features from the intermediate levels and passed through the final convolution and activation layers to generate lesion segmentation probabilities.

[0073] The modified U-Nets 205 and 210 are connected by cross-network connections 245 from the encoding block of the first modified U-Net 205 for segmenting non-enhancing lesions to the encoding block of the second modified U-Net 210 for segmenting Gd-enhancing lesions. The cross-network connections 245 may further connect the decoding block of the first modified U-Net 205 to the decoding block of the second modified U-Net 210. These cross-network connections 245 allow non-enhancing lesion features to influence Gd-enhancing lesion features. Extracted features can be shared across the cross-network connections 245 so that a first segmentation mask for representing non-enhancing lesions can be generated in parallel with a second segmentation mask for representing Gd-enhancing lesions. This design configuration, connecting a first modified U-Net 205 for segmenting non-enhancing lesions to a second modified U-Net 210 for segmenting Gd-enhancing lesions using cross-network connections 245, is advantageous instead of the reverse or using shared layers. This is because the prevalence of Gd-enhancing lesions is low, which means that there is no information available in the absence of Gd-enhancing lesions in other possible configurations. Furthermore, the modified U-Nets 205; 210 can be co-trained by minimizing a loss function. A penalty term can be included in the loss function for any overlap between lesion types and the false positive contribution from the modified U-Nets 205; 210, resulting in a reduction in false positives for smaller, weakly enhancing lesions.

[0074] In the example, because features for segmenting various types of lesions are only loosely coupled, primarily for large or rim-enhancing lesions, cross-network connections 245 are added only on the reduction path 215 and not on the expansion path 220. Feature transformation (3x3x3) and compression (1x1x1) blocks 250 are also added to adapt non-enhancing lesion features to be relevant for segmenting Gd-enhancing lesions. Each of the blocks 250 includes a set of convolutional, batch norm, and activation layers. The output of the compression block 250 is concatenated to the output of the encoding block at a level similar to that of 225 in the first modified U-Net 210 of the second modified U-Net 205. The concatenated features are then provided as input to the next-level encoding block of the modified U-Net 210 to segment Gd-enhancing lesions.

[0075] II.C. Exemplary Process 3 illustrates an exemplary process environment for segmenting representations of various types of lesions in an image using a model having multiple connected networks, according to various embodiments. At block 302, a three-dimensional MRI image representing a subject's brain is accessed. The subject's brain includes at least a first type of lesion (e.g., a T1 non-enhancing lesion) and a second type of lesion (e.g., a T1 Gd-enhancing lesion). The three-dimensional MRI image can be accessed by a segmentation controller (e.g., segmentation controller 170) from an imaging system (e.g., imaging system 130).

[0076] In block 304, the three-dimensional MRI image is input to a machine learning model including a first convolutional neural network (e.g., a first modified U-Net 205) and a second convolutional neural network (e.g., a second modified U-Net 210). The first convolutional neural network is connected to a second convolutional neural network using one or more cross-network connections between one or more layers of the first convolutional neural network and one or more layers of the second convolutional neural network. The cross-network connections may be from encoding blocks of the first convolutional neural network to encoding blocks of the second convolutional neural network. Further, the cross-network connections may be from decoding blocks of the first convolutional neural network to decoding blocks of the second convolutional neural network. Each of the cross-network connections may include a feature transformation block and a compression block.

[0077] A first segmentation mask of the first type of lesion is generated using a first convolutional neural network that takes the three-dimensional MRI image as input, at block 306. The first segmentation mask includes an estimated segmentation boundary around a representation of the first type of lesion.

[0078] At block 308, a second segmentation mask of the second type of lesion is generated using a second convolutional neural network that takes the three-dimensional MRI image as input. The second segmentation mask includes a second estimated segmentation boundary around a representation of the second type of lesion. The first and second segmentation masks are generated in parallel, and the set of extracted features is shared across one or more cross-connect network connections while the first and second segmentation masks are generated.

[0079] At block 310, a first segmentation mask and a second segmentation mask are output. The number of lesions of a first type can be determined using the first segmentation mask, and / or the number of lesions of a second type can be determined using the second segmentation mask. One or more lesion sizes or lesion burdens can be determined using the first segmentation mask and / or the second segmentation mask. Data corresponding to a previous MRI can be accessed, and a change in the volume, size, or cumulative size of one or more lesions can be determined using the first segmentation mask and / or the second segmentation mask and the data, and an output representing the change can be generated. A change in treatment strategy can be recommended based on the first segmentation mask and / or the second segmentation mask. Additionally or alternatively, an output corresponding to a possible or confirmed diagnosis of multiple sclerosis for the subject can be provided based at least in part on the first segmentation mask and / or the second segmentation mask. The subject can be diagnosed with multiple sclerosis based at least in part on the first segmentation mask and / or the second segmentation mask. In some cases, a treatment response can be assessed and / or predicted based at least in part on the first segmentation mask and / or the second segmentation mask. [Example]

[0080] III. Working Examples The systems and methods implemented in various embodiments can be better understood with reference to the following examples.

[0081] III.A. Example 1.—Lesion Segmentation Detecting treatment response to T1 Gd-enhancing lesion burden in multiple sclerosis clinical trials using deep learning.

[0082] III.Ai background Ocrelizumab (OCR) is a humanized anti-CD20+ monoclonal antibody approved for the treatment of relapsing and primary progressive forms of MS. OCR inhibits the development of new areas of inflammation, as indicated by a reduction in the number of T1 Gd-enhancing lesions. Deep learning (DL)-based segmentation of lesions has the potential to automate these manual readings, allowing for more rapid and reproducible quantification.

[0083] III.A.ii. Materials and Methods Lesion segmentation models were developed using MRI datasets from two identical Phase III, multicenter, randomized, double-blind, double-dummy, parallel-group trials in patients with relapsing-remitting multiple sclerosis (RRMS). (Opera 1: NCT01247324, n=898; Opera 2: NCT01412333, n=905) Data were acquired on 1.5T (approximately 0.8%) and 3T (approximately 0.2%) scanners from various manufacturers. Standardized conventional brain MRI, including T1w (pre- and post-contrast), T2w, FLAIR, and PD images with a resolution of approximately 1 × 1 × 3 mm, was performed at baseline, 24, 48, and 96 weeks. Axial 3 mm T1-weighted slices (3D spoiled gradient echo, repetition time = 28-30 ms, echo time = 5-11 ms, flip angle = 27-30) were acquired before and after Gd injection (0.1 mmol / kg, 10 min post-injection delay). Axial 3 mm T2-enhanced slices were acquired by 2D fast spin echo, repetition time = 4000-6190 ms, echo time = 74-91 ms, and echo train length = 7-11. MRI data preprocessing included bias field correction, rigid registration of baseline images to the MNI template, and rigid registration of skull bases at baseline and follow-up visits.

[0084] Manual expert annotations collected as part of the trial were used as a ground truth (GT) mask to train the segmentation model. T2-hyperintense lesions were segmented using a semi-automated approach, and the initial automated segmentation was manually corrected by an expert neuroradiologist to generate the final T2 lesion mask. The initial segmentation of T1-hypointense / non-enhancing lesions included regions within the T2 lesion mask whose T1w intensity was lower than the majority of white matter voxels and did not show Gd enhancement on post-contrast scans. This initial mask was then manually corrected by an expert reader to create the final T1-non-enhancing lesion mask. Gd-enhancing lesions were manually segmented. Contrast enhancement here was defined as an increase in intensity of at least 20% on post-contrast scans, and a minimum size threshold of 3 voxels was used for both non-enhancing and enhancing lesions. New T1-nonenhancing lesions were identified based on differences in lesion segmentation between the reference and follow-up time points and subsequently reviewed by expert readers to correct for spurious differences due to segmentation variability and potential misregistration.

[0085] III.A.iii. Network Architecture The network architecture of the model is shown and described herein with respect to FIG. 2. Generally, the model included a first modified U-Net used to segment non-enhancing lesions and a second modified U-Net used to segment Gd-enhancing lesions. The first modified U-Net was connected to the second modified U-Net using multiple cross-network connections from the encoding block of the first modified U-Net to the encoding block of the second modified U-Net. Each cross-network connection included a feature transformation block and a compression block.

[0086] III.A.iv. Network Training and Subsequent Inference To co-train the network, images from the Opera I trial were used, and a 70%-30% data split was used for training and validation. The network was tested on images from the Opera II trial.

[0087] FLAIR images were intensity-rescaled and z-scored. To capture the increased intensity of lesions in post-contrast T1w images relative to pre-contrast images, these volumes were jointly normalized, i.e., rescaled while maintaining the intensity relationship between both volumes, and z-scored using the mean and standard deviation of the brain in the pre-contrast scan. The model was trained on three consecutive slices from FLAIR, pre- and post-contrast T1w images, stacked along the channel dimension, which served as input for both networks in the individual configurations. For the joint configuration, the second network for segmenting T1 Gd-enhancing lesions received only pre- and post-contrast T1w images, and FLAIR-related information was primarily conveyed along the cross-network connections. For a given input geometry of 256 × 256 × 60, a sliding window approach was used to create 58 stacks of three slices each. Only stacks containing any lesions (non-enhancing or Gd-enhancing) were retained in the training and validation sets. Network parameters were optimized using the Adam optimizer (initial learning rate 1e-4, β1=0.9, β2=0.999) for 50 epochs with a batch size of 6.

[0088] Due to the better performance and potential advantages of asymmetric loss functions achieved from combined loss functions for segmenting lesions in MS, a combination of the Tversky loss (see Equation (1)) and the weighted binary cross entropy (wBCE) loss (see Equation (2)) was created to train the individual networks. The hyperparameters α and β were fine-tuned in the experiments using Equation (3). For joint training, in addition to the losses of the two networks, a penalty term was added for any overlap of the two lesion types and for false positive contributions from both networks using Equation (4).

[0089] For inference, the multimodal input was processed in the same way as during training: the 3D volume was divided into stacks of three slices, and model predictions were obtained for every stack. For every slice, three predictions were obtained by considering the same slice as part of three adjacent stacks as the top, bottom, and middle slices. The prediction probability was thresholded at 0.5, and the three predictions were combined using a majority vote. This helped reduce false positives, and slices toward the edges were treated differently. The model was implemented in Python using Keras in TensorFlow.

[0090] Because a significant proportion of false positives for Gd-enhancing lesions originated from blood vessels, vascular tracking was performed to reduce false positives. Region growing was used for voxels with intensities in the top 3 percentile (>=97th) of voxels within the brain on T1 postcontrast images. The maximal vascular tree and enhancement along the brain border were retained. Lesions overlapping the vascular tree by more than 30% were removed.

[0091] III.Av Evaluation The models were evaluated for segmentation performance at the per-voxel level and detection performance at the lesion level. The mean Dice coefficient (DC), positive predictive value (PPV), sensitivity, absolute volume difference (AVD), and Pearson's correlation coefficient of predicted and GT volumes were used as metrics to evaluate segmentation performance. DC is a metric for measuring the overlap of the predicted and ground truth binary masks and is defined by the following equation (5): JPEG0007766678000013.jpg18170

[0092] TP is a true positive, corresponding to the overlap between the model prediction and the ground truth mask, FP is a false positive, and FN is a false negative. DC values ​​range from 0 to 1, with 0 indicating no overlap and 1 indicating complete overlap. In addition to the overall mean DC, we evaluated the mean DC for volumes with low, medium, and high total lesion burden.

[0093] PPV or precision is the ratio of voxel-wise TP to the sum of TP and FP, and is a measure of the proportion of voxels correctly predicted by the model, as shown in equation (6). JPEG0007766678000014.jpg18170

[0094] Sensitivity or recall is the ratio of TP to the sum of TP and FN, which is the percentage of ground truth voxels correctly segmented by the model, as shown in equation (7). JPEG0007766678000015.jpg22170

[0095] AVD is the ratio of the absolute difference between the predicted ground truth volume and the ground truth volume. It gives a measure of the over- or under-segmentation by the model relative to the ground truth volume, as shown in Equation (8). JPEG0007766678000016.jpg21170

[0096] Individual lesions were identified as connected components (with an 18 connectivity kernel) in the ground truth and predicted masks. For a detected lesion to be considered a true lesion, it had a minimum size of 3 voxels and at least 10% overlap with the ground truth. To evaluate lesion detection performance, we used the lesion PPV (LPPV), lesion true positive rate (LTPR), lesion false positive rate (LFPR), and Pearson's correlation coefficient between the number of lesions predicted by the model and the number of lesions in the ground truth mask. High values ​​for all detection metrics are desirable except for LFPR, which is a measure of the proportion of FP lesions among the lesions predicted by the model.

[0097] III.A.vi. Reproduction of imaging endpoints To perform experiments replicating imaging endpoints, comparison of the model-predicted masks with manual neuroradiologist readings was performed only on images from Opera 2 (because images from Opera 1 were used to train the model). Statistical analyses were performed using R (R version 4.0.1, R Foundation for Statistical Computing).

[0098] To examine whether the trained model could reproduce known T1 Gd-enhancing endpoints, the mean number of Gd-enhancing lesions and the percent reduction in mean lesion counts in the treatment (ocrelizumab) compared with the control (interferon beta) group were estimated at weeks 24, 48, and 96. Similar to known analytical methods for determining T1 Gd-enhancing endpoints, a negative binomial regression of the number of Gd-enhancing lesions was performed using treatment group, baseline T1 Gd-enhancing lesions (present or absent), baseline Expanded Disability Status Scale score (<4.0 vs. ≥4.0), and geographic region (United States vs. rest of the world) as independent variables.

[0099] An automated heuristic approach based on morphological manipulations to estimate the number of new T1-nonenhancing lesions from the GT contiguous lesion mask was developed to closely approximate the values ​​reported in the original analysis by neuroradiologists. This approach was then applied to the model-predicted mask to estimate the mean number of new T1-nonenhancing lesions across all follow-up time points and the percentage reduction in the oclizumab group compared with the control group. As in known analytical methodologies, a negative binomial regression was performed on the number of new T1-nonenhancing lesions, with treatment group, baseline T1-hypointense lesion count, baseline EDSS (<4.0 vs. ≥4.0), and geographic region (United States vs. rest of the world) as independent variables.

[0100] Figure 4 shows the total T1 non-enhancing lesion volume (ml) and total T1 Gd-enhancing lesion volume (ml) for the Opera 2 dataset. Only a subset of subjects had Gd-enhancing lesions, and T1 Gd-enhancing lesions were typically smaller than T1 non-enhancing lesions. According to the neuroradiologist-read GT mask, T1 Gd-enhancing lesions were present in approximately 40% of patients at baseline and approximately 16–18% of the volume across all time points (baseline, 24 weeks, 48 ​​weeks, and 96 weeks). Lesion burdens were significantly skewed toward smaller sizes. T1 non-enhancing lesion burdens ranged from 0.012 ml to 68.86 ml with a median of 1.56 ml, and T1 Gd-enhancing lesion burdens ranged from 0.012 ml to 7.47 ml with a median of 0.213 ml. 71% of the volumes had a maximum Gd-enhancing lesion burden of 0.5 ml, and 72% of the volumes had a maximum T1 non-enhancing lesion burden of 4 ml.

[0101] The segmentation performance of the co-trained models separately for both lesion types on the test set is summarized in Table 1. Both networks had good agreement between the manual annotations and the model-predicted masks in GT, as seen by the mean Dice coefficient (DC) and Pearson's correlation between GT and model-predicted total lesion burden. Overall, the individually trained models had slightly better sensitivity than the co-trained model (0.74 vs. 0.7 for T1-nonenhancing lesions and 0.78 vs. 0.69 for Gd-enhancing lesions); the co-trained model improved DC (0.72 vs. 0.73, 0.72 vs. 0.75), positive predictive value (PPV; 0.74 vs. 0.8, 0.71 vs. 0.87), and absolute volume difference (AVD; 0.29 vs. 0.26, 0.69 vs. 0.31). The final mean DC was 0.78 for Gd-enhancing lesion segmentation, with approximately 23% of lesions being ≤10 voxels in size. (Compared to [1] I. Coronado, REGabr and aPANarayana, "Deep learning segmentation of gadolinium-enhancing lesions in multiple sclerosis," Multiple Sclerosis Journal, no. May, 2020 and [2] G. Brugnara, F. Isensee, U. Neuberger, D. Bonekamp, ​​J. Petersen, R. Diem, B. Wildemann, S. Heiland, W. Wick, M. Bendszus, K. Maier-Hein and P. Kickingereder, "Automated volumetric assessment with artificial neural networks might enable a more accurate assessment of disease burden in patients with multiple sclerosis," European Radiology, vol. 30, no. 4, pp. 2356-64, 2020, they achieved DCs of 0.77 and 0.91 for lesion segmentation, respectively. In [2], 14mm 3We excluded all lesions smaller than / voxel, which may explain the high DC here).

[0102] [Table 1]

[0103] The detection performance of the models on the test set is summarized in Table 2. Both models showed similar trends in detection performance, with the individually trained model having a slightly better lesion true positive rate (LTPR; 0.83 vs. 0.79, 0.96 vs. 0.91), and the jointly trained model having a better LPPV (0.89 vs. 0.92, 0.76 vs. 0.93) and lesion false positive rate (LFPR; 0.17 vs. 0.13, 0.25 vs. 0.06). As can be seen from the high Pearson correlation coefficients, there was good agreement between the lesion counts from the GT and the model-predicted masks of the various models. Joint training significantly reduced oversegmentation and FPs for Gd-enhancing lesion segmentation. This was further improved after FP reduction using vessel tracking, which reduced erroneous predictions of volumes without Gd-enhancing lesions according to manual GT annotations.

[0104] [Table 2]

[0105] Figure 5 shows the predicted lesion segmentation from the model compared with GT for MRI images of a portion of the brain for three subjects. The overlap between GT and the predicted T1-nonenhancing and Gd-enhancing lesions suggests that the model accurately predicts lesion size and location in the MRI images.

[0106] Additionally, we investigated the detection performance of various lesion size buckets for Gd-enhancing lesions (Table 3). Results include calculations of GT lesion count, TP value, sensitivity, FP value (the number in parentheses corresponds to the number of FP lesions in the dataset that did not contain Gd-enhancing lesions in the GT annotation), false positive rate (FDR), and Dice value. The false discovery rate (FDR), also known as FPR, is high for lesions smaller than 0.03 ml (or 10 voxels) and decreases significantly as lesion size increases. DC also showed a similar trend with increasing lesion size. Based on this analysis, a detection threshold of 0.03 ml would be preferable to balance high FDR with low sensitivity. However, this also excludes 23% of lesions. (Performance is comparable to that reported in Table III of [3] Z. Karimaghaloo, H. Rivaz, DL Arnold, DL Collins and T. Arbel, "Temporal Hierarchical Adaptive Texture CRF for Automatic Detection of Gadolinium-Enhancing Multiple Sclerosis Lesions in Brain MRI", IEEE Transactions on Medical Imaging, vol. 34, no. 6, pp. 1227-1241, 2015, except for the minimum lesion size bucket. This example does not use longitudinal information and had a larger training set (8x) compared to [3]. It is also comparable to Table 2 of [1], but with a different lesion size bucket.)

[0107] [Table 3]

[0108] The detection performance of the T1 non-enhanced lesion segmentation models for various lesion sizes is shown in Table 4 and in Table 5 for both the individually and jointly trained models for Gd-enhancing lesion segmentation.

[0109] [Table 4]

[0110] [Table 5]

[0111] III.B. Example 2.—Lesion Changes Between Treatments Detecting treatment response to T1 Gd-enhancing lesion burden in multiple sclerosis clinical trials using deep learning.

[0112] III.Bi Consideration Lesions are frequently assessed during clinical trials to predict whether a given therapeutic agent will be effective in treating multiple sclerosis to slow disease progression. In particular, study endpoints often examine the extent to which lesion count and / or lesion size (e.g., cumulative lesion size) change during the study period. Changes observed when a particular treatment is administered during the study period can be compared to corresponding changes observed when no treatment or baseline treatment is administered during the study period.

[0113] The data in this example compare the change in T1 non-enhancing lesion counts with ocrelizumab compared to interferon-beta. Co-trained models were used to process T1 non-enhanced and T1 Gd-enhanced images collected after contrast administration. Each model was trained using the Opera 1 dataset and tested using the Opera 2 dataset. For each subject and imaging session, the model prediction evaluated was the number of T1 Gd-enhancing lesions. MRIs were performed at 24, 48, and 96 weeks after treatment administration. For each treatment group, the number of lesions at follow-up was calculated by dividing the number of MRI scans at that time point.

[0114] As shown in Figure 6A, the clinical data showed that T1 Gd-enhancing lesion counts were significantly different between the two treatment groups at each of the three time points. For the GT mask, the reductions at follow-up time points were 93%, 96%, and 96% at 24, 48, and 96 weeks, respectively. Using model predictions, at the individual follow-up time points, the reductions were 79%, 84%, and 90% at 24, 48, and 96 weeks, respectively.

[0115] Figure 6B shows the results of T1 non-enhancing lesion counts from the GT mask using the empirical approach and the model-predicted mask using the same empirical approach. Using the empirical approach for the GT mask, there was a 22% reduction, from 4.78 in the control group to 3.73 in the oclizumab group. The model tended to predict fewer lesions than the GT mask using the empirical approach. Based on the model prediction, the number of new T1 non-enhancing lesions was 3.36 in the control group and 2.51 in the treatment group. Furthermore, the number of new T1 non-enhancing lesions was significantly different between groups in negative binomial regression with additional baseline imaging and clinical covariates. There was a roughly 22% reduction in the mean number of new T1 non-enhancing lesions between the treatment and control groups from the GT mask. When using the model prediction, the reduction was approximately 25%, comparable to the reduction from the GT mask. However, the empirical approach used to identify new T1 non-enhancing lesions requires improvement, as the original manual analysis had a percent reduction of approximately 60%.

[0116] IV. Further Considerations Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium containing instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0117] The terms and expressions which have been employed are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0118] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of preferred exemplary embodiments provides those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0119] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

[0120] V. Exemplary Embodiments As used below, any reference to a series of examples should be understood disjunctively as a reference to each of those examples (e.g., "Examples 1-4" should be understood as "Examples 1, 2, 3, or 4").

[0121] Example 1 is a computer-implemented method including: accessing three-dimensional magnetic resonance imaging (MRI) images, the three-dimensional MRI images displaying a region of a brain of a subject, the brain region including at least a first type of lesion and a second type of lesion; inputting the three-dimensional MRI images to a machine learning model including a first convolutional neural network and a second convolutional neural network, the first convolutional neural network being connected to the second convolutional neural network using one or more cross-network connections between one or more layers of the first convolutional neural network and one or more layers of the second convolutional neural network; and detecting a first segment of the first type of lesion using the first convolutional neural network that takes the three-dimensional MRI images as input. generating a first segmentation mask of a first type of lesion, the first segmentation mask including a first estimated segmentation boundary around a representation of the first type of lesion; generating a second segmentation mask of a second type of lesion using a second convolutional neural network that takes as input the three-dimensional MRI image, the second segmentation mask including a second estimated segmentation boundary around a representation of the second type of lesion, the first segmentation mask and the second segmentation mask being generated in parallel, and wherein a set of extracted features is shared across one or more cross-network connections while the first segmentation mask and the second segmentation mask are generated; and outputting the first segmentation mask and the second segmentation mask.

[0122] Example 2 is the computer-implemented method of example 1, wherein the first convolutional neural network is connected to the second convolutional neural network using one or more cross-network connections from an encoding block of the first convolutional neural network to an encoding block of the second convolutional neural network.

[0123] Example 3 is the computer-implemented method of example 1 or example 2, wherein the first convolutional neural network is connected to the second convolutional neural network using one or more cross-network connections from a decoding block of the first convolutional neural network to a decoding block of the second convolutional neural network.

[0124] Example 4 is the computer-implemented method of Examples 1-3, wherein each of the cross-network connections includes a feature transformation block and a compression block.

[0125] Example 5 is the computer-implemented method of any one of Examples 1 to 4, further including: generating a final image mask by combining information from the first segmentation mask and the second segmentation mask, the final image mask including an estimated segmentation boundary around a representation of a new set of lesions representing a combination of the first type of lesion and the second type of lesion; and outputting the final image mask.

[0126] Example 6 is the computer-implemented method of any one of Examples 1-5, wherein the first type of lesion is a T1 non-enhancing lesion and the second type of lesion is a T1 Gd-enhancing lesion.

[0127] Example 7 is the computer-implemented method of any one of Examples 1-6, wherein the first convolutional neural network and the second convolutional neural network are trained using a loss function comprising a combination of Tversky loss and weighted binary cross-entropy loss.

[0128] Example 8 is the computer-implemented method of Examples 1-7, wherein the first convolutional neural network includes a plurality of model parameters identified using a set of training data including a plurality of medical images having annotations associated with segmentation boundaries around various types of lesions, including a first type of lesion and a second type of lesion, and the plurality of model parameters are identified using the set of training data based on minimizing a loss function.

[0129] Example 9 is the computer-implemented method of Examples 1-8, wherein the second convolutional neural network includes a plurality of model parameters identified using the set of training data, and the second convolutional neural network is co-trained with the first convolutional neural network such that the plurality of model parameters of the second convolutional neural network are collaboratively identified using the set of training data based on minimizing a loss function.

[0130] Example 10 is the computer-implemented method of Examples 1-9, wherein the loss function further includes a penalty term for any overlap between the first type of lesion and the second type of lesion, and for false positive contributions from the first convolutional neural network and the second convolutional neural network.

[0131] Example 11 is the computer-implemented method of any one of Examples 1-10, wherein the first convolutional neural network and the second convolutional neural network are U-Nets.

[0132] Example 12 is the computer-implemented method of Examples 1-11, wherein the first convolutional neural network includes three layers of encoding blocks and decoding blocks.

[0133] Example 13 is the computer-implemented method of examples 1-12, wherein the second convolutional neural network includes three layers of encoding blocks and decoding blocks.

[0134] Example 14 is the computer-implemented method of any one of Examples 1 to 13, wherein the three-dimensional MRI image includes multiple adjacent slices stacked along the channel dimension, the adjacent slices including a first three-dimensional MRI slice generated using a first type of MRI sequence, a second three-dimensional MRI slice generated using a second type of MRI sequence, and a third three-dimensional MRI slice generated using a third type of MRI sequence.

[0135] Example 15 is the computer-implemented method of Examples 1-14, wherein the first type of MRI sequence is a post-contrast T1, the second type of MRI sequence is a pre-contrast T1, and the third type of MRI sequence is a fluid-attenuated inversion recovery.

[0136] Example 16 is the computer-implemented method of Examples 1-15, wherein the first and second three-dimensional MRI slices are normalized to capture the contrast agent, and the normalization comprises z-scoring both the first and second three-dimensional MRI slice volumes using the mean and standard deviation of the second three-dimensional MRI slice.

[0137] Example 17 is the computer-implemented method of any one of Examples 1-16, further comprising determining a count of the number of lesions of a first type using the first segmentation mask, and determining a count of the number of lesions of a second type using the second segmentation mask.

[0138] Example 18 is the computer-implemented method of Examples 1-17, further including using the first segmentation mask to determine a count of the number of lesions of a first type, using the second segmentation mask to determine a count of the number of lesions of a second type, and / or using the final image mask to determine a count of the number of lesions in the new set of lesions.

[0139] Example 19 is the computer-implemented method of any one of Examples 1-18, further comprising determining one or more lesion sizes or lesion burdens using the first segmentation mask, the second segmentation mask, and / or the final image mask.

[0140] Example 20 is the computer-implemented method of any one of Examples 1-19, further including accessing data corresponding to a previous MRI, and using the first segmentation mask, the second segmentation mask, and / or the final image mask and the data to determine a change in volume, size, or cumulative size of one or more lesions, and generating an output representative of the change.

[0141] Example 21 is the computer-implemented method of any one of Examples 1 to 20, further comprising recommending a change in treatment strategy based on the first segmentation mask, the second segmentation mask, and / or the final image mask.

[0142] Example 22 is the method of any one of Examples 1-21, further comprising providing an output corresponding to a possible or confirmed diagnosis of the subject with multiple sclerosis based at least in part on the first segmentation mask, the second segmentation mask, and / or the final image mask.

[0143] Example 23 is the method of any one of Examples 1-22, further comprising diagnosing the subject with multiple sclerosis based at least in part on the first segmentation mask, the second segmentation mask, and / or the final image mask.

[0144] Example 24 is the method of any one of Examples 1-23, further comprising assessing and / or predicting treatment response based at least in part on the first segmentation mask, the second segmentation mask, and / or the final image mask.

[0145] Example 25 is a computer-implemented method comprising: accessing three-dimensional magnetic resonance imaging (MRI) images, the three-dimensional MRI images showing a region of a brain of a subject, the brain region including at least a first type of lesion and a second type of lesion; inputting the three-dimensional MRI images to a machine learning model including a first convolutional neural network connected to a second convolutional neural network; generating a first segmentation mask of the first type of lesion using the first convolutional neural network that takes the three-dimensional MRI images as input; 10. A computer-implemented method comprising: generating a second segmentation mask of a second type of lesion using a second convolutional neural network on the MRI image that takes as input the first segmentation mask; generating a final image mask by combining information from the first segmentation mask and the second segmentation mask, wherein the final image mask includes a first estimated segmentation boundary around a representation of the first type of lesion and / or a second estimated segmentation boundary around a representation of the second type of lesion; and outputting the final image mask.

[0146] Example 26 is a system comprising one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein.

[0147] Example 27 is a computer program product tangibly embodied in a non-transitory machine-readable storage medium that includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

Claims

1. 1. A computer-implemented method comprising: accessing a three-dimensional magnetic resonance imaging (MRI) image, the three-dimensional MRI image displaying a region of a brain of a subject, the region of the brain including at least a first type of lesion and a second type of lesion; inputting the three-dimensional MRI image into a machine learning model including a first convolutional neural network and a second convolutional neural network, wherein the first convolutional neural network is connected to the second convolutional neural network using one or more cross-network connections between one or more layers of the first convolutional neural network and one or more layers of the second convolutional neural network; generating a first segmentation mask of the first type of lesion using the first convolutional neural network that takes the three-dimensional MRI image as input, the first segmentation mask including a first estimated segmentation boundary around a representation of the first type of lesion; generating a second segmentation mask of the second type of lesion using the second convolutional neural network that takes the three-dimensional MRI image as input, the second segmentation mask including a second estimated segmentation boundary around a representation of the second type of lesion, the first segmentation mask and the second segmentation mask being generated in parallel, and a set of extracted features being shared across the one or more cross-network connections while the first segmentation mask and the second segmentation mask are generated; outputting the first segmentation mask and the second segmentation mask; 20. A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, wherein the first convolutional neural network is connected to the second convolutional neural network using the one or more cross-network connections from a coding block of the first convolutional neural network to a coding block of the second convolutional neural network.

3. 3. The computer-implemented method of claim 1, wherein the first convolutional neural network is connected to the second convolutional neural network using the one or more cross-network connections from a decoding block of the first convolutional neural network to a decoding block of the second convolutional neural network.

4. The computer-implemented method of any one of claims 1 to 3, wherein each of the cross-network connections includes a feature transformation block and a compression block.

5. The computer-implemented method of any one of claims 1 to 4, wherein the first type of lesion is a T1 non-enhancing lesion and the second type of lesion is a T1 Gd-enhancing lesion.

6. 6. The computer-implemented method of claim 1, wherein the first convolutional neural network and the second convolutional neural network are trained using a loss function comprising a combination of Tversky loss and weighted binary cross-entropy loss.

7. 7. The computer-implemented method of claim 6, wherein the first convolutional neural network includes a plurality of model parameters identified using a set of training data including a plurality of medical images having annotations associated with segmentation boundaries around various types of lesions, including the first type of lesion and the second type of lesion, and the plurality of model parameters are identified using the set of training data based on minimizing the loss function.

8. 8. The computer-implemented method of claim 7, wherein the second convolutional neural network includes a plurality of model parameters identified using the set of training data, and the second convolutional neural network is co-trained with the first convolutional neural network such that the plurality of model parameters of the second convolutional neural network are collaboratively identified using the set of training data based on minimizing the loss function.

9. 9. The computer-implemented method of claim 8, wherein the loss function further comprises penalty terms for any overlap between the first type of lesion and the second type of lesion and for false positive contributions from the first convolutional neural network and the second convolutional neural network.

10. 10. The computer-implemented method of claim 1, wherein the first convolutional neural network and the second convolutional neural network are U-Nets.

11. 11. The computer-implemented method of claim 10, wherein the first convolutional neural network includes three layers of encoding and decoding blocks.

12. 12. The computer-implemented method of claim 10 or 11, wherein the second convolutional neural network includes three layers of encoding and decoding blocks.

13. 13. The computer-implemented method of claim 1, wherein the three-dimensional MRI image comprises a plurality of adjacent slices stacked along a channel dimension, the adjacent slices comprising a first three-dimensional MRI slice generated using a first type of MRI sequence, a second three-dimensional MRI slice generated using a second type of MRI sequence, and a third three-dimensional MRI slice generated using a third type of MRI sequence.

14. 14. The computer-implemented method of claim 13, wherein the first type of MRI sequence is a post-contrast T1, the second type of MRI sequence is a pre-contrast T1, and the third type of MRI sequence is a fluid-attenuated inversion recovery.

15. 15. The computer-implemented method of claim 13 or 14, wherein the first and second three-dimensional MRI slices are normalized to account for contrast agent enhancement, the normalization comprising z-scoring both the first and second three-dimensional MRI slice volumes using the mean and standard deviation of the second three-dimensional MRI slice.

16. 16. The computer-implemented method of claim 1, further comprising: determining a count of the number of lesions of the first type using the first segmentation mask; and determining a count of the number of lesions of the second type using the second segmentation mask.

17. Generating a final image mask by combining information from the first segmentation mask and the second segmentation mask, wherein the final image mask includes an estimated segmentation boundary around a representation of a new set of lesions representing a combination of the first type of lesion and the second type of lesion; outputting the final image mask; The computer-implemented method of any one of claims 1 to 16, further comprising:

18. 18. The computer-implemented method of claim 17, further comprising: determining a count of the number of lesions of the first type using the first segmentation mask; determining a count of the number of lesions of the second type using the second segmentation mask; and / or determining a count of the number of lesions in the new set of lesions using the final image mask.

19. 19. The computer-implemented method of claim 17 or 18, further comprising determining one or more lesion sizes or lesion burdens using the first segmentation mask, the second segmentation mask, and / or the final image mask.

20. accessing data corresponding to a previous MRI; determining a change in volume, size, or cumulative size of one or more lesions using the first segmentation mask, the second segmentation mask, and / or the final image mask and the data; generating an output representative of said change; The computer-implemented method of any one of claims 17 to 19, further comprising:

21. 21. The computer-implemented method of any one of claims 17 to 20, further comprising recommending a change in treatment strategy based on the first segmentation mask, the second segmentation mask, and / or the final image mask.

22. 22. The computer-implemented method of any one of claims 17-21, further comprising providing an output corresponding to a possible or confirmed diagnosis of the subject with multiple sclerosis based at least in part on the first segmentation mask, the second segmentation mask, and / or the final image mask.

23. 23. The computer-implemented method of any one of claims 17 to 22, further comprising diagnosing the subject as having multiple sclerosis based at least in part on the first segmentation mask, the second segmentation mask, and / or the final image mask.

24. 24. The computer-implemented method of any one of claims 17 to 23, further comprising assessing and / or predicting a treatment response based at least in part on the first segmentation mask, the second segmentation mask, and / or the final image mask.

25. 1. A computer-implemented method comprising: accessing a three-dimensional magnetic resonance imaging (MRI) image, the three-dimensional MRI image displaying a region of a brain of a subject, the region of the brain including at least a first type of lesion and a second type of lesion; inputting the three-dimensional MRI images into a machine learning model including a first convolutional neural network connected to a second convolutional neural network; generating a first segmentation mask of the first type of lesion using the first convolutional neural network that takes the three-dimensional MRI image as input; generating a second segmentation mask of the second type of lesion using the second convolutional neural network that takes as input the first segmentation mask on the three-dimensional MRI image; generating a final image mask by combining information from the first segmentation mask and the second segmentation mask, the final image mask including a first estimated segmentation boundary around a representation of the first type of lesion and / or a second estimated segmentation boundary around a representation of the second type of lesion; outputting the final image mask; 20. A computer-implemented method comprising:

26. A computer program comprising instructions configured to cause one or more data processors to carry out a method according to any one of claims 1 to 25.

27. ​​A non-transitory machine-readable storage medium that records the computer program described in claim 26.

28. 1. A system comprising: one or more data processors; a non-transitory computer-readable storage medium having recorded thereon the computer program of claim 26; A system comprising:

Images