Multi-arm machine learning model with attention for lesion segmentation

Abstract

Techniques are provided for segmenting a representation of a lesion in a medical image using a trained multi-arm machine learning network.SOLUTION: Embodiments disclosed herein generally relate to a multi-arm machine learning model for lesion detection. In particular, aspects of the present disclosure relate to accessing three dimensional magnetic resonance imaging (MRI) images. Each of the three dimensional MRI images displays the same volume of the subject's brain. The brain mass includes at least a portion of one or more lesions. Each three dimensional MRI image of the three dimensional MRI images is processed using one or more corresponding encoder arms of the machine learning model to generate an encoding of the three dimensional MRI image. The encodings of the three dimensional MRI images are concatenated to generate a concatenated representation. The concatenated representation is processed using a decoder arm of the machine learning model to generate a prediction identifying one or more portions of the brain mass predicted to display at least a portion of the lesion.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 074,354, filed September 3, 2020, which is incorporated herein by reference in its entirety for all purposes. [Background technology]

[0002] background Multiple sclerosis (MS) is a disease that affects the central nervous system. As a result of the disease, lesions form in the brains of subjects with MS. Most of the time, MS initially manifests as relapsing-remitting MS, which is characterized by spatial and temporal separation. For example, a subject may experience multiple symptomatic episodes affecting different body regions and / or different functional systems. As another example, a subject may experience lesions in different parts of the central nervous system. As yet another example, a subject may experience symptoms and have lesions in brain regions that do not correspond to a system. Over time, the size and number of lesions in MS subjects frequently increase, reflecting disease progression. Relapsing-remitting MS typically eventually progresses to secondary progressive MS, during which gradual deterioration of symptoms and brain health is observed rather than discrete symptomatic relapses and recovery episodes. Primary progressive MS is similarly characterized by gradual symptom worsening and brain degradation, but this diagnosis is limited to subjects who have not previously experienced relapsing-remitting MS.

[0003] Magnetic resonance imaging (MRI) can provide visualization of new (contrast-enhanced) lesions, old lesions, and brain atrophy (via brain size changes). These metrics are the foundation for supporting MS diagnosis, facilitating MS prognosis, selecting MS treatments, and evaluating MS treatments. MRI images are generated using MRI sequences, which are specific settings of radiofrequency pulses and gradients that result in specific image appearances. The three main MRI sequence types include T1 sequences, T2 sequences, and fluid-attenuated inversion recovery (FLAIR) sequences.

[0004] T1 MRI is acquired relatively quickly and can provide structural information. Dark holes, indicating axonal destruction and neuronal death, appear dark on T1 images. T2 MRI is frequently used to show total disease burden. New and old lesions appear as hyperintense areas on these scans. FLAIR MRI is similar to T2 MRI, except that the ventricles appear dark on FLAIR MRI (bright on T2 MRI). Therefore, lesions that appear near the ventricles (e.g., in the corpus callosum) are more visible on FLAIR MRI.

[0005] In some cases, a contrast agent (e.g., gadolinium) is administered to the subject (e.g., intravenously) before acquiring one or more MRI scans to improve the visibility of active lesions. If the blood-brain barrier is intact, the contrast agent does not enter the central nervous system. If the blood-brain barrier is disrupted, the contrast agent can migrate into the central nervous system and localize to areas of inflammation. The contrast agent appears very intense on T1 MRI.

[0006] Traditionally, radiologists manually annotate MRI scans. However, given that scans are collected at different depths and perspectives, as well as different sequence types, there are many scans to annotate. Furthermore, there is significant variability in annotation across radiologists. Therefore, it would be advantageous to identify automated techniques that can process MRI images to improve efficiency and consistency. Summary of the Invention

[0007] overview In some embodiments, a computer-implemented method is provided. A plurality of three-dimensional magnetic resonance imaging (MRI) images are accessed. Each of the plurality of three-dimensional MRI images displays the same mass of a subject's brain. A first three-dimensional MRI image was generated using a first type of MRI sequence (e.g., T1, T2, or fluid-attenuated inversion recovery (FLAIR)) that is different from a second type of MRI sequence used to generate the second three-dimensional MRI image. Each of the plurality of three-dimensional MRI images is processed using one or more corresponding encoder arms of a machine learning model to generate an encoding of the three-dimensional MRI image. The encodings of the plurality of three-dimensional MRI images are concatenated to generate a concatenated representation. The concatenated representation is processed using a decoder arm of the machine learning model to generate a prediction identifying one or more portions of the brain mass predicted to display at least a portion of a lesion.

[0008] In some embodiments, the computer-implemented method further includes generating, for each three-dimensional MRI image of the plurality of three-dimensional MRI images, a downsampled encoding having a lower resolution than a resolution of the encoding of the three-dimensional MRI image. The downsampled encoding of each three-dimensional MRI image of the plurality of three-dimensional MRI images is processed using one or more layers of one or more corresponding encoding arms. The downsampled encodings are concatenated to generate another concatenated representation. The prediction is further based on processing the another concatenated representation using a decoder arm of the machine learning model.

[0009] In some embodiments, the machine learning model includes a U-Net machine learning model.

[0010] In some embodiments, the machine learning model includes one or more skip attention modules, each of which connects an encoding block of an encoding arm of the machine learning model to a decoder block of a decoder arm at the same resolution.

[0011] In some embodiments, each skip attention module of the skip attention modules receives an input of the concatenated representation and an upsampled encoding of another concatenated representation at the resolution of the three-dimensional MRI image, and the prediction is further based on processing the output of the skip feature encoding from the skip attention module using a decoder arm of the machine learning model.

[0012] In some embodiments, one or more skip attention modules include residual connections between the inputs and outputs of the skip attention modules to facilitate skipping of the skip attention modules when relevant high-dimensional features are unavailable.

[0013] In some embodiments, the machine learning model was trained using weighted binary cross-entropy loss and / or Tversky loss.

[0014] In some embodiments, the machine learning model was trained using losses calculated at each of multiple depths of the machine learning model.

[0015] In some embodiments, the first type of MRI sequences includes sequences from the T1, T2 and FLAIR sequence set, and the second type of MRI sequences includes another sequence from the sequence set.

[0016] In some embodiments, the computer-implemented method further includes determining the number of lesions using the prediction.

[0017] In some embodiments, the computer-implemented method further includes using the prediction to determine one or more of lesion size or lesion burden.

[0018] In some embodiments, the computer-implemented method further includes accessing data corresponding to a previous MRI. A change in the volume, size, or cumulative size of one or more lesions can be determined using the predictions and the data. An output representative of the change is generated.

[0019] In some embodiments, the computer-implemented method further comprises recommending a change in treatment strategy based on the prediction.

[0020] In some embodiments, the computer-implemented method further includes providing an output corresponding to a possible or confirmed diagnosis of multiple sclerosis in the subject based at least in part on the prediction.

[0021] In some embodiments, the computer-implemented method further includes diagnosing the subject with multiple sclerosis based at least in part on the prediction.

[0022] Some embodiments of the present disclosure include a system including one or more data processors. The system may further include a non-transitory computer-readable storage medium including instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0023] In some embodiments, a computer program product tangibly embodied in a non-transitory machine-readable storage medium is provided, which may include instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein. [Brief explanation of the drawings]

[0024] [Figure 1] 1 illustrates an exemplary computing environment for segmenting representations of lesions in images using a multi-armed model, according to various embodiments.

[0025] [Figure 2] 1 illustrates an exemplary model including multiple encoder arms used to segment a representation of a lesion in accordance with various embodiments.

[0026] [Figure 3] 1 illustrates another exemplary model including multiple encoder arms used to segment a representation of a lesion in accordance with various embodiments.

[0027] [Figure 4] 1 illustrates an exemplary process for segmenting a representation of a lesion in an image using a model with multiple encoder arms, according to various embodiments.

[0028] [Figure 5A-5B] 1 shows exemplary results of multiple machine learning models for segmenting and detecting lesion indications in magnetic resonance imaging (MRI) images.

[0029] [Figures 6A-6B] 1 shows exemplary results of multiple machine learning models for segmenting and detecting lesion indications in MRI images.

[0030] [Figure 7] 1 shows exemplary results of detecting lesion indications in MRI images using seven different machine learning models.

[0031] [Figure 8] 1 shows exemplary results of lesion count prediction during clinical treatment using different machine learning models. DETAILED DESCRIPTION OF THE INVENTION

[0032] I. Overview This disclosure describes techniques for automated lesion segmentation in medical images. More specifically, embodiments of the present disclosure provide techniques for segmenting representations of lesions in medical images using a trained multi-arm machine learning network. In some embodiments, a machine learning model with attention can be used to segment lesions in three-dimensional magnetic resonance imaging (MRI) images.

[0033] MRI images are typically annotated manually or semi-manually to identify lesions visible in the MRI images. However, given the different sequence types (T1, T2, and fluid-attenuated inversion recovery (FLAIR)) as well as the different depths and perspectives that scans are collected at, there are often many scans to annotate. Furthermore, there is often a high degree of variability in annotations across annotators. Therefore, manual or semi-manual approaches can result in suboptimal efficiency and consistency for lesion detection.

[0034] To address these limitations and problems, some embodiments of the present disclosure provide a technique for automated object segmentation that utilizes a multi-arm deep learning network trained to segment lesion representations. For example, a data processing system accesses multiple three-dimensional MRI images, each of which depicts the same brain mass of a subject. The brain mass includes at least a portion of one or more lesions. Each of the three-dimensional MRI images is generated using a different type of MRI sequence (e.g., T1, T2, or FLAIR). The data processing system processes each three-dimensional MRI image using one or more corresponding encoder arms of a machine learning model to generate an encoding of the three-dimensional MRI image. Each encoder arm can be trained to process a different type of MRI sequence. The data processing system concatenates the encodings of the three-dimensional MRI images to generate a concatenated representation and processes the concatenated representation using a decoder arm of the machine learning model to identify one or more portions of the brain mass that are predicted to display at least a portion of the lesion. The lesion predictions can be output for further downstream analysis. For example, the predicted lesion indication can be used to predict whether a subject has any lesions, whether a subject has any enhancing lesions, the number of lesions a subject has, the number of enhancing lesions a subject has, the location (e.g., brain location) of each of one or more lesions, the location of each of one or more enhancing lesions, the size of each of one or more lesions a subject has, the size of each of one or more enhancing lesions a subject has, the cumulative lesion size of the subject, the cumulative enhancing lesion size of the subject, and / or the brain mass of the subject.

[0035] II. Techniques for Segmenting Medical Images Image segmentation is the procedure of separating an image into parts that exhibit similarities in distinct features, such as shape, size, and color. Segmentation of lesion appearances allows for visualization of the size and location of lesions within a body region (e.g., the brain) and can also provide a basis for analyzing treatments. The gold standard for lesion segmentation has long been manual segmentation, which is time-consuming, labor-intensive, and therefore unsuitable for large-scale studies. Considerable research has attempted to fully or partially automate the process of lesion segmentation. For example, image segmentation techniques such as thresholding, region growing, fuzzy clustering, and the use of watershed algorithms have been used to separate appearances of abnormal tissue (e.g., lesions) from appearances of normal tissue, such as brain white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF). Nevertheless, the segmentation process remains challenging due to the diversity in the shape, location, and size of lesion appearances.

[0036] This paper describes an end-to-end method that incorporates a model that uses multiple encoding arms of a machine learning model (e.g., U-Net) to segment lesion representations. Treatment response, along with other factors (e.g., recurrence, progression, etc.), can be characterized based on the segmented lesion representations. The developed model is devised to accommodate the complexity of three-dimensional scans, the extreme disparity between various lesion types, and the heterogeneity of input images (e.g., variable density and object size). As used herein, a "scan" is a graphical representation of signals on a single plane through a subject's body. This model has lesion segmentation performance comparable to conventional algorithms that rely on manual intervention (e.g., manual selection of seeds or manual identification of bounding boxes), such as thresholding, edge-based segmentation, or region-based segmentation methods.

[0037] II.A. Exemplary Computing Environment 1 illustrates an exemplary computing environment 100 (i.e., a data processing system) for segmenting representations of lesions in images using a multi-armed model, according to various embodiments. As shown in FIG. 1, the segmentation performed by the computing environment 100 in this example includes several stages: an image acquisition stage 105, a model training stage 110, a segmentation stage 115, and an analysis stage 120.

[0038] The image acquisition stage 105 includes one or more imaging systems 130 (e.g., MRI imaging systems) for acquiring input images 135 (e.g., MRI images) of various portions of the subject. The imaging systems 130 are configured to acquire the input images 135 using radiological imaging techniques such as MRI. The imaging systems 130 can determine differences between various structures and functions within the subject based on characteristics associated with each of the imaging systems 130 (e.g., brightness, contrast, and spatial resolution) and generate a series of two-dimensional or three-dimensional images. Once the series of two-dimensional images are collected by the scanner's computer, the two-dimensional images can be digitally "stacked" together through computer analysis to reconstruct a three-dimensional image of the subject or portion of the subject. The two-dimensional images and / or reconstructed three-dimensional input images 135 enable easier identification and location of underlying structures (e.g., organs) and possible lesions or abnormalities. Each two-dimensional image and / or reconstructed three-dimensional input image 135 corresponds to a session time and subject and can display an internal region of the subject. Each two-dimensional image and / or reconstructed three-dimensional input image 135 may further be of a standardized size, resolution, and / or magnification.

[0039] The model training stage 110 builds and trains one or more models 140a-140n (which may be referred to herein individually as models 140 or collectively as models 140) to be used by other stages. The models 140 may be machine learning ("ML") models including multiple networks, such as, for example, a convolutional neural network ("CNN"), such as an initial neural network, a residual neural network ("Resnet"), a U-Net, a V-Net, a single-shot multi-box detector ("SSD") network, or a recurrent neural network ("RNN"), such as, for example, a long short-term memory ("LSTM") model or a gated recurrent unit ("GRU") model, or any combination thereof. Model 140 may also be any other suitable ML model trained in object detection and / or segmentation from images, such as a three-dimensional CNN ("3DCNN"), a dynamic time warping and warping ("DTW") technique, a hidden Markov model ("HMM"), or the like, or a combination of one or more of such techniques, e.g., a CNN-HMM or an MCNN (multiscale convolutional neural network). The computing environment 100 may use the same or different types of models to segment different types of lesion indications. Model 140 may include an encoder arm having a set of encoder models for generating features related to the segmentation and detection of lesion indications, and a decoder arm 185 for generating lesion predictions. In a particular example, model 140 is constructed by a combined asymmetric loss function, e.g., a combination of Tversky loss and weighted binary cross entropy (wBCE) loss, to train the network.

[0040] To train model 140 in this example, training images 145 are generated by acquiring digital images, dividing the images into a subset of training images 145a for training (e.g., 90%) and a subset of training images 145b for validation (e.g., 10%), preprocessing the subset of training images 145a and the subset of training images 145b, expanding the subset of training images 145a, and optionally annotating the subset of training images 145a with labels 150. The subset of training images 145a are acquired from one or more imaging modalities (e.g., MRI T1, T2, proton density (PD), or FLAIR). Optionally, the subset of training images 145a are acquired from a data storage structure, such as a database, imaging system (e.g., one or more imaging systems 130), associated with one or more imaging modalities. Each image displays one or more lesions.

[0041] The partitioning may be performed randomly (e.g., 90 / 10% or 70 / 30%), or the partitioning may be performed according to more complex validation techniques such as K-fold cross-validation, leave-one-out cross-validation, leave-one-out cross-validation, nested cross-validation, etc. to minimize sampling bias and overfitting. Preprocessing may include standardization or normalization to place all features on the same scale (e.g., the same size scale or the same color or saturation scale). In a particular example, the image is resized with a minimum size (width or height) of a predetermined number of pixels (e.g., 2500 pixels) or a maximum size (width or height) of a predetermined number of pixels (e.g., 3000 pixels) to maintain the original aspect ratio.

[0042] Augmentation can be used to artificially increase the size of a subset of training images 145a by creating modified versions of images in the dataset. Image data augmentation can be performed by creating transformed versions of images in the dataset that belong to the same class as the original images. Transformations include a range of operations from the field of image manipulation, such as shift, flip, and zoom. In some cases, operations include random erasure, shift, brightness, rotation, Gaussian blur, and / or elastic transformation to ensure that the model 140 can perform under conditions outside those available from the subset of training images 145a.

[0043] Annotation can be performed manually by one or more humans (annotators, such as radiologists or pathologists) who identify the presence of one or more lesion indications in each image of the subset of training images 145a and provide labels 150 for the one or more lesions, e.g., using annotation software to draw a bounding box (ground truth) or segmentation boundary around an area identified by the human as containing one or more lesion indications. In certain examples, bounding boxes or segmentation boundaries can be drawn only for cases where the probability of a lesion being present exceeds 50%. For images annotated by multiple annotators, the bounding boxes or segmentation boundaries from all annotators may be used. In some cases, the annotation data can further indicate the type of lesion. For example, for various types of lesions, the annotation data can indicate the type, such as enhancing, non-enhancing, or T2 hyperintense.

[0044] In some cases, a subset of training images 145 may be sent to the annotator device for inclusion in the training dataset (i.e., a subset of training images 145a). Input can be provided to the annotator device (e.g., by a radiologist) using a mouse, trackpad, stylus, and / or keyboard indicating (for example) whether the image displays an object of interest (e.g., a lesion, organ, etc.), the number and type of displayed lesions in the image; and the perimeter (bounding box or segmentation boundary) of each displayed lesion in the image. The annotator device can be configured to generate a label 150 for each image using the provided input. For example, label 150 can include the number and / or type of displayed lesions in the image; a type classification of each displayed lesion; the number of each displayed lesion of a particular type; and the perimeter and / or mask of one or more identified lesions in the image. In some cases, label 150 can further include the perimeter and / or mask of one or more identified lesions overlaid on the medical image.

[0045] In some cases, the model 140 is trained and used to process digital images of a three-dimensional (3D) patch of the brain. The 3D patch may be 96x96x32 and may include a first portion displaying at least a portion of one or more lesions and a second portion displaying no lesions. The training data may include images generated using different MRI sequence types. For example, a training dataset may include three-dimensional MRI images generated using a T1 sequence, a T2 sequence, and a FLAIR sequence. The training data may include labels for portions displaying lesions and portions not displaying lesions. The labels may be received, identified, or derived from data received from a database or provider system. The label data may include (for example) for a single image, an indication of the portion of the image displaying at least a portion of a lesion, the amount of lesion at least partially displayed in the image, the location within the image (e.g., a particular voxel) displaying at least a portion of the lesion, the type of lesion (e.g., enhancing or non-enhancing lesion) at least partially displayed in the image or at a particular location within the image, etc.

[0046] Each of the set of encoder models of model 140 can be trained to process a particular type of MRI sequence. For example, a first machine learning model can be trained on MRI images of a T1 sequence. A second machine learning model can be trained on MRI images of a T2 sequence. A third machine learning model can be trained on MRI images of a FLAIR sequence. The training data used to train model 140 can include MRI images acquired after a contrast agent has been administered to the subject, MRI images showing at least a portion of one or more contrast-enhancing lesions, MRI images acquired without a contrast agent being administered to the subject, MRI images acquired before a contrast agent has been administered to the subject, and / or MRI images not displaying contrast-enhancing lesions. For example, the encoder models of model 140 can be trained only on images acquired without a contrast agent having been recently administered to the subject, only on images acquired after a contrast agent has been recently administered to the subject, or with some of both types of images. It will be appreciated that images collected after the contrast agent has been administered may, in some cases, not display any enhancing lesions in situations where the blood-brain barrier is intact and / or if a given scan does not display the brain region through which the contrast agent has traveled.

[0047] It will be appreciated that the machine learning model techniques disclosed herein can be trained and used to detect various types of lesions. For example, T2 lesions may be detected, enhancing T1 lesions may be detected, etc. In some cases, the machine learning model techniques disclosed herein can be trained and used to detect black holes.

[0048] In a particular example, the model 140 is trained using a weighted binary cross entropy (wBCE) loss or a Tversky loss. The loss function can capture multiple aspects, such as region overlap, voxel-wise accuracy, surface inconsistency, false positives (FP), false negatives (FN), and possible penalties for AUC. The loss function can be a weighted binary cross entropy (wBCE, to quantify voxel-wise accuracy) loss or a Tversky loss (e.g., for region overlap). The Tversky loss can reduce false positives in the prediction 185, and the wBCE can help identify smaller lesions. For example, the loss can be calculated using a Tversky loss with a beta of 0.7, weighting the contribution of false positives by 0.7 and the contribution of false negatives by 0.3. Additionally, a combination of the Tversky loss and the wBCE loss can be used to calculate the loss.

[0049] The training process for model 140 involves selecting hyperparameters for model 140 and performing iterations of inputting images from a subset of training images 145a into model 140 to find a set of model parameters (e.g., weights and / or biases) that minimizes a loss or error function for model 140. Hyperparameters are settings that can be tuned or optimized to control the behavior of model 140. Most models explicitly define hyperparameters that control different aspects of the model, such as memory or execution cost. However, additional hyperparameters can be defined to adapt the model to specific scenarios. For example, hyperparameters can include the number of hidden units in the model, the learning rate of the model, the convolution kernel width, or the number of kernels in the model. Each iteration of training involves finding a set of model parameters for model 140 (composed by the defined set of hyperparameters) such that the value of the loss or error function using that set of model parameters is smaller than the value of the loss or error function using a different set of model parameters in the previous iteration. A loss or error function can be constructed to measure the difference between the output inferred using the model 140 and the ground truth segmentation boundaries annotated to the image using the labels 150.

[0050] Once a set of model parameters has been identified, the model 140 can be trained and validated using a subset (test or validation data set) of the training images 145b. The validation process involves iteratively inputting images from the subset of training images 145b into the model 140 using a validation technique such as K-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, or nested cross-validation, and selecting the model with the minimum loss on the validation set over epochs. The validation set can also be used to tune hyperparameters and ultimately find an optimal set of hyperparameters. Once an optimal set of hyperparameters is obtained, a reserved test set of images from the subset of training images 145b is input into the model 140 to obtain an output (in this example, a segmentation boundary around one or more lesion representations), which is then evaluated against the ground truth segmentation boundary using correlation techniques such as the Bland-Altman method and Spearman's rank correlation coefficient, and calculating performance metrics such as error, precision, precision, recall, and receiver operating characteristic curves (ROC).

[0051] As should be understood, other training / validation mechanisms are envisioned and may be implemented within computing environment 100. For example, a model may be trained and hyperparameters may be tuned on images from a subset of training images 145a, and images from a subset of training images 145b may be used only to test and evaluate the model's performance. Furthermore, while the training mechanisms described herein focus on training new models 140, these training mechanisms may also be utilized to fine-tune existing models 140 trained from other datasets. For example, in some cases, model 140 may have been pre-trained using images of other objects or biological structures, or from sections from other subjects or studies (e.g., human or mouse studies). In those cases, model 140 may be used for transfer learning and retrained / validated using input images 135.

[0052] The model training stage 110 outputs a trained model including one or more trained segmentation models 165, each including a set of encoder and decoder arms. One or more input images 135 are acquired by a lesion segmentation controller 170 within the segmentation stage 115. In various examples, the input images 135 are three-dimensional MRI images. Each of the three-dimensional MRI images displays the same block of the subject's brain. The block of brain includes at least a portion of one or more lesions. Each three-dimensional MRI image includes a three-dimensional patch of the complete MRI image. The three-dimensional patch can be a subset of voxels of the complete three-dimensional MRI image. In some cases, a first three-dimensional MRI image can be generated using a first type of MRI sequence that is different from a second type of MRI sequence used to generate the second three-dimensional MRI image. In certain examples, the three-dimensional MRI images acquired using the different MRI sequences can each be input to a separate encoder arm of the segmentation model 165.

[0053] In some cases, the three-dimensional MRI images are preprocessed before being input to the segmentation model 165. For example, the FLAIR MRI images may be intensity rescaled and z-scored. Furthermore, to capture the increased intensity of lesions in the T1 post-contrast images relative to the pre-contrast images, the volumes may be normalized together (e.g., rescaled while maintaining the intensity relationship between the volumes). Both volumes may be z-scored using the mean and standard deviation of the brain in the pre-contrast scan.

[0054] In some cases, the lesion segmentation controller 170 includes a process for processing three-dimensional MRI images using a trained segmentation model 165 that includes a set of encoder and decoder arms. In some cases, each three-dimensional MRI image is input to a corresponding encoder arm of the trained segmentation model 165. Each encoder arm can generate an encoding of the three-dimensional MRI image. A downsampled encoding having a lower resolution than the resolution of the encoding of the three-dimensional MRI image can also be generated. The downsampled encoding can be processed using one or more layers of one or more corresponding encoding arms. The encodings (or downsampled encodings) at a given resolution are aggregated (e.g., concatenated) to generate a concatenated representation for each resolution. Each concatenated representation can be processed using a set of layers in a decoder arm of the segmentation model 165 to generate a prediction 185 that identifies one or more portions of the brain mass predicted to represent at least a portion of a lesion. The prediction 185 can identify the likelihood of each voxel containing a representation of a lesion. In some cases, the lesion segmentation controller 170 can output a prediction 185 of an image corresponding to a three-dimensional MRI image with an estimated segmentation boundary around a representation of the lesion.

[0055] In some cases, the segmentation model 165 may include skip features. Skip features are encoded features whose details are transferred to the decoder arm. The decoder arm of the segmentation model 165 may receive an aggregate of the skip feature encodings as the output of the skip attention module and use the skip feature encodings to generate a prediction 185 of the displayed lesion. In some cases, the skip attention module may receive a concatenated representation at a given resolution and an upsampled encoding of another concatenated representation to the given resolution. The skip attention module may output skip feature encodings. The skip feature encodings may be aggregated, which may include the concatenation or sum of the skip feature encodings generated by the skip attention module of the segmentation model 165 at the given resolution, upsampled skip feature encodings generated by upsampling the results from the skip attention module at a lower resolution, and / or downsampled skip feature encodings generated by downsampling the results from the skip attention module at a higher resolution. Alternatively, the decoder arm may receive skip feature encodings from the skip attention module at each resolution without aggregating the upsampled or downsampled skip feature encodings at other resolutions.

[0056] Although skip features contain fine details, they can be noisy because they are generated by encoding layers with reduced receptive fields (local regions of the input image seen by the current convolution operation). Skip features lack context for correct classification of voxels, resulting in false positives (FPs). For example, blood vessels may appear similar to small lesions when viewing a smaller portion of the scan, but using high-dimensional features learned from a larger portion of the scan can make it easier to learn the elongated structure of blood vessels. Furthermore, discrepancies in the edges or boundaries of structures in the low-dimensional representation can be corrected using semantic information learned from the high-dimensional representation. Therefore, the skip attention module attempts to reduce FPs by providing attention to regions within the skip layer based on higher-dimensional representations from the next-level features, which have lower resolution.

[0057] The predictions 185 can be transmitted to an analysis controller 175 within the analysis stage 120. The analysis controller 175 includes a process for obtaining or receiving one or more predictions 185 at the resolution and determining an analysis result 180 based on the one or more predictions 185. The analysis controller 175 can further include a process for determining the number of lesions using the one or more predictions 185. The analysis controller 175 can further include a process for determining one or more lesion sizes or lesion burdens using the one or more predictions 185. The number of lesions, the one or more lesion sizes, and / or the lesion burden can be output as the analysis result 180. The analysis controller 175 can further include a process for accessing data corresponding to a previous MRI and using the predictions and the data to determine a change in the amount, size, or cumulative size of one or more lesions and generate the analysis result 180 as an output representing the change. As an example, a care provider can recommend that a subject switch from a current treatment to another treatment based on changes in several lesion indications (identified in the output) detected in recent MRI data compared to older MRI data associated with the subject. Analysis controller 175 may further include a process for recommending changes in treatment strategy based on prediction 185. Analysis controller 175 may further include a process for providing analysis result 180 as an output corresponding to a possible or confirmed diagnosis of multiple sclerosis for the subject based at least in part on prediction 185. For example, a care provider may identify a diagnosis (e.g., of a type of multiple sclerosis or multiple sclerosis itself) based on the amount of lesions detected and identified in the output. Analysis controller 175 may further include a process for diagnosing the subject with multiple sclerosis based at least in part on prediction 185. Analysis controller 175 may further include a process for assessing and / or predicting treatment response based at least in part on prediction 185. For example, analysis controller 175 may compare prediction 185 for the subject with older MRI data to assess and / or predict treatment response.A diagnosis of multiple sclerosis, an assessment of treatment response and / or a prediction can be output as analysis result 180.

[0058] In some embodiments, multiple techniques can be used to reduce false positive predictions. Predictions can be masked by a white matter mask to reduce false positives by hypothesizing that there is a differential distribution of true positives (TPs) and FPs within and outside the white matter (e.g., true T2 lesions are in the white matter, and false positives are outside the white matter). Because MS lesions are caused by the immune system attacking the myelin sheath around neuronal axons, TPs can be assumed to occur in the white matter. However, it can be difficult to observe the differential distribution of FPs and TPs. In such cases, a false positive term can also be added to the loss function to reduce false positive predictions. Furthermore, a smooth truncated loss can be implemented instead of wBCE to reduce learning from outliers. Additional machine learning models (e.g., radiology-based models) can also be implemented to classify predicted lesions into true positives and false positives.

[0059] Although not explicitly shown, it will be understood that computing environment 100 can further include a developer device associated with the developer. Communications from the developer device to components of computing environment 100 can indicate the types of input images to be used for the models, the number and types of models to be used, the hyperparameters of each model, e.g., learning rate and number of hidden layers, how to format data requests, the training data to be used (e.g., and how to access the training data) and validation techniques to be used, and / or how to configure the controller process.

[0060] II.B. Exemplary Model with Multiple Encoder Arms 2 shows an exemplary architecture including an encoder arm 210 used to generate encodings of MRI images and a decoder arm 220 used to convert the encodings into lesion predictions. The encoder arm 210 can include a set of models with the same architecture or different architectures. This architecture corresponds to the segmentation model 165 of FIG. 1 used to determine lesion predictions by the lesion segmentation controller 170.

[0061] In some cases, one, more, or all of the encoder models of the encoder arm 210 can each be configured to receive as input a three-dimensional MRI image, such as the input image 135 of FIG. 1. The three-dimensional MRI image can include a three-dimensional patch of the complete MRI image. The three-dimensional patch can be a subset of voxels of the complete three-dimensional MRI image. For example, a 1×1×3 mm 3 An anisotropic voxel size of 1000 x 1000 x 32 can be used to create a patch size of 96 x 96 x 32. Alternatively, an isotropic voxel size (e.g., 1 x 1 x 1 mm) can be used to create a patch size of 96 x 96 x 32. 3 ) and the resulting patch size (96x96x96) can be used. Other patch sizes corresponding to anisotropic or isotropic voxel sizes can also be used. In some cases, the patch is a resampled (e.g., upsampled) patch.

[0062] In some cases, the set of encoder models of the encoder arm 210 includes multiple models configured to process an input image and extract features at different scales. For example, the encoder arm 210 can include one or more models configured to process an input image having dimensions D (depth) × H (height) × W (width). Each arm can include multiple depth levels (e.g., 4), with features extracted at each level (e.g., D × W × H, D / 2 × W / 2 × H / 2, D / 4 × W / 4 × H / 4, and D / 8 × W / 8 × H / 8). Features extracted at lower depth levels can be downsampled encodings generated by downsampling the encoding at full resolution. The number of downsampled encodings corresponds to the number of downsampling operations. The set of models of the encoder arm 210 can be used for encodings trained using different MRI sequence data. For example, the one or more models of the encoder arm 210 may include (for example) a first model trained using T1 MRI images, a second model trained using T2 MRI images, and a third model trained using FLAIR MRI images.

[0063] The encoding blocks 230 of the encoder arm 210 may include one or more sets of convolutional (e.g., 1x3x3 or 3x3x3) layers, batch norm layers, and rectified linear unit (ReLU) activation layers to generate an encoding of each received input image (e.g., a three-dimensional patch of an MRI image acquired using a particular MRI sequence). Each of the encoding blocks 230 may be configured to extract features from the input image. For example, for each level of the network, a set of feature arrays may be generated. The feature arrays may be downsampled using strided convolution (e.g., 2x2x2).

[0064] For a given resolution, encodings can be generated using an aggregation of downsampled encodings from higher depth levels. For example, at a depth level corresponding to ¼ resolution, the encoding block 230 can receive an aggregation of feature representations from higher depth levels (e.g., ½ resolution and full resolution) downsampled to ¼ resolution. The aggregation can include summing or concatenating feature arrays. For example, feature representations can be summed to minimize the computational requirements of the machine learning model. With increasing computational power, feature representations from each depth level can be concatenated.

[0065] The feature representations from each encoder model for a given resolution can then be aggregated. For example, all feature arrays corresponding to all encoding arms across all input images having a given resolution (e.g., depth level) can be concatenated along the channel dimension while maintaining feature shapes at the given resolution. For example, for a given resolution, the feature arrays can include elements from multiple types of MRI sequences (e.g., T1, T2, FLAIR). Furthermore, for each input data set (e.g., corresponding to a particular imaging session and a particular subject), the encoder arm 210 generates multiple concatenated feature representations 235, each associated with a particular resolution. Each of these concatenated feature representations 235 can itself include elements representing multiple types of MRI sequences represented in the input data.

[0066] The concatenated representation 235 at each resolution can be processed using a decoder arm 220 (e.g., a U-Net model). The decoder arm 220 can be trained using a loss function that quantifies the mismatch and / or discrepancy between the model predictions and the ground truth mask. A loss can be output at each level (e.g., loss per depth) so that the machine learning model learns a representation at each level. The decoder arm 220 can process the concatenated representation 235 to generate predictions 260 that identify one or more portions of the brain mass that display a portion of a lesion (e.g., a portion of a non-enhancing lesion, a portion of an enhancing lesion, or a portion of any type of lesion).

[0067] In some cases, the decoder arm 220 may include one or more skip attention modules 245. The skip attention module 245 connects two non-adjacent layers of the machine learning model to transfer the fine-grained details extracted by the encoding block 230 of the encoder arm 210 to be combined with semantic features from the upsampling block of the decoder arm 220. The decoder block 250 of the decoder arm 220 may receive the skip feature encodings generated by the skip attention module 245 to generate predictions 260 of lesions displayed in the three-dimensional MRI images. At each resolution, the decoder block 250 may receive an aggregate of the skip feature encodings generated at multiple resolutions. For example, at the full resolution level, a skip attention module 245 including one or more sets of convolutional (e.g., 1 × 1 × 1) layers, ReLU activation layers, and batch norm layers may process the concatenated representation 235. The skip feature encodings generated by the skip attention module 245 are aggregated with other skip feature encodings generated by upsampling the outputs of other skip attention modules 245 to full resolution. For lower resolutions, the skip feature encodings of the skip attention module at a given resolution can be aggregated with upsampled skip feature encodings generated by upsampling the results of the skip attention module 245 at a resolution lower than the given resolution to the given resolution. Additionally, aggregation can include downsampled skip feature encodings generated by downsampling the results of the skip attention module 245 at a resolution higher than the given resolution. For aggregation, feature representations from the same encoder model can be summed, and feature representations from different encoder models can be concatenated.

[0068] The decoder block 250 of the machine learning model can output a prediction 260, which is an example of the prediction 185 in FIG. 1 . For example, the prediction 260 can identify one or more portions of a brain mass predicted to display at least a portion of a lesion. The prediction 260 can be used in determining the number of lesions, lesion size, or lesion burden within the brain mass. The prediction 260 can further be used in combination with data corresponding to a previous MRI to determine changes in lesion volume, size, or cumulative size. Furthermore, recommendations for modifying treatment strategies can be determined based on the prediction 260. In some cases, the prediction 260 can also be used to diagnose a subject with multiple sclerosis.

[0069] 3 shows another exemplary architecture that includes an encoder arm 310 with a set of encoder models used to generate encodings of MRI images and a decoder arm 320 used to convert the encodings into lesion predictions. The set of encoder models can include sets of models with the same architecture or different architectures. This architecture corresponds to the segmentation model 165 of FIG. 1 that is used to determine lesion predictions by the lesion segmentation controller 170.

[0070] In some cases, each of one, more, or all of the encoder models of the encoder arm 310 can be configured to receive a three-dimensional MRI image (e.g., input image 135 of FIG. 1) as input. The three-dimensional MRI image can include a three-dimensional patch of the complete MRI image. The three-dimensional patch can be a subset of voxels of the complete three-dimensional MRI image. For example, a 1×1×3 mm 3 An anisotropic voxel size of 1000 x 1000 x 32 can be used to create a patch size of 96 x 96 x 32. Alternatively, an isotropic voxel size (e.g., 1 x 1 x 1 mm) can be used to create a patch size of 96 x 96 x 32. 3) and the resulting patch size (96x96x96) can be used. Other patch sizes corresponding to anisotropic or isotropic voxel sizes can also be used. In some cases, the patch is a resampled (e.g., upsampled) patch.

[0071] In some cases, the set of encoder models of the encoder arm 310 includes multiple models configured to process an input image 325 (e.g., the input image 135 of FIG. 1 ) and extract features at different scales. For example, the encoder arm 310 can include one or more models configured to process an input image 325 having dimensions D (depth) × H (height) × W (width). Each arm can include multiple depth levels (e.g., 4), with features extracted at each level (e.g., D × W × H, D / 2 × W / 2 × H / 2, D / 4 × W / 4 × H / 4, and D / 8 × W / 8 × H / 8). Features extracted at lower depth levels can be downsampled encodings generated by downsampling the encoding at full resolution. The number of downsampled encodings corresponds to the number of downsampling operations. The set of models of the encoder arm 310 can be used for encodings trained using different MRI sequence data. For example, the one or more models of the encoder arm 310 may include (for example) a first model trained using T1 MRI images, a second model trained using T2 MRI images, and a third model trained using FLAIR MRI images.

[0072] The encoding blocks 330 of the encoder arm 310 may each include one or more sets of convolutional (e.g., 3x3x3) layers, batch norm layers, and ReLU activation layers to generate an encoding of a received input image 325 (e.g., a three-dimensional patch of an MRI image acquired using a particular MRI sequence). Each of the encoding blocks 330 may be configured to extract features from the input image 325. For example, for each level of the network, a set of feature arrays may be generated. The feature arrays may be downsampled using strided convolution (e.g., 2x2x2).

[0073] For a given resolution, the encoding can be aggregated. For example, all feature arrays corresponding to all encoding arms across all input images having a given resolution (e.g., depth level) can be concatenated along the channel dimension 335 while maintaining feature shapes at the given resolution. For example, for a given resolution, the feature arrays can include elements from multiple types of MRI sequences (e.g., T1, T2, FLAIR). Furthermore, for each input data set (e.g., corresponding to a particular imaging session and a particular subject), the encoder arms 310 generate multiple concatenated feature representations 340, each associated with a particular resolution. Each of these concatenated feature representations 340 can itself include elements representing multiple types of MRI sequences represented in the input data.

[0074] The concatenated representation 340 at each resolution can be processed using a decoder arm 320 (e.g., a U-Net model). The decoder arm 320 can be trained using a loss function that quantifies the mismatch and / or discrepancy between the model predictions and the ground truth mask. A loss can be output at each level (e.g., loss per depth) such that the machine learning model learns a representation at each level. The decoder arm 320 can process the concatenated representation 340 to generate predictions that identify one or more portions of the brain mass that display a portion of a lesion (e.g., a portion of a non-enhancing lesion, a portion of an enhancing lesion, or a portion of any type of lesion).

[0075] In some cases, the decoder arm 320 may include one or more skip attention connections 345. The skip attention connections 345 connect two non-adjacent layers of the machine learning model to transfer the fine-grained details extracted by the encoding block 330 of the encoder arm 310 to be combined with semantic features from the upsampling block of the decoder arm 320. The skip features have a lower receptive field (the local region of the input image that the current convolution operation sees), and the skip features may lack context for correct classification of voxels, resulting in FP. For example, a blood vessel may appear similar to a small lesion when viewed in a smaller context, but using high-dimensional features learned from a larger context makes it easier to learn the elongated structure of the blood vessel. Furthermore, inconsistencies in the edges or boundaries of structures in the low-dimensional representation can be corrected using semantic information learned from the high-dimensional representation. The machine learning model may further include a skip attention module 350 that receives the concatenated representation 340 and propagates features through skip connections 345 to enhance and / or modulate fine features from the encoding block 330 of the encoder arm 310 with coarse features from a higher-level representation, such as a deeper level in the decoder arm 320. The skip attention module 350 can reduce FP introduced by the skip function, thereby improving the prediction of lesion indications. The skip attention module 350 uses soft attention, which can be trained by standard backpropagation techniques. In one embodiment, the skip attention module 350 includes at least one hidden layer connected to non-adjacent layers and / or adjacent preceding layers via residual connections 355. The residual connections 355 between the hidden layer and the adjacent preceding or non-adjacent layers can be used to bypass the path in which the skip features are multiplied by the voxel-wise attention learned from the sum of the skip features and the upsampled features. The residual connections 355 bypass attention based on determining that the upsampled features contain insufficient signal.Because MS lesions tend to be small, there may not be enough signal in the high-dimensional features to guide skip features, so residual connections 355 avoid attention from the high-dimensional features when necessary.

[0076] The machine learning model can output a prediction 360, which can be an example of a prediction 185. The prediction 360 can identify one or more portions of a brain mass predicted to display at least a portion of a lesion. The prediction 360 can be used in determining the number of lesions, lesion size, or lesion burden within the brain mass. The prediction 360 can further be used in combination with data corresponding to a previous MRI to determine changes in lesion volume, size, or cumulative size. Furthermore, recommendations for modifying treatment strategies can be determined based on the prediction 360. In some cases, the prediction 360 can also be used to diagnose a subject with multiple sclerosis.

[0077] II.C, Illustrative Process FIG. 4 illustrates an exemplary process for segmenting a representation of a lesion in an image using a model with multiple encoder arms, according to various embodiments. At block 402, three-dimensional MRI images (e.g., input image 135 of FIG. 1 ) are accessed. The three-dimensional MRI images represent the same mass of the subject's brain. The mass of the brain includes at least a portion of one or more lesions. Furthermore, a first one of the three-dimensional MRI images can be generated using a first type of MRI sequence (e.g., T1, T2, or FLAIR) that is different from a second type of MRI sequence used to generate a second one of the three-dimensional MRI images. The three-dimensional MRI images can be accessed by a segmentation controller (e.g., lesion segmentation controller 170) from an imaging system (e.g., imaging system 130).

[0078] At block 404, each of the three-dimensional MRI images is processed using one or more corresponding encoder arms of a machine learning model (e.g., segmentation model 165 of FIG. 1). Each encoder arm of the machine learning model can correspond to a type of MRI sequence, such that a three-dimensional MRI image generated by a particular type of MRI sequence can be processed by an encoder arm corresponding to the particular type of MRI sequence. The encoder arms can generate respective encodings of the three-dimensional MRI images.

[0079] In block 406, the encodings of the three-dimensional MRI image are concatenated to generate a concatenated representation (e.g., concatenated representation 235 or concatenated representation 340). The encodings from each encoder arm (e.g., corresponding to a particular type of MRI sequence) can be concatenated. Concatenation can be performed at each resolution so that there can be a concatenated representation generated for each resolution.

[0080] In block 408, the concatenated representation is processed using the decoder arm of the machine learning model to generate a prediction identifying one or more portions of the brain mass predicted to display at least a portion of a lesion. The prediction can then be output. Using the prediction, the number of lesions, one or more lesion sizes, and / or lesion burden can be determined. Data corresponding to previous MRIs can be accessed, and the prediction and the data can be used to determine a change in the volume, size, or cumulative size of one or more lesions, and an output representing the change can be generated. A change in treatment strategy can be recommended based on the prediction. Additionally or alternatively, an output corresponding to a possible or confirmed diagnosis of multiple sclerosis in the subject can be provided based at least in part on the prediction. The subject can be diagnosed with multiple sclerosis based at least in part on the prediction.

[0081] III. Working Examples III.A. Example 1 To evaluate model performance, datasets were collected from Opera I (NCT01247324) and tested on the Opera II (NCT01412333) datasets (baseline, 24 weeks, 48 ​​weeks, and 96 weeks). Each dataset contained MRI images and an indication of which voxels within the MRI images displayed lesions (weighted, unweighted, or T2 hyperweighted). Various model architectures were trained using the opera1 dataset and tested on the independent opera2 dataset to generate performance results.

[0082] Each model architecture used an encoder arm 310 with a set of encoder models. The encoder arm 310 included a first model configured to receive T1-weighted images, a second model configured to receive T2-weighted images, and a third model configured to receive FLAIR images. Each of the set of encoder models included multiple depth levels for extracting image features. Each of the first set of encoder models was configured to output multiple feature arrays. For each depth level, the feature arrays generated by all of the corresponding models were concatenated. The concatenated array was provided to a decoder arm 320. Subsequent models may include: ● A stacked U-net model (Stacked Unet) configured to receive as input the encodings generated in a single arm (by one encoding model) of a concatenation of three consecutive T1 slices (256 × 256 × 3 voxels), three consecutive T2 slices, and three consecutive FLAIR slices, using a single loss or a loss per depth: ● A U-net model configured to receive as input the encodings generated in a single arm (by one encoding model) of the concatenation of the T1 patch (96 × 96 × 32 voxels), the T2 patch, and the FLAIR patch using a single loss (“3D; single loss”); ● A U-net model configured to receive the encodings generated in three arms (by three encoding models) corresponding to three types of images (T1, T2 and FLAIR) trained using a single loss ("3D; 3-arm; single-loss"); ● A U-net model configured to receive encodings generated in three arms corresponding to three types of images (T1, T2, and FLAIR) trained using losses calculated at each of multiple depths ("3D; 3 arms; loss per depth"); and ● A U-net model with skip attention connections configured to receive encodings generated in three arms corresponding to three types of images (T1, T2 and FLAIR) trained using losses calculated at each of multiple depths ("3D; 3 arms; loss per depth + att").

[0083] For each model, images were processed cross-sectionally (e.g., ignoring temporal and longitudinal information). Model predictions were evaluated for segmentation (voxel-by-voxel) and detection (lesion-by-lesion) performance. In subsequent longitudinal analyses, a heuristic approach was used to identify new and expanding lesions from the continuous lesion mask.

[0084] The 3D; 3-arm; depth-per-loss + att model has the architecture shown in Figure 3. Specifically, the model included an encoder arm 310 (e.g., a multi-arm encoder) for separately extracting features from 3D T1-weighted MRI image patches, 3D T2-weighted MRI image patches, and 3D FLAIR image patches. The encoder arms were designed to have the same receptive field in all dimensions. A non-isotropic kernel was used to extract features from a non-sampled 1 × 1 × 1 mm image. 3 The anisotropic image resolution (1 × 1 × 3 mm) of the Opera 1 and 2 datasets (NCT01247324 and NCT01412333, respectively) was used to reduce memory and computational requirements compared to training and running the model using a resolution of 3A patch size of 96 × 96 × 32 was used for the Opera dataset. For the ISBI dataset, the preprocessed images were 1 × 1 × 1 mm. 3 Since the images were resampled to have a resolution of , an isotropic kernel was used.

[0085] The 3D; 3-arm; depth-specific loss + att model was a U-Net model configured to include nodes located across a U-shaped distribution of levels extending from the highest level to the lowest level (i.e., the bridge level) and back, with the output from each level feeding the next (lower or higher level). The decoder arm 320 included a skip attention module 350 so that gating signals could be provided from deeper or higher levels. The skip attention module 350 included residual connections 355 that could be used to bypass attention from high-dimensional features, if necessary.

[0086] We trained a U-net machine learning model using a combination of the Tversky loss and the weighted binary cross-entropy loss. The weighted binary cross-entropy loss is a voxel-wise loss used to aid in segmenting smaller lesions. To account for the large imbalance between the foreground / lesion class and the background, the weighted binary cross-entropy heavily weights the foreground (depending on the corresponding hyperparameter settings). Adding the weighted binary cross-entropy helped detect smaller lesions, but it also risked increasing false positives. To reduce false positives, we also used the Tversky loss. The beta variable was set to 0.7 so that the contribution from false positives was 0.7 and the contribution from false negatives was 0.3. Deep supervision with losses was used at all levels to facilitate network learning representations at each level.

[0087] Figure 5A shows the results comparing the performance of a 3D; 3-arm; single-loss model with 3D; 3-arm; loss per depth model to understand the effect of deep supervision and 3D; and 3-arm; loss per depth + att model to understand the effect of attention. Segmentation results were calculated by characterizing the accuracy of predictions at the voxel level. Notably, the average Dice score (defined as false negatives, twice the true positives divided by the sum of false positives, and twice the true positives) was higher for the model with attention supervision compared to the other models without it, and the Dice score for the model using loss per depth was higher than that of the single-loss model. This result held true regardless of whether Dice scores were calculated across all images associated with lesion loads less than 5 ml, lesion loads between 5 and 15 ml, or lesion loads greater than 15 ml. Furthermore, the attention model achieved superior precision (defined as the number of true positives divided by the sum of true positives and false positives) and superior absolute volume difference (AVD, defined as the absolute difference between predicted volume and GT volume divided by GT volume). Although the sensitivity decreased by about 1%, the precision and AVD of the attention model improved substantially.

[0088] Figure 5B shows the detection results using three different machine learning models. Detection results were calculated by characterizing the accuracy of predictions at the lesion level. The calculated statistics included the lesion positive predictive value (PPV, defined as the number of true positives for a lesion divided by the sum of true positives and false positives), the lesion-specific true positive rate (LTPR), and the lesion-specific false positive rate (LFPR, defined as the number of false positives for a lesion divided by the sum of predicted true positives and false positives). The model including attention concatenation achieved the highest PPV and lowest LFPR values ​​compared to the other models evaluated. The 3D; 3-arm; per-depth loss model also achieved a higher PPV score compared to the single-loss model. These results suggest that both per-depth loss and skip attention modules (e.g., skip attention module 245 and skip attention module 350) improve the ability to segment and detect lesions.

[0089] Figure 6A shows the segmentation results using three different machine learning models using 2.5D and 3D input MRI images. Two-dimensional and three-dimensional models are commonly used for lesion detection. Stacked Unet is a slight improvement over 2D and captures local context. Because most MS lesions are small, Stacked Unet has the best sensitivity. Patched Unet is a simple anisotropic 3D network that captures more context. Patched Unet and variants of Patched Unet were the best-performing models for the ISBI dataset. We used a multi-arm patch model based on observations in the GT mask where, in some cases, lesion information is present in one input channel but not the other.

[0090] In terms of segmentation (voxel-level performance), Stacked Unet and Multi-Arm Unet were comparable, with Stacked Unet achieving the highest Dice score and precision. However, as shown in Figure 6B, Stacked Unet had more false positives compared to the 3-arm model. Therefore, in terms of detection (lesion-level performance), Stacked Unet achieved higher performance in terms of LTPR. The 3D; Arm; Loss per Depth + att model increased LPPV and decreased LFPR, but at the expense of LTPR.

[0091] Figure 7 shows the segmentation and detection results using seven different machine learning models. The models were evaluated on the ISBI dataset. Performance metrics included score (combination of other metrics), Dice coefficient, PPV, LTPR, LTFR, and AVD.

[0092] The multi-arm Unet model achieved Dice coefficients, LTPRs, and AVDs comparable to the top-performing models. These results suggest that the multi-arm Unet model, as described in Figures 2 and 3, can accurately detect lesion voxels and lesion counts.

[0093] III.B. Example 2 Lesions are frequently assessed during clinical trials to predict whether a given therapeutic agent will be effective in treating multiple sclerosis to slow disease progression. In particular, study endpoints often examine the extent to which lesion count and / or lesion size (e.g., cumulative lesion size) change during the study period. Changes observed when a particular treatment is administered during the study period can be compared to corresponding changes observed when no treatment or baseline treatment is administered during the study period.

[0094] The data in this example compare the change in the number of new and expanding T2 lesions with ocrelizumab 400 mg compared to interferon-beta-1α 44 μg. Using lesion masks from serial MRI scans in a longitudinal analysis, a heuristic approach was developed to identify and estimate the number of new and expanding T2 lesions at weeks 24, 48, and 96 relative to the previous time points (baseline, w24, and w48, respectively). The heuristic was adjusted with the GT lesion mask to match the percentage reduction in the average number of new and expanding T2 lesions between the treatment (Ocr) and control (Inf) arms. The heuristic approach was applied to T2 lesion segmentation masks from three previously described models (Stacked Unet, Patched Unet, and Multi-Arm Unet) along with two ground truth models (automated and automated with a minimum lesion size of 3 voxels). The percentage reduction in this imaging endpoint between the treatment and control groups was estimated at weeks 24, 48, and 96. To understand whether there was a significant difference in this endpoint between groups, a negative binomial regression of the number of new enlarging T2 lesions was performed with treatment group, presence of T1 Gd-enhancing lesions at baseline (yes or no), baseline Diastolic Dysfunction Status Score (EDSS, <4 vs. >=4), and geographic region (USA vs. rest of the world) as independent variables.

[0095] Results were calculated for each of the following: ● Ground truth: Original analysis of the Opera dataset by radiologists. Two manual radiologist reads were performed on the MRI images. The first read was for lesion segmentation at each time point (e.g., baseline, w24, w48, w96). The second read was for detecting new and expanding T2 lesions, taking into account previous segmentations and candidate new / enlarging lesions. ● Automated GT: Automation of new / enlarging T2 lesion detection when a GT radiologist read baseline, w24, w48, and w96 without a second radiologist reading. For example, lesion segmentation at baseline and w24 was used to determine which lesions were new at w24. This process was performed for each successive pair (e.g., w24 and w48, w48 and w96). ● Auto-GT (GT >= 3 voxels): Lesion detection using an automatic GT approach while enforcing a minimum lesion size of 3 voxels. ● Stack: The stack U-net model was configured to receive as input the encoding generated (by one encoding model) in a single arm of a concatenation of three consecutive T1 slices (of 256 × 256 × 3 voxels), three consecutive T2 slices, and three consecutive FLAIR slices using a single loss. Lesion detection using an automated approach to identify new and expanding T2 lesions ("2.5D; single loss") was applied to T2 lesions predicted by the stack U-net model. ● Patch: The U-net model was configured to receive as input the encoding generated (by one encoding model) in a single arm of the concatenation of T1, T2, and FLAIR patches (of 96 × 96 × 32 voxels) using a single loss. Lesion detection using an automated approach to identify new and expanding T2 lesions ("3D; single loss") was applied to T2 lesions predicted by the Patch U-net model. ● Multi-arm unit: A U-net model with skip attention connections configured to receive encodings generated in three arms corresponding to three types of images (T1, T2, and FLAIR) trained using losses calculated at each of multiple depths. Lesion detection using an automated approach to identify new and expanding T2 lesions applied to T2 lesions predicted by the multi-arm U-net model with skip attention connections ("3D; 3 arms; loss per depth + att"). This is the embodiment of Figure 3 described in the previous example.

[0096] As shown in Figure 8, the clinical data showed that the number of T2 lesions differed significantly between the two treatment groups at each of the three time points. According to manual radiologist reading, subjects receiving interferon beta-1α (Inf) had an average of approximately 1.8, 1.0, and 2.5 new / enlarging T2 lesions at Weeks 24, 48, and 96, respectively. Subjects receiving oclizumab (Ocr) had an average of approximately 0.7, 0.1, and 0.1 new / enlarging T2 lesions at Weeks 24, 48, and 96, respectively. Subjects receiving Ocr had 61%, 96%, and 97% fewer new / enlarging T2 lesions than subjects receiving Inf at Weeks 24, 48, and 96, respectively. The reduction rates between the groups were significantly different at each time point (p<0.001). Although the automated GT and automated GT (GT >= 3 voxels) approaches were successful in predicting significant differences at each time point, these techniques relied on manual lesion segmentation annotation at each time point.

[0097] All three fully automated techniques (Stack, Patch, and Multi-Arm Unet) captured significant differences between treatment groups at weeks 48 and 96. However, as detailed in both Figures 2 and 3, the Multi-Arm Unet technique was the only fully automated technique whose MRI imaging output successfully captured significant differences between treatment groups at week 24.

[0098] Furthermore, the mean lesion counts generated by the multi-arm Unet technique were closer to those from the clinical database, whereas the other two fully automated techniques yielded mean lesion counts that more substantially exceeded the true mean lesion counts, indicating that these other techniques had more false positives compared to the multi-arm Unet technique in Figures 2 and 3. The reduction rates from all three models were also significantly lower than those from manual assessment, demonstrating the inadequacy of heuristic approaches for identifying new / enlarging T2 lesions.

[0099] IV. Further Considerations Some embodiments of the present disclosure include a system including one or more data processors. In some embodiments, the system includes a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein. Some embodiments of the present disclosure include a computer program product tangibly embodied in a non-transitory machine-readable storage medium containing instructions configured to cause one or more data processors to perform some or all of one or more methods and / or some or all of one or more processes disclosed herein.

[0100] The terms and expressions which have been employed are used as terms of description rather than of limitation, and there is no intention in the use of such terms and expressions to exclude any equivalents of the features shown and described, or portions thereof, but it is recognized that various modifications are possible within the scope of the invention as claimed. Thus, although the claimed invention has been specifically disclosed by embodiments and optional features, it will be understood that modifications and variations of the concepts disclosed herein may be resorted to by those skilled in the art, and that such modifications and variations are deemed to be within the scope of the invention as defined by the appended claims.

[0101] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of preferred exemplary embodiments provides those skilled in the art with an enabling description for implementing various embodiments. It will be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0102] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

[0103] V. Illustrative Embodiments As used hereinafter, any reference to a series of examples should be understood disjunctively as a reference to each of those examples (e.g., "Examples 1-4" should be understood as "Examples 1, 2, 3, or 4").

[0104] Example 1 is a computer-implemented method that includes accessing a plurality of three-dimensional magnetic resonance imaging (MRI) images, each of the plurality of three-dimensional MRI images displaying the same mass of a brain of a subject, wherein a first three-dimensional MRI image was generated using a first type of MRI sequence that is different from a second type of MRI sequence used to generate a second three-dimensional MRI image; for each three-dimensional MRI image of the plurality of three-dimensional MRI images, processing the three-dimensional MRI image using one or more corresponding encoder arms of a machine learning model to generate an encoding of the three-dimensional MRI image; concatenating the encodings of the plurality of three-dimensional MRI images to generate a concatenated representation; and processing the concatenated representation using a decoder arm of the machine learning model to generate a prediction that identifies one or more portions of the brain mass predicted to display at least a portion of a lesion.

[0105] Example 2 is the computer-implemented method of example 1, further including: for each three-dimensional MRI image of the plurality of three-dimensional MRI images, generating a down-sampled encoding having a resolution lower than a resolution of the encoding of the three-dimensional MRI image; for each three-dimensional MRI image of the plurality of three-dimensional MRI images, processing the down-sampled encoding using one or more layers of one or more corresponding encoding arms; and concatenating the down-sampled encoding to generate another concatenated representation, wherein the prediction is further based on processing the another concatenated representation using a decoder arm of the machine learning model.

[0106] Example 3 is the computer-implemented method of example 1 or 2, wherein the machine learning model comprises a U-Net machine learning model.

[0107] Example 4 is the computer-implemented method of any one of Examples 1 to 3, wherein the machine learning model includes one or more skip attention modules, each of which connects an encoding block of an encoding arm of the machine learning model to a decoder block of a decoder arm at the same resolution.

[0108] Example 5 is the computer-implemented method of Examples 1-4, wherein each skip attention module of the skip attention modules receives input of a concatenated representation and an upsampled encoding of another concatenated representation at a resolution of a three-dimensional MRI image, and the prediction is further based on processing the output of the skip feature encoding from the skip attention module using a decoder arm of the machine learning model.

[0109] Example 6 is the computer-implemented method of Examples 1-5, wherein one or more skip attention modules include residual connections between inputs and outputs of the skip attention modules to facilitate skipping of the skip attention modules when associated high-dimensional features are unavailable.

[0110] Example 7 is the computer-implemented method of any one of Examples 1-6, wherein the machine learning model is trained using weighted binary cross-entropy loss and / or Tversky loss.

[0111] Example 8 is the computer-implemented method of any one of Examples 1-7, wherein the machine learning model is trained using losses calculated at each of multiple depths of the machine learning model.

[0112] Example 9 is the computer-implemented method of any one of Examples 1 to 8, wherein the first type of MRI sequences includes sequences from a T1, T2, and fluid-attenuated inversion recovery (FLAIR) sequence set, and the second type of MRI sequences includes another sequence from the sequence set.

[0113] Example 10 is the computer-implemented method of any one of Examples 1-9, further comprising using the prediction to determine the number of lesions.

[0114] Example 11 is the computer-implemented method of any one of Examples 1-10, further comprising using the prediction to determine one or more of lesion size or lesion burden.

[0115] Example 12 is the computer-implemented method of any one of Examples 1-11, further including accessing data corresponding to a prior MRI, using the predictions and the data to determine a change in volume, size, or cumulative size of one or more lesions, and generating an output representative of the change.

[0116] Example 13 is the computer-implemented method of any one of Examples 1-12, further comprising recommending a change in treatment strategy based on the prediction.

[0117] Example 14 is the method of any one of Examples 1-13, further comprising providing an output corresponding to a probable or confirmed diagnosis of multiple sclerosis in the subject based at least in part on the prediction.

[0118] Example 15 is the method of any one of Examples 1-14, further comprising diagnosing the subject with multiple sclerosis based at least in part on the prediction.

[0119] Example 16 is a system comprising one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more methods disclosed herein.

[0120] Example 17 is a computer program product tangibly embodied in a non-transitory machine-readable storage medium that includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

Claims

1. 1. A computer-implemented method comprising: accessing a plurality of three-dimensional magnetic resonance imaging (MRI) images, each of the plurality of three-dimensional MRI images displaying the same mass of the subject's brain, a first three-dimensional MRI image being generated using a first type of MRI sequence that is different from a second type of MRI sequence used to generate a second three-dimensional MRI image; For each three-dimensional MRI image of the plurality of three-dimensional MRI images, processing the three-dimensional MRI image using one or more corresponding encoder arms of a machine learning model to generate an encoding of the three-dimensional MRI image; concatenating the encodings of the plurality of three-dimensional MRI images to generate a concatenated representation; processing the concatenated representation using a decoder arm of the machine learning model to generate a prediction identifying one or more portions of the brain mass predicted to display at least a portion of a lesion; 11. A computer-implemented method comprising:

2. generating, for each three-dimensional MRI image of the plurality of three-dimensional MRI images, a down-sampled encoding having a resolution lower than a resolution of the encoding of the three-dimensional MRI image; For each three-dimensional MRI image of the plurality of three-dimensional MRI images, processing the down-sampled encoding using one or more layers of one or more corresponding encoding arms; concatenating the downsampled encodings to generate another concatenated representation, wherein the prediction is further based on processing the another concatenated representation using the decoder arm of the machine learning model; and The computer-implemented method of claim 1 , further comprising:

3. The computer-implemented method of claim 1 or 2, wherein the machine learning model comprises a U-Net machine learning model.

4. 4. The computer-implemented method of claim 1, wherein the machine learning model includes one or more skip attention modules, each of which connects a coding block of the encoding arm of the machine learning model to a decoder block of the decoder arm at the same resolution.

5. 5. The computer-implemented method of claim 4, wherein each skip attention module of the skip attention modules receives as input the concatenated representation and an upsampled encoding of the another concatenated representation at the resolution of the three-dimensional MRI image, and the prediction is further based on processing an output of the skip feature encoding from the skip attention module using the decoder arm of the machine learning model.

6. 6. The computer-implemented method of claim 5, wherein the one or more skip attention modules include residual connections between inputs and outputs of the skip attention modules to facilitate skipping of the skip attention modules when associated high-dimensional features are unavailable.

7. The computer-implemented method of any one of claims 1 to 6, wherein the machine learning model is trained using weighted binary cross-entropy loss and / or Tversky loss.

8. 8. The computer-implemented method of claim 1, wherein the machine learning model is trained using losses calculated at each of a plurality of depths of the machine learning model.

9. 9. The computer-implemented method of claim 1, wherein the first type of MRI sequences includes a sequence from a set of T1, T2, and fluid-attenuated inversion recovery (FLAIR) sequences, and the second type of MRI sequences includes another sequence from the set of sequences.

10. The computer-implemented method of any one of claims 1 to 9, further comprising using the prediction to determine a number of lesions.

11. The computer-implemented method of any one of claims 1 to 10, further comprising using the prediction to determine one or more of lesion size or lesion burden.

12. accessing data corresponding to a previous MRI; using said prediction and said data to determine a change in volume, size, or cumulative size of one or more lesions; generating an output representative of said change; The computer-implemented method of any one of claims 1 to 11, further comprising:

13. The computer-implemented method of any one of claims 1 to 12, further comprising recommending a change in treatment strategy based on the prediction.

14. 14. The computer-implemented method of any one of claims 1 to 13, further comprising providing an output corresponding to a probable or confirmed diagnosis of multiple sclerosis in the subject based at least in part on the prediction.

15. The computer-implemented method of any one of claims 1 to 14, further comprising diagnosing the subject with multiple sclerosis based at least in part on the prediction.

16. 1. A system comprising: one or more data processors; a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein; A system comprising:

17. A computer program product tangibly embodied in a non-transitory machine-readable storage medium comprising instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

Images