Method for Medical Image Analysis Using Self-Supervised Learning

Abstract

The present invention relates to a method (1400) comprising obtaining a first teacher dataset comprising an unannotated multi-dimensional medical image (202), and performing a self-supervised mask image modeling (MIM) training process (200) to pre-train an image encoder (150) with the first teacher dataset. The method also comprises obtaining a second teacher dataset (203) comprising annotated multi-dimensional medical images (204). Here, each of the annotated multi-dimensional medical images comprises a plurality of image voxels (206) paired respectively with corresponding ground truth labels (208) indicating the class to which the corresponding image voxels belong. The method also comprises performing a supervised training process (160) to train an image analysis model (170) with the second teacher dataset to cause the image analysis model to learn a method for predicting ground truth labels corresponding to the plurality of image voxels of each of the annotated multi-dimensional medical images. The image analysis model incorporates the pre-trained image encoder.

Description

Technical Field

[0001] The present disclosure relates to medical image analysis using self-supervised learning.

Background Art

[0002] Multi-dimensional medical images, such as three-dimensional (3D) medical images, provide a detailed image of a patient's body to assist in facilitating the medical analysis, diagnosis, or treatment of the patient. Such medical images can be generated using various modalities, including, for example, computed tomography (CT) or magnetic resonance imaging (MRI).

Summary of the Invention

[0003] A first aspect of the present disclosure provides a computer-implemented method. When executed on data processing hardware, the method causes the data processing hardware to obtain a first teacher dataset including a plurality of unannotated multi-dimensional medical images, and to perform operations including pre-training an image encoder with the first teacher dataset by executing a self-supervised mask image modeling (MIM) training process. The operations also include obtaining a second teacher dataset including a plurality of annotated multi-dimensional medical images. Here, each of the annotated multi-dimensional medical images includes a plurality of image voxels that are paired with corresponding ground-truth labels indicating the class to which the corresponding image voxels belong. The operations also include executing a supervised training process for learning an image analysis model with the second teacher dataset to cause the image analysis model to learn a method for predicting ground-truth labels corresponding to the plurality of image voxels of each of the annotated multi-dimensional medical images. The image analysis model incorporates a pre-trained image encoder.

[0004] Embodiments of the present disclosure may include any one or more of the following features. In some embodiments, for each of the unannotated multi-dimensional medical images corresponding to the first teacher dataset, the step of performing a self-supervised MIM training process to pre-train the image encoder includes using an image tokenizer configured to receive the corresponding unannotated multi-dimensional medical image as input to generate a series of discrete visual tokens characterizing the corresponding unannotated multi-dimensional medical image, dividing the corresponding unannotated multi-dimensional medical image into a plurality of image patches, and randomly masking a portion of the image patches divided from the corresponding unannotated multi-dimensional medical image. For each of the masked image patches, this operation also includes generating an encoded hidden representation for the masked image patch using the image encoder, and based on the encoded hidden representation, generating a corresponding predicted token using the decoder. Here, this operation also includes determining a training loss based on the predicted token generated for the masked image patch and the corresponding visual token from the series of discrete visual tokens that match the masked image patch, and updating the parameters of the image encoder based on the training loss. In these embodiments, the image encoder may include a plurality of multi-head attention layers, and the decoder may include a plurality of multi-head attention layers. Additionally, or alternatively, the step of randomly masking a portion of the image patches includes using one of a central region masking strategy, a block-wise masking strategy, or a uniformly random masking strategy that uses various masked patch sizes and masking ratios to randomly mask a portion of the image patches. The number of visual tokens in the series of discrete visual tokens may be equal to the number of image patches in the plurality of image patches.

[0005] In some examples, for each of the unannotated multi-dimensional medical images corresponding to the first teacher dataset, the step of performing a self-supervised MIM training process to pre-train the image encoder includes splitting the corresponding unannotated multi-dimensional medical image into a plurality of image patches, where each image patch is represented by a corresponding set of raw voxel values, and randomly masking a portion of the image patches split from the corresponding unannotated multi-dimensional medical image. For each of the masked image patches, the operation includes generating an encoded hidden representation for the masked image patch using the image encoder, and generating predicted voxel values for the masked image patch using a prediction head based on the encoded hidden representation. Here, the operation also includes determining a training loss based on the predicted voxel values generated for the masked image patch and the corresponding set of raw voxel values representing the masked image patch, and updating the parameters of the image encoder based on the training loss. In these examples, the image encoder may include a plurality of multi-head attention layers, the prediction head may include a single linear layer prediction head, and may be configured to generate predicted voxel values from the encoded hidden representation without using a decoder. Additionally, or alternatively, the step of randomly masking a portion of the image patch includes using one of a central region masking strategy, a per-block masking strategy, or a uniform random masking strategy that uses various masked patch sizes and masking ratios to randomly mask a portion of the image patch. As an embodiment, the present image analysis model includes a tumor segmentation model. As an example, the present image analysis model includes a multi-organ segmentation model.

[0006] Another aspect of the present disclosure provides a system including data processing hardware and memory hardware that communicates with the data processing hardware. When executed on the data processing hardware, the memory hardware stores instructions that cause the data processing hardware to perform operations including obtaining a first teacher dataset including a plurality of unannotated multi-dimensional medical images, and performing a self-supervised mask image modeling (MIM) training process to pre-train an image encoder with the first teacher dataset. The operations also include obtaining a second teacher dataset including a plurality of annotated multi-dimensional medical images, where each of the annotated multi-dimensional medical images includes a plurality of image voxels paired with corresponding ground truth labels indicating the class to which the corresponding image voxel belongs. The operations also include performing a supervised training process of learning an image analysis model with the second teacher dataset to cause the image analysis model to learn a method of predicting corresponding ground truth labels for each of the plurality of image voxels of each of the annotated multi-dimensional medical images. The image analysis model incorporates a pre-trained image encoder.

[0007] This aspect may include any one or more of the following features. In some embodiments, for each of the unannotated multi-dimensional medical images corresponding to the first teacher dataset, the step of performing a self-supervised MIM training process to pre-train the image encoder includes using an image tokenizer configured to receive the corresponding unannotated multi-dimensional medical image as input, generating a series of discrete visual tokens characterizing the corresponding unannotated multi-dimensional medical image, dividing the corresponding unannotated multi-dimensional medical image into a plurality of image patches, and randomly masking a portion of the image patches divided from the corresponding unannotated multi-dimensional medical image. For each of the masked image patches, this operation also includes generating an encoded hidden representation for the masked image patch using the image encoder, and based on the encoded hidden representation, generating a corresponding predicted token using the decoder. Here, this operation also includes determining a training loss based on the predicted tokens generated for the masked image patches and the corresponding visual tokens from the series of discrete visual tokens that match the masked image patches, and updating the parameters of the image encoder based on the training loss. In these embodiments, the image encoder may include a plurality of multi-head attention layers, and the decoder may include a plurality of multi-head attention layers. Additionally, or alternatively, the step of randomly masking a portion of the image patches includes using one of a central region masking strategy, a per-block masking strategy, or a uniform random masking strategy that uses various masked patch sizes and masking ratios, and includes the step of randomly masking a portion of the image patches. The number of visual tokens in the series of discrete visual tokens may be equal to the number of image patches in the plurality of image patches.

[0008] In some examples, for each of the unannotated multi-dimensional medical images corresponding to the first teacher dataset, the step of performing a self-supervised MIM training process to pre-train the image encoder includes the step of dividing the corresponding unannotated multi-dimensional medical image into a plurality of image patches, where each image patch is represented by a corresponding set of raw voxel values, and randomly masking a portion of the image patches divided from the corresponding unannotated multi-dimensional medical image. For each of the masked image patches, the operation includes generating an encoded hidden representation for the masked image patch using the image encoder, and generating predicted voxel values for the masked image patch using a prediction head based on the encoded hidden representation. Here, the operation also includes determining a training loss based on the predicted voxel values generated for the masked image patch and the corresponding set of raw voxel values representing the masked image patch, and updating the parameters of the image encoder based on the training loss. In these examples, the image encoder may include a plurality of multi-head attention layers, the prediction head may include a single linear layer prediction head, and may be configured to generate predicted voxel values from the encoded hidden representation without using a decoder. Additionally, or alternatively, the step of randomly masking a portion of the image patch includes using one of a central region masking strategy, a per-block masking strategy, or a uniform random masking strategy that uses various masked patch sizes and masking ratios to randomly mask a portion of the image patch. As an embodiment, the present image analysis model includes a tumor segmentation model. As an example, the present image analysis model includes a multi-organ segmentation model.

[0009] Details of one or more embodiments of the present disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

[0011] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE INVENTION

[0012] Computer vision analysis has witnessed a paradigm shift from the use of convolutional neural networks to the use of multi-head attention-based architectures. This disclosure refers, by way of example, to a transformer-based architecture using self-attention as one form of a multi-head attention-based architecture, but the disclosure may use other forms of multi-head attention-based architectures that enhance multi-dimensional input images. Generally, a transformer-based architecture (e.g., vision transformer) divides a multi-dimensional input image into patches and creates patch embeddings as inputs to a transformer-based model for various vision tasks including image classification, object detection, and image segmentation.

[0013] Three-dimensional (3D) medical imaging techniques such as computed tomography (CT) or magnetic resonance imaging (MRI) are widely used to diagnose and treat a wide range of diseases. In general, 3D medical volumetric images can help improve the speed and accuracy of diagnosing a patient's condition. For example, detecting and measuring tumor lesions promptly and appropriately from an MRI or CT scan is important for disease prevention, early detection, and treatment plan optimization, and promotes the development of more effective clinical applications, ultimately improving the patient's life. The basic tasks performed in medical image analysis include 3D image segmentation. Another basic task performed in medical image analysis includes image classification. The image classification task classifies input images into various categories. In general, 3D image segmentation (also referred to as "3D semantic segmentation") aims to separate each specific object from one another by predicting the corresponding class for each voxel of the volumetric input image and covering each specific object with its respective segmentation mask in order to classify one or more specific objects. 3D image segmentation has the potential to relieve the workload of radiologists on a daily basis by automating or assisting the image interpretation workflow, ultimately improving clinical care and patient outcomes. The 3D image segmentation task may include multi-organ segmentation performed as a 13-class segmentation task with single-channel input, and brain tumor segmentation performed as a 3-class segmentation class with 4-channel input.

[0014] Training a robust transformer-based image analysis model requires more annotated teacher data that outperforms the performance of conventional CNNs. However, the high cost of obtaining dedicated annotations for 3D medical volume images in specific regions often hinders attempts to utilize advanced technologies for clinical outcomes, such as using deep learning approaches for 3D medical image analysis. In short, the large-scale annotation of 3D medical images by radiologists is limited in terms of being costly and time-consuming to create. Another limiting factor in 3D medical image processing is the sheer amount of data related to 3D medical images. The amount of data is determined by the increased dimensions and resolution of 3D images, resulting in significant processing complexity. As a result, the ability to effectively integrate radiomics endpoint information with other biomarker data for other downstream tasks in clinical study design, such as tumor volume assessment and overall survival prediction, can be very difficult.

[0015] Transfer learning is the use of a model learned in one context for another context. Transfer learning from ordinary images can be used for medical image analysis regardless of differences in image statistics, scale, and task-related characteristics. For example, transfer learning from ImageNet can accelerate the convergence of medical images and can be useful when medical image teacher data is limited. Transfer learning using region-specific data can also help solve problems due to regional differences. For example, improved performance can be achieved with pre-training followed by labeled data from the same region. However, this strategy can often be unrealistic because various medical scenarios require cost and time to collect labeled data. Self-supervised learning provides a viable alternative that can use unlabeled / unannotated medical data.

[0016] Self-supervised learning is a training technique that emphasizes learning representations from unlabeled data such that a low-capacity classifier can achieve high accuracy using various embeddings. Contrastive learning is another example of a self-supervised learning strategy. Contrastive learning creates image similarity and dissimilarity (or just similarity) between two or more regions with data augmentation that is important for contrastive and related approaches. Self-supervised learning can be used in the medical field for tasks that do not seem to have region-specific relationships or for contrastive learning adapted to medical data. The scope of self-supervised learning strategies can be applied to 3D medical images. For example, a model pre-trained on ImageNet can be applied to dermatology image classification. As another example, inpainting can be associated with contrastive learning in medical image segmentation.

[0017] Approaches to masked image modeling generally mask out a part of the input image or the encoded image tokens and have the model reconstruct the masked regions. Some existing MIM models use an encoder-decoder design followed by a projection head. The encoder aids in modeling the latent feature representation, while the decoder aids in resampling the latent vectors back to the original image. The encoded or decoded embeddings can then match the original signal in the regions masked by the projection head. In particular, the decoder component can be designed to be lightweight to minimize training time. A lightweight decoder can not only reduce the computational complexity but also increase the encoder's ability to learn more generalizable representations that the decoder can easily grasp, transform, and convey. The encoder can be used for fine-tuning. Encoding techniques such as SimMIM can eliminate the need for an entire decoder with a single projection layer.

[0018] When using a Vision Transformer (ViT), for example, an image can be divided into non - overlapping patches (e.g., a 96×96×96 3D volume can be divided into 216 patches of 16×16×16 small volumes), which are often considered as the basic processing units of the Vision Transformer. There are several random masking techniques. This includes, but is not limited to, the central region masking strategy, the per - complex - block masking strategy, and / or the patch - level uniform random masking method that uses various masked patch sizes and masking ratios.

[0019] In some examples, the image encoder includes a Vision Transformer (ViT) architecture such as a standard (vanilla) ViT (e.g., ViT3D, Swin Transformer 3D, and / or Attention Visual Network (e.g., VAN3D)). This can inherit the attention mechanism to derive a hierarchical representation similar to, for example, Swin Transformer 3D, but uses pure convolution instead. Other forms of multi - head attention layers can be used by an image encoder such as a conformer layer, a performer layer, or a lightweight convolution layer, but are not limited to these.

[0020] Embodiments of this document are directed to performing a self-supervised masked image modeling (MIM) training process to pre-train an image encoder on a plurality of unannotated (e.g., unlabeled) multi-dimensional medical images. As used herein, multi-dimensional images are referred to as 3D medical images, but the disclosure is not limited thereto and may also include 4D medical images. 3D medical images include volume slices from CT or MRI of a region inside (or outside) a patient's body. The image encoder includes a plurality of multi-head attention layers. For example, the image encoder may include a transformer-based architecture that uses self-attention using a stack of transformer layers. As will become apparent, the image encoder is responsible for modeling the latent feature representation of masked image patches, which can be used later to predict the original image signal within the region associated with the masked image patch. Through the self-supervised MIM training process, a pre-trained image encoder on unannotated 3D medical images can be adapted to a wide range of downstream vision tasks such as 3D image segmentation and image classification.

[0021] A pre-trained image encoder may be integrated into an image analysis model and may be fine-tuned to use annotated multi-dimensional medical images for performing a specific downstream vision task. The annotated multi-dimensional medical images used to fine-tune the pre-trained image encoder and ultimately train a medical image model to perform a specific vision task may each include a plurality of image voxels that are each paired with a corresponding ground truth label indicating the class to which the corresponding image voxel belongs. In this way, embodiments of the present disclosure perform a supervised training process of learning an image segmentation model with a plurality of annotated multi-dimensional medical images to learn a method of predicting a ground truth label corresponding to each of the plurality of image voxels of the annotated multi-dimensional medical images. Thereby, the image segmentation model includes a pre-trained image encoder that is initialized with unannotated multi-dimensional medical images via a self-supervised MIM training process and fine-tuned with annotated multi-dimensional medical images via a supervised training process. In some examples, the trained image analysis model includes an image segmentation model that performs a 3D image segmentation task, such as multi-organ segmentation or tumor segmentation, on 3D image slices separated from an MRI or CT scan of a body part. As will be described in more detail below, the trained image analysis model receives, as input, a plurality of image patches separated from a multi-dimensional medical image (i.e., volume slices from an MRI or CT scan), generates an enhanced medical image based on features extracted from the multi-dimensional medical image, and may perform image segmentation or image classification on the enhanced image. In an image segmentation scenario, the trained image analysis model may be trained to separate specific objects (e.g., tumors or organs) from each other by classifying one or more specific objects (e.g., tumors or organs) in the enhanced image and expanding the enhanced image to include a respective segmentation mask that superimposes the specific objects.As used herein, extending an enhanced image that includes a segmentation mask includes extending image voxels within the enhanced image that represent each object class and / or define the boundaries of each object class. Extending the image voxels may include changing the color of the image voxels, adjusting the intensity of the image voxels, or extending the image voxels in any suitable manner such that each classified object is distinguishable and identifiable within the enhanced image.

[0022] FIG. 1 shows an exemplary system 100 that pre-trains an image encoder 150 via a self-supervised training process 200 to learn a method for generating an encoded feature representation 225 (FIGS. 2A and 2B) from an unannotated 3D medical image 202, and fine-tunes the pre-trained image encoder 150 for performing a downstream image task via a supervised training process 160. Specifically, the pre-trained image encoder 150 may be adapted for use in an image analysis model 170 to perform a specific vision task by training the image analysis model 170 with the annotated 3D medical image 204. The system 100 includes a computing system 120 comprising data processing hardware 122, and memory hardware 124 that communicates with the data processing hardware 122 and stores instructions for causing the data processing hardware 122 to perform operations. In some embodiments, a first computing system 120, 120a executes a self-supervised training process 200 to pre-train the image encoder 150, and then executes a supervised training process 160 to train an image analysis model 170 incorporating the pre-trained image encoder 150 to perform a downstream vision task on the 3D medical image. In these embodiments, after the image analysis model 170 has been trained to perform the downstream vision task, the first computing system 120a provides the trained image analysis model 170 to a second computing system 120, 120b. Here, the second computing system 120b may execute the image analysis model 170 to generate enhanced 3D medical images 110, 110E from raw 3D medical images 110, 110R, and perform a downstream vision task with the enhanced 3D medical images 110E.

[0023] The first computing system 120a may include a distributed system (e.g., a cloud computing environment). The second computing system 120b may include a computing device (e.g., a desktop computer, a workstation, a laptop, a tablet, etc.) that downloads the image analysis model 170 from the first computing system 120a. In some other embodiments, the first computing system 120a receives the 3D medical image 110R from the second computing system 120b and executes the image analysis model 170 to perform downstream vision tasks. In additional embodiments, the second computing system 120b receives the pre-trained image encoder 150 from the first computing system 120a by the self-supervised training process 200 and executes the supervised training process 160 to fine-tune the pre-trained image encoder for downstream vision tasks. In this scenario, the annotated MD image 204 may be locally processed at the second computing system 120b via the supervised training process 160, thereby protecting privacy and confidential data.

[0024] The self-supervised training process 200 trains the image encoder 150 with a first teacher dataset 201 that includes a plurality of unannotated multi-dimensional (MD) images 202. Specifically, and as described in more detail below with reference to FIGS. 2A and 2B, the self-supervised training process includes a self-supervised mask image modeling (MIM) training process. Each unannotated MD image 202 in the first teacher dataset 201 may include an image slice segmented from a CT scan or an MRI scan of a patient's body. As a result, the first teacher dataset 201 includes a corpus of unannotated MD medical images 202 related to image slices from CT scans and / or MRI scans of the bodies of multiple patients. As one example, the first teacher dataset 201 includes unannotated 3D CT scan images 202 obtained from the The Cancer Imaging Archive-Covid19 (TCIA-Covid19) public dataset. Here, the unannotated 3D CT scan images include 771 volumes of unenhanced chest CT scans collected from 661 patients with Covid19 infection.

[0025] In particular, self-supervised MIM training as disclosed herein is particularly advantageous for modeling 3D medical images by significantly accelerating the convergence of training and improving downstream performance. For example, when comparing with simple contrastive learning, when the trained image encoder 150 is adapted and fine-tuned to perform downstream vision tasks, the convergence of training can save up to 1.40 times the training cost to achieve the same or higher Dice coefficient. Similarly, the downstream performance of downstream vision tasks for image segmentation can achieve an improvement of 5 percent (%) or more without any hyperparameter tuning. In addition, downstream applications incorporating image encoders pre-trained via self-supervised MIM training are faster and more cost-effective than transfer learning to specific downstream tasks such as prognosis, treatment sensitivity prediction, tissue segmentation, image classification, and digital representation of patients. As will become apparent, training the image encoder 150 via the self-supervised MIM training process 200 enables the prediction of raw voxel values using a high masking ratio and a relatively small patch size. To simply reconstruct the raw input 3D medical image 110R into the enhanced 3D medical image 110E, a lightweight decoder may be implemented to receive the encoded feature representation 225 output by the image encoder 150 and reconstruct the image signal with increased speed and reduced computational and memory costs. Self-supervised MIM training is versatile across the entire raw input 3D medical image 110R, having diverse image resolutions and labeled data ratios during the supervised training process 160.

[0026] Generally, MIM learning includes a learning task that involves masking a subset of the input signal (e.g., image patch 210) and predicting the masked signal. In other words, MIM learning / training is a self-supervised learning technique that learns representations via an image corrupted by a mask. The masking can be provided as a noise type. The prediction of the masked patch in self-supervised learning can predict the lost voxels by repairing a large rectangular region of the source area and grouping the voxel values for each cluster to classify the unknown voxel values. Additionally, the prediction of the masked patch in self-supervised learning can be accomplished by predicting the average color of the image.

[0027] After the image encoder 150 is pre-trained via the self-supervised training process 200, the supervised training process 160 trains the image analysis model 170 on a second teacher dataset 203 that includes a plurality of annotated MD medical images 204. The supervised training process 160 fine-tunes the pre-trained image encoder 150 integrated with the image analysis model 170 and causes the image analysis model 170 to learn to perform downstream vision tasks such as an image segmentation task or an image classification task. Each annotated MD medical image 204 includes a plurality of image voxels 206 that are paired with corresponding ground truth labels 208 indicating the class to which the corresponding image voxels 206 belong. In particular, the unannotated 3D images 202 in the first teacher dataset 201 used to pre-train the image encoder 150 may be associated with different medical regions than the annotated 3D images 204 in the annotated second teacher dataset 203. For example, the first dataset 201 may include chest CT scans, and the second dataset 203 may include abdominal CT scans or multi-modal MRI scans of brain tumors.

[0028] The image analysis model 170 may include a U-shaped encoder-decoder architecture that includes an image encoder 150 (used as a ViT-based encoder, a Swin Transformer, or a VAN) to generate hierarchical encoded features 225 (FIGS. 2A and 2B) from the image patch 210 and the decoder 152. The decoder 152 may include a UPerNet to perform an image segmentation task based on the encoded features 225 output from the image encoder 150. That is, the two-layer transposed convolution can be used as the projection head 260 (FIG. 2A) during the self-supervised MIM training process for pre-training the image encoder 150, and the UPerNet decoder 152 can be implemented for use with the pre-trained image encoder 150 by the image analysis model 170 for performing downstream image segmentation. In some examples, the image encoder 150 includes a masked autoencoder (MAE) (see FIG. 2A) that uses a stack of multi-head attention layers. For example, the MAE may include an 8-layer stack of Transformer blocks having 512 dimensions for use by the decoder 152. In other examples, the image encoder includes a simple masked image model (SimMIM) (see FIG. 2), and a simple linear layer is used as the projection head instead of the decoder.

[0029] In one example, the second teacher dataset 203 includes annotated 3D CT scans obtained from the abdominal dataset of Beyond the Cranial Vault (BTCV) that contain abdominal CT scans of 30 participants / patients with 13 organ annotations obtained by a human translator under the supervision of a clinical radiologist. Each 3D CT scan of the BTCV abdominal dataset is performed in the portal phase using contrast enhancement and includes 80 to 225 slices with a slice thickness in the range of 512×512 pixels and 1 to 6 millimeters (mm). During preprocessing, each annotated 3D image 204 may be resampled at an isotropic voxel spacing of 1.5 - 2.0. In this example, the teacher - supervised training process 160 trains the image analysis model 170 as a multi - organ segmentation model that performs 13 - class segmentation with a 1 - channel output. As a result, the ground truth label 208 of the image voxels 206 corresponding to each annotated 3D medical image 204 may include one of 13 different classes depending on the organ to which the corresponding image voxels 206 belong.

[0030] In other examples, the second teacher dataset 203 includes annotated 3D MRI scan images obtained from the Brain Tumor Segmentation (BraTS) public dataset that have ground truth labels 208 for the corresponding image voxels 206 representing regions of edema, non - enhancing core, and necrotic core. In this example, the teacher - supervised training process 160 trains the image analysis model 170 as a brain tumor segmentation model that performs 3 - class segmentation with a 4 - channel input. The voxel spacing of the MRI images may be 1.0×1.0×1.0 mm3. The intensity of the voxels may be pre - processed using normalization.

[0031] The supervised training process 200 may store the pre-trained image encoder 150 in the data storage 180 overlaid on the memory hardware 124 of the computing system 120. Similarly, the supervised training process 160 may store the trained image analysis model 170 in the data storage 180. The first computing system 120a and / or any number of second computing systems 120b may connect / obtain to / from the pre-trained image encoder 150 and / or the trained image analysis model 170 for its execution.

[0032] During inference, the image analysis model 170 incorporating the pre-trained and fine-tuned image encoder 150 executes on a second computing system 120b (or the first computing system 120a) to process and perform the image analysis task of one or more raw input 3D medical images 110R. In particular, the image analysis tasks performed by the image analysis model 170 include downstream vision tasks (i.e., image segmentation or image classification). Here, the image analysis model 170 is trained by a supervised training process 160 performed. Each of the raw input 3D medical images 110R may correspond to a 3D image slice from a 3DCT scan or 3DMRI scan of a patient's body. Optionally, the raw input 3D medical images 110R may correspond to 3D images of regions within a patient's body. Each of the raw input 3D medical images 110R may undergo initial image preprocessing 184 to divide the raw input 3D medical image 110R into a plurality of image patches 210, 210a-n. Although 9 image patches are shown as an example, this example is not restrictive and the preprocessing 184 may divide the image into any number of image patches 210. The image analysis model 170 may process the image patches 210 to generate enhanced 3D medical images 110E and perform downstream vision tasks on the enhanced 3D medical images 110E. When the image analysis model 170 performs a downstream vision task of 3D medical segmentation, the model 170 predicts a class corresponding to each voxel of the volumetric enhanced 3D medical image 110E to classify one or more specific objects (e.g., tumors, tissues, organs), and separates each of the specific objects from each other by defining a respective segmentation mask for covering the voxels that classify each object. Example 3D image segmentation tasks may include multi-organ segmentation performed as a 13-class segmentation task with a single channel and brain tumor segmentation performed as a 3-class segmentation class with a 4-channel input.

[0033] The Image Augmenter 360 may receive an enhanced 3D medical image 110E that is segmented to identify voxels representing each particular object class and generate a corresponding segmentation mask for application to at least a portion of the voxels representing the particular object class. Accordingly, the Image Augmenter 360 may augment image voxels in the enhanced image representing each object class and / or defining the boundaries of each object class. Augmentation of the image voxels may include changing the color of the image voxels, adjusting the intensity of the image voxels, or augmenting the image voxels in a suitable manner such that each classified object is distinguishable and identifiable within the enhanced image 110E. The segmentation mask may include a graphic function applied to the enhanced image to convey the location of each object identified in the enhanced image 110E. The Image Augmenter 360 may output an enhanced and augmented image 110A that represents a segmentation mask conveying the results of the segmentation performed by the analysis model 170. The graphic user interface 360 executing on the computing system 120 may display the enhanced image 110A augmented on the screen in the communication of the computing system 120. Additionally, or alternatively, the enhanced image and / or the augmented image 110A may be provided as an output to one or more additional downstream tasks.

[0034] Referring to FIGS. 2A and 2B, in some embodiments, the self-supervised MIM training process 200 pre-trains an image encoder 150 that comprises either a Mask Autoencoder (MAE) architecture (FIG. 2A) or a Simple MIM (SimMIM) architecture (FIG. 2B). For each unannotated 3D medical image 202, the training process 200 first pre-processes the image 202 in a pre-processing stage 184 to divide the image 202 into a plurality of image patches 210, 210a-n. Since it is generally difficult to directly load the volume of the full 3D image onto the data processing hardware 122 (e.g., GPU) of the computing system 120, the self-supervised MIM training process 200 may implement a sliding window training strategy where the pre-processing divides the original 3D medical image 202 into several (e.g., 96×96×96) small 3D windows. By default, the pre-processing stage 184 may implement a patch size of about 16. The pre-processing stage may downsample the image resolution of the unannotated 3D medical image 202. For example, a resolution of 96× the volume can be downsampled to a resolution of 9× the volume when the image encoder 150 includes a ViT-based image encoder, or can be downsampled to a resolution of 3× the volume when the image encoder 150 includes a Swin Transformer or a VAN.

[0035] Figure 2A shows a MIM training process 200 for training an image encoder 150 with a MAE architecture by randomly masking a part of an image patch 210 divided from a corresponding unannotated MD medical image 202. The training process 200 uses one of a central region masking strategy, a per-block masking strategy, or a uniform random masking strategy that uses various masked patch sizes and masking ratios to further randomly mask a part of the image patch 210. The training process uses an image tokenizer 230 configured to receive an unannotated MD medical image 202 as input and further generate a series of discrete visual tokens that characterize the corresponding unannotated MD medical image 202. The number of visual tokens in the series of discrete visual tokens 240 may be equal to the number of image patches 210 divided from the unannotated MD medical image 202. The tokenizer 230 may map discrete image voxels from the medical image 202 to the discrete visual tokens 240 according to a visual codebook that includes a token vocabulary containing discrete token indices. Since the visual tokens 240 are discrete, the training process 200 is not differentiable. In some examples, the tokenizer 230 is trained via an autoencoder-style reconstruction process in which the image is tokenized into discrete visual tokens according to a learned vocabulary.

[0036] As shown by way of example, the self-supervised MIM training process 200 adds a positional embedding 215 to the image patch 210. The image encoder 150 receives each masked image patch 210M, whereby each masked image patch may be replaced with a special masking embedding [M]. The special masking token [M] may be randomly initialized as a learnable vector optimized to reveal the corresponding masked image patch 210.

[0037] For each masked image patch [M], the image encoder 150 is configured to generate a corresponding encoded feature representation 225 (also referred to as the encoded latent representation 225), and the decoder 250 decodes the corresponding encoded feature representation 225 to predict a corresponding predicted token 275 as the output from the projection head 260. The purpose of the MIM training process 200 is to learn the image encoder 150 and the decoder 250 to learn a method for predicting the visual tokens 240 obtained from the original 3D image 202. Specifically, the training process 200 trains the encoder 150 to generate an encoded feature representation 225 of the masked image patch 210M used to generate a predicted token 275 that matches the visual token 240 obtained from the original 3D image 202. Here, the training process 200 may determine a training loss based on the predicted token 275 generated for the masked image patch 210M and the corresponding visual token from a series of discrete visual tokens 240 that match the masked image patch 210M (i.e., using the positional embedding 215). Thereafter, the training process 200 updates the parameters of the image encoder 150 (and optionally the decoder 250) based on the training loss.

[0038] The decoder may include a plurality of multi-head attention layers (e.g., transformer layers). In some examples, the masked image patch 210M is not noticed by the encoder 150, such that only the decoder 250 has knowledge of the various tokens. This approach may save computation and memory without interfering with training.

[0039] Figure 2B shows a self-supervised MIM training process 200 for training an image encoder 150 with a SimMIM architecture that randomly masks a portion of an image patch 210 segmented from a corresponding unannotated MD medical image 202. Each image patch 210 may be represented by a corresponding set of raw voxel values. The training process 200 uses one of a center region masking strategy, a per-block masking strategy, or a uniform random masking strategy that uses various masked patch sizes and masking ratios to further randomly mask a portion of the image patch 210.

[0040] As shown by way of example, the self-supervised MIM training process 200 adds a positional embedding 215 to the image patch 210. The image encoder 150 receives each masked image patch 210M, whereby each masked image patch may be replaced by a special masking embedding [M]. The special masking token [M] may be randomly initialized as a learnable vector optimized to reveal the corresponding masked image patch 210.

[0041] For each masked image patch 210M, the image encoder 150 is configured to generate a corresponding encoded feature representation 225, and the prediction head 260 generates predicted voxel values for the masked image patch 210M. In particular, the MIM training process 200 for pre-training an image encoder 150 with a SimMIM architecture omits the decoder and instead implements a prediction head 260 to directly predict the raw voxel values 270 of each masked image patch 210M from the encoded feature representation 225 generated by the image encoder 225 of the corresponding masked image patch 210M. The training process 200 may determine a training loss based on the predicted voxel values 270 generated from the masked image patches and the set of raw voxel values corresponding to the original unannotated MD medical image 202 representing the masked image patches.

[0042] The training loss may be based on the distance in voxel space between the restored / estimated raw voxel values 270 and the original voxel values from the corresponding set of raw voxel values representing the masked image patch. The training loss may include either the loss function l 1 or l 2 . In particular, the training loss may only compute the masked match 210M in order to prevent the encoder 150 from performing self-reconstruction and potentially dominating the learning process and ultimately the knowledge learning that is disrupted. Thereafter, the training process 200 updates the parameters of the image encoder 150 (and optionally the decoder 250) based on the training loss. The projection head can transform the predicted tokens 275 to the original voxel space when the preprocessing downsamples the resolution of the medical image 202. Optionally, two layers of transposed convolutions can upsample the compressed encoded feature representation 225 to the resolution of the original medical image 202.

[0043] Figure 3 shows examples of input, masked, and reconstructed 3D CT scan images from the validation set of TCIA-COVID19 applying a pre-trained image encoder 150 using SimMIM reconstruction. The original images are all 3D volumes, but the reconstructed images are shown in slice format for ease of explanation and understanding, and the index numbers represent depth. For each triple, the first or leftmost column shows the ground truth (e.g., the original image). The second or middle column shows the masked image. The third or rightmost column shows the machine learning model using SimMIM reconstruction. For the images shown in Figure 5, the ViT-Base backbone is applied to the encoder, the masked patch size is approximately 16 (for all dimensions), and the masking ratio is approximately 75%.

[0044] Figure 4 shows an example of an input, masked, and reconstructed 3D CT scan image from the validation set of TCIA-COVID19 to which a machine learning model using MAE reconstruction is applied. Similar to Figure 3, all the original images are 3D volumes, but the reconstructed images are shown in slice format for ease of explanation and understanding, and the index numbers represent depth. For each triple, the first or leftmost column shows the ground truth (e.g., the original image). The second or middle column shows the masked image. The third or rightmost column shows the machine learning model using MAE reconstruction. For the images shown in Figure 4, the ViT-Large backbone is applied to the encoder, the masked patch size is approximately 16 (for all dimensions), and the masking ratio is approximately 75%.

[0045] Figure 5 shows a table demonstrating that the MIM approach can generally outperform contrastive learning techniques. The pre-trained image encoder 150 has both the MAE architecture and the SimMIM architecture achieving an average Dice coefficient from 0.752 to 0.758, while SimCLR has an average Dice coefficient of approximately 0.723, which is 4.5% lower. As used here, the Dice coefficient is used to evaluate the accuracy of segmentation performed as a downstream vision task. For a given semantic task, G i and P i respectively represent the ground truth and predicted values for each corresponding voxel i. The following equation may be used to clarify the Dice coefficient.

Equation

[0046] FIG. 6 shows a table listing supplementary baseline settings of the supervised training process 160 for training the image analysis model 170 on the BTCV dataset for multi-organ image segmentation. FIG. 7 shows a table listing supplementary baseline settings of the supervised training process 160 for training the image analysis model 170 on the BraTS dataset for brain tumor segmentation. FIG. 8 shows a table listing pre-training settings of the self-supervised training process using the 3DCT image volume as the unannotated 3D medical image 202.

[0047] FIG. 9 is a table revealing the results of using a machine learning model on brain tumor segmentation images after pre-training using the BraTS teacher dataset as the annotated MD medical image 204. The segmentation results of BraTS in FIG. 8 follow a pattern similar to the segmentation results shown in FIG. 5. The average Dice coefficient of the mask image modeling approach is slightly larger than 0.80, while SimCLR obtained a Dice coefficient of 0.7739, which is 4.37% smaller than the best approach comparable to FIG. 5. Another point to note is that despite the similarity of the two MIM techniques, SimMIM can achieve slightly better performance than MAE, as revealed in both FIGS. 5 and 9. One explanation for this is that even if the encoder 150 does not obtain a generalizable representation, an efficient decoder (even a lightweight decoder) may be able to reconstruct the original image, thus periodically relaxing the motivation of the encoder 150 to learn a more efficient representation 225. One goal of self-supervised MIM learning is to learn an efficient and generalizable data representation rather than just self-convergence. In comparison, SimMIM uses a lighter design by omitting the entire decoder and runs the encoder for more complex reconstruction and learning tasks.

[0048] The self-supervised MIM training process 200 reduces the cost of pre-training the image encoder 150 with the first teacher dataset 201 while increasing the training speed. Figure 10 shows a plot depicting how the self-supervised MIM training process 200 proceeds with the supervised training process 160. Here, the average Dice coefficient of the validation set is divided into a supervised baseline and various self-supervised MIM techniques using different masking ratios throughout the training stage. Pre-training the masked image modeling saves training costs and yields better performance. The SimMIM-based architecture can obtain 1.76 times the Dice coefficient at the 1.3k training stage. Furthermore, the MIM-based approach can reach a Dice coefficient of 0.7 with 1.4 times less training time than the training time required by the supervised baseline.

[0049] In some implementations, various masked patch sizes and masking ratios are used to train a model using self-supervised MIM. The results of applying machine learning models to 3D medical images using various MIM techniques and fine-tuning the pre-trained image encoder for downstream image segmentation are summarized in the tables of Figures 11 and 12. Figure 11 includes a table showing an ablation study of various masked patch sizes and masking ratios in multi-organ segmentation. The machine learning model 160 applied to generate the results of Figure 13 is equipped with the default backbone of ViT-B applied as the pre-trained encoder 150. In addition, the machine learning model 160 is trained via the supervised training process 160 using the BTCV teacher dataset. Figure 12 is a table showing an ablation study of various masked patch sizes and masking ratios in brain tumor segmentation. Similarly, the data for pre-training includes the BraTS dataset confidence, and ViT-B is applied as the encoder backbone of UNETR for segmentation fine-tuning.

[0050] A higher masking ratio is an important self-supervised learning job that can continuously run a model to build general representations that can be efficiently converted into downstream tasks. For example, the best Dice coefficients in multi-organ segmentation and brain tumor segmentation are obtained when a masking ratio of about 0.75 is used across multiple patch sizes (e.g., 0.7183 for patch size 16 in Figure 11, 0.8041 for patch sizes 24 and 32 in Figure 12). A high masking ratio combined with a small patch size results in relatively good performance when used in combination with SimMIM. As shown in Figures 11 and 12, when the patch size is equal to 16, the model can run with Dice coefficients of about 0.7249 and 0.8077 respectively. However, as the patch size increases, the SimMIM method appears less sensitive to this masking ratio. For example, when the patch size is about 32, the model can obtain the largest Dice coefficient with a masking ratio of about 0.15, which is the smallest possible masking ratio. Medical images are generally raw and low-level signals with a relatively large amount of spatial redundancy. Repairing some unknown patches is done by directly copying nearby patches with a little comprehensive knowledge of the object and its surroundings. A single small masked patch may not be able to fully mask complex and intersecting structures or positions, while a large patch size can hide more important signals alone. As a result, a high masking ratio for a small patch size can be more critical than a high masking ratio for a large patch size.

[0051] Generally, in supervised learning, more teacher data results in improved performance. FIG. 13 includes a table showing the Dice coefficients of the image analysis model 170 incorporating the hand with the image encoder 150 pre-trained via the self-supervised MIM training process 200 and the MAE architecture (FIG. 2A). The image encoder 150 may be pre-trained for each of various data sources having various degrees of downsampling. The supervised training process 160 may train the image analysis model 170 on a multi-organ segmentation dataset having various labeled data ratios. The results in the table show that models trained with more unannotated 3D medical images 202 via the self-supervised training MIM training process 200 outperform models trained with fewer unannotated 3D medical images 202 (e.g., 4.9% improvement from 0.7184 to 0.7543, 4.6% improvement from 0.7018 to 0.7338). This advantage may be more pronounced at lower image resolutions, such as when only half of the labeled data is used for supervised training, where 0.6818 is 5.6% greater than 0.6552.

[0052] FIG. 13 also shows how different resolutions of unannotated 3D medical images for pre-training affect downstream image task performance. For example, a higher pre-training resolution results in better segmentation results because the image contains more granular information. Here, different downsampling ratios can be used to represent the degree to which the original signal is compressed into all dimensions of each volume. As can be seen from FIG. 13, pre-trained encoder models with high resolutions (e.g., 1.5×, 1.5×, 2.0) generally perform better than pre-trained models with low resolutions (e.g., 2.0×, 2.0×, 2.0). For example, a Dice coefficient of 0.7338 is 2.7% smaller than the Dice coefficient pre-trained using a higher resolution with the same data source and the same labeled ratio.

[0053] FIG. 14 is a flowchart of an exemplary arrangement of operations of a method 1400 for training an image analysis model to perform an image analysis task on multi-dimensional medical images. The data processing hardware 122 of the computing system 120 may perform operations by executing instructions stored in the memory hardware 124. In operation 1402, the method 1400 includes obtaining a first teacher dataset 201 that includes a plurality of unannotated multi-dimensional medical images 202. In operation 1404, the method 1400 includes executing a self-supervised mask image modeling (MIM) training process 200 to pre-train an image encoder 150 with the first teacher dataset 201.

[0054] In operation 1406, the method 1400 includes obtaining a second teacher dataset 203 that includes a plurality of annotated multi-dimensional medical images 204. Here, each of the annotated multi-dimensional medical images 204 includes a plurality of image voxels 206 that are paired with corresponding ground truth labels 208 that indicate the class to which the corresponding image voxels belong. In operation 1408, the method 1400 includes executing a supervised training process 160 to train an image analysis model 170 with the second teacher dataset 203, causing the image analysis model 170 to learn a method for predicting the ground truth label 208 corresponding to each of the plurality of image voxels 206 of the annotated multi-dimensional medical images 204. Here, the image analysis model 170 incorporates a pre-trained image encoder 150. The supervised training process 160 fine-tunes the pre-trained image encoder 150 initialized via the self-supervised MIM training process 200.

[0055] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an “application,” an “app,” or a “program.” Examples of applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0056] A non-transitory memory may be a physical device that serves as a temporary or permanent basis for use by a computing device and is used to store a program (e.g., a sequence of instructions) or data (e.g., state information of a program). The non-transitory memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., used for normal firmware such as a boot program). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disks or tapes.

[0057] FIG. 15 is a diagram schematically showing an example of a computing device 1500 that may be used to implement the systems and methods shown herein. Computing device 1500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The connections and relationships, and functional components shown herein are meant as examples only and are not meant to limit the implementation of the invention described and / or claimed herein.

[0058] Computing device 1500 includes a processor 1510, a memory 1520, a storage device 1530, a high-speed interface / controller 1540 connected to memory 1520 and high-speed expansion port 1550, and a low-speed interface / controller 1560 connected to low-speed bus 1570 and storage device 1530. Each of the components 1510, 1520, 1530, 1540, 1550, and 1560 may be interconnected using various buses and may be mounted on a common motherboard or in other manners as required. Processor 1510 can process instructions for execution within computing device 1500, including instructions stored on storage device 1530 for displaying graphical information for a graphical user interface (GUI) on an external input / output device such as a display 1580 coupled to memory 1520 or high-speed interface 1540. In other embodiments, multiple processors and / or multiple buses may be used, as necessary, along with multiple memories and memory types. Also, multiple computing devices 1500 may be connected, as needed, to each device that provides a portion of the required operations (e.g., a server bank, a group of blade servers, or a multi-processor system).

[0059] Memory 1520 stores information non - transiently within computing device 1500. Memory 1520 may be a computer - readable medium, a volatile memory unit, or a non - volatile memory unit. The non - transient memory 1520 is a temporary or permanent basis for use by computing device 1500 and may be a physical device used to store a program (e.g., a sequence of instructions) or data (e.g., state information of a program). Examples of non - volatile memory include, but are not limited to, flash memory and read - only memory (ROM) / programmable read - only memory (PROM) / erasable programmable read - only memory (EPROM) / electrically erasable programmable read - only memory (EEPROM) (used for normal firmware such as a boot program). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase - change memory (PCM), and disks or tapes.

[0060] Storage device 1530 can provide mass storage for computing device 1500. In some embodiments, storage device 1530 is a computer - readable medium. In various different embodiments, storage device 1530 may be an array of devices including a floppy disk device, a hard disk device, an optical disk device, or a tape device, flash memory or other similar solid - state memory devices, or a storage area network or other configured devices. In additional embodiments, a computer program product is specifically incorporated into an information medium. The computer program product includes instructions that, when executed, perform one or more of the methods as described above. The information medium is a computer or machine - readable medium such as memory 1520, storage device 1530, or memory on processor 1510.

[0061] The high-speed controller 1540 manages operations that consume the bandwidth of the computing device 1500, while the low-speed controller 1560 manages operations that consume low bandwidth. Such role assignments are merely examples. In some embodiments, the high-speed controller 1540 is coupled to the memory 1520, the display 1580 (e.g., through a graphics processor or accelerator), and the high-speed expansion port 1550 (although not shown here, various expansion cards may be applied). In some embodiments, the low-speed controller 1560 is coupled to the storage device 1530 and the low-speed expansion port 1590. The low-speed expansion port 1590 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be coupled to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router (e.g., through a network adapter).

[0062] As shown in the figure, the computing device 1500 may be implemented in several different forms. For example, it may be implemented as a standard server 1500a or a group of such servers 1500a, as a laptop computer 1500b, or as part of a rack server system 1500c.

[0063] The various embodiments of the systems and techniques described herein can be realized in digital electrical and / or optical circuits, integrated circuits (especially ASICs (application-specific integrated circuits)), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include embodiments in one or more computer programs. It may be executable and / or interpretable in a programmable system including at least one programmable processor, which may be special-purpose or general-purpose, and may be combined to receive data and instructions from a storage system, at least one input device, and at least one output device and to transmit data and instructions.

[0064] (Also known as programs, software, software applications, or code,) these computer programs include machine language instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide machine language instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine language instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine language instructions and / or data to a programmable processor.

[0065] The processes and logical flows described in this specification can be performed by one or more programmable processors, and can also be referred to as data processing hardware, and can execute one or more computer programs to perform functions by computing input data and generating output. The processes and logical flows can also be performed by special-purpose logic circuits (e.g., FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits)). Processors suitable for the execution of computer programs include, by way of example, both general-purpose and special-purpose microprocessors, and one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random access memory or both. Indispensable elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, and is operatively coupled to receive data from or transmit data to these, or both. However, a computer may not have such devices. Computer-readable media suitable for storing the instructions and data of a computer program include media and memory devices of all forms, including non-volatile memory, (by way of example semiconductor memory devices (e.g., EPROM, EEPROM) and flash memory devices, magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, CDROM and DVDROM disks). Processors and memories can also be supplemented by, and incorporated in, special-purpose logic circuits.

[0066] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer comprising a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user, and optionally, a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer). Other types of devices may be used to provide interaction with the user as well. For example, the feedback provided to the user may be any form of sensory feedback such as visual feedback, auditory feedback, tactile feedback, etc., and the input from the user can be received in any form including acoustic, voice, or tactile input. Further, the computer can act on the user by sending documents to and receiving documents from the devices used by the user, e.g., by sending a web page to a web browser on a user client device in response to a request received from the web browser.

[0067] Some embodiments have been described. Nevertheless, it is understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method (1400) executed by data processing hardware (122), the method comprising causing the data processing hardware (122) to: obtain a first teacher dataset (201) including a plurality of unannotated multi-dimensional medical images (202); execute a self-supervised mask image modeling (MIM) training process (200) to pre-train an image encoder (150) with the first teacher dataset (201); obtain a second teacher dataset (203) including a plurality of annotated multi-dimensional medical images (204), each of the annotated multi-dimensional medical images (204) including a plurality of image voxels (206) paired with corresponding ground truth labels (208) indicating the class to which the corresponding image voxels (206) belong; execute a supervised training process (160) to train an image analysis model (170) with the second teacher dataset (203), causing the image analysis model (170) to learn a method for predicting the ground truth label (208) corresponding to each of the plurality of image voxels (206) of each of the annotated multi-dimensional medical images (204); perform operations including: wherein the image analysis model (170) incorporates the pre-trained image encoder (150); Method (1400).

2. The step of executing the self-supervised MIM training process (200) to pre-train the image encoder (150) comprises, for each of the unannotated multi-dimensional medical images (202) corresponding to the first teacher dataset (201): using an image tokenizer (230) configured to receive the corresponding unannotated multi-dimensional medical image (202) as input and generate a sequence of discrete visual tokens (240) characterizing the corresponding unannotated multi-dimensional medical image (202); dividing the corresponding unannotated multi-dimensional medical image (202) into a plurality of image patches (210); randomly masking a portion of the image patches (210) divided from the corresponding unannotated multi-dimensional medical image (202), for each of the masked image patches (210M), Generating an encoded hidden representation (225) for the masked image patch (210M) using the image encoder (150); Generating a corresponding predicted token (275) based on the encoded hidden representation (225) using a decoder (250); including the steps of; Determining a training loss based on the predicted token (275) generated for the masked image patch (210M) and the corresponding visual token (240) from the series of discrete visual tokens (240) that match the masked image patch (210M); Updating the parameters of the image encoder based on the training loss; including; The method (1400) according to claim 1.

3. The image encoder (150) includes a plurality of multi-head attention layers, The decoder (250) includes a plurality of multi-head attention layers, The method (1400) according to claim 2.

4. The step of randomly masking a part of the image patch (210) uses one of a central region masking strategy, a per-block masking strategy, or a uniform random masking strategy that uses various masked patch sizes and masking ratios to randomly mask a part of the image patch (210), The method (1400) according to claim 2 or 3.

5. The number of visual tokens (240) in the series of discrete visual tokens (240) is equal to the number of image patches (210) in the plurality of image patches (210), The method (1400) according to any one of claims 2 to 4.

6. The step of pre-training the image encoder (150) by executing the self-supervised MIM training process (200) for each of the unannotated multi-dimensional medical images (202) corresponding to the first teacher dataset (201), Dividing the corresponding unannotated multi-dimensional medical image (202) into a plurality of image patches (210), where each image patch (210) is represented by a set of corresponding raw voxel values (270); Randomly masking a part of the image patches (210) divided from the corresponding unannotated multi-dimensional medical image (202), For each masked image patch (210M), generating an encoded hidden representation (225) for the masked image patch (210M) using the image encoder (150); generating, based on the encoded hidden representation (225), predicted voxel values (270) for the masked image patch (210M) using a prediction head (260); including the steps of; determining a training loss based on a set of the predicted voxel values (270) generated for the masked image patch (210M) and the corresponding raw voxel values (270) representing the masked image patch (210M); updating parameters of the image encoder (150) based on the training loss; including the method (1400) according to any one of claims 1 to 5.

7. The image encoder (150) includes a plurality of multi-head attention layers, The prediction head (260) includes a single linear layer prediction head (260) and is configured to generate the predicted voxel values (270) from the encoded hidden representation (225) without using a decoder (250). The method (1400) according to claim 6.

8. The step of randomly masking a part of the image patch (210) uses one of a central region masking strategy, a per-block masking strategy, or a uniform random masking strategy that uses various masked patch sizes and masking ratios to randomly mask a part of the image patch (210). The method (1400) according to claim 6 or 7.

9. The image analysis model (170) includes a tumor segmentation model. The method (1400) according to any one of claims 1 to 8.

10. The image analysis model (170) includes a multi-organ segmentation model. The method (1400) according to any one of claims 1 to 8.

11. A system (100) comprising: data processing hardware (122); memory hardware (124) that communicates with the data processing hardware (122) and stores instructions that, when executed on the data processing hardware (122), cause the data processing hardware (122) to perform operations; including wherein the operations are Obtaining a first teacher dataset (201) including a plurality of unannotated multi-dimensional medical images (202); Performing a self-supervised mask image modeling (MIM) training process (200) to pre-train an image encoder (150) with the first teacher dataset (201); Obtaining a second teacher dataset (203) including a plurality of annotated multi-dimensional medical images (204), wherein each of the annotated multi-dimensional medical images (204) includes a plurality of image voxels (206) paired with corresponding ground truth labels (208) indicating the class to which the corresponding image voxels (206) belong; Performing a supervised training process (160) to train an image analysis model (170) with the second teacher dataset (203), causing the image analysis model (170) to learn a method for predicting ground truth labels (208) corresponding to the plurality of image voxels (206) of each of the annotated multi-dimensional medical images (204); Including; The image analysis model (170) incorporates the pre-trained image encoder (150); System (100).

12. The step of performing the self-supervised MIM training process (200) to pre-train the image encoder (150) for each of the unannotated multi-dimensional medical images (202) corresponding to the first teacher dataset (201); Using an image tokenizer (230) configured to receive the corresponding unannotated multi-dimensional medical image (202) as an input, generating a series of discrete visual tokens (240) characterizing the corresponding unannotated multi-dimensional medical image (202); Dividing the corresponding unannotated multi-dimensional medical image (202) into a plurality of image patches (210); Randomly masking a part of the image patches (210) divided from the corresponding unannotated multi-dimensional medical image (202), For each of the masked image patches (210M), Generating an encoded hidden representation (225) for the masked image patch (210M) using the image encoder (150); Based on the encoded hidden representation (225), generating a corresponding predicted token (275) using a decoder (250); including the step of; determining a training loss based on the predicted token (275) generated for the masked image patch (210M) and the corresponding visual token (240) from the series of discrete visual tokens (240) that match the masked image patch (210M); updating the parameters of the image encoder based on the training loss; including; the system (100) according to claim 11.

13. The image encoder (150) includes a plurality of multi-head attention layers, The decoder (250) includes a plurality of multi-head attention layers. the system (100) according to claim 12.

14. The step of randomly masking a part of the image patch (210) uses one of a central region masking strategy, a per-block masking strategy, or a uniform random masking strategy that uses various masked patch sizes and masking ratios to randomly mask a part of the image patch (210), the system (100) according to claim 12 or 13.

15. The number of visual tokens (240) in a series of discrete visual tokens (240) is equal to the number of image patches (210) in a plurality of image patches (210). the system (100) according to any one of claims 12 to 14.

16. The step of pre-training the image encoder (150) by executing the self-supervised MIM training process (200) is for each of the unannotated multi-dimensional medical images (202) corresponding to the first teacher dataset (201), the step of dividing the corresponding unannotated multi-dimensional medical image (202) into a plurality of image patches (210), where each image patch (210) is represented by a set of corresponding raw voxel values (270); the step of randomly masking a part of the image patches (210) divided from the corresponding unannotated multi-dimensional medical image (202), for each of the masked image patches (210M), Generating an encoded latent representation (225) for the masked image patch (210M) using the image encoder (150); Generating predicted voxel values (270) for the masked image patch (210M) using a prediction head (260) based on the encoded latent representation (225); including the steps of; Determining a training loss based on the predicted voxel values (270) generated for the masked image patch (210M) and the set of corresponding raw voxel values (270) representing the masked image patch (210M); Updating the parameters of the image encoder (150) based on the training loss; including; The system (100) according to any one of claims 11 to 15.

17. The image encoder (150) includes a plurality of multi-head attention layers, The prediction head (260) includes a single linear layer prediction head (260) and is configured to generate the predicted voxel values (270) from the encoded latent representation (225) without using a decoder (250). The system (100) according to claim 16.

18. The step of randomly masking a part of the image patch (210) uses one of a central region masking strategy, a per-block masking strategy, or a uniform random masking strategy that uses various masked patch sizes and masking ratios, and includes randomly masking a part of the image patch (210). The system (100) according to claim 16 or 17.

19. The image analysis model (170) includes a tumor segmentation model. The system (100) according to any one of claims 11 to 18.

20. The image analysis model (170) includes a multi-organ segmentation model. The system (100) according to any one of claims 11 to 18.

Images