Medical imaging analysis using self-supervised learning

Abstract

A method (1400) comprises: obtaining a first training data set (201) comprising an unlabeled multi-dimensional medical image (202); and performing a self-supervised masked image modeling (MIM) training process (200) to pre-train an image encoder (150) on the first training data set. The method further includes obtaining a second training data set (203) including the annotated multi-dimensional medical image (204). Here, each tagged multi-dimensional medical image comprises a plurality of image voxels (206), each paired with a corresponding true value tag (208) indicating a category to which the corresponding image voxel belongs. The method also includes performing a supervised training process (160) to train an image analysis model (170) on the second training dataset to teach the image analysis model to learn how to predict a corresponding true value tag for the plurality of image voxels of each tagged multi-dimensional medical image. The image analysis model is combined with a pre-trained image encoder.

Description

Technical Field

[0001] The present disclosure relates to medical imaging analysis using self-supervised learning. Background Art

[0002] Multidimensional medical images, such as three-dimensional (3D) medical images, provide rich images of a patient's internal body to aid in medical analysis, diagnosis, or treatment of the patient. Such medical images can be generated using different modalities, including, for example, computed tomography (CT) or magnetic resonance imaging (MRI). Summary of the invention

[0003] One aspect of the present disclosure provides a computer-implemented method that is executed on data processing hardware to cause the data processing hardware to perform operations, the operations comprising: obtaining a first training data set, the first training data set comprising a plurality of unlabeled multidimensional medical images; and performing a self-supervised masked image modeling (MIM) training process to pre-train an image encoder on the first training data set. The operations also include obtaining a second training data set, the second training data set comprising a plurality of labeled multidimensional medical images. Here, each labeled multidimensional medical image comprises a plurality of image voxels, each of the plurality of image voxels being paired with a corresponding true value label indicating a category to which the corresponding image voxel belongs. The operations also include performing a supervised training process to train an image analysis model on the second training data set to teach the image analysis model to learn how to predict corresponding true value labels for the plurality of image voxels of each labeled multidimensional medical image. The image analysis model is combined with a pre-trained image encoder.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, for each corresponding unlabeled multidimensional medical image in the first training data set, performing the self-supervised MIM training process to pre-train the image encoder includes: using an image tokenizer configured to receive the corresponding unlabeled multidimensional medical image as input to generate a discrete visual token sequence characterizing the corresponding unlabeled multidimensional medical image; dividing the corresponding unlabeled multidimensional medical image into a plurality of image blocks; and randomly masking a portion of the image block divided from the corresponding unlabeled multidimensional medical image. For each masked image block, the operation also includes: using the image encoder to generate an encoded hidden representation for the masked image block; and based on the encoded hidden representation, using a decoder to generate a corresponding predicted token. Here, the operation also includes: determining a training loss based on the predicted token generated for the masked image block and a corresponding visual token from the discrete visual token sequence aligned with the masked image block; and updating parameters of the image encoder based on the training loss. In these implementations, the image encoder may include multiple multi-headed attention layers, and the decoder may include multiple multi-headed attention layers. Additionally or alternatively, randomly masking the portion of the image block comprises randomly masking the portion of the image block using one of: a center area masking strategy, a block-by-block masking strategy, or a uniform random masking strategy using different masking block sizes and masking rates. The number of visual tokens in the discrete visual token sequence may be equal to the number of image blocks in the plurality of image blocks.

[0005] In some examples, for each corresponding unlabeled multidimensional medical image in the first training data set, performing the self-supervised MIM training process to pre-train the image encoder includes: dividing the corresponding unlabeled multidimensional medical image into a plurality of image blocks, each image block represented by a corresponding set of original voxel values; and randomly masking a portion of the image block divided from the corresponding unlabeled multidimensional medical image. For each masked image block, the operation also includes: generating a coded hidden representation for the masked image block using the image encoder; and generating a predicted voxel value for the masked image block using a prediction head based on the coded hidden representation. Here, the operation also includes: determining a training loss based on the predicted voxel value generated for the masked image block and the corresponding set of original voxel values ​​representing the masked image block; and updating the parameters of the image encoder based on the training loss. In these examples, the image encoder may include a plurality of multi-head attention layers, and the prediction head may include a single linear layer prediction head and is configured to generate the predicted voxel value from the coded hidden representation without using a decoder. Additionally or alternatively, randomly masking the portion of the image block includes randomly masking the portion of the image block using one of the following: a central region masking strategy, a block-by-block masking strategy, or a uniform random masking strategy using different masking block sizes and masking rates. In some implementations, the image analysis model includes a tumor segmentation model. In some examples, the image analysis model includes a multi-organ segmentation model.

[0006] Another aspect of the present disclosure provides a system comprising data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations, the operations comprising: obtaining a first training data set comprising a plurality of unlabeled multidimensional medical images; and performing a self-supervised masked image modeling (MIM) training process to pre-train an image encoder on the first training data set. The operations also include obtaining a second training data set, the second training data set comprising a plurality of labeled multidimensional medical images. Here, each labeled multidimensional medical image comprises a plurality of image voxels, each of the plurality of image voxels being paired with a corresponding true value label indicating a category to which the corresponding image voxel belongs. The operations also include performing a supervised training process to train an image analysis model on the second training data set to teach the image analysis model to learn how to predict corresponding true value labels for the plurality of image voxels of each labeled multidimensional medical image. The image analysis model is combined with a pre-trained image encoder.

[0007] This aspect may include one or more of the following optional features. In some implementations, for each corresponding unlabeled multidimensional medical image in the first training data set, performing the self-supervised MIM training process to pre-train the image encoder includes: using an image tokenizer configured to receive the corresponding unlabeled multidimensional medical image as input to generate a discrete visual token sequence characterizing the corresponding unlabeled multidimensional medical image; dividing the corresponding unlabeled multidimensional medical image into a plurality of image blocks; and randomly masking a portion of the image block divided from the corresponding unlabeled multidimensional medical image. For each masked image block, the operation also includes: using the image encoder to generate an encoded hidden representation for the masked image block; and based on the encoded hidden representation, using a decoder to generate a corresponding predicted token. Here, the operation also includes: determining a training loss based on the predicted token generated for the masked image block and a corresponding visual token from the discrete visual token sequence aligned with the masked image block; and updating parameters of the image encoder based on the training loss. In these implementations, the image encoder may include multiple multi-headed attention layers, and the decoder may include multiple multi-headed attention layers. Additionally or alternatively, randomly masking the portion of the image block comprises randomly masking the portion of the image block using one of: a center area masking strategy, a block-by-block masking strategy, or a uniform random masking strategy using different masking block sizes and masking rates. The number of visual tokens in the discrete visual token sequence may be equal to the number of image blocks in the plurality of image blocks.

[0008] In some examples, for each corresponding unlabeled multidimensional medical image in the first training data set, performing the self-supervised MIM training process to pre-train the image encoder includes: dividing the corresponding unlabeled multidimensional medical image into a plurality of image blocks, each image block represented by a corresponding set of original voxel values; and randomly masking a portion of the image block divided from the corresponding unlabeled multidimensional medical image. For each masked image block, the operation also includes: generating a coded hidden representation for the masked image block using the image encoder; and generating a predicted voxel value for the masked image block using a prediction head based on the coded hidden representation. Here, the operation also includes: determining a training loss based on the predicted voxel value generated for the masked image block and the corresponding set of original voxel values ​​representing the masked image block; and updating the parameters of the image encoder based on the training loss. In these examples, the image encoder may include a plurality of multi-head attention layers, and the prediction head may include a single linear layer prediction head and is configured to generate the predicted voxel value from the coded hidden representation without using a decoder. Additionally or alternatively, randomly masking the portion of the image block includes randomly masking the portion of the image block using one of the following: a central region masking strategy, a block-by-block masking strategy, or a uniform random masking strategy using different masking block sizes and masking rates. In some implementations, the image analysis model includes a tumor segmentation model. In some examples, the image analysis model includes a multi-organ segmentation model.

[0009] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram of a system for pre-training an image encoder using self-supervised masked image modeling (MIM) and training an image analysis model incorporating the pre-trained image encoder.

[0011] Figure 2A and Figure 2B For pre-training Figure 1 Schematic diagram of the self-supervised MIM training process for an image encoder.

[0012] Figure 3 Example input, masked, and reconstructed 3D CT images using a pre-trained image encoder with a simple MIM architecture are shown.

[0013] Figure 4 Example input, masked, and reconstructed 3D CT images using a pre-trained image encoder with a masked autoencoder (MAE) architecture are shown.

[0014] Figure 5 is a table showing the DICE scores for multi-organ segmentation images using the image analysis model.

[0015] Figure 6 Is a list of training Figure 1 Table of supplementary baseline settings for the supervised training process of image analysis models.

[0016] Figure 7 Is a list of training Figure 1 Table of supplementary baseline settings for the supervised training process of image analysis models.

[0017] Figure 8 is listed for Figure 1 Table of the pre-training settings for the self-supervised MIM training process.

[0018] Fig. 9 is a table defining the results of using a machine learning model on brain tumor segmentation images after pre-training using the Bra TS training dataset.

[0019] Fig.10 A plot depicting how self-supervised MIM training of an image encoder advances downstream supervised fine-tuning.

[0020] Fig.11 is a table depicting an ablation study applying different masking block sizes and masking ratios on a multi-organ segmentation task.

[0021] Fig.12 is a table depicting an ablation study applying different masking block sizes and masking ratios on the brain tumor segmentation task.

[0022] Fig.13 is a table depicting the results of pre-training an image encoder using a fixed block size and a fixed masking rate.

[0023] Fig.14 is a flow diagram of an example arrangement of operations for training an image analysis model to perform vision tasks on multi-dimensional medical images.

[0024] Fig.15 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0025] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION

[0026] Computer vision analysis has witnessed a paradigm shift from using Convolutional Neural Networks (CNNs) to using multi-head attention-based architectures. The present disclosure relates to using Transformer-based architectures with self-attention as one type of multi-head attention-based architecture by way of example, however, the present disclosure may employ other types of multi-head attention-based architectures to enhance multi-dimensional input images. In general, Transformer-based architectures (i.e., visual transformers) segment multi-dimensional input images into blocks and create block embeddings as input to Transformer-based models for various visual tasks including image classification, object detection, and image segmentation.

[0027] Three-dimensional (3D) medical imaging techniques such as computed tomography (CT) or magnetic resonance imaging (MRI) are widely used to diagnose and treat a wide variety of diseases. In general, 3D medical volume images can help improve the speed and accuracy of diagnosing patient conditions. For example, correctly and quickly finding and measuring tumor lesions from MRI or CT scans may be crucial for disease prevention, early detection, and treatment plan optimization, and also inspire the development of more successful clinical applications, thereby ultimately improving the lives of patients. Basic tasks performed for medical image analysis include 3D image segmentation. Another basic task performed for medical image analysis includes image classification. The image classification task classifies the input image into different categories. In general, 3D image segmentation (also referred to as "3D semantic segmentation") aims to predict the corresponding category for each voxel of the volume input image, thereby classifying one or more specific objects, and separating each specific object from each other by covering the corresponding segmentation mask on the specific object. 3D image segmentation has the potential to reduce the daily workload of radiologists by automating or assisting image interpretation workflows to ultimately improve clinical care and patient outcomes. Example 3D image segmentation tasks may include multi-organ segmentation performed as a 13-class segmentation task with a single channel input and brain tumor segmentation performed as a three-class segmentation task with a four-channel input.

[0028] Training robust Transformer-based image analysis models requires more annotated training data to surpass the performance of conventional CNNs. However, the high cost of obtaining expert annotations of 3D medical volume images in specific fields often hinders attempts to use deep learning methods for 3D medical image analysis to make progress in clinical outcomes. In short, large-scale annotation of 3D medical images by radiologists is limited, expensive, and time-consuming to produce. Another limiting factor in 3D medical image processing is the huge amount of data associated with 3D medical images, which is driven by the increased dimensionality and resolution of 3D images, resulting in significant processing complexity. Therefore, it may be extremely difficult to effectively integrate radiological endpoint information with other biomarker data for other downstream tasks in clinical study design (such as tumor burden assessment and overall survival prediction).

[0029] Transfer learning is the use of a model trained from one context in a different context. Transfer learning from natural images can be used in medical image analysis regardless of differences in image statistics, scale, and task-related properties. Transfer learning from, for example, ImageNet can accelerate convergence on medical images, which can be useful when medical image training data is limited. Transfer learning using domain-specific data can also help address the domain difference problem. For example, improved performance can be achieved after pre-training on labeled data from the same domain. However, this strategy may often be impractical for various medical scenarios that require labeled data that is expensive and time-consuming to collect. Self-supervised learning provides a viable alternative that allows the exploitation of unlabeled / unannotated medical data.

[0030] Self-supervised learning is a training technique that focuses on learning representations from unlabeled data so that low-capacity classifiers can use various embeddings to achieve high accuracy. Contrastive learning is another example of a self-supervised learning strategy. Contrastive learning models image similarities and dissimilarities (or just similarities) between two or more views, where data augmentation is crucial for contrast and related methods. Self-supervised learning can be used in the medical field, such as domain-specific pre-tasks or customizing contrastive learning for medical data. A range of self-supervised learning strategies can be applied to 3D medical imaging. For example, a model pre-trained on the ImageNet dataset can be applied to dermatology image classification. In another example, image inpainting can be combined with contrastive learning for medical image segmentation.

[0031] Masked image modeling methods typically mask out a portion of the input image or encoded image tokens and encourage the model to recreate the masked area. Some existing MIM models adopt an encoder-decoder design followed by a projection head. The encoder facilitates the modeling of latent feature representations, while the decoder facilitates the resampling of the latent vector to the original image. The encoded or decoded embedding can then be aligned with the original signal at the masked area through a projection head. It is worth noting that the decoder component can be lightweight in design to minimize training time. A lightweight decoder can not only reduce computational complexity, but also increase the ability of the encoder to learn more general representations that the decoder can easily grasp, translate, and convey. The encoder can be used for fine-tuning. Coding techniques such as SimMIM can eliminate the entire decoder using a single projection layer.

[0032] For example, using a visual transformer (ViT), an image can be divided into regular non-overlapping blocks (e.g., a 96x 96x 96 3D volume can be divided into 216 blocks of smaller volumes of 16x 16x 16), which are generally considered as the basic processing units of the visual transformer. There are many random masking techniques, including but not limited to central region masking strategies, complex block-by-block masking strategies, and / or block-level uniform random masking methods using different masking block sizes and masking ratios.

[0033] In some examples, the image encoder includes a visual transformer (ViT) architecture, such as a vanilla ViT (e.g., ViT3D, Swin-Transformer 3D, and / or an Attention Visual Network (e.g., VAN3D)), which can inherit the attention mechanism to derive a hierarchical representation similar to, for example, Swin-Transformer 3D but instead uses pure convolutions. The image encoder can employ other types of multi-head attention layers, such as, but not limited to, Conformer layers, Performer layers, or lightweight convolutional layers.

[0034] Implementations herein involve performing a self-supervised masked image modeling (MIM) training process to pre-train an image encoder on a plurality of unlabeled (e.g., unlabeled) multidimensional medical images. As used herein, a multidimensional image is referred to as a 3D medical image, but the present disclosure may not be limited thereto and may also include a 4D medical image. A 3D medical image may include a volumetric slice from a CT or MRI scan of an internal (or external) body region of a patient. The image encoder includes a plurality of multi-head attention layers. For example, the image encoder may include a Transformer-based architecture with self-attention, the architecture employing a stack of Transformer layers. As will become clear, the image encoder is responsible for modeling a latent feature representation of a masked image block, which may then be used to predict an original image signal in a region associated with the masked image block. An image encoder pre-trained on an unlabeled 3D medical image via a self-supervised MIM training process is adaptable to a wide range of downstream visual tasks, such as 3D image segmentation and image classification.

[0035] The pre-trained image encoder can be integrated into the image analysis model and fine-tuned using the annotated multidimensional medical images to perform a specific downstream visual task. The annotated multidimensional medical images used to fine-tune the pre-trained image encoder and ultimately train the image analysis model to perform a specific visual task can each include a plurality of image voxels, each of which is paired with a corresponding true value label indicating the category to which the corresponding image voxel belongs. In this way, the implementation of the present disclosure further involves performing a supervised training process to train an image segmentation model on a plurality of annotated multidimensional medical images to teach the image segmentation model to learn how to predict the corresponding true value label for the plurality of image voxels of each annotated multidimensional medical image, wherein the image segmentation model includes a pre-trained image encoder, which is initialized on an unlabeled multidimensional image via a self-supervised MIM training process and fine-tuned on the annotated multidimensional image via a supervised training process. In some examples, the trained image analysis model includes an image segmentation model for performing 3D image segmentation tasks (such as multi-organ segmentation or tumor segmentation performed on 3D image slices divided from MRI or CT scans of internal body regions). As described in more detail below, a trained image analysis model can receive as input a plurality of image patches partitioned from a multidimensional medical image (i.e., a volume slice from an MRI or CT scan), generate an enhanced medical image based on features extracted from the multidimensional medical image, and perform image segmentation or image classification on the enhanced image. In an image segmentation scenario, a trained image analysis model can be trained to classify one or more specific objects (e.g., a tumor or an organ) in an enhanced image, and separate each specific object from one another by enhancing the enhanced image to include a corresponding segmentation mask covering the specific object. As used herein, enhancing the enhanced image to include a segmentation mask includes enhancing image voxels in the enhanced image that represent respective object categories and / or define the boundaries of the respective object categories. Enhancing the image voxels can include changing the color of the image voxels, adjusting the intensity of the image voxels, or enhancing the image voxels in any suitable manner so that each classified object is distinguishable and identifiable within the enhanced image.

[0036] Figure 1 An example system 100 is shown for pre-training an image encoder 150 via a self-supervised training process 200 to learn how to generate an encoded feature representation 225 from an unlabeled 3D medical image 202 ( Figure 2A and Figure 2B) and fine-tuning the pre-trained image encoder 150 via a supervised training process 160 to perform downstream image tasks. Specifically, the pre-trained image encoder 150 can be adapted for use in an image analysis model 170 to perform a specific visual task by training the image analysis model 170 on annotated 3D medical images 204. The system 100 includes a computing system 120 having data processing hardware 122 and memory hardware 124, which communicates with the data processing hardware 122 and stores instructions that cause the data processing hardware 122 to perform operations. In some implementations, the first computing system 120, 120a performs a self-supervised training process 200 to pre-train the image encoder 150, and then performs a supervised training process 160 to train the image analysis model 170 in conjunction with the pre-trained image encoder 150 to perform downstream visual tasks on 3D medical images. In these implementations, after the image analysis model 170 is trained to perform the downstream vision task, the first computing system 120a can provide the trained image analysis model 170 to the second computing system 120, 120b. Here, the second computing system 120b can execute the image analysis model 170 to generate an enhanced 3D medical image 110, 110E from the original 3D medical image 110, 110R, and perform the downstream vision task on the enhanced 3D medical image 110E.

[0037] The first computing system 120a may include a distributed system (e.g., a cloud computing environment). The second computing system 120b may include a computing device (e.g., a desktop computer, a workstation, a laptop computer, a tablet computer, etc.) that downloads the image analysis model 170 from the first computing system 120a. In some other implementations, the first computing system 120a receives the original 3D medical image 110R from the second computing system 120b, and executes the image analysis model 170 to perform downstream visual tasks. In another implementation, the second computing system 120b receives the image encoder 150 pre-trained by the self-supervised training process 200 from the first computing system 120a, and performs a supervised training process 160 to fine-tune the pre-trained image encoder on the downstream visual task. In this scenario, the annotated MD image 204 can be processed locally on the second computing system 120b via the supervised training process 160, thereby preserving privacy and sensitive data.

[0038] The self-supervised training process 200 trains the image encoder 150 on a first training dataset 201 comprising a plurality of unlabeled multi-dimensional (MD) images 202. Specifically, and as described below with reference to Figure 2A and Figure 2BDescribed in more detail, the self-supervised training process includes a self-supervised masked image modeling (MIM) training process. Each unlabeled MD image 202 in the first training data set 201 may include an image slice divided from a CT scan or MRI scan of the patient's internal body. Therefore, the first training data set 201 may include a corpus of unlabeled MD medical images 202 related to image slices from CT scans and / or MRI scans of the internal bodies of multiple patients. In one example, the first training data set 201 includes unlabeled 3D CT scan images 202 obtained from the Cancer Imaging Archive-Covid 19 (TCIA-Covid19) public data set. Here, the unlabeled 3D CT scan images include 771 volumes of unenhanced chest CT scans collected from 661 Covid19 infected patients.

[0039] It is noteworthy that self-supervised MIM training as disclosed herein is particularly advantageous for 3D medical image modeling by significantly accelerating training convergence and improving downstream performance. For example, when compared with naive contrastive learning, when the pre-trained image encoder 150 is adapted and fine-tuned to perform downstream visual tasks, training convergence can save up to 1.40x training cost to achieve the same or higher dice score. Similarly, downstream execution of downstream visual tasks of image segmentation can achieve an improvement of more than 5 percent (5%) without any hyperparameter adjustment. Additionally, for patient prognosis, treatment sensitivity prediction, tissue segmentation, image classification and digital representation, downstream applications of image encoders pre-trained via self-supervised MIM training are faster and more cost-effective than transfer learning to specific downstream tasks. As will become clear, training the image encoder 150 via the self-supervised MIM training process 200 enables the use of high masking rates and relatively small block sizes to predict original voxel values. To simply reconstruct the original input 3D medical image 110R into the enhanced 3D medical image 110E, a lightweight decoder may be implemented to receive the encoded feature representation 225 output by the image encoder 150 and perform reconstruction of the image signal at increased speed and reduced computational and storage costs. During the supervised training process 160, the self-supervised MIM training is generalized on the original input 3D medical image 110R with different image resolutions and labeled data rates.

[0040] In general, MIM learning includes a learning task that includes masking a subset of an input signal (e.g., image block 210) and predicting the masked signal. In other words, MIM learning / training is a self-supervised learning technique that learns representations via masking damaged images. Masking can be presented as a noise type. Masked block predictions for self-supervised learning can predict lost voxels by patching large rectangular regions of the source region and grouping voxel values ​​into different clusters to classify unknown voxel values. Additionally, masked block predictions for self-supervised learning can be implemented by predicting the average color of an image.

[0041] After the image encoder 150 is pre-trained via the self-supervised training process 200, the supervised training process 160 trains the image analysis model 170 on a second training dataset 203 including a plurality of annotated MD medical images 204. The supervised training process 160 fine-tunes the pre-trained image encoder 150 integrated with the image analysis model 170 to teach the image analysis model 170 to perform downstream visual tasks such as image segmentation tasks or image classification tasks. Each annotated MD medical image 204 includes a plurality of image voxels 206, each of which is paired with a corresponding true value label 208 indicating the category to which the corresponding image voxel 206 belongs. It is worth noting that the unlabeled 3D images 202 in the first training dataset 201 used to pre-train the image encoder 150 may be associated with a different medical field than the annotated 3D images 204 in the annotated second training dataset 203. For example, the first dataset 201 may include chest CT scans, while the second dataset 203 may include abdominal CT scans or multi-modal MRI scans of brain tumors.

[0042] The image analysis model 170 may include a U-shaped encoder-decoder architecture including a method for generating hierarchical coding features 225 ( Figure 2A and Figure 2B ) of the image encoder 150 (used as a ViT-based encoder, SwinTransformer or VAN), and a decoder 152. The decoder 152 may include a UPerNet to perform an image segmentation task based on the encoded features 225 output from the image encoder 150. That is, during the self-supervised MIM training process 200 for pre-training the image encoder 150, a two-layer convolutional transpose may be used as a projection head 260 ( Figure 2A ), and the UPerNet decoder 152 can be implemented by the image analysis model 170 to be used with the pre-trained image encoder 150 for performing downstream image segmentation. In some examples, the image encoder 150 includes a masked audio encoder (MAE) using a stack of multi-head attention layers (see Figure 2A). For example, the MAE may include an 8-layer stack of Transformer blocks with 512 dimensions for use by the decoder 152. In other examples, the image encoder includes a simple masked imaging model (SimMIM) (see Figure 2B ), and a single linear layer is used as a projection head instead of a decoder.

[0043] In one example, the second training data set 203 includes annotated 3D CT scans obtained from the Beyond the CranialVault (BTCV) abdominal data set, which includes abdominal CT scans with 13 organ annotations made by manual interpreters obtained from 30 participants / patients under the supervision of a clinical radiologist. Each 3D CT scan in the BTCV abdominal data set is performed in the portal venous phase with contrast enhancement and includes 80 to 225 slices with 512x512 pixels and a slice thickness ranging from one to six millimeters (mm). During preprocessing, each annotated 3D image 204 can be resampled to an isotropic voxel spacing of 1.5-2.0. In this example, the supervised training process 160 trains the image analysis model 170 as a multi-organ segmentation model to perform 13 class segmentations with 1 channel output. Thus, the ground truth label 208 for each corresponding image voxel 206 in each annotated 3D image 204 may include one of 13 different categories, depending on which organ the corresponding image voxel 206 belongs to.

[0044] In another example, the second training data set 203 includes annotated 3D MRI scan images obtained from a public dataset of Brain Tumor Segmentation (BraTS), which includes multi-modal and multi-site MRI scans with true value labels 208 for corresponding image voxels 206, indicating areas of edema, non-enhancing core, and necrotic core. In this example, the supervised training process 160 trains the image analysis model 170 as a brain tumor segmentation model to perform 3-class segmentation with 4-channel input. The voxel spacing of the MRI image can be 1.0 x 1.0 x 1.0 mm3. The voxel intensity can be pre-processed by z-score normalization.

[0045] The self-supervised training process 200 may store the pre-trained image encoder 150 in a data storage 180 overlaid on the memory hardware 124 of the computing system 120. Likewise, the supervised training process 160 may store the trained image analysis model 170 in the data storage 180. The first computing system 120a and / or any number of second computing systems 120b may access / retrieve the pre-trained image encoder 150 and / or the trained image analysis model 170 to execute thereon.

[0046] During inference, the image analysis model 170 in conjunction with the pre-trained and fine-tuned image encoder 150 is executed on the second computing system 120b (or the first computing system 120a) to process and perform image analysis tasks on one or more raw input 3D medical images 110R. Notably, the image analysis tasks performed by the image analysis model 170 include downstream visual tasks (i.e., image segmentation or image classification) that the image analysis model 170 is trained to perform by the supervised training process 160. Each raw input 3D medical image 110R may correspond to a 3D image slice from a 3D CT scan or a 3D MRI scan of the patient's internal body. Alternatively, the raw input 3D medical image 110R may correspond to a 3D image of an external body region of the patient. Each raw input 3D medical image 110R may undergo initial image preprocessing 184 to divide the raw input 3D medical image 110R into a plurality of image blocks 210, 210a-n. Although nine (9) image patches are shown by way of example, the example is non-limiting and the pre-processing 184 may divide the image into any number of image patches 210. The image analysis model 170 may process the image patches 210 to generate an enhanced 3D medical image 110E and perform downstream visual tasks on the enhanced 3D medical image 110E. When the image analysis model 170 performs the downstream visual task of 3D image segmentation, the model 170 predicts a corresponding class for each voxel of the volume enhanced 3D medical image 110E, thereby classifying one or more specific objects (e.g., tumors, tissues, organs), and separates each specific object from another by defining a corresponding segmentation mask, thereby covering the voxels that classify each object. Example 3D image segmentation tasks may include multi-organ segmentation performed as a 13-class segmentation task with a single channel input and brain tumor segmentation performed as a three-class segmentation task with a four-channel input.

[0047] The image enhancer 360 may receive an enhanced 3D medical image 110E segmented to identify voxels representing various specific object categories, and generate a corresponding segmentation mask to be applied to at least a portion of the voxels representing the specific object categories. Thus, the image enhancer 360 may enhance the image voxels representing various object categories and / or defining the boundaries of the corresponding object categories in the enhanced image. The enhancement of the image voxels may include changing the color of the image voxels, adjusting the intensity of the image voxels, or enhancing the image voxels in any suitable manner so that each of the classified objects is distinguishable and identifiable within the enhanced image 110E. The segmentation mask may include a graphical feature applied to the enhanced image to convey the location of the various object categories identified in the enhanced image 110E. The image enhancer 360 may output an enhanced enhanced image 110A, the image depicting the segmentation mask conveying the segmentation results performed by the analysis model 170. The graphical user interface 360 ​​executed on the computing system 120 may display the enhanced image 110A on a screen in communication with the computing system 120. Additionally or alternatively, the enhanced image and / or the intensified image 110A may be provided as output to one or more further downstream tasks.

[0048] refer to Figure 2A and Figure 2B In some implementations, the self-supervised MIM training process 200 is performed on a model having a masked autoencoder (MAE) architecture ( Figure 2A ) or the simple MIM (SimMIM) architecture ( Figure 2B ) is pre-trained on an image encoder 150 of the unlabeled 3D medical image 202. For each unlabeled 3D medical image 202, the training process 200 first pre-processes the image 202 at a pre-processing stage 184 to divide the image 202 into a plurality of image blocks 210, 210a-n. Since the complete 3D image volume is generally difficult to load directly onto the data processing hardware (e.g., GPU) 122 of the computing system 120, the self-supervised MIM training process 200 can implement a sliding window training strategy, wherein the pre-processing divides the original 3D medical image 202 into a number of (e.g., 96x 96x 96) small 3D windows. By default, the pre-processing stage 184 can implement a block size of approximately 16. The pre-processing stage can downsample the image resolution of the unlabeled 3D medical image 202. For example, a 96x volume resolution can be downsampled to a 9x volume resolution when the image encoder 150 includes a ViT-based image encoder, or can be downsampled to a 3x volume resolution when the image encoder 150 includes a SwinTransformer or a VAN.

[0049] Figure 2AA MIM training process 200 is shown, which trains an image encoder 150 having a MAE architecture by randomly masking a portion of an image block 210 divided from a corresponding unlabeled MD medical image 202. The training process 200 further randomly masks the portion of the image block 210 by using one of the following: a central region masking strategy, a block-by-block masking strategy, or a uniform random masking strategy using different masking block sizes and masking rates. The training process further uses an image tokenizer 230 configured to receive the unlabeled MD medical image 202 as input to generate a sequence of discrete visual tokens 240 that characterize the corresponding unlabeled MD medical image 202. The number of visual tokens in the sequence of discrete visual tokens 240 can be equal to the number of image blocks 210 divided from the unlabeled MD medical image 202. The tokenizer 230 can map discrete image voxels from the medical image 202 into discrete visual tokens 240 according to a visual codebook, the visual codebook comprising a token vocabulary containing discrete token indices. Since the visual tokens 240 are discrete, the training process 200 is non-differentiable. In some examples, the tokenizer 230 is trained via an auto-encoding style reconstruction process, where images are tokenized into discrete visual tokens according to a learned vocabulary.

[0050] In the example shown, the self-supervised MIM training process 200 adds a position embedding 215 to the image block 210. The image encoder 150 receives each masked image block 210M, where each masked image block can be replaced by a special mask embedding [M]. The special masking token [M] can be randomly initialized to a learnable vector that is optimized to reveal the corresponding masked image block 210.

[0051] For each masked image block [M], the image encoder 150 is configured to generate a corresponding encoded feature representation 225 (also referred to as an encoded hidden representation 225), and the decoder 250 decodes the corresponding encoded feature representation 225 to predict a corresponding prediction token 275 as an output from the projection head 260. The purpose of the MIM training process 200 is to teach the image encoder 150 and the decoder 250 to learn how to predict the visual token 240 obtained from the original 3D image 202. Specifically, the training process 200 teaches the encoder 150 to generate an encoded feature representation 225 for the masked image block 210M for generating a prediction token 275 that matches the visual token 240 obtained from the original 3D image 202. Here, the training process 200 can determine the training loss based on the prediction token 275 generated for the masked image block 210M and the corresponding visual token from the sequence of discrete visual tokens 240 that is aligned with the masked image block 210M (i.e., using the position embedding 215). Thereafter, the training process 200 updates the parameters of the image encoder 150 (and optionally the decoder 250) based on the training loss.

[0052] The decoder may include multiple multi-head attention layers (e.g., Transformer layers). In some examples, the masked image block 210M is invisible to the encoder 150, so that only the decoder 250 knows the individual tokens. This approach can save computation and storage without interfering with training.

[0053] Figure 2B A self-supervised MIM training process 200 is shown, which trains an image encoder 150 having a SimMIM architecture by randomly masking a portion of an image block 210 partitioned from a corresponding unlabeled MD medical image 202. Each image block 210 can be represented by a corresponding set of original voxel values. The training process 200 further randomly masks the portion of the image block 210 by using one of the following: a central region masking strategy, a block-by-block masking strategy, or a uniform random masking strategy using different masking block sizes and masking rates.

[0054] In the example shown, the self-supervised MIM training process 200 adds a position embedding 215 to the image block 210. The image encoder 150 receives each masked image block 210M, where each masked image block can be replaced by a special mask embedding [M]. The special masking token [M] can be randomly initialized to a learnable vector that is optimized to reveal the corresponding masked image block 210.

[0055] For each masked image block 210M, the image encoder 150 is configured to generate a corresponding encoded feature representation 225, and the prediction head 260 generates a predicted voxel value 270 for the masked image block 210M. It is worth noting that the MIM training process 200 for pre-training the image encoder 150 with the SimMIM architecture omits the decoder, but implements the prediction head 260 to predict the original voxel value 270 for each masked image block 210M directly from the encoded feature representation 225 generated by the image encoder 225 for the corresponding masked image block 210M. The training process 200 can determine the training loss based on the predicted voxel value 270 generated for the masked image block and the corresponding set of original voxel values ​​representing the masked image block from the original unlabeled MD medical image 202.

[0056] The training loss can be based on the distance in voxel space between the restored / estimated original voxel value 270 and the original voxel from the corresponding set of original voxel values ​​representing the masked image block. The training loss can include an l1 or l2 loss function. It is worth noting that the training loss can be calculated only for the masked match 210M to prevent the encoder 150 from participating in self-reconstruction and potentially dominating the learning process and ultimately hindering knowledge learning. Thereafter, the training process 200 updates the parameters of the image encoder 150 (and optionally the decoder 250) based on the training loss. When preprocessing downsamples the resolution of the medical image 202, the projection head can transform the prediction token 275 to the original voxel space. Optionally, a two-layer convolutional transpose can upsample the compressed encoded feature representation 225 to the resolution of the original medical image 202.

[0057] Figure 3 Example input, masked, and reconstructed 3D CT scan images from the TCIA-COVID19 validation set using a pre-trained image encoder 150 reconstructed using SimMIM are shown. Because the original images are all 3D volumes, the reconstructed images are in slice form for illustration and ease of understanding purposes, with the index number representing the depth. For each triplet, the first or leftmost column shows the true value (e.g., the original image). The second or middle column shows the masked image. The third or rightmost column shows the machine learning model reconstructed using SimMIM. For Figure 5 For the images shown in , the ViT-Base backbone is applied to the encoder, the masking block size is about 16 (for all dimensions), and the masking ratio is about 75%.

[0058] Figure 4 Shows example input, masked, and reconstructed 3D CT scan images from the TCIA-COVID19 validation set applying a machine learning model using MAE reconstruction. Figure 3 , because the original images are all 3D volumes, the reconstructed images are presented in slices for illustration and ease of understanding purposes, where the index number represents the depth. For each triplet, the first or leftmost column shows the true value (e.g., the original image). The second or middle column shows the masked image. The third or rightmost column shows the machine learning model reconstructed using MAE. Figure 4 For the images shown in , the ViT-Large backbone is applied to the encoder, the masking blocks are about 16 (for all dimensions), and the masking ratio is about 75%.

[0059] Figure 5A table is depicted showing that the MIM approach can generally outperform the contrastive learning technique, as the pre-trained image encoder 150 with both the MAE architecture and the SimMIM architecture achieves an average dice score of about 0.752 to 0.758, while SimCLR achieves an average dice score of about 0.723, 4.5% lower. As used herein, the Dice score is used to evaluate the accuracy of the segmentation performed as a downstream visual task. For a given semantic category, Gi and Pi represent the true value and predicted value of each corresponding voxel i, respectively. The following equation can be used to define the Dice score:

[0060] Figure 6 Included is a table listing supplemental baseline settings for a supervised training process 160 that trains an image analysis model 170 on the BTCV dataset to perform multi-organ image segmentation. Figure 7 Included is a table listing supplemental baseline settings for a supervised training process 160 that trains an image analysis model 170 on the BraTS dataset to perform brain tumor segmentation. Figure 8 A table is included that lists the pre-training settings for a self-supervised MIM training process that uses 3D CT image volumes as unlabeled 3D medical images 202 .

[0062] Fig. 9 A table defining the results of using a machine learning model on brain tumor segmentation images after pre-training using the Bra TS training dataset as annotated MD medical images 204 is shown. Figure 8 The segmentation results for BraTS follow the same Figure 5 The average dice score of the masked image modeling method is slightly greater than 0.80, however, SimCLR obtains a dice value of 0.7739, which is higher than Figure 5 It is also worth noting that although the two MIM techniques are similar, SimMIM can achieve slightly better performance than MAE, as shown in Figure 2. Figure 5 and Fig. 9 One explanation for this is that an efficient decoder (even a lightweight decoder) is able to reconstruct the original image even if the encoder 150 does not obtain a generalizable representation, thus circularly weakening the motivation for the encoder 150 to learn a more effective representation 225. One goal of self-supervised MIM learning is to learn an efficient and generalizable representation of the data, rather than just self-convergence. In contrast, SimMIM adopts a lighter design by completely omitting the decoder, which pushes the encoder to perform more complex reconstruction and learning tasks.

[0063] The self-supervised MIM training process 200 improves the training speed while reducing the cost of pre-training the image encoder 150 on the first training dataset 201. Fig.10 A drawing depicting how the self-supervised MIM training process 200 advances the supervised training process 160 is shown. Here, the average dice score on the validation set is compared between the supervised baseline and different self-supervised MIM techniques using different masking rates across training steps. Masked image modeling pre-training can save training costs and produce better performance. The SimMIM-based architecture can achieve a 1.76x better dice score at 1.3k training steps. In addition, the MIM-based method can achieve a dice score of 0.7 with 1.4x less training time than the supervised baseline.

[0064] In some implementations, the model can be trained using self-supervised MIM using various masking block sizes and masking ratios. The results of applying machine learning models to 3D medical images using several MIM techniques and then fine-tuning the pre-trained image encoder to perform downstream image segmentation are summarized in Fig.11 and Fig.12 in the table. Fig.11 Includes a table depicting ablation studies of different masking block sizes and masking ratios on multi-organ segmentation. Fig.13 The resulting machine learning model 160 in has the default backbone of ViT-B applied as the pre-trained encoder 150. Additionally, the machine learning model 160 is trained via a supervised training process 160 using the BTCV training dataset. Fig.12 is a table depicting ablation studies of different masking block sizes and masking ratios on brain tumor segmentation. Likewise, the pre-training data includes the BraTS dataset itself, and ViT-B is applied as the encoder backbone in UNETR for segmentation fine-tuning.

[0065] Higher masking rates are non-trivial for self-supervised learning, which can continuously drive the model to build generalizable representations that can be effectively transferred to downstream tasks. For example, when the Fig.11 The block size in 16 is 0.7183, and Fig.12 The best dice scores on the multi-organ segmentation task and the brain tumor segmentation task were obtained when using a masking ratio of about 0.75 for block sizes 24 and 32. When used in combination with SimMIM, the high masking ratio combined with the small block size leads to relatively good performance. Fig.11 and Fig.12As shown, when the block size is equal to 16, the model can perform with dice scores of approximately 0.7249 and 0.8077, respectively. However, as the block size increases, the SimMIM method seems to be less sensitive to this masking rate. For example, when the block size is about 32, the model can obtain the highest dice score with a masking rate of approximately 0.15 (the minimum possible masking rate). Medical images are usually raw low-level signals with large spatial redundancy, and some lost blocks can be restored by directly copying nearby blocks, while requiring less comprehensive knowledge of the object and the surrounding environment. A single small masking block may not be able to adequately mask complex and interlaced structures or locations, but a large block size may be able to independently hide more important signals. Therefore, a high masking rate for a small block size may be more critical than a high masking rate for a larger block size.

[0066] In general, in supervised learning, more training data leads to improved performance. Fig.13 Included is a table depicting the Dice scores of an image analysis model 170 that combines an image encoder 150 pre-trained via a self-supervised MIM training process 200 and a hand with a MAE architecture ( Figure 2A ). The image encoder 150 can be pre-trained on a variety of different data sources with varying degrees of downsampling. The supervised training process 160 can train the image analysis model 170 on a multi-organ segmentation dataset with varying labeled data rates. The results of the table show that the model trained on more unlabeled 3D medical images 202 via the self-supervised MIM training process 200 outperforms the model trained on fewer unlabeled 3D medical images, for example, 0.7543 to 0.7184; 4.9% improvement and 0.7338 to 0.7018; 4.6% improvement). The advantage may be more pronounced at lower image resolutions, as 0.6818 is 5.6% higher than 0.6552 when only semi-labeled data is used for supervised training.

[0067] Fig.13 We also depict how different resolutions of unlabeled 3D medical images used for pre-training affect the performance of downstream image tasks. For example, a higher pre-training resolution can lead to better segmentation results because the image contains more fine-grained information. Here, different downsampling rates can be used to represent the degree to which the original signal is compressed across all dimensions of each volume. As can be seen from Fig.13 It is observed that pre-trained encoder models with higher resolutions (e.g., 1.5x, 1.5x, 2.0x) generally perform better than pre-trained models with lower resolutions (e.g., 2.0x, 2.0x, 2.0x). For example, a dice score of 0.7338 is 2.7% lower than the dice score of a model pre-trained with a higher resolution using the same data source and labeling rate.

[0068] Fig.14 14 is a flowchart of an example arrangement of operations of a method 1400 for training an image analysis model to perform an image analysis task on a multi-dimensional medical image. The data processing hardware 122 of the computing system 120 can execute instructions stored on the memory hardware 124 to perform operations. At operation 1402, the method 1400 includes obtaining a first training data set 201, the first training data set including a plurality of unlabeled multi-dimensional medical images 202. At operation 1404, the method 1400 includes performing a self-supervised masked image modeling (MIM) training process 200 to pre-train the image encoder 150 on the first training data set 201.

[0069] At operation 1406, the method 1400 includes obtaining a second training data set 203, the second training data set including a plurality of annotated multidimensional medical images 204. Here, each annotated multidimensional medical image 204 includes a plurality of image voxels 206, each of the plurality of image voxels being paired with a corresponding true value label 208 indicating a category to which the corresponding image voxel belongs. At operation 1408, the method 1400 includes performing a supervised training process 160 to train an image analysis model 170 on the second training data set 203 to teach the image analysis model 170 to learn how to predict the corresponding true value label 208 for the plurality of image voxels 206 of each annotated multidimensional medical image 204. Here, the image analysis model 170 is combined with a pre-trained image encoder 150. The supervised training process 160 fine-tunes the pre-trained image encoder 150 initialized via the self-supervised MIM training process 200.

[0070] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0071] Non-transitory memory can be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0072] Fig.15 1 is a schematic diagram of an example computing device 1500 that can be used to implement the systems and methods described in this document. Computing device 1500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not meant to limit the implementation of the inventions described and / or claimed in this document.

[0073] The computing device 1500 includes a processor 1510, a memory 1520, a storage device 1530, a high-speed interface / controller 1540 connected to the memory 1520 and a high-speed expansion port 1550, and a low-speed interface / controller 1560 connected to a low-speed bus 1570 and the storage device 1530. Each of the components 1510, 1520, 1530, 1540, 1550, and 1560 is interconnected using various buses and can be mounted on a common motherboard or otherwise installed where appropriate. The processor 1510 can process instructions for execution within the computing device 1500, including instructions stored in the memory 1520 or on the storage device 1530, to display graphical information for a graphical user interface (GUI) on an external input / output device (such as a display 1580 coupled to the high-speed interface 1540). In other implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple types of memories where appropriate. Furthermore, multiple computing devices 1500 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0074] Memory 1520 stores information non-temporarily within computing device 1500. Memory 1520 may be a computer-readable medium, one or more volatile memory units, or one or more non-volatile memory units. Non-temporary memory 1520 may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 1500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0075] Storage device 1530 can provide mass storage for computing device 1500. In some implementations, storage device 1530 is a computer-readable medium. In various implementations, storage device 1530 can be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a device array (including a device in a storage area network or other configuration). In another implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as a memory on memory 1520, storage device 1530, or processor 1510.

[0076] The high-speed controller 1540 manages bandwidth-intensive operations for the computing device 1500, while the low-speed controller 1560 manages less bandwidth-intensive operations. This allocation of responsibilities is exemplary only. In some implementations, the high-speed controller 1540 is coupled to the memory 1520, the display 1580 (e.g., through a graphics processor or accelerator), and to a high-speed expansion port 1550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 1560 is coupled to the storage device 1530 and the low-speed expansion port 1590. The low-speed expansion port 1590, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or, for example, through a network adapter to a networking device such as a switch or a router.

[0077] As shown in the figures, computing device 1500 can be implemented in many different forms. For example, it can be implemented as a standard server 1500a or multiple implementations in a group of such servers 1500a, as a laptop computer 1500b, or as part of a rack server system 1500c.

[0078] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These different implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0079] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural programming language and / or an object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., a disk, an optical disk, a memory, a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0080] The process and logic flow described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware) that execute one or more computer programs to perform functions by operating on input data and generating outputs. The process and logic flow can also be performed by a dedicated logic circuit system (e.g., FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)). By way of example, processors suitable for executing computer programs include general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer include a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data or be operatively coupled to the one or more large-capacity storage devices to receive data from it or transmit data to it or both. However, a computer does not need to have such a device. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated in a special purpose logic circuit system.

[0081] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), an LCD (liquid crystal display) monitor, or a touch screen) and optionally a keyboard and a pointing device (e.g., a mouse or trackball), wherein the display device is used to display information to the user, and the user can provide input to the computer through the keyboard and the pointing device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, voice input, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from the device used by the user (e.g., by sending a web page to a web browser in response to a request received from a web browser on a user's client device).

[0082] Many implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method (1400) executed on data processing hardware (122) to cause the data processing hardware (122) to perform operations comprising: Obtaining a first training data set (201), wherein the first training data set includes a plurality of unlabeled multidimensional medical images (202); performing a self-supervised masked image modeling (MIM) training process (200) to pre-train an image encoder (150) on the first training dataset (201); Obtaining a second training data set (203), the second training data set comprising a plurality of annotated multidimensional medical images (204), each annotated multidimensional medical image (204) comprising a plurality of image voxels (206), each of the plurality of image voxels being paired with a corresponding true value label (208) indicating a category to which the corresponding image voxel (206) belongs; as well as A supervised training process (160) is performed to train an image analysis model (170) on the second training dataset (203) to teach the image analysis model (170) to learn how to predict corresponding true value labels (208) for the multiple image voxels (206) of each annotated multidimensional medical image (204), the image analysis model (170) being combined with a pre-trained image encoder (150).

2. The method (1400) of claim 1, wherein: For each corresponding unlabeled multidimensional medical image (202) in the first training dataset (201), performing the self-supervised MIM training process (200) to pre-train the image encoder (150) comprises: generating a sequence of discrete visual tokens (240) characterizing the corresponding unlabeled multidimensional medical image (202) using an image tokenizer (230) configured to receive the corresponding unlabeled multidimensional medical image (202) as input; Dividing the corresponding unlabeled multidimensional medical image (202) into a plurality of image blocks (210); randomly masking a portion of the image block (210) divided from the corresponding unlabeled multidimensional medical image (202); For each masked image block (210M): generating an encoded hidden representation (225) for the masked image block (210M) using the image encoder (150); and Based on the encoded hidden representation (225), generating corresponding prediction tokens (275) using a decoder (250); determining a training loss based on the predicted token (275) generated for the masked image patch (210M) and a corresponding visual token (240) from the sequence of discrete visual tokens (240) aligned with the masked image patch (210M); and Parameters of the image encoder (150) are updated based on the training loss.

3. The method (1400) of claim 2, wherein: The image encoder (150) includes a plurality of multi-head attention layers; and The decoder (250) includes multiple multi-head attention layers.

4. The method (1400) according to claim 2 or 3, wherein: Randomly masking the portion of the image block (210) includes randomly masking the portion of the image block (210) using one of the following: a center area masking strategy, a block-by-block masking strategy, or a uniform random masking strategy using different masking block sizes and masking rates.

5. The method (1400) according to any one of claims 2-4, wherein: The number of visual tokens (240) in the sequence of discrete visual tokens (240) is equal to the number of image blocks (210) in the plurality of image blocks (210).

6. The method (1400) according to any one of claims 1 to 5, wherein: For each corresponding unlabeled multidimensional medical image (202) in the first training dataset (201), performing the self-supervised MIM training process (200) to pre-train the image encoder (150) comprises: Dividing the corresponding unlabeled multidimensional medical image (202) into a plurality of image blocks (210), each image block (210) being represented by a corresponding set of original voxel values ​​(270); randomly masking a portion of the image block (210) divided from the corresponding unlabeled multidimensional medical image (202); For each masked image block (210M): generating an encoded hidden representation (225) for the masked image block (210M) using the image encoder (150); and Based on the encoded hidden representation (225), using a prediction head (260) to generate a predicted voxel value (270) for the masked image block (210M); determining a training loss based on the predicted voxel values ​​(270) generated for the masked image block (210M) and the corresponding set of original voxel values ​​(270) representing the masked image block (210M); and Parameters of the image encoder (150) are updated based on the training loss.

7. The method (1400) of claim 6, wherein: The image encoder (150) includes a plurality of multi-head attention layers; and The prediction head (260) comprises a single linear layer prediction head (260) and is configured to generate the predicted voxel value (270) from the encoded hidden representation (225) without using a decoder (250).

8. The method (1400) according to claim 6 or 7, wherein: Randomly masking the portion of the image block (210) includes randomly masking the portion of the image block (210) using one of the following: a center area masking strategy, a block-by-block masking strategy, or a uniform random masking strategy using different masking block sizes and masking rates.

9. The method (1400) according to any one of claims 1-8, wherein: The image analysis model (170) includes a tumor segmentation model.

10. The method (1400) according to any one of claims 1-8, wherein: The image analysis model (170) includes a multi-organ segmentation model.

11. A system (100), comprising: Data processing hardware (122); as well as Memory hardware (124) in communication with the data processing hardware (122) and storing instructions that, when executed on the data processing hardware (122), cause the data processing hardware (122) to perform operations comprising: Obtaining a first training data set (201), wherein the first training data set includes a plurality of unlabeled multidimensional medical images (202); performing a self-supervised masked image modeling (MIM) training process (200) to pre-train an image encoder (150) on the first training dataset (201); obtaining a second training data set (203), the second training data set comprising a plurality of annotated multidimensional medical images (204), each annotated multidimensional medical image (204) comprising a plurality of image voxels (206), each of the plurality of image voxels being paired with a corresponding true value label (208) indicating a category to which the corresponding image voxel (206) belongs; and A supervised training process (160) is performed to train an image analysis model (170) on the second training dataset (203) to teach the image analysis model (170) to learn how to predict corresponding true value labels (208) for the multiple image voxels (206) of each annotated multidimensional medical image (204), the image analysis model (170) being combined with a pre-trained image encoder (150).

12. The system (100) according to claim 11, wherein: For each corresponding unlabeled multidimensional medical image (202) in the first training dataset (201), performing the self-supervised MIM training process (200) to pre-train the image encoder (150) comprises: generating a sequence of discrete visual tokens (240) characterizing the corresponding unlabeled multidimensional medical image (202) using an image tokenizer (230) configured to receive the corresponding unlabeled multidimensional medical image (202) as input; Dividing the corresponding unlabeled multidimensional medical image (202) into a plurality of image blocks (210); randomly masking a portion of the image block (210) divided from the corresponding unlabeled multidimensional medical image (202); For each masked image block (210M): generating an encoded hidden representation (225) for the masked image block (210M) using the image encoder (150); and Based on the encoded hidden representation (225), generating corresponding prediction tokens (275) using a decoder (250); determining a training loss based on the predicted token (275) generated for the masked image patch (210M) and a corresponding visual token (240) from the sequence of discrete visual tokens (240) aligned with the masked image patch (210M); and Parameters of the image encoder (150) are updated based on the training loss.

13. The system (100) of claim 12, wherein: The image encoder (150) includes a plurality of multi-head attention layers; and The decoder (250) includes multiple multi-head attention layers.

14. The system (100) according to claim 12 or 13, wherein: Randomly masking the portion of the image block (210) includes randomly masking the portion of the image block (210) using one of the following: a center area masking strategy, a block-by-block masking strategy, or a uniform random masking strategy using different masking block sizes and masking rates.

15. The system (100) according to any one of claims 12 to 14, wherein: The number of visual tokens (240) in the sequence of discrete visual tokens (240) is equal to the number of image blocks (210) in the plurality of image blocks (210).

16. The system (100) according to any one of claims 11 to 15, wherein: For each corresponding unlabeled multidimensional medical image (202) in the first training dataset (201), performing the self-supervised MIM training process (200) to pre-train the image encoder (150) comprises: Dividing the corresponding unlabeled multidimensional medical image (202) into a plurality of image blocks (210), each image block (210) being represented by a corresponding set of original voxel values ​​(270); randomly masking a portion of the image block (210) divided from the corresponding unlabeled multidimensional medical image (202); For each masked image block (210M): generating an encoded hidden representation (225) for the masked image block (210M) using the image encoder (150); and Based on the encoded hidden representation (225), using a prediction head (260) to generate a predicted voxel value (270) for the masked image block (210M); determining a training loss based on the predicted voxel values ​​(270) generated for the masked image block (210M) and the corresponding set of original voxel values ​​(270) representing the masked image block (210M); and Parameters of the image encoder (150) are updated based on the training loss.

17. The system (100) of claim 16, wherein: The image encoder (150) includes a plurality of multi-head attention layers; and The prediction head (260) comprises a single linear layer prediction head (260) and is configured to generate the predicted voxel value (270) from the encoded hidden representation (225) without using a decoder (250).

18. The system (100) according to claim 16 or 17, wherein: Randomly masking the portion of the image block (210) includes randomly masking the portion of the image block (210) using one of the following: a center area masking strategy, a block-by-block masking strategy, or a uniform random masking strategy using different masking block sizes and masking rates.

19. The system (100) according to any one of claims 11 to 18, wherein: The image analysis model (170) includes a tumor segmentation model.

20. The system (100) according to any one of claims 11 to 18, wherein: The image analysis model (170) includes a multi-organ segmentation model.

Images