Attention-based learning for digital histopathology analysis

Abstract

Embodiments of the present disclosure leverage hierarchical encoding and self- supervised learning (SSL) objectives to provide pretrained portions of a deep learning network for digital histopathology analysis. Some embodiments of the disclosure leverage different training methods for training different portions of the network. Some embodiments use a DINOv2 objective for pretraining a patch-level encoder and / or for pretraining a region- level encoder. Some embodiments use a contrastive learning objective such as simCLR to pretrain a patch-level encoder but use a different learning objective to pretrain a region-level encoder. These and other aspects of the present disclosure are more fully detailed herein.

Description

Attention-based Learning for Digital Histopathology Analysis CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 486,625, filed on February 23, 2023. The contents of that application are incorporated by reference herein.BACKGROUND

[0002] This disclosure relates generally to technology for making computerized inferences from patient medical images.SUMMARY

[0003] The amount of pixel data in atypical whole slide digital histopathology image (WSI) is significant. This poses challenges to effective and practical computerized analysis for making inferences from, performing regression analysis on, and segmenting WSIs.

[0004] Embodiments of the present disclosure leverage hierarchical encoding and selfsupervised learning (SSL) objectives to provide pretrained portions of a deep learning network for digital histopathology analysis. Some embodiments of the disclosure leverage different training methods for training different portions of the network. Some embodiments use a DIN0v2 objective for pretraining a patch-level encoder and / or for pretraining a regionlevel encoder. Some embodiments use a contrastive learning objective such as simCLRto pretrain a patch-level encoder but use a different learning objective to pretrain a region-level encoder. These and other aspects of the present disclosure are more fully detailed below.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] FIG. 1 illustrates a deep learning network for processing digital histopathologyimages in accordance with an embodiment of the disclosure.

[0006] FIG. 2 is a flow diagram illustrating a method for training a digital histopathology deep learning network such as, for example, the deep learning network of FIG. 1. The illustrated method is in accordance with an embodiment of the disclosure.

[0007] FIG. 3 illustrates alternative learning methods for each phase of training a hierarchical deep learning network such as, for example, the deep learning network of FIG.1.

[0008] FIG. 4 is a data table illustrating results of testing various embodiments of the present disclosure.

[0009] FIG. 5 shows an example of a computer system one or more of which may be used to implement one or more of the apparatuses, systems, and methods illustrated herein.

[0010] While the embodiments are described with reference to the above drawings, the drawings are intended to be illustrative, and other embodiments are consistent with the spirit, and within the scope, of the disclosure.DETAILED DESCRIPTION

[0011] The various embodiments now will be described more fully hereinafter with reference to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific examples of practicing the embodiments. This specification may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this specification will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Among other things, this specification may be embodied as methods or devices. Accordingly, any of the various embodiments herein may take the form of an entirely hardware embodiment, an entirely software embodiment or anembodiment combining software and hardware aspects. The following specification is, therefore, not to be taken in a limiting sense.

[0012] FIG. 1 illustrates a deep-learning network 1000 for processing digital histopathology images such as, for example, whole-slide images 11. In digital histopathology, a whole slide image of hematoxylin and eosin (H&E) stained tissue is generally referred to simply as a “whole slide image” or “WSI.”

[0013] In the example shown, the system is presumed to be fully trained and fine-tuned for end-use in generating predictions or otherwise making inferences relevant to a particular task (e.g., presence of a particular type of cancer based on pathology images of a tissue sample). In the present example, patch-level and a region-level encoder have been pretrained for more general tasks using a self-supervised learning (SSL) technique (also known in the art as an “objective.”). An example of hierarchical pretraining of vision transformer (ViT) encoders for WSIs is described by Chen et al. in “Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning” (2022) (arXiv:2206.02647). This paper is incorporated by reference herein in its entirety. The whole-slide level encoder has been trained, together with a classification network, with labelled data to fine tune for a particular prediction task.

[0014] Pre-processing block 110 receives WSI 11 (e.g., digital data corresponding to an image of H&E-stained tissue on a histopathology slide, the tissue corresponding to a sample taken from a patient) and conducts pre-processing including dividing images 11 into regions and patches with each region including a plurality of patches. In one, non-limiting example, a WSI is divided into regions, each region being 4480x4480 pixels in size. In this example, each region is divided into 400 patches with each patch being 224x224 pixels in size. The number of regions in a WSI will depend on the size of the WSI and can be expected to vary across a set of WSIs. It might be 4 regions, 8 regions, 10 regions, 16 regions, or some othernumber of regions.

[0015] WSIs are known to have various types of artifacts ranging from large amounts of background to pen marks, blurred areas, and bubbles. The attention mechanism in vision transformer (ViT) networks can automatically focus on salient regions and hence do not require an extremely fine-grained quality control (QC) output. In one embodiment, background and pen marks are detected from the WSIs.

[0016] In one embodiment, pre-processing block 110 uses HistoQC (https: / / github.com / choosehappy / HistoQC / wiki), an open-source quality control tool for digital histopathology, to extract tissue regions 4480x4480 in size from the WSIs. Each region is processed through the tool, and it sequentially produces metrics and tracks the amount of background in the whole slide images and the locations of pen marks. In once example, based on these metrics, pre-processing block 110 filters out patches containing artifacts and selects regions in the image with rich features.

[0017] Pre-processing block 110 outputs patch-level digital images 12. In one example, patch-level images 12 are 224x224 pixels in size. First pretrained encoder 120 processes each patch image 12 to generate a patch-level representation 21. In one example, patch-level representations 21 are vectors having 768 dimensions. Thus, in this example, the resulting representational data of each patch-level representation 21 is of size 1x768, which consumes significantly less memory than the 224x224 pixel data of a patch 12. Of course, one skilled in the art will appreciate that the dimension size will depend on the type of encoder used and its configuration for a particular application. Therefore, the dimensions given for this embodiment are by way of example only.

[0018] Patch-level representations 21 are used to represent each patch in a region, thereby providing region-level pseudo images 20. Images 20 are referred to as “pseudo” images (or representational images) because the patches of the region image are not pixel-spacerepresentations but rather are representations in a space whose dimensions are defined by first patch-level encoder 120. In one example, each region comprises 40 patches, and, therefore, region-level pseudo images 20 are 40x768 in size.

[0019] Region-level pseudo images 20 are processed by second pretrained encoder 130. Encoder 130 processes each region-level pseudo image 20 and generates a region-level representation 31. In one example, region-level representations 31 are vectors having 384 dimensions. Thus, in this example, the resulting representational data of each region-level representation 31 is of size 1x384, which consumes significantly less memory than the 4480x4480 pixel data of the corresponding region from a WSI 11. Of course, one skilled in the art will appreciate that the dimension size will depend on the type of region-level encoder used and its configuration for a particular application. Therefore, the dimensions given for this embodiment are by way of example only.

[0020] Region-level representations 31 are used to represent each region of a whole slide image (one representation 31 for each region), thereby providing a whole-slide-level pseudo 30 image comprising region-level representations 31. The number of regions may vary depending on the slide. But in one example, a WSI is divided into about 4 regions and thus a whole-slide pseudo image 30 might be 4x384 in size.

[0021] Region-level representations 31 corresponding to a whole-slide pseudo image 30 are processed by attention-based encoder 140. In one embodiment, attention-based encoder 140 is vision transformer (ViT) encoder. In one example, the ViT has 4 attention heads, 2 layers (i.e., depth = 2), and an output dimension of 384. Attention-based encoder 140 generates a whole-slide-level representation 41 (which, in the context of a ViT encoder, is the processed CLS token). Representation 41 is then processed by a classifier network 150 which in one example may include one or more typical feed-forward hidden layers (i.e., a multi-layer perceptron or “MLP”) followed by a softmax layer to obtain predictions (e.g., classprobabilities) classifying the tissue corresponding to the relevant WSI 11 processed by system 1000.

[0022] FIG. 2 is a flow diagram illustrating a method 2000 for training a digital histopathology deep learning network such as, for example, network 1000 of FIG. 1.

[0023] Step 201 conducts self-supervised learning on patch-level training images to obtain a pretrained patch level encoder. Preferably a large training set of histopathology images is used at step 201. Those images are preferably not task specific, but include a wide variety of tissue samples, e.g., from many diverse potential disease sites. In one example, over 11,677 H&E-stained whole slide images from TCGA public data are used for pretraining. In one example, the training slides come from 33 diverse potential tumor sites. In one example, approximate 33 million patches of 224x224 pixels are extracted from the WSI training set and are used at step 201.

[0024] After a pretrained patch-level encoder is obtained from step 201, step 202 uses the pretrained patch-level encoder to encode training image patches (pixel data) into patch-level representations. In one example, each patch-level representation is a vector representation comprising 768 dimensions. In one example, the training image patches encoded at step 202 are the same training patches used in step 201. In another example, the training image patches encoded at step 202 are different image patches from a different training set than that used to conduct pretraining at step 201.

[0025] Step 203 uses the patch-level representations to form region-level pseudo images, each pseudo image corresponding to a region of a WSI, the pseudo image comprising a plurality of patch representations of patches in the corresponding region. In one example, a region-level pseudo image comprises 400 patch-level representations.

[0026] Step 204 uses the region-level pseudo images to conduct self-supervised learning to pretrain a region-level encoder.

[0027] Once patch-level and region level encoders are pretrained, they can then be used to hierarchically generate whole-slide pseudo images from a labeled training set. These wholeslide pseudo images, comprising region-level representations, can then be used to train a whole-slide-level encoder, together with a classifier, for a specific prediction task corresponding to the labeled training set. This downstream “fine-tuning” using supervised learning to train an encoder and a classifier for a specific task is described further below in the context of steps 205-209.

[0028] Step 205 uses the pretrained patch-level encoder to generate patch-level representations from patches from WSIs in a labeled training set. Step 206 then joins the patch-level representations into region-level pseudo images comprising the patch-level representations.

[0029] Step 207 uses the pretrained region-level encoder to generate region level representations from the region-level pseudo images comprising the patch-level representations. Step 208 joins the region-level representations into whole-slide-level pseudo images. Each whole-slide level pseudo image corresponds to a WSI in the labeled training set.

[0030] However, because the whole-slide-level pseudo image is generated through hierarchical encoding, first at the patch level (patch level representations based on patchlevel pixel data), and then at the region level (region-level representations generated using patch level representations, rather than patch-level pixel data), the whole-slide-level pseudo image contains a small fraction of data (in representation space) than is contained in the amount of pixel-space data corresponding a whole-slide image. For example, if each region includes data from 400 patches, and each patch is 224x224 pixels, and, for example, a WSI has 4 regions, then the entire WSI might contain 80,281,600 pixels worth of data (or multiplied by 3 in RGB pixel space). By contrast, in the embodiment of FIG. 1, a whole-slide level pseudo image can represent the whole slide (assuming 4 regions) using 4x384 or 1,536 values, a tiny fraction of the pixel-space version of the image.

[0031] Continuing with the description of FIG. 2, step 209 uses the whole-slide-level pseudo images to conduct supervised learning to train a whole slide-level attention-based encoder to effectively encode, for a specific prediction task, a whole-slide-level representations of an WSI in the training set. In one embodiment, as described further above in the context of FIG. 1 and below in the context of FIG. 3, the whole-slide-level encoder, during supervised training, generates a whole-slide level representation from the whole-slide level pseudo image. The whole-slide level representation can then be submitted to a classifier network (e.g., multi-layer perceptron) to generate a prediction. During training, the computed loss is back propagated through both the classifier layer or layers and the whole-slide level encoder to adjust the learnable parameters. In other words, the whole-slide level encoder and the classifier are trained together using supervised learning.

[0032] FIG. 3 illustrates various alternatives for each phase of the training process. Alternatives 301 show alternate self-supervised learning (SSL) approaches for pretraining a patch-level encoder, such as, for example, first SSL pretrained encoder 120 of the embodiment of FIG. 1. In some embodiments, the “SimCLR” approach may be used, as described by Chen et al. in “A Simple Framework for Contrastive Learning of Visual Representations” (2020) (arXiv:2002.05709). In some embodiments, a ViT masked auto encoder (ViTMAE) approached may be used, as described by He, K., et al. in “Masked Autoencoders Are Scalable Vision Learners” (2021) (arXiv:2111.06377). In other embodiments, a “DINOvl” approach may be used, as described by Caron et al. in “Emerging Properties in Self-Supervised Vision Transformers” (2021) (arXiv:2104.14294). In some embodiments, a “DIN0v2” approach may be used, as described by Oquab et al. in “DIN0v2: Learning Robust Visual Features without Supervision” (2023)(arXiv:2304.07193). All four papers are hereby incorporated by reference in their entirety.

[0033] Alternatives 302 show alternate SSL approaches for pretraining a region-level encoder, such as, for example, second pretrained encoder 130 of the embodiment shown in FIG. 1. In some embodiments, a ViTMAE approach may be used. In some embodiments, a DINOv 1 approach may be used. In some embodiments, a DINOv2 approach may be used.

[0034] Alternatives 303 show alternate approaches for a whole-slide level encoder such as, for example, whole-slide-level encoder 140 of the embodiment of FIG. 1, together with a classifier such as, for example, classifier 150 of the embodiment of FIG. 1. In some embodiments, the whole slide level encoder is trained together with the classifier using supervised learning. As shown, one embodiment uses a ViT encoder. When a ViT encoder is used, region-level representations for a whole-slide-level pseudo image are positionally encoded and provided to multi-headed attention layers of the ViT encoder. An additional “classification” token (CLS) is processed, together with the region-level representations, and, because the CLS token attends to each region-level representation it is processed with, the output value of this token provides an effective “aggregated” representation that can be used downstream as an input to a classifier for classifying the whole slide. The learnable parameters of the ViT, along with learnable parameters of the classifier (e.g., an MLP) can be fine-tuned via supervised learning.

[0035] In some embodiments, at the fine-tuning stage, a convolutional neural network (CNN), such as a residual CNN (ResNet), is used to further encode each regional representation of a whole slide pseudo image and then an attention mechanism assigns an attention weight to each regional encoding, which can then be used to aggregate the encoding to obtain an attention-weighted representation of the whole slide. This approached is describe in, for example, Use et al., “Attention-based Deep Multiple Instance Learning” (2018) (arXiv: 1802.04712v4). This paper is hereby incorporated by reference in its entirety.The learnable parameters of the CNN and the attention mechanism, along with learnable parameters of the classifier (e.g., an MLP), can be fine-tuned via supervised learning.

[0036] In a particular embodiment, a DIN0v2 objective is used for pretraining ViTs for both the patch-level encoder (e.g., 120 of FIG. 1) and the region-level encoder (e.g., 130 of FIG. 1). In some embodiments, the simCLR objective (contrastive learning) is used to pretrain a ResNet encoder as the patch-level encoder and a DIN0v2 objective is used to pretrain a ViT as the region-level encoder. In some embodiments, a ViTMAE objective is used to pretrain the region-level encoder.

[0037] In some embodiments, any combination of the listed alternative can be selected for each phase. That is, any one or more of the 301 pretraining objective alternatives can be selected for pretraining the patch level encoder and any one or more of the 302 pretraining objective alternatives can be selected for pretraining the region-level encoder.Results from Experiments

[0038] Various embodiments have been tested for downstream finetuning on three selected tasks as follows: (1) NSCLC (Non-Small Cell Lung Cancer) subtyping: 869 H&E images from LUAD (Lung Adenocarcinoma) and LUSC (Lung Squamous Cell Carcinoma); (2) RCC subtyping (Renal Cell Carcinoma): 810 images from KIRP (Kidney Renal Papillary Cell Carcinoma), KICH (Kidney Chromophobe), KIRC (Kidney Renal Clear Cell Carcinoma) and (3) BRCA (Breast Cancer) subtyping: 883 H&E images.

[0039] For each of these tasks, an experiment divided the images in 80 / 20 train / test split. The 80% split was then used for 10-fold cross validation and performance is reported on the test split. Note that the 20% test split was not included in the pretraining of the model at both patch and region levels. For these experiments, the patch-level and region-level encoders were not finetuned (modified) during downstream training of the whole-slide-level encoder and classifier. All used models were trained using 10-fold cross-validation on the trainingsets. Model for each fold is selected on the epoch with the lowest validation loss. The performance was reported on the test set.

[0040] FIG. 4 is a table summarizing example results using the area under the curve (AUC) metric. Results are shown for downstream predictions after using possible combinations of pretraining approaches shown in FIG. 3 for the patch-level encoder alternatives 301 and the region-level encoder alternatives 302.

[0041] With reference to Fig. 4, across both of patch-level and region-level, DIN0v2 pretraining yielded good performance. Furthermore, for region level pretraining methods that use a patch level DIN0v2 model, the performance was high. Similarly, at the region level, DIN0v2 can extract meaningful representations from models trained with lower levels of patch level performance such as ViTMAE and DINOv 1 and still provide a high level of performance on the predictive tasks. Also, using simCLR at the patch level achieves high performance in combination with using any of the other objectives for region-level pretraining.

[0042] Unlike natural images, H&E images can look very uniform in pixel space, and they have subtle variations. Hence, in some embodiments, a mean squared error (MSE) loss that is applied to the images may fail to capture granular nuances of the tumor microenvironment in these images. At a region level, for models that have high levels of patch-level performance such as SimCLR and DIN0v2, the ViTMAE region level model performs extremely well. A masked modelling type pretraining objective may therefore work well in scenarios where the data has a high level of intra image variations.

[0043] FIG. 5 shows an example of a computer system 5000, one or more of which may be used to implement one or more of the apparatuses, systems, and methods illustrated herein. Computer system 5000 executes instruction code contained in a computer program product 560. Computer program product 560 comprises executable code in an electronically readablemedium that may instruct one or more computers such as computer system 5000 to perform processing that accomplishes the exemplary method steps performed.

[0044] The electronically readable medium may be any transitory or non-transitory medium that stores information electronically and may be accessed locally or remotely, for example via a network connection. The medium may include a plurality of geographically dispersed media each configured to store different parts of the executable code at different locations and / or at different times. The executable instruction code in an electronically readable medium directs the illustrated computer system 5000 to carry out various exemplary tasks described herein. The executable code for directing the carrying out of tasks described herein would be typically realized in software. However, it will be appreciated by those skilled in the art, that computers or other electronic devices might utilize code realized in hardware to perform many or all the identified tasks. Those skilled in the art will understand that many variations on executable code may be found that implement exemplary methods within the spirit and the scope of the disclosure.

[0045] The code or a copy of the code contained in computer program product 560 may reside in one or more storage persistent media (not separately shown) communicatively coupled to system 5000 for loading and storage in persistent storage device 570 and / or memory 510 for execution by processor 520. Computer system 500 also includes I / O subsystem 530 and peripheral devices 540. I / O subsystem 530, peripheral devices 540, processor 520, memory 510, and persistent storage device 570 are coupled via bus 550. Like persistent storage device 570 and any other persistent storage that might contain computer program product 560, memory 510 is a non-transitory media (even if implemented as a typical volatile computer memory device). Moreover, those skilled in the art will appreciate that in addition to storing computer program product 560 for carrying out processing described herein, memory 510 and / or persistent storage device 570 may be configured tostore the various data elements referenced and illustrated herein.

[0046] Those skilled in the art will appreciate computer system 5000 illustrates just one example of a system in which a computer program product in accordance with the disclosure may be implemented. To cite but one example, execution of instructions contained in a computer program product may be distributed over multiple computers, such as, for example, over the computers of a distributed computing network.

[0047] Instructions for implementing an artificial neural network or other deep learning network may reside in computer program product 560. When processor 520 is executing the instructions of computer program product 560, the instructions, or a portion thereof, are typically loaded into working memory 510 from which the instructions are readily accessed by processor 520.

[0048] Processor 520 may comprise multiple processors which may comprise respective additional working memories (additional processors and memories not individually illustrated) including one or more graphics processing units (GPUs) comprising at least thousands of arithmetic logic units supporting parallel computations on a large scale. GPUs are often utilized in deep learning applications because they can perform the relevant processing tasks more efficiently than can typical general-purpose processors (CPUs).Processor 520 may additionally or alternatively comprise one or more specialized processing units comprising systolic arrays and / or other hardware arrangements that support efficient parallel processing. Such specialized hardware may work in conjunction with a CPU and / or GPU to carry out the various processing described herein. Such specialized hardware may comprise application specific integrated circuits and the like (which may refer to a portion of an integrated circuit that is application-specific), field programmable gate arrays and the like, or combinations thereof. However, a processor such as processor 520 may be implemented as one or more general purpose processors (preferably having multiple cores)without necessarily departing from the spirit and scope of the present disclosure.

[0049] While the present disclosure has been particularly described with respect to the illustrated embodiments, it will be appreciated that various alterations, modifications, and adaptations may be made based on the disclosure and are intended to be within the scope of the disclosure. While the disclosure has been described in connection with what are presently considered to be the most practical and preferred embodiments, it is to be understood that the present disclosure is not limited to the disclosed embodiments but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the scope of the underlying principles as described by the various embodiments reference above and below.ADDITIONAL EXAMPLES

[0050] Some examples include a method of obtaining a pretrained portion of a deep learning network configured to execute on one or more computers to process a histopathology image to extract representations usable for analyzing tissue corresponding to the histopathology image, the method comprising: obtaining a first pretrained encoder (“encoder” may also be referred to as a feature extraction network in that it extracts a representation of the input data), using a first encoder to pretrain a first encoder by processing a plurality of patches obtained from a first plurality of histopathology images using a first self-supervised learning (SSL) objective; using the first pretrained encoder to obtain a plurality of pseudo-images, each pseudo image corresponding to an image region and comprising a plurality of patch representations obtained by processing a plurality of image patches the image region, the plurality of pseudo images corresponding to image regions from a second plurality of histopathology images; and obtaining a second pretrained encoder, using a second encoder, by processing the plurality of pseudo images using a second self-supervised learning objective to pretrain the second encoder; wherein the pretrained portion of the deep learningnetwork comprises the first and second pretrained encoders.

[0051] In some examples, the first SSL learning objective and the second SSL learning objective comprises DIN0v2, and the first encoder and the second encoder comprise vision transformer (ViT) encoders. In some examples, the first SSL learning objective comprises simCLR. In some examples, the first SSL learning object comprises simCLR and the first encoder comprises a convolutional neural network (CNN). In some examples, the CNN is a ResNet.

[0052] In some examples, the first plurality of histopathology images and the second plurality of histopathology images are the same. If some examples, the first plurality of histopathology of histopathology images and the second plurality of histopathology images are different.

[0053] In some examples, an image region corresponds to portion of an image that is at least 100 times larger than an image patch. In some examples, an image region corresponds to portion of an image that is at least 200 times larger than an image patch. In some examples, an image region corresponds to portion of an image that is at least 400 times larger than an image patch.

[0054] Some embodiments include a computer-executable deep learning network stored in a non-transitory computer readable medium and configured to execute on one or more computers to process a histopathology image and analyze tissue corresponding to the histopathology image comprising: the first pretrained encoder; the second pretrained encoder; and a machine learning analyzer configured to receive the representation of the histopathology image from the second pretrained encoder and generate results of analysis of the tissue corresponding to a histopathology image, wherein the results of analysis comprise at least one of classification, regression, and segmentation.

[0055] In some examples, the machine learning analyzer comprises one or more feed forward neural network layers. In some examples, the machine learning analyzer comprises a vision transformer encoder. In some examples, the machine learning analyzer comprises a non-neural-network-based analyzer. In some examples, the machine learning analyzer comprises a logistic regression-based analyzer. In some examples, the machine learning analyzer comprises a machine learning classifier and analysis results comprise a classification expressed as a class probability.

Claims

CLAIMSWhat is claimed is:

1. A method of obtaining a pretrained portion of a deep learning network configured to execute on one or more computers to process a histopathology image to extract representations usable for analyzing tissue corresponding to the histopathology image, the method comprising: obtaining a first pretrained encoder, using a first encoder, by processing a plurality of patches obtained from a first plurality of histopathology images using a first self-supervised learning (SSL) objective to pretrain the first encoder; using the first pretrained encoder to obtain a plurality of pseudo-images, each pseudo image corresponding to an image region and comprising a plurality of patch representations obtained by processing a plurality of image patches in the image region, the plurality of pseudo images corresponding to image regions from a second plurality of histopathology images; and obtaining a second pretrained encoder, using a second encoder, by processing the plurality of pseudo images using a second self-supervised learning objective to pretrain the second encoder; wherein the pretrained portion of the deep learning network comprises the first and second pretrained encoders.

2. The method of claim 1 wherein the first SSL learning objective comprises DIN0v2 and the first encoder comprises a vision transformer (ViT) encoder.

3. The method of claim 1 wherein the first SSL learning objective comprises DINOv 1 and the first encoder comprises a vision transformer (ViT) encoder.

4. The method of claim 1 wherein the first SSL learning objective comprises acontrastive learning objective.

5. The method of claim 4 wherein the contrastive learning objective comprises simCLR.

6. The method of any of claims 4-5 wherein the first encoder comprises a convolutional neural network.

7. The method of any of claims 2-6 wherein the second SSL learning objective comprises DIN0v2.

8. The method of any of claims 2-6 wherein the second SSL learning objective comprises DINOvl.

9. The method of any of claims 2-6 wherein the second SSL learning objective comprises vision transformer mask autoencoding (ViTMAE).

10. The method of any of claims 1-9 wherein the first plurality of histopathology images and the second plurality of histopathology images are the same.

11. The method of any of claims 1-9 wherein the first plurality of histopathology of histopathology images and the second plurality of histopathology images are different.

12. The method of any of claims 1-9 wherein the first plurality of histopathology of histopathology images and the second plurality of histopathology images are overlapping.

13. The method of any of claims 1-12 wherein an image region corresponds to portion of an image that is at least 100 times larger than an image patch.

14. The method of any of claims 1-12 wherein an image region corresponds to portion of an image that is at least 200 times larger than an image patch.

15. The method of any of claims 1-12 wherein an image region corresponds to portion of an image that is at least 400 times larger than an image patch.

16. A computer-executable deep learning network stored in a non-transitory computerreadable medium and configured to execute on one or more processor of one or more computers to process a histopathology image and analyze tissue corresponding to the histopathology image comprising: the first pretrained encoder obtained by a method according to any one of claims 1- 15; the second pretrained encoder obtained by a method according to any one of claims 1-15; and a machine learning analyzer configured to receive the representation of the histopathology image from the second pretrained encoder and generate results of analysis of the tissue corresponding to a histopathology image, wherein the results of analysis comprise at least one of classification, regression, and segmentation.

17. The computer-executable deep learning network according to claim 16 wherein the machine learning analyzer comprises one or more feed-forward neural network layers.

18. The computer-executable deep learning network according to claim 16 wherein the machine learning analyzer comprises a vision transformer (ViT) encoder.

19. The computer-executable deep learning network according to claim 16 wherein the machine learning analyzer comprises a logistic regression-based analyzer.

20. The computer-executable deep learning network according to claim 16 wherein the machine learning analyzer comprises a machine learning classifier and analysis results comprise a classification expressed as a class probability.

21. A computer-executable deep learning network stored in a non-transitory computer readable medium and configured to execute on one or more processor of one or more computers to process a whole-slide histopathology image (WSI) and analyze tissue corresponding to the WSI, the computer-executable deep learning network comprising:a first pretrained encoder, wherein the first pretrained encoder has been pretrained using a first self-supervised learning objective to extract patch-level representations from patch-level pixel data corresponding to patches of the WSI; a second pretrained encoder, wherein the second pretrained encoder has been pretrained using a second self-supervised learning objective to extract region-level representations from a set of patch-level representations, the set of patch-level representations corresponding to a region of the WSI; and an attention-based machine learning analyzer configured to receive the region-level representations of the WSI from the second pretrained encoder, generate a WSI-level representation from the region-level representations of the WSI, and process the WSI-level representation to generate results of analysis of the tissue corresponding to a histopathology image, wherein the results of analysis comprise at least one of classification, regression, and segmentation.

22. The computer-executable deep learning network of claim 21 wherein the first SSL learning objective comprises DIN0v2 and the first encoder comprises a vision transformer (ViT) encoder.

23. The computer-executable deep learning network of claim 21 wherein the first SSL learning objective comprises DINOv 1 and the first encoder comprises a vision transformer (ViT) encoder.

24. The computer-executable deep learning network of claim 21 wherein the first SSL learning objective comprises a contrastive learning objective.

25. The computer-executable deep learning network of claim 24 wherein the contrastive learning objective comprises simCLR.

26. The computer-executable deep learning network of any of claims 24-25 wherein thefirst encoder comprises a convolutional neural network (CNN).

27. The computer-executable deep learning network of any of claims 22-26 wherein the second SSL learning objective comprises DIN0v2.

28. The computer-executable deep learning network of any of claims 22-26 wherein the second SSL learning objective comprises DINOvl.

29. The computer-executable deep learning network of any of claims 22-26 wherein the second SSL learning objective comprises vision transformer mask autoencoding (ViTMAE).

30. The computer-executable deep learning network of any of claims 21-29 wherein the attention-based machine-learning analyzer comprises a vision transformer (ViT) encoder.

31. The computer-executable deep learning network of any of claims 21-29 wherein the attention-based machine-learning analyzer comprises a convolutional neural network CNN and a multi-instance attention block configured to assign attention weights to output of the CNN to provide an aggregated representation of the WSI.

32. The computer-executable deep learning network of any of claims 30-31 wherein the attention-based machine-learning analyzer comprises one or more feed-forward neural network layers.

33. The computer-executable deep learning network of any of claims 21-32 wherein an image region corresponds to portion of an image that is at least 100 times larger than an image patch.

34. The computer-executable deep learning network of any of claims 21-32 wherein an image region corresponds to portion of an image that is at least 200 times larger than an image patch.

35. The computer-executable deep learning network of any of claims 21-32 wherein an image region corresponds to portion of an image that is at least 400 times larger than animage patch.

36. The computer-executable deep learning network of any of claims 21-35 wherein in a patch comprises an area of the WSI having a size of at least 100x100 and not more than 1000x1000 pixels.

37. The computer-executable deep learning network of any of claims 21-36 wherein a region comprises an area of the WSI having a size of at least 1000x1000 and not more than 10000x10000 pixels.

38. The computer-executable deep learning network of any of claims 21-37 wherein a patch comprises an area of the WSI having a size equal or approximately equal to 224x224 pixels (e.g., 224+ / -50 x 224+ / -50 pixels).

39. The computer-executable deep learning network of any of claims 21-38 wherein a region comprises an area of the WSI having a size equal or approximately equal to 4480x4480 pixels (e.g., 4480+ / -500 x 4480+ / -500 pixels).

Images