Attention-Based Learning for Digital Histopathology Analysis

Abstract

Embodiments of the present disclosure utilize hierarchical encoding and self-supervised learning (SSL) objectives to provide pre-trained portions of a deep learning network for digital histopathology analysis. Some embodiments of the present disclosure utilize different training methods to train different portions of the network. Some embodiments use a DINOv2 objective to pre-train a patch-level encoder and / or to pre-train a region-level encoder. Some embodiments use a contrasting learning objective, such as simCLR, to pre-train a patch-level encoder, but a different learning objective to pre-train a region-level encoder. These and other aspects of the present disclosure are detailed more fully herein.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 486,625, filed February 23, 2023, the contents of which are incorporated herein by reference. [Background technology]

[0002] The present disclosure relates generally to techniques for making computerized inferences from medical images of patients. Summary of the Invention

[0003] The amount of pixel data in a typical whole slide digital histopathology image (WSI) is substantial, which poses challenges to effective and practical computational analysis for making inferences from, performing regression analyses on, and segmenting WSI.

[0004] Embodiments of the present disclosure utilize hierarchical encoding and self-supervised learning (SSL) objectives to provide pre-trained portions of a deep learning network for digital pathology analysis. Some embodiments of the present disclosure utilize different training methods to train different portions of the network. Some embodiments use a DINOv2 objective to pre-train a patch-level encoder and / or to pre-train a region-level encoder. Some embodiments use a contrasting learning objective, such as simCLR, to pre-train a patch-level encoder, but a different learning objective to pre-train a region-level encoder. These and other aspects of the present disclosure are detailed more fully below. [Brief explanation of the drawings]

[0005] [Figure 1]1 illustrates a deep learning network for processing digital histopathology images, according to an embodiment of the present disclosure. [Figure 2] 2 is a flow diagram illustrating a method for training a digital histopathology deep learning network, such as the deep learning network of FIG. 1. The illustrated method is in accordance with an embodiment of the present disclosure. [Figure 3] For example, we illustrate alternative learning methods for each phase of training a hierarchical deep learning network, such as the deep learning network of Figure 1. [Figure 4] 1 is a data table illustrating the results of testing various embodiments of the present disclosure. [Figure 5] Illustrative computer system examples are provided, one or more of which may be used to implement one or more of the devices, systems, and methods illustrated herein.

[0006] Although embodiments are described with reference to the above drawings, the drawings are intended to be illustrative and other embodiments are consistent with the spirit and scope of this disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0007] Various embodiments are described more fully hereinafter with reference to the accompanying drawings, which form a part of this specification and which show, by way of illustration, specific examples of implementations of the embodiments. This specification may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this specification will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Among other things, the specification may be embodied as methods or devices. Accordingly, any of the various embodiments herein may take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment combining software and hardware aspects. Therefore, the following description should not be construed in a limiting sense.

[0008] 1 illustrates a deep learning network 1000 for processing digital histopathology images, such as whole slide images 11. In digital histopathology, whole slide images of hematoxylin and eosin (H&E) stained tissue are generally referred to simply as "whole slide images" or "WSIs."

[0009] In the example shown, the system is assumed to be fully trained and fine-tuned for end use in generating predictions or otherwise making inferences related to a specific task (e.g., the presence of a specific type of cancer based on pathology images of tissue samples). In this example, patch-level and region-level encoders are pre-trained for a more general task using self-supervised learning (SSL) techniques (also known in the art as "targeting"). An example of hierarchical pre-training of a vision transformer (ViT) encoder for WSI is described by Chen et al. in "Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learning" (2022) (arXiv:2206.02647), which is incorporated herein by reference in its entirety. The entire slide-level encoder, along with a classification network, is trained using labeled data to fine-tune for a specific prediction task.

[0010] The pre-processing block 110 receives the WSI 11 (e.g., digital data corresponding to an image of H&E-stained tissue on a pathology histology slide, where the tissue corresponds to a sample taken from a patient) and performs pre-processing on the image 11, including dividing the image 11 into regions and patches, each region containing multiple patches. In one non-limiting example, the WSI is divided into regions, each region being 4480 x 4480 pixels in size. In this example, each region is divided into 400 patches, each patch being 224 x 224 pixels in size. The number of regions in a WSI depends on the size of the WSI and can be expected to vary across a set of WSIs. It may be 4 regions, 8 regions, 10 regions, 16 regions, or some other number of regions.

[0011] WSI is known to have various types of artifacts, ranging from large amounts of background to pen marks, blurred regions, and bubbles. The attention mechanism of the Vision Transformer (ViT) network can automatically focus on salient regions, so it does not require an extremely fine-grained quality control (QC) output. In one embodiment, background and pen marks are detected from the WSI.

[0012] In one embodiment, the preprocessing block 110 uses HistoQC (https: / / github.com / choosehappy / HistoQC / wiki), an open-source quality control tool for digital histopathology, to extract tissue regions of size 4480x4480 from the WSI. Each region is processed through the tool, which in turn creates metrics to track the amount of background and the location of pen marks within the entire slide image. In one example, based on these metrics, the preprocessing block 110 filters out patches containing artifacts and selects regions within the image that have rich features.

[0013] The pre-processing block 110 outputs patch-level digital images 12. In one embodiment, the patch-level images 12 are 224 x 224 pixels in size. The first pre-trained encoder 120 processes each patch image 12 to generate a patch-level representation 21. In one embodiment, the patch-level representation 21 is a vector having 768 dimensions. Therefore, in this embodiment, the resulting representation data for each patch-level representation 21 is 1 x 768, which consumes significantly less memory than the 224 x 224 pixel data of the patch 12. Of course, those skilled in the art will understand that the dimension size depends on the type of encoder used and its configuration for a particular application. Therefore, the dimensions given for this embodiment are merely exemplary.

[0014] A patch-level representation 21 is used to represent each patch within the region, thereby providing a region-level pseudo-image 20. Image 20 is referred to as a "pseudo" image (or representation image) because the patches of the region image are not a pixel-space representation, but rather a representation in a space whose dimensions are defined by the first patch-level encoder 120. In one embodiment, each region contains 40 patches, and therefore region-level pseudo-image 20 is 40x768 in size.

[0015] The region-level pseudo-images 20 are processed by a second pre-trained encoder 130. The encoder 130 processes each region-level pseudo-image 20 to generate a region-level representation 31. In one embodiment, the region-level representation 31 is a vector having 384 dimensions. Therefore, in this embodiment, the resulting representation data for each region-level representation 31 is sized 1×384, which consumes significantly less memory than the 4480×4480 pixel data for the corresponding region from the WSI 11. Of course, those skilled in the art will understand that the dimension size depends on the type of region-level encoder used and its configuration for a particular application. Therefore, the dimensions given for this embodiment are merely exemplary.

[0016] Region-level representations 31 are used to represent each region of the full slide image (one representation 31 for each region), thereby providing a full slide-level pseudo-image 30 that includes the region-level representations 31. The number of regions may vary depending on the slide. However, in one embodiment, the WSI is divided into approximately four regions, and therefore the full slide pseudo-image 30 may be 4x384 in size.

[0017] The region-level representations 31 corresponding to the full-slide pseudo-images 30 are processed by an attention-based encoder 140. In one embodiment, the attention-based encoder 140 is a Vision Transformer (ViT) encoder. In one implementation, ViT has four attention heads, two layers (i.e., depth=2), and 384 output dimensions. The attention-based encoder 140 generates a full-slide-level representation 41 (which, in the context of the ViT encoder, are processed CLS tokens). The representations 41 are then processed by a classifier network 150, which in one implementation may include one or more typical feed-forward hidden layers (i.e., a multi-layer perceptron or "MLP") followed by a softmax layer, to obtain predictions (e.g., class probabilities) that classify the tissues corresponding to the associated WSIs 11 processed by the system 1000.

[0018] FIG. 2 is a flow diagram illustrating a method 2000 for training a digital histopathology deep learning network, such as, for example, network 1000 of FIG.

[0019] In step 201, self-supervised learning is performed on patch-level training images to obtain a pre-trained patch-level encoder. Preferably, in step 201, a large training set of histopathology images is used. These images are preferably not task-specific but include a wide variety of tissue samples, for example, from many different potential disease sites. In one example, over 11,677 H&E-stained whole slide images from the TCGA public data are used for pre-training. In one example, the training slides are from 33 different potential tumor sites. In one example, approximately 33 million patches of 224 x 224 pixels are extracted from the WSI training set and used in step 201.

[0020] After the pre-trained patch-level encoder is obtained from step 201, step 202 encodes training image patches (pixel data) into patch-level representations using the pre-trained patch-level encoder. In one embodiment, each patch-level representation is a vector representation containing 768 dimensions. In one embodiment, the training image patches encoded in step 202 are the same training patches used in step 201. In another embodiment, the training image patches encoded in step 202 are different image patches from a different training set than the one used to perform pre-training in step 201.

[0021] Step 203 uses the patch-level representations to form region-level pseudo-images, each pseudo-image corresponding to a region of the WSI, the pseudo-image including multiple patch representations of patches within the corresponding region. In one embodiment, the region-level pseudo-image includes 400 patch-level representations.

[0022] Step 204 uses the region-level pseudo-images to perform self-supervised learning to pre-train a region-level encoder.

[0023] Once the patch-level and region-level encoders are pre-trained, they can then be used to hierarchically generate whole-slide pseudo-images from the labeled training set. These whole-slide pseudo-images, including region-level representations, can then be used to train the whole-slide encoder, along with a classifier, for the specific prediction task corresponding to the labeled training set. This downstream "fine-tuning" using supervised learning to train the encoder and classifier for a specific task is further described below in the context of steps 205-209.

[0024] Step 205 uses a pre-trained patch-level encoder to generate patch-level representations from patches from the WSI in the labeled training set, and step 206 then combines the patch-level representations into a region-level pseudo-image containing the patch-level representations.

[0025] Step 207 uses a pre-trained region-level encoder to generate region-level representations from the region-level pseudo-images containing the patch-level representations. Step 208 combines the region-level representations into full-slide-level pseudo-images. Each full-slide-level pseudo-image corresponds to a WSI in the labeled training set.

[0026] However, because the full slide-level pseudo-image is generated through hierarchical encoding, first at the patch level (patch-level representations based on patch-level pixel data) and then at the region level (region-level representations generated using patch-level representations rather than patch-level pixel data), the full slide-level pseudo-image contains a smaller proportion of data (in representation space) than would be contained in the amount of pixel space data corresponding to the full slide image. For example, if each region contains data from 400 patches, each patch is 224 x 224 pixels, and the WSI has, for example, four regions, the entire WSI can contain 80,281,600 pixels' worth of data (or tripled in RGB pixel space). In contrast, in the embodiment of Figure 1, the full slide-level pseudo-image can represent the full slide (assuming four regions) using 4 x 384 or 1,536 values, a small fraction of the pixel space version of the image.

[0027] Continuing with FIG. 2 , step 209 uses the full-slide-level pseudo-images to perform supervised learning to train a full-slide-level attention-based encoder, effectively encoding full-slide-level representations of WSIs in the training set for a particular prediction task. In one embodiment, as further described above in the context of FIG. 1 and below in the context of FIG. 3 , the full-slide-level encoder generates full-slide-level representations from the full-slide-level pseudo-images during supervised training. The full-slide-level representations can then be submitted to a classifier network (e.g., a multilayer perceptron) to generate predictions. During training, the computed loss is backpropagated to both the classifier layer and the full-slide-level encoder to adjust the learnable parameters. In other words, the full-slide-level encoder and classifier are trained together using supervised learning.

[0028] FIG. 3 illustrates various alternatives for each phase of the training process. Alternative 301 shows an alternative self-supervised training (SSL) approach for pre-training a patch-level encoder, such as the first SSL pre-trained encoder 120 in the embodiment of FIG. 1. In some embodiments, the "SimCLR" approach may be used, as described by Chen et al. in "A Simple Framework for Contrastive Learning of Visual Representations" (2020) (arXiv:2002.05709). In some embodiments, the ViT masked autoencoder (ViTMAE) approach may be used, as described by He, K. et al. in "Masked Autoencoders Are Scalable Vision Learners" (2021) (arXiv:2111.06377). In other embodiments, the "DINOv1" approach may be used, as described by Caron et al. in "Emerging Properties in Self-Supervised Vision Transformers" (2021) (arXiv:2104.14294). In some embodiments, the "DINOv2" approach may be used, as described by Oquab et al. in "DINOv2: Learning Robust Visual Features without Supervision" (2023) (arXiv:2304.07193), all four of which are incorporated herein by reference in their entireties.

[0029] Alternative 302 illustrates an alternative SSL approach for pre-training a region-level encoder, such as, for example, the second pre-trained encoder 130 of the embodiment shown in FIG. 1. In some embodiments, the ViTMAE approach may be used. In some embodiments, the DINOv1 approach may be used. In some embodiments, the DINOv2 approach may be used.

[0030] Alternative 303 illustrates an alternative approach to a full slide-level encoder, such as full slide-level encoder 140 of the embodiment of FIG. 1, along with a classifier, such as classifier 150 of the embodiment of FIG. 1. In some embodiments, the full slide-level encoder is trained together with the classifier using supervised learning. As shown, one embodiment uses a ViT encoder. When a ViT encoder is used, region-level representations of the full slide-level pseudo-images are positionally encoded and provided to the multi-head attention layer of the ViT encoder. An additional "classification" token (CLS) is processed along with the region-level representations. A CLS token accompanies each region-level representation it processes, so that the output value of this token provides an effective "aggregate" representation that can be used downstream as input to a classifier to classify the full slide. The learnable parameters of the ViT encoder, along with the learnable parameters of the classifier (e.g., MLP), can be fine-tuned via supervised learning.

[0031] In some embodiments, a convolutional neural network (CNN), such as a residual CNN (ResNet), can be used in the fine-tuning stage to further encode each region representation of the whole-slide pseudo-image. An attention mechanism can then assign attention weights to each region encoding, which can then be used to aggregate the encodings to obtain an attention-weighted representation of the whole slide. This approach is described, for example, in Ilse et al., "Attention-based Deep Multiple Instance Learning," (2018) (arXiv:1802.04712v4), which is incorporated herein by reference in its entirety. The learnable parameters of the CNN and attention mechanism, along with the learnable parameters of the classifier (e.g., MLP), can be fine-tuned via supervised learning.

[0032] In particular embodiments, the DINOv2 target is used to pre-train ViT for both the patch-level encoder (e.g., 120 in FIG. 1) and the region-level encoder (e.g., 130 in FIG. 1). In some embodiments, the simCLR target (contrastive learning) is used to pre-train the ResNet encoder as the patch-level encoder, and the DINOv2 target is used to pre-train ViT as the region-level encoder. In some embodiments, the ViTMAE target is used to pre-train the region-level encoder.

[0033] In some embodiments, any combination of the listed alternatives can be selected for each phase, i.e., any one or more of the 301 pre-training target alternatives can be selected for pre-training the patch-level encoder, and any one or more of the 302 pre-training target alternatives can be selected for pre-training the region-level encoder.

[0034] Experimental results Various embodiments were tested for downstream fine-tuning on three selected tasks: (1) 869 H&E images from NSCLC (Non-Small Cell Lung Cancer) subtypes: LUAD (Lung Adenocarcinoma) and LUSC (Lung Squamous Cell Carcinoma), (2) 810 images from RCC subtypes (Renal Cell Carcinoma): KIRP (Kidney Renal Papillary Cell Carcinoma), KICH (Kidney Chromophobe), and KIRC (Kidney Renal Clear Cell Carcinoma), and (3) BRCA (Breast Cancer) subtypes: 883 H&E images.

[0035] For each of these tasks, experiments divided images into an 80 / 20 training / test split. The 80% split was then used for 10-fold cross-validation, and performance is reported for the test split. Note that the 20% test split was not included in the pre-training of models at both the patch and region levels. In these experiments, the patch-level and region-level encoders were not fine-tuned (modified) during the downstream training of the full slide-level encoder and classifier. All models used were trained using 10-fold cross-validation on the training set. The model for each split was selected at the epoch with the lowest validation loss. Performance was reported on the test set.

[0036] Figure 4 is a table summarizing exemplary results using the area under the curve (AUC) metric. The results of downstream prediction after using possible combinations of the pre-training approaches shown in Figure 3 for the patch-level encoder alternative 301 and the region-level encoder alternative 302 are shown.

[0037] Referring to Figure 4, both patch-level and domain-level DINOv2 pretraining yielded good performance. Furthermore, performance was high for domain-level pretraining methods using patch-level DINOv2 models. Similarly, at the domain level, DINOv2 can extract meaningful representations from models trained with lower levels of patch-level performance, such as ViTMAE and DINOv1, while still providing high levels of performance for prediction tasks. Furthermore, using simCLR at the patch level, combined with using any of the other objectives for domain-level pretraining, achieves high performance.

[0038] Unlike natural images, H&E images can appear very uniform in pixel space and have subtle variations. Therefore, in some embodiments, a mean squared error (MSE) loss applied to the images may not be able to capture the granular nuances of the tumor microenvironment in these images. At the region level, for models with high patch-level performance, such as SimCLR and DINOv2, the ViTMAE region-level model performs extremely well. Therefore, masked modeling-type pre-training objectives may perform well in scenarios where the data has a high level of intra-image variation.

[0039] 5 illustrates an example computer system 5000, one or more of which may be used to implement one or more of the devices, systems, and methods illustrated herein. The computer system 5000 executes instruction code contained in a computer program product 560. The computer program product 560 includes executable code in an electronically readable medium that may instruct one or more computers, such as the computer system 5000, to perform processes that effect the example method steps performed.

[0040] The electronically readable medium may be any transitory or non-transitory medium that stores information electronically and may be accessed locally or remotely, for example, via a network connection. The medium may include multiple geographically distributed media, each configured to store different portions of executable code at different locations and / or different times. Executable instruction code in the electronically readable medium directs the illustrated computer system 5000 to perform the various exemplary tasks described herein. Executable code for directing the performance of the tasks described herein is typically implemented in software. However, those skilled in the art will appreciate that a computer or other electronic device may utilize hardware-implemented code to perform many or all of the identified tasks. Those skilled in the art will appreciate that many variations of executable code implementing the exemplary methods may be found within the spirit and scope of the present disclosure.

[0041] Code or copies of code contained in computer program product 560 may reside on one or more storage persistent media (not separately shown) communicatively coupled to system 5000 for loading and storing in persistent storage device 570 and / or memory 510 for execution by processor 520. Computer system 500 also includes an I / O subsystem 530 and peripheral devices 540. I / O subsystem 530, peripheral devices 540, processor 520, memory 510, and persistent storage device 570 are coupled via bus 550. Memory 510, like persistent storage device 570 and any other persistent storage that may contain computer program product 560, is a non-transitory medium (even when implemented as a typical volatile computer memory device). Furthermore, those skilled in the art will understand that, in addition to storing computer program product 560 for performing the processes described herein, memory 510 and / or persistent storage device 570 may be configured to store various data elements referenced and illustrated herein.

[0042] Those skilled in the art will appreciate that computer system 5000 illustrates only one example of a system in which a computer program product according to the present disclosure may be implemented. By way of example, execution of instructions included in a computer program product may be distributed across multiple computers, such as, for example, computers in a distributed computing network.

[0043] Instructions for implementing an artificial neural network or other deep learning network may be present in a computer program product 560. When the processor 520 is executing instructions of the computer program product 560, the instructions, or portions thereof, are typically loaded into the working memory 510, from which the instructions can be easily accessed by the processor 520.

[0044] Processor 520 may comprise multiple processors, each of which may comprise additional working memory (additional processors and memory not separately illustrated), including one or more graphics processing units (GPUs) with at least thousands of arithmetic logic units supporting massively parallel computation. GPUs are often utilized in deep learning applications because they can perform associated processing tasks more efficiently than typical general-purpose processors (CPUs). Processor 520 may additionally or alternatively comprise one or more special-purpose processing units comprising systolic arrays and / or other hardware arrangements supporting efficient parallel processing. Such special-purpose hardware may operate in conjunction with the CPU and / or GPU to perform various processes described herein. Such special-purpose hardware may comprise an application-specific integrated circuit (which may refer to a portion of an application-specific integrated circuit), a field-programmable gate array, or the like, or a combination thereof. However, a processor such as processor 520 may be implemented as one or more general-purpose processors (preferably having multiple cores) without necessarily departing from the spirit and scope of the present disclosure.

[0045] While the present disclosure has been particularly described with reference to illustrated embodiments, it will be understood that various changes, modifications, and adaptations may be made based on the present disclosure and are intended to be within the scope of the present disclosure. While the present disclosure has been described in connection with what are presently considered to be the most practical and preferred embodiments, it will be understood that the present disclosure is not limited to the disclosed embodiments, but on the contrary, is intended to cover various modifications and equivalent arrangements that are within the scope of the basic principles described by the various embodiments set forth above and referenced below.

[0046] Additional Examples Some embodiments include a method for obtaining a pre-trained portion of a deep learning network configured to run on one or more computers to process histopathology images to extract representations usable for analyzing tissue corresponding to the histopathology images, the method including obtaining a first pre-trained encoder (an "encoder" may also be referred to as a feature extraction network in that it extracts representations of input data), using the first encoder to pre-train the first encoder by processing a plurality of patches obtained from a first plurality of histopathology images using a first self-supervised learning (SSL) objective, and generating a plurality of pseudorandom images. The method includes using a first pre-trained encoder to obtain pseudo-images, each pseudo-image corresponding to an image region and including a plurality of patch representations obtained by processing a plurality of image patches of the image region, the plurality of pseudo-images corresponding to image regions from a second plurality of histopathology images; and using the second encoder to obtain a second pre-trained encoder by processing the plurality of pseudo-images using a second self-supervised learning objective to pre-train the second encoder, wherein the pre-trained portion of the deep learning network includes the first pre-trained encoder and the second pre-trained encoder.

[0047] In some embodiments, the first SSL learning object and the second SSL learning object include DINOv2, and the first encoder and the second encoder include a Vision Transformer (ViT) encoder. In some embodiments, the first SSL learning object includes simCLR. In some embodiments, the first SSL learning object includes simCLR, and the first encoder includes a convolutional neural network (CNN). In some embodiments, the CNN is ResNet.

[0048] In some embodiments, the first plurality of histopathology images and the second plurality of histopathology images are the same. In some embodiments, the first plurality of histopathology images and the second plurality of histopathology images are different.

[0049] In some embodiments, the image region corresponds to a portion of the image that is at least 100 times larger than the image patch, in some embodiments, the image region corresponds to a portion of the image that is at least 200 times larger than the image patch, in some embodiments, the image region corresponds to a portion of the image that is at least 400 times larger than the image patch.

[0050] Some embodiments provide a computer-executable deep learning network stored on a non-transitory computer-readable medium and configured to execute on one or more computers to process histopathology images and analyze tissue corresponding to the histopathology images, the deep learning network comprising: a first pre-trained encoder; a second pre-trained encoder; and a machine learning analyzer configured to receive the representation of the histopathology image from the second pre-trained encoder and generate results of an analysis of the tissue corresponding to the histopathology image, wherein the results of the analysis include at least one of classification, regression, and segmentation.

[0051] In some embodiments, the machine learning analyzer comprises one or more feed-forward neural network layers. In some embodiments, the machine learning analyzer comprises a vision transformer encoder. In some embodiments, the machine learning analyzer comprises a non-neural network based analyzer. In some embodiments, the machine learning analyzer comprises a logistic regression based analyzer. In some embodiments, the machine learning analyzer comprises a machine learning classifier, and the analysis results include classifications expressed as class probabilities.

Claims

1. 1. A method for obtaining a pre-trained portion of a deep learning network configured to run on one or more computers for processing histopathology images to extract representations usable for analyzing tissue corresponding to the histopathology images, the method comprising: obtaining a first pre-trained encoder using a first encoder by processing a plurality of patches obtained from a first plurality of histopathology images using a first self-supervised learning (SSL) objective; and using the first pre-trained encoder to obtain a plurality of pseudo-images, each pseudo-image corresponding to an image region and including a plurality of patch representations obtained by processing a plurality of image patches within the image region, the plurality of pseudo-images corresponding to image regions from a second plurality of histopathology images; and processing the plurality of simulated images using a second self-supervised learning objective to pre-train a second encoder, thereby obtaining a second pre-trained encoder using the second encoder; 10. The method of claim 1, wherein the pre-trained portion of the deep learning network includes the first pre-trained encoder and the second pre-trained encoder.

2. The method of claim 1 , wherein the first SSL learning target comprises DINOv2 and the first encoder comprises a Vision Transformer (ViT) encoder.

3. The method of claim 1 , wherein the first SSL learning target comprises DINOv1 and the first encoder comprises a Vision Transformer (ViT) encoder.

4. The method of claim 1 , wherein the first SSL learning objective comprises a contrasting learning objective.

5. The method of claim 4 , wherein the contrasting learning objectives include simCLR.

6. The method of claim 4 or 5, wherein the first encoder comprises a convolutional neural network.

7. The method of any one of claims 2 to 6, wherein the second SSL learning objective comprises DINOv2.

8. The method of any one of claims 2 to 6, wherein the second SSL learning objective comprises DINOv1.

9. The method of any one of claims 2 to 6, wherein the second SSL learning objective comprises Vision Transformer Mask Auto-Encoding (ViTMAE).

10. The method of any one of claims 1 to 9, wherein the first plurality of histopathological images and the second plurality of histopathological images are the same.

11. The method of any one of claims 1 to 9, wherein the first plurality of histopathology images and the second plurality of histopathology images are different.

12. The method of any one of claims 1 to 9, wherein the first plurality of histopathology images and the second plurality of histopathology images are overlapping.

13. A method according to any one of the preceding claims, wherein the image region corresponds to a portion of the image that is at least 100 times larger than the image patch.

14. A method according to any preceding claim, wherein the image region corresponds to a portion of the image that is at least 200 times larger than the image patch.

15. A method according to any one of the preceding claims, wherein the image region corresponds to a portion of the image that is at least 400 times larger than the image patch.

16. 1. A computer-executable deep learning network stored on a non-transitory computer-readable medium and configured to execute on one or more processors of one or more computers to process histopathology images and analyze tissue corresponding to the histopathology images, the first pre-trained encoder obtained by the method of any one of claims 1 to 15; the second pre-trained encoder obtained by the method of any one of claims 1 to 15; and and a machine learning analyzer configured to receive the representation of the histopathology image from the second pre-trained encoder and generate results of an analysis of the tissue corresponding to the histopathology image, wherein the results of the analysis include at least one of classification, regression, and segmentation.

17. 17. The computer-executable deep learning network of claim 16, wherein the machine learning analyzer comprises one or more feed-forward neural network layers.

18. 17. The computer-executable deep learning network of claim 16, wherein the machine learning analyzer comprises a Vision Transformer (ViT) encoder.

19. 17. The computer-executable deep learning network of claim 16, wherein the machine learning analyzer comprises a logistic regression-based analyzer.

20. 17. The computer-executable deep learning network of claim 16, wherein the machine learning analyzer comprises a machine learning classifier and the analysis results include classifications expressed as class probabilities.

21. 1. A computer-executable deep learning network stored on a non-transitory computer-readable medium and configured to execute on one or more processors of one or more computers to process whole slide histopathology images (WSIs) and analyze tissue corresponding to the WSIs, the computer-executable deep learning network comprising: a first pre-trained encoder, the first pre-trained encoder being pre-trained using a first self-supervised learning objective to extract patch-level representations from patch-level pixel data corresponding to patches of the WSI; a second pre-trained encoder, the second pre-trained encoder being pre-trained using a second self-supervised learning objective to extract region-level representations from a set of patch-level representations corresponding to regions of the WSI; and an attention-based machine learning analyzer configured to receive the region-level representations of the WSI from the second pre-trained encoder, generate a WSI-level representation from the region-level representations of the WSI, and process the WSI-level representations to generate results of an analysis of the tissue corresponding to a histopathology image, wherein the results of the analysis include at least one of classification, regression, and segmentation.

22. 22. The computer-executable deep learning network of claim 21 , wherein the first SSL learning target comprises DINOv2 and the first encoder comprises a Vision Transformer (ViT) encoder.

23. 22. The computer-executable deep learning network of claim 21, wherein the first SSL learning goal comprises DINOv1 and the first encoder comprises a Vision Transformer (ViT) encoder.

24. 22. The computer-executable deep learning network of claim 21, wherein the first SSL learning goal comprises a contrasting learning goal.

25. 25. The computer-executable deep learning network of claim 24, wherein the contrastive learning goal comprises simCLR.

26. 26. The computer-executable deep learning network of claim 24 or 25, wherein the first encoder comprises a convolutional neural network (CNN).

27. 27. The computer-executable deep learning network of claim 22, wherein the second SSL learning goal comprises DINOv2.

28. 27. The computer-executable deep learning network of claim 22, wherein the second SSL learning goal comprises DINOv1.

29. 27. The computer-executable deep learning network of any one of claims 22 to 26, wherein the second SSL learning goal comprises Vision Transformer Mask Auto-Encoding (ViTMAE).

30. 30. The computer-executable deep learning network of any one of claims 21 to 29, wherein the attention-based machine learning analyzer comprises a Vision Transformer (ViT) encoder.

31. 30. The computer-executable deep learning network of any one of claims 21 to 29, wherein the attention-based machine learning analyzer comprises a convolutional neural network CNN and a multi-instance attention block configured to assign attention weights to outputs of the CNN to provide an aggregate representation of the WSI.

32. 32. The computer-executable deep learning network of claim 30 or 31, wherein the attention-based machine learning analyzer comprises one or more feed-forward neural network layers.

33. 33. The computer-executable deep learning network of any one of claims 21 to 32, wherein an image region corresponds to a portion of an image that is at least 100 times larger than an image patch.

34. 33. The computer-executable deep learning network of any one of claims 21 to 32, wherein an image region corresponds to a portion of an image that is at least 200 times larger than an image patch.

35. 33. The computer-executable deep learning network of any one of claims 21 to 32, wherein the image region corresponds to a portion of the image that is at least 400 times larger than the image patch.

36. 36. The computer-executable deep learning network of any one of claims 21 to 35, wherein a patch comprises an area of ​​the WSI having a size of at least 100x100 pixels and no greater than 1000x1000 pixels.

37. 37. The computer-executable deep learning network of any one of claims 21 to 36, wherein a region comprises an area of ​​the WSI having a size of at least 1000x1000 pixels and no greater than 10000x10000 pixels.

38. 38. The computer-executable deep learning network of any one of claims 21 to 37, wherein a patch comprises an area of ​​the WSI having a size equal to or approximately equal to 224 x 224 pixels (e.g., 224 + / - 50 x 224 + / - 50 pixels).

39. 39. The computer-executable deep learning network of any one of claims 21-38, wherein a region comprises an area of ​​the WSI having a size equal to or approximately equal to 4480 x 4480 pixels (e.g., 4480 + / - 500 x 4480 + / - 500 pixels).

Images