Classifying phenotypes and identifying biological mediators from digital histopathology images using deep learning models

Abstract

The present disclosure provides methods of training a deep learning (DL)-based classifier to classify a digital histopathology image with respect to a phenotype, methods of classifying a digital histopathology image with respect to a phenotype, and methods of identifying a biological mediator associated with a pathology, pathophysiology, or other clinical characteristic. The disclosure further provides systems for implementation of these methods.

Description

SYSTEMS AND METHODS FOR CLASSIFYING PHENOTYPES AND IDENTIFYING BIOLOGICAL MEDIATORS FROM DIGITAL HISTOPATHOLOGY IMAGES USINGDEEP LEARNING MODELSTECHNICAL FIELD

[0001] The present disclosure relates to methods and systems for using deep learning (DL)-based classifiers to classify a tissue phenotype in a digital histopathology image as well as methods of training such classifiers and trained DL-based classifiers. The present disclosure further relates to methods and systems for using DL-based classifiers or DL models to locate areas of interest in digital histopathology images that may then be used to identify biological mediators associated with the digital histopathology images or a pathology reflected in the images.BACKGROUND

[0002] Digital histopathology images are a common and readily accessible form of medical diagnostic information. Histopathology involves preparing microscopy slides from patient tissue samples that are treated, typically by staining with one or more specific chemicals, to enhance the visibility of microscopic structures. These microscopic structures help pathologists determine the pathophysiology of the tissue, which reflects whether the patient has a particular pathology and potentially also information about that pathology. For example, histopathology is the most common way to determine if resected tumors or tumor biopsies contain cancerous tissue and often also the subtype, stage, or other clinically relevant properties of the cancer.

[0003] Histopathology’s popularity is due, in part, to the existence of well-established protocols for preparing and staining microscopy slides from patient tissue samples. In addition to being well-established, such protocols can often be performed in even very basic laboratories, which are present in almost all hospitals. Furthermore, many histopathology slide preparation techniques are very economical as compared to other diagnostic techniques. Increasingly, images of histopathology slides are stored in digital form.

[0004] Histopathology slides are typically viewed by pathologists who make determinations about the pathophysiology of the tissue samples in the slides. Although there are also well-established and reasonably accurate protocols for assessing pathophysiologyusing histopathology slides, there is still often significant variation in assessments between different pathologists. This variation is partially due to differences in human judgements of the same visual information, which are not eliminated even by the rigorous training pathologists undertake. However, another source of variation is the fact that pathologists assessing a histopathology slide view only a few high-power magnification fields that typically represent only about 20% of the entire slide. Differences in pathophysiology assessment may, therefore, sometimes simply be due to different pathologists viewing different portions of the slide. This problem is exacerbated when indicators of the underlying pathology are only sparsely found in the patient tissue sample. Unfortunately, often the most effective window for treating a patient is when the underlying pathology is in an early stage, and is, therefore, only sparsely found in a tissue sample.

[0005] In an attempt to improve consistency of these histopathology results, others have captured digital images of histopathology slides. These digital images may be analyzed in a computer-assisted manner that reviews the entirety of the slide, which is often referred to as a whole slide image (WSI). Use of the WSI at least improves the ability to detect early- stage pathologies or other pathologies whose indicators are only sparsely found in a tissue sample. However, many methods of digital image analysis focus on features also used by pathologists to assess the disease or condition, such as the morphology of tumor cells. These analysis methods essentially attempt to accomplish what pathologists could do if they had time to examine the WSI, and also did not differ in pathophysiology assessments of the same slide. Such methods remain reliant on human abilities to detect information in histopathology images because human abilities were first used to identify the features that image analysis software then looks for.SUMMARY

[0006] The present disclosure provides a method of training a deep learning (DL)- based classifier to classify a digital histopathology image with respect to a phenotype, the method including: creating a dataset of identified digital histopathology images including, for each identified digital histopathology image, an image and a phenotype ground truth label that reflects a known phenotype classification for the image; creating, for each identified digital histopathology image, a set of image- specific tiles, wherein each tile represents an area of the identified digital histopathology image that is less than the whole digital histopathology image; extracting a features from each tile using a pre-trained featureextractor to create a tile-specific feature vector for each tile; assembling a set of tile-specific feature vectors for each identified medical image; providing, for at least a subset of identified medical images, the set of tile-specific feature vectors and the phenotype ground truth label to an untrained DL model; training the DL model in a self-supervised manner using the provided sets of tile- specific feature vectors and phenotype ground truth label to classify digital histopathology images with respect to the phenotype; assessing accuracy of the DL model in classifying digital histopathology images with respect to phenotype; and comparing assessed accuracy to an accuracy threshold and, if the accuracy threshold is met, storing the DL model.

[0007] In some embodiments, the method may further include any of the following additional aspects, which may be combined with one another in any combinations:• the digital histopathology includes an image of a hematoxylin and eosin (H&E)-stained slide;• the digital histopathology image includes an image of a Perl’s Prussian blue, Alcian blue, Masson’s trichome, Reticulin stain, silver stain, periodic acid Schiff Reaction (PAS) Stain, Congo red, Mucicarmine, Sudan black, Nissl stain (also called the Cresyl Violet Stain), or Golgi method stained slide;• the digital histopathology image includes a whole slide image (WSI) or near WSI and the set of image- specific tiles collectively represents the entire area of the WSI or near WSI;• each tile represents no more than 5% of total area of the digital histopathology image;• the phenotype classification is a binary classification;• the phenotype classification is a graduated classification;• the pre-trained feature extractor includes a vision transformer trained using paired imagetext descriptions;• the pre-trained feature extractor has been trained using paired pathology image-text descriptions in a non-task specific manner;• the untrained DL model includes an attention-based model;• the DL model includes a neural network and a final classifier that uses an output layer of the neural network to provide a phenotype classification;• accuracy is assessed by determining the area under receiving operator curve (AUROC);• the accuracy threshold is at least 70%;• further including preparing augmented data for each identified digital histopathology image;• the augmented data includes at least one of tile- shifted data, color augmented data, and pseudo-vector mixup data;• the phenotype includes a cancer pathophysiology;• the phenotype includes at least one of: tertiary lymphoid structures, B-cell proliferation, T-cell mediated cytotoxicity, FoxP3-mediated immunosuppression, antigen processing and presentation, tumor associated fibroblasts, angiogenesis, epithelial-mesenchymal transitions, tumor cell invasiveness, fatty acid metabolism, glycolysis, and oxidative phosphorylation.

[0008] The disclosure further includes a deep learning (DL)-based classifier trained to classify a digital histopathology image with respect to a phenotype, wherein the classifier includes a feature extractor and DL model trained according to any of the above methods.

[0009] The disclosure further provides a deep learning (DL)-based classifier to classify a digital histopathology image with respect to a phenotype, the system including one or more processors and a non-transitory computer-readable medium including one or more sequences of instructions that, when executed by the one or more processors, cause the system to perform operations including any of the above methods.

[0010] The disclosure further provides a method of classifying a digital histopathology image with respect to a phenotype, the method including: creating a set of tiles, wherein each tile represents an area of the digital histopathology image that is less than the whole digital histopathology image; extracting a features from each tile using a pretrained feature extractor to create a set of tile-specific feature vectors; providing the set of file- specific feature vectors to a phenotype-trained DL model trained according to any of the above methods to obtain a phenotype classification for the digital histopathology image; and storing the phenotype classification.

[0011] In some embodiments, the method may further include any of the following additional aspects, which may be combined with one another in any combinations:• the digital histopathology image includes an image of a hematoxylin and eosin (H&E)- stained slide;• the digital histopathology image includes an image of a Perl’s Prussian blue, Alcian blue, Masson’s trichome, Reticulin stain, silver stain, periodic acid Schiff Reaction (PAS)Stain, Congo red, Mucicarmine, Sudan black, Nissl stain (also called the Cresyl Violet Stain), or Golgi method stained slide;• wherein the digital histopathology image includes a whole slide image (WSI) or near WSI and the set of tiles collectively represents the entire area of the WSI or near WSI;• each tile represents no more than 5% of total area of the digital histopathology image;• the phenotype classification is a binary classification;• the phenotype classification is a graduated classification;• the pre-trained feature extractor includes a pre-trained feature extractors according to any of the above methods;• the phenotype includes a cancer pathophysiology;• the phenotype includes at least one of: tertiary lymphoid structures, B-cell proliferation, T-cell mediated cytotoxicity, FoxP3-mediated immunosuppression, antigen processing and presentation, tumor associated fibroblasts, angiogenesis, epithelial-mesenchymal transitions, tumor cell invasiveness, fatty acid metabolism, glycolysis, and oxidative pho sphory lation ;• further including providing the phenotype classification to a clinical who, based upon the phenotype classification undertakes a treatment regimen or prescribes further diagnostic assays for a patient having a tissue used to create the medical diagnostic image.

[0012] The disclosure further provides a system for classifying a digital histopathology image with respect to a phenotype, the system including one or more processors and a non-transitory computer-readable medium including one or more sequences of instructions that, when executed by the one or more processors, cause the system to perform operations of any of the above methods.

[0013] The disclosure further provides a method of identifying a biological mediator associated with a pathology, pathophysiology, or other clinical characteristic, the method including: creating a set of tiles for each image in a dataset of digital histopathology images, wherein each tile represents an area of a digital histopathology image that is less than the whole digital histopathology image, wherein each image is labeled with respect to the pathology, pathophysiology, or other clinical characteristic, and wherein the dataset includes digital histopathology images with at least two different labels for the pathology, pathophysiology, or clinical characteristic; extracting a features from each tile using a pretrained feature extractor to create a set of tile-specific feature vectors for each digitalhistopathology image; providing the sets of tile-specific feature vectors and labels to a DL model; assessing the sets of tile-specific feature vectors and labels using the DL model to identify differences between the images based on the labels and to provide a set of attention scores or an attention map associated with areas of interest in at least a portion of the digital histopathology images; analyzing a tumor tissues corresponding to the areas of interest for at least some of the digital histopathology image for a difference in a characteristic that correlates with a difference in the labels; and identifying the characteristic as a biological mediator if a difference correlating with the difference in labels exists.

[0014] In some embodiments, the method may further include any of the following additional aspects, which may be combined with one another in any combinations:• the biological mediator is a biological mediator of therapeutic resistance;• the DL model includes a phenotype-trained DL model trained accordingly to any of the above methods using a tissue type of the digital histopathology images;• the DL model identifies between 60% and 80% of the digital histopathology images on average, as not an area of interest;• analyzing the tumor tissues corresponding to the areas of interest includes in situ analysis of tissue sections adjacent tissue sections used to create the digital histopathology images;• the in situ analysis includes analysis of the genome, transcriptome, or proteome of the tissue;• the pathology, pathophysiology, or other clinical characteristic includes whether the patient responds to a particular therapy;• the pathology, pathophysiology, or other clinical characteristic includes a characteristic relating to a cancer, infection, or autoimmune disease;• each digital histopathology image includes an image of a hematoxylin and eosin (H&E)- stained slide;• each digital histopathology image includes an image of a Perl’s Prussian blue, Alcian blue, Masson’s trichome, Reticulin stain, silver stain, periodic acid Schiff Reaction (PAS) Stain, Congo red, Mucicarmine, Sudan black, Nissl stain (also called the Cresyl Violet Stain), or Golgi method stained slide;• each digital histopathology image includes a whole slide image (WSI) or near WSI and the set of tiles collectively represents the entire area of the WSI or near WSI.;• each tile represents no more than 5% of total area of the digital histopathology image;• the pre-trained feature extractor includes a pre-trained feature extractors according to any of the above methods.

[0015] The disclosure further provides a system for identifying a biological mediator associated with a pathophysiology using a digital histopathology image exhibiting the pathophysiology, the system including one or more processors and a non-transitory computer-readable medium including one or more sequences of instructions that, when executed by the one or more processors, cause the system to perform operations including any of the above methods.

[0016] The disclosure further provides a non-transitory computer-readable medium storing a deep learning (DL)-based classifier trained to classify a digital histopathology image with respect to a phenotype, wherein the DL-based classifier was trained by a process including: creating a dataset of identified digital histopathology images including, for each identified digital histopathology image, an image and a phenotype ground truth label that reflects a known phenotype classification for the image; creating, for each identified digital histopathology image, a set of image- specific tiles, wherein each tile represents an area of the identified digital histopathology image that is less than the whole digital histopathology image; extracting a features from each tile using a pre-trained feature extractor to create a tile-specific feature vector for each tile; assembling a set of tile-specific feature vectors for each identified medical image; providing, for at least a subset of identified medical images, the set of tile- specific feature vectors and the phenotype ground truth label to an untrained DL model; training the DL model in a self-supervised manner using the provided sets of tilespecific feature vectors and phenotype ground truth label to classify digital histopathology images with respect to the phenotype; assessing accuracy of the DL model in classifying digital histopathology images with respect to phenotype; and comparing assessed accuracy to an accuracy threshold and, if the accuracy threshold is met, storing the DL model.DESCRIPTION OF THE FIGURES OF THE DRAWING

[0017] Some embodiments of the disclosure are described herein with reference to the accompanying figures. The description, together with the figures, makes apparent to a person having ordinary skill in the art how some embodiments of the disclosure may be practiced. The figures are for the purpose of illustrative discussion and details of embodiments that are not necessary for a fundamental understanding of the teachings of the disclosure may beomitted. The figures are not drawn to scale and where dimensions are provided, they are merely exemplary and do not limit the scope or spirit of the disclosed invention.

[0018] Fig. 1 is a schematic diagram of sources of patient data that may be used in a DL-based classifier. Dashed lines represent optional additional data that may be used by the DL-based classifier. A representative area of interest in histopathology slides 1 and 2 and the in situ analysis sample is identified in black.

[0019] Fig. 2 is a schematic workflow illustrating how a digital histopathology image may be classified with respect to a phenotype.

[0020] Fig. 3A is a flowchart of a method of training a DL model to classify digital histopathology images with respect to a phenotype. Fig. 3B is a flowchart of a DL model training step in an embodiment in which the DL model includes a neural network and a final classifier.

[0021] Fig. 4 is a flowchart of a method of classifying a tissue sample image using a phenotype-trained DL-based classifier.

[0022] Fig. 5 is a flowchart of a method of identifying a biological mediator using attention scores from a DL model.

[0023] Fig. 6 is a schematic diagram of a tile-shifting data augmentation technique.

[0024] Fig. 7 is a schematic diagram of a color- augmentation data augmentation technique.

[0025] Fig. 8 is a schematic workflow illustration how a digital histopathology image may be classified with respect to phenotype using tile-shifting data augmentation, coloraugmentation, randomized feature vector augmentation, and random feature vector selection augmentation.

[0026] Fig. 9 is a set of digital histopathology images of H&E- stained breast cancer tumor sections and corresponding schematic diagrams illustrating three different immune- response phenotypes that may be classified using a DL-based classifier.

[0027] Fig. 10 is a set of annotated digital histopathology images of H&E-stained breast cancer tumor sections overlaid with attention maps generated using a DL model.

[0028] Fig. 10A is an image generated using a DL model to classify T cell-mediated cytotoxicity. Red zones and blue zones indicate high and low attention scores, respectively. On the right, two exemplary patches are highlighted, one corresponding to a portion of the image with abundant tumor-infiltrating lymphocytes without tumor cells, which aresuggestive of high immune activity (and which corresponds with a high attention score in the overlay), and the other corresponding to a portion of the image with areas of tumor necrosis and minimal lymphocytes, consistent with low immune activity (and which corresponds with a low attention score in the overlay).

[0029] Fig. 10B is an image generated using a DL model to classify epithelial- mesenchymal transition. Red zones and blue zones indicate high and low attention scores, respectively. On the right, two exemplary patches are highlighted, one corresponding to a portion of the image with irregular nests of invasive tumor cells (and which corresponds with a high attention score in the overlay), and the other to a portion of the image that is stromal tissue (and which corresponds with a low attention score in the overlay).

[0030] Fig. 10C is an image generated using a DL model to classify angiogenesis. Red zones and blue zones indicate high and low attention scores, respectively. On the right, two exemplary patches are highlighted, one corresponding to a portion of the image with scattered tumor cells and a possible blood vessel (and which corresponds with a high attention score in the overlay), and the other to a portion of the image with scattered lymphocytes (and which corresponds with a low attention score in the overlay).

[0031] Fig. 10D is an image generated using a DL model to classify immunosuppression. Red zones and blue zones indicate high and low attention scores, respectively. On the right, two exemplar patches are highlighted, one corresponding to a portion of the image with tumor cells with adjacent lymphocytes (and which corresponds with a high attention score in the overlay), and the other to a portion of the image with stromal tissue (and which corresponds with a low attention score in the overlay).

[0032] Fig. 10E is an image generated using a DL-based classifier to classify T-cell cytotoxicity, epithelial-mesenchymal transition, and angiogenesis. The WSI with attention map is shown in the left. Tiles with highest and lower attention scores and randomly sampled tiles that were further evaluated by human pathologists are shown in the right.DETAILED DESCRIPTION

[0033] The invention comprises, consists of, or consists essentially of the features disclosed here and in the Figures, in any combination.

[0034] Disclosed here are methods and systems for using DL-based classifiers to classify a tissue phenotype in a digital histopathology image, as well as methods of trainingsuch classifiers and trained DL-based classifiers. The present disclosure further relates to methods and systems for using DL-based classifiers or DL models to locate areas of interest in digital histopathology images that may then be used to identify biological mediators associated with the digital histopathology images or a pathophysiology reflected in the images.

[0035] Information not readily ascertainable by human medical professionals is available in histopathology images, but has been neglected. For example, Omics-based studies have established that tumor architecture and spatial patters of tumor or immune cells provide information that could be clinically useful. This information is present in many standard histopathology slides, but is not used because humans cannot determine this information from a histopathology slide, or, when such information can be determined by pathologists, human assessments based on the information exhibit too much variation or inaccuracy to be clinically useful, or take an impractical amount of time. Overall, there is currently a great deal of therapeutically relevant information in histopathology slides, even those prepared according to very standard and well-established protocols, that is not being used. The present disclosure provides systems and methods that allow the use of this information in clinical decision making, biological mediator discovery, or both.

[0036] A DL-based classifier, in some embodiments, includes a pre-trained feature extractors and a DL model, which includes a neural network. In some embodiments, the DL- based classifier also includes a final classifier.

[0037] DL models of the present disclosure may be trained using “weakly supervised” training. In particular embodiments, the weakly supervised training may employ inexact supervision, in which a label is applied to slide used to generate the digital histopathology image, but the label may not apply to every portion of the image. For example, a slide may be labeled as containing angiogenesis, or a from a cancer patient who did not respond to a drug, but many tiles representing portions of the image may contain no evidence of angiogenesis, or no evidence relevant to the patient’s non-responder status. These portions of the slide that the label does not apply to constitute noise when considered in the context of the labeled characteristic. In some embodiments, the weakly supervised DL model training may use multiple-instance learning. For example, when training data is derived from digital histopathology images and labels applied to those images by pathologists, due to errors by the pathologists, the labels may not always accurately reflectthe pathophysiology of the tissue in the image, or the pathology of the patient. However, when provided with a large data set, such as a data set including hundreds or thousands of labeled digital histopathology images, DL models may be trained to accurately classify digital histopathology images with respect to phenotypes, even though data from some portions of the images is noise in the context of certain phenotypes or pathophysiologies.

[0038] In weakly supervised training, the DL model is not provided with indicators of the types of information or features it should locate within the digital histopathology images. For example, the DL model is not provided with indicators directing the model to detect numbers of cells, cell shapes, nuclear features, or other intracellular features. Such features have been used to train other models in a more supervised and directed manner. However, the use of such features limits a model to features that humans can observe, and also imports other human biases into the ultimate classifications. DL models trained in a weakly supervised manner of the present disclosure are not bound by what humans can detect in an image, and, therefore, produce results that more accurately reflect the true biology displayed in the digital histopathology image. This reflection of the true biology also allows the identification of new and under-recognized biological features, such as new biological mediators.

[0039] In specific embodiments, DL models trained to classify or used to detect biological mediators in digital histopathology images may be provided only with a patientlevel pathology label, such as whether the patient does or does not have a disease, a clinical label, such as whether the patient has or has not responded to a treatment, or a disease state label, such as the stage of a cancer. In some embodiments, the DL models may not be provided with any label specific to a pathology or pathophysiology only determinable or, according to standard medical practices of this time, typically determined using histopathology or other microscopy-based diagnostics. In other embodiments, the DL models may not be provided with any label specific to a pathology or pathophysiology only determinable from a patient tissue sample, particularly an invasively-obtained patient tissue sample, such as a tumor or biopsy.

[0040] DL-based classifiers trained to classify images with respect to phenotypes, in some embodiments, may be used to provide therapeutically relevant information about a particular digital histopathology image from a patient.

[0041] DL-based classifiers or DL models, in other embodiments, may be used to identify biological mediators associated with a pathology or pathophysiology, such as biological mediators of therapeutic resistance. The biological mediators may have not previously been correlated with the pathology. In some embodiments, the DL models may have been trained prior to use in identifying a biological mediator. In other embodiments, the DL models are presented with information corresponding to digital histopathology images in a format similar to what would be used in training such a DL model to classify a phenotype, but, rather than being specifically trained to classify phenotypes, the DL model is instead used to provide attention scores or areas of interest in the digital histopathology images that are useful in identifying biological mediators. In some embodiments, the biological mediator may be identified without any a priori knowledge of the mediator or its properties other than association with a therapeutic resistance.

[0042] As used herein, a “phenotype” refers to the properties of a cell or tissue expressed in a digital histopathology image, such as the pathophysiology of the cell or tissue. Cells and tissues also have associated “genotypes” which reflect the genes their DNA contains in its genome. However, many genes in a cell or tissue are not active, or they may vary in their levels of activity. Additionally the genes, and particularly mutated forms of genes, found within a tumor may vary in different parts of the tumor, a phenomenon referred to as tumor heterogeneity. As a result, knowing the genotype of a cell or tissue, particularly a tissue as a whole, or a tissue representing only part of a tumor, is often not particularly therapeutically relevant.

[0043] Fig. 1 provides a general description of how diagnostic data from a tumor may be gathered and used according to embodiments of the present disclosure. As indicated by solid arrows, a tumor (or a biopsied part of a tumor) is resected from a patient and sliced into tissue sections. At least one tissue section is used to create a histopathology slide (histopathology slide 1), from which a digital histopathology image is created and provided to a trained DL-based classifier for classification with respect to a phenotype relevant to the tumor.

[0044] In some embodiments in which the DL-based classifier is being trained, the digital histopathology image and any other diagnostic data, including data relevant to a particular pathophysiology the patient exhibits, is provided to the untrained DL-based classifier, in some embodiments, along with similar digital histopathology images anddiagnostic data from other patients, some of whom do not exhibit the pathophysiology, or who exhibit variants of the pathophysiology. The this set of information is used by the DL model to train itself in a weakly supervised manner to identify at least one phenotype associated with the pathophysiology and accurately classify further digital histopathology images with respect to that phenotype.

[0045] In some embodiments where biological mediators are identified, the digital histopathology image is from a patient having one pathophysiology and is processed in the same manner as in a DL-based classifier to provide information to a DL model of the type used in the DL-based classifier, but rather than classify the image, the DL model uses the image, along with images from other patients having a different pathophysiology, to determine attention scores or areas of interest that indicate biological mediators relevant to the pathophysiology difference.

[0046] In some embodiments, the DL-based classier may be provided with one or more additional types of data about the patient or the tumor obtained via modes other than digital histopathology (or at least the original digital histopathology image protocol) to facilitate multi-modal analysis. In particular embodiments, the DL-based classifier may be provided with a digital histopathology image from a second histopathology slide (histopathology slide 2) that was prepared according to a different histopathology protocol than the first histopathology slide. In other particular embodiments, the DL-based classifier maybe provided with clinical metadata, such as patient metadata or the results of other diagnostics that did not use the tissue sample, such as radiology images. In still other particular embodiments, the DL-based classifier may be provided with molecular metadata obtained from the tumor, such as via whole tissue analysis or in situ analysis. In situ analysis may allow molecular metadata to be matched to specific locations within the digital histopathology image.

[0047] In situ analysis may be particular helpful in embodiments in which a DL model is used to detect biological mediators. Transcriptome or proteome analysis, or targeted analysis for certain proteins or molecular markers may be applied to slices of the tumor adjacent to or very near areas of interest in the digital histopathology image.

[0048] Embodiments directed to multi-modal models may be more accurate or may be better able to identify biological mediators than single-modal models. Multi-modal models may differ in terms of when data representing modes other than the digitalhistopathology images are introduced in training and / or assessment using a trained multimodal DL-based classifier. In some embodiments, the data representing other modes may be introduced prior to training the DL model, prior to final classification by a final classifier, or in a cross-modal or cross-attention classifier.Tissues and Phenotypes

[0049] DL-based classifiers of the present disclosure may be trained to recognize a phenotype in tissues that exhibit or affected by a pathophysiology (such as a particular pathology). In some embodiments, the tissue may be of any type for which a histopathology slide can be created. For example, the tissue may be a tumor. In some embodiments, the tissue may include pathological tissue, such as a tumor, and surrounding tissue, such as normal tissue in the vicinity of the tumor.

[0050] During training, in some embodiments, all of the digital histopathology images in the training dataset may be from patients having a pathology of interest, but differing in another characteristic of interest that may be reflected in pathophysiology differences, such as stage of the pathology, or response to a particular therapy. In other embodiments the digital histopathology images in the training dataset may include images of the same type of tissue in patients who do not have the pathology of interest. For example, a training dataset for breast cancer may include images of tumors and surrounding breast tissue, as well as images of healthy breast tissue from patients who do not have tumors.

[0051] In some embodiments of the present disclosure, particularly those associated with DL-based classifiers trained to classify phenotypes associated with cancers, the cancers may be any types that form solid tumors, including both primary tumors and metastatic tumors. Given that many different cancerous solid tumors and the tumor microenvironment share phenotypes relevant to the pathology, such as angiogenesis, immune activation, and invasiveness, training methods that allow DL-based classifiers to classify phenotypes for the example cancers of this disclosure may also be used to train DL-based classifiers to classify phenotypes for other cancers that form solid tumors.

[0052] In some embodiments, therapeutically relevant pathophysiologies may be associated with certain phenotype classifications. Systems and methods using DL-based classifiers may, in some embodiments, provide only a phenotype classification to the user, leaving a medical professional to connect the phenotype classification to any therapeuticallyrelevant pathophysiology. In other embodiments, the systems and methods, and, in particular embodiments, the DL model itself, may use an association between a phenotype classification a pathophysiology to provide the user with an assessment or classification of the pathophysiology as well. In some embodiments, the phenotype may be indicative of the tumor microenvironment (TME) of a cancerous tumor. The TME often affects whether the cancer is likely to respond to immune-based therapies, or the TME may indicate how aggressive the cancer is.

[0053] In some specific embodiments in which the therapeutically relevant pathophysiology is a cancerous tumor, an associated phenotype may be an immune phenotype. Certain immune phenotypes, including tertiary lymphoid structures, B-cell proliferation, T-cell mediated cytotoxicity, FoxP3-mediated immunosuppression, and antigen processing and presentation may all be classified for a digital histopathology image using a DL-based classifier of the present disclosure. All such immune phenotypes are linked to the therapeutic efficacy of radiation and / or checkpoint inhibitors. The immune phenotypes that are reflected in the presence of specific immune cells of specific types may be referred to as immune cell presence-based phenotypes.

[0054] In addition, the TME may include phenotypes that are not reflected in immune cells themselves, but are reflective of the effects of immune cells on other TME cells or the presence of other cells in the TME that may affect the function of any immune cells also in the TME, or recruitment of immune cells to the TME. For example, the presence of tumor associated fibroblasts is linked to immunosuppression. These immune cell-induced or immune cell-reflective phenotypes are also subgroups of immune phenotypes.

[0055] In other specific embodiments in which the therapeutically relevant pathophysiology is a cancerous tumor, an associated phenotype may be a tumor phenotype. Certain tumor phenotypes, such as angiogenesis, epithelial-mesenchymal transitions, and tumor cell invasiveness are related to the co-localization of different cell populations. Other tumor phenotypes, such as cell cycling, are related to mitotic figures or other intracellular image information. Tumor phenotypes are linked to tumor aggressiveness, metastasis, and resistance to oncogene-directed therapies. Choice of therapeutic regimen may be based at least in part, on tumor phenotypes, with a more aggressive therapeutic regimen being adopted when the cancer appears aggressive, invasive, or metastatic.

[0056] In other specific embodiments in which the therapeutically relevant pathophysiology is a cancerous tumor or other cancer cell, an associated phenotype may be a metabolic phenotype. Certain metabolic phenotypes, such as fatty acid metabolism, glycolysis, and oxidative phosphorylation are also linked to tumor aggressiveness and to therapeutic resistance for many systemic therapies.

[0057] In some embodiments, DL-based classifiers may also be trained to classify phenotypes associated with cancers that do not form solid tumors, such as blood cancers. The DL-based classifiers may also be used to classify phenotypes of solid tumor cancer cells no longer in their solid environment, such as circulating tumor cells in blood, or tumor cells shed in bodily wastes or secretions. Although digital histopathology images of such tumors may not contain some types of information that are specific to solid tumors, such as information relating to angiogenesis, for example, many other phenotype indicators that can be detected through histopathology, such as some cell cycling and many metabolic phenotypes, are similar in both solid and non-solid tumors, and are found in solid tumors detached from their solid environment. Features indicative of these phenotypes are expected to be similar across cancers, such that training methods that allow DL-based classifiers to classify phenotypes for the example cancers of this disclosure may also be used to train DL- based classifiers to classify phenotypes for other cancers that are not solid tumors, or for tumor cells that are no longer located in a solid tumor.

[0058] Furthermore, in some embodiments, DL-based classifiers may also be trained to classify phenotypes associated with non-cancerous pathologies that share phenotypic features with solid tumors. For example, DL-based classifiers may be trained to classify phenotypes associated with pathologies, such as infections or auto-immune diseases, where immune activation can be detected in digital histopathology images.

[0059] In specific embodiments, DL-based classifiers may be trained to use digital histopathology images of samples obtained via lung bronchoscopy or an abscess biopsy to determine whether an infection is present or properties of the infection, such as whether an immune response is occurring or details of such response, particularly whether the response is adequate to clear an infection, or whether the immune response is being adequately assisted by a given treatment, such as an antibiotic. The early ability to identify whether an infection is likely to respond to a given treatment, or whether a treatment is having an effect can greatly reduce the duration or danger of many infections, particularly in vulnerable patients orthose with multi-drug resistant infections. The presence of or type of immune cells, or properties indicating immune response can be detected in solid tumors, and are, therefore, also detectable in samples from infected tissues.

[0060] Given the ability of DL-based classifiers to recognize patterns and features not readily distinguishable by a human, in some embodiments, DL-based classifiers trained to recognize infection-related phenotypes may recognize and use information that is not readily usable by human pathologists. For example, the ability to DL-based classifiers to detect cell cycle-related phenotypes or metabolic phenotypes indicates an ability to detect and use very small features, such features present in bacterial cells, which are typically much smaller than mammalian cells. Although it is not necessary to know exactly what sort of features are being detected by the DL-based classifiers, features indicative of bacterial type beyond large- scale features, such as whether the bacteria is Gram-positive or negative, may be detected and used to classify infection-related phenotypes. Additionally, smaller features indicative of the efficacy of immune response, a treatment, or both, may be detected and used by DL-based classifiers. These smaller features might relate to disruption of a bacterial cell membrane or coat, which are common effects of antibiotics, or disruption of internal cellular processes. In the case of viral infections, smaller features might relate to indicators of whether viral replication within cells has been hampered, for example by disruption of protein synthesis or the beginning stages of apoptosis, even if individual viral proteins cannot be seen.

[0061] Smaller features similar to cell cycle and metabolic indicators in tumors might also include detecting whether the patient’s own immune cells have altered internal processes in a way that indicates activation or the cell, or engulfment of bacterial cells.

[0062] In other specific embodiments, DL-based classifiers may be trained to use digital histopathology images of samples from tissues affected or suspected to be affected by autoimmune diseases or other immunological-based dysfunctions. For example, tissues might include pancreatic tissues from patients with suspected Type I diabetes, which can be difficult to distinguish in patients who also exhibit Type II diabetes. Tissues might also include biopsies or resected portions of thyroid tissue from patients with thyroid autoimmune diseases. Samples from such patients might further indicate, for example, whether or when the presentation of such as disease switches, for example from causing hyperthyroidism to hypothyroidism. Samples might also indicate the likelihood or presence of associated malignancies, which are presently difficult to predict. Intestinal tissues might be useful inmore rapid or accurate distinction and diagnosis of various inflammatory bowel disease pathologies that often exhibit overlapping symptoms and involve an immune-response, such as Crohn’s disease, Celiac disease. Intestinal tissues might also prove useful in determining whether a primarily immune-based pathology, such as one of the above, or a primarily non- immune-based pathology, such as irritable bowel syndrome, is present. To the extent tissue samples and biopsies of affected tissues are reasonably obtainable, DL-based classifiers may be used to detect other autoimmune diseases, such as rheumatoid arthritis or lupus, which are presently difficult to diagnose in early stages and to monitor other than by very indirect methods, such as blood tests or radiology.

[0063] The ability of the DL-based classifier to detect and differentiate immune phenotypes in order to make determinations about any of the above autoimmune diseases relies on the same capabilities to detect immune phenotypes in solid tumors.Digital Histopathology Images

[0064] Digital histopathology images, in particular embodiments, may represent histopathology slides prepared from tissue samples. These slides may be prepared using any histopathological protocol designed to allow microscopic examination of a tissue sample. In some embodiments hematoxylin and eosin staining (often referred to as H&E staining or HE staining) is the histopathology protocol. Nearly every hospital pathology lab can prepare H&E-stained microscopy slides from a tissue sample. As a result, there exist large databases of H&E-stained slides and associated phenotype or pathophysiology classification labels. Access to large amounts of information of this type may facilitate DL-based classifier training. Additionally, the ability of a DL-based classifier to use slides prepared according to common protocols also facilitates use of the DL-based classifier in clinical applications, such as classifying phenotypes associated with therapeutically relevant pathophysiology, and in research applications, such as detecting new biological mediators.

[0065] Other histopathology slide preparation protocols, in some embodiments, may also be used to produce slides represented in digital histopathology images. These include staining with Perl’s Prussian blue, Alcian blue, Masson’s trichome, Reticulin stain, silver stain, periodic acid Schiff Reaction (PAS) Stain, Congo red, Mucicarmine, Sudan black, Nissl stain (also called the Cresyl Violet Stain), and Golgi method stain. In someembodiments, the stain may be of a type commonly used by pathologists to assess the therapeutically relevant pathophysiology.

[0066] In other embodiments the stain (or other elements of the histopathology protocol) may be different than that commonly used by pathologists. Such a stain may take advantage of the ability of DL-based classifiers to recognize and use information in digital histopathology images that is not recognizable or usable by humans without the assistance of a trained computational model. For example, such a stain may allow the DL-based classifier to better detect small features such as intracellular features or cell membrane features that a human simply cannot see or recognize when viewing a histopathology slide. In another example, such as stain may allow the DL-based classifier to more accurately classify a phenotype that is not normally associated with the stain, such as if a stain commonly used to detect infection allows more accurate classification of cancer response or stage. In another example, a protocol that results in staining too light or too dark, or with too subtle gradations of color to be useful to a human may improve accuracy of a DL-based classifier. As another example, such as stain may not cause a detectable visual difference in the light spectrum visible to humans, buy may cause a difference in a portion of the electromagnetic spectrum detectable by cameras, particularly the ultraviolet spectrum, that is still readily captured by a camera able to detect the relevant wavelengths. In either instance, the stain is not helpful to a human pathologist, but a reasonably predicted or known ability of the stain to bind to some cell types or cellular structures, coupled with a reasonably predicted or known ability to alter light transmission, or transmissions in another part of the electromagnetic spectrum detectable by a digital camera, indicates that such as stain might be useful in training a DL- based classifier.

[0067] In some embodiments, the DL-based classifier may be trained using and may, correspondingly, classify phenotypes using data from multiple slides from the same tissue sample (such as slides 1 and 2 in Fig. 1). In some particular embodiments, each slide may be prepared using a different histopathology protocol, such as a different stain.

[0068] Digital histopathology images may be obtained by using a microscope, particularly a microscope containing or coupled to a digital camera, to create a magnified digital image of a histopathology slide. DL-based classifiers may, in some embodiments, be trained to assess the whole slide image (WSI) of a digital histopathology image. In some embodiments, in place of a WSI, a near WSI may be used. A near WSI includes a largeportion of the digital histopathology image, such as at least 80%, at least 90%, at least 95%, or at least 99% thereof. In particular embodiments, the DL-based classifier, in particular the DL-model portion of the DL-based classifier, even when assessing the WSI, may locate portions of the digital histopathology image that are more likely to contain information relevant to the assessed phenotype(s), which may be referred to as areas of interest. These areas of interest may be determined by the DL model in the same manner as during DL model training as described below (although the DL model used for this assessment has typically already been trained), and may further be used in detecting biological mediators, also as described below. The DL-based classifier may then assign greater weight to these potions in subsequent attention-based analysis. This method allows the WSI to be used in phenotype classification of the tissue sample or biological mediator detection without devoting computational resources and time to portions of the image unlikely to substantially contribute to the classification or biological mediator detection, while still not wholly ignoring these portions of the image.

[0069] The histopathology slide represented in a particular digital histopathology image will be at a specific magnification. In some embodiments, the magnification may be a magnification commonly used by pathologists when examining histopathology slides, such as a magnification in a range between 2x and 400x. In specific embodiments, the magnification may be a low-power magnification in a range of between 2x and lOx, not including lOx, a medium-power magnification in a range between lOx and 40x, not including 40x, or a high- power magnification in a range between 40x and 400x. In some embodiments, two or more magnifications, such as 20x and 40x, may be used for the same histopathology slide to create two digital histopathology images of that slide. In some embodiments, both magnifications may be used for a WSI or near WSI. In other embodiments a lower magnification may be used for a WSI or near WSI, and a higher magnification for only a portion of the WSI or near WSI.

[0070] The DL-based classifier may be able to use digital histopathology images of different magnifications throughout training, classification or a given image by a trained DL- based classifier, or biological mediator detection. The pre-trained feature extractor, DL model, or both may be able to make adjustments to how features are measured and how feature vectors are used that can accommodate various magnifications. For similar reasons, the digital histopathology images do not have to be of the same image quality. For example,the digital histopathology images may differ in resolution, light levels, or staining quality. However, in some embodiments, some parameters, such as a minimal resolution or light level, maybe be set for the digital histopathology images, particularly images that will be classified by a trained DL-based classifier and used to obtain information used to treat a patient.DL-Based Classifier Training Methods

[0071] Certain image processing and computational techniques may be used in training methods disclosed herein and in phenotype classification of a tissue sample using a DL-based classifier. Some such techniques are illustrated in Figs. 2, 3A, and 3B.

[0072] In some embodiments, the disclosure provides a method 100 for training a DL-based classifier to classify a digital histopathology image, such as a digital histopathology image, with respect to a phenotype, as illustrated in the flowchart of Fig. 3A as supplemented, for some embodiments, by the flowchart of Fig. 3B.Creating Sets of Tiles for a Dataset of Identified Digital histopathology Images

[0073] In step 110 a set of image- specific tiles is created for each image in the dataset of identified digital histopathology images. In some embodiments, the image- specific tiles, collectively, represent the entire area of the WSI or near WSI.

[0074] A “tile” is an area of a digital histopathology image that is less than the whole image. In some embodiments, a tile may be an area of a digital histopathology image representing no more than 5% of the total area of the digital histopathology image. Tiles may also be referred to as “patches.” A set of tiles may be created from a single original digital histopathology image. In some embodiments, the set of tiles may contain no overlapping portions of the WSI or near WSI. In other embodiments, subsets of tiles may be created, for example by using tile-shifting as described below. In some such embodiments, a first subset of tiles may be created such that the first subset tiles, when compared to one another, contain no overlapping portions of the WSI or near WSI. Then a second subset of shifted tiles may be created so that the second subset tiles, when compared to one another, contain no overlapping portions of the WSI or near WSI. However, the first subset tiles and the second subset tiles will overlap with one another. For example, a tile in the first subset may overlap with one quarter, one third, one half, two thirds, three quarters, etc. with a tile in the second subset, depending on how tile-shifting is performed. Accordingly, in embodiments in whichtile-shifting is used as a form of data augmentation, the set of tiles may contain overlapping portions of the WSI or near WSI. However, the set of tiles may be formed from two or more subsets of tiles, each of which does not contain tiles that overlap with other tiles within the same subset. In some embodiments, a small amount of overlap, such as 5%, 2%, 1%, 0.1%, or 0.01% or less of the total area between tiles in a set or a subset may be acceptable, although this would result in overrepresentation of data corresponding to the overlapped portions, and substantial overlap may, in some embodiments, cause unacceptably long training or analysis times.

[0075] The dataset of identified digital histopathology images, such as digital histopathology images, are images for which a phenotype or pathophysiology of interest is known. In some embodiments, the dataset may include a sufficient number of images for the DL-based classifier to be trained to at least a selected level of accuracy, as described further below. Whether the number of images in the dataset is sufficient may be apparent by the lack of a selected level of accuracy after an epoch of training, which may indicate that additional identified images should be added to the data set for further training.

[0076] The known phenotype classification of an image in the dataset (such as a pathology or histopathology label associated with the image) may be used as the ground truth for that image when used in DL-based classifier training methods of the present disclosure.

[0077] In some embodiments the known phenotype classification of a given digital histopathology image may be a binary classification; the image either exhibits the phenotype or it does not. In other embodiments, the classification may be a graduated classification, representing degrees of exhibiting the phenotype. For example, the known phenotype classification might be a non-continuous graduation, such as number on a scale, such as a scale of 1-5 or 1-10. In other examples, the known phenotype classification might be a continuous graduation, for example, a measured number, such as an amount of RNA having a certain sequence, or the amount or activity of a particular protein. In some embodiments, a continuous graduation may be provided by regression analysis, such as linear regression or a Pearson’s correlation coefficient.

[0078] Known phenotype classifications used may be assigned, in some embodiments, based on pathologist assessments, clinical assessments of the patent from whom the tissue sample was obtained, or a direct measurement obtained using another type of assay or method, such as a gene expression assay.Although the following methods are described with respect to training a DL model to classify a digital histopathology image with respect to only one phenotype, in some embodiments the same methods may be used to simultaneously train a model to classify an image with respect to two or more phenotypes, such as between 2 and 50 phenotypes, between 2 and 25 phenotypes, or between 2 and 10 phenotypes. In some embodiments, the DL-based classifier is task invariant, meaning that it may be used to predict.Feature Extraction

[0079] In step 120 a pre-trained feature extractor extracts features from each tile to create a tiles- specific feature vector for each individual tile. These feature vectors form a set of tile-specific feature vectors that, together, represent to original digital histopathology image, such as the WSI or near WSI.

[0080] The number of tiles formed from a WSI or near WSI may vary depending on what is practical in terms of training and use of the DL-based classifier. However, in some embodiments the WSI or near WSI may be broken into at least 10,000 tiles, at least 20,000 tiles, at least 30,000 tiles, at least 40,000 tiles, at least 50,000 tiles, or tiles numbering in a range between 10,000 and 70,000, between 10,000 and 50,000, between 20,000 and 70,000, between 20,000 and 50,000, between 30,000 and 70,000, between 30,000 and 50,000, between 40,000 and 70,000, between 40,000 and 60,000, between 40,000 and 50,000, between 50,000 and 70,000, between 50,000 and 60,000, or between 20,000 and 48,000.

[0081] A “feature” may be any low dimensional representation of the tile. A low dimensional representation is a way of expressing a high-dimensional representation in a simplified form that still indicates important characteristics of the high-dimensional representation. The tile, which contains many data points, is a high-dimensional representation. A low dimensional representation might, for example, express a common property of hundreds of data points as a single data point. Low dimensional representations of the present disclosure may be based on far more data points than a human could readily recognize and draw conclusions about. Furthermore, because the feature extractor selects features in an unsupervised manner, most features will not be readily recognized or understood by a human. In some embodiments, a cutoff for each feature is established, such that whether the image exhibits the feature is a binary determination. In other embodiments, regression may be used to provide graduated information, in more specific embodiments with multiple cutoffs.

[0082] Information about features in a given tile is used to construct a feature vector for that tile. A “feature vector” is a string of numeric data organized in an order that allows comparison between tiles. Using the binary feature determination example above and the illustration in Fig. 2, if a tile exhibits a feature (for example, the feature designated by a crescent), the feature vector contains 1 as the data for that feature (illustrated as a grey background). If a tile does not exhibit that feature, the feature vector contains 0 as the data for that feature (illustrated as a white background). So, if a feature vector contains data for features A, B, C, and D in that order, and a first tile exhibits only feature B, then the feature vector for that first tile is 0010. If a second tile exhibits features A and C, then the feature vector for that second tile is 1010. The organization of features into feature vectors by the feature extractor allows the DL model to more readily and effectively compare data from different tiles.

[0083] In some embodiments, in step 120, the feature extractor may be pre-trained. A feature extractor contains instructions for the identification of features and the creation of feature vectors. In a pre-trained feature extractor, these instructions were developed by training the feature extractor to extract features from images in a non task- specific manner (z.e. the training did not include any determinations based on whether the feature extractor component extracted features that improved the ability of the DL model to classify the phenotype of interest). In some embodiments, the pre-trained feature extractor may have been pre-trained with an array of pathology images and text annotations (labels) that are not limited to the types of tissues, pathophysiology, or histopathology protocols represented in the known digital histopathology images. Feature extractors pre-trained with such pathology - specific images, rather than an image set containing varied types of visual information, such as general images found on the web, may allow more effective or efficient training of the DL model.

[0084] In some embodiments, the pre-trained feature extractor may be or have features similar to a vision transformer trained using contrastive learning with paired pathology image-text descriptions. A vision transformer is a DL-based model architecture with improved performance as compared to a convolutional neural network. One example such feature extractor is PLIP, which uses 210,000 paired pathology image-text descriptions from Twitter in the OpenPath dataset. Another example of such a feature extractor is CONCH, which uses over 1.1 million paired pathology image-text descriptions from tertiarycare centers. Both CONCH / PLIP use an image encoder and a text encoder, trained using a number of contrastive alignment objectives that seeks to align the image and text in the same representation.

[0085] One example of a feature extractor trained using an image set containing general visual information is ResNet50, which is a convolutional neural network trained in baseline ImageNet.

[0086] Although even vision transformer-based feature extractors trained using general visual information, convolutional neural network feature extractors trained using pathology images, and convolutional neural network feature extractors trained using general visual information may normally result in lower accuracy of DL models trained using feature vectors from such feature extractors, data augmentation, as discussed below, used in conjunction with these types of feature extractors may improve DL model accuracy.

[0087] A set of feature vectors containing the tile-specific feature vectors may be assembled for each image in the dataset of identified digital histopathology images.Training the DL Model

[0088] In step 130, the DL model is trained by providing, for at least a subset of images in the dataset of identified digital histopathology images, a set of tile-specific feature vectors and a phenotype ground truth label. The DL model uses this information to undertake weakly supervised training to classify images with respect to the phenotype.

[0089] DL models use a neural network-based architecture in which there are multiple layers, with each layer containing multiple nodes. An example of such as model is depicted in Fig. 2. DL models contain a visible input layer and a visible output layer, along with a large (deep) number of hidden computational layers between the input and output layers. Each computational layer takes information from the previous layer, performs computations, and sends different information, based on those computations, to the next layer. This process is referred to as forward propagation. Typically each computational layer in the model builds on the previous layer to refine the ultimate prediction, which is or directly relates to the phenotype classification.

[0090] During training, DL models also implement a process, referred to as backpropagation, in which errors in predictions are calculated and used to adjust weights and biases of the layers backwards through the DL model to train the model. During training, forward propagation and backpropagation allow the model to make predictions and correctfor errors. DL models are trained via iterations. An iteration is a single cycle in which the DL model processes a given dataset, evaluates its predictions, and corrects for errors. In some embodiments, an iteration is one forward propagation and subsequent back propagation. The total number of iterations required such that every piece of data from the original dataset has been used in at least one iteration is referred to as an epoch.

[0091] Nodes in computational layers are individual units in which a computation is performed. Nodes in a DL model are assigned weights and biases. A bias is a constant associated with a node that is not connected to any input information, but that is affects the output information. A weight is numerical value associated with connections between nodes different computational layers that adjust the impact of information coming from one node on the calculations of the other node.

[0092] During the training process, in some embodiments, the DL model may refine accuracy primarily or even solely by adjusting weights between nodes. In some embodiments, the weights assigned between nodes ultimately affect the importance that different tiles from a given image will have on the ultimate phenotype classification for the digital histopathology image. This effectively assigns weights to tiles within the digital histopathology image.

[0093] In some embodiments an attention-based DL model is used in the training process. In particular embodiments, the DL model may be a single-branch attention (AM- SB) model, which uses one mechanism to assign weights to tiles within a digital histopathology image. In other particular embodiments, the DL model may be a multiplebranch attention (AM-MB) model, which uses multiple mechanisms to assign weights to tiles. In some embodiments, single-branch attention and multiple-branch attention models may be similarly effective in training.

[0094] In some embodiments employing an attention-based DL model, such as AM- SB or AM-MB, the model uses attention-based pooling to aggregate tile-level representations, weighted by the attention score, into slide-level representations. This slidelevel representation is the input to a final classification component. In such embodiments, as illustrated in Fig. 3B, step 130 may be split into steps 130a and 130b. In step 130a, a set of tile-specific feature vectors is provided to the DL input layer and used by the neural network to generate an output layer. Then, in step 130b, the DL output layer is provided to a final classifier that classifies the image with respect to phenotype.

[0095] In a more specific embodiment, for each image or a selected subset of images from the dataset of known digital histopathology images, the set of feature vectors is provided to the input layer for self- supervised training of an attention-based DL model, which aggregates and weights by attention score the tile-level feature vectors to generate a slidelevel representation, for example in the DL output layer. Then in step 130b, the slide-level representation for each image is provided to a final classifier, along with a ground truth phenotype label, for self- supervised training. In some embodiments, the final classifier may also be a neural network.

[0096] In some embodiments other methods of refining accuracy may be used. In particular embodiments, the DL model may use a vision transformer model that accounts for spatial correlation and context between patches, such as Transformer- multiple instance learning (Trans-MIL) or Hierarchical Image Pyramid Transformer (HIPT). In some embodiments, however, such models, which require more computational time than attentionbased models, may not offer a significant improvement in accuracy.

[0097] In some embodiments, when a certain dataset of identified digital histopathology images is used to train a DL model, a portion of the dataset may be used for training, during which the model is developed. A portion may be used for validation, a process during which a preliminary trained model is presented with the validation dataset and uses information generated while assessing the validation dataset to further change the DL model to improve accuracy. Although the validation dataset is used to refine the DL model, accuracy may also be calculated for the validation dataset. Another portion of the complete dataset may be used for testing, which is a process during which the DL model is used to classify images in the dataset and its accuracy is assessed. The test dataset is typically not used to further refine the DL model.

[0098] In some embodiments, areas of the digital histopathology image for which tiles are assigned higher weights may be referred to as having a “high attention score.” Areas for which tiles are assigned lower weights may be referred to as having a “low attention score.”

[0099] In some embodiments, the DL model may use attention scores to generate attention maps, which show, on the digital histopathology image, the relative attention scores for different areas. Attention maps allow a human monitor to know which areas have a high attention score. Although, in self-supervised learning, the DL model does not receive any 1input from the human monitor as part of the training process, the human monitor can, nevertheless, gain some insight into whether the model appears to be assigning a high attention score to areas that are likely to contain information about the phenotype or therapeutically relevant pathophysiology.

[0100] Verification by a human monitor that the DL model is assigning attention scores appropriately, although not an objective measure of accuracy, may, nevertheless, provide additional assurance that the model is using biologically relevant information to make predictions.Accuracy Assessment

[0101] In step 140, accuracy of pathophysiology classifications by the DL-based image assessment model is determined. Then, in step 150, the accuracy is compared to an accuracy threshold. If the accuracy meets a selected accuracy threshold, no further training is needed and the DL model may be stored and then later used as the DL model in a phenotype-trained DL-based classifier for other purposes described herein, such as classifying a digital histopathology image of a patient tissue sample. If accuracy does not meet a selected threshold, in some embodiments, the method returns to step 130 and training is repeated. In some embodiments, biases or architecture of the DL model may be altered before training is repeated.

[0102] In some alternative embodiments in which additional identified digital histopathology images are introduced for further training, if accuracy does not meet the selected accuracy threshold, then the method may return to step 110 instead of step 130, to allow the inclusion of the additional images. In some more particular embodiments, steps 110 and 120 are then performed only for the newly added identified digital histopathology images, and previously assembled sets of tile-specific feature vectors for each identified digital histopathology images previously included in the dataset are simply reused.

[0103] In some embodiments, in step 140, accuracy is assessed using a mathematical calculation. Typically, to calculate accuracy, a testing dataset of identified digital histopathology images is assessed by the phenotype-trained DL-based classifier and whether the classifier correctly classifies each image in the testing dataset is recorded, along with information about the correct phenotype classification (e.g., in a binary system, whether the image is positive or negative for the phenotype).

[0104] In some embodiments, the testing dataset may be a portion of the original dataset of identified digital histopathology images, where the remainder or another portion of the original dataset was used to train the model. In other embodiments, the testing dataset may be from a different dataset of images for the same tissue or pathophysiology.

[0105] In some embodiments, accuracy is assessed by determining the area under receiving operator curve (AUROC). In a binary system, the AUROC indicates the DL model’s ability to discriminate between images exhibiting the phenotype and images that do not exhibit the phenotype. In some embodiments, the selected accuracy threshold for AUROC in step 150 may be at least 0.7, at least 0.8, at least 0.85, at least 0.9, or at least 0.95.

[0106] AUROC is a valuable method or assessing accuracy of many types of trained DL models. However, AUROC is known to provide misleadingly high assessments of accuracy of models that are trained with data in which negative examples greatly outnumber positive ones. Accordingly, in some embodiments, at least one other assessment of accuracy may be used in place of or in addition to AUROC.

[0107] In some embodiments, AUROC, the specificity (true positives) and / or sensitivity (true negatives) Positive Predictive Value; Negative Predictive Value, other ratios reflecting the relative likelihood of these accuracy measurements, and / or any combinations of these accuracy measurements may be calculated in step 140. In each case, the selected accuracy threshold in step 150 for the accuracy measurement may be at least 0.7 (or 70%), 0.8 (or 80%), 0.9 (or 90%), 0.95 (or 95%), in a range between any of the preceding and 0.75 (or 75%), 0.8 (or 80%), 0.9 (or 90%), 0.95 (or 95%), or 0.99 (or 99%).

[0108] Suitability of a trained DL-based classifier may also, in some embodiments, be based on whether the classifier exhibits generalizability of its phenotype classification(s) to unknown test sets. In a specific embodiment, a trained DL-based classifier may be suitable for use in biological mediator identification if the trained DL-based classifier achieves a classification accuracy with an unknown test set that does not exhibit statistically significantly variation from the accuracy of that classifier with an internal test set (a test set derived from the same dataset used to train the classifier, typically representing 10% of the dataset used to train the classifier).Data Augmentation

[0109] Data augmentation may occur at three different levels, tile-level, feature vector-level, and whole digital histopathology image (e.g. slide or WSI)-level.

[0110] Tile-level data augmentation creates additional sets of tiles from the same original digital histopathology image. Tile-level data augmentation may include tile shifting and color augmentation techniques, as well as a tile-level version of random feature vector selection.

[0111] Tile-shifting is a technique in which a set of initial tiles is created from a digital histopathology image. Then the positions within the digital image used to define tiles are shifted, typically in defined way, and a set of shifted tiles is created from the digital histopathology image. Typically, the set of shifted tiles includes the same image information as the set of original tiles, but areas of the image that are near one another, but are located in separate tiles in the set of original tiles may be located in the same tile in the set of shifted tiles. This simulates spatial variability in the digital image.

[0112] Shifted tiles may, with respect to the original tiles, be shifted horizontally, vertically, or diagonally, as illustrated in Fig. 6. Multiple directions or degrees of shifts may be applied with respect to the original tiles to create shifted tiles that are shifted in different manners. For example, one set of shifted tiles may be shifted one half a tile side length horizontally, while another set of shifted tiles that differs in degree of shift may be shifted two and a half tile side lengths horizontally. In another example, one set of shifted tiles may be shifted horizontally, while another set of shifted tiles that differs in direction of shift maybe shifted diagonally.

[0113] Color augmentation is a technique in which a set of initial tiles is created from a digital image in one color space, such as red green blue (RGB), then transformed into another color space, such as hematoxylin, eosin, and DAB (3, 3'-diaminobenzidine) (HED) to create a color-augmented tile. In some embodiments the color- augmented tile may first be transformed back into the original color space, such as RGB, before being included in a set of color- augmented tiles. Fig. 7 provides an example of this. Parameters of the colortransformation may be adjusted to emphasize one color over another. Color-augmented tiles may simulate variations in colors between separate digital histopathology images, such as variations in staining between H&E slides.

[0114] In tile-level random feature vector selection, for a given tile a set of original and augmented feature vectors is created, but, for any given set of feature vectors provided to the DL model for training, each tile is represented by only a set number of feature vectors. This set number of feature vectors is less than the total number of original and augmentedfeature vectors created for that tile. In a simple embodiment, the set number may be one. In that case, each tile is represented in the full set of tile-specific feature vectors by only one feature vector. That representative feature vector is chosen at random from the total available feature vectors for that tile. Methods of choosing this representative feature vector for inclusion in the tile-specific set of feature vectors may include any method, such as those discussed below and in Example 2, for selecting a subset of feature vectors from the full set of feature vectors representing the whole digital medial image.

[0115] Feature vector-level data augmentation creates additional sets of feature vectors from an original set of feature vectors. Feature vector-level data augmentation may include pseudo-vector mixup.

[0116] Pseudo-vector mixup is a feature vector-level data augmentation technique that creates new synthetic feature vectors by interpolating between existing, non-synthetic feature vectors. In some embodiments, such as that depicted in Fig. 8, each feature in the synthetic feature vector is randomly selected from a non-synthetic feature vector. In some embodiments, each feature in a synthetic feature vector may be derived from, but not the same as, the counterpart feature in at least two non-synthetic feature vectors. This may be accomplished, for example, by interpolation, such as linear interpolation between values for that feature in the non-synthetic feature vectors.

[0117] In some embodiments, synthetic feature vectors may be created in a similar manner, but with features selected from any feature vector representing any tile. These synthetic feature vectors may represent the digital histopathology image, rather than any particular tile created from the image.

[0118] Whole digital histopathology image-level data augmentation relies on each iteration during training of the DE model using a subset of feature vectors, rather than the full set of available feature vectors for a given digital histopathology image. Typically different subsets are used in separate iterations of DE model training, although the same subset may sometimes be used in multiple iterations.

[0119] In random feature vector selection, the feature vectors in a given iteration subset of feature vectors used in a given iteration of DL model training are randomly selected. In some embodiments, such as that depicted in Fig.8, feature vectors for an iteration subset may be selected individual at random from the full set of feature vectors. In other embodiments, such as that described in more detail in Example 2, subsets of featurevectors based on data augmentation type may be identified, and entire such subsets may be selected as the iteration subset or for inclusion in the iteration subset. In various embodiments, an example of which is also provided in Example 2, the probability of selecting any given feature vector or subset of feature vectors may be specified. In some embodiments, probability of selecting an original feature vector or the subset of original feature vectors may be favored over data augmented feature vectors. This may increase accuracy of the trained DL model. In some embodiments, probability of selecting a feature vector or subset of feature vectors with a given data augmentation may be favored over other data augmented feature vectors, or, in some embodiments, even the original feature vectors.

[0120] In some embodiments, data augmentation may occur prior to DL model training and the full set of feature vectors available for DL model training and how subsets of feature vectors for iterations are assembled may not change during DL model training. However, in other embodiments, particularly based on interim or preliminary DL model accuracy, data augmentation may be applied during DL model training such that the full set of feature vectors changes, based on data augmentation, or how subsets of feature vectors for iterations are selected changes using whole digital histopathology image-level data augmentation.

[0121] Although data augmentation may be implemented in a variety of ways at a variety or levels, with some data augmentation combinations acting synergistically for certain DL models and with respect to certain tissues or phenotype classifications, Fig. 8 provides one example in which several types of data augmentation are illustrated in the context of the steps of DL-based classifier training that may be affected.

[0122] In Fig. 8, original tiles are created from a digital histopathology image, then tile shifting data augmentation is applied to create a set of shifted tiles. Color augmentation is also applied to create color- augmented tiles. These augmentations are reflected in a set of original and augmented tiles. A pre-trained feature extractor is used to create feature vectors based on these original and augmented tiles. All of the resulting features vectors are nonsynthetic. The set of non- synthetic feature vectors are then used, in combination with pseudovector mix-up data augmentation to produce synthetic feature vectors. These synthetic feature vectors are then added to the non-synthetic feature vectors to create a set of non-synthetic and synthetic feature vectors. Next random feature vector selection is used to select a subset of these feature vectors for use in one iteration of training the DL model.Images Prepared Using Additional Histopathology Techniques

[0123] Although the training method above is described using a digital medial images created in a uniform manner, not all datasets of identified images need to so uniform. For example, a dataset of identified digital histopathology images may include images of slides prepared according to one histopathology protocol, such as H&E staining, along with images of slides prepared according to a second or subsequent histopathology protocol, such as staining with a different stain. In some embodiments, slides prepared according to second or subsequent histopathology protocols may be correlated slides from the same tissue sample as the slide prepared according to the first protocol. A correlated slide, in some embodiments, may be prepared using a slice of the tissue sample adjacent or within 10 microns of the slice used in the original slide. Correlated slides are illustrated schematically in Fig. 1.

[0124] In some embodiments, the digital histopathology images created using a second or subsequent protocol may be included in step 110 and may result in feature vectors included in the sets of feature vectors assembled for each image in step 120.

[0125] In still other embodiments, in which step 130 includes sub-steps 130a and 130b, steps 110 through 130a may be performed separately for digital histopathology images prepared using different protocols, resulting in two separate, but correlated slide-level representations that are provided as an input to the final classifier.Including Clinical or Molecular Metadata

[0126] In some embodiments, the DL-based classifier is also provided with at least one type of clinical or molecular metadata associated with each image in the dataset of identified digital histopathology images, as shown in Fig. 1. This clinical or molecular metadata, in some embodiments, be provided to the input layer of the DL model. In other embodiments, the clinical or molecular metadata may be provided, along with the output layer from the DL model, to a final classifier.

[0127] In some embodiments, the clinical metadata may include information about patient age, weight, ethnicity, other health conditions, prior treatments, histologic or tumor grade, tumor or cancer stage, or other factors that may affect the phenotype of interest. In some embodiments, the clinical metadata may include radiological medical images. In some embodiments, the clinical metadata may include a general patient-level genomics analysis, such as the patient’s genotype for one or more cancer-associated genes, such as humanepidermal growth factor receptor 2 (HER2), estrogen receptor (ER), progesterone receptor (PR).

[0128] In some embodiments, the radiological image may be an MRI image, a CT scan image, a PET scan image, or a digitized X-ray image. These images, may, in some embodiments, be of a tumor, particularly a cancerous tumor, such as a metastatic tumor. In some embodiments, the radiological image may be used in the DL-based classifier in one of the same manners as described above for an image prepared using a second histopathological technique. In such embodiments in which the radiological images are 3D, cubes or other 3D segments may be equivalent to tiles. In other embodiments in which the radiological images are 3D, set of 2D images extracted from a single 3D segment, particularly 2D images from different axes, may be equivalent to a tile or multiple tiles.

[0129] In some embodiments, the molecular metadata may include molecular data such as results from RNA analyses, protein analyses, or, in some specific embodiments, HER-2, ER, or PR-status of the patient or tissue sample, or other metadata related to the tissue sample and not obtained from digital histopathology images.

[0130] In some embodiments, the molecular metadata, particularly RNA analyses or protein analyses, may be in situ data from a tissue section correlated with areas of a digital histopathology image. For example, the molecular metadata may be obtained using a slice of the tissue sample adjacent or within 10 microns of the slice shown in the digital histopathology image.

[0131] DL-based classifiers may, in some embodiments, be able to use digital histopathology images to assess phenotypes and associated therapeutically relevant pathophysiologies that may be theoretically assessable by pathologists using the same images, but for which such assessments by pathologists would not be practical. For example, the amount of time required for an assessment by pathologist might be so long that clinicians would not receive relevant information in a timely manner or a pathologist assessment might require so much time and other resources that the availability of assessments would be limited to a much smaller pool of patients. In still other embodiments, the phenotypes or associated pathophysiologies may be theoretically assessable by pathologists using the same images, but the accuracy of pathologist assessments may be too low to be clinically acceptable, or the accuracy of pathologist assessments may be significantly lower than the accuracy of classifications by DL-based classifiers of the present disclosure.

[0132] In other embodiments, DL-based classifiers may be able to assess phenotypes or therapeutically relevant pathophysiologies that cannot be assessed by pathologists using digital histopathology images alone. For example, the DL-based classifiers may determine spatial patterns in digital histopathology images using image assessment techniques that humans cannot replicate, such as techniques able to measure features of the digital histopathology images that are not readily measurable by humans, if measurable at all. The DL-based classifiers may also consider a much wider array of features of the digital histopathology images (for example hundreds of individual features) than a pathologist could evaluate without the assistance of a trained computational model. As a result of this, the DL- based classifiers may also recognize patterns of features associated with phenotypes and, ultimately therapeutically relevant pathophysiologies that a pathologist would not be able to recognize without the assistance of a trained computational model.

[0133] In one embodiment, further illustrated in some aspects in the Examples below, a human pathologist viewing a digital histopathology image or the slide from which it was derived may only be able to assess that lymphocytes are visible in the image, indicating some sort of immune activity. A DL-based classifier of the present disclosure, in contrast, can not only assess that lymphocytes are present, but also determine the type of lymphocytes or the lymphocyte activity displayed. For instance, in the Examples, the DL-based classifier is able to determine that the lymphocytes are T-cells and whether or not those T-cells are active, or non-functional (such as exhausted T-cells). These additional details have critical implications for whether and how the patient’s immune system is responding to a tumor, which may helpful in determining the most appropriate therapy for a patient (e.g. whether to administer checkpoint inhibitors, whether to administer them prior to further chemotherapy, etc.), and which may also be helpful in identifying biological mediators relevant to how the patient’ s immune system is responding to the tumor.

[0134] In still other embodiments, the DL-based classifiers may use information extracted from a digital histopathology image in combinations not replicable by humans without the assistance of a trained computational model.Assessment of Patient or Other Unclassified Tissue Sample Images

[0135] The present disclosure also provides methods and systems using DL-based classifiers to classifying unclassified digital histopathology images, such as images obtainedfrom a patient tissue sample, with respect to phenotype.

[0136] A method 200 of classifying a digital histopathology image with respect to phenotype is illustrated in Fig. 4. Information and techniques described above in the context of DL-based classifiers may be equally applicable in method 200.

[0137] In step 210 a set of tiles is created for an unclassified digital histopathology tissue sample image. In some embodiments the tiles, collectively, represent the WSI or near WSI. In some embodiments, the tiles may be created using any of the methods described in step 110, above.

[0138] In step 220, a pre-trained feature extractor extracts features from each tile to create a set of tile- specific feature vectors. In some embodiments, the pre-trained feature extractor may be any such feature extractor described in connection with step 120, above, and may perform any methods described in connection with step 120, above.

[0139] In step 230, the set of tile-specific feature vectors is provided to a phenotype- trained DL model and used to classify the image with respect to phenotype. In specific embodiments, the set of tile-specific feature vectors is provided to an input layer of a trained DL model, which then, based on the input layer, performs calculations to classify the image with respect to phenotype.

[0140] The phenotype-trained DL-model may have been trained as set forth in method 100 and have an accuracy as indicated in steps 140 and 150 above. The tissue type and histopathology protocol or other sample preparation or image collection method reflected in identified digital histopathology images used to train.

[0141] In step 240, the phenotype classification is stored. In some embodiments, the phenotype classification may be stored with patient-identifying data. In some embodiments, the phenotype classification may be stored with other phenotype classifications of the same digital histopathology image provided by DL-based classifiers trained to classify images with respect to those phenotypes.

[0142] In step 250, the phenotype classification is reported to a user. Typically the user is a pathologist or other medical professional.

[0143] Phenotypes and therapeutically relevant pathophysiologies identified using DL-based classifiers described herein, in some embodiments, may be used to direct decisions of a clinician treating the patient whose tissue was used to create the digital histopathology image. Specific decisions directed by the clinician in response to an identified phenotype orpathophysiology will be specific for the phenotype or pathophysiology. Specific decisions may be those indicated as the standard of care for patients with the phenotype or pathophysiology. Decisions may include recommended treatment regimens, which may include selection of surgical or pharmaceutical treatments (including treatments with small molecule drugs, biologies or cell therapies), the timing of such treatments, and the duration or dosage of pharmaceutical treatments. Decisions may also include whether to conduct further diagnostic testing and the nature and timing of such diagnostic testing. In instances where no problematic pathophysiology is detected the decision may be to not prescribe any treatment or further diagnostic testing.

[0144] In some embodiments, step 250 further includes presenting the user with information based on the phenotype classification that may be useful in decisions by a clinician relating to further treatment or diagnosis of the patient. For example, if the phenotype classification indicates a pathology, then the user may be presented with further information about the pathology, its prognosis, specific treatments, information about what specific treatments are or are not likely to be effective in that patient, and complicating factors, such as co-morbidities, known to be present in the patient or that are associated with variations in prognosis or treatment efficacy. As another example, if the phenotype classification indicates a pathology, but cannot provide other clinically relevant details that must be assesses using a different diagnostic technique, then the user may be presented with information regarding the details not assessed by the DL-based classifier, why those details are clinically relevant, further diagnostic assays that should be considered for the patient, or details of what additional information might be obtained from further diagnostic assays and how it might contribute to treatment decisions. As yet another example, if the phenotype classification indicates that a particular pathology or phenotype is not present, or is inconclusive, the user may be presented with information regarding alternative ways to diagnose the suspected pathology, such as alternative diagnostic assays, other pathologies with similar symptoms, diagnostic assays for such other pathologies, or information regarding the risk of the patient having the suspected pathology based on other patient information, such as clinical metadata.

[0145] Information provided to the user may include links to web sites or other data repositories with information regarding the pathology, other diagnostics, or treatments, links that allow the user to refer or prescribe a given treatment or further diagnostic assay, links toscientific publications relating to the pathology or treatment, particularly those linking the phenotype to such pathologies or treatments, links to current clinical guidelines, treatment or diagnostic protocols specific to a given medical institution, practice, or insurance provider, and other information which the clinician may consider in prescribing a further treatment or diagnostic assay specifically based on the phenotype classification.

[0146] For purposes of using DL-based classifiers described herein to identify therapeutically relevant phenotypes and pathophysiologies, clinicians who request such assessments, professionals who perform the assessments and provide the results to a clinician, and the clinician who receives the results (if different from the requesting clinician) and whose treatment decisions are based on such results may all act in concert with one another (and other medical professionals involved in the process from determining that an assessment is needed through taking a clinical action based on the assessment) to achieve identifying and acting on therapeutically relevant phenotypes and pathophysiologies (or the absence thereof). In particular embodiments, these clinicians and medical professionals may act in concert according to medical processes and procedures recommended by a medical professional, institutional, or governmental body.

[0147] Accordingly, the disclosure also provides a method of directing medical treatment of a patient that includes obtaining a patient tissue sample from the patient, assessing the patient tissue sample using a method disclosed herein, such as method 200, to identify a phenotype or pathophysiology exhibited by the tissue sample, and, based on the phenotype or pathophysiology, prescribing a treatment or diagnostic regimen that is different from the treatment or diagnostic regimen that would be prescribed according to standards of medical care if the phenotype or pathophysiology were not known.

[0148] The disclosure also provides a method of directing medical treatment of a patient that includes selecting a patient based on a prior phenotype classification or pathophysiology assessed using a patient tissue sample and a DL-based classifier described herein, for example via method 200, treating the patient according to the phenotype classification or pathophysiology. In some embodiments, treating may include prescribing a therapeutic regimen known to be effective in patients exhibiting the phenotype or pathophysiology. In some embodiments, treating may include prescribing further diagnostic testing related to the phenotype or pathophysiology. In some embodiments, treating may include refraining from prescribing a therapeutic regimen or diagnostic test related to thephenotype or pathophysiology that under standards of care would normally (and likely unnecessarily) be prescribed.

[0149] The disclosure further provides a method for directing medical treatment of a patient that includes the patient providing a tissue sample, the patient obtaining a phenotype classification or pathophysiology assessment using the tissue sample and a trained DL-based classifiers, such as a prediction obtained via method 200, and the patient undergoing a treatment regimen based on the phenotype classification or pathophysiology. In some embodiments, the patient may direct the activities by providing informed consent for such activities, or by otherwise seeking the assistance of medical professionals to perform these activities and provide medical treatment advice.Identification of Biological Mediators

[0150] The present disclosure also provides methods for identifying biological mediators using a DL-based classifier analysis of digital histopathology images of a tissue sample with labeled clinical correlations. The DL-based classifier identifies areas of interest in the digital histopathology images. These areas of interest are then used to target corresponding portions of the tissue sample that are then subject to further analysis to identify biological mediators. For example, the corresponding portions of the tissue sample may be further analyzed to determine the amounts or types of DNA, RNA, or proteins contained in those portions. A comparison of the amounts or types of DNA, RNA, or proteins based on the clinical correlations may uncover a particular DNA, RNA, or protein that differs based on the clinical information. Biological mediators identified in this way may be used for diagnosis or for targeted drug discovery.

[0151] In some embodiments, the biological mediator may be a biological mediator of therapeutic resistance. Such biological mediators may be identified by providing a DL-based classifier with digital histopathology images labeled as being from patients who are or are not resistant to a therapy, such as a therapy for a pathology. The therapeutic resistance may be resistance to a therapy using a small molecule, protein, engineered cell-based therapy, or any combinations thereof. The DL-based classifier may be a trained DL-based classifier, such as one that has been trained using digital histopathology images from patients with the same pathology or the same tissue type. For example, the DL-based classifier used to identify biological mediators of resistance to a particular cancer therapy, such as a particularchemotherapy or checkpoint inhibitor, or combination therapy may be a DL-based classifier that was been trained using digital histopathology images from patients with the same type of cancer.

[0152] In another example, the biological mediator may be a biological mediator of response to therapeutic mechanical assistance. For example, the biological mediator may relate to patient response to therapeutic mechanical respiratory assistance (e.g. a ventilator). In some embodiments, the biological mediator may indicate whether a specific type of therapeutic mechanical assistance is beneficial to a patient, such as whether a patient is likely to benefit from being placed on a ventilator. In some embodiments, the biological mediator may indicate how long a patient will need therapeutic mechanical assistance, or how likely it is that such mechanical assistance can be withdrawn. For example, the biological mediator may indicate whether a patient, having been placed on a ventilator, is likely to have to remain on it for a short period of time, such as days or weeks, or a longer period of time, such as months.

[0153] In other embodiments, the biological mediator may relate to patient response to other non-pharmaceutical interventions, such as surgery or supplemental oxygen, or to whether a patient is likely to need any therapy for a particular pathology, such as whether a patient is likely to clear an infection without antibiotics, which could help reduce the use of antibiotics at a population level and the development of antibiotic -resistant bacteria.

[0154] In some embodiments, the DL-based classifier may be presented with a set of slides from tumor sections, some from patients that respond to checkpoint inhibitors, and some from patients who do not respond. If the DL-based classifier is used in a manner similar to the DL model training methods described above to attempt to distinguish digital histopathology images based on checkpoint inhibitor response, the classifier will, as part of that process identify areas of interest within the images. Regions in other sections of the tumor that correspond with these areas of interest may be highlighted. This is represented schematically by the black area in the histopathology slides and in situ analysis sample illustrated in Fig. 1. A corresponding region from another section of the tumor may be subject to in situ analysis for a particular protein type (defined by function, e.g. kinases, specific protein sequence, etc.). Results from the in situ analysis between responder and nonresponder patients may be compared, but rather than compare results across the entire tissue sections, the comparison may focus on results in the area of interest for each tumor instead.This allows more efficient and accurate identification of whether expression of that protein differs between responders and non-responder patients than if the entire tissue sections were compared. If expression of the protein does differ, then it is a potential biological mediator to checkpoint inhibitors. Assays for the protein might be used as diagnostic assays before prescribing a checkpoint inhibitor to the patient. The protein’s biological role in cancers and potential ways of blocking or enhancing its effects may be explored to develop further treatments that allow the use of or improve the effects of checkpoint inhibitors in patients who are non-responders or poor responders.

[0155] In some embodiments, the DL-based classifier used in biological mediator identification is a DL-based classifier trained to diagnose phenotypes associated with a pathology or pathophysiology of interest. The DL-based classifier is able to identify areas of interest while analyzing the digital histopathology images to determine differences between those exhibiting the labeled characteristic, such as response to a particular therapy.

[0156] In some embodiments, the DL-based classifier used in identifying biological mediators need not be trained to a level of accuracy as high as would be preferred for a DL- based classifier used for phenotype classification. This lower level of accuracy is tolerable because, for identifying biological mediators, the goal is reducing the amount of noise in subsequent steps, such as in situ analysis. A substantial reduction in the amount of noise can be achieved with a reasonably accurate DL-based classifier, and the consequences of not achieving a greater amount of noise reduction is only a slightly more prolonged or computationally intensive search for biological mediators. In contrast, the consequences of errors in phenotype classification may include a clinician obtaining inaccurate information about a patient and, as a result, prescribing a therapeutic or diagnostic path that is not optimal.

[0157] The trained DL-based classifier for identifying a biological mediator will still have an overall accuracy that is appropriate for the context. In some embodiments, the trained DL-based classifier will have an accuracy greater than random chance, otherwise it is unlikely to be of any meaningful assistance in identifying biological mediators. However, as noted above, accuracy need not be as high as in a diagnostic setting. For example, an AUROC of at least 55%, 60% 70%, 80%, or 90%, or an AUROC in a range between any of the preceding and 60%, 70%, 80%, 90%, 95%, or 99% may be sufficient.

[0158] Biological mediator identification based on areas of interest may allow for more rapid analysis or more accurate identification of biological mediators than if unfocused tissue analysis were used. In unfocused analysis, much of the tissue analyzed does not vary significantly based on the labeled characteristic, and therefore, is not likely to have any distinguishing biological mediators. Data from unfocused analysis includes a substantial amount of data that is not likely relevant to differences in the labeled characteristic. Such data is often referred to as “noise” and its presence can make it difficult mathematically and in terms of computational resources used to locate any information that is relevant to the labeled characteristic, and likely indicative of a biological mediator. In some embodiments, biological mediator identification methods of the present disclosure may identify, on average, between 60-80% of a digital histopathology image, such as a WSI or near WSI, as not an area of interest. This allows biological mediator identification in the corresponding tissue to similarly not focus on, or entirely exclude information from tissue corresponding to this excluded 60-80% of the digital histopathology image. For example, if in situ analysis is used, then 60-80% of a tissue section may be excluded from the in situ analysis, greatly reducing the amount of noise in the in situ analysis.

[0159] Biological mediators can be anything detectable in the tissue sample that indicate a clinically relevant difference between patients. In some embodiments, this may be a simple difference in a gene, indicated by DNA present in the sample. For example, many tumors undergo mutations in important genes, such that genomic or gene-specific analysis, particularly in situ analysis, might indicate whether a given mutation in a gene is a biological mediator.

[0160] The merely presence or absence of a gene, however, has increasingly been recognized as less meaningful that once thought, primarily because not all genes are actively expressed in a cell, and because the levels of expression may vary. Active genes are expressed by first being transcribed into RNA, which allows the cell to use the gene’s DNA instructions to make proteins. The amount of RNA for a given gene in a cell or tissue is thus a better reflection of what is actively occurring in that cell or tissue than merely the presence or absence of a gene. The RNA present in a cell is often referred to as a transcriptome and analysis of such RNA is referred to as transcriptomics. In some embodiments, biological mediators may be identified by a transcriptomic method, such as analysis, particularly in situanalysis, of the complete, nearly complete, or partial transcriptome of a cell or tissue, or for an RNA having a given sequence.

[0161] To have a therapeutically relevant effect, typically RNA has to actually be translated into proteins, and this does not always occur in a set and predictable way, particularly in cells that are not functioning normally, such as cancer cells. The activities of proteins create the pathophysiology of the tissue, which tends to be therapeutically relevant. The proteins in cells and tissues are referred to as the proteome, and their analysis is referred to a proteomics. In some embodiments, biological mediators may be identified by a proteomic method, such as analysis, particularly in situ analysis, of the complete, nearly complete, or partial proteome of a cell or tissue, or for a protein have a given sequence. Protein analysis may also include analysis for particular protein activities, such as kinase activities, or the presence of or activation state of various transcription factors. Protein analysis, particularly for specific proteins or protein activities, may include immunohistochemical or immunofluorescence analysis.

[0162] Although the above example relates to identifying biological mediators for checkpoint inhibitor response in cancer patients, the methods herein may be used to identify biological mediators for other pathologies or pathophysiologies that are likely to exhibit phenotypic differences of the types that can be detected using DL-based classifiers of the present disclosure. For example, pathologies or pathophysiologies that are likely to exhibit particular angiogenesis, immune response, cell cycle, or cell metabolism differences may be analyzed to identify biological mediators. These include cancer, infections, autoimmune diseases as described above, or other pathologies or pathophysiologies likely to be distinguished by angiogenesis, immune response, cell cycle, or metabolic differences similar to those described anywhere herein.

[0163] In some embodiments, the pathology or pathophysiology may not, prior to biological mediator identification, be conclusively tied with a phenotype of the types demonstrated to be classifiable by the DL-based classifiers of the present disclosure, in which case the biological mediator identification process may also provide information about phenotype associations. If a DL-based classifier is able to identify areas of interest in a biological mediator identification assay that yield biological mediators or that, based on a human assessment of the image, appear to be located in regions of the digital histopathology image that are expected to be associated with the pathology or pathophysiology (e.g. regionsthat have a lot of immune cells when response to a checkpoint inhibitor or antibiotic is the clinical indication), then the pathology or pathophysiology must also associated with at least some of the phenotypes the DL-based classifier was trained to distinguish.

[0164] In some embodiments, the pathology or pathophysiology may be any clinical correlation, such as a particular response to an immunosuppressant, whether a patient who requires a ventilator will be able to transition off such assistance quickly, whether a patient is likely to have a better prognosis if provided with chemotherapy or checkpoint inhibitors in addition to surgical removal of a tumor, or whether antiviral drugs or antibiotics are likely needed to clear an infection or prevent a recurrence. If a biological mediator identification method according to the present disclosure is not able to provide meaningful focus in the context of a given clinical correlation, then that is readily determined by statistical analysis, or the identification of non-sensical regions of interest. In such cases, differently trained DL- based classifiers may be used, or the clinical correlation may simply be deemed not analyzable according to methods of the present disclosure. If a biological mediator identification method according to the present disclosure is able to provide meaningful focus, then that should be apparent by identification of areas of interest that at least appear to a human observer to have some consistency based on the clinical indicator.

[0165] A method 300 of identifying a biological mediator associated with a pathophysiology, pathology, or other clinical characteristic is illustrated in Fig. 5. Information and techniques described above in the context of DL-based classifiers may be equally applicable in method 300.

[0166] In step 310 a set of tiles is created for each digital histopathology image in dataset of identified digital histopathology images, each with a labeled clinical correlation that reflects a pathology or pathophysiology. In some embodiments, the set of tiles may be created using any of the methods described in step 110, above.

[0167] In some embodiments, the digital histopathology image may be from a patient clinically diagnosed as having a specific pathology or pathophysiology or other clinically relevant characteristic, such as response or non-response to therapy.

[0168] In some embodiments, the digital histopathology image may be from a tissue sample assessed via other diagnostic methods as exhibiting or not exhibiting a specific pathophy siology .

[0169] In step 320, a pre-trained feature extractor extracts features from each tile to create a set of tile-specific feature vectors for each digital histopathology image. In some embodiments, the pre-trained feature extractor may be any such feature extractor described in connection with step 120, above, and may perform any methods described in connection with step 120, above.

[0170] In step 330, the sets of tile-specific feature vectors and corresponding image labels are provided to a DL model and assessed by the DL model in a manner that identifies differences between the images based on the different labels to provide a set of attention scores or an attention map associated with at least some of the digital histopathology images. In some embodiments, attention scores or an attention map is provided for each digital histopathology image in the dataset.

[0171] In some embodiments, the DL model is a phenotype-trained DL model that has been trained as set forth in method 100. In specific embodiments, the phenotype-trained DL model has an accuracy as set forth in method 100. In some embodiments, the phenotype-trained DL model has an accuracy that is generalizable as described above for DL models usable in biological mediator identification. In some embodiments, the phenotype- trained DL model has been trained using the same tissue type as the digital histopathology images.

[0172] In some embodiments, the DL model is phenotype-trained for a phenotype exhibited or likely exhibited by at least some patients having the pathology or pathophy siology .

[0173] In other embodiments, the DL model is of a type able to generate attention scores and, optionally, also an attention map in which areas having high attention scores, designated areas of interest, are marked on the digital histopathology image or in some manner that may be correlated with portions of the digital histopathology image. In some embodiments, the DL model may be of the same type and architecture as may be used in methods 100 and 200.

[0174] In step 340, tumor tissues corresponding to the areas of interest for at least some of the digital histopathology images are assessed for differences in other characteristics that correlate with differences in the phenotype, pathophysiology, or other clinical characteristic. In some embodiments, areas of interest are assessed for tumor tissues corresponding to all of the digital histopathology images with attention maps or attentionscores. In some embodiments, assessing includes assessing tumor tissues corresponding to the areas of interest for at least some of the digital histopathology images for a difference in a characteristic that correlates with a difference in the image labels.

[0175] In other embodiments, assessing the tumor tissues corresponding to areas of interest may be performed in a manner described above with respect to ways of obtaining molecular metadata from a tissue sample. In particular, identification of areas of interest on a digital histopathology image may be used to guide in situ analysis of an adjacent tissue section. This in situ analysis may include analysis of the genome, transcriptome or proteome as described above for biological mediator identification.

[0176] Noise may be reduced in the assessment of tumor tissues as described above.

[0177] In step 350, genes, RNA, or proteins, or combinations thereof, correlating with differences in the phenotype, pathophysiology, or other clinical characteristic, as indicated by the labels, are identified as biological mediators. In some embodiments, the mere presence or absence of the biological mediator may be the relevant correlation. In some embodiments, the amount or activity of the biological mediator may be the relevant correlation. In either case, thresholds may be identified and used to classify whether the biological mediator is present or absent or has a given amount or activity.

[0178] The biological mediator may then be investigated for use in diagnostic assays, as drug or other therapeutic targets, or to understand biological pathways underling the disease, condition, or pathophysiology. In some embodiments, the biological mediator is a biological mediator not previously identified or correlated with the phenotype or pathophy siology .Systems

[0179] The present disclosure further includes systems for performing certain methods disclosed herein. In particular embodiments, the system includes at least one processor and a non-transitory computer-readable medium storing instructions that, when executed by a processor, causes the system to perform operations according to any of the methods of training a DL-based classifier, or the methods of assessing a digital histopathology image using a trained DL-based classifier.

[0180] In some embodiments, the system may further include an imaging system able to capture a digital histopathology image suitable for use by the relevant DL-based classifier.The system, in some embodiments, may include an input / output device, such as a network input / output connection or graphical user interface (GUI). Components of the system may be interconnected via a system bus.EXAMPLESExample 1 - Feature Extraction Approaches and. Model Architectures for Breast Cancer Phenotype-Trained DL-Based Classifiers

[0181] DL models were used to assess activity of ten therapeutically relevant pathophysiologies from the hematoxylin and eosin (H&E)-stained slides of primary breast tumors. The DL models included different feature extractor components and neural network components. Using binary classification, certain DL models attained area under the receiver operating characteristic (AUROC) scores above 0.70 for nearly all therapeutically relevant pathophysiologies, and, in some cases, exceeded 0.80. Attention maps were also generated and suggested that the trained DL models recognized biologically relevant spatial patterns of cell sub-populations from H&E-stained slides.Patch Preparation and Feature Extraction

[0182] Whole slide digital histopathology images were obtained for H&E slides from the breast invasive carcinoma (BRCA) subset of The Cancer Genome Atlas (TCGA): TCGA- BRCA (Lingle et al. 2016). For quality assessment, the default parameters of HistoQC (Janowczyk et al. 2019; Chen et al. 2021). A curated segmentation mask (tissue versus background) was obtained for each whole slide digital histopathology image. Next, using this mask, each whole slide image (at 20x, 40x magnification) was split into patches of size 256x256 using the pipeline described in (Lu et al. 2021). Finally, feature vectors were extracted from each patch using two different approaches: a ResNet50 (He et al. 2016; Lu et al. 2021) backbone pre-trained on the ImageNet dataset (Russakovsky et al. 2015), the PLIP (Huang et al. 2023) image encoder pre-trained on the OpenPath dataset.Quantification of Gene Expression and Assignment of Known Classification

[0183] To provide known pathophysiology classifications for the whole slide digital histopathology images, normalized expression of individual genes (bulk RNA- sequencing data) for the TCGA-BRCA cohort was obtained from the cBioPortal for Cancer Genomics (cBioPortal for Cancer Genomics 2023). Each therapeutically relevant pathophysiology was defined as per Gene Set Enrichment Analysis (GSEA) protocols. Single-sample GSEA(ssGSEA) was used to calculate gene set enrichment scores for each sample and pathophysiology pairing. ssGSEA transforms a single sample’s gene expression profile to a gene set enrichment profile. This enrichment score represents the activity level of the pathophysiology-associated biological pathway in which the gene set’s members are coordinately up (i.e., positive values) or down-regulated (i.e., negative values) (Subramanian et al. 2005; Coscia et al. 2018; Yi et al. 2020).Training Details

[0184] Pathophysiologies associated with the digital histopathology images were assigned a binary classification. To this end, the ssGSEA scores underwent binarization, with negative scores receiving the label “0” and the rest label “1”. A random data split of the digital histopathology images allocated 70% to training, 15% to validation, and 15% to testing, ensuring exclusive assignment of a given image’s data to one split. Neural network components of the DL models were given architectures based on two general architectures, CLAM (Lu et al. 2021) or Trans-MIL (Shao et al. 2021).

[0185] The Clustering-constrained Attention Multiple instance learning approach (CLAM) architecture uses attention-based pooling to aggregate patch-level representations, weighted by the attention score, into slide-level representations. This slide-level representation, instead of the entire whole slide image is the input to the final classification module, such that CLAM requires fewer computational resources to train and optimize.

[0186] In CLAM, there are two variations of the model architecture. Single-branch CLAM (CLAM-SB) employs a fully connected layer for slide-level classification. Multibranch CLAM (CLAM-MB) uses a fully connected layer per class, using the class with the maximum score for slide-level classification. CLAM also promotes instance-level clustering under a mutual exclusivity assumption (e.g., cancer subtypes are mutually exclusive: when one subtype is present in the WSI, one may assume that there is no morphology of the other subtypes concurrently present). This assumption did not apply to here, so only the attention mechanism (along with the classifier) was used. The resulting two model architectures were designated AM-SB and AM-MB. These models were trained for 300 epochs, using crossentropy as the loss function and the adaptive moment estimation (Adam) optimizer (Kingma and Ba 2015) with a learning rate of 2xl0-4and weight decay of 1 x 10-5. The best model according to the loss on the validation set was saved.

[0187] An important limitation of multiple instance learning is that each patch is assumed to be independent of surrounding patches (Lipkova et al. 2022). This assumption may not be valid, given the importance of spatial localization of different cell populations in dictating tumor phenotype. Accordingly, another DL model, Transformer based multiple instance learning (Trans-MIL) was also tested. Trans-MIL is a vision transformer model that accounts for spatial correlation and context between patches. Although not tested in this example, Hierarchical Image Pyramid Transformer (HIPT) (Chen and Krishnan 2021; Chen et al. 2022) is another such model. Vision transformer models leverage morphological and spatial information to learn representations that encode the natural hierarchical structure inherent in whole slide images. While there is early evidence that these architectures may boost performance for tasks in computational pathology, it was not immediately clear whether this boost in performance would justify the increased computational resources that vision transformers require.

[0188] Trans-MIL was trained for 200 epochs, using cross-entropy as the loss function and the Adam optimizer with a learning rate of 2 x 10-4and weight decay of 1 x 10“5. The best model according to the loss on the validation set was saved.

[0189] Each phenotype was assigned a ground truth negative / positive phenotype classification in a binary fashion and the relative numbers of digital histopathology images that were negative for the phenotype (designated 0) and digital histopathology images that were positive for the phenotype (designated 1), were determined.

[0190] The DL models assessed predicted pathophysiology classifications, in particular tumor micro environment (TME) phenotypes from whole slide images of H&E- stained slides with reasonable accuracy. Table 1 presents the area under the receiver operating characteristic curve (AUROC) results for both types of DL model architectures, using either ResNet50-derived or PLIP-extracted features. Each phenotype was assigned a binary class.Table 1: AUROC For Ten Phenotype Classifications Using DL-Based C assifiers

[0191] Image assessment models trained to predict immune phenotypes e.g., B-cell proliferation, T-cell mediated cytotoxicity, immunosuppression, antigen presentation) achieved satisfactory performance (AUROC 0.75-0.8). Given the link between the spatial distribution of tumor-infiltrating lymphocytes and myeloid cells to immune activity (Longo et al. 2021), these findings illustrate DL’s ability to identify and learn spatial patterns of cell populations predictive of specific facets of immune function. These findings are noteworthy, as the type (e.g., T-, B-, and dendritic cell) or function (e.g., cytotoxic or exhausted) of lymphocyte is indistinguishable from manual pathology review on H&E-stained slides.

[0192] Image assessment models trained to predict facets of tumor aggressiveness (e.g., cell cycling, angiogenesis, and epithelial-mesenchymal transition (EMT)) achieved reasonable accuracy. These findings are biologically plausible, as cell cycling measures mitotic activity and aggressiveness of tumor cells, angiogenesis represents growth of new blood vessels (a hallmark of cancer growth), and EMT measures a tumor’s ability to invade surrounding tissue. Biological activity associated with these pathways can be observed on H&E-stained slides, but are often difficult for pathologists to quantify.

[0193] Image assessment models trained to predict fatty acid metabolism and oxidative phosphorylation achieved relatively poor performance. These phenotypes are metabolic pathways that are often not apparent in cell morphology and tissue architecture on H&E-stained slides and sufficiently accuracy in predicting these phenotypes may require more specific stains.

[0194] AM-SB and AM-MB-based architectures marginally outperformed TransMIL- based architectures in predicting all phenotypes assessed. This suggests that the type of DL architecture may not play a crucial role in overall image assessment model performance.

[0195] The nature of the pre-trained feature extractor component did affect performance of the image assessment model. Models containing the PLIP feature extractor component generally outperformed models containing the ResNet50 feature extractor component. To highlight this comparison, Table 2 presents data from Table 1 for the bestAUROC for each phenotype assessed (regardless of the DL component of the image assessment model).

[0196] ResNet50 is a convolutional neural network pre-trained on ImageNet (baseline), which contains a variety of images that are not pathology- specific.

[0197] PLIP is a vision transformer employing contrastive learning with 210,000 paired pathology image-text descriptions from Twitter. A vision transformer is a DL-based model with improved performance as compared to convolutional neural networks.

[0198] A protocol similar to that used to produce the data in Table 1 was implemented, but with CONCH as the feature extractor. CONCH is a vision transformer employing contrastive learning with over 1.1 million paired pathology image-captions from several tertiary care centers (Lu et al. 2024). The best AUROC for each phenotype assessed (regardless of the DL model used in the complete DL-based classifier) is also presented in Table 2.

[0199] Both PLIP and CONCH use an image encoder and a text encoder, trained using a number of contrastive alignment objections that seek to align the image and text in the same representation.

[0200] Date presented in Table 2 represents the output of the various assessed models without the benefit of any data augmentations.Table 2: Best AUROC By Phenotype for ResNet50, PLIP, and CONCH Without Data Augmentation

[0201] This data suggests that image assessment models using feature extractor components pre-trained using general pathology images such as PLIP and CONCH) can better learn essential features or relationships specific to histopathology than feature extractors pre-trained with images not specific to pathology (such as ResNet50). Which pretrained feature extractor component would perform better, or even if the component would cause a meaningful difference in AUROC for the complete, trained image assessment modelwas not clear prior to testing. For several other medical tasks, feature extraction strategies more native to the modality of interest do not always outperform natural image-based feature extraction (Komblith, Shlens, and Le 2019; Zhou et al. 2020; Morid, Borjali, and Del Fiol 2021; Li et al. 2022).

[0202] In this example, PLIP had an additional computational advantage of only extracting 512 features from an image patch to create a 512-dimensional feature vector that is easier to process and store than the 1024-dimensional feature vectors created by ResNet50.

[0203] To assess whether attention maps generated from trained image assessment models could be used to identify explainable morphological descriptors that may govern tumor and immune function from our trained models, an attention map was generate from an H&E-stained whole slide image using a trained model containing AM-SB architecture. The attention maps are presented in Fig. 10, with notations indicating areas identified by a pathologist as relevant to the phenotype classified by the trained DL-based classifier. Notations and shading also indicate which areas had high attention scores, and which had low attention scores.

[0204] The attention map presented in Fig. 10A was generated using a DL-based classifier trained to classify T-cell mediated cytotoxicity. An area identified by a pathologist as having a high degree of infiltrating lymphocytes had a high attention score. An area identified by a pathologist as necrotic tissue had a low attention score. This indicates that the DL-based classifier properly identified areas of the image relevant to the phenotype of interest.

[0205] The attention map presented in Fig. 10B was generated using a DL-based classifier trained to classify epithelial-mesenchymal transition. An area identified by a pathologist as having irregular nests of invasive tumor cells had a high attention score. An area identified by a pathologist as stromal tissue had a low attention score. This indicates that the DL-based classifier properly identified areas of the image relevant to the phenotype of interest.

[0206] The attention map presented in Fig. 10C was generated using a DL-based classifier trained to classify angiogenesis. An area identified by a pathologist as having scattered tumor cells and a possible blood vessel had a high attention score. An area identified by a pathologist as having scattered lymphocytes had a low attention score. Thisindicates that the DL-based classifier properly identified areas of the image relevant to the phenotype of interest.

[0207] The attention map presented in Fig. 10D was generated using a DL-based classifier trained to classify immunosuppression. An area identified by a pathologist as having scattered tumor cells with adjacent lymphocytes had a high attention score. An area identified by a pathologist as stromal tissue had a low attention score. This indicates that the DL-based classifier properly identified areas of the image relevant to the phenotype of interest. Furthermore, pathologists are not able to distinguish regulatory T cells (which are exhausted lymphocytes that contribute to immunosuppression) from other T cell types (which may contribute to immune activation or directly mediate immune activities), but the DL- based classifier seems to be able to differentiate various immune cell types and functions.

[0208] The attention map presented in Fig. 10E was generated using a DL-based classifier trained to classify T-cell cytotoxicity, epithelial-mesenchymal transition, and angiogenesis. The WSI with attention map is shown in the left. Tiles with highest and lower attention scores and randomly sampled tiles are shown in the right. These tiles were presented to board certified pathologists who predicted, using only these tiles, whether the WSI represented high or low expression of the relevant phenotype (or whether the phenotype was not determinable (NA))) for T-cell cytotoxicity, epithelial-mesenchymal transition, and angiogenesis. This allowed assessment of the biological accuracy of attention maps on a WSI level. Example results for T-cell cytotoxicity are presented in Table 3 (similar results were obtained for the other two phenotypes) and indicate that the attention maps reflected biology relevant to the phenotypes.T able 3: Pathologist Assessment of Phenotypes Based on Selected Image Tiles

[0209] This provides additional evidence that the trained DL-based classifiers learned therapeutically relevant patterns.

[0210] Overall, the data established that tumor and immune phenotypes can be predicted using digital histopathology images from H&E-stained slides. This ability may have major implications for cancer patients and precision oncology as many hospitals only possess the capability for H&E and lack the resources for DNA / RNA sequencing.Example 2 - Data Augmentation and Accuracy of Phenotype-Trained DL-Based Classifiers

[0211] Data augmentation is widely recognized for its ability to enhance model performance and generalizability, particularly in tasks with limited training data. Two publicly available datasets were used to evaluate various strategies for improving accuracy in phenotype classification by phenotype-trained DL-based classifiers of the present disclosure.

[0212] Despite advancements in image analysis and machine learning techniques, the scarcity of annotated samples and the high intra-class variability inherent to histological images impede the accuracy and reliability of automated diagnostic models. In the face of such limited data with which to train DL models, image augmentation is often used to increase the diversity of training datasets without acquiring new labeled data. However, while the application of image augmentation is straightforward in most image analysis tasks, it is more challenging when using state-of-the-art multiple instance learning (MIL) methods for digital pathology, such as the CLAM model and TransMIL. These DL models does not allow for “on-the- fly” image augmentation because feature vectors are pre-computed and fixed prior to training the DL model. Doing otherwise would dramatically increase training times and computational requirements. Moreover, multiple pre-computed versions of augmented tiles from an initial image result in a huge increase in storage requirements due to the very large size of the tile files (which are also image files, albeit smaller than the original image files). Consequently, in most image data sets for training DL models, there is no image-level augmentation. In the present example, various data augmentations were implemented to produce different sets of feature vectors, which were then used to train DL models. By assessing effects of different types of data augmentation, alone or in combinations, on accuracy of phenotype classification, accuracy effects of these different types and combinations of data augmentation were systematically evaluated.

[0213] Comparative evaluations using a dataset of breast cancer images, and a dataset of ovarian cancer images were performed to determine if data augmentations would have different benefits depending on the type of cancer being classified. Evaluation of an ovarian cancer dataset also allowed evaluation of the DL-based classifier’s ability to handle variations in image types, dimensions, and H&E-staining techniques.Datasets of Identified Digital Histopathology Images

[0214] Identified images were obtained from two public databases, The CancerGenome Atlas Breast Invasive Carcinoma (TCGA- BRCA) and OCEAN (also referred to asUBC-Ocean, as described in Ali Bashashati et al., 2023), each with an accompanying identification, such as cancer type or subtype.

[0215] The first dataset of identified images included 1,110 H&E-stained WSI microscopy images from breast tissue obtained from TCGA-BRCA for the task of histological subtyping. The images used corresponded to the three main types of invasive carcinoma: 700 images corresponded to ductal carcinoma, 300 to lobular carcinoma, and 110 were mixed, presenting characteristics of both types. The prevalence of ductal carcinoma over the other breast cancer subtypes could lead to a biased model susceptible to overfitting, and is, therefore, particularly useful in examining the effects of image augmentation on the ability of DL-based classifiers to correctly classify images from the minority classifications (lobular carcinoma and mixed). When using this dataset to train DL models and assess accuracy, a 10-fold cross-validation strategy was used and slides from the same patient were consistently grouped, adhering to the CLAM framework’s guidelines for training (75%), validation (12.5%), and testing (12.5%) splits.

[0216] The second dataset of identified images included 2,438 H&E-stained images from both WSIs and Tissue Microarrays, sourced from 24 centers, primarily part of the Ovarian Tumor Tissue Analysis (OTTA) consortium, and located in the OCEAN database, which is designed for ovarian cancer subtype classification and outlier detection. Specifically, 538 images are allocated to the training dataset, while 437 and 1,488 images are designated for the public and private test sets, respectively. The training set included images representing the five main subtypes of ovarian carcinoma: clear-cell ovarian carcinoma (CCOC), endometrioid carcinoma (ENOC), high-grade serous carcinoma (HGSC), low-grade serous carcinoma (LGSC), and mucinous carcinoma (MUC). The public and private test sets also include 147 outlier images (labeled as “Other”), encompassing rare ovarian cancer subtypes and normal tissues. However, in this example, only the 538 images from the public training set were used.Feature Extraction

[0217] Following the methods of Example 1, feature vectors were extracted for each tile in the WSIs using a ResNet-50 model or a CONCH foundational model. Use of both general image-trained and pathology-image trained feature extractors allowed the evaluation of whether pathology-image trained feature extractors may have learned invariance tocommon sources of variation within histopathology images, thus reducing the need for additional data augmentation when using extracted features to train a DL model.Tile-Level Tile-Shifting Data Augmentation

[0218] A tile-shifting data augmentation technique was used to enhance the DL model’s generalization capability from spatial variations. In tile-shifting, the coordinates for tile extraction from WSIs were modified to collect different sets of tiles. This technique, which is depicted in Fig. 6, simulates spatial variability by adjusting the positions from which tiles were created. To update the position of a given tile’s coordinates (x, y) extracted using the CLAM methodology, extra displacements were applied based on the specified parameters. For a vertical shift upwards, the vertical coordinate was adjusted by setting ynew = y - P / 2. For a diagonal right-upward shift, both vertical and horizontal coordinates were adjusted by setting xnew = x+ P / 2 and ynew = y - P / 2. A rightward shift increased the horizontal coordinate to xnew = x+P / 2. For a diagonal right-downward shift, both horizontal and vertical coordinates were adjusted to xnew = x + P / 2 and ynew = y + P / 2. P represents the tile side length. This process resulted in an additional four sets of shifted tiles, which were ultimately used to create an additional four sets of pre-computed feature vectors.Tile-Level HED Color Augmentation

[0219] Tiles were augmented in the Hematoxylin-Eosin-DAB (HED) color space to simulate variations in staining between slides. This technique, which is depicted in Fig. 7, included first transforming the RGB images into the HED color space, performing adjustments in the intensities of the three components (hematoxylin, eosin, and DAB), and then transforming back to the RGB space. Color- augmented tiles were created and features were extracted to create feature vectors at n = 4 equally-spaced values for each of the three components (HED) between the lower and upper limits of a = -0.5 and b = 0.5, respectively, to provide an extensive spectrum of variations. This created a total of 27 additional feature vectors for each original tile.Feature-Level Pseudo-Vector(Bag) Mixup

[0220] Pseudo-Vector (Bag) Mixup (PseMix) is a feature vector-level data augmentation technique that was applied to the sets of feature vectors. This method (and variations thereof in Liu et. al.2024) creates new synthetic feature vectors by interpolating between existing, non-synthetic feature vectors. By operating directly on the feature vectors rather than the tile images, this data augmentation method is efficient and avoids theproblems associated with storing large image files. It also enhances the DL model’s ability to generalize by promoting linear behavior between training examples. However, linear interpolation between existing feature vectors represents a more limited and less realistic source of variation than tile-level data augmentation.Whole Digital Medial Image (Slide)-Level Random Feature Vector Selection

[0221] Consider a set of feature vectors F = { Forig. Faug.i, . . . , Faug.n}, where Forig represents the original subset of feature vectors obtained without any data augmentation and Faug,i is the ith subset of feature vectors generated using data augmentation, where each feature vector subset includes all feature vectors derived from each tile in the given digital histopathology image. To incorporate diversity into the DL model training process, iteration subsets were randomly chosen using a defined probability distribution. The probability distribution over the full feature vector set P(F) was defined such that the probability of selecting Forig was p,,rig. and the probability of selecting any Faug,i was uniformly paug- It was established that porig > paug, slightly favoring the original feature vector set over the augmented feature vector sets to maintain a solid learning base. The probability of selecting a subset of augmented feature vectors was equally distributed among all Faug,i, with paug= I- P origin .

[0222] During training, the feature vector selection process for a given iteration subset was formalized as follows:> t Forig with probability of porig I Faug. i with probability of paug for each i

[0223] Selection was made by random sampling according to P(F), allowing the integration of variability in each iteration set while preserving the predominant influence of the original feature vectors.Tile-Level Random Feature Vector Selection

[0224] A similar process was used for tile-level random feature vector selection. Tile-level random feature vector selection involved randomly selecting a feature vector F for each tile, whereas the slide-level approach selected an entire set F at once for a given slide. The tile-level approach maximized the feature vector randomization during training whereas the slide-level approach ensured that the variations introduced maintain spatial coherence throughout the entire image, reflecting variabilities that could be observed under different image acquisition conditions. Both random feature vector selection techniques for dataaugmentation were implemented to improve the robustness and generalization of the DL model in the face of the inherent heterogeneity of the histological data.Interaction between Image-Level and. Feature-Level Augmentations

[0225] PseMix was used to generate synthetic feature vectors by interpolating feature vectors obtained from different tile-level data augmentations. This technique was integrated directly into the DL model training iterations, ensuring that the DL model was exposed to a range of feature vectors during each training epoch. The augmentation was performed by first collecting feature vectors that corresponded to both the original tiles and the tile-level data augmented tiles.Feature Vector Dataset and Selection

[0226] During each iteration of DL model training, feature vectors are gathered from both the original and data augmented tiles. The data augmentation techniques included HED color modifications and tile shifting. Feature vectors based on original and data augmented tiles were created and stored. PseMix was applied by selecting pairs of feature vectors, one from the original set and another from a data augmented set. The selection process followed a probabilistic approach, governed by predefined probabilities that controlled the likelihood of choosing a feature from either the original or augmented feature vectors. This probability distribution was carefully calibrated to maintain a balance between preserving the original feature characteristics and introducing variability.Interpolation and Synthetic Feature Generation

[0227] The selected feature vectors were linear interpolated to generate new synthetic feature vectors. This interpolation was a weighted combination of the original and augmented feature vectors, where the weights were determined by the selected probabilities. The resulting synthetic feature vectors were then used in the input layer for training the DL model. The interpolation was conducted at both the tile and slide levels, depending on the strategy employed.

[0228] For tile-level augmentations, feature vectors corresponding to individual tiles were interpolated, allowing for fine-grained adjustments within a slide. At the slide level, the entire set of feature vectors corresponding to the slide was treated as a single entity, with interpolation performed across the entire slide.Integration into the Training Loop

[0229] The process of feature selection, interpolation, and synthetic featuregeneration was integrated into the training loop, which generated feature vectors as the input for each training iteration for the DL model. As each new batch of feature vectors for use as the DL model input was processed, PseMix was applied dynamically, enabling the DL model to utilize a diverse set of feature representations during training. This approach ensured that the DL model was continually exposed to a wide range of data augmented feature vectors, without the need for extensive on-the-fly computation.

[0230] The implementation of PseMix required careful consideration of the computational resources and storage requirements for the sets of augmented feature vectors. By precomputing the data augmented feature vectors and efficiently managing the selection and interpolation processes within each training loop, the data augmentation strategy was optimized to operate within the constraints of the computational environment.Effects of Data Augmentation

[0231] Data augmented feature sets were used to train a CLAM DL model. For each trained model, a portion of the dataset (75%) was used for training, while other portions were reserved for validation (12.5%) and testing (12.5%).

[0232] Accuracy, expressed as AUROC, for each model is presented in Table 4.Table 4. Comparative Analysis of CLAM Performance Utilizing Various Data Augmentation Techniques, As Measured by Validation and Test AUROC Scores

[0233] The results indicate that, without data augmentation, the breast cancer subtype classification-trained DL model achieved a validation AUROC of 0.769 ± 0.096 and a test AUROC of 0.758 ± 0.096. Introducing HED color augmentation and tile shifting augmentation at different levels (tile, slide, or feature- vector) yielded varying improvements. The most significant increase was observed when both techniques were applied at the tile level, leading to a validation AUROC of 0.839 ± 0.064 and a test AUROC of 0.849 ± 0.066.

[0234] Furthermore, combining image-level data augmentation and feature-level data augmentation, specifically HED color augmentation and PseMix augmentation, resulted in the highest AUROC improvements. With the CONCH feature extractor, combining HED color augmentation and PseMix augmentation at the slide and feature levels achieved a test AUROC of 0.888, demonstrating the complementary nature of these techniques.

[0235] However, when the DL model was trained to classify ovarian cancer subtypes suing the UBC-OCEAN dataset, the same combination of HED color augmentation and PseMix augmentation only resulted in a marginal increase of 0.001 in test AUCROC. This minimal improvement indicates that the benefit of data augmentations may vary, even when the same type of digital medial image is used (here H&E-stained slides) depending on the phenotype to be classified.

[0236] Given the that the DL model accuracy in the absence of data augmentation was higher for the model trained to classify ovarian cancer subtypes than the for the model trained to classify breast cancer subtypes, the data may indicate that accuracy improvements due to data augmentation may be more marked for DL models that are less accurate absent data augmentation. Additionally, given that the TCGA BRCA dataset was characterized by high intra-class variability, while the UBC-OCEAN dataset was more homogeneous and well-labeled, the data indicate that information regarding the training dataset available in advance of DL model training may be used to select data augmentations that are likely to achieve accuracy improvements that are with the increased computational requirementsassociated with data augmentation as compared to using unaugmented data. In particular, the accuracy improvements seen in the DL model trained with the TCGA BRCA dataset when data augmentations were implemented indicates the importance of introducing spatial and color diversity to enhance DL model generalization in the case of complex training datasets. The limited accuracy improvements seen in the DL model trained with the UBC-OCEAN dataset indicates that data augmentation may be less useful when training DL models using datasets with high consistency and low overfitting risk.

[0237] The data provided in Table 4 also indicates that the choice of feature extractor greatly affected the effects of data augmentation on DL model accuracy. In particular, when the ResNet50 feature extractor was used, data augmentation resulted in substantial improvements in accuracy of the trained DL models. When the CONCH feature extractor was used, accuracy improvements from data augmentation were more limited. This suggests that data augmentation may be more useful in DL-based classifiers that use a general image pre-trained feature extractor than in those that use a pathology image pre-trained feature extractor. However, for the TCGA BRCA dataset, significant improvements in accuracy were achieved through data augmentation even when the CONCH feature extractor was used.

[0238] Combining HED color augmentation and tile shift augmentations generally improved DL model accuracy. However, the addition of feature-level data augmentation via PseMix did not always result in the highest accuracy gains. For example, the test AUROC for HED + Tile Shift + PseMix at the tile level was slightly lower than that for HED + Tile Shift alone. This suggests that certain combinations of data augmentations may not always be synergistic, and the effectiveness of data augmentations may be context-dependent.

[0239] In general tile-level data augmentation seemed to result in the greatest improvements in DL model accuracy.

[0240] Overall, the findings in this example indicated that data augmentation may be most beneficial in exploratory tasks or in scenarios with moderate DL model accuracy in the absence of data augmentation or limited datasets.Example 3 - Effects of Classification Types on DL-Based Classifier Accuracy

[0241] The effects on DL-based classifier accuracy of using binary phenotype classifications as ground truth as compared to regression-based classifications was evaluated. Specifically, a regression was performed to predict a continuous value for each phenotypeusing a Pearson’s correlation coefficient. Binary classifications, although useful in some contexts, may not be useful in many other contexts. For example, simply knowing whether a phenotype is up-regulated or down-regulated (via comparison to a threshold) may not be sufficient to determine a diagnosis or locate a biological mediator. Results are presented in Table 5.Table 5. DL-Based Classifier Binary and Regression Results ComparisonExample 4 - Comparison of Breast Cancer Phenotype DL-Based Classifier, Clinical Metadata Learning Models, and Multi-Modal Classifiers

[0242] To determine the effects of when clinical metadata is provided to the DL model, a set of clinical metadata including genomics data including assessments for HER2, ER, PR, as well as clinical data including age, race, cancer stage, and tumor grade was introduced early in the training process, late in the training process, or at an intermediate time in the training process, as described in Lipkova 2022 (which indicated that there is no clear consensus regarding when such data is best provided, also referred to as when data fusion occurs).

[0243] In the early fusion classifier, representations, such as feature vectors, from the different data sources (clinical metadata and digital histopathology images) were integrated into the training process prior to being provided as inputs to the DL model. In the late fusion classifier, representations from the different data sources were used to train a different DL model for each data source, and then outputs from the different DL models were aggregated in a final classification. In the intermediate fusion classifier, also referred to as a crossattention classifier, information from data sources is aggregated in different fashions before being provided as an input layer to train the DL model. In some instances of the intermediate fusion classifier, the data from one source may be affect what data is used from the othersource(s), or how such data from the other source(s) is used. Data presented in Table 6 compares multi-modal classifiers and single-modal classifiers. Them multi-modal classifiers demonstrate superior performance as compared to the single-modal classifiers.Table 6. DL-Based Classifier Accuracy for Single-modal and Multi-modal ClassifiersExample 5 - Predicting Breast Cancer Patient Response to Checkpoint Inhibitors

[0244] Keynote-522 includes chemotherapy in combination with the checkpoint inhibitor, pembrolizumab. Keynote-522 results in a complete response in 64% of early stage triple negative breast cancer (TNBC) patients treated. However, a substantial proportion of patients, 36%, did not benefit from Keynote-522 treatment or showed less than a compete response. (Schmid et al., NEJM 2020). Although digital H&E images correlated with RNA sequence data are not widely available, digital H&E images correlated with KN-522 outcome is much more easy to locate. Accordingly, KN-522 response was used as a surrogate for expression of increased immune activity associated with this clinical outcome.

[0245] 78% of the patients (independent of response to treatment) suffered a grade 3 or higher adverse event. A grade 3 or higher adverse event is an adverse event that is severe or medically significant. A grade 3 event is not life threatening, but may require medical intervention. A grade 4 adverse event is life threatening, while a grade 5 adverse event results in death of the patient. (Schmid et al., NEJM 2020)

[0246] Given the prevalence of quite severe and dangerous adverse events in patients administered Keynote-522, coupled with the expectation that approximately a third of patients will not obtain the full therapeutic benefit of the treatment, there is a significant clinical need to be able to identify patients who will likely not respond to Keynote-522. If such non-responder patients cannot be identified in advance, they may not only be administered different treatments that may be more effective in controlling TNBC in that patient, these non-responder patients may also avoid being unnecessarily subjected to the risk of severe side effects. Unfortunately, there is presently no predictive biological mediator for efficacy of Keynote-522. Biological mediators likely to have predictive value, such as PD- L1 expression on tumors or tumor mutational burden, have been evaluated and found to not have predictive value. (Schmid et al., NEJM 2020)

[0247] Checkpoint inhibitors efficacy relies on robust immune function in the tumor microenvironment. However, there are presently no clinically usable assays able to quantify immune function for individual patients.

[0248] A DL model according to the present disclosure trained with images of breast cancer tissue (tumor tissue and surrounding tissue) to predict immune function-related phenotypes was used to predict early stage triple negative breast cancer (TNBC) patient response to treatment with Keynote-522. Existing H&E-stained digital histopathology images of tissue samples obtained prior to Keynote-522 treatment, along with data indicating whether the corresponding patient responded fully to Keynote-522, was used to determine whether the model can accurately predict patient response to Keynote-522.

[0249] The trained model was used to predict the ten phenotypes identified in Example 2. Results for the phenotypes from all images and from a subgroup of 97 patients in the form of P values are presented on Table 7. P value represents the likelihood of a statistically significant relationship between expression of a phenotype and a therapeutic response.Table 7. Prediction of Response to Keynote-522 Using Breast Cancer Phenotype DL- Based Classifier

[0250] The data in Table 7 indicate that predictions of increased expression of T-cell cytotoxicity and increased antigen processing and presentation, and decreased expression of epithelial-mesenchymal transition correlated with the patient being likely to respond Keynote-522.Example 6 - Predicting Breast Cancer Patient Response to Chemotherapeutics

[0251] Dose-dense doxorubicin, dose-dense cyclophosphamide, paclitaxel (ddAC-T) therapy is a common regimen for breast cancer patients. There is no predictive biological mediator for efficacy of this treatment for TNBC patients. However, a high immune function and high proliferative activity are linked to chemo sensitivity for ddAC-T (i.e. response to the therapy).

[0252] A DL model according to the present disclosure trained with images of breast cancer tissue was provided a dataset of images from patients that were labeled based on whether the patient had a future recurrence of breast cancer and was in remission. The dataset included 84 digital histopathology images of H&E- stained slides from tumors taken from TNBC patients who received ddAC-T therapy. The DL model was used to predict therapeutic response to ddAC-T therapy. Accuracy was assessed and is indicated in Table 8. A correlation of high T-cell cytotoxicity and cell cycling with remission was observed.Table 8: Prediction of Response to ddAC-T Using Breast Cancer Phenotype DL-Based Classifier

[0253] It will be appreciated by those skilled in the art that while the disclosed subject matter is described above in connection with particular embodiments and examples, the invention is not necessarily so limited, and that numerous other embodiments, examples, uses, modifications and departures from the embodiments, examples and uses are intended to be encompassed by the claims attached hereto. Each reference cited herein is incorporated by reference in its entirety.

[0254] Various features and advantages of the invention are set forth in the following claims.

[0255] The various embodiments described above can be combined to provide further embodiments. All of the U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications and non-patent publications referred to in this specification and / or listed in the Application Data Sheet, including U.S. Patent Application No. 63 / 555,957, filed February 21, 2024, and U.S. Patent Application No. 63 / 687,878, filed August 28, 2024, are incorporated herein by reference, in their entirety. Aspects of the embodiments can be modified, if necessary to employ concepts of the various patents, applications and publications to provide yet further embodiments.

[0256] These and other changes can be made to the embodiments in light of the above-detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled. Accordingly, the claims are not limited by the disclosure.

Claims

CLAIMS1. A method of training a deep learning (DL)-based classifier to classify a digital histopathology image with respect to a phenotype, the method comprising: creating a dataset of identified digital histopathology images comprising, for each identified digital histopathology image, an image and a phenotype ground truth label that reflects a known phenotype classification for the image; creating, for each identified digital histopathology image, a set of image- specific tiles, wherein each tile represents an area of the identified digital histopathology image that is less than the whole digital histopathology image; extracting a features from each tile using a pre-trained feature extractor to create a tile-specific feature vector for each tile; assembling a set of tile-specific feature vectors for each identified medical image; providing, for at least a subset of identified medical images, the set of tile-specific feature vectors and the phenotype ground truth label to an untrained DL model; training the DL model in a self-supervised manner using the provided sets of tilespecific feature vectors and phenotype ground truth label to classify digital histopathology images with respect to the phenotype; assessing accuracy of the DL model in classifying digital histopathology images with respect to phenotype; and comparing assessed accuracy to an accuracy threshold and, if the accuracy threshold is met, storing the DL model.

2. The method of claim 1, wherein the digital histopathology image comprises n image of a hematoxylin and eosin (H&E)-stained slide.

3. The method of claim 1, wherein the digital histopathology image comprises an image of a Perl’s Prussian blue, Alcian blue, Masson’s trichome, Reticulin stain, silver stain, periodic acid Schiff Reaction (PAS) Stain, Congo red, Mucicarmine, Sudan black, Nissl stain (also called the Cresyl Violet Stain), or Golgi method stained slide.

4. The method of any one of claims 1-3, wherein the digital histopathology image comprises a whole slide image (WSI) or near WSI and the set of image- specific tiles collectively represents the entire area of the WSI or near WSI.

5. The method of any one of claims 1-4, wherein each tile represents no more than 5% of total area of the digital histopathology image.

6. The method of any one of claims 1-5, wherein the phenotype classification is a binary classification.

7. The method of any one of claims 1-5, wherein the phenotype classification is a graduated classification.

8. The method of any one of claims 1-7, wherein the pre-trained feature extractor comprises a vision transformer trained using paired image-text descriptions.

9. The method of any one of claims 1-8, wherein the pre-trained feature extractor has been trained using paired pathology image-text descriptions in a non-task specific manner.

10. The method of any one of claims 1-9, wherein the untrained DL model comprises an attention-based model.

11. The method of any one of claims 1-10, wherein the DL model comprises a neural network and a final classifier that uses an output layer of the neural network to provide a phenotype classification.

12. The method of any one of claims 1-11, wherein accuracy is assessed by determining the area under receiving operator curve (AUROC).

13. The method of any one of claims 1-12, wherein the accuracy threshold is at least 70%.

14. The method of any one of claims 1-13 further comprising preparing augmented data for each identified digital histopathology image.

15. The method of claim 14, wherein the augmented data comprises at least one of tile-shifted data, color augmented data, and pseudo-vector mixup data.

16. The method of any one of claims 1-15, wherein the phenotype comprises a cancer pathophysiology.

17. The method of claim 16, wherein the phenotype comprises at least one of: tertiary lymphoid structures, B-cell proliferation, T-cell mediated cytotoxicity, FoxP3- mediated immunosuppression, antigen processing and presentation, tumor associated fibroblasts, angiogenesis, epithelial-mesenchymal transitions, tumor cell invasiveness, fatty acid metabolism, glycolysis, and oxidative phosphorylation.

18. A deep learning (DL)-based classifier trained to classify a digital histopathology image with respect to a phenotype, wherein the classifier comprises feature extractor and DL model trained according to any one of claims 1-17.

19. A system for training a deep learning (DL)-based classifier to classify a digital histopathology image with respect to a phenotype, the system comprising one or more processors and a non-transitory computer-readable medium including one or more sequences of instructions that, when executed by the one or more processors, cause the system to perform operations comprising the method of any one of claims 1-17.

20. A method of classifying a digital histopathology image with respect to a phenotype, the method comprising: creating a set of tiles, wherein each tile represents an area of the digital histopathology image that is less than the whole digital histopathology image; extracting a features from each tile using a pre-trained feature extractor to create a set of tile- specific feature vectors;providing the set of file-specific feature vectors to a phenotype-trained DL model trained according to any one of claims 1-17 to obtain a phenotype classification for the digital histopathology image; and storing the phenotype classification.

21. The method of claim 20, wherein the digital histopathology image comprises an image of a hematoxylin and eosin (H&E)-stained slide.

22. The method of claim 20, wherein the digital histopathology image comprises an image of a Perl’s Prussian blue, Alcian blue, Masson’s trichome, Reticulin stain, silver stain, periodic acid Schiff Reaction (PAS) Stain, Congo red, Mucicarmine, Sudan black, Nissl stain (also called the Cresyl Violet Stain), or Golgi method stained slide.

23. The method of any one of claims 20-22, wherein the digital histopathology image comprises a whole slide image (WSI) or near WSI and the set of tiles collectively represents the entire area of the WSI or near WSI.

24. The method of any one of claims 20-23, wherein each tile represents no more than 5% of total area of the digital histopathology image.

25. The method of any one of claims 20-24, wherein the phenotype classification is a binary classification.

26. The method of any one of claims 20-24, wherein the phenotype classification is a graduated classification.

27. The method of any one of claims 20-26, wherein the pre-trained feature extractor comprises a pre-trained feature extractors according to any one of claims 1 and 7-8.

28. The method of any one of claims 20-27, wherein the phenotype comprises a cancer pathophysiology.

29. The method of claim 28, wherein the phenotype comprises at least one of: tertiary lymphoid structures, B-cell proliferation, T-cell mediated cytotoxicity, FoxP3-mediated immunosuppression, antigen processing and presentation, tumor associated fibroblasts, angiogenesis, epithelial-mesenchymal transitions, tumor cell invasiveness, fatty acid metabolism, glycolysis, and oxidative phosphorylation.

30. The method of any one of claims 20-29, further comprising providing the phenotype classification to a clinical who, based upon the phenotype classification undertakes a treatment regimen or prescribes further diagnostic assays for a patient having a tissue used to create the medical diagnostic image.

31. A system for classifying a digital histopathology image with respect to a phenotype, the system comprising one or more processors and a non-transitory computer- readable medium including one or more sequences of instructions that, when executed by the one or more processors, cause the system to perform operations comprising the method of any one of claims 20-30.

32. A method of identifying a biological mediator associated with a pathology, pathophysiology, or other clinical characteristic, the method comprising: creating a set of tiles for each image in a dataset of digital histopathology images, wherein each tile represents an area of a digital histopathology image that is less than the whole digital histopathology image, wherein each image is labeled with respect to the pathology, pathophysiology, or other clinical characteristic, and wherein the dataset comprises digital histopathology images with at least two different labels for the pathology, pathophysiology, or clinical characteristic; extracting a features from each tile using a pre-trained feature extractor to create a set of tile-specific feature vectors for each digital histopathology image; providing the sets of tile-specific feature vectors and labels to a DL model; assessing the sets of tile-specific feature vectors and labels using the DL model to identify differences between the images based on the labels and to provide a set of attention scores or an attention map associated with areas of interest in at least a portion of the digital histopathology images; analyzing a tumor tissues corresponding to the areas of interest for at least some of the digital histopathology image for a difference in a characteristic that correlates with a difference in the labels; andidentifying the characteristic as a biological mediator if a difference correlating with the difference in labels exists.

33. The method of claims 32, wherein the biological mediator comprises a biological mediator of therapeutic resistance.

34. The method of claim 32 or claim 33, wherein the DL model comprises a phenotype-trained DL model trained accordingly to any one of claims 1-17 using a tissue type of the digital histopathology images.

35. The method of any one of claims 32-34, wherein the DL model identifies between 60% and 80% of the digital histopathology images on average, as not an area of interest.

36. The method of any one of claims 32-35, wherein analyzing the tumor tissues corresponding to the areas of interest comprises in situ analysis of tissue sections adjacent tissue sections used to create the digital histopathology images.

37. The method of claim 36, wherein the in situ analysis comprises analysis of the genome, transcriptome, or proteome of the tissue.

38. The method of any one of claims 32-37, wherein the pathology, pathophysiology, or other clinical characteristic comprises whether the patient responds to a particular therapy.

39. The method of any one of claims 32-38, wherein t wherein the pathology, pathophysiology, or other clinical characteristic comprises a characteristic relating to a cancer, infection, or autoimmune disease.

40. The method of any one of claims 32-39, wherein each digital histopathology image comprises an image of a hematoxylin and eosin (H&E)-stained slide.

41. The method of any one of claims 32-40, wherein each digital histopathology image comprises an image of a Perl’s Prussian blue, Alcian blue, Masson’s trichome,Reticulin stain, silver stain, periodic acid Schiff Reaction (PAS) Stain, Congo red, Mucicarmine, Sudan black, Nissl stain (also called the Cresyl Violet Stain), or Golgi method stained slide.

42. The method of any one of claims 32-41, wherein each digital histopathology image comprises a whole slide image (WSI) or near WSI and the set of tiles collectively represents the entire area of the WSI or near WSI.

43. The method of any one of claims 32-42, wherein each tile represents no more than 5% of total area of the digital histopathology image.

44. The method of any one of claims 32-43, wherein the pre-trained feature extractor comprises a pre-trained feature extractors according to any one of claims 1 and 7-8.

45. A system for identifying a biological mediator associated with a pathophysiology using a digital histopathology image exhibiting the pathophysiology, the system comprising one or more processors and a non-transitory computer-readable medium including one or more sequences of instructions that, when executed by the one or more processors, cause the system to perform operations comprising the method of any one of claims 32-44.

46. A non-transitory computer-readable medium storing a deep learning (DIFbased classifier trained to classify a digital histopathology image with respect to a phenotype, wherein the DL-based classifier was trained by a process comprising: creating a dataset of identified digital histopathology images comprising, for each identified digital histopathology image, an image and a phenotype ground truth label that reflects a known phenotype classification for the image; creating, for each identified digital histopathology image, a set of image- specific tiles, wherein each tile represents an area of the identified digital histopathology image that is less than the whole digital histopathology image; extracting a features from each tile using a pre-trained feature extractor to create a tile-specific feature vector for each tile; assembling a set of tile-specific feature vectors for each identified medical image;providing, for at least a subset of identified medical images, the set of tile-specific feature vectors and the phenotype ground truth label to an untrained DL model; training the DL model in a self-supervised manner using the provided sets of tilespecific feature vectors and phenotype ground truth label to classify digital histopathology images with respect to the phenotype; assessing accuracy of the DL model in classifying digital histopathology images with respect to phenotype; and comparing assessed accuracy to an accuracy threshold and, if the accuracy threshold is met, storing the DL model.

Images