Analysis of histopathology samples

Abstract

Computer-implemented methods of analysing a histopathology sample are described, comprising obtaining a plurality of tile representations using a tile representation machine learning model, assigning each of the plurality of tile representations to one of a predetermined set of histomorphological phenotype clusters, obtaining a whole slide image representation using a histomorphological phenotype cluster language model, and predicting one or more biological or clinical features associated with the sample using a task specific machine learning model, wherein the task specific machine learning model is a model that has been trained using training whole slide images and optionally associated one or more ground truth biological or clinical features of interest to predict the one or more biological or clinical feature of interest for a whole slide image using as input the whole slide image representation provided by the histomorphological cluster language model for the whole slide image.

Description

[0001] Analysis of histopathology samples

[0002] Field of the Disclosure

[0003] The present disclosure relates to the analysis of histopathology samples. In particular, the present disclosure relates to methods of analysing histopathology images using a combination of selfsupervised and supervised machine learning to identify histomorphological phenotype clusters, patterns of histomorphological phenotype clusters and their relationships with phenotypic or clinical variables of interest.

[0004] Background

[0005] Analysis of histopathology images is a cornerstone of modern pathology, particularly but not exclusively in the context of cancer. Definitive cancer diagnosis and management depend upon the extraction of information from microscopy images by pathologists. These images contain complex information requiring time-consuming expert human interpretation that is prone to human bias. However, pathologist interpretation is time consuming and prone to inter-observer variation, depending on expertise, knowledge, and on the inherent difficulty in characterizing certain tumours or patterns.

[0006] Artificial intelligence, and in particular the use of supervised deep learning methods has emerged as a potential solution to this. Some such approaches have shown to be on par with specialists on tumour classification tasks, and have been employed to tackle more challenging questions, such as predicting genetic alterations, survival and immunotherapy response. However, these approaches require availability of rigorous clinical annotations to properly train supervised models, which is difficult and limits applicability and performance of the models. Further, by limiting the study to annotated features, such approaches also limit the potential discovery of new biomarkers. Finally, such approaches are often described as black-boxes, where interpretation and understanding of how decisions are taken by the network are often difficult and may affect trust, limiting the ability to take well-informed treatment decisions. Semi-supervised approaches and weakly supervised approaches such as multiple-instance learning (MIL) have been proposed for the analysis of whole slide images (see e.g. Yao et al. (2020b)). However, MIL only informs about which individual tile(s) are important for a given task without giving any further information about the broader clinical and biological significance of the tiles.

[0007] Quiros et al. 2024 described an unbiased method to extract histomorphological phenotype representations through self-supervised learning and community detection. Whole slide images are segmented into tiles without overlap, then self-supervised learning is used on a training set of tiles to capture distinct morphological patterns found in tissue and to represent them by vector representations, which can be thought of as feature vectors that describe the visually distinct patterns (e.g. texture). Next, a nearest neighbour graph between tiles, is obtained using the tile vector representations from the previous step. Finally, Leiden community detection is applied to the nearest neighbour graph in order to find Histomorphological Phenotype Clusters (HPCs). These can be used to calculate the ratio of the area covered by the HPC to the total tissue area of all slides from the same patient, which was shown to be predictive of e.g. survival in lung adenocarcinoma using a Cox proportional hazards model.

[0008] However, despite these advances in cell identification in histopathology images, there is still a need for scalable, interpretable, high-performance approaches to analyse histopathology images.

[0009] Summary of the Invention

[0010] The present inventors recognised that a problem associated with many semi-supervised and weakly supervised approaches to histopathology image analysis (e.g. MIL) is that they learn representations of these images that are specific to a given task. This means that models have to be fully retrained in a task specific way, a process that is both inefficient and fails to learn from larger amounts of non-task specific data that may be available to learn general clinical and biological features that may be significant across a plurality of tasks. In an attempt to learn image representations that at least partially alleviate these issues, the inventors designed the concept of Histomorphological Phenotype Learning, in which feature vectors are learned on a tile level, then clusters of morphologically similar tiles are identified and cluster membership proportions at the whole slide level can be used to provide a general histomorphological characterisation of the image. Building on their previous work developing Histomorphological Phenotype Learning (HPL, Quiros et al. 2024), the present inventors postulated that richer representations of whole slide images (WSIs) could be obtained based on assignment of tiles to histomorphological phenotype clusters (HPCs), by applying self-supervised learning to learn the “language of HPCs”. They designed a new method, termed “Idioma” which creates a finite set of HPCs that capture the different morphological tissue patterns, where each tile in a WSI can be assigned to an HPC, then use a language model trained in a self-supervised manner on tile assignment sequences (i.e. vectors of tile assignments to HPCs) to learn a WSI representation. Instead of creating WSI representations by accounting for the percentage of the different HPCs with respect to the total tissue area (as was done in Quiros et al. 2023), this learns WSI representations that account not only for the area of different HPCs but also for how they interact with each other. The inventors showed that this results in an extremely powerful representation that can be leveraged for prediction of a variety of clinically relevant features of the sample such as diagnostic, prognostic, and treatment response features, simply by using these learned representations as predictive features of supervised machine learning algorithms trained for these specific tasks. In particular, the same learned representations were shown to be usable in this way to predict overall survival in 23 different cancer types, classify samples between histological types and subtypes, classify samples between molecular subtypes, and predict genomic alterations, with a performance that equalled or outperformed prior art methods while being superior in applicability and interpretability.

[0011] Accordingly, a first aspect of the invention provides a method of analysing ta histopathology sample, the method comprising: receiving, by a processor, a whole slide image of the sample, obtaining a plurality of tile representations for respective tiles obtained from the whole slide image, using a tile representation machine learning model, wherein a tile representation machine learning model is a machine learning model that has been trained to learn a representation associated with a histopathology image tile, assigning each of the plurality of tile representations to one of a predetermined set of histomorphological phenotype clusters, wherein the set of histomorphological phenotype clusters are clusters of tile representations that have been previously obtained using training tile representations obtained using the tile representation machine learning model for a plurality of training tiles from a plurality of training whole slide images associated with respective samples in a cohort comprising a plurality of histopathology samples, obtaining a whole slide image representation using a histomorphological phenotype cluster language model, wherein the histomorphological phenotype cluster language model is a machine learning model that has been trained in a self-supervised manner using training whole slide images to take as input an indication of the assigned histomorphological phenotype cluster for each of a plurality of tiles of a whole slide image and learn a representation associated with the whole slide image from which histopathological phenotype cluster membership of individual tiles of the whole slide image can be reconstructed, and predicting one or more biological or clinical features associated with the sample using a task specific machine learning model, wherein the task specific machine learning model is a model that has been trained using training whole slide images and optionally associated one or more ground truth biological or clinical features of interest to predict the one or more biological or clinical feature of interest for a whole slide image, using as input the whole slide image representation provided by the histomorphology cluster language model for the whole slide image.

[0012] Therefore, the methods use a language model that exploits a finite set of histomorphological phenotype clusters as a dictionary. The method explicitly converts a whole slide image into a sequence of tokens defined by HPC membership. The histomorphological phenotype cluster language model or whole slide model is therefore no longer operating on image data but on a string of “words” in a finite size dictionary, the words having meaning in the context of histomorphologically similar images. This represents a more efficient way to analyse whole slide images than prior art methods, which does not limit the training of any part of the model other than the task specific machine learning model to task specific data. Methods according to the present aspect may have any one or more of the following optional features.

[0013] In embodiments, the histomorphological phenotype cluster language model has been trained to learn respective token embeddings of a predetermined size corresponding to each of the histomorphological phenotype cluster in the predetermined set of histomorphological phenotype clusters. In other words, the histomorphological phenotype cluster language model may have been trained to learn a dictionary of learned embeddings associated with the finite set of histomorphological phenotype clusters. Thus, the histomorphological phenotype cluster language model may comprise a trainable embedding layer.

[0014] In embodiments, the histomorphological phenotype cluster language model is a sequence model. Thus, the histomorphological phenotype cluster language model may be a model configured to take as input a sequence of tokens. In embodiments, the histomorphological phenotype cluster language model is a deep learning model, optionally recurrent neural network-based model, a long short-term memory model, a convolution-based model, a state-space model, a multilayer perceptron (MLP)-based model, a retentive neural network, or a transformer-based model. In embodiments, the tile representation machine learning model is a deep learning model, optionally a convolutional neural network, a transformer-based model, a graph neural network, a generative adversarial model, a variational autoencoder, a diffusion model, or a model combining convolutional and attention blocks. In embodiments, the tile representation machine learning model has been trained in a self-supervised manner. In embodiments, the task specific machine learning model is a classification model, a regression model, a generative model, or a survival model. In embodiments, the task specific machine learning model is a linear model or multilayer perceptron. In embodiments, the whole slide image and training whole slide images are stained image and / or colour images. In embodiments, the whole slide images and training whole slide images are chemically stained images, immunofluorescence images, immunohistochemistry images or in situ hybridisation images, optionally H&E stained image.

[0015] In embodiments, the whole slide image and training whole slide images are images associated with the same magnification. In embodiments, the magnification is between 2x and 100x, such as e.g. 5x, 10x, 20x or 40x. For example, the whole slide image may have been acquired at 5x magnification and all machine learning models may be trained using training whole slide images that have been acquired at 5x magnification. Images may have been ab initio acquired at the same magnification, or may be downsampled to a common magnification (also referred to herein as resolution).

[0016] In embodiments, the method further comprises obtaining the plurality of tiles from the whole slide image. In embodiments, the plurality of tiles are non-overlapping tiles and / or tiles of predetermined equal size. In embodiments, obtaining the plurality of tiles from the whole slide image comprises obtaining a first plurality of tiles and selecting a second plurality of tiles from the first plurality of tiles that are associated with a percentage of area showing tissue above a predetermined percentage for obtaining respective tile representations using the tile representation machine learning model, optionally wherein unselected tiles are removed or associated with a predetermined identifier.

[0017] In embodiments, the set of histomorphological phenotype clusters are clusters of tile representations that have been previously obtained by clustering the training tile representations using a clustering algorithm, optionally a community detection algorithm applied to a graph where each node is associated with a training tile representation and edges connect training tiles representations with their one or more nearest neighbours in terms of tile representation. In embodiments, the predetermined set of histomorphological phenotype clusters comprises between 20 and 150 clusters. In embodiments, the number of clusters in the predetermined set of histomorphological phenotype clusters has been determined by applying a clustering algorithm to the training tiles representations.

[0018] In embodiments, the histomorphological phenotype cluster language model has been trained using a self-supervised learning objective, optionally wherein the self-supervised learning objective is selected from masked language modelling, causal language modelling, and HPC masking in which all input tokens associated with a particular HPC are masked in each training whole slide image provided as input to the model. In embodiments, the histomorphological phenotype cluster language model takes as input a vector corresponding to a flattened version of a two-dimensional matrix with elements comprising an indication of the assigned histomorphological phenotype cluster for each of the plurality of tiles of a whole slide image. In embodiments, the vector comprises a predetermined identifier associated with any elements of the matrix corresponding to tiles that do not satisfy one or more predetermined filtering criteria, optionally wherein the one or more predetermined filtering criteria include a criterion that applies to the percentage area of the tile showing tissue. In embodiments, the histomorphological phenotype cluster language model is configured to take into account the relative position of input tokens or corresponding token embeddings, optionally by using positional encoding or linear biases. Using positional encoding can comprise adding positional embeddings to the token embeddings. Using linear biases can comprise using attention with linear biases, i.e. biasing query-key attention scores with a penalty that is proportional to their distance, for example using ALiBi as described in Press et al. 2021 . In embodiments, the histomorphological phenotype cluster language model takes as input an indication of the assigned histomorphological phenotype cluster for each of a plurality of tiles of a whole slide image, wherein each tile represents an area of tissue of between 0.1 mm2and 1 mm2, about 0.2 mm2, or between 20 microm by 20 microm and 1 mm by 1 mm. In embodiments, the plurality of tiles together represent an area of tissue of at least 10 mm by 10mm. In embodiments, the plurality of tiles comprises at least 1024 tiles each between 0.1 mm2and 1 mm2. The HCPs may have been obtained using a plurality of training tiles that have been obtained from training whole slide images as non-overlapping tiles of a predetermined size. The training tiles may have been obtained by obtaining a first set of tiles from the training whole slide images, and selecting a plurality of tiles where the tissue in the image represents at least a predetermined percentage of the area of the slide (e.g. 60%). The training tiles may have been subject to stain normalization, for example using Reinhard’s method and a reference set of slides selected randomly from the plurality of training slides. In embodiments, the histomorphological phenotype cluster language model has been trained to predict a plurality of respective output embeddings associated with each respective tile of the plurality of tiles, wherein the whole slide image representation is obtained by combining the plurality of output embeddings, optionally using global average pooling.

[0019] In embodiments, the task specific machine learning model has been trained in a supervised manner using training whole slide images and associated one or more ground truth biological or clinical features of interest. In embodiments, the predicted one or more biological or clinical feature are selected from: a prognosis, a diagnostic, a treatment response prediction, and an identification of a biological feature of the sample. In embodiments, a prognosis is selected from a classification between prognosis classes, and a prediction of a survival metric. In embodiments, a diagnostic is selected from the presence or absence of a pathological feature, a classification between different types or subtypes of a disease or disorder, and the presence or absence of a disease or disorder. In embodiments, a biological feature of the sample is selected from the presence of a histological feature, and the presence of a genomic alteration feature.

[0020] In embodiments, the predicted one or more biological or clinical feature include a classification between a plurality of histological types, a classification between a plurality of histological subtypes, a classification between a plurality of molecular subtypes, a genomic alteration prediction, or a survival prediction, optionally wherein the survival prediction uses a survival model trained through negative log-likelihood on discrete time-to-event.

[0021] In embodiments, the histomorphological phenotype cluster language model has been trained using training whole slide images comprising images of histopathology samples comprising tumour tissue from a plurality of cancer types. In embodiments, the histomorphological phenotype cluster language model has been additionally fine-tuned using training whole slide images comprising images of histopathology samples comprising tumour tissue from a particular cancer type.

[0022] A second aspect of the present invention provides a method of providing a diagnostic and / or prognostic and / or treatment response prediction for a subject, the method comprising: analysing a histopathology sample previously obtained from the subject using the method of any embodiment of the first aspect, wherein the predicted one or more biological or clinical features associated with the sample comprise a diagnostic and / or prognostic and / or treatment response prediction.

[0023] According to a third aspect, there is provided a method of screening a plurality of histopathology samples, the method comprising: analysing each histopathology sample using the method of any embodiment of the first aspect, wherein the predicted one or more biological or clinical features associated with the sample comprises a classification between a plurality of classes associated with respective risk levels (e.g. a first class associated with high risk samples, such as e.g. samples that are likely to include a pathological feature, such as a malignancy, and a second class associated with low risk samples, such as e.g. samples that are unlikely to include a pathological feature, such as a malignancy), or a predicted probability of the sample being a high risk sample (e.g. a probability of the sample including a pathological feature, such as presence of a malignancy).

[0024] According to a fourth aspect, there is provided a method of providing a trained machine learning model for analysing a histopathology sample, the method comprising: providing a plurality of training whole slide images of a plurality of training samples; obtaining a plurality of training tile representations for respective tiles obtained from the plurality of training whole slide images, using a trained tile representation machine learning model, wherein a tile representation machine learning model is a machine learning model that has been trained to learn a representation associated with a histopathology image tile, assigning each of the plurality of training tile representations to one of a predetermined set of histomorphological phenotype clusters, wherein the set of histomorphological phenotype clusters are clusters of tile representations that have been previously obtained using training tile representations obtained using the tile representation machine learning model for a plurality of training tiles from a plurality of training whole slide images associated with respective samples in a cohort comprising a plurality of histopathology samples, training a histomorphological phenotype cluster language model in a self-supervised manner using the training whole slide images to take as input an indication of the assigned histomorphological phenotype cluster for each of a plurality of training tiles of a training whole slide image and learn a representation associated with the training whole slide image from which histopathological phenotype cluster membership of individual training tiles of the training whole slide image can be reconstructed, and training a task specific machine learning model, optionally in a supervised manner, using training whole slide images and optionally associated one or more ground truth biological or clinical features of interest to predict the one or more biological or clinical feature of interest for a whole slide image using as input the whole slide image representation provided by the histomorphological cluster language model for the whole slide image, In embodiments, the method further comprises training the tile representation machine learning model and / or obtaining the set of histomorphological phenotype clusters.

[0025] Methods according to the present aspect may have any of the features described in relation to the first aspect.

[0026] According to a fifth aspect, there is provided a method comprising any of the steps of the first aspect, second or third aspect and any of the steps of the fourth aspect.

[0027] Any of the methods described herein may be computer-implemented.

[0028] According to a sixth aspect, there is provided a system including: at least one processor; and at least one non-transitory computer readable medium containing instructions that, when executed by the at least one processor, cause the at least one processor to implement any of the methods described herein. For example, the system may be configured to implement the methods of any embodiment of any of the first, second, third, fourth and / or fifth aspects.

[0029] According to a seventh aspect, there is provided a non-transitory computer readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform any of the methods described herein. For example, the instructions may cause the at least one processor to implement the methods of any embodiment of any of the first, second, third, fourth and / or fifth aspects.

[0030] According to an eighth aspect, there is provided a computer program comprising code which, when the code is executed on a computer, causes the computer to perform any of the methods described herein. For example, the code may cause the computer to implement the methods of any embodiment of any of the first, second, third, fourth and / or fifth aspects.

[0031] Brief Description of the Figures

[0032] Figure 1A is a flowchart illustrating schematically a method of analysing histopathology samples, according to the disclosure.

[0033] Figure 1 B is a flowchart illustrating schematically a method of providing a trained machine learning model for analysing histopathology samples, according to the disclosure.

[0034] Figure 2 shows an embodiment of a system for analysing histopathology samples.

[0035] Figure 3A-C show an overview of an exemplary method according to the disclosure.

[0036] Figure 3A illustrates schematically a step in which Histomorphological Phenotype Learning (HPL) is used to define Histomorphological Phenotype Clusters (HPCs). Each whole slide image (WSI) is then translated into a 2D matrix that replaces each 224 x 224 with the HPC membership.

[0037] Figure 3B illustrates schematically a step in which a transformer encoder is trained through masked language modeling (MLM). WSI representation is defined as the global average pooling of all embeddings in the sequence. The model takes as input HPC identity for each of a plurality of tiles (w x w tiles, e.g. 32 by 32 tiles for model training), flattened into a vector of size w2(although shorter vectors are also accepted; tiles assigned a “background ID” - illustrated as white regions, which are regions that do not contain enough tissue, are associated with an attention value of 0). Token embeddings are obtained corresponding to each HPC using a dictionary of learned token embeddings of dimension e. In other words, each HPC identity is associated with a token embedding of predetermined size e, which is obtained using an embedding function that is learned when training the model. Thus, the image becomes associated with a sequence of embeddings of size number of tiles x size of token embeddings (illustrated as w2x e). The model is trained in a self-supervised manner to predict respective output embeddings (typically having the same dimensionality as input embeddings, here illustrated as e) associated with each of the tiles, from which the identity of the HPC identity associated with one or more masked tiles can be predicted. The output embeddings are then combined into a single vector of dimensionality e by global average pooling. This ensures that all WSIs are associated with a final vector of the same size, and controls the dimensionality of the output to maintain computational efficiency for downstream tasks. The vector is a representation corresponding to all w x w tiles in the image, capturing the identity of the HPCs associated with each tile and their interactions and spatial relationships.

[0038] Figure 3C illustrates schematically a step in which WSI representations (whole slides) are then used in different supervised settings such as histological classification or survival analysis.

[0039] Figure 4 shows results of UMAP dimensionality reduction of WSI representations obtained using a method of the disclosure. Each point corresponds to a UMAP of a WSI vector representation. Each WSI is labelled based on its corresponding cancer type. Cancer type text labels are located based on an average of all x,y locations of corresponding UMAP WSIs. These results show that different cancer types such as LGG, KIRC, KICH, PRAD, or THCA are clearly clustered together, suggesting that WSI representations of the same cancer types have similar HPC interactions.

[0040] Figure 5 shows results of UMAP dimensionality reduction of HPC dictionary embeddings obtained using a method of the disclosure. Each point corresponds to a UMAP of the language model HPC token embedding. Each HPC token is labelled based on the correlation value between HPC and immune signature fields according to Quiros et al. (2023).

[0041] Figure 5A shows a UMAP of HPC dictionary embeddings labelled based on TIL regional fraction.

[0042] Figure 5B shows a UMAP of HPC dictionary embeddings labelled based on proliferation. Figure 5C shows a UMAP of HPC dictionary embeddings labelled based on stromal fraction.

[0043] Figure 5D shows a UMAP of HPC dictionary embeddings labelled based on wound healing.

[0044] Figure 6 shows the results of an analysis of HPC-to-HPC interactions on the TCGA (The Cancer Genome Atlas) LUAD (Lung adenocarcinoma) Cohort. Attention weights obtained using a method of the disclosure were used to quantify HPC to HPC interactions. Each row corresponds to a ’Source HPC (A)’ where the entire row is a probability distribution of interaction with a ’Destination HPC (B)’ (column). In addition, each HPC is color-labeled with correlation values between HPC and TIL Regional Fraction, Stromal Fraction, Proliferation, and Wound Healing. Positive correlation values are shown in red while negative in blue. A hierarchically-clustered heatmap of attention weights for all patients in the TCGA LUAD cohort is also displayed. Different groups of HPC interactions are highlighted: A, B, C, and D groups contain HPCs that tend to interact with the HPCs of their corresponding group. In addition, they generally see a low negative correlation with TIL Regional Fraction. In contrast, group E contains HPCs that are positively correlated with TIL Regional Fraction. These HPCs tend to interact globally with other HPCs.

[0045] Figure 7A-B show results of an analysis of HPC-to-HPC Interactions on high and low risk groups in LUAD. Attention weights obtained using a method as described herein were used to quantify HPC-to-HPC interactions. Each row corresponds to a ’Source HPC (A)’ where the entire row is a probability distribution of interaction with a ’Destination HPC (B)’ (column). In addition, each HPC is color-labeled with correlation values between HPC and TIL Regional Fraction, Stromal Fraction, Proliferation, and Wound Healing. Positive correlation values are shown in red while negative in blue. A hierarchically-clustered heatmap of attention weights for high and low risk groups of the TCGA LUAD cohort is displayed. Risk groups were obtained through a supervised model (as described in the Examples below). The figure highlights how B and C groups see an increase in interaction when moving from high to low risk. Group B contains HPC with a positive high correlation value with TIL Regional Fraction while group C sees a negative correlation of TIL Regional Fraction but also a negative correlation with Proliferation.

[0046] Figure 7A shows results for the High Risk group.

[0047] Figure 7B shows results for the Low Risk group.

[0048] Figures 8A-C show results of an analysis of attention maps of whole slide images. The figures display data for TCGA PAAD WSI TCGA- 3A-A9IV-01Z-00-DX1 .

[0049] Figure 8A shows the original WSI with an overlay of HPC membership per tile.

[0050] Figure 8B shows the WSI with HPC overlay and brightness scaling based on each HPC attention values.

[0051] Figure 8C shows the accumulated attention values per HPC for the WSI. These figure provide a visual interpretation of the tissue in each WSI, HPCs and their spatial distribution, and attention placed by the models described herein in each of them; allowing pathologist to inspect which parts of the tissue section are relevant according to the methods of the disclosure.

[0052] Detailed Description of the Invention

[0053] Certain aspects and embodiments of the invention will now be illustrated by way of example and with reference to the figures described above.

[0054] In describing the present invention, the following terms will be employed, and are intended to be defined as indicated below.

[0055] “and / or” where used herein is to be taken as specific disclosure of each of the two specified features or components with or without the other. For example “A and / or B” is to be taken as specific disclosure of each of (i) A, (ii) B and (iii) A and B, just as if each is set out individually herein.

[0056] A “sample” as used herein may be a cell sample (e.g. cytology sample, cell culture sample) or tissue sample (e.g. a biopsy or a surgically resected specimen), from which histopathology images can be obtained, typically after fixation and / or labelling. The sample may be a sample that has been obtained from a subject. The sample may be a sample of diseased tissue or a sample of normal tissue. In particular, the sample may be a tumour sample. The sample may be one which has been freshly obtained from a subject or may be one which has been processed and / or stored prior to making a determination (e.g. frozen, fixed or subjected to one or more purification, enrichment or labelling steps). Alternatively, the sample may be a cell or tissue culture sample. As such, a sample as described herein may refer to any type of sample comprising cells. The sample may be a sample comprising eukaryotic cells (e.g. mammalian cells). The sample may be a mammalian sample, such as a human or mouse sample. Further, the sample may be transported and / or stored, and collection may take place at a location remote from the sample processing location (e.g. labelling, fixation etc.), and / or the image data acquisition location. Further, the computer-implemented method steps may take place at a location remote from the sample collection and / or processing location(s) and / or remote from the image data acquisition location. For example, the computer-implemented method steps may be performed by means of a networked computer, such as by means of a “cloud” provider.

[0057] A sample may be a histopathology sample, specifically e.g. a whole tissue section, a tissue biopsy, or a tissue microarray. A histopathology sample may be a tumour section, tumour tissue microarray or tumour biopsy. A tumour section may be a whole tissue section. A whole tissue section is typically a section cut from a block containing a part of a surgically resected piece of tissue such as a tumour, and may thus represent the characteristics of the whole tumour. Thus, a tumour section may be a section of a surgically resected tumour. A histopathology sample may be a section cut from block containing the entirety of a biopsy specimen obtained from a subject. A histopathology sample may be a section of a block containing a tissue microarray. A tissue microarray is a block comprising multiple tissue cores or fragments from multiple original samples (e.g. pieces of tissue) or locations within a sample.

[0058] The histopathology sample may be a stained sample. Staining facilitates morphological analysis of tumour sections by colouring cells, subcellular structures and organelles. Any type of staining that facilitates morphological analysis may be used. For example, the sample may have been stained using Haematoxylin and eosin staining, Papanicolaou (PAP) staining (a combination of haematoxylin, Orange G, eosin Y, Light Green SF yellowish, and sometimes Bismarck Brown Y), Masson's trichrome stain, Romanowsky stain, Silver staining, or any combination of one or more known histological dyes such as acridine orange, Bismarck brown, carmine, Coomassie blue, cresyl violet, DAPI, eosin, ethidium bromide, Hoechst stains, iodine, malachite green, methyl green, methylene blue, neutral red, nile blue, nile red, osmium tetraoxide, propium iodide, rhodamine, and safranine. The histopathology sample may be stained with hematoxylin and eosin (H&E). H&E stain is the most commonly used stain in histopathology for medical diagnosis, particularly for the analysis of biopsy sections of suspected cancers by pathologists. Thus H&E stained histopathology samples are usually readily available as part of large datasets collated for the study of cancer. The histopathology sample may instead or in addition be a sample that has been stained using one or more labels that are associated with specific cellular markers of interest. A common way to associate a label with a cellular marker is through the use of labelled affinity reagents, such as labelled antibodies, or labelled nucleic acids. Thus, the cellular markers of interest may also be referred to as “antigens” (when antibodies are used). When labelled nucleic acids are used, these may be referred to as “probes”. The labels may be detectable through a chromogenic reaction, where the affinity reagent is conjugated to an enzyme (e.g. a peroxidase) that can catalyse a colour-producing reaction. Alternatively, the labels may be detected through fluorescence, where the affinity reagent is associated with a fluorophore (such as e.g. fluorescein or rhodamine) or with an enzyme (eg a peroxidase) which catalyses the deposition of a fluorophore. When using antibodies as affinity reagents, the latter may be referred to as immunofluorescence. The former may be referred to as chromogenic immunohistochemistry, or simply immunohistochemistry, when using antibodies as affinity reagents. The term “immunohistochemistry” is commonly used as an umbrella term to encompass detection via chromogenic immunohistochemistry and via immunofluorescence. The term “immunohistochemistry” also covers multiplex chromogenic or fluorescent immunohistochemistry, in which multiple antigens are detected within a single image using different channels or colours (also referred to as multiplex immunochemistry / immunofluorescence). When using labelled nucleic acids as affinity reagents, the the cellular markers may be RNA or DNA molecules of interest. The nucleic acid affinity reagents may be detected through chromogenic or fluorescent means as explained above. These methods may also be referred to as in situ hybridisation or multiplex in situ hybridisation (when multiple nucleic acid markers are detected simultaneously).

[0059] The methods described herein apply to images of histopathology samples. A “whole slide image” (WSI) refers to an image that shows a substantial portion of a histopathology sample. This is by contrast to a “tile” or “patch” which refers to a subsection of a whole slide image. Thus, a whole slide image refers to an image of a histopathology sample from which a plurality of tiles can be obtained. Tiles are typically obtained as fixed size subsections of an image by applying a grid or stride of a predetermined size, or equivalently by dividing an image of a predetermined size in a predetermined number of areas of predetermined (optionally equal) size. For example, tiles can be obtained by applying a grid of fixed dimensions expressed in number of pixels, equivalent physical area, or number of tiles along both dimensions of a whole slide image of a specific size (e.g. specific number of pixels at specific magnification). For example, a tile size may be defined by predetermined numbers of pixels along two dimensions of an image at a predetermined magnification (e.g. 224x224 pixels at 5x magnification). Individual tiles are typically nonoverlapping. In embodiments, such as e.g. for the purpose of model training, a plurality of grids of tiles each comprising a predetermined number of tiles may be obtained from one or more whole slide images. For example, a sliding window comprising a predetermined number of tiles (e.g. 32x32 tiles) may be applied to create corresponding grids of tiles of a predetermined size. Grids may be nonoverlapping or partially overlapping. When using nonoverlapping grids for model training, each tile is only represented once in the training set. When using partially overlapping grids for model training, each tile may be represented multiple times as parts of a different grid of tiles. Partially overlapping grids can overlap by a predetermined percentage of their surface area (or number of tiles) with one or more adjacent grids. For example, grids may overlap by e.g. 0.75, indicating that 25% of the tiles are unique to the grid, and 75% of the tiles are also included in an adjacent grid. For example, when using a 32x32 WSI grid of tiles (1024 tiles per grid) with 75% overlap, the maximum overlap with another WSI grid of tiles will be 768 tiles (32*32*0.75). this can be achieved by moving a sliding window for at least 8 tiles (0.25*32) in one dimension (x,y). For the purpose of training a histomorphological cluster language model, it may be beneficial to use overlapping tiles, especially in combination with image augmentation such as e.g. by rotation, and / or flipping. A predetermined size for a grid of tiles or a whole slide image can be expressed in terms of a number of pixels, number of equally sized tiles per whole slide image of known size, or area in the physical sample. As will be explained further below, tiles may be excluded or labelled as “background” if they contain less than a predetermined percentage of surface area that shows a tissue or other physical feature of interest (e.g. cells). For example, tiles may be considered to be “background” if less than 60% of their area shows tissue. For the purpose of training a histomorphological cluster language model, grids of tiles may be excluded if they have less than a predetermined percentage of their area that shows a tissue or other physical feature of interest (e.g. cells). The predetermined percentage area may be expressed as a percentage of tiles labelled as background, or as a percentage of the area covered by the grid and showing structures of interest. For example, grids of tiles that have less than 20% of the tiles that are not background tiles may be excluded from a training set used to train a histomorphological cluster language model. A whole slide image as used herein for the purpose of analysing a sample or training a model does not necessarily comprise all of the tiles of an original whole slide image. For example, a set of tiles obtained from an original whole slide image may be filtered, e.g. to remove any tiles where the percentage area of the tile that shows a tissue is below a predetermined threshold. Removal may be achieved by actually removing the information associated with the tile (resulting in a smaller input vector), or by ensuring that tiles that have been labelled as background (or any other label uniquely identifying tiles that have been filtered out) are not given any weight by neural network used to analyse the tile sequence (e.g. by setting attention weights to 0 for those tiles, when using an attention based neural network such as a transformer-based model). When obtaining a flattened version of such a whole slide image as explained further below, this is essentially equivalent to “pulling closer together” regions of the whole slide image corresponding to tiles that were kept. As another example, instead or in addition to this, a section (also referred to as a region) of an original whole slide image may be selected and analysed individually. Such a section may also be referred to as a “whole slide image”. For the purpose of analysing a whole slide image, a grid of tile covering substantially the whole image may be used. The tiles may have the same size and magnification as the tiles used to train the model. For example, a grid of tiles of 224x224 pixels at 5x magnification may be used. To cover an entire WSI there may be e.g. 120x120 of such tiles. In such an example, a whole slide image representation for an image obtained at 5x magnification may comprise 120x120 tiles each of dimension 224x224 pixels. Other choices are possible depending on the magnification and size of the images used for training and analysis. The whole slide image that is being analysed and the images used for training may have been acquired at the same magnification. The whole slide image that is being analysed and / or the images used for training may have a magnification between 2x and 100x, such as e.g. 2x, 5x, 10x, 20x, 40x, 60x, 80x or 100x.

[0060] A plurality of machine learning models are referred to herein, including a tile representation machine learning model, a histomorphological phenotype cluster language model (also referred to as a whole slide image representation model), and a task specific machine learning model (also referred to herein as a biological or clinical feature prediction model).

[0061] A tile representation machine learning model is a machine learning model that has been trained to learn a representation associated with a histopathology image tile. The tile representation machine learning model may have been trained in a self-supervised or supervised manner. In embodiments, the tile representation machine learning model has been trained in a self-supervised manner. The tile representation machine learning model may be trained as a generative model. Any machine learning model that can be used to extract features associated with an image can be used as a tile representation machine learning model. The tile representation machine learning model is typically a deep neural network (also referred to as a deep learning model), such as e.g. a convolutional neural network (CNN) such as e.g. ResNet (e.g. ResNet50, or a CNN based network trained with VICreg as described e.g. in Bardes et al. 2022), a transformer-based model such as a ViT, a statespace model (e.g. Vision Mamba, Zhu et al. 2024), a graph neural network, a generative adversarial model (e.g. PathologyGAN, Quiros et al. 2021 ), a variational autoencoder (VAE), a diffusion model, or a model combining convolutional and attention blocks (see e.g. Quiros et al. 2024). The tile representation machine learning model is configured to take as input an image tile, and produce as output a representation (also referred to as feature vector) associated with the tile.

[0062] Tile representations are used to identify histomorphological phenotype clusters (HPCs) by applying any clustering algorithm known in the art to a plurality of tile representations obtained from a plurality of whole slide images associated with respective samples in a cohort comprising a plurality of histopathology samples. The cohort may comprise samples associated with a plurality of subjects. The cohort may comprise tumour samples associated with a plurality of cancer types. Tile representations may be used to identify histomorphological clusters by obtaining a graph where each node is associated with a tile representation and edges connect tiles with their one or more nearest neighbours in terms of tile representation, and applying a community detection algorithm to the graph. A community detection algorithm may be selected from the Louvain algorithm or the Leiden algorithm. A new tile not forming part of the cohort from which the HPCs were identified can be assigned to a HCP by reclustering the tile representation with the new tile, or by assigning the tile to one of the existing clusters using any clustering method known in the art for assigning observations to predetermined clusters, such as e.g. k-nearest neighbours.

[0063] A histomorphological phenotype cluster language model is a machine learning model that has been trained in a self-supervised manner to learn a representation associated with whole slide images from which histopathological phenotype cluster membership of individual tiles of the whole slide image can be reconstructed. The histomorphological phenotype cluster language model is configured to take as input a vector of respective histomorphological phenotype cluster memberships (i.e. values or strings indicating the histomorphological phenotype cluster that a respective tile is associated with) associated with respective tiles of a plurality of tiles of a whole slide image, and produce as output a whole slide image representation for the whole slide image. The histomorphological phenotype cluster language model is trained to learn token embeddings associated with respective histomorphological phenotype cluster, and process these learned token embeddings to produce output embeddings from which the histomorphological phenotype cluster memberships of one or more masked tiles of the whole slide image can be predicted. Thus, the histomorphological phenotype cluster language model can be seen as a foundation model that learns the language of HPCs. The histomorphological phenotype cluster language model may have been trained using any self-supervised approach including masked language modelling (in which random input tokens are masked in each whole slide image provided as input to the model), HPC masking (in which all input tokens associated with a particular, randomly selected HPC are masked in each training whole slide image provided as input to the model), DINO (Oquab et al. (2024)), VIC-Reg (Bardes et al. (2022)), l-JEPA (Assran et al. (2023)), InfoNCE (Oord et al. (2019)), next token prediction, causal language modeling (CLM), and / or any variant thereof such as causal masked modeling (CMM), prefix language modeling, permutation modelling, unified language learning, next sentence prediction (NSP) (Devlin et al. (2019)), etc., or any combination of any one or more known self-supervised approaches including but not limited to the above. The histomorphological phenotype cluster language model operates on a dictionary that has a predetermined size corresponding to the number of HPCs. Each element of the dictionary may be a learned token embedding for a respective HPC. The learned token embeddings may have a predetermined size. The output embeddings of the histomorphological phenotype cluster language model can be used as a whole slide image representation or can be processed to generate a whole slide image representation, for example by global average pooling of the output embeddings corresponding to each of the plurality of tiles of the whole slide image. Thus, the histomorphological phenotype cluster language model can be used to generate a whole slide image representation based on HPC memberships assigned to tiles comprised in the WSI, and can therefore be referred to as a whole slide image representation model. The histomorphological phenotype cluster language model may be a deep learning model. The histomorphological phenotype cluster language model may be a model referred to as a “sequence model” or “language model”. A sequence model is machine learning model that takes a sequence of data as input. A sequence of data is any string of input data that has a sequence (order). This is by contrast to a set of input data representing a set of independent data points. A sequence model is typically a deep learning model. A deep learning model can be a deep neural network, such as: (i) a recurrent neural network (RNN) or RNN-based model, such as a model combining recurrence and attention mechanisms such as Receptance Weighted Key Value (RWKV) models (Peng et al. 2023) and Griffin (De et al. 2024), or a long short-term memory (LSTM) model, (ii) a convolutional neural network (CNN) or a convolution-based model such as a model combining elementwise multiplication (gating) and long convolutions (convolutions with filter sizes as long as the input) such as Hyena (Poli et al. 2023) or a model combining convolutions and attention mechanisms such as Striped Hyena (Poli et al., 2023b), (iii) a state-space model such as Mamba (Gu et al. 2023) or Hawk (De et al., 2024), or hybrid transformer-state space model such as Jamba (Lieber et al., 2024), (iv) a multilayer perceptron (MLP)-based architecture such as HyperMixer (Mai et al. 2023) or MLP-Mixer (Tolstikhin et al. 2021 ), (v) a retentive neural network (e.g. RetNet, Sun et al. 2023), or (vi) a fully attention-based model, also referred to as transformer-based model (Viswani et al. 2017). Fully or partially attention-based models are advantageous because attention maps can be obtained. In embodiments, the histomorphological cluster language model is a transformer-based model. Transformer-based models are a class of machine learning models, in particular deep learning models, that include attention mechanisms (self- and / or cross-attention mechanisms). Transformer-based models primarily or exclusively rely on attention mechanisms to learn from tokens in context, and typically do not rely on a recurrent structure for this. The histomorphological cluster language model may comprise a transformer-based encoder. A transformer-based encoder typically comprises a plurality of attention layers each comprising multi-head attention mechanisms, and one or more fully connected layers optionally including residual connections. In embodiments, the histomorphological cluster language model comprises a plurality of transformer block layers, each layer comprises a multi-head attention layer, a pre-norm layer, and a normalisation layer. In embodiments, the histomorphological cluster language model further comprises a MLP, such as a 2 layer MLP. The MLP may have a non-linear activation function, such as e.g. GELU. In embodiments, the histomorphological cluster language model is trained using a cross entropy loss, weight typing on the linear layer, and / or oReparam over the dense layers.

[0064] A task specific machine learning model is a model that has been trained in a supervised manner to predict a biological or clinical feature of interest for a whole slide image based on the whole slide image representation provided by the histomorphological cluster language model for the whole slide image. Any machine learning model architecture suitable for classification or regression tasks from a feature vector can be used for this purpose. In embodiments, the task specific machine learning model is a linear model, a transformer model (e.g. a combination of a multihead selfattention with MLP and a pre-norm layer), a multiple instance learning model, or a multilayer perceptron (MLP). A multilayer perceptron or single linear layer can be added to the histomorphological cluster language model to obtain a machine learning model that takes as input a vector of respective histomorphological phenotype cluster memberships (i.e. values or strings indicating the histomorphological phenotype cluster that a respective tile is associated with) associated with respective tiles of a plurality of tiles of a whole slide image, and produces as output a desired classification or regression. Thus, the task specific machine learning model can also be referred to as a classification head or regression head. For example, one or more class tokens can be prepended to the output embeddings of the histomorphological phenotype cluster language model or the corresponding whole slide representation (combined output embeddings which combines the output embeddings of each of the plurality of tiles of the whole slide image), and a MLP or single linear layer can be used to compute logits for the class tokens from the output embeddings I combined output embeddings. When training a model comprising the histomorphological cluster language model and the task specific machine learning model, all parameters of the histomorphological cluster language model may be frozen. In other words, only the task specific machine learning model may effectively be trained in a supervised manner for the specific task.

[0065] The term “whole slide image representation” refers to the output of a histomorphological cluster language model for a particular whole slide image, comprising the output embeddings for each of the plurality of tiles of the whole slide image produced by the histomorphological cluster language model for the whole slide image or a combined representation obtained from these output embeddings, for example by global average pooling. The term “tile representation” refers to the output of the tile representation machine learning model.

[0066] A biological or clinical feature predicted by the task specific machine learning model may be selected from: a prognosis (including e.g. a classification between prognosis classes, a prediction of a survival metric, etc.), a diagnostic (including e.g. the presence or absence of a pathological feature, a classification between different types or subtypes of a disease or disorder, the presence or absence of a disease or disorder, etc.), a treatment response prediction, and an identification of a biological feature of the sample (such as e.g. the presence of a histological feature, the presence of a genomic alteration feature, etc. ). Any embodiments comprising providing a treatment response prediction or identification of a treatment that a subject is likely to respond to may comprise treating the subject with the identified treatment.

[0067] As used herein, the terms “computer system” includes the hardware, software and data storage devices for embodying a system or carrying out a method according to the above-described embodiments. For example, a computer system may comprise a central processing unit (CPU), one or more graphics processing units (GPU), input means, output means and data storage, which may be embodied as one or more connected computing devices. Preferably the computer system has a display or comprises a computing device that has a display to provide a visual output display (for example in the design of the business process). The data storage may comprise RAM, disk drives or other computer readable media. The computer system may include a plurality of computing devices connected by a network and able to communicate with each other over that network. It is explicitly envisaged that computer system may consist of or comprise a cloud computer.

[0068] As used herein, the term “computer readable media” includes, without limitation, any non-transitory medium or media which can be read and accessed directly by a computer or computer system. The media can include, but are not limited to, magnetic storage media such as floppy discs, hard disc storage media and magnetic tape; optical storage media such as optical discs or CD-ROMs; electrical storage media such as memory, including RAM, ROM and flash memory; and hybrids and combinations of the above such as magnetic / optical storage media.

[0069] As the skilled person understands, the complexity of the operations described herein (due at least to the amount of data that is analysed and the complexity of the machine learning models used) are such that they are beyond the reach of a mental activity. Thus, unless context indicates otherwise (e.g. where sample preparation or acquisition steps are described), all steps of the methods described herein are computer implemented. The present disclosure provides machine learning based methods for analysing histopathology images, in which self-supervised learning is used to: (i) identify feature vectors for tiles of whole slide images using an image model, and identify associated histomorphological phenotype clusters (HPC), and (ii) learn WSI representations from HPC-tile assignment using a language model.

[0070] Other methods using machine learning for the analysis of histopathology images have been proposed. For example, Song et al. (2024) proposed a method termed “PANTHER” where a feature extractor is used to process each patch of a whole slide image, the feature set for each patch is then assumed to have been generated from a mixture distribution, where each mixture component represents a morphological exemplar. The parameters of the mixture components are learned by the model for each slide, and a slide representation is constructed as the concatenation of the estimated model parameters. This is then directly used for downstream tasks such as subtyping and survival prediction. Contrary to the methods of the present disclosure, this approach can at most provide information about proportions of “morphological examplars” and cannot account for how they are spatially distributed and interact with each other. Chen et al. (2022a) proposed a vision transformer architecture termed “Hierarchical Image Pyramid Transformer” (HIPT) that uses two levels of self-supervised learning to learn high-resolution image representations. A first transformer is trained through self-supervised learning (DINO, Oquab et al. (2024)) on tile tissue images at 20X magnification to learn patch-level representations. A second transformer is trained through DINO using WSI regions built from tile tissue representations to obtain region-level representations. A third transformer is trained through supervised learning and using WSI regions. By contrast, the methods of the present disclosure use a finite set of HPCs for unsupervised learning, rather than tile representations directly, and achieve WSI representations in an unsupervised manner. Therefore, the methods of the present disclosure advantageously obtain both meaningful and widely applicable whole slide image representations in a computational and data efficient manner. Jiang et al. (2023a) proposed a method termed MaskHIT which is similar to HIPT expected that the second transformer is trained through masked image modelling rather than DINO. This suffers from the same limitations as HIPT. Filiot et al. (2023b) proposed a method that uses a Vision Transformer model pretrained in a self-supervised manner using masked image modelling to learn tile representations. The model is then used for weakly supervised WSI classification tasks. By contrast, the methods of the present disclosure use a finite set of HPCs rather than tile representations directly, achieve WSI representations in an unsupervised manner, and use a language model to aggregate tile information into a WSI representation. This is in contrast to the MIL-based models used in Filiot et al. (2023b) that do not create attention weights based on a one-to-one comparison between tile instances. The methods described herein therefore yield a simpler, more computationally efficient model. As another example, Vu et al. (2023a) proposed a method called “Handcrafted Histological Transformer” (H2T) which uses convolutional neural network (CNN) trained using a self-supervised learning method called SWAV (Caron et al., 2021 ) to learn tile representations, which are then clustered to identify “prototypical patterns”. The cluster centroids (prototypical patterns) along with each tile representation are used to handcraft WSI representations through: (i) weighted averages of tile representations and cluster memberships, and (ii) co-localization of patterns (pattern colocalization matrix for the slide). By contrast, the methods of the present disclosure use a language model to learn WSI representation by learning to model interactions between different HPCs. The present model generates a single WSI representation that contains HPC content but also location information. By contrast, H2T uses 2 different types of representations per WSI, one for tissue pattern information and another one for spatial information in the WSI. Further, the methods of the present disclosure are able to summarize tissue interactions more efficiently and characterize dynamics not only for each patient but also at a cohort level. For example, patterns of HPC associations with clinical and molecular data at the cohort level can be obtained. Further, through the use of a finite number of HPCs, the present methods can summarise how tissue patterns interact with each other (HPC to HPC interactions). This can be derived at the patient level (e.g. defining HPC to HPC interactions using aggregates of all attention values tile to tile) and at the cohort level (by aggregating those per- patient HPC-HPC interactions aggregates). It is then possible to compare cohorts (e.g. high and low risk groups or subtypes). This is demonstrated in a specific example below, where the comparison of Figure 7A (high risk patients) and Figure 7B (low risk patients) shows an increase of interactions in HPCs that are positively correlated with Tumor Infiltrating Lymphocytes. Such cohort levels insights are difficult to obtain using methods like HIPT because areas of interest in a particular patient’s slide have no correspondence to those in another patient’s side. As another example, Wang et al. (2021 ) described a method called “CTransPath” that focuses only on learning tissue tile representations. These tissue tile representations are later used in a supervised manner for different tasks. By contrast, the methods of the present disclosure use a finite set of HPCs rather than tile representations directly, and use a language model to learn HPC interaction dynamics and create WSI representations without supervision. Deep Attention Multiple Instance Survival Learning (DeepAttnMISL, Yao et al. (2020b)) is a supervised approach that uses a pretrained ImageNet network to translate tile images into tile representation. K-Means is then used to cluster tile representations for all tiles per patient (i.e. clusters are strictly patient specific), and the tile representations and cluster centroids are then used in a supervised manner using ABMIL (Use et al., 2018), a deep attention-based multiple instance learning model. ABMIL is a much simpler model than a language model. For example, transformer-based language models allow to create attention weights based on a one-to-one comparison between tile instances, ABMIL does not. By contrast, the methods of the present disclosure use a finite set of HPCs rather than tile representations or patient cluster-centroids directly, achieve WSI representations in an unsupervised manner, and use a language model to aggregate tile tissue information into a WSI representation. Multiple multimodal approaches have also been proposed which combine WSI and clinical report text (e.g. PRISM, Shaikovski et al. (2024), which aligns WSI representations and language reports and uses these to reconstruct the report text), genomic profile information (e.g. CMTA (Zhou and Chen, 2023) and MCAT (Chen et al., 2021 ), both of which are supervised approaches that work with tile representations directly) or transcriptomic information (e.g. TANGLE (Jaume et al., 2024), which uses MIL and unsupervised training aiming to find a common representation space of WSI and transcriptomic). By contrast, the methods described herein can be fully unsupervised up to the point of task-specific training, focussing on learning the spatial distribution of HPCs and their interactions, and do not need any kind of additional information such as pathology reports or genomic profile information.

[0071] The methods described herein are unique in many ways including: (i) the methods define a finite dictionary of morphological tissue patterns (in the form of HPCs); (ii) the methods uses a language model and an unsupervised learning task (such as e.g. mask language modelling) to learn interactions between HPCs; (iii) the methods can create WSI representations in an unsupervised manner, without labels or annotations, which WSI representations can later be used in different classification and survival tasks without the need for retraining of the model that generates the WSI representation (although the model may be additionally fine-tuned in embodiments if desired - such as e.g. fine tuning the model using data from a specific cancer type to optimize the model on learning the spatial distributions of HPCs that are more common in that specific cancer type - without losing the benefits since those representations would be more context specific through unsupervised training with a particular cohort, but still not task specific and therefore maintain wide applicability); and (iv) using a finite set of HPCs allows to more easily scrutinize HPC interactions through the language model’s attention maps (or other visualisation such as grad-CAM and categorical Jacobian (e.g. as described in Zhang et al. 2024), depending e.g. on the architecture of the language model), at the patient and cohort levels. Other methods show attention maps for each tile globally (i.e. attention values from each tile to all other tiles - i.e. for a WSI of size NxM tiles, this leads to an attention matrix of (N*M x N*M), typically averaged across all tiles for a given tile so it can be displayed as a 2D matrix like in Fig. 4 of Chen et al. 2022a), not between tissue tiles. Using the set of HPCs allows to summarize attention maps (or other visualisations) more efficiently. Indeed, by using HPCs, we can define the patient’s WSI attention maps by the attentions HPC to HPC, by translating each tile to its HPC membership and aggregating values, transforming the attention matrix of (N*M x N*M) to (T x T) where T is the number of HPCs. Additionally, the mean values of these HPC to HPC attentions can also be used to define cohort level interactions. This provides additional information about how much a source HPC pays attention to another destination HPC, allowing to see interactions of HPC patterns (see examples below and e.g. Figure 6). This is not possible to obtain with prior art methods.

[0072] Analysing pathology images

[0073] The present disclosure provides method for analysing histopathology samples, using image data acquired from the sample. An illustrative method will be described by reference to Figure 1A. Figure 1A describes steps performed in analysing a histopathology sample, and Figure 1 B describes steps performed in providing trained model for analysing a histopathology sample. The methods of Figures 1A and 1 B may each be performed independently, or may both be performed. For example, the steps of Figure 1 B may optionally be performed in a method of analysing a histopathology sample according to Figure 1 A.

[0074] At step 10, an image of a histopathology sample is obtained. The sample may be a stained sample, for example a H&E stained sample as described elsewhere herein. The image may be referred to as a digital pathology image. The image may have been acquired using any imaging technology suitable for obtaining images of histopathology samples. The image may be received from a user interface, a data store or an image acquisition means. The image may be a multicolour image such as an RGB image.

[0075] At step 12, a plurality of tiles are obtained from the whole slide image. The plurality of tiles may be non-overlapping tiles of predetermined equal size. At step 14, the plurality of tiles may be pre- processed, such as e.g. by selecting a second plurality of tiles from the first plurality of tiles obtained at step 12, where the selected tiles are associated with a percentage of area showing tissue above a predetermined percentage. These may be selected t proceed to the next step of obtaining respective tile representations using the tile representation machine learning model. Unselected tiles may be removed or associated with a predetermined identifier. The predetermined identifier may be used by the histomorphological phenotype cluster language model as an indication that the tile associated with the predetermined identifier is to be ignored (e.g. assigned an attention of 0).

[0076] At step 16, a plurality of tile representations are obtained for respective tiles obtained from the whole slide image, using a tile representation machine learning model, wherein a tile representation machine learning model is a machine learning model that has been trained to learn a representation associated with a histopathology image tile.

[0077] At step 18, each of the plurality of tile representations is assigned to one of a predetermined set of histomorphological phenotype clusters, wherein the set of histomorphological phenotype clusters are clusters of tile representations that have been previously obtained using training tile representations obtained using the tile representation machine learning model for a plurality of training tiles from a plurality of training whole slide images associated with respective samples in a cohort comprising a plurality of histopathology samples.

[0078] At step 20, a whole slide image representation is obtained using a histomorphological phenotype cluster language model, wherein the histomorphological phenotype cluster language model is a machine learning model that has been trained in a self-supervised manner using training whole slide images to take as input an indication of the assigned histomorphological phenotype cluster for each of a plurality of tiles of a whole slide image and learn a representation associated with the whole slide image from which histopathological phenotype cluster membership of individual tiles of the whole slide image can be reconstructed.

[0079] At step 22, one or more biological or clinical features associated with the sample are predicted using a task specific machine learning model. The task specific machine learning model is a model that has been trained using training whole slide images and optionally associated one or more ground truth biological or clinical features of interest to predict the one or more biological or clinical feature of interest for a whole slide image using as input the whole slide image representation provided by the histomorphological cluster language model for the whole slide image. The one or more biological or clinical features of interest may be a prediction of an augmented slide image for a whole slide image. In such cases the task specific machine learning model may simply be the histomorphological phenotype cluster language model used as a generative model.

[0080] Optionally, instead or in addition to predicting biological I clinical features using a task specific machine learning model, the results of step 20 may be used to analyse the attention patterns associated with the histomorphological phenotype cluster language model for one or more whole slide images by: averaging attention values across one or more heads and layers of the histomorphological phenotype cluster language model to obtain an attention matrix of size (N*M x N*M) for each of one or more WSI of size (NxM), and aggregating the attention values based on each tile’s assigned histomorphological phenotype cluster, thereby obtain a matrix of dimension (T x T) where T is the number of histomorphological phenotype clusters. In embodiments, aggregating comprises averaging and normalising the values such that each row of the (T x T) aggregated matrix sums to 1 . In embodiments, the method further comprises averaging the aggregated matrix over a plurality of whole slide images.

[0081] In embodiments, the histomorphological phenotype cluster language model has been trained with a dataset comprising WSI from at least 10, at least 15 or at least 20 different cancer types. In embodiments, the histomorphological phenotype cluster language model has been trained with a dataset comprising at least 1000 WSI images from one or more cancer types selected from ACC= Adrenocortical carcinoma. BLCA= Bladder Urothelial Carcinoma, BRCA= Breast invasive carcinoma, CESC= Cervical squamous cell carcinoma and endocervical adenocarcinoma, CHOL= Cholangiocarcinoma, COAD= Colon adenocarcinoma, ESCA= Esophageal carcinoma, GBM= Glioblastoma multiforme, HNSC= Head and Neck squamous cell carcinoma, KICH= Kidney Chromophobe, KIRC= Kidney renal clear cell carcinoma, KIRP= Kidney renal papillary cell carcinoma, LGG= Brain Lower Grade Glioma, LIHC= Liver hepatocellular carcinoma, LUAD=lung adenocarcinoma, LUSC=Lung squamous cell carcinoma, MESO= Mesothelioma, OV= Ovarian serous cystadenocarcinoma, PAAD= Pancreatic adenocarcinoma, PRAD= Prostate adenocarcinoma, READ= Rectum adenocarcinoma, SARC= Sarcoma, SKCM= Skin Cutaneous Melanoma, STAD= Stomach adenocarcinoma, TGCT= Testicular Germ Cell Tumors, THCA= Thyroid carcinoma, UCEC= Uterine Corpus Endometrial Carcinoma, UVM=Uveal Melanoma.

[0082] In embodiments, the histomorphological phenotype cluster language model has been trained using a plurality of training tiles obtained from a plurality of training whole slide images by obtaining grids of tiles of a predetermined size optionally with overlap (e.g. 75%), and optionally filtering out grids with less than a predetermined percentage of tissue area (e.g. 20%). Each grid of tiles may comprise a fixed number of tiles, such as 32 by 32 tiles. In embodiments, the training grids of tiles have been obtained by additionally performing image augmentation by rotation and / or flipping. In embodiments, the training tiles comprised at least 500000 grids of tiles, at least 1 million grids of tiles, or more.

[0083] [In embodiments, the method further comprises obtaining a whole slide image representation for each of a plurality of whole slide images using the histomorphological phenotype cluster language model, and visualising the whole slide image representations using a dimensionality reduction technique (e.g. UMAP). In embodiments, the method further comprises visualising the learned token embeddings for the HPCs using a dimensionality reduction technique (e.g. UMAP). In embodiments, the method further comprises analysing and / or visualising the attention patterns associated with the histomorphological phenotype cluster language model for a particular whole slide image or set of whole slide images.

[0084] At step 24 one or more results of the analysis may be provided to a user. This may include any of the information obtained at steps 20 and. or 22 or any information derived therefrom, such as e.g. attention patterns, recommended therapies, etc.

[0085] The models used at steps 16, 20 and 22 may have been previously obtained using a method as described in relation to Figure 1 B. At step 110, a plurality of training histopathology images are obtained. These images may have any of the features described herein in relation to histopathology images, and in particular any of the features described in relation to the image used in Figure 1 A. The training whole slide images may comprise images of histopathology samples comprising tumour tissue from a plurality of cancer types.

[0086] At step 112, a plurality of training tiles are obtained, e.g. as explained by reference to Fig. 1 A. At step 114, the plurality of training tiles may be pre-processed, such as e.g. as explained by reference to Fig. 1 A. In embodiments, step 114 additionally comprises obtaining a plurality of grids of tiles comprising a predetermined number of tiles. Each grid can be used as a whole slide image for the purpose of training the histomorphological cluster language model as explained further below. The plurality of training tiles may be obtained from training whole slide images as nonoverlapping tiles of a predetermined size. The training tiles may be obtained by obtaining a first set of tiles from the training whole slide images, and selecting a plurality of tiles where the tissue in the image represents at least a predetermined percentage of the area of the slide (e.g. 60%). The training tiles may be subject to stain normalization, for example using Reinhard’s method and a reference set of slides selected randomly from the plurality of training slides.

[0087] At optional step 116, the plurality of training tiles are used to train a tile representation model. The tile representation machine learning model may be trained to learn a representation associated with a histopathology image tile. In embodiments, a pre-trained model may be used and step 116 may instead comprise obtaining a plurality of training tile representations for respective tiles obtained from the plurality of training whole slide images, using a trained tile representation machine learning model.

[0088] At optional step 118, histomorphological clusters are obtained. These are clusters of tile representations that have been previously obtained using training tile representations obtained using the tile representation machine learning model for a plurality of training tiles from a plurality of training whole slide images associated with respective samples in a cohort comprising a plurality of histopathology samples. This is optional and instead previously obtained clusters may be used such that step 118 may comprise assigning each of the plurality of training tile representations to one of a predetermined set of histomorphological phenotype clusters. Step 118 may comprise applying a clustering algorithm to the training tiles representations. The algorithm may be a community detection algorithm applied to a graph where each node is associated with a training tile representation and edges connect training tiles representations with their one or more nearest neighbours in terms of tile representation.

[0089] At step 120, an expanded set of training tiles is optionally obtained, for example by applying image augmentation to tiles of grids of tiles in the training set.

[0090] At step 122, a histomorphological phenotype cluster language model is trained in a self-supervised manner using the training whole slide images to take as input an indication of the assigned histomorphological phenotype cluster for each of a plurality of training tiles of a training whole slide image and learn a representation associated with the training whole slide image from which histopathological phenotype cluster membership of individual training tiles of the training whole slide image can be reconstructed. Step 122 may comprise training the histomorphological phenotype cluster language model using training whole slide images of samples of a plurality of cancer types, and additionally fine-tuning the model using training whole slide images comprising or consisting of images of histopathology samples comprising tumour tissue from a particular cancer type. The histomorphological phenotype cluster language model may be trained to predict a plurality of respective output embeddings associated with each respective tile of the plurality of tiles, e.g. from which HPC cluster assignments for masked tiles can be reconstructed. The whole slide image representation may be obtained by combining the plurality of output embeddings, optionally using global average pooling. The histomorphological phenotype cluster language model may be trained using a self-supervised learning objective. For example, a self-supervised learning objective may be selected from masked language modelling, causal language modelling, and HPC masking in which all input tokens associated with a particular HPC are masked in each training whole slide image provided as input to the model.

[0091] At optional step 124, the token embeddings and / or attention weights associated with one or more WSI of the histomorphological phenotype cluster language model may be analysed as explained above.

[0092] At optional step 126, a task specific machine learning model is trained, optionally in a supervised manner, using training whole slide images and optionally associated one or more ground truth biological or clinical features of interest to predict the one or more biological or clinical feature of interest for a whole slide image using as input the whole slide image representation provided by the histomorphological cluster language model for the whole slide image. This is optional because the trained histomorphological cluster language model itself may be used directly to predict biological or clinical features of interest, for example when the model is used in a generate manner to obtain augmented images.

[0093] At step 128, the trained models obtained at steps 116, 122 and / or 126 may be provided to a user, computing device or data store.

[0094] The methods of the present invention are performed on samples (such as e.g. histopathology samples) and in particular on images of such samples. The methods of the present invention are therefore ex vivo or in vitro methods, that is, the methods of the present invention are not practiced on the human body.

[0095] Applications

[0096] The above methods find applications in a variety of clinical contexts. For example, the methods described herein find applications in any context in which it is desirable to predict a phenotypic or clinical feature associated with a histopathology, including but not limited to a diagnosis (including e.g. presence or absence of a disease or disorder, including a particular type or subtype of a disease or disorder, classification between subtypes of a disorder, etc.), prognosis, treatment response prediction, cell or tissue type characterisation or segmentation, etc.

[0097] The methods of the present disclosure can enable better prediction of outcomes (such as e.g. prognosis, treatment response) than current clinical gold standards. The methods of the present disclosure can enable better diagnoses than current clinical gold standard (such as e.g. cancer typing or subtyping, molecular subtyping, etc.). Indeed, the methods of the present disclosure can make highly accurate predictions about patients from tissue samples, where the predictions can include any one or more of: outcome (prognosis), therapeutic response, diagnosis, in a consistent way which outperforms expert human assessment by pathologists. Without wishing to be bound by theory, the present inventors believe that the superior performance of the present methods is due at least in part to the use of a model that has been trained for ab initio learning of the fundamental and potentially human-unapparent image features which compose a training set of images, using these features to discover and provide a ‘dictionary’ of tissue appearances, converting a diagnostic image into ‘words’ derived from this dictionary, and then subjecting these ‘words’ to analysis using large language models. The end result is an essentially unsupervised analytical pipeline capable of discovering subtle, complex image features which are influential / meaningful to any question for which annotations are available for supervised learning of a task-specific model built on top of the unsupervised model (e.g. patient survival, drug sensitivity, precise diagnosis).

[0098] Reducing the sample number requirement for Al algorithm development: Importantly, the method described acquires great experience and authority from the ingestion of a large training set in the unsupervised learning part. This knowledge can then be applied to much smaller image sets (e.g. biopsy collections from clinical trials) for the supervised learning part. This is important because labelled data for supervised learning is typically available in small amounts, leading to machine learning models that are either poorly constrained or overfitted. By contrast, the present disclosure proposed a method that makes use of unsupervised learning to learn the “language” of histopathology images, then uses these learnings to perform specific classification or regression tasks for which labelled data is available.

[0099] Further, the method can process large numbers of digitalised images (e.g. whole slide images) of many different tissues (e.g. different cancer types) without expert pathologist annotation, assign the images into different tissue types and disease categories (e.g. tumour vs non tumour), and / or identify regions of the images which are influential (e.g. associated with particular predictions such as poor survival, using e.g. attention maps and HPC associations as described herein, e.g. focussing on tiles associated with HPCs that have been shown to be relevant to a particular prediction or associated with a particular characteristic) and therefore worthy of additional human scrutiny and / or research. An important application of this would be screening for malignant / tumour samples from normal samples. This task currently makes up a large proportion of the workflow in pathology departments of hospitals in the UK. The methods described herein can offer a viable solution to the problem of reducing processing time of these samples.

[0100] The methods described herein can be deployed at the fraction of the cost of manual expert review of histopathology slides as well as other alternative modalities for diagnosis, prognosis and / or therapeutic response prediction such as molecular tests, sequencing based technologies etc. Further, they can be applied to obtain diagnostic, therapeutics and / or prognostic insights that would have otherwise required the use of such alternative assays using standard of care diagnostic material such as H&E sections, that are often already widely available. The methods of the present disclosure can also be applied in addition to other tests such as molecular tests and other clinical tests when available, supplementing them to generate more accurate predictions.

[0101] Further, the methods of the present disclosure can be used resolves issues with intra-tumoral heterogeneity. Intra-tumoral heterogeneity poses a significant issue when only a portion of the tumour is being examined or sequenced and not the whole tumour, which makes prediction of therapeutic response or resistance highly variable. When using the algorithm, one is able to predict what the rest of the tumour may look like with some confidence, hence a better understanding of the rest of the tumour and the degree of intra-tumoral heterogeneity, hence a better prediction of patient response and prognosis. Indeed, the deep learning models described herein can be used as parts of generative models to output HPC sequences based on a partial tissue area. This is similar to prompting a language model with questions. The heterogeneity here is explicitly captured by the HPC composition and their spatial coordinates (i.e. tokens and their locations in the sequences). This explicit representation of spatial heterogeneity makes the present approach stronger than multiple-instance learning or other supervised approach at handling heterogeneity.

[0102] Thus, also described herein are methods of providing a prognosis for a subject that has been diagnosed as having a cancer, the method comprising analysing a histopathology sample from the subject as described herein, wherein the predicted biological or clinical features comprise a prognosis indication.

[0103] Also described herein are methods of identifying a subject as having cancer or having a particular type or subtype of cancer, the method comprising analysing a histopathology sample from the subject as described herein, wherein the predicted biological or clinical feature comprises an identification of the subject as being likely to have cancer or a particular type or subtype of cancer. Similarly, also described herein is a method of screening a plurality of histopathology samples, the method comprising analysing each histopathology sample using a method as described herein, wherein the task-specific machine learning model is configured to classify a histopathology samples between a plurality of classes associated with respective risk levels (e.g. a first class associated with high risk samples, such as e.g. samples that are likely to include a pathological feature, such as a malignancy, and a second class associated with low risk samples, such as e.g. samples that are unlikely to include a pathological feature, such as a malignancy), or to predict a probability of a sample being a high risk sample (e.g. a probability of the sample including a pathological feature, such as presence of a malignancy). Such methods may further comprise selecting one or more samples of the plurality of samples for further review using the output of the task-specific machine learning model. For example, one or more samples classified in a first class associated with high-risk samples or associated with a probability of being a high risk sample above a predetermined threshold may be selected for further review. The samples selected for further review may be further reviewed by a trained pathologist, and / or the subjects from which these samples originated may be selected for further diagnostic testing and / or treatment for a disease or disorder associated with the pathological feature, or included in a report indicating subjects that may benefit from further diagnostic testing and / or treatment for a disease or disorder associated with the pathological feature.

[0104] Further, also described herein are methods of treating a subject that has been diagnosed as having a cancer with a particular treatment, or identifying a subject that has been diagnosed as having a cancer as likely to benefit from a particular treatment. The method may comprise analysing a histopathology sample from the subject as described herein to obtain a predicted biological and / or clinical feature associated with the sample. The method may further comprise treating the subject with the particular treatment if the histopathology sample is classified in a class that is likely to benefit from the particular treatment. For example, specific subtypes of cancers or cohorts of subjects are known to be more or less likely to respond to some therapeutic strategies than others. Therefore, a subject who is identified as having a first type of cancer or belonging to a first cohort of subjects that is likely to respond to a particular treatment may be selected for treatment or treated with the particular treatment. Conversely, a subject who is identified as having a second type of cancer or belonging to a second cohort of subjects that is unlikely to respond to a particular treatment may be selected for treatment or treated with a treatment that is different from the particular treatment. As another example, the methods described herein can be used to train a model that can classify a histopathology sample between a class that is associated with response to a particular therapy and a class that is not associated with response to therapy. This can use training data comprising e.g. histology and details of patient response to therapy (e.g. from a randomised control trial), such as direct measures of response (advantageously since they represent more direct evidence of therapeutic effect) or patient outcome metrics (such as e.g. survival metrics). For example, in a randomised controlled trial (e.g. treatment A in a first arm vs treatment B in a second arm), the methods described herein can be used to predict the prognosis within each arm, which is the same as predicting treatment response, as the treatment is homogeneous. The same principles can be applied to any cohort of patient who have received the same treatment and for which histology and response data is available.

[0105] The disclosure also relates to a method of providing a diagnostic and / or prognostic and / or treatment response prediction for a subject, the method comprising analysing a histopathology sample of the subject using a method as described herein, wherein the output of the task specific machine learning model is indicative of a diagnostic and / or prognostic and / or treatment response. Any classification method described herein may be binary or multiclass. Thus, classifying a histopathology sample between a plurality of classes may comprise classifying the histopathology sample between a first class and a second class, or between a plurality of classes comprising more than two classes (i.e. three or more classes). Each class may be associated with a different diagnosis, prognosis or treatment response.

[0106] Providing a diagnostic can comprise classifying the histopathology sample between a plurality of classes associated with different diagnoses. For example, a first class may be associated with the presence of a pathological feature (e.g. a malignancy), and a second class may be associated with the absence of the pathological feature. As another example, a first class may be associated with a first subtype of a disease or disorder, and a second class may be associated with a second, different subtype of disease or disorder. Providing a prognosis can comprise classifying the histopathology sample between a plurality of classes associated with different prognosis. For example, a first class may be associated with a first prognosis, and a second class may be associated with a second prognosis, the second prognosis being a poorer prognosis than the first prognosis. Prognosis can be associated with any survival metric, range of survival metric or combination of survival metrics known in the art (such as e.g. overall survival, progression free survival, etc.) As another example, a first class may be associated with a first subtype of a disease or disorder that is known to have a better prognosis than a second subtype of disease or disorder, and a second class may be associated with the second, different subtype of disease or disorder. Further, providing a prognosis can comprise predicting a value of one or more survival metrics. A survival metric can be any survival metric known in the art. Providing a treatment response can comprise classifying the histopathology sample between a plurality of classes associated with different treatment responses. For example, a first class may be associated with samples that are expected to respond to one or more therapies, and a second class may be associated with samples that are not expected to respond to the one or more therapies. The one or more therapies may be any therapies known in the art and for which training histopathology samples images are available comprising histopathology samples images associated with subjects that are known or expected to respond to the therapy and histopathology samples images associated with subjects that are known or expected to not respond to the therapy. The one or more therapies may be selected from: immunotherapy, surgery, radiotherapy, and chemotherapy.

[0107] The cancer may be a cancer type or subtype selected from, breast cancer (including e.g. Breast invasive carcinoma), central nervous system cancer (including glioblastoma multiforme and brain lower grade glioma), endocrine cancer (including adrenocortical carcinoma, thyroid carcinoma, paraganglioma & pheochromocytoma), gastrointestinal cancer (including Cholangiocarcinoma, Colon Adenocarcinoma, Esophageal carcinoma, Rectum adenocarcinoma, Stomach adenocarcinoma), gynaecologic I reproductive system cancer (including Cervical squamous cell carcinoma and endocervical adenocarcinoma, Ovarian Serous Cystadenocarcinoma, Uterine Corpus Endometrial Carcinoma, Testicular Germ Cell Tumors), Liver cancer (such as Liver Hepatocellular Carcinoma), pancreatic cancer (including Pancreatic Ductal Adenocarcinoma), head and neck cancer (including Head and Neck Squamous Cell Carcinoma, Uveal Melanoma), skin cancer (including Skin Cutaneous Melanoma), soft tissue cancer (including Sarcoma), thoracic cancer (including Lung Adenocarcinoma (LUAD), Lung Squamous Cell Carcinoma (LUSC), Mesothelioma) and urologic cancer (including Chromophobe Renal Cell Carcinoma, Clear Cell Kidney Carcinoma, Kidney renal papillary cell carcinoma, Prostate Adenocarcinoma, Urothelial Bladder Carcinoma). Each of these cancers is the subject of study as part of The Cancer Genome Atlas project, and methods of the disclosure have been demonstrated on each of these.

[0108] In specific embodiments, the present disclosure provides methods of identifying a subject as having a particular type or subtype of cancer (including e.g. a histological subtype, molecular subtype, histological type, cancer having or not having a particular genomic alteration, cancer having or not having a particular histological feature such as keratinization), the method comprising analysing a histopathology sample from the subject as described herein, wherein the task specific machine learning model has been trained to classify histopathology samples between a plurality of classes associated with respective types or subtypes of cancers.

[0109] In some embodiments, the plurality of classes comprise a first class associated with lung adenocarcinoma (LUAD) and a second class associated with lung squamous cell carcinoma (LUSC). Thus, also described herein are methods of identifying a subject who has been diagnosed as having or being likely to have non-small cell lung carcinoma cases as having lung adenocarcinoma (LUAD) or lung squamous cell carcinoma (LUSC) types.

[0110] In some embodiments, the plurality of classes comprise a first class associated with colon adenocarcinoma and a second class associated with rectal adenocarcinoma. Thus, also described herein are methods of identifying a subject who has been diagnosed as having or being likely to have colorectal cancer cases as having Colon adenocarcinoma (COAD) or rectum adenocarcinoma (READ).

[0111] In some embodiments, the plurality of classes comprise a first class associated with gliobastoma multiforme (GBM) and a second class associated with brain lower grade glioma (LGG). Thus, also described herein are methods of identifying a subject who has been diagnosed as having or being likely to have brain cancer as having gliobastoma multiforme (GBM) or brain lower grade glioma (LGG).

[0112] In some embodiments, the plurality of classes comprise a first class associated with invasive ductal carcinoma (IDC) and a second class associated with lobular carcinoma (LC). Thus, also described herein are methods of identifying a subject who has been diagnosed as having or being likely to have a breast carcinoma as having invasive ductal carcinoma (IDC) or lobular carcinoma (LC).

[0113] In some embodiments, the plurality of classes comprise a first class associated with kidney renal clear cell carcinoma (KIRC), a second class associated with kidney renal papillary cell carcinoma (KIRP), and a third class associated with kidney chromophobe (KICH). Thus, also described herein are methods of identifying a subject who has been diagnosed as having or being likely to have renal cell cancer cases as having kidney renal clear cell carcinoma (KIRC), kidney renal papillary cell carcinoma (KIRP), or kidney chromophobe (KICH). Similarly, the disclosure also encompasses embodiments that use a binary classifier that classifies samples between a subset of these classes, or combine two of these classes in a single class.

[0114] In some embodiments, the plurality of classes comprise a first class associated with ADEX (Aberrantly differentiated endocrine exocrine) pancreatic cancer, a second class associated with immunogenic pancreatic cancer, a third class associated with progenitor pancreatic cancer, and a fourth class associated with squamous pancreatic cancer. Thus, also described herein are methods of identifying a subject who has been diagnosed as having or being likely to have pancreatic ductal adenocarcinoma as having ADEX, immunogenic, progenitor, or squamous pancreatic ductal adenocarcinoma. Similarly, the disclosure also encompasses embodiments that use a classifier that classifies samples between a subset of these classes, or combine two or more of these classes in a single class. For example, in some embodiments, the plurality of classes comprise a first class associated with progenitor pancreatic cancer, and a second class associated with squamous pancreatic cancer. Thus, also described herein are methods of identifying a subject who has been diagnosed as having or being likely to have pancreatic ductal adenocarcinoma as having progenitor or squamous pancreatic ductal adenocarcinoma.

[0115] In some embodiments, the plurality of classes comprise a first class associated with normal breast carcinoma, a second class associated with basal breast carcinoma, a third class associated with luminal A breast carcinoma, a fourth class associated with luminal B breast carcinoma, and a fifth class associated with Her2-enriched breast carcinoma. Thus, also described herein are methods of identifying a subject who has been diagnosed as having or being likely to have breast carcinoma as having normal, basal, luminal A, luminal B, or Her2-enriched breast carcinoma. Similarly, the disclosure also encompasses embodiments that use a classifier that classifies samples between a subset of these classes, or combine two or more of these classes in a single class. Luminal A breast cancers are known to be likely to benefit from hormonal therapy in addition to chemotherapy. Her2-enriched breast cancers are known to be likely to respond better to specific therapies that target HER2, including trastuzumab, trastuzumab combined with emtasin (T-DM1 ), pertuzumab, and tyrosine kinase inhibitors such as lapatinib and neratinib, among others. Therefore, such methods may further comprise selecting a subject who has been identified as having a Her2- enriched breast cancer for treatment with a tyrosine kinase inhibitor or HER2-specific therapy, and / or selecting a subject who has been identified as having a luminal A breast cancer with a combination of chemotherapy and hormonal therapy. Such methods may further comprise treating the subject. In some embodiments, the plurality of classes comprise a first class associated with cancers that have microsatellite instability (MSI), and a second class associated with cancers that are microsatellite stable (MSS). Thus, also described herein are methods of identifying a subject who has been diagnosed as having or being likely to have cancer as having a microsatellite unstable (MSI) or a microsatellite stable (MSS) cancer. The cancer may be any cancer type that is likely to have MSI, including e.g. colon cancer, rectal cancer, stomach cancer, ovarian cancer, breast cancer, etc. The cancer may be selected from colon adenocarcinoma, rectal adenocarcinoma, and stomach adenocarcinoma. MSI cancers are known to be more likely to respond to immunotherapy, as well treatments that cause double strand breaks, such as PARP inhibitors and platinum-based therapies. Therefore, such methods may further comprise selecting a subject who has been identified as having a MSI cancer for treatment with e.g. an immunotherapy, a PARP inhibitor or platinum-based therapy. Such methods may further comprise treating the subject. The immunotherapy may be a checkpoint inhibitor therapy (CPI), such as e.g. a CTLA-4 inhibitor (e.g. an anti-CTLA-4 antibody such as Ipilimumab), PD-1 inhibitor (e.g. an anti-PD-1 inhibitor such as nivolumab or pembrolizumab) or PD-L1 inhibitor (e.g. an anti-PD-L1 antibody such as atezolizumab, avelumab or durvalumab).

[0116] In some embodiments, the plurality of classes comprise a first class associated with papillary transitional Urothelial bladder carcinoma and a second class associated with transitional Urothelial bladder carcinoma. Thus, also described herein are methods of identifying a subject who has been diagnosed as having or being likely to have Urothelial bladder carcinoma as having papillary transitional or transitional urothelial bladder carcinoma.

[0117] In some embodiments, the plurality of classes comprise a first class associated with astrocytoma and a second class associated with oligodendroglioma. Thus, also described herein are methods of identifying a subject who has been diagnosed as having or being likely to have brain lower grade glioma cases as having astrocytoma or oligodendroglioma.

[0118] In some embodiments, the plurality of classes comprise a first class associated with liposarcoma and a second class associated with leiomyosarcoma. Thus, also described herein are methods of identifying a subject who has been diagnosed as having or being likely to have sarcoma as having liposarcoma or leiomyosarcoma.

[0119] In some embodiments, the plurality of classes comprise a first class associated with intestinal stomach adenocarcinoma, a second class associated with diffuse stomach adenocarcinoma, and a third class associated with tubular stomach adenocarcinoma. Thus, also described herein are methods of identifying a subject who has been diagnosed as having or being likely to have stomach adenocarcinoma cases as having intestinal, diffuse, or tubular stomach adenocarcinoma. Similarly, the disclosure also encompasses embodiments that use a binary classifier that classifies samples between a subset of these classes, or combine two of these classes in a single class. In some embodiments, the plurality of classes comprise a first class associated with the presence of keratinization and a second class associated with the absence of keratinization. Thus, also described herein are methods of identifying a subject who has been diagnosed as having or being likely to have cancer as having a cancer with keratinization or having a cancer without keratinization. The cancer may be selected from lung squamous cell carcinoma (LUSC), headneck squamous cell carcinoma (HNSC), cervical squamous cell carcinoma and endocervical adenocarcinoma (CESC), and oesophageal carcinoma (ESCA).

[0120] In specific embodiments, the present disclosure provides methods of providing a prognosis for a subject, the method comprising analysing a histopathology sample from the subject as described herein, wherein the task specific machine learning model has been trained to classify histopathology samples between a plurality of classes associated with respective prognoses (e.g. associated with distinct ranges of values of one or more prognostic metrics), or to predict the value of one or more prognostic metrics associated with the sample. The one or more prognostic metrics may include overall survival. Thus, such methods may comprise predicting the expected duration of time until a patient’s death event. In specific embodiments, the task specific machine learning model has been trained using images associated with a specific type of cancer. The specific type of cancer may be selected from : breast cancer, colorectal cancer, Kidney renal clear cell carcinoma, Kidney renal papillary cell carcinoma, lung adenocarcinoma, stomach adenocarcinoma, colon adenocarcinoma, lower grade glioma, Ovarian serous cystadenocarcinoma, Bladder Urothelial Carcinoma, Cervical squamous cell carcinoma and endocervical adenocarcinoma, Esophageal carcinoma, glioblastoma multiforme, Head and Neck squamous cell carcinoma, liver hepatocellular carcinoma, lung squamous carcinoma, mesothelioma, Pancreatic adenocarcinoma, sarcoma, skin cutaneous melanoma, uterine corpus endometrial carcinoma, and uveal melanoma. Without wishing to be bound by theory, the present inventors believe that a task specific machine learning model can be obtained to predict survival for any cancer type for which a set of training histopathology images and associated survival metric values is available.

[0121] Systems

[0122] Figure 2 shows an embodiment of a system according to the present disclosure. The system comprises a computing device 1 , which comprises a processor 101 and computer readable memory 102. In the embodiment shown, the computing device 1 also comprises a user interface 103, which is illustrated as a screen but may include any other means of conveying information to a user such as e.g. through audible or visual signals. The computing device 1 is communicably connected, such as e.g. through a network, to image acquisition means 3, such as a microscope or scanner, and / or to one or more data storing means 2 storing image data. The computing device may be a smartphone, tablet, personal computer, cloud computer, or other computing device. The computing device is configured to implement a method for analysing images, as described herein. In alternative embodiments, the computing device 1 is configured to communicate with a remote computing device (not shown), which is itself configured to implement a method of analysing images, as described herein. In such cases, the remote computing device may also be configured to send the result of the method of analysing images to the computing device. Communication between the computing device 1 and the remote computing device may be through a wired or wireless connection, and may occur over a local or public network such as e.g. over the public internet.

[0123] The image data acquisition means 3 may be in wired connection with the computing device 1 , or may be able to communicate through a wireless connection as illustrated, such as e.g. through WiFi. The connection between the computing device 1 and the image data acquisition means 3 may be direct or indirect (such as e.g. through a remote computer). The image data acquisition means 3 are configured to acquire image data from samples, for example histopathology samples or other samples of cells and / or tissues. In some embodiments, the sample may have been subject to one or more preprocessing steps such as fixation, labelling, etc. The image acquisition means 3 acquire signal associated with cells or cellular structures in the sample. The samples may be labelled with one or more histological stains or dyes that are detectable by the image data acquisition means 3. In some embodiments, it may not be necessary to actively label the target. Signal detection may be done by any method for detecting electromagnetic radiation (e.g., light) such as a method selected from far-field optical microscopy, near-field scanning optical microscopy, epi-fluorescence microscopy, confocal microscopy, two-photon microscopy, optical microscopy, and total internal reflection microscopy, where the cell or cellular structure is labelled with an electromagnetic radiation emitter. Thus, the image acquisition means 3 may comprise a far-field optical microscope, a near-field scanning optical microscope, an epi-fluorescence microscope, a confocal microscope, a two-photon microscope, an optical microscope, or a total internal reflection microscope. Conveniently, the image acquisition means may comprise an optical or confocal microscope. The image acquisition means 3 may be configured to acquire optical images. The methods of the present disclosure find particular use in the context of analysis of histopathology images obtained by optical imaging of stained samples, such as e.g. hematoxylin and eosin stained samples. Thus, any image acquisition means that is suitable to acquire images of samples stained with a tissue stain can be used.

[0124] The following is presented by way of example and is not to be construed as a limitation to the scope of the claims.

[0125] Examples

[0126] Introduction - A new approach to learn the language of cancer tissue

[0127] The present examples introduce Idioma, a method to learn whole slide image (WSI) representations without the need of labels or annotations. This is achieved at least in part by training a language model that learns the interaction dynamics between a finite set of morphological patterns found in tissue images. These WSI representations capture different histological and survival information that can be used in different histological classification and survival tasks. In addition, the proposed approach is highly interpretable. Idioma can provide interaction maps between the different tissue patterns contained in the WSI that are directly used in the creation of the WSI representation. Furthermore, by working with a finite set of morphological patterns, it is possible to expand the study and scrutiny of the interaction dynamics to an entire cohort.

[0128] Figure 3A-C together outline an example of the methodology. First, Histomorphological Phenotype Learning (HPL, Quiros et al. (2023) is used to define clusters of 224 x 224 tissue tiles with distinct morphological patterns. These clusters are referred to as Histomorphological Phenotype Clusters (HPCs) (Figure 3A). This step allows us to define large sections of tissue, such as Whole Slide Images (WSIs), as a spatial set of HPC identifiers. Next, a transformer model is trained with masked language modeling (MLM) to learn the interaction dynamics between HPCs. After training, the transformer model is used to obtain a single vector representation per WSI (Figure 3B). Finally, WSI representations can be used in downstream classification and survival tasks (Figure 3C).

[0129] By using a finite set of HPCs, we can further scrutinize HPC interactions through the language model’s attention maps. Other methods show attention maps for each tile globally, not between tissue patterns. In contrast, using the set of HPCs allows to summarize tissue interactions more efficiently and characterize dynamics not only for each patient but also at a cohort level.

[0130] Supervised tasks. Finally, WSI vector representations can be used in different supervised tasks. As illustrated further below, we demonstrate how WSI representations can be used for overall survival analysis and in histological or molecular subtype classification. Furthermore, we show how we can scrutinize Idioma’s attention patterns between HPCs to further study interactions among and between risk groups.

[0131] In particular, the model is evaluated over 38 tasks that include classification and overall survival modeling. The classification tasks include histological type and subtype, molecular subtype, and genomic alteration prediction. For survival, the model is evaluated on overall survival prediction over 23 different cancer types.

[0132] Methods

[0133] Defining Histomorphological Phenotype Clusters. WSIs are pre-processed to divide them into 224 x 224 tiles without overlap and at 5X magnification (2pm / px). Note that other magnifications could also be used such as e.g. 20x or 10x. This would increase the number of tiles that compose a WSI given the same pixel size, which would increase computational time for processing an image given the same architecture. However, this can be offset by using more computationally efficient models such as Mamba (Gu et al. 2024) or training with flash-attention (Dao et al. 2022). Tiles are filtered out if the tissue in the image does not cover at least 60% of the area. Afterward, stain normalization is applied by using Reinhard’s method (Reinhard et al., 2001 ). This method aims to match the mean and standard deviation of an image to a target, which in this case was set as the average values of 100 random tiles form the TCGA dataset (although a single image can also be used).

[0134] Next, Histomorphological Phenotype Clusters are learned in a manner similar to that described in Quiros et al. 2023. Briefly, self-supervised learning is used to learn tissue vector representations from tile images using VIC-Reg (Bardes et al. (2022) - note the original method used Barlow twins, using a CNN as feature extractor comprising ResNet layers and a self-attention layer). Alternative models including the Vision transformer (ViT, Dosovitskiy et al. 2021 ), and ResNet50 were also tested, with similar results. Through this process, we capture distinct morphological and cellular information into vector representations without the need of labels.

[0135] Finally, Leiden community detection (Traag et al., 2018) is used to cluster tile vector representations and define HPCs, i.e. clusters with specific morphological patterns found in tissue. This process allows to translate each 224 x 224 tissue tile image into its HPC membership. Therefore, we can define WSIs as spatial sets of HPC identifiers.

[0136] The Louvain algorithm starts by assigning each node in the graph to a different community, then it iterates through the steps described below until there are no further changes in the network. Firstly, for each node the algorithm will evaluate the gain of modularity when the node is moved from the current community to a neighboring one; if there’s a positive gain, the node is kept in the new community. After an initial pass, this process is repeated for nodes that changed community until it reaches a local maximum, finding a partition of the graph. Secondly, the previous partition is refined by possibly further splitting some of the previously define communities. This is done by randomly merging a node to a community if it increases modularity. Modularity is defined by: H = — Xc ( ec- y-^j, where ecis the number of edges in a community c, is the expected number of edges where Kc is the sum of the degrees of the nodes in a community c, and m is the total number of edges in the network, y is the resolution parameter where higher values lead to more communities and lower values to fewer communities. In order to find Histomorphological Phenotype Clusters (HPCs) from self-supervised tissue representations, we created a graph by using K nearest neighbors (K = 250) over 200,000 randomly sampled tiles from the training set (although larger sets have also been successfully tested, such as e.g. 1 million tiles), and used Leiden community detection to define clusters. Subsequently, we assigned clusters to vector representations of additional sets by using K-nearest neighbors between vectors of the training set and each of the vectors of the additional set. The resolution parameter can be set using a method balancing compactness and average presence of different cohorts as explained in Quiros et al. 2023. This was set to 2.0 in the present work, which resulted in 60 different clusters. However, as explained further below, the inventors have shown that the methods described herein are robust to the choice of value for the resolution parameter between e.g. 1 and 5.

[0137] Learning the language of HPCs. At this stage, WSIs have been translated into corresponding spatial sets of HPCs and a transformer model is used as explained below to learn the interaction dynamics between HPCs. We train a transformer through masked language modeling, randomly masking HPCs at different locations and asking the model to predict the correct HPC from the context of non-masked HPCs. We train the transformer encoder through WSI regions of 32 x 32 tiles (i.e. 1024 equally sized tiles per image, also referred to herein as grids), which is equivalent to 7168 x 7168 pixels or 14.336 x 14.336 mm of tissue. After training, we use the transformer to define WSI as a vector representation. The WSI vector representation is the result of the global average pooling of all output embeddings. The output embeddings are representations of the spatial distribution of HPCs, they can be understood as a mix of information of the HPC embeddings. Since the output embeddings have information about the spatial distribution of HPCs, a linear layer can be used to predict the likely original HPC in any location that was masked out. Other methods of training the model can be used such as e.g. combining random MLM with a sequence SSL loss (such as e.g. DINO, VIC-Reg, l-JEPA, next sentence prediction). These approaches could be introduced by including a class token that learns the WSI representation and fits the sequence SSL loss. It is also not necessary to use global average pooling of the output embeddings (e.g. the supervised task can be performed using all output embeddings and a multiple instance learning model to aggregate all output embeddings such as e.g. Attention MIL, Transformer, Mamba). GAP advantageously enables to maintain the dimensionality across WSI and improves computational efficiency for the downstream task. However, using a sequence representation learning framework (such as e.g. DINO, VIC-Reg, l-JEPA, next sentence prediction) may also give alternatives to using GAP as a method to define a WSI representation. For example, random or HPC-based training can be combined with VIC-Reg.

[0138] Architecture. Idioma uses a transformer encoder consisting of 12 block layers with 12 heads per multi-head attention layer and a pre-norm layer arrangement. It uses layer normalization as normalization layer and a multilayer perceptron (MLP) of 2 layers with GELU non-linearity.

[0139] In addition, it uses a learnable dictionary of token embeddings R768, one per HPC and a ‘mask’ token. The embedding layer is a lookup table that stores embeddings of a fixed dictionary and size. It stores a set of embeddings and retrieves them using indices (e.g. HPC ID).

[0140] Finally, we use attention with linear biases (ALiBi, Press et al. 2022) for positional encoding. ALiBi has been shown to extrapolate efficiently to higher context windows during inference, with minimal loss in perplexity. In our case, we use context windows of 32 x 32 during training and 120 x 120 during inference.

[0141] After training, we define WSI vector representations as global average pooling (GAP) over all output embeddings from the transformer encoder (see Fig. 3B). Training. Idioma is trained through masked language modeling (MLM) and WSI regions of 32 x 32 tiles. For each WSI region, we sample from a Gaussian distribution m ~ N(0.3, 0.05) and mask out m tiles. Masked tiles are replaced with the ’mask’ token while the remaining tiles are replaced by each of their corresponding HPC token embedding.

[0142] For each location where we initially masked out HPCs, we use the output embedding along with a linear layer to predict the masked HPC. The model is trained through cross-entropy loss and we use weight typing on the linear layer of the transformer blocks (Press and Wolf, 2017). Weight typing has shown to improve performance of language models by sharing the weights of the embedding and softmax layers. We make use of oReparam (Zhai et al., 2023) overall dense layers of the method, oReparam uses a learnable scalar and spectral normalization on each layer’s weights in order stabilize training. In our case, we see an additional effect on limiting the model to over-fitting.

[0143] We use Adam optimizer (pi = 0.9, p2 = 0.95), weight decay of 0.1 , and a cosine schedule for learning rate, with base of 1 e-3and warm up of 10% of total epochs. Finally, we train the model for 20 epochs with a batch size of 20 and gradient accumulation over 12 batches. We keep the model with highest performance in predicting the correct HPC in a held-out validation set.

[0144] We train our model in a 5-fold cross-validation setting. Each train set consists of samples from the train sets of the overall survival tasks, applying the same principle on the test set. In this manner, we have no data leakage between train samples on the Idioma model training and the downstream survival tasks. With respect to the classification tasks, we apply the same procedure whenever possible, such as in histological type prediction for non-small cell lung carcinoma, colorectal cancer, brain related cancer types, and renal cell cancer cases.

[0145] Idioma pre-training: WSI Region Sizes. To generate all of the results shown, the inventors trained Idioma by using WSI sections of 32 x 32 tiles, equivalent to 14.336 x 14.336 mm of tissue. The inventors postulate that they could use smaller WSI sections of e.g. 16 x 16 tiles or 8 x 8 tiles, and still capture the spatial distribution of HPCs. These smaller WSI sections would yield x4 and x16 more samples, providing the training process with a larger sample size and potentially improving the model’s ability to predict masked out HPC locations. Consequently, this could improve Idioma’s performance on classification and survival tasks.

[0146] Idioma pre-training and fine-tuning. Idioma is currently pretrained with a dataset of 28 different cancer types (see Figure 3B). In addition, Idioma could be fine-tuned for each cancer type after completing the pancancer pre-training. Each cancer type has its own spatial distribution of HPCs and fine-tunning would allow Idioma to better fit those spatial HPC dynamics for each of the types. As a result, Idioma’s performance on classification and survival tasks. Figure 4 displays each WSI representation labelled by the cancer type associated. It shows that Idioma can distinguish cancer types by identifying the different spatial distribution of HPCs and therefore suggesting that further training on them would allow to better fit each cancer own spatial dynamics. HPC to HPC interactions. We define HPC interactions as the mean attention value from a source HPC X to a destination HPC Y across a set of patients. We obtained the HPC to HPC interactions in the following manner, where T represents the number of HPCs:

[0147] 1 . For each WSI (size N x M):

[0148] 1 .A We first average the attention value across heads and layers, leaving an attention matrix of (N*M x N*M)

[0149] 1.B We then aggregate and normalize attention values based on each tile’s HPC membership, translating the attention matrix from (N*M x N*M) to (T x T). Each row of the attention matrix sums to 1 .

[0150] 2. Finally, we average WSI attention matrices (T x T) across the entire cohort (LUAD cohort is shown in the examples below).

[0151] This process leaves a summary of HPC to HPC interactions across an entire cohort in an unsupervised manner, allowing to interpret how Idioma mixes HPCs representations in order to create WSI representations. Through this process and HPCs quantifications (e.g. immune signatures or pathologist annotations), we can further analyze and study how different HPC patterns may be related across cancer subtypes or cohorts.

[0152] Training Data. We use WSI regions of 32 x 32 tiles to train our model. We use WSI from The Cancer Genome Atlas (TCGA) of 28 different cancer types. All slides used in these examples are H&E stained, but the approach is applicable to any type of histology images, including e.g. slides stained with other chemical stains, immunofluorescence slides, immunohistochemistry slides, and in situ hybridisation.

[0153] For each WSI available, we create regions (grids) of 32 x 32 tiles by applying a 32x32 tiles sliding window, allowing an overlap of 0.75 (i.e. the sliding window moves by 0.25 of its size to create a new 32x32 region - e.g. if the first 32x32 region A starts at (0,0) and ends at (31 ,31 ). The next region being considered B would start (7,0) and end at (38,31 )). We then filter out regions (grids) with less than 0.2 of tissue area. This is a relatively laxed criterion to ensure that a large amount of training data was kept. In addition, we perform a rotation of 90°, and 180°, and vertical and horizontal flips. This process yields a total of 1 .468 million samples with a mean of approximately 493 HPC tokens per sample and a standard deviation of 204.

[0154] We used samples of 28 cancer types from The Cancer Genome Atlas (TCGA). Table 1 describes the complete cohort of patients and whole slide images per cancer type. The Cancer Genome Atlas (TCGA) whole slide images and corresponding annotations are available at the Genomics Data Commons portal (gdc.cancer.gov / ). This data is publicly available without restriction, authentication or authorization necessary.

[0155] Table 1 : TCGA pancancer dataset description. ACC= Adrenocortical carcinoma. BLCA= Bladder Urothelial Carcinoma, BRCA= Breast invasive carcinoma, CESC= Cervical squamous cell carcinoma and endocervical adenocarcinoma, CHOL= Cholangiocarcinoma, COAD= Colon adenocarcinoma, ESCA= Esophageal carcinoma, GBM= Glioblastoma multiforme, HNSC= Head and Neck squamous cell carcinoma, KICH= Kidney Chromophobe, KIRC= Kidney renal clear cell carcinoma, KIRP= Kidney renal papillary cell carcinoma, LGG= Brain Lower Grade Glioma, LIHC= Liver hepatocellular carcinoma, LUAD=lung adenocarcinoma, LUSC=Lung squamous cell carcinoma, MESO= Mesothelioma, OV= Ovarian serous cystadenocarcinoma, PAAD= Pancreatic adenocarcinoma, PRAD= Prostate adenocarcinoma, READ= Rectum adenocarcinoma, SARC= Sarcoma, SKCM= Skin Cutaneous Melanoma, STAD= Stomach adenocarcinoma, TGCT= Testicular Germ Cell Tumors, THCA= Thyroid carcinoma, UCEC= Uterine Corpus Endometrial Carcinoma, UVM=Uveal Melanoma.

[0156] Supervised Tasks. After training Idioma, we translate each WSI into a single vector representation of dimensionality R768. We now a have set of WSI vector representations that we can use in any supervised setting. Details of each of the different histological classification and survival tasks are provided below.

[0157] Architecture. In both classification and survival tasks, we use a MLP network to extract information from the WSI representation to the corresponding task. The MLP network consists of 3 blocks of 2 dense layers with GELU non-linearity followed by layer normalization. After these 3 blocks, it uses a final dense layer.

[0158] Training. Classification models are trained though cross-entropy loss with rescaling class weights to account for unbalanced datasets. In addition, we use Adam optimizer (Pi = 0.9, P2 = 0.95), weight decay of 0.1 , and a cosine schedule for learning rate, with a base of 1 e-4and warm up of 10% of total epochs. Finally, we train the classification models for 20 epochs with a batch size of 32 and no gradient accumulation.

[0159] Survival models are trained through negative log-likelihood on discrete time-to-event (Zadeh and Schmid, 2021). We use Adam optimizer (pi = 0.9, p2 = 0.95), weight decay of 0.1 , and a cosine schedule for learning rate, with a base of 1e-4 and warm up of 10% of total epochs. The survival models are trained for 200 epochs with a batch size of 256 and no gradient accumulation. Survival analysis models the expected duration of time until an event occurs, where we may consider cases in which such event is not observed. In our overall survival analysis, we consider events as a patient death and not observed as patient's last follow-up time (i.e. right censored). We use months as time unit. For our survival models, we use the Negative Log-Likelihood (NLL) survival loss on discrete time-to-event (Zadeh and Schmid, 2021 ), we use the WSI representations as patient representations (only one per patient, z) along with the death event time and censoring rate c (c=1 represents a censored sample). In the NLL setup, we discretize the complete range time-to-event T into a finite set of possible time intervals {yj=[tji, tj) where j e (1 ,n)} defined by the quartiles of death event for uncensored patients n. Therefore, each patient / will be defined by their WSI representation, c, and time interval (z', c', y)). The MLP network taking WSI representations z' aims to predict the time to event interval y), providing the corresponding logit p). Finally, the NLL survival loss is defined as the following (Zadeh and Schmid, 2021 ) where f hazard (y'j / z') is the hazard function and fsur ivaify'j / z') is the survival function:

[0160] The hazard function and survival function are defined as the following: f azard(yij / zi)=sigmoid(pij)

[0161] Other approaches are possible, such as e.g. using the Cox proportional hazards loss (Wong et al. 1986) or the Ranking loss (Luck et al. 2018).

[0162] We use a 5-fold cross-validation setting for evaluation of survival models while we use a 10-fold cross-validation in classification. In both cases, we keep the model that performs higher on the test set through training.

[0163] Tasks. As mentioned above, the model is evaluated over 38 different tasks, the details of which are provided below. For each of these tasks, the prediction is made at the patient or whole slide image (WSI) level.

[0164] 1 . Histological type prediction:

[0165] 1.a. TCGA-NSLC. Non-small cell lung carcinoma cases are divided between lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC) types.

[0166] 1.b. TCGA-CRC. Colorectal cancer cases divided between Colon (COAD) and rectum (READ). 1.c. TCGA-Brain. Brain related cancer types divided between gliobastoma multiforme (GBM) and brain lower grade glioma (LGG).

[0167] 2. Histological subtype prediction:

[0168] 2. a. TCGA-BRCA. Breast carcinoma cases are divided between invasive ductal carcinoma (IDC) and lobular carcinoma (LC).

[0169] 2.b. TCGA-RCC. Renal cell cancer cases are divided between kidney renal clear cell carcinoma (KIRC), kidney renal papillary cell carcinoma (KIRP), and kidney chromophobe (KICH).

[0170] 3. Molecular subtype prediction

[0171] 3. a. TCGA-PAAD. Pancreatic ductal adenocarcinoma cases are divided between ADEX, immunogenic, progenitor, and squamous. In addition, we define another task where only consider two molecular subtypes: progenitor and squamous.

[0172] 3.b. TCGA-BRCA. Breast carcinoma cases are divided between normal, basal, luminal A, luminal B, and Her2-enriched molecular subtypes.

[0173] 4. Genomic alteration prediction

[0174] 4. a. TCGA-MSI. Colorectal and stomach (COAD, READ, and STAD) cases are divided between high microsatellite instable (MSI) and microsatellite stable (MSS).

[0175] 5. Histological annotations

[0176] 5.a. TCGA-BLCA. Urothelial bladder carcinoma cases are divided between papillary transitional and transitional samples.

[0177] 5.b. TCGA-LGG. Brain lower grade glioma cases are divided between astrocytoma and oligodendroglioma samples.

[0178] 5.c. TCGA-SARC. Sarcoma cases are divided between liposarcoma and leiomyosarcoma samples. 5.d. TCGA-STAD. Stomach adenocarcinoma cases are divided between intestinal, diffuse, and tubular samples.

[0179] 5.e. TCGA-Keratinization. Keratinization prediction on samples from lung squamous cell carcinoma (LUSC), head-neck squamous cell carcinoma (HNSC), cervical squa-mous cell carcinoma and endocervical adenocarcinoma (CESC), and esophageal carcinoma (ESCA). In this case we define to different tasks. ’Keratinization’ divides samples between specific annotations of keratinization and nonkeratinization. ’Keratinization Alt.’ uses any sample that it is not explicitly annotated as keratinization annotation as a nonkeratinization case.

[0180] 6. Overall survival

[0181] The overall survival task aims to analyze the expected duration of time until a patient’s death event. Patients with a loss of follow-up are included as right-censored samples. The overall survival analysis includes the following cancer types: BRCA IDC, COADRE (COAD & READ), KIRC KIRP, LUAD, STAD, BRCA, COAD, LGG, OV (ovarious serous cystodeocarcinoma), BLCA, CESC, ESCA, GBM, HNSC, LIHC (hepatocellular carcinoma), LUSC, MESO (mesothelioma), PAAD, SARC, SKCM (skin cutaneous melanoma), UCEC (uterine corpus endometrial carcinoma), and UVM (uveal melanoma). ACC (adrenocortical carcinoma), CHOL (cholangiocarcinoma), PRAD (prostate adeno-carcinoma), READ (rectum adenocarcinoma), TGCT (testicular germ cell tumors), KICH (kidney chromophobe), and THCA (thyroid cancer) were excluded from the overall survival analysis due to the lack of death event cases and high censorship rates.

[0182] Table 2 contains details of the death events, censored samples, and total sample size for each cancer type.

[0183] Table 2: TCGA pancancer overall surival description. We include the total number of patients, patients that see a death event, and patients that have been censored due to a lack of follow-up.

[0184] Results

[0185] The inventors tested the quality of Idioma’s WSI representation in classification and survival tasks. The results below show the Idioma’s WSI representations performance for all tasks listed above, including different histological and molecular subtype classifications and overall survival over many cancer types. The inventors also provide UMAP dimensionality reductions of WSI representations and HPC token representations, showing their relation to cancer types and immune signature correlations respectively. Next, they illustrate the interpretability behind attention maps and HPCs to show how these interact in the creation of WSI representations. Finally, they showcase summaries of HPC to HPC interaction for entire cohorts and survival risk groups.

[0186] Classification analysis. Table 3 presents a comparative analysis between the new methods described herein and other existing methods at classification of histological type and subtype, molecular subtype, and genomic alteration prediction. The inventors compare these performances to relevant methods such as iBOT(Filiot et al., 2023a), MaskHIT(Jiang et al., 2023a), HIPT(Chen et al., 2022a), DeepAttnMISL(Yao et al., 2020a), H2T(Vu et al., 2023b), and MHATTN(Jiang et al., 2023b). Each datapoint represents the mean ROC AUG (Receiver operating characteristic area under the curve) over a 10-fold cross-validation for a given task. The method described herein is competitive in almost all classification tasks such as lung and kidney histological typing, out-performing other existing methods at brain type differentiation between LGG and GBM.

[0187] Table 3: Classification typing performances: The table reports the mean ROC AUG (Receiver operating characteristic area under the curve) over a 10-fold cross-validation for 15 classification tasks. Embodiments of the presently described methods (Idioma) are compared to other existing methods: iBOT(Filiot et al., 2023a), MaskHIT(Jiang et al., 2023a), HIPT(Chen et al., 2022a), DeepAttnMISL(Yao et al., 2020a), H2T(Vu et al., 2023b), and MHATTN(Jiang et al., 2023b). Best performing model per task is highlighted in bold.

[0188] Overall Survival Analysis. Table 4 shows a comparative analysis between methods of embodiments of the disclosure (Idioma) and other existing methods for overall survival analysis. We compare these performances to other relevant methods such as HPL(Quiros et al., 2023), iBOT(Filiot et al., 2023a), MaskHIT(Jiang et al., 2023a), HIPT(Chen et al., 2022a), PANTHER(Song et al., 2024), H2T(Vu et al., 2023b), and MHATTN(Jiang et al., 2023b). Each data point represents the mean concordance index over a 5-fold cross-validation for a given cancer type.

[0189] Out of the 23 different survival types, the method described herein outperforms other existing methods in 15 of them, with BRCA IDC, KIRP, LGG, PAAD, SKCM, and LIHC showing the largest differences. It is worth noting that the proposed method allows to analyze tissue patterns in the form of clusters, providing further insight into how the patient and WSI representations are created. By working with a finite set of clusters that define tissue pattern, the new method enables to easily analyze entire cohorts and the interaction of tissue patterns within those patients. This is in contrast to other existing methods which limit themselves to analyzing attention maps per patient. Further analysis is provided below in this regard.

[0190] Table 4: Overall survival performances: The table reports the mean concordance index over a 5- fold cross-validation for 23 cancer types. In The performance of embodiments of the methods of the disclosure (Idioma) is compared to other existing methods: HPL(Quiros et al., 2023), iBOT(Filiot et al., 2023a), MaskHIT(Jiang et al., 2023a), HIPT(Chen et al., 2022a), PANTHER(Song et al., 2024), and ABMIL(Chen et al., 2022b; Use et al., 2018). Best performing model per cancer type is highlighted in bold.

[0191] Interpretability analysis. In this section, the inventors provide further insight into Idioma’s results and features such as WSI representations, HPC dictionary embeddings, and attention maps. The representations obtained using the methods of the disclosure, as well as token embeddings and attention patterns were analysed to investigate whether the representations themselves could provide valuable information, as well as validate that the model is capturing biologically relevant information.

[0192] First, the inventors analysed the results from a pancancer model trained on 28 different cancer types. Figure 4 shows the results of a UMAP (Uniform Manifold Approximation and Projection - Mclnnes et al. 2020) analysis of the WSI representations for slides across 28 cancer types. On Figure 4, each point corresponds to a UMAP of a WSI vector representation, coloured based on its cancer type. This figure shows the complete set of TOGA WSIs labelled by its corresponding cancer type. Cancer type text labels are located on the plot based on an average of all x,y locations of UMAP WSIs for slides of the cancer type. This shows that the representations learned by Idioma are highly indicative of cancer type, since the representations clearly cluster by cancer type, with cancers of similar origin clustering together (e.g. LGG and GBM, KIRK and BLCA, etc.).

[0193] Figure 8 displays TCGA PAAD WSI TCGA-3A-A9IV-01 Z-00-DX1 . It contains the original WSI with an overlay of HPC membership per tile (A), the WSI with HPC overlay and brightness scaling based on each HPC attention values (B), and the accumulated attention values per HPC for the WSI (C). This figure provides interpretability into the amount of attention placed in each HPC when creating the WSI representation. In addition, it shows the spatial distribution of HPCs and the scale of attention placed on each tile, allowing pathologist to inspect which parts of the tissue section are relevant according to Idioma.

[0194] Next, the token embeddings associated with Idioma’s HPCs were analysed by UMAP dimensionality reduction. In particular, the inventors analysed HPC dictionary embeddings and HPC interactions on an Idioma model trained only on LUAD samples. The results are shown on Figure 5, where each point corresponds to a UMAP of the language model learned HPC token embedding (a vector of 768 dimensions), and HPC tokens are coloured based on the correlation value between HPC and immune signatures of proliferation, wound healing, stromal fraction and tumour infiltrating lymphocytes (TIL) regional fraction. The results on Figure 5 show that HPCs 0, 1 , 36, 43, 40, and 9 have similar embedding representations, showing positive correlations with TIL regional fraction and stromal fraction. Similarly, HPCs 18, 19, 6, and 21 show similar representations and negative correlations on stromal fraction. Finally, HPCs 28, 19, 26, and 27 show similar representations and negative correlations on proliferation. These results show that Idioma learns HPC embeddings that contain relevant histological information. We note that these representations are learned solely from HCP identifiers and their spatial distributions. These correlations were calculated as described in Quiros et al. (2023). Briefly, a Spearman rank correlation was calculated between cluster contributions in a slide and transcriptomic-based immune signatures. The immune landscape features used here are derived from a immune signature analysis of 33 cancer types (10,000K) published by (Thorsson et al. 2018). These immune signatures were derived from bulk RNASeq data from TCGA. The resulting Spearman correlation value is an average of per type correlation values. Cancer types and number of patients are an intersection between available TCGA cancer types and immune signature information; we also selected cancer types with at least 150 annotated samples. The complete cohort used for the correlation is composed of 279 patients of bladder urothelial carcinoma (BLCA), 364 patients of breast invasive carcinoma (BRCA), 187 patients of cervical squamous cell carcinoma and endocervical adenocarcinoma (CESC), 369 patients of colon adenocarcinoma (COAD), 366 patients of lung adenocarcinoma (LUAD), 367 patients of lung squamous cell carcinoma (LUSC), 247 patients of prostate adenocarcinoma (PRAD), 363 patients of skin cutaneous melanoma (SKCM), 278 patients of stomach adenocarcinoma (STAD), and 395 patients of uterine corpus endometrial carcinoma (UCEC). Finally, the inventors study HPC interactions using Idioma’s attention maps. They defined HPC interactions as the mean attention value from a source HPC X to a destination HPC Y across a set of patients. Given Idioma’s masking and HPC prediction training, we can interpret attention values as how predictive a destination HPC Y can be of a source HPC X based on their spatial location. After the model is trained, we use Idioma’s attention maps to quantify the attention placed between HPCs; defining HPC interactions. HPC interactions and HPC quantification (e.g. RNASeq derived signature correlations, pathologist annotations) provide a tool to scrutinize the possible relationships among histomorphological patterns; indicating if they are likely to co-occur in a WSI and if they are spatially related. Furthermore, HPC interactions also indicate on how HPCs are combined to create WSI representations. We can later use WSI representations on any given task such as overall survival. Through this process, we can relate HPC interactions and the relevance of WSI representations for the task. We can then compare the difference in HPC interactions across sets of patients such as high and low risk groups, providing a tool to analyse the relevance of HPC interactions in the task. In other words, attention values indicate how HPC information is mixed through Idioma. Given the masking training, Idioma will mix HPC information (through attention values) based on how predictive each HPC might be of others. Each attention value is computed based on a learnable similarity measure of a source HPC i and a destination HPC j, these can be broadly understood as how predictive HPC j might be of HPC i. After training, we can use these attention patterns (i.e. interaction patterns) to further scrutinize how each of the HPCs (i.e. morphological phenotypes) relate to each other. Along with individual characterization of HPCs such as immune signature correlations, cell density measures, or pathologist annotations; we can get a general view of how morphological phenotypes behave across subsets of patients (e.g. cancer subtype or risk groups).

[0195] Attention patterns of HPC to HPC were then analysed over the entire TCGA LUAD cohort, with results on Figure 6. In particular, Figure 6 shows a bi-hierarchical clustering of HPC interactions for the entire TCGA LUAD cohort where we colour-labelled each HPC with correlation values between the HPC and TIL Regional Fraction, Stromal Fraction, Proliferation, and Wound Healing. Figure 6 shows an example of how Idioma’s interaction patterns can be used to get a general understanding of how HPCs relate to each other for a cancer subtype. In this case, LUAD. From these figures, we can infer different groups of HPC interactions. The model’s attention weights were used to quantify HPC to HPC interactions. On Figure 6, each row corresponds to a ’Source HPC (A)’ where the entire row is a probability distribution of interaction with a ’Destination HPC (B)’ (column). On Figure 6, each HPC is additionally colour-labeled with correlation values between HPC and TIL Regional Fraction, Stromal Fraction, Proliferation, and Wound Healing as in Figure 5. Different groups of HPC interactions are highlighted on the figure: A, B, C, and D groups contain HPCs that tend to interact with the HPCs of their corresponding group. All these groups contain HPCs with negative correlations with TIL regional fraction and a combination of positive and negative correlations with stromal fraction, proliferation, and wound healing. In contrast, group E contains HPCs that are positively correlated with TIL Regional Fraction , stromal fraction, proliferation, and wound healing. These HPCs (HPCs 25, 0, 13, 9, 1 , 17, 10, 2, 29, and 33) tend to interact globally with other HPCs.

[0196] Finally, attention patterns of HPC to HPC were analysed in the same way as for Figure 6, but separately for LUAD samples classified in a high-risk group vs. a low-risk group using the above survival analysis. These results belong to a 5-fold cross-validation with no overlapping test sets. For each fold, we used the train set to fit the survival model and the test set to measure performance (c-index) and get high / low risk groups. This way we can loop over the entire dataset and get a high / low risk group based on patient samples unseen by the model during training. We used high / low risk groups to verify the model's ability to generalize across unseen samples. In this manner we can relate the model’s ability to generalize to new samples, survival performance, and the model’s interpretation of HPC interaction patterns with respect to survival. The high and low risk groups used show a statistically significant split through the logrank test with an associated p- value of 2.45 e-7. The results are shown on Figure 7, where each row corresponds to a ’Source HPC (A)’ where the entire row is a probability distribution of interaction with a ’Destination HPC (B)’ (column). Each HPC is colour labelled with correlation values between HPC and TIL Regional Fraction, Stromal Fraction, Proliferation, and Wound Healing, as explained above. The data shows that B and C groups see an increase in interaction when moving from high to low risk (compare Fig. 7A and Fig. 7B). Group B contains HPC with a positive high correlation value with TIL Regional Fraction while group C sees a negative correlation of TIL Regional Fraction but also a negative correlation with Proliferation. Figures 7A and 7B are an example of how we can use HPC interactions and two different sets of patients to scrutinize how HPCs could change interaction behaviours across patients. In this case, we present Figures 7A (high risk) and 7B (low risk) along with Group A, which shows positive correlations with Proliferation / Wound Healing and negative correlations with TILs / stroma cells; Group B shows positive correlations with Proliferation / Wound Healing / TIL / stroma cells, and Group C that shows negative correlations with Proliferation / Wound Healing / TIL / stroma cells. Comparing 7A and 7B, we highlight with arrows the increase of interactions in Groups B and C from high to low risk groups. These differences in interactions suggest that low risk patients have an increase of phenotypes with TILs, proliferation of cancer cells, wound healing, and stroma cells possibly suggesting a larger activation of an immune response. These two figures are an example of how we can use two different cohorts of patients to quantify how morphological phenotypes may behave differently across cohorts.

[0197] Figures 6 and 7 provide an example of how Idioma’s interpretation of HPC interactions can be used to study and scrutinize how HPCs could change interaction behaviours across patients. HPC Dictionary Size. As explained above, the HPL method described in Quiros et al. 2023 is used in this work to define Histomorphological Phenotype Clusters (HPCs) (see Figure 3A). HPL uses Leiden community detection and a resolution parameter to determine the number of HPCs. The resolution parameter is proportional to the number of HPCs. The inventors studied the impact of the number of HPCs in Idioma’s supervised tasks. They did so by sweeping along different resolution parameter values and training Idioma with the resulting HPCs.

[0198] Tables 5 and 6 display the classification and survival performance for different HPC set sizes. The results show that the method is flexible with respect to the size of HPC sets, in both types of supervised tasks the performance remains stable on average. The difference in performance ranges from 0.78 (38 HPCs) to 0.80 (60 HPCs) ROC AUG in classification tasks, and between

[0199] 0.644 (38 HPCs) and 0.661 (129 HPCs) concordance index in overall survival tasks.

[0200] Table 5: Classification performances across different HPC dictionary sizes: The table shows the mean ROC AUG (Receiver operating characteristic area under the curve) over a 10-fold cross- validation of each classification task. Idioma shows to be flexible with respect to the size of HPC sets, the average ROC AUG across the 15 classification tasks remains consistent between 0.782 and 0.799.

[0201] Table 6: Overall survival performances across different HPC dictionary sizes: The table shows the mean concordance index over a 5-fold cross-validation of each cancer type. Idioma shows to be flexible with respect to the size of HPC sets, the average concordance index across the 23 overall survival analyses remains consistent between 0.644 and 0.661 .

[0202] HPC masking methods. The training process used to train Idioma to generate all of the results above comprises training Idioma by predicting randomly masking out HPC memberships at different tile locations in the WSI regions. The inventors explored a different masking process as an alternative. Instead of random masking of tile locations, this process randomly samples an HPC present in the WSI region and mask out all tiles that have that membership. We refer to this process as HPC masking, which can be compared to the causal masking in GPT models. In this case, we ask Idioma to model the conditional probability of the HPC masked out given the context of all other HPCs present in the WSI region.

[0203] We study the impact of the two different masking processes (random and HPC based) by training two Idioma models with an HPC dictionary size of 60 HPCs (Leiden resolution 2.0). As explained above, the inventors believe that any combination of self-supervised learning approaches (e.g, MLM, CLM, CMM, prefix language modeling, unified language learning) and / or sequence representation learning (DINO, VIC-Reg, InfoNCE, or NSP) could be used. Further, we report performance for classification and survival tasks in Tables 7A and 7B. On average, both masking methods are comparable and report similar performance. Some cancer types such as UVM, ESCA, COAD, COADREAD benefit from the HPC masking process while KIRC or KIRP benefit from random masking. The inventors chose Random masking for further ablation studies based on the following factors: (i) HPC-based masking performed slightly worse on classification tasks; (ii) Random masking performs better in overall survival for BRCA, KIRC, KIRP, LUAD, BRCA, LUSC which are cancer types that are more commonly used for comparisons to other methods; (iii) previous work that only used LUAD patients (-440 samples) for HPC definition and Idioma training, instead of pancancer showed a drop of 8.7% on LUAD overall survival between Random masking and HPC-Based masking. This datapoint suggested that Random masking is more robust to lower sample sizes. That being said, HPC-based masking is in average as competitive as Random masking with respect to overall survival on the 23 cancer types.

[0204] Table 7A: Overall survival performances for masking methods: The table shows the mean concordance index over a 5-fold cross-validation for each cancer type. On average, both masking methods are comparable and report similar performance. However, some cancer types such as UVM, ESCA, COAD, COADREAD benefit from the HPC masking process while KIRC or KIRP benefit from random masking.

[0205] Table 7B. Classification performances for masking methods: We report the mean ROC AUG (Receiver operating characteristic area under the curve) over a 10-fold cross-validation of each classification task. On average, HPC-based masking shows a 4.1% performance drop on classification tasks showing that random masking could be more suitable for these.

[0206] WSI supervised classifiers. The inventors also studied the difference in performance when using simpler models on supervised tasks. As explained above and shown in Figure 3C, the results above are generated using an MLP that takes as input the WSI representations for classification and survival tasks. Alternatively, we could use a linear layer that takes as input the WSI representations (i.e. a linear model that takes the WSI representations as inputs). To evaluate the impact of the classifiers, the inventors trained an Idioma model with an HPC dictionary size of 60 HPCs (Leiden resolution 2.0) as described above, then used the two different classifiers (MLP or single linear layer, i.e. linear model) for the supervised tasks.

[0207] Table 8 shows a comparison between an MLP and linear classifier on overall survival analysis. The data show a small performance drop of 4.4% when replacing the MLP with a linear classifier, showing that Idioma’s WSI representations are robust even when using simpler models for supervised tasks.

[0208] Table 8: Overall survival performances of MLP and linear classifiers: The table shows the mean concordance index over a 5-fold cross-validation for each cancer type. We can see a small performance drop of 4.4% when we introduce a linear classifier instead of an MLP, showing that Idioma’s WSI representations are robust even when we use simpler models for supervised tasks.

[0209] Dataset size: pancancer vs cancer type. We study the difference in performance when we train Idioma models across all cancer types versus a specific cancer type. Table 9 shows the difference in performance on LUAD overall survival for two Idioma models, one on pancancer samples for 28 cancer types and another solely on LUAD samples. The pancancer Idioma model was trained on 1 .46 million WSI region samples and the LUAD Idioma model on 66 thousand WSI region samples. We report the mean concordance index over a 5-fold cross-validation on LUAD overall survival for both models. Random masking reports a drop of performance of 2.6% while HPC-based masking reports a 6.8% drop; showing that training using smaller number of samples (e.g. cancer type specific cohorts) is feasible, and that random masking provides more robust WSI representations against a lower amount of WSI region samples.

[0210] Table 10. Overall survival performances for Pancancer and LUAD Idioma mod- els: We trained two Idioma models, one on pancancer samples for 28 cancer types and another solely on LUAD samples. The pancancer Idioma model was trained on 1.46 million WSI region samples and the LUAD Idioma model on 66 thousand WSI region samples. We report the mean concordance index over a 5-fold cross-validation on LUAD overall survival for both models. Random masking reports a drop of performance of 2.6% while HPC-based masking reports a 6.8% drop; showing that random masking provides more robust WSI representations against a lower amount of WSI region samples.

[0211] Conclusion

[0212] The present examples describe and demonstrate the use of a new method, termed “Idioma”, that can learn whole slide image representations without annotations or labels. It does so by using a language model that learns the interaction dynamics between a finite set of morphological tissue patterns i.e. Histomorphological Phenotype Clusters (HPCs). The examples demonstrate the applicability of WSI representations in a range of 38 tasks including different histological classifications and survival analysis. The data provided show that Idioma is competitive on classification tasks, and outperforms other existing methods in 12 out of 16 cancer types in overall survival analysis. ■

[0213] The examples also demonstrate that Idioma enables further interpretability among tissue pattern interactions. By using a finite set of HPCs, one can more easily characterize interaction dynamics between different tissue patterns, not only for each patient but also through different risk groups or cohorts. The examples finally show how attention maps from the language model can be used to derive HPC interaction patterns.

[0214] The methods described have a number of advantages over existing solutions.

[0215] Diagnostic pathological services are under immense strain; there are insufficient trained staff, and the subclassification of disease into distinct entities with individualised therapies continually increases the diagnostic burden. Al algorithms are beginning to come into use to assist with these tasks, but at present they are very narrow in their capabilities, and approved examples are all highly supervised and therefore likely to be limited to a human ‘gold standard’. The methods described herein can train powerful algorithms which can encompass all relevant information in the tissue image, including elements which are very difficult for a human operator to perceive or interpret, and make use of all of this information by linking it to diagnosis or disease subgroup.

[0216] In this way the method can generate tools to expedite rapid precise diagnosis. The algorithm is trained with a large set of unannotated images, yielding a ‘dictionary’ of the disease type and superimposing additional learning from the implementation of a large language model. By linkage with known outcomes, the algorithm can further acquire the ability to predict patient outcome and refine diagnosis. An application or any other software implementation derived from this could be used by a pathologist to: (i) scan through their daily workload to identify priority or high-risk images to report first, and / or (ii) suggest important diagnoses for them to confirm or exclude. In addition, cases of diagnostic difficulty could be submitted to the app, and a likelihood of several differential diagnoses would be returned, linked to probabilities, and highlighting regions of the image supportive of each diagnosis. The former could use either individual binary classifiers or a single multiclass classifier. The latter would be most beneficial and could e.g. provide individual indices of suspicion for several diagnoses, e.g. "5% change of disease A, 2% chance of disease B". The latter could be performed using HPC-to-HPC interaction information for a single patient (e.g. as shown on Fig. 6) and / or information on which HPCs are most relevant for a WSI (e.g. as shown on Fig. 8), using a supervised model trained for the given task as described herein (i.e. analysing which HPC interactions are relevant to that prediction). The methods described herein can also be useful in the context of biomarker and drug development. Accurate biomarkers are crucial to the success of therapeutic development pipelines; accurate patient selection makes the difference between a useable drug and one which fails in phase II or III. Modern therapeutic development of targeted therapy relies on accurate identification of patients likely to benefit from the therapy. Hence identification of biomarker of therapeutic response (usable as “companion diagnostics”) is often crucial in every stage of the development and sometimes a requirement for regulatory approval. Currently this process is guided by biological expectation, such as e.g. the detection of the drug target. However, many drugs still fail at least in part because no good biomarker or a weakly performing biomarker has been identified.

[0217] The methods described herein are advantageous because they require no biological knowledge to link patient response to tumour histology (or any other variable of interest to be predicted), and can be used to discover very subtle and complex indicators of likely tumour response (e.g. spatial arrangements of cell types and tissue architectures). This could be particularly useful in various contexts including in clinical trial. For example, an early-phase trial could be used to provide tissue images and treatment response, using patients who are either unselected or selected with best available knowledge. These data can then be used to train a supervised model using an unsupervised model as described herein, which has previously learned all the meaningful morphological features of the relevant tumour type(s) from a large external training set. This would train the model to discover a precise morphological feature signature linked to treatment response. This can be investigated orthogonally (e.g. predicted high- or low-risk tumours could be biologically investigated for relevant altered biology, thereby linking morphology to response), and if deemed promising investigated for its ability to predict outcome in later-phase trials. A successful digital biomarker can then be brought to market as a companion diagnostic.

[0218] The methods described herein can also be advantageous in the context of automated digital image archiving for commercial access. Large amounts of routine diagnostic slides are being digitalised in Diagnostic Pathology Labs around the world every day. These however are not conveniently annotated, hence even though such data exist, they are not routinely useable even if stored or hosted in trusted research environments. The methods described herein can provide label free annotation, without significant input from pathologists, to create a catalogued pathology “library” of different tissue origin, disease type, and correlations to clinical outcomes if available.

[0219] The methods described herein are unique in their approach to summarise information across tiles of a whole slide image. The methods proposed explicitly convert a whole slide image into a sequence of tokens defined by HPCs. Each tile is assigned to an HPC, and the identity of the HPC becomes a token. Represented by iBOT, DeepAttnMISL, ABMIL, many Al methods instead use a technique called multiple instance learning and work with tile representations directly (rather than tile histomorphological phenotype cluster identifiers as is the case here). They are inferior in predicting patient outcome compared to the proposed approach (see iBOT, DeepAttnMISL and ABMIL in Table 4). Notably, iBOT uses a 3 times bigger encoder to extract tile representations but still is outperformed by the proposed approach in the three most common cancer types, namely breast, lung and colon cancers. Thus, the methods described herein also provide a more computational efficient way of obtaining informative tile representations for analysis of whole slide images. Further, with approaches using Vision Transformers (ViT), the tokens are inexplicably constructed within a neural network, and are therefore not interpretable. This approach is also inferior in performance (such as HIPT and H2T in table 4).

[0220] The HPL method that constructs the HPCs (Quiros et al. 2023) is the only prior art directly comparable to the approach described here. It uses the frequencies of HPCs for downstream tasks. The present approach instead trains a language model such as a transformer model on token sequences obtained from HPCs. This provides a much richer representation that is able to account not only for the area of different HPCs but for their respective localisations and how the HPCs interact with each other. This in turns translates to a much more complex, richer representation of biologically relevant information in the whole slide images, ultimately enabling better predictions of biologically and clinically relevant information.

[0221] References

[0222] All references cited herein are incorporated herein by reference in their entirety and for all purposes to the same extent as if each individual publication or patent or patent application was specifically and individually indicated to be incorporated by reference in its entirety.

[0223] Adrien Bardes, Jean Ponce, Yann LeCun. VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning. arXiv:2105.04906v3 28 Jan 2022Mathilde Caron, et al. Unsupervised learning of visual features by contrasting cluster assignments. arXiv:2006.09882v5 8 Jan 2021 . doi.org / 10.48550 / arXiv.2006.09882

[0224] Richard J Chen, et al. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 4015-4025, 2021 .

[0225] Richard J. Chen, Chengkuan Chen, Yicong Li, Tiffany Y. Chen, Andrew D. Trister, Rahul G. Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, June 2022a. doi: 10.1109 / cvpr52688. 2022.01567.

[0226] Richard J Chen, Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Muhammad Shaban, Maha Shady, Mane Williams, Bumjin Joo, Zahra Noor, et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell, 2022b Aug 8;40(8):865- 878. e6. doi: 10.1016 / j.ccell.2022.07.004.

[0227] Alexandre Filiot, et al. Scaling self-supervised learning for histopathology with masked image modeling. medRxiv, 2023a. doi: 10.1101 / 2023.07.21. 23292757.

[0228] Alexandre Filiot, et al. Scaling self-supervised learning for histopathology with masked image modeling. medRxiv, 2023b. doi: 10.1101 / 2023.07.21 . 23292757.

[0229] Maximilian Use, Jakub M. Tomczak, and Max Welling. Attention-based deep multiple instance learning. arXiv:1802.04712 2018v4, 28 June 2018. doi.org / 10.48550 / arXiv.1802.04712 Guillaume Jaume, et al. Transcriptomics-guided slide representation learning in computational pathology. arXiv:2405.11618v1 , 19 May 2024. doi.org / 10.48550 / arXiv.2405.11618

[0230] Shuai Jiang, Liesbeth Hondelink, Ariel A. Suriawinata, and Saeed Hassanpour. Masked pretraining of transformers for histology image analysis, 2023a. arXiv.2304.07434v1 14 April 2023. doi.org / 10.48550 / arXiv.2304.07434

[0231] Shuai Jiang, Arief A. Suriawinata, and Saeed Hassanpour. Mhattnsurv: Multi-head attention for survival prediction using whole-slide pathology images. Comput Biol Med. 2023 May;158:106883. doi: 10.1016 / j. compbiomed.2023.106883. Epub 2023 Apr 5.

[0232] Maxime Oquab, Timothee Darcet, Theo Moutakanni, et al. Dinov2: Learning robust visual features without supervision, arXiv.2304.07193v2, 2 Feb 2024. doi.org / 10.48550 / arXiv.2304.07193

[0233] Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Mirella Lapata, Phil Blunsom, and Alexander Koller, editors, Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157-163, Valencia, Spain, April 2017. Association for Computational Linguistics. aclanthology.org / E17-2025.

[0234] Quiros, A. C., Coudray, N., Yeaton, A. et al. Mapping the landscape of histomorphological cancer phenotypes using self-supervised learning on unannotated pathology slides. Nat Commun 15, 4596 (2024). E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley. Color transfer between images. IEEE Computer Graphics and Applications, 21 (5):34-41 , 2001 . doi: 10.1109 / 38.946629.

[0235] George Shaikovski, Adam Casson, Kristen Severson, et al. Prism: A multi-modal generative foundation model for slide-level histopathology. arXiv.2405.102542024v2, 22 May 2024. doi.org / 10.48550 / arXiv.2405.10254

[0236] Zhuchen Shao, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. arXiv.2106.00908v2, 31 Oct 2021. doi.org / 10.48550 / arXiv.2106.00908

[0237] Andrew H Song, Richard J Chen, Tong Ding, Drew FK Williamson, Guillaume Jaume, and Faisal Mahmood. Morphological prototyping for unsupervised slide representation learning in computational pathology. arXiv.2405.11643v1 19 May 2024. doi.org / 10.48550 / arXiv.2405.11643

[0238] Vincent Traag, Ludo Waltman, and Nees Jan van Eck. From Louvain to Leiden: guaranteeing well-connected communities. Scientific Reports, vol. (9): 5233 (2019). Also available as arXiv.1810.08473v3 30 Oct 2019, at doi.org / 10.48550 / arXiv.1810.08473

[0239] Quoc Dang Vu, Kashif Rajpoot, Shan E. Ahmed Raza, and Nasir Rajpoot. Handcrafted histological transformer (h2t): Unsupervised representation of whole slide images. Medical Image Analysis, 85:102743, 2023a. ISSN 1361 -8415. doi.org / 10.1016 / j.media.2023.102743.

[0240] Quoc Dang Vu, Kashif Rajpoot, Shan E. Ahmed Raza, and Nasir Rajpoot. Handcrafted histological transformer (h2t): Unsupervised representation of whole slide images. Medical Image Analysis, 85:102743, April 2023b. doi: 10.1016 / j. media.2023.102743.

[0241] Xiyue Wang, et al. Transpath: Transformer-based self-supervised learning for histopathological image classification. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pages 186-195. Springer, 2021.

[0242] Jiawen Yao, et al. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, 65:101789, 2020a. doi.org / 10.1016 / j. media.2020.101789.sciencedirect.com / science / article / pii / S1361841520301535.

[0243] Jiawen Yao, et al. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, 65:101789, October 2020b. doi: 10.1016 / j.media.2020.101789.

[0244] Shekoufeh Gorgi Zadeh and Matthias Schmid. Bias in cross-entropy-based training of deep survival networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43 (9):3126— 3137, 2021. doi: 10.1109 / TPAMI.2020.2979450. Shuangfei Zhai, et al. Stabilizing transformer training by preventing attention entropy collapse. arXiv.2303.06296v2, 11 March 2023. doi.org / 10.48550 / arXiv.2303.06296

[0245] Fengtao Zhou and Hao Chen. Cross-modal translation and alignment for survival analysis. arXiv.2309.12855v1 , 22 Sept 2023. doi.org / 10.48550 / arXiv.2309.12855

[0246] Ofir Press, Noah A. Smith, Mike Lewis. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. arXiv.2108.12409v2, 22 April 2022. doi.org / 10.48550 / arXiv.2108.12409

[0247] Alexey Dosovitskiy, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv.2010.11929v2, 3 June 2021 . doi.org / 10.48550 / arXiv.2010.11929

[0248] Poli, M. et al. 2023b. StripedHyena: Moving Beyond Transformers with Hybrid Signal Processing Models, github.com / togethercomputer / stripedhyena. doi: 10.57967 / hf / 1595. December 2023.

[0249] Peng Bo. Blinkdl / rwkv-lm: 0.01 , Aug 2021. doi.org / 10.5281 / zenodo.5196577

[0250] De et al. 2024. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models. arXiv:2402.19427v1 [cs.LG] 29 Feb 2024

[0251] Ashish Vaswani, et al. Attention is all you need. In 31 st Conference on Neural Information Processing Systems (NIPS), page 6000-6010. Curran Associates, Inc., 2017

[0252] Michael Poli, et al. Hyena hierarchy: Towards larger convolutional language models. arXiv preprint arXiv:2302.10866, 2023.

[0253] Lieber et al. 2024. Jamba: A Hybrid Transformer-Mamba Language Model. arXiv:2403.19887. Thu, 28 Mar 2024

[0254] Albert Gu, Tri Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. arXiv.2312.00752v2, 31 May 2024. doi.org / 10.48550 / arXiv.2312.00752

[0255] Adalberto Claudio Quiros, Roderick Murray-Smith, Ke Yuan. PathologyGAN: Learning deep representations of cancer tissue. Melba Journal, Volume 1 , MIDL 2020 special issue, 2021 :004.

[0256] Lianghui Zhu, et al. Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model. arXiv:2401 .09417v2. 10 Feb 2024.

[0257] Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805v2. 24 May 2019

[0258] Aaron van den Oord, Yazhe Li, Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. arXiv:1807.03748v2. 22 Jan 2019

[0259] Mahmoud Assran, et al. Self-Supervised Learning from Images with a Joint-Embedding Predictive

[0260] Architecture. arXiv:2301 ,08243v3. 13 Apr 2023

[0261] Yutao Sun, et al. Retentive Network: A Successor to Transformer for Large Language Models. arXiv:2307.08621 v4. 9 Aug 2023.

[0262] Florian Mai, et al. HyperMixer: An MLP-based Low Cost Alternative to Transformers. arXiv:2203.03691 v3. 13 Nov 2023

[0263] Ilya Tolstikhin, et al. MLP-Mixer: An all-MLP Architecture for Vision. arXiv:2105.01601 v4. 11 Jun 2021.

[0264] Zhidian Zhang, et al. Protein language models learn evolutionary statistics of interacting sequence motifs. bioRxiv. 31 Jan 2024. doi.org / 10.1101 / 2024.01 .30.577970.

[0265] Tri Dao, et al. Flash Attention: Fast and Memory-Efficient Exact Attention with IO-Awareness. arXiv:2205.14135v2. 23 Jun 2022

[0266] Margaux Luck, et al. Learning to rank for censored survival data. arXiv:1806.01984v2. 8 Jun 2018 Wong, W. H. Theory of Partial Likelihood. The Annals of Statistics. Vol. 14, No. 1 (Mar., 1986), pp. 88-123

[0267] Mclnnes L., Healy J., Melville J. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv:1802.03426v3. 18 Sep 2020.

[0268] Thorsson et al. The Immune Landscape of Cancer. Immunity. Volume 48, Issue 4, 17 April 2018, Pages 812-830. e14

[0269] The specific embodiments described herein are offered by way of example, not by way of limitation. Various modifications and variations of the described compositions, methods, and uses of the technology will be apparent to those skilled in the art without departing from the scope and spirit of the technology as described. Any sub-titles herein are included for convenience only, and are not to be construed as limiting the disclosure in any way.

[0270] The methods of any embodiments described herein may be provided as computer programs or as computer program products or computer readable media carrying a computer program which is arranged, when run on a computer, to perform the method(s) described above.

[0271] Unless context dictates otherwise, the descriptions and definitions of the features set out above are not limited to any particular aspect or embodiment of the invention and apply equally to all aspects and embodiments which are described.

[0272] Throughout the specification and claims, the following terms take the meanings explicitly associated herein, unless the context clearly dictates otherwise. The phrase “in one embodiment” as used herein does not necessarily refer to the same embodiment, though it may. Furthermore, the phrase “in another embodiment” as used herein does not necessarily refer to a different embodiment, although it may. Thus, as described below, various embodiments of the invention may be readily combined, without departing from the scope or spirit of the invention.

[0273] It must be noted that, as used in the specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from “about” one particular value, and / or to “about” another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when values are expressed as approximations, by the use of the antecedent “about,” it will be understood that the particular value forms another embodiment. The term “about” in relation to a numerical value is optional and means for example + / - 10%.

[0274] Throughout this specification, including the claims which follow, unless the context requires otherwise, the word “comprise” and “include”, and variations such as “comprises”, “comprising”, and “including” will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps. Other aspects and embodiments of the invention provide the aspects and embodiments described above with the term “comprising” replaced by the term “consisting of” or ’’consisting essentially of”, unless the context dictates otherwise.

[0275] The features disclosed in the foregoing description, or in the following claims, or in the accompanying drawings, expressed in their specific forms or in terms of a means for performing the disclosed function, or a method or process for obtaining the disclosed results, as appropriate, may, separately, or in any combination of such features, be utilised for realising the invention in diverse forms thereof.

Claims

62Claims:1 . A method of analysing a histopathology sample, the method comprising: receiving, by a processor, a whole slide image of the sample, obtaining a plurality of tile representations for respective tiles obtained from the whole slide image, using a tile representation machine learning model, wherein a tile representation machine learning model is a machine learning model that has been trained to learn a representation associated with a histopathology image tile, assigning each of the plurality of tile representations to one of a predetermined set of histomorphological phenotype clusters, wherein the set of histomorphological phenotype clusters are clusters of tile representations that have been previously obtained using training tile representations obtained using the tile representation machine learning model for a plurality of training tiles from a plurality of training whole slide images associated with respective samples in a cohort comprising a plurality of histopathology samples, obtaining a whole slide image representation using a histomorphological phenotype cluster language model, wherein the histomorphological phenotype cluster language model is a machine learning model that has been trained in a self-supervised manner using training whole slide images to take as input an indication of the assigned histomorphological phenotype cluster for each of a plurality of tiles of a whole slide image and learn a representation associated with the whole slide image from which histopathological phenotype cluster membership of individual tiles of the whole slide image can be reconstructed, and predicting one or more biological or clinical features associated with the sample using a task specific machine learning model, wherein the task specific machine learning model is a model that has been trained using training whole slide images and optionally associated one or more ground truth biological or clinical features of interest to predict the one or more biological or clinical feature of interest for a whole slide image using as input the whole slide image representation provided by the histomorphological cluster language model for the whole slide image.

2. The method of claim 1 , wherein the histomorphological phenotype cluster language model has been trained to learn respective token embeddings of a predetermined size corresponding to each of the histomorphological phenotype cluster in the predetermined set of histomorphological phenotype clusters.

3. The method of any of claims 1 or 2, wherein the histomorphological phenotype cluster language model is a sequence model.

4. The method of any preceding claim, wherein the histomorphological phenotype cluster language model is a deep learning model, optionally recurrent neural network-based model, a long short-63 term memory model, a convolution-based model, a state-space model, a multilayer perceptron (MLP)-based model, a retentive neural network, or a transformer-based model.

5. The method of any preceding claim, wherein the tile representation machine learning model is a deep learning model, optionally a convolutional neural network, a transformer based model, a graph neural network, a generative adversarial model, a variational autoencoder, a diffusion model, or a model combining convolutional and attention blocks., and / or wherein the tile representation machine earning model has been trained in a self-supervised manner6. The method of any preceding claim, wherein the task specific machine learning model is a classification model, a regression model, a generative model, or a survival model, optionally wherein the task specific machine learning model is a linear model or multilayer perceptron.

7. The method of any preceding claim, wherein the whole slide image and training whole slide images are stained image and / or colour images, optionally wherein the whole slide images and training whole slide images are chemically stained images, immunofluorescence images, immunohistochemistry images or in situ hybridisation images, optionally H&E stained image.

8. The method of any preceding claim wherein the whole slide image and training whole slide images are images associated with the same magnification.

9. The method of any preceding claim, further comprising obtaining the plurality of tiles from the whole slide image, optionally wherein the plurality of tiles are non-overlapping tiles and / or tiles of predetermined equal size.

10. The method of claim 9, wherein obtaining the plurality of tiles from the whole slide image comprises obtaining a first plurality of tiles and selecting a second plurality of tiles from the first plurality of tiles that are associated with a percentage of area showing tissue above a predetermined percentage for obtaining respective tile representations using the tile representation machine learning model, optionally wherein unselected tiles are removed or associated with a predetermined identifier.11 . The method of any preceding claim, wherein the set of histomorphological phenotype clusters are clusters of tile representations that have been previously obtained by clustering the training tile representations using a clustering algorithm, optionally a community detection algorithm applied to a graph where each node is associated with a training tile representation and edges connect training tiles representations with their one or more nearest neighbours in terms of tile representation.6412. The method of any preceding claim, wherein the predetermined set of histomorphological phenotype clusters comprises between 20 and 150 clusters, and / or wherein the number of clusters in the predetermined set of histomorphological phenotype clusters has been determined by applying a clustering algorithm to the training tiles representations.

13. The method of any preceding claim, wherein the histomorphological phenotype cluster language model has been trained using a self-supervised learning objective, optionally wherein the self-supervised learning objective is selected from masked language modelling, causal language modelling, and HPC masking in which all input tokens associated with a particular HPC are masked in each training whole slide image provided as input to the model.

14. The method of any preceding claim, wherein the histomorphological phenotype cluster language model takes as input a vector corresponding to a flattened version of a two-dimensional matrix with elements comprising an indication of the assigned histomorphological phenotype cluster for each of the plurality of tiles of a whole slide image, optionally wherein the vector comprises a predetermined identifier associated with any elements of the matrix corresponding to tiles that do not satisfy one or more predetermined filtering criteria, optionally wherein the one or more predetermined filtering criteria include a criterion that applies to the percentage area of the tile showing tissue.

15. The method of any preceding claim, wherein the histomorphological phenotype cluster language model is configured to take into account the relative position of input tokens or corresponding token embeddings, optionally by using positional encoding or linear biases.

16. The method of any preceding claim, wherein the histomorphological phenotype cluster language model takes as input an indication of the assigned histomorphological phenotype cluster for each of a plurality of tiles of a whole slide image, wherein each tile represents an area of tissue of between 0.1 mm2and 1 mm2, about 0.2 mm2, or between 20 pm by 20 pm and 1 mm by 1 mm, and / or wherein the plurality of tiles together represent an area of tissue of at least 10 mm by 10mm, and / or wherein the plurality of tiles comprises at least 1024 tiles each between 0.1 mm2and 1 mm2.

17. The method of any preceding claim, wherein the histomorphological phenotype cluster language model has been trained to predict a plurality of respective output embeddings associated with each respective tile of the plurality of tiles, wherein the whole slide image representation is obtained by combining the plurality of output embeddings, optionally using global average pooling.6518. The method of any preceding claim, wherein the task specific machine learning model has been trained in a supervised manner using training whole slide images and associated one or more ground truth biological or clinical features of interest, optionally wherein the predicted one or more biological or clinical feature are selected from: a prognosis, a diagnostic, a treatment response prediction, and an identification of a biological feature of the sample, optionally wherein a prognosis is selected from a classification between prognosis classes, and a prediction of a survival metric, and / or wherein a diagnostic is selected from the presence or absence of a pathological feature, a classification between different types or subtypes of a disease or disorder, and the presence or absence of a disease or disorder , and / or wherein a biological feature of the sample is selected from the presence of a histological feature, and the presence of a genomic alteration feature.

19. The method of any preceding claim, wherein the predicted one or more biological or clinical feature include a classification between a plurality of histological types, a classification between a plurality of histological subtypes, a classification between a plurality of molecular subtypes, a genomic alteration prediction, or a survival prediction, optionally wherein the survival prediction uses a survival model trained through negative log-likelihood on discrete time-to-event.

20. The method of any preceding claim, wherein the histomorphological phenotype cluster language model has been trained using training whole slide images comprising images of histopathology samples comprising tumour tissue from a plurality of cancer types, optionally wherein the histomorphological phenotype cluster language model has been additionally finetuned using training whole slide images comprising images of histopathology samples comprising tumour tissue from a particular cancer type.21 . A method of providing a diagnostic and / or prognostic and / or treatment response prediction for a subject, the method comprising: analysing a histopathology sample previously obtained from the subject using the method of any preceding claim, wherein the predicted one or more biological or clinical features associated with the sample comprise a diagnostic and / or prognostic and / or treatment response prediction.

22. A method of screening a plurality of histopathology samples, the method comprising: analysing each histopathology sample using the method of any of claims 1 to 20, wherein the predicted one or more biological or clinical features associated with the sample comprises a classification between a plurality of classes associated with respective risk levels (e.g. a first class associated with high risk samples, such as e.g. samples that are likely to include a pathological feature, such as a malignancy, and a second class associated with low risk samples, such as e.g. samples that are unlikely to include a pathological feature, such as a malignancy), or a predicted probability of66 the sample being a high risk sample (e.g. a probability of the sample including a pathological feature, such as presence of a malignancy).

23. A method of providing a trained machine learning model for analysing a histopathology sample, the method comprising: providing a plurality of training whole slide images of a plurality of training samples; obtaining a plurality of training tile representations for respective tiles obtained from the plurality of training whole slide images, using a trained tile representation machine learning model, wherein a tile representation machine learning model is a machine learning model that has been trained to learn a representation associated with a histopathology image tile, assigning each of the plurality of training tile representations to one of a predetermined set of histomorphological phenotype clusters, wherein the set of histomorphological phenotype clusters are clusters of tile representations that have been previously obtained using training tile representations obtained using the tile representation machine learning model for a plurality of training tiles from a plurality of training whole slide images associated with respective samples in a cohort comprising a plurality of histopathology samples, training a histomorphological phenotype cluster language model in a self-supervised manner using the training whole slide images to take as input an indication of the assigned histomorphological phenotype cluster for each of a plurality of training tiles of a training whole slide image and learn a representation associated with the training whole slide image from which histopathological phenotype cluster membership of individual training tiles of the training whole slide image can be reconstructed, and training a task specific machine learning model, optionally in a supervised manner, using training whole slide images and optionally associated one or more ground truth biological or clinical features of interest to predict the one or more biological or clinical feature of interest for a whole slide image using as input the whole slide image representation provided by the histomorphological cluster language model for the whole slide image, optionally wherein the method further comprises training the tile representation machine learning model and / or obtaining the set of histomorphological phenotype clusters.

24. A system comprising one or more processor and one or more computer readable medium / media comprising instructions that, when executed by the processor, cause the processor to perform the steps of any of claims 1 to 23.

25. One or more non-transitory computer readable media comprising instructions that, when executed by at least one processor, cause the at least one processor to perform the method of any of claims 1 to 23.

Images