Machine learning enabled hepatocellular carcinoma molecular subtype classification

A machine learning-based digital pathology platform segments tumor images into tiles and applies models to classify HCC subtypes accurately, addressing data scarcity and variability issues, enhancing therapeutic target identification.

CN119072722APending Publication Date: 2024-12-03GENENTECH INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380035959.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-29
Filing Date
2023-04-26
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing methods for classifying hepatocellular carcinoma (HCC) molecular subtypes are limited by high costs and scarcity of patient tumor-specific transcriptome data, and traditional digital pathology methods suffer from significant result variability.

Method used

A machine learning-enabled digital pathology platform segments tumor images into tiles, applies machine learning models to determine molecular subtypes, and generates visual representations of subtype distributions using techniques like ANN, MIL, and dimensionality reduction to accurately classify HCC subtypes without relying on transcriptome data.

Benefits of technology

Improves the accuracy of HCC molecular subtype classification, reducing result variability and enabling effective therapeutic target identification by providing precise molecular subtype insights.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119072722A_ABST
    Figure CN119072722A_ABST
Patent Text Reader

Abstract

The present disclosure relates to a method for image-based classification of hepatocellular carcinoma (HCC) molecular subtypes, which may include determining a plurality of tiles within an image depicting a plurality of cells of a biological sample, where each tile depicts a portion of the plurality of cells contained in the sample. A machine learning model may be applied to determine a molecular subtype for the portion of the plurality of cells depicted in each tile. Further, an overall molecular subtype of the plurality of cells depicted in the image for the biological sample may be determined based on the molecular subtype of the portion of the plurality of cells depicted in each of the plurality of tiles. For example, another machine learning model may be applied to determine the overall molecular subtype of the plurality of cells depicted in the image of the biological sample. Related systems and computer program products are also provided.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 337,006, filed on April 29, 2022, the entire contents of which are hereby incorporated by reference for all purposes. Technical Field

[0003] The subject matter described herein relates generally to digital pathology, and more specifically to machine learning-based techniques for molecular subtype classification of hepatocellular carcinoma (HCC). Background Art

[0004] Hepatocellular carcinoma (HCC) is a common disease with high mortality but few effective treatment options. Although combination immunotherapies such as atezolizumab (anti-PD-L1) and bevacizumab (anti-VEGF) have shown strong anti-tumor activity in clinical trials, the disease still progresses in a large proportion of patients. In fact, an accurate understanding of HCC tumor heterogeneity and the corresponding immune response mechanisms remains elusive. Therefore, biological insights into HCC heterogeneity remain crucial for identifying effective new therapeutic targets. Summary of the invention

[0005] Systems, methods, and articles of manufacture, including computer program products, for image-based classification of hepatocellular carcinoma (HCC) subtypes are provided. In some exemplary embodiments, a system comprising at least one processor and at least one memory is provided. The at least one memory may include program code that provides operations when executed by at least one processor. The operations may include: determining a plurality of tiles within an image of a biological sample, each of the plurality of tiles depicting a portion of the biological sample; applying a first machine learning model to determine a molecular subtype for a portion of the biological sample depicted in each of the plurality of tiles; and determining an overall molecular subtype for the biological sample based at least on the molecular subtype of each of the plurality of tiles.

[0006] In some variations, one or more features disclosed herein including the following features may optionally be included in any feasible combination: The overall molecular subtype of the biological sample may be determined by applying a second machine learning model.

[0007] In some variations, the second machine learning model may be trained to determine the overall molecular subtype by at least determining a representative encoding of the plurality of tiles.

[0008] In some embodiments, the second machine learning model may be further trained to assign a higher attention score to a first tile in the plurality of tiles compared to a second tile in the plurality of tiles while determining a representative encoding of the plurality of tiles.

[0009] In some variations, a higher attention score may indicate that a first molecular subtype of a first tile contributes more to the representative encoding of the plurality of tiles than a second molecular subtype of a second tile.

[0010] In some variations, a higher attention score may indicate that a first molecular subtype of a first tile is more relevant to the overall molecular subtype of the biological sample than a second molecular subtype of a second tile.

[0011] In some variations, the second machine learning model may include a multiple instance learning (MIL) model.

[0012] In some variations, the second machine learning model may include an attention mechanism.

[0013] In some variations, the operations may further include generating a first visual representation of the reduced-dimensional representation of the plurality of tiles.

[0014] In some variations, the first visual representation may include one or more visual indicators configured to provide visual distinction between tiles of different subtypes.

[0015] In some variations, the first visual representation may be generated by applying a dimensionality reduction technique to at least a pixel-wise representation of each tile in the plurality of tiles.

[0016] In some variations, dimensionality reduction techniques may include one or more of principal component analysis (PCA), uniform manifold approximation and projection (UMAP), and t-distributed stochastic neighbor embedding (t-SNE).

[0017] In some variations, the first visual representation may be generated to include one or more visual indications configured to provide visual distinction between one or more clusters of similar tiles within the plurality of tiles.

[0018] In some variations, the operations may further include generating a second visual representation depicting the plurality of tiles organized according to one or more similar tile clusters.

[0019] In some variations, the operations may further include generating a second visual representation depicting a spatial distribution of the one or more clusters of similar tiles within the biological sample.

[0020] In some variations, one or more clusters of similar tiles may be identified by applying a cluster analysis technique.

[0021] In some embodiments, clustering analysis techniques may include one or more of k-means clustering, mean shift clustering, density-based spatial clustering of applications with noise (DBSCAN), expectation maximization (EM) clustering using a Gaussian mixture model (GMM), and agglomerative hierarchical clustering.

[0022] In some variations, the overall molecular subtype of a biological sample may be determined based at least on the amount of each molecular subtype present within a plurality of cells.

[0023] In some variations, the operations may further include generating a visual representation depicting a spatial distribution of one or more molecular subtypes within the biological sample based at least on the molecular subtypes of each tile in the plurality of tiles.

[0024] In some variations, the operations may further include generating a visual representation depicting a first tile having a first subtype and a second tile having the first subtype from among a plurality of tiles from the same biological sample or different biological samples.

[0025] In some variations, the visual representation is further generated to depict a third tile having the second subtype and a fourth tile of the second subtype from among the plurality of tiles from the same biological sample or different biological samples.

[0026] In some variations, the plurality of tiles may exclude one or more tiles in the image having a background of the image above a threshold ratio or an average color channel variance below a threshold.

[0027] In some variations, the first machine learning model may be trained to determine a molecular subtype associated with each tile in the plurality of tiles based on a morphological pattern present within a portion of the biological sample depicted in each tile.

[0028] In some variations, the first machine learning model may include an artificial neural network (ANN).

[0029] In some variations, the biological sample may include a hepatocellular carcinoma (HCC) tissue sample. Each of the plurality of tiles may be assigned a molecular subtype, the molecular subtype including one of a cholangioid subtype, a hepatocyte-like subtype, or a progenitor-like subtype. The overall molecular subtype of the plurality of cells depicted in the image of the biological sample may include one of a cholangioid subtype, a hepatocyte-like subtype, or a progenitor-like subtype.

[0030] In some variations, the operations may further include identifying a plurality of molecular subtypes based at least on transcriptome data associated with the plurality of tumor tissue samples.

[0031] In some variations, the plurality of tumor tissue samples may include a plurality of hepatocellular carcinoma (HCC) tumor tissue samples.The plurality of molecular subtypes may include a cholangioid subtype, a hepatocyte-like subtype, and a progenitor-like subtype.

[0032] In some variations, the first machine learning model may be trained to assign to each of the plurality of tiles a label corresponding to one of a plurality of molecular subtypes identified based on the transcriptome data.

[0033] In some variations, the overall molecular subtype of the plurality of cells depicted in the image of the biological sample may include one of a plurality of molecular subtypes identified based on the transcriptome data.

[0034] In some variations, wherein the image may depict a plurality of cells comprised by the biological sample, each of the plurality of tiles may depict a portion of the plurality of cells comprised by the biological sample.

[0035] In another aspect, a method for image-based hepatocellular carcinoma (HCC) subtype classification is provided. The method may include: determining a plurality of tiles within an image of a biological sample, each of the plurality of tiles depicting a portion of the biological sample; applying a first machine learning model to determine a molecular subtype for the portion of the biological sample depicted in each of the plurality of tiles; and determining an overall molecular subtype for the biological sample based at least on the molecular subtype of each of the plurality of tiles.

[0036] In some variations, one or more features disclosed herein including the following features may optionally be included in any feasible combination: The overall molecular subtype of the biological sample may be determined by applying a second machine learning model.

[0037] In some variations, the second machine learning model may be trained to determine the overall molecular subtype by at least determining a representative encoding of the plurality of tiles.

[0038] In some embodiments, the second machine learning model may be further trained to assign a higher attention score to a first tile in the plurality of tiles compared to a second tile in the plurality of tiles while determining a representative encoding of the plurality of tiles.

[0039] In some variations, a higher attention score may indicate that a first molecular subtype of a first tile contributes more to the representative encoding of the plurality of tiles than a second molecular subtype of a second tile.

[0040] In some variations, a higher attention score may indicate that a first molecular subtype of a first tile is more relevant to the overall molecular subtype of the biological sample than a second molecular subtype of a second tile.

[0041] In some variations, the second machine learning model may include a multiple instance learning (MIL) model.

[0042] In some variations, the second machine learning model may include an attention mechanism.

[0043] In some variations, the method may further include generating a first visual representation of the reduced-dimensional representation of the plurality of tiles.

[0044] In some variations, the first visual representation may include one or more visual indicators configured to provide visual distinction between tiles of different subtypes.

[0045] In some variations, the first visual representation may be generated by applying a dimensionality reduction technique to at least a pixel-wise representation of each tile in the plurality of tiles.

[0046] In some variations, dimensionality reduction techniques may include one or more of principal component analysis (PCA), uniform manifold approximation and projection (UMAP), and t-distributed stochastic neighbor embedding (t-SNE).

[0047] In some variations, the first visual representation may be generated to include one or more visual indications configured to provide visual distinction between one or more clusters of similar tiles within the plurality of tiles.

[0048] In some variations, the method may further include generating a second visual representation depicting the plurality of tiles organized according to one or more similar tile clusters.

[0049] In some variations, the method may further include generating a second visual representation depicting the spatial distribution of the one or more clusters of similar tiles within the biological sample.

[0050] In some variations, one or more clusters of similar tiles may be identified by applying a cluster analysis technique.

[0051] In some embodiments, clustering analysis techniques may include one or more of k-means clustering, mean shift clustering, density-based spatial clustering of applications with noise (DBSCAN), expectation maximization (EM) clustering using a Gaussian mixture model (GMM), and agglomerative hierarchical clustering.

[0052] In some variations, the overall molecular subtype of a biological sample may be determined based at least on the amount of each molecular subtype present within a plurality of cells.

[0053] In some variations, the method may further comprise generating a visual representation depicting a spatial distribution of one or more molecular subtypes within the biological sample based at least on the molecular subtypes of each tile in the plurality of tiles.

[0054] In some variations, the method may further include generating a visual representation depicting a first tile having a first subtype and a second tile having the first subtype from among a plurality of tiles from the same biological sample or different biological samples.

[0055] In some variations, the visual representation is further generated to depict a third tile having the second subtype and a fourth tile of the second subtype from among the plurality of tiles from the same biological sample or different biological samples.

[0056] In some variations, the plurality of tiles may exclude one or more tiles in the image having a background of the image above a threshold ratio or an average color channel variance below a threshold.

[0057] In some variations, the first machine learning model may be trained to determine a molecular subtype associated with each tile in the plurality of tiles based on a morphological pattern present within a portion of the biological sample depicted in each tile.

[0058] In some variations, the first machine learning model may include an artificial neural network (ANN).

[0059] In some variations, the biological sample may include a hepatocellular carcinoma (HCC) tissue sample. Each of the plurality of tiles may be assigned a molecular subtype, the molecular subtype including one of a cholangioid subtype, a hepatocyte-like subtype, or a progenitor-like subtype. The overall molecular subtype of the plurality of cells depicted in the image of the biological sample may include one of a cholangioid subtype, a hepatocyte-like subtype, or a progenitor-like subtype.

[0060] In some variations, the method may further comprise identifying a plurality of molecular subtypes based at least on transcriptome data associated with the plurality of tumor tissue samples.

[0061] In some variations, the plurality of tumor tissue samples may include a plurality of hepatocellular carcinoma (HCC) tumor tissue samples.The plurality of molecular subtypes may include a cholangioid subtype, a hepatocyte-like subtype, and a progenitor-like subtype.

[0062] In some variations, the first machine learning model may be trained to assign to each of the plurality of tiles a label corresponding to one of a plurality of molecular subtypes identified based on the transcriptome data.

[0063] In some variations, the overall molecular subtype of the plurality of cells depicted in the image of the biological sample may include one of a plurality of molecular subtypes identified based on the transcriptome data.

[0064] In some variations, wherein the image may depict a plurality of cells comprised by the biological sample, each of the plurality of tiles may depict a portion of the plurality of cells comprised by the biological sample.

[0065] In another aspect, a computer program product is provided that includes a non-transitory computer-readable medium storing instructions. The instructions may cause operations that may be performed by at least one data processor. The operations may include: determining a plurality of tiles within an image of a biological sample, each tile in the plurality of tiles depicting a portion of the biological sample; applying a first machine learning model to determine a molecular subtype for the portion of the biological sample depicted in each tile in the plurality of tiles; and determining an overall molecular subtype for the biological sample based at least on the molecular subtype of each tile in the plurality of tiles.

[0066] Specific implementations of the current subject matter may include, but are not limited to, methods consistent with the description provided herein and articles including tangibly embodied machine-readable media that are operable to cause one or more machines (e.g., computers, etc.) to cause operations that implement one or more of the features described. Similarly, a computer system that may include one or more processors and one or more memories coupled to the one or more processors is also described. A memory that may include a non-transitory computer-readable or machine-readable storage medium may include, encode, store, etc., one or more programs that cause one or more processors to perform one or more of the operations described herein. A computer-implemented method consistent with one or more implementations of the current subject matter may be implemented by one or more data processors present in a single computing system or multiple computing systems. Such multiple computing systems may be connected and may exchange data and / or commands or other instructions, etc., via one or more connections, including, for example, via a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, etc.) via a direct connection between one or more computing systems in the multiple computing systems, etc.

[0067] The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features and advantages of the subject matter described herein will become apparent with reference to the description and drawings, as well as the claims. Although certain features of the subject matter disclosed herein are described for illustrative purposes related to hepatocellular carcinoma (HCC), it should be readily understood that such features are not intended to be limiting. The claims following this disclosure are intended to define the scope of the subject matter protected. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate certain aspects of the subject matter disclosed herein and, together with the description, help explain some principles associated with the disclosed embodiments.

[0069] Figure 1 depicts a system diagram illustrating an example of a digital pathology system according to some exemplary embodiments;

[0070] Figure 2 depicts the association between hepatocellular carcinoma (HCC) molecular subtypes and liver epithelial cell lineages according to some exemplary embodiments;

[0071] Figure 3A depicts a schematic diagram showing an example of an image divided into separate tiles according to some example embodiments;

[0072] Figure 3B depicts examples of individual tiles from an image of a tumor sample according to some exemplary embodiments;

[0073] Figure 4 depicts a schematic diagram illustrating an example of a machine learning model trained to perform image-based molecular subtype classification according to some exemplary embodiments;

[0074] Figure 5A depicts screenshots showing examples of visual representations according to some exemplary embodiments;

[0075] Figure 5B depicts a screenshot showing another example of a visual representation according to some exemplary embodiments;

[0076] Fig. 6A depicts a screenshot showing another example of a visual representation according to some exemplary embodiments;

[0077] Figure 6B depicts a screenshot showing another example of a visual representation according to some exemplary embodiments;

[0078] Figure 7depicts a screenshot showing another example of a visual representation according to some exemplary embodiments;

[0079] Fig. 8A depicts a screenshot showing another example of a visual representation according to some exemplary embodiments;

[0080] Figure 8B depicts a screenshot showing another example of a visual representation according to some exemplary embodiments;

[0081] Figure 8C depicts a screenshot showing another example of a visual representation according to some exemplary embodiments;

[0082] Fig.8D depicts a screenshot showing another example of a visual representation according to some exemplary embodiments;

[0083] Fig. 8E depicts a screenshot showing another example of a visual representation according to some exemplary embodiments;

[0084] Fig. 9 depicts a flow chart illustrating an example of a process for image-based classification of molecular subtypes of hepatocellular carcinoma (HCC) according to some exemplary embodiments;

[0085] Fig.10 depicts a graph showing a receiver operating characteristic (ROC) curve representing the performance of a machine learning model trained to perform image-based molecular subtype classification according to some exemplary embodiments;

[0086] Fig.11 Depicted is a block diagram illustrating an example of a computing system in accordance with some example embodiments.

[0087] When applicable, like reference numerals refer to like structures, features, or elements. DETAILED DESCRIPTION

[0088] Hepatocellular carcinoma (HCC) is a highly heterogeneous disease with complex etiology and various molecular and cellular dysfunctions. As noted, biological insights into the heterogeneity of hepatocellular carcinoma remain critical for identifying effective new therapeutic targets. For example, in hepatocellular carcinoma (HCC) and other cancers, molecular subtypes present in tumors can serve as key biomarkers for predicting patient response to treatment and survival. However, due to the high cost and scarcity of patient tumor-specific transcriptome data, the practicality of traditional transcriptome-based molecular subtype classification (e.g., RNA sequence-based molecular subtype classification) is limited. At the same time, traditional digital pathology methods for image-based molecular subtype classification are associated with significant result variability. Therefore, in some exemplary embodiments, the digital pathology platform can be configured to perform image-based molecular subtype classification enabled by machine learning, wherein the molecular subtype of a tumor sample (such as a hepatocellular carcinoma (HCC) tumor sample) is determined by applying one or more machine learning models to images of tumor samples instead of transcriptome data and / or in addition to transcriptome data. Therefore, even in the absence of transcriptome data, one or more molecular subtypes present in a tumor sample can be determined based on morphological patterns detected in an image of the tumor sample.

[0089] In some exemplary embodiments, an image depicting a tumor sample may exhibit an overall molecular subtype determined based on the molecular subtypes of one or more separate portions of the image. For example, a digital pathology platform may segment an image depicting cells of a tumor sample (e.g., a whole slide microscopic image, etc.) into a plurality of tiles. Thus, each of the resulting tiles may depict a portion of a tumor sample. In addition, the digital pathology platform may apply a first machine learning model, such as an artificial neural network (ANN), to each tile to determine a molecular subtype for a portion of the tumor sample depicted therein. The overall molecular subtype of the tumor sample may be determined based at least on the molecular subtype of each tile.

[0090] In some exemplary embodiments, the overall molecular subtype of the tumor sample depicted in the image can be determined based on the amount (such as a relative proportion) of each molecular subtype present in the tumor sample. Alternatively and / or additionally, the digital pathology platform can apply a second machine learning model to determine the overall molecular subtype of the tumor sample based at least on the molecular subtype of each tile. For example, the second machine learning model can be a multi-instance learning (MIL) model trained to determine the overall molecular subtype by determining the representative encoding of the tiles included in the image. In some cases, the second machine learning model may include an attention mechanism that is configured to assign to each tile an attention score representing how the molecular subtype of each tile will be related to the overall molecular subtype of the image of the tumor sample. Therefore, if the first molecular subtype of the first tile is more related to the overall molecular subtype of the image than the second molecular subtype of the second tile, the first tile with the first molecular subtype can be assigned a higher attention score than the second tile with the second molecular subtype.

[0091] In some exemplary embodiments, the digital pathology platform may generate one or more visual representations of at least a portion of the results of the image-based molecular subtype classification performed on the tumor sample. For example, in some cases, the digital pathology platform may generate a visual representation depicting the spatial distribution of different subtypes present within the tumor sample. Alternatively and / or additionally, the digital pathology platform may generate a visual representation in which tiles of the same molecular subtype are aligned adjacent to other tiles of the same molecular subtype from the same tumor sample and / or different tumor samples.

[0092] In some exemplary embodiments, the digital pathology platform can generate a visual representation of one or more similar tile subgroups present within an image depicting a tumor sample. In some cases, one or more similar tile subgroups can be identified by applying a clustering analysis technique to the pixel-by-pixel representation of each tile, such as k-means clustering, mean shift clustering, density-based spatial clustering of applications with noise (DBSCAN), expectation maximization (EM) clustering using a Gaussian mixture model (GMM), agglomerative hierarchical clustering, etc. Alternatively and / or additionally, one or more similar tile subgroups can be identified by applying a dimensionality reduction technique, such as principal component analysis (PCA), uniform manifold approximation and projection (UMAP), T-distributed stochastic neighbor embedding (t-SNE), etc. The resulting dimensionality reduction representation of the tile can correspond to a projection of the m-dimensional pixel-by-pixel representation of each tile onto a low n-dimensional subspace (where n<<m).

[0093] Therefore, the digital pathology platform can generate a visual representation in which the distribution of tiles provides a visual indication of similar and dissimilar tiles present within the image. In some cases, the visual representation of the reduced-dimensional representation of the tiles may include visual indicators (e.g., symbols of different colors, shapes, sizes, etc.) to enable further visual distinction between tiles of different molecular phenotypes. In doing so, the visual representation of the reduced-dimensional representation of the tiles can provide a visual indication of the overlap between different molecular subtypes present within the image of the tumor sample. Alternatively and / or additionally, the visual representation can depict the spatial distribution of similar tiles within the tumor sample. Such a visual representation can include a visual indicator (e.g., symbols of different colors, shapes, sizes, etc.) within the image of the tumor sample, which provides a visual distinction between tiles from different clusters of similar tiles.

[0094] Figure 1 A system diagram illustrating an example of a digital pathology system 100 according to some exemplary embodiments is depicted. Figure 1 , the digital pathology system 100 may include a digital pathology platform 110, an imaging system 120, and a client device 130. Figure 1 As shown, the digital pathology platform 110, the imaging system 120, and the client device 130 may be communicatively coupled via a network 140. The network 140 may be a wired network and / or a wireless network, including, for example, a local area network (LAN), a virtual local area network (VLAN), a wide area network (WAN), a public land mobile network (PLMN), the Internet, etc. The imaging system 120 may include one or more imaging devices, including, for example, a microscope, a digital camera, a whole slide scanner, a robotic microscope, etc. The client device 130 may be a processor-based device, including, for example, a workstation, a desktop computer, a laptop computer, a smart phone, a tablet computer, a wearable device, etc.

[0095] Reference again Figure 1 , the digital pathology platform 110 may include an analysis engine 115 configured to determine one or more molecular subtypes associated with the tumor sample based at least on one or more images of the tumor sample. The one or more images of the tumor sample may be, for example, a whole slide image (WSI) received from the imaging system 120. In some cases, the tumor sample may be a hepatocellular carcinoma (HCC) tumor sample. Figure 2As shown, hepatocellular carcinoma (HCC) is associated with cholangioid subtypes, progenitor cell-like subtypes, and hepatocyte-like subtypes, which are identified based on transcriptome data of various hepatocellular carcinoma tumor samples (e.g., non-negative matrix decomposition (NFM) or other clustering analysis of transcriptome data). Each molecular subtype of hepatocellular carcinoma can be associated with a different liver epithelial cell lineage. For example, cholangioid subtypes, progenitor cell-like subtypes, and hepatocyte-like subtypes correspond to cholangiocytes, bipotent progenitor cells, and hepatocytes, respectively. In addition, each molecular subtype of hepatocellular carcinoma can present a unique combination of tumor cell-intrinsic features and tumor microenvironment (TME) features. Therefore, in hepatocellular carcinoma (HCC) and other cancers, the molecular subtypes present in tumors can be used as key biomarkers for predicting patient response to treatment and survival.

[0096] Thus, the analysis engine 115 can apply one or more machine learning models to determine one or more molecular subtypes associated with a tumor sample based at least on the image of the tumor sample. In some exemplary embodiments, an overall molecular subtype for a tumor sample depicted in an image can be determined based on the molecular subtypes of individual tiles within the image. For example, FIG. 3A to FIG. 3B As shown, the analysis engine 115 can determine one or more tiles 305 within the image 300 of the tumor sample, including, for example, a first tile 305a, a second tile 305b, etc. Each tile 305 may include a portion of the tumor sample depicted in the image. For example, the image 300 of the tumor sample may depict cells present in the tumor sample, in which case each tile 305 may include a portion of the cells present in the tumor sample. In addition, in some cases, the analysis engine 115 may exclude tiles that do not depict cells present in the tumor sample above a threshold amount from further image-based molecular subtype classification. Examples of tiles excluded from further image-based molecular subtype classification may include tiles of backgrounds of images having a ratio above a threshold, tiles having an average color channel variance below a threshold (e.g., gray tiles), etc.

[0097] Reference now Figure 4 In some exemplary embodiments, the analysis engine 115 may apply a machine learning model 400 to each tile 305 within the image 300 of the tumor sample, the machine learning model being trained to determine a molecular subtype for each tile 305. Figure 4 In the example shown, the machine learning model 400 is an artificial neural network (ANN) having one or more of a convolutional layer, a maximum pooling layer, and a fully connected layer. The machine learning model 400 can be trained based on training data that includes images of tumor samples (or image patches depicting portions of tumor samples) that have been annotated with true molecular subtype labels. Figure 4As shown, the fully connected layer of the machine learning model 400 can generate an output indicating the molecular subtypes present within the tile 305 based at least on the morphological features of the tile 305 extracted by the convolutional layer and the maximum pooling layer.

[0098] In some exemplary embodiments, an overall molecular subtype for the tumor sample depicted in the image 300 may be determined based on the molecular subtypes of the individual tiles 305 within the image 300. For example, in some cases, the analysis engine 115 may determine the overall subtype for the tumor sample based on the amount of each molecular subtype present within the tumor sample. Alternatively and / or additionally, the analysis engine 115 may apply a second machine learning model to determine the overall molecular subtype of the tumor sample depicted in the image 300 based at least on the molecular subtype of each tile 305. For example, the second machine learning model may be a multiple instance learning (MIL) model trained to determine the overall molecular subtype by determining a representative encoding of the tiles 305 included in the image 300. In some cases, the second machine learning model may include an attention mechanism that is configured to assign to each tile 305 an attention score that represents how the molecular subtype of each tile 305 will be related to the overall molecular subtype of the image 300 of the tumor sample. Thus, if the first molecular subtype of the first tile 305 is more relevant to the overall molecular subtype of the image 300 than the second molecular subtype of the second tile 305b , the first tile 305a may be assigned a higher attention score than the second tile 305b .

[0099] In some exemplary embodiments, analysis engine 115 may generate one or more visual representations of at least a portion of the results of image-based molecular subtype classification performed on a tumor sample, for display in user interface 135 at client device 130. Figure 5A In one example shown, analysis engine 115 can generate visual representation 500 in which tiles 305 from image 300 are arranged according to the molecular subtype represented by each of tiles 305. Figure 5A As shown, visual representation 500 may include a first tile grouping representing a first subtype A (e.g., a cholangioid subtype), a second tile grouping representing a second subtype B (e.g., a hepatocyte-like subtype), and a third tile grouping representing a third subtype C (e.g., a progenitor-like subtype).

[0100] Figure 5B Another example of a visual representation 550 is depicted, which may be a distribution map 505 depicting the spatial distribution of patches of different molecular subtypes within the tumor sample shown in image 300. Figure 5BIn the example of visual representation 550 shown in , each tile 305 within image 300 can be represented using a visual indicator corresponding to the molecular subtype of the tile. Thus, symbols of different shapes, sizes, and / or colors can be used to enable visual distinction between tiles of, for example, cholangioid subtype, progenitor-like subtype, and hepatocyte-like subtype.

[0101] FIG. 6A to FIG. 6B An example of a visual representation is depicted in which a distribution map 505 of tiles of different molecular subtypes within a tumor sample shown in image 300 is juxtaposed adjacent to image 300 and / or a portion of image 300. For example, Fig. 6A An example of a visual representation 600 is depicted in which a distribution map 500 of tiles of different molecular subtypes within a tumor sample shown in image 300 is superimposed on a portion of image 300. Alternatively and / or additionally, Figure 6B An example of a visual representation 650 including a visual indicator (eg, a bounding box) configured to identify one or more corresponding portions 605 of a tumor sample within the image 300 and the distribution map 505 is depicted.

[0102] In some exemplary embodiments, the analysis engine 115 may also generate a visual representation of one or more similar tile subgroups present within the image 300 of, for example, a tumor sample for display in the user interface 135 at the client device 130. For example, each tile 305 in the image 300 may be associated with an x ​​amount of pixels across one or more color channels (e.g., a single channel of a grayscale image, and three channels of a color image). Thus, in some cases, each tile 305 in the image 300 may be encoded as a vector of m values, each value corresponding to an intensity value of a corresponding pixel in the image 300. In the case where the image 300 is a color image, the vector encoding each tile 305 may include a separate intensity value for each color channel for each pixel in the image 300 (e.g., m=3x). One or more similar tile subgroups in the image 300 of a tumor sample may be identified based on a pixel-by-pixel representation of each tile 305 included in the image 300.

[0103] In some exemplary embodiments, the analysis engine 115 may identify one or more similar subgroups of tiles in the image 300 by applying a dimensionality reduction technique such as principal component analysis (PCA), uniform manifold approximation and projection (UMAP), T-distributed stochastic neighbor embedding (t-SNE), etc. to the pixel-by-pixel representation of each tile 305 in the image 300. The resulting dimensionality reduction representation of the tiles 305 in the image 300 may correspond to a projection of the m-dimensional pixel-by-pixel representation of each tile 305 onto a low n-dimensional subspace (where n<<m).

[0104] Figure 7 An example of a visual representation 700 is depicted that includes a reduced dimensionality representation 705 (e.g., a uniform manifold approximation and projection (UMAP)) of a tile 305 included in an image 300. Figure 7 As shown, the reduced dimensionality representation 705 can depict the distribution of the tiles 305 across the two-dimensional subspace. Figure 7 It is shown that each tile 305 in the reduced dimensionality representation 705 can be represented using a different visual indicator corresponding to the molecular subtype of the tile. Thus, symbols of different shapes, sizes, and / or colors can be used to enable visual distinction between tiles of, for example, the cholangioid subtype, the progenitor-like subtype, and the hepatocyte-like subtype.

[0105] In some exemplary embodiments, one or more similar tile subgroups in the image 300 can be identified by applying a cluster analysis technique to the pixel-by-pixel representation of each tile 305 in the image 300, such as k-means clustering, mean shift clustering, density-based spatial clustering of applications with noise (DBSCAN), expectation maximization (EM) clustering using a Gaussian mixture model (GMM), agglomerative hierarchical clustering, etc. In doing so, the analysis engine 115 can identify the amount of clusters that maximize the intra-cluster correlation between members of each cluster. In addition, the analysis engine 115 can identify the amount of clusters associated with the minimum Bayesian information criterion, which means that the distribution of tiles 305 between different clusters accurately reflects the distribution of tiles 305. Fig. 8A One example of a visual representation 800 is depicted, which may be a distribution graph 805 that depicts the spatial distribution of tiles assigned to different clusters of similar tiles along a low n-dimensional subspace occupied by the reduced-dimensional representation of the tiles 305 . Fig. 8A The example of the visual representation 800 shown in includes twelve different clusters (e.g., Gaussian mixture model (GMM) clusters), where the members of each cluster are represented using different visual indicators (e.g., symbols of different colors, sizes, and / or shapes) to enable visual distinctions therebetween. For example, the twelve different clusters depicted in the visual representation 800 can be identified based on a low-dimensional representation of the corresponding tiles. The mixture model (e.g., Gaussian mixture model (GMM)) can represent the probability distribution of each similar tile grouping or subgroup across the total cluster as a whole.

[0106] Figure 8B Another example of a visual representation 810 is depicted, which may include a distribution map 815 showing the spatial distribution of tiles of different clusters of similar tiles on the image 300 of the tumor sample. Figure 8BIn the example shown, each tile 305 in the image 300 can be represented using a visual indicator corresponding to one of twelve different clusters of similar tiles (e.g., Gaussian mixture model (GMM) clusters) associated with the tile. Thus, symbols of different shapes, sizes, and / or colors can be used to enable visual distinction between tiles 305 that can be represented using a visual indicator corresponding to a cluster of tiles from each cluster.

[0107] Figure 8C Another example of a visual representation 820 is depicted in which tiles 305 in image 300 are arranged according to their membership in clusters of similar tiles. Figure 8C In the example of visual representation 820 shown in , each row of the grid may be occupied by tiles belonging to a separate cluster of similar tiles. Thus, visual representation 820 may provide a visual juxtaposition of similar tiles and dissimilar tiles within image 300.

[0108] Fig.8D Another example of a visual representation 830 depicting the distribution of different molecular subtypes across clusters of similar tiles is depicted in accordance with some exemplary embodiments. Fig.8D In the example shown in , the visual representation 830 can provide textual and graphical indications of the frequency of each molecular subtype (e.g., cholangioid subtype, progenitor-like subtype, and hepatocyte-like subtype) within each similar tile cluster (e.g., Gaussian mixture model (GMM) cluster). For example, in addition to numerical values ​​corresponding to the amount of tiles representing each molecular subtype within each cluster, each numerical value can be associated with a corresponding color to provide a heat map display of the frequency of each molecular subtype across the tile cluster.

[0109] Fig. 8E Another example of a visual representation 840 is depicted that depicts a composite of various images of a tumor sample in terms of the amount of constituent tiles from each tile cluster, in accordance with some exemplary embodiments. Fig. 8E , the visual representation 840 may include a bar graph 845, wherein each bar 850 in the bar graph 845 corresponds to an image of a tumor sample, such as, for example, image 300. Fig. 8E As shown, each bar 850 in bar graph 845 may include separate portions, each portion corresponding to a single cluster of similar tiles. Thus, each portion of a bar 850 in bar graph 845 may have a length representing the amount of tiles belonging to the corresponding cluster of similar tiles.

[0110] Fig. 9A flow chart illustrating an example of a process 900 for image-based molecular subtype classification according to some exemplary embodiments is depicted. For example, in some exemplary embodiments, the analysis engine 115 at the digital pathology platform 110 may perform the process 900 to determine one or more molecular subtypes present in the tumor sample based at least on an image of the tumor sample received from the imaging system 120. In some cases, the analysis engine 115 may further perform the process 900 to determine a treatment for a patient associated with the tumor sample based at least on the molecular subtypes present in the tumor sample.

[0111] At 902, the analysis engine 115 can determine a plurality of tiles within an image of a biological sample. FIG. 3A to FIG. 3B As shown, the analysis engine 115 can determine one or more tiles 305 within the image 300 of the tumor sample, including, for example, a first tile 305a, a second tile 305b, etc. The image 300 can depict cells of the tumor sample, in which case each tile 305 can depict at least a portion of the cells included in the tumor sample. In addition, in some cases, the analysis engine 115 can exclude tiles that do not depict cells present in a tumor sample above a threshold amount from further image-based molecular subtype classification. For example, tiles (e.g., gray tiles) with backgrounds of images above a threshold ratio and / or average color channel variances below a threshold can be excluded from further image-based molecular subtype classification.

[0112] At 904, the analysis engine 115 may apply a machine learning model to determine a molecular subtype for a portion of the biological sample depicted in each of the plurality of tiles. In some exemplary embodiments, the analysis engine 115 may apply a machine learning model, such as Figure 4 , to determine a molecular subtype for each tile 305 in image 300. As noted, in some cases, machine learning model 400 can be an artificial neural network (or another type of machine learning model) trained to recognize morphological patterns associated with each molecular subtype.

[0113] At 906, the analysis engine 115 may determine an overall molecular subtype for the biological sample based at least on the molecular subtype of each of the plurality of tiles. In some exemplary embodiments, the analysis engine 115 may determine an overall molecular subtype of the tumor sample depicted in the image 300 based at least on the amount of each molecular subtype present in the tumor sample. Alternatively and / or additionally, the analysis engine 115 may apply another machine learning model to determine an overall molecular subtype of the tumor sample depicted in the image 300 based at least on the molecular subtype of each tile 305. The another machine learning model may be a multiple instance learning (MIL) model trained to determine an overall molecular subtype by determining a representative encoding of the tiles 305 included in the image 300.

[0114] In some cases, the other machine learning model may include an attention mechanism configured to assign an attention score to each tile 305 that represents how the molecular subtype of each tile 305 will relate to the overall molecular subtype of the image 300 of the tumor sample. Thus, if a first molecular subtype of the first tile 305 is more relevant to the overall molecular subtype of the image 300 than a second molecular subtype of the second tile 305b, the first tile 305a may be assigned a higher attention score than the second tile 305b.

[0115] It should be appreciated that the aforementioned machine learning-enabled techniques for image-based molecular subtyping classification can improve the accuracy of image-based molecular subtyping classification by at least minimizing the variability of results associated with traditional digital pathology methods. To further illustrate, Fig.10 Depicted is a graph 1000 showing a receiver operating characteristic (ROC) curve representing the performance of the machine learning enabled image-based molecular subtype classification technique described herein when applied to a hepatocellular carcinoma (HCC) tumor sample. As shown in Figure 1000, the machine learning-based method achieved an area under the curve (AUC) of 0.698 for individual tile molecular subtype classification and an area under the curve (AUC) of 0.733 for overall image molecular subtype classification. These high area under the curve (AUC) values ​​indicate that the machine learning-based method described herein can partition between different hepatocellular carcinoma (HCC) molecular subtypes with high accuracy.

[0116] At 908, the analysis engine 115 may generate one or more visual representations of one or more molecular subtypes associated with the biological sample. In some exemplary embodiments, the analysis engine 115 may generate a visual representation (e.g., Figure 5BAlternatively and / or additionally, the analysis engine 115 may generate a visual representation in which the tiles 305 of the image 300 are organized according to their corresponding molecular subtypes (eg, Figure 5A In some cases, analysis engine 115 may generate visual representations depicting various relationships between similar tile subgroups within image 300 (e.g., Figure 7 and FIG. 8A to FIG. 8E ). For example, the analysis engine 115 can identify one or more similar tile subgroups by applying dimensionality reduction techniques, cluster analysis techniques, etc. In addition, the analysis engine 115 can generate one or more corresponding visual representations depicting the distribution of molecular subtypes across similar tile subgroups, the distribution of similar tiles across tumor samples in the image 300, etc.

[0117] At 910, the analysis engine 115 can determine treatment for a patient associated with the biological sample based at least on the overall molecular subtype for the biological sample. As noted, molecular subtypes of certain cancers, including hepatocellular carcinoma (HCC), can serve as key biomarkers for predicting patient response to treatment and survival. Thus, in the case where the tumor sample is a hepatocellular carcinoma (HCC) tumor sample, for example, the analysis engine 115 can determine whether the hepatocellular carcinoma (HCC) tumor sample is associated with a cholangioid subtype, a progenitor cell-like subtype, or a hepatocellular-like subtype. In addition, the molecular subtype of the hepatocellular carcinoma tumor sample can be used to determine whether treatment for a patient associated with the hepatocellular tumor sample should include a combination immunotherapy, such as atezolizumab (anti-PD-L1) plus bevacizumab (anti-VEGF) combination therapy, and additional therapies that overcome subtype-specific resistance to certain therapies (e.g., GPC3 / CD3 bispecific antibodies to overcome resistance to combination immunotherapy associated with progenitor cell-like subtypes).

[0118] Fig.11 A block diagram illustrating an example of a computing system 1100 is depicted in accordance with some example embodiments. Figure 1 and Fig.11 , the computing system 1100 may be used to implement the digital pathology platform 110 , the client device 130 , and / or any components thereof.

[0119] like Fig.11As shown, the computing system 1100 may include a processor 1110, a memory 1120, a storage device 1130, and an input / output device 1140. The processor 1110, the memory 1120, the storage device 1130, and the input / output device 1140 may be interconnected via a system bus 1150. The processor 1110 is capable of processing instructions for execution within the computing system 1100. Such executed instructions may implement, for example, one or more components of the digital pathology platform 110, the client device 130, etc. In some exemplary embodiments, the processor 1110 may be a single-threaded processor. Alternatively, the processor 1110 may be a multi-threaded processor. The processor 1110 is capable of processing instructions stored on the memory 1120 and / or the storage device 1130 to display graphical information for a user interface provided via the input / output device 1140.

[0120] The memory 1120 is a computer-readable medium, such as a volatile or non-volatile computer-readable medium, that stores information within the computing system 1100. For example, the memory 1120 may store a data structure representing a configuration object database. The storage device 1130 is capable of providing persistent storage for the computing system 1100. The storage device 1130 may be a floppy disk device, a hard disk device, an optical disk device, or a magnetic tape device or other suitable persistent storage member. The input / output device 1140 provides input / output operations for the computing system 1100. In some exemplary embodiments, the input / output device 1140 includes a keyboard and / or a pointing device. In various specific implementations, the input / output device 1140 includes a display unit for displaying a graphical user interface.

[0121] According to some exemplary embodiments, the input / output device 1140 may provide input / output operations for a network device. For example, the input / output device 1140 may include an Ethernet port or other networking port to communicate with one or more wired and / or wireless networks (e.g., a local area network (LAN), a wide area network (WAN), the Internet).

[0122] In some exemplary embodiments, the computing system 1100 can be used to execute various interactive computer software applications that can be used to organize, analyze and / or store data in various formats. Alternatively, the computing system 1100 can be used to execute any type of software application. These applications can be used to perform various functions, such as planning functions (e.g., generating, managing, editing electronic spreadsheet documents, word processing documents and / or any other objects, etc.), computing functions, communication functions, etc. The application can include various additional functions or can be an independent computing product and / or function. After activation within the application, the function can be used to generate a user interface provided via the input / output device 1140. The user interface can be generated by the computing system 1100 and presented to the user (e.g., on a computer screen monitor, etc.).

[0123] One or more aspects or features of the subject matter described herein may be implemented in digital electronic circuits, integrated circuits, specially designed ASICs, field programmable gate arrays (FPGAs) computer hardware, firmware, software, and / or combinations thereof. These various aspects or features may include a specific implementation in one or more computer programs, which are executable and / or interpretable on a programmable system, which includes at least one programmable processor (which may be dedicated or general, coupled to receive data and instructions from it and send data and instructions to it), a storage system, at least one input device, and at least one output device. A programmable system or computing system may include a client and a server. Typically, the client and the server are remotely arranged from each other and generally interact through a communication network. The relationship between the client and the server is generated by means of computer programs running on respective computers and the client-server relationship between each other.

[0124] These computer programs may also be referred to as programs, software, software applications, applications, components or codes, including machine instructions for programmable processors, and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the term "machine-readable medium" refers to any computer program product, device and / or device (such as, for example, a disk, an optical disk, a memory and a programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor. A machine-readable medium may store such machine instructions non-temporarily (such as, for example, a non-temporary solid-state memory or a magnetic hard drive or any equivalent storage medium). A machine-readable medium may store such machine instructions in a temporary manner (such as, for example, a processor cache or other random access memory associated with one or more physical processor cores) alternatively or additionally.

[0125] To provide interaction with a user, one or more aspects or features of the subject matter described herein may be implemented on a computer having a display device (such as, for example, a cathode ray tube (CRT) or a liquid crystal display (LCD) or a light emitting diode (LED) monitor for displaying information to a user) and a keyboard and pointing device (such as, for example, a mouse or trackball, through which a user can provide input to the computer). Other types of devices may also be used to provide interaction with a user. For example, the feedback provided to the user may be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or tactile feedback; input from the user may be received in any form, including sound, voice, or tactile input. Other possible input devices include touch screens or other touch-sensitive devices, such as single-point or multi-point resistive or capacitive tracking pads, voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices and associated interpretation software, and the like.

[0126] Example

[0127] The examples provided are:

[0128] 1. A system comprising:

[0129] at least one data processor; and

[0130] At least one memory storing instructions that, when executed by at least one data processor, result in operations including:

[0131] A plurality of tiles are defined within the image of the biological sample, each tile in the plurality of tiles describing

[0132] Draw a portion of a biological sample;

[0133] Applying a first machine learning model to determine a method for describing a

[0134] molecular subtypes of a portion of a biological sample; and

[0135] An overall molecular subtype for the biological sample is determined based at least on the molecular subtype of each tile in the plurality of tiles.

[0136] 2. A system according to embodiment 1, wherein the overall molecular subtype of the biological sample is determined by applying a second machine learning model.

[0137] 3. A system according to embodiment 2, wherein the second machine learning model is trained to determine the overall molecular subtype by at least determining a representative encoding of multiple tiles.

[0138] 4. A system according to embodiment 3, wherein the second machine learning model is further trained to: assign a higher attention score to a first tile among the multiple tiles compared to a second tile among the multiple tiles, while determining representative encodings of the multiple tiles.

[0139] 5. The system of embodiment 4, wherein a higher attention score indicates that a first molecular subtype of a first tile contributes more to the representative encoding of the plurality of tiles than a second molecular subtype of a second tile.

[0140] 6. The system of any one of embodiments 4 to 5, wherein a higher attention score indicates that the first molecular subtype of the first tile is more relevant to the overall molecular subtype of the biological sample than the second molecular subtype of the second tile.

[0141] 7. A system according to any one of embodiments 2 to 6, wherein the second machine learning model includes a multiple instance learning (MIL) model.

[0142] 8. A system according to any one of embodiments 2 to 7, wherein the second machine learning model includes an attention mechanism.

[0143] 9. The system of any one of embodiments 1 to 8, wherein the operations further comprise:

[0144] A first visual representation of the reduced-dimensional representation of the plurality of tiles is generated.

[0145] 10. The system of embodiment 9, wherein the first visual representation comprises one or more visual indicators configured to provide visual distinction between tiles of different subtypes.

[0146] 11. A system according to any one of embodiments 9 to 10, wherein the first visual representation is generated by applying a dimensionality reduction technique to at least a pixel-by-pixel representation of each tile in the plurality of tiles.

[0147] 12. A system according to embodiment 11, wherein the dimensionality reduction technique includes one or more of principal component analysis (PCA), uniform manifold approximation and projection (UMAP), and T-distributed stochastic neighbor embedding (t-SNE).

[0148] 13. A system according to any one of embodiments 9 to 12, wherein the first visual representation is further generated to include one or more visual indications configured to provide visual distinction between one or more similar tile clusters within the plurality of tiles.

[0149] 14. The system of embodiment 13, wherein the operations further comprise:

[0150] A second visual representation is generated that depicts the plurality of tiles organized according to one or more similar tile clusters.

[0151] 15. The system of any one of embodiments 13 to 14, wherein the operations further comprise:

[0152] A second visual representation is generated depicting the spatial distribution of the one or more clusters of similar tiles within the biological sample.

[0153] 16. The system of any one of embodiments 13 to 15, wherein one or more clusters of similar tiles are identified by applying a cluster analysis technique.

[0154] 17. A system according to embodiment 16, wherein the clustering analysis technique includes one or more of k-means clustering, mean shift clustering, density-based spatial clustering of applications with noise (DBSCAN), expectation maximization (EM) clustering using Gaussian mixture model (GMM), and agglomerative hierarchical clustering.

[0155] 18. A system according to any one of embodiments 1 to 17, wherein the overall molecular subtype of the biological sample is determined based at least on the amount of each molecular subtype present in multiple cells.

[0156] 19. The system of any one of embodiments 1 to 18, wherein the operations further comprise:

[0157] A visual representation depicting a spatial distribution of one or more molecular subtypes within the biological sample is generated based at least on the molecular subtypes of each tile in the plurality of tiles.

[0158] 20. The system of any one of embodiments 1 to 19, wherein the operations further comprise:

[0159] A visual representation is generated depicting a first tile having a first subtype and a second tile having the first subtype from among a plurality of tiles from the same biological sample or different biological samples.

[0160] 21. The system of embodiment 20, wherein the visual representation is further generated to depict a third tile having the second subtype and a fourth tile having the second subtype from a plurality of tiles from the same biological sample or different biological samples.

[0161] 22. A system according to any one of embodiments 1 to 21, wherein the plurality of tiles exclude one or more tiles in the image having a background of the image above a threshold ratio or an average color channel variance below a threshold.

[0162] 23. A system according to any one of embodiments 1 to 22, wherein the first machine learning model is trained to determine a molecular subtype associated with each of a plurality of tiles based on morphological patterns present within a portion of a biological sample depicted in each tile.

[0163] 24. A system according to any one of embodiments 1 to 23, wherein the first machine learning model includes an artificial neural network (ANN).

[0164] 25. A system according to any one of embodiments 1 to 24, wherein the biological sample includes a hepatocellular carcinoma (HCC) tissue sample, wherein each of the multiple tiles is assigned a molecular subtype, the molecular subtype including one of a cholangioid subtype, a hepatocyte-like subtype, or a progenitor-like subtype, and wherein the overall molecular subtype of the multiple cells depicted in the image of the biological sample includes one of a cholangioid subtype, a hepatocyte-like subtype, or a progenitor-like subtype.

[0165] 26. The system of any one of embodiments 1 to 25, wherein the operations further comprise:

[0166] A plurality of molecular subtypes are identified based at least on transcriptome data associated with the plurality of tumor tissue samples.

[0167] 27. A system according to embodiment 26, wherein the multiple tumor tissue samples include multiple hepatocellular carcinoma (HCC) tumor tissue samples, and wherein the multiple molecular subtypes include cholangiocyte-like subtype, hepatocyte-like subtype and progenitor-like subtype.

[0168] 28. A system according to any one of embodiments 26 to 27, wherein the first machine learning model is trained to assign a label corresponding to one of a plurality of molecular subtypes identified based on transcriptome data to each of the plurality of tiles.

[0169] 29. A system according to any one of embodiments 26 to 28, wherein the overall molecular subtypes of the plurality of cells depicted in the image of the biological sample include one of the plurality of molecular subtypes identified based on the transcriptome data.

[0170] 30. A system according to any one of embodiments 1 to 29, wherein the image depicts a plurality of cells contained in the biological sample, and wherein each of the plurality of tiles depicts a portion of the plurality of cells contained in the biological sample.

[0171] 31. A computer-implemented method comprising:

[0172] determining a plurality of tiles within the image of the biological sample, each tile of the plurality of tiles depicting a portion of the biological sample;

[0173] applying a first machine learning model to determine a molecular subtype for a portion of the biological sample depicted in each of the plurality of tiles; and

[0174] An overall molecular subtype for the biological sample is determined based at least on the molecular subtype of each tile in the plurality of tiles.

[0175] 32. A method according to embodiment 31, wherein the overall molecular subtype of the biological sample is determined by applying a second machine learning model.

[0176] 33. A method according to embodiment 32, wherein the second machine learning model is trained to determine the overall molecular subtype by at least determining a representative encoding of multiple tiles.

[0177] 34. A method according to embodiment 33, wherein the second machine learning model is further trained to: assign a higher attention score to a first tile among the multiple tiles compared to a second tile among the multiple tiles, while determining representative encodings of the multiple tiles.

[0178] 35. The method of embodiment 34, wherein a higher attention score indicates that a first molecular subtype of a first tile contributes more to the representative encoding of the plurality of tiles than a second molecular subtype of a second tile.

[0179] 36. The method of any one of embodiments 34 to 35, wherein a higher attention score indicates that a first molecular subtype of a first tile is more relevant to the overall molecular subtype of the biological sample than a second molecular subtype of a second tile.

[0180] 37. A method according to any one of embodiments 32 to 36, wherein the second machine learning model comprises a multiple instance learning (MIL) model.

[0181] 38. A method according to any one of embodiments 32 to 37, wherein the second machine learning model includes an attention mechanism.

[0182] 39. The method of any one of embodiments 31 to 38, further comprising:

[0183] A first visual representation of the reduced-dimensional representation of the plurality of tiles is generated.

[0184] 40. The method of embodiment 39, wherein the first visual representation includes one or more visual indicators configured to provide visual distinction between tiles of different subtypes.

[0185] 41. A method according to any one of embodiments 39 to 40, wherein the first visual representation is generated by applying a dimensionality reduction technique to at least a pixel-by-pixel representation of each tile in a plurality of tiles.

[0186] 42. A method according to embodiment 41, wherein the dimensionality reduction technique includes one or more of principal component analysis (PCA), uniform manifold approximation and projection (UMAP), and T-distributed stochastic neighbor embedding (t-SNE).

[0187] 43. A method according to any one of embodiments 39 to 42, wherein the first visual representation is further generated to include one or more visual indications, wherein the one or more visual indications are configured to provide visual distinction between one or more similar tile clusters within the plurality of tiles.

[0188] 44. The method of embodiment 43, further comprising:

[0189] A second visual representation is generated that depicts the plurality of tiles organized according to one or more similar tile clusters.

[0190] 45. The method of any one of embodiments 43-44, further comprising:

[0191] A second visual representation is generated depicting the spatial distribution of the one or more clusters of similar tiles within the biological sample.

[0192] 46. ​​The method of any one of embodiments 43 to 45, wherein one or more clusters of similar tiles are identified by applying a cluster analysis technique.

[0193] 47. A method according to embodiment 46, wherein the clustering analysis technique includes one or more of k-means clustering, mean shift clustering, density-based spatial clustering of applications with noise (DBSCAN), expectation maximization (EM) clustering using Gaussian mixture model (GMM), and agglomerative hierarchical clustering.

[0194] 48. A method according to any one of embodiments 31 to 47, wherein the overall molecular subtype of the biological sample is determined based at least on the amount of each molecular subtype present in multiple cells.

[0195] 49. The method of any one of embodiments 31 to 48, further comprising:

[0196] A visual representation depicting a spatial distribution of one or more molecular subtypes within the biological sample is generated based at least on the molecular subtypes of each tile in the plurality of tiles.

[0197] 50. The method of any one of embodiments 31 to 49, further comprising:

[0198] A visual representation is generated depicting a first tile having a first subtype and a second tile having the first subtype from among a plurality of tiles from the same biological sample or different biological samples.

[0199] 51. The method of embodiment 50, wherein the visual representation is further generated to depict a third tile having the second subtype and a fourth tile having the second subtype from a plurality of tiles from the same biological sample or different biological samples.

[0200] 52. A method according to any one of embodiments 31 to 51, wherein the plurality of tiles exclude one or more tiles in the image having a background of the image above a threshold ratio or an average color channel variance below a threshold.

[0201] 53. A method according to any one of embodiments 31 to 52, wherein a first machine learning model is trained to determine a molecular subtype associated with each of a plurality of tiles based on morphological patterns present within a portion of a biological sample depicted in each tile.

[0202] 54. A method according to any one of embodiments 31 to 53, wherein the first machine learning model comprises an artificial neural network (ANN).

[0203] 55. A method according to any one of embodiments 31 to 54, wherein the biological sample comprises a hepatocellular carcinoma (HCC) tissue sample, wherein each of the multiple tiles is assigned a molecular subtype, the molecular subtype comprising one of a cholangioid subtype, a hepatocyte-like subtype, or a progenitor-like subtype, and wherein the overall molecular subtype of the multiple cells depicted in the image of the biological sample comprises one of a cholangioid subtype, a hepatocyte-like subtype, or a progenitor-like subtype.

[0204] 56. The method of any one of embodiments 31 to 55, further comprising:

[0205] A plurality of molecular subtypes are identified based at least on transcriptome data associated with the plurality of tumor tissue samples.

[0206] 57. A method according to embodiment 56, wherein the multiple tumor tissue samples include multiple hepatocellular carcinoma (HCC) tumor tissue samples, and wherein the multiple molecular subtypes include cholangiocyte-like subtype, hepatocyte-like subtype and progenitor cell-like subtype.

[0207] 58. A method according to any one of embodiments 56 to 57, wherein the first machine learning model is trained to assign a label corresponding to one of a plurality of molecular subtypes identified based on transcriptome data to each of a plurality of tiles.

[0208] 59. A method according to any one of embodiments 56 to 58, wherein the overall molecular subtypes of multiple cells depicted in the image of the biological sample include one of the multiple molecular subtypes identified based on transcriptome data.

[0209] 60. A method according to any one of embodiments 31 to 59, wherein the image depicts a plurality of cells contained in the biological sample, and wherein each of the plurality of tiles depicts a portion of the plurality of cells contained in the biological sample.

[0210] 61. A non-transitory computer readable medium storing instructions that, when executed by at least one data processor, result in operations comprising:

[0211] determining a plurality of tiles within the image of the biological sample, each tile of the plurality of tiles depicting a portion of the biological sample;

[0212] applying a first machine learning model to determine a molecular subtype for a portion of the biological sample depicted in each of the plurality of tiles; and

[0213] An overall molecular subtype for the biological sample is determined based at least on the molecular subtype of each tile in the plurality of tiles.

[0214] In the above description and claims, phrases such as "at least one" or "one or more" may appear, followed by a list of combinations of elements or features. The term "and / or" may also appear in a list of two or more elements or features. Unless otherwise implicitly or explicitly contradicted by the context in which it is used, the phrase is intended to represent any element or feature listed alone, or any other described element or feature combined with any other described element or feature. For example, the phrase "at least one of A and B"; "one or more of A and B"; "A and / or B" are each intended to represent "single A, single B, or A and B together". A similar interpretation also applies to lists including three or more items. For example, the phrases "at least one of A, B, and C"; "one or more of A, B, and C" and "A, B, and / or C" are each intended to represent "single A, single B, single C, A and B together, A and C together, B and C together, or A and B and C together". The use of the term "based on" above and in the claims is intended to represent "based at least in part", so that undescribed features or elements are also permissible.

[0215] Depending on the desired configuration, the subject matter described herein may be embodied in systems, devices, methods and / or articles. The embodiments described in the foregoing description do not represent all embodiments consistent with the subject matter described herein. Instead, they are only some examples consistent with aspects related to the described subject matter. Although some variations have been described in detail above, other modifications or additions are possible. In particular, in addition to those features and / or variations described herein, other features and / or variations may also be provided. For example, the above-mentioned specific implementations may be directed to various combinations and sub-combinations of the disclosed features and / or to combinations and sub-combinations of several further features disclosed above. In addition, the logical flows depicted in the drawings and / or described herein do not necessarily require the specific order or sequential order shown to achieve the desired results. Other specific implementations may be within the scope of the following claims.

Claims

1. A system comprising: at least one data processor; as well as at least one memory storing instructions that, when executed by the at least one data processor, result in operations including: determining a plurality of tiles within the image of the biological sample, each tile of the plurality of tiles depicting a portion of the biological sample; applying a first machine learning model to determine a molecular subtype for the portion of the biological sample depicted in each tile of the plurality of tiles; as well as An overall molecular subtype for the biological sample is determined based at least on the molecular subtype of each tile in the plurality of tiles.

2. The system of claim 1, wherein the overall molecular subtype of the biological sample is determined by applying a second machine learning model.

3. The system of claim 2, wherein the second machine learning model is trained to determine the overall molecular subtype by at least determining a representative encoding of the plurality of tiles.

4. A system according to claim 3, wherein the second machine learning model is further trained to: assign a higher attention score to a first tile among the multiple tiles compared to a second tile among the multiple tiles, while determining the representative encoding of the multiple tiles. 5 . The system of claim 4 , wherein the higher attention score indicates that a first molecular subtype of the first tile contributes more to the representative encoding of the plurality of tiles than a second molecular subtype of the second tile.

6. The system of any one of claims 4 to 5, wherein the higher attention score indicates that the first molecular subtype of the first tile is more relevant to the overall molecular subtype of the biological sample than the second molecular subtype of the second tile.

7. A system according to any one of claims 2 to 6, wherein the second machine learning model comprises a multiple instance learning (MIL) model.

8. A system according to any one of claims 2 to 7, wherein the second machine learning model includes an attention mechanism.

9. The system according to any one of claims 1 to 8, wherein the operations further comprise: A first visual representation of the reduced-dimensional representation of the plurality of tiles is generated. 10 . The system of claim 9 , wherein the first visual representation comprises one or more visual indicators configured to provide visual distinction between tiles of different subtypes.

11. The system of any one of claims 9 to 10, wherein the first visual representation is generated by applying a dimensionality reduction technique to at least a pixel-by-pixel representation of each tile of the plurality of tiles.

12. The system of claim 11, wherein the dimensionality reduction technique comprises one or more of principal component analysis (PCA), uniform manifold approximation and projection (UMAP), and t-distributed stochastic neighbor embedding (t-SNE).

13. The system of any one of claims 9 to 12, wherein the first visual representation is further generated to include one or more visual indications configured to provide visual distinction between one or more clusters of similar tiles within the plurality of tiles.

14. The system of claim 13, wherein the operations further comprise: A second visual representation is generated that depicts the plurality of tiles organized according to the one or more similar tile clusters.

15. The system of any one of claims 13 to 14, wherein the operations further comprise: A second visual representation is generated depicting a spatial distribution of the one or more clusters of similar patches within the biological sample.

16. The system of any one of claims 13 to 15, wherein the one or more clusters of similar tiles are identified by applying a cluster analysis technique.

17. A system according to claim 16, wherein the clustering analysis technique includes one or more of k-means clustering, mean shift clustering, density-based spatial clustering of applications with noise (DBSCAN), expectation maximization (EM) clustering using Gaussian mixture model (GMM), and agglomerative hierarchical clustering.

18. The system of any one of claims 1 to 17, wherein the overall molecular subtype of the biological sample is determined based at least on the amount of each molecular subtype present in a plurality of cells.

19. The system of any one of claims 1 to 18, wherein the operations further comprise: A visual representation depicting a spatial distribution of one or more molecular subtypes within the biological sample is generated based at least on the molecular subtypes of each tile in the plurality of tiles.

20. The system of any one of claims 1 to 19, wherein the operations further comprise: A visual representation is generated that depicts a first tile having a first subtype and a second tile of the first subtype from the plurality of tiles from the same biological sample or different biological samples.

21. The system of claim 20, wherein the visual representation is further generated to depict a third tile having a second subtype and a fourth tile of the second subtype among the plurality of tiles from the same biological sample or the different biological sample.

22. The system of any one of claims 1 to 21, wherein the plurality of tiles exclude one or more tiles in the image having a background of the image above a threshold ratio or an average color channel variance below a threshold.

23. A system according to any one of claims 1 to 22, wherein the first machine learning model is trained to determine the molecular subtype associated with each of the multiple tiles based on morphological patterns present within the portion of the biological sample depicted in each tile.

24. The system of any one of claims 1 to 23, wherein the first machine learning model comprises an artificial neural network (ANN).

25. A system according to any one of claims 1 to 24, wherein the biological sample comprises a hepatocellular carcinoma (HCC) tissue sample, wherein each of the multiple tiles is assigned a molecular subtype, the molecular subtype comprising one of a cholangioid subtype, a hepatocyte-like subtype, or a progenitor-like subtype, and wherein the overall molecular subtype of the multiple cells depicted in the image of the biological sample comprises one of the cholangioid subtype, the hepatocyte-like subtype, or the progenitor-like subtype.

26. The system of any one of claims 1 to 25, wherein the operations further comprise: A plurality of molecular subtypes are identified based at least on transcriptome data associated with the plurality of tumor tissue samples.

27. The system of claim 26, wherein the plurality of tumor tissue samples comprises a plurality of hepatocellular carcinoma (HCC) tumor tissue samples, and wherein the plurality of molecular subtypes comprises a cholangiogenic subtype, a hepatocellular subtype, and a progenitor-like subtype.

28. The system of any one of claims 26 to 27, wherein the first machine learning model is trained to assign to each of the plurality of tiles a label corresponding to one of the plurality of molecular subtypes identified based on the transcriptome data.

29. The system of any one of claims 26 to 28, wherein the overall molecular subtype of the plurality of cells depicted in the image of the biological sample comprises one of the plurality of molecular subtypes identified based on the transcriptome data.

30. The system of any one of claims 1 to 29, wherein the image depicts a plurality of cells comprised by the biological sample, and wherein each of the plurality of tiles depicts a portion of the plurality of cells comprised by the biological sample.

31. A computer-implemented method comprising: determining a plurality of tiles within the image of the biological sample, each tile of the plurality of tiles depicting a portion of the biological sample; applying a first machine learning model to determine a molecular subtype for the portion of the biological sample depicted in each tile of the plurality of tiles; as well as An overall molecular subtype for the biological sample is determined based at least on the molecular subtype of each tile in the plurality of tiles.

32. The method of claim 31 , wherein the overall molecular subtype of the biological sample is determined by applying a second machine learning model.

33. The method of claim 32, wherein the second machine learning model is trained to determine the overall molecular subtype by at least determining a representative encoding of the plurality of tiles.

34. A method according to claim 33, wherein the second machine learning model is further trained to: assign a higher attention score to a first tile among the multiple tiles compared to a second tile among the multiple tiles, while determining the representative encoding of the multiple tiles.

35. The method of claim 34, wherein the higher attention score indicates that a first molecular subtype of the first tile contributes more to the representative encoding of the plurality of tiles than a second molecular subtype of the second tile.

36. The method of any one of claims 34 to 35, wherein the higher attention score indicates that the first molecular subtype of the first tile is more relevant to the overall molecular subtype of the biological sample than the second molecular subtype of the second tile.

37. A method according to any one of claims 32 to 36, wherein the second machine learning model comprises a multiple instance learning (MIL) model.

38. A method according to any one of claims 32 to 37, wherein the second machine learning model includes an attention mechanism.

39. The method according to any one of claims 31 to 38, further comprising: A first visual representation of the reduced-dimensional representation of the plurality of tiles is generated.

40. The method of claim 39, wherein the first visual representation comprises one or more visual indicators configured to provide visual distinction between tiles of different subtypes.

41. A method according to any one of claims 39 to 40, wherein the first visual representation is generated by applying a dimensionality reduction technique to at least a pixel-by-pixel representation of each tile in the plurality of tiles.

42. The method of claim 41, wherein the dimensionality reduction technique comprises one or more of principal component analysis (PCA), uniform manifold approximation and projection (UMAP), and t-distributed stochastic neighbor embedding (t-SNE).

43. The method of any one of claims 39 to 42, wherein the first visual representation is further generated to include one or more visual indications configured to provide visual distinction between one or more clusters of similar tiles within the plurality of tiles.

44. The method of claim 43, further comprising: A second visual representation is generated that depicts the plurality of tiles organized according to the one or more similar tile clusters.

45. The method according to any one of claims 43 to 44, further comprising: A second visual representation is generated depicting a spatial distribution of the one or more clusters of similar patches within the biological sample.

46. ​​A method according to any one of claims 43 to 45, wherein the one or more clusters of similar tiles are identified by applying a cluster analysis technique.

47. A method according to claim 46, wherein the clustering analysis technique includes one or more of k-means clustering, mean shift clustering, density-based spatial clustering of applications with noise (DBSCAN), expectation maximization (EM) clustering using Gaussian mixture model (GMM), and agglomerative hierarchical clustering.

48. The method of any one of claims 31 to 47, wherein the overall molecular subtype of the biological sample is determined based at least on the amount of each molecular subtype present in a plurality of cells.

49. The method according to any one of claims 31 to 48, further comprising: A visual representation depicting a spatial distribution of one or more molecular subtypes within the biological sample is generated based at least on the molecular subtypes of each tile in the plurality of tiles.

50. The method according to any one of claims 31 to 49, further comprising: A visual representation is generated that depicts a first tile having a first subtype and a second tile of the first subtype from the plurality of tiles from the same biological sample or different biological samples.

51. The method of claim 50, wherein the visual representation is further generated to depict a third tile having a second subtype and a fourth tile having the second subtype among the plurality of tiles from the same biological sample or the different biological sample.

52. The method of any one of claims 31 to 51, wherein the plurality of tiles exclude one or more tiles in the image having a background of the image above a threshold ratio or an average color channel variance below a threshold.

53. A method according to any one of claims 31 to 52, wherein the first machine learning model is trained to determine the molecular subtype associated with each of the multiple tiles based on morphological patterns present within the portion of the biological sample depicted in each tile.

54. A method according to any one of claims 31 to 53, wherein the first machine learning model comprises an artificial neural network (ANN).

55. A method according to any one of claims 31 to 54, wherein the biological sample comprises a hepatocellular carcinoma (HCC) tissue sample, wherein each of the plurality of tiles is assigned a molecular subtype, the molecular subtype comprising one of a cholangioid subtype, a hepatocyte-like subtype, or a progenitor-like subtype, and wherein the overall molecular subtype of the plurality of cells depicted in the image of the biological sample comprises one of the cholangioid subtype, the hepatocyte-like subtype, or the progenitor-like subtype.

56. The method according to any one of claims 31 to 55, further comprising: A plurality of molecular subtypes are identified based at least on transcriptome data associated with the plurality of tumor tissue samples.

57. The method of claim 56, wherein the plurality of tumor tissue samples comprises a plurality of hepatocellular carcinoma (HCC) tumor tissue samples, and wherein the plurality of molecular subtypes comprises a cholangiogenic subtype, a hepatogenic subtype, and a progenitor-like subtype.

58. A method according to any one of claims 56 to 57, wherein the first machine learning model is trained to assign a label to each of the plurality of tiles corresponding to one of the plurality of molecular subtypes identified based on the transcriptome data.

59. The method of any one of claims 56 to 58, wherein the overall molecular subtype of the plurality of cells depicted in the image of the biological sample comprises one of the plurality of molecular subtypes identified based on the transcriptome data.

60. The method of any one of claims 31 to 59, wherein the image depicts a plurality of cells comprised by the biological sample, and wherein each of the plurality of tiles depicts a portion of the plurality of cells comprised by the biological sample.

61. A non-transitory computer readable medium storing instructions that, when executed by at least one data processor, result in operations comprising: determining a plurality of tiles within the image of the biological sample, each tile of the plurality of tiles depicting a portion of the biological sample; applying a first machine learning model to determine a molecular subtype for the portion of the biological sample depicted in each tile of the plurality of tiles; as well as An overall molecular subtype for the biological sample is determined based at least on the molecular subtype of each tile in the plurality of tiles.