Machine learning histological analysis for the identification of molecular features

Abstract

The method may include determining a first plurality of tiles having a first tile size and a second plurality of tiles having a second tile size within an image of the biological sample. A feature extraction model may be applied to extract features from tiles of different sizes. Connected feature sets may be formed, each including a first feature of a first tile from the first plurality of tiles, a second feature of a second tile from the second plurality of tiles, and a third feature of a third tile from the second plurality of tiles. Molecular features present in the biological sample may be determined based on an attention-weighted positional embedding of the connected feature sets and a joint representation of features across clusters of spatially adjacent tiles within the image. Related systems and computer program products are also provided.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 387,462, filed December 14, 2022, entitled "Machine Learning Histological Analysis for Identification of Molecular Signatures," the entire contents of which are incorporated herein by reference.

[0002] The subject matter described herein relates generally to digital and computational pathology, and more specifically to deep learning approaches for identifying molecular features in histological images. [Background technology]

[0003] A cellular phenotype may refer to a unique combination of morphological and functional characteristics resulting from various cellular processes, including, for example, gene expression, protein expression, and / or the like. In some cases, the complex interplay between a cell's genome, epigenome, and local environment can result in a combination of observable characteristics collectively known as the cellular phenotype. Cellular phenotypes, including those of tumor cells, are typically attributed to genomic instability, but in recent years, the influence of epigenetics and the microenvironment has gained attention. Such non-genetic factors may further increase the inherent diversity and plasticity of tumor cells. At the tumor level, non-genetic factors may contribute to greater phenotypic heterogeneity, allowing tumor cells to evade immune responses and resist drug intervention. Summary of the Invention

[0004] Systems, methods, and articles, including computer program products, are provided for machine-learning-enabled identification of molecular features in histological images. In one aspect, a system is provided that includes at least one processor and at least one memory. The at least one memory may include program code that, when executed by the at least one processor, provides operations. The operations may include determining a first plurality of tiles having a first tile size within an image of the biological sample; determining a second plurality of tiles having a second tile size within the image of the biological sample; applying a feature extraction model to extract a first plurality of features from the first plurality of tiles of the first size; applying the feature extraction model to extract a second plurality of features from the second plurality of tiles of the second size; and determining one or more molecular features present in the biological sample depicted in the image based at least on the first plurality of features and the second plurality of features.

[0005] In another aspect, a method for machine-learning-enabled identification of molecular features in a histological image is provided. The method may include determining a first plurality of tiles having a first tile size within an image of the biological sample, determining a second plurality of tiles having a second tile size within the image of the biological sample, applying a feature extraction model to extract a first plurality of features from the first plurality of tiles of the first size, applying the feature extraction model to extract a second plurality of features from the second plurality of tiles of the second size, and determining one or more molecular features present in the biological sample depicted in the image based at least on the first plurality of features and the second plurality of features.

[0006] In another aspect, a computer program product for machine-learning-enabled identification of molecular features in histological images is provided. The computer program product may include a non-transitory computer-readable medium storing instructions that, when executed by at least one data processor, cause operations to occur. The operations may include determining a first plurality of tiles having a first tile size within an image of the biological sample, determining a second plurality of tiles having a second tile size within the image of the biological sample, applying a feature extraction model to extract a first plurality of features from the first plurality of tiles of the first size, applying the feature extraction model to extract a second plurality of features from the second plurality of tiles of the second size, and determining one or more molecular features present in the biological sample depicted in the image based at least on the first plurality of features and the second plurality of features.

[0007] Implementations of the present subject matter may include, but are not limited to, methods according to the description provided herein, as well as articles comprising tangibly embodied machine-readable media operable to cause one or more machines (e.g., computers, etc.) to perform operations that implement one or more of the described features. Similarly, computer systems are described that may include one or more processors and one or more memories coupled to the one or more processors. The memory, which may include a non-transitory computer-readable or machine-readable storage medium, may include, encode, or store one or more programs that cause the one or more processors to perform one or more of the operations described herein. Computer-implemented methods consistent with one or more implementations of the present subject matter may be implemented by one or more data processors present in a single computing system or in multiple computing systems. Such multiple computing systems may be connected, e.g., to exchange data and / or commands or other instructions, etc., via one or more connections, including, for example, connections via a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, etc.), direct connections between one or more of the multiple computing systems, etc.

[0008] Details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features and advantages of the subject matter described herein will be apparent from the description and drawings, and from the claims. While certain features of the presently disclosed subject matter are described for illustrative purposes in connection with machine-learning-enabled identification of gene expression, protein expression, and gene signature expression in histological images, it should be readily understood that such features are not intended to be limiting. The claims following this disclosure define the scope of the protected subject matter. [Brief explanation of the drawings]

[0009] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate certain aspects of the subject matter disclosed herein and, together with the description, serve to explain some of the principles associated with the disclosed embodiments.

[0010] [Figure 1] FIG. 1 depicts a system diagram showing an example of a digital pathology system, according to some exemplary embodiments.

[0011] [Figure 2A] 1 depicts a flowchart illustrating an example of a process for machine learning-enabled identification of molecular features in histological images, according to some exemplary embodiments.

[0012] [Figure 2B] 1 depicts a flowchart showing another example of a process for machine learning-enabled identification of molecular features in histological images, according to some exemplary embodiments.

[0013] [Figure 3] 1 depicts a schematic diagram showing an example of a histological computational model, according to some exemplary embodiments.

[0014] [Figure 4A] 1 depicts a schematic diagram illustrating an example of a tile extractor and a feature extractor, according to some exemplary embodiments;

[0015] [Figure 4B] 1 depicts a schematic illustrating an example of cross-cluster attention, according to some exemplary embodiments.

[0016] [Figure 5] 1 depicts an example of pre-processing of a histological image, according to some exemplary embodiments.

[0017] [Figure 6]FIG. 1 depicts a graph illustrating the structural similarity (SSIM) index as a measure of agreement between transforming growth factor (TGF)-β inhibitory membrane-associated protein (TIMAP) cell type predictions and tile-level gene expression predictions made by a histological computational model, according to some exemplary embodiments.

[0018] [Figure 7A] 1 depicts histological images illustrating the concordance between tumor cells identified by transforming growth factor (TGF)-β inhibitory membrane-associated protein (TIMAP) cell type prediction and tile-level gene expression prediction made by a histological computational model, according to some exemplary embodiments.

[0019] [Figure 7B] 1 depicts histological images showing the concordance between lymphocytes identified by transforming growth factor (TGF)-β inhibitory membrane-associated protein (TIMAP) cell type prediction and tile-level gene expression prediction made by a histological computational model, according to some exemplary embodiments.

[0020] [Figure 7C] 1 depicts histological images illustrating the concordance between fibroblasts identified by transforming growth factor (TGF)-β inhibitory membrane-associated protein (TIMAP) cell type prediction and tile-level gene expression prediction made by a histological computational model, according to some exemplary embodiments.

[0021] [Figure 8A] 1 depicts a histological image of a tumor region localized based on molecular features identified by a histological computational model, according to some exemplary embodiments.

[0022] [Figure 8B] 1 depicts histological images of intratumor heterogeneity captured based on molecular features identified by a histological computational model, according to some exemplary embodiments.

[0023] [Figure 8C] 1 depicts the correspondence between cyclin spatial patterns identified by histological computational models and cyclin bulk RNA sequence expression patterns, according to some exemplary embodiments.

[0024] [Figure 9A] 10 depicts various examples of signatures associated with tiles depicting lymphocytes in a histological image, according to some exemplary embodiments.

[0025] [Figure 9B] 10 depicts various examples of signatures associated with tiles depicting adipose, tumor, and mucinous tissue structures within a histological image, according to some exemplary embodiments.

[0026] [Figure 10A] 1 depicts histological images showing fatty acid oxidation and proton transport signature co-localization, which are predictive biomarkers for the prediction of clinical outcomes, according to some exemplary embodiments.

[0027] [Figure 10B] 1 depicts histological images showing amino acid catabolism and neuronal signature co-localization, which are predictive biomarkers for the prediction of clinical outcomes, according to some exemplary embodiments.

[0028] [Figure 11] 1 depicts a block diagram illustrating an example of a computing system, according to some illustrative embodiments.

[0029] Wherever practical, like reference numerals refer to like structures, features, or elements. DETAILED DESCRIPTION OF THE INVENTION

[0030] In highly heterogeneous diseases such as cancer, insight into molecular features present in diseased tissues and the surrounding microenvironment can be essential for accurate clinical endpoint prediction. For example, specific molecular features, such as gene expression, protein expression, and gene signature expression, can serve as biomarkers for diagnosing disease subtypes, prognosing disease progression, and predicting response to various treatments. Nevertheless, traditional histological analysis techniques for identifying molecular features within microscopic images (e.g., hematoxylin and eosin (H&E)-stained whole slide images, multiplex immunofluorescence (MxIF)-stained whole slide images, etc.), including deep learning-based approaches, focus on fixed-size features. However, important insights are often found across a range of different sizes of features, from millimeter-scale features such as blood vessels to cellular-scale features such as the tissue microenvironment.

[0031] In some exemplary embodiments, the histological computational model may apply a hybrid multi-instance learning (MIL) approach to tiles of different sizes in an image of a biological sample (e.g., a whole slide image (WSI) or the like). For example, the histological computational model may extract from the image of the biological sample a first plurality of tiles of a first size (e.g., 224×224 pixels) capturing features at a first scale (e.g., cellular scale) and a second plurality of tiles of a second size (e.g., 56×56 pixels) capturing features at a second scale (e.g., millimeter scale). Furthermore, the histological computational model may concatenate a first plurality of features extracted from the first plurality of tiles of the first size with a second plurality of features extracted from the second plurality of tiles of the second size. For example, in some cases, the histological computational model may apply pyramidal concatenation, in which a feature from a larger tile covering a portion of the image is concatenated with features from two or more smaller tiles covering the same (or similar) portion of the image. Thus, a first feature associated with a first tile of a first size is concatenated with at least a second feature associated with a second tile of a second size and a third feature associated with a third tile of a second size. Additionally, in some cases, a first feature associated with a first tile of a first size may be concatenated with a second feature from the first tile of a first size, a third feature from the second tile of a second size, and a fourth feature from the third tile of a second size.

[0032] In some exemplary embodiments, the histological computational model may determine one or more bag levels in an image of a biological sample based on a combined representation of key instances from a first plurality of tiles of a first size and a second plurality of tiles of a second size. For example, the bag level indicators in the image may indicate whether a biological sample depicted in the image is associated with a molecular feature, such as gene expression, protein expression, or gene signature expression. In this regard, if the biological sample is positive for (or exhibits) the molecular feature, the biological sample may be associated with the molecular feature; if the biological sample is negative for (or does not exhibit) the molecular feature, the biological sample may not be associated with the molecular feature.

[0033] In some exemplary embodiments, the bag-level indicators may be determined based at least on a combined representation of key instances included in the first and second tiles. In some cases, the bag-level indicators in the image may be determined based on a positional embedding of a first plurality of features extracted from a first plurality of tiles of a first size concatenated with a second plurality of features extracted from a second plurality of tiles of a second size. For example, the positional embedding may include a first position in a first tile of a first size embedded with a first feature extracted from the first tile, a second position in a second tile of a second size embedded with a second feature extracted from the second tile, and a third position in a third tile of a second size embedded with a third feature extracted from the third tile. Thus, the bag-level indicators in the image may be determined to take into account different scale features from tiles of different sizes as well as the spatial distribution of these features within the image.

[0034] In some exemplary embodiments, the histological computational model may include an attention mechanism for identifying one or more key instances across individual tiles when determining bag-level indicators in an image. Thus, in some cases, the histological computational model may include an attention generator network trained to determine, for each positional embedding (e.g., of a first feature in a first tile of a first size concatenated with a second feature in a second tile of a second size and a third feature in a third tile of a second size), a corresponding attention weight indicating whether the corresponding instance triggers a bag-level indicator in the image. For example, in some cases, the bag-level indicator in an image of a biological sample may be a binary value indicating whether the biological sample is associated with a particular molecular feature. The key instance in this case may refer to a tile (or cluster of tiles) that triggers a bag-level indicator in the image by causing at least the bag-level indicator to assume either a first value indicating that the biological sample is associated with the molecular feature (or is positive for the molecular feature) or a second value indicating that the biological sample is not associated with the molecular feature (or is negative for the molecular feature).

[0035] In some exemplary embodiments, the histological computational model may determine multiple bag-level indicators in an image of a biological sample, each of which indicates whether the biological sample depicted in the image is associated with a molecular feature, such as gene expression, gene signature expression, or protein expression. For example, the histological computational model may determine a first bag-level indicator in the image based on attention-weighted instances of positionally embedded and connected Feature sets from a first plurality of tiles and / or a second plurality of tiles. In this regard, each instance included in the image of the biological sample may refer to a positional embedding of a connected Feature set, for example, including a first feature associated with a first tile of a first size connected with at least a second feature associated with a second tile of a second size and a third feature associated with a third tile of the second size. Furthermore, in some cases, the histological computational model may perform attention-based tile selection and pooling, followed by instance regression, to determine a first bag-level indicator in the image of the biological sample based at least on the attention-weighted instances.

[0036] In some exemplary embodiments, the histological computational model may also determine a second bag-level indicator in the image based on different tile clusters within the image of the biological sample. For example, in some cases, the histological computational model may perform position-based clustering to identify one or more clusters of spatially adjacent tiles within a first plurality of tiles and a second plurality of tiles within the image. For each tile cluster, the histological computational model may perform cross-cluster attention map distillation to determine indicators that identify molecular features present in tiles found in other tile clusters. Furthermore, for each tile cluster, the histological computational model may determine a set of cross-cluster attention weights that include first average attention weights for tiles within the tile cluster and second average attention weights for tiles within the other tile clusters. In some cases, the histological computational model may determine a second bag-level indicator in the image of the biological sample based on the set of cross-cluster attention weights associated with each tile cluster. For example, in some cases, the histological computational model may perform attention-based cluster selection and pooling, followed by bag-level regression, to determine a second bag-level indicator in the entire image based at least on the attention-weighted tile clusters. In some cases, the histological computational model may determine a global label in the image indicating whether the biological sample depicted in the image is associated with a molecular feature, such as gene expression, gene signature expression, protein expression, etc., based at least on the first bag-level label determined by the instance-level regression and the second bag-level label determined by the bag-level regression.

[0037] FIG. 1 depicts a system diagram illustrating an example of a digital pathology system 100, according to some exemplary embodiments. Referring to FIG. 1 , digital pathology system 100 may include a digital pathology platform 110, an imaging system 120, and a client device 130. As shown in FIG. 1 , digital pathology platform 110, imaging system 120, and client device 130 may be communicatively coupled via a network 140. Network 140 may be a wired and / or wireless network, including, for example, a local area network (LAN), a virtual local area network (VLAN), a wide area network (WAN), a public land mobile network (PLMN), the Internet, etc. Imaging system 120 may include one or more imaging devices, including, for example, a microscope, a digital camera, a whole slide scanner, a robotic microscope, etc. Client device 130 may be a processor-based device, including, for example, a workstation, a desktop computer, a laptop computer, a smartphone, a tablet computer, a wearable device, etc.

[0038] Referring again to FIG. 1 , the digital pathology platform 110 may include a histological computational model 115 and an analysis engine 117. In the example shown in FIG. 1 , the digital pathology platform 110 may apply the histological computational model 115 to an image 125 of a biological sample to identify one or more molecular features present in the biological sample. Examples of molecular features may include gene expression, gene signature expression, and protein expression, as well as gene mutations, copy number alterations (CNAs), cellular phenotypes, etc. In some cases, the first image 125 may be a stained whole slide image (WSI), including, for example, a hematoxylin and eosin (H&E)-stained whole slide image, a multiplex immunofluorescence (MxIF)-stained whole slide image, an immunohistochemistry (IHC)-stained whole slide image, etc. In some cases, the analysis engine 117 may determine at least one of a disease diagnosis, disease progression, disease burden, treatment, treatment response, and survival prediction in a patient associated with the biological sample based at least on the one or more molecular features present in the biological sample. Alternatively and / or additionally, the analysis engine 117 may identify one or more biomarkers and disease-modifying target genes based at least on one or more molecular features present in the biological sample. In some cases, the analysis engine 117 may also perform bulk RNA sequence prediction and in silico spatial transcriptomics to determine the spatial distribution of genetic activity occurring within the biological sample based at least on one or more molecular features present in the biological sample.

[0039] 2A depicts a flowchart illustrating an example of a process 200 for machine-learning-enabled identification of molecular features in histological images, according to some exemplary embodiments. Referring to FIG. 2A, process 200 may be performed by digital pathology platform 110 to determine bag-level indicators indicating whether a biological sample depicted in image 125 is associated with molecular features, including, for example, gene expression, gene signature expression, and protein expression, gene mutations, copy number alterations (CNAs), cellular phenotypes, etc.

[0040] At 202, the digital pathology platform 110 may determine a first plurality of tiles having a first size within the image of the biological specimen. In some exemplary embodiments, the digital pathology platform 110 may extract tiles of different sizes from the image of the biological specimen 125 to capture features of different scales, such as millimeter-scale features, such as blood vessels, and cellular-scale features, such as the tissue microenvironment. Further explaining, FIG. 3 depicts a schematic diagram of an example of a histological computational model 115, according to some exemplary embodiments. As shown in FIG. 3, the histological computational model 115 may include a tile extractor 302. FIG. 4A depicts a schematic diagram of an example of the tile extractor 302, according to some exemplary embodiments. As shown in FIG. 4A, the tile extractor 302 may perform patch extraction to extract a first plurality of tiles 410 of a first size (e.g., 224×224 pixels) from the image of the biological specimen 125. In some cases, the first plurality of tiles 410 of the first size may capture features of a first scale, such as global features or cellular-scale features present in the image 125.

[0041] In some cases, prior to application of the histological computational model 115, the image 125 may undergo various forms of image preprocessing. For example, in some cases, the image 125 may be preprocessed to reduce and / or remove artifacts. Alternatively and / or additionally, in some cases, the image 125 may be preprocessed to remove one or more background portions of the image 125. FIG. 5 depicts an example in which the image 125 is preprocessed to remove artifacts and background. Furthermore, in some cases, when determining the first plurality of tiles 410, the tile extractor 302 may exclude one or more tiles in which less than a threshold portion of the tile (e.g., less than 50% or another threshold portion of the tile) is covered by the biological sample.

[0042] At 204, the digital pathology platform 110 may determine a second plurality of tiles having a second size within the image of the biological specimen. Referring again to FIGS. 3 and 4A , in some exemplary embodiments, the tile extractor 302 (or a different tile extractor) of the histological computational model 115 may extract a second plurality of tiles 420 of a second size from the image 125 of the biological specimen. In some cases, the second plurality of tiles 420 of the second size may include a different number of pixels (e.g., 64×64 pixels) than the first plurality of tiles 420. Further, in some cases, a single tile of the first plurality of tiles 410 may cover the same (or a similar) portion of the image 125 as two or more tiles of the second plurality of tiles 420. Thus, in some cases, the second plurality of tiles 420 of the second size may capture second-scale features, such as local features or millimeter-scale features, present in the image 125. 4A illustrates that each tile in the first plurality of tiles 410 and the second plurality of tiles 420 is an equal-sized tile, the first plurality of tiles 410 and / or the second plurality of tiles 420 may include tiles of different sizes. For example, in some cases, two or more tiles in the second plurality of tiles 410 that cover the same (or similar) portion of the image 125 as a single tile in the first plurality of tiles 420 may have the same size or different sizes. Furthermore, the same or a different number of tiles in the second plurality of tiles 420 of the second size may be associated with each tile in the first plurality of tiles 410 of the first size. For example, a first number of tiles from the second plurality of tiles 420 of the second size may be associated with a first tile in the first plurality of tiles 410 of the first size, while the same first or a different second number of tiles from the second plurality of tiles 420 may be associated with a second tile in the first plurality of tiles 410. In some cases, when determining the second plurality of tiles 420 of the second size, the tile extractor 302 may exclude one or more tiles where less than a threshold portion of the tile (e.g., less than 50% or another threshold portion of the tile) is covered by the biological sample.

[0043] At 206, the digital pathology platform 110 may extract a first plurality of features from the first plurality of tiles of a first size. Referring again to FIGS. 3 and 4A, the histological computational model 115 may include a feature extractor 304 (e.g., including a machine learning model 400, such as a visual transducer) trained to extract a first plurality of features from the first plurality of tiles 410 of a first size. In some cases, the first plurality of features may be at a first scale (e.g., a global scale or a cellular scale) corresponding to the first size of the first plurality of tiles 410. In the example shown in FIG. 3, the feature extractor 304 is a view-specific feature extractor trained to recognize and extract features associated with a particular disease or a particular subclass of disease, such as cancer. However, it should be understood that the feature extractor 304 may also be implemented as a general feature extractor trained to recognize and extract features associated with multiple diseases or multiple subclasses of diseases.

[0044] At 208, the digital pathology platform 110 may extract a second plurality of features from the second plurality of tiles of the second size. In some cases, the feature extractor 304 (or a different feature extractor) of the histological computational model 115 may also be trained to extract a second plurality of features from the second plurality of tiles 420 of the second size. In some cases, the second plurality of features may be at a second scale (e.g., a local scale or a millimeter scale) corresponding to the second size of the second plurality of tiles 420.

[0045] At 210, the digital pathology platform 110 determines one or more molecular features present in the biological sample depicted in the image based at least on the first plurality of features and the second plurality of features. In some exemplary embodiments, the histological computational model 115 may determine one or more bag-level indicators in the image 125 based at least on the first plurality of features extracted from the first plurality of tiles 410 of the first size and the second plurality of features extracted from the second plurality of tiles 420 of the second size. In some cases, the bag-level indicators in the image 125 may be determined based at least on the positional embedding 308 of the linked Feature set 306. For example, in some cases, the linked Feature set 306 may include a first feature associated with a first tile from the first plurality of tiles 420 of the first size linked to at least a second feature associated with a second tile from the second plurality of tiles 410 of the second size and a third feature associated with a third tile from the second plurality of tiles 420 of the second size. Additionally, in some cases, the linked feature set 306 may include a first feature associated with a first tile of a first size, a second feature from the first tile of a first size, a third feature from a second tile of a second size, and a fourth feature from a third tile of a second size. Meanwhile, the position embedding 308 of the linked feature set 306 may further include a first position of the first tile, a second position of the second tile, and / or a third position of the third tile. In some cases, the first position of the first tile, the second position of the second tile, and the third position of the third tile may each include a set of coordinates of one or more pixels included in the corresponding tile, such as, for example, one or more pixels occupying a corner (e.g., the upper left corner) of the corresponding tile. As described in more detail below, in some cases, a bag-level indicator indicating whether a biological sample depicted in image 125 is associated with a molecular feature may be determined based at least on attention-weighted instances, each of which corresponds to a positional embedding of the connected feature across the first plurality of tiles 410 and the second plurality of tiles 420.

[0046] At 212, the digital pathology platform 110 may perform one or more downstream analysis tasks based at least on one or more molecular features present in the biological sample. In some exemplary embodiments, the digital pathology platform 110, e.g., the analysis engine 117, may perform various downstream analysis tasks based on one or more molecular features, such as gene expression, gene signature expression, and protein expression, as well as gene mutations, copy number alterations (CNAs), cellular phenotypes, etc., identified as present (or absent) in the biological sample depicted in the image 125. For example, in some cases, the analysis engine 117 may determine at least one of a disease diagnosis, disease progression, treatment, treatment response, and survival prediction in a patient associated with the biological sample based at least on one or more molecular features identified as present (or absent) from the biological sample. In some cases, the analysis engine 117 may also identify one or more biomarkers and disease-modifying target genes based at least on one or more molecular features identified as present (or absent) from the biological sample. Alternatively and / or additionally, in some cases, the analysis engine 117 may perform bulk RNA sequence prediction and in silico spatial transcriptomics to determine the spatial distribution of genetic activity occurring within the biological sample, based at least on one or more molecular features identified as present (or absent) in the biological sample.

[0047] 2B depicts a flowchart illustrating an example of a process 250 for machine-learning-enabled identification of molecular features in histological images, according to some exemplary embodiments. Referring to FIG. 2B, process 250 may be performed by digital pathology platform 110 to determine, based on features extracted from tiles of different sizes within image 125, a bag-level indicator indicating whether the biological sample depicted in image 125 is associated with molecular features including, for example, gene expression, gene signature expression, and protein expression, gene mutations, copy number alterations (CNAs), cellular phenotypes, etc. In some cases, process 250 may implement operation 210 of process 200 described with respect to FIG. 2A.

[0048] At 252, the digital pathology platform 110 may concatenate a first plurality of features extracted from the first plurality of tiles of a first size with a second plurality of features extracted from the second plurality of tiles of a second size. For example, as shown in Figures 3 and 4A, the digital pathology platform 110 may generate a concatenated Feature set 306 based at least on the first plurality of features extracted from the first plurality of tiles of a first size 410 and the second plurality of features extracted from the second plurality of tiles of a second size 420. In some cases, the concatenated Feature set 306 may include, for example, a concatenation of a first feature of a first tile from the first plurality of tiles of a first size 410, a second feature of a second tile from the second plurality of tiles of a second size 420, and a third feature of a third tile from the second plurality of tiles of a second size 420.

[0049] At 254, the digital pathology platform 110 may determine a location embedding for each concatenated Feature Set, the location embedding including a first feature of a first tile from the first plurality of tiles of the first size, a second feature of a second tile from the second plurality of tiles of the second size, and a third feature of a third tile from the second plurality of tiles of the second size. For example, in some cases, the histological computational model 115 may generate a corresponding location embedding 308 for each concatenated Feature Set 306. In some cases, the location embedding 308 for a concatenated Feature Set 306 may include, for example, a first position of the first tile, a second position of the second tile, and / or a third position of the third tile. Further, in some cases, the first position of the first tile, the second position of the second tile, and the third position of the third tile may each include a set of coordinates of one or more pixels included in the corresponding tile. For example, in some cases, the positional embedding 308 of the concatenated feature set 306 may be generated based on pixels occupying one or more corners (e.g., the upper left corner) of a first tile from a first plurality of tiles 410 of a first size, a second tile from a second plurality of tiles 420 of a second size, and a third tile from the second plurality of tiles 420 of a second size.

[0050] At 256, the digital pathology platform 110 may determine a first bag-level indicator in the image of the biological specimen based at least on the attention-weighted positional embeddings of the concatenated Feature sets. Referring again to FIG. 3 , in some exemplary embodiments, the histological computational model 115 may include an attention generator network 310 configured to determine, for the positional embeddings 308 of each concatenated Feature set 306, an attention weight indicating the relative importance of individual instances, including the corresponding tile and the features contained therein. In this regard, the attention weight assigned to the positional embeddings 308 of the concatenated Feature sets 306 may be a value corresponding to how much that particular instance contributes to the bag-level indicator in the image 125. Thus, an important instance (or key instance) that triggers (or contributes to) the bag-level indicator may be associated with a higher attention weight than a less important instance that is less relevant to the bag-level indicator. In the example shown in FIG. 3 , for example, the attention generator network 310 may determine, for each of the N positional embeddings 308, the corresponding attention weights a1, a2, ..., a N may be determined.

[0051] 3 , the histological computational model 110 may include an attention-based tile selection and pooling network 312 followed by an instance regressor 314 trained to determine, based at least on the attention-weighted instances (e.g., attention-weighted positional embeddings 308 of the concatenated feature sets 306), a first bag-level indicator indicating whether the biological sample depicted in the image 125 is associated with a molecular feature, such as gene expression, gene signature expression, protein expression, gene mutation, copy number alteration (CNA), cellular phenotype, etc. For example, in some cases, the first bag-level indicator may be a binary indicator having a first value (e.g., “1”) indicating that the biological sample is associated with the molecular feature (or is positive for the molecular feature) or a second value (e.g., “0”) indicating that the biological sample is not associated with the molecular feature (or is negative for the molecular feature). In some cases, the instance regressor 314 may be implemented using a neural network, a Hopfield network, etc.

[0052] At 258, the digital pathology platform 110 may identify one or more tile clusters based at least on the location of each of the first and second plurality of tiles. In some exemplary embodiments, the histological computational model 115 may apply a clustering algorithm 316 to identify one or more clusters of similar tiles within the first plurality of tiles 410 and / or the second plurality of tiles 420. In some cases, the histological computational model 115 may apply the clustering algorithm 316 to perform position-based clustering such that the resulting clusters of tiles include spatially proximate tiles (e.g., tiles occupying the same or similar regions of the image 125). In the example shown in FIG. 3 , the histological computational model 115 applies the clustering algorithm 316 to identify one or more clusters of tiles C1, C2, ..., C3 within the first plurality of tiles 410 of the first size and the second plurality of tiles 420 of the second size. k A number of tile clusters may be determined, denoted as k tile clusters.

[0053] At 260, the digital pathology platform 110 may determine, for each tile cluster, a set of cross-cluster attention weights including a first average attention weight of tiles within the tile cluster and a second average attention weight of tiles within other tile clusters. Referring again to FIG. 3 , the histological computational model 115 may determine, for each tile cluster, a first average attention weight C of tiles within tile cluster k, e.g., k a and a second average attention weight nC for tiles that are not in tile cluster k but are in other tile clusters. k In the example shown in FIG. 3, the histological computational model 115 may determine the set of cross-cluster attention weights by performing cross-cluster attention 320 across tile clusters. To further explain, FIG. 4B depicts a schematic diagram illustrating an example of cross-cluster attention 320, in which the histological computational model 115 determines a first average attention weight C of tiles within tile cluster k based at least on the cross-cluster attention map 450. k a and the second average attention weight nC for tiles not in the tile cluster k As shown in Figure 3, the first average attention weight C k a may be associated with a first combined representation of the features of tiles within tile cluster k, while a second average attention weight nC k a may be associated with a second combined representation of features of tiles not within the tile cluster.

[0054] At 262, the digital pathology platform 110 may determine a second bag-level indicator indicating whether the biological sample depicted in the image is associated with the molecular feature based at least on the cross-cluster attention weights for each tile cluster. In some exemplary embodiments, the example histological computational model 115 shown in FIG. 3 may include an attention-based cluster selection and pooling network 322 and a bag regressor 324 trained to determine a second bag-level indicator indicating whether the biological sample depicted in the image 125 is associated with the molecular feature based at least on the attention-weighted tile clusters. For example, as shown in FIG. 3, the second bag-level indicator may be determined based on the first average attention weight C for each tile cluster k. k A first combined representation of features present in tiles in tile cluster k weighted by a and a second average attention weight nC k a second combined representation of features present in tiles outside tile cluster k weighted by a. In some cases, bag regressor 324 may be implemented using a neural network, a Hopfield network, etc.

[0055] At 264, the digital pathology platform 110 may determine an overall indicator indicating whether the biological sample depicted in the image is associated with a molecular feature based at least on the first bag-level indicator and the second bag-level indicator. For example, as shown in FIG. 3 , the histological computational model 115 may determine an overall indicator indicating whether the biological sample depicted in the image 125 is associated with a molecular feature, such as gene expression, gene signature expression, protein expression, gene mutation, copy number alteration (CNA), cellular phenotype, etc., based at least on the first bag-level indicator in the image 125 determined by the instance regressor 314 and the second bag-level indicator in the image 125 determined by the bag regressor 324.

[0056] In some exemplary embodiments, the performance of the histological computational model 115 in determining whether the biological sample depicted in the image 125 is associated with molecular biomarkers (e.g., gene expression, gene signature expression, protein expression, gene mutations, copy number alterations (CNAs), cell phenotypes, etc.) may be evaluated based on, for example, the agreement between a transforming growth factor (TGF)-β inhibitory membrane-associated protein (TIMAP) cell mask and tile-level gene expression predictions made by the histological computational model 115. FIG. 6 depicts a graph illustrating a structural similarity index (SSIM) as a measure of agreement between TIMAP cell-type predictions and tile-level gene expression predictions made by the histological computational model, according to some exemplary embodiments. FIG. 7A depicts a histological image showing agreement between tumor cells identified through TIMAP cell-type predictions and tile-level gene expression predictions made by the histological computational model 115. FIG. 7B depicts a histological image showing agreement between lymphocytes identified through TIMAP cell-type predictions and tile-level gene expression predictions made by the histological computational model 115. FIG. 7C depicts histological images illustrating the concordance between fibroblasts identified through TIMAP cell type prediction and tile-level gene expression prediction made by the histological computational model 115.

[0057] In some cases, the performance of the histological computational model 115 may also be evaluated based on the agreement with expert annotations. For example, FIG. 8A depicts a histological image of a tumor region localized based on molecular features identified by the histological computational model 115 and corresponding expert annotations of the same image. FIG. 8B depicts a histological image of intratumoral heterogeneity captured based on molecular features identified by the histological computational model 115 and corresponding expert annotations of the same image. In some cases, predictions made by the histological computational model 115 may also be validated based on the underlying bulk RNA-seq expression pattern. For example, FIG. 8C depicts the agreement between the cyclin spatial pattern identified by the histological computational model 115 and the same cyclin pattern identified by bulk RNA-seq expression.

[0058] As mentioned above, in some exemplary embodiments, digital pathology platform 110 may apply histological computational model 115 to determine whether a biological sample depicted in image 125 is associated with one or more molecular features, including, for example, gene expression, gene signature expression, protein expression, genetic mutations, copy number alterations (CNAs), cellular phenotypes, etc. For example, in some cases, histological computational model 115 may output, for a particular molecular feature, a binary indicator having either a first value (e.g., “1”) indicating that the biological sample is associated with the molecular feature (or is positive for the molecular feature) or a second value (e.g., “0”) indicating that the biological sample is not associated with the molecular feature (or is negative for the molecular feature). FIG. 9A depicts various examples of signatures associated with tiles depicting lymphocytes within a histological image, such as image 125, while FIG. 9B depicts various examples of signatures associated with tiles depicting fat, tumor, and mucinous tissue structures within a histological image, such as image 125.

[0059] In some exemplary embodiments, the digital pathology platform 110 may perform various downstream analysis tasks based on at least one or more molecular features. For example, in some cases, one or more molecular features identified in the biological sample depicted in the image 125 may serve as biomarkers for determining at least one of a disease diagnosis, disease progression, disease burden, treatment, treatment response, and survival prediction in a patient associated with the biological sample. For example, FIG. 10A depicts a histological image showing co-localization of fatty acid oxidation and proton transport signatures, which are predictive biomarkers for predicting clinical outcomes, according to some exemplary embodiments. FIG. 10B depicts a histological image showing co-localization of amino acid catabolism and neuronal signatures, which are predictive biomarkers for predicting clinical outcomes, according to some exemplary embodiments.

[0060] 11 depicts a block diagram illustrating an example of a computing system 1100 according to some exemplary embodiments. Referring to FIG. 1 and FIG. 11, the computing system 1100 may be used to implement the digital pathology platform 110, the imaging system 120, the client device 130, and / or any components therein.

[0061] 11 , computing system 1100 may include a processor 1110, a memory 1120, a storage device 1130, and an input / output device 1140. The processor 1110, the memory 1120, the storage device 1130, and the input / output device 1140 may be interconnected via a system bus 1150. The processor 1110 is capable of processing instructions for execution within the computing system 1100. Such executed instructions may implement one or more components, such as, for example, the digital pathology platform 110, the imaging system 120, the client device 130, etc. In some exemplary embodiments, the processor 1110 may be a single-threaded processor. Alternatively, the processor 1110 may be a multi-threaded processor. The processor 1110 is capable of processing instructions stored in the memory 1120 and / or the storage device 1130 to display graphical information for a user interface provided via the input / output device 1140.

[0062] Memory 1120 is a computer-readable medium, such as a volatile or non-volatile medium, that stores information within computing system 1100. Memory 1120 may store, for example, data structures representing a configuration object database. Storage device 1130 may provide persistent storage for computing system 1100. Storage device 1130 may be a floppy disk drive, a hard disk drive, an optical disk drive, or a tape drive, or other suitable persistent storage means. Input / output device 1140 provides input / output operations for computing system 1100. In some exemplary embodiments, input / output device 1140 includes a keyboard and / or a pointing device. In various embodiments, input / output device 1140 includes a display unit for displaying a graphical user interface.

[0063] According to some demonstrative embodiments, input / output devices 1140 may provide input / output operations for network devices. For example, input / output devices 1140 may include an Ethernet port or other networking port for communicating with one or more wired and / or wireless networks (e.g., a local area network (LAN), a wide area network (WAN), the Internet).

[0064] In some exemplary embodiments, computing system 1100 may be used to execute various interactive computer software applications that may be used for organizing, analyzing, and / or storing various types of data. Alternatively, computing system 1100 may be used to execute any type of software application. These applications may be used to perform various functions, such as planning functions (e.g., creating, managing, editing spreadsheet documents, word processing documents, and / or any other objects), computing functions, communication functions, etc. Applications may include various add-in functions or may be standalone computing products and / or functions. When active within an application, functionality may be used to generate a user interface that is provided via input / output devices 1140. The user interface may be generated and presented to a user by computing system 1100 (e.g., on a computer screen monitor, etc.).

[0065] One or more aspects or features of the subject matter described herein may be implemented in digital electronic circuitry, integrated circuits, specially designed ASICs, field programmable gate array (FPGA) computer hardware, firmware, software, and / or combinations thereof. These various aspects or features may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device. The programmable system or computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0066] These computer programs, which may also be referred to as programs, software, software applications, applications, components, or code, include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language and / or assembly / machine language. As used herein, the term “machine-readable medium” refers to any computer program product, apparatus, and / or device used to provide machine instructions and / or data to a programmable processor, such as, for example, magnetic disks, optical disks, memory, and programmable logic devices (PLDs), and includes a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor. A machine-readable medium may non-transitory store such machine instructions, such as, for example, a non-transitory solid-state memory, a magnetic hard drive, or any equivalent storage medium. Alternatively or additionally, a machine-readable medium may temporarily store such machine instructions, such as, for example, a processor cache or other random access memory associated with one or more physical processor cores.

[0067] To provide for user interaction, one or more aspects or features of the subject matter described herein may be implemented on a computer having, for example, a display device, such as a cathode ray tube (CRT) or liquid crystal display (LCD) or light-emitting diode (LED) monitor, for displaying information to a user, and a keyboard and pointing device, such as a mouse or trackball, through which the user may provide input to the computer. Other types of devices may also be used to provide for user interaction. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Other possible input devices include touchscreens or other touch-sensitive devices, such as single-point or multi-point resistive or capacitive trackpads, voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices and associated interpretation software.

[0068] Embodiment Among the embodiments provided are the following: 1. Determining a first plurality of tiles having a first tile size within an image of a biological sample; determining a second plurality of tiles within the image of the biological sample, the second plurality of tiles having a second tile size; applying a feature extraction model to extract a first plurality of features from a first plurality of tiles of a first size; applying a feature extraction model to extract a second plurality of features from the second plurality of tiles of the second size; determining one or more molecular features present in the biological sample depicted in the image based at least on the first plurality of features and the second plurality of features; 10. A computer-implemented method comprising: 2. The method of embodiment 1, wherein the feature extraction model includes a first machine learning model trained to extract a first plurality of features from a first plurality of tiles having a first tile size, and the feature extraction model further includes a second machine learning model trained to extract a second plurality of features from a second plurality of tiles having a second tile size. 3. The method of any one of embodiments 1 to 2, wherein a first tile size of the first plurality of tiles includes a different number of pixels than a second tile size of the second plurality of tiles. 4. The method of any one of embodiments 1 to 3, wherein the first plurality of features extracted from the first plurality of tiles having a first tile size include global features present in the image of the biological sample, and the second plurality of features extracted from the second plurality of tiles having a second tile size include local features present in the image of the biological sample. 5. The method of any one of embodiments 1 to 4, wherein the first plurality of features extracted from the first plurality of tiles having a first tile size include cell-scale features, and the second plurality of features extracted from the second plurality of tiles having a second tile size include millimeter-scale features. 6. The method of any one of embodiments 1 to 5, wherein a first tile size of the first plurality of tiles is 56 pixels by 56 pixels. 7. The method of any one of embodiments 1 to 6, wherein the second tile size of the second plurality of tiles is 224 pixels by 224 pixels. 8. Determining a third plurality of tiles having a third size within the image of the biological specimen; applying a feature extraction model to extract a third plurality of features from the third plurality of tiles; determining one or more molecular features in the biological sample based at least on the third plurality of features; 8. The method of any one of embodiments 1 to 7, further comprising: 9. Concatenating a first plurality of features with a second plurality of features; determining one or more molecular features present in the biological sample based at least on a linkage between the first plurality of features and the second plurality of features; 9. The method of any one of embodiments 1 to 8, further comprising: 10. The method of embodiment 9, wherein linking the first plurality of features and the second plurality of features includes linking a first feature associated with a first tile of the first plurality of tiles of a first size with a second feature associated with a second tile of the second plurality of tiles of a second size. 11. The method of embodiment 10, wherein linking the first plurality of features and the second plurality of features further includes linking a second feature associated with a first tile of the first plurality of tiles of the first size with a second feature associated with a second tile of the second plurality of tiles of the second size. 12. The method of embodiment 10, wherein linking the first plurality of features and the second plurality of features further includes linking the first feature of the first tile and the second feature of the second tile with a third feature of a third tile of the second plurality of tiles of the second size. 13. Generating a positional embedding of a concatenated feature set including a first feature of a first tile, a second feature of a second tile, and a third feature of a third tile; 13. The method of embodiment 12, further comprising: 14. The method of embodiment 13, wherein the position embedding includes a first position in a first tile, a second position in a second tile, and / or a third position in a third tile. 15. The method of embodiment 14, wherein the first location of the first tile, the second location of the second tile, and the third location of the third tile each include a set of coordinates of at least one pixel from the corresponding tile. 16. Determining a first bag-level signature indicative of one or more molecular features present in the biological sample based at least on the positional embedding of each tile of the first plurality of tiles and the linked feature set associated with two or more corresponding tiles of the second plurality of tiles; 16. The method of any one of embodiments 1 to 15, further comprising: 17. The method of embodiment 16, wherein the one or more molecular features are further determined based at least on an attention weight associated with each position embedding. 18. The method of embodiment 17, wherein the one or more molecular features are determined by applying an attention-based tile selection and pooling network and an instance regressor to a plurality of attention-weighted position embeddings. 19. Determining a second bag-level signature indicative of one or more molecular features present in the biological sample based at least on the combined representation of features associated with one or more clusters of tiles; 17. The method of embodiment 16, further comprising: 20. Clustering the first plurality of tiles and the second plurality of tiles into one or more clusters of tiles; 20. The method of embodiment 19, further comprising: 21. The method of embodiment 20, wherein the clustering is performed based at least on position information of each tile of the first plurality of tiles and the second plurality of tiles. 22. The method of embodiment 20, wherein the first plurality of tiles and the second plurality of tiles are clustered into a configurable number of clusters. 23. For each cluster of tiles, determining a first average attention weight for features of tiles in the cluster and a second average attention weight for features of tiles not in the cluster; 21. The method of embodiment 20, further comprising: 24. Determining one or more molecular features present in the biological sample based at least on a first average attention weight applied to a first combined representation of tile features within the cluster and a second average attention weight applied to a second combined representation of tile features not within the cluster; 24. The method of embodiment 23, further comprising: 25. The method of embodiment 24, wherein the one or more molecular features present in the biological sample are determined by applying an attention-based cluster selection and pooling network and a bag-level regressor to at least a first average attention weight applied to a first combined representation of tile features in the cluster and a second average attention weight applied to a second combined representation of tile features not in the cluster. 26. Determining an overall label indicative of one or more molecular features present in the biological sample based at least on the first bag-level label and the second bag-level label; 20. The method of embodiment 19, further comprising: 27. The method of any one of embodiments 1 to 26, wherein the feature extraction model is a visual transformer. 28. The method of any one of embodiments 1 to 27, wherein the image is a whole slide image. 29. The method of any one of embodiments 1 to 28, wherein the image is a hematoxylin and eosin (H&E) stained whole slide image. 30. The method of any one of embodiments 1 to 29, wherein the biological sample comprises one or more tissue fragments, free cells, and / or body fluids. 31. The method of any one of embodiments 1 to 30, wherein the biological sample comprises tumor tissue. 32. The method of any one of embodiments 1 to 31, wherein the feature extraction model is trained to extract features associated with a particular disease or a particular subtype of a disease. 33. The method of any one of embodiments 1 to 32, wherein the feature extraction model is trained to extract features associated with a particular cancer or a particular subtype of cancer. 34. The method of any one of embodiments 1 to 33, wherein the one or more molecular features comprise gene expression, gene signature expression, protein expression, gene mutations, copy number alterations (CNAs), and / or cellular phenotypes. 35. Identifying one or more biomarkers and disease-modifying target genes based at least on one or more molecular features present in a biological sample; 35. The method of any one of embodiments 1 to 34, further comprising: 36. Making bulk RNA sequence predictions based at least on one or more molecular features present in a biological sample; 36. The method of any one of embodiments 1 to 35, further comprising: 37. Performing in silico spatial transcriptomics to determine the spatial distribution of genetic activity occurring within a biological sample based at least on one or more molecular features present in the biological sample; 37. The method of any one of embodiments 1 to 36, further comprising: 38. Determining at least one of disease diagnosis, disease progression, disease burden, treatment, treatment response, and survival prediction in a patient associated with a biological sample based at least on one or more molecular features present in the biological sample; 38. The method of any one of embodiments 1 to 37, further comprising: 39. at least one data processor; At least one memory storing instructions that, when executed by at least one data processor, result in operations comprising the method of any one of embodiments 1 to 38; A system comprising: 40. A non-transitory computer-readable medium storing instructions that, when executed by at least one data processor, result in operations comprising the method of any one of embodiments 1 to 38.

[0069] In the above description and in the claims, phrases such as "at least one of" or "one or more of" may appear followed by a conjunctive list of elements or features. The term "and / or" may also be used in listings of two or more elements or features. Unless otherwise implicitly or explicitly stated by the context in which it is used, such phrases are intended to refer to any of the listed elements or features individually, or any of the listed elements or features in combination with any of the other listed elements or features. For example, the phrases "at least one of A and B," "one or more of A and B," and "A and / or B" are intended to mean "A only, B only, or A and B together," respectively. A similar interpretation is intended for lists containing more than two items. For example, the phrases "at least one of A, B, and C;," "one or more of A, B, and C;," and "A, B, and / or C" are intended to mean "A alone, B alone, C alone, A and B, A and C, B and C, or A, B, and C," respectively. Use of the term "based on" above and in the claims is intended to mean "based at least in part on," allowing for unrecited features or elements.

[0070] The subject matter described herein may be embodied in systems, devices, methods, and / or articles, depending on the desired configuration. The implementations set forth in the above description do not represent all implementations of the subject matter described herein. Instead, these implementations are merely some examples consistent with aspects associated with the described subject matter. While some variations have been described in detail above, other modifications or additions are possible. In particular, additional features and / or variations may be provided in addition to those described herein. For example, the implementations described above may be directed to various combinations and subcombinations of the disclosed features and / or combinations and subcombinations of certain additional features described above. Furthermore, the logic flow illustrated in the accompanying drawings and / or described herein does not necessarily require the particular order shown, or sequential order, to achieve desirable results. Other implementations may be within the scope of the following claims.

Claims

1. determining a first plurality of tiles within an image of the biological specimen, the tiles having a first tile size; determining a second plurality of tiles within the image of the biological specimen, the second plurality of tiles having a second tile size; applying a feature extraction model to extract a first plurality of features from the first plurality of tiles of the first tile size; applying the feature extraction model to extract a second plurality of features from the second plurality of tiles of the second tile size; determining one or more molecular features present in the biological sample depicted in the image based at least on the first plurality of features and the second plurality of features; 11. A computer-implemented method comprising:

2. 2. The method of claim 1 , wherein the feature extraction model comprises a first machine learning model trained to extract the first plurality of features from the first plurality of tiles having the first tile size, and the feature extraction model further comprises a second machine learning model trained to extract the second plurality of features from the second plurality of tiles having the second tile size.

3. The method of claim 1 , wherein the first tile size of the first plurality of tiles includes a different number of pixels than the second tile size of the second plurality of tiles.

4. 2. The method of claim 1, wherein the first plurality of features extracted from the first plurality of tiles having the first tile size comprise global features present in the image of the biological specimen, and the second plurality of features extracted from the second plurality of tiles having the second tile size comprise local features present in the image of the biological specimen.

5. 2. The method of claim 1, wherein the first plurality of features extracted from the first plurality of tiles having the first tile size comprises cell-scale features, and the second plurality of features extracted from the second plurality of tiles having the second tile size comprises millimeter-scale features.

6. The method of claim 1 , wherein the first tile size of the first plurality of tiles is 56 pixels by 56 pixels.

7. The method of claim 1 , wherein the second tile size of the second plurality of tiles is 224 pixels by 224 pixels.

8. determining a third plurality of tiles within the image of the biological specimen, the third plurality of tiles having a third tile size; applying the feature extraction model to extract a third plurality of features from the third plurality of tiles; determining the one or more molecular features in the biological sample based at least on the third plurality of features; The method of claim 1 further comprising:

9. Concatenating the first plurality of features and the second plurality of features; determining the one or more molecular features present in the biological sample based at least on a linkage between the first plurality of features and the second plurality of features; The method of claim 1 further comprising:

10. 10. The method of claim 9, wherein concatenating the first plurality of features and the second plurality of features comprises concatenating a first feature associated with a first tile of the first plurality of tiles of the first tile size with a second feature associated with a second tile of the second plurality of tiles of the second tile size.

11. 11. The method of claim 10, wherein concatenating the first plurality of features and the second plurality of features further comprises concatenating a second feature associated with the first tile of the first plurality of tiles of the first tile size with the second feature associated with the second tile of the second plurality of tiles of the second tile size.

12. 11. The method of claim 10, wherein concatenating the first plurality of features and the second plurality of features further comprises concatenating the first feature of the first tile and the second feature of the second tile with a third feature of a third tile of the second plurality of tiles of the second tile size.

13. 13. The method of claim 12, further comprising generating a positional embedding of a concatenated feature set including the first feature of the first tile, the second feature of the second tile, and the third feature of the third tile.

14. The method of claim 13 , wherein the position embedding comprises a first position of the first tile, a second position of the second tile, and / or a third position of the third tile.

15. 15. The method of claim 14, wherein the first location of the first tile, the second location of the second tile, and the third location of the third tile each include a set of coordinates of at least one pixel from a corresponding tile.

16. 2. The method of claim 1, further comprising determining a first bag-level signature indicative of the one or more molecular features present in the biological sample based at least on a positional embedding of a linked feature set associated with each tile of the first plurality of tiles and two or more corresponding tiles of the second plurality of tiles.

17. 17. The method of claim 16, wherein the one or more molecular features are further determined based at least on an attention weight associated with each position embedding.

18. 20. The method of claim 17, wherein the one or more molecular features are determined by applying an attention-based tile selection and pooling network and an instance regressor to multiple attention-weighted positional embeddings.

19. 17. The method of claim 16, further comprising determining a second bag-level signature indicative of the one or more molecular features present in the biological sample based at least on the combined representation of features associated with one or more clusters of tiles.

20. The method of claim 19 , further comprising clustering the first plurality of tiles and the second plurality of tiles into one or more clusters of tiles.

21. The method of claim 20 , wherein the clustering is performed based at least on position information of each tile of the first plurality of tiles and the second plurality of tiles.

22. The method of claim 20 , wherein the first plurality of tiles and the second plurality of tiles are clustered into a configurable number of clusters.

23. 21. The method of claim 20, further comprising: for each cluster of tiles, determining a first average attention weight for features of tiles in the cluster and a second average attention weight for features of tiles not in the cluster.

24. 24. The method of claim 23, further comprising determining the one or more molecular features present in the biological sample based at least on the first average attention weight applied to a first combined representation of the features of the tiles within the cluster and the second average attention weight applied to a second combined representation of the features of the tiles not within the cluster.

25. 25. The method of claim 24, wherein the one or more molecular features present in the biological sample are determined by applying an attention-based cluster selection and pooling network and a bag-level regressor to at least the first average attention weights applied to the first combined representation of the features of the tiles within the cluster and the second average attention weights applied to the second combined representation of the features of the tiles not within the cluster.

26. 20. The method of claim 19, further comprising determining an overall label indicative of the one or more molecular features present in the biological sample based at least on the first bag-level label and the second bag-level label.

27. The method of claim 1 , wherein the feature extraction model is a visual transformer.

28. The method of claim 1 , wherein the image is a whole slide image.

29. The method of claim 1 , wherein the image is a hematoxylin and eosin (H&E) stained whole slide image.

30. The method of claim 1 , wherein the biological sample comprises one or more tissue fragments, free cells, and / or body fluids.

31. The method of claim 1 , wherein the biological sample comprises tumor tissue.

32. The method of claim 1 , wherein the feature extraction model is trained to extract features associated with a particular disease or a particular subtype of a disease.

33. The method of claim 1 , wherein the feature extraction model is trained to extract features associated with a particular cancer or a particular subtype of cancer.

34. 10. The method of claim 1, wherein the one or more molecular features comprise gene expression, gene signature expression, protein expression, genetic mutations, copy number alterations (CNAs), and / or cellular phenotypes.

35. 10. The method of claim 1, further comprising identifying one or more biomarkers and disease-modifying target genes based at least on the one or more molecular features present in the biological sample.

36. 10. The method of claim 1, further comprising performing bulk RNA sequence prediction based at least on the one or more molecular features present in the biological sample.

37. 10. The method of claim 1, further comprising performing in silico spatial transcriptomics to determine a spatial distribution of genetic activity occurring within the biological sample based at least on the one or more molecular features present in the biological sample.

38. 10. The method of claim 1, further comprising determining at least one of a disease diagnosis, disease progression, disease burden, treatment, treatment response, and survival prediction in a patient associated with the biological sample based at least on the one or more molecular features present in the biological sample.

39. at least one data processor; at least one memory storing instructions that, when executed by said at least one data processor, result in operations comprising the method of any one of claims 1 to 38; A system comprising:

40. 29. A non-transitory computer readable medium storing instructions that, when executed by at least one data processor, result in operations comprising the method of any one of claims 1 to 28.

Images