System and method for generating pathology predictions

Abstract

A method of generating a prediction corresponding to a patient includes extracting a plurality of image embeddings from a plurality of image tiles corresponding to a whole slide image of a tissue sample of the patient; clustering the plurality of image embeddings into a plurality of image data groups based on a degree of similarity between the image embeddings; generating a plurality of weighted image embeddings by applying a plurality of first attention modules of the prediction system to the image embeddings of the plurality of image data groups; generating a slide-level image embedding by applying a second attention module of the prediction system to the weighted image embeddings such that the slide-level image embedding inherently includes genomic-based data corresponding to the tissue sample, which is not input to the prediction system; and generating the prediction based on the slide-level image embedding..

Description

PCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11 SYSTEM AND METHOD FOR GENERATING PATHOLOGY PREDICTIONSCROSS-REFERENCE TO RELATED APPLICATIONS5

[0001] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 679,894, filed August 6, 2024; U.S. Provisional Application 63 / 706,500, filed October 11 , 2024; and U.S. Provisional Application No. 63 / 778,302, filed March 26, 2025, the entire contents of which are incorporated herein by reference.10FIELD

[0002] Aspects of some embodiments of the present disclosure relate to a prediction system and a method of training the same.15BACKGROUND

[0003] Computational pathology has seen significant advancements through the integration of digital pathology systems and artificial intelligence (Al) techniques. By utilizing high-throughput imaging technologies and machine learning (ML) methodologies, computational pathology enhances the analysis and interpretation of20 pathology data, such as histopathological images, to assist in disease diagnosis, prognosis, and treatment planning. Modem Al techniques, particularly deep learning, allow for automatic feature extraction, leading to improved accuracy and efficiency in tasks like tumor detection, grading, and classification.

[0004] In computational pathology, multimodal approaches may enable25 integrating whole slide images (WSIs) with bulk transcriptom ics data to enhance patient outcome predictions by providing a more comprehensive understanding of cancer prognosis. However, due to the acquisition cost of genetic data, both WSI and genetic data for a given patient may not always be available, which limits the applicability of multi-modal prediction systems.30

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background and therefore the information discussed in this Background section does not necessarily constitute prior art.SUMMARY35

[0006] Aspects of some embodiments of the present disclosure are directed to a prediction system that extracts feature embeddings infused with both image-level and genomic-level data from an input WSI and / or genomic data and utilizes the embeddings to make a prediction based on the input data.PCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11

[0007] Aspects of some embodiments of the present disclosure are directed to a method utilizing clustering techniques for different modalities, including more refined gene clustering for transcriptom ics and tile clustering for histology images, which may significantly improve the resulting whole-slide representations. Some5 embodiments according to the present disclosure enhance the quality of multimodal embeddings and are helpful in addressing lung cancer subtype classification. Aspects of some embodiments may include multimodal pre-training and clustering in advancing both prognostic accuracy and cancer subtype identification.

[0008] According to some embodiments of the present disclosure, there is10 provided a method of generating a prediction corresponding to a patient by a prediction system, the method including: extracting a plurality of image embeddings from a plurality of image tiles corresponding to a whole slide image of a tissue sample of the patient; clustering the plurality of image embeddings into a plurality of image data groups based on a degree of similarity between the image embeddings;15 generating a plurality of weighted image embeddings by applying a plurality of first attention modules of the prediction system to the image embeddings of the plurality of image data groups; generating a slide-level image embedding by applying a second attention module of the prediction system to the weighted image embeddings such that the slide-level image embedding inherently includes genomic-based data20 corresponding to the tissue sample, which is not input to the prediction system; and generating the prediction based on the slide-level image embedding.

[0009] In some embodiments, the extracting the plurality of image embeddings is performed by a feature extractor trained on pathology images.

[0010] In some embodiments, the clustering the plurality of image embeddings25 includes applying a k-means algorithm to the plurality of image embeddings to generate the plurality of image data groups.

[0011] In some embodiments, each one of the weighted image embeddings corresponds to one of the image data groups.

[0012] In some embodiments, the plurality of first attention modules and the30 second attention module are trained to extract the genomic-based data from the plurality of image tiles and to embed the plurality of weighted image embeddings and the slide-level image embedding with the genomic-based data.

[0013] In some embodiments, the second attention module includes: a transformer configured to generate a plurality of tile-level embeddings based on pair-35 wise relationships between each one of weighted image embeddings and other ones of the weighted image embeddings; and an attention pooling network configured to generate the slide-level image embedding based on a weighted average of the tilelevel embeddings.PCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11

[0014] In some embodiments, a number of the first attention modules is a same as a number of groups of the image data groups, and each one of the image embeddings, each one of the weighted image embeddings, each one of the tile-level embeddings, and the slide-level image embedding is a one-dimensional vector of a5 same length.

[0015] In some embodiments, the generating the prediction is performed by a classifier trained to generate the prediction based on the slide-level image embedding.

[0016] In some embodiments, the generating the prediction includes: aggregating10 the slide-level image embedding and a slide-level genomic embedding to generate an aggregated slide-level embedding; and classifying the aggregated slide-level embedding to determine the prediction, wherein the slide-level genomic embedding includes genomic-based data corresponding to the patient.

[0017] According to some embodiments of the present disclosure, there is15 provided a method of generating a prediction corresponding to a patient by a prediction system, the method including: clustering genomic data associated with a tissue sample of the patient into a plurality of genomic data groups; extracting a plurality of genomic embeddings from the plurality of genomic data groups; generating a slide-level genomic embedding by applying an attention module of the20 prediction system to the genomic embeddings such that the slide-level genomic embedding inherently includes image-based data corresponding to the tissue sample, which is not input to the prediction system; and generating the prediction based on the slide-level genomic embedding.

[0018] In some embodiments, each one of the genomic data groups includes a25 corresponding group of genomic data that are clinically relevant, and wherein each one of the genomic embeddings corresponds to one of the genomic data groups.

[0019] In some embodiments, the extracting the plurality of genomic embeddings is performed by a plurality of self-normalizing networks (SNNs) of the prediction system, and each one of the SNNs corresponds to one group of the genomic data30 groups.

[0020] In some embodiments, the plurality of SNNs and the attention module are trained to extract the image-based data from the plurality of genomic data groups and to embed the plurality of genomic embeddings and the slide-level genomic embedding with the image-based data.35

[0021] In some embodiments, the attention module includes: a transformer configured to generate a plurality of group-level embeddings based on pair-wise relationships between each one of genomic embeddings and other ones of the genomic embeddings; and an attention pooling network configured to generate thePCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11 slide-level genomic embedding based on a weighted average of the group-level embeddings.

[0022] In some embodiments, a number of the SNNs is a same as a number of groups of the genomic data groups, and each one of the genomic embeddings, each5 one of the group-level embeddings, and the slide-level genomic embedding is a onedimensional vector of a same length.

[0023] In some embodiments, the generating the prediction includes: aggregating the slide-level genomic embedding and a slide-level image embedding to generate an aggregated slide-level embedding; and classifying the aggregated slide-level10 embedding to determine the prediction, wherein the slide-level image embedding includes data corresponding to a whole slide image of the tissue sample of the patient.

[0024] According to some embodiments of the present disclosure, there is provided a method of pre-training a prediction system, the method including:15 extracting a plurality of image embeddings from a plurality of image tiles corresponding to a whole slide image of a tissue sample of a patient; clustering the plurality of image embeddings into a plurality of image data groups based on a degree of similarity between the image embeddings; generating a plurality of weighted image embeddings by applying a plurality of first attention modules of the20 prediction system to the image embeddings of the plurality of image data groups; generating a slide-level image embedding by applying a second attention module of the prediction system to the weighted image embeddings; clustering genomic data associated with the patient into a plurality of genomic data groups; extracting a plurality of genomic embeddings from the plurality of genomic data groups; and25 generating a slide-level genomic embedding by applying a third attention module of the prediction system to the genomic embeddings; and aligning the slide-level image embeddings and the slide-level genomic embeddings to increase cross-modal interaction by applying a pre-training loss function to the slide-level image and genomic embeddings.30

[0025] In some embodiments, the whole slide image and the genomic data correspond to a same tissue sample from the patient.

[0026] In some embodiments, each one of the genomic data groups includes a corresponding group of genomic data that are clinically relevant, each one of the genomic embeddings corresponds to one of the genomic data groups, and the35 image data groups has a same number of groups as the genomic data groups.

[0027] In some embodiments, the aligning the slide-level image and genomics embeddings is such that the slide-level image embedding inherently includes genomic information corresponding to the tissue sample and the slide-level genomicPCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11 embedding inherently includes image-based information corresponding to the tissue sample.

[0028] In some embodiments, the pre-training loss function includes a meansquare error function.5

[0029] In some embodiments, the method further includes: storing model weights of the prediction system for subsequent fine-tuning of the prediction system.

[0030] According to some embodiments of the present disclosure, there is provided a prediction system including: a processor; and a memory coupled to the processor, the memory storing instructions that, when executed by the processor,10 cause the processor to perform: extracting a plurality of image embeddings from a plurality of image tiles corresponding to a whole slide image of a tissue sample of a patient; clustering the plurality of image embeddings into a plurality of image data groups based on a degree of similarity between the image embeddings; generating a plurality of weighted image embeddings by applying a plurality of first attention15 modules of the prediction system to the image embeddings of the plurality of image data groups; generating a slide-level image embedding by applying a second attention module of the prediction system to the weighted image embeddings; clustering genomic data associated with the patient into a plurality of genomic data groups; extracting a plurality of genomic embeddings from the plurality of genomic20 data groups; and generating a slide-level genomic embedding by applying a third attention module of the prediction system to the genomic embeddings; and aligning the slide-level image embeddings and the slide-level genomic embeddings to increase cross-modal interaction by applying a pre-training loss function to the slidelevel image and genomic embeddings.25BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Non-limiting and non-exhaustive embodiments according to the present disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise30 specified.

[0032] FIG. 1 is a block diagram illustrating a prediction system, according to some embodiments of the present disclosure.

[0033] FIG. 2 is a block diagram illustrating the whole slide image (WSI) processor and its operation, according to some embodiments of the present35 disclosure.

[0034] FIG. 3A is a block diagram illustrating the internal structure of the primary prediction network of the prediction system and the pre-training stage of the trainingPCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11 process of the prediction system, according to some embodiments of the present disclosure.

[0035] FIG. 3B is a block diagram illustrating the internal structure of the secondary prediction network of the prediction system and the fine-tuning stage of5 the training process of the prediction system, according to some embodiments of the present disclosure.

[0036] FIG. 4 is a flow diagram illustrating a process of generating a prediction 30 associated with patient by the prediction system based in image data alone, according to some embodiments of the present disclosure.10

[0037] FIG. 5 is a flow diagram illustrating a process of generating a prediction 30 associated with patient by the prediction system based in genomic data alone, according to some embodiments of the present disclosure.

[0038] FIG. 6 is a flow diagram illustrating a process of pre-training the prediction system, according to some embodiments of the present disclosure.15DETAILED DESCRIPTION

[0039] Hereinafter, aspects of some example embodiments will be described in more detail with reference to the accompanying drawings, in which like reference numbers refer to like elements throughout. The present invention, however, may be20 embodied in various different forms, and should not be construed as being limited to only the illustrated embodiments herein. Rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the aspects and features of the present invention to those skilled in the art. Accordingly, processes, elements, and techniques that are not necessary to those25 having ordinary skill in the art for a complete understanding of the aspects and features of the present invention may not be described. Unless otherwise noted, like reference numerals denote like elements throughout the attached drawings and the written description, and thus, descriptions thereof will not be repeated. In the drawings, the relative sizes of elements, layers, and regions may be exaggerated for30 clarity.

[0040] In the field of computational pathology, the integration of histology images and genomic data has shown significant promise in enhancing prognostic models for cancer diagnosis and treatment. However, existing methods often require both modalities to be available at the same time, which poses a challenge due to the high35 cost and limited availability of comprehensive datasets. This limitation hinders the practical application of these models in clinical settings, where genomic data may not be accessible in all cases. Furthermore, current approaches may not fully leverage the potential of each modality to improve prediction accuracy, as they often treatPCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11 image and genomic data as separate entities rather than as complementary sources of information.

[0041] In this domain, current attempts may involve the use of multimodal models that integrate imaging and genomic data to enhance cancer prognosis. While these5 models may show improved performance over single-modality approaches, they are often constrained by the need for both types of data to be present during inference. This requirement limits their applicability in real-world scenarios where genomic data might be unavailable. Additionally, many current models do not adequately address the heterogeneity of cancer, which can lead to suboptimal predictions. The reliance10 on traditional machine learning techniques, such as random grouping of data, further limits the ability to capture the complex interactions between different data types.

[0042] The present disclosure addresses these challenges by introducing a novel multimodal pre-training framework that enhances the integration of histology images and genomic data. This framework employs advanced clustering techniques to15 refine the representation of both modalities, allowing for the generation of robust joint embeddings. By leveraging models pre-trained on both pathology images and genomic data, the framework facilitates the learning of cross-modal information, enabling the model to perform effectively even when only one modality is available during inference. This approach not only improves the accuracy of predictions but20 also enhances the model's adaptability to various downstream tasks, such as survival analysis and gene mutation prediction. The framework's ability to generate genomics-enhanced image representations from image data alone represents a significant advancement in the field, offering a practical solution for clinical applications where comprehensive data is not consistently available.25

[0043] FIG. 1 is a block diagram illustrating a prediction system 100, according to some embodiments of the present disclosure.

[0044] The prediction system 100 is a comprehensive framework capable of integrating and processing both slide image data 10 and genomic data 20 associated with a patient to generate predictions related to a patient outcome. In some30 examples, the prediction system 100 is responsible for managing the input data, processing the information through a series of networks, and producing a prediction that can be utilized for clinical decision-making.

[0045] In some embodiments, the prediction system 100 includes a primary prediction network 200 and a secondary prediction network 300. The primary35 prediction network 200 utilizes machine learning techniques, such as deep learning models and attention mechanisms, to process whole slide image (WSI) data and genomic data (e.g., transcriptom ic information such as RNA sequencing) and to extract meaningful patterns and relationships from the input data in the form ofPCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11 image and / or genomic embeddings. The secondary prediction network 300, which serves as an additional layer of analysis within the prediction system 100, then generates a prediction 30 based on the image and / or genomic embeddings generated by the primary prediction network 200.5

[0046] According to some embodiments, the primary prediction network 200 is separately (e.g., independently) trained (e.g., pretrained) on multimodal data (e.g., WSI and genomic data) during a pre-training stage of the training process using unsupervised learning, and the primary and secondary prediction networks 200 and 300 are then jointly trained on either single modality data or multimodal data (e.g.,10 (e.g., WSI and / or genomic data) during a fine-tuning stage of the training process using supervised learning. The pre-training of the primary prediction network 200 involves learning joint embeddings for all modalities using unsupervised learning, which allows the primary prediction network 200 to identify features of one modality in data of another modality. This enables the overall framework to function15 effectively even when only one type of data is available, thereby enhancing its applicability in real-world clinical settings. The fine-tuning stage may employ supervised learning that is tailored to each downstream task (e.g., cancer detection, survival prediction, etc.).

[0047] This prediction 30 may be a quantitative or qualitative assessment of a20 patient's condition, such as a prognosis or risk score, which can be used to inform clinical decisions. The prediction 30 may be interpretable and actionable, providing healthcare professionals with insights that can guide treatment planning and patient management. In some examples, the prediction 30 may be a multi-level output (e.g., ‘O’, T, ‘2’, etc.) indicating a classification category for which the prediction system25 100 is trained (such as lung cancer subtype categories). In other examples, prediction 30 may be a confidence level or probability associated with the various classification categories (e.g., lung cancer subtype categories). However, these are merely examples, and embodiments of the present disclosure are not limited thereto. The prediction 30 may be provided to a display device 40 and / or a server 50.30

[0048] The display device 40 may be an interface component that presents the prediction 30 to the end-user, such a healthcare professional (e.g., a clinician or pathologist). This device is responsible for visualizing the prediction in a clear and accessible manner, allowing users to easily interpret and act upon the information provided. The display device 40 may include features such as graphical35 representations, interactive dashboards, or alerts, which enhance the usability and effectiveness of the prediction system 100 in a clinical setting.

[0049] The server (e.g., remote or cloud server) 50 may be a backend component that supports the operation of the prediction system 100, providing computationalPCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11 resources and data storage capabilities. The server 50 may be responsible for managing the flow of data between the various components of the system, and may also facilitate remote access to the prediction system 100, thus allowing users to interact with the system from different locations and devices.5

[0050] In some embodiments, the prediction system 100 also includes a WSI processor 110 that is configured to preprocess the WSI data 10 to ensure uniformity in the tiles that are supplied to the segmentation and classification block 120. Given that different labs that generate whole slide images based on tissue samples may use different stainers and / or settings, the resulting WSIs produced by such labs may10 have different stains (e.g., different colorations). Therefore, in some embodiments, the WSI processor 110 performs stain normalization, that is, standardizes the stains across all tiles, and generates a plurality of normalized tiles that are then passed onto the primary prediction network 200 for further analysis and processing. The WSI processor 110 may also perform the function of extracting tiles from an original15 WSI.

[0051] However, embodiments of the present disclosure are not limited thereto. For example, one or more functions of the WSI processor 110 may be omitted from this component and integrated into other component blocks, or omitted from the prediction system 100 altogether. For example, stain normalization may be omitted20 from the WSI processor 110 and the function may be integrated into the input stage of primary prediction network 200. Further, the stain normalization function may be omitted from the prediction system 100 and the primary prediction network 200 may operate on the raw tiles with potentially different staining profiles.

[0052] FIG. 2 is a block diagram illustrating the WSI processor and its operation,25 according to some embodiments of the present disclosure.

[0053] In some examples, the WSI data 10 includes a WSI 11 and a region-of- interest (ROI) map 12 that identifies the potential areas of interest (e.g., potential tumor regions) of the WSI 11 that are relevant to the analysis of the prediction system 100. The ROI map 12 may be manually annotated by a pathologist.30

[0054] In some examples, the ROI map 12 may be generated by applying a series of filters to the WSI 11 . The filters may include at least one of a background filter, an out-of-focus filter, a crush filter, a pen mark filter, a hemorrhage filter, a necrosis filter, a fat tissue filter, or non-lymphoid filter.

[0055] Each of these filters may represents a function parametrized by a35 convolutional neural network (CNN) that takes a WSI or a portions thereof as input and returns a single Boolean value as output. The CNN model underlying each filter may be trained to identify a specific histologic concept in WSIs and to classify its portions depending on whether the concept is present or absent. An output of zeroPCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11 may mean that the component did not identify the concept in a given WSI portion, and an output of one may means that the filter did identify the concept in the WSI portion.

[0056] The application of the above noted filters produces an analysis region of5 interest as output. This region that may be continuous or may be spread out in multiple parts, i.e. , not continuously connected. In some examples, the region identified by the ROI map 12 includes areas enriched with lymphoid elements (e.g., lymphoma tissue, lymph node parenchyma, lymphoid-rich stroma, lymphoid aggregates) that may be encountered in nodal and extranodal anatomic sites. The10 ROI may be free of artifacts and non-lymphoid tissue (e.g., be free of background regions, out-of-focus regions, crush tissue, pen mark regions, hemorrhage tissue, necrotic tissue, and fat tissue). In some examples, the ROI map 12 may be further examined and modified by a human user (e.g., pathologist) as desired.

[0057] In some embodiments, the WSI processor 110 includes a tile extractor 11215 and a stain normalizer 114.

[0058] The tile extractor 112 may apply the ROI map 12 to (e.g., overlay the ROI map 12 onto) the WSI 11 to identify regions of interest in the WSI 11 and to then extract a plurality of non-overlapping tiles 113 of equal size from the regions of interest in the WSI 11. In some examples, the tile extractor 112 may also extract20 tiles from the WSI 11 and discard those tiles that do not fall within the ROI (e.g., tiles that have greater than 10% overlap with non-ROI regions). In some examples, the tile extractor 112 may extract a number of (e.g., more than 10,000) non-overlapping tiles of a particular size (e.g., 224x224, 512x512, or 1024x1024 pixels) from the WSI 11 . The tile size may be chosen to ensure that each tile contains a sufficient number25 of cells to derive robust cellular feature statistics.

[0059] To accommodate for the different stains that the tiles 113 may exhibit, the stain normalizer 114 standardizes the stains across the plurality of tiles to generate a plurality of normalized image tiles 115 that that have a uniform stain irrespective of the stain used in the WSI 11.30

[0060] In some embodiments, the stain normalizer 114 includes a first model, which may utilize a ll-Net architecture having a neural network (e.g., a convolutional neural network) that expresses an input image in short form as a vector and then upscales the image in the desired (e.g., standardized) stain. However, embodiments of the present disclosure are not limited thereto, and the first model of the stain35 normalizer 114 may use any suitable architecture.

[0061] The stain normalizer 114 provides the normalized tiles 115 to the primary prediction network 200 for further processing.PCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11

[0062] FIG. 3A is a block diagram illustrating the internal structure of the primary prediction network 200 of the prediction system 100 and the pre-training stage of the training process of the prediction system 100, according to some embodiments of the present disclosure.5

[0063] Referring to FIG. 3A, the primary prediction network 200 is a dual-stream model including an image processing pipeline 210 for processing image data, and a genomic processing pipeline 220 for processing genomic data (e.g., transcriptom ic information such as RNA sequencing). The image and genomic processing pipelines 210 and 220 may process the image data 10 (in the form of image tiles10 115) and the genomic data 20 in parallel (e.g., concurrently or simultaneously) or process them serially.

[0064] According to some embodiments, the image processing pipeline 210 includes a feature extractor 212, a tile clustering block 214, a plurality of first attention modules 216, and a second attention module 218.15

[0065] The feature extractor 212 processes the image tiles 115 to identify and extract a plurality of image embeddings (e.g., 384 1 -dimensional vectors), which are meaningful features from the image data that indicate the underlying biological structures. In so doing, the feature extractor 212 transform the image data into compact representations that can be effectively utilized by subsequent components20 in the primary prediction network 200. In some examples, the feature extractor 212 may employ advanced machine learning models, such as convolutional neural networks (CNNs), to perform automatic feature extraction. The feature extractor 212 may be an advances vision transformer model that is pretrained on pathology images.25

[0066] In some embodiments, the tile clustering block 214 clusters / organizes the plurality of image embeddings into a plurality of image data groups / clusters 215 based on a degree of similarity between the image embeddings. In some examples, the tile clustering block 214 organizes the image embeddings into 8 groups based on their morphological features. For example, embeddings of similar tiles from a30 tumorous region are grouped together, and ones from a non-tumorous region are grouped together. Grouping similar embeddings together (as opposed to grouping them randomly) helps to reduce the complexity of the data and enhances the ability of the primary prediction network 200 to focus on the most relevant patterns and structures within the image data. According to some embodiments, the clustering is35 performed using a k-means algorithm, which partition the embeddings into a predefined number of groups / clusters 215 (e.g., 8 clusters) based on their similarity. K-means algorithm is an iterative, centroid-based clustering method that partitions a dataset into similar groups based on the distance between their centroids. ThePCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11 centroid, or cluster center, represents either the mean or median of all the points within a cluster depending on the characteristics of the data being grouped. In some examples, the number of embeddings in each cluster may not be the same (i.e. , the clusters may contain different number of embeddings).5

[0067] In some embodiments, each one of the plurality of first attention modules216 applies an attention mechanism to the image embeddings of a corresponding one of groups / clusters 215 to generate a weighted image embedding (e.g., a 1 *256 feature vector) 217. The first attention modules 216 have a one-to-one correspondence with the plurality of image data groups / clusters (e.g., there are the10 same number of first attention modules 216 as image data groups 215). Each of the first attention modules 216 assigns different weights to the image embeddings within a corresponding image data groups / cluster 215 in a manner that emphasizes the most informative features within the group / cluster. In some examples, the first attention modules 216 may be multi head self-attention (MHSA) network. The15 attention mechanism allows the system to focus on specific parts of the image data that are most relevant to the prediction task. The set of weighted image embeddings217 are then passed to the second attention module 218 for further processing and generation of slide-level image embeddings 219.

[0068] According to some embodiments, the second attention module 21820 includes a first transformer 218a and a first attention pooling network (or first attention pooling block) 218b. The first transformer 218a generates a plurality of tilelevel embeddings (e.g., eight 1 *256 feature vectors) based on the pair-wise relationships between each one of weighted image embeddings 217 and other ones of the weighted image embeddings 217. The first transformer 218a may utilize self¬25 attention mechanisms to model the relationships between different parts of the image data, capturing complex interactions and dependencies. The first attention pooling network 218b aggregates the outputs of the first transformer 218a (i.e., applies a pooling operation that combines the information from all the tile-level embeddings) into a single slide-level image embedding 219. In so doing, the first30 attention pooling network 218b generates the slide-level image embedding (e.g., a 1 *256 feature vector) 219 based on a weighted average of the tile-level embeddings produced by the first transformer 218a. The attention weights applied to the tile-level embeddings ensure that the most relevant features are emphasized in the final slidelevel image embedding 219. As a result, the slide-level image embedding 21935 encapsulates and emphasizes the fundamental features of the entire WSI.

[0069] In some examples, each one of the image embeddings, each one of the weighted image embeddings 217, each one of the tile-level embeddings, and thePCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11 slide-level image embedding 219 is a one-dimensional vector of the same length (of, e.g., 256).

[0070] According to some embodiments, the genomic processing pipeline 220 includes a gene selector 224, a plurality of self-normalizing networks (SNNs) 226,5 and third attention module 228.

[0071] The gene selector 224 identifies and selects relevant genes from the genomic data that are associated with the patient, and clusters them into a plurality of genomic data groups / clusters 215 based on their similarity and relevance. For example, the genomics data (e.g., transcriptom ic data) 20 may include over a 100010 genes, and the gene selector 224 may select the ones (e.g., a few hundred) that are most relevant to cancer development and group the related ones together to obtain eight clinically relevant (e.g., “hallmarks of cancer”) gene groups 225. In some examples, these may be clustered based on the conceptualization of human cell functional capabilities acquired via mechanisms of cancer development and15 malignant progression.

[0072] In some embodiments, each one of the SNNs 226 extracts a genomic embedding (e.g., a 1 *256 feature vector) from a corresponding one of the plurality of genomic data groups / clusters 225, which captures the fundamental genomic information in that group / cluster 225. In effect, the SNNs 226 transform the20 clustered genomic data 20 into a format suitable for further analysis. Here, the SNNs 226 have a one-to-one correspondence with the plurality of genomic data groups / clusters (e.g., there are the same number of SNNs 226 as genomic data groups 225).

[0073] According to some embodiments, the third attention module 228 includes25 a second transformer 228a and a second attention pooling network (or second attention pooling block) 228b, which operate similar to those described above with respect to the image processing pipeline 210. The second transformer 228a generates a plurality of group-level embeddings based on pair-wise relationships between each one of genomic embeddings 227 and other ones of the genomic30 embeddings 227. The second transformer 228a may utilize self-attention mechanisms to model the relationships between different parts of the genomic data, capturing complex interactions and dependencies. The second attention pooling network 228b aggregates the outputs of the second transformer 228a (i.e. , applies a pooling operation that combines the information from all the group-level embeddings)35 into a single slide-level genomic embedding (e.g., a 1 *256 feature vector) 229. In so doing, the second attention pooling network 228b generates the slide-level genomic embedding 229 based on a weighted average of the group-level embeddings produced by the second transformer 228a. The attention weights applied to thePCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11 group-level embeddings ensure that the most relevant features are emphasized in the final slide-level genomic embedding 229.

[0074] In some embodiments, when in pre-training mode, the outputs of the second and third attention modules 218 and 228 are fed to a loss function 230,5 which is configured to align the slide-level image and genomic embeddings 219 and 229 by minimizing a loss function, such as mean square error (MSE), which measures the discrepancy between the two slide-level embeddings 219 and 229. The aligned embeddings 219 and 229 are then used in the fine-tuning stage to generate predictions based on the integrated image and genomic data.10

[0075] During the pre-training phase, the model parameters (e.g., weights) of some or all of the components of the prediction system 100 may be adjusted based on the loss function minimization. In some examples, the feature extractor 212 may be a frozen network (i.e. , not be adjusted during pre-training), while the remaining components may be trainable. Once pre-trained, the model weights of the prediction15 system 100 are stored for subsequent fine-tuning of the prediction system.

[0076] After the pre-training phase when the two slide-level embeddings 219 and 229 are aligned, the output of the image processing pipeline 210, which is sourced solely by the image data 10 (in the form of image tiles 115), inherently includes / is imbued / infused with the genomic information from the tissue sample that the image20 data is sourced from. In other words, the first attention modules 216 and the second attention module 218 are trained (through the pre-training process) to effectively extract the genomic data from the plurality of image tiles 115 and to embed / infuse the plurality of weighted image embeddings 217 and the slide-level image embedding 219 with this genomic data.25

[0077] Similarly, post -alignment of the pre-training process, the output of the genomic processing pipeline 220, which is sourced solely on genomic data 20, inherently includes / is imbued / infused with the image-based information from the tissue sample that the genomic data is sourced from. In other words, the SNNs 226 and the third attention module 228 are trained (through the pre-training process) to30 effectively extract the image data from the plurality of genomic data 20 and to embed / infuse the plurality of genomic embeddings 227 and the slide-level genomic embedding 229 with this image data.

[0078] As described, the pre-training process enhances the model's ability to integrate information from both modalities, improving the model's performance on35 downstream tasks.

[0079] FIG. 3B is a block diagram illustrating the internal structure of the secondary prediction network 300 of the prediction system 100 and the fine-tuningPCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11 stage of the training process of the prediction system 100, according to some embodiments of the present disclosure.

[0080] In some embodiments, the secondary prediction network 300 includes an aggregator 310 and a classifier 320, which respectively aggregate and generate a5 prediction based on the slide-level embeddings 219a and 229.

[0081] The aggregator 310 combines (e.g., concatenates) the slide-level image embedding 219A and the slide-level genomic embedding 229B received from the pre-trained primary prediction network 200 into an aggregated slide-level embedding (e.g., a 1 x512 feature vector). The aggregator 310 ensures that the combined10 embedding retains the most relevant features from both the image and genomic data, facilitating the subsequent classification process.

[0082] The classifier 320 generates a prediction 30 based on the aggregated slide-level embedding produced by the aggregator 310. The classifier 320 is trained to interpret the combined information from both image and genomic data, providing15 outputs that can be used for various clinical decision-making tasks. The classifier 320 may employ machine learning techniques such as a neural network to perform this task. The neural network may include a convolutional neural network (ConvNet / CNN), a recurrent neural network (RNN) with convolution operation, a random forest network, and / or the like.20

[0083] During the fine-tuning stage of the training of the prediction system 100, a fine-tuning loss function 330 may be used to set (e.g., optimize) the model parameters of the aggregator 310 and classifier 320. The fine-tuning loss function 330 may be tailored to the particular requirements of the task for which the prediction system 100 is being utilized, such as survival analysis, gene mutation prediction, or25 biomarker (e.g., immunohistochemistry (IHC)) detection and is applied to the predictions generated by the classifier 320. Through minimizing the fine-tuning loss function 330, the accuracy of the predictive output of the prediction system 100 may be improved. In the example of survival analysis, the fine-tuning loss function 330 may be a survival loss function for optimizing the output of the prediction system30 100. The continuous overall survival timescale may be partitioned into 8 nonoverlapping bins using quantile binning to ensure an even distribution of patients across bins. The negative log-likelihood survival loss may then be used to supervise the training process. In the example of gene mutation prediction, the task may be treated as a classification problem and a cross-entropy loss function may be used as35 the fine-tuning loss function 330 to optimize the output of the prediction system 100.

[0084] Once the fine-tuning training is complete, the prediction system 100 may be used in inference to generate predictions based on WSI and / or genomic data. According to some embodiments, due to the alignment of the slide-level image andPCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11 genomic embeddings 219 and 229 resulting from the pre-training of the primary prediction network 200, inference performed solely using pathology data 10 may generate predictions that are partially derived from and enriched by insight into the genomics information inherent in the pathology data 10. Thus, even in situations5 where the genomic data corresponding to some pathology data is not available due to real-world constraints (e.g., due to cost or instrumentation access constraints), the prediction system 100 is capable of producing predictions that are more accurate than those produced by other systems of the related art that rely only on pathology data. Similarly, inference performed solely using genomic data 20 may generate10 predictions that are partially derived from and enriched by insight into the pathology information inherent in the genomic data 20.

[0085] FIG. 4 is a flow diagram illustrating a process 400 of generating a prediction 30 associated with patient by the prediction system 100 based in image data alone, according to some embodiments of the present disclosure.15

[0086] During the inference stage, the prediction system 100 extracts a plurality of image embeddings from a plurality of image tiles 115 corresponding to a whole slide image 10 of a tissue sample of the patient (S402). The extraction of the plurality of image embeddings may be performed by a feature extractor 212 trained on pathology images.20

[0087] The prediction system 100 may cluster the plurality of image embeddings into a plurality of image data groups 215 based on a degree of similarity between the image embeddings (S404). The clustering may include applying a k-means algorithm to the plurality of image embeddings to generate the plurality of image data groups 215.25

[0088] The prediction system 100 may generate a plurality of weighted image embeddings 217 by applying the first attention modules 216 to the image embeddings of the plurality of image data groups 215 (S406), such that each one of the weighted image embeddings 217 corresponds to one of the image data groups 215.30

[0089] The prediction system 100 may the generate a slide-level image embedding 219 by applying a second attention module 218 of the prediction system to the weighted image embeddings 217 such that the slide-level image embedding 219 inherently includes genomic-based data corresponding to the tissue sample, which is not input to the prediction system (S408). In some examples, the first35 attention modules 216 and the second attention module 218 are trained to extract the genomic-based data from the plurality of image tiles 115 and to embed the plurality of weighted image embeddings 217 and the slide-level image embedding with the genomic-based data.PCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11

[0090] The prediction system 100 may then generate the prediction based on the slide-level image embedding (S410).

[0091] FIG. 5 is a flow diagram illustrating a process 500 of generating a prediction 30 associated with patient by the prediction system 100 based in genomic5 data alone, according to some embodiments of the present disclosure.

[0092] During the inference stage, the prediction system 100 may cluster genomic data 20 associated with the patient into a plurality of genomic data groups225 (S502). Each one of the genomic data groups 225 may include a corresponding group of genomic data 20 that are clinically relevant, and each one of the genomic10 embeddings may correspond to one of the genomic data groups 225.

[0093] The prediction system 100 may extract a plurality of genomic embeddings from the plurality of genomic data groups (S504). The extraction of the genomic embeddings may be performed by a plurality of self-normalizing networks (SNNs)226 of the prediction system 100, and each one of the SNNs 226 may corresponds15 to one group of the genomic data groups 225. The number of the SNNs 226 may be the same as a number of groups of the genomic data groups 225, and each one of the genomic embeddings, each one of the group-level embeddings, and the slidelevel genomic embedding 229 may be a one-dimensional vector of the same length.

[0094] The prediction system 100 may generate a slide-level genomic embedding20 229 by applying the second attention module 228 of the prediction system 100 to the genomic embeddings 227 such that the slide-level genomic embedding 229 inherently includes image-based data corresponding to the tissue sample, which is not input to the prediction system (S506).

[0095] Here, the SNNs 226 and the second attention module 228 may be trained25 to extract image-based data from the plurality of genomic data groups and to embed the plurality of genomic embeddings and the slide-level genomic embedding with the image-based data.

[0096] The prediction system 100 may then generate the prediction 30 based on the slide-level genomic embedding 229 (S508).30

[0097] FIG. 6 is a flow diagram illustrating a process 600 of pre-training the prediction system 100, according to some embodiments of the present disclosure.

[0098] During pre-training, the prediction system 100 extracts a plurality of image embeddings from a plurality of image tiles 115 corresponding to a whole slide image 10 of a tissue sample of the patient (S602).35

[0099] The prediction system 100 may cluster (e.g., using a K-mean process) the plurality of image embeddings into a plurality of image data groups 215 based on a degree of similarity between the image embeddings (S604).PCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11

[0100] The prediction system 100 may generate a plurality of weighted image embeddings 217 by applying the first attention modules 216 to the image embeddings of the plurality of image data groups 215 (S606).

[0101] The prediction system 100 may the generate a slide-level image5 embedding 219 by applying a second attention module 218 of the prediction system to the weighted image embeddings 217 (S608).

[0102] the prediction system 100 may cluster genomic data 20 associated with the patient into a plurality of genomic data groups 225 (S610). Each one of the genomic data groups 225 may include a corresponding group of genomic data that10 are clinically relevant.

[0103] The prediction system 100 may extract a plurality of genomic embeddings from the plurality of genomic data groups (S612).

[0104] The prediction system 100 may generate a slide-level genomic embedding 229 by applying a third attention module 228 of the prediction system 100 to the15 genomic embeddings (S614).

[0105] The prediction system 100 may then align the slide-level image embeddings and the slide-level genomic embeddings to increase cross-modal interaction by applying a pre-training loss function (e.g., a mean-square error function) to the slide-level image and genomic embeddings (S616). In some20 examples, the aligning the slide-level image and genomics embeddings is such that the slide-level image embedding 219 inherently includes genomic information corresponding to the tissue sample and the slide-level genomic embedding inherently includes image-based information corresponding to the tissue sample.

[0106] Once aligned, the prediction system 100 may store the model weights for25 subsequent fine-tuning of the prediction system 100.

[0107] Accordingly, as described above, some embodiments of the present disclosure provide a two-stage pathology-gene multimodal framework. The first stage involves unsupervised pre-training, aiming to align genomic and image embeddings and facilitate the learning of cross-modal information. The second stage30 involves supervised fine-tuning using labels specific for the downstream tasks. According to some embodiments, the pre-training phase addresses the heterogeneity between image and genomic features by employing a loss function to align their embeddings. By minimizing the mean square error (MSE) loss, the image and genomics embeddings are brought closer in the same latent space, thus35 enhancing their interactions. This pre-training strategy allows the prediction system 100 to capture intrinsic relationships between the different data types effectively. In some embodiments, during the fine-tuning stage, the prediction system 100 can utilize either single-modality or multimodal data, enhancing its versatility for variousPCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11 downstream tasks. For multimodal fine-tuning, the prediction system 100 concatenates slide-level co-representations from both image and genomics data, using a risk classifier for tasks such as survival stratification or a classification loss function for gene mutation prediction. In single-modality fine-tuning, the pre-trained5 slide-level co-representation from either image or genomic data can be directly used for prediction. Model parameters are updated using an appropriate loss function tailored to the specific downstream task. The fine-tuning stage benefits significantly from the pre-training phase. For example, even when only image data is available, the pre-trained prediction system 100 can generate genomics-enhanced image10 representation that carry the multimodal knowledge acquired during pre-training, removing the dependency on having both modalities during fine-tuning and inferencing.

[0108] According to various embodiments of the present disclosure, the prediction system 100 is implemented using one or more processing circuits or electronic15 circuits configured to perform various operations as described above. Types of electronic circuits may include a central processing unit (CPU), a graphics processing unit (GPU), an artificial intelligence (Al) accelerator (e.g., a vector processor, which may include vector arithmetic logic units configured efficiently perform operations common to neural networks, such dot products and softmax), a20 field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a digital signal processor (DSP), or the like. For example, in some circumstances, aspects of embodiments of the present disclosure are implemented in program instructions that are stored in a non-volatile computer readable memory where, when executed by the electronic circuit (e.g., a CPU, a GPU, an Al25 accelerator, or combinations thereof), perform the operations described. The operations performed by the virtual pathology system 1 may be performed by a single electronic circuit (e.g., a single CPU, a single GPU, or the like) or may be allocated between multiple electronic circuits (e.g., multiple GPUs or a CPU in conjunction with a GPU). The multiple electronic circuits may be local to one another30 (e.g., located on a same die, located within a same package, or located within a same embedded device or computer system) and / or may be remote from one other (e.g., in communication over a network such as a local personal area network such as Bluetooth®, over a local area network such as a local wired and / or wireless network, and / or over wide area network such as the internet, such a case where35 some operations are performed locally and other operations are performed on a server hosted by a cloud computing service). One or more electronic circuits operating to implement the virtual pathology system 1 may be referred to herein as a computer or a computer system, which may include memory storing instructions that,PCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11 when executed by the one or more electronic circuits, implement the systems and methods described herein.

[0109] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present invention. As5 used herein, the singular forms “a” and “an” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” "includes," and "including," when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or10 addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. Expressions such as “at least one of,” when preceding a list of elements, modify the entire list of elements and do not modify the individual elements of the list.15

[0110] It will be understood that, although the terms “first,” “second,” “third,” etc., may be used herein to describe various elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms are used to distinguish one element, component, region, layer or section from another element, component, region, layer20 or section. Thus, a first element, component, region, layer or section described below could be termed a second element, component, region, layer or section, without departing from the spirit and scope of the present invention.

[0111] As used herein, the term "substantially," "about," and similar terms are used as terms of approximation and not as terms of degree, and are intended to25 account for the inherent deviations in measured or calculated values that would be recognized by those of ordinary skill in the art. Further, the use of “may” when describing embodiments of the present invention refers to “one or more embodiments of the present invention.” As used herein, the terms "use," "using," and "used" may be considered synonymous with the terms "utilize," "utilizing," and30 "utilized," respectively.

[0112] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should35 be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and / or the present specification, and should not be interpreted in an idealized or overly formal sense, unless expressly so defined herein.PCT / US25 / 40905 06 August 2025 (06.08.2025)274396WO - - P39492-WO-11

[0113] Although aspects of some example embodiments of the prediction system have been described and illustrated herein, various modifications and variations may be implemented, as would be understood by a person having ordinary skill in the art, without departing from the spirit and scope of embodiments according to the present5 disclosure. Accordingly, it is to be understood that a pathology slide manufacturing system and method according to the principles of the present disclosure may be embodiment other than as specifically described herein. The disclosure is also defined in the following claims, and equivalents thereof.101520253035

Claims

WHAT IS CLAIMED IS:1 . A method of generating a prediction corresponding to a patient by a prediction system, the method comprising: extracting a plurality of image embeddings from a plurality of image tiles corresponding to a whole slide image of a tissue sample of the patient; clustering the plurality of image embeddings into a plurality of image data groups based on a degree of similarity between the image embeddings; generating a plurality of weighted image embeddings by applying a plurality of first attention modules of the prediction system to the image embeddings of the plurality of image data groups; generating a slide-level image embedding by applying a second attention module of the prediction system to the weighted image embeddings such that the slide-level image embedding inherently comprises genomic-based data corresponding to the tissue sample, which is not input to the prediction system; and generating the prediction based on the slide-level image embedding.

2. The method of claim 1 , wherein the extracting the plurality of image embeddings is performed by a feature extractor trained on pathology images.

3. The method of claim 1 , wherein the clustering the plurality of image embeddings comprises applying a k-means algorithm to the plurality of image embeddings to generate the plurality of image data groups.

4. The method of claim 1 , wherein each one of the weighted image embeddings corresponds to one of the image data groups.

5. The method of claim 1 , wherein the plurality of first attention modules and the second attention module are trained to extract the genomic-based data from the plurality of image tiles and to embed the plurality of weighted image embeddings and the slide-level image embedding with the genomic-based data.

6. The method of claim 1 , wherein the second attention module comprises: a transformer configured to generate a plurality of tile-level embeddings based on pair-wise relationships between each one of weighted image embeddings and other ones of the weighted image embeddings; andan attention pooling network configured to generate the slide-level image embedding based on a weighted average of the tile-level embeddings.

7. The method of claim 6, wherein a number of the first attention modules is a same as a number of groups of the image data groups, and wherein each one of the image embeddings, each one of the weighted image embeddings, each one of the tile-level embeddings, and the slide-level image embedding is a one-dimensional vector of a same length.

8. The method of claim 1 , wherein the generating the prediction is performed by a classifier trained to generate the prediction based on the slide-level image embedding.

9. The method of claim 1 , wherein the generating the prediction comprises: aggregating the slide-level image embedding and a slide-level genomic embedding to generate an aggregated slide-level embedding; and classifying the aggregated slide-level embedding to determine the prediction, wherein the slide-level genomic embedding comprises genomic-based data corresponding to the patient.

10. A method of generating a prediction corresponding to a patient by a prediction system, the method comprising: clustering genomic data associated with a tissue sample of the patient into a plurality of genomic data groups; extracting a plurality of genomic embeddings from the plurality of genomic data groups; generating a slide-level genomic embedding by applying an attention module of the prediction system to the genomic embeddings such that the slide-level genomic embedding inherently comprises image-based data corresponding to the tissue sample, which is not input to the prediction system; and generating the prediction based on the slide-level genomic embedding.11 . The method of claim 10, wherein each one of the genomic data groups comprises a corresponding group of genomic data that are clinically relevant, and wherein each one of the genomic embeddings corresponds to one of the genomic data groups.

12. The method of claim 10, wherein the extracting the plurality of genomic embeddings is performed by a plurality of self-normalizing networks (SNNs) of the prediction system, and wherein each one of the SNNs corresponds to one group of the genomic data groups.

13. The method of claim 12, wherein the plurality of SNNs and the attention module are trained to extract the image-based data from the plurality of genomic data groups and to embed the plurality of genomic embeddings and the slide-level genomic embedding with the image-based data.

14. The method of claim 12, wherein the attention module comprises: a transformer configured to generate a plurality of group-level embeddings based on pair-wise relationships between each one of genomic embeddings and other ones of the genomic embeddings; and an attention pooling network configured to generate the slide-level genomic embedding based on a weighted average of the group-level embeddings.

15. The method of claim 14, wherein a number of the SNNs is a same as a number of groups of the genomic data groups, and wherein each one of the genomic embeddings, each one of the group-level embeddings, and the slide-level genomic embedding is a one-dimensional vector of a same length.

16. The method of claim 10, wherein the generating the prediction comprises: aggregating the slide-level genomic embedding and a slide-level image embedding to generate an aggregated slide-level embedding; and classifying the aggregated slide-level embedding to determine the prediction, wherein the slide-level image embedding comprises data corresponding to a whole slide image of the tissue sample of the patient.

17. A method of pre-training a prediction system, the method comprising: extracting a plurality of image embeddings from a plurality of image tiles corresponding to a whole slide image of a tissue sample of a patient; clustering the plurality of image embeddings into a plurality of image data groups based on a degree of similarity between the image embeddings;generating a plurality of weighted image embeddings by applying a plurality of first attention modules of the prediction system to the image embeddings of the plurality of image data groups; generating a slide-level image embedding by applying a second attention module of the prediction system to the weighted image embeddings; clustering genomic data associated with the patient into a plurality of genomic data groups; extracting a plurality of genomic embeddings from the plurality of genomic data groups; and generating a slide-level genomic embedding by applying a third attention module of the prediction system to the genomic embeddings; and aligning the slide-level image embeddings and the slide-level genomic embeddings to increase cross-modal interaction by applying a pre-training loss function to the slide-level image and genomic embeddings.

18. The method of claim 17, wherein the whole slide image and the genomic data correspond to a same tissue sample from the patient.

19. The method of claim 17, wherein each one of the genomic data groups comprises a corresponding group of genomic data that are clinically relevant, wherein each one of the genomic embeddings corresponds to one of the genomic data groups, and wherein the image data groups has a same number of groups as the genomic data groups.

20. The method of claim 17, wherein the aligning the slide-level image and genomics embeddings is such that the slide-level image embedding inherently comprises genomic information corresponding to the tissue sample and the slidelevel genomic embedding inherently comprises image-based information corresponding to the tissue sample.21 . The method of claim 17, wherein the pre-training loss function comprises a mean-square error function.

22. The method of claim 17, further comprising: storing model weights of the prediction system for subsequent fine-tuning of the prediction system.

23. A prediction system comprising: a processor; and a memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to perform: extracting a plurality of image embeddings from a plurality of image tiles corresponding to a whole slide image of a tissue sample of a patient; clustering the plurality of image embeddings into a plurality of image data groups based on a degree of similarity between the image embeddings; generating a plurality of weighted image embeddings by applying a plurality of first attention modules of the prediction system to the image embeddings of the plurality of image data groups; generating a slide-level image embedding by applying a second attention module of the prediction system to the weighted image embeddings; clustering genomic data associated with the patient into a plurality of genomic data groups; extracting a plurality of genomic embeddings from the plurality of genomic data groups; and generating a slide-level genomic embedding by applying a third attention module of the prediction system to the genomic embeddings; and aligning the slide-level image embeddings and the slide-level genomic embeddings to increase cross-modal interaction by applying a pre-training loss function to the slide-level image and genomic embeddings.

Images