Methods and systems for detection of glioma infiltration

Abstract

Novel approaches for training medical foundation models are provided, in conjunction with novel techniques for fine-tuning models based on ordinally labeled data, in order to fully leverage expert knowledge and advances in AI technology in addressing a wide range of critically important clinical tasks such as, e.g., residual tumor detection. In addition, in order to ensure that the clinically actionable information obtained using the models of the disclosure is safe, reliable, and trustworthy, methods for generating interpretable visualizations via the trained and / or fine-tuned foundational models are also provided. The methods and systems of the disclosure may be used to rapidly, accurately, and verifiably identify tumor infiltration across a wide range of clinically significant settings.

Description

[0001] METHODS AND SYSTEMS FOR DETECTION OF GLIOMA INFILTRATION

[0002] CROSS-REFERENCE TO RELATED APPLICATION

[0003] This application claims benefit under 35 U.S.C. § 119(e) of provisional application 63 / 709,240, filed October 18, 2024, which application is hereby incorporated by reference in its entirety.

[0004] INTRODUCTION

[0005] The importance of detecting tumor infiltration within surgical specimens during an operation has been recognized for over a century. Through the detection of tumor infiltration in specimens obtained, e.g.. from the resection cavity of a patient, residual tumor may be identified and, if feasible, removed — effectively optimizing the extent of resection and achieving safe maximal tumor removal. This is important as, for solid cancers and brain tumors, residual tumor results in worse quality-of-life, decreased patient survival, and increased burden on healthcare systems. For example, in glioma surgery, extent of resection is known to predict overall patient survival in both lower grade gliomas and glioblastomas. Unfortunately, despite efforts to deliver precision healthcare to cancer patients, rates of residual tumor have not improved over the last two decades. Further, conventional microscopic techniques for residual tumor detection during surgery (such as, e.g., hematoxylin and eosin staining) are slow, resource intensive, and reliant on a shrinking pathology workforce. Accordingly, residual tumor after surgery continues to be a major public health problem within the United States and globally.

[0006] SUMMARY

[0007] Thus, there is a critical need for improved and useful methods and systems for fast and accurate detection of tumor infiltration in freshly sampled surgical tissues. This invention provides such new and useful methods and systems, addressing the limitations mentioned above. To accomplish this, the invention utilizes human domain expertise and cutting-edge imaging technology, as well as recent advances in machine learning techniques, to create novel methods for training medical foundation models for use in addressing a wide range of downstream clinical tasks (including, e.g., residual tumor detection). In order to enable the medical foundation models of the disclosure to fully leverage the insights provided by expertly labeled data, and to generate more intuitive clinically actionable outputs, novel methods of fine-tuning a model based on ordinally labeled data are provided. Further, methods for generating interpretable visualizations via the trained and / or fine-tuned foundational models are also disclosed, ensuring that the clinically actionable information obtained using the models is safe, reliable, and trustworthy. Thus, the methods and systems of the invention, e.g., as described in greater detail below, find use in a variety of applications where it is desirable to quickly, accurately, and verifiably identify tumor infiltration at microscopic resolution — effectively minimizing the risk of residual tumor after surgery resection and improving patient outcomes.

[0008] In one aspect, methods of training a visual machine learning (ML) model (such as, e.g., a medical foundation model) to identify and encode tissue image features are provided. Aspects of the methods include: obtaining a set of tissue images including a plurality of neoplasm and / or tissue types, wherein each tissue image includes multiple patches; training a tokenizer of the model to convert patches into tokens by identifying and encoding patch features using the patches of the tissue image set, wherein the tokenizer includes a convolutional neural network (CNN) and the training includes self- supervised contrastive learning; generating two or more views of each of a plurality of images of the tissue image set, wherein the plurality of images have been passed through the tokenizer; and training an image encoder of the model to identify and encode tissue image features using the two or more views of the plurality of tokenized images, wherein the image encoder includes a vision transformer and the training includes self-supervised learning.

[0009] In certain embodiments, the method further includes dividing images of the tissue image set into multiple patches. In these instances, the tokenizer may be configured to divide each image of the set into a predetermined number and / or layout of patches. For example, each image may be divided (e.g.. via the tokenizer) into non-overlapping rectangular patches having a fixed 300x300-pixel size.

[0010] In certain embodiments, the set of tissue images each include a slide of tissue. In some embodiments, the set of tissue images are obtained via stimulated Raman histology (SRH). In some cases, the set of tissue images are obtained from a plurality of subjects such as, e.g., 500 or more subjects, or 1,000 or more, or 3,000 or more, or 5,000 or more, or 10,000 or more. In some instances, the set of tissue images includes 1,000 or more images such as, e.g., 5,000 or more images, or 10,000 or more, or 50,000 or more, or 100,000 or more. In some embodiments, the set of tissue images includes a plurality of neoplasm types. In these cases, the plurality of neoplasm types may include central nervous system (CNS) tumors, wherein the CNS tumors include one or more of: high-grade gliomas (HGG), low- grade gliomas (LGG), meningiomas, pituitary adenomas, schwannomas, and / or metastatic tumors. In some embodiments, the set of tissue images includes tissues from multiple different organ systems. In these instances, the tissue image set may include neoplasms from multiple different organ systems.

[0011] In certain embodiments, the self-supervised contrastive learning includes hierarchical self- supervised learning via a hierarchical discriminative learning task. For example, the tokenizer may be trained to identify and encode patch features for each level of a multileveled hierarchy, wherein the hierarchy includes a patch level, an image level, and a subject level. In some embodiments, the hierarchical self- supervised learning of the tokenizer includes: generating a plurality of positive patch pairs for each of the patch, image, and subject levels of the hierarchy from the set of tissue images; passing the positive patch pairs through the CNN; and adjusting the CNN to minimize a contrastive learning objective function, wherein the contrastive learning objective function includes patch, image, and subject level components. In some embodiments, the method further includes generating a plurality of negative patch pairs for each of the patch, image, and subject levels of the hierarchy from the set of tissue images and passing the negative patch pairs through the CNN. In some instances, the CNN is a residual CNN such as, e.g., ResNet-18, ResNet-34, ResNet-50. ResNet-101, or ResNet-152.

[0012] In certain embodiments, the two or more views of each of the plurality of images are generated by randomly: splitting each image into mutually exclusive patch sets; spatially cropping each patch set; and masking multiple patches from each spatially cropped patch set. In some embodiments, the self-supervised learning of the image encoder includes: passing each view generated for an image of the plurality of images through a different vision transformer of a Siamese architecture, wherein the transformers of the architecture share weights; and adjusting the transformers of the architecture to minimize a self- supervised objective function, wherein the self-supervised objective function includes variance, invariance, and covariance components. In these cases, the self-supervised objective function may include Variance-Tnvariance-Covariance Regularization (VICReg). In some embodiments, the transformers of the architecture include two or more hidden layers, wherein each hidden layer includes a dimension of 500 or more and multiple attention heads (such as, e.g., a dimension of 512 and 4 attention heads). In some embodiments, the transformers of the architecture include a component for generating positional embeddings such as, e.g.. a Fourier feature positional embedding generator network.

[0013] In certain embodiments, the method further includes fine-tuning the model and / or training an output module of the model to perform one or more specific tasks. In some embodiments, an output module of the model is trained to perform a specific task by passing tissue images through the tokenizer and the image encoder of the model. In some embodiments, the output module includes a K-Nearest Neighbors (KNN) classifier. In some embodiments, the output module includes one or more output layers. For example, the output module may include a one layer linear output layer. In some embodiments, the output module is trained to diagnose brain tumors. In some cases, the output module is trained to quantify tumor infiltration or tumor burden.

[0014] In certain embodiments, the output module is fine-tuned using a second set of tissue images, wherein the second set of tissue images includes a smaller number of images than the initial set of images used to train the tokenizer, the image encoder, and / or the output module of the model. In some embodiments, the image encoder of the model is fine-tuned using the second set of tissue images. In some embodiments, each image of the second set of tissue images includes an ordinal label corresponding to a category of a discrete and ordered scale. For example, the scale may include a tumor infiltration scale quantifying the degree of tumor infiltration in each image of the second set of tissue images. In these instances, the tumor infiltration scale may include four categories, wherein: the first category corresponds to tissue images where no tumor is present; the second category corresponds to tissue images including atypical cells and / or images where it is possible, but not definite, that a tumor is present: the third category corresponds to tissue images including sparse tumor infiltration; and the fourth category corresponds to tissue images including dense tumor infiltration. In some embodiments, the method further includes generating the ordinal labels for the second set of tissue images. In these instances, the ordinal labels may be generated using domain specific knowledge and one or more of: SRH imaging, hematoxylin and eosin (H&E) staining, and / or immunohistochemistry (IHC).

[0015] In certain embodiments, fine-tuning the visual ML model (e.g., the tokenizer, the image encoder, and / or the output module of the model) includes: passing tissue images of the second set through the tokenizer, the image encoder, and the output module of the model; and adjusting the output module and / or the image encoder to: maximize the latent distance between one or more encoded features of tissue images of the second set having different ordinal labels, and / or minimize the latent distance between one or more encoded features of tissue images of the second set having the same ordinal label.

[0016] In certain embodiments, the output module includes an image scorer (such as, e.g., a one layer linear scorer) fine-tuned to output a continuous score corresponding to the discrete and ordered scale of the ordinal labels of the second set of tissue images. In these instances, the fine-tuning may include: passing a batch of tissue images of the second set through the tokenizer, the image encoder, and the image scorer of the model in order to produce a continuous score for each tissue image of the batch; performing a pairwise comparison between all tissue images of the batch using the continuous scores and a binary cross entropy objective function; and adjusting the image scorer and / or the image encoder of the model to minimize the binary cross entropy objective function. In some embodiments, class imbalance is addressed by oversampling tissue images of the second set corresponding to ordinal labels of a minority class. In some instances, the fine-tuned model may be used to detect and quantify tumor infiltration in one or more unprocessed, label-free, surgical samples. In these instances, the one or more surgical samples may be imaged at lower resolutions and / or faster acquisition rates than the tissue images used to train and / or fine-tune the model.

[0017] In certain embodiments, the method further includes generating a visualization identifying one or more regions of interest (ROIs) within a tissue image passed through the model. In some cases, the visualization is generated via principal component analysis (PCA). In some embodiments, the visualization is generated using a set of patch keys including features or characteristics of the one or more ROIs. In these instances, the set of patch keys may include two or more image patches selected by a medical professional. In some embodiments, the set of patch keys includes examples of normal tissue and irregular tissue. For example, the set of patch keys may include diverse examples of normal brain parenchyma and diffuse glioma subtypes. In some instances, regions of interest are identified by comparing the set of patch keys, having passed through the model, with each of the patches of the tissue image passed through the model.

[0018] In certain embodiments, generating the visualization includes: calculating a few-shot visualization score for one or more of the patches of the tissue image, wherein for each of the one or more patches the calculating includes: determining a maximum similarity value between a patch of the tissue image and each of the irregular tissue patch keys by calculating a dot product between the tissue image patch and each of the irregular tissue patch keys; determining a maximum similarity value between the tissue image patch and each of the normal tissue patch keys by calculating a dot product between the tissue image patch and each of the normal tissue patch keys; and calculating a few-shot visualization score for the tissue image patch by determining a difference between the irregular tissue maximum similarity value and the normal tissue maximum similarity value; and generating the visualization by indicating patches of the tissue image having at or above a threshold fewshot visualization score. In some embodiments, the irregular tissue maximum similarity value and the normal tissue maximum similarity value are determined using cosine similarity. In some instances, few-shot visualization scores are only calculated for patches of the tissue image having at or above a threshold irregular tissue maximum similarity value, wherein tissue image patches having below the threshold irregular tissue maximum similarity value are classified as background patches. In some embodiments, generating the visualization further includes: constructing a heatmap indicating one or more ROIs by rescaling each fewshot visualization score, wherein each patch of the tissue image is colored based on its rescaled few-shot visualization score; and overlaying the heatmap on the tissue image to generate the visualization. In some instances, patches of the tissue image for which no fewshot visualization score was calculated are colored in the same manner as patches of the tissue image having below a threshold few-shot visualization score.

[0019] In certain embodiments, the model is used as a surgical adjunct. For example, the model may be used to guide a surgical resection of a tumor. In some embodiments, one or more hyperparameters used to train or fine-tune the model are optimized such as, e.g., via performing an ablation study. In these instances, the one or more hyperparameters may be optimized using a validation set of tissue images held out of the set of tissue images used to train or fine-tune the model.

[0020] In another aspect, systems for performing the methods of training a visual ML model (e.g., a medical foundation model) to identify and encode tissue image features, as well as systems for using the trained visual ML model to perform one or more clinical tasks (such as, e.g., detecting and / or quantifying tumor infiltration in freshly sampled surgical tissue) are provided. Further, non-transitory computer readable storage media including the instructions of the memory of the systems, and kits including one or more of the system components and / or the non-transitory computer readable storage media, are also provided.

[0021] In another aspect, methods of fine-tuning a model (such as, e.g., a medical foundation model) to output a continuous score corresponding to a discrete and ordered scale of ordinal categories are provided. Aspects of the methods include: obtaining a set of tissue images including a plurality of neoplasm and / or tissue types, wherein each tissue image of the set includes an ordinal label corresponding to a category of the discrete and ordered scale; passing a batch of tissue images of the tissue image set through the model in order to produce a continuous score for each tissue image of the batch, wherein the model includes: an image encoder trained to identify and encode tissue image features for each tissue image of the batch; and an output module including an image scorer configured to output the continuous score for each tissue image of the batch using the encoded tissue image features; performing a pairwise comparison between all tissue images of the batch using the continuous scores and a binary cross entropy objective function; and adjusting the image scorer of the model to minimize the binary cross entropy objective function such that the continuous scores output by the model better correspond to the discrete and ordered scale.

[0022] In certain embodiments, the image encoder of the model is adjusted to minimize the binary cross entropy objective function. In some cases, the image scorer includes a one layer linear scorer. In some embodiments, the model is initially trained via the methods of training a visual ML model to identify and encode tissue image features, e.g., as described above and herein. In other embodiments, the model includes a CTransPath, Phycon, or UNI model.

[0023] In certain embodiments, the set of tissue images used to fine-tune the model includes a smaller number of images than an initial set of tissue images used to train the model. In some embodiments, the scale includes a tumor infiltration scale quantifying the degree of tumor infiltration in each image of the set of tissue images. Tn these instances, the tumor infiltration scale may include four categories, wherein: the first category corresponds to tissue images where no tumor is present: the second category corresponds to tissue images including atypical cells and / or images where it is possible, but not definite, that a tumor is present; the third category corresponds to tissue images including sparse tumor infiltration; and the fourth category corresponds to tissue images including dense tumor infiltration. In some embodiments, the method may further include generating the ordinal labels for the set of tissue images. In these instances, the ordinal labels may be generated using domain specific knowledge and one or more of: SRH imaging, H&E staining, and / or IHC.

[0024] In certain embodiments, the fine-tuning includes: adjusting the output module and / or the image encoder to: maximize the latent distance between one or more encoded features of tissue images of the set having different ordinal labels, and / or minimize the latent distance between one or more encoded features of tissue images of the set having the same ordinal label. In some embodiments, class imbalance is addressed by oversampling tissue images of the set corresponding to ordinal labels of a minority class. In some embodiments, the finetuned model is used to detect and quantify tumor infiltration in one or more unprocessed, label-free, surgical samples. In these instances, the one or more surgical samples may be imaged at lower resolutions and / or faster acquisition rates than the tissue images used to initially train and / or fine-tune the model.

[0025] In another aspect, systems for performing the methods of fine-tuning a model (such as, e.g., a medical foundation model) to output a continuous score corresponding to a discrete and ordered scale of ordinal categories, as well as systems for using the fine-tuning model to perform one or more clinical tasks (such as, e.g., detecting and / or quantifying tumor infiltration in freshly sampled surgical tissue) are provided. Further, non-transitory computer readable storage media including the instructions of the memory of the systems, and kits including one or more of the system components and / or the non-transitory computer readable storage media, are also provided.

[0026] In another aspect, methods of generating a visualization identifying one or more regions of interest (ROIs) within a tissue image passed through a model (such as, e.g., a medical foundation model) are provided. Aspects of the methods include: passing a set of patch keys including features or characteristics of the one or more ROIs through the model, wherein the set of patch keys includes examples of normal tissue and irregular tissue; calculating a few-shot visualization score for one or more patches of the tissue image, wherein for each of the one or more patches, the calculating includes: determining a maximum similarity value between a patch of the tissue image and each of the irregular tissue patch keys by calculating a dot product between the tissue image patch and each of the irregular tissue patch keys; determining a maximum similarity value between the tissue image patch and each of the normal tissue patch keys by calculating a dot product between the tissue image patch and each of the normal tissue patch keys; and calculating a few-shot visualization score for the tissue image patch by determining a difference between the irregular tissue maximum similarity value and the normal tissue maximum similarity value; and generating the visualization by indicating patches of the tissue image having at or above a threshold few-shot visualization score.

[0027] In certain embodiments, the model is initially trained via the methods of training a visual ML model to identify and encode tissue image features, e.g., as described above and herein. In other embodiments, the model includes a CTransPath, Phycon, or UNI model. In some embodiments, the model is fine-tuned via the methods of fine-tuning a model to output a continuous score corresponding to a discrete and ordered scale of ordinal categories, e.g., as described above and herein.

[0028] In certain embodiments, the set of patch keys includes two or more image patches selected by a medical professional. In some embodiments, the set of patch keys includes diverse examples of normal brain parenchyma and diffuse glioma subtypes. In some instances, the irregular tissue maximum similarity value and the normal tissue maximum similarity value are determined, at least in part, using cosine similarity. In some embodiments, few- shot visualization scores are only calculated for patches of the tissue image having at or above a threshold irregular tissue maximum similarity value, wherein tissue image patches having below the threshold irregular tissue maximum similarity value are classified as background patches. In some embodiments, generating the visualization further includes: constructing a heatmap indicating one or more ROIs by rescaling each fewshot visualization score, wherein each patch of the tissue image is colored based on its rescaled few-shot visualization score; and overlaying the heatmap on the tissue image to generate the visualization. In these instances, patches of the tissue image for which no few- shot visualization score was calculated may be colored in the same manner as patches of the tissue image having below a threshold few-shot visualization score.

[0029] In another aspect, systems for performing the methods of generating a visualization identifying one or more regions of interest (ROIs) within a tissue image passed through a model (such as, e.g., a medical foundation model), as well as systems for using generated visualizations to perform one or more clinical tasks (such as, e.g., verifying the detection and / or quantification of tumor infiltration by medical foundation models, as described above and herein) are provided. Further, non-transitory computer readable storage media including the instructions of the memory of the systems, and kits including one or more of the system components and / or the non-transitory computer readable storage media, are also provided.

[0030] BRIEF DESCRIPTION OF THE FIGURES

[0031] FIG. 1 demonstrates the workflow of a fine-tuned visual foundational machine learning (ML) model (i.e., FastGlioma), in accordance with embodiments of the invention.

[0032] FIGS. 2A to 2E provide prospective clinical testing results for FastGlioma. (A) depicts prediction results for the full prospective, international, multicenter testing cohort of diffuse gliomas patients. (B) provides box and whisker plots, shown in the standardized quartile format, of FastGlioma infiltration scores by ground truth value. (C) illustrates FastGlioma performance on full resolution versus low resolution SRH images. (D) provides slide SRH representations plotted on a linear discriminant axis. (E) illustrates subgroup analysis by WHO adult- type diffuse glioma subtypes.

[0033] FIG. 3 illustrates general and interpretable FastGlioma predictions via a visualization method (i.e., few-shot visualizations) highlighting regions of interest (ROIs) in accordance with embodiments of the invention.

[0034] FIGS. 4A to 4B provide simulated clinical trial results for FastGlioma. (A) illustrates results of a prospective simulated clinical trial comparing FastGlioma to current standard-of- care surgical adjuncts. (B) provides a graphical summary of patient-level false negative rates for the simulated clinical trial.

[0035] FIGS. 5A to 5C depict the FastGlioma workflow and dataset generation, in accordance with embodiments of the invention. (A) illustrates surgical specimens sampled from the patient resection cavity and sent for clinical pathologic testing. (B) depicts a surgical specimen loaded into a premade microscope slide and imaged at the patient’s bedside using the NIO Imaging System. (C) demonstrates tumor infiltration scoring via H&E and immunohistochemistry (IHC) staining.

[0036] FIGS. 6A to 6C provide charts characterizing the SRH foundation model and FastGlioma training datasets, in accordance with embodiments of the invention. (A) illustrates the institutions and diagnoses of the training dataset. (B) depicts infiltration scores for diffuse glioma training data. (C) provides details regarding patient demographics, WHO glioma subtypes, recurrence status, tumor grade, and institution for testing data.

[0037] FIGS. 7A to 7C illustrate self-supervised foundation model training, in accordance with embodiments of the invention. (A) demonstrates patch tokenizer training via selfsupervised hierarchical contrastive learning. (B) provides training results as t-SNE patch representations plotted and colored according to the ground truth whole slide label. (C) demonstrates the whole slide self-supervised training strategy.

[0038] FIGS. 8A to 8D illustrate ordinal metric learning for efficient ordinal representation learning, in accordance with embodiments of the invention. (A) depicts an ordering of each whole slide via ordinal metric learning. (B) shows the performance of ordinal metric learning versus other state-of-the-art ordinal regression methods on a well-studied face age estimation dataset. (C) depicts the results of hold-out cross validation studies performed for ordinal metric learning. (D) shows tumor infiltration scores for ordinal metric learning versus other baseline strategies.

[0039] FIGS. 9A to 9D illustrate speed versus performance trade-off ablation studies of a fine-tuned visual foundational ML model, in accordance with embodiments of the invention. (A) demonstrates increased imaging speed via single-channel imaging and decreasing line scanning density. (B) depicts results of Hold-out cross validation ablation studies examining the trade-off between image resolution / speed and model performance. (C) illustrates how tumor infiltration scores are related via full resolution and fast SRH imaging. (D) demonstrates the similarity between whole slide encoder self-attention coefficients for full resolution versus fast SRH image.

[0040] FIGS. 10A to 10E illustrate extended prospective testing results with subgroup analysis for FastGlioma. (A) shows FastGlioma scores by sex, age, and race using box and whisker plots in the standardized quartile format. (B) depicts subgroup analysis by medical center. (C) provides mean AUROC values plotted by tumor grade. (D) illustrates subgroup analysis by index / first surgery versus surgery for recurrent tumor. (E) shows whole slide SRH representations for IDH mutant and IDH wildtype diffuse gliomas.

[0041] FIGS. 11A to 11B provide cytologic analysis of tumor infiltration scores for FastGlioma. (A) depicts the relationship between cellularity and FastGlioma scores. (B) provides a chart of cellularity versus FastGlioma scores plotted by molecular subgroup.

[0042] FIGS. 12A to 12B provide few-shot visualizations of diffuse glioma infiltration. (A) shows a panel of whole slide SRH images across diffuse glioma molecular subgroups. (B) illustrates principal component analysis (PCA) used on patch features extracted by FastGlioma.

[0043] FIGS. 13A to 13B illustrate zero-shot predictions and visualizations obtained via FastGlioma in accordance with embodiments of the invention.

[0044] FIGS. 14A to 14C show a comparison of FastGlioma and surgical adjuncts for diffuse glioma resection. (A) demonstrates classification performance for distinguishing between normal brain and dense tumor by FastGlioma versus surgical adjuncts on diffuse gliomas. (B) demonstrates classification performance of FastGlioma versus surgical adjuncts on glioblastomas. (C) provides illustrative examples of matched surgical specimens with FastGlioma predictions, neuronavigation coordinates, radiographic features, and 5-ALA status.

[0045] FIG. 15 provides an overview of ordinal metric learning for FastGlioma training, in accordance with embodiments of the invention, via PyTorch-like pseudocode.

[0046] FIG. 16 provides charts comparing FastGlioma and surgical adjuncts (i.e., Fluid- Attenuated Inversion Recovery (FLAIR) MRI and 5 -aminolevulinic acid (5-ALA) fluorescence) for diffuse glioma resection.

[0047] FIGS. 17A to 17B provide extent of resection versus cumulative survival charts for both (A) lower grade gliomas and (B) glioblastomas.

[0048] FIGS. 18A to 18B demonstrate an exemplary workflow of a fine-tuned visual foundational ML model, in accordance with embodiments of the invention. (A) depicts a workflow for label-free, microscopic detection of diffuse glioma infiltration in the operating room (OR). (B) demonstrates an end-to-end Al system for automating the detection and scoring of the degree of diffuse glioma infiltration FIG. 19 provides exemplary FLAIR MRT, 5-ALA fluorescence, IHC staining, and SRH images depicting glioma infiltrated brain tissue.

[0049] FIGS. 20A to 20D provide exemplary SRH images wherein tumor infiltration has been scored and ROIs have been highlighted using models and visualization techniques in accordance with embodiments of the invention.

[0050] FIG. 21 illustrates self-supervised whole slide learning, in accordance with embodiments of the invention, compared to supervised whole slide learning.

[0051] FIGS. 22A to 22B provide an overview of Slide Pre-trained Transformers (SPT), in accordance with embodiments of the invention. (A) illustrates a SPT framework consisting of a two-stage model architecture. (B) depicts exemplary learning paradigms including SimCLR, BYOL, VICReg, and supervised contrastive learning.

[0052] FIG. 23 demonstrates the limited effect of pixel-level patch augmentations by visualizing a tSNE plot of SimCLR pre-trained patch representations sampled from a single whole slide image (WSI).

[0053] FIG. 24 illustrates an SPT transformation strategy comprising a combination of splitting, cropping, and masking, used to generate views for model training in accordance with embodiments of the invention.

[0054] FIGS. 25A to 25B provide SPT benchmarks with different patch encoders. (A) characterizes a number of different patch encoders. (B) provides SPT results on H&E Glioma.

[0055] FIG. 26 shows attention heatmap visualizations in accordance with embodiments of the invention.

[0056] FIG. 27 illustrates cosine similarity of learned positional embeddings generated in accordance with embodiments of the invention.

[0057] FIG. 28 provides SPT results for SRH imaged central nervous system (CNS) tumors and H&E stained gliomas with a wide range of patch encoders.

[0058] FIGS. 29A to 29B illustrate SPT transformation parameter ablation, in accordance with embodiments of the invention, for the H&E glioma molecular classification tasks of Example 2. (A) provides the results of a max token limit ablation study. (B) provides the results of a cropping size range ablation study. FIG. 30 provides attention heatmaps for H&E whole slides of oligodendroglioma, astrocytoma, and glioblastoma tumors, in accordance with embodiments of the invention.

[0059] FIG. 31 provides attention heatmaps differentiating tumor regions from nondiagnostic regions for SRH whole slides, in accordance with embodiments of the invention.

[0060] FIG. 32 provides an illustration of the implementation of SPT transformations, in accordance with embodiments of the invention, via PyTorch-like pseudocode.

[0061] DETAILED DESCRIPTION

[0062] Novel approaches for training medical foundation models are provided, in conjunction with novel techniques for fine-tuning models based on ordinally labeled data, in order to fully leverage expert knowledge and advances in Al technology in addressing a wide range of critically important clinical tasks (such as, e.g., residual tumor detection). In addition, in order to ensure that the clinically actionable information obtained using the models of the disclosure is safe, reliable, and trustworthy, methods for generating interpretable visualizations via the trained and / or fine-tuned foundational models are also provided. Thus, methods and systems for rapidly, accurately, and verifiably identifying tumor infiltration across a wide range of clinically significant settings are disclosed herein.

[0063] Before the present invention is described in greater detail, it is to be understood that this invention is not limited to particular embodiments described, as such may, of course, vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting, since the scope of the present invention will be limited only by the appended claims.

[0064] Where a range of values is provided, it is understood that each intervening value, to the tenth of the unit of the lower limit unless the context clearly dictates otherwise, between the upper and lower limit of that range and any other stated or intervening value in that stated range, is encompassed within the invention. The upper and lower limits of these smaller ranges may independently be included in the smaller ranges and are also encompassed within the invention, subject to any specifically excluded limit in the stated range. Where the stated range includes one or both of the limits, ranges excluding either or both of those included limits are also included in the invention. Certain ranges are presented herein with numerical values being preceded by the term “about.” The term “about” is used herein to provide literal support for the exact number that it precedes, as well as a number that is near to or approximately the number that the term precedes. In determining whether a number is near to or approximately a specifically recited number, the near or approximating unrecited number may be a number which, in the context in which it is presented, provides the substantial equivalent of the specifically recited number.

[0065] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the present invention, representative illustrative methods and materials are now described.

[0066] All publications and patents cited in this specification are herein incorporated by reference as if each individual publication or patent were specifically and individually indicated to be incorporated by reference and are incorporated herein by reference to disclose and describe the methods and / or materials in connection with which the publications are cited. The citation of any publication is for its disclosure prior to the filing date and should not be construed as an admission that the present invention is not entitled to antedate such publication by virtue of prior invention. Further, the dates of publication provided may be different from the actual publication dates which may need to be independently confirmed.

[0067] It is noted that, as used herein and in the appended claims, the singular forms “a”, “an”, and “the” include plural referents unless the context clearly dictates otherwise. It is further noted that the claims may be drafted to exclude any optional element. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely.” “only” and the like in connection with the recitation of claim elements, or use of a “negative” limitation.

[0068] As will be apparent to those of skill in the art upon reading this disclosure, each of the individual embodiments described and illustrated herein has discrete components and features which may be readily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present invention. Any recited method can be carried out in the order of events recited or in any other order which is logically possible.

[0069] While the apparatus and method has or will be described for the sake of grammatical fluidity with functional explanations, it is to be expressly understood that the claims, unless expressly formulated under 35 U.S.C. §112, are not to be construed as necessarily limited in any way by the construction of “means” or “steps” limitations, but are to be accorded the full scope of the meaning and equivalents of the definition provided by the claims under the judicial doctrine of equivalents, and in the case where the claims are expressly formulated under 35 U.S.C. § 112 are to be accorded full statutory equivalents under 35 U.S.C. § 112.

[0070] METHODS

[0071] As summarized above, methods for rapidly, accurately, and verifiably identifying tumor infiltration across a wide range of clinically significant settings are provided. In one aspect, methods of training a visual machine learning (ML) model such as, e.g., a medical foundation model, to identify and encode tissue image features are provided. The training methods of the disclosure enable the creation of models addressing a wide range of downstream clinical tasks including, but not limited to, residual tumor detection. In another aspect, methods of fine-tuning a visual ML model to output a continuous score corresponding to a discrete and ordered scale of ordinal categories are provided. The fine-tuning methods of the disclosure, especially when used with the aforementioned training methods, enable ML models to fully leverage the insights provided by expertly labeled data, and to generate more intuitive clinically actionable outputs. In another aspect, methods of generating a visualization identifying one or more regions of interest (ROIs) within a tissue image passed through a visual ML model are provided. The disclosed visualization generating methods allow a practitioner to verify clinically actionable information obtained via, e.g., ML models trained and / or fine-tuned using the methods described above and herein, ensuring that the obtained information is safe, reliable, and trustworthy.

[0072] The terms “subject”, “individual”, “patient”, and “participant” are used interchangeably herein and refer to an individual for which tissue images have been obtained for use with the visual ML models of the disclosure, e.g., in order to obtain clinically actionable information for the individual. The individual is preferably human, e.g., a child, an adolescent, or an adult (such as a young, middle-aged, or elderly adult) human who may benefit from the methods and systems disclosed herein, or who may facilitate understanding of a disease or condition in which they are afflicted through use of the subject methods and systems disclosed herein. The individual or subject may belong to any demographic, and may be characterized by any number of different prognostic variables. In some embodiments, the subject may have a solid cancer. In some cases, the subject may have a brain tumor such as, e.g., a malignant or a benign brain tumor. Malignant brain tumors may include, but are not limited to, gliomas, choroid plexus tumors, embryonal tumors, germ cell tumors, pineal tumors, meningiomas, nerve tumors, and pituitary tumors. In some embodiments, the subject may have a glioma such as, e.g., an astrocytoma, glioblastoma, oligodendroglioma, ependymoma, diffuse midline gliomas (DMG), diffuse intrinsic pontine gliomas (DIPG), etc.

[0073] Visual Foundation Model Training

[0074] As described above, methods of training a visual machine learning (ML) model to identify and encode tissue image features are provided. Aspects of the methods include: obtaining a set of tissue images including a plurality of neoplasm and / or tissue types, wherein each tissue image includes multiple patches; training a tokenizer of the model to convert patches into tokens by identifying and encoding patch features using the patches of the tissue image set, wherein the training includes self-supervised contrastive learning; generating two or more views of each of a plurality of images of the tissue image set, wherein the plurality of images have been passed through the tokenizer; and training an image encoder of the model to identify and encode tissue image features using the two or more views of the plurality of tokenized images, wherein the image encoder includes a vision transformer and the training includes self- supervised learning.

[0075] The visual ML model trained using the methods of the disclosure, e.g., as described herein, may be a foundation model. By foundation model is meant an ML model that is trained on a large and diverse dataset, using self- supervision at scale, that can be utilized for a wide range of downstream tasks. In some embodiments, foundation models of the disclosure (i.e., foundation models trained using the training methods of the disclosure) are utilized for a specific downstream task by fine-tuning the model and / or by training an output module of the model to perform the specific downstream task, e.g., as described below or herein. In some embodiments, foundation models of the disclosure demonstrate zero-shot generalization. By zero-shot generalization is meant the foundation models of the disclosure are able to perform tasks or make predictions on data they have never seen before or data for which they were not explicitly trained. In some embodiments, the foundation models of the disclosure are medical foundation models.

[0076] In some embodiments, the ML models of the disclosure (e.g., the foundation models of the disclosure) are particularly suited for performing downstream clinical and / or diagnostic tasks. In these cases, the models may be trained on microscopy images, wherein the set of tissue images each include a slide of tissue. In order to enable medical foundation models of the disclosure to demonstrate zero-shot generalization, the set of tissue images may include a wide range of different tissue types and different neoplasm types. For example, the set of tissue images may include two or more tissue types such as, e.g., three or more tissue types, or five or more, or ten or more, or twenty or more, and two or more neoplasm types such as. e.g., three or more neoplasm types, or five or more, or ten or more, or twenty or more. By different types of tissue is meant tissues having a different structure and / or function. By different types of neoplasms is meant neoplasms having at least one differing classification (such as, e.g., different tumor grades, different tumor types or subtypes, etc.). In some embodiments, the plurality of neoplasm types may include central nervous system (CNS) tumors, wherein the CNS tumors include one or more of: high-grade gliomas (HGG), low-grade gliomas (LGG), meningiomas, pituitary adenomas, schwannomas, and / or metastatic tumors. In some embodiments, the set of tissue images includes tissues from multiple different organ systems. In these instances, the tissue image set may include neoplasms from multiple different organ systems.

[0077] In some embodiments, the set of tissue images is obtained using stimulated Raman histology (SRH). In some embodiments, the foundation models of the disclosure are SRH foundation models. In some cases, SRH images are obtained using multiple image channels. For example, the SRH images may be acquired sequentially at 2845 cm’1(CH2 channel) and 2930 cm’1(CH3 channel) Raman wavenumber shifts. In these instances, the SRH images may be obtained using the NIO Imaging System.

[0078] In some embodiments, the set of tissue images are obtained from a plurality of subjects such as, e.g., 500 or more subjects, or 1,000 or more, or 3,000 or more, or 5,000 or more, or 10,000 or more, or 20,000 or more, or 50,000 or more. In some instances, the set of tissue images includes 1,000 or more images such as, e.g., 5,000 or more images, or 10,000 or more, or 50.000 or more, or 100,000 or more, or 200,000 or more, or 500,000 or more, or 1,000,000 or more.

[0079] Patch Tokenization

[0080] As discussed above, embodiments of the visual ML model training methods include training a tokenizer of the model to encode features from patches in order to convert patches into tokens using self-supervised contrastive learning. In some embodiments, the method further includes dividing images of the tissue image set into multiple patches such as, e.g., a predetermined number and / or layout of patches. In these instances, the tokenizer may be configured to divide each image of the set into a predetermined number and / or layout of patches. For example, each image may be divided (e.g., via the tokenizer) into nonoverlapping rectangular patches having a fixed 300x300-pixel size.

[0081] In some embodiments, the tokenizers (i.e., of the visual ML models of the disclosure) comprise an ML model. The tokenizer ML models, in accordance with embodiments of the methods, may vary and may include, but are not limited to, any of the models discussed below, or any model known in the art capable of identifying and encoding image features and passing said features to the whole slide encoder models of the disclosure, as discussed herein. In some embodiments, the tokenizer model includes a vision transformer. In some embodiments, the tokenizer model includes an artificial neural network (NN) such as, e.g., convolutional NN (CNN). In some instances, the CNN is a residual CNN such as, e.g., ResNet-18, ResNet-34, ResNet-50, ResNet-101, or ResNet-152.

[0082] In certain embodiments, the self-supervised contrastive learning includes hierarchical self- supervised learning via a hierarchical discriminative learning task. For example, the tokenizer may be trained to identify and encode patch features for each level of a multileveled hierarchy, wherein the hierarchy includes a patch level, an image level, and a subject level. In some embodiments, the hierarchical self- supervised learning of the tokenizer includes: generating a plurality of positive patch pairs for each of the patch, image, and subject levels of the hierarchy from the set of tissue images; passing the positive patch pairs through the CNN; and adjusting the CNN to minimize a contrastive learning objective function, wherein the contrastive learning objective function includes patch, image, and subject level components. In some embodiments, the method further includes generating a plurality of negative patch pairs for each of the patch, image, and subject levels of the hierarchy from the set of tissue images and passing the negative patch pairs through the CNN.

[0083] Whole Slide Encoding

[0084] As discussed above, embodiments of the visual ML model training methods include generating two or more views of each of a plurality of images of the tissue image set having been passed through the tokenizer, wherein the two or more views of the plurality of images are used to train an image encoder of the model, including a vision transformer, to encode tissue image features through self- supervised learning. In certain embodiments, the two or more views of each of the plurality of images are generated by randomly: splitting each image into mutually exclusive patch sets; spatially cropping each patch set; and masking multiple patches from each spatially cropped patch set.

[0085] In some embodiments, the self- supervised learning of the image encoder includes: passing each view generated for an image of the plurality of images through a different vision transformer of a Siamese architecture, wherein the transformers of the architecture share weights; and adjusting the transformers of the architecture to minimize a selfsupervised objective function, wherein the self-supervised objective function includes variance, invariance, and covariance components. In these cases, the self- supervised objective function may include Variance-Invariance-Covariance Regularization (VICReg). In some embodiments, the transformers of the architecture include two or more hidden layers, wherein each hidden layer includes a dimension of 500 or more and multiple attention heads (such as, e.g., a dimension of 512 and 4 attention heads). In some embodiments, the transformers of the architecture include a component for generating positional embeddings such as, e.g., a Fourier feature positional embedding generator network.

[0086] Task Outputs and Output Modules

[0087] As discussed above, embodiments of the visual ML model training methods include training a tokenizer and a whole slide encoder of the model to, in conjunction, identify and encode tissue image features from tissue images passed through the model. In some embodiments, the method further includes training an output module of the visual ML model to perform one or more specific tasks. In some embodiments, an output module of the model is trained to perform a specific task by passing tissue images through the tokenizer and the image encoder of the model. In some embodiments, the output module includes a K-Nearest Neighbors (KNN) classifier. In some embodiments, the output module includes one or more output layers. For example, the output module may include a one layer linear output layer. In some embodiments, the output module is trained to diagnose brain tumors. In some cases, the output module is trained to quantify tumor infiltration or tumor burden.

[0088] In certain embodiments, the method further includes fine-tuning the visual ML model (e.g., the tokenizer, the image encoder, and / or the output module of the model) to perform one or more specific tasks. In some embodiments, the output module is fine-tuned using a second set of tissue images, wherein the second set of tissue images includes a smaller number of images than the initial set of images used to train the tokenizer, the image encoder, and / or the output module of the model. In some embodiments, the image encoder of the model is fine-tuned using the second set of tissue images.

[0089] In some embodiments, each image of the second set of tissue images includes an ordinal label corresponding to a category of a discrete and ordered scale. For example, the scale may include a tumor infiltration scale quantifying the degree of tumor infiltration in each image of the second set of tissue images. In these instances, the tumor infiltration scale may include four categories, wherein: the first category corresponds to tissue images where no tumor is present; the second category corresponds to tissue images including atypical cells and / or images where it is possible, but not definite, that a tumor is present; the third category corresponds to tissue images including sparse tumor infiltration; and the fourth category corresponds to tissue images including dense tumor infiltration. In some embodiments, the method further includes generating the ordinal labels for the second set of tissue images. In these instances, the ordinal labels may be generated using domain specific knowledge and one or more of: SRH imaging, hematoxylin and eosin (H&E) staining, and / or immunohistochemistry (IHC).

[0090] In some embodiments, fine-tuning the visual ML model (e.g., the tokenizer, the image encoder, and / or the output module of the model) includes: passing tissue images of the second set through the tokenizer, the image encoder, and the output module of the model; and adjusting the output module and / or the image encoder to: maximize the latent distance between one or more encoded features of tissue images of the second set having different ordinal labels, and / or minimize the latent distance between one or more encoded features of tissue images of the second set having the same ordinal label.

[0091] In certain embodiments, the output module includes an image scorer (such as, e.g., a one layer linear scorer) fine-tuned to output a continuous score corresponding to the discrete and ordered scale of the ordinal labels of the second set of tissue images. In these instances, the fine-tuning may include: passing a batch of tissue images of the second set through the tokenizer, the image encoder, and the image scorer of the model in order to produce a continuous score for each tissue image of the batch; performing a pairwise comparison between all tissue images of the batch using the continuous scores and a binary cross entropy objective function; and adjusting the image scorer and / or the image encoder of the model to minimize the binary cross entropy objective function. In some embodiments, class imbalance is addressed by oversampling tissue images of the second set corresponding to ordinal labels of a minority class. In some instances, the fine-tuned model may be used to detect and quantify tumor infiltration in one or more unprocessed, label-free, surgical samples. In these instances, the one or more surgical samples may be imaged at lower resolutions and / or faster acquisition rates than the tissue images used to train and / or fine-tune the model.

[0092] In some embodiments, the method further includes generating a visualization identifying one or more regions of interest (ROIs) within a tissue image passed through the model. In some cases, the visualization is generated via principal component analysis (PCA). In some embodiments, the visualization is generated using a set of patch keys including features or characteristics of the one or more ROIs. In these instances, the set of patch keys may include two or more image patches selected by a medical professional. In some embodiments, the set of patch keys includes examples of normal tissue and irregular tissue. For example, the set of patch keys may include diverse examples of normal brain parenchyma and diffuse glioma subtypes. In some instances, regions of interest are identified by comparing the set of patch keys, having passed through the model, with each of the patches of the tissue image passed through the model.

[0093] In certain embodiments, generating the visualization includes: calculating a few-shot visualization score for one or more of the patches of the tissue image, wherein for each of the one or more patches the calculating includes: determining a maximum similarity value between a patch of the tissue image and each of the irregular tissue patch keys by calculating a dot product between the tissue image patch and each of the irregular tissue patch keys; determining a maximum similarity value between the tissue image patch and each of the normal tissue patch keys by calculating a dot product between the tissue image patch and each of the normal tissue patch keys; and calculating a few-shot visualization score for the tissue image patch by determining a difference between the irregular tissue maximum similarity value and the normal tissue maximum similarity value; and generating the visualization by indicating patches of the tissue image having at or above a threshold fewshot visualization score. In some embodiments, the irregular tissue maximum similarity value and the normal tissue maximum similarity value are determined using cosine similarity.

[0094] In some instances, few-shot visualization scores are only calculated for patches of the tissue image having at or above a threshold irregular tissue maximum similarity value, wherein tissue image patches having below the threshold irregular tissue maximum similarity value are classified as background patches. In some embodiments, generating the visualization further includes: constructing a heatmap indicating one or more ROIs by rescaling each few- shot visualization score, wherein each patch of the tissue image is colored based on its rescaled few-shot visualization score; and overlaying the heatmap on the tissue image to generate the visualization. In some instances, patches of the tissue image for which no few-shot visualization score was calculated are colored in the same manner as patches of the tissue image having below a threshold few-shot visualization score.

[0095] In certain embodiments, the model is used as a surgical adjunct. For example, the model may be used to guide a surgical resection of a tumor. In some embodiments, one or more hyperparameters used to train or fine-tune the model are optimized such as, e.g., via performing an ablation study. In these instances, the one or more hyperparameters may be optimized using a validation set of tissue images held out of the set of tissue images used to train or fine-tune the model.

[0096] Fine-Tuning Using Ordinally Labeled Data

[0097] As described above, methods of fine-tuning a model (such as, e.g., a medical foundation model) to output a continuous score corresponding to a discrete and ordered scale of ordinal categories are provided. Aspects of the methods include: obtaining a set of tissue images including a plurality of neoplasm and / or tissue types, wherein each tissue image of the set includes an ordinal label corresponding to a category of the discrete and ordered scale; passing a batch of tissue images of the tissue image set through the model in order to produce a continuous score for each tissue image of the batch, wherein the model includes: an image encoder trained to identify and encode tissue image features for each tissue image of the batch; and an output module including an image scorer configured to output the continuous score for each tissue image of the batch using the encoded tissue image features; performing a pairwise comparison between all tissue images of the batch using the continuous scores and a binary cross entropy objective function; and adjusting the image scorer of the model to minimize the binary cross entropy objective function such that the continuous scores output by the model better correspond to the discrete and ordered scale.

[0098] In certain embodiments, the image encoder of the model is adjusted to minimize the binary cross entropy objective function. In some cases, the image scorer includes a one layer linear scorer. In some embodiments, the model is initially trained via the methods of training a visual ML model to identify and encode tissue image features, e.g., as described above and herein. In other embodiments, the model includes a CTransPath, Phycon, or UNI model.

[0099] In some embodiments, the set of tissue images used to fine-tune the model includes a smaller number of images than an initial set of tissue images used to train the model. In some embodiments, the scale includes a tumor infiltration scale quantifying the degree of tumor infiltration in each image of the set of tissue images. In these instances, the tumor infiltration scale may include four categories, wherein: the first category corresponds to tissue images where no tumor is present; the second category corresponds to tissue images including atypical cells and / or images where it is possible, but not definite, that a tumor is present; the third category corresponds to tissue images including sparse tumor infiltration; and the fourth category corresponds to tissue images including dense tumor infiltration. In some embodiments, the method may further include generating the ordinal labels for the set of tissue images. In these instances, the ordinal labels may be generated using domain specific knowledge and one or more of: SRH imaging, H&E staining, and / or IHC.

[0100] In some embodiments, the fine-tuning includes: adjusting the output module and / or the image encoder to: maximize the latent distance between one or more encoded features of tissue images of the set having different ordinal labels, and / or minimize the latent distance between one or more encoded features of tissue images of the set having the same ordinal label. In some embodiments, class imbalance is addressed by oversampling tissue images of the set corresponding to ordinal labels of a minority class. In some embodiments, the finetuned model is used to detect and quantify tumor infiltration in one or more unprocessed, label-free, surgical samples. In these instances, the one or more surgical samples may be imaged at lower resolutions and / or faster acquisition rates than the tissue images used to initially train and / or fine-tune the model. For example, for an SRH foundation model of the disclosure, fine-tuned using the methods as described above and herein, SRH images passed into the model may be down-sampled by a downsampling factor of, e.g., Vi or less, or or less, or 1 / 8 or less, without any significant reduction in the performance of the model.

[0101] Few-Shot Visualizations

[0102] As described above, methods of generating a visualization identifying one or more regions of interest (ROIs) within a tissue image passed through a model (such as, e.g., a medical foundation model) are provided. Aspects of the methods include: passing a set of patch keys including features or characteristics of the one or more ROIs through the model, wherein the set of patch keys includes examples of normal tissue (i.e., non-cancerous or healthy) and irregular tissue (e.g., neoplasms, cancerous or tumor infiltrated tissue); calculating a few-shot visualization score for one or more patches of the tissue image, wherein for each of the one or more patches, the calculating includes: determining a maximum similarity value between a patch of the tissue image and each of the irregular tissue patch keys by calculating a dot product between the tissue image patch and each of the irregular tissue patch keys; determining a maximum similarity value between the tissue image patch and each of the normal tissue patch keys by calculating a dot product between the tissue image patch and each of the normal tissue patch keys; and calculating a few-shot visualization score for the tissue image patch by determining a difference between the irregular tissue maximum similarity value and the normal tissue maximum similarity value; and generating the visualization by indicating patches of the tissue image having at or above a threshold few-shot visualization score. In some embodiments, the model is initially trained via the methods of training a visual ML model to identify and encode tissue image features, e.g., as described above and herein. In other embodiments, the model includes a CTransPath, Phycon, or UNI model. In some embodiments, the model is fine-tuned via the methods of fine-tuning a model to output a continuous score corresponding to a discrete and ordered scale of ordinal categories, e.g., as described above and herein.

[0103] In certain embodiments, the set of patch keys includes two or more image patches selected by a medical professional. In some embodiments, the set of patch keys includes diverse examples of normal brain parenchyma and diffuse glioma subtypes. In some instances, the irregular tissue maximum similarity value and the normal tissue maximum similarity value are determined, at least in part, using cosine similarity. In some embodiments, few- shot visualization scores are only calculated for patches of the tissue image having at or above a threshold irregular tissue maximum similarity value, wherein tissue image patches having below the threshold irregular tissue maximum similarity value are classified as background patches.

[0104] In some embodiments, generating the visualization further includes: constructing a heatmap indicating one or more ROIs by rescaling each few-shot visualization score, wherein each patch of the tissue image is colored based on its rescaled few-shot visualization score; and overlaying the heatmap on the tissue image to generate the visualization. In these instances, patches of the tissue image for which no few-shot visualization score was calculated may be colored in the same manner as patches of the tissue image having below a threshold few- shot visualization score.

[0105] In some embodiments, the method further includes using the generated visualization to verify or confirm clinically actionable information produced by a visual ML model. For example, the generated visualization may be displayed to a surgeon in order to verify or ensure the accuracy of a tumor infiltration score output by the ML model. In these instances, a visualization indicating patches of the tissue image as irregular (e.g., cancerous or tumor- infiltrated) when the surgeon deems them to be regular (i.e„ non-cancerous or healthy), or indicating patches of the tissue image as regular when the surgeon deems them to be irregular, might result in the surgeon ignoring a tumor infiltration score output by the ML model. SYSTEMS AND COMPUTER IMPLEMENTED METHODS

[0106] Aspects of the present disclosure further include systems, such as computer- controlled systems, for practicing embodiments of the above methods.

[0107] In one aspect, systems for performing the methods of training a visual ML model (e.g., a medical foundation model) to identify and encode tissue image features, as well as systems for using the trained visual ML model to perform one or more clinical tasks (such as, e.g., detecting and / or quantifying tumor infiltration in freshly sampled surgical tissue) are provided. Aspects of the systems include: a processor including memory operably coupled to the processor wherein the memory includes instructions stored thereon, which when executed by the processor, cause the processor to: obtain a set of tissue images including a plurality of neoplasm and / or tissue types, wherein each tissue image includes multiple patches; train a tokenizer of the model to convert patches into tokens by identifying and encoding patch features using the patches of the tissue image set, wherein the tokenizer includes a convolutional neural network (CNN) and the training includes self- supervised contrastive learning; generate two or more views of each of a plurality of images of the tissue image set, wherein the plurality of images have been passed through the tokenizer; and train an image encoder of the model to identify and encode tissue image features using the two or more views of the plurality of tokenized images, wherein the image encoder includes a vision transformer and the training includes self-supervised learning.

[0108] In some embodiments, the memory includes instructions stored thereon, which when executed by the processor, further cause the processor to apply the trained visual ML model (i.e., trained according to the methods of the disclosure, as described herein) to perform one or more specific clinical and / or diagnostic tasks (such as, e.g., any of the tasks disclosed above or herein). In some cases, the one or more tasks include detecting and / or quantifying tumor infiltration in a freshly sampled surgical tissue in order to, e.g.. guide a surgical resection of a tumor. For example, the processor may be configured to receive one or more images from an external device, wherein the instructions then cause the processor to pass the one or more received images through the trained ML model in order to produce clinically actionable information. In these instances, the clinically actionable information may include, e.g., a tumor infiltration score quantifying the degree of tumor infiltration in each received image, as discussed above. In some embodiments, the system further includes a stimulated Raman histology (SRH) imaging device or system operably coupled with the processor, wherein the processor is configured to receive images from the SRH imaging device / system. In these instances, the SRH imaging device / system may include the NIO Imaging System (Invenio Imaging, Inc., Santa Clara, CA). In some embodiments, the system further includes a display configured to provide clinically actionable information produced by the trained ML model to a user. For example, a tumor infiltration score output by the trained ML model may be displayed to a surgeon in order to guide a surgical resection of a tumor. In some embodiments, the system further includes operator input devices configured to receive input from a user. For example, several images may be generated via the SRH imaging device / system and displayed, wherein a user may then select which images to pass through the trained ML model. In some cases, the operator input devices include a keyboard, a mouse, and / or a touch screen.

[0109] In another aspect, systems for performing the methods of fine-tuning a model (such as, e.g.. a medical foundation model) to output a continuous score corresponding to a discrete and ordered scale of ordinal categories, as well as systems for using the fine-tuning model to perform one or more clinical tasks (such as, e.g., detecting and / or quantifying tumor infiltration in freshly sampled surgical tissue) are provided. Aspects of the systems include: a processor including memory operably coupled to the processor wherein the memory includes instructions stored thereon, which when executed by the processor, cause the processor to: obtain a set of tissue images including a plurality of neoplasm and / or tissue types, wherein each tissue image of the set includes an ordinal label corresponding to a category of the discrete and ordered scale; pass a batch of tissue images of the tissue image set through a visual ML model in order to produce a continuous score for each tissue image of the batch, wherein the model includes: an image encoder trained to identify and encode tissue image features for each tissue image of the batch; and an output module including an image scorer configured to output the continuous score for each tissue image of the batch using the encoded tissue image features; perform a pairwise comparison between all tissue images of the batch using the continuous scores and a binary cross entropy objective function; and adjust the image scorer of the model to minimize the binary cross entropy objective function such that the continuous scores output by the model better correspond to the discrete and ordered scale. In some embodiments, the processor is configured to train the visual ML model prior to the fine-tuning, e.g., using the training methods described above. In other cases, the processor is configured to receive the visual ML from an external source (i.e., wherein the model has been trained prior to being received by the processor).

[0110] In some embodiments, the memory includes instructions stored thereon, which when executed by the processor, further cause the processor to apply the fine-tuned visual ML model (i.e.. fine-tuned using ordinally labeled data according to the methods of the disclosure, as described herein) to perform one or more specific clinical and / or diagnostic tasks (such as, e.g., any of the tasks disclosed above or herein). In some cases, the one or more tasks include detecting and / or quantifying tumor infiltration in a freshly sampled surgical tissue in order to, e.g., guide a surgical resection of a tumor. For example, the processor may be configured to receive one or more images from an external device, wherein the instructions then cause the processor to pass the one or more received images through the fine-tuned ML model in order to produce clinically actionable information. In these instances, the clinically actionable information may include, e.g., a tumor infiltration score quantifying the degree of tumor infiltration in each received image, as discussed above.

[0111] In some embodiments, the system further includes an SRH imaging device or system operably coupled with the processor, wherein the processor is configured to receive images from the SRH imaging device / system. In these instances, the SRH imaging device / system may include the NIO Imaging System (Invenio Imaging, Inc., Santa Clara, CA). In some embodiments, the system further includes a display configured to provide clinically actionable information produced by the fine-tuned ML model to a user. For example, a tumor infiltration score output by the fine-tuned ML model may be displayed to a surgeon in order to guide a surgical resection of a tumor. In some embodiments, the system further includes operator input devices configured to receive input from a user. For example, several images may be generated via the SRH imaging device / system and displayed, wherein a user may then select which images to pass through the fine-tuned ML model. In some cases, the operator input devices include a keyboard, a mouse, and / or a touch screen.

[0112] In another aspect, systems for performing the methods of generating a visualization identifying one or more regions of interest (ROIs) within a tissue image passed through a visual ML model (such as, e.g.. a medical foundation model), as well as systems for using generated visualizations to perform one or more clinical tasks (such as, e.g., verifying the detection and / or quantification of tumor infiltration by medical foundation models, as described above and herein) are provided. Aspects of the systems include: a processor including memory operably coupled to the processor wherein the memory includes instructions stored thereon, which when executed by the processor, cause the processor to: pass a set of patch keys including features or characteristics of the one or more ROIs through a visual ML model, wherein the set of patch keys includes examples of normal tissue and irregular tissue; calculate a few-shot visualization score for one or more patches of a tissue image passed through the model, wherein for each of the one or more patches, the calculating includes: determining a maximum similarity value between a patch of the tissue image and each of the irregular tissue patch keys by calculating a dot product between the tissue image patch and each of the irregular tissue patch keys; determining a maximum similarity value between the tissue image patch and each of the normal tissue patch keys by calculating a dot product between the tissue image patch and each of the normal tissue patch keys; and calculating a few-shot visualization score for the tissue image patch by determining a difference between the irregular tissue maximum similarity value and the normal tissue maximum similarity value; and generate the visualization by indicating patches of the tissue image having at or above a threshold few-shot visualization score.

[0113] In some embodiments, the processor is configured to train and / or fine-tune the visual ML model prior to generating the visualization, e.g., using the training methods described above. In other cases, the processor is configured to receive the visual ML from an external source (i.e., wherein the model has been trained and / or fine-tuned prior to being received by the processor). In some embodiments, the processor is configured to receive the set of patch keys and the tissue image from an external device. In some embodiments, the processor is configured to receive the set of patch keys and the tissue image from the same external device. In other instances, the processor is configured to receive the set of patch keys and the tissue image from different external devices. For example, the processor may be configured to receive the set of patch keys from an external memory device (such as, e.g., a USB flash drive) or a wireless connection, and the processor may be configured to receive the tissue image from an SRH imaging device or system. In some embodiments, the system further includes the SRH imaging device / system (e.g., a NIO Imaging System) operably coupled with the processor. In some embodiments, the system further includes a display configured to verify or confirm clinically actionable information produced by the ML model to a user. For example, the generated visualization may be displayed to a surgeon in order to verify or ensure the accuracy of a tumor infiltration score output by the ML model. In these instances, a visualization indicating patches of the tissue image as irregular (e.g., cancerous or tumor- infiltrated) when the surgeon deems them to be regular (i.e., non-cancerous or healthy), or indicating patches of the tissue image as regular when the surgeon deems them to be irregular, might result in the surgeon ignoring a tumor infiltration score output by the ML model. In some embodiments, the system further includes operator input devices configured to receive input from a user. For example, several patch keys may be displayed, wherein a user may then select which pass keys to use in order to generate visualization (i.e.. as described above / herein). In some cases, the operator input devices include a keyboard, a mouse, and / or a touch screen.

[0114] In some instances the systems further include one or more computers for complete automation or partial automation of the methods described herein. In some embodiments, systems include a computer having a computer readable storage medium with a computer program stored thereon.

[0115] In embodiments, the system includes an input module, a processing module and an output module. The subject systems may include both hardware and software components, where the hardware components may take the form of one or more platforms, e.g., in the form of servers, such that the functional elements, i.e., those elements of the system that carry out specific tasks (such as managing input and output of information, processing information, etc.) of the system may be carried out by the execution of software applications on and across the one or more computer platforms represented of the system.

[0116] Systems may include a display and operator input device. Operator input devices may, for example, be a keyboard, mouse, or the like. The processing module includes a processor which has access to a memory having instructions stored thereon for performing the steps of the subject methods. The processing module may include an operating system, a graphical user interface (GUI) controller, a system memory, memory storage devices, and input-output controllers, cache memory, a data backup unit, and many other devices. The processor may be a commercially available processor or it may be one of other processors that are or will become available. The processor executes the operating system and the operating system interfaces with firmware and hardware in a well-known manner, and facilitates the processor in coordinating and executing the functions of various computer programs that may be written in a variety of programming languages, such as Java, Perl, C++, Python, other high-level or low-level languages, as well as combinations thereof, as is known in the art. The operating system, typically in cooperation with the processor, coordinates and executes functions of the other components of the computer. The operating system also provides scheduling, input-output control, file and data management, memory management, and communication control and related services, all in accordance with known techniques. The processor may be any suitable analog or digital system. In some embodiments, the processor includes analog electronics which provide feedback control, such as for example negative feedback control.

[0117] The system memory may be any of a variety of known or future memory storage devices. Examples include any commonly available random access memory (RAM), magnetic medium such as a resident hard disk or tape, an optical medium such as a read and write compact disc, flash memory devices, or other memory storage device. The memory storage device may be any of a variety of known or future devices, including a compact disk drive, a tape drive, a removable hard disk drive, or a diskette drive. Such types of memory storage devices typically read from, and / or write to, a program storage medium (not shown) such as, respectively, a compact disk, magnetic tape, removable hard disk, or floppy diskette. Any of these program storage media, or others now in use or that may later be developed, may be considered a computer program product. As will be appreciated, these program storage media typically store a computer software program and / or data. Computer software programs, also called computer control logic, typically are stored in system memory and / or the program storage device used in conjunction with the memory storage device.

[0118] In some embodiments, a computer program product is described including a computer usable medium having control logic (computer software program, including program code) stored therein. The control logic, when executed by the processor the computer, causes the processor to perform functions described herein. In other embodiments, some functions are implemented primarily in hardware using, for example, a hardware state machine. Implementation of the hardware state machine so as to perform the functions described herein will be apparent to those skilled in the relevant arts.

[0119] Memory may be any suitable device in which the processor can store and retrieve data, such as magnetic, optical, or solid-state storage devices (including magnetic or optical disks or tape or RAM, or any other suitable device, either fixed or portable). The processor may include a general-purpose digital microprocessor suitably programmed from a computer readable medium carrying necessary program code. Programming can be provided remotely to processor through a communication channel, or previously saved in a computer program product such as memory or some other portable or fixed computer readable storage medium using any of those devices in connection with memory. For example, a magnetic or optical disk may cany the programming, and can be read by a disk writer / reader. Systems of the invention also include programming, e.g., in the form of computer program products, algorithms for use in practicing the methods as described above. Programming according to the present invention can be recorded on computer readable media, e.g., any medium that can be read and accessed directly by a computer. Such media include, but are not limited to: magnetic storage media, such as floppy discs, hard disc storage medium, and magnetic tape; optical storage media such as CD-ROM; electrical storage media such as RAM and ROM; portable flash drive; and hybrids of these categories such as magnetic / optical storage media.

[0120] The processor may also have access to a communication channel to communicate with a user at a remote location. By remote location is meant the user is not directly in contact with the system and relays input information to an input manager from an external device, such as a computer connected to a Wide Area Network (“WAN”), telephone network, satellite network, or any other suitable communication channel, including a mobile telephone (i.e., smartphone).

[0121] In some embodiments, systems according to the present disclosure may be configured to include a communication interface. In some embodiments, the communication interface includes a receiver and / or transmitter for communicating with a network and / or another device. The communication interface can be configured for wired or wireless communication, including, but not limited to, radio frequency (RF) communication (e.g., Radio-Frequency Identification (RFID), Zigbee communication protocols, WiFi, infrared, wireless Universal Serial Bus (USB), Ultra-Wide Band (UWB), Bluetooth® communication protocols, and cellular communication, such as code division multiple access (CDMA) or Global System for Mobile communications (GSM).

[0122] In one embodiment, the communication interface is configured to include one or more communication ports, e.g., physical ports or interfaces such as a USB port, an RS-232 port, or any other suitable electrical connection port to allow data communication between the subject systems and other external devices such as a computer terminal (for example, at a physician’s office or in hospital environment) that is configured for similar complementary data communication.

[0123] In one embodiment, the communication interface is configured for infrared communication, Bluetooth® communication, or any other suitable wireless communication protocol to enable the subject systems to communicate with other devices such as computer terminals and / or networks, communication enabled mobile telephones, personal digital assistants, or any other communication devices which the user may use in conjunction.

[0124] In one embodiment, the communication interface is configured to provide a connection for data transfer utilizing Internet Protocol (IP) through a cell phone network, Short Message Service (SMS), wireless connection to a personal computer (PC) on a Local Area Network (LAN) which is connected to the internet, or WiFi connection to the internet at a WiFi hotspot.

[0125] In one embodiment, the subject systems are configured to wirelessly communicate with a server device via the communication interface, e.g., using a common standard such as 802.11 or Bluetooth® RF protocol, or an IrDA infrared protocol. The server device may be another portable device, such as a smart phone, Personal Digital Assistant (PDA) or notebook computer; or a larger device such as a desktop computer, appliance, etc. In some embodiments, the server device has a display, such as a liquid crystal display (LCD), as well as an input device, such as buttons, a keyboard, mouse or touch-screen.

[0126] In some embodiments, the communication interface is configured to automatically or semi-automatically communicate data stored in the subject systems, e.g., in an optional data storage unit, with a network or server device using one or more of the communication protocols and / or mechanisms described above.

[0127] Output controllers may include controllers for any of a variety of known display devices for presenting information to a user, whether a human or a machine, whether local or remote. Tf one of the display devices provides visual information, this information typically may be logically and / or physically organized as an array of picture elements. A graphical user interface (GUI) controller may include any of a variety of known or future software programs for providing graphical input and output interfaces between the system and a user, and for processing user inputs. The functional elements of the computer may communicate with each other via system bus. Some of these communications may be accomplished in alternative embodiments using network or other types of remote communications. The output manager may also provide information generated by the processing module to a user at a remote location, e.g., over the Internet, phone or satellite network, in accordance with known techniques. The presentation of data by the output manager may be implemented in accordance with a variety of known techniques. As some examples, data may include SQL, HTML or XML documents, email or other files, or data in other forms. The data may include Internet URL addresses so that a user may retrieve additional SQL, HTML, XML, or other documents or data from remote sources. The one or more platforms present in the subject systems may be any type of known computer platform or a type to be developed in the future, although they typically will be of a class of computer commonly referred to as servers. However, they may also be a main-frame computer, a workstation, or other computer type. They may be connected via any known or future type of cabling or other communication system including wireless systems, either networked or otherwise. They may be co-located or they may be physically separated. Various operating systems may be employed on any of the computer platforms, possibly depending on the type and / or make of computer platform chosen. Appropriate operating systems include Windows, iOS, Oracle Solaris, Linux, IBM, Unix, and others.

[0128] Aspects of the present disclosure further include non-transitory computer readable storage mediums having instructions for practicing the subject methods. Computer readable storage mediums may be employed on one or more computers for complete automation or partial automation of a system for practicing methods described herein. In certain embodiments, instructions in accordance with the method described herein can be coded onto a computer-readable medium in the form of “programming”, where the term "computer readable medium" as used herein refers to any non-transitory storage medium that participates in providing instructions and data to a computer for execution and processing. Examples of suitable non-transitory storage media include a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, magnetic tape, non-volatile memory card, ROM, DVD-ROM, Blue -ray disk, solid state disk, and network attached storage (NAS), whether or not such devices are internal or external to the computer. A file containing information can be “stored” on computer readable medium, where “storing” means recording information such that it is accessible and retrievable at a later date by a computer. The computer-implemented method described herein can be executed using programming that can be written in one or more of any number of computer programming languages. Such languages include, for example, Python, Java, Java Script, C, C#, C++, Go, R, Swift, PHP, as well as many others.

[0129] The non-transitory computer readable storage medium may be employed on one or more computer systems having a display and operator input device. Operator input devices may, for example, be a keyboard, mouse, or the like. The processing module includes a processor which has access to a memory having instructions stored thereon for performing the steps of the subject methods. The processing module may include an operating system, a graphical user interface (GUI) controller, a system memory, memory storage devices, and input-output controllers, cache memory, a data backup unit, and many other devices. The processor may be a commercially available processor, or it may be one of other processors that are or will become available. The processor executes the operating system and the operating system interfaces with firmware and hardware in a well-known manner, and facilitates the processor in coordinating and executing the functions of various computer programs that may be written in a variety of programming languages, such as those mentioned above, other high level or low-level languages, as well as combinations thereof, as is known in the art. The operating system, typically in cooperation with the processor, coordinates and executes functions of the other components of the computer. The operating system also provides scheduling, input-output control, file and data management, memory management, and communication control and related services, all in accordance with known techniques. KITS

[0130] Kits are also provided for carrying out the methods described herein. In some embodiments, the kit includes software and / or memory for carrying out the methods of the disclosure via a computer. In some embodiments, the kit includes one or more components of a system for carrying out the methods of the disclosure, as described herein.

[0131] In some embodiments, the kit includes software for carrying out computer implemented methods of training a visual ML model to identify and encode tissue image features, as described herein. In some embodiments, the kit includes one or more components of a system for training a visual ML model to identify and encode tissue image features, as described herein. Such a system may include: a processor; memory operably coupled to the processor wherein the memory includes instructions stored thereon, which when executed by the processor, cause the processor to perform the training methods; a stimulated Raman histology (SRH) imaging device or system; a display; and one or more operator input devices. In some embodiments, the kit includes non-transitory computer readable storage media including the instructions of the memory of the system.

[0132] In some embodiments, the kit includes software for carrying out computer implemented methods of fine-tuning a visual ML model to output a continuous score corresponding to a discrete and ordered scale of ordinal categories, as described herein. In some embodiments, the kit includes one or more components of a system for fine-tuning a visual ML model to output a continuous score corresponding to a discrete and ordered scale of ordinal categories, as described herein. Such a system may include: a processor; memory operably coupled to the processor wherein the memory includes instructions stored thereon, which when executed by the processor, cause the processor to perform the fine-tuning methods; a SRH imaging device or system; a display; and one or more operator input devices. In some embodiments, the kit includes non-transitory computer readable storage media including the instructions of the memory of the system.

[0133] In some embodiments, the kit includes software for carrying out computer implemented methods of generating a visualization identifying one or more regions of interest (ROIs) within a tissue image passed through a visual ML model, as described herein. In some embodiments, the kit includes one or more components of a system for training a visual ML model to identify and encode tissue image features, as described herein. Such a system may include: a processor; memory operably coupled to the processor wherein the memory includes instructions stored thereon, which when executed by the processor, cause the processor to perform the visualization generating methods: a SRH imaging device or system; a display; and one or more operator input devices. In some embodiments, the kit includes non-transitory computer readable storage media including the instructions of the memory of the system.

[0134] In addition, the kits may further include, in certain embodiments, instructions for practicing the subject methods. These instructions may be present in the subject kits in a variety of forms, one or more of which may be present in the kit. For example, instructions may be present as printed information on a suitable medium or substrate, e.g., a piece or pieces of paper on which the information is printed, in the packaging of the kit, in a package insert, and the like. Another form of these instructions is a computer readable medium, e.g., diskette, compact disk (CD), flash drive, and the like, on which the information has been recorded. Yet another form of these instructions that may be present is a website address which may be used via the internet to access the information at a removed site.

[0135] UTILITY

[0136] The methods and systems of the present disclosure, e.g., as described above, find use in a variety of applications wherein it is desirable to rapidly, accurately, and reliably perform any number of a wide variety of clinical and / or diagnostic tasks. In some embodiments, the methods and systems described herein find use wherein it is desirable to quickly and verifiably identify tumor infiltration at microscopic resolution in order to, e.g., minimize the risk of residual tumor after surgery resection and improve patient outcomes. In some embodiments, the methods and systems described herein find use in quantifying tumor infiltration in order to, e.g., provide clinical decision support.

[0137] In certain aspects, the methods and systems of the present disclosure find use as a surgical adjunct. For example, the methods and systems of the present disclosure may be used to guide a surgical resection of a tumor. In some embodiments, the methods and systems described herein find use as a diagnostic tool. In these cases, the methods and systems of the present disclosure may be used to diagnose any number of solid cancer and / or brain tumor types. In some embodiments, the methods and systems described herein find use as a clinical decision support tool. For example, the methods and systems of the present disclosure may be used to verify the accuracy of a diagnosis or prognosis and / or to provide specific solid cancer and / or brain tumor characterizations having an impact on the efficacy of potential treatment options.

[0138] In some embodiments, the methods and systems of the present disclosure may advance the development of clinically useful devices, methods, or systems to monitor and / or therapeutically treat solid cancers and / or brain tumors in patients. In some embodiments, the subject methods and systems described herein find use wherein it is desirable to provide accurate and reliable clinically actionable information at a reduced cost and with enhanced efficiency. In some embodiments, the subject methods and systems may improve the ability to provide personalized therapies to patients afflicted with a variety of solid cancers and brain tumors.

[0139] EXEMPLARY NON-LIMITING ASPECTS OF THE DISCLOSURE

[0140] Aspects, including embodiments, of the present subject matter described above may be beneficial alone or in combination, with one or more other aspects or embodiments. Without limiting the foregoing description, certain non-limiting aspects of the disclosure are provided below. As will be apparent to those of ordinary skill in the art upon reading this disclosure, each of the individually numbered aspects may be used or combined with any one of the preceding or following individually numbered aspects. This is intended to provide support for all such combinations of aspects and is not limited to combinations of aspects explicitly provided below. It will be apparent to one of ordinary skill in the art that various changes and modifications can be made without departing from the spirit or scope of the invention.

[0141] 1. A method of training a visual machine learning model to identify and encode tissue image features, the method comprising: obtaining a set of tissue images comprising a plurality of neoplasm and / or tissue types, wherein each tissue image comprises multiple patches; training a tokenizer of the model to convert patches into tokens by identifying and encoding patch features using the patches of the tissue image set, wherein the tokenizer comprises a convolutional neural network (CNN) and the training comprises self-supervised contrastive learning; generating two or more views of each of a plurality of images of the tissue image set, wherein the plurality of images have been passed through the tokenizer; and training an image encoder of the model to identify and encode tissue image features using the two or more views of the plurality of tokenized images, wherein the image encoder comprises a vision transformer and the training comprises self-supervised learning.

[0142] 2. The method according to Aspect 1, wherein the method further comprises dividing images of the tissue image set into multiple patches.

[0143] 3. The method according to Aspect 1 or 2, wherein the model is a foundation model.

[0144] 4. The method according to any one of the preceding Aspects, wherein the set of tissue images each comprise a slide of tissue.

[0145] 5. The method according to Aspect 4, wherein the set of tissue images are obtained via stimulated Raman histology (SRH).

[0146] 6. The method according to Aspect 4 or 5, wherein the set of tissue images are obtained from a plurality of subjects.

[0147] 7. The method according to Aspect 6, wherein the set of tissue images are obtained from 1,000 or more subjects.

[0148] 8. The method according to Aspect 6 or 7, wherein the set of tissue images comprises a plurality of neoplasm types.

[0149] 9. The method according to Aspect 8, wherein the plurality of neoplasm types comprise central nervous system (CNS) tumors, wherein the CNS tumors comprise one or more of: high-grade gliomas (HGG), low-grade gliomas (LGG), meningiomas, pituitary adenomas, schwannomas, and / or metastatic tumors.

[0150] 10. The method according to any one of Aspects 6 to 9, wherein the set of tissue images comprises tissues from multiple different organ systems.

[0151] 11. The method according to Aspect 10, wherein the set of tissue images comprises neoplasms from multiple different organ systems.

[0152] 12. The method according to any one of Aspects 6 to 11, wherein the set of tissue images comprises 10,000 or more images. 13. The method according to any one of Aspects 6 to 12, wherein the self-supervised contrastive learning comprises hierarchical self-supervised learning via a hierarchical discriminative learning task.

[0153] 14. The method according to Aspect 13, wherein the tokenizer is trained to identify and encode patch features for each level of a multi-leveled hierarchy, wherein the hierarchy comprises a patch level, an image level, and a subject level.

[0154] 15. The method according to Aspect 14, wherein the hierarchical self-supervised learning of the tokenizer comprises: generating a plurality of positive patch pairs for each of the patch, image, and subject levels of the hierarchy from the set of tissue images; passing the positive patch pairs through the CNN; and adjusting the CNN to minimize a contrastive learning objective function, wherein the contrastive learning objective function comprises patch, image, and subject level components.

[0155] 16. The method according to any one of Aspects 13 to 15, wherein the CNN is a residual CNN.

[0156] 17. The method according to Aspect 16, wherein the residual CNN comprises ResNet-18, ResNet-34, ResNet-50, ResNet-101, or ResNet-152.

[0157] 18. The method according to any one of Aspects 6 to 17, wherein the two or more views of each of the plurality of images are generated by randomly: splitting each image into mutually exclusive patch sets; spatially cropping each patch set; and masking multiple patches from each spatially cropped patch set.

[0158] 19. The method according to Aspect 18, wherein the self- supervised learning of the image encoder comprises: passing each view generated for an image of the plurality of images through a different vision transformer of a Siamese architecture, wherein the transformers of the architecture share weights; and adjusting the transformers of the architecture to minimize a self-supervised objective function, wherein the self-supervised objective function comprises variance, invariance, and covariance components. 20. The method according to Aspect 19, wherein the self- supervised objective function is Variance-Invariance-Covariance Regularization (VICReg).

[0159] 21. The method according to Aspect 19 or 20. wherein the transformers of the architecture comprise two hidden layers, wherein each hidden layer comprises a dimension of 512 and 4 attention heads.

[0160] 22. The method according to any one of Aspects 19 to 21, wherein the transformers of the architecture comprise a component for generating positional embeddings.

[0161] 23. The method according to Aspect 22, wherein the positional embedding component comprises a Fourier feature positional embedding generator network.

[0162] 24. The method according to any one of Aspects 6 to 23, wherein the method further comprises fine-tuning the model and / or training an output module of the model to perform one or more specific tasks.

[0163] 25. The method according to Aspect 24, wherein the method further comprises training an output module of the model to perform a specific task by passing tissue images through the tokenizer and the image encoder of the model.

[0164] 26. The method according to Aspect 25, wherein the output module comprises a K- Nearest Neighbors (KNN) classifier.

[0165] 27. The method according to Aspect 25, wherein the output module comprises one or more output layers.

[0166] 28. The method according to Aspect 26, wherein the output module comprises a one layer linear output layer.

[0167] 29. The method according to Aspect 27 or 28. wherein the output module is trained to diagnose brain tumors.

[0168] 30. The method according to any one of Aspects 27 to 29, wherein the output module is trained to quantify tumor infiltration or tumor burden.

[0169] 31. The method according to any one of Aspects 27 to 30, wherein the output module is fine-tuned using a second set of tissue images, wherein the second set of tissue images comprises a smaller number of images than the initial set of images used to train the tokenizer and the image encoder of the model.

[0170] 32. The method according to Aspect 31, wherein the image encoder of the model is finetuned using the second set of tissue images. 33. The method according to Aspect 31 or 32, wherein each image of the second set of tissue images comprises an ordinal label corresponding to a category of a discrete and ordered scale.

[0171] 34. The method according to Aspect 33, wherein the method further comprises generating the ordinal labels for the second set of tissue images.

[0172] 35. The method according to Aspect 33 or 34, wherein the scale comprises a tumor infiltration scale quantifying the degree of tumor infiltration in each image of the second set of tissue images.

[0173] 36. The method according to Aspect 35, wherein the ordinal labels are generated using domain specific knowledge and one or more of: SRH imaging, hematoxylin and eosin (H&E) staining, and / or immunohistochemistry.

[0174] 37. The method according to Aspect 35 or 36, wherein the tumor infiltration scale comprises four categories, wherein: the first category corresponds to tissue images where no tumor is present; the second category corresponds to tissue images comprising atypical cells and / or images where it is possible, but not definite, that a tumor is present; the third category corresponds to tissue images comprising sparse tumor infiltration; and the fourth category corresponds to tissue images comprising dense tumor infiltration.

[0175] 38. The method according to any one of Aspects 33 to 37, wherein the fine-tuning comprises: passing tissue images of the second set through the tokenizer, the image encoder, and the output module of the model; and adjusting the output module and / or the image encoder to: maximize the latent distance between one or more encoded features of tissue images of the second set having different ordinal labels, and / or minimize the latent distance between one or more encoded features of tissue images of the second set having the same ordinal label.

[0176] 39. The method according to any one of Aspects 33 to 38, wherein the output module comprises an image scorer fine-tuned to output a continuous score corresponding to the discrete and ordered scale of the ordinal labels of the second set of tissue images. 40. The method according to Aspect 39, wherein the image scorer comprises a one layer linear scorer.

[0177] 41. The method according to Aspect 39 or 40. wherein the fine-tuning comprises: passing a batch of tissue images of the second set through the tokenizer, the image encoder, and the image scorer of the model in order to produce a continuous score for each tissue image of the batch; performing a pairwise comparison between all tissue images of the batch using the continuous scores and a binary cross entropy objective function; and adjusting the image scorer and / or the image encoder of the model to minimize the binary cross entropy objective function.

[0178] 42. The method according to Aspect 41, wherein class imbalance is addressed by oversampling tissue images of the second set corresponding to ordinal labels of a minority class.

[0179] 43. The method according to any one of Aspects 35 to 42, wherein the fine-tuned model is used to detect and quantify tumor infiltration in one or more unprocessed, label-free, surgical samples.

[0180] 44. The method according to Aspect 43, wherein the one or more surgical samples are imaged at lower resolutions and faster acquisition rates than the tissue images used to train and / or fine-tune the model.

[0181] 45. The method according to any one of Aspects 1 to 44, wherein the method further comprises generating a visualization identifying one or more regions of interest (ROIs) within a tissue image passed through the model.

[0182] 46. The method according to Aspect 45, wherein the visualization is generated via principal component analysis (PC A).

[0183] 47. The method according to Aspect 45. wherein the visualization is generated using a set of patch keys comprising features or characteristics of the one or more ROIs.

[0184] 48. The method according to Aspect 47, wherein the set of patch keys comprises two or more image patches selected by a medical professional.

[0185] 49. The method according to Aspect 47 or 48, wherein the set of patch keys comprises examples of normal tissue and irregular tissue. 50. The method according to Aspect 49, wherein the set of patch keys comprises diverse examples of normal brain parenchyma and diffuse glioma subtypes.

[0186] 51. The method according to Aspect 49 or 50. wherein regions of interest are identified by comparing the set of patch keys, having passed through the model, with each of the patches of the tissue image passed through the model.

[0187] 52. The method according to Aspect 51. wherein generating the visualization comprises: calculating a few-shot visualization score for one or more of the patches of the tissue image, wherein for each of the one or more patches the calculating comprises: determining a maximum similarity value between a patch of the tissue image and each of the irregular tissue patch keys by calculating a dot product between the tissue image patch and each of the irregular tissue patch keys; determining a maximum similarity value between the tissue image patch and each of the normal tissue patch keys by calculating a dot product between the tissue image patch and each of the normal tissue patch keys; and calculating a few-shot visualization score for the tissue image patch by determining a difference between the irregular tissue maximum similarity value and the normal tissue maximum similarity value; and generating the visualization by indicating patches of the tissue image having at or above a threshold few-shot visualization score.

[0188] 53. The method according to Aspect 52, wherein the irregular tissue maximum similarity value and the normal tissue maximum similarity value are determined using cosine similarity.

[0189] 54. The method according to Aspect 52 or 53, wherein few-shot visualization scores are only calculated for patches of the tissue image having at or above a threshold irregular tissue maximum similarity value, wherein tissue image patches having below the threshold irregular tissue maximum similarity value are classified as background patches.

[0190] 55. The method according to any one of Aspects 52 to 54, wherein generating the visualization further comprises: constructing a heatmap indicating one or more ROIs by rescaling each few-shot visualization score, wherein each patch of the tissue image is colored based on its rescaled few-shot visualization score; and overlaying the heatmap on the tissue image to generate the visualization.

[0191] 56. The method according to Aspect 55, wherein patches of the tissue image for which no few-shot visualization score was calculated are colored in the same manner as patches of the tissue image having below a threshold few-shot visualization score.

[0192] 57. The method according to any one of Aspects 1 to 56, wherein the model is used as a surgical adjunct.

[0193] 58. The method according to Aspect 57, wherein the model is used to guide a surgical resection of a tumor.

[0194] 59. The method according to any one of Aspects 1 to 58, wherein one or more hyperparameters used to train or fine-tune the model are optimized.

[0195] 60. The method according to Aspect 59, wherein the one or more hyperparameters are optimized by performing an ablation study.

[0196] 61. The method according to Aspect 59 or 60, wherein the one or more hyperparameters are optimized using a validation set of tissue images held out of the set of tissue images used to train or fine-tune the model.

[0197] 62. A method of fine-tuning a model to output a continuous score corresponding to a discrete and ordered scale of ordinal categories, the method comprising: obtaining a set of tissue images comprising a plurality of neoplasm and / or tissue types, wherein each tissue image of the set comprises an ordinal label corresponding to a category of the discrete and ordered scale; passing a batch of tissue images of the tissue image set through the model in order to produce a continuous score for each tissue image of the batch, wherein the model comprises: an image encoder trained to identify and encode tissue image features for each tissue image of the batch; and an output module comprising an image scorer configured to output the continuous score for each tissue image of the batch using the encoded tissue image features; performing a pairwise comparison between all tissue images of the batch using the continuous scores and a binary cross entropy objective function; and adjusting the image scorer of the model to minimize the binary cross entropy objective function such that the continuous scores output by the model better correspond to the discrete and ordered scale.

[0198] 63. The method according to Aspect 62, wherein the model comprises a model generated using the methods of any one of Aspects 1 to 61.

[0199] 64. The method according to Aspect 62, wherein the model comprises a CTransPath, Phycon, or UNI model.

[0200] 65. The method according to any one of Aspects 62 to 64, wherein the set of tissue images used to fine-tune the model comprises a smaller number of images than an initial set of tissue images used to train the model.

[0201] 66. The method according to any one of Aspects 62 to 65, wherein the image encoder of the model is adjusted to minimize the binary cross entropy objective function.

[0202] 67. The method according to any one of Aspects 62 to 66, wherein the method further comprises generating the ordinal labels for the set of tissue images.

[0203] 68. The method according to Aspect 67, wherein the scale comprises a tumor infiltration scale quantifying the degree of tumor infiltration in each image of the set of tissue images

[0204] 69. The method according to Aspect 68, wherein the ordinal labels are generated using domain specific knowledge and one or more of: SRH imaging, hematoxylin and eosin (H&E) staining, and / or immunohistochemistry.

[0205] 70. The method according to Aspect 68 or 69, wherein the tumor infiltration scale comprises four categories, wherein: the first category corresponds to tissue images where no tumor is present; the second category corresponds to tissue images comprising atypical cells and / or images where it is possible, but not definite, that a tumor is present; the third category corresponds to tissue images comprising sparse tumor infiltration; and the fourth category corresponds to tissue images comprising dense tumor infiltration.

[0206] 71. The method according to any one of Aspects 62 to 70, wherein the image scorer comprises a one layer linear scorer.

[0207] 72. The method according to any one of Aspects 62 to 71, wherein the fine-tuning comprises: adjusting the output module and / or the image encoder to: maximize the latent distance between one or more encoded features of tissue images of the set having different ordinal labels, and / or minimize the latent distance between one or more encoded features of tissue images of the set having the same ordinal label.

[0208] 73. The method according to any one of Aspects 62 to 72, wherein class imbalance is addressed by oversampling tissue images of the set corresponding to ordinal labels of a minority class.

[0209] 74. The method according to any one of Aspects 62 to 73, wherein the fine-tuned model is used to detect and quantify tumor infiltration in one or more unprocessed, label-free, surgical samples.

[0210] 75. The method according to Aspect 74, wherein the one or more surgical samples are imaged at lower resolutions and faster acquisition rates than the tissue images used to train and / or fine-tune the model.

[0211] 76. A method of generating a visualization identifying one or more regions of interest (ROIs) within a tissue image passed through a model, the method comprising: passing a set of patch keys comprising features or characteristics of the one or more ROIs through the model, wherein the set of patch keys comprises examples of normal tissue and irregular tissue; calculating a few-shot visualization score for one or more patches of the tissue image, wherein for each of the one or more patches, the calculating comprises: determining a maximum similarity value between a patch of the tissue image and each of the irregular tissue patch keys by calculating a dot product between the tissue image patch and each of the irregular tissue patch keys; determining a maximum similarity value between the tissue image patch and each of the normal tissue patch keys by calculating a dot product between the tissue image patch and each of the normal tissue patch keys; and calculating a few-shot visualization score for the tissue image patch by determining a difference between the irregular tissue maximum similarity value and the normal tissue maximum similarity value; and generating the visualization by indicating patches of the tissue image having at or above a threshold few-shot visualization score.

[0212] 77. The method according to Aspect 76. wherein the model comprises a model generated using the methods of any one of Aspects 1 to 61.

[0213] 78. The method according to Aspect 76, wherein the model comprises a CTransPath, Phycon, or UNI model.

[0214] 79. The method according to any one of Aspects 76 to 78, wherein the set of patch keys comprises two or more image patches selected by a medical professional.

[0215] 80. The method according to any one of Aspects 76 to 79, wherein the set of patch keys comprises diverse examples of normal brain parenchyma and diffuse glioma subtypes.

[0216] 81. The method according to any one of Aspects 76 to 80, wherein the irregular tissue maximum similarity value and the normal tissue maximum similarity value are determined, at least in part, using cosine similarity.

[0217] 82. The method according to any one of Aspects 76 to 81, wherein few-shot visualization scores are only calculated for patches of the tissue image having at or above a threshold irregular tissue maximum similarity value, wherein tissue image patches having below the threshold irregular tissue maximum similarity value are classified as background patches.

[0218] 83. The method according to any one of Aspects 76 to 82, wherein generating the visualization further comprises: constructing a heatmap indicating one or more ROIs by rescaling each few-shot visualization score, wherein each patch of the tissue image is colored based on its rescaled few-shot visualization score; and overlaying the heatmap on the tissue image to generate the visualization.

[0219] 84. The method according to Aspect 83, wherein patches of the tissue image for which no few-shot visualization score was calculated are colored in the same manner as patches of the tissue image having below a threshold few-shot visualization score. EXPERIMENTAL

[0220] As demonstrated in the above disclosure, the present invention has a wide variety of applications. The following examples are put forth so as to provide those of ordinary skill in the art with a complete disclosure and description of how to make and use the present invention and are not intended to limit the scope of what the inventors regard as their invention nor are they intended to represent that the experiments below are all or the only experiments performed. Those of skill in the art will readily recognize a variety of noncritical parameters that could be changed or modified to yield essentially similar results. Efforts have been made to ensure accuracy with respect to numbers used (e.g. amounts, percentages, etc.) but some experimental errors and deviations should be accounted for.

[0221] Example 1: Foundation models for fast, label-free detection of glioma infiltration

[0222] 1.1. Overview

[0223] A critical challenge in glioma treatment is detecting tumor infiltration during surgery to achieve safe maximal resection [1-3]. Unfortunately, safely resectable residual tumor is found in the majority of glioma patients after surgery, causing early recurrence and decreased survival [4-6], FastGlioma, a visual foundation model for fast (< 10 seconds) and accurate detection of glioma infiltration in fresh, unprocessed surgical tissue, is presented herein. FastGlioma was pretrained using large-scale self-supervision (~4 million images) on rapid, label-free, optical microscopy, and fine-tuned to output a normalized score that indicates the degree of tumor infiltration within whole slide optical images. In a prospective, multicenter, international testing cohort of diffuse glioma patients (n=220), FastGlioma was able to detect and quantify the degree of tumor infiltration with an average area under the ROC curve of 92.1 + / - 0.9%. FastGlioma outperformed image-guided and fluorescence-guided adjuncts for detecting tumor infiltration during surgery by a wide margin in a head-to-head, prospective study (n=129). FastGlioma performance remained high across diverse patient demographics, medical centers, and diffuse glioma molecular subtypes as defined by the World Health Organization (WHO). FastGlioma shows zero-shot generalization to other adult and pediatric brain tumor diagnoses, demonstrating the potential for the foundation model to serve as a general-purpose adjunct for guiding brain tumor surgeries. These findings represent the transformative potential of medical foundation models to unlock the role of artificial intelligence in the care of cancer patients.

[0224] 1.2. Introduction

[0225] The importance of detecting tumor infiltration within surgical specimens during an operation has been recognized for over a century [7], Despite efforts to deliver precision healthcare to cancer patients, residual tumor after surgery is a major public health problem within the United States and globally [8], For solid cancers and brain tumors, residual tumor results in worse quality-of-life, decreased patient survival, and increased burden on healthcare systems [2,3,9,10]. For example, in glioma surgery, extent of resection is known to predict overall patient survival in both lower grade gliomas and glioblastomas (FIGS 17A- 17B). Rates of residual tumor have not improved over the last two decades, and corrective surgical procedures and post-surgical treatment have an estimated total cost of more than $1 billion annually in the U.S. [9,11],

[0226] Here, FastGlioma, an open-source, artificial intelligence (Al)-based diagnostic system for detecting brain tumor infiltration in fresh, unprocessed, unlabeled surgical tissue at the patient’s bedside, is presented. Conventional microscopic analysis with hematoxylin and eosin (H&E)-stained tissue during surgery is slow, resource intensive, and reliant on a shrinking pathology workforce

[0012] , FastGlioma solves these limitations by combining rapid, user-friendly, bedside optical microscopy and visual foundation models trained on a diverse dataset of over UK surgical specimens and 4 million unique microscopic fields of view. Foundation models, such as the GPT-4 and DALL-E3 models, are Al models trained on massive, diverse datasets and can be adapted to a wide range of downstream tasks [13-16], Foundation models for medical Al have the potential to solve challenging clinical tasks by requiring little to no task- specific annotations to achieve good generalization performance and clinically meaningful results [17-19], FastGlioma combines visual foundation model training and efficient fine-tuning strategies to generalize across patient demographics, healthcare systems, and WHO brain tumor diagnoses with minimal supervised training. As a general-purpose surgical adjunct, FastGlioma provides physicians with real-time, accurate, and clinically actionable diagnostic information within seconds of tissue biopsy that can improve the surgical care of brain tumor patients, thereby increasing patient’s quality-of-life and overall survival.

[0227] 1.3. Results

[0228] 1.3.1. Visual Foundation Model Training

[0229] The FastGlioma workflow starts during the surgical resection of a brain tumor (FIG. 1). Fresh tissue specimens are sampled at the surgical margins of a resection cavity to detect microscopic tumor infiltration. Specimens are imaged at the patient’s bedside using stimulated Raman histology (SRH), a rapid, label-free, sub-micron-resolution, optical imaging method [20,21], A major advantage of SRH over other intraoperative imaging methods is that image contrast is generated from the intrinsic biochemical properties of the specimen and does not rely on stains, dyes, or labels. SRH images can be acquired in either full-resolution mode (~ 100 seconds) or fast mode (~10 seconds, lower resolution) depending on clinician’s preference and clinical context (FIG. 5). Whole slide images range from 1 mm2-10 mm2and are divided into smaller, non-overlapping fields-of-view, or patches, for model input. The FastGlioma foundation model training dataset was acquired from 13 medical centers and includes imaging data from over 3000 patients, spanning the diagnostic spectrum of central nervous system tumors and human cancers (FIG. 6). A two-stage selfsupervised learning method specifically designed for training vision transformer architectures on whole slide microscopy images was developed

[0022] , First, patch features are extracted using a patch tokenizer trained using hierarchical self-supervised learning (FIGS. 7A-7B)

[0023] . Second, whole slide image features are learned by generating two views of the same whole slide image by randomly splitting, cropping, and masking the patch tokens. Both views then undergo a feedforward pass through a vision transformer, with the patches as input tokens, and a whole slide self-supervised objective is minimized (FIG. 7C)

[0024] .

[0230] Using this two-stage strategy to train a vision foundation model on the full SRH dataset, high-quality whole slide representations and state-of-the-art performance on a previously benchmarked multiclass brain tumor diagnosis task were obtained

[0025] , The classification task includes diagnosing the most common brain tumor types, including diffuse lower grade and malignant gliomas with variable amounts of tumor infiltration. The high- quality patch-level and slide-level self- supervised training results in minimal trade-off between imaging speed / resolution versus model performance, with fast SRH reaching a mean class accuracy of 88.0 + / - 2.1% versus full resolution SRH at 90.2 + / - 3.0% (FIG. 7C). The proposed vision foundation model training strategy allows for clinicians to seamlessly analyze multiple specimens throughout the tumor resection without sacrificing model performance.

[0231] FIG. 1 demonstrates the FastGlioma workflow. During tumor resection, the surgeon samples tissue from the surgical margin. The portable SRH imaging system acquires microscopic images in the operating room, performed by a single technician using simple touchscreen instructions. A freshly excised surgical specimen is loaded directly into a custom microscope slide and inserted into the SRH imager without the need for tissue processing. Additional details on image acquisition can be found in FIG. 5. SRH images can be virtually stained using an H&E-like color scheme for clinician review

[0020] . A whole slide SRH image is divided into patches and each patch undergoes a feedforward pass through a patch tokenizer (FIG. 7A). The patch tokens, plus an appended classification token <CLS>, are then input into a whole slide SRH encoder that is a vision transformer. The patch tokenizer and whole slide encoder are pretrained as a visual foundation model using large-scale selfsupervision (FIG. 7B). For tumor infiltration scoring, a slide scorer model is fine-tuned to output a normalized continuous score between 0-1 that predicts the degree of tumor infiltration within the whole slide image that corresponds to a 4-tier whole slide ordinal infiltration scale as defined by expert neuropathologists (FIGS. 6 and 8). Ordinal labels are weak because they apply to the slide-level only. Despite the weak labels, FastGlioma provides regional interpretability by identifying area within whole slides SRH images with high probability of tumor infiltration. Scale bars, 100 microns.

[0232] FIGS. 5A to 5C illustrate the FastGlioma workflow and dataset generation. FIG. 5A: FastGlioma is intended for patients with a suspected diffuse glioma on preoperative imaging studies. Surgical specimens are sampled from the patient resection cavity. During sampling, neuronavigation coordinates and 5-ALA fluorescence status are recorded by a laboratory technician in real time. Core specimens are sent for clinical pathologic testing, including histologic and molecular classification with SRH

[0044] , and the margin specimens are sent for FastGlioma prediction. Surgeons were encouraged to sample within the resection cavity according to their clinical judgement and context. FIG. 5B: Each surgical specimen is loaded into a premade microscope slide and imaged at the patient’s bedside using the NIO Imaging System (Invenio Imaging, Inc., Santa Clara, CA). A scout photograph of the specimen is taken, and the imaging field of view selected by the user using simple touchscreen instructions. SRH imaging does not require a skilled imaging technician. SRH images are acquired sequentially as strips at two Raman shifts, 2845 cm’1and 2930 cm-1. The size and number of strips to be acquired is set by the operator who defines the desired image size. Images can be acquired in either two-channel, full resolution mode or one-channel (2845 cm’1only), low resolution fast mode to decrease imaging time by a factor of 10. FIG. 5C: Following SRH imaging, surgical specimens were removed from the microscope, formalin- fixed, and paraffin-embedded for H&E and immunohistochemistry (IHC) staining. Similar to previous protocols, tumor-specific IHC labels were used, if available, to determine the degree of tumor infiltration based on molecular genetics [4], A neuropathologist from each institution then scores the degree of tumor infiltration within the SRH images and the H&E / IHC- stained slides on a 0-3 scale. Slide-level predictions are generated from imagelevel FastGlioma predictions by taking the average image-level prediction. Scale bars, 100 microns.

[0233] FIGS. 6A to 6C depict SRH foundation model and FastGlioma training datasets. FIG. 6A: The foundation model training dataset consisted of over 1 IK whole slide SRH images from 13 medical centers across the United States and Europe. Similar to other foundation model training datasets, such as Google’s JTF-3B58 or Meta’s SEER59 datasets, the SRH dataset was generated ‘in the wild’ at the discretion of the medical centers and treating physicians. The dataset consisted of a diverse set of central nervous system tumors, but also contained specimens from other organ systems, including head and neck tumors, breast, prostate, and lung surgical specimens. FIG. 6B: The diffuse glioma infiltration dataset consisted of two parts. The training dataset was generated entirely from a previously published study evaluating tumor margin specimens with SRH [4], No additional annotation or data collection was performed for this study. The multicenter diffuse glioma testing dataset was generated from three tertiary medical centers with brain tumor programs. Details of data collection and annotation can be found in the Methods section, below. FIG. 6C: Details regarding the patient demographics, WHO glioma subtypes, recurrence status, tumor grade, and institution for the prospective testing dataset is shown. FIGS. 7 A to 7C illustrate self-supervised foundation model training. FIG. 7A: The FastGlioma patch tokenizer is trained via self- supervised hierarchical contrastive learning on over 4 million 300 x 300 pixel SRH patches. Positive patch pairs are generated using the patch-slide-patient hierarchy of clinical SRH images. Patches undergo a feedforward pass through a ResNet-34 model and a contrastive learning objective is minimized

[0023] , FIG. 7B: t-SNE patch representations are plotted and colored according to the ground truth whole slide label. Patch-level self- supervised training results in representations that span a tumor infiltration axis without the need for patch-level supervision. FIG. 7C: Whole slide selfsupervision is accomplished by generating two views of a whole slide SRH image via random splitting, cropping, and masking of the patch tokens. The random sequence of patches in each view then undergo a feedforward pass through a slide transformer. The slidelevel representations are then projected into a lower-dimensional projection space where a self-supervised objective is minimized (VICReg

[0050] ). The whole slide self-supervised training strategy was benchmarked using several internal SRH classification metrics, including multiclass brain tumor classification. Mean class accuracy of ablation studies are plotted. Whole slide SRH representations are plotted via t-SNE and show tumor diagnosis discrimination. Trained slide transformer produces interpretable multi-headed self-attention maps that differentiate tumor infiltration from normal brain and nondiagnostic regions. Scale bars, 100 microns.

[0234] 1.3.2. Fine-Tuning for Infiltration Scoring

[0235] Fine-tuning can improve task-specific foundation model performance

[0026] ; however, a major disadvantage of finetuning is the need for new large and annotated datasets

[0016] . Biomedical datasets are challenging to obtain, making efficient fine-tuning strategies essential to ensure safe and effective medical Al performance. To adapt the SRH foundation model for tumor infiltration detection and scoring, a data-efficient ordinal representation learning method was developed, called ordinal metric learning, that was developed and implemented using an existing, previously annotated SRH tumor infiltration dataset [4], The tumor infiltration dataset is lOOx smaller than the foundation SRH dataset (~100 versus ~ 10,000 images) and was annotated by three expert neuropathologists. The pathologists ranked the degree of tumor infiltration within each SRH image on a consensus 4-tier scale: normal brain tissue / no tumor (0), atypical cells / possible tumor but not definitive (1), sparse tumor infiltration (2), dense tumor infiltration (3). Each specimen then underwent H&E staining and tumor marker- specific immunohistochemistry, such as isocitrate dehydrogenase- 1 / 2 (IDH) and p53, to confirm tumor infiltration scores [4],

[0236] Using this tumor infiltration dataset, ordinal metric learning fine-tunes the SRH foundation model by maximizing the latent distance, or metric, between whole slide SRH images with different degrees of tumor infiltration (FIGS. 9A-9B). In addition, the increased efficiency of ordinal metric learning stems from enforcing that the model ranks images based on their tumor infiltration by performing a pairwise comparison between all images in a training mini-batch. The model then implicitly learns similar representations for whole slide SRH images with the same degree of tumor infiltration and the representations are appropriately ordered in the whole slide embedding space (FIG. 9C). The fine-tuned model uses a linear slide scoring layer to output a single scalar value between 0-1 that indicates the degree of tumor infiltration within a whole slide SRH image, which provides clinically actionable information for each specimen within seconds of biopsy. In addition to assessing surgical margins, FastGlioma can identify specimens with dense tumor infiltration early in the surgical resection to obtain high-yield diagnostic tissue for intraoperative and final pathologic diagnosis. Ordinal metric learning is demonstrated herein to outperform other state-of-the-art ordinal regression methods, especially when training data is limited, achieving a mean area under the receiver operator characteristic curve (mAUROC) of 88.7 + / - 1.6% on hold-out testing using the SRH infiltration dataset.

[0237] FIGS. 9A to 9D illustrate speed versus performance trade-off ablation studies. FIG. 9A: The NIO Imaging System can speed imaging time by acquiring only a single-channel image or by decreasing line scanning density. Fast SRH is a single channel greyscale image acquired at Raman wavenumber 2845 cm4and scanning every 5th line, which corresponds to approximately 10X speedup in imaging time. Examples of full resolution and low resolution images are shown. FIG. 9B: Hold-out cross validation ablation studies (n=161) to examine the trade-off between image resolution / speed and model performance (plotted as mean + / - s.d.). A less than 2% decrease in mAUROC was found with the 10X speed-up of Fast SRH. FIG. 9C: How tumor infiltration scores are related was examined using both full resolution and fast SRH imaging. Scores are strongly correlated with a Pearson correlation coefficient, r, of 0.90 and score residual standard deviation of 0.102. FIG. 9D: A demonstration of the similarity between the whole slide encoder self-attention coefficients for full resolution versus fast SRH image. Despite significant pixel-level differences between the two imaging domains, self-attention and tumor infiltration scores on the whole slides are similar. Note in the Fast SRH row above, virtual H&E color-scheme is shown for visualization purposes only. The true underlying image is the low-resolution Fast SRH image. Scale bars, 100 microns.

[0238] 1.3.3. Prospective Testing of FastGlioma

[0239] The fine-tuned FastGlioma model was tested in a multicenter, prospective cohort of diffuse glioma patients to evaluate how the model generalizes across different continents, medical centers, patient demographics, and WHO diffuse glioma molecular subgroups. Model testing was designed as a single-arm, non-inferiority, diagnostic clinical trial with a minimum sample size of 565 SRH specimens for both IDH-mutant and IDH-wildtype diffuse gliomas (1130 total specimens). All specimen processing and annotation was performed using the validated 4-tiered tumor infiltration rating system mentioned above. Three tertiary medical centers across the United States and Europe were included as testing recruitment sites: University of California San Francisco (UCSF), New York University (NYU), and Medical University of Vienna (MUV). Patients were recruited as a consecutive cohort of adult patients (>18 years) with diffuse gliomas who underwent tumor resection. A total of 220 patients were included (FIG. 6), resulting in 767 IDH-mutant specimens and 659 IDH- wildtype specimens.

[0240] FastGlioma achieved a mean AUROC of 92.1 + / - 0.9% for differentiating the four degrees of diffuse glioma infiltration (FIG. 2A). Normalized infiltration scores were strongly correlated with ground truth ordinal labels, with a correlation coefficient of p = 0.77 (95% confidence interval 0.74-0.78, p-0.00) (FIG. 2B). Importantly, visual foundation model pretraining allowed FastGlioma to generalize to the fast, low-resolution images acquired at 10X imaging speed without a clinically significant reduction in prediction performance (<1% decrease) (FIG. 2C). SRH foundation model fine-tuning with ordinal metric learning resulted in the FastGlioma representing whole slide SRH images on a linear infiltration axis that stratifies SRH images according to their ground truth infiltration label (FIG. 2D). Model performance and infiltration scores were consistent across patient demographics, including sex, age, and race. mAUROC remained high across medical centers: UCSF 92.1 + / - 0.2%, MUV 88.6 + / - 0.13%. NYU 92.9 + / - 0.1% (FIGS. 10A-10B). While other surgical adjuncts can be limited to a single brain tumor diagnosis, FastGlioma was intentionally designed as a general-purpose model for all diffuse gliomas and degrees of tumor infiltration. FastGlioma maintained accurate tumor infiltration scores despite the significant cytologic and histoarchitectural differences related to tumor grade, molecular genetics, treatment effect, or WHO subtypes (FIG. 2E and FIGS 10C-10E). Lastly, FastGlioma outperforms cellularity / segmentation-based methods for tumor detection by a large margin with a > 10% mAUROC performance increase, especially in regions of lower cellularity or tumor densities (FIG. 11).

[0241] FIGS. 2A to 2E provide prospective clinical testing results. FIG. 2A: Prediction results for the full prospective, international, multicenter testing cohort of diffuse gliomas patients (n = 220) are shown. ROC curves (plotted as mean + / - s.d.) show average performance for predicting four levels of tumor infiltration (see FIG. 6 for subgroup analysis). SRH foundation model pretraining showed strong prediction performance without fine-tuning. FastGlioma that included fine-tuning with ordinal metric learning had a 3.2% increase in overall performance. FastGlioma outperforms models trained using standard supervised training (84.7 + / - 1.1% mAUC). FIG. 2B: Box and whisker plots, shown in the standardized quartile format, of FastGlioma infiltration scores by ground truth value are shown. Scores had strong correlation with ground truth ordinal scores (p = 0.77 95% confidence interval 0.74-0.78). Individual scores are shown in a histogram and correspond to AUROC values in FIG. 2A. FIG. 2C: FastGlioma performance on full resolution versus low resolution SRH images is shown (plotted as mean + / - s.d.). FastGlioma allows for 10X increase in imaging speed with minimal performance tradeoff. FIG. 2D: Whole slide SRH representations are plotted on a linear discriminant axis. FastGlioma learned representations that rank whole slide SRH images on a near-linear tumor infiltration axis. FIG. 2E: Subgroup analysis by WHO adult-type diffuse glioma subtypes (ROC curves plotted as mean + / - s.d.). FastGlioma performs well across all three adult-type diffuse gliomas. Importantly, FastGlioma performs well on lower grade gliomas where tumor infiltration and tissue cellularity can be low (FTG. 1 1). Low grade and lower tumor infiltration are major challenges for other surgical adjuncts, such as fluorescence-guided surgery.

[0242] FIGS. 10A to 10E illustrate extended prospective testing results with subgroup analysis. FIG. 10A: Subgroup analysis by patient demographics. FastGlioma scores by sex, age, and race are shown using box and whisker plots in the standardized quartile format. FIG. 10B: Subgroup analysis by medical center. mAUROC value is plotted for both FastGlioma and the SRH foundation model (plotted as mean + / - s.d.). Performance remained high across the three external testing medical centers. FIG. IOC: Mean AUROC values are plotted by tumor grade. Importantly, similar performance is seen across all diffuse glioma grades, indicating that FastGlioma scores are not reliant on specific histologic features found within diffuse glioma grades. Lower grade cytologic features and low tumor infiltration were preliminary concerns about the performance of FastGlioma that did not bear out in prospective testing. Mean AUROC values and standard deviations are shown. FIG. 10D: Subgroup analysis by index / first surgery versus surgery for recurrent tumor (ROC curves plotted as mean + / - s.d.). A total of 442 surgical specimens were sampled from patients with recurrent tumors. FastGlioma maintained good performance for detecting tumor infiltration in recurrent tumors, which is known to be challenging for intraoperative frozen sectioning [42,43]. FIG. 10E: Whole slide SRH representations are shown for IDH mutant and IDH wildtype diffuse gliomas. Points are colored by their ground truth tumor infiltration scores. Despite differences in molecular features, the representation of whole slide SRH images is similarly distributed according to tumor infiltration. These findings demonstrate that FastGlioma is invariant to molecular markers, tumor subtypes, and histologic features for defining the degree of tumor infiltration within SRH images. This is contrasted with 5-ALA fluorescence which is only approved for use in glioblastomas, IDH-wildtype.

[0243] FIGS. 11 A to 11B provide cytologic analysis of FastGlioma tumor infiltration scores. FIG. 11A: Increased cellularity, or cellular density, is a cytologic feature of diffuse gliomas and is correlated with the degree of tumor infiltration [55,64], However, cellularity alone is a poor predictor for tumor infiltration across tumor grade, diffuse glioma molecular subgroups, and degree of infiltration. Cellularity as a predictor performs progressively worse as degree of tumor infiltration decreases and with lower tumor grades [4], The relationship between cellularity and FastGlioma scores was evaluated. Cellularity and FastGlioma scores have a Pearson correlation coefficient of 0.65. Consistent with previous results, surgical specimens were identified without tumor infiltration (Os) that have cellular densities comparable to specimens with a low to moderate degree of tumor infiltration (Is and 2s). Additionally, some dense tumor specimens (3s) were found to have relatively low cellularity (< 10). Examples of both of these scenarios are shown. The kernel density estimates (KDE) of the FastGlioma scores demonstrate the expected ordinal distribution across the ground truth labels; cellularity KDEs are poorly distributed according to ground truth labels, especially for 0-2 scores. FastGlioma has a clinically significant increase in mAUROC compared to cellularity-based predictions. FIG. 11B: Cellularity versus FastGlioma scores are plotted by molecular subgroup. Despite differences in cytologic features (anaplasia, pleomorphism, nuclear-cytoplasmic ratios) across molecular subgroups, FastGlioma tumor infiltration scoring is consistently accurate. Diffuse glioma molecular subgroups show a similar correlation between cellularity and score values. FastGlioma effectively uses the full tumor infiltration score range to quantify the degree of tumor infiltration across all molecular subgroups. FastGlioma outperforms cellularity-based metrics for predicting degree of tumor infiltration. Scale bars, 100 microns.

[0244] 1.3.4. Interpretability and Zero-Shot Results

[0245] Interpretable visualizations that highlight regions of tumor infiltration within whole slide images are essential to ensure safe, reliable, and trustworthy predictions

[0027] , A visualization strategy was developed that leverages FastGlioma’ s self-supervised training and generalization performance, called few-shot visualizations. For a query SRH region or patch, few-shot visualizations use a small support set of physician- selected SRH patch examples as keys to assess feature similarity (~10 images). The support set includes a diverse selection of diffuse gliomas and normal brain parenchyma SRH patch examples.

[0246] FastGlioma few-shot visualizations compare the query cosine similarity with tumor keys and the dissimilarity with normal keys to generate tumor infiltration heatmaps. Fewshot visualizations are a flexible foundation model-based interpretability framework that can accommodate any support set selected by pathologists, clinicians, or investigators. Adding or changing support set examples does not require model retraining. FIG. 3 contains interpretable few-shot FastGlioma visualizations from the prospective testing medical centers. Heatmaps demonstrate slide-level segmentation of tumor infiltrated regions. Visualization quality generalizes across medical centers, degrees of tumor infiltration, and molecular subgroups (FIG. 12). Similar to FastGlioma tumor infiltration scoring, few-shot visualizations reliably identify regions of tumor infiltration despite variations in the underlying histologic features. FastGlioma learned invariant tumor infiltration representations across the molecular subgroups of diffuse gliomas (FIG. 12A-12B). Importantly, FastGlioma demonstrates medical foundation model properties with zero- shot generalization. FastGlioma achieves accurate tumor infiltration detection and few-shot visualization for non-glioma brain tumor diagnoses, including metastatic brain tumors, primary central nervous system lymphomas, embryonal tumors, and meningiomas (FIG. 13). These results demonstrate the advantage of visual foundation models for medical Al applications and the potential to generalize to other human cancers without requiring extensive model retraining or fine-tuning.

[0247] FIG. 3 illustrates general and interpretable FastGlioma predictions. Diffuse glioma specimens from the four study medical centers are shown. Specimens span the full diagnostic spectrum of WHO adult-type diffuse gliomas. Few-shot visualizations highlight regions of tumor infiltration contained within surgical specimens using a support set of expert-physician selected SRH patch keys in order to match the underlying SRH query features using FastGlioma. As a visual foundation model, FastGlioma requires few examples (~10, UM images only) to achieve interpretable tumor infiltration visualizations that generalize across medical centers and degrees of tumor infiltration. FastGlioma predictions and visualizations are consistent across diverse histologic features, including dense tumor, infiltrated normal brain parenchyma, tumor-associated immune cell infiltration, and atypical cells. FastGlioma differentiates between increased cellularity due to tumor cell infiltration versus non- neoplastic cells, such as red blood cells (RBCs). Additional detailed few-shot visualizations can be found in FIG. 12. Scale bars, 100 microns.

[0248] FIGS. 12A to 12B provide few-shot visualizations of diffuse glioma infiltration. FIG. 12A: A panel of whole slide SRH images across diffuse glioma molecular subgroups are shown. Cases have been selected across the prospective testing medical centers. Few-shot visualizations identify tumor infiltrated regions in molecular oligodendrogliomas, astrocytomas, and glioblastoma. Insets are shown from each whole slide image to demonstrate the diversity of underlying histologic and cytologic features found within diffuse glioma subgroups (Right). Also shown are several SRH images with 0 ground truth scores from patients diagnosed with glioblastomas who previously underwent surgery and chemoradiation. SRH images show evidence of treatment effect, including reactive astrocytes and hyalinized blood vessels. Despite these non-neoplastic pathologic findings, FastGlioma gives low tumor infiltration scores and does not identify regions of dense tumor infiltration. FIG. 12B: Principal component analysis (PC A) was used on patch features extracted by FastGlioma. PCA provides an unsupervised visualization strategy to better elucidate the learned patch features. It was observed that tumor infiltrated regions are matched between SRH images despite changes in degree of tumor infiltration, molecular subtype, and histologic features. Detailed comparisons between FastGlioma scores, few-shot visualizations, and SRH images can be found at (fastglioma.mlins.org). Scale bars, 100 microns.

[0249] FIGS. 13A to 13B provide zero-shot predictions and visualizations. FastGlioma’ s zero-shot generalization to non-adult-type diffuse glioma brain tumor diagnoses was evaluated. FastGlioma fine-tuning was restricted to adult- type diffuse gliomas; here FastGlioma predictions are shown on non-adult-type diffuse gliomas (zero-shot). Moreover, few-shot visualization heatmaps are shown for non-glioma brain tumors using diffuse glioma SRH keys. The figure shows examples of brain tumors from several broad categories such as ring-enhancing lesions, pediatric gliomas, embryonal tumors, non-tumor lesions, and extra- axial tumors. CNS lymphomas are infiltrative brain tumors and can be challenging to differentiate from glioblastomas intraoperatively. SRH images show classic perivascular arrangement (angiocentricity) with tumor cells forming layers around the blood vessels. Residual microscopic metastatic tumor is the major cause of tumor recurrence after surgery. FastGlioma identified regions of microscopic residual metastatic tumor. Tumor infiltration from pediatric gliomas, such as diffuse midline gliomas and pilocytic astrocytomas, is detected by FastGlioma. Pediatric diffuse midline gliomas can have a spectrum of histologic morphologies that can differ in appearance compared to glioblastomas. Rosenthal fibers (black lobules), present in pilocytic astrocytomas but rare in diffuse gliomas, are not a source of error for FastGlioma. Acellular and nondiagnostic regions are not segmented as regions of tumor infiltration. Embryonal tumors have distinctive ‘small round blue cell’ cytologic features that FastGlioma correctly identifies as tumor infiltration. FastGlioma provides realtime confirmation of the absence of tumor infiltration during surgery, such as infarcts. Meningiomas are known to invade adjacent dura, which is the major source of tumor recurrence. FastGlioma can identify meningioma infiltration within normal dura sampled at the dural margin. These results provide evidence for the potential of FastGlioma to generalize beyond brain tumors, such as breast, lung, and prostate cancer. Scale bars, 100 microns.

[0250] 1.3.5. FastGlioma as a Surgical Adjunct

[0251] Finally, the feasibility and safety of FastGlioma as a surgical adjunct was evaluated by simulating an interventional clinical trial where surgical resections are guided by FastGlioma predictions. FastGlioma predictions (experimental arm) were compared in a head-to-head, prospective comparison against standard-of-care intraoperative surgical adjuncts (control arm): image-guided surgery with MRI-based neuronavigation and fluorescence-guided surgery with 5-aminolevulinic acid (5-ALA). Both adjuncts have been studied in clinical trials that demonstrated improved extent of resection and decreased postoperative residual tumor volume [28,29], A total of 129 diffuse glioma patients, a subset of the above prospective cohort, were included, resulting in 624 surgical specimens with matched FastGlioma predictions and radiographic features (contrast enhancement / FLAIR) and / or 5-ALA status for each surgical specimen according to previously published and validated protocols [4,30].

[0252] Both study arms were evaluated on the classification task of differentiating surgical specimens with the ground truth label of normal brain (score 0) versus dense tumor infiltration (score 3). Errors on this task are clinical high-risk errors because they represent actionable and decisive predictions: normal brain predictions signals to stop resection, dense tumor signals to continue resection if otherwise safe. FastGlioma outperformed both image- guided and fluorescence-guided methods for detecting tumor infiltration by a wide margin (FIG. 4A). FastGlioma achieved an AUROC of 98.1% compared to 76.3% for FLAIR positivity, 71.8% for contrast enhancement, and 89.0% for 5-ALA fluorescence. A major challenge in diffuse glioma surgery is interpreting FLAIR positivity, which can indicate tumor infiltration or cerebral edema, or both. FastGlioma was able to correctly differentiate tumor infiltration and cerebral edema in FLAIR-positive regions, with an AUROC of 98.7% (FIG. 16)

[0031] . Next, these results were analyzed for each study patient to identify the number of patients that had one or more high-risk tumor miss errors, or false negative predictions, for both study arms. Tumor miss errors place patients at high risk of dense, safely resectable residual tumor left within the resection cavity after surgery. Only 3.8% (5 / 129) of patients in the FastGlioma arm had > 1 high-risk tumor misses compared to 24.0% (31 / 129) in the surgical adjuncts arm (FIG. 4B). Patients who undergo diffuse glioma resections guided using current standard-of-care surgical adjuncts alone are potentially at 6.3X increased relative risk of dense, safely resectable residual tumor after surgery compared to including FastGlioma as a surgical adjunct. FastGlioma powered by real-time, label-free, optical imaging with SRH overcomes the inherent limitations of existing adjuncts, such as insufficient tumor fluorescence, nonspecific radiographic features, and brain shift, enabling more precise and effective surgical care of diffuse gliomas patients (FIG. 14) [30,32,33].

[0253] FIGS. 4A to 4B provide simulated clinical trial results. FIG. 4A: Results of the prospective simulated clinical trial comparing FastGlioma to current standard-of-care surgical adjuncts. Both FastGlioma and SRH foundation models outperform image-guided and fluorescence-guided surgical methods by a wide margin for differentiating normal / gliotic brain (score 0) from dense tumor infiltration (score 3). The performance boost results from FastGlioma predicting directly on high-resolution, label-free SRH images without requiring indirect labels of infiltration, such as 5-ALA fluorescence or contrast enhancement. FIG. 4B: Graphical summary of the patient-level false negative rates in the prospective cohort. To ensure fair and clinical representative comparison, false negative rates were calculated for the optimal surgical adjunct given the patient’s diffuse glioma subtypes: FLAIR for IDH-mutant diffuse gliomas and contrast enhancement and 5-ALA was used for IDH- wildtype tumors. False negative errors on the score 0 versus score 3 classification task represent high-risk tumor misses and place the patient at the highest risk of residual tumor after surgery resection. The FastGlioma study arm had a 3.8% false negative / tumor miss rate compared to 24.0% in the standard-of-care surgical adjuncts arm. The results indicate a potential 6.3X decrease in relative risk of residual tumor within resection cavities by using FastGlioma to guide tumor resections. Detailed classification results can be found in FIG. 14.

[0254] FIGS. 14A to 14C depict a comparison of FastGlioma and surgical adjuncts for diffuse glioma resection. FIG. 14A: Classification performance for distinguishing between normal brain and dense tumor by FastGlioma versus surgical adjuncts on diffuse gliomas, IDH-mutation. FLAIR positivity in IDH-mutant gliomas is considered a radiographic marker of tumor infiltration and is used to define extent of resection

[0033] , In the matched cohort, FastGlioma balanced accuracy is +11.8% compared to FLAIR positivity. FLAIR had relatively lower specificity due to a higher number of false positives. Non-specific causes of FLAIR positivity, such as vasogenic edema, result in decreased accuracy. FIG. 14B: Classification performance of FastGlioma versus surgical adjuncts on glioblastomas, IDH- wildtype. Patients with complete removal of contrast enhancing regions have improved progression-free and overall survival [2,6], In the matched cohort, FastGlioma balanced accuracy is +13.8% compared to contrast enhancement for IDH- wildtype glioblastomas. Tumor infiltration is known to extend beyond regions of contrast enhancement and dense, viable tumor is identified in non-enhancing regions [6,54], The findings of poor sensitivity are consistent with previous radio-pathologic correlation studies [64,65], 5-ALA fluorescence is used in glioblastoma surgery to guide surgical resection

[0028] , Previous studies have reported a wide range of sensitivity and specificity values of 5-ALA fluorescence as an indicator of tumor infiltration depending on tumor types, tumor grade, and recurrence status [30,66,67], In the matched cohort, FastGlioma achieved a +13.5% increase in balanced accuracy compared to 5-ALA. FIG. 14C: Illustrative examples of matched surgical specimens with FastGlioma predictions, neuronavigation coordinates, radiographic features, and 5-ALA status. The lower left shows concordance between FLAIR positivity and FastGlioma prediction with dense tumor within the specimen. Center shows a specimen outside the contrast enhancing rim of a glioblastoma with white matter tumor infiltration identified by FastGlioma. Lower right shows an example of 5-ALA fluorescent positive tissue. The patient had a recurrent glioblastoma with associated treatment effect and reactive astrocytes.

[0255] FIG. 16 provides a comparison of FastGlioma and surgical adjuncts for diffuse glioma resection. Performance of FastGlioma in the prospective comparison study for specimen subgroups defined by the surgical adjuncts are shown. This subgroup analysis aimed to determine how well FastGlioma performs in surgical scenarios that are a diagnostic challenge for detecting tumor infiltration: FLAIR-positive regions, contrast enhancing (CE)- negative regions, FLAIR / CE negative regions. For specimens collected in FLAIR-positive regions (left), FastGlioma achieves an AUG of 98.7%. For specimens collected in CE- negative regions (middle), FastGlioma achieves an AUC of 99.2%. For specimens collected in CE / FLAIR-negative regions (right). FastGlioma achieves an AUC of 98.6%. FastGlioma maintains a high diagnostic performance across all scenarios because SRH is a label-free imaging modality that is not dependent on radiographic features. These results emphasize the advantage FastGlioma in conjunction with standard surgical adjuncts to improve the efficacy of diffuse glioma surgery.

[0256] 1.4. Analysis and Conclusions

[0257] FastGlioma is an open-source medical foundation model for fast, label-free detection of diffuse glioma infiltration during surgery. Within seconds of tissue sampling, FastGlioma accurately predicts the degree of tumor infiltration in fresh, unprocessed surgical specimens from both IDH-wildtype glioblastomas and IDH-mutant diffuse gliomas. FastGlioma outperforms standard-of-care image-guided and fluorescence-guided intraoperative methods for tumor infiltration detection by a wide margin. FastGlioma can be generalized to other pediatric and adult brain tumor diagnoses, demonstrating its potential as a general-purpose medical foundation model for guiding brain tumor surgery.

[0258] FastGlioma has the potential for immediate clinical impact on improving the comprehensive management of diffuse glioma patients. The prognostic significance of extent of resection and residual tumor burden has been reproducibly demonstrated in multiple, independent, international clinical studies [2,3,6,34], FastGlioma represents an innovative departure from existing surgical adjuncts by rapidly identifying tumor infiltration at microscopic resolution using Al, nearly eliminating the potential risk of residual dense tumor within glioma resection cavities. Al-based computer vision and visual foundation models can minimize reliance on radiographic features, contrast enhancement, or extrinsic fluorescent labels to optimize extent of resection and achieve safe maximal tumor removal.

[0259] Worldwide, over 18 million people will be diagnosed with cancer

[0035] , The importance of delivering safe, timely, and affordable cancer surgery has been identified as a major area of improvement in the global fight against cancer [8], Global cancer initiatives have recommended incorporating new technologies, including advanced imaging modalities and Al, into cancer surgery

[0036] , SRH and FastGlioma are accessible and affordable surgical adjuncts for cancer surgery. Future studies will focus on applying a similar FastGlioma workflow to other human cancers, including lung, prostate, head and neck, and breast cancer [37.38],

[0260] As part of the growing field of Al-based diagnostics [39-41], FastGlioma is a rapid and scalable alternative to conventional wet lab methods. Conventional intraoperative pathology using smear preparation and frozen sectioning is limited in diagnostic and clinical value. Histologic diagnosis [20,42], tumor recurrence

[0043] , molecular classification

[0044] , and tumor infiltration detection are now possible through an integrated bedside SRH-AI platform. SRH visual foundation models can enable computational staining for histologic diagnoses, molecular markers, and tumor infiltration within fresh, unprocessed cancer specimens.

[0261] The SRH visual foundation model can facilitate generalization of FastGlioma to other human cancers. Foundation models will play a central role in the safe and effective deployment of Al in healthcare and clinical medicine. Large and diverse medical datasets combined with high-quality self-supervised training results in better performance and less bias across a wide range of complex clinical and diagnostic tasks, as demonstrated by the results of the disclosure. For example, the above example demonstrates how visual foundation models can facilitate tumor detection across patient demographics, medical centers, molecular genetics, and brain tumor diagnoses. In conclusion, FastGlioma represents the transformative potential of medical foundation models to unlock the role of artificial intelligence in the care of cancer patients.

[0262] 1.5. Materials and Methods

[0263] 1.5.1. Study Design

[0264] This study has three main objectives: (1) train a vision transformer model on the largest and most diverse dataset of SRH images using self-supervision to develop the first SRH visual foundation model, (2) fine-tune the visual foundation models to develop FastGlioma for detection and quantification of diffuse glioma infiltration in fresh, unprocessed surgical specimens, (3) validate FastGlioma in a prospective, multicenter, international cohort of diffuse glioma patients and compare results to current surgical adjuncts. The common working definition of a foundation model was adopted: (1) any machine learning model that is (2) trained on a large and diverse dataset using (3) self- supervision at scale and (4) can be adapted to a wide range of downstream tasks

[0013] , The following was also added to this definition: (5) evidence of zero-shot generalization to new, unseen data.

[0265] Foundation modeling have not been previously investigated in studies on the clinical applications of SRH and tumor infiltration was focused on as the most clinically important and ubiquitous problem in cancer surgery as the major downstream tasks. FastGlioma was designed to detect microscopic tumor infiltration for all diffuse glioma molecular subtypes. A major data-centric contribution of this work was developing a multicenter, international, label-free SRH tumor infiltration dataset annotated by expert neuropathologists. The results demonstrated the feasibility of generating this complex biomedical dataset. Moreover, previous studies that combine SRH and Al were done such that the same imaging dataset was used both for human interpretation and Al model training [42,44], Here, the limits of AI- based computer vision performance in lower image resolution / faster image acquisition regimes was pushed at lOx the speed of conventional SRH imaging. Finally, the feasibility of using FastGlioma as a surgical adjunct was demonstrated and compared to tumor detection performance with existing image-guided and fluorescence-guide surgical adjuncts.

[0266] 1.5.2. Stimulated Raman Histology

[0267] All images in the study were acquired using intraoperative fiber-laser-based stimulated Raman scattering microscopy [21,48]. The NIO Imaging System (Invenio Imaging, Inc., Santa Clara, CA) was used for all training and testing data collection. A detailed description of the imager and laser configuration can be found in previous studies [20,48], In short, a pump beam at 790 nm and a Stokes beam with a tunable range from 1015 nm-1050 nm was used to stimulate the surgical specimens. The settings allow for access to the Raman shift spectral range between 2800 cm-1-3130 cm4. Images were acquired as 1000 pixel-width strips with an imaging speed of 0.4 Mpixel(s) per strip. In normal imaging mode, each strip row is acquired independently in a left-right fashion using a custom beam-scanning [20,21], Two image channels are acquired sequentially at 2845 cm-1(CH2 channel) and 2930 cm-1(CH3 channel) Raman wavenumber shifts. A stimulated Raman signal at 2845 cm-1represents the CH2 symmetric stretching mode of lipid-rich structures, such as myelinated axons. A second Raman peak at 2930 cm-1corresponds to protein- and nucleic acid-rich regions such as the cell nucleus and collagen. Because all SRH strips are acquired via standard horizontal line scanning [20,21,48], low-resolution SRH images can be generated by directly downsampling SRH strip rows by a downsampling factor, such as 1 / 2, 1 / 4, 1 / 8, etc. Halving the line sampling factor corresponds to a 2X imaging time savings. In fast imaging mode, single channel images with a user specified downsampling factor are acquired. The whole slide SRH images are then split into 300x300-pixel patches without overlap using a sliding raster window over the full image. All models are trained using 16-bit, raw, greyscale SRH images. For the purposes of the study, SRH images were acquired as two-channel images (2845 cm-1, 2930 cm-1) for pathologist’s review to determine ground truth tumor infiltration labels.

[0268] 1.5.3. SRH Dataset

[0269] Clinical SRH imaging began at the University of Michigan (UM). All patients with a suspected brain tumor are recruited for intraoperative SRH imaging in a prospective fashion. Inclusion criteria were the following: patients who were undergoing surgery for (1) suspected central nervous system tumor and / or (2) epilepsy, (3) subject or durable power of attorney was able to provide consent, and (4) preoperative assessment that additional tumor specimens would be available in addition to what is required for clinical pathologic diagnosis. Exclusion criteria were (1) grossly inadequate tissue. (2) insufficient diagnostic tissue (e.g. hemorrhagic, necrotic, etc.), or (3) imaging malfunction. A similar imaging protocol was implemented at 12 other medical centers with clinical SRH imaging deployed in their operating rooms. A total of 2799 patients, 11,462 whole slide SRH images, and approximately 4 million unique 300x300 pixel SRH patches were included for SRH foundation model training. Dataset statistics and diagnostic information can be found in FIG. 6.

[0270] 1.5.4. SRH Foundation Model Training

[0271] SRH foundation models consist of two modular components trained using selfsupervision: the patch tokenizer and the whole slide encoder.

[0272] Patch tokenizer training with hierarchical discrimination'. In standard vision transformers, converting small, fixed-size image patches, such as 8x8 or 16x16 pixel patches, into tokens can be done by flattening. This tokenization strategy is not feasible due to the size of whole slide SRH images (> 6000 x 6000 pixels). Therefore, a data-driven patch tokenization method was developed that leverages the inherent patient-slide-patch hierarchy of SRH images to define a hierarchical discriminative learning tasks. Hierarchical discrimination, called HiDisc, was demonstrated to outperform instance discrimination methods for biomedical microscopy computer vision tasks. HiDisc uses self-supervised contrastive learning such that positive image patches are defined based on a shared ancestry in the patient-slide-patient data hierarchy. The HiDisc loss is a summation of three losses, each of which corresponds to instance discrimination at a level of the patch-slide-patient hierarchy. The HiDisc loss at the level f was defined to be: where £ G {Patch, Slide, Patient} is the level of discrimination, and I is the set of all images in the minibatch. Ae (0 is the set of all images in I except for the anchor image i, a set of images that are positive pairs of ?' at the £-level,

[0273] P / (z) = {p G A^i) : ancestryf(p) = ancest where ancestry / •) is the f’-level ancestry for the anchor patch. For example, patches ® i andx:i from the same patient would have the same patient ancestry, i.e., ancestryPatieu / a? / = ancestryPatient(a?7)_ The component HiDisc losses calculate the same overall contrastive objective with positive pairs at different levels in the hierarchy. Finally, the complete HiDisc loss is the sum of the patch-, slide-, and patient-level losses defined above:

[0274] ^HiDisc = HiDisc

[0275] / “ {Patch, Slide. Patient} , (4) a weighting hyperparameter for level £ in the total loss. Because HiDisc is a self-supervised representation learning method, the full SRH dataset was used as shown in FIG. 6A. It was found that HiDisc patch tokenization improved classification performance compared to ImageNet transfer learning (FIG. 3C).

[0276] Patch encoding was accomplished using the ResNet-34 architecture as the backbone feature extractor and a one-layer multilayer perceptron (MLP) to project the embedding to 128-dimensional latent space for HiDisc self- supervised training

[0049] . Ablation studies were performed over the batch size, learning rate, and loss hyperparameters to optimize performance on the SRH7 dataset. The encoder was trained using a batch size of 512 and an AdamW optimizer with a learning rate of 0.001 on a cosine decay schedule with warmup for the first 10% of training iterations for a total of 100K iterations on the foundation SRH dataset. To train using the HiDisc loss, the mini-batches were constructed by first selecting 64 patients, followed by sampling two slides per patient, two patches per slide, and finally applying two random augmentations per patch, yielding 512 patches. The patch, slide, and patient losses were weighted equally, and the temperature was set to 0.7. All patch experiments were performed using mixed-precision and data parallelism on 4 NVIDIA A40 GPUs, taking up to 3 days. Additional ablation experiments were performed with open- source foundational patch encoders to assess the quality of HiDisc feature learning compared to other pre trained models

[0019] ,

[0277] Whole slide encoder. A major contribution of this work was developing an efficient and effective method for whole slide self-supervised training with vision transformer architectures. The major advantage of vision transformers for whole slide inference in computational pathology and optical imaging is their ability to handle large and variably sized images. The whole slide self-supervised learning strategy is a Siamese architecture that requires two random transformations of the same whole slide image. The slide-level transformation strategy is as follows: First, the whole slide is split into two mutually exclusive patch sets (splitting). Next, two random spatial crops are selected from the whole slide image (cropping). Finally, 10-80% of patches from a crop are dropped (masking). This strategy is ideally suited for vision transformers because it allows for variable sized inputs and random dropping of patch tokens / spatial regions within a whole slide image. After generating two transformed views, a variance-invariance-covariance (VICReg) self- supervised objective function was then minimized

[0050] . VTCReg is well suited for whole slide encoding because it is computationally efficient, does not require negative examples, and maintains high expressivity by avoiding dimensional collapse

[0051] ,

[0278] The whole slide transformer consists of 2 hidden layers with dimension 512, with 4 attention heads per layer. The output of the transformer is distilled into a <CLS> token, with seven additional register tokens employed to stabilize training

[0052] , Positional information is learned concurrently in a Fourier feature positional embedding generator network

[0053] , The Fourier feature and MLP hidden dimension of this network are 96 and 36, respectively. For self- supervision purposes, a one-layer MLP was trained to project the embedding to 128- dimensional latent space. The VICReg objective was used for whole-slide self-supervised training, with the coefficients being 10, 10, and 1 for the variance, invariance, and covariance losses, respectively. Pretraining was done with an effective batch size of 256 and the AdamW optimizer with a learning rate of 3e-4 for 100 epochs on a single NVIDIA Titan V100 GPU. Checkpoints were saved every 10 epochs, with the optimal one selected using slide-level metrics on the histological brain tumor diagnosis task with a hold-out validation set. Schematic of SRH foundation model training is shown in FIG. 7. Detailed model training configurations, including batch size, learning rate, and other hyperparameters can be found at github. com / MLNeurosurg / fastglioma.

[0279] 1.5.5. SRH Foundation Model Evaluation

[0280] Validation of the foundation model was performed on a multiclass SRH brain tumor diagnostic task. This dataset consists of 3560 whole slide images from 896 patients (852K total patches). Diagnostic classes are normal brain, high-grade glioma (HGG), low-grade glioma (LGG), meningioma, pituitary adenoma, schwannoma, and metastatic tumor. In all previous benchmarking studies, training required supervised filtering of nondiagnostic patches and patch-level average pooling for whole slide inference, which is known to degrade performance [42-44], Here, it was demonstrated that high-quality self-supervised patch and whole slide representation learning with vision transformers bypasses the need for preprocessing, filtering, or patch-level voting / averaging. Nearest neighbor classification (kNN) was used for SRH foundation model evaluation. First, whole slide representations were generated for both the training and testing data. Next, the kNN classifier was used to match each slide in the testing dataset to the k most similar representations in the training dataset as determined by their cosine similarity, hi the experiments, k=10 was set for all models to ensure consistent results. This allows for a class prediction to be determined for each slide in the testing dataset. The mean class accuracy (MCA) and mean average precision (mAP) was then calculated for the 7-class task for slide metrics (FIG. 8). Whole slide representations were visualized using t-distributed stochastic neighbor embedding (tSNE) to qualitatively assess slide representations with respect to tumor classes. Embeddings for kNN and subsequent evaluations were generated on a single NVIDIA Titan V100 GPU.

[0281] FIGS. 8A to 8D illustrate ordinal metric learning for efficient ordinal representation learning. FIG. 8A: A major design challenge for FastGlioma was enforcing the model output to be a semantically meaningful continuous tumor infiltration score while being trained using discrete, ordinal labels. This design choice was warranted because biologic tumor infiltration is continuous given a surgical specimen or microscopy image. Each whole slide SRH image undergoes a feedforward pass through the slide encoder and slide scorer, which outputs a single scalar value. A pairwise distance matrix was then computed between each instance in the minibatch using the scores. Finally, a binary cross entropy (BCE) loss is computed between the pair’s dot product, 'xj, and the label associated with the pair, defined as 1 if label; isgreater than bibelj,ancj Q otherwise. All BCE losses are summed and minimized. Intuitively, this representation learning strategy corresponds to ordering each whole slide according to the amount of accumulated ‘force’ generated by the other examples within the minibatch. For example, in FIG. 8A, whole slides with label 3 will be maximally pushed to the right by all other examples. Whole slides with label 2 will be pushed to the right by two examples (Os and Is) and to left by one example (3s). FIG. 8B: An ablation study was performed for ordinal metric learning using the well- studied and benchmarked face age estimation dataset (n=24085) [60,61]. Ordinal metric learning was shown to outperform other state-of-the-art ordinal regression methods and is especially effective when annotated data is sparse (plotted as mean + / - s.d.). Ordinal metric learning produces a near linear subspace that orders each face image according to their respective ages. FIG. 8C: Hold-out cross validation studies were performed on the training dataset (n=161) that showed improved ordinal regression performance on tumor infiltration scoring when compared to other methods (ROC curves plotted as mean + / - s.d.). FIG. 8D: Tumor infiltration scores are shown for ordinal metric learning versus other baseline strategies.

[0282] FIG. 15 provides PyTorch-like pseudocode of FastGlioma training with ordinal metric learning. An overview of ordinal metric learning for tumor infiltration scoring is illustrated. Ordinal metric learning was motivated by the need to learn discrete labels along a continuous axis in a data-sparse environment, resulting in the pairwise loss function shown. Ordinal metric learning is a general ordinal regression representation learning method that can be applied to any ranking task. Results shown in FIG. 8 show state-of-the-art results on face age estimation.

[0283] 1.5.6. Fine-Tuning With Ordinal Representation Learning

[0284] The SRH foundation models of this study were specifically developed to adapt to downstream diagnostic tasks for clinical decision support. Here, it was aimed to fine-tune the foundation model for the detection and quantification of tumor infiltration using intraoperative SRH imaging. While diffuse glioma infiltration is a continuous random variable, the majority of previous work modeled glioma infiltration as an ordinal variable [4,54], such that expert pathologists score the degree of tumor infiltration on a discrete, ordered scale. The foundation model was fine-tuned using the glioma infiltration dataset from Pekmezci et al [4], The dataset consists of 161 surgical specimens imaged using SRH from 35 patients. The degree of tumor infiltration in each SRH image was scored on a scale from 0-3 by three independent expert neuropathologists: 0 is no tumor present, 1 is mildly cellular tissue either due to reactive gliosis or with scattered atypical cells, without definitive tumor, 2 is tumor present but in mild / sparse density, 3 is moderate to severe density of tumor cells. This dataset is approximately lOOx smaller than the SRH foundation model training dataset and approximately lOx smaller than the calculated sample size for model testing. Due to this extreme data sparsity for fine-tuning, a general, data-efficient, few-shot ordinal representation learning method was developed called ordinal metric learning. Ordinal metric learning aims to minimize the feature distance, or metric, between images with the same ordinal rank. Moreover, it implicitly learns to order images based on their ordinal label by performing a pairwise comparison between all images in a mini-batch. Ordinal metric learning accomplishes this by applying a binary cross entropy objective on the distance between scores for all possible pairs of images in a mini-batch to enforce the image with the higher label is assigned a higher score. The following loss equation accomplishes this: where:

[0285] BCEf .r. y) = y ■ log a(.r) (1 — y) ■ log (1 - rr(x)\ (6) and:

[0286] T is the set of all images in the minibatch. is the set of all images in T except for the anchor image i,

[0287] B(i) = I \ {»},(8)

[0288] A schematic of ordinal metric learning can be found in FIG. 8. Ordinal metric learning was used to train the FastGlioma model and included fine-tuning the slide encoder and a one layer linear slide scorer. Tumor infiltration labels were balanced via whole slide oversampling of the minority classes during training. The model was trained with a batch size of 16 and adjusted learning rate of 1.875e-5 for 100 epochs. The best checkpoint was selected using a hold-out validation set. To evaluate the SRH foundation model, a standard linear evaluation protocol was followed with only the slide scorer being trained. The linear evaluation protocol is similar to other self- supervised visual representation learning methods, such as SimCLR55 or DINO56, where the visual feature extractor is frozen and a final classification / regression layer is trained.

[0289] 1.5.7. Fine-Tuning With Ordinal Representation Learning

[0290] To evaluate the performance of FastGlioma in distinguishing various levels of diffuse glioma infiltration, two key metrics were employed: mean area under the receiver operating curve (mAUROC) and mean absolute error (MAE). The MAE is calculated by passing the FastGlioma whole slide logit through a sigmoid activation function to rescale between 0 to 1. Similarly, the ground truth labels, which range from 0 to 3, are also normalized to the 0 to 1. The MAE was then computed by measuring the average absolute difference between the rescaled logits and the normalized labels. The mAUROC provides a straightforward metric to assess FastGlioma’ s ability to discern between different degrees of tumor infiltration. mAUROC is calculated as the average of the area under the receiver operating curves for three binary classification tasks: 0vl23, 01v23, and 012v3. This metric reflects the ordinal label distribution and emphasizes the clinical diagnostic task.

[0291] 1.5.8. Prospective Testing of FastGlioma

[0292] The prospective FastGlioma clinical testing included a primary and secondary endpoint. The primary endpoint was to validate FastGlioma’ s ability to reproducibly and accurately detect tumor infiltration within SRH images across patient populations, demographics, medical centers, and diffuse glioma subgroups. The secondary endpoint was to compare the performance of FastGlioma with the standard-of-care methods for intraoperative tumor infiltration detection currently in use for brain tumor surgery.

[0293] Primary testing endpoint — SRH-based tumor infiltration detection: the primary study endpoint was to achieve a diagnostic performance for detecting diffuse glioma infiltration in SRH images on par with previous SRH classification tasks, such as intraoperative tissue diagnosis and molecular classification [42,44]. The primary testing was designed using the same principles as a single-arm, non-inferiority diagnostic clinical trial [42,44], In order to obtain a minimum sample size estimate, previous studies that combined SRH and Al were used to classify normal brain versus any tumor tissue. Previously reported accuracy values range from 89.3-95.8% with an average value of 93.2% (+ / - 3.6%) [42-44,54], This average value was used to define the expected performance, the equivalence / non-inferiority limit was set to 5%, the alpha value to 2%, and the power to 90%, resulting in a sample size value of 565 SRH images obtained from surgical margins. This sample size was aimed to be achieved for both IDH-wildtype and IDH-mutant diffuse gliomas to ensure generalizability and reproducibility across diffuse glioma molecular subtypes as defined by the WHO. The calculation resulted in a final minimum sample size of 1130 surgical specimens. Prospective patient recruitment was continued until minimum sample sizes were reached in both IDH- mutant and TDH-wildtype cohorts. All sample size calculations were performed using the epiR package (version 2.0.46) in R (version 3.6.3). Ground truth SRH tumor infiltration labels were provided by on-site pathologists. All pathologists were provided written and video instructions for SRH tumor infiltration scoring using the 4-tiered system by the primary study pathologist.

[0294] Secondary testing endpoint: FastGlioma comparison with image- and fluorescencebased surgical adjuncts for tumor infiltration detection. The secondary study endpoint was to compare the FastGlioma intraoperative workflow (experimental arm) with the two most common surgical adjuncts for identifying tumor infiltration intraoperatively (control arm) in a simulated prospective surgical trial. ‘Simulated’ terminology is used because FastGlioma has not yet been approved by the Food and Drug Administration or European Medicines Agency to guide treatment decisions, such as extent of tumor resections. However, it was aimed to demonstrate the feasibility and safety of using FastGlioma to guide resections via predicting on surgical specimens sampled at the resection margin of diffuse glioma patients. FastGlioma predictions in this setting produce the actionable information needed to guide resection and simulates the clinical setting that FastGlioma would be deployed. FastGlioma is compared in a head-to-head prospective comparison study to (1) image-guided surgery with MRI-based neuronavigation and (2) fluorescence-guided surgery with 5- ALA. Both methods have been shown to improve extent of resection in randomized controlled trials [28,29], In general, neuronavigation and 5-ALA fluorescence can indicate presence of tumor infiltration, but, unlike FastGlioma, do not quantify the degree of infiltration. For the purposes of this study and others

[0030] , tumor detection using neuronavigation or 5-ALA was treated as a binary indicator, for example, yes / no contrast enhancement, yes / no 5-ALA fluorescence. To perform a fair comparison between FastGlioma and the surgical adjuncts, this secondary endpoint was designed to differentiate normal brain tissue (score 0) versus dense tumor (score 3). This task was focused specifically on because errors are clinical high- risk errors and these tumor infiltration scores are actionable and decisive: score 0 means stop resection, score 3 means continue resection if otherwise safe. Moreover, this strategy avoids biasing performance results in favor of FastGlioma, which provides a continuous score that can differentiate degrees of tumor infiltration. It was aimed to show that FastGlioma was non-inferior to both neuronavigation and 5-ALA fluorescence for detecting tumor within surgical specimens collected at the margin of resection cavities during surgical resection. Details of generating the matched SRH / MRI / 5-ALA specimen dataset as a subset of the primary testing endpoint data are described herein.

[0295] 1.5.9. Prospective Testing Dataset

[0296] Three medical centers acted as external FastGlioma testing sites: University of California San Francisco, New York University, and Medical University of Vienna. Each medical center prospectively enrolled patients for testing. Inclusion criteria were: (1) patient age > 18 years old, (2) a suspected diffuse glioma on preoperative radiographic imaging, and (3) planned brain tumor resection. Exclusion criteria included: (1) aborted tumor resection, (2) non-glioma final pathology, and (3) SRH imager malfunction. An aim was to accurately simulate the clinical setting that FastGlioma would be implemented for surgical interventions. Therefore, study neurosurgeons were instructed to sample surgical margins at their discretion to identify microscopic tumor infiltration within the tumor resection cavity. It was aimed to provide as minimal instruction as possible to account for surgeon / user variability during FastGlioma testing. After intraoperative SRH imaging, surgical specimens were removed from the premade microscope slide and preserved in formalin for downstream tissue processing (FIG. 5). Each SRH image was scored postoperatively by an onsite, board- certified neuropathologist with dedicated training and expertise in intraoperative SRH imaging. The primary study pathologist provided verbal and video instructions for tumor infiltration scoring. The previously developed and validated protocol was used for 0-3 tumor infiltration scoring [4]. For the primary testing endpoint that is evaluated at the image-level, SRH tumor infiltration scores provided by the neuropathologists were used as the ground truth. For the secondary testing endpoint that is evaluated at the specimen-level, neuronavigation coordinates, radiographic features (i.e. contrast enhancement, FLAIR positive), and 5-ALA fluorescence status were recorded in real-time by a study technician for each specimen. Secondary endpoint testing was completed at UCSF by a dedicated study technician and the central neuropathologist to standardize all matched data collection. To optimize for the secondary testing endpoint, annotated data from UM, NYU, and MUV were used to fine-tune FastGlioma. After intraoperative SRH imaging, the specimen was extracted from the premade microscope slide and sent for downstream whole slide / specimen analysis using H&E / IHC testing as previously detailed [4], Specimen-level ground truth tumor infiltration scores were determined based on whole slide analysis. This strategy allows for an unbiased comparison between all three surgical adjuncts.

[0297] 1.5.10. FastGlioma Versus Cellularity-Based Tumor Infiltration Scoring

[0298] The cellularity within the SRH whole-slide images was calculated to be the average number of cells per 300x300 pixel SRH patch. The number of cells was determined using an SRH single-cell segmentation model training using full supervision. Specifically, a Mask R- CNN model with a ResNet-50 backbone pre-trained on the Microsoft COCO dataset was fine-tuned on 1000 annotated SRH patches of normal brain and 6 different brain tumor diagnoses

[0057] , The final model predictions were filtered with a non-maximal suppression algorithm to remove overlapping cell bounding boxes with >20% area and predictions with less than 80% confidence. Correlation between cellularity and FastGlioma tumor infiltration score was calculated using Pearson’s correlation coefficient. To evaluate whether cellularity can be used to detect diffuse glioma infiltration, surrogate tumor infiltration score for a whole slide was calculated using the cellularity value. This was then used to calculate the mAUROC across the three different tumor infiltration tasks to compare with FastGlioma infiltration scores as shown in FIG. 11.

[0299] 1.5.11. Few-Shot Visualizations and Model Interpretability

[0300] It was aimed to develop a whole slide visualization method that can accurately and flexibly identify regions of tumor infiltration within SRH images to improve model interpretability. Studies on vision transformers have generally relied on plotting self-attention coefficients to generate data visualization

[0056] , Unfortunately, this strategy does not guarantee uniformly high attention coefficients on foreground / tumor infiltrated regions and is known to produce spurious high attention coefficients in background regions

[0052] , Therefore, a few-shot visualization strategy was developed that leverages a curated support set of expert physician-selected SRH patches, or keys, that include diverse examples of normal brain parenchyma and diffuse glioma subtypes. This strategy leverages the representational power of the self- supervised patch tokenizer to identify similar SRH features within any given whole slide field-of-view. Specifically, for any SRH patch query,x<i, within a whole slide SRH image, the dot product was calculate between the tokenized query patch A? and a support set of tokenized keys, S. It was first determined if the query patch is foreground / diagnostic by determining if the maximal dot product across the support set exceeds a threshold,(P. If not, then the patch is classified as background. If the query dot product exceeds ® for any patch in the support set, it was then assigned a few-shot visualization score,sq. This is defined as the difference between maximum dot product from the tumor exemplars in the support subset, Stumor, and the maximum dot product from the normal exemplars. ^normaJ: where: sirnf.r, y) = . - and sint is the cosine similarity, I HI H l? / l I and 0 was 0.5 for the visualizations. This visualization strategy has the advantage of leveraging both the feature similarity between tumor patches and the dissimilarity between tumor and normal patches. If a patch has a high similarity to any of the tumor exemplars and high dissimilarity with the normal exemplars, thens<? > 0, and vice versa. Empirically, 10 or less patch exemplars per subset can yield high-quality and interpretable visualizations using FastGlioma. Moreover, this strategy demonstrates good zero-shot generalization to non-glioma brain tumor diagnoses without needing to add tumor- specific examples (e.g., meningioma or medulloblastoma exemplars) to the support set, as shown in FIG. 14.

[0301] 1.5.12. Computational Hardware and Software

[0302] SRH images were processed using an Intel Core i76700K Skylake QuadCore 4.0 central processing unit with a custom Python-based (version 3.9) mlins-package. The pydicom package (version 2.3.1) was used to process the SRH images from the NIO Imaging System. All archived postprocessed image patches were saved as 16-bit TIFF images and handled using the tifffile package (version 2022.10.10). All models were trained using the University of Michigan Advanced Research Computing (ARC) Armis2 high-performance computing cluster. Visual patch and whole slide encoders were trained on NVIDIA A40 and Titan V100 graphical processing units (GPUs), respectively. Evaluations were performed on NVIDIA Titan V100 GPUs. All custom code for training and inference can be found in an open-source FastGlioma repository. The models were implemented in PyTorch Lightning (version 1.8.4). The ImageNet pretrained ResNet-34 model from torchvision (0.14.0) was used. Scikit-learn (version 1.4.1) was used to compute performance metrics on model predictions at both training and inference.

[0303] All code was implemented in Python (version 3.9) using PyTorch Lightning (1.8.4) as the primary machine learning framework. The following packages were used for data analysis: pydicom (2.3.1), tifffile (2020.10.10), PyTorch (1.13.0), torchvision (0.14.0), pandas (1.5.3), NumPy (1.23.5), matplotlib (3.6.3), opencv-python (3.7.0), and scikit-learn (1.4.1). For data visualization and scientific plotting, R (3.5.2) packages ggplot2 (3.3.5), dplyr (2.1.1), and tidyverse (1.3.1) were used. Code and scripts to reproduce the experiments of this disclosure are available on GitHub (https: / / github. com / MLNeurosurg / fastglioma).

[0304] 1.6. References

[0305] [1] Karschnia, P. et al. Prognostic validation of a new classification system for extent of resection in glioblastoma: A report of the RANG resect group. Neuro. Oncol. 25, 940-954 (2023).

[0306] [2] Sanai, N., Polley, M.-Y., McDermott, M. W., Parsa, A. T. & Berger, M. S. An extent of resection threshold for newly diagnosed glioblastomas. J. Neurosurg. 115, 3-8 (2011).

[0307] [3] Smith, J. S. et al. Role of extent of resection in the long-term outcome of low-grade hemispheric gliomas. J. Clin. Oncol. 26. 1338-1345 (2008).

[0308] [4] Pekmezci, M. et al. Detection of glioma infiltration at the tumor margin using quantitative stimulated Raman scattering histology. Sci. Rep. 11, 12162 (2021).

[0309] [5] Petrecca, K., Guiot, M.-C., Panet-Raymond, V. & Souhami, L. Failure pattern following complete resection plus radiotherapy and temozolomide is at the resection margin in patients with glioblastoma. J. Neurooncol. I l l, 19-23 (2013). [6] Hervey-Jumper, S. L. et al. Interactive Effects of Molecular, Therapeutic, and Patient Factors on Outcome of Diffuse Low-Grade Glioma. J. Clin. Oncol. 41, 2029-2042 (2023).

[0310] [7] Gal, A. A. & Cagle, P. T. The 100-year anniversary of the description of the frozen section procedure. JAMA 294, 3135-3137 (2005).

[0311] [8] Sullivan, R. et al. Global cancer surgery: delivering safe, affordable, and timely cancer surgery. Lancet Oncol. 16, 1193-1224 (2015).

[0312] [9] Orosco, R. K. et al. Positive Surgical Margins in the 10 Most Common Solid Cancers. Sci. Rep. 8, 5686 (2018).

[0313]

[0010] Brown, P. D. et al. A prospective study of quality of life in adults with newly diagnosed high-grade gliomas: the impact of the extent of resection on quality of life and survival. Neurosurgery 57, 495-504; discussion 495-504 (2005).

[0314]

[0011] The big question. ARPA-H https: / / arpa-h.gov / research-and-funding / programs / psi.

[0315]

[0012] Robboy, S. J. et al. Pathologist workforce in the United States: I. Development of a predictive model to examine factors influencing supply. Arch. Pathol. Lab. Med. 137, 1723- 1732 (2013).

[0316]

[0013] Bommasani, R. et al. On the Opportunities and Risks of Foundation Models. arXiv [cs.LG] (2021).

[0317]

[0014] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C. & Chen, M. Hierarchical textconditional image generation with CLIP latents. ArXiv abs / 2204.06125, (2022).

[0318]

[0015] OpenAI et al. GPT-4 Technical Report. arXiv [cs.CL] (2023).

[0319]

[0016] Brown, T. B. et al. Language Models are Few-Shot Learners. Adv. Neural Inf. Process. Syst. abs / 2005.14165, (2020).

[0320]

[0017] Moor, M. et al. Foundation models for generalist medical artificial intelligence. Nature 616, 259-265 (2023).

[0321]

[0018] Azizi, S. et al. Robust and data-efficient generalization of self-supervised machine learning for diagnostic imaging. Nat Biomed Eng 7, 756-779 (2023).

[0322]

[0019] Chen, R. J. et al. Towards a general-purpose foundation model for computational pathology. Nat. Med. 30, 850-862 (2024).

[0323]

[0020] Orringer, D. A. et al. Rapid intraoperative histology of unprocessed surgical specimens via fibre-laser-based stimulated Raman scattering microscopy. Nat Biomed Eng 1, (2017).

[0021] Freudiger, C. W. et al. Label-free biomedical imaging with high sensitivity by stimulated Raman scattering microscopy. Science 322, 1857-1861 (2008).

[0324]

[0022] Dosovitskiy, A. et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv [cs.CV] (2020).

[0325]

[0023] Jiang, C. et al. Hierarchical discriminative learning improves visual representations of biomedical microscopy. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 2023, 19798-19808 (2023).

[0326]

[0024] Chen, T., Kornblith, S., Norouzi, M. & Hinton, G. A Simple Framework for Contrastive Learning of Visual Representations, in Proceedings of the 37th International Conference on Machine Learning (eds. lii, H. D. & Singh, A.) vol. 119 1597-1607 (PMLR, 13-18 Jul 2020).

[0327]

[0025] Jiang, C. et al. OpenSRH: optimizing brain tumor surgery using intraoperative stimulated Raman histology. Adv. Neural Inf. Process. Syst. 35, 28502-28516 (2022).

[0328]

[0026] Ouyang, L. et al. Training language models to follow instructions with human feedback. Adv. Neural Inf. Process. Syst. abs / 2203.02155, (2022).

[0329]

[0027] Wiens, J. et al. Do no harm: a roadmap for responsible machine learning for health care. Nat. Med. 25, 1337-1340 (2019).

[0330]

[0028] Stummer, W. et al. Fluorescence-guided surgery with 5 -aminolevulinic acid for resection of malignant glioma: a randomised controlled multicentre phase III trial. Lancet Oncol. 7, 392-401 (2006).

[0331]

[0029] Senft, C. et al. Intraoperative MRI guidance and extent of resection in glioma surgery: a randomised, controlled trial. Lancet Oncol. 12, 997-1003 (2011).

[0332]

[0030] Lau, D. et al. A prospective Phase II clinical trial of 5 -aminolevulinic acid to assess the correlation of intraoperative fluorescence intensity and degree of histologic cellularity during resection of high-grade gliomas. J. Neurosurg. 124, 1300-1309 (2016).

[0333]

[0031] Broggi, G. et al. Histologic Definition of Enhancing Core and FLAIR Hyperintensity Region of Glioblastoma, IDH-Wild Type: A Clinico-Pathologic Study on a Single-Institution Series. Brain Sci 13, (2023).

[0334]

[0032] Gerard, I. J. et al. Brain shift in neuronavigation of brain tumors: A review. Med. Image Anal. 35, 403-420 (2017).

[0033] Wen, P. Y. et al. RANO 2.0: Update to the Response Assessment in Neuro-Oncology Criteria for High- and Low-Grade Gliomas in Adults. J. Clin. Oncol. 41, 5187-5199 (2023).

[0335]

[0034] Jakola, A. S. et al. Comparison of a strategy favoring early surgical resection vs a strategy favoring watchful waiting in low-grade gliomas. JAMA 308, 1881-1888 (2012).

[0336]

[0035] Sung, H. et al. Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J. Clin. 71. 209-249 (2021).

[0337]

[0036] Are, C. et al. Global Cancer Surgery: pragmatic solutions to improve cancer surgery outcomes worldwide. Lancet Oncol. 24, e472-e518 (2023).

[0338]

[0037] Mannas, M. P. et al. Stimulated Raman histology, a novel method to allow for rapid pathologic examination of unprocessed, fresh prostate biopsies. Prostate 83, 1060-1067 (2023).

[0339]

[0038] Hoesli, R. C., Orringer, D. A., McHugh, J. B. & Spector, M. E. Coherent Raman Scattering Microscopy for Evaluation of Head and Neck Carcinoma. Otolaryngol. Head Neck Surg. 157, 448-453 (2017).

[0340]

[0039] Tiu. E. et al. Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning. Nat Biomed Eng 6, 1399-1406 (2022).

[0341]

[0040] Zhang, K. et al. Clinically Applicable Al System for Accurate Diagnosis, Quantitative Measurements, and Prognosis of COVID-19 Pneumonia Using Computed Tomography. Cell 182, 1360 (2020).

[0342]

[0041] He, B. et al. Blinded, randomized trial of sonographer versus Al cardiac function assessment. Nature 616. 520-524 (2023).

[0343]

[0042] Hollon, T. C. et al. Near real-time intraoperative brain tumor diagnosis using stimulated Raman histology and deep neural networks. Nat. Med. 26, 52-58 (2020).

[0344]

[0043] Hollon, T. C. et al. Rapid, label-free detection of diffuse glioma recurrence using intraoperative stimulated Raman histology and deep neural networks. Neuro. Oncol. (2020) doi: 10.1093 / neuonc / noaa 162.

[0345]

[0044] Hollon, T. et al. Artificial-intelligence-based molecular classification of diffuse gliomas using rapid, label-free optical imaging. Nat. Med. 29, 828-832 (2023).

[0346]

[0045] Jiang, L. Y. et al. Health system-scale language models are all-purpose prediction engines. Nature 619, 357-362 (2023).

[0046] Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172-180 (2023).

[0347]

[0047] Mittermaier, M„ Raza, M. M. & Kvedar, J. C. Bias in Al-based models for medical applications: challenges and mitigation strategies. NPJ Digit Med 6, 113 (2023).

[0348]

[0048] Freudiger, C. W. et al. Stimulated Raman Scattering Microscopy with a Robust Fibre Laser Source. Nat. Photonics 8, 153-159 (2014).

[0349]

[0049] He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 770-778 (2015).

[0350]

[0050] Bardes, A., Ponce, J. & LeCun, Y. Variance-invariance-covariance regularization for self-supervised learning. ICLR, Vicreg 1, (2022).

[0351]

[0051] Jing, L„ Vincent, P., LeCun, Y. & Tian, Y. Understanding Dimensional Collapse in Contrastive Self-supervised Learning. arXiv [cs.CV] (2021).

[0352]

[0052] Darcet, T., Oquab, M., Mairal, J. & Bojanowski, P. Vision Transformers Need Registers. arXiv [cs.CV] (2023).

[0353]

[0053] Li, Y., Si, S., Li, G., Hsieh, C.-J. & Bengio, S. Learnable Fourier features for multidimensional spatial positional encoding. Adv. Neural Inf. Process. Syst. abs / 2106.02795, (2021).

[0354]

[0054] Ji, M. et al. Detection of human brain tumor infiltration with quantitative stimulated Raman scattering microscopy. Sci. Transl. Med. 7, 309ral63 (2015).

[0355]

[0055] Chen, R. J. et al. Scaling vision Transformers to gigapixel images via hierarchical self-supervised learning. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 16123-16134 (2022).

[0356]

[0056] Caron, M. et al. Emerging properties in self- supervised vision transformers. ICCV 9630-9640 (2021).

[0357]

[0057] He. K... Gkioxari, G., Dollar, P. & Girshick, R. Mask R-CNN. IEEE Trans. Pattern Anal. Mach. Intell. 42, 386-397 (2020).

[0358]

[0058] Zhai, X., Kolesnikov, A., Houlsby, N. & Beyer, L. Scaling Vision Transformers. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 1204-1213 (2021).

[0359]

[0059] Goyal, P. et al. Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision. arXiv [cs.CV] (2022).

[0060] Cao, W., Mirjalili, V. & Raschka, S. Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognit. Lett. 140, 325-331 (2020).

[0360]

[0061] Paplham, J. & Franc, V. A call to reflect on evaluation practices for age estimation: Comparative analysis of the state-of-the-art and a unified benchmark, of the IEEE / CVF Conference on ... (2023).

[0361]

[0062] Hollon, T. C. et al. Rapid Intraoperative Diagnosis of Pediatric Brain Tumors Using Stimulated Raman Histology. Cancer Res. 78, 278-289 (2018).

[0362]

[0063] Oquab, M. et al. DINOv2: Learning Robust Visual Features without Supervision. arXiv (cs.CV) (2023).

[0363]

[0064] Eidel, O. et al. Tumor Infiltration in Enhancing and Non-Enhancing Parts of Glioblastoma: A Correlation with Histopathology. PLoS One 12, e0169292 (2017).

[0364]

[0065] Kubben, P. L. et al. Correlation between contrast enhancement on intraoperative magnetic resonance imaging and histopathology in glioblastoma. Surg. Neurol. Int. 3, 158 (2012).

[0365]

[0066] Roberts, D. W. et al. Coregistered fluorescence-enhanced tumor resection of malignant glioma: relationships between 6-aminolevulinic acid-induced protoporphyrin IX fluorescence, magnetic resonance imaging enhancement, and neuropathological parameters. Clinical article. J. Neurosurg. 114, 595-603 (2011).

[0366]

[0067] Zhao, S. et al. Intraoperative Fluorescence-Guided Resection of High-Grade Malignant Gliomas Using 5-Aminolevulinic Acid-Induced Porphyrins: A Systematic Review and Meta- Analysis of Prospective Studies. PLoS One 8, e63682 (2013).

[0367] Example 2: A self-supervised framework for learning whole slide representations

[0368] 2.1. Overview

[0369] Whole slide imaging is fundamental to biomedical microscopy and computational pathology. Previously, learning representations for gigapixel-sized whole slide images (WSIs) has relied on multiple instance learning with weak labels, which do not annotate the diverse morphologic features and spatial heterogeneity of WSIs. A high-quality selfsupervised learning method for WSIs would provide transferable visual representations for downstream computational pathology tasks, without the need for dense annotations. Slide Pre-trained Transformers (SPT) is presented for gigapixel-scale self-supervision of WSIs. Treating WSI patches as tokens, SPT combines data transformation strategies from language and vision modeling into a general and unified framework to generate views of WSIs for self-supervised pretraining. SPT leverages the inherent regional heterogeneity, histologic feature variability, and information redundancy within WSIs to learn high-quality whole slide representations. SPT visual representations were benchmark on five diagnostic tasks across three biomedical microscopy datasets. SPT significantly outperforms baselines for histopathologic diagnosis, cancer subtyping, and genetic mutation prediction. Finally, it was demonstrated that SPT consistently improves whole slide representations when using off-the- shelf, in-domain, and foundational patch encoders for whole slide multiple instance learning.

[0370] FIG. 21 illustrates self- supervised whole slide learning. Previous work in computational pathology relies on multiple instance learning with weak supervision from slide or patient-level labels to learn whole slide representations [1,2, 3, 4, 5, 6, 7]. A selfsupervised framework is presented for learning whole slide representations, called Slide Pretrained Transformers (SPT), by combining data transformations from vision and language modeling to generate high-quality paired views.

[0371] 2.2. Introduction

[0372] Whole slide imaging is an integral part of tissue diagnosis and laboratory medicine. Computational pathology can provide rapid tissue analysis of complex WSIs, such as cancer detection, subtyping, and grading. Modern computational methods have advanced beyond morphology-based diagnostics to “omics" predictions, prognostication, and treatment response predictions from WSIs alone [8-10], Supervised multiple instance learning (MIL) methods, including AB MIL [2], CLAM [3], DSMIL

[0011] , TransMIL [4], can achieve good performance on the above diagnostic tasks. Unfortunately, these methods rely on slide annotations to learn whole slide representations. WSI annotations are weak, sparse, incomplete, and expensive to obtain

[0012] , Moreover, WSIs are gigapixel-size and contain diverse morphologic and histopathologic features with extensive spatial heterogeneity. Weak slide labels may annotate only a small region within WSIs, demonstrating the limitation of relying on weak supervision to achieve high-quality and transferable whole slide representations. To compensate for weak whole slide labels, self-supervised learning (SSL) has been increasingly used in computational pathology. The majority of previous SSL work in computational pathology has focused on region or patch learning. HiDisc, PLIP, and UNI are examples of WSI patch encoders that can be used to obtain visual features for downstream whole slide learning [13,7,14], However, few previous studies have investigated learning whole slide representations using SSL [15,5], A general and unified framework for whole slide SSL would enable transferable whole slide feature learning and generalize to a wide range of downstream pathology tasks with minimal to no annotations required.

[0373] In this example, Slide Pre-trained Transformers (SPT) for self- supervised whole slide representation learning is presented. SPT treats gigapixel WSIs as a sequence of patch tokens and applies a domain-informed set of vision-language transformations, including splitting, cropping, and masking, to generate distinct views for self-supervised training. Over a range of patch encoders, SPT learns high-quality patch feature aggregation and whole slide representations compared to the state-of-the-art baselines. The main contributions are:

[0374] • The introduction of SPT, a general, flexible, and unified learning framework for training on WSIs at scale evaluated benchmarked on five computational pathology tasks.

[0375] • SPT outperforms previous state-of-the-art self-supervised and supervised methods for WSI representation learning.

[0376] • SPT offers a consistent performance boost across a wide range of patch encoders.

[0377] 2.3. Summary of Methods

[0378] 2.3.1. The SPT framework

[0379] WSIs are partitioned into smaller non-overlapping fields-of-view or patches.

[0380] Let I={Ii}ni=i be a WSI, where n is the number of patches in a WSI, and each A G y3xHxW, with coordinate piEZ2. p ={pi}fli=1are the coordinates of all patches in the WSI. SPT is a two- stage learning framework to leam WSI representations: 1) a pre-trained patch encoder 6"; and 2) a transformer whole slide encoder / . The pre-trained patch encoder can have the architecture of any visual feature extractor. The overall model architecture of SPT is illustrated in FIG. 22.

[0381] In the SPT framework, the patch encoder ^encodes each A into a patch token, Xi=&x(Ii). All patch tokens were represented in a WSI as x={xz}",= / =<? (A). The whole- slide encoder / serves as an aggregation function that learns a whole-slide representation using patch tokens and their corresponding coordinates: / r= / x(x,p)=f (£(T),p).f can be any learned aggregation architecture, such as in AB MIL [2] or a transformer [4,42], Due to the scale of WSIs, it is infeasible to both train the patch encoder 6“ and whole-slide encoder / jointly in an end-to-end manner. Thus, the patch encoder was frozen to allow for large mini-batch training of the whole-slide encoder / The aim of the SPT framework is to learn high-quality whole slide representations with / using self- supervision.

[0382] Self-supervised methods share a common strategy: apply random transformations to a single data example to generate distinct views, called positive pairs. The SPT framework was designed to be compatible with existing SSL objectives, where a WSI undergoes transformations into different views. With a two-view SSL paradigm, r(Z) = are randomly drawn from a set of transformations described below. The patches and their corresponding coordinates from these transformed views are processed through $ and / to obtain the whole slide representations:

[0383] H = / (6W), t(p)). / T = / (W)). t'(p))-

[0384] An SSL loss is used to self-supervise whole slide training:

[0385] TSPT ( / I, hr).

[0386] In the experiments, representative methods were selected from different SSL families

[0043] , including SimCLR

[0016] from contrastive learning, BYOL

[0017] from selfdistillation, and VICReg

[0018] from canonical component analysis. The self- supervised SPT was denoted as ssSPT.

[0387] SPT with supervision: SPT can be adapted to fully supervised training using weak slide- or patient-level labels, by applying a supervised contrastive loss. Positive pairs for supervised contrastive learning are defined by class labels

[0019] , The supervised variant of SPT was denote as suSPT.

[0388] FIGS. 22A to 22B provide an overview of Slide Pre-trained Transformers (SPT).

[0389] FIG. 22A: The SPT framework consists of a two-stage model architecture: 1) a pre-trained patch encoder ^and 2) a transformer whole slide encoder / . WSIs are first divided into small patches, and the patch encoder extracts patch-level features. Whole slide transformations were then applied to the patch tokens to create two views of the same WSI. The transformations combine splitting, cropping, and masking, which are informed by the structure and unique properties of WSIs. The transformed views are encoded by the transformer whole slide encoder, and the slide-level feature learning can use any paradigm. FIG. 22B: Example learning paradigms. In the experiments, three representative self-supervised paradigms were focused on, including SimCLR

[0016] , BYOL

[0017] , and VICReg

[0018] , and supervised contrastive learning

[0019] ,

[0390] 2.3.2. SPT transformation

[0391] While previous work focused on pixel space transformations

[0015] , it was hypothesized that these augmentations at patch level are insufficient to generate high-quality views for whole slide SSL. A patch encoder trained with instance discrimination, for example, should be invariant to pixel- space augmentations. Thus, pixel space augmentations, such as color jittering, should have minimal effect on the representation space and resulting similar embeddings, as illustrated with SimCLR in FIG. 23, because it is explicitly enforced as the pretext task. It was opted to bypass patch augmentations altogether, thereby reducing SPT memory and compute burden, and focusing on transformations at WSI level.

[0392] Transformation strategy: WSIs are divided into a sequence of patch tokens. The SPT transformation strategy, as shown in FIG. 24, is inspired by both vision and language modeling, and was selected to address the domain-specific properties of WSI:

[0393] • Splitting to decrease mutual information between views. Reducing mutual information between SSL views, while keeping task-relevant information intact, improves downstream performance [44,45], The splitting transformation randomly partitions patch tokens into disjoint sets, ensuring mutually exclusive set membership for each token. Enforcing mutual exclusion reduces mutual information between SSL views.

[0394] • Cropping to capture regional heterogeneity. WSIs contain regional differences in histologic features. Similar to multi-cropping in visual SSL, cropping generates spatially diverse views of WSI with variable sizes and features. • Masking to reduce redundant visual features. Due to their gigapixel size,

[0395] WSIs can contain large regions of similar histologic features and tissue phenotypes, causing redundancy. Masking is an effective transformation for vision and language modeling [5,34,48], For each view, the masking transformation samples a subset of patch tokens without replacement. Additionally, masking increases training efficiency by reducing sequence lengths.

[0396] While splitting, cropping and masking can be used individually, here they were applied jointly to generate versatile and high-quality views for SPT training.

[0397] FIG. 23 demonstrates the limited effect of pixel-level patch augmentations. The effect of pixel-level augmentation was qualitatively evaluated on the patch representations by visualizing the tSNE plot of SimCLR pre-trained patch representations sampled from a single WSI. It was observed that strong augmentations at the pixel level have a minimal effect on the patch embeddings. The invariant behavior of the patch encoder is explicitly enforced by the SimCLR pretext task.

[0398] FIG. 24 illustrates the SPT transformation strategy. SPT combines splitting, cropping, and masking to generate views, and they are motivated by the size, region diversity, and information redundancy of WSIs. Splitting partitions patches into mutually exclusive sets decreases mutual information between views; cropping can generate spatially diverse views covering different regions on the WSI; masking reduces redundant visual features and improves training efficiency. The combination of these transformations can create optimal positive pairs for whole slide representation learning.

[0399] 2.3.3. SPT Implementation

[0400] End-to-end whole slide learning with large batch sizes is infeasible due to the gigapixel scale. To enable efficient training, all unaugmented patch tokens were computed on a frozen patch encoder, and SPT transformations were then applied to patch tokens. Each whole slide was represented as an embedding x G RraxJand a coordinate p G Z"x2. Splitting and masking were implemented as row-wise partitioning and dropout on x, p, respectively. Cropping was implemented as coordinate filtering on p. An illustration of the implementation with pseudocode can be found in FIG. 32. FIG. 32 provides an illustration of the implementation of SPT transformations, in accordance with embodiments of the invention, via PyTorch-like pseudocode.

[0401] 2.4. Experiments

[0402] 2.4.1. Benchmarks

[0403] SRH CNS benchmark'. Stimulated Raman Histology (SRH) is a novel optical microscopy method that enables fast imaging of unprocessed tissues [49,50], The benchmark includes six central nervous system (CNS) tumors and normal brains, with 2035 and 925 WSIs for training and evaluation, respectively. These data were collected at University of Michigan, following the imaging protocol in

[0051] , and were labeled by board-certified pathologists.

[0404] H&E glioma molecular classification benchmark'. SPT was also evaluated using publicly available diffuse glioma H&E stained WSIs from the Cancer Genome Atlas (TCGA) and Digital Brain Tumour Atlas (DBTA)

[0052] . The classification of three molecular subgroups was focused on, as defined by the World Health Organization

[0053] , Molecular classification is a challenging computer vision task because the diagnoses are made via molecular testing (such as genetic sequencing), and are not possible for expert pathologists using H&E images alone. The glioma dataset is comprised of 2309 training slides and 341 slides from the TCGA dataset set for evaluation.

[0405] TCGA BRCA, TCGA NSCLC, and TCGA RCC benchmarks'. SPT was further evaluated using three widely used TCGA H&E classification benchmarks: 1) invasive breast carcinoma (BRCA) subtyping, 2) non-small cell lung carcinoma (NSCLC) subtyping, and 3) renal cell carcinoma (RCC) subtyping. For these benchmarks, the well-established study design of [5] was followed.

[0406] 2.4.2. Implementation details

[0407] In-domain ResNet-34

[0054] patch encoders were trained to extract features for all patches in each slide. These patch encoders were trained with SimCLR

[0016] and HiDisc

[0013] , Additional off-the-shelf and foundational patch encoders are described in FIG. 25 A. SPT slide encoders are six-layer transformers

[0042] with four heads, and learnable Fourier positional embeddings

[0055] (visualized in FIG. 27). A two-layer projection head was used for SimCLR and VTCReg, and one-layer projection and prediction heads were used for BYOL. AdamW optimizers and cosine decay schedulers were used after warm-up in the initial 10% of the iterations. The learning rate was adjusted between 103and 107to accommodate the training dynamics of different SSL methods and tasks. With the SPT slide transformation strategy, cropping and masking sizes were adjusted for each experiment and up to 64 patches were utilized per slide for each view. ssSPT and suSPT experiments were trained up to 800 and 100 epochs, respectively, with an effective batch size of up to 1024 WSIs. All models were trained with mixed-precision on a NVIDIA A40 GPU, taking up to 8 hours.

[0408] 2.4.3. Evaluation Protocol

[0409] SPT was benchmarked using standard linear evaluation protocols. Since linear classifiers are sensitive to hyperparameters

[0047] , k nearest neighbor (kNN) was also employed for direct evaluation. H&E glioma molecular evaluation used only WSI from TCGA. For SRH and H&E Glioma, experiments were repeated with three random seeds, and for TCGA BRCA, NSCLC, and RCC, 10-fold cross-validation was used for error bars, following the protocols in previous work [5,15], For the baselines, original embeddings or logits were used when available. Mean class accuracy (MCA), Fl scores, and area under the receiver operating characteristic (AUC) were used to evaluate all benchmarks.

[0410] 2.5. Results ssSPT performance with self- supervised WSI learning strategies was first benchmarked in section 2.5.1, below. ssSPT and suSPT with supervised WSI MIL strategies were compared in section 2.5.2. The ability of SPT to generalize across a wide range of patch encoders in was then evaluated section 2.5.3. Next, it was shown that SPT improves MIL results with SOTA foundation model patch encoders in section 2.5.4. Finally, SPT selfattention heatmaps were visualized in section 2.5.5. Additional results with error bars, ablation studies on the SPT transformations and parameters, and additional visualizations are in section 2.8. 2.5.1. ssSPT learns high-quality whole slide representations ssSPT was benchmarked with self-supervised baselines using in-domain patch encoders. WSI features were evaluated directly using a kNN classifier. As shown in Table 1 below, ssSPT surpasses all baselines across all self- supervised objectives in all metrics, except for AUC on the highly imbalanced BRCA benchmark. ssSPT outperforms all existing self-supervised methods by a large margin, outperforming the previous best self-supervised method with a 10 and 5 points increase in MCA on the SRH CNS and H&E glioma benchmarks, respectively.

[0411] Table 1: Self-supervised benchmarks. A kNN classifier was used to evaluate ssSPT and baselines. The best performing SSL objective for ssSPT was reported, with additional SSL objectives in Table 6. Mean values are reported.

[0412] 2.5.2. SPT outperforms previous fully supervised methods ssSPT and suSPT were benchmarked with linear evaluation using in-domain patch encoders. As shown in Table 2 below, SPT outperforms previous fully supervised methods across all tasks. Remarkably, ssSPT outperforms or matches previous fully supervised methods on MCA in SRH CNS (+1.0), H&E glioma (match), TCGA BRCA (+2.2), and TCGA RCC (+1.4) benchmarks. suSPT outperforms existing MIL methods across all five benchmarks on nearly all metrics. In comparison with the best-performing baselines on MCA of these five benchmarks, suSPT achieves a performance increase of 1.2, 0.7, 3.4, 2.1, and 0.3 points, respectively. Thus, SPT provides a performance increase for both self-supervised and supervised learning, generalizing to different tissue types and diagnostic tasks.

[0413] Table 2: SPT benchmarks. Linear evaluation results were reported on five histology benchmarks. The best performing SSL objective was reported for ssSPT, with additional SSL objectives in Table 6. Mean values are reported.

[0414] 2.5.3. SPT offers performance boost across a range of patch encoders

[0415] The generalizability of SPT to different patch encoders was explored using the more challenging H&E Glioma benchmark. As depicted in FIG. 25 A, ImageNet, HIPT [5], PLIP

[0014] , and UNI [7], in addition to in-domain patch encoders trained with SimCLR and HiDisc were tested. The ImageNet patch encoder is off-the-shelf and out-of-distribution (OOD). HIPT [5] is near-domain since it was trained with H&E WSIs from TCGA, including multiple organ systems and institutions. PLIP

[0014] and UNI [7] are visual foundational models for histology, trained with OpenPath and large-scale institutional datasets, respectively. SPT performance is shown in FIG. 25B, with additional results for SRH CNS benchmark in FIG. 28, and extended metrics with error bars are in Table 4.

[0416] For all patch encoders, SPT training significantly improves whole slide representations over pooling baselines. This improvement is the most significant for ImageNet and HIPT patch encoders. As expected, the performance boost is smaller for indomain patch encoders, especially HiDisc, where slide discrimination was learned during patch training. SPT also achieves a large performance boost using PLIP patch encoders, as it bridges the domain gap between the benchmark and the pre-training dataset OpenPath which is collected on Twitter. As for the state-of-the-art UNI patch encoder, a large performance boost was still observed achieving MCA near 90 points for suSPT. Overall, these results demonstrate that SPT can enhance whole-slide representation learning using encoders from different domains, thereby reducing the reliance on a specialized in-domain patch encoder. SPT training time ranges from 6-8 GPU hours, while the patch encoder is computationally intensive and usually took over 80 GPU hours to train.

[0417] FIGS. 25A to 25B provide SPT benchmarks with different patch encoders. ssSPT and suSPT offer performance boosts with a wide range of patch encoders. ssSPT approaches supervised performance upperbound. Additional metrics with error bars are in Table 4. FIG. 25A: Patch encoders. ImageNet is an off-the-shelf model trained on OOD data. HIPT is neardomain for H&E glioma. SimCLR and HiDisc patch encoders are trained with in-domain data. PLIP and UNI are SOTA foundational patch encoders. FIG. 25B: SPT results on H&E Glioma.

[0418] 2.5.4. SPT improves state-of-the-art MIL results

[0419] Most recently, foundation models have been developed for computational pathology, promising a general approach for WSI diagnostic tasks. UNI is a foundational patch encoder that achieves state-of-the-art results [7], The UNI authors found that “ABMIU with UNI features outperforms many sophisticated MIL architectures” [7], suSPT was benchmarked using UNI patch features, in comparison to ABMIL, in Table 3 below. suSPT outperforms AB MIL in all benchmarks with UNI features, improving upon the previous state-of-the-art performance on these tasks. This showcases that SPT is complementary to the innovative medical foundation models for learning patch representations.

[0420] Table 3: suSPT improves state-of-the-art MIL results with UNI patch features. Mean values are reported here, and standard deviations are in Table 5.

[0421] 2.5.5. Self-attention visualizations reveal tissue phenotypes in full gigapixel WSIs

[0422] Finally, the strategy in

[0047] was adopted and ssSPT was evaluated qualitatively by visualizing self-attention on the full WSI. As illustrated in FIG. 26, self-attention maps on H&E WSI can distinguish different tissue phenotypes such as blood, dense tumor, and necrosis. The CLS token attends to dense tumor regions instead of non-diagnostic regions such as blood and necrosis for slide representations. This observation is consistent with existing literature on self-supervised vision transformers, where attention maps can serve as unsupervised segmentation [47,5], It is believed, this work is the first to generate selfsupervised transformer-based attention maps on full gigapixel WSIs, instead of localized regions, showing the strong representation capacity, flexibility, and efficiency of ssSPT. More H&E and SRH attention map visualizations are in section 2.8.4.

[0423] FIG. 26 shows attention heatmap visualization. Self-attention visualization demonstrates ssSPT’ s ability to distinguish between different tissue phenotypes such as blood, dense tumor, and necrosis. It is believed that these are the first self-supervised transformer-based attention maps on full gigapixel WSIs, instead of localized regions.

[0424] 2.6. Analysis and Conclusions

[0425] Slide Pre-trained Transformers (SPT), a self-supervised approach for whole slide representation learning, in a general, versatile, and lightweight framework was present. SPT uses domain-informed, vision-language transformations for high-quality WSI view generation and self- supervised learning. It was demonstrated that SPT achieves superior performance on five computational pathology benchmarks across three clinical tasks, including histopathologic diagnosis, cancer subtyping, and molecular genetic prediction. It is understood that this is the first systematic study of whole slide representation learning evaluating the role of SSL objectives, transformation strategies, and patch encoders.

[0426] This study advances the field of whole slide representation learning and computational pathology. The work enables an automated diagnostic support system for biomedical microscopy and histopathology, paving the way for more accurate diagnoses and personalized treatment recommendations. SPT can serve as a potential foundation model training strategy within computational pathology. Further, while experiments are in medical applications, there is a potential for impact on other similar gigapixel image modalities, such as geospatial and astronomical images. 2.7. Extended Methods and Implementation

[0427] 2.7.1. SPT transformation details

[0428] As described above. SPT transformations were implemented with pre-computed embeddings and patch coordinates. Recall that each whole slide is represented as a (x,p) tuple, where x E is the embedding tensor and p E Z"x2is a corresponding coordinate tensor for each patch in the WSI. Each transformation is implemented by indexing into the rows (corresponding to patches / tokens) of x and p. A PyTorch style pseudocode is in FIG. 32.

[0429] Splitting has one hyperparameter: the ratio of tokens between two views. The ratio specifies the fraction of tokens in the first view and is used to compute the number of tokens in each view. Tokens are split randomly between views accordingly, and two disjoint views are returned.

[0430] Cropping has two sets of hyperparameters: cropping area range, and cropping aspect ratio range. The cropping area and cropping aspect ratio are randomly chosen from their respective ranges. A random token in the view with coordinates ( / u, CA) is selected to be an anchor as the crop center of the view. A coordinate range (ro, cd), (ri, ci) is computed based on the area and aspect ratio:

[0431] II — ^Area / aspect

[0432] TV — H ■ aspect where H. IF are the height and width of the crop. Tokens in this coordinate range are included in the transformed view.

[0433] Masking has two hyperparameters: masking ratio range and max masking token limit. The masking ratio is randomly drawn from the masking ratio range, and it is used to compute the number of tokens m to keep in the augmented view. Max masking token limit is a cap of the number of tokens in each view, i.e., m = min(m, max_token_lim). This parameter may be omitted (set to co) when training on datasets with smaller WSIs. Finally, m tokens were randomly select to remain in the augmented view. 2.7.2. Relative positional embedding

[0434] To work with WSIs of different sizes, it is non-trivial to use a fixed-size learnable position embedding with absolute distance

[0034] , The relative positional encoding introduced in

[0055] was adopted, which utilizes the learnable Fourier feature, modulated with a multilayer perceptron. This positional encoding uses the coordinates of the patch encoder as input and the output is added directly to the transformer with the same dimension. The positional embedding similarity is visualized in FIG. 27.

[0435] FIG. 27 illustrates cosine similarity of learned positional embeddings. Visualization is generated wherein each patch shows the cosine similarity between the position embedding of the token with the indicated row and column and the position embeddings of all other tokens.

[0436] 2.8. Extended Results

[0437] 2.8.1. ssSPT results with different patch encoders

[0438] In FIG. 25B, the ssSPT’s performance was compared to the pooling baseline and suSPT performance upper bound with the H&E glioma benchmark. Here, the same experiment is presented with the SRH CNS benchmark in FIG. 28. In the SRH benchmark, WSI representations improved dramatically in OOD and foundation patch encoders, which are not trained on SRH data. After training with SPT, a maximum 18.6% boost in MCA for ImageNet was demonstrated. The expected performance boost for in-domain patch encoder SimCLR and HiDisc was also observed. Similar to the H&E glioma benchmark, ssSPT performance approaches suSPT upperbound on all patch encoders, within 4 points in MCA for SRH CNS benchmarks. In addition, full metrics for each patch encoder for both SRH CNS and H&E glioma are in Table 4.

[0439] Table 4: SPT results with different patch encoder. Metrics reported for results in FIG. 25B. Standard deviations are in (parentheses).

[0440] FIG. 28 provides SPT results on SRH CNS with a wide range of patch encoders. Publicly available foundation patch encoders are trained using H&E data and are out-of- distribution (OOD) for SRH. In-domain patch encoders are trained using SRH data.

[0441] 2.8.2. suSPT benchmarks with UNI patch features suSPT was evaluated with UNI patch features in Table 3, above. Error bars are reported in Table 5, below.

[0442] Table 5: suSPT benchmark with UNI features. Extended results for Table 3 with standard deviations reported in (parentheses). 2.8.3. Ablation studies

[0443] In this section, ablation studies are presented over SSL paradigms, SPT transformation choices, and SPT transformation parameters.

[0444] SSL paradigms: The effect of different SSL paradigms on ssSPT was examine in Table 6, below. VICReg performs the best on the SRH CNS benchmark, BYOL performs the best on the TCGA RCC benchmark, while SimCLR performs the best on H&E glioma, TCGA BRCA, and TCGA NSCLC. These results demonstrate that the flexibility of SPT allows different algorithms to accommodate different training dynamics of different diagnostic tasks and image modalities. Table 6: SPT benchmark with different SSL paradigms. Different SPT training algorithms were compared. Standard deviations are reported in (parentheses)

[0445] SPT transformation choices: Table 7 below reports the performance of individual and combinations of slide-level transformations. Splitting alone was left out since it does not reduce the total number of patch tokens for both views, making it prohibitive to train. For the masking and cropping alone, it is shown that cropping performs better than masking for both SRH CNS and H&E Glioma. Combining cropping and masking shows similar performance to cropping alone, but is more efficient to train due to the reduced number of tokens per view. Combining each transformation with splitting improves or maintains performance, showing the benefit of further decreasing mutual information (MT) between views. Overall, cropping is the most important transformation, because capturing regional heterogeneity is the most effective and challenging pre- text task.

[0446] Table 7: Ablation over transformations. X stands for transformations used. Standard deviations reported in (parentheses).

[0447] SPT transformation parameters: The relationship between mutual information (MI) and model performance was examined by combining and adjusting SPT transformation parameters in the H&E glioma benchmark.

[0448] Token limit: the effect of the number of tokens, with splitting + cropping + masking and only masking was studied. A higher token limit results in a higher likelihood of overlapping area coverage between views. The results are in FIG. 29A. As expected, MI levels can be adjusted to optimize model performance. Previous studies have concluded that fewer patch tokens resulted in better model performance

[0015] , but only in the case of using masking transformation alone. Cropping size: cropping size range was varied in FIG. 29B. Cropping also allows the amount of MI to be adjusted between views, where a larger crop size leads to a higher probability of overlapping regions between views. The ablation study shows that for SimCLR training on the H&E glioma benchmark, [100,400] is the optimal cropping size range, achieving an MCA of 76.5. These results correspond to the hypothesis in

[0044] , where an optimal amount of MI between views achieves the best performance.

[0449] FIGS. 29A to 29B illustrate SPT transformation parameter ablation for H&E glioma molecular classification. FIG. 29A: Max token limit ablation. SCM, splitting-cropping- masking transformation. FIG. 29B: Cropping size range ablation. Cropping size range [min, max cropping size] is on the x-axis. None of the size ranges represent no cropping applied. The shaded area represents the standard deviation across three different random seeds.

[0450] SPT model size: additional ablation studies were performed with different SPT model sizes, and the results are reported in Table 8 below. The six- and two-layer transformers have the best performance for H&E glioma and SRH CNS, respectively. Smaller models performed better for SRH CNS, likely because of a smaller slide size and a relatively more uniform image. All models ranging from two to eight-layer transformers outperformed previous best self- supervised models.

[0451] Table 8: SPT ablation studies on model size. Standard deviations are in (parentheses).

[0452] 2.8.4. Extended atention heat maps

[0453] Extended attention heatmap visualization for SRH and H&E are presented in FIG. 30 and FIG. 31, respectively. Recall these attentions are the self-attention of the CLS token from the last layer of the whole slide encoder. On both SRH and H&E heatmaps, it can be seen that the whole slide encoder attends to clinically significant regions such as dense tumors, and regions with tumor infiltration, and avoids non-diagnostic regions such as blood, tissue processing artifacts, and blood vessels.

[0454] FIG. 30 provides attention heatmaps for H&E whole slides. The attention map shows the ssSPT-trained whole slide transformer can differentiate between different morphologies like dense tumors, blood, and different artifacts on the gigapixel WSIs. Visualization shows high attention to varying degrees of tumor infiltration, and low attention to low cellularity, blood, necrotic regions, and tissue processing artifacts. Oligo-oligodendroglioma, Astro=astrocytoma, GBM=glioblastoma.

[0455] FIG. 31 provides attention heatmaps for SRH whole slides. On this SRH WSI, relatively small compared to H&E, the model shows a strong capacity for unsupervised segmentation of histological features. Visualization shows high attention values to tumor regions, and low attention values to non-diagnostic regions such as blood vessels, laser noise, and empty space.

[0456] 2.9. References

[0457] [1] Xinliang Zhu, Jiawen Yao, Feiyun Zhu, and Junzhou Huang. Wsisa: Making survival prediction from whole slide histopathological images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7234-7242, 2017.

[0458] [2] Maximilian Use, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning, pages 2127-2136. PMLR, 2018.

[0459] [3] Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555-570, 2021.

[0460] [4] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al. Transmit: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136-2147, 2021.

[0461] [5] Richard J Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 16144-16155, 2022. [6] Syed Ashar Javed, Dinkar Juyal, Harshith Padigela, Amaro Taylor-Weiner, Limin Yu, and Aaditya Prakash. Additive mil: intrinsically interpretable multiple instance learning for pathology. Advances in Neural Information Processing Systems, 35:20689-20702, 2022.

[0462] [7] Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shahan, et al. Towards a general-purpose foundation model for computational pathology. Nature Medicine. 30(3):850— 862, 2024.

[0463] [8] Todd Hollon, Cheng Jiang, Asadur Chowdury, Mustafa Nasir-Moin, Akhil Kondepudi, Alexander Aabedi, Arjun Adapa, Wajd ALHolou, Jason Heth, Oren Sagher, et al. Artificial-intelligence based molecular classification of diffuse gliomas using rapid, label- free optical imaging. Nature Medicine, 29(4):828-832, 2023.

[0464] [9] Nicolas Coudray, Paolo Santiago Ocampo, Theodore Sakellaropoulos, Navneet Narula, Matija Snuderl, David Fenyo, Andre L Moreira, Narges Razavian, and Aristotelis Tsirigos. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nature medicine, 24(10): 1559-1567, 2018.

[0465]

[0010] Richard J Chen, Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Jana Lipkova, Zahra Noor, Muhammad Shaban, Maha Shady, Mane Williams, Bumjin Joo, et al. Pancancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell, 40(8):865— 878, 2022.

[0466]

[0011] Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 14318— 14328, 2021.

[0467]

[0012] Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Wemeck Krauss Silva, Klaus J Busam, Edi Brogi, Victor E Reuter, David S Klimstra. and Thomas J Fuchs. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine, 25(8): 1301-1309, 2019.

[0468]

[0013] Cheng Jiang, Xinhai Hou, Akhil Kondepudi, Asadur Chowdury, Christian W Freudiger, Daniel A Orringer, Honglak Lee, and Todd C Hollon. Hierarchical discriminative learning improves visual representations of biomedical microscopy. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 19798-19808, 2023.

[0469]

[0014] Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual-language foundation model for pathology image analysis using medical twitter. Nature medicine, 29(9):2307-2316, 2023.

[0470]

[0015] Tristan Lazard, Marvin Lerousseau, Etienne Decenciere. and Thomas Walter. Giga- ssl: Self-supervised learning for gigapixel images. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 4305-4314, 2023.

[0471]

[0016] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.

[0472]

[0017] Jean-Bastien Grill, Florian Strub, Florent Altche, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo. Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284, 2020.

[0473]

[0018] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-invariance- covariance regularization for self- supervised learning. In International Conference on Learning Representations, 2021.

[0474]

[0019] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18673, 2020.

[0475]

[0020] Ming Y Lu, Tiffany Y Chen, Drew FK Williamson, Melissa Zhao, Maha Shady, Jana Lipkova, and Faisal Mahmood. Ai-based pathology predicts origins for cancers of unknown primary. Nature. 594(7861): 106— 110, 2021.

[0476]

[0021] Todd C Hollon, Balaji Pandian, Arjun R Adapa, Esteban Urias, Akshay V Save, Siri Sahib S Khalsa, Daniel G Eichberg, Randy S D’Amico, Zia U Farooq, Spencer Lewis, et al. Near real-time intraoperative brain tumor diagnosis using stimulated raman histology and deep neural networks. Nature medicine, 26(1):52— 58, 2020.

[0477]

[0022] Richard J Chen, Ming Y Lu, Wei-Hung Weng, Tiffany Y Chen, Drew FK Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. Tn Proceedings of the IEEE / CVF International Conference on Computer Vision, pages 4015-4025, 2021.

[0478]

[0023] Chengkuan Chen, Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Andrew J Schaumberg, and Faisal Mahmood. Fast and scalable search of whole-slide images via selfsupervised deep learning. Nature Biomedical Engineering, 6(12): 1420-1434, 2022.

[0479]

[0024] Thomas G Dietterich, Richard H Lathrop, and Tomas Lozano-Perez. Solving the multiple instance problem with axis-parallel rectangles. Artificial intelligence, 89(l-2):31- 71, 1997.

[0480]

[0025] Le Hou, Dimitris Samaras, Tahsin M Kurc, Yi Gao, James E Davis, and Joel H Saltz. Patch-based convolutional neural network for whole slide tissue image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2424-2433, 2016.

[0481]

[0026] Jiawen Yao, Xinliang Zhu, and Junzhou Huang. Deep multi-instance learning for survival prediction from whole slide images. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part I 22, pages 496-504. Springer, 2019.

[0482]

[0027] Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins, and Junzhou Huang. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, 65:101789, 2020.

[0483]

[0028] Weijian Li, Viet-Duy Nguyen, Haofu Liao, Matt Wilder, Ke Cheng, and Jiebo Luo. Patch transformer for multi-tagging whole slide histopathology images. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13-17, 2019, Proceedings, Part I 22, pages 532-540. Springer, 2019.

[0484]

[0029] Xiyue Wang, Sen Yang, Jun Zhang, Minghui Wang, Jing Zhang, Wei Yang, Junzhou Huang, and Xiao Han. Transformer-based unsupervised contrastive learning for histopathological image classification. Medical image analysis, 81:102559. 2022.

[0485]

[0030] Jiahui Yu, Tianyu Ma, Yu Fu, Hang Chen, Maode Lai, Cheng Zhuo, and Yingke Xu. LocaLto global spatial learning for whole-slide image representation and classification. Computerized Medical Imaging and Graphics, 107:102230, 2023.

[0031] Tiancheng Lin, Zhimiao Yu, Hongyu Hu, Yi Xu, and Chang-Wen Chen. Interventional bag multi-instance learning on whole-slide pathological images. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 19830-19839, 2023.

[0486]

[0032] Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2020.

[0487]

[0033] Tomas Mikolov. Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv: 1301.3781, 2013.

[0488]

[0034] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.

[0489]

[0035] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 (8):9, 2019.

[0490]

[0036] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.

[0491]

[0037] Ming Y Lu, Bowen Chen, Andrew Zhang, Drew FK Williamson, Richard J Chen, Tong Ding, Long Phi Le, Yung-Sung Chuang, and Faisal Mahmood. Visual language pretrained multiple instance zero-shot transfer for histopathology images. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 19764- 19775, 2023.

[0492]

[0038] Maxime Oquab, Timothee Darcet, Theo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza. Francisco Massa. Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.

[0493]

[0039] Sobhan Hemati, Shivam Kalra, Cameron Meaney, Morteza Babaie, Ali Ghodsi, and Hamid Tizhoosh. Cnn and deep sets for end-to-end whole slide image representation learning. In Medical Imaging with Deep Learning, pages 301-311. PMLR, 2021.

[0040] Xiyue Wang, Jinxi Xiang, Jun Zhang, Sen Yang, Zhongyi Yang, Ming-Hui Wang, Jing Zhang, Wei Yang, Junzhou Huang, and Xiao Han. Scl-wc: Cross-slide contrastive learning for weakly-supervised whole-slide image classification. Advances in neural information processing systems, 35:18009-18021, 2022.

[0494]

[0041] Thomas E Tavolara, Metin N Gurcan, and M Khalid Khan Niazi. Contrastive multiple instance learning: An unsupervised framework for learning slide-level representations of whole slide histopathology images without labels. Cancers, 14(23):5778, 2022.

[0495]

[0042] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.

[0496]

[0043] Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, et al. A cookbook of self-supervised learning. arXiv preprint arXiv:2304.12210, 2023.

[0497]

[0044] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in neural information processing systems, 33:6827-6839, 2020.

[0498]

[0045] Ravid Shwartz Ziv and Yann LeCun. To compress or not to Compress-Self- Supervised learning and information theory: A review. Entropy, 26(3), March 2024.

[0499]

[0046] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924, 2020.

[0500]

[0047] Mathilde Caron, Hugo Touvron, Ishan Misra, Herve Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE / CVF international conference on computer vision, pages 9650- 9660, 2021.

[0501]

[0048] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.

[0502]

[0049] Christian W. Freudiger, Wei Min, Brian G. Saar, Sijia Lu, Gary R. Holtom, Chengwei He, Jason C. Tsai, Jing X. Kang, and X. Sunney Xie. Label-free biomedical imaging with high sensitivity by stimulated raman scattering microscopy. Science, 322(5909): 1857-1861, 2008.

[0503]

[0050] Daniel A Orringer, Balaji Pandian, Yashar S Niknafs, Todd C Hollon, Julianne Boyle, Spencer Lewis, Mia Garrard, Shawn L Hervey- Jumper, Hugh JL Garton, Cormac O Maher, et al. Rapid intraoperative histology of unprocessed surgical specimens via fibrelaser-based stimulated raman scattering microscopy. Nature biomedical engineering, l(2):0027, 2017.

[0504]

[0051] Cheng Jiang, Asadur Zaman Chowdury, Xinhai Hou, Akhil Kondepudi, Christian Freudiger, Kyle Stephen Conway, Sandra Camelo-Piragua, Daniel A Orringer, Honglak Lee, and Todd Hollon. OpenSRH: optimizing brain tumor surgery using intraoperative stimulated raman histology. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.

[0505]

[0052] Thomas Roetzer-Pejrimovsky, Anna-Christina Moser, Baran Atli, Clemens Christian Vogel, Petra A Mercea, Romana Prihoda, Ellen Gelpi, Christine Haberler, Romana Hoftberger, Johannes A Hainfellner, et al. the digital brain tumour atlas, an open histopathology resource. Scientific Data, 9(1):55, 2022.

[0506]

[0053] David N Louis, Arie Perry, Pieter Wesseling, Daniel J Brat, Ian A Cree, Dominique Figarella Branger, Cynthia Hawkins, HK Ng, Stefan M Pfister, Guido Reifenberger, et al. The 2021 who classification of tumors of the central nervous system: a summary. Neurooncology, 23(8): 1231-1251, 2021.

[0507]

[0054] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.

[0508]

[0055] Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. Learnable fourier features for multi-dimensional spatial positional encoding. Advances in Neural Information Processing Systems, 34:15816-15829, 2021.

[0509]

[0056] Marc Macenko, Marc Niethammer, James S Marron, David Borland, John T Woosley, Xiaojun Guan, Charles Schmitt, and Nancy E Thomas. A method for normalizing histology slides for quantitative analysis. In 2009 IEEE international symposium on biomedical imaging: from nano to macro, pages 1107-1110. IEEE, 2009.

[0057] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Geliy, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.

[0510] In at least some of the previously described embodiments, one or more elements used in an embodiment can interchangeably be used in another embodiment unless such a replacement is not technically feasible. It will be appreciated by those skilled in the art that various other omissions, additions and modifications may be made to the methods and structures described above without departing from the scope of the claimed subject matter. All such modifications and changes are intended to fall within the scope of the subject matter, as defined by the appended claims.

[0511] It will be understood by those within the art that, in general, terms used herein, and especially in the appended claims (e.g., bodies of the appended claims) are generally intended as “open” terms (e.g., the term “including” should be interpreted as “including but not limited to,” the term “having” should be interpreted as “having at least,” the term “includes” should be interpreted as “includes but is not limited to,” etc.). It will be further understood by those within the art that if a specific number of an introduced claim recitation is intended, such an intent will be explicitly recited in the claim, and in the absence of such recitation no such intent is present. For example, as an aid to understanding, the following appended claims may contain usage of the introductory phrases “at least one” and “one or more” to introduce claim recitations. However, the use of such phrases should not be construed to imply that the introduction of a claim recitation by the indefinite articles “a” or “an” limits any particular claim containing such introduced claim recitation to embodiments containing only one such recitation, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an” (e.g., “a” and / or “an” should be interpreted to mean “at least one” or “one or more”); the same holds true for the use of definite articles used to introduce claim recitations. In addition, even if a specific number of an introduced claim recitation is explicitly recited, those skilled in the art will recognize that such recitation should be interpreted to mean at least the recited number (e.g., the bare recitation of “two recitations,” without other modifiers, means at least two

[0512] I ll recitations, or two or more recitations). Furthermore, in those instances where a convention analogous to “at least one of A, B, and C, etc.” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “ a system having at least one of A, B, and C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). In those instances where a convention analogous to “at least one of A, B, or C, etc.” is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., “ a system having at least one of A, B, or C” would include but not be limited to systems that have A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B, and C together, etc.). It will be further understood by those within the art that virtually any disjunctive word and / or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase “A or B” will be understood to include the possibilities of “A” or “B” or “A and B.”

[0513] In addition, where features or aspects of the disclosure are described in terms of Markush groups, those skilled in the art will recognize that the disclosure is also thereby described in terms of any individual member or subgroup of members of the Markush group. As will be understood by one skilled in the art, for any and all purposes, such as in terms of providing a written description, all ranges disclosed herein also encompass any and all possible sub-ranges and combinations of sub-ranges thereof. Any listed range can be easily recognized as sufficiently describing and enabling the same range being broken down into at least equal halves, thirds, quarters, fifths, tenths, etc. As a non-limiting example, each range discussed herein can be readily broken down into a lower third, middle third and upper third, etc. As will also be understood by one skilled in the art all language such as “up to,” “at least,” “greater than,” “less than,” and the like include the number recited and refer to ranges which can be subsequently broken down into sub-ranges as discussed above. Finally, as will be understood by one skilled in the art, a range includes each individual member. Thus, for example, a group having 1-3 articles refers to groups having 1, 2, or 3 articles. Similarly, a group having 1-5 articles refers to groups having 1, 2, 3, 4, or 5 articles, and so forth. Although the foregoing invention has been described in some detail by way of illustration and example for purposes of clarity of understanding, it is readily apparent to those of ordinary skill in the art in light of the teachings of this invention that certain changes and modifications may be made thereto without departing from the spirit or scope of the appended claims.

[0514] Accordingly, the preceding merely illustrates the principles of the invention. It will be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope. Furthermore, all examples and conditional language recited herein are principally intended to aid the reader in understanding the principles of the invention and the concepts contributed by the inventors to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and embodiments of the invention as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents and equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims.

[0515] The scope of the present invention, therefore, is not intended to be limited to the exemplary embodiments shown and described herein. Rather, the scope and spirit of present invention is embodied by the appended claims. In the claims, 35 U.S.C. § 112(f) or 35 U.S.C. § 112(6) is expressly defined as being invoked for a limitation in the claim only when the exact phrase "means for" or the exact phrase "step for" is recited at the beginning of such limitation in the claim; if such exact phrase is not used in a limitation in the claim, then 35 U.S.C. § 112 (f) or 35 U.S.C. §112(6) is not invoked.

Claims

WHAT IS CLAIMED IS:

1. A method of training a visual machine learning model to identify and encode tissue image features, the method comprising: obtaining a set of tissue images comprising a plurality of neoplasm and / or tissue types, wherein each tissue image comprises multiple patches; training a tokenizer of the model to convert patches into tokens by identifying and encoding patch features using the patches of the tissue image set, wherein the tokenizer comprises a convolutional neural network (CNN) and the training comprises self-supervised contrastive learning; generating two or more views of each of a plurality of images of the tissue image set, wherein the plurality of images have been passed through the tokenizer; and training an image encoder of the model to identify and encode tissue image features using the two or more views of the plurality of tokenized images, wherein the image encoder comprises a vision transformer and the training comprises self-supervised learning.

2. The method according to Claim 1, wherein the method further comprises dividing images of the tissue image set into multiple patches.

3. The method according to Claim 1 or 2, wherein the model is a foundation model.

4. The method according to any one of the preceding claims, wherein the set of tissue images each comprise a slide of tissue.

5. The method according to Claim 4, wherein the set of tissue images are obtained via stimulated Raman histology (SRH).

6. The method according to Claim 4 or 5, wherein the set of tissue images are obtained from a plurality of subjects.

7. The method according to Claim 6, wherein the set of tissue images are obtained from 1,000 or more subjects.

8. The method according to Claim 6 or 7, wherein the set of tissue images comprises a plurality of neoplasm types.

9. The method according to Claim 8, wherein the plurality of neoplasm types comprise central nervous system (CNS) tumors, wherein the CNS tumors comprise one or more of: high-grade gliomas (HGG), low-grade gliomas (LGG), meningiomas, pituitary adenomas, schwannomas, and / or metastatic tumors.

10. The method according to any one of Claims 6 to 9, wherein the set of tissue images comprises tissues from multiple different organ systems.

11. The method according to Claim 10, wherein the set of tissue images comprises neoplasms from multiple different organ systems.

12. The method according to any one of Claims 6 to 11, wherein the set of tissue images comprises 10,000 or more images.

13. The method according to any one of Claims 6 to 12, wherein the self-supervised contrastive learning comprises hierarchical self-supervised learning via a hierarchical discriminative learning task.

14. The method according to Claim 13, wherein the tokenizer is trained to identify and encode patch features for each level of a multi-leveled hierarchy, wherein the hierarchy comprises a patch level, an image level, and a subject level.

15. The method according to Claim 14, wherein the hierarchical self- supervised learning of the tokenizer comprises: generating a plurality of positive patch pairs for each of the patch, image, and subject levels of the hierarchy from the set of tissue images; passing the positive patch pairs through the CNN; andadjusting the CNN to minimize a contrastive learning objective function, wherein the contrastive learning objective function comprises patch, image, and subject level components.

16. The method according to any one of Claims 13 to 15, wherein the CNN is a residual CNN.

17. The method according to Claim 16, wherein the residual CNN comprises ResNet-18, ResNet-34, ResNet-50, ResNet-101, or ResNet-152.

18. The method according to any one of Claims 6 to 17, wherein the two or more views of each of the plurality of images are generated by randomly: splitting each image into mutually exclusive patch sets; spatially cropping each patch set; and masking multiple patches from each spatially cropped patch set.

19. The method according to Claim 18, wherein the self-supervised learning of the image encoder comprises: passing each view generated for an image of the plurality of images through a different vision transformer of a Siamese architecture, wherein the transformers of the architecture share weights; and adjusting the transformers of the architecture to minimize a self-supervised objective function, wherein the self-supervised objective function comprises variance, invariance, and covariance components.

20. The method according to Claim 19, wherein the self-supervised objective function is Variance-Invariance-Covariance Regularization (VICReg).

21. The method according to Claim 19 or 20, wherein the transformers of the architecture comprise two hidden layers, wherein each hidden layer comprises a dimension of 512 and 4 attention heads.

22. The method according to any one of Claims 19 to 21, wherein the transformers of the architecture comprise a component for generating positional embeddings.

23. The method according to Claim 22, wherein the positional embedding component comprises a Fourier feature positional embedding generator network.

24. The method according to any one of Claims 6 to 23, wherein the method further comprises fine-tuning the model and / or training an output module of the model to perform one or more specific tasks.

25. The method according to Claim 24, wherein the method further comprises training an output module of the model to perform a specific task by passing tissue images through the tokenizer and the image encoder of the model.

26. The method according to Claim 25, wherein the output module comprises a K- Nearest Neighbors (KNN) classifier.

27. The method according to Claim 25, wherein the output module comprises one or more output layers.

28. The method according to Claim 26, wherein the output module comprises a one layer linear output layer.

29. The method according to Claim 27 or 28, wherein the output module is trained to diagnose brain tumors.

30. The method according to any one of Claims 27 to 29, wherein the output module is trained to quantify tumor infiltration or tumor burden.31 . The method according to any one of Claims 27 to 30, wherein the output module is fine-tuned using a second set of tissue images, wherein the second set of tissue images comprises a smaller number of images than the initial set of images used to train the tokenizer and the image encoder of the model.

32. The method according to Claim 31, wherein the image encoder of the model is finetuned using the second set of tissue images.

33. The method according to Claim 31 or 32, wherein each image of the second set of tissue images comprises an ordinal label corresponding to a category of a discrete and ordered scale.

34. The method according to Claim 33, wherein the method further comprises generating the ordinal labels for the second set of tissue images.

35. The method according to Claim 33 or 34, wherein the scale comprises a tumor infiltration scale quantifying the degree of tumor infiltration in each image of the second set of tissue images.

36. The method according to Claim 35, wherein the ordinal labels are generated using domain specific knowledge and one or more of: SRH imaging, hematoxylin and eosin (H&E) staining, and / or immunohistochemistry.

37. The method according to Claim 35 or 36, wherein the tumor infiltration scale comprises four categories, wherein: the first category corresponds to tissue images where no tumor is present; the second category corresponds to tissue images comprising atypical cells and / or images where it is possible, but not definite, that a tumor is present; the third category corresponds to tissue images comprising sparse tumor infiltration; and the fourth category corresponds to tissue images comprising dense tumor infiltration.

38. The method according to any one of Claims 33 to 37, wherein the fine-tuning comprises: passing tissue images of the second set through the tokenizer, the image encoder, and the output module of the model; and adjusting the output module and / or the image encoder to: maximize the latent distance between one or more encoded features of tissue images of the second set having different ordinal labels, and / or minimize the latent distance between one or more encoded features of tissue images of the second set having the same ordinal label.

39. The method according to any one of Claims 33 to 38, wherein the output module comprises an image scorer fine-tuned to output a continuous score corresponding to the discrete and ordered scale of the ordinal labels of the second set of tissue images.

40. The method according to Claim 39, wherein the image scorer comprises a one layer linear scorer.

41. The method according to Claim 39 or 40, wherein the fine-tuning comprises: passing a batch of tissue images of the second set through the tokenizer, the image encoder, and the image scorer of the model in order to produce a continuous score for each tissue image of the batch; performing a pairwise comparison between all tissue images of the batch using the continuous scores and a binary cross entropy objective function; and adjusting the image scorer and / or the image encoder of the model to minimize the binary cross entropy objective function.

42. The method according to Claim 41, wherein class imbalance is addressed by oversampling tissue images of the second set corresponding to ordinal labels of a minority class.

43. The method according to any one of Claims 35 to 42, wherein the fine-tuned model is used to detect and quantify tumor infiltration in one or more unprocessed, label-free, surgical samples.

44. The method according to Claim 43, wherein the one or more surgical samples are imaged at lower resolutions and faster acquisition rates than the tissue images used to train and / or fine-tune the model.

45. The method according to any one of Claims 1 to 44, wherein the method further comprises generating a visualization identifying one or more regions of interest (ROIs) within a tissue image passed through the model.

46. The method according to Claim 45, wherein the visualization is generated via principal component analysis (PC A).

47. The method according to Claim 45, wherein the visualization is generated using a set of patch keys comprising features or characteristics of the one or more ROIs.

48. The method according to Claim 47, wherein the set of patch keys comprises two or more image patches selected by a medical professional.

49. The method according to Claim 47 or 48, wherein the set of patch keys comprises examples of normal tissue and irregular tissue.

50. The method according to Claim 49, wherein the set of patch keys comprises diverse examples of normal brain parenchyma and diffuse glioma subtypes.

51. The method according to Claim 49 or 50, wherein regions of interest are identified by comparing the set of patch keys, having passed through the model, with each of the patches of the tissue image passed through the model.

52. The method according to Claim 51 , wherein generating the visualization comprises: calculating a few-shot visualization score for one or more of the patches of the tissue image, wherein for each of the one or more patches the calculating comprises: determining a maximum similarity value between a patch of the tissue image and each of the irregular tissue patch keys by calculating a dot product between the tissue image patch and each of the irregular tissue patch keys; determining a maximum similarity value between the tissue image patch and each of the normal tissue patch keys by calculating a dot product between the tissue image patch and each of the normal tissue patch keys; and calculating a few-shot visualization score for the tissue image patch by determining a difference between the irregular tissue maximum similarity value and the normal tissue maximum similarity value; and generating the visualization by indicating patches of the tissue image having at or above a threshold few-shot visualization score.

53. The method according to Claim 52, wherein the irregular tissue maximum similarity value and the normal tissue maximum similarity value are determined using cosine similarity.

54. The method according to Claim 52 or 53, wherein few-shot visualization scores are only calculated for patches of the tissue image having at or above a threshold irregular tissue maximum similarity value, wherein tissue image patches having below the threshold irregular tissue maximum similarity value are classified as background patches.

55. The method according to any one of Claims 52 to 54, wherein generating the visualization further comprises: constructing a heatmap indicating one or more ROIs by rescaling each few-shot visualization score, wherein each patch of the tissue image is colored based on its rescaled few-shot visualization score; and overlaying the heatmap on the tissue image to generate the visualization.

56. The method according to Claim 55, wherein patches of the tissue image for which no few-shot visualization score was calculated are colored in the same manner as patches of the tissue image having below a threshold few-shot visualization score.

57. The method according to any one of Claims 1 to 56, wherein the model is used as a surgical adjunct.

58. The method according to Claim 57, wherein the model is used to guide a surgical resection of a tumor.

59. The method according to any one of Claims 1 to 58, wherein one or more hyperparameters used to train or fine-tune the model are optimized.

60. The method according to Claim 59, wherein the one or more hyperparameters are optimized by performing an ablation study.

61. The method according to Claim 59 or 60, wherein the one or more hyperparameters are optimized using a validation set of tissue images held out of the set of tissue images used to train or fine-tune the model.

62. A method of fine-tuning a model to output a continuous score corresponding to a discrete and ordered scale of ordinal categories, the method comprising: obtaining a set of tissue images comprising a plurality of neoplasm and / or tissue types, wherein each tissue image of the set comprises an ordinal label corresponding to a category of the discrete and ordered scale; passing a batch of tissue images of the tissue image set through the model in order to produce a continuous score for each tissue image of the batch, wherein the model comprises: an image encoder trained to identify and encode tissue image features for each tissue image of the batch; andan output module comprising an image scorer configured to output the continuous score for each tissue image of the batch using the encoded tissue image features; performing a pairwise comparison between all tissue images of the batch using the continuous scores and a binary cross entropy objective function; and adjusting the image scorer of the model to minimize the binary cross entropy objective function such that the continuous scores output by the model better correspond to the discrete and ordered scale.

63. The method according to Claim 62, wherein the model comprises a model generated using the methods of any one of Claims 1 to 61.

64. The method according to Claim 62, wherein the model comprises a CTransPath, Phycon, or UNI model.

65. The method according to any one of Claims 62 to 64, wherein the set of tissue images used to fine-tune the model comprises a smaller number of images than an initial set of tissue images used to train the model.

66. The method according to any one of Claims 62 to 65, wherein the image encoder of the model is adjusted to minimize the binary cross entropy objective function.

67. The method according to any one of Claims 62 to 66, wherein the method further comprises generating the ordinal labels for the set of tissue images.

68. The method according to Claim 67, wherein the scale comprises a tumor infiltration scale quantifying the degree of tumor infiltration in each image of the set of tissue images69. The method according to Claim 68, wherein the ordinal labels are generated using domain specific knowledge and one or more of: SRH imaging, hematoxylin and eosin (H&E) staining, and / or immunohistochemistry.

70. The method according to Claim 68 or 69, wherein the tumor infiltration scale comprises four categories, wherein: the first category corresponds to tissue images where no tumor is present; the second category corresponds to tissue images comprising atypical cells and / or images where it is possible, but not definite, that a tumor is present; the third category corresponds to tissue images comprising sparse tumor infiltration; and the fourth category corresponds to tissue images comprising dense tumor infiltration.

71. The method according to any one of Claims 62 to 70, wherein the image scorer comprises a one layer linear scorer.

72. The method according to any one of Claims 62 to 71, wherein the fine-tuning comprises: adjusting the output module and / or the image encoder to: maximize the latent distance between one or more encoded features of tissue images of the set having different ordinal labels, and / or minimize the latent distance between one or more encoded features of tissue images of the set having the same ordinal label.

73. The method according to any one of Claims 62 to 72, wherein class imbalance is addressed by oversampling tissue images of the set corresponding to ordinal labels of a minority class.

74. The method according to any one of Claims 62 to 73, wherein the fine-tuned model is used to detect and quantify tumor infiltration in one or more unprocessed, label-free, surgical samples.

75. The method according to Claim 74, wherein the one or more surgical samples are imaged at lower resolutions and faster acquisition rates than the tissue images used to train and / or fine-tune the model.

76. A method of generating a visualization identifying one or more regions of interest (ROIs) within a tissue image passed through a model, the method comprising: passing a set of patch keys comprising features or characteristics of the one or more ROIs through the model, wherein the set of patch keys comprises examples of normal tissue and irregular tissue; calculating a few-shot visualization score for one or more patches of the tissue image, wherein for each of the one or more patches, the calculating comprises: determining a maximum similarity value between a patch of the tissue image and each of the irregular tissue patch keys by calculating a dot product between the tissue image patch and each of the irregular tissue patch keys; determining a maximum similarity value between the tissue image patch and each of the normal tissue patch keys by calculating a dot product between the tissue image patch and each of the normal tissue patch keys; and calculating a few-shot visualization score for the tissue image patch by determining a difference between the irregular tissue maximum similarity value and the normal tissue maximum similarity value; and generating the visualization by indicating patches of the tissue image having at or above a threshold few-shot visualization score.

77. The method according to Claim 76, wherein the model comprises a model generated using the methods of any one of Claims 1 to 61.

78. The method according to Claim 76, wherein the model comprises a CTransPath, Phycon, or UNI model.

79. The method according to any one of Claims 76 to 78, wherein the set of patch keys comprises two or more image patches selected by a medical professional.

80. The method according to any one of Claims 76 to 79, wherein the set of patch keys comprises diverse examples of normal brain parenchyma and diffuse glioma subtypes.

81. The method according to any one of Claims 76 to 80, wherein the irregular tissue maximum similarity value and the normal tissue maximum similarity value are determined, at least in part, using cosine similarity.

82. The method according to any one of Claims 76 to 81, wherein few-shot visualization scores are only calculated for patches of the tissue image having at or above a threshold irregular tissue maximum similarity value, wherein tissue image patches having below the threshold irregular tissue maximum similarity value are classified as background patches.

83. The method according to any one of Claims 76 to 82, wherein generating the visualization further comprises: constructing a heatmap indicating one or more ROIs by rescaling each few-shot visualization score, wherein each patch of the tissue image is colored based on its rescaled few-shot visualization score; and overlaying the heatmap on the tissue image to generate the visualization.

84. The method according to Claim 83, wherein patches of the tissue image for which no few-shot visualization score was calculated are colored in the same manner as patches of the tissue image having below a threshold few-shot visualization score.

Images