Method for determining biomarker from pathological tissue slide image

Abstract

To provide a novel technology capable of easily diagnosing TIL, PD-L1 and other biomarkers using H&E images to identify and characterize biomarkers in an efficient manner across population groups, generate further optimized drug treatment recommendations and protocols, and improve prediction of disease progression.SOLUTION: A generalizable and interpretable deep learning model is provided for predicting biomarker status and biomarker metrics from pathological tissue slide images.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation-in-part of U.S. Patent Application No. 16 / 732,242, filed December 31, 2019, which claims priority to U.S. Provisional Patent Application No. 62 / 787,047, filed December 31, 2018, and is a continuation-in-part of U.S. Patent Application No. 16 / 41, filed May 14, 2019, which claims priority to U.S. Provisional Patent Application No. 62 / 671,300, filed May 14, 2018. This application is a continuation-in-part of U.S. Provisional Patent Application No. 2,362, which claims priority to U.S. Provisional Patent Application No. 62 / 824,039, filed March 26, 2019, U.S. Provisional Patent Application No. 62 / 889,521, filed August 20, 2019, and U.S. Provisional Patent Application No. 62 / 983,524, filed February 28, 2020, the entire disclosures of each of which are expressly incorporated herein by reference.

[0002] The present disclosure relates to detecting, quantifying, and / or characterizing biomarkers associated with cancer, and more particularly, to examining digital images to detect, quantify, and / or characterize such biomarkers from analysis of one or more tissue pathology slide images. [Background technology]

[0003] The description of the background art provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors to the extent described in this Background section, and aspects of the description that may not otherwise be considered prior art at the time of filing, are not admitted, expressly or impliedly, as prior art to the present disclosure.

[0004] It is common to extract and examine tumor samples from patients to guide medical professionals in the diagnosis, prognosis, and treatment evaluation of a patient's cancer. Visual inspection can reveal the growth pattern of cancer cells within the tumor in relation to nearby healthy cells, and the presence of immune cells within the tumor. Traditionally, thin slices of tumor tissue mounted on glass microscope slides are visually analyzed by a pathologist, a member of a pathology team, other trained medical professionals, or other human analysts to identify each area of ​​tissue that corresponds to one of the many tissue types present in the tumor sample. This information helps the pathologist determine the characteristics of the patient's cancer tumor and may aid in treatment decisions. Pathologists often assign one or more numerical scores to the slides based on their visual approximation.

[0005] To make these visual approximations, medical professionals attempt to identify many features of the tumor, including, for example, the tumor's grade, the tumor's purity, the degree of invasiveness of the tumor, the degree of immune infiltration of the tumor, the stage of the cancer, and the tumor's anatomical site of origin, which may be important in diagnosing and treating metastatic tumors. These details about the cancer may help physicians monitor the progression of the cancer within a patient and predict which anti-cancer treatments are likely to be successful in eliminating cancer cells from the patient's body.

[0006] Another feature of tumors is the presence of certain biomarkers or other cell types, including immune cells, in or near tumors. For example, tumor infiltrating lymphocytes (TILs), present at high levels, have been recognized as a biomarker of antitumor immune responses across a broad range of tumors. TILs are mononuclear immune cells that infiltrate tumor tissue or stroma and have been reported in multiple tumor types, including breast cancer. The population of TILs consists of various types of cells (i.e., T cells, B cells, natural killer (NK) cells, etc.). Although the population of TILs that naturally occurs in cancer patients is largely ineffective in destroying tumors, the presence of TILs has been associated with improved prognosis in many types of cancer, such as epithelial ovarian cancer, colon cancer, esophageal cancer, melanoma, endometrial cancer, and breast cancer (see, e.g., Melichar et al., Anticancer Res. 2014;34(3):1115-25; Naito et al., Cancer Res. 1998;58(16):3491-4).

[0007] Yet another feature of tumors is the presence of certain molecules as biomarkers, including a molecule known as programmed death-ligand 1 (PD-L1). PD-L1 is relevant in the diagnosis and evaluation of non-small cell lung cancer (NSCLC). NSCLC is the most common type of lung cancer, affecting more than 1.5 million people worldwide. NSCLC has a poor response to standard treatment chemoradiotherapy and a high incidence of recurrence, resulting in a low 5-year survival rate. Advances in immunology have revealed that expression of PD-L1, which binds to programmed death-1 (PD-1) expressed on the surface of T cells, is frequently elevated in NSCLC. Binding of PD-1 to PD-L1 deactivates the antitumor response of T cells, allowing NSCLC to avoid targeting by the immune system. The discovery of the interplay between tumor progression and immune response has led to the development and regulatory approval of PD-1 / PD-L1 checkpoint blockade immunotherapies, such as nivolumab and pembrolizumab. Anti-PD-1 and anti-PD-L1 antibodies restore antitumor immune responses by disrupting the interaction between PD-1 and PD-L1. Notably, patients with PD-L1-positive NSCLC treated with these checkpoint inhibitors have achieved durable tumor regression and improved survival.

[0008] As the role of immunotherapy in oncology expands, accurate assessment of tumor PD-L1 status may help identify patients who may benefit from PD-1 / PD-L1 checkpoint blockade immunotherapy. Currently, immunohistochemistry (IHC) staining of tumor tissue obtained from biopsy or surgical specimens is employed to assess PD-L1 status. However, such IHC staining is often limited due to insufficient tissue samples or lack of resources in some settings.

[0009] Hematoxylin and eosin (H&E) staining is a long-standing method used by pathologists to analyze the morphological characteristics of tissue for the diagnosis of malignant tumors. For example, H&E slides can reveal visual characteristics of tissue structures, such as cell nuclei and cytoplasm, to aid in the identification of cancer tumors.

[0010] Advances in technology have allowed histopathological H&E and IHC slides to be digitized into high-resolution whole slide images (WSIs), providing the opportunity to develop computer vision tools for a wide range of clinical applications. High-resolution digital images of microscope slides allow computer-aided analysis of the slides, enabling classification of tissue types or pathologies. Broadly speaking, deep learning applications have shown promise, for example, as a tool in medical diagnostic applications and predicting treatment outcomes. Deep learning is a subset of machine learning, where models can be built with multiple distinct neural node layers. A convolutional neural network ("CNN") is a neural network that employs convolutional techniques. For example, a CNN can provide a deep learning process, whereby digital images are analyzed by assigning a class label to each input image. However, WSIs contain two or more types of tissues, including the boundaries between adjacent tissue classes. To analyze the boundaries between adjacent tissue classes and the presence of immune cells among tumor cells, it is necessary to classify partially distinct regions as different tissue classes. For a traditional CNN to assign multiple tissue classes to a single slide image, the CNN must process each section of the image separately that needs to be assigned a tissue class label. However, because sections of adjacent images overlap, processing each section separately requires a lot of redundant computation and is time consuming.

[0011] A Fully Convolutional Network (FCN) is another type of deep learning process. FCNs can analyze images and assign a classification label to each pixel in the image. As a result, compared to CNNs, FCNs are useful for analyzing images that represent objects with more than one classification. FCNs generate an overlay map that shows the location of each classified object in the original image. However, for FCN deep learning algorithms to be effective, they must be trained on a dataset of images where each pixel is labeled as a tissue class, which requires too much annotation time and processing time to be practical. Digital WSI images may contain more than 10,000 to 100,000 pixels on each edge of the image. A complete image requires at least 10,000 2 ~100,000 2 A slide may contain pixels, which would require very long algorithm run times to attempt tissue classification. Due to the large number of pixels, it is not possible to segment a digital image of a slide using a traditional FCN.

[0012] New technologies that can easily diagnose TILs, PD-L1, and other biomarkers using H&E images are needed to identify and characterize such biomarkers across population groups in an efficient manner, generate more optimized drug treatment recommendations and protocols, and improve prediction of disease progression. Summary of the Invention

[0013] The present application presents an imaging-based biomarker prediction system formed of a deep learning framework configured and trained to learn directly from histopathology slide images and predict the presence of biomarkers in medical images. In examples, the deep learning framework is configured and trained to analyze histopathology images and identify a plurality of different biomarkers. In various examples, these deep learning frameworks are configured to include different trained biomarker classifiers, each configured to receive unlabeled histopathology images and provide different biomarker predictions for those images. These biomarker predictions can then be used to reduce a large set of available immunotherapies to a reduced small subset of targeted immunotherapies that can be used by medical professionals to treat patients. Thus, in various examples, a deep learning framework is provided that identifies biomarkers indicative of the presence of a tumor, the state / condition of the tumor, or information about the tumor in a tissue sample, from which a set of targeted immunotherapies can be determined.

[0014] In examples, the system includes a deep learning framework that is trained to analyze and predict biomarker status of tissue pathology images received from a network-accessible image source, such as a medical laboratory or medical imaging machine, and generate and store and display reports of the predicted biomarker status. These predicted biomarker status reports can be provided to be stored and displayed in network-accessible systems, such as pathology laboratories and primary care physician systems, and used to determine a patient's cancer treatment protocol (i.e., immunotherapy or chemotherapy treatment). In some examples, the predicted biomarker status reports can be input to a network-accessible next generation sequencing system to drive subsequent genomic sequencing, or to a computerized cancer treatment decision system to filter treatment lists to biomarker-determined corresponding treatments.

[0015] The technology herein can identify biomarkers associated with any of a wide variety of cancers. Exemplary cancers include, but are not limited to, adrenocortical carcinoma, lymphoma, anal cancer, anorectal cancer, basal cell carcinoma, skin cancer (non-melanoma), cholangiocarcinoma, extrahepatic bile duct cancer, intrahepatic bile duct cancer, bladder cancer, urinary bladder cancer, osteosarcoma, brain tumor, brain stem glioma, breast cancer (including triple-negative breast cancer), cervical cancer, colon cancer, colorectal cancer, lymphoma, endometrial cancer, esophageal cancer, gastric (stomach) cancer, head and neck cancer, hepatocellular (liver) cancer, renal cancer, kidney cancer, lung cancer, melanoma, tongue cancer, oral cancer, ovarian cancer, pancreatic cancer, prostate cancer, uterine cancer, testicular cancer, and vaginal cancer.

[0016] In some examples, the imaging-based biomarker prediction system is formed in a deep learning framework, the deep learning framework having a multi-scale configuration designed to perform classification of tissue images (labeled or unlabeled) using a classifier trained to classify tiles of received tissue images. In some examples, the multi-scale configuration includes a tile-level tissue classifier, i.e., a classifier trained using tile-based deep learning training. In some examples, the multi-scale configuration includes a pixel-level cell classifier and a cell segmentation model. In some examples, classifications from the tile-level tissue classifier and the pixel-level cell classifier are analyzed to predict the status of biomarkers in the tissue images. In further examples, the multi-scale configuration includes a tile-level biomarker classifier.

[0017] In some examples, the imaging-based biomarker prediction system is formed in a deep learning framework, which has a single-scale configuration designed to perform classification of histopathology images (labeled or unlabeled) using a classifier trained using a multi-instance learning (MIL) technique. In some examples, the single-scale configuration includes a slide-level classifier trained using gene sequencing data, such as RNA sequencing data. That is, using RNA sequencing data, a slide-level classifier is trained to develop an image-based classifier that can predict the status of a biomarker in a histopathology image.

[0018] According to one example, a computer-implemented method for identifying biomarkers in a digital image of a hematoxylin and eosin (H&E) stained slide of a target tissue includes receiving the digital image into an image-based biomarker prediction system having one or more processors; performing, using the one or more processors, an image tiling process on the digital image by separating the digital image into a plurality of tile images, each of the plurality of tile images including a different portion of the digital image; applying, using the one or more processors, the plurality of tile images to a multi-scale deep learning framework including one or more trained deep learning multi-scale classifier models, each trained to classify a different tissue classification for each tile image, and determining, using the multi-scale deep learning framework, a tissue classification for each of the plurality of tile images; identifying cells in the digital image using the trained cell segmentation model, using the one or more processors; and identifying a predicted presence of one or more biomarkers associated with the digital image from the tissue classification determined for each tile image and from the identified cells in the digital image.

[0019] According to another example, a computer-implemented method for identifying biomarkers in digital images of hematoxylin and eosin (H&E) stained slides of a target tissue includes receiving a molecular training dataset of a plurality of training tissue samples, the molecular training dataset including RNA transcriptome counts from sequencing of substantially similar samples associated with each training tissue sample; performing a clustering process on the molecular training dataset to identify one or more molecular data subsets, each corresponding to a distinct biomarker; and, for each of the one or more molecular data subsets, performing an image-based biomarker clustering process with one or more processors. receiving a plurality of digital images of H&E stained training slides of training tissue samples corresponding to respective biomarkers for the prediction system; generating, using one or more processors, a trained image-based biomarker classifier model for each of the one or more molecular data subsets based on the plurality of digital images of the H&E stained training slides; receiving, using one or more processors, a subsequent digital image of an H&E stained slide of a subsequent tissue sample; and applying, using one or more processors, the subsequent digital image to the trained image-based biomarker classifier model to identify a predicted presence of one or more biomarkers in the subsequent tissue sample.

[0020] According to another example, a computer-implemented method for identifying biomarkers in a digital image of a hematoxylin and eosin (H&E) stained slide of a target tissue includes receiving the digital image into an image-based biomarker prediction system having one or more processors; separating, using the one or more processors, the digital image into a plurality of tile images, each of the plurality of tile images comprising a different portion of the digital image; applying, using the one or more processors, the plurality of tile images to a deep learning framework including one or more trained biomarker classification models, each of the plurality of tile images trained to classify a different tissue classification; predicting, using the one or more processors, a biomarker classification for each of the plurality of tile images using the one or more trained biomarker classification models; determining a predicted presence of one or more biomarkers in the target tissue from the predicted biomarker classification for each of the tile images; and generating a report including the digital image and a digital overlay visualizing the predicted presence of the one or more biomarkers.

[0021] In some examples, the deep learning framework includes a multi-scale deep learning framework.

[0022] In some examples, separating the digital image into a plurality of tile images includes performing, using one or more processors, an image tiling process by applying a tiling mask to the digital image to separate the digital image into a plurality of tile images.

[0023] In some examples, the tiling mask includes tiles of the same size and / or tiles having a rectangular shape.

[0024] In some examples, applying the plurality of tile images to a deep learning framework and predicting a biomarker classification for each of the plurality of tile images each includes applying each of the tile images to one or more trained deep learning multi-scale classifier models, each trained to classify a different tissue classification for each tile image, and determining a tissue classification for each of the plurality of tile images using the multi-scale deep learning framework; identifying cells in the digital image using the trained cell segmentation model, using one or more processors; and predicting a biomarker classification for each tile image from the tissue classification determined for each tile image and from the identified cells in the digital image.

[0025] In some examples, the method further includes training one or more trained deep learning multi-scale classifier models in a multi-scale deep learning framework by receiving a plurality of H&E slide training images from a training image dataset, each H&E slide training image having a label corresponding to a biomarker to be trained on, performing a tile-based tissue classification analysis on each of the H&E slide training images, performing a pixel-based cell segmentation analysis on each of the H&E slide training images, and optionally performing a tile-based biomarker classification analysis on each of the H&E slide training images, and generating one or more trained deep learning multi-scale classifier models accordingly.

[0026] In some examples, each H&E slide training image includes multiple tile images, each with a tile-level label.

[0027] In some examples, the method includes, for each H&E slide training image, labeling each of a plurality of tile images of the H&E slide training image with a tile-level label.

[0028] In some examples, the method further includes performing, for each H&E slide training image, a tile selection process that infers a class status for each tile image in the H&E slide training image, and discarding tile images that do not correspond to a class of interest before performing the tile-based tissue classification analysis on each of the H&E slide training images based on the inferred class status, such that the tile-based tissue classification analysis is performed only on selected tile images of the H&E slide training images.

[0029] In some examples, each of the one or more trained deep learning multi-scale classifier models is configured as a tile-resolution fully convolutional network (FCN) classification model.

[0030] In some examples, identifying cells in a digital image tile using a trained cell segmentation model includes applying, using one or more processors, each of a plurality of tile images to the cell segmentation model and, for each tile, assigning a cell classification to one or more pixels in the tile image.

[0031] In some examples, assigning a cell classification to one or more pixels in the tile image includes using one or more processors to identify the one or more pixels as a cell interior, a cell boundary, or a cell exterior, and classifying the one or more pixels as a cell interior, a cell boundary, or a cell exterior.

[0032] In some examples, the trained cell segmentation model is a pixel-resolution, three-dimensional UNet classification model trained to classify cell interiors, cell boundaries, and cell exteriors.

[0033] In some examples, the one or more biomarkers are selected from the group consisting of tumor infiltrating lymphocytes (TIL), nuclear cytoplasmic ratio (NC), ploidy, signet ring morphology, and programmed death-ligand 1 (PD-L1).

[0034] In some examples, the deep learning framework includes a single-scale deep learning framework.

[0035] In some examples, separating the digital image into a plurality of tile images includes performing an image tiling process using one or more processors by applying the digital image to a trained multi-instance learning controller that separates the digital image into a plurality of tile images.

[0036] In some examples, the method further includes providing each tile image to a tile selection process that infers a class status for each tile image in the H&E slide training image, and selectively discarding tile images based on a tile selection criterion before applying the remaining plurality of tile images to the deep learning framework based on the inferred class status.

[0037] In some examples, the method further includes providing each tile image to a tile selection process that infers a class status for each tile image in the H&E slide training image, and randomly discarding tile images based on the inferred class status before applying the remaining plurality of tile images to the deep learning framework.

[0038] In some examples, the method further includes receiving a molecular training dataset of a plurality of training tissue samples, the molecular training dataset including RNA transcriptome counts from sequencing of substantially similar samples associated with each training tissue sample; performing a clustering process on the molecular training dataset to identify one or more molecular data subsets, each corresponding to a distinct biomarker; receiving, for each of the one or more molecular data subsets, a plurality of digital images of H&E stained training slides of the training tissue samples corresponding to the respective biomarkers for an image-based biomarker prediction system having one or more processors; and generating, using the one or more processors, one of the trained biomarker classification models based on the plurality of digital images of the H&E stained training slides for each of the one or more molecular data subsets.

[0039] In some examples, generating one of the trained biomarker classification models for each of the one or more molecular data subsets includes performing a multi-instance learning process on a plurality of digital images of H&E stained training slides.

[0040] In some examples, each of the multiple digital images of H&E stained training slides of training tissue samples has a slide-level label.

[0041] In some examples, each of the multiple digital images of H&E stained training slides of training tissue samples is unlabeled.

[0042] In some examples, the single-scale deep learning framework is a convolutional neural network having a ResNet configuration or an Inception-v3 configuration.

[0043] In some examples, the one or more biomarkers are selected from the group consisting of consensus molecular subtype (CMS) and homologous recombination deficiency ("HRD").

[0044] In some examples, the one or more processors are one or more graphics processing units (GPUs), tensor processing units (TPUs), and / or central processing units (CPUs).

[0045] In some examples, a computing device (e.g., an image-based biomarker prediction system) is communicatively coupled to a pathology slide scanner system via a communications network, such that the image-based biomarker prediction system receives digital images from the pathology slide scanner system via the communications network.

[0046] In some examples, the computing device is included within a pathology slide scanner system.

[0047] In some examples, the pathology slide scanner system includes an image-based, adversarially trained, and / or predictive model of microsatellite instability (MSI).

[0048] In some examples, generating a report including the digital image and the digital overlay includes generating the digital overlay to include an overlay element that identifies the tumor content of the digital image or the tumor percentage of the digital image.

[0049] According to another example, a computing device configured to identify biomarkers in a digital image of a hematoxylin and eosin (H&E) stained slide of a target tissue, the computing device including one or more memories and one or more processors configured to: receive the digital image; perform an image tiling process on the digital image by separating the digital image into a plurality of tile images, each of the plurality of tile images comprising a different portion of the digital image; apply the plurality of tile images to a multi-scale deep learning framework including one or more trained deep learning multi-scale classifier models, each trained to classify a different tissue classification for each tile image, and determine a tissue classification for each of the plurality of tile images using the multi-scale deep learning framework; identify cells in the digital image using the trained cell segmentation model; and identify a predicted presence of one or more biomarkers associated with the digital image from the tissue classification determined for each tile image and from the identified cells in the digital image.

[0050] According to another example, a computing device configured to identify biomarkers in digital images of hematoxylin and eosin (H&E) stained slides of a target tissue, the computing device comprising one or more memories and one or more processors, the computing device including: receiving a molecular training dataset of a plurality of training tissue samples, the molecular training dataset including RNA transcriptome counts from sequencing of substantially similar samples associated with each training tissue sample; performing a clustering process on the molecular training dataset to identify one or more molecular data subsets, each corresponding to a distinct biomarker; and clustering the one or more molecular data subsets. and one or more processors configured to: receive, for each of the one or more molecular data subsets, a plurality of digital images of H&E stained training slides of the training tissue sample corresponding to the respective biomarkers to an image-based biomarker prediction system having one or more processors; generate, for each of the one or more molecular data subsets, a trained image-based biomarker classifier model based on the plurality of digital images of the H&E stained training slides; receive subsequent digital images of H&E stained slides of the subsequent tissue sample; and apply the subsequent digital images to the trained image-based biomarker classifier model to identify a predicted presence of the one or more biomarkers in the subsequent tissue sample.

[0051] According to another example, a computing device configured to identify biomarkers in a digital image of a hematoxylin and eosin (H&E) stained slide of a target tissue, the computing device including one or more memories and one or more processors configured to: receive the digital image into an image-based biomarker prediction system having the one or more processors; separate the digital image into a plurality of tile images, each of the plurality of tile images comprising a different portion of the digital image; apply the plurality of tile images to a deep learning framework including one or more trained biomarker classification models, each of the plurality of tile images trained to classify a different tissue classification; predict a biomarker classification for each of the plurality of tile images using the one or more trained biomarker classification models; determine a predicted presence of one or more biomarkers in the target tissue from the predicted biomarker classification for each of the tile images; and generate a report including the digital image and a digital overlay visualizing the predicted presence of the one or more biomarkers. [Brief description of the drawings]

[0052] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the U.S. Patent and Trademark Office upon request and payment of the necessary fee.

[0053] The drawings described below depict various aspects of the systems and methods disclosed herein. It should be understood that each figure depicts an example of an aspect of the systems and methods. [Figure 1] FIG. 1 is a block diagram of a schematic diagram of a prediction system having an imaging-based biomarker prediction system, according to an example. [Diagram 2] FIG. 1 is a block diagram of a conventional pathologist's cancer diagnosis workflow process. [Diagram 3]FIG. 2 is a block diagram of a schematic diagram of a deep learning framework that may be implemented in the system of FIG. 1 , according to an example. [Figure 4] FIG. 1 is a block diagram of a schematic diagram of machine learning data flow, according to an example. [Diagram 5] FIG. 4 is a block diagram of a schematic diagram of a deep learning framework that may be implemented in the systems of FIGS. 1 and 3 to form multiple different marker classification models, according to one example. [Figure 6] FIG. 1 is a block diagram of a process for imaging-based biomarker prediction with an example of a multi-scale configuration. [Figure 7] FIG. 7 is a block diagram of an exemplary process for determining the status of a predicted biomarker, according to an exemplary embodiment of the process of FIG. 6. [Figure 8] FIG. 1 is a block diagram of a process for imaging-based biomarker prediction with an example of a single-scale configuration. [Figure 9] FIG. 4 is a block diagram of a process for generating a biomarker prediction report and overlay map that may be performed by the systems of FIGS. 1 and 3, according to an example. [Figure 10A] 10A-B show examples of overlay maps generated by the process of FIG. 9, showing a tissue overlay map (FIG. 10A) and a cell overlay map (FIG. 10B), according to one example. [Figure 10B] 10A-B show examples of overlay maps generated by the process of FIG. 9, showing a tissue overlay map (FIG. 10A) and a cell overlay map (FIG. 10B), according to one example. [Figure 11] FIG. 1 is a block diagram of a process for preparing digital images of histopathology slides for classification, according to one example. [Figure 12A] 1 illustrates an example of a neural network architecture that may be used for a classification model, according to an example. [Figure 12B] 1 illustrates an example of a neural network architecture that may be used for a classification model, according to an example. [Figure 12C]1 illustrates an example of a neural network architecture that may be used for a classification model, according to an example. [Figure 13] 1 illustrates a histopathology image showing tile images for classification, according to an example. [Figure 14] FIG. 13 is a block diagram of a schematic diagram of an imaging-based biomarker prediction system using separate pipelines, according to another example. [Figure 15A] FIG. 15 is a block diagram of a schematic diagram of an exemplary biomarker prediction process that may be implemented by the system of FIG. 14, according to one example. [Figure 15B] 15 is a schematic block diagram of an exemplary training process that may be implemented by the system of FIG. 14, according to one example. [Figure 16] 16A-16C show an example of a representative PD-L1 positive biomarker classification. FIG. 16A shows an input H&E image, FIG. 16B shows a probability map overlaid on the H&E image, and FIG. 16C shows PD-L1 IHC staining for reference. FIG. 16D-16F show an example of a representative PD-L1 negative biomarker classification. FIG. 16D shows an input H&E image, FIG. 16E shows a probability map overlaid on the H&E image, and FIG. 16F shows PD-L1 IHC staining for reference. The color bars indicate the predicted probability of tumor PD-L1+ class. [Figure 17] FIG. 16 is a block diagram of an exemplary multi-view strategy for PD-L1 classification, which may be performed by the process of FIGS. 14, 15A, and 15B, according to an example. [Figure 18] FIG. 1 is a block diagram of a schematic machine learning architecture capable of performing label-free annotation training in a deep learning framework and having a multi-instance learning controller, according to one example. [Figure 19] FIG. 20 is a block diagram of a framework operation that may be implemented by the multi-instance learning controller of FIG. 18, according to an example. [Figure 20]FIG. 20 is a block diagram of a framework operation that may be implemented by the multi-instance learning controller of FIG. 18, according to an example. [Figure 21] FIG. 20 is a block diagram of a framework operation that may be implemented by the multi-instance learning controller of FIG. 18, according to an example. [Figure 22] FIG. 20 is a block diagram of a framework operation that may be implemented by the multi-instance learning controller of FIG. 18, according to an example. [Diagram 23] 1 is an example of a resultant overlap map showing biomarker classification of CMS, according to one example. [Figure 24] FIG. 20 is a block diagram of another framework operation that may be implemented by the multi-instance learning controller of FIG. 18, according to another example. [Diagram 25] 13 is an example of overlap map results showing biomarker classification of CMS according to another example. [Figure 26] FIG. 20 is a block diagram of another framework operation that may be implemented by the multi-instance learning controller of FIG. 18, according to another example. [Figure 27] According to another example, an example of a neural network architecture that may be used for a classification model is shown. [Figure 28] FIG. 1 is a block diagram of a process for determining a list of possible corresponding therapies (eg, immunotherapy), according to an example. [Figure 29] FIG. 1 is a block diagram of data flow for generating a list of potential corresponding treatments, according to an example. [Diagram 30] FIG. 1 is a block diagram of a system for performing imaging-based biomarker prediction in conjunction with a pathology scanner system, according to an example. [Diagram 31] 1, 3, and 30 show various screenshots of an example generated graphic user interface display that may be generated by a system such as those of FIGS. [Diagram 32]1, 3, and 30 show various screenshots of an example generated graphic user interface display that may be generated by a system such as those of FIGS. [Diagram 33] 1, 3, and 30 show various screenshots of an example generated graphic user interface display that may be generated by a system such as those of FIGS. [Diagram 34] 1, 3, and 30 show various screenshots of an example generated graphic user interface display that may be generated by a system such as those of FIGS. [Diagram 35] 1, 3, and 30 show various screenshots of an example generated graphic user interface display that may be generated by a system such as those of FIGS. [Diagram 36] 1, 3, and 30 show various screenshots of an example generated graphic user interface display that may be generated by a system such as those of FIGS. [Figure 37] 1, 3, and 30 show various screenshots of an example generated graphic user interface display that may be generated by a system such as those of FIGS. [Figure 38] FIG. 1 is a block diagram of an exemplary computing device for use in implementing various systems herein, according to one example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0054] The imaging-based biomarker prediction system comprises a deep learning framework configured and trained to learn directly from pathology tissue slides and predict the presence of biomarkers in medical images. The deep learning framework can be configured and trained to analyze the medical images and identify biomarkers that indicate the presence of a tumor, the state / condition of the tumor, or information about the tumor in the tissue sample.

[0055] In an implementation, a cloud-based deep learning framework is used for medical image analysis. Deep learning algorithms automatically learn advanced imaging features to enhance diagnosis, prognosis, treatment indication, and treatment effect prediction. In an example, the deep learning framework can directly connect to cloud storage and leverage resources on the cloud platform for efficient deep learning algorithm training, comparison, and deployment.

[0056] In some examples, the deep learning framework includes multi-scale configurations that use tiling strategies to accurately capture structural and local tissue structures of various diseases (e.g., predicting cancer tumors). These multi-scale configurations perform classification of histopathology images (labeled or unlabeled) using a classifier trained to classify tiles of received histopathology images. In some examples, the multi-scale configurations include tile-level tissue classifiers, i.e., classifiers trained using tile-based deep learning training. In some examples, the multi-scale configurations include pixel-level cell classifiers and cell segmentation models. In some examples, classifications from the tile-level tissue classifier and pixel-level cell classifier are analyzed to predict the status of biomarkers in the histopathology images. In further examples, the multi-scale configurations include tile-level biomarker classifiers. Once tracked, the multi-scale classifiers can receive new labeled or unlabeled histopathology images and predict the presence of specific biomarkers in the associated histopathology slides.

[0057] In some examples, the deep learning framework herein includes a single-scale configuration trained using a multi-instance learning (MIL) strategy to predict the presence of biomarkers in histopathology images. A classifier trained using the single-scale configuration can be trained to perform classification of histopathology images (labeled or unlabeled) using a classifier trained using one or more multi-instance learning (MIL) techniques. In some examples, the single-scale configuration includes a slide-level classifier trained using gene sequencing data, such as RNA sequencing data, to analyze histopathology images with slide-level labels rather than tile-level labels. That is, a slide-level classifier is trained using RNA sequencing data to develop an image-based classifier that can predict the status of biomarkers in histopathology images.

[0058] Both the multi-scale and single-scale configurations herein can incorporate various algorithmic optimizations to accelerate computations for such disease analysis.

[0059] In a multi-scale classifier configuration implementation, the deep learning framework can be trained to include classifiers that perform automatic cell segmentation, determine cell / biomarker types, and determine tissue type classification from histopathology images, thereby providing image-based biomarker development. Even single-scale classifiers can be trained to include tissue type classification and biomarker classification.

[0060] In the case of a multi-scale classifier configuration, for example, aggregate and spatial imaging features for various cell types (e.g., tumor, stromal, lymphocyte) of a digital hematoxylin & eosin (H&E) slide can be determined by a deep learning framework and used to predict clinical and treatment outcomes. Instead of rudimentary manual cell type classification, examples herein use a multi-scale configuration in a deep learning framework to classify each sub-region of an H&E slide histopathology image into a specific cell segmentation, cell type, and tissue type. From there, biomarker detection is performed by another deep learning framework configured to identify various types of imaging metrics. Examples of imaging metrics include tumor shape, including tumor minimum and maximum shape, tumor area, tumor perimeter, tumor %, cell shape, including cell area, cell perimeter, cell convex area ratio, cell circularity, cell convex perimeter area, cell length, lymphocyte %, cell characteristics, cell texture, including saturation, intensity, and hue.

[0061] Examples of tissue classes include, but are not limited to, tumor, stromal, normal, lymphocyte, adipose, muscle, vascular, immune cluster, necrosis, hyperplasia / dysplasia, erythrocyte, and tissue classes or cell types that are positive (particularly containing an amount of the IHC stained target molecule greater than a certain threshold) or negative (not containing the molecule or containing an amount of the molecule less than a certain threshold) for an IHC stained target molecule.

[0062] In some instances, biomarker detection can be enhanced by combining imaging metrics with structured clinical and sequencing data to develop enhanced biomarkers.

[0063] Biomarkers can be identified by any of the following models. Any model referred to herein may be implemented as an artificial intelligence engine and may include a gradient boosting model, a random forest model, a neural network (NN), a regression model, a naive Bayes model, or a machine learning algorithm (MLA). The MLA or NN can be trained by a training dataset. In an exemplary predictive profile, the training dataset may include imaging, disease state, clinical, and / or molecular reports, and patient details (e.g., curated from EHR or gene sequencing reports). MLA includes supervised algorithms (e.g., algorithms where features / classifications in the dataset are annotated) using linear regression, logistic regression, decision trees, classification and regression trees, naive Bayes, nearest neighbor clustering; unsupervised algorithms (e.g., algorithms where features / classifications in the dataset are not annotated) using a priori, average clustering, principal component analysis, random forest, adaptive boosting; and semi-supervised algorithms (e.g., algorithms where features / classifications in the dataset are annotated) using generative approaches (e.g., mixtures of Gaussians, mixtures of multinomial distributions, hidden Markov models), sparse separation, graph-based approaches (e.g., min-cut, harmonic functions, manifold regularization), heuristic approaches, or support vector machines. NNs include conditional random fields, convolutional neural networks, attention-based neural networks, deep learning, long short-term memory networks, or other neural models, and the training dataset includes pathology reports covering multiple tumor samples, RNA expression data for each sample, and imaging data for each sample. Although MLA and neural networks identify different approaches to machine learning, these terms may be used interchangeably herein. Thus, unless otherwise specified, a reference to an MLA may include a corresponding NN, and a reference to a NN may include a corresponding MLA.Training may include providing an optimized dataset, labeling those characteristics found in patient records, and training the MLA to predict or classify based on new inputs. Artificial NNs are efficient computational models that have demonstrated strengths in solving difficult problems in artificial intelligence. Artificial NNs have been shown to be universal approximators (capable of representing a wide variety of functions given appropriate parameters). Some MLAs may identify important features and identify coefficients or weights for them. The coefficients may be multiplied by the frequency of occurrence of the features to generate a score, and if the score of one or more features exceeds a threshold, a particular classification may be predicted by the MLA. The coefficient schema may be combined with a rule-based schema to generate more complex predictions, such as predictions based on multiple features. For example, ten key features may be identified for various classifications. There may be a list of coefficients for the key features, and there may be a rule set for classification. The rule set may be based on the number of occurrences of the features, scaled weights of the features, or other qualitative and quantitative evaluation of the features coded in logic known to those skilled in the art. In other MLAs, the features may be organized in a binary tree structure. For example, a key feature that may distinguish most classifications may be present as the root of a binary tree and each subsequent branch in the tree until a classification is assigned based on reaching a terminal node of the tree. For example, a binary tree may have a root node that tests the first feature. There must be an occurrence or non-occurrence of this feature (a binary decision), and the logic can traverse the branch that is true for the item being classified. Additional rules may be based on thresholds, ranges, or other qualitative and quantitative tests. Supervised methods are useful when there are many known values ​​or annotations in the training dataset, but due to the nature of EMR / EHR documents, there may not be many annotations provided. When exploring large amounts of unlabeled data, unsupervised methods are useful for binning / bucketing instances in the dataset.As used herein, a single instance of the above model, or two or more such instances may be combined to constitute a model for purposes of a model, artificial intelligence, neural network, or machine learning algorithm.

[0064] In some examples, the technology provides machine learning assisted histopathology image review, including automatically identifying and outlining tumor regions and / or characteristics of regions or cell types within the regions (e.g., lymphocytes, PD-L1 positive cells, tumors with high tumor budding, etc.), counting cells within the tumor regions, and generating a decision score to improve the efficiency and objectivity of pathology slide review.

[0065] As used herein, the term "biomarker" refers to image-derived information related to the screening, diagnosis, prognosis, treatment, selection, disease monitoring, progression, and disease recurrence of cancer or other diseases, particularly information in the form of morphological features identifiable in histologically stained samples. Biomarkers herein may, in some instances, be morphological features determined from labeling-based images. Biomarkers herein may be morphological features determined from labeled RNA data.

[0066] A biomarker herein can be image-derived information that correlates with the presence of or susceptibility to cancer in a subject, the likelihood of the cancer being of one subtype or another, the presence or proportion of a biological characteristic such as a type or class of tissue, cell, or protein, the probability that a patient will respond or not respond to a particular treatment or class of treatment, the degree of positive response expected to a treatment or class of treatment (e.g., survival time and / or progression-free survival), whether a patient is responding to a treatment, or the likelihood that the cancer will regress, progress, or progress beyond the site of origin (i.e., metastasize).

[0067] Examples of biomarkers predicted from histopathological images using the various techniques herein include the following:

[0068] Tumor infiltrating lymphocytes (TILs) as used herein refer to mononuclear immune cells that infiltrate tumor tissue or stroma. TILs include, for example, T cells, B cells, and NK cells, and their populations can be subclassified based on function, activity, and / or biomarker expression. For example, TIL populations can include, for example, cytotoxic T cells expressing CD3 and / or CD8, and regulatory T cells (also known as suppressor T cells), which are often characterized by FOXP3 expression. Information regarding the density, location, organization, and composition of TILs provides valuable insights into prognosis and potential treatment options. In various embodiments, the present disclosure provides methods for predicting TIL density in a sample, distinguishing subpopulations of TILs in a sample (e.g., distinguishing cytotoxic T cells expressing CD3 / CD8 from FOXP3 Tregs), distinguishing between stromal and intratumoral TILs, and the like.

[0069] Programmed death-ligand 1 (PD-L1) is a 40 kDa type 1 transmembrane protein that affects immune system suppression, especially in patients with autoimmune diseases, cancer, and other pathologies. In the context of cancer immunotherapy, PD-L1 is expressed on the surface of tumor cells, tumor-associated macrophages (TAMs), and T lymphocytes, and can subsequently inhibit PD-1-positive T cells.

[0070] Polyploidy refers to the number of sets of homologous chromosomes in the genome of a cell or organism. Examples include haploid, meaning one set of chromosomes, and diploid, meaning two sets of chromosomes. The presence of more than one set of paired chromosomes in an organism's genome is described as "polyploidy." Three sets of chromosomes, 3n, are triploid, and four sets of chromosomes, 4n, are tetraploid. Extremely large numbers of sets can be designated by a number (e.g., 15 sets is 15ploid).

[0071] The nucleus-to-cytoplasm (NC) ratio is a measure of the ratio between the size of a cell's nucleus and the size of the cytoplasm of that cell. The NC ratio can be expressed as a volume ratio or cross-sectional area. The NC ratio can indicate the maturity of a cell, with the size of the cell nucleus decreasing with the maturity of the cell. In contrast, a high NC ratio of a cell may indicate malignancy of the cell.

[0072] The signet ring morphology is the morphology of signet ring cells, i.e. cells with large vacuoles, which appear in their malignant form mainly in cases of carcinoma. Signet ring cells are most commonly associated with gastric cancer, but can arise from a variety of tissues, including the prostate, bladder, gallbladder, breast, colon, ovarian stroma, and testis. For example, signet ring cell carcinoma (SRCC) is a rare form of highly malignant adenocarcinoma. It is an epithelial malignant tumor characterized by the histological appearance of signet ring cells.

[0073] These biomarkers, TIL, NC ratio, ploidy, signet ring morphology, and PD-L1, are examples of biomarkers of morphological features determined from labeling-based images by the technology herein.

[0074] Consensus molecular subtypes ("CMS") are a set of classification subtypes of colorectal cancer (CRC) developed based on comprehensive gene expression profile analysis. CMS classification in primary colorectal cancer includes CMS1-immune infiltration (often BRAFmut, MSI-High, TMB-High), CMS2-canonical (often ERBB / MYC / WNT driven), CMS3-metabolic (often KRASmut), and CMS4-mesenchymal (often TGF-B driven). More broadly, CMS herein includes these and other subtypes of colorectal cancer. Even more broadly, CMS herein refers to subtypes derived from comprehensive gene expression profile analysis of other cancer types described herein.

[0075] Homologous recombination deficiency ("HRD") status is a classification indicating a deficiency in the normal homologous recombination DNA damage repair process that results in loss of replication of a chromosomal region referred to as genomic loss of heterozygosity (LOH).

[0076] Biomarkers such as CMS and HRD are examples of morphological feature biomarkers determined from labeled RNA data by the techniques herein.

[0077] By way of example, biomarkers herein include HRD status, DNA ploidy score, karyotype, CMS score, chromosomal instability (CIN) status, signet ring morphology score, NC ratio, cellular pathway activation status, cell state, tumor signature, and splice variants.

[0078] As used herein, "histopathology image" refers to a digital (including digitized) image of microscopically, histopathologically developed tissue. Examples include images of histologically stained specimen tissue, where histological staining is a process performed in the preparation of sample tissue to aid in microscopic investigation. In some examples, the histopathology image is a digital image of a hematoxylin and eosin (H&E) stained histopathology slide, an immunohistochemistry (IHC) stained slide, a Romanowsky-Giemsa stained slide, a Gram stained slide, a Trichrome stained slide, a Carmine stained slide, and a silver nitrate stained slide. Other examples include blood smear slides and tumor smear slides. In other examples, the histopathology image is of other stained slides known in the art. As used herein, references to digital images, digitized images, slide images, and medical images refer to "histopathology image."

[0079] These histopathology images can be captured in the visible wavelength range and outside the visible range, such as infrared digital images obtained using spectroscopic examination of tissues developed histopathologically. In some examples, the histopathology images include z-stack images representing horizontal sections of a three-dimensional specimen or histopathology slide, captured at various levels of the specimen or various focal points of the slide. In some examples, the two or more images can be images from adjacent or near adjacent sections of tissue from the specimen, and one of the two or more images can have a tissue feature that corresponds to another of the two or more images. There can be a vertical and / or horizontal shift between the location of the corresponding tissue feature in the first image and the location of the corresponding tissue feature in the second image. Thus, a histopathology image also refers to an image, set of images, or a movie generated from multiple different images. It is understood that the following exemplary embodiments can be interchanged or trained on models with different styles of staining, unless expressly excluded.

[0080] Various examples herein are described with reference to a particular class of histopathological images, H&E slide images. Digital H&E slide images can be generated by capturing digital photographs of H&E slides. Alternatively, or in addition, such images can be generated from images derived from unstained tissues via machine learning systems, such as deep learning. For example, digital H&E slide images can be generated from wide-field autofluorescence images of unlabeled tissue sections. See, for example, Rivenson et al., "Virtual histological staining of unlabelled tissue-autofluorescence images via deep learning," Nature Biomedical Engineering, 3(6):466, 2019.

[0081] FIG. 1 illustrates a predictive system 100 that can analyze digital images of histopathology slides of a tissue sample and determine the likely presence of a biomarker in that tissue, where the presence of the biomarker indicates other information about the tumor in the tissue sample, such as a predicted presence of a tumor, a predicted tumor state / condition, or the likelihood of a clinical response to the use of a treatment associated with the biomarker.

[0082] System 100 includes an imaging-based biomarker prediction system 102 that implements, among other things, image processing operations, a deep learning framework, and report generation operations to analyze histopathological images of tissue samples and predict the presence of biomarkers in the tissue samples. In various examples, system 100 is configured to predict the presence of these biomarkers, tissue locations associated with these biomarkers, and / or cellular locations of these biomarkers.

[0083] The imaging-based biomarker prediction system 102 can be implemented on one or more computing devices, such as a computer, tablet, or other mobile computing device, or a server, such as a cloud server. The imaging-based biomarker prediction system 102 may include multiple processors, controllers, or other electronic components for processing or facilitating image capture, generation, or storage and image analysis, as well as deep learning tools for analysis of the images, as described herein. An exemplary computing device 3800 for implementing the imaging-based biomarker prediction system 102 is shown in FIG.

[0084] As shown in FIG. 1, the imaging-based biomarker prediction system 102 is connected to one or more medical data sources via a network 104. The network 104 can be a public network, such as the Internet, a private network, such as a research institute or a company's private network, or any combination thereof. The network can include a local area network (LAN), a wide area network (WAN), a cellular, satellite, or other network infrastructure, whether wireless or wired. The network 104 can be part of a cloud-based platform. The network 104 can utilize communication protocols, including packet-based and / or datagram-based protocols, such as Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), or other types of protocols. Additionally, the network 104 can include multiple devices that facilitate network communication and / or form the hardware infrastructure of the network, such as switches, routers, gateways, access points (such as wireless access points as shown), firewalls, base stations, repeaters, backbone devices, etc.

[0085] The imaging-based biomarker prediction system 102 is communicatively coupled via a network 104 to receive medical images, e.g., histopathology slides, such as digital H&E stained slide images, IHC stained slide images, or digital images of other staining protocols from a wide variety of different sources. These sources may include a physician's clinical record system 106 and a histopathology imaging system 108. The system 100 may be used to access any number of sources of medical image data. The histopathology images may be images captured by any suitable optical histopathology slide scanner, including any dedicated digital medical image scanner, e.g., 20x and 40x resolution magnification scanners. Additionally, the biomarker prediction system 102 may receive images from a histopathology image repository 110. In yet another example, images may be received from a partner genomic sequencing system 112, e.g., TCGA and NCI Genomic Data Commons. Additionally, the biomarker prediction system 102 may receive histopathology images from an organoid modeling lab 116. These image sources may communicate image data, genomic data, patient data, treatment data, historical data, etc., according to the techniques and processes described herein. Each of the image sources may represent multiple image sources. Additionally, each of these image sources may be considered a different data source, which may generate and provide different image data than other providers, hospitals, etc. The imaging data between the different sources may potentially differ in one or more respects, resulting in biases inherent to the various data sources, such as different dyes, biological sample fixation, embedding, staining protocols, and different pathology imaging equipment and settings.

[0086] In the example of FIG. 1, the imaging-based biomarker prediction system 102 includes an image pre-processing subsystem 114 that performs initial image processing to enhance image data to speed up processing in training a machine learning framework and perform biomarker prediction using a trained deep learning framework. In the illustrated example, the image pre-processing subsystem 114 performs a normalization process on the received image data, including one or more of color normalization 114a, intensity normalization 114b, and imaging source normalization 114c, to compensate and correct differences in the received image data. In some examples, the imaging-based biomarker prediction system 102 receives medical images, while in other examples, the subsystem 114 can generate medical images from either received histopathology slides or other received images, for example, by aligning shifted histopathology images to compensate for vertical / horizontal shifts to generate a composite histopathology image. This image pre-processing enables deep learning frameworks to more efficiently analyze images across large datasets (e.g., >1000, >10,000, >100,000, >1,000,000 medical images), thereby speeding up the training and analysis processes.

[0087] The image pre-processing subsystem 114 may perform further image processing to remove artifacts and other noise from the received images by performing preliminary tissue detection 114d, for example to identify regions of the images that correspond to histopathologically stained tissue for subsequent analysis, classification, and segmentation.

[0088] As further described herein, in a multi-scale configuration where image data is analyzed on a tile basis, in some examples, image pre-processing includes receiving an initial tissue pathology image at a first image resolution, downsampling the image to a second image resolution, and then performing normalization on the downsampled tissue pathology image, such as color and / or intensity normalization, to remove non-tissue objects from the image.

[0089] In contrast, the single-scale configuration does not use downsampling of the received histopathology images. The single-scale configuration analyzes the image data on a slide-level basis, rather than a tile-based basis.

[0090] In some further hybrid versions of each of the multi-scale and single-scale configurations, a tiling process is applied to the received histopathology images to generate tiles for tile-based analysis.

[0091] The imaging-based biomarker prediction system 102 may be a standalone system that interfaces with external (i.e., third-party) network-accessible systems 106, 108, 110, 112, and 116. In some examples, the imaging-based biomarker prediction system 102 may be integrated with one or more of these systems, including as part of a distributed cloud-based platform. For example, the system 102 may be integrated with a pathology tissue imaging system, such as a digital H&E stain imaging system, to enable, for example, rapid biomarker analysis and reporting at the imaging station. Indeed, any of the functionality described in the technology herein may be distributed across one or more network-accessible devices, including cloud-based devices.

[0092] In some examples, the imaging-based biomarker prediction system 102 is part of a comprehensive biomarker prediction, patient diagnosis, and patient treatment system. For example, the imaging-based biomarker prediction system 102 may be coupled to communicate predicted biomarker information, tumor prediction, and tumor status information to external systems, including a computer-based pathology lab / oncology system 118, which may receive the generated biomarker report including the image overlay mapping and use it to further diagnose the patient's cancer status and identify corresponding treatments for use in treating the patient. The imaging-based biomarker prediction system 102 may further transmit the generated report to the patient's primary care provider's computer system 120 and the physician's clinical record system 122 for databased patient reports using a database of previously generated reports for the patient and / or generated reports for other patients for use in future patient analyses (including deep learning analyses as described herein).

[0093] To analyze the received histopathological imaging data and other data, the imaging-based biomarker prediction system 102 includes a deep learning framework 150 that implements various machine learning techniques to generate a trained classifier model for image-based biomarker analysis from a received set of image data or a set of image data and other patient information. Using the trained classifier model, the deep learning framework 150 is further used to analyze and diagnose the presence of image-based biomarkers in subsequent images collected from the patient. In this manner, images and other data of previously treated and analyzed patients are utilized through the trained model to provide analysis and diagnosis capabilities for future patients.

[0094] In the exemplary system 100, the deep learning framework 150 includes a histopathology image-based classifier training module 160 that can access data received and stored from external systems 106, 108, 110, 112, and 116, as well as any other systems, where the data can be analyzed from the received data stream and databased into various data types. The various data types can be split into image data 162a, which can be associated with other data types, molecular data 162b, demographic data 162c, and tumor response data 162d. The associations can be formed by labeling the image data 162a with one or more different data types. By labeling the image data 162a according to the associations with other data types, the imaging-based biomarker prediction system can train the image classifier module to predict one or more different data types from the image data 162a.

[0095] In the illustrated data, the deep learning framework 150 includes image data 162a. For example, to train or use a multi-scale PD-L1 biomarker classifier, this image data 162a may include pre-processed image data received from subsystem 114, images from H&E slides, or images from IHC slides (with or without human annotations), including IHC slides targeting PD-L1, PTEN, EGFR, beta-catenin / catenin beta 1, NTRK, HRD, PIK3CA, and hormone receptors such as HER2, AR, ER, PR, etc. To train or use other biomarker classifiers, whether multi-scale or single-scale classifiers, the image data 162A may include images from other stained slides. Additionally, in the example of training a single-scale classifier, the image data 162A is image data related to RNA-seq data of a particular biomarker cluster enabling the multi-instance learning (MIL) techniques herein.

[0096] Molecular data 162b may include DNA sequences, RNA sequences, metabolomic data, proteomic / cytokine data, epigenomic data, organoid data, live karyotype data, transcriptional data, transcriptomics, metabolomics, microbiomics, and immunonomics, and may include identification of SNPs, MNPs, InDels, MSI, TMB, CNV fusions, loss of heterozygosity, loss or gain of function. Epigenomic data includes DNA methylation, histone modifications, or other factors that inactivate genes or cause alterations in gene function without changing the nucleotide sequence of the gene. Microbiomics includes data regarding viral infections that may affect the treatment and diagnosis of certain diseases, as well as data regarding bacteria present in the patient's gastrointestinal tract that may affect the effectiveness of medications taken by the patient. Proteomic data can include information about the composition, structure and activity of a protein, when and where it is expressed, its rates of production, degradation, and steady-state abundance, how it is modified (e.g., post-translational modifications such as phosphorylation), its trafficking between subcellular compartments, its participation in metabolic pathways, how it interacts with each other, or post-translational modifications of RNA to proteins, such as phosphorylation, ubiquitination, methylation, acetylation, glycosylation, oxidation, or nitrosylation.

[0097] The deep learning framework 150 may further include demographic data 162c and tumor response data 162d (including, for example, data regarding the reduction in tumor growth after being subjected to a particular therapy, such as immunotherapy, DNA damaging therapy such as PARP inhibitors or platinum, or HDAC inhibitors). The demographic data 162c may include age, sex, race, country of origin, etc. The tumor response data 162d may include epigenomic data, examples of which include changes in chromatin morphology and histone modifications.

[0098] Tumor response data 162d may include cellular pathways, examples of which include IFN gamma, EGFR, MAP kinase, mTOR, CYP, CIMP, and AKT pathways, as well as pathways downstream of HER2 and other hormone receptors. Tumor response data 162d may include cellular state indicators, examples of which include collagen composition, appearance, or refractive index (e.g., extracellular vs. fibroblastic, nodular fasciitis), stromal density or other stromal characteristics (e.g., stromal thickness, wet vs. dry), and / or general appearance of angiogenesis or vasculature (including distribution of vasculature in collagen / stroma, also referred to as epithelial-mesenchymal transition or EMT). Tumor response data 162d may include tumor characteristics, such as, for example, the presence of tumor budding or other morphological features / characteristics indicative of tumor complexity, tumor size (including bulky or low-grade tumor status), tumor aggressiveness (e.g., known as high-grade basaloid tumors, particularly in colorectal cancer, or high-grade dysplasia, particularly in Barrett's esophagus), and / or the immune status of the tumor (e.g., inflammatory / "hot" tumors versus non-inflammatory / "cold" tumors versus immune-excluded tumors).

[0099] The histopathology image-based classifier training module 160 may be configured with machine learning techniques adapted for image analysis, including, for example, deep learning techniques, including, for example, CNN models, more specifically, tile-resolution CNNs, which in some examples are implemented as FCN models, more specifically, tile-resolution FCN models. Any of the data types 162a-162d may be included within the histopathology image and obtained directly from data communicated to the imaging-based biomarker prediction system 102, such as data communicated with the histopathology image. The data types 162a-162d may be used by the histopathology image-based classifier training module 160 to develop a classifier for identifying one or more biomarkers discussed herein.

[0100] In one example, a histopathology image can be segmented and each segment of the image can be labeled according to one or more data types that can be classified into that segment. In another example, the histopathology image as a whole can be labeled according to one or more data types that can be classified into the image or at least one segment of the image. The data types can indicate one or more biomarkers, and by labeling the histopathology image or segments with the data types, biomarkers can be identified.

[0101] In the exemplary system 100, the deep learning framework 150 further includes a trained image classifier module 170, which may be configured with deep learning techniques, including those implementing the module 160. In some examples, the trained image classifier module 170 accesses the image data 162 for analysis and biomarker classification. In some examples, the module 170 further accesses the molecular data 162, the demographic data 162c, and / or the tumor response data 162d for analysis and tumor prediction, corresponding treatment prediction, and the like.

[0102] The trained image classifier module 170 includes a trained tissue classifier 172 that has been trained by module 160 using one or more training image sets to identify and classify tissue types in regions / areas of received image data. In some examples, these trained tissue classifiers are trained to identify biomarkers through tissue classification and include single-scale constituent classifiers 172a and multi-scale classifiers 172b.

[0103] Module 170 may further include other trained classifiers, including a trained cell classifier 174 that identifies biomarkers through cell classification. Module 170 may further include a cell segmenter 176 that identifies cells in the histopathology image, including cell boundaries, cell interiors, and cell exteriors.

[0104] In the examples herein, the tissue classifier 172 may include a biomarker classifier specifically trained to identify tumor infiltration (e.g., ratio of lymphocytes in tumor tissue to all cells in tumor tissue), PD-L1 (e.g., positive or negative status), ploidy (e.g., score), CMS (e.g., distinguishing subtypes), NC ratio (e.g., distinguishing nuclear size), signet ring morphology (e.g., signet cell classification and vacuole size), HRD (e.g., by score or positive or negative classification), etc., according to the biomarkers herein.

[0105] As described in detail herein, the trained image classifier module 170 and associated classifiers may be comprised of machine learning techniques adapted for image analysis, including, for example, deep learning techniques, including, by way of example, CNN models, more specifically, tiled resolution CNNs, which in some examples are implemented as FCN models, more specifically, tiled resolution FCN models, etc.

[0106] The system 102 further includes a tumor report generator 180 configured to receive classification data from the trained tissue (biomarker) classifier 172, the trained cell (biomarker) classifier 174, and the cell segmenter 172, determine tumor metrics of the image data, and generate digital image and statistical data reports, where such output data may be provided to the pathology laboratory 118, the primary care physician system 120, the genome sequencing system 112, a tumor board, a tumor board electronic software system, or other external computer system for display or consumption in further processes.

[0107] A conventional cancer diagnosis workflow 200 using histopathology images is shown in FIG. 2. A biopsy is performed to collect a tissue sample from a patient. In a medical laboratory, digital histopathology images of the tissue sample are generated (202) using known staining techniques, e.g., H&E or IHC staining, and a digital medical imager (e.g., a slide scanner). These histopathology images are provided to a pathologist who visually analyzes them and identifies tumors in the received images (204). The pathologist can optionally receive the patient's genomic sequencing data (e.g., DNA Seq data or RNA Seq data from a genomic sequencing lab) and analyze that data (206). The pathologist then diagnoses other characteristics of the type of cancer of the tumor / cancer cells from the visual analysis of the histopathology slide and any genomic sequencing data (208) and generates a pathology report (210).

[0108] 3 illustrates an exemplary implementation of the imaging-based biomarker prediction system 102, and more specifically, the deep learning framework 150 in the form of a deep learning framework 300. The framework 300 may be communicatively coupled to receive histopathology imaging data and other data (e.g., molecular data, tumor response data, demographic data, etc.) from external systems, such as a physician's clinical record system 106, a histopathology imaging system 108, a genome sequencing system 112, a medical image repository 110, and / or the organoid modeling lab 116 of FIG. 1, via the network 104. The organoid modeling lab 116 may collect various types of data, such as, for example, the sensitivity of the organoids to a drug (e.g., determined by measuring cell death or cell viability after exposure to a drug), single-cell analysis data, or detection of cellular products (including proteins, lipids, and other molecules) indicative of the presence of a particular cell population, including effector data, stimulatory data, regulatory data, inflammatory data, chemoattractant data, as well as organoid imaging data, any of which may be stored within the molecular data 162b.

[0109] The framework 300 includes a pre-processing controller 302, a deep learning framework cell segmentation module 304, a deep learning framework multi-scale classifier module 306, a deep learning framework single-scale classifier module 307, and a deep learning post-processing controller 308.

[0110] To prepare medical images for multi-scale and single-scale deep learning, in one example, the pre-processing controller 302 includes a normalization process 310, which may include color normalization, intensity normalization, and imaging source normalization. The normalization process 310 is optional and can be omitted to facilitate deep learning training, image analysis, and / or biomarker prediction.

[0111] The image discriminator 314 receives the histopathology image normalized by the normalization process 310 and examines the image including image metadata to determine the type of image. The image discriminator 314 can analyze the image data to determine whether the image is a training image, e.g., an image from a training dataset. The image discriminator 314 can analyze the image data to determine the type of labeling on the image, e.g., whether the image has tile level labeling, slide level labeling, or no labeling. The image discriminator 314 can analyze the image data to determine the slide stain used to generate the digital image, H&E, IHC, etc.

[0112] In response to examining this image data, the image discriminator 314 determines which images are provided to a slide-level label pipeline 313 to feed to a single-scale classifier module 307 of the deep learning framework, as well as which images are provided to a tile-level label pipeline 315 to feed to a multi-scale classifier 306 of the deep learning framework.

[0113] In the illustrated example, images with tile-level labeling in the pipeline 315 include a tissue detection process and an image tiling process. These processes may be performed on all image data received, only on training image data, only on image data received for analysis, or some combination thereof. In some examples, for example, the tissue detection process may be omitted to facilitate deep learning training, image analysis, and / or biomarker prediction. Indeed, any of the processes of the controller 302 may be performed in a dedicated biomarker prediction system or may be distributed for performance by an externally connected system. For example, the tissue imaging system may be configured to perform a normalization process before sending image data to the biomarker prediction system. In some examples, the biomarker prediction system may communicate an executable normalization software package to a connected external system, which configures their system to perform normalization or other pre-processing.

[0114] In examples where the image discriminator 314 sends unlabeled images to the pipeline 315, the pipeline 315 includes a multi-instance learning (MIL) controller, further described herein, that is configured to convert all or a portion of these histopathology images into tiled labeled images. The MIL controller can be configured to perform processes such as those described herein in FIGS.

[0115] To facilitate tissue detection of the trained tissue classifier, the tissue detection process of the pipeline 315 can perform an initial tissue identification to locate and segment tissue regions of interest for biomarker analysis. Such tissue identification of interest may include, for example, identifying tissue boundaries and segmenting the image into tissue and non-tissue regions, such that metadata identifying the tissue regions is stored with the image data to facilitate processing and prevent attempted biomarker analysis in non-tissue regions or regions that do not correspond to the tissue of interest.

[0116] To facilitate deep learning classification in various multi-scale configurations, the deep learning framework multi-scale classifier module 306 is configured to classify tissues using tiling analysis. For example, in the pipeline 315, the tissue detection process sends histopathological images (e.g., image data enriched with tissue detection metadata) to the image tiling process, which selects and applies a tiling mask to the received image to break the image into smaller sub-images for analysis by the framework module 306. The pipeline 315 can store multiple different tiling masks and select a tiling mask. In some examples, the image tiling process selects one or more tiling masks optimized for different biomarkers. That is, in some examples, the image tiling is biomarker specific. This allows for the use of tiles of different pixel sizes and different pixel shapes that are specifically selected to, for example, improve accuracy and / or reduce processing time associated with a particular biomarker. For example, a tile size optimized for identifying the presence of TILs in an image may be different than a tile size optimized for identifying PD-L1 or another biomarker. Thus, in some examples, the pre-processor controller 302 is configured to perform image processing and tiling specific to a type of biomarker, and after the system 300 analyzes the image data for that biomarker, the controller 302 can reprocess the original image data for analysis for the next biomarker until all biomarkers have been examined.

[0117] Generally speaking, the tiling mask applied by the image tiling process of the pipeline 315 may be selected to increase the efficiency of operation of the deep learning framework module 306. The tiling mask may be selected based on a size of the received image data, based on a configuration of the deep learning framework 306, based on a configuration of the framework module 304, or some combination thereof.

[0118] The tiling masks may have different sizes of tiling blocks. Some tiling masks have uniform (i.e., each of the same size) tiling blocks. Some tiling masks have different sizes of tiling blocks. The tiling mask applied by the image tiling process may be selected based on, for example, the number of classification layers in the deep learning framework 306. In some examples, the tiling mask may be selected based on the processor configuration of the biomarker prediction system, for example, if multiple parallel processors are available or if a graphical processing unit or a tensor processing unit is used.

[0119] In the illustrated example, the deep learning multi-scale classifier module 304 is configured to perform cell segmentation via the cell segmentation model 316. Here, the cell segmentation can be a pixel-level process of the histopathology image from the normalization process 310. In other examples, this pixel-level process may be performed on image tiles received from the pipeline 315. In some examples, the cell segmentation process of the framework 304 results in biomarker classification, since some of the biomarkers identified herein are determined from a cell-level analysis as opposed to a tissue-level analysis. These include, for example, signet rings, large nuclei, and high NC ratios. The module 304 can be configured using a CNN configuration, and in particular, an FCN configuration to implement each separate segmentation.

[0120] The deep learning framework multi-scale classifier module 306 includes a tissue segmentation model 318, a tissue classification model 320, and a biomarker classification model 320. Similar to module 304, module 306 can be configured using a CNN configuration, and in particular, an FCN configuration to implement each separate segmentation.

[0121] In one example, the cell segmentation model 316 of the module 304 may be configured as a three-class semantic segmentation FCN model developed by modifying a UNet classifier to replace the loss function with a cross-entropy function, a focal loss function, or a mean squared error function to form a three-class segmentation model. The three-class nature of the FCN model means that the cell segmentation model 316 may be configured as a first pixel-level FCN model that identifies and assigns each pixel of the image data to a cell subunit class, namely (i) cell interior, (ii) cell boundary, or (iii) cell exterior. This is provided as an example. The segmentation size of the module model 316 may be determined based on the type of cell being segmented. For example, for both TIL biomarkers, the model 316 may be configured to perform lymphocyte identification and segmentation using a three-class FCN model. For example, the cell segmentation model 316 may be configured to classify pixels in an image as corresponding to (i) the interior, (ii) the boundary, or (iii) the exterior of a lymphocyte cell. The cell segmentation model 316 can be configured to identify and segment any number of cells, examples of which include tumor positive, tumor negative, lymphocyte positive, lymphocyte negative, immune cells including lymphocytes, cytotoxic T cells, B cells, NK cells, macrophages, etc.

[0122] In some examples, module 304 receives tiled sub-images from pipeline 315, cell segmentation model 316 determines a list of all lymphocyte locations, and those locations are compared to a list of all cells of the other three class models determined from model 316 to eliminate falsely detected lymphocytes that are not cells. System 300 then takes the new list of confirmed lymphocyte locations from this module 304 and compares it to a list of tissues in tissue segmenter module 318, e.g., tumor and non-tumor tissue locations determined from tissue classification model 320, to determine whether the lymphocytes are in tumor or non-tumor regions.

[0123] The three-class model makes it easier to count individual cells and allows for more accurate classification, especially when two or more cells overlap each other. Tumor-infiltrating lymphocytes overlap with tumor cells. In a traditional two-class cell contour model, which labels only whether a pixel contains the outer edge of a cell, each clump of two or more overlapping cells is counted as one cell.

[0124] In addition to using a three-class model, the cell segmentation model 316 can be configured to avoid the possibility of a cell that spans two tiles being counted twice by adding a buffer around all four sides of each tile that is slightly wider than an average cell. The intention is to count only cells that appear in the central unbuffered region of each tile. In this case, the tiles are positioned such that the central unbuffered regions of adjacent tiles are adjacent and do not overlap. Adjacent tiles overlap in their respective buffer regions.

[0125] In one example, the cell segmentation algorithm of model 316 may be formed from two UNet models. One UNet model may be trained with images of mixed tissue classes where a human analyst has highlighted the outer edge of each cell and classified each cell according to its tissue class. In one example, the training data includes digital slide images where every pixel is labeled as either the interior of a cell, the outer edge of a cell, or background which is outside all cells. In another example, the training data includes digital slide images where every pixel is labeled with "yes" or "no" to indicate whether it shows the outer edge of a cell. This UNet model may recognize the outer edges of many types of cells and may classify each cell according to its cell shape or its location within the tissue class region assigned by tissue classification module 320.

[0126] Another UNet model may be trained on images of many cells of a single tissue class, or images of a diverse set of cells where only cells of one tissue class are outlined with a binary mask. In one example, the training set is labeled by associating a first value with all pixels that exhibit a cell type of interest and a second value with all other pixels. Visually, an image so labeled appears as a black and white image, where all pixels that exhibit the tissue class of interest are white and all other pixels are black, and vice versa. For example, an image may have only lymphocytes labeled. This UNet model can recognize the outer edge of that particular cell type and assign a label to cells of that type in the digital image of the slide.

[0127] The cell segmentation model 316 is a trained cell segmentation model that can be used for cell detection, although in some examples, the model 316 is configured as a biomarker detection model configured as a pixel-level classifier that classifies pixels as corresponding to a biomarker.

[0128] Turning to the multi-scale classifier module 306 of the deep learning framework, the tissue segmentation model 318 may be configured in a similar manner to the segmentation model 316, i.e., as a three-class semantic segmentation FCN model developed by modifying the UNet classifier by replacing the loss function with a cross-entropy function, a focal loss function, or a mean squared error function to form a three-class segmentation model. The model 318 may identify the interiors, exteriors, and boundaries of various tissue types within a tile.

[0129] The tissue classification model 320 is a tile-based classifier configured to classify tiles as corresponding to one of a number of different tissue classifications. Examples of tissue classes include, but are not limited to, tumor, stroma, normal, lymphocyte, fat, muscle, blood vessel, immune cluster, necrosis, hyperplasia / dysplasia, red blood cell, and tissue classes or cell types that are positive (particularly containing an amount of the IHC stained target molecule greater than a certain threshold) or negative (not containing the molecule or containing an amount of the molecule less than a certain threshold) for an IHC stained target molecule. Examples also include tumor positive, tumor negative, lymphocyte positive, and lymphocyte negative.

[0130] With the cell segmentation in the histopathology image generated by the cell segmentation model 316 and the tissue classification from the tissue classification model 302, the biomarker classification model 322 receives data from both and determines the presence of predictive biomarkers in the histopathology image, and in particular in a multi-scale configuration, the presence of predictive biomarkers in each tile image of the histopathology image. The biomarker classification model 322 may be a trained classifier implemented in the deep learning framework model 306 as shown, or implemented separately from the model 306, such as in the deep learning post-processing controller 308.

[0131] In some examples of biomarker classification models for detecting TIL biomarkers, the tissue classification model 320 is trained to identify the percentage of TILs within a tile image, the cell segmenter 316 determines cell boundaries, and the biomarker classification model 322 classifies the tile image based on the percentage of TILs within the cell interior, resulting in a classification of (i) tumor-IHC / lymphocyte positive or (ii) non-tumor-IHC / lymphocyte positive.

[0132] In some examples of biomarker classification models for detecting ploidy, the biomarker classification model 322 can be trained on a ploidy model based on histopathology images and associated ploidy scores, for example, using techniques provided in Coudray N, Ocampo PS, Sakellaropoulos T, Narula N, Snuderl M, Fenyo D, et al., "Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning," Nat Med. 2018;24:1559-67.

[0133] In one example, the training data may be data such as an actual karyotype determined by a cytogeneticist, but in some examples, the biomarker classification model 322 may be configured to infer such data. The ploidy data may be formatted in columns of chromosome number, start position, stop position, and length of the region. The ploidy score may be determined from DNA sequencing data and may be specific to a gene, chromosome, or chromosome arm. The ploidy score may be global and represent the entire genome of the sample (global CNVs / copy number variations may cause changes in hematoxylin staining of tumor nuclei), or may be a score calculated by averaging the ploidy scores of each region in the genome, and the local regional ploidy scores may be weighted according to the length of each region associated with that score. The trained ploidy model of the biomarker classification model 322 may be specific to a gene, chromosome arm, or entire chromosome. This is because each section may have a different impact on the cell morphology seen in the histopathology images. The predicted ploidy biomarker data may impact the accept / reject analysis, since even if the tumor purity or cell count on a slide is low, if the ploidy is higher than normal there may still be enough material remaining for genetic testing. The biomarker metric processor 326 may be configured to make such a determination prior to report generation.

[0134] In some examples of biomarker classification models to detect signet ring morphology, the biomarker classification model 322 may be trained with a signet ring morphology model based on classification techniques such as poorly cohesive (PC), signet ring cell (SRC), and Laurent's subclassifications, as well as Mariette, C., Carneiro, F., Grabsch, HI et al., "Consensus on the pathological definition and classification of poorly cohesive gastric carcinoma," Gastric Cancer 22, 1-9 (2019) and other signet ring morphology classifications.

[0135] In some examples of biomarker classification models to detect NC ratios, the cell segmentation model 316 may be configured with a three-class UNet as described herein, which is trained to identify three classes: nucleus, cytoplasm, and cell boundary / non-cellular background. In one example, the training data may be images in which each pixel has been manually annotated with one of these three classes and / or may be images so annotated by a trained model, as illustrated in the example of updated training images in FIG.

[0136] Thus, the cell segmentation model 316 can be trained to analyze the input image and assign one of three classes to each pixel, defining a cell as the group of all cytoplasmic pixels between an adjacent nuclear pixel and the next closest border pixel. The biomarker classification model 322 can then be configured to calculate, for each cell, a nuclear:cytoplasmic ratio as the area (in pixels) of the cell's nucleus divided by the area (in pixels) of the entire cell (nucleus and cytoplasm).

[0137] To identify tumor tissue and tumor status of the tissue, the deep learning framework 306 can be configured, in one example, using an FCN classifier. In one example, the deep learning framework 304 can be configured as a pixel-resolution FCN classifier, while the deep learning framework 306 can be configured as a tile-resolution FCN classification model, or a tile-resolution CNN model, i.e., a model that performs classification on an entire received tile of image data.

[0138] The classification model 320 of the module 306 can be configured to classify tissues in the tile as corresponding to one of a number of tissue classes, such as, for example, biomarker status, tumor status, tissue type, and / or tumor state / condition, or other information. In the illustrated example, the module 306 is configured with a tissue classification 320 and a tissue segmentation model 322. In an exemplary implementation of a TIL biomarker, the tissue classification model 320 can classify tissues using tissue classifications such as tumor-IHC positive, tumor-IHC negative, necrotic, stromal, epithelial, or blood. The tissue segmentation model 328 identifies boundaries of the different tissue types identified by the tissue classification model 320 and generates metadata for use by the post-processing controller 308 to visually display the boundaries and color coding of the different tissue types in an overlay mapping report generator.

[0139] In an example implementation, the deep learning framework 300 performs tile-based classification by receiving tiles (i.e., sub-images) from the classification models 320 and 322 processes. In some examples, the tiling may be performed by the framework 306 using a tiling mask, and the module 306 itself may send the generated sub-images to the module 304 for pixel-level segmentation in addition to performing the tissue classification. The module 306 may examine each tile sequentially one after the other, or the module 306 may examine each tile in parallel for faster processing of the image due to the nature of the matrix generated by the FCN model.

[0140] In some examples, the tissue segmentation model 318 receives pixel-resolution cell segmentation data and / or pixel-resolution biomarker segmentation data from module 304 and performs statistical analysis on a tile-by-tile basis. In some examples, the statistical analysis determines (i) the area of ​​the image data covered by tissue, e.g., the area of ​​a stained histopathology slide covered by tissue, and (ii) the number of cells in the image data, e.g., the number of cells in a stained histopathology slide. For example, the tissue segmentation model 318 can accumulate cell and tissue classifications for each tile of the image until all tiles forming the image have been classified.

[0141] If the deep learning framework is a multi-scale classifier module for classifying biomarkers on a tile-based basis, the deep learning framework 300 is further configured to classify biomarkers using classifications trained from slide-level training images without the need for tile-level labeling. For example, as discussed further below, the slide-level training images received by the image discriminator may be provided to a slide-level label piper 313 having a MIL controller configured to perform processes such as those described in Figures 18-26 herein to generate a plurality of tile images with inferred classifications and optionally perform tile selection on the tiles to train the tissue classification model 317 and the biomarker classification model 319. Examples of single-scale classifiers include a CMS biomarker classification model whose output is a CMS class, and an HRD biomarker classification model whose output is HRD+ or HRD-. These classifications may be performed on the entire histopathology image to determine biomarker predictions, or may be performed on each tile image of the digital image and analyzed by a biomarker metric processor 326 to determine biomarker predictions from the tile images.

[0142] In some examples of biomarker classification models to detect HRD, the biomarker classification model 319 can be configured to predict HRD. Training of the HRD model in the classifier 318 can be based on histopathology images and matched HRD scores. For example, training data can be generated by H&E images and RNA-seq data, which in some examples includes RNA expression profile data, which is fed to the HRD model and fed back for further training, similar to the updated training data 403 in FIG. 4. The training data can be derived from tumor organoids, with H&E images of the organoids paired with measurements of the sensitivity of the organoids to PARP inhibitors indicative of HRD, or results of the HRD model run by the RNA team based on the RNA expression profile of the organoids. Exemplary HRD prediction models of biomarker classification model 319 are described in Peng, Guang et al., "Genome-wide transcriptome profiling of homologous recombination DNA repair," Nature communications vol. 5 (2014): 3361 and van Laar, RK, Ma, X.-J., de Jong, D., Wehkamp, ​​D., Floore, AN, Warmoes, MO, Simon, I., Wang, W., Erlander, M., van't Veer, LJ and Glas, AM (2009), "Implementation of a novel microarray-based diagnostic test for cancer of unknown primary," Int. J. Cancer, 125: 1390-1397.

[0143] In one example, a deep learning framework can identify HRD from H&E slides using RNA expression and identify slide-level labels indicating the percentage of the slide containing biomarker-expressing cells. In one example, an RNA label activation map approach can be applied to the entire slide as a binary label (i.e., positive or negative HRD expression anywhere in the tissue) or as a continuous percentage (i.e., 62% of the cells in the image are found to express HRD). Binary RNA labels can be generated by next generation sequencing of the specimen, and cell percentage labels can be generated by applying single-cell RNA sequencing. In one example, single-cell sequencing can identify cell types and amounts present in RNA expression from NGS.

[0144] Training of the tile-based deep learning network to predict biomarker classification labels for each tile of a whole slide image can be performed using any of the methods described herein. Once training is complete, the model can be applied to each tile using a method of activation mapping. Activation mapping can be performed using Grad-CAM (Gradient Class Activation Mapping) or guided backpropagation. Both can identify which regions of the tile contribute most to the classification. In one example, the portion of the tile that contributes most to the HRD positive class can be the cells clustered in the upper right corner of the tile. Cells within the identified active regions can then be labeled as HRD positive cells.

[0145] Proving that a model works reliably clinically may include comparing the model results to a source of ground truth. One possible method of generating ground truth involves isolating small regions of tissue, each containing less than 100 cells, by segmenting through tissue microarrays and sequencing each region individually to obtain RNA labels for each region. A method of generating ground truth may further include classifying these regions using a biomarker classification model to determine the accuracy with which the activation map highlights cells in regions of high HRD expression and ignores most cells in regions of low HRD expression.

[0146] When training a tile-based deep learning network to predict the biomarker classification label for each tile, a strong supervised approach is utilized to generate biomarker labels and identify the HRD status (positive or negative) of individual cells. Single-cell RNA sequencing can be used alone or in combination with laser-guided microdissection to extract one cell at a time and create a label for each cell. In one example, a cell segmentation model can be incorporated to first obtain the cell outline and then an artificial intelligence engine can be incorporated to classify the pixel values ​​within the outline of each of the cells according to the biomarker status. In another example, a mask of the image can be generated, with HRD-positive cells assigned a first value and HRD-negative cells assigned a second value. The slide with the mask can then be used to train a single-scale deep learning framework to identify cells expressing HRD.

[0147] In some examples of biomarker classification models to detect CMS, the biomarker classification model 319 can be configured to predict CMS. Such biomarker classifications can be configured to classify segmented cells as corresponding to cancer-specific classifications. For example, four trained CMS classifications in primary colorectal cancer include: 1-Immune Infiltration (often BRAFmut, MSI-High, TMB-High), 2-Canonical (often ERBB / MYC / WNT driven), 3-Metabolic (often KRASmut), and 4-Mesenchymal (often TGF-B driven). In other examples, more trained CMS classifications can be used, but generally, two or more CMS subtypes are classified in the examples herein. Additionally, other cancer types may have their own trained CMS categories, and the classifier 318 can be configured to have models for subtyping each cancer type. Examples of techniques for developing CMS callers for 4, 5, 6, 7 or more CMS callers are described in Eide, PW, Bruun, J., Lothe, RA et al., "CMScaller: an R package for consensus molecular subtyping of colorectal cancer pre-clinical models," Sci Rep 7, 16618 (2017) and https: / / github.com / peterawe / CMScaller.

[0148] Training of the CMS model in the classifier 318 may be based on histopathology images matching CMS category assignments. CMS category assignments may be based on RNA expression profiles, and in one example are generated by an R program using closest template predictions called CMS Caller (see Eide, PW, Bruun, J., Lothe, RA et al., "CMScaller: an R package for consensus molecular subtyping of colorectal cancer pre-clinical models," Sci Rep 7, 16618 (2017) and https: / / github.com / peterawe / CMScaller). An alternative classification using a random forest model is described in Guinney, J., Dienstmann, R., Wang, X. et al., "The consensus molecular subtypes of colorectal cancer," Nat Med 21, 1350-1356 (2015). For example, a CMS caller looks at each RNA-seq data sample and determines whether each gene is above or below the average, resulting in a binary classification of each gene. This avoids batch effects between, for example, different RNA-seq datasets. Training data may also include DNA data, IHC data, mucin markers, and treatment response / survival data from clinical reports. These may or may not be associated with the CMS category assignment. For example, CMS4 IHC may stain positive for TGFbeta, CMS1 IHC may be positive for CD3 / CD8, CMS2 and 3 have alterations in mucin genes, CMS2 responds to cetuximab, and CMS1 responds well to Avastin. CMS1 has the best survival prognosis and CMS4 has the worst prognosis. (See slide 12 in the CMS slides). CMS categories 1 and 4 can be detected from H&E. By training, for example, a model can be trained using the architecture in Figure 4 to identify and classify the difference between CMS2 and 3.

[0149] In one example, a biomarker classification model 319 can be configured to identify five CRC intrinsic subtypes (CRIS) with unique molecular, functional and phenotypic specificities: (i) CRIS-A: mucinous, glycolytic, enriched microsatellite instability or KRAS mutation; (ii) CRIS-B: TGF-β pathway activity, epithelial-mesenchymal transition, poor prognosis; (iii) CRIS-C: elevated EGFR signaling, sensitivity to EGFR inhibitors; (iv) CRIS-D: WNT activation, IGF2 gene overexpression and amplification; and (v) CRIS-E: Paneth cell-like phenotype, TP53 mutation. Although the CRIS subtypes successfully classify an independent set of primary and metastatic CRCs, there is limited overlap with existing transcriptional classes and the predictive and prognostic performance has been unprecedented. See, for example, Isella, C., Brundu, F., Bellomo, S. et al., "Selective analysis of cancer-cell intrinsic transcriptional traits defines novel clinically relevant subtypes of colorectal cancer," Nat Commun 8, 15107 (2017).

[0150] For biomarker detection, the biomarker classification model 319 may be trained with a CMS model that predicts for each tile, a CMS classification that identifies different tissue types (e.g., stroma) for that classification, instead of attempting to simply average CMS classification across all tiles. In one example, each tile is processed and the CMS model generates a compressed representation using the pixel data associated with each tile, and each tile is assigned to a class (cluster 1, cluster 2, etc.) based on the pattern of pixel data for each tile and the similarity between tiles. A list of the percentage of tiles in the image that belong to each cluster is the cluster profile of the image, and may be provided by a report generator. In one example, each profile is fed to the model along with the corresponding CMS designation or RNA expression profile (which is the original method used to define CMS categories) for training. In another example, each tile of all training slide images is annotated according to the overall CMS category assigned to the entire slide from which the tile originated, and the tiles are clustered and analyzed to determine the cluster most closely associated with a CMS category.

[0151] In some examples, instead of performing the same classification on each tile and weighting the tile classifications equally, the biomarker classification model 319 (and biomarker classification model 322) can cluster each tile into a discrete number of clusters. One way to achieve this is to include an attention layer in the biomarker classification model. In one example, every tile in every training slide can be classified into a cluster, and then if a number of tiles in a cluster are not statistically associated with a biomarker, that cluster is not weighted as highly as a cluster that is associated with a biomarker. In other examples, a majority voting technique can be used to train the biomarker classification model 319 (or model 322).

[0152] Although shown as separate models, biomarker classification models 322 and 319 may each be configured to include all or a portion of a corresponding tissue classification model, cell segmentation model, and tissue segmentation model, as in the case of classifying various biomarkers herein. Additionally, while biomarker classification model 322 is shown to be included within multi-scale classifier module 306 and biomarker classification model 319 is shown to be included within single-scale classifier module 307, in some examples, all or a portion of these biomarker classification models may be implemented in post-processing controller 308, as in the case of classifying various biomarkers herein. Additionally, while described as tile-level or slide-level classification models, in some examples, biomarker classification models 322 and 319 may be configured as pixel-level classifiers, as in some examples.

[0153] From the decisions made by modules 304, 306, and 307, post-processing controller 308 can determine whether the image data contains an amount of tissue that exceeds a threshold and / or meets a criteria, e.g., whether the image data contains an amount of tissue sufficient for genetic analysis, to use the image data as a training image in the learning phase of a deep learning framework, or to combine the image data with an existing trained classifier model.

[0154] Thus, in various examples herein, including those described with reference to FIG. 3 and elsewhere herein, a patient report may be generated that may be presented to a patient, physician, medical professional, or researcher as a digital copy (e.g., a JSON object, a PDF file, or an image on a website or portal), a hard copy (e.g., a printout on paper or another tangible medium), audio (e.g., recorded or streamed), or in another format.

[0155] The report may include information related to the gene expression call (e.g., over- or under-expression of a particular gene), the genetic variants detected, other characteristics of the patient's sample, and / or clinical records. The report may further include clinical trials for which the patient may be eligible based on the genetic variants detected, other characteristics of the sample, and / or clinical records, treatments for which the patient may be eligible, and / or predicted side effects if the patient receives a particular treatment.

[0156] The results contained in the report and / or additional results (e.g., from a bioinformatics pipeline) can be used to analyze a database of clinical data to determine, among other things, whether there is a trend indicating that the treatment slows the progression of cancer in other patients with the same or similar outcome as the specimen. The results can also be used to design tumor organoid experiments. For example, organoids can be genetically engineered to have the same characteristics as the specimen and observed after being subjected to a treatment to determine whether the treatment can slow the growth rate of the organoids and therefore the growth rate of the patient associated with the specimen.

[0157] In one example, the post-processing controller 308 is further configured to determine a plurality of different biomarker predictive metrics and a plurality of tumor predictive metrics, for example, using the biomarker metric processing module 326. Example predictive metrics include tumor purity, number of tiles classified as a particular tissue class, number of cells, number of tumor infiltrating lymphocytes, clustering of cell types or tissue classes, density of cell types or tissue classes, tumor cell features (circularity, length, nuclear density), stromal thickness surrounding the tumor tissue, image pixel data statistics, predicted patient survival, PD-L1 status, MSI, TMB, tumor origin, and immunotherapy / treatment response.

[0158] For example, the biomarker metric processing module 326 can determine, for each tissue class, the number of tiles classified into one or more single tissue classes, the percentage of tiles classified into each tissue class, for any two classes, the ratio of the number of tiles classified into the first tissue class to the number of tiles classified into the second tissue class, and / or the total area of ​​tiles classified into a single tissue class. The module 326 can determine tumor purity based on the number of tiles classified as tumor relative to other tissue classes, or based on the number of cells located in tumor tiles relative to the number of cells located in other tissue class tiles. The module 326 can determine cell counts for the entire tissue pathology image, within an area predefined by the user, within tiles classified as any of the tissue classes, within a single grid tile, or across an area or region of interest, whether predefined, selected by the user during operation of the system 300, or automatically selected by the system 300, for example, by determining the most likely region of interest based on image analysis. The module 326 can determine the clustering of cell types of tissue classes based on the spacing and density of classified cells, the spacing and distance of tiles classified into tissue classes, or any visually detectable feature. In some examples, module 326 determines the probability that two adjacent cells are, for example, two immune cells, two tumor cells, or one of each. Module 326 determines tumor cell characteristics by determining the average circularity, perimeter, and / or nuclear density of the identified tumor cells. The thickness of the identified stroma can be used as a predictor of patient response to treatment. Image pixel data statistics determined by module 326 may include the mean, standard deviation, and sum of each tile of either a single image or a collection of images of any pixel data including red, green, blue (RGB) values, optical density, hue, saturation, grayscale, and stain deconvolution.Additionally, module 326 may calculate the location of lines, patterns of alternating brightness, contours of shapes, segmented tissue classes and / or staining patterns of segmented cells within the image. In any of these examples, module 326 may be configured to create a determined / predicted status, and then overlay display generation module 324 generates a report to display the determined information. For example, overlay map generation module 324 may generate a network-accessible user interface that allows a user to select different types of data to be displayed. Module 324 generates an overlay map showing the selected different types of data overlaid on a rendition of the original stained image data.

[0159] FIG. 4 shows a machine learning data input / flow schematic 400 that may be implemented in the system 300 of FIG. 3, or more generally, in any of the systems and processes described herein.

[0160] In a training mode in which the deep learning framework of the system 300 is trained, various training data can be acquired. In the illustrated example, training image data 401 in the form of high-resolution and low-resolution histopathology images is provided to the pre-processing controller 302. As shown, the training images can include annotated tissue image data from various tissue types, e.g., tumor, stroma, normal, immune cluster, necrosis, hyperplasia / dysplasia, and red blood cells. As shown, the training images can include computer-generated synthetic image data, as well as image data of segmented cells (cell image data) and image data of biomarkers (e.g., biomarkers discussed herein) labeled with slide-level labels or tile-level labels (collectively, biomarker-labeled image data). These training images can be digitally annotated, although in some examples, the tissue annotation is done manually. In some images, the training image data includes molecular data and / or demographic data, e.g., as metadata within the image data. In the illustrated example, such data is separately fed to a deep learning framework 402 (consisting of exemplary implementations of a multi-scale deep learning framework 306' and a single-scale deep learning framework 307'). Other training data, such as pathway activation scores, may also be provided to the controller 302 for additional training of the deep learning framework.

[0161] In some examples, the deep learning framework 402 generates updated training images 403, which are annotated and segmented by the deep learning framework 402 and fed back to the framework 402 (or the pre-processing controller 302) for use in updated training of the framework 402.

[0162] In the diagnostic mode, patient image data 405 is provided to the controller 302 and used in accordance with the examples herein.

[0163] Any of the image data herein, including patient image data and training images, may be histopathology image data, such as H&E slide images and / or IHC slide images, for example, in the case of IHC training images, the images may be segmented images that distinguish between cytotoxic T cells and regulatory T cells, or other cell types.

[0164] In some examples, the controller 302 generates image tiles 407 and accesses one or more tiling masks 409, and tile metadata 411, which are supplied as inputs to a deep learning framework 402 for the controller 302 to determine predicted biomarkers and / or tumor status and metrics, which are then provided to an overlay report generator 404 for generating a biomarker and tumor report 406. Optionally, the report 406 may include an overlay of the histopathology image, and may further include biomarker scoring data, such as percentage TILs, in one example.

[0165] In some examples, clinical data 413 is provided to the deep learning framework 402 for use in analyzing the image data. The clinical data 413 may include health records, biopsy tissue type, and anatomical location of the biopsy. In some examples, tumor response data 415 collected from the patient following treatment is additionally provided to the deep learning framework 402 to determine biomarker status, tumor status, and / or changes in those metrics.

[0166] FIG. 5 illustrates an exemplary deep learning framework 500 formed from multiple different biomarker classification models. The elements of FIG. 5 are provided as follows: "Cell" refers to a cell segmentation model, e.g., a trained pixel-level segmentation model, according to examples herein. "Multi" refers to a multi-scale (tile-based) tissue classification model, according to examples herein. "Post" refers to arithmetic calculations that may be performed in a final stage of a biomarker classification model configured to predict a biomarker status of an image or tile image, according to examples herein, in response to one or more data from the "Cell" or "Multi" stages. In one example, "Post" may include majority voting, such as identifying summing the number of tiles in an image associated with each biomarker label and assigning the biomarker status in the image to the biomarker label with the largest sum. A two-tier "Post" configuration refers to a two-stage post-processing configuration, where arithmetic calculations may be stacked. In one example, a first tier of Post may include summing the tissue and cells in a tile labeled and summing lymphocyte cells in the same tile. The second layer can divide the lymphocyte cell count by the cell count to generate a ratio, which can be used to assign the status of the biomarker in the image based on whether the ratio exceeds the threshold when compared to the threshold. The final "post" configuration can include other post-processing functions such as a report generation process as described herein. "Single" refers to a single-scale classification model, according to examples herein. "MIL" refers to a MIL controller, according to examples herein. In the illustrated example, the deep learning framework 500 includes a TIL biomarker classification model 502, a PD-L1 classification model 504, a first CMS classification model 506 based on a "single" classification architecture, and a second CMS classification model 508 based on a "multiple" classification architecture, as well as an HRD classification model 510. Patient data 512, such as molecular data, demographic data, tumor response data, and patient images 514, are stored in a dataset accessible by the deep learning framework 500.

[0167] The training data is also shown in the form of cell segmentation training data 516 , single-scale classification biomarker training data 518 , multi-scale classification biomarker training data 520 , MIL training data 522 , and post-processing training data 524 .

[0168] FIG. 6 illustrates a process 600 that may be performed by the imaging-based biomarker prediction system 102, the deep learning framework 300, or the deep learning framework 402, particularly in a deep learning framework having a multi-scale configuration.

[0169] As part of the training process, in block 602, tile-labeled histopathology images are received at the deep learning framework 300. Herein, the histopathology images can be of any type, but in this example are shown as digital H&E slide images. These images can be training images of previously determined and labeled (and therefore known) cancer types (e.g., in a supervised learning configuration). In some examples, the images can be training images of multiple different cancer types. In some examples, the images can be training images that include some or all of the images of unknown or unlabeled cancer types (e.g., in an unsupervised learning configuration). In some examples, the training images include digital H&E slide images annotated with tissue classes (to train a tile resolution FCN tissue classifier) ​​and other digital H&E slide images annotated with each cell. In an example of training for TIL biomarker classification, each lymphocyte can be annotated with an H&E slide image (e.g., to train a pixel resolution FCN segmentation classifier annotated images to train a UNet model classifier). In some examples, the training images may be digital IHC stained images for training a pixel-resolution FCN segmentation classifier, particularly images where the IHC staining targets lymphocyte markers. In some examples, the training images include images combined with molecular data, clinical data, or other annotations (such as pathway activation scores).

[0170] In the illustrated example, pre-processing is performed on the training images, such as a normalization process described herein, at block 604. Other pre-processing processes described herein may also be performed at block 604.

[0171] In block 606, the tile-labeled H&E slide training images are provided to a deep learning framework and analyzed within a machine learning construct such as a CNN, more specifically a tile resolution CNN implemented as an FCN model, to analyze tile images of training images for tissue classification training, pixels of training images for cell segmentation training, and in some cases tile images for biomarker classification training. As a result, in block 608, a trained deep learning framework multi-scale biomarker classification model is generated, which may include a cell segmentation model and a tissue classification model. In training multiple biomarker classification models, in block 608, a separate model can be generated for each of the biomarkers TIL, PD-L1, ploidy, NC ratio, and signet ring morphology.

[0172] As a prediction process, in block 610, a new unlabeled histopathology image, such as an H&E slide image, is received and provided to a multi-scale biomarker classification model, and in block 612, the status of the biomarkers in the received histopathology image is predicted as determined by the one or more biomarker classification models.

[0173] For example, in block 610, a new (unlabeled or labeled) histopathology image may be received from a physical clinical record system or primary care system and applied to the trained deep learning framework applying its trained cell segmentation, tissue classification model, and biomarker classification model, and in block 612, a biomarker prediction score is determined. The prediction score may be determined for the entire histopathology image or various regions of the entire image. For example, for each image, block 612 may generate an absolute number of biomarkers on the image, a percentage of the number of cells in the tumor region associated with each of the biomarkers, and / or a designation of the biomarker classification or other information. In some examples, the deep learning framework may identify predicted biomarkers for all tissue classes identified in the image. As such, the biomarker prediction probability score may vary across images. For example, in predicting the presence of TILs, process 612 may predict the presence of TILs at different locations in the histopathology image. As a result, the prediction of TILs varies across images. This is provided as an example, and in block 612, any number of metrics discussed herein may be determined.

[0174] As shown in process 900 of Figure 9, in block 902, after the prediction is made, the predicted biomarker classification may be received at a report generator. In block 904, a tissue pathology image and therefore a clinical report for the patient may be generated, which includes the status of the predicted biomarkers. In block 906, an overlay map showing the predicted biomarker status may be generated, which may be provided to a pathologist for display to a clinician or to determine a preferred immunotherapy corresponding to the predicted biomarkers.

[0175] 7, an exemplary process 700 for determining predicted biomarker status, and in particular for predicting the status of TILs, is provided. Process 700 can nevertheless be used to predict the status of any number of biomarkers and other metrics, in accordance with the examples described herein.

[0176] The pre-processing controller receives the histopathology image and performs initial image processing (702) as described herein. In one example, the deep learning pre-processing controller receives the entire image file in any pyramidal TIFF format and identifies edges and contours of viable tissue in the image (e.g., performs segmentation). The output of block 702 can be, for example, a binary mask of the input image, with each pixel having a value of 0 or 1, where 0 indicates background and 1 indicates foreground / tissue. The dimensions of the mask can be the dimensions of the input slide when downsampled by 128 times. This binary mask can be temporarily buffered and provided to a tiling process 704.

[0177] In process 704, the pre-processing controller applies a tissue mask process using a tiling procedure to divide the image into sub-images (i.e., tiles) that are examined separately. Since the deep learning framework is configured to run two different learning models, one for tissue classification and the other for cell / lymphocyte segmentation, a different tiling procedure for each model can be run in step 704. Process 704 can generate two outputs, each including, for example, a list of coordinates defined from the top left corner of the tile. The output lists can be temporarily buffered and passed to the tissue classification and cell segmentation processes.

[0178] In the example of FIG. 7, tissue classification is performed in process 706, which receives the pathology tissue images from process 704 and performs tissue classification on each received tile using a trained tissue classification model. The trained tissue classification model is configured to classify each tile into a different tissue class (e.g., tumor, stroma, normal epithelium, etc.). To reduce computational redundancy, multiple layers of tilings can be used by process 706. For each tile, the trained tissue classification model calculates a class probability for each class stored in the model. Process 706 then determines the most likely class and assigns that class to the tile. Process 706 can output a single list of lists as a result. Each nested inner list acts as a nested classification, which describes a single tile and includes the location of the tile, the probability that the tile is each class included in the model, and the ID of the most likely class. This information is listed for each tile. The single list of lists can be saved to an output json file of the pipeline of the deep learning framework.

[0179] In processes 708 and 710, cell segmentation and lymphocyte segmentation are performed, respectively. In processes 708 and 710, the histopathology images and tile list are received from processes 704 and 706. In process 708, a trained cell segmentation model is applied. In process 710, a trained lymphocyte segmentation model is applied. That is, in the illustrated example, two pixel resolution models are run in parallel for each tile in the cell segmentation tile list. In one example, the two models both use UNet architecture, but are trained on different training data. The cell segmentation model identifies cells and draws a border around all cells in the received tile. The lymphocyte segmentation model identifies lymphocytes and draws a border around all lymphocytes in the tile. Because hematoxylin binds to DNA, performing "cell segmentation" using digital H&E slide images is sometimes referred to as nuclear segmentation. That is, in the cell segmentation model process 708, nucleus segmentation is performed for all cells, and in the lymphocyte segmentation model process 710, nucleus segmentation is performed for lymphocytes.

[0180] In this example, since the same UNet architecture is used for both, processes 708 and 710 each generate two identically formatted mask array outputs. Each output is a mask array of the same shape and size as the received tile. Each array element is either 0, 1, or 2, where 0 indicates a pixel / location predicted as background (i.e., outside the object), 1 indicates a pixel / location predicted as the object boundary, and 2 indicates a pixel / location predicted as the object interior. In the case of the cell segmentation model output, the object refers to a cell. In the case of the lymphocyte segmentation model, the object refers to a lymphocyte. These mask array outputs may be temporarily buffered and provided to processes 712 and 714, respectively.

[0181] Processes 712 and 714 receive the output mask array of the cell segmentation (UNet) model and the output mask array of the lymphocyte segmentation (UNet) model, respectively. Processes 712 and 714 are performed for each received tile and are used to express the information in the mask array in coordinates in the coordinate space of the original whole slide image.

[0182] In one example, process 712 may access a stored image processing library and use that library to find contours around the cell interior class, i.e., contours that correspond to locations that have a value of 2 in each mask. In this way, process 712 may perform a cell registration process. In the cell boundary class (indicated by locations that have a value of 1 in each mask), separation between adjacent cell interiors is guaranteed. This generates a list of all contours for each mask. Then, by treating each contour as a filled polygon, process 712 determines the coordinates of the contour's centroid (center of mass), from which process 712 generates a centroid list. Next, each coordinate in the contour list and centroid list is shifted to generate output that is in a coordinate space defined by the entire received image, rather than a coordinate space specific to a single tile in the image. Without this shift, each coordinate would be in the coordinate space of the image tile that contains it. The value of each shift is the same as the coordinate of the top left corner of the parent tile of the received image. In this example, process 714 performs the same process as process 712 for the lymphocyte class.

[0183] Processes 712 and 714 each generate a contour list output and a centroid list output corresponding to a respective UNet segmentation model. A contour is a set of coordinates that, when connected, outlines the detected object. Each contour can be represented as a line of text consisting of the constituent coordinates printed in order as number pairs. Each contour list from processes 712 and 714 can be saved as a text file consisting of many such lines. A centroid list is a list of number pairs. Each of these outputs can be temporarily buffered and provided to process 716.

[0184] Process 716 receives the tissue classification output (a single list of multiple lists) from process 706, the cell centroid and contour list from process 712, and the lymphocyte centroid and contour list from process 714, and performs cell segmentation integration.

[0185] For example, process 716 can combine the paired outputs of processes 712 and 714 to generate a single concise list containing the most important information about the cells. In one example, there are two main components of process 716.

[0186] In the first component of process 716, the information found in the cell segmentation model and the lymphocyte segmentation model are combined. Before the information is combined, it exists as a list of cell outlines and a list of lymphocyte outlines, which are not necessarily a subset of cell outlines since lymphocyte outlines are the output of two independent models (712 and 714). Thus, it is desirable to make lymphocytes a subset of cells since (1) lymphocytes are a type of cell, so if an object is not a cell, it cannot biologically be a lymphocyte, and (2) it is desired to report percentages of the dataset that have the same denominator. Thus, the cell segmentation integration process 716 can be performed by comparing the location of each cell to the locations of all lymphocytes (in one example, this can only be performed for objects in a single tile, so that the number of comparisons is not excessive). In one example, a cell is considered a lymphocyte only if it is "close enough" to a lymphocyte. The definition of "close enough" can be established by empirically determining the median radius of objects detected by the lymphocyte segmentation model across a set of training histopathology images. Note that this updated set of training images (e.g., 403 in FIG. 4) differs from the training set of images used to train the model because this set of training histopathology images are annotated images generated by the model itself, resulting in orders of magnitude more images, e.g., millions of automatically annotated images, that form the new or updated training set. In fact, the training set of the model may continue to grow with new received medical images that meet the accept / reject criteria. This may be the case for tissue classification models, as well as cell segmentation and lymphocyte segmentation models. Generating a new training set from the model and evaluating subsequent images allows the model to (1) use the median of millions of cells, not just those outlined in the training tiles, and (2) compare the actual size of the detected objects, not the size of the annotations drawn by humans.Lymphocyte nuclei are typically spherical, so in one example, these objects were all modeled as circles (since they are two-dimensional slices of a sphere). The radii of these circles were calculated and the median was used to determine the typical size for lymphocyte detections. As a result, the final cell list will be exactly the same as the objects detected by the cell segmentation model, but the goal of the lymphocyte segmentation model is to provide a Boolean true / false label for each cell in that list.

[0187] In the second component of process 716, each cell is binned into one of the tissue classification tiles (from process 706) based on its location. Note that in the example described here, the size of the cell segmentation tile may differ from the tissue classification tile due to the different architecture of the model. Nevertheless, process 716 is configured to determine the parent tile of each cell based on the location of the centroid, having the coordinates of each cell centroid, the coordinates of the top left corner of each tissue classification tile, and the size of each tissue classification tile.

[0188] Process 716 generates an output, which is a list of lists. Each nested inner list serves as a nested classification describing a single cell, including the coordinates of the cell's centroid, the tile number of the parent tile, the tissue class of the parent tile, and whether the cell is classified as a lymphocyte. This information is listed for each cell, and the output list is saved to the output json file of the deep learning framework pipeline.

[0189] In process 718, which may be implemented by a post-processing controller such as the biomarker metric processing module 326, in this particular example, any of a number of different biomarker metrics of the predicted TIL status and other TIL metrics are determined, as described.

[0190] For example, process 718 can be configured to perform a tissue area calculation based on the tissue mask used in process 704 to determine the area covered by tissue. In some examples, the tissue mask is a Boolean array that takes a value of 1 where tissue is present and 0 elsewhere, so process 718 can count the number of 1s to give a measurement of the tissue area. This value is the number of square pixels at 128x downsampling. Multiplying this by 16384 (i.e., for a 128*128 tissue mask) gives the number of square pixels at the native resolution (referred to as "x"). The native resolution of the image indicates the number of pixels per micron, and taking this number squared gives the number of square pixels per square micron (referred to as "y"). Dividing the number of square pixels at the native resolution by this resolution scaling factor (or x / y using the variables defined above) gives the number of square microns covered by tissue, and thus the tissue area is calculated. That is, process 718 can generate a floating point number in [0,∞] representing the tissue area in square microns, which can be used in the accept / reject model process described below.

[0191] As an example of other biomarker statistics, process 718 can be further configured to perform a total nuclei calculation using the cell segmentation integrated output from process 716. For example, the total number of nuclei on a slide is determined as the number of entries in the cell segmentation integrated output. Process 718 can also perform a % tumor nuclei calculation based on the output from this process 716. The total number of tumor nuclei on an image is the number of entries in the cell segmentation integrated output that meet the following requirements: (i) the tissue class of the parent tile is tumor, and (ii) the cells are not classified as lymphocytes.

[0192] In addition to determining biomarker statistics, process 718 can be further configured to perform an acceptance / rejection process based on the determined tissue area, total nuclei number, and tumor nuclei number. In one example, process 718 can be configured with a logistic regression model, where these three variables are used as inputs and the output of the model is a binary recommendation on whether the slide should be accepted or rejected for molecular sequencing. The logistic regression model can be trained on a training set of images using these derived variables. For example, the training images can be formed of histopathology images that have been previously sent for sequencing and accepted, and histopathology images that have been rejected during routine pathology review. Alternatively, there can be a pre-set threshold, for example, 20% of the nuclei on a slide are tumor, or a minimum number of tumor cells are required. In some examples, the model can take into account the DNA ploidy of the tumor cells (data from karyotyping or DNA sequence information) and calculate an adjusted estimate of available genetic material by dividing the number of tumor nuclei by the average copy number of chromosomes detected in each tumor nucleus by the normally expected copy number of 2. In some examples, the logistic regression model can be configured to have three possible outputs (instead of two) by adding an uncertainty zone between accept and reject that recommends manual review. For example, the penultimate output of the logistic regression model is a real number, and the final step of the model thresholds this number to 0 to generate a binary classification. Instead, in some examples, the uncertainty zone is defined as a range of numbers including 0, where values ​​higher than this range correspond to "reject," values ​​within the range correspond to manual review, and values ​​lower than this range correspond to "accept." Process 718 can be configured to calculate the size of this uncertainty zone by performing a cross-validation experiment. For example, process 718 can perform a training process that is repeated many times, but in each iteration, a different random subset of images in the training set is used.This produces many similar but not identical final models, and process 718 can use this variation to determine the uncertainty range of the final logistic regression model. Thus, process 718 can produce a binary accept / reject output in some cases and an accept / reject / manual review output in some cases.

[0193] Using the recommendations from process 718, a decision is made. For example, the deep learning output post-processing controller can generate a report of images marked "accept" and automatically send those images to the genome sequencing system (112) for molecular sequencing, while images with a "reject" recommendation are rejected and not sent for molecular sequencing. If a "manual review" option is configured and recommended, the images can be sent to a pathologist or team of pathologists (118) to review the slide and determine whether to send for molecular sequencing or reject it.

[0194] FIG. 8 shows an example process 800 that may be performed by the imaging-based biomarker prediction system 102, the deep learning framework 300, or the deep learning framework 402, particularly in a deep learning framework having a single-scale configuration.

[0195] In process 802, molecular training data is received at the imaging-based biomarker prediction system. This molecular training data is for a plurality of patients and can be obtained from a gene expression dataset, such as from sources described herein. In some examples, the molecular training data includes RNA-seq data. In block 804, the molecular training data is labeled with biomarkers. One form of biomarker clustering includes labeling that can be performed by taking an existing label associated with the specimen, such as a tumor subtype, and associating that label with the molecular training data. Alternatively, or in addition, labeling can be performed by clustering, such as using an automated clustering algorithm. One exemplary algorithm in the case of CMS subtype biomarkers is an algorithm for identifying CMS subtypes in the molecular training data and cluster training data according to the CMS subtype. This automated clustering can be performed, for example, within a single class classifier module of a deep learning framework or within a slide-level label pipeline, such as the one in the deep learning framework 300. In some examples, the molecular training data received in block 802 is RNA-seq data, for example, generated using an RNA wet lab and processed using a bioinformatics pipeline.

[0196] In various embodiments, for example, each transcriptome data set can be generated by processing patient or tumor organoid samples through RNA whole exome next generation sequencing (NGS) to generate RNA sequencing data, and the RNA sequence data can be processed by bioinformatics pipeline to generate the RNA-seq expression profile of each sample.Patient samples can be tissue samples or blood samples that contain cancer cells.

[0197] RNA can be isolated from blood samples or tissue sections using commercially available reagents, such as proteinase K, TURBO DNase-I, and / or RNA Clean XP beads. The isolated RNA can be subjected to quality control protocols to determine the concentration and / or quantity of RNA molecules, including using fluorescent dyes and fluorescence microplate readers, standard spectrofluorometers, or filter fluorometers.

[0198] A cDNA library can be prepared from the isolated RNA, purified, and selected for cDNA molecule size selection using commercially available reagents, e.g., Roche KAPA Hyperbeads. Preparation of the cDNA library can include reverse transcription. In another example, a New England Biolab (NEB) kit can be used. Preparation of the cDNA library can include ligation of adapters to the cDNA molecules. For example, UDI adapters, including Roche SeqCap dual-end adapters, or UMI adapters (e.g., full-length or partial (stubby) Y-shaped adapters) can be ligated to the cDNA molecules. In this example, the adapters are nucleic acid molecules that can function as barcodes to identify cDNA molecules according to the sample they originate from and / or to facilitate downstream bioinformatics processing and / or next-generation sequencing reactions. The sequence of nucleotides within the adapters can be unique to the sample to distinguish the sequencing data obtained for different samples. The adaptors may facilitate binding of the cDNA molecules to anchor oligonucleotide molecules on the sequencer flow cell and may act as seeds for the sequencing process by providing a starting point for the sequencing reaction.

[0199] The cDNA library can be amplified and purified using reagents such as Axygen MAG PCR clean-up beads. Amplification can include polymerase chain reaction (PCR) techniques different from quantitative or reverse transcription quantitative PCR (qPCR or RT-qPCR). The concentration and / or amount of cDNA molecules can then be quantified using fluorescent dyes and fluorescent microplate readers, standard spectrofluorometers, or filter fluorometers.

[0200] Before drying in a vacuum, the cDNA libraries can be pooled and treated with reagents to reduce off-target capture, such as Human COT-1 and / or IDT xGen Universal Blockers. The pools can then be resuspended in a hybridization mix, such as IDT xGen Lockdown, and probes can be added to each pool, such as IDT xGen Exome Research Panel v1.0 probes, IDT xGen Exome Research Panel v2.0 probes, other IDT probe panels, Roche probe panels, or other probes. The pools can be incubated in an incubator, PCR machine, water bath, or other temperature-regulating device to allow the probes to hybridize. The pools can then be mixed with streptavidin-coated beads or another means to capture the hybridized cDNA probe molecules, particularly cDNA molecules representing exons of the human genome. In another embodiment, polyA capture can be used. The pool can be amplified and purified again using commercially available reagents, such as the KAPA HiFi Library Amplification kit and Axygen MAG PCR clean-up beads, respectively.

[0201] The cDNA library may be analyzed to determine the concentration or amount of cDNA molecules, for example, by using fluorescent dyes (e.g., PicoGreen pool quantification) and a fluorescent microplate reader, a standard spectrofluorometer, or a filter fluorometer. The cDNA library may also be analyzed to determine the fragment size of the cDNA molecules. This can be done via gel electrophoresis techniques and may include using a device such as the LabChip GX Touch. The pools can be cluster amplified using a kit (e.g., Illumina Paired-End Cluster Kit with PhiX Spike). In one example, the cDNA library preparation and / or whole exome capture steps may be performed in an automated system using a liquid handling robot (e.g., SciClone NGSx).

[0202] Library amplification can be performed on a device, e.g., an Illumina C-Bot2, and the resulting flow cells containing the amplified target capture cDNA libraries can be sequenced on a next-generation sequencer, e.g., an Illumina HiSeq4000 or Illumina NovaSeq6000, to a unique on-target depth selected by the user, e.g., 300x, 400x, 500x, 10,000x. The next-generation sequencer can generate FASTQ, BCL, or other files for each patient sample or each flow cell.

[0203] When two or more patient samples are processed simultaneously on the same sequencer flow cell, the reads from multiple patient samples are initially contained in the same BCL file and then split into separate FASTQ files for each patient. If the adapter sequences used for each patient sample are different, they can act as barcodes to facilitate associating each read with the correct patient sample and placing it in the correct FASTQ file.

[0204] Each FASTQ file may contain paired-end or single-read reads, and may be short or long reads. Here, each read represents one detected nucleotide sequence within an mRNA molecule isolated from a patient sample, which is inferred by using a sequencer to detect the sequence of nucleotides contained in a cDNA molecule generated from the isolated mRNA molecule during library preparation. Each read in a FASTQ file is also associated with a quality assessment. The quality assessment may reflect the likelihood that an error during the sequencing procedure affected the associated read.

[0205] Each FASTQ file can be processed by a bioinformatics pipeline. In various embodiments, the bioinformatics pipeline can filter the FASTQ data. Filtering the FASTQ data can include correcting sequencer errors and removing (trimming) low-quality sequences or bases, adapter sequences, contamination, chimeric reads, over-represented sequences, bias caused by library preparation, amplification, or capture, and other errors. Entire reads, individual nucleotides, or multiple nucleotides that may be erroneous may be discarded based on a quality assessment associated with the reads in the FASTQ file, the known error rate of the sequencer, and / or a comparison of each nucleotide in the read to one or more nucleotides in other reads aligned to the same position in the reference genome. Filtering can be performed in part or in whole by various software tools. FASTQ files can be analyzed by sequencing data QC software, such as AfterQC, Kraken, RNA-SeQC, FastQC (BaseSpace Labs by Illumina or see https: / / www.illumina.com / products / by-type / informatics-products / basespace-sequence-hub / apps / fastqc.html), or other similar software programs for quality control and rapid assessment of reads. In the case of paired-end reads, the reads can be merged.

[0206] For each FASTQ file, each read in the file can be aligned to the location in the reference genome that has a sequence that most closely matches the sequence of the nucleotides in the read. Many software programs exist that are designed to align reads, such as Bowtie, Burrows Wheeler Aligner (BWA), and programs that use the Smith-Waterman algorithm. The alignment can be directed using a reference genome by comparing the nucleotide sequence of each read to a portion of the nucleotide sequence of a reference genome (e.g., GRCh38, hg38, GRCh37, other reference genomes developed by the Genome Reference Consortium, etc.) to determine the portion of the reference genome sequence that most likely corresponds to the sequence of the read. The alignment may take into account RNA splice sites. The alignment can generate a SAM file that stores the start and end positions of each read in the reference genome, as well as the coverage (number of reads) of each nucleotide in the reference genome. SAM files can be converted to BAM files, BAM files can be sorted, and duplicate reads can be marked for deletion.

[0207] In one example, kallisto software can be used for alignment and quantification of RNA reads (see Nicolas L Bray, Harold Pimentel, Pall Melsted and Lior Pachter, "Near-optimal probabilistic RNA-seq quantification" Nature Biotechnology 34, 525-527 (2016), doi:10.1038 / nbt.3519). In another embodiment, quantification of RNA reads can be performed using another software, for example, Sailfish or Salmon (see Rob Patro, Stephen M. Mount, and Carl Kingsford (2014) "Sailfish enables alignment-free isoform quantification from RNA-seq reads using lightweight algorithms" Nature Biotechnology (doi:10.1038 / nbt.2862) or Patro, R., Duggal, G., Love, MI, Irizarry, RA, & Kingsford, C. (2017) "Salmon provides fast and bias-aware quantification of transcript expression" Nature Methods). These RNA-seq quantification methods may not require alignment. There are many software packages available for normalization, quantitative analysis, and differential expression analysis of RNA-seq data.

[0208] For each gene, the raw RNA read count for that particular gene can be calculated. The raw read counts can be stored in a tabular file for each sample, where a column represents a gene and each entry represents the raw RNA read count for that gene. In one example, the kallisto alignment software calculates the raw RNA read count for each read as the sum of the probabilities that the read aligns to the gene. Thus, in this example, the raw counts are not integers.

[0209] The raw RNA read counts can then be normalized to correct for GC content and gene length, for example using full quantile normalization, and the sequencing depth can be adjusted using a size factor method. In one example, normalization of the RNA read counts is performed according to the methods disclosed in U.S. Patent Application No. 16 / 581,706, entitled "Methods of Normalizing and Correcting RNA Expression Data," filed September 24, 2019, or PCT19 / 52801, which are incorporated herein by reference in their entirety. The rationale for normalization is that the copy number of each cDNA molecule in the sequencer may not reflect the distribution of mRNA molecules in a patient sample. For example, during library preparation, amplification, and capture steps, certain portions of mRNA molecules may be over- or under-represented due to artifacts occurring in various aspects of priming reverse transcription caused by random hexamers, amplification (PCR enrichment), rRNA depletion, and probe binding and errors generated during sequencing that may be due to GC content, read length, gene length, and other characteristics of the sequence of each nucleic acid molecule. Each raw RNA read count for each gene can be adjusted to eliminate or reduce over- or under-representation caused by biases or artifacts in the NGS sequencing protocol. The normalized RNA read counts can be saved in a tabular file for each sample, with columns representing genes and each entry representing the normalized RNA read count for that gene.

[0210] The set of transcriptome values ​​may refer to either normalized RNA read counts or raw RNA read counts, as described above.

[0211] Returning to Figure 8, in block 804, the molecular training data (e.g., such RNA sequence data) is labeled with biomarkers and clustered using an automated clustering algorithm, such as an algorithm that identifies CMS subtypes in the molecular training data and an algorithm that identifies cluster training data according to CMS subtype. This automated clustering can be performed, for example, within a single class classifier module of a deep learning framework or within a slide-level labeling pipeline such as that in deep learning framework 300.

[0212] In block 806, for each biomarker cluster (each corresponding to a different biomarker, such as a different CMS subtype or HRD), histopathology images from relevant patients are obtained. These histopathology images can be, for example, H&E slide images with slide-level labels. In block 808, for each biomarker cluster, these labeled histopathology images are provided to a deep learning framework for training a biomarker classification model, such as a multiple CMS classification model for predicting different CMS subtypes. As a result, in block 810, a set of trained biomarker classifiers (classification models) is generated. Thus, blocks 802-810 represent a training process.

[0213] The prediction process begins at block 812, where a new (unlabeled or labeled) tissue pathology image, such as an H&E slide image, is received and provided to the single-scale biomarker classifier generated by block 810, and in block 814, a biomarker classification on the received tissue pathology image is predicted as determined by one or more biomarker classification models, such as one or more CMS subtypes or HRD.

[0214] Similar to block 610, in block 814, new histopathology images may be received from a physical clinical record system or primary care system, and the tissue classification model and / or biomarker classification model may be applied to the trained deep learning framework to determine biomarker predictions. The prediction scores may be determined, for example, for the entire histopathology image.

[0215] 9, similar to process 600, after prediction, the predicted biomarker classification from block 814 may be received in block 902. In block 904, a tissue pathology image and therefore a clinical report for the patient may be generated, which includes the predicted biomarker status. In block 906, an overlay map showing the predicted biomarker status may be generated, which may be provided to a pathologist for display to a clinician or to determine a preferred immunotherapy corresponding to the predicted biomarkers.

[0216] 10A and 10B, for example, show examples of digital overlay maps created by the overlay map generator 324 of the system 300. These overlay maps can be generated as static digital reports displayed to a clinician or as dynamic reports that allow for user interaction via a graphical user interface (GUI). FIG. 10A shows a tissue class overlay map generated by the overlay map generator 324. FIG. 10B shows a cell perimeter overlay map generated by the overlay map generator 324.

[0217] In one example, the overlay map generator 324 can display the digital overlay as a transparent or opaque layer over the histopathology image, aligned such that the image location shown in the overlay and the histopathology image are in the same location on the display. The transparency of the overlay map can vary. The transparency can be adjustable by the user in a dynamic reporting mode of the overlay map generator 324. The overlay map generator 326 can report the percentage of labeled tiles associated with each tissue class label, the proportion of the number of tiles classified into each tissue class, the total area of ​​all grid tiles classified into a single tissue class, and the proportion of the area of ​​tiles classified into each tissue class. The overlay map can be displayed as a heat map with different pixel intensity levels showing different tissue classifications and corresponding to different biomarker status levels. For example, in the TIL example, high intensity pixels are shown in tissue regions with high predicted TIL status (high %) and low intensity pixels in tissue regions with low predicted TIL status (low %).

[0218] In one example, the deep learning output post-processing controller 308 can also report the total number of cells or the percentage of cells that are in an area defined by the user, either the entire slide, a single grid tile, all grid tiles classified as each tissue class, or cells classified as immune cells. The controller 308 can also report the number of cells classified as lymphocyte cells that are located within an area classified as tumor or any other tissue class.

[0219] In one example, the digital overlays and reports generated by the controller 308 can be used to help medical professionals more accurately estimate tumor purity and locate or diagnose areas of interest, including invasive tumors that have tumor cells protruding into non-tumor tissue areas surrounding the tumor. They can also assist medical professionals in prescribing treatments. For example, the number of lymphocytes in an area classified as tumor can predict whether immunotherapy will be successful in treating a patient's cancer.

[0220] In one example, the digital overlay and report generated by the controller 308 can be used to determine whether a slide sample has sufficient high quality tissue for successful genetic sequence analysis of the tissue, e.g., to implement an accept / reject / manual decision as discussed in process 700. Genetic sequence analysis of tissue on a slide can be successful if the slide contains a certain amount of tissue and / or has a tumor purity value that exceeds a user-defined tissue amount and tumor purity threshold. The controller 308 can use process 700 to label slides as accepted or rejected for sequence analysis depending on the amount of tissue present on the slide and the tumor purity of the tissue on the slide. The controller 308 can also label slides as uncertain using a user-defined tissue amount threshold and a user-defined uncertainty range obtained from a user having interaction with the digital overlay and report from the generator 324, similarly according to process 700.

[0221] In one example, the controller 308 implementing the process 700, for example using the biomarker metric processing module 326, calculates the amount of tissue on the slide by measuring the total area covered by tissue in the histopathology image or by counting the number of cells on the slide. The number of cells on the slide may be determined by the number of cell nuclei visible on the slide. In one example, the controller 308 calculates the percentage of tissue that is cancer cells by dividing the number of cell nuclei in a grid area labeled tumor by the total number of cell nuclei on the slide. The controller 308 may exclude cell nuclei or outer edges of cells that are located in tumor areas but belong to cells characterized as lymphocytes. The percentage of tissue that is cancer cells is known as the tumor purity of the sample. The controller 308 then compares the tumor purity to a user selected minimum tumor purity threshold and compares the number of cells in the digital image to a user selected (entered by a user interacting with the overlay map generator 324) minimum cell threshold and, if both thresholds are exceeded, approves the tissue slide depicted in the image for molecular testing, including gene sequence analysis. In one example, the user selected minimum tumor purity threshold is 0.20, or 20%, although any number of tumor purity thresholds may be selected, including 1%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 45%, 50%, or more.

[0222] In another example, the controller 308 multiplies the total area covered by tissue detected on the slide by a first multiplier value, multiplies the number of cells counted on the slide by a second multiplier value, and gives the image a composite tissue volume score that sums the products of these multiplications.

[0223] In one example, the controller 308 can calculate whether the grid areas labeled as tumor are spatially integrated or dispersed among the non-tumor grid areas. If the controller 308 determines that the tumor areas are spatially integrated, the overlay map generator 324 can generate a digital overlay of a recommended cutting boundary that separates image areas classified as tumor and non-tumor, or image areas within the non-tumor classified areas that are proximal to the areas classified as tumor. The recommended cutting boundary can be a guide to assist a technician in dissecting the slide to separate the maximum amount of tumor or non-tumor tissue from the slide, particularly for molecular testing including gene sequence analysis.

[0224] In one example, the controller 308 may include a clustering algorithm that calculates and reports information about spacing and density of type-classified cells, tissue-classified tiles, or visually detectable features on a slide. Spacing information includes distribution patterns and heat maps of lymphocytes, immune cells, tumor cells, or other cells. These patterns may include clustered, dispersed, dense, and absent. This information helps determine whether immune and tumor cells cluster together and what percentage of the cluster areas overlap, which may facilitate predicting immune infiltration and patient response to immunotherapy.

[0225] The controller 308 can also calculate and report the average tumor cell circularity, the average tumor cell perimeter, and the average tumor nuclei density.

[0226] The interval information also includes the level of mixing of tumor and immune cells: clustering algorithms can calculate the probability that two adjacent cells on a given slide or in a region of a slide are either two tumor cells, two immune cells, or one tumor cell and one immune cell.

[0227] The clustering algorithm can also measure the thickness of some stromal patterns located around areas classified as tumors, and the thickness of the stroma surrounding this tumor region may be a predictor of a patient's response to treatment.

[0228] In one example, the controller 308 can also calculate and report statistics including mean, standard deviation, sum, etc. for red, green, blue (RGB) values, brightness, hue, saturation, grayscale, and stain deconvolution information within each grid tile aggregated from a single slide image or many slide images. Deconvolution involves the removal of visual signals created by several individual stains or combinations of stains, including hematoxylin, eosin, or IHC stains.

[0229] The controller 308 can also incorporate known mathematical formulas from the fields of physics and image analysis to calculate basic visually detectable features of each grid tile. Basic visually detectable features including lines, alternating brightness patterns, and outlineable shapes can be combined to create complex visually detectable features including cell size, cell circularity, cell shape, and staining patterns referred to as texture features.

[0230] In other examples, the digital overlays, reports, statistics, and estimates generated by the overlay map generator 324 may be useful in predicting patient survival, patient response to a particular cancer treatment, PD-L1 status of tumor or immune clusters, microsatellite instability (MSI), tumor mutation burden (TMB), and tumor origin when the tumor origin is unknown or the tumor is metastatic. The biomarker metric processing module 326 may also calculate quantitative measures of predicted patient survival, patient response to a particular cancer treatment, PD-L1 status of tumor or immune clusters, microsatellite instability (MSI), and tumor mutation burden (TMB).

[0231] In one example, the controller 308 can calculate the relative density of each type of immune cell across the entire slide in an area designated as tumor or another tissue class, including lymphocytes, cytotoxic T cells, B cells, NK cells, macrophages, etc.

[0232] In one example, the act of scanning or otherwise digitally capturing a histopathology slide automatically triggers the deep learning framework 300 to analyze the digital image of the histopathology slide.

[0233] In one example, the overlay map generator 324 allows a user to edit the cell edge or boundary between two tissue classes on a tissue class overlay map or a cell edge overlay map and save the modified map as a new overlay.

[0234] 11 illustrates a process 1100 for preparing digital images of pathology slides for tissue classification, biomarker detection, and mapping analysis that may be implemented using the system 300. The process 1100 may be performed on each image received for analysis and biomarker prediction. In some examples, the process 1100 may be performed in whole or in part on training images initially received. Each of the processes described in FIG. 9 may be performed by the pre-processing controller 302, where any one or more of the processes may be performed by the normalization module 310 and / or the tissue detector 314.

[0235] In one example, such as when training a classifier model, each digital image file received at 1102 by the pre-processing controller 302 contains multiple versions of the same image content, each version having a different resolution. The file stores these copies in stacked layers, arranged by resolution, with the highest resolution image containing the greatest number of bytes at the bottom. This is known as a pyramid structure. In one example, the highest resolution image is the highest resolution achievable by the scanner or camera that created the digital image file.

[0236] In one example, each digital image file also includes metadata indicating the resolution of each layer. The pre-processing controller 302 can detect the resolution of each layer in this metadata and compare it to a user-selected resolution criteria to select the layer with the optimal resolution for analysis, in process 1104. In one example, the optimal resolution is 1 pixel per micron (downsampled by 4).

[0237] In one example, the pre-processing controller 302 receives a tagged image file format (TIFF) file with a lowest resolution of 4 pixels per micron. This 4 pixels per micron resolution corresponds to the resolution achieved by a microscope objective with a magnification of "40x". In one example, the area where tissue may be present on the slide is up to 100,000 x 100,000 pixels in size.

[0238] In one example, a TIFF file has about 10 layers, with each layer having half the resolution of the layer below it. If the resolution of the high-resolution layer is 4 pixels per micron, then the resolution of the layer above it will be 2 pixels per micron. The area represented by one pixel of the upper layer is the size of the area represented by four pixels of the lower layer, i.e., the length of each side of the area represented by one upper layer pixel is twice the length of each side of the area represented by one lower layer pixel.

[0239] Each layer may be downsampled by twice the amount of the layer below it, as performed in process 1106. Downsampling is a method by which a new version of an original image may be created at lower resolution values ​​than the original image. There are many methods known in the art for downsampling, including nearest neighbor, bilinear, Hermite, Bell, Mitchell, bicubic, and Lanczos resampling.

[0240] In one example, downsampling by a factor of 2 means that the red, green, and blue (RGB) values ​​of three of four pixels in a square in the high-resolution layer are replaced with the RGB value of the fourth pixel, creating a new, larger pixel in the layer above that occupies the same space as the four averaged pixels.

[0241] In one example, the digital image file does not contain a layer or image at the optimum resolution, in which case the pre-processing controller 302 may receive an image from the file having a higher resolution than the optimum resolution in process 1106 and downsample the image at a ratio that achieves the optimum resolution.

[0242] In one example, the optimum resolution is 2 pixels per micron, or a "20x" magnification, but the bottom layer of the TIFF file is 4 pixels per micron, and each layer is downsampled by 4x compared to the layer below it. In this case, the TIFF file has one layer with 40x magnification and the next layer with 10x magnification, but no layer with 20x magnification. In this example, pre-processing controller 302 reads the metadata, compares the resolution of each layer to the optimum resolution, and does not find a layer with the optimum resolution. Instead, pre-processing controller 302 finds the layer with 40x magnification, and then downsamples the image in that layer by a downsampling ratio of 2x to create an image with an optimum resolution of 20x magnification.

[0243] Also in process 1106, the pre-processing controller 302, after acquiring the image at optimal resolution, identifies all portions of the image that depict tumor sample tissue and digitally removes debris, pen marks, and other non-tissue objects.

[0244] In one example, also in process 1106, the pre-processing controller 302 distinguishes between tissue and non-tissue regions of the image and uses Gaussian blur removal to edit out pixels with non-tissue objects. In one example, any control tissue on the slide that is not part of the tumor sample tissue can be detected by a tissue detector and labeled as control tissue or manually labeled by a human analyst as control tissue to be excluded from downstream tile grid projections.

[0245] Non-tissue objects include artifacts, markings, and debris in the images. Debris includes keratin, badly compressed or pulverized tissue that cannot be visually analyzed, and objects that were not collected with the sample.

[0246] In one example, also in process 1106, the slide image contains marker ink or other writing that is detected and digitally removed by controller 302. The marker ink or other writing may be transparent on the tissue, meaning that the tissue on the slide may be visible through the ink. Because the ink of each marking is one color, the ink causes a consistent shift in RGB values ​​of pixels that contain stained tissue underneath the ink compared to pixels that contain stained tissue without the ink.

[0247] In one example, also in process 1106, the controller 302 identifies portions of the slide image that contain ink by detecting portions that have RGB values ​​that are different from the RGB values ​​of the remainder of the slide image, where the difference in RGB values ​​from the two portions is consistent. The tissue detector can then subtract the difference between the RGB values ​​of the pixels of the ink portions and the pixels of the non-ink portions from the RGB values ​​of the pixels of the ink portions to digitally remove the ink.

[0248] In one example, also in process 1106, the controller 302 eliminates pixels in the image that have low local variation. These pixels represent artifacts, markings, or blurry areas caused by an out-of-focus tissue slice, an air bubble trapped between two glass layers of a slide, or a pen mark on a slide.

[0249] In one example, also in process 1106, the controller 302 removes these pixels by converting the image to a grayscale image, and then passing the grayscale image through a Gaussian blur filter to mathematically adjust the original grayscale value of each pixel to a blurred grayscale value to create a blurred image. Other filters can be used to blur the image. Then, for each pixel, the controller 302 subtracts the blurred grayscale value from the original grayscale value to create a difference grayscale value. In one example, if the difference grayscale value of the pixel is less than a user-defined threshold, it may indicate that the blur filter did not significantly change the original grayscale value and the pixel of the original image was in a blurred area. The difference grayscale value may be compared to a threshold to create a binary mask that indicates where the blurred areas are, which may be designated as non-tissue areas. The mask may be a copy of the image, with the colors, RGB values, or other values ​​in the pixels adjusted to indicate the presence or absence of a particular type of object, and the location of all objects of that type. For example, a binary mask may be generated by setting the binary value of each pixel to 0 if the pixel has a differential grayscale value below a user-defined blur threshold, and setting the binary value of each pixel to 1 if the pixel has a differential grayscale value equal to or greater than the user-defined blur threshold. Regions of the binary mask with pixel binary values ​​of 0 indicate blurred areas of the original image that may be designated as non-tissue.

[0250] The controller 302 can also mute or remove extreme brightness or darkness in the image in process 1108. In one example, the controller 302 converts the input image to a grayscale image, with each pixel receiving a numerical value according to the brightness of the pixel. In one example, the grayscale values ​​range from 0 to 255, where 0 represents black and 255 represents white. For pixels with grayscale values ​​above the brightness threshold, the tissue detector replaces the grayscale values ​​of those pixels with the brightness threshold. For pixels with grayscale values ​​below the darkness threshold, the tissue detector replaces the grayscale values ​​of those pixels with the darkness threshold. In one example, the brightness threshold is about 210. In one example, the darkness threshold is about 45. The tissue detector stores the image with the new grayscale values ​​in a data file.

[0251] In one example, the controller 302 analyzes the altered image for artifacts, debris, or markings remaining after the initial analysis in process 1110. A tissue detector scans the image and classifies remaining groups of pixels having a particular color, size, or smoothness as non-tissue.

[0252] In one example, a slide has H&E staining and most of the tissue in the histopathology image has a pink staining. In this example, the controller 302 classifies all objects that do not have a pink or red hue as determined by the RGB values ​​of the pixel representing the object as non-tissue. The tissue detector 314 can interpret any color or lack of any color in a pixel to indicate the presence or absence of tissue in that pixel.

[0253] In one example, the controller 302 detects the outline of each object in the image to measure the size and smoothness of each object. Very dark pixels may be debris and very bright pixels may be background, both of which are non-tissue objects. Thus, the controller 302 can detect the outline of each object by converting the image to grayscale, comparing the grayscale value of each pixel to a user-determined range of values ​​that are not too light or too dark, and determining if the grayscale value is within the range to generate a binary image in which each pixel is assigned one of two numerical values.

[0254] For example, to threshold an image, controller 302 can compare the grayscale value of each pixel to a user-defined range of values ​​and replace each grayscale value outside the user-defined range with a value of 0, and replace each grayscale value within the user-defined range with a value of 1. Controller 302 then draws all the outlines of all objects as the outer perimeter of each group of adjacent pixels that have a value of 1. A closed outline indicates the presence of an object, and controller 302 measures the area within the outline of each object to measure the size of the object.

[0255] In one example, tissue objects on a slide are unlikely to come into contact with the outer edge of the slide, and the controller 302 classifies all objects that come into contact with the edge of the slide as non-tissue.

[0256] In one example, after measuring the size of each object, the controller 302 ranks the sizes of all objects and designates the maximum value as the size of the largest object. The controller 302 divides the size of each object by the size of the largest object and compares the resulting size quotient to a user-defined size threshold. If the size quotient of an object is less than the user-defined size threshold, the controller 302 designates the object as non-tissue. In one example, the user-defined size threshold is 0.1.

[0257] Prior to measuring the size of each object, in process 1106, the controller 302 may first downsample the input image to reduce the likelihood that a portion of a tissue object will be designated as non-tissue. For example, a single tissue object may appear as a first tissue object portion surrounded by one or more additional tissue object portions having smaller sizes. After thresholding, the additional tissue object portions may have a size quotient smaller than a user-defined size threshold and may be erroneously designated as non-tissue. Downsampling before thresholding may result in a small group of adjacent pixels with a value of 1 surrounded by pixels with a value of 0 in the original image being included in a large group of proximal pixels with a value of 1. The opposite may also be true where a small group of adjacent pixels with a value of 0 surrounded by pixels with a value of 1 in the original image being included in a large group of proximal pixels with a value of 0.

[0258] In one example, the controller 302 downsamples an image having a magnification of 40x by a ratio of 16x, so that the resulting downsampled image has a magnification of 40 / 16x, and each pixel in the downsampled image represents 16 pixels of the original image.

[0259] In one example, in process 1110, the controller 302 detects the boundary of each object on the slide as a cluster of pixels having a binary or RGB value not equal to zero and surrounded by pixels having an RGB value equal to zero that indicates the boundary of the object. If the pixels forming the boundary are relatively straight, the controller 302 classifies the object as non-tissue. For example, the controller 302 uses a closed polygon to outline a shape. If the number of vertices of a polygon is less than a user-defined minimum vertex threshold, the polygon is considered to be a simple inorganic shape that is too smooth and is marked as non-tissue. The controller 302 then applies a tiling process to the normalized image in process 1112.

[0260] 12A-12C show an example architecture 1200 that may be used for the classification model of module 306. For example, the same architecture 1200 may be used for each of tissue segmentation model 322 and tissue classification model 320, both implemented using the FCN configuration or any neural network of the present specification. Tissue classifier module 306 includes a tissue classification algorithm (see FIGS. 12A-12C) that assigns a tissue class label to the image represented by each received tile (an example tile 1302 is labeled in a first portion 1304 of a histopathology image 1300 shown in FIG. 13). In one example, overlay map generator 324 may report the assigned tissue class label associated with each small square tile by displaying a grid-based digital overlay map in which each tissue class is represented by a unique color (see FIG. 12A).

[0261] A small tile size may increase the time required for the tissue classifier module 306 to analyze the input image. Alternatively, a large tile size may increase the likelihood that a tile contains more than one tissue class, making it difficult to assign a single tissue class label to the tile. In this case, instead of calculating a higher probability that one of the tissue class labels describes the image in a small square tile compared to the other tissue class labels, the architecture 1200 may calculate an equal probability that two or more tissue class labels are accurately assigned to a single small square tile.

[0262] In one example, each side of each small square tile is approximately 32 microns long, allowing approximately 5-10 cells to fit into each small square tile. This small tile size allows the tissue classifier module 306 to create more spatially accurate boundaries when determining the boundary between two adjacent small square tile regions representing two different tissue classes. In one example, each side of the small square tile can be as short as 1 micron.

[0263] In one example, the size of each tile can be set by a user to contain a particular number of pixels. In this example, the resolution of the input image will determine the length of each side of the tile measured in microns. At different resolutions, the tile sides may have different lengths in microns, and each tile may have a different number of cells.

[0264] The architecture 1200 recognizes various pixel data patterns in portions of the digital image located within or near each small square tile and assigns a tissue class label to each small square tile based on those detected pixel data patterns. In one example, a medium square tile centered on a small square tile includes an area of ​​the slide image that is sufficiently close to the small square tile to contribute to the label assignment of the small square tile.

[0265] In one example, each side of the medium square tile is approximately 466 microns long, and each medium square tile contains approximately 225 (15x15) small square tiles. In one example, this medium tile size increases the likelihood that structural tissue features can fit within a single medium tile, providing context to the algorithm when labeling the central small square tile. Structural tissue features may include glands, ducts, blood vessels, immune clusters, etc.

[0266] In one example, this medium tile size is chosen to counteract the shrinkage that occurs during convolution.

[0267] During convolution by architecture 1200, the input image matrix is ​​multiplied by a filter matrix to create a result matrix. Shrinkage refers to when the result matrix is ​​smaller than the input image matrix. The dimensions of the filter matrices of a convolutional layer affect the number of rows and columns lost by shrinkage. The total number of matrix entries lost by shrinkage by processing an image through a particular CNN can be calculated depending on the number of convolutional layers of the CNN and the dimensions of the filter matrices of each convolutional layer (see Figures 12A-12C).

[0268] In the example shown in FIG. 12B, the combined convolutional layer loses a total of 217 matrix rows or columns from the top, bottom, and two side edges of the matrix, so the medium square tile is set to be equal to the small square tile plus 217 pixels on either side of the small square tile.

[0269] In one example, two adjacent small square tiles share a side and each is at the center of a medium square tile. The two medium square tiles overlap. Of the 466*466 small pixels in each medium square tile, the two medium square tiles share all but 32*466 pixels. In one example, each convolutional layer of the algorithm (see Figures 12A and 12B) analyzes both medium square areas simultaneously, such that the algorithm produces two vectors of values, one for each of the two small square tiles.

[0270] The vector of values ​​includes a probability value for each tissue class label, indicating the likelihood that the small square tile represents that tissue class. The vector of values ​​may be arranged in a matrix to form a three-dimensional probability data array. The location of each vector in the three-dimensional probability data array relative to other vectors corresponds to the location of the associated small square tile relative to other small square tiles included in the algorithmic analysis.

[0271] In this example, 434x434 (188, 356) of the 466x466 (217, 156) pixels in each medium square tile are common to both medium square tiles. By analyzing both medium square tiles simultaneously, the algorithm increases efficiency.

[0272] In one example, architecture 1200 can further increase efficiency by analyzing large tiles formed by multiple overlapping medium square tiles, each of which contains many small square tiles surrounding one central small square tile that receives a tissue class label. In this example, the algorithm generates a data structure in the form of a three-dimensional probability data array that contains one vector of probabilities for each small square tile, with the location of the vectors in the three-dimensional array corresponding to the location of the small tiles within the large tile.

[0273] The architecture 1200 stores this three-dimensional probability data array, for example in the tissue classifier module 306, and the overlay map generator 324 converts the tissue class label probability for each small square tile into a tissue class overlay map. In one example, the overlay map generator 324 can compare the probabilities stored in each vector to determine the maximum probability value associated with each small square tile. The tissue class label associated with that maximum value can be assigned to that small square tile, and only the assigned labels will be displayed in the tissue class overlay map.

[0274] In one example, the matrices generated by each layer of the architecture 1200 for a large square tile are stored in a graphics processing unit (GPU) memory. The capacity of the GPU memory and the amount of GPU memory required for each entry of the three-dimensional probability data array may determine the maximum possible size of the large square tile. In one example, the GPU memory capacity is 250 MB, and each entry of the matrix requires 4 bytes of GPU memory. This allows for a large tile size of 4,530 pixels by 4,530 pixels, calculated as follows: 4 bytes / entry * 4530 * 4530 * 3 entries for each large tile = 246 (approximately 250) MB of GPU memory is required for each large square tile. In another example, each entry of the matrix requires 8 bytes of GPU memory. In this example, a 16 GB GPU can process 32 large tiles simultaneously, and the size of each large tile has dimensions of 4,530 pixels by 4,530 pixels, calculated as follows: 32 big tiles * 8 bytes / entry * 4530 * 4530 * 3 entries for each big tile = 14.7 (approximately 16) GB of GPU memory required.

[0275] In one example, each entry in the three-dimensional probability data array is a data entry in single-precision floating-point format (float32).

[0276] In one example, there are 16,384 (1282) non-overlapping small square tiles that form a large square tile. Each small square tile is the center of a medium square tile with sides each approximately 466 pixels long. The small square tiles form the central region of the large square tile with sides each approximately 4,096 pixels long. The medium square tiles all overlap, creating a border around all four sides of the central region that is approximately 217 pixels wide. Including the border, each large square tile has sides each approximately 4,530 pixels long.

[0277] In this example, this large square tile size allows for simultaneous calculations and reduces the percentage of redundant calculations by 99%. This can be calculated as follows: First, a pixel inside the large square tile (any pixel at least 434 pixels from the edge of the large square tile) is selected, and a region the size of a medium square tile (466 pixels per edge) is created with this model pixel as its center, then for a small square tile that is in the center of this constructed region, the model pixel is contained within the corresponding medium square tile of that small square tile. There are (466 / 32)^2=approximately 217 such small square tiles inside the large square tile. For pixels that are not inside the large square tile, the number of small square tiles that satisfy this condition will be smaller. As the distance between the selected small square tile and the edge of the large square tile decreases, the number decreases linearly, and then as the distance between the selected small square tile and the corner decreases, a small number of pixels (approximately 0.005%) will only contribute to the classification of a single small square tile. Performing the classification on a single large square tile means that the calculation for each pixel is performed only once, instead of once for each small square tile, and thus the redundancy is reduced by a factor of almost 217. In one example, a slide may contain several large square tiles, each of which may overlap slightly with adjacent tiles, so the redundancy is not completely eliminated.

[0278] An upper limit can be set on the redundant calculation percentage (the slight deviation from this upper limit depends on the number of large square tiles required to cover the tissue and the relative placement of these tiles). The redundancy percentage is 1-1 / r, where r is the redundancy ratio, and r can be calculated as (T / N+1)(sqrt(N)*E+434)^2 / (sqrt(T)*E+434)^2, where T is the total number of small square tiles on the slide, N is the number of small square tiles per large square tile, and E is the border size of the small square tiles.

[0279] FIG. 12A shows layers of an example layer structure of the architecture 1200. FIG. 12B shows an example of the different layers of the architecture 1200 and the resulting sub-layer output sizes, illustrating the tile-resolution FCN configuration. As shown, the tile-resolution FCN configuration included in the tissue classifier module 306 has an additional layer of 1×1 convolution with skip connections, 8x downsampling with skip connections, uses a belief map layer, replaces the average pooling layer with a concatenation layer, and replaces the fully connected FCN layer with a 1×1 convolution and softmax layer. The additional layers transform the classification task into a classification segmentation task. This means that instead of receiving and classifying the entire image as one tissue class label, the additional layers enable the tile-resolution FCN to classify each small tile in the user-defined grid as a tissue class.

[0280] These append and permutation layers convert the CNN into a tile-resolution FCN without the need for upsampling, which is performed in later layers of a traditional pixel-resolution FCN. Upsampling is a method that allows for the creation of a new version of an original image with higher resolution values ​​than the original. However, upsampling is a time-consuming and computationally intensive process that can be avoided in our architecture.

[0281] There are many methods known in the art for upsampling, including nearest neighbor, bilinear, Hermite, Bell, Mitchell, bicubic, and Lantzos resampling. In one example, 2x upsampling means that a pixel with red, green, and blue (RGB) values ​​may be split into four pixels, and the RGB values ​​of the three new pixels may be selected to match the RGB values ​​of the original pixel. In another example, the RGB values ​​of the three new pixels may be selected as the average of the RGB values ​​from the pixels adjacent to the original pixel and the neighboring pixels.

[0282] Upsampling can introduce errors into the final image overlay map generated by overlay map generator 224 because the RGB values ​​of the new pixels may not accurately reflect the visible tissue of the original slide captured by the digital slide image.

[0283] In one example, instead of labeling individual pixels, a tiled FCN is programmed to analyze a large square tile made up of smaller square tiles, generating a 3D array of values, each representing the probability that one tissue class classification label matches the tissue class depicted in each smaller tile. A convolutional layer, as known in the art, performs multiplication of at least one input image matrix by at least one filter matrix. After the first convolution, the input image matrix has values ​​for every pixel in the large square tile input image, representing the visual data for that pixel (e.g., values ​​from 0 to 255 for each channel of RGB).

[0284] The filter matrix may have user-selected dimensions and may contain weight values ​​selected by the user or determined by backpropagation during training of the CNN model. In one example, in the first convolutional layer, the filter matrix has dimensions of 7×7 and there are 64 filters. The filter matrix may represent visual patterns that can distinguish one tissue class from another.

[0285] In one example where RGB values ​​are input to the input image matrix, the input image matrix and the filter matrix are three-dimensional (see FIG. 12C). Each filter matrix is ​​multiplied with each input image matrix to generate a result matrix. All the result matrices generated by the filters of one convolutional layer can be stacked to create a three-dimensional result matrix with dimensions such as rows, columns, and depth. The last dimension of the 3D result matrix, the depth, has a depth equal to the number of filter matrices. The result matrix from one convolutional layer becomes the input image matrix of the next convolutional layer.

[0286] Returning to FIG. 12A, the title of a convolutional layer that includes " / n" (where n is a number) indicates that there is downsampling (known as pooling) of the result matrix produced by that layer. The n indicates the factor by which the downsampling occurs. Downsampling by a factor of 2 means that a downsampled result matrix with half the rows and half the columns of the original result matrix is ​​created by replacing the squares of four values ​​in the result matrix with one of those values, or a statistic calculated from those values. For example, the minimum, maximum, or average of the values ​​may be replaced with the original value.

[0287] Architecture 1200 also adds skip connections (shown in FIG. 12A as black lines with arrows that connect the blue convolutional layers directly to the concatenation layers). The skip connection on the left involves 8x downsampling, while the skip connection on the right involves two convolutional layers that multiply the input image matrix by a filter matrix, each with dimensions 1×1. Because the filter matrices in these layers are of dimension 1×1, only individual small square tiles contribute to the corresponding probability vectors in the result matrix created by the purple convolutional layer. These result matrices represent a small field of view.

[0288] In all other convolutional layers, the large dimensions of the filter matrices allow pixels in each medium square tile, including the small square tile in the center of a medium square tile, to contribute to a probability vector in the result matrix corresponding to that small square tile. These result matrices allow the contextual pixel data patterns surrounding the small square tile to influence the probability that each tissue class label is applied to the small square tile. These result matrices represent a large field of view.

[0289] The skip-connection 1×1 convolutional layer allows the algorithm to consider the pixel data pattern of the small central square tile as more or less important than the remaining pixel data patterns of the surrounding medium square tiles. This is reflected by the weights by which the trained model is multiplied by the final result matrix from the skip connection layer (shown on the right side of FIG. 12A) compared to the weights by which the trained model is multiplied by the final result matrix from the medium tile convolutional layer between the connection layers (shown in the middle column of FIG. 10A).

[0290] The downsampling skip connection shown on the left side of FIG. 12A creates a result matrix of depth 64. A 3×3 convolution layer with 512 filter matrices creates a result matrix of depth 512. A 1×1 convolution layer with 64 filter matrices creates a result matrix of depth 64. All three of these result matrices have the same number of rows and the same number of columns. The concatenation layer concatenates these three result matrices to form a final result matrix with the same number of rows, the same number of columns, and a depth of 64+512+64 (640) as the three concatenated matrices. This final result matrix combines the small and large foci of the view matrices.

[0291] The final result matrix can be flattened to two dimensions by multiplying every entry by a coefficient and summing the products along each depth. Each element can be selected by the user or by backpropagation during model training. Flattening does not change the number of rows and columns of the final result matrix, but it does change the depth to 1.

[0292] The 1x1 convolutional layer takes the final result matrix and filters it with one or more filter matrices. The 1x1 convolutional layer may contain one filter matrix associated with each tissue class label of the trained algorithm. This convolutional layer produces a 3D result matrix with a depth equal to the number of tissue class labels. Each depth corresponds to one filter matrix, and there may be a probability vector for each small square tile along the depth of the result matrix. This 3D result matrix is ​​a three-dimensional probability data array, and the 1x1 convolutional layer stores this 3D probability data array.

[0293] A softmax layer can create a two-dimensional probability matrix from a 3D probability data array by comparing all values ​​of each probability vector, selecting the tissue class associated with the maximum value, and assigning that tissue class to the small square tile associated with that probability vector.

[0294] The stored three-dimensional probability data array or 2D probability matrix can then be converted into a tissue class overlay map at the final confidence map layer in FIG. 10A to efficiently assign tissue class labels to each tile.

[0295] In one example, to counteract shrinkage, the input image matrix adds rows and columns to all four outer edges of the matrix, with each value entry of the added rows and columns being zero. These rows and columns are referred to as padding. In this case, the training data input matrix would have the same number of added rows and columns with value entries equal to zero. Any difference in the number of padding rows or columns in the training data input matrix would result in values ​​in the filter matrix that would not cause the tissue class locator 216 to accurately label the input image.

[0296] In the FCN shown in Figure 12A, because of the gray and blue layers, a total of 217 outer rows or columns on each side of the input image matrix are lost to shrinkage before skip connections. Only pixels in the small square tiles have vectors that correspond to the result matrix created after the green layer.

[0297] In one example, each medium square tile is not padded by adding rows and columns with zero value entries around the input image matrix corresponding to each medium square tile, because the zeros replace image data values ​​from adjacent medium square tiles that the tissue class locator 216 needs to analyze. In this case, the training data input matrix is ​​also not padded.

[0298] FIG. 12C is a visualization of the depths of an exemplary three-dimensional input image matrix being convolved with two exemplary three-dimensional filter matrices.

[0299] In an example where the input image matrix contains RGB channels for each medium square tile, the input image matrix and each filter matrix are three-dimensional, in one of which the input image matrix and each filter matrix has three depths, one for the red channel, one for the green channel, and one for the blue channel.

[0300] The red channel (first depth) 1202 of the input image matrix is ​​multiplied by the corresponding first depth of the first filter matrix. The green channel (second depth) 1204 is multiplied in a similar manner, and the blue channel (third depth) 1206 is multiplied in a similar manner. The red, green, and blue product matrices are then summed to create the first depth of a three-dimensional result matrix. This is repeated for each filter matrix, creating additional depths of three-dimensional result matrices corresponding to each filter.

[0301] A wide variety of training sets can be used to train the CNN or FCN included in the tissue classifier module 306.

[0302] In one example, the training set may include JPEG images of medium square tiles, each with a tissue class label assigned to its central small square tile taken from at least 50 digital images of histopathology slides at a resolution of approximately 1 pixel per micron. In one example, a human analyst has outlined and labeled (annotated various tissue classes) all relevant tissue classes or labeled each small square tile of each histopathology slide as non-tissue or a specific type of cell. The tissue classes may include tumor, stromal, normal, immune cluster, necrosis, hyperplasia / dysplasia, and red blood cells. In one example, each side of each central small square tile is approximately 32 pixels long.

[0303] In one example, the training set images are converted into an input training image matrix and processed by the tissue classifier module 306 to assign a tissue class label to each tile image of the training images. If the tissue classifier module 306 does not accurately label the validation set of training images to match the corresponding annotations added by a human analyst, the weights of each layer of the deep learning network may be automatically adjusted by stochastic gradient descent with backpropagation until the tissue classifier module 306 accurately labels the majority of the validation set of training images.

[0304] In one example, a training dataset has multiple classes, where each class represents a tissue class. The training set generates a unique model with certain hyperparameters (e.g., number of epochs, learning rate, etc.) that can recognize the content of digital slide images and classify them into various classes. The tissue classes may include tumor, stroma, immune cluster, normal epithelium, necrosis, hyperplasia / dysplasia, and red blood cells. In one example, if there are sufficient training sets for each tissue class, the model can classify an unlimited number of tissue classes.

[0305] In one example, the training set images are converted into grayscale masks for annotation, where different values ​​(0-255) in the mask image represent different classes.

[0306] Because each histopathology image may exhibit significant variation in visual features, including tumor appearance, the training set may contain highly disparate digital slide images to better train the model for the wide variety of slides that may be analyzed. Images in the training data may be subjected to data augmentation (including rotation, scaling, color jitter, etc.) before being used to train the model.

[0307] The training set may also be cancer type specific, where all histopathology slides from which digital images were generated in a particular training set include tumor samples from the same type of cancer. Cancer types may include breast, colorectal, lung, pancreatic, liver, stomach, skin, etc. Each training set may generate a unique model specific to the cancer type. Each cancer type may also be divided into cancer subtypes known in the art.

[0308] In one example, the training set may be derived from a pair of histopathology slides. The pair of histopathology slides includes two histopathology slides, each with one slice of tissue, where the two slices of tissue were located substantially proximal / near adjacent to each other in the tumor sample. Thus, the two slices of tissue are substantially similar. One of the pair of slides is stained with only H&E staining, and the other of the pair of slides is stained with IHC staining of a particular molecular target. Areas on the H&E stained slide that correspond to areas where IHC staining appears in the pair of slides are annotated by a human analyst as containing the particular molecular target, and the tissue class locator receives the annotated H&E slide as the training set. Substantially similar slides may include, for example, a pair of slides including an H&E stained slide and a slide formed with molecular sequence data cropped from one of the adjacent slides, or other combinations, such as one with IHC staining and the other with molecular sequence data, or both with similar molecular sequence data.

[0309] For example, in some embodiments, two or more samples are obtained from a subject (e.g., two or more tissue slices adjacent to each other can be taken). In some cases, tissue slices are acquired such that a portion of a pathology slide prepared from each slice is imaged, while a portion of the pathology slide is used to acquire sequencing information.

[0310] A suitable training dataset can be used to train an optimization model according to embodiments of the present disclosure. In some embodiments, curating a training dataset can include collecting a series of pathology reports and associated sequencing information from multiple patients. For example, a physician can perform a tumor biopsy on a patient by extracting a small amount of tumor tissue / specimen from the patient and sending the specimen to a laboratory. In the laboratory, slides can be produced from the specimen using slide production techniques such as freezing the specimen and slice layers, setting the specimen in paraffin and slice layers, smearing the specimen on a slide, or other methods known to those skilled in the art. For purposes of the following disclosure, slides and slices can be used interchangeably. A slide stores a slice of tissue from a specimen and receives a label that identifies the specimen from which the slice was extracted and the sequence number of the slice from the specimen. Traditionally, pathology slides can be produced by staining a specimen to reveal cellular features (such as cell nuclei, lymphocytes, stroma, epithelium, or whole or part of other cells). The pathology slide selected for staining is traditionally the terminal slide of a specimen block. Slicing of the specimen proceeds with a series of initial slides that can be produced for staining and diagnostic purposes. The next consecutive slice in the series can be used for sequencing and the final terminal slide can be processed for additional staining. If the terminal stained slide is too far away from the sequenced slide, another slide closer to the sequenced slide may be stained and the sequenced slide divided by the stained slide. Although slight deviations exist from slice to slice, deviations are expected to be minimal since tissue is sliced ​​at thicknesses approaching 4um for paraffin slides and 35um for frozen slides. Laboratories have generally found that distances of typically less than 40um (approximately 10 slides / slice) do not result in substantial deviations in tissue slices.

[0311] If slices of the specimen vary significantly from slice to slice (which is infrequent), the outliers may be discarded and not processed further. The pathology slides 510 may be various stained slides taken from tumor samples from patients. Some slides and sequencing data may be taken from the same specimen to ensure data robustness, while other slides and sequencing data may be taken from each unique specimen. The more tumor samples in the dataset, the higher the accuracy can be expected from the prediction of cell type RNA profiles. In some embodiments, the stained tumor slides may be reviewed by a pathologist for identification of cellular features (such as the amount of cells and how different they are from normal cells of that type or similar types).

[0312] In this case, the trained tissue classification model 320 receives a digital image of H&E stained tissue and predicts tiles that may contain IHC staining or a given molecular target, and the overlay map generator 326 generates an overlay map indicating which tiles are likely to contain the IHC target or the given molecule. In one example, the resolution of the overlay is at the level of individual cells.

[0313] Overlays generated by a model trained on one or more training sets can be reviewed by a human analyst to annotate and add digital slide images to one of the training sets.

[0314] The pixel data patterns that the algorithm detects may represent visually detectable features, some examples of which may include color, texture, cell size, shape, and spatial organization.

[0315] For example, the color of the slide provides contextual information. Purple areas on a slide are more likely to be cellular and invasive tumors. Tumors also make the surrounding stroma more fibrotic in a desmoplastic reaction, making the normally pink stroma appear blue-gray. Color intensity also helps identify individual cells of a particular type (e.g., lymphocytes are uniformly very dark blue).

[0316] Texture refers to the distribution of staining within the cells. Most tumor cells have a rough, heterogeneous appearance with bright pockets in the nucleus and dark nucleoli. A zoomed out view with many tumor cells will give this rough appearance. Many non-tumor tissue classes each have characteristic features. Furthermore, the pattern of tissue classes present in an area can indicate the type of tissue or cellular structure present in that area.

[0317] Additionally, cell size often indicates tissue class: if a cell is several times larger than normal cells elsewhere on the slide, it is more likely to be a tumor cell.

[0318] The shape of individual cells, particularly how round they are, can indicate what type of cell they are. Fibroblasts (stromal cells) are usually elongated and thin, while lymphocytes are very round. Tumor cells may be more irregularly shaped.

[0319] The organization of groups of cells can also indicate tissue classes. Often, normal cells are organized in structured, recognizable patterns, while tumor cells grow in denser, disorganized clusters. Each type and subtype of cancer may produce tumors with specific growth patterns, including the location of cells relative to tissue features, the spacing of tumor cells relative to each other, the formation of geometric elements, etc.

[0320] The techniques herein can be extended to other architectures. FIG. 14, for example, shows an imaging-based biomarker prediction system 1400 that also uses separate pipelines for tissue and cell classification. The system 1400 can be used for the determination of various biomarkers, including PD-L1, as described in the examples herein. Additionally, similar to other architectures herein, the system 1400 can be configured to predict biomarker status and tumor status and tumor statistics based on 3D image analysis.

[0321] The system 1400 receives one or more digital images of a histopathology slide and creates a dense grid-based digital overlay map that identifies the majority of classes of tissue visible within each grid tile of the image. The system 1400 can also generate a digital overlay drawing that identifies each cell in the histopathology image at a resolution level of individual pixels.

[0322] The system 1400 includes a tissue detector 1402 that detects areas of the digital image that have tissue and stores data including the location of the areas detected as having tissue (e.g., the location of the pixel using a reference location in the image, such as the location of the 0,0 pixel). The tissue detector 1402 forwards tissue area location data 1403 to a tissue class tile grid projector 1404 and a cell tile grid projector 1406. The tissue class tile grid projector 1404 receives the tissue area location data 1403 and performs tissue classification on the tiles for each of a number of tissue class labels. A tissue class locator 1408 receives the resulting tile classifications and determines where each tissue class is located in the digital image by calculating a percentage that represents the likelihood that the tissue class label accurately describes the image in each tile. For each tile, all percentages calculated for all tissue class labels sum to 1, reflecting 100%. In one example, the tissue class locator 1408 assigns one tissue class label to each tile to determine where each tissue class is located in the digital image. The tissue class locator stores the calculated percentages and the assigned tissue class labels associated with each tile.

[0323] In one example, the system 1400 includes a multi-tile algorithm that simultaneously analyzes many tiles in an image, both individually and in combination with the portion of the image surrounding each tile. The multi-tile algorithm can provide multi-scale, multi-resolution analysis that captures both the content of the individual tiles and the context of the portion of the image surrounding the tile. Because the portion of the image surrounding two adjacent tiles overlaps, analyzing many tiles and their surroundings simultaneously, rather than analyzing each tile individually with its surroundings, reduces computational redundancy and improves processing efficiency.

[0324] In one example, the system 1400 can store the analysis results in a three-dimensional probability data array, which includes one one-dimensional data vector for each tile analyzed. In one example, each data vector includes a list of percentages that sum to 100%, each indicating the probability that each grid tile contains one of the analyzed tissue classes. The location of each data vector in the orthogonal two-dimensional plane of the data array relative to the other vectors corresponds to the location of the tile associated with that data vector in the digital image relative to the other tiles.

[0325] The cell type tile grid projector 1406 receives the tissue area location data 1403, identifies and classifies cells within the tiles, and projects a cell type tile grid onto the area of ​​the image containing tissue. The cell type locator 1410 detects each biological cell of the digital image within each grid, creates an outline around the outer edge of each cell, and can classify each cell by cell type. The cell type locator 1410 stores data including each pixel including the location of each cell and the outer edge of the cell, as well as a cell type label assigned to each cell.

[0326] The overlay map generator and metric calculator 1412 can take the stored three-dimensional probability data array from the tissue class locator 1408 and convert it into an overlay map that displays the tissue class label assigned to each tile. The tissue class assigned to each tile may be displayed as a transparent color unique to each tissue class. In one example, the tissue class overlay map displays the probability of each grid tile for a user-selected tissue class. The overlay map generator and metric calculator 1412 also retrieves stored cell location and type data from the cell type locator 1410 and calculates a metric related to the number of cells in the entire image or within a tile assigned to a particular tissue class.

[0327] FIG. 15A shows an overview of an exemplary process 1500 implemented by the imaging-based biomarker prediction system 1400, showing a model inference pipeline for predicting biomarkers in this exemplary PD-L1. In the process 1500, many tiles are processed in parallel, taking advantage of the fully convolutional model architecture of the system 1400. In one example, using a GeForce GTX 1080 Ti GPU and a 6th Generation Intel® Core™ i7 processor, the process 1500 took 2.8 seconds to classify a single 4096×4096 pixel image. The process 1500 further included tissue detection and artifact removal algorithms to work in a fully automated manner in a real-world setting where slides may contain artifacts.

[0328] In the first process 1502, an initial tissue segmentation is performed by the tissue detector 1402, e.g., applying a tissue masking algorithm to automatically outline the tissue (red contour) to generate a bounding box (not shown) around the tissue of interest. Aligned with the upper left corner of the bounding box, the tissue region is divided into large non-overlapping 4096x4096 input windows (blue dashed lines). Typically, 10-30 input windows are needed to cover the tissue. Large window areas extending beyond the bounding region are padded with zeros (grey areas).

[0329] In process 1504, trained classification model prediction is performed. In the illustrated example, the large input window contained 128×128=16,384 small 32×32 tiles (the grid is much finer than shown). The large input window was padded with zeros on all sides (length 217) to account for the overlapping 466×466 tile edges in the center of each 32×32 small tile. Each large window was passed through one or more trained models 1506 of a deep learning framework (including the tissue classification process of projector 1404 and the cell classification process of projector 1406). If the trained models 1506 are fully convolutional, in one example, each tile in the large input window is processed in parallel to generate a 128×128×3 probability cube (in which there are three classes). Each 1×1×3 vector of this probability cube corresponds to a 32×32 pixel region in the center of each 466×466 tile of the original image. The resulting probability cubes are assembled into a probability map for the entire image.

[0330] Process 1508 displays the image associated with the tissue masking step of process 1502. The assembled probability map generated by process 1504 is passed through this tissue mask to remove the background. In this example, both the background and marker areas have been removed by the masking algorithm of process 1508.

[0331] Process 1510 displays an image showing a classification map identifying one or more classified regions, such as each distinct region for each of the biomarker classes. In one example, the maximum probability class (argmax) is assigned to each tile via process 1508, and a classification map of three biomarker classes (PD-L1+, PD-L1-, Other) is generated in process 1510. The classification map shows each of these biomarker classes and their identified locations relative to the original histopathology image.

[0332] Process 1512 performs statistical analysis on the biomarker classifications from the classification map and displays the resulting prediction scores for the biomarkers. In this example, the number of predicted PD-L1 positive tiles is divided by the total number of predicted tumor tiles to arrive at an exemplary model score.

[0333] The tissue class locator 1408 may have an architecture such as architecture 1200, for example. The architecture is similar to that of the FCN tile resolution classifier. The architecture 1550 may be formed from three main components: 1) a fully convolutional residual neural network (e.g., built on ResNet-18) backbone that processes wide field of view images (FOVs), 2) two branches that process small FOVs, and 3) a concatenation of small and large FOV features for multi-FOV classification. The ResNet-18 backbone contains multiple shortcut connections, shown as dotted lines, and the feature maps are also downsampled by 2x. The small FOV branch emerges after the second convolution block. The feature maps of the small FOV branch are downsampled by 8x to match the dimensions of the ResNet-18 feature maps. These feature maps are concatenated before passing through a softmax output to generate a PD-L1 biomarker prediction (confidence) map.

[0334] In one example, the model backbone includes an 18-layer version of ResNet (ResNet-18) with some modifications. The ResNet-18 backbone was converted to a fully convolutional network (FCN) by removing the global average pooling layer and eliminating zero padding in downsampled layers. This allows for the output of 2D probability maps instead of 1D probability vectors (see FIG. 15B). In the illustrated example, the tile size (466×466 pixels) is more than twice that of standard ResNet, providing the model with a large FOV from which it can learn morphological features of its surroundings. The tissue class locator 1408 in this example adds multiple fields of view (multi-FOV) capabilities to the ResNet architecture, but it should be understood that the tissue class locator 1408 may be comprised of a separate network architecture adapted to incorporate the multi-FOV approach disclosed herein.

[0335] The FCN configuration of architecture 1505 offers many advantages, including overcoming the accuracy degradation challenges traditionally associated with “very deep” neural networks (including neural networks with more than 16 convolutional layers) (see, for example, He et al., “Deep Residual Learning for Image Recognition” (2015), arXiv ID:1512.03385v1, and Simonyan et al., “Very Deep Convolution Networks for Large-Scale Image Recognition” (2014), arXiv ID:1409.1556v6). Architecture 1550 includes a stack of convolutional layers interleaved with “shortcut connections” that skip intermediate layers. These shortcut connections use past layers as reference points to guide deeper layers to learn residuals between layer outputs, rather than learning identity mappings between layers. This innovation improves convergence speed and stability during training, leading to better performance of deep networks over shallow networks.

[0336] The tissue class locator 1408 may include two additional branches with receptive fields restricted to a small FOV (32 × 32 pixels) centered on the second convolutional feature map (see Figure 15B). One branch passes a copy of the small FOV to a convolutional filter, while the other branch is a standard shortcut connection with downsampling. The features generated by these additional branches are concatenated to the features from the main backbone in a softmax layer just before the model output is converted to a probability. In this way, the tissue class locator 1408 combines information from multiple FOVs, just as a pathologist might rely on different zoom levels when diagnosing a slide. Furthermore, this architecture ensures that the central region of each tile contributes more to the classification than the edges of the tile, resulting in a more accurate classification map across the entire tissue pathology image.

[0337] FIG. 15B illustrates an exemplary training process 1570 for the imaging-based biomarker prediction system 1400 and generation of an overlay map output, where the location of the PD-L1 biomarker is predicted from analysis of IHC and H&E histopathology images. In the model training process 1570, matching areas in the IHC and H&E digital images were annotated by a medical professional. However, in some cases, staining in adjacent tissue slices may be automatically annotated. Images showing PD-L1+ and PD-L1- were annotated and used to train the system. The annotated regions of the H&E images were tiled into overlapping tiles (466×466 pixels) with a stride of 32 pixels to generate training data. The tissue class locator 1408 was then trained using a cross-entropy loss function. The yellow box in the model schematic indicates the central region that was cropped for the small FOV. The resulting PD-L1 classification model is saved in a trained deep learning framework 1574.

[0338] FIG. 15B also shows an exemplary prediction process 1572 where each image was divided into large non-overlapping 4096×4096 input windows (blue dashed lines). Each large window was passed through the trained model. Because the deep learning framework 1574 is fully convolutional, each tile in the large input window was processed in parallel to generate a probability cube of 128×128×3 (the last dimension represents the three classes). The resulting probability cubes were slotted into place and assembled to generate a probability map for the entire image. The class with the highest probability was assigned to each tile, and the PD-L1 prediction report is generated.

[0339] 16A-16F show input histopathology images received by imaging-based biomarker prediction system 1400, corresponding overlay maps generated by system 1400 to predict the location of IHC PD-L1 biomarkers, and corresponding IHC stained tissue images used as references to determine the accuracy of the overlay maps. The IHC stained tissue images were obtained from a test cohort but were not applied to system 1200 during model training. FIGs. 16A-16C show representative PD-L1 positive biomarker classification examples. FIG. 16A shows an input H&E image, FIG. 16B shows a probability map overlaid on the H&E image, and FIG. 16C shows PD-L1 IHC staining for reference. FIGs. 16D-16F show representative PD-L1 negative biomarker classification examples. FIG. 16D shows an input H&E image, FIG. 16E shows a probability map overlaid on the H&E image, and FIG. 16F shows PD-L1 IHC staining for reference. The color bar indicates the predicted probability of tumor PD-L1+ class.

[0340] Among the benefits offered by the deep learning framework herein, breaking shift invariance can improve accuracy. Shift invariance, or uniformity, is a property of linear filters, such as convolutions, whose response does not depend explicitly on position. That is, if you shift the signal, the output image is the same but for the shift applied. While shift invariance is desirable for most image classification tasks (Le Cun (1989)), in our examples it is generally undesirable for objects close to the edge of the tile to contribute equally to the classification.

[0341] FIG. 17 illustrates an exemplary benefit of a multi-FOV strategy as described with reference to FIGS. 14, 15A, and 15B. The top large FOV (red box) contains both PD-L1+ tumor cells (purple, top left) and stroma (pink). Only the stroma is contained in the small FOV (green box). After passing through the convolutional layers of the tissue class locator 1408, the tumor area generates a unique pattern (colored squares) that differs from the pattern generated by the stromal area (white squares). After the patterns from the large and small FOV branches are concatenated, the model may predict "others". In the lower part, the field of view shifts and the PD-L1+ tumor area falls within the small FOV. This tumor area generates the same convolutional filter pattern in both the large and small FOV branches (colored squares). Concatenating the learned features makes the tissue class locator 1408 more likely to predict PD-L1+ tumors. Thus, without the small FOV used to train the tissue class locator 1408, the system 1400 may have predicted PD-L1+ tumors for both images. Instead, the multi-FOV strategy of the architecture of FIG. 15 allows the network to preferentially classify what is in the center of the image while taking advantage of the rich contextual information of the surrounding areas.

[0342] Still other architectures can be used with any of the classifier examples herein to predict biomarker status, tumor status, and / or metrics thereof, particularly using multi-instance learning techniques.

[0343] In the examples discussed herein, for example, a classification model architecture based on the FCN architecture described in FIG. 12A was trained on digital images of histopathology slides that may include a matrix of annotations. Training from such digital images is performed on a tile-by-tile basis. For example, only tiles with annotations (i.e., labels) are provided to the deep learning framework as training tiles. Tiles of the digital image without annotations may be discarded. Furthermore, each column and row of the matrix corresponds to a separate grid having NxM pixels of the digital image. To properly assign annotations from a matrix with columns and rows to a digital image having multiple tiles, in some examples, it may be advantageous to take a column (i) and row (j) from the matrix and assign annotations to a central region of the grid starting at pixels N(i) and M(j) and extending to the next [N-1] to [M-1] pixels. Here, the tiles that the central region spans are assigned annotations of the i, j matrix. Thus, the matrix can accurately represent annotations for each tile in the larger digital image.

[0344] The FCN architecture can use large tiles as inputs while labels are obtained from the central region that is mapped to the annotation mask points. The FCN architecture can learn from both the central region of the large tiles and the pixels surrounding the central region, with the central region contributing further to the prediction. Additionally, slide metadata can be stored in feature vectors, such as a vector associated with slides that identify slide-level labeling. This includes patient features that may improve the model's performance. Multiple tiles from a grid of digital images and the corresponding annotation matrix are sequentially provided to the FCN architecture, and the FCN architecture is trained to classify the N×M sized tiles according to the annotations contained in the matrix and the slides themselves according to the annotations contained in the feature vector. The output of the FCN architecture may include a matrix of predicted classifications per tile, which can be aggregated into a vector of predicted classifications per slide. The matrix can be converted into a digital overlay by associating the best classification of each tile with a color that can be superimposed on the corresponding grid location in the digital image. In some examples, the matrix may be converted into multiple digital overlays, each corresponding to a respective classification, and associated color intensities are assigned to the overlays based on the percentage of confidence associated with each classification. For example, a tile with a 30% probability of being tumor, a 50% probability of being stroma, and a 20% probability of being normal may be assigned a single overlay of stroma, which is the most likely tissue within the tile, or a first overlay with a first color intensity of 30% and a second overlay with a second color intensity of 50% may be assigned to identify the type of tissue that the tile may comprise.

[0345] However, even classification models based solely on architectures that may not support per-tile annotations, such as architectures similar to Resnet-34 or Inception-v3, can be trained using only digital images of pathology slides that contain only vectors of annotations, where each entry in the vector is an annotation for a patient feature or metadata that is applied to the slide. In some instances, even architectures that support per-tile annotations may not have access to per-tile annotations for certain features.

[0346] To use pathology tissue images in training neural networks images without tile annotations or to train neural networks to identify biomarkers trained on molecular training data, in some examples, the present technology includes deep learning training architectures configured for label-free training. In examples, there are architectures that do not require tile-level annotations when training tissue classification models. Additionally, the label-free training architectures are neural network agnostic in that they enable label-free training that does not depend on the configuration of the neural network (i.e., ResNet-34, FCN, Inception-v3, UNet, etc.). The architectures can analyze a set of possible training images and predict tiles to exclude from training. Thus, in some examples, images may be discarded from training, while in other examples, tiles may be discarded but the remainder of the images may be used for training. These techniques result in significantly less training data, significantly reducing the time required for training, and in some cases, significantly reducing the time required for updating training of classification models herein. Additionally, since the technology does not require labeling by a pathologist, it significantly reduces the time it takes to train a classification model and avoids annotation errors and annotation variability between experts.

[0347] Instead, in some instances, training can be performed with weakly supervised learning, which includes only image-level labeling and does not include local labeling of tissues, cells, tumors, etc. The architecture can consist of an unlabeled training front-end with an algorithm with a customized cost function that selects tiles to use as inputs for a particular label. This process can be iterative, first treating each histopathology image as a collection of tiles, where a single label for the image is applied to all tiles in the collection. The tiles can be applied to an inference pipeline, such as through a network such as ResNet34, Inception-v3, or FCN. Also, a predefined tile selection criterion, such as the probability of the neural network output, can be used to select the output image tile that will be provided as input to the same neural network for the next round. This process can be repeated many times, and given enough collections and tiles as inputs to the neural network, it will learn to distinguish tiles of different classes with increasing accuracy as more iterations are performed.

[0348] FIG. 18 illustrates an exemplary machine learning architecture 1800 in an exemplary configuration for performing label-free annotation training of a deep learning framework to perform the processes described in the examples herein. The deep learning framework 1802 may be similar to other deep learning frameworks described herein having multi-scale and single-scale classification modules and includes a pre-processing and post-processing controller 1804, and the execution process is described in the similar examples of FIGS. 1 and 3. The deep learning framework 1802 includes a cell segmentation module 1806 and a tissue classifier module 1808, each of which is configured as a tile-based neural network classifier. The deep learning framework 1802 further includes a number of different biomarker classification models 1810, 1812, 1814, and 1816, each of which may be configured to have different neural network architectures, some of which may have multi-scale configurations and some of which may have single-scale configurations. These different neural network architectures may be configured for training using annotated or unannotated images. Some of these architectures may be configured for training using training images with tile annotations, while others of these architectures may be configured for training using training images without tile annotations. For example, some architectures may be configured to only accept slide-level annotations of images (i.e., annotations of the entire image, not annotations that identify characteristics of specific tiles, cell segmentation, or tissue segmentation). Examples of neural network architecture types for modules 1810-1816 include ResNet-34, FCN, Inception-v3, and UNet.

[0349] The annotated images 1818 may be provided to the deep learning framework 1802 for training of various classification modules using the techniques described above. In some examples, the entire histopathology image is provided to the framework 1802 for training. In some examples, the annotated images 1818 are passed directly to the deep learning framework 1802. In some examples, the annotated images 1818 may be annotated with a granularity to be reduced. Thus, in some examples, the multi-instance learning (MIL) controller 1821 may be configured to further separate the annotated images 1818 into multiple tile images, each corresponding to a different portion of the digital image, and the MIL controller 1821 applies the tile images to the deep learning framework 1802. However, in the architecture 1800, the unannotated images 1820 may be used for training of the classification modules by first providing the images 1820 to the MIL controller 1821 with the front-end tile selection controller 1822. In some examples, the MIL controller 1821 can be configured to separate the unannotated image 1820 into multiple tile images, each corresponding to a different portion of the digital image, and the MIL controller 1821 applies the tile images to the deep learning framework 1802. In one example, the architecture 1800 deploys weakly supervised learning to train a convolutional neural network architecture (e.g., FCN, ResNet34, Inception-v3, etc.) to classify local tissue regions using only slide-level labels. Since weakly supervised learning does not require localized annotations, labeling can be performed faster, resulting in a larger set of labeled slides. Thus, this architecture 1800 can be used to train models that complement or improve FCN classification, or to further train the FCN-based model itself.

[0350] In this illustrated example, the front-end tile section controller 1822 is configured with a feedback arrangement that allows the tile section process to be combined with a classification model, allowing the tile section process to be informed by a neural network architecture, such as an FCN architecture. In some examples, the tile selection process performed by the controller 1822 is a trained MIL process. For example, one of the biomarker classification models 1810-1816 can generate an output during training, which is used as an initial input to guide the tile selection controller 1822. The MIL process is typically an iterative process where initial tile selection is difficult, and by informing the MIL process of the controller 1822 with guidance from, for example, the FCN architecture predictions, the MIL process of the controller 1822 starts with better examples and converges faster to a stable and useful FCN classifier. In yet another example, combining the results from the FCN (or other neural network) architecture with the MIL process of the controller 1822 may include combining only the results of the vector outputs in the concatenation layer, such that the matrix output is obtained by voting for the best. The FCN architecture and the MIL process of the controller 1822 may configure the same prediction task (i.e., look for the same biomarkers). However, in some cases, the prediction outputs of the two classification processes may differ, in which case combining the results by voting for the best one may be performed. In yet another example, the output from the MIL framework and the FCN architecture can be combined to obtain better slide-level predictions. In this case, the MIL uses a slide-level loss function as a learning criterion and uses the output from the FCN architecture to derive a guided truth for the MIL loss calculation.

[0351] The tile section controller 1822 can be implemented in a number of different ways.

[0352] An example of the tile selection process of the controller 1822 is described with reference to the single-class basic framework. In the single-class example, tiles of a histopathology image are classified as belonging to a target class (class 1) or not (class 0). Class 0 can be considered background and not in the target class. In the instance-based MIL process, tiles must be selected to be used as examples to be used in training. For a single-class problem, the controller 1822 can be configured with a trained model that returns a classification such that if a slide has no tiles that belong to the target class, all tiles should return a low inference score of zero and during training, the slide is labeled as class 0, and if a slide has tiles that belong to the target class, the slide is labeled as class 1 and tiles that belong to the target class should return a high inference score of 1 and all other tiles should return a low inference score of 0. To train a classifier model (e.g., models 1810-1816), tiles that represent slide-level classes must be identified. Since it is known that class 0 slides do not have tiles of class 1, any tile can be used as a training example. However, the best choice is to use the tile with the poorest model performance. Since all tiles should have an inference score of 0, the tiles with the highest scores should be used to train the model. These highest scoring tiles are referred to as the "top k" tiles, where k is an integer (e.g., 5, 10, 15) that indicates the number of tiles from that slide that are being used for training.

[0353] For class 1 slides, the controller 1822 may identify tiles that are most likely to be of class 1. This decision may be complicated because class 1 slides may contain tiles of both class 0 and class 1. However, for a classifier model trained on class 0 tiles of class 0 slides, this means that tiles in class 1 slides that are similar to class 0 tiles will have a lower inference score. Similarly, this means that tiles that are not similar to class 0 tiles should have a higher score. Thus, the tiles that are most likely to be in fact class 1 are the tiles with the highest inference scores. Thus, the top k tiles need to be selected again as examples for training the model.

[0354] This means that the top k scored tiles for both class 0 and class 1 slides should be used as training examples, as seen in the tile selection framework 1900 in Figure 19. The rows of numbers represent the inference scores for the various tiles. First, an annotated histopathology image with no tiles is provided in 1902, and model inference is performed in process 1904 for each of the tiles in the image.

[0355] The framework 1900 may be used to compute class prediction scores for all tiles of all slides by performing model inference 1904, and the tiles with the highest scores from each slide (the top k tiles labeled 1906) are selected to train the model. The tiles 1906 are given the same label as the slide-level label received in 1902 (e.g., the tile for a slide of class 0 is given a label of class 0). After selecting the tiles from all slides, the model is trained for a single epoch (or iteration) in process 1908.

[0356] At the end of a training epoch (where the tile was used once to update the model weights), new prediction scores for all tiles for all slides are computed, again using framework 1900. The new prediction scores are used to identify new tiles to use for training. This process of scoring tiles and selecting the top-k tiles for training is repeated until the model reaches a stopping criterion (e.g., convergence of performance on the set of held out slides used for validation), as determined by process 1910.

[0357] There are multiple advantages to using a weakly supervised learning configuration such as 1800 in FIG. 18. Strongly supervised annotation and local annotation, such as having a pathologist manually mark examples of tissue classes across images, are costly and inefficient. Architecture 1800 can train models using a single label on datasets with orders of magnitude more slides. Furthermore, one may have classification targets that cannot be annotated. For example, genetic mutations currently discovered by genotypic biomarkers may correlate with tissue features, but it was previously unclear what these tissue features were. However, with the present technology, genotypic biomarkers can be used as slide-level labels, and the training framework of architecture 1800 can be used to identify tissue morphologies that can be used to predict the genotype present. While traditional RNA / DNA analysis to predict genotypes can take weeks, image classification to predict genotypes using the present technology can be performed in hours and on a much larger training set.

[0358] To avoid overfitting situations, in some examples, the tile selection controller 1822 can be configured to perform randomized tile selection. For example, in case of training with a small dataset (<300 slide images), a framework such as the framework 1900 of FIG. 19 may overfit some tiles of slides of class 1. This situation can occur because if a tile of class 1 is used for training, its score will be higher in the next epoch. Therefore, the tile of class 1 will likely be selected again for training in the next epoch, which will further increase the inference score and make it more likely to be selected again.

[0359] FIG. 20 illustrates a framework 2000 that may be used to avoid overfitting. Framework 2000 is similar to framework 1900 and has similar reference numbers, but uses a random tile selector 2012 to randomly select among the high-scoring tiles, and the randomly selected tiles are then sent to a training process 2008. Model 2004 continues to be used to determine the inference scores, and tiles are selected based on the scores. However, if a tile has a high inference score, it may be considered more likely to be a tile of class 1, even if it is not one of the top k tiles. For example, framework 2000 may set a low threshold score of 0.9 (or any value), and then any tile with a score of 0.9 or greater may be used as a training example. That is, any of tiles 2006 may be used for training because their scores exceed a determined threshold (e.g., threshold 0.9). Random high-scoring tile selector 2012 then randomly determines which of these tiles are provided to model training process 2008. In other examples, the tiles sent to the random high scoring tile selector 2012 are the top k tiles. Additionally, in some examples, the tile selection probabilities applied by the selector 2012 may be completely random across all tile scores, while in other examples the selection probabilities may be partially random, with tiles with particular scores or within particular score ranges having a different random selection probability than tiles with other scores or within other score ranges.

[0360] FIG. 21 shows another framework 2100 for dealing with overfitting situations. In some examples, when training on a smaller dataset (<300 slide images), the framework of FIG. 19 may overfit slide images by predicting all tiles of class 0 slides as class 0 and all tiles of class 1 slides as class 1 (although not all tiles of class 1 slides are actually class 1). This means that tiles of class 1 slides may be misclassified. However, in framework 2100, random tile selection is performed on high-scoring tiles 2101, as in framework 2000, but additionally, random tile selection is performed on low-scoring tiles 2103. In the illustrated example, random low-scoring tile selector 2102 feeds a class 0 model training process 2104. Random high-scoring tile selector 2106 is provided to a class 1 model training process 2108.

[0361] The examples in Figures 19-26 are described in the context of single-class training. The label-free training of the present technique can also be used for multi-class training. For multi-class problems, a set of slide-level labels can be set such that there are no slides with class 0 labels. For example, when training a model to predict consensus molecular subtypes (CMS), in colorectal cancer (CRC), the CMS classes are used to guide targeted therapy, but only genotypic biomarkers, i.e., mutations in RNA data, are used. However, the present technique predicts genotypes through imaging, which allows targeted therapy to be initiated and tested in patients in hours, rather than having to wait weeks to perform RNA analysis. While an example is described in reference CMS, classifier modules for other image-based biomarkers herein can be trained with architecture 1800, including PD-L1, TMB, etc.

[0362] One problem with multi-class training is that not all images contain all classes. For example, in the case of CMS, the available labels for training images are CMS1, CMS2, CMS3, or CMS4. However, not all tiles in each image contain biomarkers indicative of one of these four classes. This can lead to situations where there are no class 0 slides that could be used to identify class 0 tiles, leading to the trained model misclassifying tissue types that have no predictive value, reducing the accuracy of the model. Additionally, slides may contain features that represent classes other than the slide-level labels. For example, in CMS, there may be two or more subtypes of CRC. This means that a particular image may be labeled CMS1 because it is the predominant subtype in that sample, but the whole-slide image may contain CMS2 tissue.

[0363] To achieve multi-class training, in some examples, the tile selection controller 1822 of FIG. 18 is configured to execute a series of processes to identify tissue features that correlate with different class subtypes, e.g., the four CMS subtypes. First, the tile selection controller 1822 may be trained by identifying only positive in-class examples. Second, the tile selection controller 1822 may apply model training by identifying positive in-class tiles and negative out-of-class tiles with low positive in-class scores. Third, the tile selection controller 1822 may apply model training by identifying positive in-class tiles and negative out-of-class tiles with high negative scores. An example of each process will now be described with reference to training a CMS biomarker classifier, as shown in FIGS. 22-26.

[0364] FIG. 22 illustrates a framework 2200 that may be used to identify the CMS class with which each tissue feature is most correlated, regardless of whether that tissue feature is relevant for classification. FIG. 22 illustrates an example of the first process, which identifies only examples within the positive class and shows only two classes for simplicity. For each tile of the histopathology image, a list of possible scores is shown, with each row corresponding to a class (0, 1, and 2) shown on the left. As shown, for class 1 slide images (left side of FIG. 22), tiles with high class 1 scores (shaded) are used for training, and for class 2 slide images (right side of FIG. 22), tiles with high class 2 scores (shaded) are used for training.

[0365] In this example, all tiles are classified as 1 or 2, with a probability of being classified as class 0 of 0.00. Figure 23 shows the resulting overlay map showing the classification of CMS biomarkers with four classes. The tiles are color-coded based on the inference scores of the four CMS classes, with CMS1 (microsatellite instability immune) shown in red, CMS2 (epithelial gene expression profile, WNT and MYC signaling activation) shown in green, CMS3 (epithelial profile with evident metabolic dysregulation) shown in dark blue, and CMS4 (mesenchymal, prominent transforming growth factor-β activation) shown in light blue. In the illustrated example, the transparency of the tiles is adjusted to indicate the inference score, with higher scores being more opaque and lower scores being more transparent.

[0366] FIG. 24 shows the framework 2200 applied in the second process. That is, model training continues by identifying positive in-class tiles and negative out-of-class tiles with low positive in-class scores. The first process in FIG. 22 classifies all tiles as one of the non-zero classes, while the second process in FIG. 24 identifies tiles that are likely background class 0. In the illustrated example, the process does this by identifying tiles with scores below a slide level class threshold. If a tile of a class 1 slide image has a score below 0.1, the low score is marked as a slide image that can be used as an example of class 0. A similar process is performed for class 2 slide images. FIG. 25 shows the resulting overlay map showing the CMS classification. If a tile is predicted to be class 0, it becomes transparent. Compared to FIG. 23, in FIG. 25 this second process identifies some tissue types as class 0. This means that the image is background tissue that does not correlate with any of the CMS classes. At the same time, the second process can identify various tissue types or tissue features associated with each of the four classes. The tiles are colored based on the inference scores of the four CMS classes as in Figure 23, except that tiles that do not show a color are predicted to be class 0 tiles with no class prediction value.

[0367] FIG. 26 illustrates the framework 2200 applied in the third process, i.e., continuing model training by identifying positive in-class tiles and negative out-of-class tiles by high negative scores. As mentioned above, for CMS biomarkers, the CMS classes of training images are not mutually exclusive. A slide image of CMS1 can contain tissue type or tissue features of CMS2. Thus, in some examples, if the tile selection controller 1822 continues to label tiles as class 0 due to low slide level class scores, some tiles may be mislabeled. From the second process above, class 0 tissues have already been identified that have low or no correlation with any of the CMS classes. There are tiles with high class 0 scores. Thus, in the third process shown in FIG. 26, class 0 tiles may be identified based on having high class 0 scores.

[0368] Returning to FIG. 18, the architecture 1800 can be used to train any of the biomarker classification models 1810-1816 to classify different biomarkers. CMS is described with reference to FIGS. 22-26 as an example. Furthermore, the architecture 1800 may be agnostic to the convolutional neural network configuration, i.e., each module 1810-1816 may have the same or different configuration. In addition to the FCN architecture of FIGS. 10A-10C, the modules 1810-1816 may be configured with a ResNet architecture as shown in FIG. 27. The ResNet architecture provides skip connections, which help avoid the vanishing gradient problem during training as well as the degradation problem of large architectures. Pre-trained ResNet models of various sizes can be used to initialize the training of new models, including but not limited to ResNet-18 and ResNet-34. In addition to ResNet, architecture 1800 can be used to train neural networks with convolutional layers that create feature maps that are input into a fully connected classification layer, such as AlexNet or VGG; networks with layered modules that apply multiple convolution kernels simultaneously, such as Inception v3; networks designed to require fewer parameters to increase processing speed, such as MobileNet, SqueezeNet, and MNASNet; or customized architectures designed to better extract relevant pathological features, either designed manually or using neural architecture search networks such as NASNet.

[0369] By using a tile-based training architecture 1800 with a tile selection controller that is independent of the neural network configuration, a deep learning framework can be created that can identify biomarkers where a single architecture (e.g., FCN) alone is less accurate or requires a more timely process.

[0370] Additionally, because the tile-based training architecture 1800 has a feedback configuration, in some examples, the deep learning framework 1802 can classify regions in training images, for example, using an FCN architecture, and feed those classified images back to the tile selection controller 1822 as weakly supervised training images. For example, an FCN architecture can be used to first identify specific tissue regions (e.g., tumor, stroma), which can then be used as input to a weakly supervised training pipeline, such as MIL. Models trained by the weakly supervised pipeline can be used in combination with the FCN architecture to validate or improve the results of the FCN architecture, or to detect new features to complement the FCN architecture.

[0371] Additionally, there exist biomarkers that are not possible to annotate, e.g., previously undiscovered tissue features that correlate with genotype, gene expression, or patient metadata. In such cases, the genotype, gene expression, or patient metadata can be used to create slide-level labels that are used to train the secondary model or the FCN architecture itself, using a weakly supervised framework to discover novel classifications.

[0372] Additionally, architecture 1800 can provide for the detection of tissue and tissue artifacts. Regions in a slide image containing tissue can be first detected for use as input to the FCN architecture model. Imaging techniques such as color or texture thresholding can be used to identify tissue regions, and the deep learning convolutional neural network model (e.g., FCN architecture) herein can be used to further improve the generalizability and accuracy of tissue detection. Color or texture thresholding can also be used to identify spurious artifacts in tissue images, and weakly supervised deep learning techniques can be used to improve generalizability and accuracy.

[0373] Additionally, architecture 1800 can provide marker detection. Histopathology images may contain notations or annotations drawn by a pathologist with a marker on the slide. For example, this indicates macro-anatomical regions where tissue DNA / RNA analysis should be performed. A marker detection model similar to the tissue detection model can be used to identify the regions selected by the pathologist for analysis. This further complements the data processing with weakly supervised training to isolate regions where DNA / RNA analysis was performed resulting in slide-level labels.

[0374] In FIG. 28, a process 2800 is provided for determining a proposed immunotherapy treatment for a patient using the biomarker predictions of the imaging-based biomarker prediction system 102 of FIG. 1, and in particular the deep learning framework 300 of FIG. 3. Initially, a tissue pathology image, such as a stained H&E image, is received by the system 102 (2802). In process 2804, each tissue pathology image is applied to a trained deep learning framework, such as one that implements one or more FCN classification configurations described herein. In process 2806, the trained deep learning framework applies the image to a trained tissue classifier model and a trained biomarker segmentation model to determine the status of biomarkers in tissue regions of the image. In some examples, the trained cell segmentation classifier model is further used by process 2806. In process 2806, a biomarker status and a biomarker metric for the image are generated. 29, the output from process 2806 is provided to process 2808, which may be implemented in a tumor treatment decision system 2900 (which may be part of a genomic sequencing system, an oncology system, a chemotherapy decision system, an immunotherapy decision system, or other treatment decision system) that determines the tumor type based on the received data, including based on biomarker metrics, genomic sequencing data, etc. The system 2900 analyzes the biomarker status and / or biomarker metrics as well as other received molecular data against available immunotherapies 2902 in process 2810, and the system 2900 recommends a matched list of possible tumor type specific immunotherapies 2904, filtered from the list of available immunotherapies 2902, in the form of a corresponding treatment report.

[0375] In various examples, the imaging-based biomarker prediction system herein may be deployed partially or entirely within a dedicated slide imager, such as a high-throughput digital scanner. FIG. 30 illustrates an exemplary system 3000 having a dedicated ultra-high-speed pathology (slide) scanner system 3002, such as the Philips IntelliSite Pathology Solution available from Koninklijke Philips, Amsterdam, The Netherlands. In some examples, the pathology scanner system 3002 may include multiple trained biomarker classification models. Exemplary models may include, for example, models disclosed in U.S. Patent Application Serial No. 16 / 412,362. The scanner system 3002 is coupled to an imaging-based biomarker prediction system 3004 to implement the processes described and illustrated in the examples herein. For example, in the illustrated example, the system 3004 includes a deep learning framework 3006 based on a tile-based multi-scale and / or single-scale classification module according to examples herein, having one or more trained biomarker classifiers 3008, trained cell classifiers 3010, and trained tissue classifiers 3012. The deep learning framework 3006 performs biomarker and tumor classification on the histopathology images and stores the classification data as overlay data with the original images in a generated image database 3014. For example, the images can be stored as TIFF files. However, the database 3014 may include JSON files and other data generated by the classification process herein. In some examples, the deep learning framework may be integrated in whole or in part within the scanner 3002, as shown in optional block 3015.

[0376] To manage the generated images, which can be quite large, an image management system and viewer generator 3016 is provided. In the illustrated example, the system 3016 is shown as being external to the imaging-based biomarker prediction system 3004, connected by a private or public network. However, in other examples, all or part of the system 3016 can be deployed in the system 3004, as shown at 3019. In some examples, the system 3016 is cloud-based and stores the generated images from (or instead of) the database 3014. In some examples, the system 3016 generates a web-accessible cloud-based viewer that allows a pathologist to access, view, and manipulate the histopathology images with various classification overlays via a graphic user interface. Examples are shown in FIGS. 31-37.

[0377] In some examples, the image management system 3016 manages the receipt of scanned slide images 3018 from the scanner 3002 , which are generated from an imager 3020 .

[0378] In the illustrated example, the image management system 3016 generates an executable viewer app 3024 and deploys the app 3024 to an app deployment engine 3022 of the scanner 3002. The app deployment engine 3022 can provide functionality such as GUI generation to allow a user to interact with the viewer app 3024, an app marketplace to allow a user to download the viewer app 3024 from the image management system 3016 or other network accessible sources.

[0379] 31-37, in one example, show various digital screenshots generated by the embedded viewer 3024, which are displayed in a GUI format that allows a user to manipulate the displayed images to zoom in and out and to view various classifications of tissues, cells, biomarkers, and / or tumors.

[0380] Referring to FIG. 31, there is shown a GUI generated display 3100 having a panel 3102 showing an entire histopathology image 3104 and a magnified portion of that image 3104 (1.3x magnification) displayed as a window 3106. The panel 3102 further includes a magnification corresponding to the window 3106 and a tumor content report. FIG. 32 shows the display 3100 but at a 3.0x magnification after the user has zoomed in on the window 3106. FIG. 33 is similar but at a 5.7x magnification. FIG. 34 shows a drop down menu 3108 listing a series of classifications that the user may select to generate a classification overlay map that is displayed on the display 3100. FIG. 35 shows the resulting display 3100 with an overlay map showing tumor classified tissue, in this example showing the tissue divided into tiles and tiles with classifications. In the example of FIG. 35, the classification shown is a tumor classification. Figure 36 shows another exemplary classification overlay mapping, which is one of cell classification, epithelial, immune, stromal, tumor, or other. Figure 37 shows a magnified cell classification overlay mapping showing that the classification can actually be displayed at sufficient magnification to distinguish different cells in a histopathology image.

[0381] FIG. 38 illustrates an exemplary computing device 3800 for implementing the imaging-based biomarker prediction system 100 of FIG. 1. As illustrated, the system 100 can be implemented on the computing device 3800, particularly on one or more processing units 3810, which may represent central processing units (CPUs), and / or on one or more graphics processing units (GPUs) 3811, including clusters of CPUs and / or GPUs, and / or on one or more tensor processing units (TPUs) (also labeled 3811), any of which may be cloud-based. Features and functionality described for the system 100 can be stored on and implemented from one or more non-transitory computer-readable media 3812 of the computing device 3800. The computer-readable media 3812 can include, for example, an operating system 3814, and a deep learning framework 3816 having elements corresponding to elements of the deep learning framework 300, including the pre-processing controller 302, the classifier modules 304 and 306, and the post-processing controller 308. More generally, the computer readable medium 3812 can store trained deep learning models, executable code, etc. used to implement the techniques herein. The computer readable medium 3812 and processing unit 3810 and TPU / GPU 3811 can store image data, tissue classification data, cell segmentation data, leukocyte segmentation data, TIL metrics, and other data in one or more databases 3813 herein. The computing device 3800 includes a network interface 3824 communicatively coupled to a network 3850 for communicating to and / or from a portable personal computer, a smartphone, an electronic document, a tablet, and / or a desktop personal computer, or other computing device. The computing device further includes an I / O interface 3826 connected to devices such as a digital display 3828, a user input device 3830, etc.In some examples, the computing device 3800 generates biomarker predictions as electronic documents 3815 that can be accessed and / or shared over a network 3850, as described herein. In the illustrated example, the system 100 is implemented on a single server 3800. However, the functionality of the system 100 may be implemented across distributed devices 3800, 3802, 3804, etc., connected to each other via communication links. In other examples, the functionality of the system 100 may be distributed across any number of devices, including the illustrated portable personal computers, smartphones, electronic documents, tablets, and desktop personal computing devices. In other examples, the functionality of the system 100 may be cloud-based, such as, for example, one or more connected cloud TPUs customized to perform machine learning processes. The network 3850 may be a public network, such as the Internet, a private network, such as a research institute's or a company's private network, or any combination thereof. The network may include a local area network (LAN), a wide area network (WAN), cellular, satellite, or other network infrastructure, whether wireless or wired. A network may utilize communication protocols including packet-based and / or datagram-based protocols, such as Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), or other types of protocols. Additionally, a network may include a number of devices that facilitate network communications and / or form the hardware infrastructure of the network, such as switches, routers, gateways, access points (such as wireless access points as shown), firewalls, base stations, repeaters, backbone devices, etc.

[0382] The computer-readable medium may include executable computer-readable code stored on a computer (including, for example, a processor and a GPU) for programming a computer to the techniques of the present specification. Examples of such computer-readable storage media include hard disks, CD-ROMs, digital versatile disks (DVDs), optical storage devices, magnetic storage devices, ROMs (read-only memories), PROMs (programmable read-only memories), EPROMs (erasable programmable read-only memories), EEPROMs (electrically erasable programmable read-only memories), and flash memories. More generally, the processing unit of the computing device 1300 may represent a CPU-type processing unit, a GPU-type processing unit, a TPU-type processing unit, a field programmable gate array (FPGA), another class of digital signal processor (DSP), or other hardware logic components that can be driven by a CPU.

[0383] Although the exemplary deep learning framework herein has been described as being configured with an exemplary machine learning architecture (FCN configuration), it should be noted that any number of suitable convolutional neural network architectures can be used. Generally speaking, the deep learning framework herein can implement any suitable statistical model (e.g., a neural network or other model implemented through a machine learning process) that is applied to each of the received images. As discussed herein, the statistical model can be implemented in a wide variety of ways. In some examples, a classifier is developed that uses machine learning to evaluate training images and correlates predefined image features to specific categories of TIL status. In some examples, image features can be identified as training classifiers using learning algorithms such as neural networks, support vector machines (SVMs), or other machine learning processes. Once the classifiers in the statistical model are properly trained with a set of training images, the statistical model can be used in real time to analyze subsequent images that are provided as inputs to the statistical model to predict the status of the biomarkers. In some examples, when the statistical model is implemented using a neural network, the neural network can be configured in a wide variety of ways. In some examples, the neural network may be a deep neural network and / or a convolutional neural network. In some examples, the neural network may be a distributed and scalable neural network. The neural network may be customized in a wide variety of ways, such as providing a specific top layer, such as a logistic regression top layer. A convolutional neural network may be viewed as a neural network that includes a set of nodes with associated parameters. A deep convolutional neural network may be viewed as having a structure in which multiple layers are stacked. Neural networks or other machine learning processes may include various sizes, numbers of layers, and levels of connectivity. Some layers may correspond to stacked convolutional layers (optionally followed by contrast normalization and max pooling) followed by one or more fully connected layers.For neural networks trained with large datasets, the number of layers and layer size can be increased by using dropout to address the potential problem of overfitting. In some cases, neural networks can be designed to forego the use of fully connected upper layers at the top of the network. By forcing the network to reduce the dimensionality of the intermediate layers, very deep neural network models can be designed while dramatically reducing the number of learned parameters.

[0384] A system for performing the methods described herein may include a computing device, and more specifically may be implemented in one or more processing units, e.g., a central processing unit (CPU), and / or one or more graphics processing units (GPUs), including a cluster of CPUs and / or GPUs. The features and functions described may be stored on and implemented from one or more non-transitory computer-readable media of the computing device. The computer-readable media may include, for example, an operating system and software modules, or "engines," that implement the methods described herein. More generally, the computer-readable media may store batch normalization process instructions of the engine for implementing the techniques herein. The computing device may be a distributed computing system, such as an Amazon Web Services cloud computing solution.

[0385] The computing device includes a network interface communicatively coupled to a network for communicating to and / or from a portable personal computer, a smart phone, an electronic document, a tablet, and / or a desktop personal computer, or other computing device. The computing device further includes an I / O interface connected to devices such as a digital display, a user input device, etc.

[0386] The engine's functionality may be implemented in distributed computing devices, etc., interconnected via communication links. In other examples, the system's functionality may be distributed across any number of devices, including the illustrated portable personal computer, smartphone, electronic document, tablet, and desktop personal computing devices. The computing devices may be communicatively coupled to a network and another network. The network may be a public network, such as the Internet, a private network, such as a research or corporate network, or any combination thereof. The network may include a local area network (LAN), a wide area network (WAN), a cellular, satellite, or other network infrastructure, whether wireless or wired. The network may utilize communication protocols, including packet-based and / or datagram-based protocols, such as the Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), or other types of protocols. Additionally, the network may include multiple devices, such as switches, routers, gateways, access points (such as wireless access points as illustrated), firewalls, base stations, repeaters, backbone devices, etc., that facilitate network communication and / or form the hardware foundation of the network.

[0387] The computer-readable medium may include executable computer-readable code stored on a computer (including, for example, a processor and a GPU) for programming a computer to the technology herein. Examples of such computer-readable storage media include hard disks, CD-ROMs, digital versatile disks (DVDs), optical storage devices, magnetic storage devices, ROMs (read-only memories), PROMs (programmable read-only memories), EPROMs (erasable programmable read-only memories), EEPROMs (electrically erasable programmable read-only memories), and flash memories. More generally, the processing unit of a computing device may represent a CPU-type processing unit, a GPU-type processing unit, a field programmable gate array (FPGA), another class of digital signal processor (DSP), or other hardware logic components that can be driven by a CPU.

[0388] Throughout this specification, multiple instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods have been illustrated and described as separate operations, one or more of the individual operations may be performed simultaneously, and the operations need not be performed in the order illustrated. Structures and functions presented as separate components in an example configuration may be implemented as a combined structure or component. Similarly, structures and functions presented as a single component may be implemented as a separate component or multiple components.

[0389] Further, certain embodiments are described herein as including logic or a number of routines, subroutines, applications, or instructions. These may constitute either software (e.g., code embodied on a machine-readable medium or in a transmission signal) or hardware. In hardware, routines, etc., are tangible units capable of performing certain operations and may be configured or arranged in a particular way. In an exemplary embodiment, one or more computer systems (e.g., standalone, client, or server computer systems) or one or more hardware modules of a computer system (e.g., a processor or processors) may be configured as hardware modules that operate by software (e.g., an application or portion of an application) to perform certain operations described herein.

[0390] In various embodiments, a hardware module can be implemented mechanically or electronically. For example, a hardware module can include dedicated circuitry or logic (e.g., a microcontroller, a special-purpose processor such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC)) that is permanently configured to perform certain operations. A hardware module can also include programmable logic or circuitry (e.g., contained within a general-purpose processor or other programmable processor) that is temporarily configured by software to perform certain operations. It will be appreciated that the decision as to whether a hardware module is implemented mechanically, with dedicated and permanently configured circuitry, or with temporarily configured circuitry (e.g., configured by software) can be determined based on cost and time considerations.

[0391] Thus, the term "hardware module" should be understood to encompass tangible entities, that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. Considering embodiments in which the hardware modules are temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any one instance of time. For example, if the hardware modules include a general-purpose processor configured using software, the general-purpose processor may be configured as different hardware modules at different times. Thus, the software may configure the processor, for example, to configure a particular hardware module at one time and another hardware module at another time.

[0392] Hardware modules can provide information to and receive information from other hardware modules. Thus, the described hardware modules can be considered to be communicatively coupled. When multiple such hardware modules are present simultaneously, communication can be achieved via signal transmission (e.g., via appropriate circuits and buses) connecting the hardware modules. In embodiments in which multiple hardware modules are configured or instantiated at different times, communication between such hardware modules can be achieved, for example, via storage and retrieval of information in a memory structure accessed by the multiple hardware modules. For example, a hardware module can perform an operation and store the output of that operation in a memory device to which the hardware module is communicatively coupled. An additional hardware module can then later access the memory device to retrieve and process the stored output. Hardware modules can also initiate communication with input or output devices to operate on resources (e.g., collect information).

[0393] Various operations of the example methods described herein may be performed, at least in part, by one or more processors that are temporarily (e.g., by software) or permanently configured to perform the associated operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. Modules referred to herein may, in some example embodiments, include processor-implemented modules.

[0394] Similarly, methods or routines described herein may be at least partially processor-implemented. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented hardware modules. Certain performance of an operation may reside not only in a single machine, but also be distributed among one or more processors deployed across several machines. In some embodiments, one or more processors may reside in a single location (e.g., in a home environment, in a work environment, or as a server farm), while in other embodiments, the processors may be distributed across multiple locations.

[0395] Certain performance aspects of an operation may reside not only within a single machine, but may also be distributed among one or more processors deployed across several machines. In some exemplary embodiments, one or more processors or processor-implemented modules may reside in a single location (e.g., in a home environment, in a work environment, or as a server farm). In other exemplary embodiments, one or more processors or processor-implemented modules may be distributed across multiple locations.

[0396] Unless otherwise indicated, descriptions herein using words such as "processing," "computing," "calculating," "determining," "presenting," and "displaying" may refer to machine (e.g., computer) operations or processes that manipulate or transform data represented as physical (e.g., electronic, magnetic, or optical) quantities in one or more memories (e.g., volatile memory, nonvolatile memory, or a combination thereof), registers, or other mechanical components that receive, store, transmit, or display information.

[0397] As used herein, any reference to "one embodiment" or "an embodiment" means that a particular element, feature, structure or characteristic described in connection with an embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

[0398] Some embodiments may be described using the terms "coupled" and "connected," along with their derivatives. For example, some embodiments may be described using the term "coupled" to indicate that two or more elements are in direct physical or electrical contact. However, the term "coupled" can also mean that two or more elements are not in direct contact with each other, but still co-operate or interact with each other. The embodiments are not limited in this context.

[0399] As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," or any other variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Furthermore, unless expressly stated to the contrary, "or" means an inclusive or, not an exclusive or. For example, a condition A or B is satisfied by any one of A being true (or present) and B being false (or absent), A being false (or absent) and B being true (or present), and both A and B being true (or present).

[0400] In addition, the use of "a" or "an" is used to describe elements and components of the embodiments herein. This is done merely for convenience and to give a general sense of the description. This description should be read to include one or at least one, and the singular also includes the plural unless it is clear that otherwise is meant.

[0401] This detailed description should be construed as merely exemplary and does not describe every possible embodiment, as describing every possible embodiment would be impractical, if not impossible. Many alternative embodiments may be implemented using either current technology or technology developed after the filing date of this application.

[0402] [1] 1. A computer-implemented method for identifying biomarkers in digital images of hematoxylin and eosin (H&E) stained slides of a target tissue, the method comprising: receiving the digital image into an image-based biomarker prediction system having one or more processors; using the one or more processors, performing an image tiling process on the digital image by separating the digital image into a plurality of tile images, each of the plurality of tile images comprising a different portion of the digital image; using the one or more processors to apply the plurality of tile images to a multi-scale deep learning framework including one or more trained deep learning multi-scale classifier models, each trained to classify a different tissue classification for each tile image, and using the multi-scale deep learning framework to determine a tissue classification for each of the plurality of tile images; using the one or more processors to identify cells within the digital image using a trained cell segmentation model; and and identifying a predicted presence of one or more biomarkers associated with the digital image from the tissue classification determined for each tile image and from the identified cells within the digital image. [2] 2. The method of claim 1, wherein performing the image tiling process on the digital image comprises applying a tiling mask to the digital image to separate the digital image into the plurality of tile images. [3] 3. The method of claim 2, wherein the tiling mask comprises tiles of equal size. [4] 3. The method of claim 2, wherein the tiling mask comprises tiles of different sizes. [5] 3. The method of claim 2, wherein the tiling mask comprises tiles having a rectangular shape. [6] 3. The method of claim 2, wherein the tiling mask comprises tiles characterized by the topology and / or morphology of pixels or groups of pixels. [7] Receiving the digital image using the one or more processors, acquiring the digital image at a first image resolution; downsampling the digital image to a second image resolution; performing a luminance normalization on the pixels of the digital image; and removing non-tissue objects from the digital image. [8] said one or more trained deep learning multi-scale classifier models; receiving, in the multi-scale deep learning framework, a plurality of H&E slide training images from a training image dataset, each H&E slide training image having a label corresponding to a biomarker to be trained; performing a tile-based tissue classification analysis on each of the H&E slide training images; performing a pixel-based cell segmentation analysis on each of the H&E slide training images; Optionally, performing a tile-based biomarker classification analysis on each of said H&E slide training images; 2. The method of claim 1, further comprising: generating the one or more trained deep learning multi-scale classifier models accordingly; and training the one or more trained deep learning multi-scale classifier models accordingly. [9] 9. The method of claim 8, wherein each H&E slide training image includes a plurality of tile images, each having a tile-level label.

[10] 9. The method of claim 8, further comprising, for each H&E slide training image, labeling each of a plurality of tile images of the H&E slide training image at a tile level.

[11] performing, for each H&E slide training image, a tile selection process that infers a class status for each tile image in the H&E slide training image; 9. The method of claim 8, further comprising discarding tile images that do not correspond to a class of interest before performing the tile-based tissue classification analysis on each of the H&E slide training images based on the inferred class status, thereby performing the tile-based tissue classification analysis only on selected tile images of the H&E slide training images.

[12] said one or more trained deep learning multi-scale classifier models; receiving a molecular training dataset for a plurality of training tissue samples, the molecular training dataset including RNA transcriptome counts from sequencing of substantially similar samples associated with each training tissue sample; performing a clustering process on the molecular training data set to identify one or more molecular data subsets, each corresponding to a distinct biomarker; for each of the one or more molecular data subsets, identifying a plurality of digital images of H&E stained training slides of training tissue samples corresponding to the respective biomarkers for an image-based biomarker prediction system having one or more processors; performing a tile-based tissue classification analysis on each of the H&E slide training images; performing a pixel-based cell segmentation analysis on each of the H&E slide training images; Optionally, performing a tile-based biomarker classification analysis on each of said H&E slide training images; 2. The method of claim 1, further comprising: generating the one or more trained deep learning multi-scale classifier models accordingly; and training the one or more trained deep learning multi-scale classifier models accordingly.

[13] Item 2. The method of item 1, wherein each of the one or more trained deep learning multi-scale classifier models is configured as a tile resolution fully convolutional network (FCN) classification model.

[14] 2. The method of claim 1, wherein the trained deep learning multi-scale classifier model is trained to classify tissues as corresponding to a classification selected from the group consisting of tumor, stromal, normal, lymphocyte, fat, muscle, vascular, immune cluster, necrosis, hyperplasia / dysplasia, and red blood cell, respectively.

[15] identifying cells within the digital image tiles using the trained cell segmentation model; 2. The method of claim 1, further comprising using the one or more processors to apply each of the plurality of tile images to the cell segmentation model and, for each tile, assigning a cell classification to one or more pixels in the tile image.

[16] assigning the cell classification to one or more pixels in the tile image; 16. The method of claim 15, comprising using the one or more processors to identify the one or more pixels as a cell interior, a cell boundary, or a cell exterior, and classifying the one or more pixels as the cell interior, the cell boundary, or the cell exterior.

[17] Item 2. The method of item 1, wherein the trained cell segmentation model is a pixel-resolution three-dimensional UNet classification model trained to classify cell interiors, cell boundaries, and cell exteriors.

[18] identifying cells within the digital image tiles using the trained cell segmentation model; applying, using the one or more processors, each of the plurality of tile images to the cell segmentation model; and performing alignment of each segmented cell of the tile images by using the one or more processors to determine a cell boundary of each cell, determine a center of gravity of each cell, and shift the coordinates of the center of gravity into a universal coordinate space of the digital image.

[19] Item 14. The method of item 1, wherein the trained cell segmentation model is trained using a set of annotated H&E slide training images that identify cell boundaries, cell interiors, and cell exteriors.

[20] 2. The method of claim 1, wherein the digital image is an unlabeled digital image or a slide-level labeled image. [twenty one] 2. The method of claim 1, wherein the digital image is a tile-level labeled image. [twenty two] 2. The method of claim 1, wherein the one or more biomarkers are selected from the group consisting of tumor infiltrating lymphocytes (TIL), nuclear to cytoplasmic (NC) ratio, ploidy, signet ring morphology, and programmed death-ligand 1 (PD-L1). [twenty three] wherein the one or more biomarkers are TILs, and the method further comprises: 2. The method of claim 1, further comprising: using the one or more processors to identify lymphocyte cells in the digital image using the lymphocyte segmentation model by integrating, using the one or more processors, the cell boundaries identified using the cell segmentation model with the lymphocytes identified using a trained lymphocyte segmentation model, and generating, using the one or more processors, a nested classification of each cell. [twenty four] wherein the one or more biomarkers are TILs, and the method further comprises: 2. The method of claim 1, further comprising using the one or more processors to identify lymphocyte cells in the digital image using a trained lymphocyte segmentation model by applying each of a plurality of tile images to a trained lymphocyte segmentation model and, for each tile image, assigning a lymphocyte classification to one or more pixels in the tile image. [twenty five] 2. The method of claim 1, wherein the one or more biomarkers are TILs, and the method further comprises using the one or more processors to identify lymphocyte cells in the digital image using a trained lymphocyte segmentation model, the lymphocyte segmentation model being a pixel-resolution two-dimensional UNet classification model trained to distinguish between lymphocyte cell classifications within cell boundaries and non-lymphocyte cell classifications within cell boundaries.

[26] 2. The method of claim 1, wherein the one or more processors are one or more graphics processing units (GPUs), tensor processing units (TPUs), and / or central processing units (CPUs).

[27] 2. The method of claim 1, wherein the image-based biomarker prediction system is communicatively coupled to a pathology slide scanner system via a communications network, whereby the image-based biomarker prediction system receives the digital images from the pathology slide scanner system via the communications network.

[28] 2. The method of claim 1, wherein the image-based biomarker prediction system is comprised within a pathology slide scanner...

Claims

1. 1. A computer-implemented method for identifying biomarkers in digital images of hematoxylin and eosin (H&E) stained slides of a target tissue, the method comprising: Receiving a digital image into an image-based biomarker prediction system having one or more processors. using the one or more processors to separate the digital image into a plurality of tile images using an image tiling technique, where each of the plurality of tile images includes a different portion of the digital image; applying, using the one or more processors, the plurality of tile images to a deep learning framework including one or more trained biomarker classification models, where each trained biomarker classification model is trained to classify a different biomarker, where the deep learning framework includes a single-scale deep learning framework or a multi-scale deep learning framework; predicting, with the one or more processors, a biomarker classification for each of the plurality of tile images using the one or more trained biomarker classification models; wherein the one or more biomarker classification models are trained using a molecular training dataset corresponding to a plurality of training tissue samples, comprising molecular data based on sequencing of substantially similar samples related to each training tissue sample, and comprising a plurality of molecular data subsets clustered by biomarker; determining a predicted presence of one or more biomarkers in the target tissue from the predicted biomarker classifications of each tile image; and generating a report comprising the digital image and a digital overlay visualizing the predicted presence of one or more biomarkers.

2. The method further comprises:

10. The computer-implemented method of claim 1, comprising receiving the digital image from a pathology slide scanner system via an electrical network.

3. The method further includes receiving a molecular training dataset for a plurality of training tissue samples, where the molecular training dataset includes RNA transcriptome counts from sequencing of substantially the same samples associated with each training tissue sample; performing a clustering process on the molecular training data set to identify one or more molecular data subsets each corresponding to a distinct biomarker; receiving, for each of the one or more molecular data subsets, a plurality of digital images of H&E stained training slides of training tissue samples corresponding to respective biomarkers into an image-based biomarker prediction system having one or more processors; applying the plurality of tiled images to a deep learning framework that includes a plurality of digital images of the H&E stained training slides and one or more biomarker classification models trained to classify different biomarkers based on the molecular training dataset.

3. The computer-implemented method of claim 1, comprising:

4. performing a multiple instance learning process on a plurality of digital images of the H&E stained training slide, wherein the multiple instance learning process comprises: providing a tile selection process that assigns each tile image a class status for each tile image in the H&E slide training images; Based on the assigned class status, a subset of tile images is discarded before applying the remaining tile images to a deep learning framework. The method of claim 3 , comprising:

5. 4. The method of claim 3, wherein each of the plurality of digital images of H&E stained training slides of the training tissue samples comprises a slide level indicia.

6. 4. The method of claim 3, wherein each of the plurality of digital images of H&E stained training slides of the training tissue samples is unlabeled.

7. The method of claim 1 , wherein the single-scale deep learning framework is a convolutional neural network having a ResNet configuration or a starting configuration.

8. 2. The method of claim 1, wherein the one or more biomarkers are selected from the group consisting of consensus molecular subtype (CMS) and homologous recombination deficiency ("HRD").

9. The method of claim 1 , wherein the digital overlay includes an overlay element that identifies the tumor content of the digital image or the tumor percentage of the digital image.

Images