Method and system for ai-based immunospectrometry for cancer patient

Abstract

Systems and methods for predicting a patient's response to a therapeutic agent are disclosed. The systems and methods may include a data repository for storing an image of a tissue sample from a patient, the image including one or more features of the tissue sample; a calculation, the calculation comprising: a first neural network model configured to generate one or more Programmed Death-Ligand 1 (PD-L1) expression level indicators or configured to generate one or more Programmed Death-Ligand 1 (PD-1) expression level indicators, a second neural network model configured to generate one or more Programmed Death-Ligand 1 (PD-L1) expression level indicators, and a third neural network model configured to generate one or more Programmed Death-Ligand 1 (PD-1) expression level indicators, a second neural network model configured to calculate one or more tumor infiltrating lymphocyte (TIL) correlation metrics; and / or a third neural network model configured to predict a response status of the patient to the immunooncology therapeutic agent based on the one or more PD-L1 expression level indicators and / or one or more PD-1 expression level indicators and / or the one or more TIL-related metrics, respectively; and a display system configured to display a result indicative of the response status of the patient.

Description

[0001] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 403,652, filed on September 2, 2022, which is incorporated herein by reference in its entirety. Background Art

[0002] Cancer immunotherapy is an emerging discipline in cancer treatment that aims to harness the patient's own immune system to recognize and destroy tumor cells. Various forms of immunotherapy are under development and are at different stages of preclinical and clinical development. Immunotherapy modalities include, but are not limited to, monoclonal antibodies and adoptive cell transfer.

[0003] Checkpoint inhibition is an aspect of cancer immunotherapy and has shown efficacy in both clinical and preclinical studies as an adjuvant and alternative to traditional cancer therapy. The effectiveness of checkpoint inhibition stems from the restriction of T cells by the inhibitory effects of checkpoint molecules. T cells in the tumor microenvironment (TME) increase the expression of checkpoint molecules such as programmed cell death 1 (PD-1) and programmed cell death 1 ligand 1 (PD-L1) in response to a variety of TME-derived factors.

[0004] Cancer therapeutics, including checkpoint inhibitors (including monoclonal antibodies), present a challenge when they lack efficacy in patients with either primary or acquired resistance. Primary resistance can occur when cancer does not respond to immunotherapy strategies and, in the case of checkpoint inhibitors, when cancer cells fail to express them, thereby providing no therapeutic target. Summary of the invention

[0005] Embodiments of the present disclosure include a system for predicting a patient's response to an immuno-oncology therapeutic agent, the system comprising: a data repository for storing an image of a tissue sample from a patient, the image comprising one or more features of the tissue sample; a computing device communicatively connected to the data repository, the computing device comprising: a first neural network model, the first neural network model configured to generate one or more tumor mutation burden (TMB) indicators from the one or more features; a second neural network model, the second neural network model configured to generate one or more programmed death-ligand 1 (PD-L1) expression level indicators from the one or more features and / or configured to generate one or more tumor mutation burden (TMB) indicators from the one or more features; and a second neural network model configured to generate one or more programmed death-ligand 1 (PD-L1) expression level indicators from the one or more features. One or more features generate one or more programmed death-1 (PD-1) expression level indicators; a third neural network model configured to calculate one or more tumor infiltrating lymphocyte (TIL)-related metrics from the one or more features; or a fourth neural network model configured to predict a patient's response status to an immuno-oncology therapeutic agent based on one or more TMB indicators, one or more PD-L1 expression level indicators and / or one or more PD-1 expression level indicators and / or one or more TIL-related metrics; and a display system communicatively connected to the computing device and configured to display a result indicating the patient's response status.

[0006] In a specific embodiment, the prediction of the patient's response status to the immuno-oncology therapeutic agent is performed based on one or more medical history features of the patient. The third neural network model may include a segmentation algorithm configured to generate an identification of a tumor region in a tissue sample based on one or more features, wherein the identification is used to further generate one or more TIL-related metrics. In a specific embodiment, the segmentation algorithm is configured to predict a TIL mask, which is used in combination with a tumor mask to calculate one or more TIL-related metrics. In certain embodiments, one or more TIL-related metrics include the predicted number of TIL clusters in the identified tumor area; the predicted size of the TIL clusters in the identified tumor area; the predicted TIL cluster spread in the identified tumor area; the total count of TILs in the identified tumor area; the distance between TIL cells to the distance between tumor cells; the distance between TIL cells and tumor cells; the average size of TIL cells; stromal TIL% (sTIL%) (which can be calculated as the percentage of the stroma in the identified tumor area in the tissue sample infiltrated by TIL); and / or intratumoral TIL% (iTIL%), which can be calculated as the percentage of the identified tumor area in the tissue sample infiltrated by TIL. In various embodiments, the first neural network model includes a classification algorithm, and one or more TMB indicators include a TMB indicator of the tissue sample, which is a TMB positive classification or a TMB negative classification. If the tissue sample has a TMB value exceeding a preset TMB threshold, the classification algorithm can generate a TMB positive classification. In some cases, the first neural network model includes a regression algorithm, and one or more TMB indicators include a TMB indicator as a quantitative TMB score. In various embodiments, the second neural network model includes a classification algorithm, and one or more PD-L1 expression level indicators include a PD-L1 expression level indicator, which is a PD-L1 expression positive classification or a PD-L1 expression level negative classification. The second neural network model may include a classification algorithm, and one or more PD-1 expression level indicators include a PD-1 expression level indicator, which is a PD-1 expression positive classification or a PD-1 expression level negative classification. In certain embodiments, if the tissue sample has a PD-L1 expression level value exceeding a preset PD-L1 expression level value, the classification algorithm generates a PD-L1 expression positive classification. In some embodiments, if the tissue sample has a PD-1 expression level value exceeding a preset PD-1 expression level value, the classification algorithm generates a PD-1 expression positive classification. The second neural network model may include a regression algorithm, and one or more PD-L1 expression level indicators include a quantitative PD-L1 score. In certain embodiments, the second neural network model includes a regression algorithm, and one or more PD-1 expression level indicators include a quantitative PD-1 score.The quantitative PD-L1 score and / or the quantitative PD-1 score may be a tumor proportion score (TPS). The quantitative PD-L1 score and / or the quantitative PD-1 score may be a combined positive score (CPS).

[0007] In some embodiments, the third neural network model is further configured to generate one or more TIL masks from the one or more features; and wherein the fourth neural network model is further configured to use the one or more TIL masks to predict the patient's response status to the immuno-oncology therapeutic agent.

[0008] In various embodiments, the immuno-oncology therapeutic agent is a checkpoint inhibitor, and the checkpoint inhibitor can be a PD-L1 inhibitor or a PD-1 inhibitor. Any tissue sample may be from the patient's bladder; and in some embodiments, the patient's response to the immuno-oncology therapeutic agent is associated with bladder cancer. The tissue sample may be from a patient known to have PD-L1 positive cancer. The tissue sample may be from a patient known to have PD-1 positive cancer. In a specific embodiment, the tissue sample is from a patient known to have any one or more of the following: non-small cell lung cancer, melanoma, Hodgkin's lymphoma, bladder cancer, kidney cancer, and breast cancer. In a specific embodiment, the immuno-oncology therapeutic agent targets PD-L1 or PD-1. In some embodiments, the immuno-oncology therapeutic agent includes one or more antibodies, one or more adoptive cell therapy agents, or a combination thereof. One or more antibodies may include at least one of the following: a monoclonal antibody, a bispecific antibody, or a trispecific antibody. In some embodiments, one or more adoptive cell therapy agents include immune cells expressing one or more engineered antigen receptors. In some embodiments, one or more engineered antigen receptors include at least one of the following: a chimeric antigen receptor, a non-natural T cell receptor, or a combination thereof. In certain embodiments, the one or more adoptive cell therapy agents include at least one of: a T cell, a natural killer cell, a natural killer T cell, or a combination thereof.

[0009] In certain embodiments, the third neural network model includes a region / TIL segmentation algorithm and a cell segmentation algorithm. The cell segmentation algorithm can calculate one or more cell-based features. In certain embodiments, one or more cell-based features include morphology, color (nucleus), color (outside the nucleus), texture (energy, correlation, contrast, uniformity, dissimilarity or angular second moment). In some embodiments, the cell segmentation algorithm generates a certain number of TIL cells or an average number of TIL cells. In certain embodiments, the cell segmentation algorithm can generate an average tumor cell size, an average TIL cell size, an average distance between TIL cells, an average color of TIL cells and / or an average distance between TIL cells to an average distance between tumor cells.

[0010] Embodiments of the present disclosure include a method for predicting a patient's response to an immuno-oncology therapeutic agent, the method comprising: obtaining an image of a tissue sample from a patient by one or more processors, the image comprising one or more features of the tissue sample; generating one or more tumor mutation burden (TMB) indicators from one or more features via a first neural network model by one or more processors; generating one or more programmed death-ligand 1 (PD-L1) expression level indicators from one or more features via a second neural network model by one or more processors and / or generating one or more programmed death-1 (PD-1) expression level indicators from one or more features; generating one or more tumor infiltrating lymphocyte (TIL) related metrics from one or more features via a third neural network model by one or more processors; and predicting a patient's response status to an immuno-oncology therapeutic agent using one or more TMB indicators, one or more PD-L1 expression level indicators or one or more PD-1 level indicators and one or more related metrics via a fourth neural network model by one or more processors. In some embodiments, the method also includes selecting whether to include the patient in a patient group participating in a clinical trial based on the patient's response status. In some embodiments, the method also includes displaying a graphical representation indicating the patient's response status on a display screen.

[0011] In certain embodiments, the prediction of the response state of the patient to the immuno-oncology therapeutic agent also uses one or more medical history features of the patient. In a specific embodiment, the third neural network is a segmentation algorithm for identifying the tumor area in the tissue sample. In some embodiments, the segmentation algorithm predicts one or more TIL mask structure features in the tumor area of ​​the mark to generate one or more TIL related metrics. The predicted TIL mask structure feature is the predicted number of TIL clusters in the tumor area of ​​the mark, the predicted size of the TIL clusters in the tumor area of ​​the mark, or the predicted TIL cluster diffusion in the tumor area of ​​the mark. In some embodiments, one or more TIL mask structure features include the total count of TILs in the tumor area of ​​the mark, the distance between TIL cells to the distance between tumor cells, the distance between TIL cells and tumor cells, the average color of TIL cells and / or the average size of TIL cells. In some embodiments, the TIL% value is intratumoral TIL% (iTIL%), and iTIL% is calculated as the percentage of the tumor area of ​​the mark in the tissue sample infiltrated by TIL. In certain embodiments, the TIL% value is stromal TIL% (sTIL%), and the sTIL% value is calculated as the percentage of stroma in an identified tumor region in a tissue sample that is infiltrated by TILs.

[0012] In certain embodiments, the first neural network model is a classification algorithm that generates a TMB index, which is a TMB positive classification or a TMB negative classification of a tissue sample. In certain aspects, in a specific embodiment, if the tissue sample has a TMB value exceeding a preset TMB threshold, the classification algorithm generates a TMB positive classification. In various embodiments, the first neural network model can be a regression algorithm that generates a TMB index as a quantitative TMB score, and in certain embodiments, the second neural network model is a classification algorithm that generates a PD-L1 expression level index, which is a PD-L1 expression positive classification or a PD-L1 expression level negative classification. In a specific embodiment, the second neural network model is a classification algorithm that generates a PD-1 expression level index, which is a PD-1 expression positive classification or a PD-1 expression level negative classification. In some embodiments, if the tissue sample has a PD-L1 expression level value exceeding a preset PD-L1 expression level value, the classification algorithm generates a PD-L1 expression positive classification. In a specific embodiment, if the tissue sample has a PD-1 expression level value exceeding a preset PD-1 expression level value, the classification algorithm generates a PD-1 expression positive classification. In some embodiments, the second neural network model is a regression algorithm that generates a PD-L1 expression level indicator as a quantitative PD-L1 score. In various embodiments, the second neural network model is a regression algorithm that generates a PD-1 expression level indicator as a quantitative PD-1 score. In some embodiments, the quantitative PD-L1 score and / or the PD-1 score is a tumor proportion score (TPS). In some embodiments, the quantitative PD-L1 score and / or the PD-1 score is a combined positive score (CPS).

[0013] In some embodiments, the immuno-oncology therapeutic is a checkpoint inhibitor, such as a PD-L1 inhibitor or a PD-1 inhibitor.

[0014] In various embodiments, the method further comprises generating, by the one or more processors, one or more TIL masks from the one or more features via a third neural network model, wherein the prediction of the patient's response status to the immuno-oncology therapeutic agent further comprises using the one or more TIL masks.

[0015] In some embodiments, the tissue sample is from the bladder, and the patient's response to the immuno-oncology therapeutic agent is associated with bladder cancer. In various embodiments, it is determined that the patient is or will be a responder based on the response status; and wherein according to the determination, a therapeutically effective amount of an immuno-oncology therapeutic agent is administered to the patient. In some embodiments, it is determined that the patient is not or will not be a responder based on the response status; and wherein according to the determination, an immuno-oncology therapeutic agent is not administered to the patient. In specific embodiments, the tissue sample is from a patient known to have PD-L1 positive cancer or PD-1 positive cancer. The tissue sample is from a patient known to have any of the following: non-small cell lung cancer, melanoma, Hodgkin's lymphoma, bladder cancer, kidney cancer, or breast cancer.

[0016] In certain embodiments, the immuno-oncology therapeutic targets PD-L1 or PD-1. In specific embodiments, the immuno-oncology therapeutic includes one or more antibodies, adoptive cell therapy agents, immunomodulators, or combinations thereof, and the antibody may be a monoclonal antibody, a bispecific antibody, or a trispecific antibody. In specific embodiments, the adoptive cell therapy agent includes immune cells expressing one or more engineered antigen receptors. In some embodiments, the engineered antigen receptor is a chimeric antigen receptor, a non-natural T cell receptor, or a combination thereof. In various embodiments, the adoptive cell therapy agent includes T cells, natural killer cells, natural killer T cells, or a combination thereof.

[0017] The method may further comprise obtaining a sample from the patient and / or diagnosing the patient as having cancer.

[0018] In a specific embodiment, there is a method comprising: obtaining, by one or more processors, an image of a tissue sample from a patient and one or more features of the tissue sample from a data source; performing, by one or more processors, via one or more neural network models, one or more of the following: (a) generating one or more tumor mutation burden (TMB) indicators from the one or more features; (b) generating one or more programmed death-ligand 1 (PD-L1) expression level indicators from the one or more features or generating one or more programmed death-1 (PD-1) expression level indicators from the one or more features; and (c) generating one or more tumor infiltrating lymphocyte (TIL) percentage (%) values ​​from the one or more features; predicting, by one or more processors, via one or more neural network models, the patient's response status to an immuno-oncology therapeutic agent using one or more of the following: one or more TMB indicators, one or more PD-L1 expression level indicators, one or more PD-1 expression level indicators, and / or one or more TIL% values; and determining whether to include the patient in a group of patients participating in a clinical trial based on the patient's response status.

[0019] In certain embodiments, there is a method comprising: obtaining, by one or more processors, an image of a tissue sample from a patient and one or more features of the tissue sample from a data source; and generating, by the one or more processors, a prediction of the patient's response status to an immuno-oncology therapeutic agent using the image via one or more neural network models.

[0020] In certain embodiments, there is a system for predicting a patient's response to an immuno-oncology therapeutic agent, the system comprising: a data repository for storing an image of a tissue sample from a patient, the image comprising one or more features of the tissue sample; a computing device communicatively connected to the data repository, the computing device comprising: a first neural network model, the first neural network model configured to generate one or more tumor mutation burden (TMB) indicators from the one or more features; a second neural network model, the second neural network model configured to generate one or more programmed death-ligand 1 (PD-L1) expression level indicators from the one or more features and / or configured to generate one or more programmed death-1 (PD-1) expression level indicators from the one or more features; a level indicator; a third neural network model configured to calculate one or more tumor infiltrating lymphocyte (TIL)-related metrics from the one or more features; a fourth neural network model configured to calculate one or more cell-based features from the one or more features; and a fifth neural network model configured to predict the patient's response status to an immuno-oncology therapeutic agent based on one or more TMB indicators, one or more PD-L1 expression level indicators or one or more PD-1 expression level indicators, one or more TIL-related metrics, and one or more cell-based features, respectively; and a display system communicatively connected to the computing device and configured to display a result indicating the patient's response status.

[0021] In some embodiments, there is a method comprising: obtaining, by one or more processors, an image of a tissue sample from a patient and one or more features of the tissue sample from a data source; and training, by one or more processors, one or more neural network models to predict a patient's response status to an immuno-oncology therapeutic agent using the image.

[0022] In a specific embodiment, there is a method for predicting a patient's response to an immuno-oncology therapeutic agent, the method comprising: obtaining, by one or more processors, an image of a tissue sample from the patient, the image comprising one or more features of the tissue sample; generating, by one or more processors, one or more tumor mutation burden (TMB) indicators (optionally) from the one or more features; generating, by one or more processors, one or more programmed death-ligand 1 (PD-L1) expression level indicators or one or more programmed death-1 (PD-1) expression level indicators from the one or more features; generating, by one or more processors, one or more tumor infiltrating lymphocyte (TIL)-related metrics from the one or more features; and training, by one or more processors, a fourth neural network model to predict the patient's response status to the immuno-oncology therapeutic agent based on one or more of the following: one or more TMB indicators (when used), one or more PD-L1 expression level indicators, one or more PD-1 level indicators, and one or more related metrics. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] For a more complete understanding of the principles and advantages disclosed herein, reference is now made to the following description in conjunction with the accompanying drawings, in which:

[0024] Figure 1A Depicted are systems for predicting a patient or treatment response to a therapeutic agent, according to various embodiments.

[0025] Figure 1B

[0013] Another system for predicting a patient or treatment response to a therapeutic agent, according to various embodiments, is depicted.

[0026] Figure 1C

[0013] Another system for predicting a patient or treatment response to a therapeutic agent, according to various embodiments, is depicted.

[0027] Figure 2A Methods for predicting a patient / treatment response to a therapeutic agent according to various embodiments are illustrated.

[0028] Figure 2B

[00136] Another method for predicting a patient / treatment response to a therapeutic agent, according to various embodiments, is illustrated.

[0029] Figure 2C

[00136] Another method for predicting a patient / treatment response to a therapeutic agent, according to various embodiments, is illustrated.

[0030] Figure 2D Methods for predicting a patient / treatment response to a therapeutic agent according to various embodiments are illustrated.

[0031] Figure 2E

[00136] Another method for predicting a patient / treatment response to a therapeutic agent, according to various embodiments, is illustrated.

[0032] Figure 2F

[00136] Another method for predicting a patient / treatment response to a therapeutic agent, according to various embodiments, is illustrated.

[0033] Figure 3 is a block diagram of a computing system configured to perform a method of predicting a patient / treatment response to a therapeutic agent, according to various embodiments.

[0034] It should be understood that the drawings are not necessarily drawn to scale, nor are the objects in the drawings necessarily drawn to scale relative to each other. The drawings are intended to make the various embodiments of the apparatus, systems, and methods disclosed herein clear and easy to understand. Wherever possible, the same reference numerals will be used throughout the drawings to refer to the same or similar components. In addition, it should be understood that the drawings are not intended to limit the scope of the present teachings in any way. DETAILED DESCRIPTION

[0035] I. Overview

[0036] Embodiments of the present disclosure include methods for predicting whether an individual will respond to one or more specific therapies. In a specific embodiment, one or more specific therapies include one or more immuno-oncology therapeutic agents of any kind. In some cases, one or more immuno-oncology therapeutic agents target one or more checkpoint inhibitors. In a specific embodiment, one or more checkpoint inhibitors include PD-L1 or PD-1 or a mixture thereof. The type of immuno-oncology therapeutic agent can be any kind, including therapeutic agents that at least partially utilize any kind of antibodies. In a specific embodiment, antibodies include monoclonal antibodies and may or may not be humanized. The antibody itself can be used as an immuno-oncology therapeutic agent, or the antibody can be configured as a part of any kind of chimeric molecule (such as an engineered antigen receptor). The antibody can be configured as a part of any kind of fusion protein.

[0037] In a specific embodiment, an individual (such as a patient) is subjected to a method of measuring multiple parameters in order to determine whether the individual will respond to a particular therapy. In certain cases, the parameters include one or more tumor mutation load indicators, one or more PD-L1 expression level indicators, and / or one or more PD-1 expression level indicators; and one or more TIL metrics. In some embodiments, patient characteristics, such as biological sex, age, individual or family health history (such as whether the individual and / or one or more family members have previously suffered from any disease, including cancer, including PD-L1 positive cancer and / or PD-1 positive cancer), etc., may also be included as one or more parameters to be considered in order to determine whether the individual will respond to a particular therapy.

[0038] In certain embodiments, an individual (such as a patient) is predicted or determined to be a responder to one or more immuno-oncology therapeutic agents. In some cases, an individual is predicted or determined to have a higher probability of responding to one or more immuno-oncology therapeutic agents when compared to a specific population (such as a general population, a cancer patient population, a population lacking expression of one or more checkpoints (such as PD-L1 or PD-1), etc.).

[0039] The present disclosure is not limited to these exemplary embodiments and applications, nor to the manner in which the exemplary embodiments and applications are operated or described herein.In addition, the drawings may show simplified or partial views, and the sizes of the elements in the drawings may be exaggerated or otherwise not to scale.

[0040] Unless otherwise defined, scientific and technical terms used in conjunction with the present teachings described herein shall have the meanings commonly understood by those of ordinary skill in the art. In addition, unless the context otherwise requires, singular terms shall include the plural, and plural terms shall include the singular. Generally, the glossaries used in conjunction with the chemistry, biochemistry, molecular biology, pharmacology and toxicology described herein and the techniques of chemistry, biochemistry, molecular biology, pharmacology and toxicology are those glossaries and techniques well known and commonly used in the art.

[0041] Although the present teachings are described in conjunction with various embodiments, it is not intended that the present teachings be limited to such various embodiments. On the contrary, the present teachings encompass various alternatives, modifications, and equivalents, as will be understood by those skilled in the art.

[0042] When describing various embodiments, this specification may present a method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of steps set forth herein, the method or process should not be limited to the specific sequence of steps described, and those skilled in the art can readily understand that the sequence can be varied and still remain within the spirit and scope of the various embodiments.

[0043] II. Systems and Workflows for Predicting Patient or Treatment Response to Therapeutic Agents

[0044] Figure 1A A system 100 for predicting a patient or treatment response to a therapeutic agent is depicted in accordance with various embodiments. Although the system 100 may be applicable to any suitable therapeutic agent (such as for any type of cancer), Figure 1A The system 100 depicted in FIG. 1 is illustrated as being used to predict a patient's response to an immuno-oncology therapeutic agent. Figure 1A As shown, system 100 includes one or more neural networks 110 that can be used to analyze images 105 to make a prediction of a patient response 190 (also referred to herein as “response 190”) based on the analysis of images 105. In various embodiments, one or more patient characteristics 108 can also be used to analyze and predict the patient's response 190 to the therapeutic agent.

[0045] According to various embodiments, image 105 may be an image of a tissue sample from a patient. In various embodiments, image 105 may include one or more features of a tissue sample (e.g., shape, size, distribution, and / or density of cells, etc.) that may be used for analysis by one or more neural networks 110 to predict a patient's response 190 to a therapeutic agent. In various embodiments, image 105 may include a hematoxylin and eosin (H&E) stained tissue sample from a patient or any other sample that may be used for conventional histopathology techniques to diagnose a disease, such as cancer, including bladder cancer. In various embodiments, image 105 may be obtained from a laboratory or clinical department (such as a histology department) prior to analysis, or may be stored in a data repository, database, or data storage space for retrieval prior to analysis. Image 105 may be fresh or frozen tissue.

[0046] According to various embodiments, one or more patient characteristics 108 may include any characteristics associated with and / or related to the patient that may help analyze or predict the patient's response to the therapeutic agent. Such characteristics 108 may include, for example, but not limited to: age, biological sex, weight, smoking history, one or more symptoms, previous treatment regimens, previous treatment plans, time since diagnosis, number of previous diagnoses of the same disease or any other disease that may affect related organs or tissues, patient family medical information, presence or absence of one or more biomarkers of the disease or any anatomical part of the patient. In various embodiments, one or more patient characteristics 108 may include one or more medical history characteristics of the patient. In various embodiments, one or more patient characteristics 108 may be extracted from a patient health record, such as from an electronic health record system.

[0047] like Figure 1AAs shown, system 100 includes one or more neural networks 110 for analyzing an input image, such as image 105, and predicting a patient response 190 based on the analysis. In various embodiments, one or more neural networks 110 include a first neural network model 120 (also referred to herein as "model 120"), a second neural network model 140 (also referred to herein as "model 140"), and a third neural network model 160 (also referred to herein as "model 160"). Each of models 120, 140, and / or 160 may include a machine learning algorithm that may input image 105 along with one or more features of a tissue sample to generate a corresponding output 130, 150, and / or 170, such as Figure 1A shown.

[0048] In various embodiments, one or more neural networks 110 include a fourth neural network model 180 (also referred to herein as “model 180”), which may include a machine learning algorithm that may ingest one or more of outputs 130, 150, and 170 as input into model 180 to generate patient response 190. In various embodiments, model 180 may input one or more features of image 105 and / or tissue sample, such as Figure 1A In various embodiments, the model 180 may also input one or more patient characteristics 108 as part of the analysis to generate a patient response 190, such as Figure 1A shown.

[0049] In various embodiments, the one or more neural networks 110 and, therefore, the models 120, 140, 160, and 180 may include, for example, but not limited to, at least one of a convolutional neural network (CNN), a feedforward neural network (FNN), a recurrent neural network (RNN), a modular neural network (MNN), a residual neural network (ResNet), an ordinary differential equation neural network (neural-ODE), or another type of neural network, or any combination thereof. The one or more neural networks 110 and, therefore, each of the models 120, 140, 160, and 180 may include one or more algorithms, one or more mathematical techniques, one or more machine learning algorithms, or a combination thereof.

[0050] In various embodiments, a large amount of data sets may be used to train one or more neural networks 110 and, by extension, models 120, 140, 160, and 180. In various embodiments, the training of one or more neural networks 110 and, by extension, models 120, 140, 160, and 180 may be self-supervised learning (SSL) or performed in a supervised manner. In various embodiments, models 120, 140, 160, and 180 may be pre-trained using SSL and subsequently trained for one or more specific tasks (e.g., the various indicators or metrics listed below) via supervised learning. Thus, the data sets used to train the models may include images annotated by clinicians / pathologists. In various embodiments, each of the trained models 120, 140, 160, and / or 180 may then be configured to parse one or more features of a tissue sample from image 105, for example, to extract the necessary data, learn from it, and then make a determination or prediction, such as output 130, 150, 170, and / or patient response 190, respectively.

[0051] For example, according to various embodiments, in order to predict a patient's response to an immuno-oncology therapeutic agent, model 120 may be configured to generate one or more tumor mutation burden (TMB) indicators based on one or more features of image 105 as output 130. Similarly, according to various embodiments, model 140 may be configured to generate one or more programmed death-ligand 1 (PD-L1) expression level indicators from one or more features or configured to generate one or more programmed death-1 (PD-1) expression level indicators from one or more features of image 105 as output 150. In various embodiments, the immuno-oncology therapeutic agent is a checkpoint inhibitor, wherein the checkpoint inhibitor may be a PD-L1 inhibitor or a PD-1 inhibitor. Similarly, according to various embodiments, model 160 may be configured to calculate one or more tumor infiltrating lymphocyte (TIL) related metrics from one or more features of image 105 as output 170.

[0052] In various embodiments, the model 120 includes a classification algorithm or a regression algorithm, both of which are trained using an annotation data set as described above. In order to generate one or more TMB indicators as output 130, the classification algorithm of the model 120 can, for example, ingest one or more features of the tissue sample from the image 105 to generate an output 130 as a TMB positive classification or a TMB negative classification. In various embodiments, when the TMB value exceeds a preset TMB threshold, a TMB positive classification is generated as output 130. In various embodiments, the regression algorithm of the model 120 can generate a TMB value or a quantitative TMB score as a TMB indicator of the output 130.

[0053] In various embodiments, the model 140 includes a classification algorithm or a regression algorithm, both of which are trained using an annotation data set as described above. In order to generate one or more PD-L1 expression level indicators or one or more PD-1 expression level indicators as output 150, a classification algorithm or a regression algorithm may be used to perform analysis. In various embodiments, the classification algorithm of the model 140 may generate a PD-L1 expression level indicator as output 150, which is a PD-L1 expression positive classification or a PD-L1 expression level negative classification. In various embodiments, if the tissue sample of the image 105 has a PD-L1 expression level value exceeding a preset PD-L1 expression level value, the classification algorithm of the model 140 may generate a PD-L1 expression positive classification. In various embodiments, the regression algorithm of the model 140 may generate one or more PD-L1 expression level indicators as output 150 including a quantitative PD-L1 score. In various embodiments, the quantitative PD-L1 score may be a tumor proportion score (TPS) or a combined positive score (CPS).

[0054] In various embodiments, the classification algorithm of the model 140 may generate a PD-1 expression level indicator as an output 150, which is a PD-1 expression positive classification or a PD-1 expression level negative classification. In various embodiments, if the tissue sample of the image 105 has a PD-1 expression level value that exceeds a preset PD-1 expression level value, the classification algorithm of the model 140 may generate a PD-1 expression positive classification. In various embodiments, the regression algorithm of the model 140 may generate one or more PD-1 expression level indicators as outputs 150 including a quantitative PD-1 score. In various embodiments, the quantitative PD-1 score may be a tumor proportion score (TPS) or a combined positive score (CPS).

[0055] In various embodiments, the model 160 may include one or more segmentation algorithms, each of which may be configured to generate one or more TIL masks, one or more tumor masks, one or more matrix masks, one or more necrosis masks, or one or more nucleus masks from one or more features of the image 105 as output 170. In various embodiments, the model 160 may include a region segmentation algorithm. In various embodiments, the model 160 may include a cell segmentation algorithm. In various embodiments, the model 160 may calculate a TIL-related metric such as, but not limited to, TIL%, sTIL%, or the number of TIL clusters, etc., by using one or more TIL masks, one or more tumor masks, one or more matrix masks, one or more necrosis masks, or one or more nucleus masks as output 170.

[0056] In various embodiments, the model 160 may include a segmentation algorithm that can be trained using an annotation dataset as described above. In various embodiments, according to various embodiments, a segmentation algorithm may be employed to generate an identification of a tumor region in a tissue sample based on one or more features of the image 105. The identification may be used to further generate or calculate one or more TIL-related metrics from one or more features of the image 105 as an output 170. In various embodiments, the segmentation algorithm may be referred to as a region segmentation algorithm or a TIL region segmentation algorithm. In various embodiments, the segmentation algorithm may be referred to as a tumor region segmentation algorithm. In various embodiments, the segmentation algorithm may be configured to predict a TIL mask that is used in combination with a tumor mask to calculate one or more TIL-related metrics as an output 170. In various embodiments, one or more TIL-related metrics may include a predicted number of TIL clusters in the identified tumor region as an output 170. In various embodiments, one or more TIL-related metrics may include a predicted size of a TIL cluster in the identified tumor region as an output 170. In various embodiments, one or more TIL-related metrics may include a predicted TIL cluster spread in the identified tumor region as an output 170. In various embodiments, the one or more TIL-related metrics may include intratumoral TIL% (iTIL%) as output 170. In various embodiments, iTIL% may be calculated as a percentage of an identified tumor region in a tissue sample infiltrated by TILs as output 170. In various embodiments, the one or more TIL-related metrics may include stromal TIL% (sTIL%) as output 170. In various embodiments, the sTIL% value may be calculated as a percentage of stroma in an identified tumor region in a tissue sample infiltrated by TILs as output 170.

[0057] In various embodiments, the model 160 may include a segmentation algorithm for cell segmentation, which may be referred to as a cell segmentation algorithm. In various embodiments, the cell segmentation algorithm may input an image 105 and perform the segmentation of cells as an output 170. In various embodiments, the segmented cells may be labeled based on the regional segmentation output. For example, if the cell is in the TIL region, the segmented cells are labeled as TIL cells. In various embodiments, the cell segmentation algorithm is configured to calculate one or more features related to cells contained in the image 105 of the tissue sample, also referred to herein as cell-based features. In various embodiments, the calculated cell-based features may include morphology, color (nucleus), color (outside the nucleus), texture (energy, correlation, contrast, uniformity, dissimilarity, angular second moment), chromatin skeleton morphology and color, and many other features that may be extracted. In various embodiments, the cell segmentation algorithm of the model 160 may be configured to generate a certain number of TIL cells or an average number of TIL cells as an output 170. In various embodiments, the cell segmentation algorithm of the model 160 may be configured to generate an average tumor cell size as an output 170. In various embodiments, the cell segmentation algorithm of the model 160 may be configured to generate an average TIL cell size as an output 170. In various embodiments, the cell segmentation algorithm of the model 160 may be configured to generate an average distance between TIL cells as an output 170. In various embodiments, the cell segmentation algorithm of the model 160 may be configured to generate an average color of TIL cells as an output 170. In various embodiments, the cell segmentation algorithm of the model 160 may be configured to generate an average distance between TIL cells versus an average distance between tumor cells, etc. as an output 170. In various embodiments, one or more TIL-related metrics may include a total count of TILs in the identified tumor region as an output 170. In various embodiments, one or more TIL-related metrics may include a distance between TIL cells versus a distance between tumor cells as an output 170. In various embodiments, one or more TIL-related metrics may include a distance between TIL cells and tumor cells as an output 170. In various embodiments, one or more TIL-related metrics may include an average color of TIL cells as an output 170. In various embodiments, one or more TIL-related metrics may include an average size of TIL cells as an output 170.

[0058] In various embodiments, model 180 may be configured to generate, for example, a response state of a patient or treatment to an immuno-oncology therapeutic agent 190 by applying one or more of outputs 130, 150, and 170 as input to model 180. In various embodiments, one or more TMB indicators of output 130, one or more PD-L1 expression level indicators or one or more PD-1 expression level indicators of output 150, and one or more TIL-related metrics of output 170 may be used for analysis of model 180 to predict the patient's response state to an immuno-oncology therapeutic agent, according to various embodiments. As described above, model 180 may include a machine learning algorithm trained using the patient's immuno-oncology therapeutic agent response data, wherein response 190 may include an annotation label of a responder, a partial responder, or a non-responder as the basis for training. In various embodiments, model 180 may be configured so that each of outputs 130, 150, and 170 is weighted relative to each other in the analysis performed by the machine learning algorithm of model 180. In various embodiments, different weights applied to each of outputs 130, 150, and 170 may influence and / or significantly alter output 190 of a patient's response to a therapeutic agent, such as an immuno-oncology therapeutic agent.

[0059] In various embodiments, model 180 may be trained using a set of H&E images from patients having response data available for a desired treatment or therapeutic agent, such as an immuno-oncology therapeutic agent.

[0060] For example, the following four patients may be used to train model 180 .

[0061] - Patient 1, H&E image, responder

[0062] - Patient 2, H&E images, non-responder

[0063] - Patient 3, H&E image, responder

[0064] - Patient 4, H&E image, responder

[0065] For each of these patients, each of the models 120, 140, and 160 may be run to generate the following datasets from the image 105, outputs 130 (TMB score), 150 (PD-L1 score), and 170 (TIL mask and / or TIL features), respectively:

[0066] - Patient 1, H&E images, TMB score, PD-L1 score, TIL mask, TIL characteristics, responders

[0067] - Patient 2, H&E images, TMB score, PD-L1 score, TIL mask, TIL characteristics, non-responders

[0068] - Patient 3, H&E images, TMB score, PD-L1 score, TIL mask, TIL characteristics, responders

[0069] - Patient 4, H&E images, TMB score, PD-L1 score, TIL mask, TIL characteristics, responders

[0070] Using the above datasets of inputs 130, 150, and 170, model 180 may be trained as follows: "H&E image, TMB score, PD-L1 score, TIL mask, TIL features," along with ground truth or labeling as a "responder" or "non-responder," or in some cases, as a "partial responder." Once model 180 is trained, model 180 may be used to generate patient responses 190 based on a single image similar to image 105 and outputs generated by trained models 120, 140, and 160 using one or more features of the single image.

[0071] Figure 1B Depicted is a system 102 for predicting a patient or treatment response to a therapeutic agent according to various embodiments. The system 102 is similar to Figure 1A The system 100 described, wherein there is an image 105, a neural network 110 having models 120 and 140 that generate outputs 130 and 150, respectively, and a model 180 that must ingest the outputs 130 and 150 and / or the image 105 and / or the patient features 108 in order to generate a response 190. The system 102 differs from the system 100 in that the system 102 includes two models 160a and 160b that generate outputs 170a and 170b, respectively, that can be ingested by the model 180 to generate the response 190, rather than having the model 160 generate an output 170 that can be ingested by the model 180 to generate the response 190 (as in the system 100). In other words, the models 160a and 160b represent two independent segmentation algorithms. In various embodiments, the model 160a may include a region / TIL segmentation algorithm. In various embodiments, the model 160b may include a cell segmentation algorithm.

[0072] In various embodiments, the region / TIL segmentation algorithm of the model 160a may be used to generate an identification of a tumor region in a tissue sample based on one or more features of the image 105. In various embodiments, the identification may be used to further generate or calculate one or more TIL-related metrics from one or more features of the image 105 as an output 170a. In various embodiments, the region / TIL segmentation algorithm of the model 160a may be configured to predict a TIL mask, which is used in combination with the tumor mask to calculate one or more TIL-related metrics as an output 170a. In various embodiments, the one or more TIL-related metrics may include a predicted number of TIL clusters in the identified tumor region as an output 170a. In various embodiments, the one or more TIL-related metrics may include a predicted size of a TIL cluster in the identified tumor region as an output 170a. In various embodiments, the one or more TIL-related metrics may include a predicted TIL cluster spread in the identified tumor region as an output 170a. In various embodiments, the one or more TIL-related metrics may include intratumoral TIL% (iTIL%) as an output 170a. In various embodiments, iTIL% may be calculated as the percentage of the identified tumor region in the tissue sample infiltrated by TILs as output 170a. In various embodiments, one or more TIL-related metrics may include stromal TIL% (sTIL%) as output 170a. In various embodiments, the sTIL% value may be calculated as the percentage of stroma in the identified tumor region in the tissue sample infiltrated by TILs as output 170a.

[0073] In various embodiments, the cell segmentation algorithm of model 160b can input image 105 and perform the segmentation of cells as output 170b. In various embodiments, the cell segmentation algorithm of model 160b can input the region / TIL mask generated by model 160a and perform the segmentation of cells as output 170b. In various embodiments, the segmented cells can be marked based on the regional segmentation output. For example, if the cell is in the TIL region, the segmented cells are marked as TIL cells. In various embodiments, the cell segmentation algorithm is configured to calculate one or more features related to cells contained in the image 105 of the tissue sample, also referred to as cell-based features herein. In various embodiments, the calculated cell-based features may include morphology, color (nucleus), color (outside the nucleus), texture (energy, correlation, contrast, uniformity, dissimilarity, angular second moment), chromatin skeleton morphology and color, and many other features that can be extracted. In various embodiments, the cell segmentation algorithm of model 160b can be configured to generate a certain number of TIL cells or an average number of TIL cells as output 170b. In various embodiments, the cell segmentation algorithm of model 160b may be configured to generate an average tumor cell size as output 170b. In various embodiments, the cell segmentation algorithm of model 160b may be configured to generate an average TIL cell size as output 170b. In various embodiments, the cell segmentation algorithm of model 160b may be configured to generate an average distance between TIL cells as output 170b. In various embodiments, the cell segmentation algorithm of model 160b may be configured to generate an average color of TIL cells as output 170b. In various embodiments, the cell segmentation algorithm of model 160b may be configured to generate an average distance between TIL cells to an average distance between tumor cells, etc. as output 170b. In various embodiments, one or more TIL-related metrics may include a total count of TILs in the identified tumor region as output 170b. In various embodiments, one or more TIL-related metrics may include a distance between TIL cells to a distance between tumor cells as output 170b. In various embodiments, one or more TIL-related metrics may include a distance between TIL cells to a distance between tumor cells as output 170b. In various embodiments, one or more TIL-related metrics may include a distance between TIL cells and tumor cells as output 170b. In various embodiments, the one or more TIL-related metrics may include an average color of TIL cells as output 170b. In various embodiments, the one or more TIL-related metrics may include an average size of TIL cells as output 170b.

[0074] In various embodiments, model 180 may be configured to generate, for example, a response state of a patient or treatment to an immuno-oncology therapeutic agent 190 by applying one or more of outputs 130, 150, 170a, and 170b as input to model 180. In various embodiments, one or more TMB indicators of output 130, one or more PD-L1 expression level indicators or one or more PD-1 expression level indicators of output 150, and one or more TIL-related metrics of output 170 may be used for analysis of model 180 to predict the patient's response state to an immuno-oncology therapeutic agent, according to various embodiments. As described above, model 180 may include a machine learning algorithm trained using the patient's immuno-oncology therapeutic agent response data, wherein response 190 may include an annotation label of a responder, a partial responder, or a non-responder as the basis for training. In various embodiments, model 180 may be configured so that each of outputs 130, 150, 170a, and 170b is weighted relative to each other in the analysis performed by the machine learning algorithm of model 180. In various embodiments, different weights applied to each of the outputs 130, 150, 170a, and 170b may influence and / or significantly alter the output 190 of a patient's response to a therapeutic agent, such as an immuno-oncology therapeutic agent.

[0075] For each of these patients, each of the models 120, 140, 160a, and 160b may be run to generate the following datasets from the image 105, outputs 130 (TMB score), 150 (PD-L1 score), 170a (TIL mask and / or TIL features), and 170b (cell-based features), respectively:

[0076] - Patient 1, H&E images, TMB score, PD-L1 score, TIL mask, TIL signature, cell-based signature, responder

[0077] - Patient 2, H&E images, TMB score, PD-L1 score, TIL mask, TIL signature, cell-based signature, non-responder

[0078] - Patient 3, H&E images, TMB score, PD-L1 score, TIL mask, TIL signature, cell-based signature, responder

[0079] - Patient 4, H&E images, TMB score, PD-L1 score, TIL mask, TIL signature, cell-based signature, responder

[0080] Using the above datasets of inputs 130, 150, 170a, and 170b, model 180 may be trained as follows: "H&E images, TMB scores, PD-L1 scores, TIL masks, TIL features, cell-based features," along with ground truth or labels as "responders" or "non-responders," or in some cases as "partial responders." Once model 180 is trained, model 180 may be used to generate patient responses 190 based on a single image similar to image 105 and outputs generated by trained models 120, 140, 160a, and 160b using one or more features of the single image.

[0081] Figure 1C Another system 200 for predicting a patient or treatment response to a therapeutic agent is depicted in accordance with various embodiments. Figure 1A The systems 100 and 200 described herein may be applicable to any suitable therapeutic agent (such as for any type of cancer). Figure 1C 2 is depicted as being used to predict a patient's response to an immuno-oncology therapeutic agent, but the system 200 is suitable for generating a patient's response to any other therapeutic agent. Figure 1C As shown, system 200 includes one or more neural networks 210 that can be used to analyze images 205 to make a prediction of a patient response 290 (also referred to herein as a "response 290") based on the analysis of images 205. In various embodiments, one or more patient features 208 can also be used to analyze and predict the patient's response 290 to the therapeutic agent. Figure 1A The images 105 and the one or more patient characteristics 108 are substantially similar or identical, and therefore they will not be described in further detail.

[0082] like Figure 1C As shown, the system 200 includes one or more neural networks 210 for analyzing an input image, such as image 205, and predicting a patient response 290 based on the analysis. In various embodiments, the one or more neural networks 210 include one or more classification algorithms, one or more regression algorithms, and / or one or more segmentation algorithms for analyzing and generating one or more final outputs and intermediate outputs. In various embodiments, the intermediate outputs may be generated by one or more of the classification algorithms, regression algorithms, and / or segmentation algorithms and then used as input to another of the one or more of the classification algorithms, regression algorithms, and / or segmentation algorithms to generate the final output. Each of the classification algorithms, regression algorithms, and / or segmentation algorithms of the one or more neural networks 210 is a machine learning algorithm that may input the image 205 along with one or more features of the tissue sample to generate the response 290, such as Figure 1Cshown.

[0083] In various embodiments, the one or more neural networks 210 may include, for example, but not limited to, at least one of a convolutional neural network (CNN), a feedforward neural network (FNN), a recurrent neural network (RNN), a modular neural network (MNN), a residual neural network (ResNet), an ordinary differential equation neural network (Neural-ODE), or another type of neural network or any combination thereof.

[0084] In various embodiments, a large dataset may be used to train the one or more neural networks 210. In various embodiments, the training of the one or more neural networks 210 may be self-supervised learning (SSL) or performed in a supervised manner. Thus, the dataset used to train the model may include images annotated by clinicians / pathologists. In various embodiments, each of the trained classification algorithms, regression algorithms, and / or segmentation algorithms of the one or more neural networks 210 may then be configured to parse one or more features of the tissue sample from the image 105, for example, to extract the necessary data, learn from it, and then make a determination or prediction, such as a patient response 190, respectively.

[0085] In with Figure 1A , Figure 1B or Figure 1C In various related embodiments, one or more neural network models 110 or 210 may be configured to generate a prediction of a patient's response status to an immuno-oncology therapeutic agent using image 105 or image 205. For example, according to various embodiments, in order to predict a patient's response to an immuno-oncology therapeutic agent, one or more neural networks 210 may be configured to generate one or more tumor mutation burden (TMB) indicators based on one or more features of image 205. In various embodiments, according to various embodiments, one or more neural networks 210 may be configured to generate one or more programmed death-ligand 1 (PD-L1) expression level indicators from one or more features or configured to generate one or more programmed death-1 (PD-1) expression level indicators from one or more features of image 205. In various embodiments, according to various embodiments, one or more neural networks 210 may be configured to calculate one or more tumor infiltrating lymphocyte (TIL) related metrics from one or more features of image 205.

[0086] In various embodiments, one or more neural networks 210 are trained using an annotation data set as described above. For example, the classification algorithm of one or more neural networks 210 may ingest one or more features of a tissue sample from the image 205 to generate a TMB positive classification or a TMB negative classification. These features may be used to generate one or more TMB indicators. In various embodiments, when the TMB value exceeds a preset TMB threshold, a TMB positive classification is generated as an intermediate output. In various embodiments, the regression algorithm of one or more neural networks 210 may generate a TMB value or a quantitative TMB score as a TMB indicator of the output 130.

[0087] In various embodiments, one or more neural networks 210 may generate one or more PD-L1 expression level indicators or one or more PD-1 expression level indicators. In various embodiments, one or more neural networks 210 may include a classification algorithm that generates a PD-L1 expression level indicator as an output 150, which is a PD-L1 expression level indicator that is a PD-L1 expression positive classification or a PD-L1 expression level negative classification. In various embodiments, if the tissue sample of the image 105 has a PD-L1 expression level value that exceeds a preset PD-L1 expression level value, the classification algorithm may generate a PD-L1 expression positive classification. In various embodiments, one or more neural networks 210 may include a regression algorithm to generate one or more PD-L1 expression level indicators as an output 150 including a quantitative PD-L1 score. In various embodiments, the quantitative PD-L1 score may be a tumor proportion score (TPS) or a combined positive score (CPS).

[0088] In various embodiments, the classification algorithm may generate a PD-1 expression level indicator as an intermediate output, the PD-1 expression level indicator being a PD-1 expression positive classification or a PD-1 expression level negative classification. In various embodiments, if the tissue sample of image 205 has a PD-1 expression level value that exceeds a preset PD-1 expression level value, the classification algorithm may generate a PD-1 expression positive classification. In various embodiments, the regression algorithm may generate one or more PD-1 expression level indicators including a quantitative PD-1 score. In various embodiments, the quantitative PD-1 score may be a tumor proportion score (TPS) or a combined positive score (CPS).

[0089] In various embodiments, one or more neural networks 210 may include a segmentation algorithm that is trained using an annotation data set as described above. According to various embodiments, a segmentation algorithm is used to generate an identification of a tumor region in a tissue sample based on one or more features of the image 205. These metrics can be used to generate or calculate one or more TIL-related metrics from one or more features of the image 205. In various embodiments, the segmentation algorithm uses one or more TIL mask structure features in the identified tumor region to generate one or more TIL-related metrics as an intermediate output. In various embodiments, one or more TIL mask structure features may include the predicted number of TIL clusters in the identified tumor region as an intermediate output. In various embodiments, one or more TIL mask structure features may include the predicted size of the TIL clusters in the identified tumor region as an intermediate output. In various embodiments, one or more TIL mask structure features may include the predicted TIL cluster diffusion in the identified tumor region as an intermediate output. In various embodiments, one or more TIL mask structure features may include the total count of TILs in the identified tumor region as an intermediate output. In various embodiments, one or more TIL mask structure features may include the distance between TIL cells to the distance between tumor cells as an intermediate output. In various embodiments, one or more TIL mask structure features may include the distance between TIL cells and tumor cells as an intermediate output. In various embodiments, one or more TIL mask structure features may include the average color of TIL cells as an intermediate output. In various embodiments, one or more TIL mask structure features may include the average size of TIL cells as an intermediate output. In various embodiments, one or more TIL related metrics may include intratumoral TIL% (iTIL%) as an intermediate output. In various embodiments, iTIL% may be calculated as the percentage of the tumor area of ​​the mark in the tissue sample infiltrated by TIL as an intermediate output. In various embodiments, one or more TIL related metrics may include matrix TIL% (sTIL%) as an intermediate output. In various embodiments, the sTIL% value may be calculated as the percentage of the matrix in the tumor area of ​​the mark in the tissue sample infiltrated by TIL as an intermediate output.

[0090] In various embodiments, the one or more neural networks 210 may include one or more segmentation algorithms, each of which may be configured to generate one or more TIL masks, one or more tumor masks, one or more matrix masks, one or more necrosis masks, or one or more nucleus masks as intermediate outputs. In various embodiments, the one or more segmentation algorithms of the one or more neural networks 210 may calculate TIL-related metrics such as, but not limited to, TIL%, sTIL%, or the number of TIL clusters, etc., by using one or more TIL masks, one or more tumor masks, one or more matrix masks, one or more necrosis masks, or one or more nucleus masks.

[0091] In various embodiments, one or more neural networks 210 may be configured to generate, for example, a patient or treatment response 290 to an immuno-oncology therapeutic agent by applying one or more of the above-mentioned intermediate outputs as input to one or more neural networks 210. In various embodiments, according to various embodiments, one or more TMB indicators, one or more PD-L1 expression level indicators or one or more PD-1 expression level indicators and one or more TIL-related metrics can be used for analysis to predict the patient's response state to an immuno-oncology therapeutic agent. As described above, one or more neural networks 210 may include a machine learning algorithm trained using the patient's immuno-oncology therapeutic agent response data, wherein the response 290 may include an annotation label of a responder, a partial responder or a non-responder as the basis for training. In various embodiments, one or more neural networks 210 may be configured so that each of the intermediate outputs is weighted relative to each other in the analysis performed by the machine learning algorithm. In various embodiments, different weights applied to various intermediate outputs can affect and / or significantly change the patient's response 290 to a therapeutic agent (such as an immuno-oncology therapeutic agent).

[0092] In various embodiments, no biomarkers are generated. In various embodiments, one or more neural networks 210 predict the patient's response directly from the images 205 .

[0093] Figure 2A A method S100 for predicting a patient / treatment response to a therapeutic agent according to various embodiments is illustrated. The method S100 may be performed, for example, as described below with respect to Figure 3 The computing system 300 is implemented as described above. Figure 2AAs shown, method S100 includes: at step S110, obtaining or receiving an image of a tissue sample from a patient, such as image 105, by one or more processors of a computing system 300; at step S120, generating one or more tumor mutation burden (TMB) indicators (e.g., as output 130) from one or more features by one or more processors via a first neural network model (such as model 120); at step S130, generating one or more programmed death-ligand 1 (PD-L1) expression level indicators from one or more features or generating one or more programmed death-ligand 1 (PD-L1) expression level indicators from one or more features by one or more processors via a second neural network model (such as model 140). at step S140, generating one or more tumor infiltrating lymphocyte (TIL)-related metrics from one or more features via a third neural network model (such as model 160) by one or more processors; and at step S150, predicting a patient's response status to an immuno-oncology therapeutic agent, such as response 190, using one or more TMB metrics, one or more PD-L1 expression level metrics or one or more PD-1 level metrics, and one or more TIL-related metrics, respectively, via a fourth neural network model (such as model 180) by one or more processors.

[0094] In various embodiments, method S100 may optionally include displaying a graphical representation of the response status of the patient on a display screen. For example, the predicted response status of an individual may be displayed to a clinician or other medical service personnel, a clinical trial administrator, a researcher, or other entity via a graphical user interface (e.g., as part of a clinical trial management software or other software). In various embodiments, method S100 may optionally include selecting whether to include the patient in a patient group participating in a clinical trial according to the patient's response status. In a specific embodiment, it may be desirable to determine whether a candidate for a clinical trial can respond to an immuno-oncology therapeutic agent. Certain embodiments include treating an individual with a therapeutically effective amount of an immuno-oncology therapeutic agent, wherein the individual is a responder determined by a system or method disclosed herein, and the individual may or may not be a participant in a clinical trial. In a specific embodiment, an individual predicted to be a responder based on the measurement, determination, and / or generation of a plurality of neural network models covered herein is administered a therapeutically effective amount of an immuno-oncology therapeutic agent.

[0095] In various embodiments of method S100, the prediction of the patient's response state to the immuno-oncology therapeutic agent also uses one or more medical history features of the patient. In various embodiments of method S100, the third neural network is a segmentation algorithm that identifies the tumor region in the tissue sample. In various embodiments of method S100, the segmentation algorithm predicts a TIL mask, which is used in combination with the tumor mask to calculate one or more TIL-related metrics. In various embodiments of method S100, one or more TIL-related metrics include the predicted number of TIL clusters in the identified tumor region. In various embodiments, one or more TIL-related metrics include the predicted size of the TIL clusters in the identified tumor region.

[0096] In various embodiments of method S100, TIL-related metrics include intratumoral TIL% (iTIL%). In various embodiments, iTIL% is calculated as the percentage of the tumor area of ​​the mark in the tissue sample infiltrated by TIL. In various embodiments, the TIL% value is the matrix TIL% (sTIL%). In various embodiments, the sTIL% value is calculated as the percentage of the matrix in the tumor area of ​​the mark in the tissue sample infiltrated by TIL.

[0097] In various embodiments of method S100, the third neural network is a cell segmentation algorithm that calculates one or more cell-based features. In various embodiments, one or more cell-based features include morphology, color (nucleus), color (outside the nucleus), texture (energy, correlation, contrast, uniformity, dissimilarity, angular second moment) or chromatin skeleton morphology and color. In various embodiments, the cell segmentation algorithm generates a certain number of TIL cells or an average number of TIL cells. In various embodiments, the cell segmentation algorithm generates an average tumor cell size. In various embodiments, the cell segmentation algorithm generates an average TIL cell size. In various embodiments, the cell segmentation algorithm generates an average distance between TIL cells. In various embodiments, the cell segmentation algorithm generates an average color of TIL cells. In various embodiments, the cell segmentation algorithm generates an average distance between TIL cells to an average distance between tumor cells. In various embodiments, one or more TIL-related metrics include a predicted TIL cluster diffusion in the tumor region of the mark. In various embodiments, one or more TIL-related metrics include a total count of TIL in the tumor region of the mark. In various embodiments, one or more TIL-related metrics include a distance between TIL cells to a distance between tumor cells. In various embodiments, one or more TIL-related metrics include the distance between the TIL cells and the tumor cells. In various embodiments, one or more TIL-related metrics include the average color of the TIL cells. In various embodiments, one or more TIL-related metrics include the average size of the TIL cells.

[0098] In various embodiments of method S100, the first neural network model is a classification algorithm that generates a TMB indicator, which is a TMB positive classification or a TMB negative classification of a tissue sample. In various embodiments, if the tissue sample has a TMB value that exceeds a preset TMB threshold, the classification algorithm generates a TMB positive classification. In various embodiments, the first neural network model is a regression algorithm that generates a TMB indicator as a quantitative TMB score.

[0099] In various embodiments of method S100, the second neural network model is a classification algorithm that generates a PD-L1 expression level indicator, and the PD-L1 expression level indicator is a PD-L1 expression positive classification or a PD-L1 expression level negative classification. In various embodiments, the second neural network model is a classification algorithm that generates a PD-1 expression level indicator, and the PD-1 expression level indicator is a PD-1 expression positive classification or a PD-1 expression level negative classification. In various embodiments, if the tissue sample has a PD-L1 expression level value exceeding a preset PD-L1 expression level value, the classification algorithm generates a PD-L1 expression positive classification. In various embodiments, if the tissue sample has a PD-1 expression level value exceeding a preset PD-1 expression level value, the classification algorithm generates a PD-1 expression positive classification. In various embodiments, the second neural network model is a regression algorithm that generates a PD-L1 expression level indicator as a quantitative PD-L1 score. In various embodiments, the second neural network model is a regression algorithm that generates a PD-1 expression level indicator as a quantitative PD-1 score. In various embodiments, the quantitative PD-L1 score is a tumor proportion score (TPS) or a combined positive score (CPS). In various embodiments, the quantitative PD-1 score is a tumor proportion score (TPS) or a combined positive score (CPS). In various embodiments, the immuno-oncology therapeutic agent is a checkpoint inhibitor. In various embodiments, the checkpoint inhibitor is a PD-L1 inhibitor or a PD-1 inhibitor.

[0100] In various embodiments, the tissue sample is from the bladder, and the patient's response to the immuno-oncology therapeutic is associated with bladder cancer. In various embodiments, the patient is or will be a responder based on the response status. In various embodiments, based on the determination, a therapeutically effective amount of the immuno-oncology therapeutic is administered to the patient.

[0101] In various embodiments, it is determined that the patient is not or will not be a responder based on the response status. In various embodiments, based on this determination, the immuno-oncology therapeutic agent is not administered to the patient.

[0102] In various embodiments, the tissue sample is from a patient known to have or suspected of having PD-L1 positive cancer or PD-1 positive cancer. In various embodiments, the tissue sample is from a patient known to have or suspected of having non-small cell lung cancer, melanoma, Hodgkin's lymphoma, bladder cancer, kidney cancer or breast cancer. In various embodiments, the immuno-oncology therapeutic agent targets PD-L1 or PD-1. In various embodiments, the immuno-oncology therapeutic agent includes one or more antibodies, adoptive cell therapy agents, immunomodulators, or combinations thereof. In various embodiments, the antibody is a monoclonal antibody, a bispecific antibody, or a trispecific antibody. In various embodiments, the adoptive cell therapy agent includes immune cells expressing one or more engineered antigen receptors. In various embodiments, the engineered antigen receptor is a chimeric antigen receptor, a non-natural T cell receptor, or a combination thereof. In various embodiments, the adoptive cell therapy agent includes T cells, natural killer cells, natural killer T cells, or a combination thereof. In various embodiments, method S100 may optionally include the step of obtaining a sample from a patient. In various embodiments, method S100 may include the step of diagnosing a patient as having cancer.

[0103] Figure 2B Another method S200 for predicting a patient / treatment response to a therapeutic agent according to various embodiments is illustrated. The method S200 may be performed, for example, as described below with respect to Figure 3 The computing system 300 is implemented as described above. Figure 2B As shown, method S200 includes: at step S210, obtaining or receiving an image of a tissue sample from a patient and one or more features of the tissue sample from a data source by one or more processors; at step S220, performing the following items by one or more processors of the computing system 300 via one or more neural network models: (a) generating one or more tumor mutation burden (TMB) indicators from one or more features; (b) generating one or more programmed death-ligand 1 (PD-L1) expression level indicators from one or more features or generating one or more programmed death-1 (PD-1) expression level indicators from one or more features; and (c) generating one or more tumor infiltrating lymphocyte (TIL)-related metrics from one or more features; at step S230, using one or more TMB indicators, one or more PD-L1 expression level indicators or one or more PD-1 expression level indicators and one or more TIL-related metrics respectively by one or more processors via one or more neural network models to predict the patient's response status to an immuno-oncology therapeutic agent; and at step S240, selecting whether to include the patient in the patient group participating in the clinical trial based on the patient's response status. In various embodiments, method S200 can optionally include displaying a graphical representation on a display screen indicating the patient's response status.

[0104] In alternative embodiments, step S220 may be omitted, and step S230 may involve predicting the patient's response status to the immuno-oncology therapeutic agent from the image itself, without using any calculated / generated biomarkers.

[0105] In various embodiments of method S200, the prediction of the patient's response state to the immuno-oncology therapeutic agent also uses one or more medical history features of the patient. In various embodiments of method S200, one or more neural network models may include a segmentation algorithm configured to predict a TIL mask, which is used in combination with a tumor mask to calculate one or more TIL-related metrics. In various embodiments of method S200, the TIL-related metric is the predicted number of TIL clusters in the identified tumor region. In various embodiments, the TIL-related metric is the predicted size of the TIL cluster in the identified tumor region.

[0106] In various embodiments of method S200, TIL-related metrics include intratumoral TIL% (iTIL%). In various embodiments, iTIL% is calculated as the percentage of the tumor area of ​​the mark in the tissue sample infiltrated by TIL. In various embodiments, the TIL% value is the matrix TIL% (sTIL%). In various embodiments, the sTIL% value is calculated as the percentage of the matrix in the tumor area of ​​the mark in the tissue sample infiltrated by TIL.

[0107] In various embodiments of method S200, one or more neural network models may include a segmentation algorithm for cell segmentation, which may be referred to as a cell segmentation algorithm. In various embodiments, a cell segmentation algorithm may input an image and perform the segmentation of a cell. In various embodiments, the segmented cells may be labeled based on the regional segmentation output. For example, if the cell is in the TIL region, the segmented cells are labeled as TIL cells. In various embodiments, the cell segmentation algorithm is configured to calculate one or more features related to the cell contained in the image, also referred to herein as cell-based features. In various embodiments, the calculated cell-based features may include morphology, color (nucleus), color (outside the nucleus), texture (energy, correlation, contrast, uniformity, dissimilarity, angular second moment), chromatin skeleton morphology and color, and many other features that may be extracted. In various embodiments, the cell segmentation algorithm may be configured to generate a certain number of TIL cells or an average number of TIL cells. In various embodiments, the cell segmentation algorithm may be configured to generate an average tumor cell size. In various embodiments, the cell segmentation algorithm may be configured to generate an average TIL cell size. In various embodiments, the cell segmentation algorithm may be configured to generate an average distance between TIL cells. In various embodiments, the cell segmentation algorithm may be configured to generate an average color of TIL cells. In various embodiments, the cell segmentation algorithm may be configured to generate an average distance between TIL cells to an average distance between tumor cells. In various embodiments, the TIL-related metric is a predicted TIL cluster diffusion in the tumor region of the mark. In various embodiments, the TIL-related metric includes a total count of TILs in the tumor region of the mark. In various embodiments, the TIL-related metric may include the distance between TIL cells to the distance between tumor cells. In various embodiments, the TIL-related metric may include the distance between TIL cells and tumor cells. In various embodiments, the TIL-related metric may include the average color of TIL cells. In various embodiments, the TIL-related metric may include the average size of TIL cells.

[0108] In various embodiments of method S200, one or more neural network models may include a classification algorithm that generates a TMB indicator, which is a TMB-positive classification or a TMB-negative classification of a tissue sample. In various embodiments, if the tissue sample has a TMB value that exceeds a preset TMB threshold, the classification algorithm generates a TMB-positive classification.

[0109] In various embodiments, one or more neural network models may include a regression algorithm that generates a TMB indicator as a quantitative TMB score.

[0110] In various embodiments of method S200, one or more neural network models may include another classification algorithm that generates a PD-L1 expression level indicator, which is a PD-L1 expression positive classification or a PD-L1 expression level negative classification. In various embodiments, the classification algorithm generates a PD-1 expression level indicator, which is a PD-1 expression positive classification or a PD-1 expression level negative classification. In various embodiments, if the tissue sample has a PD-L1 expression level value that exceeds a preset PD-L1 expression level value, the classification algorithm generates a PD-L1 expression positive classification. In various embodiments, if the tissue sample has a PD-1 expression level value that exceeds a preset PD-1 expression level value, the classification algorithm generates a PD-1 expression positive classification.

[0111] In various embodiments, one or more neural network models may include another regression algorithm that generates a PD-L1 expression level indicator as a quantitative PD-L1 score. In various embodiments, another regression algorithm generates a PD-1 expression level indicator as a quantitative PD-1 score. In various embodiments, the quantitative PD-L1 score is a tumor proportion score (TPS) or a combined positive score (CPS). In various embodiments, the quantitative PD-1 score is a tumor proportion score (TPS) or a combined positive score (CPS). In various embodiments, the immuno-oncology therapeutic agent is a checkpoint inhibitor. In various embodiments, the checkpoint inhibitor is a PD-L1 inhibitor or a PD-1 inhibitor.

[0112] In various embodiments, the tissue sample is from the bladder, and the patient's response to the immuno-oncology therapeutic is associated with bladder cancer. In various embodiments, the patient is or will be a responder based on the response status. In various embodiments, based on the determination, a therapeutically effective amount of the immuno-oncology therapeutic is administered to the patient.

[0113] In various embodiments, it is determined that the patient is not or will not be a responder based on the response status. In various embodiments, based on this determination, the immuno-oncology therapeutic agent is not administered to the patient.

[0114] In various embodiments, the tissue sample is from a patient known to have or suspected of having PD-L1 positive cancer or PD-1 positive cancer. In various embodiments, the tissue sample is from a patient known to have or suspected of having non-small cell lung cancer, melanoma, Hodgkin's lymphoma, bladder cancer, kidney cancer or breast cancer. In various embodiments, the immuno-oncology therapeutic agent targets PD-L1 or PD-1. In various embodiments, the immuno-oncology therapeutic agent includes one or more antibodies, adoptive cell therapy agents, immunomodulators, or combinations thereof. In various embodiments, the antibody is a monoclonal antibody, a bispecific antibody, or a trispecific antibody. In various embodiments, the adoptive cell therapy agent includes immune cells expressing one or more engineered antigen receptors. In various embodiments, the engineered antigen receptor is a chimeric antigen receptor, a non-natural T cell receptor, or a combination thereof. In various embodiments, the adoptive cell therapy agent includes T cells, natural killer cells, natural killer T cells, or a combination thereof. In various embodiments, method S100 may optionally include the step of obtaining a sample from a patient. In various embodiments, method S100 may include the step of diagnosing a patient as having cancer.

[0115] Figure 2C A method S300 for predicting a patient / treatment response to a therapeutic agent according to various embodiments is illustrated. The method S300 may be performed, for example, as described below with respect to Figure 3 The computing system 300 is implemented as described above. Figure 2CAs shown, method S300 includes: at step S310, obtaining or receiving an image of a tissue sample from a patient, such as image 105, by one or more processors of the computing system 300; at step S320, generating one or more tumor mutation burden (TMB) indicators (e.g., as output 130) from one or more features by one or more processors via a first neural network model (e.g., model 120); at step S330, generating one or more programmed death-ligand 1 (PD-L1) expression level indicators from one or more features or generating one or more programmed death-1 (PD-1) expression level indicators from one or more features by one or more processors via a second neural network model (e.g., model 140); at step S340, generating one or more programmed death-ligand 1 (PD-L1) expression level indicators from one or more features by one or more processors via a second neural network model (e.g., model 140); at step S350, generating one or more programmed death-ligand 1 (PD-L1) expression level indicators from one or more features by one or more processors via a second neural network model (e.g., model 140); at step S360, generating one or more programmed death-1 (PD-1) expression level indicators from one or more features by one or more processors via a second neural network model (e.g., model 150); at step S371, generating one or more programmed death-1 (PD-1) expression level indicators from one or more features by one or more processors via a second neural network model (e.g., model 150); at step S382, generating one or more programmed death-1 (PD-1) expression level indicators from one or more features by one or more processors via a second neural network model (e.g., model 150); at step S393, generating one or more programmed death-1 (PD-1) expression level indicators from one or more features by one or more processors via a second neural network model (e. The processor generates one or more tumor infiltrating lymphocyte (TIL)-related metrics from one or more features via a third neural network model (such as model 160a) (e.g., as output 170a); at step S350, one or more processors generate one or more cell-based features from one or more features via a fourth neural network model (such as model 160b) (e.g., as output 170b); and at step S360, one or more processors predict a patient's response status to an immuno-oncology therapeutic agent, such as response 190, using one or more TMB indicators, one or more PD-L1 expression level indicators or one or more PD-1 level indicators, one or more TIL-related metrics, and one or more cell-based features, respectively, via a fourth neural network model (such as model 180).

[0116] exist Figure 2A-2C In any of the methods, the method may lack a neural network model associated with one or more tumor mutation burden (TMB), such as Figure 2D-2F shown.

[0117] In method S105 Figure 2D In the example, S110-2 corresponds to Figure 2A The S110, Figure 2D In the example, S120-2 corresponds to Figure 2A The S130, and in Figure 2D In the example, S130-2 corresponds to Figure 2A The S140. Figure 2D In the example, S140-2 corresponds to Figure 2A S150.

[0118] In method S205 Figure 2E In the example, S210-2 corresponds to Figure 2B S210. Figure 2E In the example, S220-2 corresponds to Figure 2BS220 (except S220-2 lacks the steps related to TMB indicator). Figure 2E In the example, S230-2 corresponds to Figure 2B S230 (except S230-2 lacks the steps related to TMB indicator). Figure 2E In the example, S240-2 corresponds to Figure 2B S240.

[0119] In method 305 Figure 2F In the example, S310-2 corresponds to Figure 2C S310. Figure 2F In the example, S320-2 corresponds to Figure 2C The S330. Figure 2F In the example, S330-2 corresponds to Figure 2C The S340. Figure 2F In the example, S340-2 corresponds to Figure 2C The S350. Figure 2F In the S350-2, Figure 2C of S360 (except that S350-2 lacks the steps related to the TMB indicator).

[0120] III. Computer-Implemented Systems

[0121] Figure 3 300 is a block diagram illustrating a computer system 300 according to various embodiments, the computer system being configured to perform a method of predicting a patient / treatment response to a therapeutic agent, with which embodiments of the disclosed systems and methods or portions thereof may be implemented. For example, the illustrated computer system may be a local or remote computer system operably connected to a control system for controlling or monitoring the systems and methods of various embodiments herein. In various embodiments of the present teachings, the computer system 300 may include a bus 302 or other communication mechanism for conveying information and a processor 304 coupled to the bus 302 to process information. In various embodiments, the computer system 300 may also include a memory coupled to the bus 302 for determining instructions to be executed by the processor 304, which memory may be a random access memory (RAM) 306 or other dynamic storage device. The memory may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 304. In various embodiments, the computer system 300 may also include a read-only memory (ROM) 308 or other static storage device coupled to the bus 302 for storing static information and instructions for the processor 304. A storage device 310 , such as a magnetic or optical disk, may be provided and coupled to bus 302 for storing information and instructions.

[0122] In various embodiments, the computer system 300 may be coupled to a display 312, such as a cathode ray tube (CRT) or a liquid crystal display (LCD), via the bus 302 for displaying information to a computer user. An input device 314 including alphanumeric and other keys may be coupled to the bus 302 for communicating information and command selections to the processor 304. Another type of user input device is a cursor control 316, such as a mouse, trackball, or cursor direction keys, which is used to communicate direction information and command selections to the processor 304, and to control cursor movement on the display 312. The input device 314 typically has two degrees of freedom in two axes, a first axis (i.e., x-axis) and a second axis (i.e., y-axis), which allows the device to specify a position in a plane. However, it should be understood that an input device 314 that allows 3-dimensional (x, y, and z) cursor movement is also contemplated herein. According to various embodiments, components 312 / 314 / 316 may together or individually constitute a control system that connects the rest of the components of the computer system to the systems herein and methods performed on such systems, and controls the execution of the methods and operation of the associated systems.

[0123] Consistent with certain implementations of the present teachings, computer system 300 may provide results in response to processor 304 executing one or more sequences of one or more instructions contained in memory 306. Such instructions may be read into memory 306 from another computer-readable medium or a computer-readable storage medium, such as storage device 310. Execution of the sequences of instructions contained in memory 306 may cause processor 304 to perform the processes described herein. Alternatively, hard-wired circuitry may be used in place of or in combination with software instructions to implement the present teachings. Thus, implementations of the present teachings are not limited to any specific combination of hardware circuitry and software.

[0124] As used herein, the term "computer-readable medium" (e.g., data repository, data storage space, etc.) or "computer-readable storage medium" refers to any medium that participates in providing instructions to processor 304 for execution. Such media may take a variety of forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Examples of non-volatile media may include, but are not limited to, dynamic memory, such as memory 306. Examples of transmission media may include, but are not limited to, coaxial cables, copper wire, and optical fiber, including the wires that make up bus 302.

[0125] Common forms of computer readable media include, for example, a floppy disk, a floppy disk, a hard disk, magnetic tape or any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM and EPROM, FLASH-EPROM, another memory chip or cartridge, or any other tangible medium from which a computer can read.

[0126] In addition to computer-readable media, instructions or data may also be provided as signals on a transmission medium included in a communication device or system to provide a sequence of one or more instructions to the processor 304 of the computer system 300 for execution. For example, a communication device may include a transceiver with signals indicating instructions and data. These instructions and data are configured to cause one or more processors to implement the functions outlined in the disclosure of this article. Representative examples of data communication transmission connections may include, but are not limited to, telephone modem connections, wide area networks (WANs), local area networks (LANs), infrared data connections, NFC connections, etc.

[0127] It should be understood that the methods, flow charts, diagrams, and the accompanying disclosure described herein can be implemented using computer system 300 as a standalone device or on a distributed network or shared computer processing resources (such as a cloud computing network). In one or more embodiments, computer system 300 may include a single computer (or computer system) or multiple computers that communicate with each other in a distributed implementation.

[0128] The methods described herein can be implemented in various ways depending on the application. For example, these methods can be implemented in hardware, firmware, software, or any combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, or combinations thereof.

[0129] In various embodiments, the methods of the present teachings may be implemented as firmware and / or software programs and applications written in conventional programming languages ​​(such as C, C++, Python, etc.). If implemented as firmware and / or software, the embodiments described herein may be implemented on a non-transitory computer-readable medium in which a program for causing a computer to perform the above methods is stored. It should be understood that the various engines described herein may be provided on a computer system (such as computer system 300), whereby the processor 304 will perform the analysis and determination provided by these engines, subject to instructions provided by any one or a combination of memory components 306 / 308 / 310 and user input provided via input device 314.

[0130] Although the present teachings are described in conjunction with various embodiments, it is not intended that the present teachings be limited to such various embodiments. On the contrary, the present teachings encompass various alternatives, modifications, and equivalents, as will be understood by those skilled in the art.

[0131] When describing various embodiments, this specification may present a method and / or process as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of steps set forth herein, the method or process should not be limited to the specific sequence of steps described, and those skilled in the art can readily understand that the sequence can be varied and still remain within the spirit and scope of the various embodiments.

[0132] V. Tumor Mutation Burden (TMB)

[0133] In certain embodiments, the systems, compositions and methods disclosed herein utilize tumor mutation burden (TMB) as at least one parameter or metric to determine whether a patient will be a responder to a particular type of therapy. In various embodiments, tumor mutation burden may include the total number of mutations (changes) present in cancer cell DNA. In specific cases, cancers (including solid tumors) with a large number of mutations may be more likely to respond to certain types of immunotherapy (including immuno-oncology therapeutic agents). In specific embodiments, TMB is defined as the number of non-genetic mutations per million bases (Mb) in the genomic sequence studied (Merino et al., (2020). J Immunother Cancer. 8 (1): e000147, the entire document is incorporated herein by reference). The measurement of TMB can be performed by any suitable method, including by targeted panel sequencing, gene targeted sequencing or whole exome sequencing, and in specific embodiments by, for example, next generation sequencing (NGS) is performed. The quantitative value of the TMB of a sample can be considered as a TMB index, and in specific cases, the amount is measured as the number of mutations per megabase (Mb). In the systems, compositions and methods of the present disclosure, one or more tumor mutation burden (TMB) indicators may be generated from one or more features of an image of a tissue sample from an individual suffering from or suspected of suffering from bladder cancer (by one or more processors and via a neural network model). In a specific embodiment, an image of a tissue sample (from a patient) may be obtained by one or more processors, the image comprising an image of one or more features of the tissue sample; then one or more TMB indicators may be generated from the one or more features by one or more processors and via a first neural network model.

[0134] In certain embodiments, the neural network model is configured to generate one or more TMB indicators from one or more features of a patient tissue sample. In a specific embodiment, the neural network model includes a classification algorithm, and one or more TMB indicators include a TMB indicator of a tissue sample, which is a TMB positive classification or a TMB negative classification. In some embodiments, the neural network model includes a classification algorithm, and one or more TMB indicators include a TMB indicator of a tissue sample, and if the tissue sample has a TMB value exceeding a preset TMB threshold, the TMB indicator is a positive classification. In such cases where a classification algorithm is utilized, in certain embodiments, the output may be considered to be TMB high or TMB low. In certain embodiments, TMB high may be >= 10, while TMB low is less than 10.

[0135] However, in certain embodiments, the associated neural network model comprises a regression algorithm, and the one or more TMB indicators comprise a TMB indicator as a quantitative TMB score. In particular instances, the TMB score can be in the range of 0-100, and the preset TMB threshold can be any value therein, including 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65 , 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99 or 100.

[0136] In certain cases, hematoxylin and eosin (H&E) images from muscle invasive bladder cancer (MIBC) patients are used, for example, to train the TMB prediction parameters of the model. In various embodiments, TMB is calculated from an NGS molecular panel applied to the tissue in the generation of the model and when it is used.

[0137] VI. Checkpoint Expression Levels

[0138] In embodiments of the present disclosure, systems, methods, and compositions utilize checkpoint expression levels as at least one parameter or metric to determine whether a patient will be a responder to a particular type of therapy. Although in some cases, the checkpoint is any kind of checkpoint, in specific embodiments, the checkpoint is an inhibitory checkpoint. In certain embodiments, the checkpoint is programmed death-1 (PD-1), programmed death ligand-1 (PD-L1). In specific embodiments, the expression of the checkpoint is quantitative expression based on images of tissue samples from patients. In specific embodiments, the expression of the checkpoint is quantitative expression based on H&E images of tissue samples from patients. In specific embodiments, a neural network model can be trained based on one or more features of one or more PD-L1 expression level indicators or PD-1 expression level indicators from H&E images (e.g., from bladder cancer patients, including muscle-invasive bladder cancer patients).

[0139] In certain embodiments, the neural network model includes a classification algorithm, and one or more PD-L1 expression level indicators include a PD-L1 expression level indicator, which is a PD-L1 expression positive classification or a PD-L1 expression level negative classification. In other embodiments, the neural network model includes a classification algorithm, and one or more PD-1 expression level indicators include a PD-1 expression level indicator, which is a PD-1 expression positive classification or a PD-1 expression level negative classification. In some embodiments, if the tissue sample has a PD-L1 expression level value exceeding a preset PD-L1 expression level value, the classification algorithm generates a PD-L1 expression positive classification. In such cases, the output may be PD-L1 high or PD-L1 low. In some embodiments, if the tissue sample has a PD-1 expression level value exceeding a preset PD-1 expression level value, the classification algorithm generates a PD-1 expression positive classification. In such cases, the output may be PD-1 high or PD-1 low. The thresholds for PD-L1 high and PD-L1 low or PD-1 high and PD-1 low may vary according to immuno-oncology therapeutic agents and / or cancers. In certain embodiments, there are different thresholds that apply based on the drug and the cancer.

[0140] In certain embodiments, the neural network model comprises a regression algorithm, and the one or more PD-L1 expression level indicators comprise a quantitative PD-L1 score, which may be a tumor proportion score (TPS) or a combined positive score (CPS). In certain embodiments, the neural network model comprises a regression algorithm, and the one or more PD-1 expression level indicators comprise a quantitative PD-1 score, which may be a tumor proportion score (TPS) or a combined positive score (CPS). For the regression algorithms for both PD-L1 and PD-1, the TPS score may range from 0-100, such as 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45 , 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99 or 100. For the regression algorithms for both PD-L1 and PD-1, the CPS score may range from 0-100, such as 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45 , 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99 or 100.In specific embodiments, PD-L1 high or PD-1 high can be TPS or CPS ≥ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81 7, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99 or 100.

[0141] In some embodiments, the model directly predicts TPS score and / or CPS score (ie, regression model). In some embodiments, the model directly predicts PD-L1 high versus PD-L1 low for different thresholds, or directly predicts PD-1 high versus PD-1 low for different thresholds. For example, in specific cases, TPS>=1, TPS>=10, or TPS>=50 may be used.

[0142] VII. Tumor-infiltrating lymphocyte (TIL) related metrics

[0143] Tumor infiltrating lymphocytes are a class of white blood cells that migrate from the blood to the tumor. They may include T cells and B cells, and they recognize and kill cancer cells. In a specific embodiment, the presence of lymphocytes in a tumor is generally associated with better clinical outcomes. The present disclosure provides systems, methods, and compositions that utilize TIL-related metrics as at least one parameter to determine whether a patient will be a responder to a specific type of therapy. In a specific embodiment, the presence of TILs in a tumor sample is measured by any suitable method, including immunohistochemical staining, H&E staining, or both. In a specific embodiment, the presence of TILs in a tumor is determined by a pathologist, and the pathologist may observe the TIL cells using IHC staining on a tissue sample, or they may be observed in an H&E slide. TILs have distinguishing features to allow pathologists to distinguish them from other cells, such as being more purple, darker, and rounder than tumor cells. In a specific embodiment, the algorithm is trained using TIL pathologists' annotations to H&E images. In a specific embodiment, a pathologist evaluates TILs in the stroma compartment of a tumor. In a specific embodiment, a pathologist evaluates TILs at the border of, for example, an invasive tumor. In specific cases, TILs outside the tumor border, areas of necrosis, areas of fibrosis, and / or associated with abscesses are excluded. A pathologist may evaluate one or more H&E-stained sections of the tumor.

[0144] In certain embodiments, the neural network model is configured to calculate one or more TIL-related metrics from one or more features of the tissue sample. In certain embodiments, TIL-related metrics can be calculated based on TIL masks, tumor masks, necrosis masks, matrix masks, and cell nucleus segmentation masks. Several masks can be used to calculate "TIL-related" metrics (such as iTIL%, sTIL%, TIL clusters, etc.). The neural network can predict the segmentation mask (i.e., the TIL mask at the top of the tissue image, together with other masks such as tumors, matrix, necrosis, and cell nucleus segmentation). These segmentation masks are then used to calculate metrics, such as the following features: tumor infiltrating lymphocytes (TIL) percentage (%), total count of TIL, number of TIL clusters, distance between TIL cells to distance between tumor cells; distance between TIL cells and tumor cells; average color of TIL cells, average size of TIL cells, etc.

[0145] In various embodiments, the neural network model can be a cell segmentation algorithm that can input an image and perform the segmentation of cells. In various embodiments, the cells segmented can be labeled based on the regional segmentation output. For example, if the cell is in the TIL region, the segmented cells are labeled as TIL cells. In various embodiments, the cell segmentation algorithm can calculate one or more cell-based features. In various embodiments, cell-based features may include morphology, color (nucleus), color (outside the nucleus), texture (energy, correlation, contrast, uniformity, dissimilarity, angular second moment), chromatin skeleton morphology and color, and many other features that can be extracted. In various embodiments, the cell segmentation algorithm can be configured to generate a certain number of TIL cells or an average number of TIL cells, an average tumor cell size, an average TIL cell size, an average distance between TIL cells, an average color of TIL cells, and an average distance between TIL cells to an average distance between tumor cells. In various embodiments, the TIL-related metric is the predicted TIL cluster spread in the identified tumor region, the total count of TILs in the identified tumor region, the distance between TIL cells versus the distance between tumor cells, the distance between TIL cells and tumor cells, the average color of TIL cells and / or the average size of TIL cells.

[0146] VIII. Immuno-Oncology Therapeutics

[0147] The present disclosure provides the following embodiments, wherein individuals in need of immuno-oncology therapy are evaluated as responders of immuno-oncology therapy. Embodiments of the present disclosure include predicting the response of patients to immuno-oncology therapeutic agents. In a specific embodiment, the immuno-oncology therapeutic agent is an immune checkpoint inhibitor. The immuno-oncology therapeutic agent can be an antibody or can be an adoptive cell therapy agent using, for example, an immune cell, or both can be used. In a specific embodiment, the antibody is a monoclonal antibody, and the monoclonal antibody can target any checkpoint. In a specific case, the antibody can be monospecific or bispecific. For example, the antibody can target PD-L1 or PD-1. When adoptive cell therapy agents are used, the cell can be an immune cell, such as a T cell, a natural killer (NK) cell, or an NK T cell. In a specific embodiment, the cell can be engineered to express a non-natural protein, such as an antigen receptor, and the antigen receptor can include an extracellular domain, which is an antibody targeting an immune checkpoint. In other cases, the antigen receptor includes an extracellular domain that binds to an immune checkpoint. In a specific embodiment, the immuno-oncology therapeutic agent is a checkpoint inhibitor, such as a PD-L1 inhibitor or a PD-1 inhibitor. In certain embodiments, immuno-oncology therapeutics may be combined with one or more other types of cancer therapy, such as surgery, radiation, hormone therapy, other immunotherapies, or chemotherapy.

[0148] **********

[0149] In a specific embodiment, the method includes utilizing an image of a tissue sample from a patient, wherein the tissue sample includes one or more features. In a specific embodiment, the method includes obtaining an image of a tissue sample from a patient and one or more features of a tissue sample from a data source by one or more processors. In either case, the method may include performing the following such as by one or more processors via one or more neural network models: (a) generating one or more tumor mutation burden (TMB) indicators from one or more features; (b) generating one or more programmed death-ligand 1 (PD-L1) expression level indicators from one or more features or generating one or more programmed death-1 (PD-1) expression level indicators from one or more features; and (c) generating one or more tumor infiltrating lymphocytes (TIL) metrics from one or more features; using one or more TMB indicators, one or more PD-L1 expression level indicators or one or more PD-1 expression level indicators and one or more TIL metrics respectively by one or more processors via one or more neural network models to predict the patient's response status to an immuno-oncology therapeutic agent; and selecting whether to include the patient in a patient group participating in a clinical trial or whether to administer a therapeutically effective amount of one or more immuno-oncology therapeutic agents to the patient based on the patient's response status.

[0150] In a specific embodiment, the method includes utilizing an image of a tissue sample from a patient, wherein the tissue sample includes one or more features; obtaining, by one or more processors, an image of the tissue sample from the patient and one or more features of the tissue sample from a data source; performing, by one or more processors, via one or more neural network models, the following: (a) generating one or more tumor mutation burden (TMB) indicators from the one or more features; (b) generating one or more programmed death-ligand 1 (PD-L1) expression level indicators from the one or more features or generating one or more programmed death-1 (PD-1) expression level indicators from the one or more features; and (c) generating one or more tumor infiltrating lymphocyte (TIL) percentage (%) values ​​(as an example of a TIL metric) from the one or more features; predicting, by one or more processors, via one or more neural network models, the response status of a patient to an immuno-oncology therapeutic agent using, respectively, one or more TMB indicators, one or more PD-L1 expression level indicators or one or more PD-1 expression level indicators, and one or more TIL% values; and selecting whether to include the patient in a group of patients participating in a clinical trial based on the patient's response status.

[0151] Various embodiments of the method include obtaining an image of a tissue sample from a patient by one or more processors, the image including one or more features of the tissue sample. In some cases, the party obtaining the image is the same party of the treatment process and / or generation step. In other cases, the party obtaining the image is not the same party of the treatment process and / or generation step. The tissue sample may be obtained from a storage space, or may be evaluated shortly after collection. The tissue sample may or may not be derived from a biopsy of an individual who has undergone a cancer diagnosis. Based on one or more symptoms, the individual may be known to have cancer or may be suspected of having cancer.

[0152] In specific embodiments, the systems, methods, and compositions of the present disclosure can be used for any type of cancer, including hematological tumors and solid tumors. The cancer can be at any stage or grade; it can be refractory or not; it can be metastatic or not. The cancer can be bladder cancer, brain cancer, breast cancer, lung cancer, colon cancer, pancreatic cancer, prostate cancer, liver cancer, spleen cancer, kidney cancer, stomach cancer, cervical cancer, uterine cancer, ovarian cancer, testicular cancer, gallbladder cancer, bone cancer, thyroid cancer, skin cancer, endometrial cancer, rectal cancer, etc.

[0153] In some embodiments, the subject has one or more symptoms of bladder cancer, such as hematuria; urinating more frequently than usual; pain or burning during urination; feeling the need to urinate immediately, even if the bladder is not full; difficulty urinating or a weak urine stream; getting up many times at night to urinate; inability to urinate; pain on one side of the back; loss of appetite, weight loss; feeling tired or weak; swollen feet; bone pain; or a combination thereof. In such cases, the subject may be suspected of having bladder cancer or may be known to have bladder cancer. The diagnosis of bladder cancer can be performed using cystoscopy, biopsy, blood tests, urine cytology, computed tomography (CT) urography, retrograde pyelography, or a combination thereof. Other tests may be performed to determine the extent of the cancer, such as magnetic resonance imaging (MRI), positron emission tomography (PET), bone scan, and / or chest X-ray. Bladder cancer can be urothelial carcinoma, squamous cell carcinoma, or adenocarcinoma. Bladder cancer can be non-invasive, non-muscle invasive, or muscle invasive. Bladder cancer can be at any stage or grade, including non-invasive papillary carcinoma, carcinoma in situ, T1, T2, T3, or T4.

[0154] Embodiments of the present disclosure include methods for treating individuals determined as responders based on systems, methods, and compositions as contemplated herein. In specific embodiments, based on measurements performed by systems, methods, and compositions as contemplated herein, a therapeutically effective amount of one or more immuno-oncology therapeutic agents is administered to responders. In specific embodiments, based on measurements performed by systems, methods, and compositions as contemplated herein, one or more immuno-oncology therapeutic agents are not administered to individuals who are not responders. In various embodiments, an individual determined to be or predicted to be responsive to one or more immuno-oncology therapeutic agents is administered a therapeutically effective amount of one or more immuno-oncology therapeutic agents based on having appropriate one or more TMB indicators, one or more PD-L1 and / or PD-1 expression level indicators, and one or more TIL-related metrics (all from one or more features of an image of a tissue sample). In certain embodiments, a therapeutically effective amount of one or more immuno-oncology therapeutics is administered to an individual determined or predicted to be responsive to one or more immuno-oncology therapeutics based on: providing an image of a tissue sample from a patient, the tissue sample comprising one or more features of the tissue sample; generating, by one or more processors, one or more TMB indicators from the one or more features via a first neural network model; generating, by one or more processors, one or more PD-L1 expression level indicators from the one or more features via a second neural network model; generating, by one or more processors, one or more TIL-related metrics from the one or more features via a third neural network model; and predicting, by one or more processors, via a fourth neural network model, that the individual will respond to the immuno-oncology therapeutic using the one or more TMB indicators, the one or more PD-L1 expression level indicators, and the one or more TIL-related metrics. In certain embodiments, a therapeutically effective amount of one or more immuno-oncology therapeutics is administered to an individual who is determined or predicted to be responsive to one or more immuno-oncology therapeutics based on: obtaining, by one or more processors, an image of a tissue sample from a patient and one or more features of the tissue sample from a data source; performing, by one or more processors, via one or more neural network models: (a) generating one or more tumor mutation burden (TMB) indicators from the one or more features; (b) generating one or more programmed death-ligand 1 (PD-L1) expression level indicators from the one or more features; and (c) generating one or more tumor infiltrating lymphocyte (TIL) percentage (%) values ​​from the one or more features; and determining or predicting, by one or more processors, via one or more neural network models, that the patient will respond to the immuno-oncology therapeutic using one or more TMB indicators, one or more PD-L1 expression level indicators, and one or more TIL-related metric values.

[0155] IX. Detailed description of implementation plan

[0156] Embodiment 1. A system for predicting a patient's response to an immuno-oncology therapeutic agent, the system comprising: a data repository for storing images of tissue samples from patients, the images comprising one or more features of the tissue samples; a computing device, the computing device communicatively connected to the data repository, the computing device comprising: a first neural network model, the first neural network model is configured to generate one or more tumor mutation burden (TMB) indicators from the one or more features; a second neural network model, the second neural network model is configured to generate one or more programmed death-ligand 1 (PD-L1) expression level indicators from the one or more features or is configured to generate one or more tumor mutation burden (TMB) indicators from the one or more features; features to generate one or more programmed death-1 (PD-1) expression level indicators; a third neural network model, the third neural network model is configured to calculate one or more tumor infiltrating lymphocyte (TIL)-related metrics from the one or more features; or a fourth neural network model, the fourth neural network model is configured to predict the patient's response status to the immuno-oncology therapeutic agent based on the one or more TMB indicators, the one or more PD-L1 expression level indicators or the one or more PD-1 expression level indicators and / or the one or more TIL-related metrics; and a display system, the display system is communicatively connected to the computing device and is configured to display a result indicating the response status of the patient.

[0157] Embodiment 2. A system according to embodiment 1, wherein the prediction of the response status of the patient to the immuno-oncology therapeutic agent is performed based on one or more medical history characteristics of the patient.

[0158] Embodiment 3. A system according to embodiment 1 or 2, wherein the third neural network model includes a segmentation algorithm, which is configured to generate an identification of the tumor region in the tissue sample based on the one or more features, wherein the identification is used to further generate the one or more TIL-related metrics.

[0159] Embodiment 4. A system according to embodiment 3, wherein the segmentation algorithm is configured to predict a TIL mask, which is used in combination with a tumor mask to calculate one or more TIL-related metrics.

[0160] Embodiment 5. A system according to embodiment 4, wherein the one or more TIL-related metrics include a predicted number of TIL clusters in the identified tumor region.

[0161] Embodiment 6. A system according to embodiment 4, wherein the one or more TIL-related metrics include a predicted size of a TIL cluster in the identified tumor region.

[0162] Embodiment 7. A system according to embodiment 4, wherein the one or more TIL-related metrics include predicted TIL cluster spread in the identified tumor region.

[0163] Embodiment 8. A system according to embodiment 4, wherein the one or more TIL-related metrics include a total count of TILs in the identified tumor region.

[0164] Embodiment 9. A system according to embodiment 4, wherein the one or more TIL-related metrics include the distance between TIL cells versus the distance between tumor cells.

[0165] Embodiment 10. A system according to embodiment 4, wherein the one or more TIL-related metrics include the distance between the TIL cells and the tumor cells.

[0166] Embodiment 11. A system according to embodiment 4, wherein the one or more TIL-related metrics include an average color of TIL cells.

[0167] Embodiment 12. A system according to embodiment 4, wherein the one or more TIL-related metrics include the average size of TIL cells.

[0168] Embodiment 13. A system according to any one of embodiments 1-12, wherein the one or more TIL-related metrics include intratumoral TIL% (iTIL%).

[0169] Embodiment 14. A system according to embodiment 13, wherein the iTIL% is calculated as the percentage of the identified tumor area in the tissue sample that is infiltrated by TILs.

[0170] Embodiment 15. A system according to embodiment 1, wherein the one or more TIL-related metrics include stromal TIL% (sTIL%).

[0171] Embodiment 16. A system according to embodiment 15, wherein the sTIL% value is calculated as the percentage of the stroma in the identified tumor region in the tissue sample that is infiltrated by TILs.

[0172] Embodiment 17. A system according to any one of embodiments 1-16, wherein the first neural network model includes a classification algorithm, and the one or more TMB indicators include a TMB indicator of the tissue sample, and the TMB indicator is a TMB-positive classification or a TMB-negative classification.

[0173] Embodiment 18. The system of embodiment 17, wherein the classification algorithm generates the TMB-positive classification if the tissue sample has a TMB value exceeding a preset TMB threshold.

[0174] Embodiment 19. A system according to any one of Embodiments 1-16, wherein the first neural network model comprises a regression algorithm and the one or more TMB indicators comprise a TMB indicator as a quantitative TMB score.

[0175] Embodiment 20. A system according to any one of Embodiments 1-19, wherein the second neural network model comprises a classification algorithm, and the one or more PD-L1 expression level indicators comprise a PD-L1 expression level indicator, and the PD-L1 expression level indicator is a PD-L1 expression positive classification or a PD-L1 expression level negative classification.

[0176] Embodiment 21. A system according to any one of Embodiments 1-19, wherein the second neural network model includes a classification algorithm, and the one or more PD-1 expression level indicators include a PD-1 expression level indicator, and the PD-1 expression level indicator is a PD-1 expression positive classification or a PD-1 expression level negative classification.

[0177] Embodiment 22. A system according to Embodiment 20, wherein if the tissue sample has a PD-L1 expression level value exceeding a preset PD-L1 expression level value, the classification algorithm generates the PD-L1 expression positive classification.

[0178] Embodiment 23. A system according to Embodiment 21, wherein if the tissue sample has a PD-1 expression level value exceeding a preset PD-1 expression level value, the classification algorithm generates the PD-1 expression positive classification.

[0179] Embodiment 24. A system according to any one of Embodiments 1-19, wherein the second neural network model comprises a regression algorithm and the one or more PD-L1 expression level indicators comprise a quantitative PD-L1 score.

[0180] Embodiment 25. A system according to embodiment 1, wherein the second neural network model comprises a regression algorithm and the one or more PD-1 expression level indicators comprise a quantitative PD-1 score.

[0181] Embodiment 26. A system according to embodiment 24, wherein the quantitative PD-L1 score is a tumor proportion score (TPS).

[0182] Embodiment 27. A system according to Embodiment 24, wherein the quantitative PD-L1 score is a combined positive score (CPS).

[0183] Embodiment 28. A system according to Embodiment 25, wherein the quantitative PD-1 score is a tumor proportion score (TPS).

[0184] Embodiment 29. A system according to Embodiment 25, wherein the quantitative PD-1 score is a combined positive score (CPS).

[0185] Embodiment 30. A system according to any one of Embodiments 1-29, wherein the immuno-oncology therapeutic agent is a checkpoint inhibitor.

[0186] Embodiment 31. A system according to embodiment 30, wherein the checkpoint inhibitor is a PD-L1 inhibitor or a PD-1 inhibitor.

[0187] Embodiment 32. A system according to any one of Embodiments 1-31, wherein the third neural network model is further configured to generate one or more TIL masks from the one or more features; and wherein the fourth neural network model is further configured to use the one or more TIL masks to predict the response status of the patient to the immuno-oncology therapeutic agent.

[0188] Embodiment 33. A system according to any one of Embodiments 1, wherein the tissue sample is from the bladder of the patient; and wherein the patient's response to the immuno-oncology therapeutic agent is associated with bladder cancer.

[0189] Embodiment 34. A system according to any one of Embodiments 1-33, wherein the tissue sample is from a patient known to have PD-L1 positive cancer.

[0190] Embodiment 35. A system according to any one of Embodiments 1-33, wherein the tissue sample is from a patient known to have a PD-1 positive cancer.

[0191] Embodiment 36. A system according to any one of embodiments 1-35, wherein the tissue sample is from a patient known to have any one or more of the following: non-small cell lung cancer, melanoma, Hodgkin lymphoma, bladder cancer, kidney cancer, and breast cancer.

[0192] Embodiment 37. A system according to any one of Embodiments 1-36, wherein the immuno-oncology therapeutic targets PD-L1 or PD-1.

[0193] Embodiment 38. A system according to any one of Embodiments 1-37, wherein the immuno-oncology therapeutic agent comprises one or more antibodies, one or more adoptive cell therapy agents, or a combination thereof.

[0194] Embodiment 39. A system according to Embodiment 38, wherein the one or more antibodies include at least one of the following: a monoclonal antibody, a bispecific antibody, or a trispecific antibody.

[0195] Embodiment 40. A system according to Embodiment 38, wherein the one or more adoptive cell therapy agents include immune cells expressing one or more engineered antigen receptors.

[0196] Embodiment 41. A system according to Embodiment 40, wherein the one or more engineered antigen receptors include at least one of the following: a chimeric antigen receptor, a non-natural T cell receptor, or a combination thereof.

[0197] Embodiment 42. A system according to any one of Embodiments 38-41, wherein the one or more adoptive cell therapy agents include at least one of the following: T cells, natural killer cells, natural killer T cells, or a combination thereof.

[0198] Embodiment 43. A system according to embodiment 1 or 2, wherein the third neural network model includes a region / TIL segmentation algorithm and a cell segmentation algorithm.

[0199] Embodiment 44. A system according to embodiment 43, wherein the cell segmentation algorithm calculates one or more cell-based features.

[0200] Embodiment 45. A system according to embodiment 44, wherein the one or more cell-based features include morphology, color (cell nucleus), color (outside cell nucleus), texture (energy, correlation, contrast, uniformity, dissimilarity or angular second moment).

[0201] Embodiment 46. A system according to Embodiment 43, wherein the cell segmentation algorithm generates a certain TIL cell number or an average TIL cell number.

[0202] Embodiment 47. A system according to embodiment 43, wherein the cell segmentation algorithm generates an average tumor cell size.

[0203] Embodiment 48. A system according to embodiment 43, wherein the cell segmentation algorithm generates an average TIL cell size.

[0204] Embodiment 49. A system according to Embodiment 43, wherein the cell segmentation algorithm generates an average distance between TIL cells.

[0205] Embodiment 50. A system according to embodiment 43, wherein the cell segmentation algorithm generates an average color of TIL cells.

[0206] Embodiment 51. A system according to Embodiment 43, wherein the cell segmentation algorithm generates the average distance between TIL cells versus the average distance between tumor cells.

[0207] Embodiment 52. A method for predicting a patient's response to an immuno-oncology therapeutic agent, the method comprising: obtaining an image of a tissue sample from a patient through one or more processors, the image comprising one or more features of the tissue sample; generating one or more tumor mutation burden (TMB) indicators from the one or more features through a first neural network model through the one or more processors; generating one or more programmed death-ligand 1 (PD-L1) expression level indicators from the one or more features or generating one or more programmed death-1 (PD-1) expression level indicators from the one or more features through a second neural network model through the one or more processors; generating one or more tumor infiltrating lymphocyte (TIL)-related metrics from the one or more features through a third neural network model through the one or more processors; and predicting the patient's response status to the immuno-oncology therapeutic agent using the one or more TMB indicators, the one or more PD-L1 expression level indicators or the one or more PD-1 level indicators and the one or more related metrics through the one or more processors through a fourth neural network model.

[0208] Embodiment 53. The method according to embodiment 52, further comprising:

[0209] Whether to include the patient in a group of patients participating in a clinical trial is selected based on the response status of the patient.

[0210] Embodiment 54. The method according to embodiment 52 or 53 further comprises: displaying a graphical representation indicating the response status of the patient on a display screen.

[0211] Embodiment 55. A method according to any one of Embodiments 52-54, wherein said prediction of said response status of said patient to said immuno-oncology therapeutic agent further uses one or more medical history characteristics of said patient.

[0212] Embodiment 56. A method according to any one of Embodiments 52-55, wherein the third neural network is a segmentation algorithm that identifies tumor regions in the tissue sample.

[0213] Embodiment 57. A method according to Embodiment 56, wherein the segmentation algorithm predicts one or more TIL mask structural features in the identified tumor region to generate the one or more TIL-related metrics.

[0214] Embodiment 58. A method according to Embodiment 57, wherein the predicted TIL mask structural feature is the predicted number of TIL clusters in the identified tumor region.

[0215] Embodiment 59. A method according to Embodiment 57, wherein the predicted TIL mask structural feature is a predicted size of TIL clusters in the identified tumor region.

[0216] Embodiment 60. A method according to Embodiment 57, wherein the predicted TIL mask structural feature is a predicted TIL cluster spread in the identified tumor region.

[0217] Embodiment 61. A method according to Embodiment 57, wherein the one or more TIL mask structure features include a total count of TILs in the identified tumor region.

[0218] Embodiment 62. A method according to Embodiment 57, wherein the one or more TIL mask structural features include the distance between TIL cells to the distance between tumor cells.

[0219] Embodiment 63. A method according to Embodiment 57, wherein the one or more TIL mask structural features include the distance between the TIL cells and the tumor cells.

[0220] Embodiment 64. A method according to Embodiment 57, wherein the one or more TIL mask structure features include an average color of TIL cells.

[0221] Embodiment 65. A method according to Embodiment 57, wherein the one or more TIL mask structural features include the average size of TIL cells.

[0222] Embodiment 66. A method according to any one of Embodiments 52-65, wherein the TIL% value is intratumoral TIL% (iTIL%).

[0223] Embodiment 67. A method according to Embodiment 66, wherein the iTIL% is calculated as the percentage of the identified tumor area in the tissue sample that is infiltrated by TILs.

[0224] Embodiment 68. A method according to any one of Embodiments 52-65, wherein the TIL% value is stromal TIL% (sTIL%).

[0225] Embodiment 69. A method according to Embodiment 68, wherein the sTIL% value is calculated as the percentage of the stroma in the identified tumor region in the tissue sample that is infiltrated by TILs.

[0226] Embodiment 70. A method according to any one of Embodiments 52-69, wherein the first neural network model is a classification algorithm that generates a TMB indicator, and the TMB indicator is a TMB-positive classification or a TMB-negative classification of the tissue sample.

[0227] Embodiment 71. The method of Embodiment 70, wherein the classification algorithm generates the TMB-positive classification if the tissue sample has a TMB value exceeding a preset TMB threshold.

[0228] Embodiment 72. A method according to any one of Embodiments 52-69, wherein the first neural network model is a regression algorithm that generates a TMB indicator as a quantitative TMB score.

[0229] Embodiment 73. A method according to any one of Embodiments 52-72, wherein the second neural network model is a classification algorithm that generates a PD-L1 expression level indicator, and the PD-L1 expression level indicator is a PD-L1 expression positive classification or a PD-L1 expression level negative classification.

[0230] Embodiment 74. A method according to any one of Embodiments 52-72, wherein the second neural network model is a classification algorithm that generates a PD-1 expression level indicator, and the PD-1 expression level indicator is a PD-1 expression positive classification or a PD-1 expression level negative classification.

[0231] Embodiment 75. The method according to Embodiment 73, wherein if the tissue sample has a PD-L1 expression level value exceeding a preset PD-L1 expression level value, the classification algorithm generates the PD-L1 expression positive classification.

[0232] Embodiment 76. A method according to Embodiment 74, wherein if the tissue sample has a PD-1 expression level value exceeding a preset PD-1 expression level value, the classification algorithm generates the PD-1 expression positive classification.

[0233] Embodiment 77. A method according to any one of Embodiments 52-72, wherein the second neural network model is a regression algorithm that generates a PD-L1 expression level indicator as a quantitative PD-L1 score.

[0234] Embodiment 78. A method according to any one of Embodiments 52-72, wherein the second neural network model is a regression algorithm that generates a PD-1 expression level indicator as a quantitative PD-1 score.

[0235] Embodiment 79. The method of Embodiment 77, wherein the quantitative PD-L1 score is a tumor proportion score (TPS).

[0236] Embodiment 80. The method of Embodiment 78, wherein the quantitative PD-1 score is a tumor proportion score (TPS).

[0237] Embodiment 81. The method of Embodiment 77, wherein the quantitative PD-L1 score is a combined positive score (CPS).

[0238] Embodiment 82. The method of Embodiment 78, wherein the quantitative PD-1 score is a combined positive score (CPS).

[0239] Embodiment 83. A method according to any one of Embodiments 52-82, wherein the immuno-oncology therapeutic agent is a checkpoint inhibitor.

[0240] Embodiment 84. A method according to Embodiment 83, wherein the check point inhibitor is a PD-L1 inhibitor or a PD-1 inhibitor.

[0241] Embodiment 85. A method according to any one of Embodiments 52-84, the method further comprising: generating one or more TIL masks from the one or more features via the third neural network model by the one or more processors, wherein the prediction of the patient's response status to the immuno-oncology therapeutic agent also includes using the one or more TIL masks.

[0242] Embodiment 86. The method of any one of Embodiments 52-85, wherein the tissue sample is from a bladder and the patient's response to the immuno-oncology therapeutic is associated with bladder cancer.

[0243] Embodiment 87. A method according to any one of Embodiments 52-86, wherein the patient is or will be a responder based on the response status; and wherein based on the determination, a therapeutically effective amount of the immuno-oncology therapeutic is administered to the patient.

[0244] Embodiment 88. A method according to any one of Embodiments 52-86, wherein it is determined that the patient is not or will not be a responder based on the response status; and wherein based on the determination, the immuno-oncology therapeutic is not administered to the patient.

[0245] Embodiment 89. The method of any one of Embodiments 52-88, wherein the tissue sample is from a patient known to have a PD-L1-positive cancer or a PD-1-positive cancer.

[0246] Embodiment 90. The method of any one of Embodiments 52-89, wherein the tissue sample is from a patient known to have any of the following: non-small cell lung cancer, melanoma, Hodgkin lymphoma, bladder cancer, kidney cancer, or breast cancer.

[0247] Embodiment 91. The method of any one of Embodiments 52-90, wherein the immuno-oncology therapeutic targets PD-L1 or PD-1.

[0248] Embodiment 92. A method according to any one of Embodiments 52-91, wherein the immuno-oncology therapeutic agent comprises one or more antibodies, adoptive cell therapy agents, immunomodulators, or a combination thereof.

[0249] Embodiment 93. A method according to Embodiment 92, wherein the antibody is a monoclonal antibody, a bispecific antibody or a trispecific antibody.

[0250] Embodiment 94. A method according to Embodiment 92, wherein the adoptive cell therapy agent comprises immune cells expressing one or more engineered antigen receptors.

[0251] Embodiment 95. A method according to Embodiment 94, wherein the engineered antigen receptor is a chimeric antigen receptor, a non-natural T cell receptor, or a combination thereof.

[0252] Embodiment 96. A method according to any one of Embodiments 92-94, wherein the adoptive cell therapy agent comprises T cells, natural killer cells, natural killer T cells, or a combination thereof.

[0253] Embodiment 97. A method according to any one of embodiments 52-96, further comprising: obtaining the sample from the patient.

[0254] Embodiment 98. The method of any one of Embodiments 52-97, further comprising: diagnosing the patient as having cancer.

[0255] Embodiment 99. A method, comprising: obtaining, by one or more processors, an image of a tissue sample from a patient and one or more features of the tissue sample from a data source; performing, by the one or more processors, via one or more neural network models, one or more of the following: (a) generating one or more tumor mutation burden (TMB) indicators from the one or more features; (b) generating one or more programmed death-ligand 1 (PD-L1) expression level indicators from the one or more features or generating one or more programmed death-1 (PD-1) expression level indicators from the one or more features; and (c) generating one or more tumor infiltrating lymphocyte (TIL) percentage (%) values ​​from the one or more features; predicting, by the one or more processors, via the one or more neural network models, the patient's response status to an immuno-oncology therapeutic agent using one or more of the following: the one or more TMB indicators, the one or more PD-L1 expression level indicators, the one or more PD-1 expression level indicators and / or the one or more TIL% values; and determining whether to include the patient in a group of patients participating in a clinical trial based on the response status of the patient.

[0256] Embodiment 100. A method comprising: obtaining, by one or more processors, an image of a tissue sample from a patient and one or more features of the tissue sample from a data source; and generating, by the one or more processors, a prediction of the patient's response status to an immuno-oncology therapeutic agent using the image via one or more neural network models.

[0257] Embodiment 101. A system for predicting a patient's response to an immuno-oncology therapeutic agent, the system comprising: a data repository for storing images of tissue samples from patients, the images comprising one or more features of the tissue samples; a computing device, the computing device communicatively connected to the data repository, the computing device comprising: a first neural network model, the first neural network model being configured to generate one or more tumor mutation burden (TMB) indicators from the one or more features; a second neural network model, the second neural network model being configured to generate one or more programmed death-ligand 1 (PD-L1) expression level indicators from the one or more features or being configured to generate one or more programmed death-1 (PD-1) expression level indicators from the one or more features; a third a neural network model, wherein the third neural network model is configured to calculate one or more tumor infiltrating lymphocyte (TIL)-related metrics from the one or more features; a fourth neural network model, wherein the fourth neural network model is configured to calculate one or more cell-based features from the one or more features; and a fifth neural network model, wherein the fifth neural network model is configured to predict the patient's response status to the immuno-oncology therapeutic agent based on the one or more TMB indicators, the one or more PD-L1 expression level indicators or the one or more PD-1 expression level indicators, the one or more TIL-related metrics, and the one or more cell-based features, respectively; and a display system, wherein the display system is communicatively connected to the computing device and is configured to display a result indicating the response status of the patient.

[0258] Embodiment 102. A method comprising: obtaining, by one or more processors, an image of a tissue sample from a patient and one or more features of the tissue sample from a data source; and training, by the one or more processors, one or more neural network models to use the image to predict the patient's response status to an immuno-oncology therapeutic agent.

[0259] Embodiment 103. A method for predicting a patient's response to an immuno-oncology therapeutic agent, the method comprising: obtaining an image of a tissue sample from a patient by one or more processors, the image comprising one or more features of the tissue sample; generating one or more tumor mutation burden (TMB) indicators from the one or more features by the one or more processors; generating one or more programmed death-ligand 1 (PD-L1) expression level indicators or one or more programmed death-1 (PD-1) expression level indicators from the one or more features by the one or more processors; generating one or more tumor infiltrating lymphocyte (TIL) related metrics from the one or more features by the one or more processors; and training a fourth neural network model by the one or more processors to predict the patient's response status to the immuno-oncology therapeutic agent based on one or more of the following: the one or more TMB indicators, the one or more PD-L1 expression level indicators, the one or more PD-1 level indicators, and the one or more related metrics.

[0260] Embodiment 104. A method of treating a subject having bladder cancer, the method comprising: administering an immuno-oncology therapeutic to the subject, wherein the subject has been determined to be responsive to the immuno-oncology therapeutic via a trained machine learning classifier, the trained machine learning classifier distinguishing between responsive subjects and non-responsive subjects who have received the immuno-oncology therapeutic based at least in part on analyzing one or more of the following items for the subject: PD-L1 expression level, PD-1 expression level, and one or more tumor infiltrating lymphocyte (TIL)-related metrics.

Claims

1. A system for predicting a patient's response to an immuno-oncology therapeutic agent, the system comprising: a data repository for storing images of tissue samples from patients, the images comprising one or more features of the tissue samples; A computing device, the computing device being communicatively connected to the data repository, the computing device comprising: a first neural network model configured to generate one or more programmed death-ligand 1 (PD-L1) expression level indicators from the one or more features and / or configured to generate one or more programmed death-1 (PD-1) expression level indicators from the one or more features; a second neural network model configured to calculate one or more tumor infiltrating lymphocytes (TILs) from the one or more features; Related metrics; or a third neural network model configured to predict the patient's response status to the immuno-oncology therapeutic agent based on (a) the one or more PD-L1 expression level indicators and / or the one or more PD-1 expression level indicators and / or (b) the one or more TIL-related metrics; and A display system is communicatively connected to the computing device and configured to display results indicative of the response status of the patient.

2. The system according to claim 1, wherein: The predicting of the response status of the patient to the immuno-oncology therapeutic agent is performed based on one or more medical history characteristics of the patient.

3. The system according to claim 1, wherein: The second neural network model includes a segmentation algorithm configured to generate an identification of a tumor region in the tissue sample based on the one or more features, wherein the identification is used to further generate the one or more TIL-related metrics.

4. The system according to claim 3, wherein: The segmentation algorithm is configured to predict a TIL mask that is used in combination with a tumor mask to calculate one or more TIL-related metrics.

5. The system according to claim 4, wherein: The one or more TIL-related metrics include a predicted number of TIL clusters in the identified tumor region.

6. The system according to claim 4, wherein: The one or more TIL-related metrics include a predicted size of a TIL cluster in the identified tumor region.

7. The system according to claim 4, wherein: The one or more TIL-related metrics include a predicted TIL cluster spread in the identified tumor region.

8. The system according to claim 4, wherein: The one or more TIL-related metrics include a total count of TILs in the identified tumor region.

9. The system according to claim 4, wherein: The one or more TIL-related metrics include distances between TIL cells versus distances between tumor cells.

10. The system according to claim 4, wherein: The one or more TIL-related metrics include the distance between the TIL cells and the tumor cells.

11. The system according to claim 4, wherein: The one or more TIL-related metrics include an average color of TIL cells.

12. The system according to claim 4, wherein: The one or more TIL-related metrics include an average size of TIL cells.

13. The system of claim 1, wherein: The one or more TIL-related metrics include intratumoral TIL % (iTIL %).

14. The system according to claim 13, wherein: The iTIL% was calculated as the percentage of the identified tumor area in the tissue sample that was infiltrated by TILs.

15. The system of claim 1, wherein: The one or more TIL-related metrics include stromal TIL % (sTIL %).

16. The system of claim 15, wherein: The sTIL% value was calculated as the percentage of the stroma in the identified tumor region in the tissue sample that was infiltrated by TILs.

17. The system of claim 1, further comprising: a fourth neural network model configured to generate one or more tumor mutation burden (TMB) indicators from the one or more features, Wherein the third neural network model is further configured to predict the patient's response status to the immuno-oncology therapeutic agent based on the one or more TMB indicators.

18. The system of claim 17, wherein: The fourth neural network model includes a classification algorithm, and the one or more TMB indicators include a TMB indicator of the tissue sample, and the TMB indicator is a TMB-positive classification or a TMB-negative classification.

19. The system of claim 18, wherein: If the tissue sample has a TMB value exceeding a preset TMB threshold, the classification algorithm generates the TMB-positive classification.

20. The system of claim 17, wherein: The fourth neural network model comprises a regression algorithm, and the one or more TMB indicators comprise a TMB indicator that is a quantitative TMB score.

21. The system of claim 1, wherein: The first neural network model includes a classification algorithm, and the one or more PD-L1 expression level indicators include a PD-L1 expression level indicator, and the PD-L1 expression level indicator is a PD-L1 expression positive classification or a PD-L1 expression level negative classification.

22. The system of claim 1, wherein: The first neural network model includes a classification algorithm, and the one or more PD-1 expression level indicators include a PD-1 expression level indicator, and the PD-1 expression level indicator is a PD-1 expression positive classification or a PD-1 expression level negative classification.

23. The system of claim 22, wherein: If the tissue sample has a PD-L1 expression level value exceeding a preset PD-L1 expression level value, the classification algorithm generates the PD-L1 expression positive classification.

24. The system of claim 22, wherein: If the tissue sample has a PD-1 expression level value exceeding a preset PD-1 expression level value, the classification algorithm generates the PD-1 expression positive classification.

25. The system of claim 1, wherein: The first neural network model comprises a regression algorithm, and the one or more PD-L1 expression level indicators comprise a quantitative PD-L1 score.

26. The system of claim 1, wherein: The first neural network model comprises a regression algorithm, and the one or more PD-1 expression level indicators comprise a quantitative PD-1 score.

27. The system of claim 25, wherein: The quantitative PD-L1 score is the tumor proportion score (TPS).

28. The system of claim 25, wherein: The quantitative PD-L1 score is the combined positive score (CPS).

29. The system of claim 26, wherein: The quantitative PD-1 score is the tumor proportion score (TPS).

30. The system of claim 26, wherein: The quantitative PD-1 score is the combined positive score (CPS).

31. The system of claim 1, wherein: The immuno-oncology therapeutic agent is a checkpoint inhibitor.

32. The system of claim 31, wherein: The checkpoint inhibitor is a PD-L1 inhibitor or a PD-1 inhibitor.

33. The system of claim 1, wherein: The second neural network model is further configured to generate one or more TIL masks from the one or more features; and Wherein the third neural network model is further configured to use the one or more TIL masks to predict the response status of the patient to the immuno-oncology therapeutic agent.

34. The system of claim 1, wherein: The tissue sample is from the patient's bladder; and wherein said response of said patient to said immuno-oncology therapeutic agent is associated with bladder cancer.

35. The system of claim 1, wherein: The tissue samples were from patients known to have PD-L1-positive cancer.

36. The system of claim 1, wherein: The tissue samples were from patients known to have PD-1-positive cancer.

37. The system of claim 1, wherein: The tissue samples are from patients known to have any one or more of the following: non-small cell lung cancer, melanoma, Hodgkin lymphoma, bladder cancer, kidney cancer, and breast cancer.

38. The system of claim 1, wherein: The immuno-oncology therapeutic targets PD-L1 or PD-1.

39. The system of claim 1, wherein: The immuno-oncology therapeutic agent includes one or more antibodies, one or more adoptive cell therapy agents, or a combination thereof.

40. The system of claim 39, wherein: The one or more antibodies include at least one of the following: a monoclonal antibody, a bispecific antibody, or a trispecific antibody.

41. The system of claim 39, wherein: The one or more adoptive cell therapy agents include immune cells expressing one or more engineered antigen receptors.

42. The system of claim 41, wherein: The one or more engineered antigen receptors include at least one of: a chimeric antigen receptor, a non-natural T cell receptor, or a combination thereof.

43. The system of claim 39, wherein: The one or more adoptive cell therapy agents include at least one of the following: T cells, natural killer cells, natural killer T cells, or a combination thereof.

44. The system of claim 1, wherein: The second neural network model includes a region / TIL segmentation algorithm and a cell segmentation algorithm.

45. The system of claim 44, wherein: The cell segmentation algorithm computes one or more cell-based features.

46. ​​The system of claim 45, wherein: The one or more cell-based features include morphology, color (nucleus), color (outside nucleus), texture (energy, correlation, contrast, homogeneity, dissimilarity, or angular second moment).

47. The system of claim 44, wherein: The cell segmentation algorithm generates a certain TIL cell number or an average TIL cell number.

48. The system of claim 44, wherein: The cell segmentation algorithm generates an average tumor cell size.

49. The system of claim 44, wherein: The cell segmentation algorithm generates the average TIL cell size.

50. The system of claim 44, wherein: The cell segmentation algorithm generates the average distance between TIL cells.

51. The system of claim 44, wherein: The cell segmentation algorithm generates the average color of TIL cells.

52. The system of claim 44, wherein: The cell segmentation algorithm generates the average distance between TIL cells versus the average distance between tumor cells.

53. A method for predicting a patient's response to an immuno-oncology therapeutic agent, the method comprising: obtaining, by one or more processors, an image of a tissue sample from a patient, the image comprising one or more features of the tissue sample; generating, by the one or more processors via a first neural network model, one or more programmed death-ligand 1 (PD-L1) expression level indicators from the one or more features and / or generating one or more programmed death-1 (PD-1) expression level indicators from the one or more features; generating, by the one or more processors, one or more tumor infiltrating lymphocyte (TIL)-related metrics from the one or more features via a second neural network model; as well as The one or more processors use the one or more PD-L1 expression level indicators and / or the one or more PD-1 level indicators and the one or more related metrics via a third neural network model to predict the patient's response status to the immuno-oncology therapeutic agent.

54. The method of claim 53, further comprising: Whether to include the patient in a group of patients participating in a clinical trial is selected based on the response status of the patient.

55. The method of claim 53, further comprising: A graphical representation indicative of the response status of the patient is displayed on a display screen.

56. The method of claim 53, wherein: The prediction of the response status of the patient to the immuno-oncology therapeutic agent also uses one or more medical history characteristics of the patient.

57. The method of claim 53, wherein: The third neural network is a segmentation algorithm that identifies tumor regions in the tissue sample.

58. The method of claim 57, wherein: The segmentation algorithm predicts one or more TIL mask structural features in the identified tumor region to generate the one or more TIL-related metrics.

59. The method of claim 58, wherein: The predicted TIL mask structural feature is the predicted number of TIL clusters in the identified tumor region.

60. The method of claim 58, wherein: The predicted TIL mask structural feature is the predicted size of the TIL clusters in the identified tumor region.

61. The method of claim 58, wherein: The predicted TIL mask structure features predicted TIL cluster spread in the identified tumor region.

62. The method of claim 58, wherein: The one or more TIL mask structure features include a total count of TILs in the identified tumor region.

63. The method of claim 58, wherein: The one or more TIL mask structural features include the distance between TIL cells versus the distance between tumor cells.

64. The method of claim 58, wherein: The one or more TIL mask structural features include the distance between the TIL cells and the tumor cells.

65. The method of claim 58, wherein: The one or more TIL mask structural features include an average color of TIL cells.

66. The method of claim 58, wherein: The one or more TIL mask structural features include an average size of TIL cells.

67. The method of claim 53, wherein: The TIL% values ​​are intratumoral TIL% (iTIL%).

68. The method of claim 67, wherein: The iTIL% was calculated as the percentage of the identified tumor area in the tissue sample that was infiltrated by TILs.

69. The method of claim 53, wherein: The TIL% values ​​are stromal TIL% (sTIL%).

70. The method of claim 69, wherein: The sTIL% value was calculated as the percentage of the stroma in the identified tumor region in the tissue sample that was infiltrated by TILs.

71. The method of claim 53, further comprising: generating, by the one or more processors, one or more tumor mutation burden (TMB) indicators from the one or more features via a fourth neural network model, Wherein the prediction via the third neural network model is further performed using the one or more TMB indicators.

72. The method of claim 71, wherein: The fourth neural network model is a classification algorithm for generating a TMB index, and the TMB index is a TMB-positive classification or a TMB-negative classification of the tissue sample.

73. The method of claim 72, wherein: If the tissue sample has a TMB value exceeding a preset TMB threshold, the classification algorithm generates the TMB-positive classification.

74. The method of claim 71, wherein: The fourth neural network model is a regression algorithm that generates a TMB indicator as a quantitative TMB score.

75. The method of claim 53, wherein: The first neural network model is a classification algorithm for generating a PD-L1 expression level indicator, and the PD-L1 expression level indicator is a PD-L1 expression positive classification or a PD-L1 expression level negative classification.

76. The method of claim 53, wherein: The first neural network model is a classification algorithm for generating a PD-1 expression level indicator, and the PD-1 expression level indicator is a PD-1 expression positive classification or a PD-1 expression level negative classification.

77. The method of claim 75, wherein: If the tissue sample has a PD-L1 expression level value exceeding a preset PD-L1 expression level value, the classification algorithm generates the PD-L1 expression positive classification.

78. The method of claim 76, wherein: If the tissue sample has a PD-1 expression level value exceeding a preset PD-1 expression level value, the classification algorithm generates the PD-1 expression positive classification.

79. The method of claim 53, wherein: The first neural network model is a regression algorithm that generates a PD-L1 expression level indicator as a quantitative PD-L1 score.

80. The method of claim 53, wherein: The first neural network model is a regression algorithm that generates a PD-1 expression level indicator as a quantitative PD-1 score.

81. The method of claim 79, wherein: The quantitative PD-L1 score is the tumor proportion score (TPS).

82. The method of claim 80, wherein: The quantitative PD-1 score is the tumor proportion score (TPS).

83. The method of claim 79, wherein: The quantitative PD-L1 score is the combined positive score (CPS).

84. The method of claim 80, wherein: The quantitative PD-1 score is the combined positive score (CPS).

85. The method of claim 53, wherein: The immuno-oncology therapeutic agent is a checkpoint inhibitor.

86. The method of claim 85, wherein: The checkpoint inhibitor is a PD-L1 inhibitor or a PD-1 inhibitor.

87. The method of claim 53, further comprising: generating, by the one or more processors, one or more TIL masks from the one or more features via the third neural network model, Wherein said prediction of said response status of said patient to said immuno-oncology therapeutic agent further comprises using said one or more TIL masks.

88. The method of claim 53, wherein: The tissue sample is from a bladder, and the patient's response to the immuno-oncology therapeutic agent is associated with bladder cancer.

89. The method of claim 53, wherein: determining that the patient is or will be a responder based on the response status; and Wherein based on said determination, a therapeutically effective amount of said immuno-oncology therapeutic agent is administered to said patient.

90. The method of claim 53, wherein: determining that the patient is not or will not be a responder based on the response status; and wherein based on said determination, said immuno-oncology therapeutic is not administered to said patient.

91. The method of claim 53, wherein: The tissue samples are from patients known to have PD-L1-positive cancer or PD-1-positive cancer.

92. The method of claim 53, wherein: The tissue sample is from a patient known to have any of the following: non-small cell lung cancer, melanoma, Hodgkin lymphoma, bladder cancer, kidney cancer, or breast cancer.

93. The method of claim 53, wherein: The immuno-oncology therapeutic targets PD-L1 or PD-1.

94. The method of claim 53, wherein: The immuno-oncology therapeutic agent includes one or more antibodies, adoptive cell therapy agents, immunomodulators, or a combination thereof.

95. The method of claim 94, wherein: The antibody is a monoclonal antibody, a bispecific antibody or a trispecific antibody.

96. The method of claim 94, wherein: The adoptive cell therapy agent includes immune cells expressing one or more engineered antigen receptors.

97. The method of claim 96, wherein: The engineered antigen receptor is a chimeric antigen receptor, a non-natural T cell receptor, or a combination thereof.

98. The method of claim 94, wherein: The adoptive cell therapy agent includes T cells, natural killer cells, natural killer T cells or a combination thereof 99. The method of claim 53, further comprising: The sample is obtained from the patient.

100. The method of claim 53, further comprising: The patient is diagnosed as having cancer.

101. A method comprising: obtaining, by one or more processors, from a data source an image of a tissue sample from a patient and one or more characteristics of the tissue sample; One or more of the following are performed by the one or more processors via one or more neural network models: (a) generating one or more programmed death-ligand 1 (PD-L1) expression level indicators from the one or more features or generating one or more programmed death-1 (PD-1) expression level indicators from the one or more features; and (b) generating one or more tumor infiltrating lymphocyte (TIL) percentage (%) values ​​from the one or more features; predicting, by the one or more processors via the one or more neural network models, a response status of the patient to an immuno-oncology therapeutic agent using one or more of the following: the one or more PD-L1 expression level indicators, the one or more PD-1 expression level indicators, and / or the one or more TIL% values; as well as A determination is made based on the response status of the patient whether to include the patient in a group of patients participating in a clinical trial.

102. A method comprising: obtaining, by one or more processors, from a data source an image of a tissue sample from a patient and one or more characteristics of the tissue sample; as well as The images are used, by the one or more processors, to generate a prediction of the patient's response status to an immuno-oncology therapeutic agent via one or more neural network models.

103. A system for predicting a patient's response to an immuno-oncology therapeutic, the system comprising: a data repository for storing images of tissue samples from patients, the images comprising one or more features of the tissue samples; A computing device, the computing device being communicatively connected to the data repository, the computing device comprising: a first neural network model configured to generate one or more programmed death-ligand 1 (PD-L1) expression level indicators from the one or more features or configured to generate one or more programmed death-1 (PD-1) expression level indicators from the one or more features; a second neural network model configured to calculate one or more tumor infiltrating lymphocyte (TIL)-related metrics from the one or more features; a third neural network model configured to compute one or more cell-based features from the one or more features; and a fifth neural network model configured to predict the patient's response status to the immuno-oncology therapeutic agent based on the one or more PD-L1 expression level indicators or the one or more PD-1 expression level indicators, the one or more TIL-related metrics, and the one or more cell-based features, respectively; and A display system is communicatively connected to the computing device and configured to display results indicative of the response status of the patient.

104. A method comprising: obtaining, by one or more processors, from a data source an image of a tissue sample from a patient and one or more characteristics of the tissue sample; as well as One or more neural network models are trained, by the one or more processors, to predict a response status of the patient to an immuno-oncology therapeutic agent using the image.

105. A method for predicting a patient's response to an immuno-oncology therapeutic agent, the method comprising: obtaining, by one or more processors, an image of a tissue sample from a patient, the image comprising one or more features of the tissue sample; generating, by the one or more processors, one or more programmed death-ligand 1 (PD-L1) expression level indicators and / or one or more programmed death-1 (PD-1) expression level indicators from the one or more features; generating, by the one or more processors, one or more tumor infiltrating lymphocyte (TIL)-related metrics from the one or more features; as well as A fourth neural network model is trained, by the one or more processors, to predict the patient's response status to the immuno-oncology therapeutic agent based on one or more of the following: the one or more PD-L1 expression level indicators, the one or more PD-1 level indicators, and the one or more related metrics.

106. A method of treating a subject having bladder cancer, the method comprising: An immuno-oncology therapeutic is administered to the subject, wherein the subject has been determined to be responsive to the immuno-oncology therapeutic via a trained machine learning classifier that distinguishes between responsive subjects and non-responsive subjects who have received the immuno-oncology therapeutic based at least in part on analyzing one or more of the following for the subject: PD-L1 expression level, PD-1 expression level, and one or more tumor infiltrating lymphocyte (TIL)-related metrics.

Images