Method and system for digital pathological assessment of cancer by deep learning

Abstract

The present disclosure provides methods and systems for classifying and / or monitoring cancer in a subject. A method for assessing cancer in a subject may include obtaining a dataset including image and / or tabular data from the subject, and processing the data using one or more trained algorithms to classify cancer in the subject. The cancer of the subject may be evaluated based on the classification results.

Description

[0001] Cross-references

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 287,158 filed on December 8, 2021, U.S. Provisional Patent Application No. 63 / 345,804 filed on May 25, 2022, and U.S. Provisional Patent Application No. 63 / 418,125 filed on October 21, 2022, each of which is incorporated herein by reference in its entirety. Background Art

[0003] Prostate cancer is the leading cause of cancer death in men. However, international standards for predicting patient outcomes rely on nonspecific and insensitive tools that often lead to over- or under-treatment. Summary of the invention

[0004] Determining the best cancer treatment for a patient is a challenging task in which oncologists must select the treatment with the highest probability of success and the lowest probability of toxicity. The difficulty in treatment selection stems from the enormous molecular, phenotypic, and prognostic heterogeneity exhibited by cancer. This article recognizes the need for accurate, globally scalable tools to support personalized cancer treatment.

[0005] The present disclosure provides methods and systems for identifying or monitoring cancer-related states by processing biological samples obtained from or derived from subjects (e.g., cancer patients). Biological samples (e.g., tissue samples) obtained from subjects can be analyzed to predict clinical outcomes (which may include, for example, distant metastasis, biochemical recurrence, death, progression-free survival, and overall survival).

[0006] In one aspect, the present disclosure provides a method for assessing cancer in a subject, the method comprising: (a) obtaining a dataset comprising image data and tabular data originating from the subject; (b) processing the dataset using a trained algorithm to classify the dataset into a category among a plurality of categories, wherein the classification comprises applying an image processing algorithm to the image data; and (c) assessing cancer in the subject based at least in part on the category among the plurality of categories classified in (b).

[0007] In some embodiments, the trained algorithm is trained using self-supervised learning. In some embodiments, the trained algorithm comprises a deep learning algorithm. In some embodiments, the trained algorithm comprises a first trained algorithm for processing image data and a second trained algorithm for processing tabular data. In some embodiments, the trained algorithm further comprises a third trained algorithm for processing the outputs of the first trained algorithm and the second trained algorithm. In some embodiments, the cancer is bladder cancer, breast cancer, cervical cancer, colorectal cancer, gastric cancer, kidney cancer, liver cancer, ovarian cancer, pancreatic cancer, prostate cancer, or thyroid cancer. In some embodiments, the cancer is prostate cancer. In some embodiments, the tabular data comprises clinical data of the subject. In some embodiments, the clinical data of the subject comprises laboratory data, therapeutic interventions, or long-term results. In some embodiments, the image data comprises digital histopathology data. In some embodiments, the histopathology data comprises images of biopsy samples from the subject. In some embodiments, the images are obtained by microscopic examination of the biopsy samples. In some embodiments, the digital histopathology data is obtained from the subject before the subject receives treatment. In some embodiments, the treatment comprises radiation therapy (RT). In some embodiments, RT comprises pre-specified use of short-term androgen deprivation therapy (ST-ADT), long-term ADT (LT-ADT), dose escalation RT (DE-RT), or any combination thereof. In some embodiments, the digital histopathology data is obtained from the subject after the subject receives treatment. In some embodiments, the treatment comprises radiation therapy (RT). In some embodiments, RT comprises pre-specified use of short-term androgen deprivation therapy (ST-ADT), long-term ADT (LT-ADT), dose escalation RT (DE-RT), or any combination thereof. In some embodiments, the method further comprises processing the image data using image segmentation, image stitching, an object detection algorithm, or any combination thereof. In some embodiments, the method further comprises extracting features from the image data.

[0008] In another aspect, the present disclosure provides a method for assessing cancer in a subject, the method comprising: (a) obtaining a dataset comprising at least image data originating from the subject; (b) processing the dataset using a trained algorithm to classify the dataset into a category among a plurality of categories, wherein the classification comprises applying an image processing algorithm to the image data, wherein the trained algorithm is trained using self-supervised learning; and (c) assessing cancer in the subject based at least in part on the category among the plurality of categories classified in (b).

[0009] In some embodiments, the trained algorithm comprises a deep learning algorithm. In some embodiments, the cancer is bladder cancer, breast cancer, cervical cancer, colorectal cancer, gastric cancer, kidney cancer, liver cancer, ovarian cancer, pancreatic cancer, prostate cancer or thyroid cancer. In some embodiments, the cancer is prostate cancer. In some embodiments, the image data comprises digital histopathology data. In some embodiments, the histopathology data comprises an image of a biopsy sample derived from the object. In some embodiments, the image is obtained by microscopic examination of the biopsy sample. In some embodiments, the digital histopathology data is obtained from the object before the object receives treatment. In some embodiments, treatment includes radiotherapy (RT). In some embodiments, RT includes pre-specified use of short-term androgen deprivation therapy (ST-ADT), long-term ADT (LT-ADT), dose escalation RT (DE-RT) or any combination thereof. In some embodiments, the digital histopathology data is obtained from the object after the object receives treatment. In some embodiments, treatment includes radiotherapy (RT). In some embodiments, RT comprises pre-specified use of short-term androgen deprivation therapy (ST-ADT), long-term ADT (LT-ADT), dose-escalation RT (DE-RT), or any combination thereof. In some embodiments, the method further comprises processing the image data using an image segmentation, image stitching, or object detection algorithm. In some embodiments, the method further comprises extracting features from the image data. In some embodiments, the data set comprises image data and tabular data. In some embodiments, the trained algorithm comprises a first trained algorithm that processes the image data and a second trained algorithm that processes the tabular data. In some embodiments, the trained algorithm further comprises a third trained algorithm that processes the outputs of the first trained algorithm and the second trained algorithm. In some embodiments, the tabular data comprises clinical data of the subject. In some embodiments, the clinical data comprises laboratory data, therapeutic interventions, or long-term outcomes.

[0010] Another aspect of the present disclosure provides a non-transitory computer-readable medium comprising machine-executable code that, when executed by one or more computer processors, implements any of the methods described above or elsewhere herein.

[0011] Another aspect of the present disclosure provides a system comprising one or more computer processors and a computer memory coupled thereto. The computer memory comprises machine executable code that, when executed by the one or more computer processors, implements any method described above or elsewhere herein.

[0012] Additional aspects and advantages of the present disclosure will become apparent to those skilled in the art from the following detailed description, wherein only illustrative embodiments of the present disclosure are shown and described. The present disclosure is capable of other different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the present disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0013] Incorporation by reference

[0014] All publications, patents, and patent applications mentioned in this specification are incorporated herein by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated herein by reference. In the event that a publication, patent, or patent application incorporated by reference contradicts the disclosure contained in the specification, the specification is intended to supersede and / or take precedence over any such contradictory material. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The novel features of the invention are set forth with particularity in the appended claims. A better understanding of the features and advantages of the invention will be obtained by reference to the following detailed description and accompanying drawings (also referred to herein as "figures") which set forth illustrative embodiments in which the principles of the invention are utilized, in which:

[0016] Figure 1 A computer system is shown that is programmed or otherwise configured to implement the methods provided herein.

[0017] Figure 2A-2C Examples of multimodal deep learning systems and datasets are shown. Figure 2A The multimodal architecture is shown to consist of three parts: a tower stack for parsing tabular clinical data, a tower stack for parsing a variable number of digital histopathology slides, and a third tower stack for combining outcome features and predicting a binary outcome. Figure 2B Shown is the training of a self-supervised model for an image tower stack. Figure 2C The first five columns of the table are shown, which show statistics from each trial. The "Combined" column shows statistics for the final dataset in which all five trials were used for training and validation. ***RTOG 9413 randomized patients in a 2 × 2 manner, testing the effects of hormonal therapy timing (before RT vs. starting with RT) and field size (RT to the prostate only vs. RT to the entire pelvis). New abbreviations used: disease-free survival (DFS), progression-free survival (PFS), prostate cancer-specific mortality (PCSM).

[0018] Figure 3A-3H Examples of deep learning systems compared to established clinical guidelines in trials and results are shown. Figure 3A Performance results using time-dependent receiver operator characteristics reporting the area under the curve (AUC) for sensitivity and specificity of the MMAI (blue bars) versus the NCCN (grey bars) model are shown. The following binary outcomes were compared at the 5- and 10-year time points: distant metastasis (DM), biochemical recurrence (BCR), prostate cancer-specific survival (PCaSS), and overall survival (OS). Figure 3B A summary table showing the relative improvement of the AI ​​model over the NCCN model in various outcomes broken down by performance on each trial data in the test set. The relative improvement is given by (PAI-PNCCN) / PNCCN, where P is the performance of the model. Figure 3C Results of ablation studies showing model performance when trained on successively smaller data input sets are shown. NCCN denotes the following three variables: combined Gleason, baseline PSA, t-stage; NCCN+3 denotes NCCN plus: Gleason primary, Gleason secondary, age; Pathway refers to digitized histopathology images. Figure 3D-Figure 3H shows a comparison of performance on various clinical trial subsets of the test set—the five subsets together make up Figure 3A The entire test set is shown in .

[0019] Figure 4 An example of pathologists’ interpretation of SSL tissue clusters is shown. A self-supervised model in the multimodal model was trained to identify whether an enhanced version of a small patch of tissue came from the same original patch without looking at clinical data labels. After training, each patch in the dataset of 10.05M patches was fed through this model to extract a 128-dimensional feature vector, and the resulting vectors were clustered and visualized using the UMAP algorithm31. Pathologists were then asked to interpret the 20 patches closest to each of the 25 cluster centroids—descriptions are shown next to the inset. For clarity, we highlight only 6 clusters (in color) and display the rest in grey. See Figure 7 for full pathologist notes.

[0020] Figure 5 An example of an image quilt from four example patients is shown. The dataset contains patients with a variable number of histopathology sections. To standardize the image input to the model, the tissue from each section was segmented and all tissues were pasted into a single square image of 51200×51200 pixels and divided into 200×200 blocks representing all histopathology data for a single patient. An image quilt from four patients is shown here.

[0021] Figure 6An example of kernel density sampling for an example image patch is shown. The tan box indicates a kernel detection, which is used to compute the kernel density. We oversample the patches input to the self-supervised training protocol based on the kernel density. Each patch is divided into deciles based on density, and each decile is oversampled so that the MMAI model sees an equal amount of total images from each decile.

[0022] Figure 7 An example of a pathologist-interpreted block cluster is shown. Using UMAP, 25 clusters were generated from the SSL features of all histopathology blocks from trial RTOG-9202. Each row in the image corresponds to the 25 nearest neighbor image patches of the cluster centroid. These have been examined by a pathologist to determine human-interpretable descriptions of the clusters listed in the table.

[0023] Figure 8 An example of the NCCN model algorithm is shown. The rule-based algorithm models the guidelines published annually by the National Cancer Center Network based on the D'Amico risk groups.

[0024] Fig. 9 is a schematic representation of the multimodal AI system described in this article.

[0025] Fig.10 is a flow chart representing the clinical trial pool used to test and develop the model described herein.

[0026] Fig.11 is a table summarizing the patient characteristics of the data analyzed by the models described herein.

[0027] Fig.12 Depicted are the distributions of MMAI scores for distant metastasis (DM) and prostate cancer-specific mortality (PCSM) between racial subgroups in the test (top graph) and development (bottom graph) cohorts, as determined by the MMAI model described herein.

[0028] Fig.13 is a table showing the MMAI scores summarized by racial subgroup in the development and test cohorts.

[0029] Fig.14 Shown are the MMAI model scores determined by the MMAI model described herein summarized by racial subgroups in the training and test cohorts.

[0030] Figures 15A-15D Subdistribution hazard ratio (HR) results of the Fine & Gray regression model for distant metastasis (DM) MMAI and prostate cancer-specific mortality (PCSM) MMAI in racial subgroups in the development and test cohorts are shown. Fig.15A DM results for the test cohort are shown. Fig. 15BDM results for the development cohort are shown. Fig. 15C The PCSM results of the test cohort are shown. Fig.15D DM results for the development cohort are shown.

[0031] Fig.16 Depicts sub-distribution hazard ratio (HR) results of the Fine & Gray regression model in the racial subgroups of the MMAI model described herein. Shown are the HRs for 5-year biochemical failure (BF5yr MMAI), 10-year BF (BF10yr MMAI), 5-year distant metastasis (DM5yr MMAI), 10-year DM (DM10yr MMAI), 10-year prostate cancer-specific mortality (PCSM10yrMMAI), and 10-year overall survival (OS10yr MMAI) in the test cohort.

[0032] Fig.17 The subdistribution hazard ratio (HR) results of the Fine & Gray regression model in racial subgroups for the MMAI model described herein for DM 5-year MMAI (Panel a) and PCSM 10-year MMAI (Panel b) in the test and training cohorts are depicted in tabular form.

[0033] Fig.18A and Fig.18B It shows that in the complete group DM ( Fig.18A ) and PCSM( Fig.18B Estimated hazard / cumulative incidence curves by ethnic subgroup for .

[0034] Fig.19A Risk stratification of the MMAI model within racial subgroups (DM MMAI) in the development cohort, test cohort, and full cohort is shown.

[0035] Fig.19B Risk stratification of the MMAI model within racial subgroups (PCSM) MMAI in the development cohort, test cohort, and full cohort is shown.

[0036] Fig. 20A Risk stratification of the MMAI model within racial subgroups (DM MMAI) in the development cohort, test cohort, and full cohort is shown.

[0037] Fig. 20B Risk stratification of the MMAI model within racial subgroups (PCSM) MMAI in the development cohort, test cohort, and full cohort is shown.

[0038] Fig.21 Cumulative incidence curves of distant metastasis (DM) in a cohort of prostate cancer patients are shown.

[0039] Fig. 22is a table summarizing the patient characteristics of risk-stratified groups predicted by the artificial intelligence model described in this article.

[0040] Fig.23 is a table showing the differential risk stratification of the same patient cohort by National Comprehensive Cancer Network (NCCN) risk stratification and by multimodal artificial intelligence risk stratification.

[0041] Fig.24 The MMAI predicted risk of distant metastasis after ten years (DM 10 years) for the patient cohort compared with the NCCN classification is shown.

[0042] Fig.25A and Fig.25B A graphical representation of the differences in stratification of patient groups by the NCCN method and the method disclosed herein is shown.

[0043] Fig.26 Flow chart of patient enrollment from the parent clinical trial NRG / RTOG 9902. H&E = hematoxylin and eosin; MMAI = multimodal artificial intelligence; DPEP = digital pathology evaluable population; RT = radiotherapy; AS = androgen suppression; CT = chemotherapy.

[0044] Fig.27A Population characteristics of participants from the parent clinical trial, NRG / RTOG 9902, are shown. Fig.27B MMAI scores between treatment groups for individual populations in NRG / RTOG 9902 are shown.

[0045] Fig.28A is a table showing univariate analysis of the association between the MMAI algorithm and DM and PCSM endpoints. Fig.28B is a table showing multivariate analysis of the association between the MMAI algorithm and DM and PCSM endpoints while adjusting for individual clinical risk factors.

[0046] Fig.29A is a table showing the prognostic performance of MMAI for distant metastasis (DM). Fig.29B is a table showing the prognostic performance of MMAI on prostate cancer specific mortality (PCSM) within subgroup classifications.

[0047] Fig. 30A is a table showing the multivariate analysis of the MMAI algorithm for PM after adjusting for all clinical risk factors. Fig. 30B is a table showing the multivariate analysis of the MMAI algorithm on PCSM after adjusting for all clinical risk factors.

[0048] Fig.31Ais a table showing the multivariate analysis of the DM-prognostic MMAI algorithm on BF, CSM, and OS. Fig.31B is a table showing the multivariate analysis of the PCSM-prognostic MMAI algorithm for BF, CSM, and OS.

[0049] Fig.32A Cumulative incidence curves of the estimated risk of distant metastasis (DM) for quartile 4 versus quartiles 1-3 predicted by multimodal artificial intelligence optimized for DM (DM MMAI) are depicted. Fig.32B Cumulative incidence curves for quartile 4 versus quartiles 1-3 of estimated prostate cancer-specific mortality (PCSM) predicted by a multimodal artificial intelligence optimized for PCSM (PCSM MMAI) are depicted.

[0050] Fig.33A Cumulative incidence curves of the risk of distant metastasis (DM) by treatment group according to quartile 4 versus quartile 1-3 DM MMAI are depicted. Fig.33B Cumulative incidence curves of prostate cancer-specific mortality (PCSM) by treatment group for quartile 4 versus quartile 1-3 PCSM MMAI estimates are depicted. DETAILED DESCRIPTION

[0051] Although various embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that these embodiments are provided by way of example only. Many variations, changes and substitutions may occur to those skilled in the art without departing from the present invention. It should be understood that various alternatives to the embodiments of the present invention described herein may be employed.

[0052] As used in the specification and claims, the singular forms "a", "an" and "the" include plural references unless the context clearly dictates otherwise. For example, the term "nucleic acid" includes a plurality of nucleic acids, including mixtures thereof.

[0053] As used herein, the term "subject" generally refers to an entity or medium with testable or detectable genetic information. The object can be a person, an individual, or a patient. The object can be a vertebrate, such as, for example, a mammal. Non-limiting examples of mammals include humans, apes, farm animals, sports animals, rodents, and pets. The object can be a male object. The object can be a female object. The object can show symptoms indicating the health or physiological state or condition of the object, such as the health or physiological state or condition associated with the cancer of the object. As an alternative, the object can be asymptomatic in terms of this health or physiological state or condition. The object can be suspected of having a health or physiological state or condition. The object can be at risk of developing a health or physiological state or condition. The health or physiological state can correspond to a disease (e.g., cancer). The object can be an individual diagnosed with a disease. The object can be an individual at risk of becoming ill.

[0054] As used herein, "diagnosis of cancer," "diagnosing cancer," and related or derived terms include identifying cancer in a subject, determining the malignancy of a cancer, or determining the stage of a cancer.

[0055] As used herein, "cancer prognosis," "predicting cancer," and related or derived terms include predicting a patient's clinical outcome, assessing the risk of cancer recurrence, determining treatment modalities, or determining the effectiveness of treatment.

[0056] As used herein, the term "nucleic acid" generally refers to a polymeric form of nucleotides of any length, whether deoxyribonucleotides (dNTPs) or ribonucleotides (rNTPs), or analogs thereof. Nucleic acids can have any three-dimensional structure and can perform any known or unknown function. Non-limiting examples of nucleic acids include deoxyribonucleic acid (DNA), ribonucleic acid (RNA), coding or non-coding regions of genes or gene fragments, sites defined by linkage analysis, exons, introns, messenger RNA (mRNA), transfer RNA, ribosomal RNA, short interfering RNA (siRNA), short hairpin RNA (shRNA), microRNA (miRNA), ribozymes, cDNA, recombinant nucleic acids, branched nucleic acids, plasmids, vectors, isolated DNA of any sequence, isolated RNA of any sequence, nucleic acid probes and primers. Nucleic acids can include one or more modified nucleotides, such as methylated nucleotides and nucleotide analogs. If present, the nucleotide structure can be modified before or after nucleic acid assembly. The nucleotide sequence of nucleic acids can be interrupted by non-nucleotide components. Nucleic acids can be further modified after polymerization, such as by conjugation or bonding with a reporter.

[0057] As used herein, the term "target nucleic acid" generally refers to a nucleic acid molecule in a starting nucleic acid molecule population having a nucleotide sequence whose presence, amount and / or sequence or one or more changes thereof need to be determined. The target nucleic acid can be any type of nucleic acid, including DNA, RNA and analogs thereof. As used herein, "target ribonucleic acid (RNA)" generally refers to a target nucleic acid as RNA. As used herein, the term "target deoxyribonucleic acid (DNA)" generally refers to a target nucleic acid as DNA.

[0058] As used herein, the term "amplification" generally refers to increasing the size or quantity of a nucleic acid molecule. A nucleic acid molecule can be single-stranded or double-stranded. Amplification can include generating one or more copies or "amplification products" of a nucleic acid molecule. Amplification can be performed, for example, by extension (e.g., primer extension) or ligation. Amplification can include performing a primer extension reaction to generate a strand complementary to a single-stranded nucleic acid molecule, and in some cases generating one or more copies of the strand and / or single-stranded nucleic acid molecule. The term "DNA amplification" generally refers to generating one or more copies or "amplified DNA products" of a DNA molecule. The term "reverse transcription amplification" generally refers to generating deoxyribonucleic acid (DNA) from a ribonucleic acid (RNA) template by the action of a reverse transcriptase.

[0059] Embodiments of the present disclosure

[0060] Despite its prevalence, accurate, sensitive, and specific prostate cancer diagnosis remains elusive. While prostate cancer is typically slow-growing and can be cured with treatment, it is the leading cause of cancer-related disability worldwide and remains one of the leading causes of cancer death in men due to the negative effects of over- and under-treatment. Determining the best treatment for a man with prostate cancer is a daunting medical task that involves considering the patient’s overall health, the characteristics of his cancer, the side effects of multiple possible treatments, outcome data from clinical trials involving patients with similar diagnoses, and predicting expected future outcomes for the current patient. This challenge is exacerbated by the lack of readily available prognostic tools to better risk stratify patients.

[0061] Artificial intelligence (AI) has allowed insights to be gleaned from massive data sets that were previously uninterpretable. While standard risk stratification tools are fixed and based on a small number of variables, AI can learn from large amounts of minimally processed data across a variety of modalities. AI systems can be low-cost, massively scalable, and improve incrementally through use.

[0062] There is an urgent need for accurate, globally scalable tools to support personalized therapy. The methods and systems disclosed herein demonstrate personalized prostate cancer therapy by using a novel multimodal deep learning model trained on digital histopathology and clinical data of prostate biopsies to predict long-term clinically relevant outcomes (e.g., distant metastasis, biochemical recurrence, partial remission, complete remission, death, relative survival, cancer-specific survival, progression-free survival, disease-free survival, five-year survival, and overall survival).

[0063] The present disclosure provides methods, systems and kits for identifying or monitoring cancer-related categories and / or states by processing biological samples obtained from or obtained from an object (e.g., a male patient suffering from or suspected of having prostate cancer). A biological sample obtained from an object (e.g., a prostate biopsy sample) can be analyzed to identify cancer-related categories (which can include, for example, measuring the presence, absence or quantitative assessment (e.g., risk, predicted outcome) of a cancer-related category). Such objects can include objects with one or more cancer-related categories and objects without cancer-related categories. Cancer-related categories or states can include, for example, cancer-positive, cancer-negative, cancer stage, predicted response to cancer treatment and / or predicted long-term outcomes (e.g., disease metastasis, biochemical recurrence, partial response, complete response, relative survival, cancer-specific survival, progression-free survival, disease-free survival, five-year survival, or overall survival).

[0064] Measurement of biological samples

[0065] The biological sample can be obtained or obtained from a human subject (e.g., a male subject). The biological sample can be stored under various storage conditions (e.g., different temperatures (e.g., at room temperature, under refrigerated or frozen conditions, at 25° C., at 4° C., at -18° C., at -20° C., or at -80° C.), different suspensions (e.g., formalin, EDTA collection tubes, cell-free RNA collection tubes, or cell-free DNA collection tubes)) before processing. The biological sample can be obtained from a subject having or suspected of having cancer (e.g., prostate cancer), or from a subject who has never had or is not suspected of having cancer.

[0066] Biological samples can be used to diagnose, detect or identify the disease or health or physiological condition of an object by analyzing the biological sample. Biological samples or parts thereof can be analyzed to determine the possibility that the sample is positive for a disease or health condition (e.g., prostate cancer). Alternatively or additionally, the methods described herein may include diagnosing an object with a disease or health condition, monitoring the disease or health condition of an object and / or determining the tendency of an object to a health disease / condition. In some embodiments, biological samples can be used to classify samples and / or objects into cancer-related categories and / or identify an object as having a specific cancer-related state. Cancer-related categories or states may include diagnosis (e.g., positive or negative for cancer), a specific type of cancer (e.g., prostate cancer), a cancer stage, a predicted outcome or prognosis, a predicted response to one or more treatments, or any combination thereof.

[0067] Any measurable substance can be used as the source of the sample. The substance can be a fluid, such as a biological fluid. The fluid substance can include blood (e.g., whole blood, plasma, serum), cord blood, saliva, urine, sweat, serum, semen, vaginal fluid, gastric juice and digestive juice, cerebrospinal fluid, placental fluid, cavity fluid, eye fluid, serum, milk, lymph fluid or a combination thereof.

[0068] The substance can be a solid, such as a biological tissue. The substance can include normal healthy tissue. The tissue can be associated with various types of organs. Non-limiting examples of organs can include the brain, breast, liver, lung, kidney, prostate, ovary, spleen, lymph nodes (including tonsils), thyroid, pancreas, heart, skeletal muscle, intestine, larynx, esophagus, stomach, or a combination thereof.

[0069] The substance may include a tumor. The tumor may be a benign tumor (non-cancer), a premalignant tumor, a malignant tumor (cancer) or any metastasis thereof. Non-limiting examples of tumors and associated cancers may include: acoustic neuroma, acute lymphocytic leukemia, acute myeloid leukemia, adenocarcinoma, adrenocortical carcinoma, AIDS-related cancers, AIDS-related lymphomas, anal cancer, angiosarcoma, appendix cancer, astrocytoma, basal cell carcinoma, bile duct cancer, bladder cancer, bone cancer, brain tumors (such as cerebellar astrocytoma, cerebral astrocytoma / malignant glioma), ependymoma, medulloblastoma, supratentorial primitive neuroectodermal tumor, visual pathway and hypothalamic glioma, breast cancer, bronchial adenoma, Burkitt's lymphoma, primary unknown cancer, central nervous system lymphoma, bronchial carcinoma, cerebellar astrocytoma, cervical cancer, childhood cancer, chondromalacia, cerebral astrocytoma, cervical cancer, pediatric cancer, chondromalacia, cerebral astrocytoma ... tumor, chordoma, choriocarcinoma, chronic lymphocytic leukemia, chronic myeloproliferative disease, colon cancer, colorectal cancer, craniopharyngioma, cutaneous T-cell lymphoma, cystadenocarcinoma, desmoplastic small round cell tumor, embryonal carcinoma, endocrine system cancer, endometrial cancer, endothelial sarcoma, ependymoma, epithelial cancer, esophageal cancer, Ewing's sarcoma, fibrosarcoma, germ cell tumor, gallbladder cancer, gastric cancer, gastrointestinal carcinoid tumor, gastrointestinal stromal tumor, gastrointestinal system cancer, genitourinary system cancer, glioma, hairy cell leukemia, head and neck cancer, heart cancer, hemangioblastoma, hepatocellular (liver) cancer, Hodgkin lymphoma, hypopharyngeal cancer, intraocular melanoma, islet cell carcinoma, Kaposi's sarcoma, kidney cancer, laryngeal cancer cancer, leiomyosarcoma, lip and oral cancer, liposarcoma, liver cancer, lung cancer (e.g., non-small cell and small cell lung cancer), lung tumors, lymphangiosarcoma, lymphangioendothelial sarcoma, lymphoma, leukemia, macroglobulinemia, malignant fibrous histiocytoma of bone / osteosarcoma, medulloblastoma, medullary carcinoma, melanoma, meningioma, mesothelioma, metastatic squamous neck cancer with occult primary, oral cancer, multiple endocrine neoplasms syndrome, myelodysplastic syndrome, myeloid leukemia, myxosarcoma, nasal cavity and paranasal sinus cancer, nasopharyngeal cancer, neuroblastoma, non-Hodgkin lymphoma, non-small cell lung cancer, oligodendroneuroma, oral cancer, oropharyngeal cancer, osteosarcoma / malignant fibrous histiocytoma of bone, ovarian cancer, ovarian epithelial cancer, Ovarian germ cell tumors, pancreatic cancer, pancreatic islet cell tumors, papillary adenocarcinoma, papillary carcinoma, paranasal sinus and nasal cavity cancer, parathyroid cancer, penile cancer, pharyngeal cancer, pheochromocytoma, pineal astrocytoma, pineal germ cell tumor, pituitary adenoma, pleuropulmonary blastoma, plasma cell tumor, primary central nervous system lymphoma, prostate cancer, rectal cancer, renal cell carcinoma, transitional cell carcinoma of the renal pelvis and ureter, retinoblastoma, rhabdomyosarcoma, salivary gland cancer, sarcoma, sebaceous gland carcinoma, seminoma, skin cancer, Merkel cell skin cancer, small intestine cancer, soft tissue sarcoma, squamous cell carcinoma, gastric cancer, sweat gland cancer, synovioma, T-cell lymphoma, testicular tumors, pharyngeal cancer, thymoma, thymic cancer, thyroid cancer,Trophoblastic tumor (pregnancy), cancer of unknown primary site, urethral cancer, uterine sarcoma, vaginal cancer, vulvar cancer, Waldenstrom's macroglobulinemia, Wilms' tumor, or a combination thereof. Tumors can be associated with various types of organs. Non-limiting examples of organs can include brain, breast, liver, lung, kidney, prostate, ovary, spleen, lymph nodes (including tonsils), thyroid, pancreas, heart, skeletal muscle, intestine, larynx, esophagus, stomach, or a combination thereof.

[0070] The material can include a mixture of normal healthy tissue or tumor tissue. Tissue can be associated with various types of organs. Non-limiting examples of organs can include brain, breast, liver, lung, kidney, prostate, ovary, spleen, lymph nodes (including tonsils), thyroid, pancreas, heart, skeletal muscle, intestine, larynx, esophagus, stomach or a combination thereof. In some embodiments, tissue is associated with the prostate of the object. In the case where the biological sample includes cells and / or tissues (e.g., biopsy samples), the biological sample can be further analyzed or measured. In some embodiments, the biopsy sample can be fixed, processed (e.g., dehydrated), embedded, frozen, stained and / or examined under a microscope. In some embodiments, digital slices are generated from the processed sample.

[0071] In some embodiments, the substance can include a variety of cells, including eukaryotic cells, prokaryotic cells, fungal cells, heart cells, lung cells, kidney cells, liver cells, pancreatic cells, germ cells, stem cells, induced pluripotent stem cells, gastrointestinal cells, blood cells, cancer cells, bacterial cells, bacterial cells isolated from a human microbiome sample, and circulating cells in human blood. In some embodiments, the substance can include the contents of a cell, such as, for example, the contents of a single cell or the contents of a plurality of cells.

[0072] In some embodiments, the substance may include one or more markers, the presence or absence of which indicates certain phenomena, such as disease, disorder, infection, or environmental exposure. The marker may be, for example, a cell, a small molecule, a macromolecule, a protein, a glycoprotein, a carbohydrate, a sugar, a polypeptide, a nucleic acid (e.g., deoxyribonucleic acid (DNA), ribonucleic acid (RNA)), a cell-free nucleic acid (e.g., cf-DNA, cf-RNA), a lipid, a cellular component, or a combination thereof.

[0073] Biological samples can be collected before and / or after treating a subject with cancer. Biological samples can be obtained from a subject during treatment or a treatment regimen. Multiple biological samples can be obtained from a subject to monitor the effect of treatment over time. Biological samples can be collected from a subject known or suspected of having cancer (e.g., prostate cancer). Biological samples can be collected from subjects with unexplained symptoms (such as fatigue, nausea, weight loss, aches, weakness, or bleeding). Biological samples can be collected from subjects with explained symptoms. Biological samples can be collected from subjects at risk of cancer due to factors such as family history, age, hypertension or prehypertension, diabetes or prediabetes, overweight or obesity, environmental exposures, lifestyle risk factors (e.g., smoking, drinking, or drug use), or the presence of other risk factors.

[0074] After obtaining the biological sample from the object, the biological sample can be processed to generate a data set indicating the disease, illness, cancer-related category or health status of the object. For example, the tissue sample can be subjected to histopathological determination (e.g., microscopy, including digital image acquisition, such as full-slice imaging) to generate image data based on the biological sample. Alternatively, the liquid sample or marker separated from the sample can be tested (e.g., clinical laboratory test) to generate tabular data. In some embodiments, the sample is measured to determine the presence, absence or amount of one or more metabolites (e.g., prostate-specific antigen (PSA)).

[0075] Data Types

[0076] The methods and systems described herein take one or more data sets as input. One or more data sets may include table and / or image data. The table and / or image data may be derived from a biological sample of a subject. In some embodiments, the data is not derived from a biological sample.

[0077] The data may include images of tissue samples taken from a biopsy of a subject. The image data may be acquired by microscopy of a biopsy sample. Microscopy may include optical microscopy, virtual or digital microscopy (such as whole slide imaging (WSI)), or any suitable microscopy technique known in the art. The microscopy image may be subjected to one or more processing steps, such as filtering, segmentation, stitching, or object detection.

[0078] Tabular data as described herein may include any non-image data related to the health state or condition (e.g., disease) of a subject. Tabular data may include clinical data, such as laboratory data at one or more time points (e.g., prostate serum antigen (PSA) levels), qualitative measurements of cytopathology (e.g., Gleason grade, Gleason score), structured or unstructured health data (e.g., digital rectal examination results), medical imaging data or results (e.g., results of X-rays, computed tomography (CT) scans, magnetic resonance imaging (MRI) scans, positron emission tomography (PET) scans, or ultrasound examinations (such as transrectal ultrasound examination results)), age, medical history, previous or current cancer status (e.g., remission, metastasis) or stage, current or previous therapeutic interventions, long-term results, and / or the National Comprehensive Cancer Network (NCCN) classification or its components (e.g., combined Gleason score, t stage, baseline PSA).

[0079] In some embodiments, the therapeutic intervention may include radiation therapy (RT). In some embodiments, the therapeutic intervention may include chemotherapy. In some embodiments, the therapeutic intervention may include surgical intervention. In some embodiments, the therapeutic intervention may include immunotherapy. In some embodiments, the therapeutic intervention may include hormone therapy. In some embodiments, RT may include RT pre-specified with short-term androgen deprivation therapy (ST-ADT). In some embodiments, RT may include RT pre-specified with long-term ADT (LT-ADT). In some embodiments, RT may include RT pre-specified with dose-escalating RT (DE-RT). In some embodiments, surgical intervention may include radical prostatectomy (RP). In some embodiments, the therapeutic intervention may include any combination of the therapeutic interventions disclosed herein. In some embodiments, the long-term outcome may include distant metastasis (DM). In some embodiments, the long-term outcome may include biochemical recurrence (BR). In some embodiments, the long-term outcome may include partial remission. In some embodiments, the long-term outcome may include complete remission. In some embodiments, the long-term outcome may include death. In some embodiments, the long-term outcome may include relative survival. In some embodiments, the long-term outcome may include cancer-specific survival. In some embodiments, cancer-specific survival may include prostate cancer-specific survival (PCaSS). In some embodiments, long-term outcomes may include progression-free survival. In some embodiments, long-term outcomes may include disease-free survival. In some embodiments, long-term outcomes may include five-year survival. In some embodiments, long-term outcomes may include overall survival (OS). In some embodiments, long-term outcomes may include any combination of long-term outcomes disclosed herein.

[0080] The data used in the methods and systems described herein can be processed through one or more processing steps. In some embodiments, data (e.g., image data) is processed through image processing, image segmentation, and / or target detection processes, such as those encoded in image processing, image segmentation, or target detection algorithms. The image processing process can filter, transform, scale, rotate, mirror, shear, combine, compress, segment, splice, extract features, and / or smooth the image before downstream processing. In some embodiments, multiple images (e.g., histopathology sections) are combined to form image splicing. Image splicing can be converted into a representation (e.g., tensor) that can be used for downstream processing of image data. The image segmentation process can segment the image into one or more fragments containing factors or regions of interest. For example, an image segmentation algorithm can process digital histopathology sections to determine tissue regions that are completely different from blank areas or artifacts. In some embodiments, the image segmentation algorithm can include machine learning or artificial intelligence algorithms. In some embodiments, image segmentation can precede image processing. In some embodiments, image processing can precede image segmentation. The target detection process can include detecting the presence or absence of a target target (e.g., a cell or cell part, such as a cell nucleus). In some embodiments, target detection can precede image processing and / or image segmentation. For example, an image containing one or more targets of interest found by an image detection algorithm can be spliced ​​together in a subsequent image processing step. Alternatively or additionally, image processing can precede target detection and / or image segmentation. For example, the original image data can be processed (e.g., filtered), and the target detection algorithm is performed on the processed image data. Multiple image processing, image segmentation and / or target detection steps can be performed on the image data in any appropriate order. In an example, one or more image processing steps are optionally performed on the image data to improve image quality. The processed image is then subjected to an image segmentation algorithm to detect an area of ​​interest (e.g., a tissue area in a set of histopathology sections). The target detection algorithm (e.g., an algorithm for detecting cell nuclei in tissue images) is then performed on the area of ​​interest, and the area found to have at least one target target is spliced ​​together to generate processed image data for downstream use.

[0081] In some embodiments, data (e.g., tabular data) may be processed through one or more processing steps. Processing steps may include, but are not limited to, standardization or normalization. One or more processing steps may, for example, discard data containing false values ​​or containing very few observations. One or more processing steps may further or alternatively standardize the encoding of data values. Depending on the source of the data set, different input data sets may have the same parameter values ​​encoded in different ways. For example, "900", "900.0", "904", "904.0", "-1", "-1.0", "nothing" and "NaN" can all encode "missing" parameter values. One or more processing steps may identify the encoding changes of the same value and standardize the data set to uniformly encode a given parameter value. Therefore, the processing steps may reduce irregularities in the input data for downstream use. One or more data sets may standardize parameter values. For example, numerical data may be scaled, whitened, colored, de-correlated or standardized. For example, data may be scaled or shifted to be located in a specific interval (e.g., [0,1] or [-1,1]) and / or to eliminate correlation. In some embodiments, categorical data may be encoded as a unique hot vector. In some embodiments, one or more different types of tabular (eg, numerical, categorical) data may be stitched together. In some embodiments, the data does not undergo a processing step.

[0082] Data can be obtained at one or more time points. In some embodiments, data can be obtained at the initial time point and a later time point. The initial time point and a later time point can be spaced apart by any appropriate amount of time, such as 1 hour, 1 day, 1 week, 2 weeks, 3 weeks, 4 weeks, 6 weeks, 12 weeks, 4 months, 5 months, 6 months, 7 months, 8 months, 9 months, 10 months, 11 months, 1 year, 2 years, 3 years, 4 years, 5 years, 6 years, 7 years, 8 years, 9 years, 10 years or longer. In some embodiments, data are from more than two time points. In some embodiments, data are from 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 or more time points.

[0083] Trained Algorithm

[0084] After one or more biological samples derived from a subject are processed using one or more assays to generate one or more data sets indicating a cancer state (e.g., one or more cancer-related categories) of the subject, the one or more data sets (e.g., visual data and / or tabular data) can be processed using a trained algorithm to determine the cancer state of the subject. For example, a trained algorithm can be used to determine the presence or absence of cancer (e.g., prostate cancer) in a subject based on image data and / or laboratory data. The trained algorithm can be configured to identify cancer status with an accuracy of at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 85%, at least about 90%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, or more than 99% for at least about 25, at least about 50, at least about 100, at least about 150, at least about 200, at least about 250, at least about 300, at least about 350, at least about 400, at least about 450, at least about 500, or more than about 500 independent samples.

[0085] The trained algorithm may include an unsupervised machine learning algorithm. The trained algorithm may include a supervised machine learning algorithm. The trained algorithm may include a classification and regression tree (CART) algorithm. The supervised machine learning algorithm may include, for example, a random forest, a support vector machine (SVM), a neural network, or a deep learning algorithm. The trained algorithm may include a self-supervised machine learning algorithm.

[0086] In some embodiments, the machine learning algorithm of the method or system described herein utilizes one or more neural networks. In some cases, a neural network is a computing system that can learn the relationship between an input data set and a target data set. A neural network can be a software representation of a human nervous system (e.g., a cognitive system) that is intended to capture the "learning" and "generalization" capabilities used by humans. In some embodiments, a machine learning algorithm includes a neural network comprising a CNN. Non-limiting examples of structural components of the machine learning algorithm described herein include: CNN, recurrent neural network, expanded CNN, fully connected neural network, deep generative model, and Boltzmann machine.

[0087] In some embodiments, a neural network includes a series of layers called "neurons". In some embodiments, a neural network includes an input layer (to which data is presented); one or more internal and / or "hidden" layers; and an output layer. Neurons can be connected to neurons in other layers through connections with weights, which are parameters that control the strength of the connections. The number of neurons in each layer can be related to the complexity of the problem to be solved. The minimum number of neurons required in a layer can be determined by the problem complexity, and the maximum number can be limited by the generalization ability of the neural network. The input neurons can receive the presented data and then transmit the data to the first hidden layer through the weights of the connections, which are modified during training. The first hidden layer can process the data and transmit its results to the next layer through a second set of weighted connections. Each subsequent layer can "aggregate" the results from the previous layers into more complex relationships. In addition, while traditional software programs require specific instructions to be written to perform functions, neural networks are programmed by training them using known sample sets and allowing them to modify themselves during (and after) training to provide the desired output (such as output values). After training, when new input data is presented to the neural network, it is configured to generalize what it "learned" during training and apply what it learned from training to new, previously unseen input data in order to generate an output associated with that input.

[0088] In some embodiments, the neural network includes an artificial neural network (ANN). The ANN can be a machine learning algorithm that can be trained to map an input data set to an output data set, wherein the ANN includes an interconnected node group organized into multiple layers of nodes. For example, the ANN architecture can include at least an input layer, one or more hidden layers, and an output layer. The ANN can include any total number of layers and any number of hidden layers, wherein the hidden layers are used as trainable feature extractors that allow the input data set to be mapped to an output value or set of output values. As used herein, a deep learning algorithm (such as a deep neural network (DNN)) is an ANN including multiple hidden layers (e.g., two or more hidden layers). Each layer of the neural network can include multiple nodes (or "neurons"). The node receives input directly from the input data or the output of the node in the previous layer, and performs a specific operation, such as a summation operation. The connection from the input to the node is associated with a weight (or weighting factor). The node can add the product of all input pairs and their associated weights. The weighted sum can be offset with a bias. The output of a node or neuron can be gated using a threshold or an activation function. The activation function can be a linear or nonlinear function. The activation function can be, for example, a rectified linear unit (ReLU) activation function, a leaky ReLU activation function, or other functions such as a saturated hyperbolic tangent function, an identity function, a binary step function, a logistic function, an inverse tangent function, a soft sign function, a parametric rectified linear unit function, an exponential linear unit function, a softplus function, a curved identity function, a soft exponential function, a sine function, a sinc function, a Gaussian function or a sigmoid function, or any combination thereof.

[0089] The weight factors, bias values, and thresholds or other computational parameters of a neural network can be "taught" or "learned" during a training phase using one or more training data sets. For example, the parameters can be trained using input data from a training data set and gradient descent or back-propagation methods so that the output values ​​computed by the ANN are consistent with the examples contained in the training data set.

[0090] The number of nodes used in the input layer of the ANN or DNN can be at least about 10, 50, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10,000, 20,000, 30,000, 40,000, 50,000, 60,000, 70,000, 80,000, 90,000, 100,000 or more. In other cases, the number of nodes used in the input layer can be at most about 100,000, 90,000, 80,000, 70,000, 60,000, 50,000, 40,000, 30,000, 20,000, 10,000, 9000, 8000, 7000, 6000, 5000, 4000, 3000, 2000, 1000, 900, 800, 700, 600, 500, 400, 300, 200, 100, 50, 10 or less. In some cases, the total number of layers (including input and output layers) used in the ANN or DNN can be at least about 3, 4, 5, 10, 15, 20 or more. In other cases, the total number of layers can be at most about 20, 15, 10, 5, 4, 3 or less.

[0091] In some cases, the total number of learnable or trainable parameters (e.g., weighting factors, biases, or thresholds) used in an ANN or DNN can be at least about 10, 50, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, 10,000, 20,000, 30,000, 40,000, 50,000, 60,000, 70,000, 80,000, 90,000, 100,000, or more. In other cases, the number of learnable parameters may be at most about 100,000, 90,000, 80,000, 70,000, 60,000, 50,000, 40,000, 30,000, 20,000, 10,000, 9000, 8000, 7000, 6000, 5000, 4000, 3000, 2000, 1000, 900, 800, 700, 600, 500, 400, 300, 200, 1000, 900, 800, 700, 600, 500, 400, 300, 200, 100, 50, 10, or less.

[0092] In some embodiments of the machine learning algorithm described herein, the machine learning algorithm includes a neural network, such as a deep CNN. In some embodiments in which a CNN is used, the network is composed of any number of convolutional layers, expansion layers, or fully connected layers. In some embodiments, the number of convolutional layers is between 1-10, and the number of expansion layers is between 0-10. The total number of convolutional layers (including input layers and output layers) can be at least about 1, 2, 3, 4, 5, 10, 15, 20 or more, and the total number of expansion layers can be at least about 1, 2, 3, 4, 5, 10, 15, 20 or more. The total number of convolutional layers can be at most about 20, 15, 10, 5, 4, 3 or less, and the total number of expansion layers can be at most about 20, 15, 10, 5, 4, 3 or less. In some embodiments, the number of convolutional layers is between 1-10, and the number of fully connected layers is between 0-10. The total number of convolutional layers (including input and output layers) may be at least about 1, 2, 3, 4, 5, 10, 15, 20, or more, and the total number of fully connected layers may be at least about 1, 2, 3, 4, 5, 10, 15, 20, or more. The total number of convolutional layers may be at most about 20, 15, 10, 5, 4, 3, 2, 1, or less, and the total number of fully connected layers may be at most about 20, 15, 10, 5, 4, 3, 2, 1, or less.

[0093] In some embodiments, the machine learning algorithm comprises a neural network comprising a CNN, a RNN, a dilated CNN, a fully connected neural network, a deep generative model, and / or a deep restricted Boltzmann machine.

[0094] In some embodiments, the machine learning algorithm includes one or more CNNs. The CNN can be a deep and feed-forward ANN. The CNN can be suitable for analyzing visual images. The CNN can include an input, an output layer, and multiple hidden layers. The hidden layers of the CNN can include convolutional layers, pooling layers, fully connected layers, and normalization layers. These layers can be organized in three dimensions (width, height, and depth).

[0095] A convolutional layer can apply a convolution operation to the input and pass the result of the convolution operation to the next layer. For processing images, the convolution operation can reduce the number of free parameters, allowing the network to become deeper with fewer parameters. In a neural network, each neuron can receive input from a certain number of locations in the previous layer. In a convolutional layer, a neuron can receive input only from a restricted sub-region of the previous layer. The parameters of a convolutional layer can include a set of learnable filters (or kernels). Learnable filters can have a small receptive field and extend through the entire depth of the input volume. During the forward pass, each filter can be convolved across the width and height of the input volume, the dot product between the filter entries and the input is calculated, and a two-dimensional activation map of the filter is produced. Therefore, the network can learn filters that are activated when it detects certain specific types of features at a certain spatial location in the input.

[0096] In some embodiments, the machine learning algorithm includes an RNN. An RNN is a neural network with recurrent connections that can encode and process sequential data. The RNN may include an input layer configured to receive an input sequence. The RNN may additionally include one or more hidden recurrent layers that maintain states. At each step, each hidden recurrent layer can calculate the output and next state of the layer. The next state can depend on the previous state and the current input. The state can be maintained across steps and can capture dependencies in the input sequence.

[0097] The RNN may be a long short-term memory (LSTM) network. The LSTM network may be composed of LSTM cells. An LSTM cell may include a cell, an input gate, an output gate, and a forget gate. The cell may be responsible for tracking dependencies between elements in an input sequence. The input gate may control the extent to which new values ​​flow into the cell, the forget gate may control the extent to which values ​​remain in the cell, and the output gate may control the extent to which values ​​in the cell are used to compute the output activation of the LSTM cell.

[0098] Alternatively, an attention mechanism (e.g., a transformer). An attention mechanism can focus on or "attend to" certain regions of the input while ignoring other regions. This can improve model performance because certain regions of the input may be less relevant. At each step, the attention unit can compute the dot product of the context vector and the input to that step, among other operations. The output of the attention unit can define where in the input sequence the most relevant information is located.

[0099] In some embodiments, the pooling layer includes a global pooling layer. The global pooling layer can combine the outputs of the neuron clusters of one layer into a single neuron in the next layer. For example, the maximum pooling layer can use the maximum value of each neuron cluster from the previous layer; and the average pooling layer can use the average value of each neuron cluster of the previous layer.

[0100] In some embodiments, a fully connected layer connects each neuron in one layer to each neuron in another layer. In a neural network, each neuron can receive input from a certain number of locations in the previous layer. In a fully connected layer, each neuron can receive input from every element in the previous layer.

[0101] In some embodiments, the normalization layer is a batch normalization layer. The batch normalization layer can improve the performance and stability of the neural network. The batch normalization layer can provide inputs with zero mean / unit variance for any layer in the neural network. Advantages of using a batch normalization layer can include faster trained networks, higher learning rates, easier initialization of weights, more feasible activation functions, and a simpler process for creating deep networks.

[0102] The trained algorithm can be configured to accept multiple input variables and generate one or more output values ​​based on the multiple input variables. The multiple input variables can include one or more data sets indicating cancer-related categories. For example, the input variables can include microscopic images of biopsy samples of the subject. The multiple input variables can also include clinical health data of the subject.

[0103] The trained algorithm may include a classifier so that each of the one or more output values ​​includes one of a fixed number of possible values ​​(e.g., a linear classifier, a logistic regression classifier, etc.), which indicates the classification of biological samples and / or objects performed by the classifier. The trained algorithm may include a binary classifier so that each of the one or more output values ​​includes one of two values ​​(e.g., {0, 1}, {positive, negative}, or {high risk, low risk}), which indicates the classification of biological samples and / or objects performed by the classifier. The trained algorithm may be another type of classifier so that each of the one or more output values ​​includes more than two values ​​(e.g., {0, 1, 2}, {positive, negative or uncertain}, or {high risk, medium risk, or low risk}), which indicates the classification of biological samples and / or objects performed by the classifier. The output value may include a descriptive label, a numerical value, or a combination thereof. Some output values ​​may include a descriptive label. Such a descriptive label may provide identification or indication of a disease or condition state of an object, and may include, for example, positive, negative, high risk, medium risk, low risk, or uncertain. Such descriptive labels can provide identification of a treatment for a cancer-related category of a subject, and can include, for example, the therapeutic intervention, the duration of the therapeutic intervention, and / or a dosage of the therapeutic intervention appropriate for treating a subject classified into a particular cancer-related category.

[0104] Some output values ​​may include numerical values, such as binary values, integer values, or continuous values. Such binary output values ​​may include, for example, {0, 1}, {positive, negative}, or {high risk, low risk}. Such integer output values ​​may include, for example, {0, 1, 2}. Such continuous output values ​​may include, for example, a probability value that is at least 0 and not greater than 1. Such continuous output values ​​may include, for example, a non-normalized probability value that is at least 0. Such continuous output values ​​may indicate the prognosis of a cancer-related category of an object. Some numerical values ​​may be mapped to descriptive labels, for example, by mapping 1 to "positive" and 0 to "negative".

[0105] Some output values ​​can be assigned based on one or more cutoff values. For example, if the sample indicates that the subject has at least a 50% probability of having a cancer-related state (e.g., a type or stage of cancer) or belonging to a cancer-related class, the binary classification of the sample can assign an output value of "positive" or 1. For example, if the sample indicates that the subject has less than a 50% probability of belonging to a cancer-related class, the binary classification of the sample can assign an output value of "negative" or 0. In this case, a single cutoff value of 50% is used to classify the sample into one of two possible binary output values. Examples of single cutoff values ​​can include about 1%, about 2%, about 5%, about 10%, about 15%, about 20%, about 25%, about 30%, about 35%, about 40%, about 45%, about 50%, about 55%, about 60%, about 65%, about 70%, about 75%, about 80%, about 85%, about 90%, about 91%, about 92%, about 93%, about 94%, about 95%, about 96%, about 97%, about 98%, and about 99%.

[0106] As another example, if the sample indicates that the subject is at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 85%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99% or more belongs to a cancer-related category (e.g., cancer diagnosis or prognosis), then the classification of the sample can be assigned an output value of "positive" or 1. If the sample indicates that the probability of the subject belonging to a cancer-related category (e.g., a long-term outcome) is greater than about 50%, greater than about 55%, greater than about 60%, greater than about 65%, greater than about 70%, greater than about 75%, greater than about 80%, greater than about 85%, greater than about 90%, greater than about 91%, greater than about 92%, greater than about 93%, greater than about 94%, greater than about 95%, greater than about 96%, greater than about 97%, greater than about 98%, or greater than about 99%, then the classification of the sample can be assigned an output value of "positive" or 1.

[0107] If the sample indicates that the subject has a cancer-related condition or belongs to a cancer-related category (e.g., positive for prostate cancer) with a probability of less than about 50%, less than about 45%, less than about 40%, less than about 35%, less than about 30%, less than about 25%, less than about 20%, less than about 15%, less than about 10%, less than about 9%, less than about 8%, less than about 7%, less than about 6%, less than about 5%, less than about 4%, less than about 3%, less than about 2%, or less than about 1%, then the classification of the sample can be assigned an output value of "negative" or 0. If the sample indicates that the subject has a probability of no greater than about 50%, no greater than about 45%, no greater than about 40%, no greater than about 35%, no greater than about 30%, no greater than about 25%, no greater than about 20%, no greater than about 15%, no greater than about 10%, no greater than about 9%, no greater than about 8%, no greater than about 7%, no greater than about 6%, no greater than about 5%, no greater than about 4%, no greater than about 3%, no greater than about 2%, or no greater than about 1%, then the classification of the sample can be assigned an output value of "negative" or 0.

[0108] If a sample is not classified as "positive", "negative", 1, or 0, the classification of the sample can be assigned an output value of "uncertain" or 2. In this case, two cutoff value sets are used to classify the sample into one of three possible output values. Examples of cutoff value sets can include {1%, 99%}, {2%, 98%}, {5%, 95%}, {10%, 90%}, {15%, 85%}, {20%, 80%}, {25%, 75%}, {30%, 70%}, {35%, 65%}, {40%, 60%}, and {45%, 55%}. Similarly, n cutoff value sets can be used to classify a sample into one of n+1 possible output values, where n is any positive integer.

[0109] The trained algorithm can be trained with multiple independent training samples. Each independent training sample can include a biological sample from an object, an associated data set obtained by measuring the biological sample (as described elsewhere herein), clinical data from the object, and one or more known output values ​​corresponding to the biological sample and / or the object (e.g., clinical diagnosis, prognosis, absence or therapeutic effect of the cancer-related state of the object). Independent training samples can include biological samples obtained or obtained from multiple different objects and associated data sets and outputs. Independent training samples can include biological samples obtained or obtained from the same object at multiple different time points (e.g., regularly, such as weekly, biweekly, monthly, annually, etc.) and associated data sets and outputs. Independent training samples can be associated with the presence of cancer-related states (e.g., training samples include biological samples obtained or obtained from multiple objects known to have cancer-related states and associated data sets and outputs). Independent training samples can be associated with the absence of cancer-related states (e.g., training samples include biological samples obtained or obtained from multiple objects known to have no previous diagnosis of cancer-related states or have received negative test results for cancer-related states and associated data sets and outputs).

[0110] The trained algorithm can be trained using at least about 5, at least about 10, at least about 15, at least about 20, at least about 25, at least about 30, at least about 35, at least about 40, at least about 45, at least about 50, at least about 100, at least about 150, at least about 200, at least about 250, at least about 300, at least about 350, at least about 400, at least about 450, or at least about 500 independent training samples. The independent training samples can include cell-free biological samples and clinical data associated with the presence of cancer-associated classes and / or cell-free biological samples and clinical data associated with the absence of cancer-associated classes. The trained algorithm can be trained using no more than about 500, no more than about 450, no more than about 400, no more than about 350, no more than about 300, no more than about 250, no more than about 200, no more than about 150, no more than about 100, or no more than about 50 independent training samples associated with the presence of cancer-associated classes. In some embodiments, the biological samples and / or clinical data are separate from the samples used to train the trained algorithm.

[0111] The trained algorithm may be trained using a first number of independent training samples associated with the presence of a cancer-related class and a second number of independent training samples associated with the absence of a cancer-related class. The first number of independent training samples associated with the presence of a cancer-related class may be no greater than the second number of independent training samples associated with the absence of a cancer-related class. The first number of independent training samples associated with the presence of a cancer-related class may be equal to the second number of independent training samples associated with the absence of a cancer-related class. The first number of independent training samples associated with the presence of a cancer-related class may be greater than the second number of independent training samples associated with the absence of a cancer-related class.

[0112] The trained algorithm can be configured to obtain a training set of at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 76%, at least about 77%, at least about 78%, at least about 79%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, at least about 100%, at least about 100%, at least about 150%, at least about 200%, at least about 250%, at least about 300%, at least about 350%, at least about 400%, at least about 450, or at least about 500 independent training samples. , at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99% or more accuracy to identify the cancer-associated class. The accuracy of identifying the cancer-associated class by the trained algorithm can be calculated as the percentage of independent test samples (e.g., subjects known to belong to the cancer-associated class or subjects with a negative clinical test result for the cancer-associated class) that are correctly identified or classified as belonging to or not belonging to the cancer-associated class.

[0113] The trained algorithm can be configured to identify a cancer-associated class with a positive predictive value (PPV) of at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, or more. The PPV of using a trained algorithm to identify a cancer-associated class can be calculated as the percentage of cell-free biological samples identified or classified as having a cancer-associated class that correspond to objects that truly belong to the cancer-associated class.

[0114] The trained algorithm can be configured to identify a cancer-associated class with a negative predictive value (NPV) of at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, or more. The NPV of using a trained algorithm to identify cancer-related classes can be calculated as the percentage of a dataset of objects identified or classified as not belonging to the cancer-related class that correspond to objects that truly do not belong to the cancer-related class.

[0115] The trained algorithm can be configured to improve the performance of the algorithm by at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, at least about 100%, at least about 101%, at least about 102%, at least about 103%, at least about 104%, at least about 105%, at least about 106%, at least about 107%, at least about 108%, at least about 109%, at least about 110%, at least about 111%, at least about 112%, at least about 113%, at least about 114%, at least about 115%, at least about The clinical sensitivity of using the trained algorithm to identify a cancer-associated class can be calculated as the percentage of independent test samples associated with the cancer-associated class (e.g., objects known to belong to the cancer-associated class) that are correctly identified or classified as having the cancer-associated class.

[0116] The trained algorithm can be configured to improve the performance of the algorithm by at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, at least about 100%, at least about 101%, at least about 102%, at least about 103%, at least about 104%, at least about 105%, at least about 106%, at least about 107%, at least about 108%, at least about 109%, at least about 110%, at least about 111%, at least about 112%, at least about 113%, at least about 114%, at least about 115%, at least about 0%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, at least about 99.1%, at least about 99.2%, at least about 99.3%, at least about 99.4%, at least about 99.5%, at least about 99.6%, at least about 99.7%, at least about 99.8%, at least about 99.9%, at least about 99.99%, at least about 99.999% or more clinical specificity to identify a cancer-associated class. The clinical specificity of using a trained algorithm to identify a cancer-associated class can be calculated as the percentage of independent test samples (e.g., subjects with a negative clinical test result for a cancer-associated class) associated with the absence of a cancer-associated class that are correctly identified or classified as not belonging to the cancer-associated class.

[0117] The trained algorithm can be configured to identify a cancer-associated class with an area under the curve (AUC) of at least about 0.50, at least about 0.55, at least about 0.60, at least about 0.65, at least about 0.70, at least about 0.75, at least about 0.80, at least about 0.81, at least about 0.82, at least about 0.83, at least about 0.84, at least about 0.85, at least about 0.86, at least about 0.87, at least about 0.88, at least about 0.89, at least about 0.90, at least about 0.91, at least about 0.92, at least about 0.93, at least about 0.94, at least about 0.95, at least about 0.96, at least about 0.97, at least about 0.98, at least about 0.99, or more. AUC can be calculated as the integral of a receiver operating characteristic (ROC) curve (eg, the area under the ROC curve) associated with a trained algorithm in classifying a subject-derived dataset as belonging to or not belonging to a cancer-related class.

[0118] The trained algorithm can be adjusted or tuned to improve one or more of performance, accuracy, PPV, NPV, clinical sensitivity, clinical specificity, or AUC for identifying cancer-related classes. The trained algorithm can be adjusted or tuned by adjusting parameters of the trained algorithm (e.g., a set of cutoff values ​​for classifying biological samples, as described elsewhere herein, or weights of a neural network). The trained algorithm can be adjusted or tuned continuously during the training process or after the training process is completed.

[0119] After the trained algorithm is initially trained, the input subset can be identified as the most influential or important subset to be included for high-quality classification. For example, a clinical data subset can be identified as the most influential or important subset to be included for high-quality classification or identification of cancer-related categories (or subtypes of cancer-related categories). Clinical data or a subset thereof can be sorted based on a classification index that indicates the impact or importance of each parameter on high-quality classification or identification of cancer-related categories (or subtypes of cancer-related categories). In some embodiments, such an index can be used to significantly reduce the number of input variables (e.g., predictors) that can be used to train a trained algorithm to a desired performance level (e.g., based on a desired minimum accuracy, PPV, NPV, clinical sensitivity, clinical specificity, AUC, or a combination thereof). For example, if training a trained algorithm using a plurality of input variables comprising tens or hundreds of input variables results in a classification accuracy greater than 99%, then instead training the trained algorithm using only a selected subset of no more than about 5, no more than about 10, no more than about 15, no more than about 20, no more than about 25, no more than about 30, no more than about 35, no more than about 40, no more than about 45, no more than about 50, or no more than about 100 of such most influential or important input variables among the plurality of input variables may produce a reduced but still acceptable classification accuracy. class accuracy (e.g., at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, or at least about 99%). The subset can be selected by ranking the entire plurality of input variables and selecting a predetermined number (e.g., no more than about 5, no more than about 10, no more than about 15, no more than about 20, no more than about 25, no more than about 30, no more than about 35, no more than about 40, no more than about 45, no more than about 50, or no more than about 100) of input variables with the best classification index.

[0120] The systems and methods described herein can use more than trained algorithms to determine outputs (e.g., cancer-related categories of an object). Systems and methods can include 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20 or more trained algorithms. A trained algorithm in a plurality of trained algorithms can be trained on a particular type of data (e.g., image data or tabular data). Alternatively, a trained algorithm can be trained on more than one type of data. An input to a trained algorithm can include the output of one or more other trained algorithms. In addition, a trained algorithm can receive the output of one or more trained algorithms as its input.

[0121] Identify or monitor cancer-related categories or states

[0122] After processing the dataset using the trained algorithm, cancer-associated classes can be identified or monitored in the subject. Identification can be based at least in part on quantitative or qualitative measurements of biological samples (e.g., histopathological sections of biopsy samples), proteomic data (quantitative measurements of proteins of a dataset including a set of cancer-associated proteins), and / or metabolomic data (quantitative measurements of a set of cancer-associated metabolites).

[0123] Cancer-related categories can characterize a cancer-related state of a subject. By way of non-limiting example, a cancer-related state can include whether a subject has or does not have cancer (e.g., prostate cancer), whether a subject is at risk for cancer or at a level of cancer risk (e.g., high risk, low risk), predicted long-term outcome of cancer (e.g., distant metastasis, biochemical recurrence, partial remission, complete remission, overall survival, cancer-specific survival, progression-free survival, disease-free survival, five-year survival, death), response or receptivity to therapeutic intervention, or any combination thereof.

[0124] The subject can be identified as belonging to a cancer-associated category with an accuracy of at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99% or more. The accuracy of identifying an individual's cancer-associated category by a trained algorithm can be calculated as the percentage of independent test samples (e.g., objects known to belong to a cancer-associated category or objects with a negative clinical test result corresponding to a cancer-associated category) that are correctly identified or classified as belonging to or not belonging to a cancer-associated category.

[0125] A subject can be identified as belonging to a cancer-associated category with a positive predictive value (PPV) of at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, or more. The PPV of using a trained algorithm to identify a cancer-related class can be calculated as the percentage of biological samples identified or classified as belonging to the cancer-related class that correspond to objects that truly belong to the cancer-related class.

[0126] A cancer-associated class can be identified in a subject with a negative predictive value (NPV) of at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, or more. The NPV of using a trained algorithm to identify a cancer-associated class can be calculated as the percentage of biological samples identified or classified as not having the cancer-associated class that correspond to subjects that truly do not have the cancer-associated class.

[0127] The amount of the polyurethane foam may be at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, at least about 100%, at least about 101%, at least about 102%, at least about 103%, at least about 104%, at least about 105%, at least about 106%, at least about 107%, at least about 108%, at least about 109%, at least about 110%, at least about 111%, at least about 112%, at least about 113%, at least about 114%, at least about 115%, at least about 116%, at least about The clinical sensitivity of using a trained algorithm to identify a cancer-associated category can be calculated as the percentage of independent test samples (e.g., objects known to belong to a cancer-associated category) associated with belonging to a cancer-associated category that are correctly identified or classified as belonging to the cancer-associated category.

[0128] The amount of the polyurethane foam may be at least about 5%, at least about 10%, at least about 15%, at least about 20%, at least about 25%, at least about 30%, at least about 35%, at least about 40%, at least about 50%, at least about 55%, at least about 60%, at least about 65%, at least about 70%, at least about 75%, at least about 80%, at least about 81%, at least about 82%, at least about 83%, at least about 84%, at least about 85%, at least about 86%, at least about 87%, at least about 88%, at least about 89%, at least about 90%, at least about 91%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, at least about 100%, at least about 101%, at least about 102%, at least about 103%, at least about 104%, at least about 105%, at least about 106%, at least about 107%, at least about 108%, at least about 109%, at least about 110%, at least about 111%, at least about 112%, at least about 113%, at least about 114%, at least about 115%, at least about 116%, at least about 1%, at least about 92%, at least about 93%, at least about 94%, at least about 95%, at least about 96%, at least about 97%, at least about 98%, at least about 99%, at least about 99.1%, at least about 99.2%, at least about 99.3%, at least about 99.4%, at least about 99.5%, at least about 99.6%, at least about 99.7%, at least about 99.8%, at least about 99.9%, at least about 99.99%, at least about 99.999% or more clinical specificity in identifying a cancer-associated class in a subject. The clinical specificity of using a trained algorithm to identify a cancer-associated class can be calculated as the percentage of independent test samples associated with not belonging to a cancer-associated class (e.g., subjects with a negative clinical test result for a cancer-associated class) that are correctly identified or classified as not belonging to the cancer-associated class.

[0129] After identifying cancer related categories in an object, the subtype of cancer related categories (e.g., selected from a plurality of subtypes of cancer related categories) can be further identified. The subtype of cancer related categories can be determined at least in part based on the quantitative or qualitative measurement of biological samples (e.g., histopathological sections of biopsy samples), proteomic data (quantitative measurement of proteins of a data set comprising a group of cancer associated proteins) and / or metabolome data (quantitative measurement comprising a group of cancer associated metabolites). For example, an object can be identified as being in the risk of a prostate cancer subtype (e.g., from a plurality of prostate cancer subtypes). After an object is identified as being in the risk of a prostate cancer subtype, a clinical intervention for an object can be selected based at least in part on the prostate cancer subtype identified as being in its risk. In some embodiments, a clinical intervention is selected from a plurality of clinical interventions (e.g., clinically indicated as being used for different prostate cancer subtypes).

[0130] After the subject is identified as belonging to a cancer-related category, a therapeutic intervention can be selectively provided to the subject (e.g., prescribing an appropriate course of treatment to treat the type, subtype, or state of the subject's cancer). The therapeutic intervention may include prescribing an effective dose of a drug or other therapy (e.g., radiotherapy, chemotherapy), a surgical intervention (e.g., radical prostatectomy), further testing or evaluation of the cancer-related category, further monitoring of the cancer-related category, or a combination thereof. If the subject is currently being treated for a cancer-related category with a course of treatment, the therapeutic intervention may include a subsequent different course of treatment (e.g., for increasing the therapeutic effect due to the ineffectiveness of the current course of treatment).

[0131] The therapeutic intervention may include recommending that the subject undergo a secondary clinical test to confirm the diagnosis of the cancer-related category. The secondary clinical test may include an imaging test, a blood test, a computed tomography (CT) scan, a magnetic resonance imaging (MRI) scan, an ultrasound scan, an X-ray, a positron emission tomography (PET) scan, a PET-CT scan, a bone scan, a lymph node biopsy, or any combination thereof.

[0132] Analysis of biopsy samples (e.g., analysis of microscopic images of prostate tissue), proteomic data (quantitative measurements of proteins in a dataset including a set of cancer-associated proteins), and / or metabolomic data (quantitative measurements of metabolites in a dataset including a set of cancer-associated metabolites) can be evaluated over a period of time to monitor a patient (e.g., a subject suffering from cancer or at risk of suffering from cancer or undergoing cancer treatment). In this case, the measurements of the patient's dataset can change during the course of treatment. For example, the measurements of a dataset of a patient with a reduced cancer-associated risk due to effective treatment can shift toward the profile or distribution of a healthy subject (e.g., a subject without cancer or in remission of cancer). Conversely, for example, the measurements of a dataset of a patient with an increased cancer-associated risk due to ineffective treatment can shift toward the profile or distribution of a subject with a higher cancer-associated risk or a more advanced cancer-associated category.

[0133] The cancer-related categories of the object can be monitored by monitoring the course of treatment for the cancer or cancer-related state of the object. Monitoring can include evaluating the cancer-related categories or states of the object at two or more time points. The assessment can be based at least on quantitative or qualitative measurements of biological samples (e.g., histopathological sections of biopsy samples) determined at each time point in two or more time points, proteomic data (quantitative measurements of proteins in a data set including a group of cancer-related proteins) and / or metabolomic data (quantitative measurements including a group of cancer-associated metabolites).

[0134] In some embodiments, differences in quantitative or qualitative measurements of a biological sample (e.g., a histopathological section of a biopsy sample), proteomic data (including quantitative measurements of proteins in a dataset of a set of cancer-associated proteins), and / or metabolomic data (including quantitative measurements of a set of cancer-associated metabolites) determined between two or more time points can indicate one or more clinical indications, such as (i) diagnosis of a cancer-associated condition in a subject, (ii) prognosis of a cancer-associated condition in a subject, (iii) increased risk of a cancer-associated condition in a subject, (iv) decreased risk of a cancer-associated condition in a subject, (v) effectiveness of a course of treatment for treating a cancer-associated condition in a subject, and (vi) ineffectiveness of a course of treatment for treating a cancer-associated condition in a subject.

[0135] In some embodiments, the difference of quantitative or qualitative measurement of biological samples (e.g., histopathological sections of biopsy samples), proteomic data (including quantitative measurement of proteins of a set of data sets of proteins associated with a group of cancer-related categories) and / or metabolomic data (including quantitative measurement of a set of cancer-associated metabolites) determined between two or more time points can indicate the diagnosis of the cancer-related state or category of the object. For example, if a cancer-related state is not detected in the object at an earlier time point, but a cancer-related state is detected in the object at a later time point, the difference indicates the diagnosis of the cancer-related state of the object. Clinical action or decision can be made based on this indication of the diagnosis of the cancer-related state of the object, such as, for example, prescribing a new therapeutic intervention for the object. Clinical action or decision can include suggesting that the object undergo a secondary clinical test to confirm the diagnosis of the cancer-related category. The secondary clinical test can include imaging tests, blood tests, computed tomography (CT) scans, magnetic resonance imaging (MRI) scans, ultrasound scans, X-rays, positron emission tomography (PET) scans, PET-CT scans, bone scans, lymph node biopsies, or any combination thereof.

[0136] In some embodiments, differences in quantitative or qualitative measurements of biological samples (e.g., histopathological sections of biopsy samples), proteomic data (quantitative measurements of proteins in a dataset comprising a set of cancer-associated proteins), and / or metabolomic data (including quantitative measurements of a set of cancer-associated metabolites) determined between two or more time points can indicate a prognosis for a cancer-associated category of a subject.

[0137] In some embodiments, a difference in a quantitative or qualitative measurement of a biological sample (e.g., a histopathological section of a biopsy sample), proteomic data (quantitative measurement of a protein of a data set including a set of cancer-associated proteins of associated classes), and / or metabolomic data (quantitative measurement of a set of cancer-associated metabolites) determined between two or more time points can indicate that the subject has an increased risk of a cancer-associated state. For example, if a cancer-associated state is detected in the subject at both an earlier time point and a later time point, and if the difference is a negative difference (e.g., a quantitative or qualitative measurement of a biological sample (e.g., a histopathological section of a biopsy sample), proteomic data (quantitative measurement of a protein of a data set including a set of cancer-associated proteins of associated classes), and / or metabolomic data (quantitative measurement of a set of cancer-associated metabolites) increases from an earlier time point to a later time point), then the difference can indicate that the subject has an increased risk of a cancer-associated state. A clinical action or decision can be made based on this indication of an increased risk of a cancer-associated state, such as prescribing a new therapeutic intervention or switching therapeutic interventions (e.g., ending current treatment and prescribing a new treatment) for the subject. The clinical action or decision may include recommending that the subject undergo a secondary clinical test to confirm the increased risk of the cancer-related category. The secondary clinical test may include an imaging test, a blood test, a computed tomography (CT) scan, a magnetic resonance imaging (MRI) scan, an ultrasound scan, an X-ray, a positron emission tomography (PET) scan, a PET-CT scan, a bone scan, a lymph node biopsy, or any combination thereof.

[0138] In some embodiments, a difference in a quantitative or qualitative measurement of a biological sample (e.g., a histopathological section of a biopsy sample), proteomic data (quantitative measurement of a protein of a data set including a set of cancer-associated proteins), and / or metabolomic data (quantitative measurement of a set of cancer-associated metabolites) determined between two or more time points can indicate that the subject has a reduced risk of a cancer-associated state. For example, if a cancer-associated state is detected in the subject at both an earlier time point and a later time point, and if the difference is a positive difference (e.g., a quantitative or qualitative measurement of a biological sample (e.g., a histopathological section of a biopsy sample), proteomic data (quantitative measurement of a protein of a data set including a set of cancer-associated proteins), and / or metabolomic data (quantitative measurement of a set of cancer-associated metabolites) decreases from an earlier time point to a later time point), the difference can indicate that the subject has a reduced risk of a cancer-associated state. A clinical action or decision (e.g., to continue or terminate a current therapeutic intervention) can be made based on this indication of the risk of a reduced cancer-associated state of the subject. A clinical action or decision can include advising the subject to undergo a secondary clinical test to confirm that the risk of a cancer-associated category is reduced. The secondary clinical test may include an imaging test, a blood test, a computed tomography (CT) scan, a magnetic resonance imaging (MRI) scan, an ultrasound scan, an X-ray, a positron emission tomography (PET) scan, a PET-CT scan, a bone scan, a lymph node biopsy, or any combination thereof.

[0139] In some embodiments, the difference in quantitative or qualitative measurements of biological samples (e.g., histopathological sections of biopsy samples), proteomic data (including quantitative measurements of proteins of a set of cancer-associated proteins) and / or metabolomic data (including quantitative measurements of a set of cancer-associated metabolites) determined between two or more time points can indicate the effectiveness of the course of treatment for the cancer-related state of the subject. For example, if a cancer-related state is detected in the subject at an earlier time point, but not detected in the subject at a later time point, the difference can indicate the effectiveness of the course of treatment for the cancer-related state of the subject. Clinical actions or decisions can be made based on this indication of the effectiveness of the course of treatment for the cancer-related state of the subject, such as continuing or ending the current therapeutic intervention for the subject. The clinical action or decision can include recommending that the subject undergo a secondary clinical test to confirm the effectiveness of the course of treatment for the cancer-related category. The secondary clinical test can include an imaging test, a blood test, a computed tomography (CT) scan, a magnetic resonance imaging (MRI) scan, an ultrasound scan, an X-ray, a positron emission tomography (PET) scan, a PET-CT scan, a bone scan, a lymph node biopsy, or any combination thereof.

[0140] In some embodiments, a difference in a quantitative or qualitative measurement of a biological sample (e.g., a histopathological section of a biopsy sample), proteomic data (quantitative measurement of a protein of a dataset comprising a set of cancer-associated proteins), and / or metabolomic data (quantitative measurement of a set of cancer-associated metabolites) determined between two or more time points can indicate the ineffectiveness of a course of treatment for a cancer-associated class of a subject. For example, if a cancer-associated state is detected in the subject at both an earlier time point and a later time point, and if the difference is a negative difference or a zero difference (e.g., a quantitative or qualitative measurement of a biological sample (e.g., a histopathological section of a biopsy sample), proteomic data (quantitative measurement of a protein of a dataset comprising a set of cancer-associated proteins), and / or metabolomic data (quantitative measurement of a set of cancer-associated metabolites) increases or remains at a constant level from an earlier time point to a later time point), and if effective treatment is indicated at the earlier time point, the difference can indicate the ineffectiveness of a course of treatment for a cancer-associated state of a subject. A clinical action or decision can be made based on this indication of the ineffectiveness of the course of treatment for treating the cancer-related condition of the subject, such as terminating the subject's current therapeutic intervention and / or switching (e.g., prescribing) a different new therapeutic intervention for the subject. The clinical action or decision can include recommending that the subject undergo a secondary clinical test to confirm the ineffectiveness of the course of treatment for treating the cancer-related condition. The secondary clinical test can include an imaging test, a blood test, a computed tomography (CT) scan, a magnetic resonance imaging (MRI) scan, an ultrasound scan, an X-ray, a positron emission tomography (PET) scan, a PET-CT scan, a bone scan, a lymph node biopsy, or any combination thereof.

[0141] Output reports on cancer-related status

[0142] After identifying a cancer-related state in a subject or monitoring an increased risk of a cancer-related state, a report indicating (e.g., identifying or providing an indication of) the cancer-related state of the subject can be output electronically. The subject may not display a cancer-related state (e.g., no symptoms of a cancer-related state (such as the presence or risk of prostate cancer)). The report can be presented on a graphical user interface (GUI) of the user's electronic device. The user can be a subject, a caregiver, a doctor, a nurse, or another healthcare worker.

[0143] The report may include one or more clinical indications, such as (i) diagnosis of the subject's cancer-related state, (ii) prognosis of the subject's cancer-related class, (iii) risk of an increased cancer-related class for the subject, (iv) risk of a reduced cancer-related class for the subject, (v) effectiveness of a course of treatment for treating the subject's cancer-related class, (vi) ineffectiveness of a course of treatment for treating the subject's cancer-related class, and (vii) long-term outcomes for the cancer-related class. The report may include one or more clinical actions or decisions made based on these one or more clinical indications. Such clinical actions or decisions may be directed to therapeutic intervention or further clinical evaluation or testing of the subject's cancer-related state.

[0144] For example, the clinical indication of the diagnosis of the cancer-related state of the object can be accompanied by the clinical action of prescribing a new therapeutic intervention for the object. As another example, the clinical indication of the risk of the increased cancer-related state of the object can be accompanied by the clinical action of prescribing a new therapeutic intervention or converting the therapeutic intervention (e.g., ending the current treatment and prescribing a new treatment) for the object. As another example, the clinical indication of the risk of the reduced cancer-related state of the object can be accompanied by the clinical action of continuing or ending the current therapeutic intervention for the object. As another example, the clinical indication of the effectiveness of the cancer-related state of the treatment course for the treatment of the object can be accompanied by the clinical action of continuing or ending the current therapeutic intervention for the object. As another example, the clinical indication of the invalidity of the cancer-related state of the treatment course for the treatment of the object can be accompanied by the clinical action of ending the current therapeutic intervention and / or converting to (e.g., prescribing) a different new therapeutic intervention.

[0145] In some embodiments, therapeutic intervention may include radiation therapy (RT), chemotherapy, surgical intervention (e.g., radical prostatectomy), further testing or evaluation of cancer-related categories, further monitoring of cancer-related categories, or a combination thereof. If the subject is currently receiving treatment for a cancer-related category, the therapeutic intervention may include subsequent different courses of treatment (e.g., for increasing the therapeutic effect due to the ineffectiveness of the current course of treatment). The therapeutic intervention may include advising the subject to undergo secondary clinical testing to confirm the diagnosis of the cancer-related category. The secondary clinical test may include imaging tests, blood tests, computed tomography (CT) scans, magnetic resonance imaging (MRI) scans, ultrasound scans, X-rays, positron emission tomography (PET) scans, PET-CT scans, bone scans, lymph node biopsies, or any combination thereof.

[0146] Computer Systems

[0147] The present disclosure provides a computer system programmed to implement the method of the present disclosure. Figure 1A computer system 101 is shown that is programmed or otherwise configured to, for example, (i) train and test a trained algorithm, (ii) process images and / or tabular data using the trained algorithm to determine a cancer-related category or a cancer-related state for a subject, (iii) assess a subject's cancer based on the classified category, (iv) identify or monitor a subject's cancer-related category or state, and (v) electronically output a report indicating a subject's cancer-related category or state.

[0148] The computer system 101 can regulate various aspects of the analysis, calculation, and generation of the present disclosure, such as, for example, (i) training and testing a trained algorithm, (ii) processing images and / or tabular data using a trained algorithm to determine a cancer-related category or cancer-related state of a subject, (iii) assessing a subject's cancer based on the classified category, (iv) identifying or monitoring a subject's cancer-related category or state, and (v) electronically outputting a report indicating a subject's cancer-related category or state. The computer system 101 can be a user's electronic device or a computer system remotely located relative to the electronic device. The electronic device can be a mobile electronic device.

[0149] Computer system 101 includes a central processing unit (CPU, also referred to herein as a "processor" and "computer processor") 105, which may be a single-core or multi-core processor or multiple processors for parallel processing. Computer system 101 also includes memory or memory location 110 (e.g., random access memory, read-only memory, flash memory), an electronic storage unit 115 (e.g., a hard disk), a communication interface 120 (e.g., a network adapter) for communicating with one or more other systems, and peripherals 125 (such as cache, other memory, data storage, and / or an electronic display adapter). Memory 110, storage unit 115, interface 120, and peripherals 125 communicate with CPU 105 via a communication bus (solid lines) such as a motherboard. Storage unit 115 may be a data storage unit (or data repository) for storing data. Computer system 101 may be operably coupled to a computer network ("network") 130 by means of communication interface 120. The network 130 may be the Internet, the internet and / or an extranet, or an intranet and / or an extranet in communication with the Internet.

[0150] In some embodiments, network 130 is a telecommunications and / or data network. Network 130 may include one or more computer servers that may support distributed computing such as cloud computing. For example, one or more computer servers may support cloud computing on network 130 ("cloud") to perform various aspects of analysis, calculation, and generation of the present disclosure, such as, for example, (i) training and testing trained algorithms, (ii) processing data using trained algorithms to determine cancer-related categories of objects, (iii) determining quantitative measurements indicating cancer-related categories of objects, (iv) identifying or monitoring cancer-related categories of objects, and (v) electronically outputting reports indicating cancer-related categories of objects. Such cloud computing may be provided by cloud computing platforms such as, for example, Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform, and IBM cloud. In some embodiments, network 130 may implement a peer-to-peer network with the aid of computer system 101, which may enable devices coupled to computer system 101 to act as clients or servers.

[0151] The CPU 105 may include one or more computer processors and / or one or more graphics processing units (GPUs). The CPU 105 may execute a sequence of machine-readable instructions that may be embodied in a program or software. The instructions may be stored in a memory location such as the memory 110. The instructions may be directed to the CPU 105, which may then program or otherwise configure the CPU 105 to implement the methods of the present disclosure. Examples of operations performed by the CPU 105 may include fetching, decoding, executing, and writing back.

[0152] CPU 105 may be part of a circuit, such as an integrated circuit. One or more other components of system 101 may be included in the circuit. In some cases, the circuit is an application-specific integrated circuit (ASIC).

[0153] Storage unit 115 may store files such as drivers, libraries, and saved programs. Storage unit 115 may store user data, such as user preferences and user programs. In some embodiments, computer system 101 may include one or more additional data storage units that are external to computer system 101, such as located on a remote server that communicates with computer system 101 via an intranet or the Internet.

[0154] Computer system 101 may communicate with one or more remote computer systems via network 130. For example, computer system 101 may communicate with a user's remote computer system. Examples of remote computer systems include personal computers (e.g., portable PCs), tablet computers or tablet PCs (e.g., iPad, Galaxy Tab), phones, smartphones (e.g. iPhone, Android-enabled devices, ) or a personal digital assistant. A user can access the computer system 101 via the network 130.

[0155] The methods described herein may be implemented by machine (e.g., computer processor) executable code stored in an electronic storage location of the computer system 101, such as, for example, the memory 110 or the electronic storage unit 115. The machine executable code or machine readable code may be provided in the form of software. During use, the code may be executed by the processor 105. In some embodiments, the code may be retrieved from the storage unit 115 and stored on the memory 110 for access by the processor 105 at any time. In some cases, the electronic storage unit 115 may not be included, and the machine executable instructions are stored on the memory 110.

[0156] The code may be precompiled and configured for use by a machine having a processor suitable for executing the code, or may be compiled during runtime. The code may be provided in a programming language that may be selected to enable the code to be executed in a precompiled or as-compiled manner.

[0157] Embodiments of the systems and methods provided herein, such as computer system 101, can be embodied in programming. Various aspects of the technology can be considered as "products" or "articles", which are generally in the form of machine (or processor) executable code and / or related data carried or embodied on a type of machine-readable medium. The machine executable code can be stored in an electronic storage unit, such as a memory (e.g., read-only memory, random access memory, flash memory) or a hard disk. "Storage" type media can include any or all tangible memories, processors, etc. of a computer, or their associated modules, such as various semiconductor memories, tape drives, or disk drives, which can provide non-temporary storage for software programming at any time. All or part of the software can sometimes communicate through the Internet or various other telecommunications networks. For example, such communication can enable software to be loaded from one computer or processor to another computer or processor, such as from a management server or host to a computer platform of an application server. Therefore, another type of medium that can carry software elements includes light waves, radio waves, and electromagnetic waves, such as across physical interfaces between local devices, through wired and optical landline networks, and through various air links. Physical elements that carry such waves, such as wired or wireless links, optical links, etc., can also be considered as media that carry the software. As used herein, unless limited to non-transitory, tangible "storage" media, terms such as computer or machine "readable media" refer to any medium that participates in providing instructions to a processor for execution.

[0158] Thus, machine-readable media such as computer executable code may take a variety of forms, including tangible storage media, carrier media, or physical transmission media. Non-volatile storage media include, for example, optical or magnetic disks, such as any storage device in any computer, such as may be used to implement a database such as shown in the accompanying drawings. Volatile storage media include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media include coaxial cables, copper wire, and optical fiber, including the wires that make up a bus within a computer system. Carrier transmission media may take the form of electrical or electromagnetic signals, or acoustic or light waves, such as those generated during radio frequency (RF) and infrared (IR) data communications. Thus, common forms of computer readable media include, for example: a floppy disk, a flexible disk, a hard disk, a magnetic tape, any other magnetic medium, a CD-ROM, a DVD or DVD-ROM, any other optical medium, a punched card stock tape, any other physical storage medium having a pattern of holes, a RAM, a ROM, a PROM and an EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave that transmits data or instructions, a cable or link that transmits such a carrier wave, or any other medium from which a computer can read programming code and / or data. Many of these forms of computer readable media may participate in carrying one or more sequences of one or more instructions to a processor for execution.

[0159] The computer system 101 may include or communicate with an electronic display 135, the electronic display 135 including a user interface (UI) 140 for providing, for example, (i) a visual display indicating training and testing of a trained algorithm, (ii) a visual display of data indicating a cancer-related category for a subject, (iii) a quantitative measurement of a cancer-related category for a subject, (iv) an identification that a subject has a cancer-related category, or (v) an electronic report indicating a cancer-related category for a subject. Examples of UIs include, but are not limited to, graphical user interfaces (GUIs) and web-based user interfaces.

[0160] The methods and systems of the present disclosure may be implemented by one or more algorithms. The algorithms may be implemented by software when executed by the central processing unit 205. The algorithms may, for example, (i) train and test the trained algorithms, (ii) process images and / or tabular data using the trained algorithms to determine a cancer-related category or cancer-related state of a subject, (iii) assess a subject's cancer based on the classified category, (iv) identify or monitor a subject's cancer-related category or state, and (v) electronically output a report indicating a subject's cancer-related category or state.

[0161] Example

[0162] Example 1: Personalization of prostate cancer therapy through multimodal deep learning

[0163] The methods and systems disclosed herein demonstrate the personalization of prostate cancer therapy by using a novel multimodal deep learning model trained on digital histopathology and clinical data of prostate biopsy to predict long-term clinically relevant outcomes (distant metastasis, biochemical recurrence, death caused by prostate cancer, and overall survival). The example system disclosed herein includes a trained algorithm that is trained and validated using data sets from five phase III randomized multinational trials running at hundreds of clinical sites. Clinical and histopathological data were available for 5,654 patients (71.1%) of 7,957 patients, which produced 16.1 terabytes of histopathological images, with subjects followed up for 10-20 years. Compared with the most commonly used risk stratification tool (National Cancer Center Network (NCCN) risk group), the deep learning model has excellent prognostic and discriminant performance in all test results. This artificial intelligence system can allow oncologists to perform computational modeling of the most likely outcome for any particular patient, thereby determining its best treatment. As long as it is equipped with a digital histopathology scanner and Internet access, any clinic can provide such capabilities, thereby enabling the universal acquisition of important therapy personalization at a low cost.

[0164] NCCN risk groups are based on international risk stratification standards, which were formulated in the late 1990s and are called D'Amico risk groups. The system is based on rectal examination, serum prostate-specific antigen (PSA) measurement and tumor grade assessed by histopathology. This three-tier system continues to form the backbone of treatment recommendations around the world, but has suboptimal prognostic and discriminative performance in risk stratification of patients. This is partly due to the highly subjective and nonspecific nature of the core variables in these models. For example, Gleason grading was formulated in the 1960s and is still highly subjective, and even among professional urological pathologists, the repeatability between observers is unacceptable. Recently, tissue-based genomic biomarkers have shown improved prognostic performance. However, almost all of these tests lack validation in prospective randomized clinical trials of the intended use population, and due to cost and processing time, there is little adoption internationally. Therefore, there is still a serious unmet clinical need for improved tools for personalized therapy of prostate cancer.

[0165] Artificial intelligence (AI) has demonstrated remarkable capabilities in many use cases in medicine, from physician-level diagnostics to workflow optimization, and has the potential to support cancer therapy as digital histopathology continues to be adopted in the clinic. AI has begun to make progress in histopathology-based prognosis, for example by predicting short-term patient outcomes or by improving the accuracy of Gleason-based cancer grading on postoperative surgical samples. While standard risk stratification tools are fixed and based on a small number of variables, AI can learn from large amounts of minimally processed data across a variety of modalities. Compared to genomic biomarkers, AI systems are low-cost, highly scalable, and improve incrementally with use. In addition, a key challenge for any biomarker is having the best data to train and validate the relevant endpoints, and some commercial prognostic biomarkers in oncology may have been trained on retrospective convenience sampling.

[0166] The methods and systems disclosed herein may include a multimodal artificial intelligence (MMAI) system that can effectively overcome the unmet need for outcome prognosis in localized prostate cancer, thereby creating a generalizable biomarker with global adoption potential. Five Phase III randomized clinical trials of prognostic biomarkers for localized prostate cancer were used to train the algorithm described herein by utilizing multimodal deep learning on digital tissue pathology.

[0167] A unique dataset was generated from five large, multinational, randomized, phase III clinical trials (NRG / RTOG 9202, 9408, 9413, 9910, and 0126) in men with localized prostate cancer. All patients received definitive radiation therapy (RT) with a prespecified use of short-term androgen deprivation therapy (ST-ADT), long-term ADT (LT-ADT), and / or dose-escalated RT (DE-RT) ( Figure 2C Of the 7,957 patients enrolled in the five trials, 7,752 had complete baseline clinical data, and 5,654 had complete baseline and digital histopathology data. This represented 16.1 TB of histopathology images from histopathology slides of 16,204 pre-treatment biopsy specimens.

[0168] The MMAI architecture can ingest both tabular (clinical) data and image-based (histopathology) data, making it particularly well suited for randomized clinical trial data. The complete architecture is shown in Figure 2AEach patient in the dataset was represented by clinical variables (including laboratory and pathology data, treatment interventions, and long-term outcomes) and digitized histopathology slides (a median of 3.5 slides). Joint learning across the two data streams was complex and involved building three separate deep learning pipelines—one for images, one for tabular data, and a third to join them together. The data were normalized across trials for consistency.

[0169] Efficiently learning relevant features from a varying number of digitized histopathology slides involves several preprocessing steps to standardize the images, followed by self-supervised training. For each patient, all tissue sections in their biopsy slides are segmented and combined into a single large image with fixed width and height across all patients, a process called image stitching ( Figure 5 ). An H×W grid is overlaid on the image mosaic, which cuts the image mosaic into patches of size 256×256 pixels across its RGB channels. These patches are then used to train a self-supervised (SSL) model to learn histopathological features useful for downstream AI tasks. Figure 2B This part of the pipeline is shown. Once trained, the SSL model can take patches of the image mosaic and output a 128-dimensional vector representation for each patch. All of these vectors are concatenated in the same spatial orientation as the original patch, producing a H×W×128 tensor (feature concatenation) that compresses the initially huge image mosaic into a compact representation useful for further downstream learning.

[0170] SSL is a method that can be used to learn from unannotated datasets. Typical ML settings utilize supervised learning, where a dataset consists of data points (e.g., images) and data labels (e.g., target classes). In contrast, during SSL, synthetic data labels are extracted from the original data and used to train a general feature representation that can be used for downstream tasks. Momentum contrast (a technique that takes a set of image patches, generates augmented copies of each patch, and then trains a model to predict whether any two augmented copies are from the same original patch) can effectively learn features for medical tasks. The structural setting is shown in Figure 2B , further details of which are described elsewhere herein.

[0171] In order to direct the SSL process to regions of the block that are likely to be more clinically useful, the blocks in the dataset were oversampled based on kernel density. Using an object detection model trained to detect nuclei, the number of nuclei in each block can be approximated. The blocks were divided into deciles based on this count, and each decile was oversampled so that the net number of images seen during one training epoch was the same for each decile. Example images are shown in Figure 6 middle.

[0172] The system described in this paper can learn from patient-level annotations, while histopathology slides are not annotated. In addition, self-supervised learning of the image model allows it to learn from new image data without any annotations.

[0173] Learning from tabular data involves two steps. First, clinical data is normalized across trials and used to pre-train a TabNet architecture via self-supervision by masking parts of the data and training the model to learn them. Each patient’s data is then fed through the TabNet to extract a feature vector, which is then concatenated with the output of the image pipeline. The concatenated vector is then fed through further neural layers, and the model outputs a binary result for the task at hand.

[0174] The internal data representation of the SSL model is shown in Figure 4 In . Image patches of the entire dataset are fed through the SSL model, and model features (128-dimensional vectors output by the model) are extracted for each patch. The Uniform Manifold Approximation and Projection Algorithm (UMAP) is then applied to these features, projecting them from 128 dimensions to 2 dimensions and plotting each patch as a separate point. Adjacent data points represent image patches that the model considers similar. UMAP groups the feature vectors into 25 clusters, some of which are displayed in various colors. The inset shows an example image patch that is close to the cluster centroid in feature space. The 20 image patches closest to the cluster centroid are then interpreted by a pathologist. Example interpretations are shown in Figure 4 and a full explanation is shown in Figure 7 Despite never being trained with clinical annotations, the SSL model learns human-interpretable image features that indicate complex aspects of cancer, such as Gleason grade or tissue type.

[0175] Six different MMAI models were trained and tested across four endpoints (DM, BCR, PCaSS, OS) and two time frames (5 years and 10 years). Taking into account competing events, the performance of these models was measured using the area under the time-dependent receiver operating characteristic curve (AUC) of sensitivity and specificity. Sensitivity is defined as the ratio of correct positive predictions to displayed positive events (sensitivity = predicted positive / positive number), and specificity is defined as the ratio of correct negative predictions to displayed negative events (specificity = predicted negative / negative number). For this indicator, 0.5 represents random chance accuracy, and 1.0 represents perfect accuracy.

[0176] The NCCN model was used as a baseline comparator, e.g. Figure 8 Three variables (Gleason score, tumor t stage, and baseline PSA) were used to stratify patients into low-risk, intermediate-risk, and high-risk groups.

[0177] The results are shown in Figure 3A-3H A separate model was trained for each outcome and time point. Figure 3A 2D-2H , the blue bars represent the performance of the MMAI model trained on a specific task, and the gray bars represent the performance of the corresponding NCCN model. Figure 2B The relative improvement of MMAI over NCCN is shown across outcomes and across subsets of test sets from five trials. The MMAI model consistently outperformed the NCCN model across all test outcomes. The relative improvement in AUC ranged from 11.45% to 19.72%. In addition, the trial subsets consistently showed relative improvement over NCCN.

[0178] To evaluate the effect of various data components specific to the MMAI model, ablation studies were performed. Additional MMAI models were trained using the following data settings: NCCN variables only, pathology images only, pathology images + NCCN variables (combined Gleason score, t-stage, baseline PSA), and pathology images + NCCN variables + 3 additional variables used in the model (age, Gleason primary, Gleason secondary). Each additional data component improved performance, with the full setting (pathology, 6 clinical variables) producing the best results ( Figure 2C ).

[0179] The MMAI system substantially outperformed the NCCN risk stratification tool encoded as a model in predicting four key future outcomes for patients (distant metastasis, biochemical recurrence, prostate cancer-specific survival, and overall survival). By creating a deep learning architecture that simultaneously ingests multiple (variable-size) data types from patients as well as clinical data, a deep learning system was built that was able to infer long-term outcomes for patients with substantially higher accuracy than established clinical models.

[0180] The methods and systems described herein can leverage robust and large-scale clinical data from five different prospective, randomized, multinational trials with 10-20 years of patient follow-up for 5,654 patients across diverse populations. The validation of these prognostic classifiers on a large set of clinical trial data (in the intended use population) uniquely positions these tools as an aid to treatment decision making. A key drawback of similar genomics-based assays is their high cost and long turnaround time; AI tools do not have these limitations, which substantially lowers the barriers to large-scale international adoption. Nearly 60% of the world has access to the internet, but only approximately 4% (the U.S. population) can easily access genomics-based assays. The increasing availability of digital histopathology, coupled with internet connectivity, can support the global distribution of AI-based prognostic and predictive tests, enabling low-cost access to critical therapy personalization.

[0181] method

[0182] Tabular pipeline. Tabular clinical data are divided into numerical variables and categorical variables. Numerical variables are whitened (mean subtraction + max normalization) to the range [-1, 1]. Categorical variables are treated as one-hot vectors, which are embedded into 2-3 dimensional vectors according to traditional word-to-vector techniques, and their dimensionality is given by the formula D = Round (1.6 numcategories 0.56 ). The TabNet model with concatenation of categorical and numerical variables as input is used (parameters: learning rate 0.2, Adam optimizer with step learning rate scheduler, batch size 1024, maximum 50 epochs, early stopping patience 10 epochs).

[0183] Image pipeline. ResNet50 model is used for training with MoCo-v2 training protocol (parameters: learning rate = 0.03 with cosine learning rate schedule, moco-t = 0.2, multi-layer perceptron head, batch size of 256, MoCo-v2 data augmentation, 200 epochs) Figure 2B The SSL model used in the system architecture. Figure 3A For each held-out test set in , the corresponding SSL model is pre-trained using only images from the training data. Some image patches are oversampled using kernel density sampling as described elsewhere in this paper. Once SSL pre-training is completed, all W×H patches are fed into an SSL pre-trained ResNet50 model to produce a W×H×128 feature concatenation for each image concatenation. The final image model used for prediction is a 2-layer CNN model with batch normalization and dropout that takes the feature tensor as input. The final CNN model is trained with a batch size of 32, 150 max epochs, and an Adam optimizer with a learning rate of 0.01 and a step learning rate scheduler.

[0184] Downstream pipeline. A joint fusion approach is used to leverage information from both modalities (image and tabular features). Images are featurized into feature tensors and fed into the final image model to produce a feature vector, while tabular features are fed into the TabNet model separately to generate another feature vector. Two fully connected layers process the concatenated feature vectors of each pipeline and output predicted probabilities. For patients with missing histopathology data, the image-based feature vectors are zeroed before concatenation.

[0185] Dataset preparation. Complete patient-level baseline clinical data, digital histopathology slides of prostate biopsies, and longitudinal outcomes from five landmark, large-scale, prospective, randomized, international clinical trials were used, which included 5,654 patients, 16,204 histopathology slides, and a median patient follow-up of 10–20 years. These trials were RTOG 9202, 9408, 9910, 0126, and 9413 ( Figure 2C ). The trials randomized to various combinations of radiation therapy (RT) and androgen deprivation therapy (ADT): RT + short-term ADT (RTS), RT + intermediate-term ADT (RTM), RT + long-term ADT (RTL), and dose and volume levels of RT (RT+). Slides were digitized by NRG Oncology using a Leica Biosystems Aperio AT2 digital pathology scanner at 20x resolution over a two-year period. Histopathology images were manually inspected to ensure quality and clarity. Prior to model training, digital slides were converted to a single image mosaic of 200×200 blocks for each unique patient. Each clinical trial collected slightly different clinical variables. Six clinical variables available across all trials (combined Gleason, Gleason primary, Gleason secondary, t-stage, baseline PSA, age) as well as digital histopathology were used for model training and validation.

[0186] Tissue segmentation. After cutting the sections into 256×256 pixel blocks at 10x magnification, an artifact classifier was developed by training ResNet-18 to classify whether the block showed usable tissue or whether it showed blanks or artifacts. The artifact classifier was trained for 25 epochs and optimized using SGD with a learning rate of 0.001. The learning rate was reduced by 10% every 7 epochs. 3661 blocks (tissue vs. non-tissue) were manually annotated and the classifier was trained on 3366 of them, achieving a validation accuracy of 97.6% on the remaining 295 blocks. The artifact classifier was then used to segment tissue sections during image mosaic formation.

[0187] Nuclear density sampling. Due to significant variations in staining intensity and stain degradation, off-the-shelf pre-trained models for nuclear detection and segmentation were unable to accurately detect nuclei in most of our sections. To address this issue, a nuclear detector was trained using the YOLOv5 (github.com / ultralytics / yolov5) object detection method.

[0188] To train the YOLOv5 model, a representative sample of 34 carefully selected slices was manually labeled using the QuPath image analysis platform. First, the tissue was segmented using the "Simple Tissue Detection" module. Next, the cells were segmented using the "Watershed Cell Detection" module and the parameters were manually adjusted for each slice. The YOLOv5-Large model was then trained on annotations from 29 of the slices and evaluated on the remaining 5 slices. The model was trained using 256×256 blocks at 10x magnification.

[0189] Model performance metric. (AUC) For each model and each outcome, the time-dependent receiver operating characteristic (ROC) was estimated using the R package timeROC, taking into account competing events. This was done by sweeping a threshold t in the interval [0,1] and defining the model’s prediction as The time-dependent sensitivity and specificity curves are calculated, where P is the probability of the outcome output by the model. The area under this curve defines the performance of the model on the current task.

[0190] NCCN model. Three clinical variables (Gleason, t stage, and baseline PSA) are used to categorize patients into low-risk, intermediate-risk, and high-risk groups. Figure 8 The algorithm in encodes the NCCN model.

[0191] Example 2: Evaluating the Algorithmic Fairness of the MMAI Algorithm

[0192] In this example, algorithmic fairness or the performance of a multimodal AI (MMAI) model utilizing clinical and digital histopathology data of AA and non-AA prostate cancer patients treated in the NRG / RTOG prostate cancer trial is described.

[0193] Prostate cancer (PCa) is the second leading cause of cancer-related death in men, and it is well documented that African American (AA) men have an increased burden due to later onset and younger age of diagnosis.

[0194] There are limitations associated with using population-based datasets and retrospective studies to investigate outcomes and associated differences among AA men with PCa, as these data often underrepresent the AA men within the cohort and lack the long-term follow-up needed to develop prognostic risk models. Ideally, studies that include large numbers of AA men enrolled in prospective randomized controlled trials (RCTs) would allow for the use of a representative sample of AA to assess long-term outcomes while minimizing the risk of selection bias and other co-founders. The Radiation Oncology Group (RTOG) and the NRG Oncology Cooperative have prioritized the recruitment of a representative proportion of AA to prostate cancer trials. Large RCTs provide an opportunity to model long-term outcomes and risk differences, which could allow for more nuanced clinical decisions regarding treatment selection and optimization of therapy for AA men with PCa.

[0195] method

[0196] Description of multi-group data

[0197] A unique dataset was collected from five large, multinational, randomized, Phase III clinical trials (NRG / RTOG-9202, 9408, 9413, 9910, and 0126) in men with localized prostate cancer, with permission from NRG Oncology, a National Clinical Trials Network (NCTN) group funded by the National Cancer Institute (NCI). All patients received definitive external radiation therapy (RT) with or without prespecified androgen deprivation therapy (ADT). The duration of combined RT with short-term ADT was 4 months, the duration of intermediate-term ADT was 36 weeks, and the duration of long-term ADT was 28 months. A total of 7,752 eligible participants were randomized to the five trials.

[0198] Description of the Multimodal AI (MMAI) Model

[0199] Four MMAI models described herein were trained and deployed on the dataset. The MMAI models jointly learned the relevant features of the digital histopathology sections and clinical data from each patient. Through self-supervised pre-training, image vector representations are learned and extracted from the tissue sections in the biopsy sections. The combination of these image feature vectors and feature vectors derived from clinical data is fed into the multimodal fusion pipeline to output the risk scores of the required clinical endpoints, including distant metastasis (DM) and prostate cancer-specific mortality (PCSM). The group is divided into 80 / 20 development and validation datasets, where the MMAI models are trained and optimized on the development set and subsequently validated on the remaining validation set. The first MMAI for predicting the risk of DM and PCSM is as described in Example 1. The second MMAI model for predicting the risk of DM and PCSM includes multimodal learning of a neural network based on multi-instance learning, which has an attention mechanism that uses the time when the required clinical endpoint event occurs as a label, as described in more detail below. Fig. 9 A schematic overview of the second set of MMAI models is shown. Comparison of findings based on the MMAI model as described in Example 1 is also discussed below.

[0200] Methods for Multimodal Deep Learning Model Development

[0201] Model Development Overview

[0202] The five trials were stratified by 1) trial, 2) distant metastasis status, and 3) patient clinical risk and randomly divided into a development set (80%) and a validation set (20%) for model development and validation, respectively. Each MMAI model was trained and optimized on the development set using a 5-fold cross-validation scheme, where the development set was further divided into a training subset and a conditioning subset in each fold. The training subset was used to update the learnable model parameters, while the conditioning subset was used to monitor unbiased performance during training and adjust hyperparameters. Since this training process generated five independent models, an integrated model was constructed by averaging the five model outputs to form a single risk score for each patient.

[0203] Clinical data preprocessing

[0204] Clinical variables (T stage, Gleason score, and primary / secondary Gleason pattern) were treated as numerical variables and standardized based on the mean and standard deviation of the training data. Any missing clinical data were imputed using the k-nearest neighbor method, where missing values ​​were imputed using the mean of the five nearest neighbors found in the training set.

[0205] Image feature extraction model development

[0206] Effectively learning relevant features from a variable number of digitized histopathology sections involves both image normalization and self-supervised pre-training. For each patient, all pre-treatment biopsy tissue sections from their biopsy sections were segmented and divided into blocks of size 256×256 pixels across each RGB channel. A tissue classifier was developed by training ResNet-18 to classify whether a block shows usable tissue or whether the block shows blanks or artifacts. The artifact classifier was trained for 25 epochs and optimized using stochastic gradient descent with a learning rate of 0.001. The learning rate was reduced by 10% every 7 epochs. 3661 blocks (tissue vs. non-tissue) were manually annotated and the classifier was trained on 3366 of them, achieving a validation accuracy of 97.6% on the remaining blocks. The artifact classifier was then used to segment tissue sections and filter out low-quality images during image feature generation.

[0207] The blocks filtered by the artifact classifier were then used to train a self-supervised learning model to learn tissue morphological features useful for downstream tasks. The ResNet-50 model was used with the MoCo-v2 training protocol (parameters: learning rate = 0.03, cosine learning rate schedule of 200 epochs, moco-t = 0.2, multi-layer perceptron head, batch size of 256, default MoCo-v2 parameters for enhancement) to train the self-supervised learning model. The corresponding self-supervised learning model was pre-trained using patient images with Gleason primary ≥ 4 to effectively learn relevant tissue morphological features. Once the self-supervised pre-training was completed, all blocks with available tissue of the full-slice image were fed into the self-supervised pre-trained ResNet-50 model to generate a 128-dimensional vector representation for each block.

[0208] Downstream multimodal prognostic model development

[0209] The downstream prognostic model takes as input for each patient an image feature tensor, a concatenation of feature vectors from all patches for each patient, and preprocessed clinical data. In the second model, an attention multi-instance learning network is employed to learn the weights of each image feature vector from each patch. A single 128-dimensional image vector is generated from the image feature tensor for each patient by taking the weighted sum of the image vectors from all patches from the same patient, where the weights are learned by the attention mechanism. The preprocessed clinical data are all treated as numerical variables and processed through a single linear layer to learn a 6-dimensional clinical vector representation. The concatenation of the 128-dimensional image vector and the 6-dimensional clinical vector are further processed through a neural network-based joint fusion pipeline to efficiently learn from clinical and image data to output a risk score for the outcome of interest ( Fig. 9 ).

[0210] The negative log partial likelihood is used as the training objective, where the model prediction score is the estimated relative log hazard. The binary indicators of the events of interest and the corresponding event occurrence times are used as labels for model development. The negative log partial likelihood loss is parameterized by the model weights θ and is formulated as follows:

[0211]

[0212] Among them, the value T i 、E i and x i are the event time or the time of the last follow-up, the indicator variable for whether the event was observed, and the model input for the i-th observation. Function f θ represents the fact branch of the multimodal model, and f θ (x) is the estimated relative risk for a given input x. The value N E＝1 represents the number of patients with observable events. The set of patients with observable events is denoted as E i = 1. Risk set is the set of patients who are still at risk of failure at time t. Breslow’s approximation is used to handle bound event times.

[0213] Model performance metrics in subgroups

[0214] Distant metastasis (DM), prostate cancer-specific mortality (PCSM), biochemical failure (BF), and overall survival (OS) were assessed. DM and PCSM were chosen because they are closely associated with prostate cancer incidence and mortality and may represent a more clinically applicable measure to reflect the broader burden of prostate cancer in the AA population compared with the non-AA population.

[0215] For each MMAI model, the model performance of DM MMAI relative to DM endpoints and PCSM MMAI relative to PCSM endpoints between AA and non-AA PCa patients were evaluated. Patients with unknown or missing racial identity were excluded from the analysis group. All endpoints evaluated were the results of the time of the event, in which the lost patients were censored, and the death before the event of interest was considered as a competing event. The racial subgroup analysis was performed by comparing the distribution of clinical variables and MMAI scores (median and interquartile range (IQR) of continuous variables and the proportion of reported categorical variables) and evaluating the prognostic ability of the MMAI model in AA and non-AA men. Both MMAI continuous scores (each increase of 0.05 points) and classified risk groups were used to evaluate the fairness of the algorithm. For the MMAI classification group, the model scores were sorted by decile, and then decomposed into three groups by dividing the decile with similar prognosis based on the corresponding endpoints originally trained by the MMAI model. For example, the DM MMAI model was grouped into the 1st-4th, 5th-9th, and 10th deciles, and the PCSM MMAI model was grouped into the 1st-5th, 6th-9th, and 10th deciles. Using DM and PCSM as the primary endpoints, and BF, OS as secondary endpoints, the performance of the models was compared using Fine-Gray or Cox proportional hazard models. Kaplan-Meier or cumulative incidence estimates were calculated and compared using log-rank or Gray tests. The Bonferroni method was then used to adjust the p-value post hoc for pairwise cumulative incidence comparisons between subgroups.

[0216] result

[0217] Fig.10 A summary of eligible clinical trial participants is depicted. There were 948 (16.6%) AA patients, 4,731 non-AA patients (82.9%), and 29 (0.5%) patients whose ethnic status was unknown or missing and who were excluded from all analyses.

[0218] In both the development and test cohorts, the median age of AA patients and non-AA patients was 69 and 71 years, respectively. In the development cohort, AA patients had a higher median baseline PSA (13 ng / mL vs. 10 ng / mL), more T1-T2a (61% vs. 55%), more Gleason 8-10 (17% vs. 13%), and a higher National Comprehensive Cancer Network (NCCN) risk (42% vs. 35%) compared with non-AA patients; similar results were observed in the test cohort, except for T stage ( Fig.11 For all MMAI models in both the development and test cohorts, the distribution overlap between the AA and non-AA subgroups was ( Fig.12). In the development cohort, the median (IQR) score for the model optimized for DM (DM MMAI) was 0.36 (0.26–0.47) in AAs and 0.36 (0.26–0.49) in non-AAs; and in the test cohort, 0.38 (0.29–0.47) vs. 0.37 (0.27–0.48). In the development cohort, the median score for DM MMAI was 0.38 (0.30–0.38) in AAs and 0.40 (0.32–0.50) in non-AAs; and in the test cohort, 0.40 (0.32–0.49) vs. 0.40 (0.32–0.50) ( Fig.13 ). The results of the first MMAI model were reported in Fig.14 middle.

[0219] Performance of the Multimodal AI (MMAI) Model in Subgroups

[0220] In the test cohort, the DM MMAI model score showed a strong prognostic signal for DM in both the AA subgroup (hazard ratio [HR] for each 0.05 score increase: 1.2 for DM, p = 0.007) and the non-AA subgroup (1.4 for DM, p < 0.001). Fig.15A Similarly, the PCSM MMAI score showed a strong prognostic signal for PCSM in both the AA subgroup (hazard ratio per 0.05 score increase: 1.2 for PCSM, p = 0.01) and the non-AA subgroup (1.5 for PCSM, p < 0.001) ( Fig. 15B All original models demonstrated similar results in both the AA and non-AA subgroups ( Fig.16 and Fig.17 ).

[0221] Cumulative incidence rates were compared among racial subgroups in the full cohort; at 10 years, the estimated DM rates were 5% (3%-6%) for the AA subgroup and 7% (6%-8%) for the non-AA subgroup ( Fig.18A Both MMAI models can stratify the risk of patients in the AA subgroup and the non-AA subgroup ( Fig.19A and Fig.19B). In the test cohort, for the DM MMAI model, the 5-year estimated DM rates were 3% (95% CI: 0%-6%), 8% (95% CI: 3%-14%), and 20% (95% CI: 2%-38%) for the AA subgroup, and 1% (95% CI: 0%-1%), 5% (95% CI: 3%-7%), and 23% (95% CI: 14%-32%) for the non-AA subgroup. The adjusted pairwise comparisons between AAs and non-AAs of different risk groups were not statistically significant (p-values ​​= 0.36, 1.00, 1.00, respectively). Similarly, for the PCSM MMAI model, the 10-year estimated PCSM rates were 5% (95% CI: 0%-10%), 8% (95% CI: 2%-14%), and 30% (95% CI: 9%-51%) for the AA subgroup, and 1% (95% CI: 0%-3%), 8% (95% CI: 5%-11%), and 19% (95% CI: 11%-28%) for the non-AA subgroup ( Fig.18B The adjusted pairwise comparisons between AA and non-AA in different risk groups were not statistically significant (p-values ​​= 1.00, 1.00, 1.00, respectively). The original MMAI model showed similar results in both the AA subgroup and the non-AA subgroup ( Fig. 20A and Fig. 20B ).

[0222] discuss

[0223] AI-based biomarkers could help physicians tailor treatment recommendations for prostate cancer patients. However, AA men may be underrepresented in population data used to develop novel biomarkers. Previous biomarker studies have questioned their value when developed in primarily non-AA cohorts and applied to AA men. This paucity of genomic data that includes AA populations could exacerbate known health disparities experienced by this population by algorithmically encoding these inequities. These observations highlight the need to examine the performance of biomarkers across ethnic lines using more clinically relevant endpoints and applying rigorous methods to control for selection bias.

[0224] A large amount of data was used to train the MMAI model, and the prognostic performance of the AI ​​model was found to be comparable between the AA and non-AA subgroups. The DM and PCSM MMAI models performed similarly in the AA and non-AA patient populations, demonstrating algorithmic fairness in the application of our tool. This approach supports the use of these AI biomarkers to provide personalized treatment options for men with prostate cancer across ethnicities. Additionally, this analysis provides a method for integrating algorithmic fairness principles into routine biomarker discovery and validation.

[0225] Example 3: Risk stratification of prostate cancer patients using MMAI

[0226] In this example, the ability of the multimodal artificial intelligence (MMAI) model described herein to stratify patients into risk groups was compared to the National Comprehensive Cancer Network (NCCN) risk stratification scheme.

[0227] The 5,569 individuals with a clear NCCN risk classification from the dataset described in Example 1 were classified into one of ten deciles based on the 10-year risk of distant metastasis (DM 10 years) according to the corresponding MMAI score predicted by the MMAI model described in Example 2. Each decile was then stratified into one of three MMAI prognostic risk groups ("MMAI low", "MMAI intermediate" and "MMAI high") based on the MMAIDM 10-year score (<10%, 10%-25% and >25%, respectively). Fig.21 The baseline characteristics of each MMAI prognostic risk group were as follows Fig. 22 shown. Fig.23 Each MMAI prognostic risk group (row) is depicted with the number of individuals classified according to the NCCN risk scheme as "low," "intermediate" (favorable and unfavorable), or "high" (NCCN high or very high).

[0228] For each individual, the MMAI model was again used to calculate the probability of DM at 10 years to determine whether the MMAI was better than the NCCN risk categories in predicting distant metastasis at 10 years. Fig.24 The mean 10-year risk of DM for individuals with a given NCCN and MMAI classification is shown (with confidence intervals in parentheses). Fig.24 As shown in Figure 2, the 10-year risk of DM was approximately the same for individuals classified as low risk by either the NCCN or MMAI schemes. However, the MMAI model was better able to identify which individuals classified as intermediate or high risk by the NCCN scheme were actually at higher risk for DM. Fig.24 As shown, the subset of individuals classified as NCCN “intermediate” and classified as MMAI “high” had a 60% MMAI-predicted probability of DM 10 years, while the subset of individuals classified as NCCN “high” and classified as MMAI “high” had a 36% MMAI-predicted probability of DM 10 years. Therefore, the risk classification based on MMAI can stratify those NCCN intermediate individuals who are at risk for metastasis. Fig.25AAs shown, the MMAI model identified 6 times more patients with the lowest risk of metastasis than the NCCN. Among those individuals with NCCN "intermediate" risk, approximately 83% had a low MMAI score and therefore a low risk of metastasis, while among those with NCCN "high" risk, approximately 13.2% had a low MMAI score. Among those individuals classified as "high" risk by the NCCN, approximately 28% had a "high" MMAIDM 10-year score (risk ≥ 30%) ( Fig.25B ).

[0229] Compared to the NCCN classification, the MMAI system disclosed herein can better stratify those individuals at risk for prostate cancer metastasis.

[0230] Example 4: External Validation of the MMAI Model

[0231] This example describes the validation of a multimodal artificial intelligence (MMAI) model as described herein for predicting prostate cancer risk.

[0232] patient

[0233] NRG / RTOG-9902 enrolled 397 high-risk localized prostate cancer (PCa) patients who were randomized to receive long-term androgen suppression (AS) together with radiation therapy (RT) alone (AS+RT) or together with adjuvant combined chemotherapy (CT) (AS+RT+CT) between January 2000 and October 2004. CT was four 21-day cycles of paclitaxel, estramustine, and oral etoposide, starting 28 days after 70.2Gy RT. The AS regimen was luteinizing hormone-releasing hormone (LHRH) for 24 months starting 2 months before RT, plus oral antiandrogens for 4 months before and during RT. Men were enrolled if they had a PSA between 20-100 and a Gleason score ≥7, or if they had a clinical stage ≥T2 and a Gleason score ≥8. The 10-year results showed no statistically significant differences in overall survival (OS), biochemical failure (BF), local progression (LP), distant metastasis (DM), or disease-free survival (DFS) between the two treatment groups (p>0.05 for all endpoints listed). Therefore, in this example, all men were pooled into one cohort regardless of treatment group.

[0234] Sample processing and scanning

[0235] NRG Oncology used a Leica Biosystems Aperio AT2 digital pathology scanner to digitize pre-treatment biopsy slides from RTOG-9902 at 20x resolution. The NRG Biobank operators and the AI ​​data acquisition team reviewed the quality and clarity of the histopathology images. A previously constructed artifact classifier was used to filter out low-quality images.

[0236] Description of the Multimodal AI (MMAI) Model

[0237] Using five Phase III NRG trials (RTOG 9202, 9408, 9413, 9910, 0126), digital histopathology slides and clinical data from each patient were used to develop and validate an MMAI architecture with six MMAI algorithms, as described in Example 2 and as described in Example 3. Fig. 9 From this MMAI architecture, two locked MMAI algorithm risk scores were optimized for the desired clinical endpoints: distant metastasis (DM) and prostate cancer-specific mortality (PCSM).

[0238] end

[0239] The primary endpoints of this study were (1) DM and (2) time to PCSM, with DM defined as the number of days from randomization to the date of distant metastasis and time to PCSM defined as the number of days from randomization to death from prostate cancer.

[0240] Secondary endpoints were time to biochemical failure (BF), defined as the number of days from randomization to the date of biochemical failure (prostate-specific antigen (PSA) failure or the first day of initiation of salvage hormone therapy), and time to overall survival (OS), defined as the number of days from randomization to the date of death.

[0241] Subjects lost to follow-up before experiencing an event of interest were censored at their last follow-up. For DM, PCSM, and BF, death before experiencing an event of interest was considered a competing event.

[0242] Statistical analysis

[0243] The digital pathology evaluable population (DPEP) was defined as subjects randomized to RTOG-9902 with high-quality histopathology data and no missing baseline clinical variables (including age, clinical T stage, primary and secondary Gleason grade, and baseline PSA) to generate the MMAI algorithm score for analysis.

[0244] Baseline demographic and clinical characteristics of the DPEP and intention-to-treat (ITT) populations were summarized descriptively and compared between subgroups of patients in the DPEP and from the ITT, but high-quality histopathology data were not available. Descriptive summaries of categorical variables were provided using counts and proportions (%) and continuous variables using medians and interquartile ranges (IQRs). P values ​​were calculated for continuous variables using the Wilcoxon rank-sum test, and for categorical variables using the Pearson chi-square test or Fisher's exact test.

[0245] The prognostic performance of the MMAI algorithm was evaluated using univariate and multivariate analyses. Fine and Gray regression was used to estimate subdistribution hazard ratios (sHRs) and 95% confidence intervals (CIs) for DM, PCSM, and BF endpoints. Cox proportional hazards regression was used to estimate HRs and 95% CIs for OS endpoints. MMAI algorithm scores were split into quartiles and summarized using cumulative incidence curves, which provide five- and ten-year estimated DM and PCSM rates and corresponding two-sided 95% CIs. MMAI-treatment interactions were also tested as exploratory analyses.

[0246] All statistical analyses were performed using R version 4.1.2 (R Foundation for Statistical Computing, Vienna, Austria). All statistical tests were two-sided, and a significance level of 0.05 was used. Results of prognostic model validation were reported using the TRIPOD reporting standards.

[0247] result

[0248] Participants

[0249] MMAI algorithm scores were generated for 318 of the 397 original clinical trial patients enrolled in NRG / RTOG-9902 (85% of the entire trial cohort had available slides; 5.6% of these were not included due to poor image quality). Fig.26The process of patient flow from the NRG / RTOG 9902 clinical trial to the DPEP included for model validation is shown. The baseline characteristics of the DPEP are shown in Figure 27. The evaluable population included men with a median baseline PSA of 23.0 ng / mL, 32% of whom had cT3-4 disease, 67% had Gleason grade group 4 or 5 disease, and 54% had >1 NCCN high-risk feature. At a median follow-up of 10.1 years, 42 men experienced DM and 29 men had PCSM. There were no statistically significant differences in baseline characteristics between the patients in whom the MMAI algorithm score could be obtained, compared to the 62 patients excluded from this validation study. Similarly, there were no differences in baseline characteristics between the two treatment groups in the DPEP. The median (IQR) score for the algorithm optimized for DM (DM MMAI) was 0.54 (0.44-0.62), and the median (IQR) score for the algorithm optimized for PCSM (PCSM MMAI) was 0.53 (0.47-0.60). Both scores were similar between the two treatment groups in NRG / RTOG 9902 ( Fig.27B ).

[0250] Model performance

[0251] Compared with clinical and pathological factors, the MMAI algorithm was significantly prognostic for outcome measures. In univariate analysis, the DM MMAI algorithm continuous score was statistically associated with the DM endpoint (sHR 2.33, 95% CI 1.60-3.38, p<0.001), and the PCSM MMAI algorithm was statistically associated with the PCSM endpoint (HR 2.63, 95% CI 1.70-4.08, p<0.001). Fig.28A ). When evaluating the non-optimized secondary endpoints, DM MMAI was statistically significantly associated with the risk of BF, PCSM, and OS. Similarly, PCSM MMAI was statistically significantly associated with the risk of DM and OS ( Fig.28B ).

[0252] For the DM endpoint, DM MMAI was prognostic in most clinical subgroups, including both treatment groups, age, non-African Americans, both PSA groups, Gleason 8-10, clinical T stage, and patients with 1 NCCN high-risk factor ( Fig.29A Similarly, for the PCSM endpoint, PCSM MMAI was also prognostic in most subgroups, including both treatment groups, age, both racial subgroups, PSA < 20 ng / mL, Gleason 8-10, clinical T stage, and patients with any NCCN high-risk factor ( Fig.29B In multivariate analysis ( Figure 30A-Figure 30B and Figure 31A-Figure 31B ), controlling for individual clinical factors (including age, baseline PSA, Gleason, T stage, or number of NCCN high-risk factors), both DM MMAI and PCSM MMAI had consistently significant prognostic value.

[0253] Using quartiles on the DM MMAI, patients in the lower 75% (Q1-3) had estimated 5- and 10-year DM rates of 4% (95% CI 1%-6%) and 7% (95% CI 4%-10%), and patients in the highest quartile (Q4) had estimated 5- and 10-year DM rates of 19% (95% CI 10%-28%) and 32% (95% CI 21%-43%), with an sHR of 5.1 (95% CI 2.7-9.3, p<0.001) ( Fig.32A Similar results were observed for PCSM MMAI (sHR 4.1, 95% CI 2.0-8.4, p<0.001) ( Fig.32B ).

[0254] There were no statistically significant interactions between DM MMAI quartile groups (Q4 vs. Q1-3) and CT treatment effect (p for interaction = 0.08) or between PCSM MMAI quartile groups and CT treatment effect (p for interaction = 0.79). Fig.33A and Fig.33B Among the top 25% of patients ranked by DM MMAI, the estimated absolute benefit from the additional use of CT was 14% at 5 years and 18% at 10 years ( Fig.33B ).

[0255] discuss

[0256] The prognostic power of the MMAI classifier, previously developed using men from five Phase III PCa trials (NRG / RTOG 9202, 9408, 9413, 9910, and 0126), was further validated using an external validation set using men from NRG / RTOG 9902, which enrolled men at high risk of disease progression. The parent trial (NRG / RTOG 9902) did not produce statistically significant clinical results when comparing treatment groups, so the validation sample was treated as a single cohort (the study groups were balanced and the mean MMAI classifier scores were similar between treatment groups). Even in the NCCN high-risk and very high-risk populations, the differences in DM and PCSM between the lower-risk and higher-risk groups of the MMAI classifier were large and statistically significant. In multivariate analysis, the MMAI score was independently prognostic, even after controlling for variables known to be associated with prognostic risk (patient age, Gleason score, T stage). The association of the MMAI classifier with DM and PCSM within subgroups suggests that the MMAI has increased discriminatory and prognostic power across the continuum of high-risk and very-high-risk disease.

[0257] Although preferred embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that these embodiments are provided by way of example only. The present invention is not intended to be limited by the specific examples provided in the specification. Although the present invention has been described with reference to the above description, the description and illustration of the embodiments herein are not meant to be interpreted in a limiting sense. Without departing from the present invention, those skilled in the art will now conceive of many variations, changes and substitutions. In addition, it should be understood that all aspects of the present invention are not limited to the specific descriptions, configurations or relative proportions depending on various conditions and variables set forth herein. It should be understood that in practicing the present invention, various alternatives to the embodiments of the present invention described herein may be adopted. Therefore, it is contemplated that the present invention should also cover any such alternatives, modifications, variations or equivalents. The appended claims are intended to define the scope of the present invention, and thus cover methods and structures within the scope of these claims and their equivalents.

Claims

1. A method for assessing cancer in a subject, the method comprising: (a) obtaining a data set comprising image data and tabular data derived from the object; (b) processing the data set using a trained algorithm to classify the data set into a category among a plurality of categories, wherein the classification comprises applying an image processing algorithm to the image data; as well as (c) assessing the cancer in the subject based at least in part on the category among the plurality of categories classified in (b).

2. The method of claim 1, wherein the trained algorithm is trained using self-supervised learning.

3. A method according to claim 1 or claim 2, wherein the trained algorithm comprises a deep learning algorithm.

4. The method of any one of claims 1-3, wherein the trained algorithms include a first trained algorithm that processes the image data and a second trained algorithm that processes the tabular data. 5 . The method of claim 4 , wherein the trained algorithm further comprises a third trained algorithm that processes outputs of the first trained algorithm and the second trained algorithm.

6. The method of any one of claims 1-5, wherein the cancer is bladder cancer, breast cancer, cervical cancer, colorectal cancer, gastric cancer, kidney cancer, liver cancer, ovarian cancer, pancreatic cancer, prostate cancer, or thyroid cancer.

7. The method of claim 6, wherein the cancer is prostate cancer.

8. The method according to any one of claims 1-7, wherein the tabular data comprises clinical data of the subject.

9. The method of claim 8, wherein the clinical data of the subject comprises laboratory data, therapeutic interventions, or long-term outcomes.

10. The method of any one of claims 1-9, wherein the image data comprises digital histopathology data.

11. The method of claim 10, wherein the histopathology data comprises an image of a biopsy sample derived from the subject.

12. The method of claim 11, wherein the image is acquired by microscopic examination of the biopsy sample.

13. The method of any one of claims 10-12, wherein the digital histopathology data is obtained from the subject before the subject receives treatment.

14. The method of claim 13, wherein the treatment comprises radiation therapy (RT).

15. The method of claim 14, wherein the RT comprises pre-specified use of short-term androgen deprivation therapy (ST-ADT), long-term ADT (LT-ADT), dose-escalation RT (DE-RT), or any combination thereof.

16. The method of any one of claims 10-12, wherein the digital histopathology data is obtained from the subject after the subject has received treatment.

17. The method of claim 16, wherein the treatment comprises radiation therapy (RT).

18. The method of claim 17, wherein the RT comprises pre-specified use of short-term androgen deprivation therapy (ST-ADT), long-term ADT (LT-ADT), dose-escalation RT (DE-RT), or any combination thereof.

19. The method of any one of claims 1-18, further comprising processing the image data using image segmentation, image stitching, object detection algorithms, or any combination thereof.

20. The method of any one of claims 1-19, further comprising extracting features from the image data.

21. A method for assessing cancer in a subject, the method comprising: (a) obtaining a dataset comprising at least image data originating from the object; (b) processing the data set using a trained algorithm to classify the data set into a category among a plurality of categories, wherein the classification comprises applying an image processing algorithm to the image data, wherein the trained algorithm is trained using self-supervised learning; as well as (c) assessing the cancer in the subject based at least in part on the category among the plurality of categories classified in (b).

22. The method of claim 21, wherein the trained algorithm comprises a deep learning algorithm.

23. The method of claim 21 or claim 22, wherein the cancer is bladder cancer, breast cancer, cervical cancer, colorectal cancer, gastric cancer, kidney cancer, liver cancer, ovarian cancer, pancreatic cancer, prostate cancer or thyroid cancer.

24. The method of claim 23, wherein the cancer is prostate cancer.

25. The method of any one of claims 21-24, wherein the image data comprises digital histopathology data.

26. The method of claim 25, wherein the histopathology data comprises an image of a biopsy sample derived from the subject.

27. The method of claim 26, wherein the image is acquired by microscopic examination of the biopsy sample.

28. The method of any one of claims 25-27, wherein the digital histopathology data is obtained from the subject before the subject receives treatment.

29. The method of claim 28, wherein the treatment comprises radiation therapy (RT).

30. The method of claim 29, wherein the RT comprises pre-specified use of short-term androgen deprivation therapy (ST-ADT), long-term ADT (LT-ADT), dose-escalation RT (DE-RT), or any combination thereof.

31. The method of any one of claims 25-27, wherein the digital histopathology data is obtained from the subject after the subject has received treatment.

32. The method of claim 31 , wherein the treatment comprises radiation therapy (RT).

33. The method of claim 32, wherein the RT comprises pre-specified use of short-term androgen deprivation therapy (ST-ADT), long-term ADT (LT-ADT), dose-escalation RT (DE-RT), or any combination thereof.

34. The method of any one of claims 21-33, further comprising processing the image data using an image segmentation, image stitching or object detection algorithm.

35. The method of any one of claims 21-34, further comprising extracting features from the image data.

36. The method of any one of claims 21-35, wherein the data set comprises image data and tabular data.

37. The method of claim 36, wherein the trained algorithms include a first trained algorithm that processes the image data and a second trained algorithm that processes the tabular data.

38. The method of claim 37, wherein the trained algorithm further comprises a third trained algorithm that processes outputs of the first trained algorithm and the second trained algorithm.

39. The method of any one of claims 36-38, wherein the tabular data comprises clinical data of the subject.

40. The method of claim 39, wherein the clinical data comprises laboratory data, therapeutic interventions, or long-term outcomes.

Images