Systems and methods for analyzing microbiome using artificial intelligence

Abstract

The disclosed systems and methods include performing a convolutional neural network to detect microbiome levels in a full slide image associated with a patient; classifying the full glass slide image into one of a low microbiome and a high microbiome based on the output of the convolutional neural network; and determining a characteristic of a cancer associated with the patient based on the classification of the full slide image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 63 / 341,559, filed on May 13, 2022, and U.S. Provisional Patent Application No. 63 / 395,163, filed on August 4, 2022. The entire disclosures of U.S. Provisional Patent Application No. 63 / 341,559 and U.S. Provisional Patent Application No. 63 / 395,163 are incorporated herein by reference. Technical Field

[0003] The present invention relates generally to systems and methods for analyzing image data, and in particular to identifying microbiome in whole slide images and determining relative microbiome levels in hematoxylin and eosin (H&E) stained images. Background Art

[0004] The human microbiome profoundly affects cancer and plays a role in the progression of carcinogenesis. The microbiome is a community of microorganisms (such as fungi, bacteria and viruses) present in a specific environment. In humans, the term is often used to describe microorganisms that survive in or on a specific part of the body (such as the skin or gastrointestinal tract) or in a specific area (such as the tumor microenvironment (TME)). These microorganisms are dynamic and change in quantity and identity in response to many environmental factors (such as exercise, diet, drugs and other influencing factors).

[0005] Recent studies have demonstrated significant microbial contributions in selected cancer types, including, for example, cancer development, progression, and response to therapy, but the full extent and diagnostic significance of the microbial contribution across cancer types remains unknown.

[0006] Manual examination of tissue and blood samples (e.g., by using a microscope) is fraught with problems such as imprecision, inaccuracy, and inefficiency. From a patient's perspective, manual examination of tissue and blood samples is inadequate because the patient needs to make an appointment with a specialist and travel to a specific hospital or clinic to provide the sample.

[0007] The human microbiome profoundly influences cancer and plays a role in the progression of carcinogenesis. These microbial communities are dynamic and change in response to many environmental factors, such as exercise, diet, drugs, and other influencing factors.

[0008] Recent studies have demonstrated a significant microbial contribution in selected cancer types, but the magnitude and diagnostic significance of the microbial contribution across cancer types remains unknown. Summary of the invention

[0009] There is a need for a system and method that can automatically and / or remotely diagnose and / or measure the severity or stage of cancer in a patient. The human microbiome has a profound impact on cancer development, progression, and response to treatment. As described herein, an image-based computational pathology system can be trained to assess microbiome levels in a whole slide image (WSI). Such a system can be trained to classify an input WSI as microbiome low or microbiome high. This classification can be used to determine the severity of cancer in a patient from which the input WSI was captured. To facilitate microbiome analysis in cancer, as described herein, the present disclosure provides an artificial intelligence (AI)-driven computational pathology system configured to determine relative microbiome levels in tissue specimens, including hematoxylin and eosin (H&E) stained slides from formalin-fixed paraffin-embedded tissue samples or biopsies.

[0010] Whole genome and whole transcriptome sequencing data that can be used for microbiome detection as described herein are available in The Cancer Genome Atlas (TCGA) Bladder Cancer, and images of this TCGA Bladder Cancer are available to define microbiome low and microbiome high signatures.

[0011] The systems and methods described herein can be used to determine the correlation between the microbiome levels determined by the AI ​​system and the therapeutic response in a patient (e.g., a patient with non-muscle invasive bladder cancer). This correlation can be used to determine the severity of cancer in a patient based on the microbiome levels determined by the AI ​​system. By comparing the WSI of a patient captured at different times, the level of cancer progression can be assessed.

[0012] As described herein, the TCGA bladder cancer group can be assigned to a training set, a validation set, and a test set; each of which has an equal number of microbiome-low patients and microbiome-high patients. The group can be characterized by an average of one or more diagnostic (DX) images per patient, with hundreds of DX images in total. In an exemplary study provided by the present disclosure, the TCGA bladder cancer group (n=408) was assigned to a training set (66.7%, n=272), a validation set (8.3%, n=34), and a test set (25.0%, n=102); each of which has an equal number of microbiome-low patients and microbiome-high patients. The group is characterized by an average of 1.05 DX images per patient, with a total of 429 DX images.

[0013] According to one or more of the embodiments described herein, a deep network can be trained using hundreds or thousands of randomly selected blocks from hundreds of diagnostic images in a training set. Testing on unprocessed (i.e., original, unaltered, unedited) patient data provided an area under the receiver operating characteristic (ROC) curve of greater than about 0.7 and an F1 score of greater than about 0.7. The present disclosure provides an exemplary study in which a deep network was trained using a total of 303,104 blocks (100×100 microns in size, equivalent to 400×400 pixels at 40x magnification) randomly selected from 296 DX images in a training set. Testing on unprocessed patient data (n=102) provided an area under the ROC curve of 0.74 and an F1 score of 0.74.

[0014] Typically, the aggregated results of three different training checkpoints for each CNN can make the performance on the validation set more stable. Table 1 below shows the evaluation metrics of the three highest checkpoints for each CNN using the validation set. Each CNN also includes a collection of the top three checkpoints.

[0015]

[0016]

[0017] Table 1: Performance of microbiome detection applications on the TCGA bladder cancer validation set (34 WSIs).

[0018] The areas under the ROC curve and precision-recall curves of the ensemble using Inceptionv3, ResNet-152, and DenseNet-201 ranged from 0.77 to 0.82 (ROC-AUC) and from 0.70 to 0.78 (PR-AUC), with DenseNet-201 achieving the best results on the TCGA validation set.

[0019] In the methods and systems of the present disclosure, the developed AI system can be used to determine the correlation between the survival rate and microbiome levels of TCGA bladder cancer patients. For example, patients with bladder cancer who were identified as having a low microbiome by the image-based system of the present disclosure were considered to have a higher survival rate, with a hazard ratio of 1.45.

[0020] Further, the correlation between pathological complete remission (pCR) can be evaluated for days, weeks, months or even years after diagnosis or treatment, and the microbiome levels in patients with non-muscle invasive bladder cancer in clinical trials can be obtained. Patients who achieve pCR with treatment can be found to have a significantly higher probability of microbiome high pre-treatment compared to patients who do not achieve pCR. A decrease in the predicted microbiome levels after treatment can be observed in pCR patients. Conversely, non-responders can be found to have an increase in microbiome high probability with treatment.

[0021] As described herein, novel AI-driven image-based computational pathology systems have the potential to provide data that can not only inform clinical decisions, but also allow for further study of the role of the TME microbiome in cancer. The AI-driven image-based microbiome analysis systems and methods of the present disclosure can be used to reveal the correlation of higher pre-treatment microbiome levels with complete remission in trial patients.

[0022] These and other needs are addressed by various embodiments and configurations of the present disclosure. The systems and methods of the present disclosure provide multiple advantages depending on the specific configuration, and these and other advantages will become apparent from the present disclosure.

[0023] As described herein, systems and methods can be used to analyze microbiomes in images of blood and / or tissue samples and, based on the analysis, perform cancer pathology, such as cancer diagnosis and staging, i.e., describe the extent of cancer in the patient from which the image was captured and determine treatment options for the patient. Such systems and methods can be used, for example, to diagnose the stage or severity of bladder cancer in a patient.

[0024] As described herein, an image-based system utilizing a convolutional neural network (CNN) can be configured to detect the microbiome in H&E-stained pathology slides of frozen tissue.

[0025] One or more means for performing any one or more of the above-described embodiments or aspects of the embodiments are described herein.

[0026] The present disclosure includes any aspect combined with any one or more other aspects.

[0027] The present disclosure includes any one or more of the features disclosed herein.

[0028] The present disclosure includes any one or more of the features as substantially disclosed herein.

[0029] The present disclosure includes any one or more of the features as substantially disclosed herein in combination with any one or more other features as substantially disclosed herein.

[0030] The present disclosure includes any one of the aspects / features / embodiments in combination with any one or more other aspects / features / embodiments.

[0031] The present disclosure provides uses of any one or more of the aspects or features as disclosed herein.

[0032] It should be appreciated that any feature described herein may be claimed in combination with any other feature(s) as described herein, regardless of whether the features are from the same described embodiment.

[0033] The foregoing is a simplified overview of the present invention, for providing an understanding of some aspects of the present invention. This overview is neither a broad overview nor an exhaustive overview of the present invention and its various embodiments. It is neither intended to identify the key or important elements of the present invention, nor to describe the scope of the present invention, but is intended to present the selected concepts of the present invention in a simplified form, as an introduction to the more detailed description presented below. As will be appreciated, other embodiments of the present invention may utilize one or more features set forth above or described in detail below, individually or in combination. In addition, although the present disclosure is presented according to exemplary embodiments, it should be appreciated that the individual aspects of the present disclosure may be claimed separately. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present disclosure is described in conjunction with the accompanying drawings, in which:

[0035] Figure 1 is an illustration of a computing environment according to one or more of the embodiments described herein;

[0036] Figure 2 is a block diagram of a machine learning system according to one or more of the embodiments described herein;

[0037] FIG. 3A to FIG. 3H is an illustrative whole slide image that may be used for analysis according to one or more of the embodiments described herein;

[0038] FIG. 4A to FIG. 4I is a graph showing data used by methods and systems according to one or more of the embodiments described herein and the results thereof;

[0039] Figure 5 is a graph showing the microbiome of each patient according to one or more of the embodiments described herein;

[0040] Fig. 6A and Figure 6Bis a graph showing microbiome probabilities according to one or more of the embodiments described herein; and

[0041] Figure 7 is a flow chart of a method according to one or more of the embodiments described herein. DETAILED DESCRIPTION

[0042] The whole body human microbiome and the human microbiome associated with the tumor microenvironment (TME) affect the development and progression of cancer and the response to therapy, which is at least in part due to the effect on the immune system. As described herein, a system can be configured to determine relative microbiome levels from WSI (such as H&E stained pathology slides). Based on the relative microbiome levels in images from patients, the TME microbiome can be analyzed and used to generate an estimated level of cancer in the patient.

[0043] Such a system as described herein may include one or more neural networks, such as a convolutional neural network (CNN). The neural network may be trained using tumor images labeled as microbiome high or microbiome low. Such tumor images may be, for example, tumor images from TCGA bladder cancer samples for which whole transcriptome-based microbiome analysis has been performed.

[0044] The QUILT 3032 trial (Chamie K, Lee JH, Rock A et al.: Preliminary phase 2 clinical results of IL-15RαFc superagonist N-803 with BCG in BCG-unresponsive non-muscle invasive bladder cancer (NMIBC) patients) was conducted by analyzing biopsy images (e.g., from patients with non-muscle invasive bladder cancer who were unresponsive to BCG) using BCG and the interleukin-15 superagonist N-803 (nogapendekin alfainbakicept; NAI). J Clin Oncol [Journal of Clinical Oncology] 37:4561-4561, 2019) performed an analysis of the image and found that patients who achieved complete remission (CR) with therapy were more likely to have a high microbiome at baseline compared with those who did not achieve CR.

[0045] These findings, along with the improved efficiency of the image-based analysis systems described herein, enable applications that provide insights into the relationship between microbiome levels and response to therapy in cancer patients. Using a system as described herein, microbiome levels in tumor histopathology images can be assessed, and the association between microbiome levels and response to therapy can serve as a useful tool for studying this relationship, and in order to inform treatment decisions.

[0046] Using a system as described herein, one or more neural networks can be used to estimate microbiome levels based on an input image of bladder cancer. Although the morphological changes introduced by the microbiome into solid tumor tissue may be too subtle to be easily discerned by eye even by a trained pathologist, the system as described herein is able to identify visual patterns that can distinguish tissues with high microbiome loads.

[0047] The microbiome includes a range of microbial groups (bacteria, fungi, etc.) that, combined with a spectrum of activities including proteins, peptides, nucleic acids, polysaccharides, toxins, and signaling molecules associated with the microbiome of the human gut and specifically the tumor microenvironment (TME), exert a profound influence on the development of cancer and response to therapy. This relationship may be based on the influence of the microbiome on tumor initiation and progression and the immune system.

[0048] The conventional method for evaluating some components of the TME microbiome is to perform immunohistochemical analysis, but this method is inherently limited by the number of microbiome-directed antibodies that can be used. As an alternative, microbiome analysis based on whole genome and transcriptome sequencing has become more widely used, but is not applied as part of standard diagnosis. A recent study, Poore, GD, Kopylova, E., Zhu, Q. et al., Nature [Natural] 579, 567-574 (2020), ("Poore et al.") (which is incorporated herein by reference) electronically examined the whole genome and whole transcriptome sequencing data of first-line patients in thirty-three cancer types in the Cancer Genome Atlas (TCGA) to obtain microbial readings, and found that most samples showed evidence that specific microbial signatures corresponded only to cancer. Poore et al. found that most samples from specific types of cancer showed significant intensity in several microbial groups.

[0049] As disclosed herein, to promote the feasibility and utility of TME microbiome analysis as a biomarker, the analysis of Poore et al. was included in a computational method we developed for detecting microbiome levels in hematoxylin and eosin (H&E) stained tissue section images. Based on the data of Poore et al., TCGA bladder cancer images were first classified as microbiome low or microbiome high. Next, a CNN-based system as described herein was trained based on the classified samples. Next, the performance of the CNN-based system was evaluated based on the output of the system when independent patient samples were provided as input. Finally, the CNN-based system was then used to evaluate the association between baseline, pre-treatment TME microbiome levels and response to therapy in patients in a clinical trial of the novel interferon-15 superagonist N-803 (Norgeleukin α Inchicept, NAI) combined with BCG in non-muscle invasive bladder cancer that was unresponsive to BCG.

[0050] Using a system or method as described herein, microbiome information can be identified from an input whole slide image (WSI) using the systems and methods described herein. The systems and methods described herein use a trained neural network (such as a CNN) to identify microbiome characteristics and locate microbiome pixels from one or more tiles generated from the input WSI. In certain embodiments, a determination can be generated as to whether the WSI or tile contains a specific amount of microbiome. The microbiome characteristics identified by the CNN can be used to determine the stage or severity of the cancer. Although bladder cancer and other cancer types are described herein as examples, it should be appreciated that the systems and methods described herein can be suitable for any type of cancer and / or a specific type of microbiome.

[0051] The systems and methods described herein apply a deep learning-based system to evaluate tumor tissue images to characterize the microbiome probability and response in the patient. This analysis can be extended to any type of cancer and / or can be used to implement a pan-cancer microbiome level detector. Further, the system described herein can be used not only to detect high microbiome levels or low microbiome levels, but also to distinguish microbial types, such as to generate a series of microbial signatures and action domain signatures. This system can be used to evaluate the response to therapy in a variety of tumor types and treatment methods, and can also be used to evaluate prognosis and metastasis potential, and reveal the characteristics of the TME microbiome that responds best to therapy. This application is likely to become a viable method to help guide doctors' decisions for cancer patients. As described herein, a system and method for determining one or more characteristics of a cancer associated with a patient can be determined. Such characteristics may include, for example, factors such as the stage of cancer, the grade of cancer and / or tumors, the location of cancer and / or tumors, the presence of metastasis, the expression of biomarkers that may lead to therapeutic or immune escape (such as the presence of increased expression of PD-L1 in tumors), and the distribution of lymphocytes and other immune cell types within the tumor environment. Different types of cancer (such as bladder cancer, head and neck cancer, and ovarian cancer) can be associated with different characteristics that can be used to determine their severity. As used herein, a characteristic can refer to any one or more of any such factors.

[0052] The following description provides only embodiments and is not intended to limit the scope, applicability or configuration of the claims. Rather, the following description will provide a description for enabling implementation of the embodiments to those skilled in the art. It will be understood that various changes may be made to the function and arrangement of elements without departing from the spirit and scope of the appended claims.

[0053] Any reference numeral in the specification including a numerical reference numeral without an alphabetical sub-reference numeral, when used in the plural, is a reference to any two or more elements having the same reference numeral when sub-reference numerals are present in the drawings. When such a reference is made in the singular without identifying a sub-reference numeral, it is a reference to one of the like numbered elements but is not limited to the specific one of the referenced elements. Any explicit use herein to the contrary or to provide further quantification or identification shall take precedence.

[0054] Exemplary systems and methods of the present disclosure will also be described with respect to analysis software, modules, and associated analysis hardware. However, to avoid unnecessarily obscuring the present disclosure, the following description omits well-known structures, components, and devices that may be omitted from the drawings or shown in simplified form or otherwise outlined.

[0055] The phrases "at least one," "one or more," "or," and "and / or" are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," "A, B and / or C," and "A, B, or C" refers to A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0056] The term "a" or "an" entity refers to one or more of that entity. Thus, the terms "a" (or "an"), "one or more" and "at least one" can be used interchangeably herein. It should also be noted that the terms "comprising", "including" and "having" can be used interchangeably.

[0057] As used herein, the term "automatic" and variations thereof refer to any generally continuous or semi-continuous process or operation that is performed without substantial human input in order to complete the process or operation. However, if input is received prior to the performance of the process or operation, then the process or operation may be automatic, even if the performance of the process or operation uses substantial or insubstantial human input. Human input is considered substantial if it affects the manner in which the process or operation is performed. Human input that consents to the performance of the process or operation is not considered "substantial."

[0058] Aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.

[0059] A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) would include the following: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible, non-transitory medium that can contain or store a program used by or in conjunction with an instruction execution system, apparatus, or device.

[0060] Computer readable signal media may include a propagated data signal in which a computer readable program code is implemented, such as in a baseband or as part of a carrier wave. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic forms, optical forms, or any suitable combination thereof. Computer readable signal media may be any computer readable medium that is not a computer readable storage medium and that may transmit, propagate, or transmit a program for use by an instruction execution system, device, or equipment or for use in conjunction with it. The program code implemented on a computer readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

[0061] As used herein, the terms "determine," "calculate," "compute," and variations thereof may be used interchangeably and include any type of methodology, process, mathematical operation, or technique.

[0062] Pursuant to 35 U.S.C., Section 112(f) and / or Section 112, paragraph 6, the term "means" as used herein should be given its broadest possible interpretation. Therefore, a claim incorporating the term "means" will cover all structures, materials, or acts set forth herein and all equivalents thereof. Further, structures, materials, or acts and their equivalents should include all that are described in the Summary of the Invention, the Description of the Figures, the Detailed Description, the Abstract, and the claims themselves.

[0063] For the purpose of explanation, numerous details are set forth in order to provide a thorough understanding of the present disclosure. However, it should be appreciated that the present disclosure can be practiced in various ways without being limited to the specific details set forth herein.

[0064] Figure 1Depicted is a computer system 103 in a computing environment 100 according to an embodiment of the present disclosure. In some embodiments, the method of analyzing whole slide images as described herein can be implemented by a computer system 103 present in a computer system such as Figure 1 The computer system 103 in the illustrated computing environment 100 is implemented. In one embodiment, the apparatus for identifying microbiome information in input image data may be implemented in whole or in part as the computer system 103, which includes various components and connections with other components and / or systems.

[0065] These components are implemented in various ways and may include a processor 106. As used herein, the term "processor" refers exclusively to an electronic hardware component that includes a circuit system having connections (e.g., pinouts) to transmit encoded electrical signals to and from the circuit system. The processor 106 may include programmable logic functions such as determined at least in part by accessing machine-readable instructions maintained in a non-transitory data storage device, which may be implemented as a circuit system, an on-chip read-only memory, memory 109, a data storage device 112, etc., which cause the processor 106 to perform the steps of the instructions.

[0066] Processor 106 may further be implemented as a single electronic microprocessor or multiprocessor device (e.g., multi-core) having circuitry therein, which may further include control unit(s), input / output unit(s), arithmetic logic unit(s), register(s), main memory, and / or other components that access information (e.g., data, instructions, etc.) received, such as via bus 115, execute instructions, and output data again, such as via bus 115. In other embodiments, processor 106 may include a shared processing device that may be utilized by other processes and / or process owners, such as in a processing array within a system (e.g., blade, multi-processor board, etc.) or a distributed processing system (e.g., a "cloud," a cluster, etc.). It should be appreciated that processor 106 is a non-transitory computing device (e.g., an electronic machine that includes circuitry and connections for communicating with other components and devices).

[0067] Processor 106 may operate a virtual processor, such as to process machine instructions that are not native to the processor (e.g., translating an operating system and machine instruction code set to enable an application to execute on a virtual processor), however, as will be appreciated by those of ordinary skill, such a virtual processor is an application executed by hardware, more specifically by the underlying circuitry and other hardware of a processor (e.g., processor 106). Processor 106 may be executed by a virtual processor. A virtual processor enables an application to appear to be a static and / or dedicated processor executing the application's instructions, although the underlying non-virtual processor(s) are executing the instructions and may be dynamic and / or split between multiple processors.

[0068] In addition to the components of the processor 106, the computer system 103 can utilize the memory 109 and / or the data storage device 112 to store accessible data, such as instructions, values, etc. The communication interface 118 facilitates communication with components, such as communicating with the processor 106 via the bus 115, and communicating with components that cannot be accessed via the bus 115. The communication interface 118 can be implemented as a network port, a card, a cable, or other configured hardware device. Additionally or alternatively, the manual input / output interface 121 is connected to one or more interface components to receive and / or present information (e.g., instructions, data, values, etc.) to and / or from manual and / or electronic devices. Examples of input / output devices 133 that can be connected to the input / output interface include, but are not limited to, keyboards, mice, trackballs, printers, displays, sensors, switches, relays, speakers, microphones, cameras, and / or video cameras, etc. In another embodiment, the communication interface 118 may include the manual input / output interface 121 or be included by the manual input / output interface. Communication interface 118 may be configured to communicate directly with networking components or utilize one or more networks, such as network 124 and / or network 127 .

[0069] The network 124 may be a wired network (e.g., Ethernet), a wireless network (e.g., Wi-Fi, Bluetooth, cellular, etc.), or a combination thereof, and may enable the computer system 103 to communicate with the networking component(s) 130. In other embodiments, the network 124 may be implemented in whole or in part as a telephone network (e.g., a public switched telephone network (PSTN), a private branch exchange (PBX), a cellular telephone network, etc.).

[0070] Additionally or alternatively, one or more other networks may be utilized. For example, network 127 may represent a second network that may facilitate communications with components used by computer system 103. For example, network 127 may be an internal network to a business entity or other organization, whereby components are trusted (or at least trusted to a certain extent), whereas networked component(s) 130 connected to a public network (e.g., the Internet) such as network 124 may not be trusted (or at least trusted to a lesser extent).

[0071] Components attached to the network 127 may include memory 136, data storage 139, input / output device(s) 133, and / or other components accessible to the processor 106. For example, the memory 136 and / or data storage 139 may supplement or replace the memory 109 and / or data storage 112 in its entirety or for a particular task or purpose. As another example, the memory 136 and / or data storage 139 may be an external data repository (e.g., a server farm, an array, a "cloud," etc.) and enable the computing device 103 and / or other devices to access data thereon. Similarly, the input / output device(s) 133 may be accessed by the processor 106 via the manual input / output interface 121 and / or via the communication interface 118 directly, via the network 127, via the network 124 alone (not shown), or via both the networks 127 and 124. Each of the memory 109, the data storage 112, the memory 136, the data storage 139 comprises a non-transitory data storage comprising a data storage device.

[0072] It should be appreciated that computer-readable data can be sent, received, stored, processed and presented by various components. It should also be appreciated that the components shown can control other components, whether it is shown herein or other. For example, an input / output device 133 can be a router, switch, port or other communication component, so that the specific output of the processor 106 enables (or disables) the input / output device 133 that can be associated with the network 124 and / or the network 127, to allow or not allow the communication between two or more nodes on the network 124 and / or the network 127. Those of ordinary skill in the art will recognize that, without departing from the scope of the embodiment, as a supplementary alternative to those communication equipment described herein, other communication equipment can be utilized.

[0073] Thus, and in one embodiment, processor 106 may execute instructions to perform the systems and methods described herein. In another embodiment, networking component(s) 130 may perform one or more systems and methods, while processor 106 may perform one or more other systems and methods. Memory values ​​may be read from memory 109 or memory of one or more network components 130. In another embodiment, output from systems and methods as described herein may be maintained in memory 136 and / or data storage 139.

[0074] The computer system 103 may be configured to perform the following operations: Figure 2 The illustrated machine learning (ML) system 200. In some embodiments, the ML system 200 may include a training element and a testing element. In other embodiments, the ML system 200 may include a pre-trained CNN and may not include a training element.

[0075] The training elements of ML system 200 may be used to train neural network 212 of ML system 200 to generate block-based results 215 based on inputs of blocks 206. The training elements of ML system 200 may be, for example, Figure 1 One or more processes are shown executed by processor 106 based on instructions in memory 109. Segments 206 may be stored in data storage 112 or received by computer system 103 via communication interface 118, and / or may be created from one or more images or WSIs stored in data storage 112 or received by computer system 103 via communication interface 118.

[0076] The WSI may be associated with a patient. In some embodiments, the method may include associating the WSI with a patient in a database. For example, the method may include identifying a patient associated with the WSI and determining the patient's cancer level at a particular time (such as a time before the date the sample imaged in the WSI was collected).

[0077] The WSI may be an image of a pathology slide including H&E staining. The WSI may be an image of a frozen tissue or blood sample.

[0078] Such a WSI can be, for example, a formalin-fixed paraffin-embedded (FFPE) tissue section slide diagnostic image, such as can be obtained from the comprehensive molecular characterization of the muscle-invasive bladder cancer (BLCA) cancer cohort in TCGA.

[0079] The tissue imaged in the image can be stored in an ultra-low temperature refrigerator at less than eighty degrees. By using frozen tissue images, natural non-denatured proteins can be separated in a static active form for biochemical analysis. In some embodiments, the input of one or more neural networks includes multiple blocks extracted from the WSI. The image may include, for example, a 400×400 pixel image with a 40x magnification. The size of each block in the multiple blocks is 100×100 microns. Each block in the multiple blocks may include a block of one hundred square microns. Each block may represent a tissue portion of 100 square microns.

[0080] Each of the input tiles 206 can be associated with a patient and / or a WSI associated with the patient. Generating a tile-based result 215 can include first preprocessing an input whole slide image (WSI) to decompose the WSI into a plurality of tiles. For example, and as described in more detail herein, a WSI associated with a patient can be divided into a plurality of tiles. Each tile created from a WSI can be associated with metadata indicating the WSI from which the tile was created and / or the patient associated with the WSI from which the tile was created. For example, during training, a WSI with microbiome levels established by other means (such as WGS or WTS of Poore et al.) or by manual review can be divided into a plurality of tiles.

[0081] The tiles 206 may be automatically generated by one or more training WSIs 203. The input to the neural network for training or testing may include image tiles of a particular size. The size may represent, for example, 100×100 microns, which may correspond to image tiles of size 400×400 pixels for a WSI with 40× magnification. It should be appreciated that any tile size and any WSI size may be used, and the numbers provided herein are provided only as examples.

[0082] Each input segment in the input segments 206 may also be associated with metadata indicating microbiome levels, such as a number or percentage indicating the amount of microbiome in the segment, or a classification such as microbiome high or microbiome low. The metadata indicating microbiome levels may be described as a ground truth microbiome label or classification.

[0083] The ground truth microbiome labels can be used to train the neural network 212 to generate tile-based results. The tile-based results can include an indication that the tile is associated with one of microbiome high or microbiome low, such as a confidence level. In some embodiments, the neural network can be trained to output a confidence score for each of microbiome high and microbiome low for each input tile.

[0084] In some embodiments, multiple CNNs can be trained separately. By training different CNNs, each trained CNN 218 can be evaluated separately, and the most accurate trained CNN 218 can be selected for use. In some embodiments, the results of different training checkpoints from different CNNs can be aggregated. This aggregation can produce more stable performance.

[0085] In this set, 7 patients (patient ID 020-001, -004, -007, -009, -013, -015, and -018) had a positive malignancy response (dark grey in the pCR column) at week 12 or 27. Six of the 7 non-responders were found to have an increase in microbiome high probability with treatment.

[0086] Generating a set of training data for the system as described herein can include utilizing any available data. Such data can be, for example, the microbiome analysis of Poore et al., which uses whole genome (WGS) and whole transcriptome sequencing (WTS) of solid tissue (i.e., non-blood) samples of bladder (TCGA-BLCA; n=408) cancer. The set of training data can include one or more WSIs for each patient in a plurality of patients. For example, in Poore et al., each of 71.8% of the patients had a sample sequenced for the microbiome, while the remaining 28.2% each had between two and six samples. The readings of each sample associated with the corresponding patient can be averaged to give an average level for each patient. The average level for each patient can be used to sort the patients and divide the patients into a low microbiome category or a high microbiome category relative to the median of the entire group.

[0087] In some embodiments, classifying patient samples into microbiome low and microbiome high categories can involve classifying certain patients whose standard deviation exceeds a median threshold that separates microbiome low and microbiome high patients. For example, such patients may be associated with ambiguous whole transcriptome sequencing (WTS) readings, emphasizing that omics-based microbiome estimation methods have inherent noise, and microbiome levels may exhibit a continuous spectrum rather than being clearly divided into microbiome high or microbiome low categories. However, systems for classifying problems based on microbiome levels as described herein provide clinical utility.

[0088] In some embodiments, the neural network 212 can be trained for multiple cycles with a starting learning rate (e.g., 0.1 or 0.2). For example, the neural network 212 can be initialized with multiple weights. In some embodiments, random values ​​can be automatically used as weights. During training, the input block 206 can be fed to the neural network 212. The neural network 212 can process the block 206 layer by layer, and ultimately generate a probability distribution over possible categories, i.e., microbiome low and microbiome high. Next, in some embodiments, the predicted category probabilities generated by the neural network 212 can be compared with the true labels associated with the training WSI 203, such as by using a loss function (e.g., cross entropy loss). Next, the initial weights of the neural network 212 can be updated based on the calculated loss, such as by using an optimization algorithm (e.g., stochastic gradient descent (SGD)). These steps can be repeated for multiple cycles until the neural network 212 converges to a constant loss. During testing, a result 215 based on the test block can be generated. This result 215 based on the test block can be used, which can be stored in a Figure 1 The data in the memory of the computer system 103 is shown in the form of a trained convolutional neural network (CNN) 218 ​​that can be used in the test element of the ML system 200. The trained CNN can be trained as discussed above with respect to the neural network 212. The trained CNN 218 can be, for example, a neural network such as Figure 1 Processes performed by the processor 106 of the computer system 103 shown. In some embodiments, the same neural network 212, once trained, can be used as a trained CNN 218. The trained CNN 218 can be a class of one or more computational models in deep learning that are particularly suitable for image analysis. The trained CNN 218 can utilize variations in multi-layer perception, which requires a certain amount of preprocessing of the input image (such as WSI). It should be appreciated that the trained CNN 218 can include input and output layers and multiple other layers, such as convolutional layers, pooling layers, fully connected layers, normalization layers, hidden layers, etc. In some embodiments, the convolutional blocks can use shared weights, wherein each input is processed in the same or similar manner. In some embodiments, the convolutional blocks can use different weights to process inputs in different ways.

[0089] The layers of the trained CNN 218 can be configured to apply convolution operations to the input and pass the results to the next layer. Pooling layers can be used to reduce the dimensionality of the input data by combining the outputs of a cluster of neurons at one layer into a single neuron in the next layer. In some embodiments, max pooling can use the maximum value of each neuron in the cluster of neurons from the previous layer, while in some embodiments, average pooling can use the average value from each neuron.

[0090] Although the embodiments described herein relate to a single CNN, it should be appreciated that multiple CNNs may be utilized together and may effectively form a single trained CNN 218. Furthermore, the methods and systems described herein may be used to train multiple CNNs to each recognize different types of microbiomes in different types of WSIs (e.g., images of blood samples, images of tissue samples, etc.) and for head and neck cancer, ovarian cancer, bladder cancer, or other types of cancer.

[0091] The input to the trained CNN 218 may include WSIs 221 divided into overlapping or non-overlapping blocks 224. As described above, the ground truth labels for each test WSI 221 may be used for each block 224 from the corresponding WSI 221. In some embodiments, a majority voting mechanism 230 may be used to aggregate block-level results into WSI-level prediction labels for testing purposes, as described below. The microbiome classification labels for testing the trained CNN 218 may be obtained from a source (e.g., each patient from Poore et al.) and may be used to annotate training and / or test WSIs 203, 221 and their corresponding blocks. When evaluating a new WSI 221 from a test set (using all blocks 224 of each test WSI 221), a majority voting mechanism 230 may be used to aggregate block-level results 227 into WSI-level prediction labels 233, as described in more detail below. As described herein, the probability that a test WSI 221 is microbiome high can be determined based on the percentage of tiles 224 associated with the test WSI 221 that are classified as microbiome high by the trained CNN 218. Similarly, the probability that a test WSI 221 is microbiome low can be determined based on the percentage of tiles 224 associated with the test WSI 221 that are classified as microbiome low by the trained CNN 218. During the test process, the tile-based results are aggregated to give an image-based probability of the microbiome in the test WSI.

[0092] As described above, the trained CNN 218 can be configured to generate a tile-based result 227 based on the input tile 224. The input tile 224 can be automatically generated by the trained CNN 218 from one or more test WSIs 221. The tile-based result 227 can be used to generate a microbiome label 233. In some embodiments, a majority voting mechanism 230 can be used to generate a microbiome label 233 based on the tile-based result.

[0093] For example, a WSI that can be used to train and / or test a neural network and to perform analysis using a trained neural network can be used. FIG. 3A to FIG. 3H What is shown.

[0094] Figure 3Ais an image of a WSI captured from a patient with head and neck cancer. This patient has a low microbiome. Figure 3A When an image of is input into a CNN trained as described herein, the trained CNN can output that the WSI is an indication of a low microbiome.

[0095] Figure 3B is an image of a WSI captured from a patient with head and neck cancer. This patient has a high microbiome. Figure 3B When an image of is input into a CNN trained as described herein, the trained CNN can output an indication that the WSI is high in the microbiome.

[0096] Figure 3C is an image of a WSI captured from a patient with ovarian cancer. This patient has a low microbiome. Figure 3C When an image of is input into a CNN trained as described herein, the trained CNN can output that the WSI is an indication of a low microbiome.

[0097] Figure 3D is an image of a WSI captured from a patient with ovarian cancer. This patient has a high microbiome. Figure 3D When an image of is input into a CNN trained as described herein, the trained CNN can output an indication that the WSI is high in the microbiome.

[0098] FIG. 3E to FIG. 3H Shown are WSIs from the TCGA bladder cancer cohort patients, each of which has been classified as either microbiome low or microbiome high. Figure 3E and Figure 3F The test image is microbiome low, and Figure 3G and Figure 3H The test images are of patients with a high microbiome.

[0099] when Figure 3E When the WSI of is input into a trained CNN 218 as described herein, the trained CNN 218 may output a probability of nine percent that the tissue imaged in the WSI is microbiome-high and a probability of ninety-one percent that the tissue imaged in the WSI is microbiome-low. Thus, a Figure 3E A WSI of 1.5 is indicative of a low microbiome.

[0100] when Figure 3F When the WSI of is input into a trained CNN 218 as described herein, the trained CNN 218 may output a probability of 61 percent that the tissue imaged in the WSI is microbiome-high and a probability of 39 percent that the tissue imaged in the WSI is microbiome-low. Thus, a Figure 3FA WSI of 1.5 is indicative of a high microbiome.

[0101] when Figure 3G When the WSI of is input into a trained CNN 218 as described herein, the trained CNN 218 may output a probability of 15 percent that the tissue imaged in the WSI is microbiome-high and an 85 percent probability that the tissue imaged in the WSI is microbiome-low. Thus, a Figure 3G A WSI of 1.5 is indicative of a low microbiome.

[0102] when Figure 3H When the WSI of is input into a trained CNN 218 as described herein, the trained CNN 218 may output a probability of ninety-nine percent that the tissue imaged in the WSI is microbiome-high and a probability of one percent that the tissue imaged in the WSI is microbiome-low. Thus, a Figure 3H A WSI of 1.5 is indicative of a high microbiome.

[0103] By analyzing an input image (such as test WSI 221), methods as described herein can be used to enable computer system 103 to use trained CNN 218 to identify the level or amount of microbiome in the input image and make certain determinations related to the microbiome and the presence of cancer in a patient associated with the input image based on the microbiome identified in the input image. The results of using the systems or methods as described herein are FIG. 4A to FIG. 4I and will be described in detail below. Figure 4A A receiver operating characteristic for a first patient is shown, plotted with sensitivity from zero to one on the vertical axis and specificity from zero to one on the horizontal axis. Curve 403 indicates the performance of a system as described herein. Line 406 indicates a one-to-one relationship between sensitivity and specificity. The performance of the system can be measured using the area under the curve (AUC) of the system performance plotted by curve 403. An AUC closer to one indicates that the system is able to accurately determine whether the input WSI associated with the first patient is microbiome high or microbiome low. Figure 4A In the example shown, the AUC is 0.875.

[0104] Figure 4B The receiver operating characteristic for the second patient is shown, plotted with sensitivity from zero to one on the vertical axis and specificity from zero to one on the horizontal axis. Curve 403 indicates the performance of the system as described herein. Line 406 indicates a one-to-one relationship between sensitivity and specificity. The performance of the system can be measured using the area under the curve (AUC) of the system performance plotted by curve 403. An AUC closer to one indicates that the system is able to accurately determine whether the input WSI associated with the second patient is microbiome high or microbiome low. Figure 4B In the example shown, the AUC is 0.7416.

[0105] Figure 4C The receiver operating characteristic for a third patient is shown, plotted with sensitivity from zero to one on the vertical axis and specificity from zero to one on the horizontal axis. Curve 403 indicates the performance of the system as described herein. Line 406 indicates a one-to-one relationship between sensitivity and specificity. The performance of the system can be measured using the area under the curve (AUC) of the system performance plotted by curve 403. An AUC closer to one indicates that the system is able to accurately determine whether the input WSI associated with the third patient is microbiome high or microbiome low. Figure 4C In the example shown, the AUC is 0.7416.

[0106] The area under the ROC curve (ROC-AUC), the area under the precision-recall curve (PR-AUC), precision, recall, and F1 scores can be used as metrics to evaluate the performance of the trained CNN 218. Here, both ROC-AUC and PR-AUC look at the WSI-based prediction score (p(microbiome high)) of the trained CNN 218 on the entire validation set or test set. This score can have any value from 0 to 1. However, the precision, recall, and F1 score values ​​are calculated using the counts of the class assignments (WSI true vs. predicted microbiome high labels and / or microbiome low labels) based on the optimal threshold. In some embodiments, the optimal threshold can be the threshold that minimizes the Euclidean distance to the point (0,1) on the ROC curve (indicating 0% false positives and 100% true positives). All performance evaluation metrics used in this study range from zero to one, where higher values ​​indicate better performance.

[0107] FIG. 4D to FIG. 4H An example of survival analysis of TCGA bladder test patients using ground truth labels as described above is shown. In each illustration, the estimated probability of survival is plotted on the vertical axis, and the amount of time in months is plotted on the horizontal axis. In each illustration, microbiome high patients 409 are plotted separately from microbiome low patients 412. As should be appreciated, microbiome high patients generally respond to therapy and exhibit better outcomes, including higher survival rates over time.

[0108] Figure 5 We present sequenced microbiome readouts from TCGA cohorts using whole genome and whole transcriptome sequencing data. Microbiome profiling of blood and tissue suggests approaches for cancer diagnostics. Figure 5In the graph of , the x-axis shows each patient sorted by the mean of the microbiome level. Patients to the left of the median threshold are associated with low microbiome levels, and patients to the right of the median threshold are associated with high microbiome levels. Figure 5 The y-axis of the graph of shows the number of microbiome readings for each patient. As should be appreciated, patients with the highest microbiome levels have relatively higher numbers of microbiomes. The increase in the average number of microbiome levels is shown by the diagonal line 503. Figure 5 The flat line 500 on the graph of shows the median threshold, which is the patient with a microbiome readout of 391,501.50 on the y-axis. Patients below the median threshold can be classified as microbiome low, while patients above the median threshold can be classified as microbiome high. As can be appreciated from the graph, 50% of the patients are indicated as microbiome low and 50% of the patients are indicated as microbiome high. Although in Figure 5 In the illustrated example, only two classifications are provided, microbiome low and microbiome high, but it should be appreciated that in some embodiments, patients may be classified into any number of groups, such as microbiome low, microbiome medium, and microbiome high. Figure 5 In the presented patient cohort, each patient was measured an average of 1.37 times. The median microbiome level was found to be 391,501.50.

[0109] As an example of training, testing, and validation of a neural network for creating a trained CNN 218 capable of detecting microbiome content in WSIs associated with bladder cancer patients as described herein, a bladder cancer cohort from TCGA can be used. In this example, the TCGA bladder cancer cohort includes 408 patients. In order to train a convolutional neural network as described herein, a training set with 272 patients can be used. Of the 272 patients, TCGA includes an average of approximately 1 WSI per patient, for a total of 296 WSIs. Each WSI can be divided into an average of 25,265.80 blocks, for a total of 7,478,677 blocks. In this example, for training, 1,024 blocks of each WSI can be used, leaving a total of 303,104 blocks for training.

[0110] Although the image-based microbiome level detection system described herein uses labels informed by sequencing data, it should be recognized that in some embodiments, the system can be used to determine whether sequencing is needed, such as for new patients. Because the data used for analysis (such as TCGA data) can include information about genomics relative to phenotypic information, the system as described herein can also be trained on data about phenotypic information (such as the actual structure and / or properties of the tissue, cell size and / or shape). Such a system can be trained to provide results for genomic analysis. Therefore, once such a system is trained, the system is able to observe tissue phenotypic properties and provide the possibility that the tissue includes mutations that can be discovered by sequencing.

[0111] For validation purposes, thirty-four patients in a cohort may be designated, wherein the thirty-four WSIs are divided into a total of 859,037 blocks. Each of the 859,037 blocks may be used for validation purposes.

[0112] For testing purposes, one hundred and two patients in the cohort can be designated for testing purposes. Ninety-nine diagnostic WSIs can be obtained from the test set, providing 2,501,314 partitions. Each of the 2,501,314 partitions can be used for testing purposes.

[0113] Each number provided herein is provided for example only and should not be considered limiting in any way.

[0114] Fig. 6A and Figure 6B Demonstrated image-based microbiome detection and response to therapy in clinical trials. Fig. 6A In the study, a correlation between the probability of a high microbiome at screening and the probability of complete remission (CR) was demonstrated. Figure 6B In the study, we showed a correlation between the probability of a high microbiome after screening (e.g., 12-27 weeks after screening) and the probability of CR.

[0115] Once the CNN is tested and trained using the system 200 as described above, Figure 1 and Figure 2 The computer system 103 shown can be used to perform a method for classifying new WSI using microbiome signatures. This method can be as follows Figure 7 As shown in the flowchart of and can start at 700, the computer system 103 can start the trained CNN 218. The trained CNN can be used as Figure 2 The trained CNN may be a process executed by the processor 106 of the computer system 103.

[0116] At 703, computer system 103 may receive raw input data. The raw input data may be, for example, as described herein and as described herein. FIG. 3A to FIG. 3H The raw input data can be obtained by Figure 1 The network shown receives. The computer system 103 is capable of receiving multiple WSIs. Each WSI can be associated with the same or different patients. For example, two or more WSIs can each relate to the same patient. Each WSI can include metadata and / or other information describing the patient associated with the WSI, or be associated with the metadata and / or other information. The computer system 103 is also or alternatively capable of receiving blocks created from the WSI. For example, another system can be used to divide the WSI into blocks. As described below, in step 706, the computer system 103 can divide the input WSI into blocks.

[0117] At 706, the computer system 103 may pre-process the input data (whether one or more WSIs or one or more blocks) to use it as a proxy for the following example: Figure 2 The input of the trained CNN218 is shown. Preprocessing the input can be automatic, that is, no human interaction is required. Preprocessing can include one or more of the following: adjusting image quality, contrast, size, brightness, orientation, etc. Preprocessing can also include dividing the input WSI into multiple tiles as described herein. In some embodiments, a single WSI can be divided into any number of tiles. As an example and not to be considered limiting in any way, each WSI can be divided into more than 25,000 tiles. Each tile created from the WSI can represent, for example, 100 μm 2 area.

[0118] At 709, the processed input data may be fed as input to a processor as described above with respect to Figure 2 The trained CNN 218 discussed. For example, each of the plurality of blocks may be individually fed to the trained CNN 218 as input. Based on the input, the trained CNN 218 may perform analysis and generate output.

[0119] Analyzing the WSI using a CNN may include separately analyzing multiple tiles associated with the WSI using a CNN. Analyzing the tiles using a trained CNN 218 may include obtaining the tiles from the input layer of the trained CNN 218 and feeding the tiles into the neural network. A series of convolutional layers of the CNN 218 may then apply a set of filters (or kernels) to the input tiles or the output of the previous layer to create a feature map representing higher-level features in the tiles. The filters are convolved across the input image or feature map to capture local patterns and spatial information.

[0120] CNN 218 can use a pooling layer to reduce the spatial dimension of the feature map. The pooling layer of CNN 218 can use one or more pooling techniques, such as maximum pooling and average pooling. CNN 218 can also use a fully connected layer to combine the learned features and generate a final output. The output layer of CNN 218 can be configured to generate an upper probability distribution of the categories microbiome low and microbiome high for the input block.

[0121] At 712, the computing system may receive the output of the trained CNN 218. In some embodiments, the output of the trained CNN 218 may be a microbiome label indicating whether the tile or WSI is microbiome low or microbiome high. For example, the trained CNN 218 may be configured to output a tile-based result, such as a microbiome label, for each tile created from a particular WSI. In this way, the trained CNN 218 may create a separate output for each of the multiple tiles fed as input.

[0122] At 715, the output of the AI ​​can be post-processed. The post-processing can include determining a microbiome label for the input WSI based on each microbiome label output for each of the plurality of tiles. The microbiome label for the input image can include an aggregation of each microbiome label for each of the plurality of tiles.

[0123] In some embodiments, aggregation can include performing or executing a majority voting mechanism. Alternatively, aggregation can be performed by averaging the probabilities, calculating the standard deviation, and defining the predicted value as fuzzy if the standard deviation around the average probability value overlaps with the determined optimal classification threshold. The microbiome label can be determined based on the aggregation of multiple tile processing results from the received image. In some embodiments, such as Figure 2 The illustrated majority voting mechanism 230 may be used to aggregate the tile-level results generated by the trained CNN 218 into WSI-level predicted labels for testing purposes, as described below.

[0124] At 718, the post-processed output of the AI ​​can be used to determine the level or amount of cancer in the input image data. In some embodiments, it can be determined whether the WSI contains any microbiome levels. In response to determining that the image contains a high microbiome level, a high microbiome label can be assigned to the WSI. In response to determining that the image does not contain a microbiome level, a low microbiome label can be assigned to the image.

[0125] The microbiome signature can be an identification of a microbial signature in tissue and / or blood found by the CNN in the input image data. Based on the microbiome signature, the computer system can be configured to automatically diagnose a patient with one or more of head and neck cancer, ovarian cancer, and bladder cancer.

[0126] In some embodiments, the output of the CNN can be compared to the patient's previous cancer level associated with the WSI. For example, the computer system can be configured to determine growth / shrinkage rates, or determine the efficacy of a treatment for an immediate patient or a large number of patients.

[0127] Based on the output of the CNN, the computer system can be configured to determine an estimated survival rate, diagnosis, and / or treatment or prescription for a patient associated with the WSI. In some embodiments, based on the output of the CNN, the computer system can be configured to compare different treatments for different patients over time. At 521, the method can end.

[0128] The systems or methods as described herein can be used in a variety of ways. In some embodiments, the systems and methods described herein can be used in the context of drug trials. For example, using a system as described herein, the results generated by a CNN can be automatically displayed on a graphical user interface that can be updated in real time with new results. Such a system can enable a single laboratory with any number of patients around the world to share results with patients and doctors simultaneously.

[0129] In one or more of the embodiments described herein, the system can be used for patient monitoring. For example, based on the results from the CNN, real-time notifications can be transmitted to doctors and / or patients based on the results from the CNN. Patients can be imaged in local clinics, and image data can be remotely transmitted to laboratories without the need for patients to go to any particular laboratory. In this way, patients may not need to make appointments with radiologists or specialist physicians, thereby saving time and money. Systems as described herein can be used to administer treatment to patients based on diagnoses created using CNN. Diagnosing a patient can include, for example, calculating the likelihood of a patient's head and neck cancer, ovarian cancer, and bladder cancer.

[0130] The systems and methods described herein can be used to implement a tile-based microbiome detection system. These systems and methods can be used to classify one or more tiles of a frozen tissue image and aggregate the tile-based results to label the frozen tissue image. If a patient is associated with multiple WSIs, each result can be aggregated to produce a single patient-level label.

[0131] As described herein, AI-driven image-based microbiome analysis can reveal a correlation between higher pre-treatment microbiome levels and complete remission in trial patients. Because the human microbiome exerts a profound influence on cancer development, progression, and response to treatment, the presence and level of the microbiome in H&E-stained slides as described herein can be used to determine the level or severity of cancer in a patient and predict patient survival.

[0132] The AI-driven computational pathology system as described herein can be configured to determine relative microbiome levels in H&E stained slides. The system can be further configured to classify WSI as microbiome low or microbiome high.

[0133] The system as described herein can be trained using FFPE diagnostic (DX) images from the TCGA bladder cancer cohort (n=408). The system can be used to determine the correlation between survival and microbiome levels of TCGA bladder cancer patients.

[0134] The system as described herein can be used to analyze microbiome levels in patients with pathological complete response (pCR) and non-muscle invasive bladder cancer in patient trials.

[0135] Using a system as described herein, a microbiome level detector that computes tumor tissue-specific images can be developed for bladder cancer. The performance of the microbiome level detector for H&E-stained FFPE tissues shows that the morphological changes introduced by the microbiome into solid tumor tissues are visible to the AI ​​image system. Although the morphological changes are too subtle to be easily seen by eye even by trained pathologists, these changes are detectable for the AI-based image system developed as described herein. Using a system as described herein, an analysis was performed on the relationship between the survival rate of TCGA bladder patients and the microbiome load, and it was seen that the survival of patients with low tumors of the microbiome was extended.

[0136] Further, using a system as described herein, the correlation between pathological complete remission (pCR) at 12 or 27 weeks in a clinical trial and microbiome levels in patients with non-muscle invasive bladder cancer can be evaluated. It has been found that patients who achieve pCR with treatment have a significantly higher probability of high pre-treatment of the microbiome than those who do not achieve pCR. A reduction in the predicted microbiome levels after treatment can also be observed in pCR patients. In contrast, it was found that 6 of the 7 non-responders had an increase in the high probability of the microbiome with treatment. This finding suggests that the novel AI-driven image-based computational pathology system as described herein has the potential to provide data that can not only inform clinical decisions, but also allow further study of the role of the TME microbiome in cancer.

[0137] The systems and methods and / or uses described herein can be used to assess a patient's response to a cancer treatment at more than one time point. For example, the response to treatment can be assessed at intervals of about 1 week, 2 weeks, 3 weeks, 4 weeks, 1 month, 2 months, 3 months, 4 months, 5 months, 6 months, 7 months, 8 months, 9 months, 10 months, 11 months, 12 months, 13 months, 15 months, 18 months, 2 years, 3 years, 4 years, or longer.

[0138] In addition, the patient's response to the cancer treatment can be assessed at different time points before or during treatment. In certain embodiments, the level of the microbiome is determined as disclosed herein before or before the cancer treatment is administered to the patient. The level is then compared to the level of the microbiome determined as disclosed herein after or after the cancer treatment is administered to the patient. The level of the microbiome can be assessed about 1 week, 2 weeks, 3 weeks, 4 weeks, 1 month, 2 months, 3 months, 4 months, 5 months, 6 months, 7 months, 8 months, 9 months, 10 months, 11 months, 12 months, 13 months, 15 months, 18 months, 2 years, 3 years, 4 years or longer after the cancer treatment begins.

[0139] In certain embodiments, the patient's response to the cancer treatment can be assessed after the cancer treatment begins, for example, to estimate or evaluate the effectiveness of the cancer treatment. For example, the patient's response to the cancer treatment can be assessed again as disclosed herein before the cancer treatment begins and then at a second time point after the cancer treatment begins. The level of the microbiome determined by the methods and / or uses disclosed herein can then be compared with the cancer treatment efficacy. The cancer treatment efficacy assessment can then be used to decide whether to continue or terminate the cancer treatment. In another example, the patient's response to the cancer treatment can be assessed at a time point after the cancer treatment, and then the patient's response to the cancer treatment can be assessed again at one or more additional time points after additional rounds of cancer treatment. The microbiome levels determined at each time point by the methods and / or uses disclosed herein can then be compared. The cancer treatment efficacy assessment can then be used to decide whether to continue or terminate the cancer treatment.

[0140] In one aspect, the patient has not yet received cancer treatment. In another aspect, the patient has been receiving cancer treatment for a certain period of time.

[0141] The phrases "cancer therapy" or "anticancer therapy" are used interchangeably to refer to a therapy used to treat cancer. Examples of anticancer therapeutic agents include, but are not limited to, surgery, chemotherapeutic agents, immunotherapy, growth inhibitory agents, cytotoxic agents, radiation therapy, anti-angiogenic agents, apoptotic agents, anti-tubulin agents, and other agents for treating cancer, such as anti-HER-2 antibodies (e.g., HERCEPTIN TM ), anti-CD20 antibodies, epidermal growth factor receptor (EGFR) antagonists (e.g., tyrosine kinase inhibitors), HER1 / EGFR inhibitors (e.g., erlotinib (TARCEVA TM )), platelet-derived growth factor inhibitors (eg, GLEEVEC TM (imatinib mesylate)), COX-2 inhibitors (e.g., celecoxib), interferons, cytokines, antagonists (e.g., neutralizing antibodies) that bind to one or more of the following targets ErbB2, ErbB3, ErbB4, PDGFR-β, BlyS, APRIL, BCMA or VEGF receptors, TRAIL / Apo2 and other biologically active and organic chemical agents, etc. Combinations of these therapies and therapeutic agents are also contemplated for use within the methods described herein.

[0142] A "chemotherapeutic agent" is a chemical compound that can be used to treat cancer. Examples of chemotherapeutic agents include erlotinib (TARCEVA TM , Genentech / OSI Pharmaceuticals, Inc. (OSI Pharm.), bortezomib (VELCADE TM, Millennium Pharm.), FASLODEX TM , AstraZeneca), sunitinib (SU11248, Pfizer), letrozole (FEMARA TM , Novartis), imatinib mesylate (GLEEVEC TM , Novartis), PTK787 / ZK222584 (Novartis), oxaliplatin (Eloxatin TM , Sanofi), 5-FU (5-fluorouracil), leucovorin, rapamycin (sirolimus, RAPAMUNE TM , Wyeth), lapatinib (GSK572016, GlaxoSmithKline), lonafarnib (SCH 66336), sorafenib (BAY43-9006, Bayer Labs.), and gefitinib (IRESSA TM , AstraZeneca), AG1478, AG1571 (SU5271; Sugen), alkylating agents such as thiotepa and CYTOXAN TMCyclophosphamide; alkyl sulfonates, such as busulfan, improsulfan, and piposulfan; aziridines, such as benzodopa, carboquinone, meturedopa, and uredopa; ethyleneimines and methylamines, including altretamine, triethylenemelamine, triethylenephosphoramide, triethylenethiophosphoramide, and triethylenethiophosphoramide. amide and trimethylomelamine; acetogenins (especially bullatacin and bullatacinone); camptothecins (including the synthetic analog topotecan); bryostatin; callystatin; CC-1065 (including its synthetic analogs adozcicsin, carzcicsin and bizcicsin); cryptophycins (specifically cryptophycin 1 and cryptophycin 8); dolastatin; duocarmycin (including synthetic analogs, KW-2189 and CB1-TM1); eleutherobin; pancratistatin; sarcodictyin; spongistatin; nitrogen mustards, such as chlorambucil, naphthyl mustard, cholophosphamide, estramustine, ifosfamide, nitrogen mustard, mechlorethamine oxide hydrochloride, mefenamic acid, len, novembichin, phenesterine, prednimustine, trofosfamide, uramustine; nitrosoureas, such as carmustine, chlorozotocin, fotemustine, lomustine, nimustine, and ranimnustine; antibiotics, such as enediyne antibiotics (e.g., calicheamicin, especially calicheamicin gamma 1 and calicheamicin omega 1 (Angew Chem. Intl. Ed. Engl [Applied Chemistry English International Edition]).(1994) 33:183-186); dynemicins, including dynemicin A; bisphosphonates, such as clodronate; esperamicins; and neocarzinostatin chromophores and related chromoprotein enediyne antibiotic chromophores), aclacinomycins, actinomycins, authramycins, azaserine, bleomycins, actinomycin C, carabicin, carminomicin, carzinophilin, chromomycinis, actinomycin D, daunomycin, detorubicin, 6-diazo-5-oxo-L-norleucine, ADRIAMYCIN. TMDoxorubicin (including morpholino-doxorubicin, cyanomorpholino-doxorubicin, 2-pyrrolino-doxorubicin and deoxydoxorubicin), epirubicin, esorubicin, idarubicin, marcellomycin, mitomycins such as mitomycin C, mycophenolic acid, nogamycin, olivomycins, peplomycin, potfiromycin, puromycin, quelamycin, rhodoxorubicin, rodorubicin, streptonigrin, streptozocin, tubercidin, ubenimex, zinostatin, zorubicin; antimetabolites such as methotrexate and 5-fluorouracil (5-FU); folic acid analogs such as denopterin, methotrexate, pteropterin, trimetrexate; purine analogs such as fludarabine, 6-mercaptopurine, thiopurine, thioguanine; pyrimidine analogs such as ancitabine, azacitidine, 6-azauridine, carmofur, cytarabine, dideoxyuridine, doxifluridine, enocitabine, floxuridine, loxuridine; androgens, such as calusterone, dromostanolone propionate, epitiostanol, mepitiostane, and testolactone; antiadreners, such as aminoglutethimide, mitotane, and trilostane; folic acid supplements, such as frolinic acid; aceglatone; aldophosphamide glycoside; aminolevulinic acid; eniluracil; amsacrine; bestrabucil; bisantrene; edatraxate;defofamine; demecolcine; diaziquone; elfornithine; elliptinium acetate; epothilone; etoglucid; gallium nitrate nitrate; hydroxyurea; lentinan; lonidainine; maytansinoids, such as maytansine and ansamitocin; mitoguazone; mitoxantrone; mopidanmol; nitraerine; pentostatin; phenamet; pirarubicin; losoxantrone; podophyllinic acid; 2-ethylhydrazide; procarbazine; PSK; TM Polysaccharide complex (JHS Natural Products, Eugene, OR); razoxane; rhizoxin; sizofuran; spirogermanium; tenuazonic acid acid; triaziquone; 2,2',2"-trichlorotriethylamine; trichothecenes (specifically T-2 toxin, verracurin A, baculocin A, and serpentin); urethan; vindesine; dacarbazine; mannomustine; mitobronitol; mitolactol; pipobroman; gasytosinc; arabinoside ("Ara-C"); cyclophosphamide; thiotepa; taxanes, such as TAXOL TM Paclitaxel (Bristol-Myers Squibb Oncology, Princeton, NJ), ABRAXANE TM(without Cremophor), albumin-engineered nanoparticle formulation of paclitaxel (American Pharmaceutical Partners, Schaumberg, IL), and TAXOTERE TM Docetaxel (Rhone-Poulenc Rorer, Antony, France); chloranbucil; GEMZAR TM gemcitabine; 6-thioguanine; mercaptopurine; methotrexate; platinum analogs, such as cisplatin and carboplatin; vinblastine; platinum; etoposide (VP-16); ifosfamide; mitoxantrone; vincristine; NAVELBINE TM Vinorelbine; novantrone; teniposide; edatrexate; daunomycin; aminopterin; xeloda; ibandronate; CPT-11; topoisomerase inhibitor RFS2000; difluoromethylornithine (DMFO); retinoids, such as retinoic acid; capecitabine; and pharmaceutically acceptable salts, acids or derivatives of any of the above.

[0143] Also included within this definition of "chemotherapeutic agent" are: (i) antihormonal agents that modulate or inhibit the effects of hormones on tumors, such as antiestrogens and selective estrogen receptor modulators (SERMs), including, for example, tamoxifen (including NOLVADEX TM (tamoxifen), raloxifene, droloxifene, 4-hydroxytamoxifen, troloxifene, keoxifene, LY117018, onapristone, and FARESTON TM (toremifene); (ii) aromatase inhibitors that inhibit aromatase, which regulates estrogen production in the adrenal glands, e.g., 4(5)-imidazole, aminoglutethimide, MEGASE TM (Megestrol acetate), AROMASIN TM (exemestane), formestane, fadrozole, RIVISOR TM (Furozole), FEMARA TM (letrozole) and ARIMIDEX TM(anastrozole); (iii) antiandrogens, such as flutamide, nilutamide, bicalutamide, leuprolide, and goserelin; and troxacitabine (a 1,3-dioxolane nucleoside cytosine analog); (iv) aromatase inhibitors; (v) protein kinase inhibitors; (vi) lipid kinase inhibitors; (vii) antisense oligonucleotides, particularly those that inhibit the expression of genes in signal transduction pathways involved in abnormal cell proliferation (e.g., PKC-α, Ralf, and H-Ras); (viii) ribozymes, such as inhibitors of VEGF expression (e.g., ANGIOZYME TM (ribozymes) and inhibitors of HER2 expression; (ix) vaccines, such as gene therapy vaccines, for example ALLOVECTIN TM Vaccines, LEUVECTIN TM Vaccines and VAXIDs TM Vaccine; PROLEUKIN TM rIL-2;LURTOTECAN TM Topoisomerase 1 inhibitor; ABARELIX TM rmRH; (x) anti-angiogenic agents, such as bevacizumab (AVASTIN TM , Genentech); and (xi) a pharmaceutically acceptable salt, acid or derivative of any of the above.

[0144] Treatment for cancer patients may include any of the following: adjuvant therapy (also called adjuvant therapy or auxiliary therapy) to eliminate residual tumor cells that may be present after a known tumor is removed by initial therapy (such as surgery) to prevent possible recurrence of the cancer; neoadjuvant therapy given before surgery to shrink the cancer; induction therapy to induce remission, usually for acute leukemias; consolidation therapy (also called intensification therapy) given once remission is achieved to maintain remission; maintenance therapy given at a lower or less frequent dose to help prolong remission; first-line therapy (also called standard therapy); second (or 3rd, 4th, etc.) line therapy (also called salvage therapy) given if the disease does not respond or relapses after first-line treatment.

[0145] As used herein, "tumor" means a mass of transformed cells involving uncontrolled cell proliferation of neoplastic nature and, at least in part, the inclusion of angiogenic vasculature. Abnormal tumor cells grow rapidly and continue even after the stimulus that triggers new growth has ceased. "Tumor" is broadly used to include tumor parenchymal cells as well as supporting matrix, including angiogenic blood vessels that infiltrate tumor parenchymal cell masses. Although tumors are typically malignant tumors, i.e., cancers with the ability to metastasize (i.e., metastatic tumors), tumors can also be non-malignant (i.e., non-metastatic tumors).

[0146] "Patient" or "individual" or "subject" are used interchangeably herein and refer to a mammalian subject, typically a human patient, to be treated.

[0147] The systems and methods described herein can also be extended in other ways. For example, instead of modeling the problem as a classification problem and predicting a label, the problem can be modeled as a regression task, where the goal can be to predict the number of microbial groups in a pathology image.

[0148] Additionally, cancers may be enriched in several microbial signatures. In the future, the microbiome population of each cancer could be studied from the images alone. Thus, instead of starting with a binary true label (microbiome high vs. microbiome low), a microbiome-specific analysis could be performed, such as whether microbiome X is present in the image data or not, or some combination thereof. This analysis could be used to determine the optimal microbiome environment for immune stimulation.

[0149] Still further, the systems and methods and / or uses disclosed herein can further include integrating supplemental data to enhance images associated with the patient (i.e., pathology slides). Such supplemental data can be obtained, for example, from RNA sequencing (RNA-seq). Further, for example, the supplemental data can come from a vector consisting of several transcript expression levels, which can be processed as parallel inputs into a neural network that can encode the supplemental information, thereby improving the accuracy or ability of the system to summarize patient samples, tissues, and cancer types.

[0150] Any of the steps, functions, and operations discussed herein can be performed continuously and automatically.

[0151] Exemplary systems and methods and / or uses of the present invention have been described with respect to head and neck cancer, ovarian cancer, and bladder cancer, as well as components and methods for analyzing and determining the presence of a microbiome in a WSI. However, in order to avoid unnecessarily obscuring the present invention, the foregoing description omits many known structures and devices. Such omissions should not be construed as limiting the scope of the claimed invention. Specific details are set forth to provide an understanding of the present invention. However, it should be appreciated that the present invention can be practiced in a variety of ways, without being limited to the specific details set forth herein.

[0152] In addition, although the exemplary embodiments shown herein show various system components as being co-located, some components of the system may be remotely located in a distant part of a distributed network such as a LAN and / or the Internet, or in a dedicated system. Therefore, it should be recognized that the components of the system or parts thereof (e.g., microprocessors, memory / storage devices, interfaces, etc.) may be combined into one or more devices (e.g., a server, multiple servers, computers, computing devices, terminals, "clouds" or other distributed processing), or co-located on a specific node of a distributed network (e.g., an analog and / or digital telecommunications network, a packet switching network, or a circuit switching network). In another embodiment, these components may be physically or logically distributed across multiple components (e.g., a microprocessor may include a first microprocessor on one component and a second microprocessor on another component, each microprocessor performing a shared task and / or a portion of an assigned task). Based on the foregoing description and for reasons of computational efficiency, it should be recognized that the components of the system may be arranged at any location within the distributed network of components without affecting the operation of the system. For example, various components may be located in a switch (e.g., a PBX and a media server, a gateway), in one or more communication devices, in one or more user residences, or in some combination thereof. Similarly, one or more functional portions of the system may be distributed between telecommunication device(s) and associated computing devices.

[0153] In addition, it should be appreciated that the various links connecting the elements may be wired or wireless links, or any combination thereof, or any other known or later developed element(s) capable of supplying data to and / or transmitting data from the connected elements. These wired or wireless links may also be secure links and capable of transmitting encrypted information. For example, the transmission medium used as the link may be any suitable carrier of an electrical signal (including coaxial cable, copper wire, and optical fiber), and may take the form of sound or light waves, such as those generated during radio wave and infrared data communications.

[0154] Furthermore, while the flow charts have been discussed and illustrated with respect to a particular sequence of events, it should be appreciated that changes, additions, and omissions may be made to the sequence without materially affecting the operation of the invention.

[0155] Many variations and modifications of the invention may be used. Some features of the invention may be provided without providing other features.

[0156] In yet another embodiment, the systems and methods and / or uses of the present invention may be implemented in conjunction with a special purpose computer, a programmed microprocessor or microcontroller and (multiple) peripheral integrated circuit components, an ASIC or other integrated circuit, a digital signal microprocessor, a hard-wired electronic or logic circuit (such as a discrete component circuit, a programmable logic device or a gate array (such as a PLD, PLA, FPGA, PAL)), a special purpose computer, any similar device, etc. In general, any (multiple) devices or devices capable of implementing the methods presented herein may be used to implement various aspects of the present invention. Exemplary hardware that may be used for the present invention includes computers, handheld devices, phones (e.g., cellular phones, Internet-enabled phones, digital phones, analog phones, hybrid phones, etc.), and other hardware known in the art. Some of these devices include a microprocessor (e.g., a single or multiple microprocessors), memory, non-volatile storage devices, input devices, and output devices. In addition, alternative software implementations may also be constructed to implement the methods described herein, including but not limited to distributed processing or component / object distributed processing, parallel processing, or virtual machine processing, such as provided by one or more processing components.

[0157] In yet another embodiment, the disclosed method can be easily implemented in conjunction with software using objects or object-oriented software development environments that provide portable source code that can be used on a variety of computer or workstation platforms. Alternatively, the disclosed system can be implemented partially or completely in hardware using standard logic circuits or VLSI designs. Whether software or hardware is used to implement the system according to the present invention depends on the speed and / or efficiency requirements of the system, the specific functions, and the specific software or hardware system or microprocessor or microcomputer system utilized.

[0158] In yet another embodiment, the disclosed method may be implemented in part in the form of software that may be stored on a storage medium and executed on a programmed general purpose computer, a special purpose computer, a microprocessor, etc. that cooperates with a controller and memory. In these cases, the system and method of the present invention may be implemented as a program (such as an applet, or CGI scripts), implemented as a resource resident on a server or computer workstation, implemented as a routine embedded in a dedicated measurement system or system component, etc. The system may also be implemented by physically incorporating the system and / or method into a software and / or hardware system.

[0159] The embodiments including software herein are executed or stored for subsequent execution by one or more microprocessors and executed as executable code. The executable code is selected to execute instructions including specific embodiments. The instructions executed are constrained instruction sets selected from a discrete set of native instructions understood by the microprocessor, and are handed over to the microprocessor accessible memory before execution. In another embodiment, before being executed by one or more microprocessors, human-readable "source code" software is first converted into system software to include a platform-specific instruction set selected from the native instruction set of the platform (e.g., computer, microprocessor, database, etc.).

[0160] A neural network as described herein may include a layer of logical nodes having inputs and outputs. If the output is below a self-determined threshold level, the output may be omitted (i.e., the input may be within the inactive response portion of the scale and no output is provided), and if the output is above the threshold, the output may be provided (i.e., the input may be within the active response portion of the scale and the output is provided). Specific placement of active and inactive depictions may be provided as one or more steps. Multiple inputs into a node may generate a multi-dimensional plane (e.g., a hyperplane) to depict combinations of active or inactive inputs.

[0161] Although the present invention describes the components and functions implemented in the embodiments with reference to specific standards and protocols, the present invention is not limited to such standards and protocols. Other similar standards and protocols not mentioned herein exist and are considered to be included in the present invention. In addition, the standards and protocols mentioned herein and other similar standards and protocols not mentioned herein are regularly replaced by faster or more efficient equivalents with substantially the same functions. Such replacement standards and protocols with the same functions are considered to be equivalents included in the present invention.

[0162] The present invention includes, in various embodiments, configurations and aspects, components, methods, uses, processes, systems and / or devices substantially as depicted and described herein, including various embodiments, subcombinations and subsets thereof. After understanding the present invention, those skilled in the art will understand how to proceed and use the present invention. The present invention includes, in various embodiments, configurations and aspects, providing devices and processes in the absence of items not depicted and / or described herein or in its various embodiments, configurations or aspects (including in the absence of such items that may have been used in previous devices or processes), such as to improve performance, achieve ease of use and / or reduce implementation costs.

[0163] The foregoing discussion of the present invention is presented for the purpose of illustration and description. The foregoing is not intended to limit the present invention to one or more forms disclosed herein. For example, in the foregoing detailed description, various features of the present disclosure are combined together in one or more embodiments, configurations or aspects for the purpose of simplifying the present invention. Features of embodiments, configurations or aspects of the present invention may be combined in alternative embodiments, configurations or aspects other than those discussed above. This approach to the present disclosure should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly listed in each claim. Rather, as reflected in the following claims, the inventive aspects have less features than all the features of a single previously disclosed embodiment, configuration or aspect. Therefore, the appended claims are hereby incorporated into the present detailed description, with each claim existing independently as a separate preferred embodiment of the present invention.

[0164] In addition, although the description of the invention includes descriptions of one or more embodiments, configurations or aspects and certain variations and modifications, other variations, combinations and modifications are also within the scope of the invention, such as within the skill and knowledge of a person skilled in the art after understanding the present disclosure. It is intended to obtain rights that include alternative embodiments, configurations or aspects within the permitted scope, including alternative, interchangeable and / or equivalent structures, functions, ranges or steps of the claimed structures, functions, ranges or steps, regardless of whether such alternative, interchangeable and / or equivalent structures, functions, ranges or steps are disclosed herein, and it is not intended to publicly dedicate any patentable subject matter.

[0165] In the foregoing description, for the purpose of illustration, the method is described in a specific order. It should be appreciated that in alternative embodiments, without departing from the scope of the embodiment, the methods may be performed in an order different from the described order. It should also be appreciated that the above method may be performed as an algorithm executed by a hardware component (e.g., a circuit system), which is specially constructed to implement one or more algorithms described herein or parts thereof. In another embodiment, the hardware component may include a general-purpose microprocessor (e.g., CPU, GPU) that is first converted to a dedicated microprocessor. Then, the encoded signal has been loaded in the dedicated microprocessor, so that the dedicated microprocessor now maintains the machine-readable instruction so that the microprocessor can read and execute the machine-readable instruction set obtained from the algorithm and / or other instructions described herein. The machine-readable instructions for executing (multiple) algorithms or parts thereof may be limited, but using a limited instruction set known to the microprocessor. In one or more embodiments, the machine-readable instruction can be encoded in the microprocessor by the voltage in the memory circuit, the configuration of the switch circuit, and / or by selectively using a specific logic gate circuit as a signal or value in the signal generating component. Additionally or alternatively, machine-readable instructions may be accessible to a microprocessor and encoded in a medium or device as a magnetic field, voltage value, charge value, reflective / non-reflective portion, and / or physical marking.

[0166] In another embodiment, the microprocessor further comprises one or more of the following: a single microprocessor, a multi-core processor, a plurality of microprocessors, a distributed processing system (e.g., arrays, blades, server farms, "clouds", multi-function processor arrays, clusters, etc.), and / or may be co-located with microprocessors that perform other processing operations. Any one or more microprocessors may be integrated into a single processing device (e.g., a computer, server, blade, etc.), or may be located in whole or in part in discrete components and connected via a communication link (e.g., a bus, a network, a backplane, etc., or multiple thereof).

[0167] Examples of general purpose microprocessors may include a central processing unit (CPU) having data values ​​encoded in an instruction register (or other circuitry for maintaining instructions) or including data values ​​of memory locations, which in turn include values ​​used as instructions. The memory locations may further include memory locations external to the CPU. Such CPU external components may be implemented as one or more of the following: a field programmable gate array (FPGA), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a random access memory (RAM), a bus accessible storage device, a network accessible storage device, etc.

[0168] These machine executable instructions may be stored on one or more machine readable media, such as CD-ROM or other types of optical disks, floppy disks, ROM, RAM, EPROM, EEPROM, magnetic or optical cards, flash memory or other types of machine readable media suitable for storing electronic instructions. Alternatively, these methods may be performed by a combination of hardware and software.

[0169] In another embodiment, the microprocessor can be a system or collection of processing hardware components, such as a microprocessor on a client device and a microprocessor on a server, a collection of devices and their corresponding microprocessors, or a shared or remote processing service (e.g., a "cloud" based microprocessor). The microprocessor system can include task-specific allocations for processing tasks and / or shared or distributed processing tasks. In yet another embodiment, the microprocessor can execute software to provide services that simulate one or more different microprocessors. Therefore, a first microprocessor consisting of a first set of hardware components can virtually provide the services of a second microprocessor, whereby the hardware associated with the first microprocessor can operate using an instruction set associated with the second microprocessor.

[0170] While machine-executable instructions may be stored and executed locally on a particular machine (e.g., a personal computer, mobile computing device, laptop computer, etc.), it should be recognized that storage of data and / or instructions and / or execution of at least a portion of the instructions may be provided via a connection to a remote data storage device and / or processing device or collection of devices (commonly referred to as the "cloud"), which may include public, private, dedicated, shared and / or other service providers, computing services and / or "server farms."

[0171] Examples of microprocessors as described herein may include, but are not limited to, at least one of the following: 800 and 801, with 4G LTE integration and 64-bit computing 610 and 615, with 64-bit architecture A7 microprocessor, M7 motion microprocessor, series, Core TM Microprocessor series, Microprocessor series, Atom TM Microprocessor series, Intel Microprocessor series, i5-4670K and i7-4770K 22nm Haswell, i5-3570K 22nm Ivy Bridge, FX TM Microprocessor series, FX-4300, FX-6300 and FX-8350 32nm Vishera, Kaveri microprocessor, Texas Jacinto C6000 TM Automotive infotainment microprocessors, Texas OMAP TM Automotive-grade mobile microprocessors, Cortex TM -M microprocessor, Cortex-A and ARM926EJ-S TM microprocessor, other industry equivalent microprocessor, and may use any known or future developed standard, instruction set, library and / or architecture to perform computing functions.

[0172] Embodiments include a method of assessing a patient's response to a cancer therapy in a patient having cancer, the method comprising: receiving a first image associated with the patient before administration of the cancer therapy, and automatically determining a microbiome level in the first image using a neural network; receiving a second image associated with the patient after administration of the cancer therapy, and automatically determining a microbiome level in the second image using a neural network; comparing the microbiome level in the first image with the microbiome level in the second image, wherein a decrease in the microbiome level in the first image compared to the microbiome level in the second image indicates that the patient is responding to the cancer therapy, and wherein an increase in the microbiome level in the first image compared to the microbiome level in the second image indicates that the patient is not responding to the cancer therapy and continuing to treat the patient using the cancer therapy if there is a decrease in the microbiome level in the first image compared to the microbiome level in the second image, or adjusting the cancer therapy if there is an increase in the microbiome level in the first image compared to the microbiome level in the second image.

[0173] Embodiments further include a computer system for assessing a patient's response to a cancer therapy in a patient having cancer, the system comprising: a processor; and a computer-readable storage medium storing computer-readable instructions that, when executed by the processor, cause the processor to: receive a first image associated with the patient before administration of the cancer therapy, and automatically determine a microbiome level in the first image using a neural network; receive a second image associated with the patient after administration of the cancer therapy, and automatically determine a microbiome level in the second image using a neural network; compare the microbiome level in the first image with the microbiome level in the second image, wherein a decrease in the microbiome level in the first image compared to the microbiome level in the second image indicates that the patient is responding to the cancer therapy, and wherein an increase in the microbiome level in the first image compared to the microbiome level in the second image indicates that the patient is not responding to the cancer therapy; and continue to treat the patient using the cancer therapy if there is a decrease in the microbiome level in the first image compared to the microbiome level in the second image, or adjust the cancer therapy if there is an increase in the microbiome level in the first image compared to the microbiome level in the second image.

[0174] Embodiments further include a computer system configured to: execute a convolutional neural network to detect microbiome levels in a whole slide image associated with a patient; classify the whole slide image as one of microbiome low and microbiome high based on an output of the convolutional neural network; and determine a characteristic of the cancer based on the classification of the whole slide image.

[0175] Aspects of the above methods and computer systems include where the whole slide image comprises a hematoxylin and eosin (H&E) stained pathology slide.

[0176] Aspects of the above methods and computer systems include: wherein the cancer is selected from the group consisting of: head cancer, neck cancer, ovarian cancer, and bladder cancer.

[0177] Aspects of the above methods and computer systems include where the convolutional neural network is trained based on FFPE diagnostic images from the Cancer Genome Atlas Bladder Cohort.

[0178] Aspects of the above-mentioned computer system include: wherein the output of the convolutional neural network is used to determine patient survival statistics.

[0179] Aspects of the above computer system include wherein microbiome low is associated with a microbiome level of less than about 390,000 and microbiome high is associated with a microbiome level of greater than about 390,000.

[0180] Aspects of the above-described computer system include where the instructions further cause the processor to: determine that the patient is responding to the cancer therapy based on the classification of the whole slide image, and continue to treat the patient with the cancer therapy in response to determining that the patient is responding to the cancer therapy.

[0181] Aspects of the above-described computer system include: wherein the instructions further cause the processor to determine a pathological complete response (pCR) based on the output of the convolutional neural network.

[0182] Embodiments include a method of assessing a patient's response to a cancer treatment in a patient having cancer, the method comprising: executing a convolutional neural network to detect a microbiome level in a whole slide image associated with the patient; classifying the whole slide image as one of microbiome low and microbiome high based on an output of the convolutional neural network; and determining a characteristic of the cancer based on the classification of the whole slide image.

[0183] Aspects of the above method and computer system include: wherein the microbiome level in the image is determined based on an aggregation of processing results from a plurality of tiles from a received first image and a second image. Aspects of the above method and computer system include: wherein the first image and the second image comprise whole slide images (WSIs). Aspects of the above method and computer system include: wherein the input to the one or more neural networks comprises a plurality of tiles extracted from the WSIs. Aspects of the above method and computer system include: wherein the first image and the second image comprise 400×400 pixel images with 40 times magnification. Aspects of the above method and computer system include: wherein each of the plurality of tiles has a size of 100×100 microns. Aspects of the above method and computer system include: wherein each of the plurality of tiles comprises a tile of one hundred square microns. Aspects of the above method and computer system include: wherein the neural network outputs a microbiome label for each of the plurality of tiles. Aspects of the above methods and computer systems include: wherein the level of the microbiome in the image includes an aggregation of each microbiome label for each of the multiple blocks. Aspects of the above methods and computer systems include: wherein the aggregation includes a majority voting mechanism. Aspects of the above methods and computer systems include: wherein the first image and the second image each include a frozen tissue image. Aspects of the above methods and computer systems include: wherein the neural network is a convolutional neural network. Aspects of the above methods and computer systems include: wherein the convolutional neural network is trained with images from the Cancer Genome Atlas (TCGA). Aspects of the above methods and computer systems include: wherein the level of the microbiome is associated with microbial signatures in the tissue and blood shown in the image. Aspects of the above methods and computer systems include: automatically determining the level of cancer in the image includes calculating the severity of the patient's cancer.

Claims

1. A computer system for evaluating a patient's response to cancer treatment, wherein: The patient has cancer and the system includes: Processor; and A computer readable storage medium storing computer readable instructions that, when executed by a processor, cause the processor to: executing a convolutional neural network to detect microbiome levels in whole slide images associated with the patient; classifying the whole slide image as one of microbiome low and microbiome high based on an output of the convolutional neural network; and Based on the classification of the whole slide image, a characteristic of the cancer associated with the patient is determined.

2. The system of claim 1, wherein: This whole slide image includes a hematoxylin and eosin (H&E) stained pathology slide.

3. The system of claim 1, wherein: The cancer is bladder cancer.

4. The system of claim 3, wherein: The convolutional neural network was trained on formalin-fixed paraffin-embedded (FFPE) diagnostic images from a bladder cohort.

5. The system of claim 1, wherein: The output of this convolutional neural network is used to determine the survival statistics for this patient.

6. The system of claim 1, wherein: Microbiome low is associated with a microbiome level that is less than the median microbiome level for the group, and microbiome high is associated with a microbiome level that is greater than the median microbiome level for the group.

7. The system of claim 1, wherein: These instructions further enable the processor to: determining whether the patient is responding to the cancer treatment based on the classification of the whole slide image; as well as In response to determining that the patient is responding to the cancer therapy, continuing to treat the patient with the cancer therapy.

8. The system of claim 1, wherein: The instructions further cause the processor to determine a pathological complete response (pCR) based on the output of the convolutional neural network.

9. A method for evaluating a patient's response to cancer treatment, wherein: The patient has cancer and the method comprises: executing a convolutional neural network to detect microbiome levels in whole slide images associated with the patient; classifying the whole slide image as one of microbiome low and microbiome high based on an output of the convolutional neural network; determining a characteristic of a cancer associated with the patient based on the classification of the whole slide image; determining that the patient is responding to the cancer treatment based on the classification of the whole slide image; and In response to determining that the patient is responding to the cancer therapy, continuing to treat the patient with the cancer therapy.

10. The method of claim 9, wherein: This whole slide image includes a hematoxylin and eosin (H&E) stained pathology slide.

11. The method of claim 9, wherein: The cancer is bladder cancer.

12. The method of claim 11, wherein: The convolutional neural network was trained on formalin-fixed paraffin-embedded (FFPE) diagnostic images from a bladder cohort.

13. The method of claim 9, wherein: The output of this convolutional neural network is used to determine patient survival statistics.

14. The method of claim 9, wherein: Microbiome low is associated with a microbiome level that is less than the median microbiome level for the group, and microbiome high is associated with a microbiome level that is greater than the median microbiome level for the group.

15. The method of claim 9, further comprising determining a pathological complete response (pCR) based on an output of the convolutional neural network.

16. At least one machine-readable non-transitory medium, the at least one machine-readable non-transitory medium comprising a plurality of instructions, the plurality of instructions being executed on a computing device to facilitate the computing device: Convolutional neural networks were implemented to detect microbiome levels in whole-slide images associated with patients; classifying the whole slide image as one of microbiome low and microbiome high based on an output of the convolutional neural network; and Based on the classification of the whole slide image, a characteristic of the cancer associated with the patient is determined.

17. The machine-readable non-transitory medium of claim 16, wherein: This whole slide image includes a hematoxylin and eosin (H&E) stained pathology slide.

18. The machine-readable non-transitory medium of claim 16, wherein: The cancer is bladder cancer.

19. The machine-readable non-transitory medium of claim 18, wherein: The convolutional neural network was trained on formalin-fixed paraffin-embedded (FFPE) diagnostic images from a bladder cohort.

20. The machine-readable non-transitory medium of claim 16, wherein: The output of this convolutional neural network is used to determine the survival statistics for this patient.

Images