System and method for analyzing microbiome using artificial intelligence

Abstract

The disclosed system and method include the steps of executing a convolutional neural network to detect the level of microbiota in a whole slide image associated with a patient, classifying the whole slide image as either low microbiota or high microbiota based on the output of the convolutional neural network, and determining cancer characteristics associated with the patient based on the classification of the whole slide image.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 63 / 341,559, filed May 13, 2022, and U.S. Provisional Patent Application No. 63 / 395,163, filed Aug. 4, 2022. The entire disclosures of U.S. Provisional Patent Application No. 63 / 341,559 and U.S. Provisional Patent Application No. 63 / 395,163 are hereby incorporated by reference into this specification.

[0002] The present invention generally relates to systems and methods for analyzing image data, and more particularly to identifying the microbiome in whole - slide images and determining the relative microbiome levels in hematoxylin and eosin (H&E) stained images.

Background Art

[0003] The human microbiome deeply affects cancer and plays a role in the progression of carcinogenesis. The microbiome is a community of microorganisms (such as fungi, bacteria, and viruses) that exist in a specific environment. In humans, this term is often used to represent the microorganisms that inhabit the inside or surface of specific parts of the body, such as the skin or the digestive tract, or within specific regions such as the tumor microenvironment (TME). These microorganisms are dynamic and their numbers and identities change in response to many environmental factors such as exercise, diet, medication, and other exposures.

[0004] Recent studies have shown that microorganisms significantly contribute in selected cancer types, including, for example, cancer initiation, progression, and response to treatment, but the full extent of the contribution of microorganisms across diverse cancer types and its implications for diagnosis remain unclear.

[0005] For example, manual inspection of tissue and blood samples using a microscope is associated with problems such as inaccuracy, errors, and inefficiency. Manual inspection of tissue and blood samples is inadequate from the patient's perspective as the patient has to visit a specific hospital or clinic to make an appointment with an expert and provide the sample.

[0006] The human microbiome deeply affects cancer and plays a role in the progression of carcinogenesis. These microbiota are dynamic and change in response to many environmental factors such as exercise, diet, medication, and other exposures.

[0007] Recent studies have shown that microorganisms significantly contribute to selected cancer types, but the extent of the contribution of microorganisms across diverse cancer types and its implications for diagnosis remain unclear.

Summary of the Invention

Problems to be Solved by the Invention

[0008] There is a need for systems and methods that can automatically and / or remotely diagnose and / or measure the severity or stage of a patient's cancer. The human microbiome has a significant impact on the development, progression, and response to treatment of cancer. As described herein, an image-based computational pathology system can be trained to evaluate the microbiome level in whole slide images (WSIs). Such a system can be trained to classify an input WSI as microbiome low or microbiome high. Using such a classification, the severity of cancer in the patient from whom the input WSI was obtained can be determined. As described herein, to facilitate microbiome analysis in cancer, the present disclosure provides an artificial intelligence (AI)-enabled computational pathology system configured to determine the relative microbiome level in a tissue specimen comprising a formalin-fixed paraffin-embedded tissue sample or a hematoxylin and eosin (H&E)-stained slide from a biopsy.

Means for Solving the Problems

[0009] All genomic and transcriptomic sequencing data that can be used for microbiome detection described in this specification may be available in bladder cancer of The Cancer Genome Atlas (TCGA) where images are available to define microbiome-low and microbiome-high labels.

[0010] Using the systems and methods described in this specification, the correlation between the microbiome level determined by an AI system and the treatment response in patients such as non-muscle-invasive bladder cancer patients can be determined. Using such a correlation, the severity of a patient's cancer can be determined based on the microbiome level determined by the AI system. By comparing WSIs obtained from a patient at different times, the progression of cancer can be evaluated.

[0011] As described in this specification, the TCGA bladder cancer cohort can be allocated to a training set, a validation set, and a test set, each having an equal number of microbiome-low patients and microbiome-high patients. This cohort may be characterized by an average of more than one diagnostic (DX) image per patient and a total of hundreds of DX images. In an exemplary study provided in this disclosure, the TCGA bladder cancer cohort (n = 408) was allocated to a training set (66.7%, n = 272), a validation set (8.3%, n = 34), and a test set (25.0%, n = 102). Each had an equal number of microbiome-low patients and microbiome-high patients. This cohort was characterized by an average of 1.05 DX images per patient and a total of 429 DX images.

[0012] In accordance with one or more of the embodiments described herein, a deep network can be trained using hundreds or thousands of patches randomly selected from hundreds of diagnostic images in a training set. In an examination using raw (i.e., unprocessed, unedited) patient data, the area under the receiver operating characteristic (ROC) curve exceeds about 0.7, and the F1 score exceeds about 0.7. The present disclosure provides an exemplary study in which a deep network was trained using a total of 303,104 patches (sized 100×100 microns, corresponding to 400×400 pixels at 40× magnification) randomly selected from 296 DX images in a training set. In an examination using raw patient data (n = 102), the area under the ROC curve was 0.74, and the F1 score was 0.74.

[0013] Generally, aggregating the results of three different training checkpoints for each CNN can lead to an improvement in the stability of the performance of the validation set. Table 1 below shows the evaluation metrics for the top three checkpoints for each CNN using the validation set. An ensemble of the top three checkpoints is also included for each CNN.

[0014]

Table 1

[0015] The area under the ROC curve and the area under the precision-recall curve using the Inception v3, ResNet-152, and DenseNet-201 ensemble systems are in the range of 0.77 - 0.82 (ROC-AUC) and 0.70 - 0.78 (PR-AUC), and DenseNet-201 has achieved the best results in the TCGA validation set.

[0016] In the methods and systems of the present disclosure, the developed AI system can be used to determine the correlation between the survival of TCGA bladder cancer patients and the microbiome level. For example, it was found that untreated bladder cancer patients identified as having a low microbiome by the image-based system of the present disclosure have a high survival rate and a hazard ratio of 1.45.

[0017] Furthermore, the correlation with pathological complete response (pCR) can be evaluated days, weeks, months, or even years after diagnosis or treatment, and the microbiome levels of non-muscle-invasive bladder cancer patients in clinical trials can be obtained. It can be observed that patients who achieved pCR with treatment have a significantly higher probability of having a high microbiome before treatment compared to patients who did not achieve pCR. A decrease in the predicted microbiome levels after treatment can be observed in pCR patients. Conversely, non-responders may be found to have a high probability of having a high microbiome with treatment.

[0018] As described herein, the novel AI-powered image-based computational pathology system has the potential to provide data that not only conveys clinical judgment but also enables further investigation of the role of the TME microbiome in cancer. Using the AI-powered image-based microbiome analysis system and the methods of the present disclosure, the correlation between higher pre-treatment microbiome levels and complete response in test patients can be elucidated.

[0019] These and other needs are addressed by various embodiments and configurations of the present disclosure. These systems and methods of the present disclosure provide many advantages by virtue of their particular configurations, and these and other advantages will become apparent from the present disclosure.

[0020] As described herein, the systems and methods are used to analyze the microbiome in images of blood and / or tissue samples and, based on that analysis, perform cancer pathological diagnoses such as cancer diagnosis and staging, i.e., explain the extent of cancer in the patient from whom the image was obtained and determine the patient's treatment options. Such systems and methods can be used, for example, to diagnose the stage or severity of a patient's bladder cancer.

[0021] As described herein, an image-based system that utilizes a convolutional neural network (CNN) can be configured to detect the microbiome in H&E stained pathology slides of frozen tissue.

[0022] One or more means for implementing any one or more of the above-described embodiments or aspects of the embodiments are described herein.

[0023] The present disclosure includes any aspect combined with one or more other aspects.

[0024] The present disclosure includes any one or more of the features disclosed herein.

[0025] The present disclosure includes any one or more of the features substantially disclosed herein.

[0026] The present disclosure includes any one or more of the features substantially disclosed herein in combination with any one or more other features substantially disclosed herein.

[0027] The present disclosure includes any one of the aspects / features / embodiments combined with one or more other aspects / features / embodiments.

[0028] The present disclosure provides for the use of any one or more of the aspects or features disclosed herein.

[0029] It will be understood that any feature described herein may be claimed in combination with any other feature described herein, regardless of whether the feature is from the same embodiment in which it is described.

[0030] The above is a simplified summary of the present invention to provide an understanding of some aspects of the present invention. This summary is neither an extensive nor an exhaustive overview of the present invention and its various embodiments. It is not intended to identify key or critical elements of the present invention, nor to delineate the scope of the present invention. It is presented in a simplified form to introduce selected concepts of the present invention as a prelude to the more detailed description that follows. As will be appreciated, other embodiments of the present invention may utilize one or more of the features described above or in detail below, alone or in combination. The present disclosure is presented from the perspective of exemplary embodiments, but it should also be understood that individual aspects of the present disclosure may be separately claimed.

[0031] The present disclosure is described in conjunction with the accompanying drawings.

Brief Description of the Drawings

[0032]

Figure 1

Figure 2

Figures 3A - 3B

Figures 3C - 3D

Figures 3E - 3F

Figures 3G - 3H

Figures 4A - 4B

Figures 4C - 4D

Figures 4E - 4F

Figures 4G - 4H

Figure 4I

Figure 5

Figures 6A - 6B

Figure 7

DETAILED DESCRIPTION OF THE INVENTION

[0033] The human microbiome related to the whole body and the tumor microenvironment (TME) affects cancer development, progression, and response to treatment, at least in part, by influencing the immune system. As described herein, the system can be configured to determine relative microbiome levels from WSIs such as H&E stained pathology slides. Based on the relative microbiome levels in the patient's image, the TME microbiome can be analyzed and used to generate an estimated level of the patient's cancer.

[0034] Such a system may include one or more neural networks, such as a convolutional neural network (CNN), as described herein. The neural network can be trained using tumor images labeled as high microbiome or low microbiome. Such tumor images can be, for example, tumor images from TCGA bladder cancer samples on which whole transcriptome-based microbiome analysis has been performed.

[0035] By analyzing biopsy images such as those from the QUILT 3032 trial of Bacillus Calmette-Guérin (BCG) and interleukin-15 superagonist N-803 (nogapendekin alfa inbaxcept; NAI) in non-muscle invasive bladder cancer patients who do not respond to BCG (Chamie K, Lee JH, Rock A, et al: Preliminary phase 2 clinical results of IL-15RαFc superagonist N-803 with BCG in BCG-unresponsive non-muscle invasive bladder cancer (NMIBC) patients. J Clin Oncol 37:4561-4561, 2019), it will be found that the probability of high microbiome at baseline was higher in patients who achieved a complete response (CR) to treatment than in those who did not achieve CR.

[0036] These findings, along with the improved efficiency of the image-based analysis system described herein, strengthen the application that provides insights into the relationship between the microbiome level and treatment response in cancer patients. Using the system described herein, the microbiome level in tumor tissue pathology images can be evaluated, and the relationship between the microbiome level and response to treatment can be used as a useful tool for studying such relationships and communicating treatment decisions.

[0037] Using the system described herein, one or more neural networks based on input images in bladder cancer can be used to estimate the microbiome level. Although the morphological changes brought about by the microbiome in solid tumor tissue may be too fine to be easily visually distinguished even by trained pathologists, the system described herein can identify visual patterns that distinguish tissues with high microbiome loads.

[0038] The microbiome, which includes various microbial communities (such as bacteria, fungi, etc.) in combination with the sites of activity containing proteins, peptides, nucleic acids, polysaccharides, toxins, and signaling molecules related to the human gut microbiota, particularly the microbiota of the tumor microenvironment (TME), has a significant impact on cancer development and the response to treatment. This relationship can be based on the influence of the microbiome on tumor development and progression as well as on the immune system.

[0039] Conventional methods for assessing some components of the TME microbiome are by immunohistochemical analysis, but this method is essentially limited by the number of microbiota-directed antibodies that can be used. As another method, microbiome analysis based on whole-genome and transcriptome sequencing has become more widely used, but it has not been applied as part of standard diagnosis. A recent study, Poore, G.D., Kopylova, E., Zhu, Q. et al., Nature 579, 567 - 574 (2020) (hereinafter referred to as "Poore et al."), which is incorporated herein by reference, electronically examined whole-genome and whole-transcriptome sequencing data from untreated patients across 33 types of cancer in the Cancer Genome Atlas (TCGA) for microbial leads and identified the majority of samples showing evidence of specific microbial signatures uniquely corresponding to cancer. Poore et al. found that most samples from certain types of cancer showed significant intensities in several microbiomes.

[0040] As disclosed herein, in the inventors' development of a computational approach for detecting microbiome levels in hematoxylin and eosin (H&E)-stained tissue section images to facilitate the feasibility and thus the usefulness of TME microbiome analysis as a biomarker, there is the analysis of Poore et al. First, TCGA bladder cancer images based on the data of Poore et al. were classified as microbiome low or microbiome high. Next, the CNN-based system described herein was trained based on the classified samples. Next, the performance of the CNN-based system was evaluated based on the output when independent patient samples were provided as input. Finally, the CNN-based system was used to evaluate the association between baseline pre-treatment TME microbiome levels in patients and response to treatment in a clinical trial of the novel interferon-15 superagonist N-803 (nogapentin alfa inbaccept; NAI) combined with Bacillus Calmette-Guerin (BCG) in BCG-non-responsive non-muscle-invasive bladder cancer.

[0041] By using the systems or methods described herein, microbiome information can be identified from whole slide images (WSIs) input using the systems and methods described herein. The systems and methods described herein use a trained neural network such as a CNN to identify the characteristics of the microbiome and to locate the positions of microbiome pixels from within one or more patches generated from the input WSI. In certain embodiments, it can provide a certainty as to whether the WSI or patch contains a particular amount of microbiome. The characteristics of the microbiome identified by the CNN can be used to determine the cancer stage or severity. Bladder and other cancer types are described herein as examples, but it should be understood that the systems and methods described herein can be adapted to any type of cancer and / or a particular type of microbiome.

[0042] The systems and methods described herein apply deep learning-based systems to evaluate tumor tissue images and characterize the probability and response of the microbiome in a patient. Such analysis can be extended to any type of cancer and / or used to implement a pan-cancer microbiome level detector. Further, the systems described herein can be used not only to detect high or low microbiome levels, but also to distinguish between types of microorganisms, such as by generating signatures of a range of microorganisms and their sites of activity. Such systems can also be used to evaluate the response to treatment, as well as the prognosis and likelihood of metastasis, in various tumor types and treatment approaches, and to identify the characteristics of the TME microbiome that are optimal for response to treatment. Such applications may provide actionable ways to guide a physician's judgment regarding cancer patients. As described herein, it is possible to determine systems and methods for determining one or more characteristics of cancer relevant to a patient. Such characteristics may include, for example, the stage of the cancer, the malignancy of the cancer and / or tumor, the location of the cancer and / or tumor, the presence of metastasis, the expression of biomarkers that may lead to treatment or immune evasion (such as increased PD-L1 expression in a tumor), and factors such as the distribution of lymphocytes and other immune cell types within the tumor microenvironment. Various types of cancer, such as bladder cancer, head and neck cancer, and ovarian cancer, may be associated with various characteristics that can be used to determine their severity. As used herein, a characteristic may refer to any one or more of any such factors.

[0043] The following description provides only embodiments and is not intended to limit the scope, application, or configuration of the claims. Rather, the following description will provide those skilled in the art with an explanation that will be useful in implementing the embodiments. It will be understood that various modifications may be made to the functions and arrangements of the elements without departing from the spirit and scope of the appended claims.

[0044] Any reference in a description that includes a numerical reference number, if an identifier following the reference number exists in the figure, does not include an alphabetical identifier following the reference number, and when used in the plural form, is a reference to any two or more elements with similar reference numbers. Such a reference is made in the singular form, but if there is no identification of an identifier following the reference number, it is a reference to one of the elements with the same number, but not limited to a specific one of the elements being referenced. Explicit use in this specification to the contrary or additional requirements or identifications provided shall prevail.

[0045] The exemplary systems and methods of the present disclosure are also described in relation to analysis software, modules, and related analysis hardware. However, to avoid unnecessarily obscuring the present disclosure, in the following description, well-known structures, components, and devices that are omitted from the drawings or shown in a simplified form in the drawings or otherwise summarized are omitted.

[0046] The phrases "at least one", "one or more", "or", and "and / or" are open-ended expressions that function in both a conjunctive and a disjunctive manner. For example, each of the expressions "at least one of A, B, and C", "at least one of A, B, or C", "one or more of A, B, and C", "one or more of A, B, or C", "A, B, and / or C", and "A, B, or C" means only A, only B, only C, A and B together, A and C together, B and C together, or A, B, and C together.

[0047] The term "one (a)" or "one (an)" entity refers to one or more of that entity. Accordingly, the terms "one (a)" (or "one (an)", "one or more", and "at least one") may be used interchangeably in this specification. Note that the terms "comprising", "including", and "having" may also be used interchangeably.

[0048] As used herein, the term "automatic" and variations thereof refer to any process or operation that is typically continuous or semi - continuous and is performed without significant human input when the process or operation is executed. However, even if the execution of a process or operation uses significant or non - significant human input, the process or operation can be automatic if that input is received prior to the execution of the process or operation. Human input is considered significant if such input affects the way the process or operation is executed. Human input that merely consents to the execution of a process or operation is not considered "material".

[0049] Aspects of the present disclosure may take the form of embodiments that are entirely hardware, embodiments that are entirely software (including firmware, resident software, microcode, etc.), or embodiments that combine software and hardware aspects that may generally be referred to herein as "circuits", "modules" or "systems". Any combination of one or more computer - readable media may be utilized. The computer - readable media may be a computer - readable signal medium or a computer - readable storage medium.

[0050] A computer - readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non - exhaustive list) of the computer - readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read - only memory (ROM), an erasable programmable read - only memory (EPROM or flash memory), an optical fiber, a portable compact disc read - only memory (CD - ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. As relevant to the present specification, a computer - readable storage medium can be any tangible non - transitory medium that can store or retain a program for use by or in connection with an instruction execution system, apparatus, or device.

[0051] A computer-readable signal medium can include a propagated data signal in which computer-readable program code is embodied, for example, in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device, rather than a computer-readable storage medium. Program code embodied on a computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wireline, fiber optic cable, RF, or any suitable combination of the foregoing.

[0052] The terms "determine," "calculate," "compute," and variations thereof, as used herein, are used interchangeably and include any type of methodology, process, mathematical operation, or technique.

[0053] As used herein, the term "means" should be given the broadest possible interpretation in accordance with 35 U.S.C. § 112(f) and / or § 112, paragraph 6. Accordingly, claims that include the term "means" are intended to cover all structures, materials, or acts described herein, and all equivalents thereof. Further, structures, materials, or acts and their equivalents are intended to include all those described in the summary, brief description of the drawings, detailed description, summary, and claims themselves.

[0054] For purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It should be understood, however, that the present disclosure may be practiced in various ways without these specific details.

[0055] FIG. 1 shows a computer system 103 in a computing environment 100 according to an embodiment of the present disclosure. In some embodiments, the method of analyzing whole slide images described herein can be executed by a computer system 103 existing within the computing environment 100, as illustrated in FIG. 1. In one embodiment, an apparatus for identifying microbiome information in input image data can be embodied in whole or in part as a computer system 103 that includes various components as well as connections to other components and / or systems.

[0056] The components can be embodied in various ways and can include a processor 106. As used herein, the term "processor" refers only to an electronic hardware component that includes a connection (e.g., a pin arrangement) for transmitting encoded electrical signals to and from an electrical circuit. The processor 106 can include, for example, programmable logic functions that are at least partially determined from access to machine-readable instructions maintained in a non-transitory data storage device, and this programmable logic function can be embodied as a circuit, on-chip read-only memory, memory 109, data storage device 112, etc., that cause the processor 106 to execute steps of the instructions.

[0057] Processor 106 may be further embodied as a single - electron microprocessor or a multi - processor device (e.g., multi - core) having an electrical circuit, and this single - electron microprocessor or multi - processor device accesses information (e.g., data, instructions, etc.) received via a control unit, an input / output unit, an arithmetic logic unit, registers, primary memory, and / or bus 115, executes instructions, and may further include other components that similarly output data via bus 115, etc. In other embodiments, processor 106 may include a shared processing device that can be utilized by other processes and / or process owners, such as a processing array (e.g., blade, multi - processor board, etc.) within a system or a distributed processing system (e.g., “cloud,” farm, etc.). It should be understood that processor 106 is a non - transient computing device (e.g., an electromechanical device including circuits and connections for communicating with other components and devices).

[0058] Processor 106 may operate a virtual processor, for example, to process machine instructions that are not specific to the processor (e.g., to convert an operating system and machine instruction code set so that an application can execute on a virtual processor). However, as understood by a person of ordinary skill in the art, such a virtual processor is an application executed by the underlying electrical circuit and other hardware of the hardware, more specifically, the processor (e.g., processor 106). Processor 106 may be executed by a virtual processor. The virtual processor can present to an application something that appears to be a static processor and / or a dedicated processor that executes the instructions of the application, but the underlying non - virtual processor executes the instructions, can be dynamic, and / or can be divided among several processors.

[0059] In addition to the components of the processor 106, the computer system 103 may utilize the memory 109 and / or the data storage device 112 for storing accessible data such as instructions and values. The communication interface 118 facilitates communication with components such as the processor 106 via the bus 115, although there are also components that cannot be accessed via the bus 115. The communication interface 118 may be embodied as a network port, card, cable, or other configured hardware device. Additionally or alternatively, the input / output human interface 121 connects to one or more interface components to receive information (e.g., instructions, data, values, etc.) from and / or present it to humans and / or electronic devices. Input / output devices 133 that may be connected to the input / output interface include, but are not limited to, keyboards, mice, trackballs, printers, displays, sensors, switches, relays, speakers, microphones, still and / or video cameras, etc. In another embodiment, the communication interface 118 may include or be constituted by the input / output human interface 121. The communication interface 118 may be configured to communicate directly with networked components or to utilize one or more networks such as the network 124 and / or the network 127.

[0060] The network 124 may be a wired network (e.g., Ethernet), a wireless network (e.g., Wi-Fi, Bluetooth, cellular, etc.), or a combination thereof, and may enable the computer system 103 to communicate with networked components 130. In other embodiments, the network 124 may be embodied in whole or in part as a telephone network (e.g., the public switched telephone network (PSTN), a private branch exchange (PBX), a cellular telephone network, etc.).

[0061] Additionally or alternatively, one or more other networks may be utilized. For example, network 127 may represent a second network that facilitates communication with components utilized by computer system 103. For example, network 127 may be an internal network for an enterprise or other organization, whereby components are trusted (or at least somewhat trusted), whereas networked components 130 connected to a public network such as network 124 (e.g., the Internet) are untrusted (or at least have a low level of trust).

[0062] Components connected to network 127 may include memory 136, data storage device 139, input / output device 133, and / or other components that may be accessible to processor 106. For example, memory 136 and / or data storage device 139 may supplement or replace memory 109 and / or data storage device 112 either completely or for a particular task or purpose. As another example, memory 136 and / or data storage device 139 may be an external data repository (e.g., a server farm, an array, a "cloud", etc.) that enables computing device 103 and / or other devices to access the data therein. Similarly, input / output device 133 may be accessed by processor 106 either directly or via network 127, network 124 (not shown) only, or either of networks 127 and 124 via input / output human interface 121 and / or communication interface 118. Each of memory 109, data storage device 112, memory 136, and data storage device 139 includes a non-transitory data storage device including a data storage device.

[0063] It should be understood that computer-readable data can be transmitted, received, stored, processed, and presented by various components. It should also be understood that the exemplary components, whether shown herein or not, can control other components. For example, since one input / output device 133 can be a router, switch, port, or other communication component, a particular output of processor 106 can enable (or disable) an input / output device 133 associated with network 124 and / or network 127 to permit or prohibit communication between two or more nodes on network 124 and / or network 127. One of ordinary skill in the art will understand that other communication devices can be utilized in addition to or in place of those described herein without departing from the scope of the embodiments.

[0064] As a result, in one embodiment, processor 106 can execute instructions for implementing the systems and methods described herein. In another embodiment, networked component 130 can implement one or more systems and methods, and processor 106 can execute one or more other systems and methods. Memory values can be read from memory 109 or the memory of one or more network components 130. In another embodiment, the output from the systems and methods described herein can be held in memory 136 and / or data storage device 139.

[0065] Computer system 103 can be configured to execute a machine learning (ML) system 200, as illustrated in FIG. 2. In some embodiments, ML system 200 can include a training element and an inspection element. On the other hand, in other embodiments, ML system 200 can include a pre-trained CNN, but may not include a training element.

[0066] Using the training elements of the ML system 200, the neural network 212 of the ML system 200 can be trained to generate patch-based results 215 based on the input of patch 206. The training elements of the ML system 200 can be one or more processes executed by the processor 106 based on instructions in the memory 109, as illustrated in FIG. 1. Patch 206 can be stored in the data storage device 112 or received by the computer system 103 via the communication interface 118, and / or can be generated from one or more images or WSIs stored in the data storage device 112 or received by the computer system 103 via the communication interface 118.

[0067] The WSI can be associated with a patient. In some embodiments, the method can include associating the WSI with a patient in a database. For example, the method can include identifying the patient associated with the WSI and determining the level of cancer of the patient at a specific time, such as a time prior to the collection date of the sample imaged by the WSI.

[0068] The WSI can be an image including an H&E stained pathology slide. The WSI can be an image of a frozen tissue or a blood sample.

[0069] Such a WSI can be, for example, a formalin-fixed paraffin-embedded (FFPE) tissue section diagnostic image that can be available from the Comprehensive Molecular Characterization of Muscle-Invasive Bladder Cancer (BLCA) cancer cohort in TCGA.

[0070] The tissues depicted in the image can be stored in an ultra-low temperature freezer below -80 degrees Celsius. By using frozen tissue images, native, non-denatured proteins can be isolated in an active form for biochemical analysis. In some embodiments, the input to one or more neural networks includes a plurality of patches extracted from a WSI. The image can include, for example, an image of 400×400 pixels at a magnification of 40x. Each patch of the plurality of patches is 100×100 micrometers in size. Each patch of the plurality of patches can include a patch that is 100 micrometers square. Each patch can represent a 100 micrometer square portion of the tissue.

[0071] Each of the input patches 206 can be related to a patient and / or a WSI associated with the patient. Generating patch-based results 215 can include first preprocessing an input whole slide image (WSI) for dividing the WSI into a plurality of patches. For example, as described in more detail herein, a WSI associated with a patient can be divided into several patches. Each patch generated from the WSI can be associated with metadata indicating the WSI that is the source of the patch and / or the patient associated with the WSI that is the source of the patch. For example, in a training process, a WSI having a microbiome level established by other means such as WGS or WTS of Poore et al. or by manual review can be divided into patches.

[0072] The patches 206 can be automatically generated from one or more training WSIs 203. The input to a neural network for training or testing can include image patches of a specific size. The size can be, for example, 100×100 micrometers, which can correspond to an image patch of a size of 400×400 pixels of a WSI at a magnification of 40x. It should be understood that any patch size and any WSI size can be used, and the numerical values described herein are merely illustrative.

[0073] Each of the input patches 206 may also be associated with metadata indicating a microbiome level, such as a numerical value or percentage indicating the amount of microbiome in the patch, or a classification such as microbiome high or microbiome low. Metadata indicating the level of the microbiome may be described as a label or classification of the ground truth microbiome.

[0074] The neural network 212 can be trained to generate patch-based results using the ground truth microbiome label. The patch-based results can include an indicator that the patch is associated with either microbiome high or microbiome low, such as a confidence level. In some embodiments, the neural network can be trained to output a confidence score for each input patch for both microbiome high and microbiome low.

[0075] In some embodiments, multiple CNNs can be trained separately. By training different CNNs, each trained CNN 218 can be evaluated separately and the most accurate trained CNN 218 can be selected for use. In some embodiments, the results from different training checkpoints of different CNNs can be aggregated. Such aggregation can result in more stable performance.

[0076] In this set, seven patients (patient IDs 020-001, -004, -007, -009, -013, -015, and -018) are positive for malignant tumor response (dark gray in the pCR column) at week 12 or week 27. Six of the seven non-responders were found to have an increased probability of having a high microbiome with treatment.

[0077] Generating a set of training data for the systems described herein can involve using any available data. Such data can be, for example, the microbiome analysis of Poore et al. using whole-genome sequencing (WGS) and whole-transcriptome sequencing (WTS) of solid tissue (i.e., not blood) samples of bladder cancer (TCGA-BLCA; n = 408). The set of training data can include one or more WSIs for each of a plurality of patients. For example, in Poore et al., 71.8% of the patients had one sample each sequenced for the microbiome, and the remaining 28.2% had 2 - 6 samples each. The readings of each sample associated with each patient can be averaged to obtain a mean level for each patient. The mean levels for each patient can be used to rank the patients and divide the patients into categories of low or high microbiome relative to the median of the entire cohort.

[0078] Classifying patient samples into low and high microbiome categories can, in some embodiments, involve classifying certain patients having a standard deviation exceeding the median of the thresholds that divide low and high microbiome patients. For example, such patients can be associated with ambiguous measurements in whole-transcriptome sequencing (WTS), highlighting that there is inherent noise in omics-based microbiome estimation methods and that the microbiome levels can exhibit a continuous spectrum rather than a clear classification into low or high microbiome classes. Nevertheless, the systems for classifying problems based on the microbiome levels described herein provide clinical utility.

[0079] In some embodiments, the neural network 212 can be trained for several epochs with an initial learning rate such as 0.1 or 0.2. For example, the neural network 212 can be initialized with some weights. In some embodiments, random values can be automatically used as the weights. During training, the input patch 206 can be supplied to the neural network 212. The neural network 212 can process the patch 206 layer by layer and finally generate a probability distribution over the possible classes of low microbiome and high microbiome. Next, in some embodiments, the predicted class probabilities generated by the neural network 212 can be compared with the true labels associated with the training WSI 203, for example, by using a loss function such as cross-entropy loss. Next, the initial weights of the neural network 212 can be updated based on the calculated loss, for example, by using an optimization algorithm such as stochastic gradient descent (SGD). These steps can be repeated for multiple epochs until the neural network 212 converges to a certain loss. During the inspection, inspection patch-based results 215 can be generated. Such inspection patch-based results 215 can be in a data format that can be stored in the memory of the computer system 103 as illustrated in FIG. 1, and the trained convolutional neural network (CNN) 218 can be trained as described above in relation to the neural network 212 and can be used in the inspection element of the ML system 200. The trained CNN 218 can be, for example, a process executed by the processor 106 of the computer system 103 as illustrated in FIG. 1. In some embodiments, the same neural network 212 can be used as the trained CNN 218 when it is trained. The trained CNN 218 can be a class of one or more computational models in deep learning that is particularly suitable for image analysis. The trained CNN 218 can utilize a variation of the multi-layer perceptron that requires a certain amount of preprocessing of the input image such as a WSI. It should be understood that the trained CNN 218 can include an input layer, an output layer, and several other layers such as convolutional layers, pooling layers, fully connected layers, normalization layers, hidden layers, etc.In some embodiments, the convolutional block may use shared weights where each input is processed in the same or a similar way. In some embodiments, the convolutional block may use different weights to process the inputs in different ways.

[0080] The layers of the trained CNN 218 may be configured to apply a convolutional operation to the input and pass the result to the next layer. A pooling layer can be utilized to reduce the dimensionality of the input data by combining the outputs of neuron clusters in one layer to a single neuron in the next layer. In some embodiments, max pooling may use the maximum value from each neuron of the neuron cluster in the previous layer, while in some embodiments, average pooling may use the average value from each neuron.

[0081] It should be understood that the embodiments described herein refer to a single CNN, but multiple CNNs can be utilized together and can effectively generate a single trained CNN 218. Using the methods and systems described herein, multiple CNNs can be trained to separately identify the microbiome in diverse types of WSIs, such as images of blood samples, images of tissue samples, etc., for head and neck cancer, ovarian cancer, bladder cancer, or other types of cancer.

[0082] The input to the trained CNN218 can include the WSI221 classified into overlapping patches 224 or non - overlapping patches 224. As described above, the ground - truth labels of each test WSI221 can be used for each patch 224 from each WSI221. In some embodiments, a majority - voting scheme 230 can be used to aggregate patch - level results into WSI - level prediction labels, as described below. The microbiome classification labels used for testing the trained CNN218 can be obtained from information sources such as for each patient of Poore et al., and these microbiome classification labels can be used to annotate the training and / or testing of WSI203, 221 and their corresponding patches. When evaluating a new WSI221 from the test set (using all patches 224 for each test WSI221), the majority - voting scheme 230 can be used to aggregate the patch - level results 227 into WSI - level prediction labels 233, as described in more detail below. As described herein, the probability that a test WSI221 is high in microbiome can be determined based on the proportion of patches 224 associated with the test WSI221 classified as high in microbiome by the trained CNN218. Similarly, the probability that a test WSI221 is low in microbiome can be determined based on the proportion of patches 224 associated with the test WSI221 classified as low in microbiome by the trained CNN218. In the testing process, patch - based results are aggregated to obtain an image - based probability of the microbiome of the test WSI.

[0083] As described above, the trained CNN218 can be configured to generate patch - based results 227 based on the input patches 224. The input patches 224 can be automatically generated by the trained CNN218 from one or more test WSI221. Patch - based results 227 can be used to generate microbiome labels 233. In some embodiments, a majority - voting scheme 230 can be utilized to generate microbiome labels 233 based on the patch - based results.

[0084] As illustrated in FIGS. 3A-3H, such WSIs can be used for neural network learning and / or testing, and can also be used for analysis using a trained neural network.

[0085] FIG. 3A is an image of a WSI obtained from a patient with head and neck cancer. The patient has a low microbiome. When the image of FIG. 3A is input into the trained CNN described herein, the trained CNN can output an indicator that the WSI has a low microbiome.

[0086] FIG. 3B is an image of a WSI obtained from a patient with head and neck cancer. The patient has a high microbiome. When the image of FIG. 3B is input into the trained CNN described herein, the trained CNN can output an indicator that the WSI has a high microbiome.

[0087] FIG. 3C is an image of a WSI obtained from a patient with ovarian cancer. The patient has a low microbiome. When the image of FIG. 3C is input into the trained CNN described herein, the trained CNN can output an indicator that the WSI has a low microbiome.

[0088] FIG. 3D is an image of a WSI obtained from a patient with ovarian cancer. The patient has a high microbiome. When the image of FIG. 3D is input into the trained CNN described herein, the trained CNN can output an indicator that the WSI has a high microbiome.

[0089] FIGS. 3E-3H illustrate WSIs of patients in the TCGA bladder cancer cohort, each classified as either having a low or high microbiome. The test images of FIGS. 3E and 3F are of patients with a low microbiome, and the test images of FIGS. 3G and 3H are of patients with a high microbiome.

[0090] When the WSI of FIG. 3E is input into the trained CNN218 described in this specification, the trained CNN218 can output a 9% probability that the tissue imaged by the WSI has a high microbiome and a 91% probability that the tissue imaged by the WSI has a low microbiome. As a result, an indicator that the WSI of FIG. 3E has a low microbiome can be generated.

[0091] When the WSI of FIG. 3F is input into the trained CNN218 described in this specification, the trained CNN218 can output a 61% probability that the tissue imaged by the WSI has a high microbiome and a 39% probability that the tissue imaged by the WSI has a low microbiome. As a result, an indicator that the WSI of FIG. 3F has a high microbiome can be generated.

[0092] When the WSI of FIG. 3G is input into the trained CNN218 described in this specification, the trained CNN218 can output a 15% probability that the tissue imaged by the WSI has a high microbiome and an 85% probability that the tissue imaged by the WSI has a low microbiome. As a result, an indicator that the WSI of FIG. 3G has a low microbiome can be generated.

[0093] When the WSI of FIG. 3H is input into the trained CNN218 described in this specification, the trained CNN218 can output a 99% probability that the tissue imaged by the WSI has a high microbiome and a 1% probability that the tissue imaged by the WSI has a low microbiome. As a result, an indicator that the WSI of FIG. 3H has a high microbiome can be generated.

[0094] By analyzing input images such as inspection WSI221, computer system 103 can use the methods described herein to confirm the level or amount of microbiota in the input image using the trained CNN218, and based on the microbiota confirmed in the input image, make specific determinations regarding the presence of cancer in the patient associated with the microbiota and the input image. The results using the systems or methods described herein are illustrated in FIGS. 4A-4I, which are described in detail below. FIG. 4A illustrates the receiver operating characteristics of a first patient graphed by sensitivity from 0 to 1 on the vertical axis and specificity from 0 to 1 on the horizontal axis. Curve 403 shows the performance of the system described herein. Straight line 406 shows the one-to-one relationship between sensitivity and specificity. The performance of the system can be measured using the area under the curve (AUC) for the performance of the system graphed by curve 403. An AUC close to 1 indicates that the system can accurately determine whether the input WSI associated with the first patient is high microbiota or low microbiota. In the example illustrated in FIG. 4A, the AUC is 0.875.

[0095] FIG. 4B illustrates the receiver operating characteristics of a second patient graphed by sensitivity from 0 to 1 on the vertical axis and specificity from 0 to 1 on the horizontal axis. Curve 403 shows the performance of the system described herein. Straight line 406 shows the one-to-one relationship between sensitivity and specificity. The performance of the system can be measured using the area under the curve (AUC) for the performance of the system graphed by curve 403. An AUC close to 1 indicates that the system can accurately determine whether the input WSI associated with the second patient is high microbiota or low microbiota. In the example illustrated in FIG. 4B, the AUC is 0.7416.

[0096] Figure 4C illustrates the receiver operating characteristics of a third patient graphed with sensitivity from 0 to 1 on the vertical axis and specificity from 0 to 1 on the horizontal axis. Curve 403 shows the performance of the system described herein. Straight line 406 shows a one-to-one relationship between sensitivity and specificity. The performance of the system can be measured using the area under the curve (AUC) for the system performance graphed by curve 403. When the AUC is close to 1, it indicates that the system can accurately determine whether the input WSI associated with the third patient is high microbiome or low microbiome. In the example illustrated in Figure 4C, the AUC is 0.7416.

[0097] The area under the ROC curve (ROC-AUC), the area under the precision-recall curve (PR-AUC), precision, recall, and F1 score can be used as metrics to evaluate the performance of the trained CNN218. Here, both ROC-AUC and PR-AUC examine the WSI-based prediction scores (p(high microbiome)) of the trained CNN218 across the entire validation or test set. Such scores can have any value from 0 to 1. However, the values of precision, recall, and F1 score are calculated using the number of class assignments (labels of WSI true vs. predicted high and / or low microbiome) based on an optimal threshold. In some embodiments, the optimal threshold can be the threshold that minimizes the Euclidean distance to the point (0,1) on the ROC curve, which indicates 0% false positives and 100% true positives. All performance evaluation metrics used in this investigation are in the range of 0 to 1, and the higher the value, the higher the performance.

[0098] Figures 4D - 4H illustrate examples of survival analysis of TCGA bladder examination patients using ground truth labels as described above. In each illustration, the estimated survival probability is plotted on the vertical axis and time (months) is plotted on the horizontal axis. In each illustration, patients 409 with high microbiome are plotted separately from patients 412 with low microbiome. Naturally, patients with high microbiome generally show better results, including improved long-term survival rates, in response to treatment.

[0099] Figure 5 illustrates the ranked microbiome read values in the TCGA cohort using whole genome and whole transcriptome sequencing data. Blood and tissue microbiome analysis suggests a cancer diagnosis approach. In the graph of Figure 5, the x-axis shows each patient ranked by the average value at the microbiome level. Patients to the left of the median of the threshold are associated with low microbiome levels, and patients to the right of the median of the threshold are associated with high microbiome levels. The y-axis of the graph in Figure 5 shows the number of microbiome reads per patient. Naturally, the patient with the highest level of microbiome has a relatively large number of microbiomes. The increase in the average number of microbiome levels is illustrated by the sloping line 503. The flat line 500 on the graph of Figure 5 illustrates the median of the threshold, and this median of the threshold is for the patient whose microbiome read value on the y-axis is 391,501.50. Patients below the median of the threshold are classified as having low microbiome, and patients above the median of the threshold are classified as having high microbiome. As can be seen from the graph, it is shown that 50% of the patients have low microbiome and 50% of the patients have high microbiome. In the example illustrated in Figure 5, only two classifications of low microbiome and high microbiome are provided, but it should be understood that in some embodiments, patients can be classified into any number of groups such as low microbiome, medium microbiome, high microbiome. In the patient cohort illustrated in Figure 5, each patient was measured an average of 1.37 times. It was found that the median of the microbiome level is 391,501.50.

[0100] As an example of the training, testing, and validation of the neural networks described herein for generating a trained CNN218 that can detect the microbiome content in WSI associated with bladder cancer patients, a cohort of bladder cancer from TCGA can be used. In this example, the TCGA bladder cancer cohort includes 408 patients. In the training of the convolutional neural network described herein, a training set of 272 patients can be used. From the 272 patients, TCGA includes on average about one WSI per patient, for a total of 296 WSIs. Each WSI can be divided into an average of 25,265.80 patches, for a total of 7,478,677 patches. In this example, for training, 1,024 patches can be used per WSI, leaving a total of 303,104 patches for training.

[0101] The image-based microbiome level detection system described herein uses labels provided by sequencing data, but it should be understood that in some embodiments, the system can be used to determine whether sequencing is necessary for a new patient or the like. Since the data used for analysis such as TCGA data can include information related to genomics for phenotypic information, the system described herein can also be trained based on phenotypic information, such as data related to the actual structure and / or nature of the tissue, the size and / or shape of the cells. Such a system can be trained to provide the results of genomic analysis. As a result, when such a system is trained, the system can observe the phenotypic properties of the tissue and provide the possibility that the tissue contains mutations that can be found by sequencing.

[0102] For validation purposes, 34 patients of the cohort can be designated and the 34 WSIs can be divided into a total of 859,037 patches. Each of the 859,037 patches can be used for validation purposes.

[0103] For examination purposes, 102 patients in the cohort can be designated for examination purposes. 99 diagnostic WSIs can be obtained from this examination set, providing 2,501,314 patches. Each of the 2,501,314 patches can be used for examination purposes.

[0104] Each numerical value shown in this specification is merely an example and should never be regarded as limiting.

[0105] Figures 6A and 6B show the detection of image-based microbiome in clinical trials and the response to treatment. Figure 6A illustrates the correlation between the probability of high microbiome and the probability of complete response (CR) at the time of screening. Figure 6B illustrates the correlation between the probability of high microbiome and the probability of CR after screening, for example, 12 to 27 weeks after screening.

[0106] When the CNN is tested and trained using the system 200 as described above, as illustrated in FIGS. 1 and 2, a method of classifying new WSIs with microbiome labels can be implemented using the computer system 103. Such a method can be as illustrated by the flowchart of FIG. 7, and the computer system 103 can start at 700 where the trained CNN 218 can be started. The trained CNN can be trained using the training and examination system as illustrated in FIG. 2. The trained CNN can be a process executed by the processor 106 of the computer system 103.

[0107] At 703, computer system 103 may receive raw input data. The raw input data may be, for example, the WSI described herein and illustrated in FIGS. 3A-3H. The raw input data may be received via a network as illustrated in FIG. 1. Computer system 103 may receive multiple WSIs. Each WSI may be related to the same patient or different patients. For example, two or more WSIs may each be related to the same patient. Each WSI may include or be associated with metadata that describes the patient to which the WSI is related and / or other information. Computer system 103 may, in addition or alternatively, be capable of receiving patches generated from the WSIs. For example, another system may be used to split the WSI into patches. As described below, at step 706, the input WSI may be split into patches by computer system 103.

[0108] At 706, whether the input data is one or more WSIs or one or more patches, this input data may be preprocessed by computer system 103 so as to be used as an input to the trained CNN 218 as illustrated in FIG. 2. The preprocessing of the input may be automatic, i.e., it does not require human intervention. The preprocessing may include one or more adjustments such as image quality, contrast, size, brightness, orientation, etc. The preprocessing may also include splitting the input WSI into multiple patches as described herein. In some embodiments, a single WSI can be split into any number of patches. As an example, and not to be construed as a limitation in any way, each WSI may be split into more than 25,000 patches. Each patch generated from the WSI may correspond to an area of, for example, 100μm 2 in area.

[0109] At 709, the processed input data may be supplied as an input to the trained CNN 218 as described above in connection with FIG. 2. For example, each of the multiple patches may be supplied individually as an input to the trained CNN 218. Based on the input, the trained CNN 218 may be analyzed to generate an output.

[0110] The analysis of WSI using a CNN may include analyzing a plurality of patches associated with the WSI separately with the CNN. The analysis of patches using a trained CNN218 may include an input layer of the trained CNN218 that takes in the patches and feeds the patches into a neural network. Next, a series of convolutional layers of the CNN218 can apply a series of filters (or kernels) to the input patch or the output of the previous layer to generate a feature map representing high-level features within the patch. The filters are convolved across the input image or feature map to obtain local patterns and spatial information.

[0111] The CNN218 may use a pooling layer to reduce the spatial dimensions of the feature map. The pooling layer of the CNN218 may utilize one or more pooling techniques such as max-pooling and average pooling. The CNN218 can also use a fully connected layer to combine the learned features and generate a final output. The output layer of the CNN218 may be configured to generate a probability distribution for the classes of low microbiome and high microbiome of the input patch.

[0112] At 712, the output of the trained CNN218 can be received by a computing system. In some embodiments, the output of the trained CNN218 can be a microbiome label indicating whether the patch or WSI is low microbiome or high microbiome. For example, the trained CNN218 can be configured to output patch-based results such as microbiome labels for each patch generated from a particular WSI. Thus, a separate output can be generated by the trained CNN218 for each of the plurality of patches supplied as input.

[0113] In 715, the output of the AI can be post - processed. The post - processing can include determining the microbiome label for the input WSI based on the output of each microbiome label for each of a plurality of patches. The microbiome label of the input image can include the aggregation of each microbiome label for each of the plurality of patches.

[0114] In some embodiments, this aggregation can include implementing or performing a majority - voting scheme. Alternatively, the aggregation can also be performed by averaging probabilities, calculating a standard deviation, and ambiguously defining a prediction when the standard deviation near the average probability value overlaps with a determined optimal classification threshold. The microbiome label can be determined based on the aggregation of the results of processing a plurality of patches from the received image. In some embodiments, as illustrated in FIG. 2, the patch - level results generated by the trained CNN 218 can be aggregated into a WSI - level prediction label for inspection purposes as described below using a majority - voting scheme 230.

[0115] In 718, the post - processed output of the AI can be used to determine the level or amount of cancer in the input image data. In some embodiments, it can be determined whether the WSI contains microbiome at any level. In response to a determination that the image contains a high level of microbiome, a high microbiome label can be assigned to the WSI. In response to a determination that the image does not contain microbiome at any level, a low microbiome label can be assigned to the image.

[0116] The microbiome label can be the identification of microbial signatures in the tissue and / or blood found by the CNN in the input image data. Based on the microbiome label, the computer system can be configured to automatically diagnose patients having one or more of head and neck cancer, ovarian cancer, and bladder cancer.

[0117] In some embodiments, the output of the CNN can be compared to the previous cancer level of the patient associated with the WSI. For example, the computer system can be configured to determine the rate of growth / shrinkage or to determine the effectiveness of treatment for an immediate patient or a number of patients.

[0118] Based on the output of the CNN, the computer system can be configured to determine an estimated survival period, diagnosis, and / or treatment or prescription for the patient associated with the WSI. In some embodiments, based on the output of the CNN, the computer system can be configured to compare different treatments for different patients over a long period of time. The method may end at 521.

[0119] The systems or methods described herein can be used in several ways. In some embodiments, the systems and methods described herein can be used in connection with drug testing. For example, the system described herein can be used to automatically display the results generated by the CNN on a graphical user interface that can be updated in real time with new results. Such a system can enable a single laboratory with a fairly large number of patients worldwide to share results with patients and physicians simultaneously.

[0120] In one or more embodiments described herein, the system can be used for patient monitoring. For example, based on the results from the CNN, real-time notifications can be sent to physicians and / or patients based on the results from the CNN. Images of the patient can be taken at a local clinic and the image data can be sent remotely to a laboratory without the patient having to visit a particular laboratory. In this way, the patient may not need to make an appointment with a radiologist or specialist and can save time and money. The system described herein can be used to administer treatment to a patient based on a diagnosis generated using the CNN. The diagnosis of the patient can include, for example, calculating the likelihood of the patient having head and neck cancer, ovarian cancer, and bladder cancer.

[0121] Using the systems and methods described herein, a patch-based microbiome detection system can be implemented. Using the systems and methods, one or more patches of a frozen tissue image can be classified and patch-based results can be aggregated to label the frozen tissue image. If a patient is associated with multiple WSIs, each result can be aggregated to generate a single patient-level label.

[0122] As described herein, AI-powered image-based microbiome analysis can reveal a correlation between high pre-treatment microbiome levels and complete response in test patients. Since the human microbiome has a significant impact on cancer development, progression, and response to treatment, the presence and levels of the microbiome in the H&E-stained slides described herein can be used to determine the level or severity of a patient's cancer and predict the patient's survival rate.

[0123] The AI-powered computational pathology system described herein can be configured to determine the relative microbiome levels in H&E-stained slides. The system can be further configured to classify WSIs as microbiome-low or microbiome-high.

[0124] The system described herein can be trained using FFPE diagnostic (DX) images from the TCGA bladder cancer cohort (n = 408). Using this system, the correlation between the survival rate and microbiome levels of TCGA bladder cancer patients can be determined.

[0125] The system described herein can be used to analyze pathologic complete response (pCR) and microbiome levels in non-muscle invasive bladder cancer patients in a patient trial.

[0126] By using the system described herein, a computer-utilizing tumor tissue-specific image-based microbiome level detector can be developed for bladder cancer. The performance of the microbiome level detector for H&E stained FFPE tissue suggests that morphological changes brought about by the microbiome in solid tumor tissue can be visualized by an AI image system. The morphological changes are too subtle to be easily seen by the naked eye even by a trained pathologist, but the changes are detectable by the developed AI-based image system described herein. When the system described herein was used to analyze the relationship between survival and microbiome burden in TCGA bladder patients, it was found that patients with tumors with low microbiome had prolonged survival.

[0127] Furthermore, by using the system described herein, the correlation between pathological complete response (pCR) at 12 weeks or 27 weeks in a clinical trial and the microbiome level in patients with non-muscle invasive bladder cancer can be evaluated. It was found that patients who achieved pCR by treatment had a significantly higher probability of having a high microbiome before treatment compared to patients who did not achieve pCR. A decrease in the predicted microbiome level after treatment can also be observed in pCR patients. Conversely, 6 out of 7 non-responders were found to have an increased probability of having a high microbiome by treatment. Such findings suggest that the novel AI-equipped image-based computational pathology system described herein has the potential to provide data that can not only convey clinical judgment but also enable further investigation of the role of the TME microbiome in cancer.

[0128] The response of a patient to cancer treatment can be evaluated at multiple time points using the systems, methods, and / or uses described herein. For example, the response to treatment can be evaluated at about 1 week, 2 weeks, 3 weeks, 4 weeks, 1 month, 2 months, 3 months, 4 months, 5 months, 6 months, 7 months, 8 months, 9 months, 10 months, 11 months, 12 months, 13 months, 15 months, 18 months, 2 years, 3 years, 4 years or at time points beyond that.

[0129] In addition, the response of a patient to cancer treatment can be evaluated at various times before or during treatment. In certain embodiments, the level of the microbiota is determined as disclosed herein prior to or before a patient is administered cancer treatment. This level is then compared to the level of the microbiota determined as disclosed herein following or after administration of cancer treatment to the patient. The level of the microbiota can be evaluated at about 1 week, 2 weeks, 3 weeks, 4 weeks, 1 month, 2 months, 3 months, 4 months, 5 months, 6 months, 7 months, 8 months, 9 months, 10 months, 11 months, 12 months, 13 months, 15 months, 18 months, 2 years, 3 years, 4 years or beyond after the start of cancer treatment.

[0130] In certain embodiments, the response of a patient to cancer treatment can be evaluated, for example, after the start of cancer treatment to assess or assay the effectiveness of the cancer treatment. For example, the response of a patient to cancer treatment can be evaluated as disclosed herein before the start of cancer treatment and then again at a second time point after the start of cancer treatment. The level of the microbiota determined by the methods and / or uses disclosed herein can then be compared to the effectiveness of the cancer treatment. The effectiveness assessment of the cancer treatment can then be used to determine whether to continue or discontinue the cancer treatment. In another example, the response of a patient to cancer treatment can be evaluated at a time point after cancer treatment and then again at one or more additional time points after an additional round of cancer treatment. The level of the microbiota at each time point determined by the methods and / or uses disclosed herein can then be compared. The effectiveness assessment of the cancer treatment can then be used to determine whether to continue or discontinue the cancer treatment.

[0131] In one aspect, the patient has not received cancer treatment. In another aspect, the patient has received cancer treatment for a period of time.

[0132] The terms "cancer treatment" or "anticancer treatment" are used interchangeably to convey treatment methods useful for treating cancer. Examples of anticancer therapeutic agents include, but are not limited to, surgery, chemotherapeutic agents, immunotherapy, growth inhibitors, cytotoxic drugs, radiation therapy, anti-angiogenic agents, apoptosis agents, anti-tubulin agents, and other agents for treating cancer, such as anti-HER-2 antibodies (e.g., HERCEPTIN (trademark)), anti-CD20 antibodies, epidermal growth factor receptor (EGFR) antagonists (e.g., tyrosine kinase inhibitors), HER1 / EGFR inhibitors (e.g., erlotinib (TARCEVA (trademark))), platelet-derived growth factor inhibitors (e.g., GLEEVEC (trademark) (imatinib mesylate)), COX-2 inhibitors (e.g., celecoxib), interferons, cytokines, antagonists (e.g., neutralizing antibodies) that bind to one or more targets such as ErbB2, ErbB3, ErbB4, PDGFR-β, BlyS, APRIL, BCMA or VEGF receptor, TRAIL / Apo2, and other bioactive agents and organic chemicals. Combinations of these treatment methods and therapeutic agents are also contemplated for use within the methods described herein.

[0133] "Chemotherapeutic agents" are compounds useful for the treatment of cancer. Examples of chemotherapeutic agents include erlotinib (TARCEVA™, Genentech / OSI Pharm.), bortezomib (VELCADE™, Millennium Pharm.), fulvestrant (FASLODEX™, Astrazeneca), sunitinib (SU11248, Pfizer), letrozole (FEMARA™, Novartis), imatinib mesylate (GLEEVEC™, Novartis), PTK787 / ZK 222584 (Novartis), oxaliplatin (Eloxatin™, Sanofi), 5-FU (5-fluorouracil), leucovorin, rapamycin (sirolimus, RAPAMUNE™, Wyeth), lapatinib (GSK572016, GlaxoSmithKline), lonafarnib (SCH 66336), sorafenib (BAY43-9006, Bayer Labs.) and gefitinib (IRESSA™, Astrazeneca), AG1478, AG1571 (SU 5271; Sugen), alkylating agents such as thiotepa and CYTOXAN™ cyclophosphamide; alkyl sulfonates such as busulfan, improsulfan and piposulfan; aziridines such as benzodopa, carboquone, meturedopa and uredopa; ethyleneimines and methylamelamines (including altretamine, triethylenemelamine, triethylenephosphoramide, triethylenethiophosphoramide and trimethylolomelamine); acetogenins (particularly bullatacin and bullatacinone); camptothecin (including the synthetic analogue topotecan); bryostatin; calistatin; CC-1065 (including its adozcicsin, carzcicsin and bizcicsin synthetic analogues); cryptophycins (particularly cryptophycin 1 and cryptophycin 8); dolastatin; duocarmycin (including the synthetic analogues, KW-2189 and CB1-TM1); erythrobins; pancratistatin; sarcodictyin; spongistatin;Nitrogen mustards, such as chlorambucil, chloronaphazine, chlorophosphamide, estramustine, ifosfamide, mechlorethamine, mechlorethamine oxide hydrochloride, melphalan, nobenbitin, phenesterine, prednimustine, trophosphamide, uracil mustard; nitrosureas, such as carmustine, chloroozotocin, fotemustine, lomustine, nimustine and ranimustine; antibiotics, such as enediyne antibiotics (e.g., calicheamicin, especially calicheamicin γ1 and calicheamicin ω1 (Angew Chem.Intl.Ed.Engl. (1994) 33:183-186); dynemicin A including dinemicin; bisphosphonates, such as clodronic acid; esperamicin; and neocarzinostatin chromophore and related chromoprotein enediyne antibiotic chromophores), aclacinomycin, actinomycin, anthramycin, azaserine, bleomycin, cactinomycin, carabicin, caminomycin, cardinophilin, chromomycinsis, dactinomycin, daunorubicin, detorubicin, 6-diazo-5-oxo-L-norleucine, ADRIAMYCIN™ doxorubicin (including morpholino-doxorubicin, cyanomorpholino-doxorubicin, 2-pyrrolino-doxorubicin and deoxydoxorubicin), epirubicin, esorubicin, idarubicin, marcellomycin, mitomycin, such as mitomycin C, mycophenolic acid, nogalamycin, olivomycin, peplomycin, potfiromycin, puromycin, rutilomycin, rhodomycin, streptozocin, streptozocin, tubercidin, ubenimex, dinostatin, zorubicin; antimetabolites, such as methotrexate and 5-fluorouracil (5-FU); folic acid analogs, such as denopterin, methotrexate, pteropterin, trimetrexate; purine analogs, such as fludarabine, 6-mercaptopurine, thiampurine, thioguanine; pyrimidine analogs, such as ancitabine, azacitidine, 6-azauridine, carmofur, cytarabine, dideoxyuridine, doxifluridine, enocitabine, floxuridine;Androgens, such as calusterone, drostanolone propionate, epithiostanol, mepitiostane, testolactone; anti-adrenal agents, such as aminoglutethimide, mitotane, trilostane; folic acid supplements, such as folinic acid; aceglatone; aldophosphamide glycoside; aminolevulinic acid; eniluracil; amsacrine; bestrabucil; bisantrene; edatraxate; defofamine; demeclocycline; diaziquone; elfomithine; elliptinium acetate; epothilone; etoglucid; gallium nitrate; hydroxyurea; lentinan; lonidamine; maytansinoids, such as maytansine and ansamitocin; mitoguazone; mitoxantrone; mopidamol; nitraerine; pentostatin; phenamet; pirarubicin; losoxantrone; podophyllinic acid; 2-ethylhydrazide; procarbazine; PSK (trademark) polysaccharide complex (JHS Natural Products, Eugene, Oreg.); razoxane; rizoxin; sizofiran; spirogermanium; tenuazonic acid; triaziquone; 2,2’,2’’-trichlorotriethylamine; trichothecene (especially T-2 toxin, verracurin A, roridin A and anguidine); urethane; vindesine; dacarbazine; mannomustine; mitobronitol; mitolactol; pipobroman; gascitosine sink; arabinoside (“Ara-C”); cyclophosphamide; thiotepa; taxoids, such as TAXOL (trademark) paclitaxel (Bristol-Myers Squibb Oncology, Princeton, N.J.), ABRAXANE (trademark) Cremophor-free, albumin-modified nanoparticle formulation of paclitaxel (American Pharmaceutical Partners, Schaumberg, Ill.) and TAXOTERE (trademark) docetaxel (Rhone-Poulenc Rorer, Antony, France); chlorambucil; GEMZAR (trademark) gemcitabine; 6-thioguanine; mercaptopurine; methotrexate; platinum analogs, such as cisplatin and carboplatin; vinblastine; platinum; etoposide (VP-16); ifosfamide; mitoxantrone; vincristine; NAVELBINE (trademark) vinorelbine; novantrone; teniposide;Edatrexate; Daunomycin; Aminopterin; Xeloda; Ibandronate; CPT-11; Topoisomerase inhibitor RFS 2000; Difluoromethylornithine (DMFO); Retinoids, such as retinoic acid; Capecitabine; and pharmaceutically acceptable salts, acids or derivatives of any of the foregoing are included.;

[0134] The definition of this "chemotherapeutic agent" includes: (i) antihormonal agents that regulate or inhibit the hormonal action on tumors, such as antiestrogens and selective estrogen receptor modulators (SERMs) (for example, including tamoxifen (NOLVADEX® (tamoxifen)), raloxifene, droloxifene, 4-hydroxytamoxifen, trioxifene, keoxifene, LY117018, onapristone, and FARESTON® (toremifene)); (ii) aromatase inhibitors that inhibit the enzyme aromatase that regulates estrogen production in the adrenal glands (for example, 4(5)-imidazole, aminoglutethimide, MEGASE® (megestrol acetate), AROMASIN® (exemestane), formestane, fadrozole, RIVISOR® (vorozole), FEMARA® (letrozole), and ARIMIDEX® (anastrozole), etc.); (iii) antiandrogens, such as flutamide, nilutamide, bicalutamide, leuprolide, and goserelin; and troxacitabine (1,3-dioxolane nucleoside cytosine analog); (iv) aromatase inhibitors; (v) protein kinase inhibitors; (vi) lipid kinase inhibitors; (vii) antisense oligonucleotides, especially those that inhibit the expression of genes in signal transduction pathways involved in abnormal cell proliferation (for example, PKC-α, Ralf, and H-Ras, etc.); (viii) ribozymes, such as VEGF expression inhibitors (for example, ANGIOZYME® (ribozyme)) and HER2 expression inhibitors; (ix) vaccines such as gene therapy vaccines, for example, ALLOVECTIN® vaccine, LEUVECTIN® vaccine, and VAXID® vaccine; PROLEUKIN® rIL-2; LURTOTECAN® topoisomerase 1 inhibitor; ABARELIX® rmRH; (x) anti-angiogenic agents, such as bevacizumab (AVASTIN®, Genentech); and (xi) also includes any pharmaceutically acceptable salts, acids, or derivatives of the above.

[0135] The treatment of cancer patients may include any of the following: adjuvant therapy (also called adjunct therapy or adjunctive therapy) that destroys residual tumor cells that may be present after a known tumor has been removed by initial treatment (e.g., surgery), thereby preventing the possibility of cancer recurrence; neoadjuvant therapy performed before a surgical procedure to shrink the cancer; induction therapy typically to achieve remission of acute leukemia; consolidation therapy (also called intensification therapy) performed after remission has been achieved to maintain the remission; maintenance therapy administered at low doses or low frequencies to help extend the remission; first-line therapy (also called standard therapy); second (or third, fourth, etc.)-line therapy (also called salvage therapy) is performed when the disease does not respond or recurs after first-line therapy.

[0136] As used herein, "tumor" means a mass of transformed cells involved in neoplastic, unregulated cell growth and containing at least partially a neovascularized vasculature. The abnormal neoplastic cell growth is rapid and continues even after the stimulus that initiated the new growth has ended. "Tumor" is used broadly to include not only the tumor parenchymal cells but also the supporting stroma containing the neovascularized blood vessels that infiltrate the tumor parenchymal cell mass. Tumors are generally malignant tumors, i.e., cancers with the ability to metastasize (i.e., metastatic tumors), but tumors can also be non-malignant (i.e., non-metastatic tumors).

[0137] "Patient", or "individual", or "subject" are used interchangeably herein and refer to a mammalian subject being treated, typically a human patient.

[0138] The systems and methods described herein can also be extended in other ways. For example, instead of modeling the problem as a classification problem to predict labels, the problem can be modeled as a regression task and aimed at predicting the amount of microbiota in a pathological image.

[0139] Furthermore, cancer may be rich in several microbial signatures. In the future, the number of microbiomes per cancer will be studied separately from the images. Thus, instead of starting with a single binary true label (high microbiome vs. low microbiome), microbiome-specific analysis can be performed, such as whether microbiome X appears in the image data or some combination thereof. Using such analysis, the optimal microbiome environment for immune stimulation can be determined.

[0140] Even further, the systems and methods and / or uses disclosed herein may further include the integration of supplementary data for magnifying an image (i.e., a pathology slide) associated with a patient. Such supplementary data can be obtained, for example, from RNA sequencing (RNA-seq). Further, for example, the supplementary data can be derived from a vector of some transcript expression levels that can be processed as parallel inputs to a neural network that can encode complementary information that can improve the accuracy or ability of the system to generalize across patient samples, tissues, and cancer types.

[0141] Any of the steps, functions, and operations discussed herein can be performed continuously and automatically.

[0142] The exemplary systems and methods and / or uses of the present invention are described in connection with components and methods for analyzing and determining the presence of microbiomes in head and neck cancer, ovarian cancer, and bladder cancer, as well as WSI. However, to avoid unnecessarily obscuring the present invention, some known structures and devices have been omitted in the foregoing description. This omission should not be construed as a limitation on the scope of the claimed invention. Specific details are set forth to provide an understanding of the present invention. However, it should be understood that the present invention can be practiced in various ways beyond the specific details described herein.

[0143] Furthermore, although the exemplary embodiments illustrated herein show various components of a co-located system, certain components of the system can be located remotely in separate portions of a distributed network such as a LAN and / or the Internet or within a dedicated system. Thus, it is to be understood that components of the system or portions thereof (e.g., microprocessors, memory / storage devices, interfaces, etc.) can be integrated into one or more devices such as a single server, multiple servers, computers, computing devices, terminals, a "cloud", or other distributed processing, or can be co-located at specific nodes of a distributed network such as an analog and / or digital communication network, a packet switched network, or a circuit switched network. In another embodiment, the components can be physically or logically distributed among a plurality of components (e.g., a microprocessor can include a first microprocessor on one component and a second microprocessor on another component, each performing a part of a shared task and / or an assigned task). From the foregoing description and for reasons of computational efficiency, it is to be understood that the components of the system can be located anywhere within the distributed network of components without affecting the operation of the system. For example, the various components can be located in a switch such as a PBX and a media server, a gateway, one or more communication devices, on-premises of one or more users, or some combination thereof. Similarly, one or more functional portions of the system can be distributed between a communication device and an associated computing device.

[0144] Furthermore, it is to be understood that the various links connecting the elements can be wired or wireless links or any combination thereof or other known or later developed elements capable of supplying and / or communicating data between the connected elements. These wired or wireless links can also be protected links and can be capable of communicating encrypted information. The transmission medium used as a link can be any carrier suitable for electrical signals, including, for example, coaxial cables, copper wire, and optical fibers, and can take the form of acoustic or light waves such as those generated during radio and infrared data communication.

[0145] The flowchart is discussed and illustrated in connection with a particular series of events, but it should be understood that changes, additions, and omissions to this sequence can be made without materially affecting the operation of the present invention.

[0146] Some variations and modifications of the present invention may be used. It would be possible to provide some of the features of the present invention without providing other features.

[0147] In yet another embodiment, the systems, methods, and / or uses of the present invention may be implemented with a dedicated computer, a programmed microprocessor or microcontroller and peripheral integrated circuit elements, an ASIC or other integrated circuit, a digital signal microprocessor, a discrete element circuit, or other wired electronic circuit or logic circuit such as a PLD, PLA, FPGA, PAL, a dedicated computer, any equivalent means, or a programmable logic circuit or gate array. In general, the various aspects of the present invention can be implemented using any device or means capable of implementing the methodologies illustrated herein. Exemplary hardware that may be used in the present invention includes computers, mobile terminals, telephones (e.g., cellular, Internet-enabled, digital, analog, and hybrid, etc.), and other hardware known in the art. These devices include microprocessors (e.g., single or multiple microprocessors), memory, non-volatile storage devices, input devices, and output devices. Further, without limitation, alternative software implementations including distributed processing or component / object distributed processing, parallel processing, or virtual machine processing can be constructed and the methods described herein can be implemented as provided by one or more processing components.

[0148] In yet another embodiment, the disclosed method can be readily implemented with software using an object or object-oriented software development environment that provides portable source code that can be used on various computer or workstation platforms. Alternatively, the disclosed system can be implemented partially or fully in hardware using standard logic circuits or VLSI designs. Whether to implement the system according to the present invention using software or hardware depends on the speed and / or efficiency requirements of the system, the particular functions, and the particular software or hardware system or microprocessor or microcomputer system utilized.

[0149] In yet another embodiment, the disclosed method can be stored on a storage medium and can be implemented partially in software executed on a programmed general-purpose computer in cooperation with a control device and memory, a dedicated computer, or a microprocessor, etc. In these cases, the system and method of the present invention can be implemented as a program incorporated into a personal computer such as an applet, JAVA (registered trademark), or CGI script, as a resource existing on a server or computer workstation, or as a routine incorporated into a dedicated measurement system or system component. The system can also be implemented by physically incorporating the system and / or method into a software and / or hardware system.

[0150] Embodiments of the specification including software are executed by one or more microprocessors or stored for subsequent execution and executed as executable code. The executable code is selected to execute instructions including particular embodiments. The instructions to be executed are a set of restricted instructions selected from a discrete set of native instructions understood by the microprocessor and placed in memory accessible by the microprocessor prior to execution. In another embodiment, human-readable "source code" software is first converted to system software to include a particular set of instructions for a platform (e.g., a computer, microprocessor, database, etc.) selected from a set of native instructions of the platform before being executed by one or more microprocessors.

[0151] As described herein, a neural network can include layers of logical nodes having inputs and outputs. If the output is below a self-determined threshold level, the output can be omitted (i.e., the input can be within the non-active response portion of the scale and no output is provided), and if the output exceeds the threshold, the output can be provided (i.e., the input can be within the active response portion of the scale and an output is provided). A particular arrangement of active and non-active depictions can be provided as one or more steps. Multiple inputs to a node can generate a multi-dimensional plane (such as a hyperplane) to depict a combination of active or non-active inputs.

[0152] The present invention describes components and functions implemented in embodiments with reference to specific standards and protocols, but the present invention is not limited to such standards and protocols. There are other similar standards and protocols not described herein that are considered to be included in the present invention. Further, the standards and protocols described herein and other similar standards and protocols not described herein are periodically replaced by more rapid or more effective equivalents having essentially the same function. Such alternative standards and protocols having the same function are considered equivalents included in the present invention.

[0153] In various embodiments, configurations, and aspects, the present invention substantially includes the components, methods, uses, processes, systems, and / or apparatuses illustrated and described herein, including various embodiments, partial combinations, and subsets thereof. Those skilled in the art will understand, upon understanding this disclosure, how to make and use the present invention. The present invention, in various embodiments, configurations, and aspects, provides apparatuses and processes in the absence of items not described and / or illustrated herein, or provides apparatuses and processes in various embodiments, configurations, or aspects herein in the absence of items that may have been used in previous apparatuses or processes, for example, for performance improvement, ease of implementation, and / or cost reduction of implementation.

[0154] The foregoing discussion of the present invention has been presented for purposes of illustration and description. The foregoing is not intended to limit the present invention to one or more of the forms disclosed herein. In the foregoing detailed description, for example, various features of the present invention are grouped together in one or more embodiments, configurations, or aspects for purposes of streamlining the disclosure. The features of an embodiment, configuration, or aspect of the present invention may be combined in alternative embodiments, configurations, or aspects other than those discussed above. The methods of this disclosure should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, aspects of the invention are less than all of the features of the single disclosed embodiment, configuration, or aspect described above. Accordingly, the following claims are incorporated herein by this detailed description, and each claim stands on its own as a separate preferred embodiment of the present invention.

[0155] Furthermore, the description of the present invention includes descriptions of one or more embodiments, configurations, or aspects, and specific variations and modifications. However, other variations, combinations, and modifications are within the scope of the present invention, such as may be within the skill and knowledge of those skilled in the art after understanding the present disclosure. Whether or not alternative, replaceable, and / or equivalent structures, functions, scopes, or steps are disclosed herein, it is not intended to publicly disclose patentable subject matter. Instead, within the allowable scope, it is intended to obtain the right to include alternative embodiments, configurations, or aspects that include such alternative, replaceable, and / or equivalent structures, functions, scopes, or steps for what is claimed.

[0156] In the foregoing description, the method was described in a particular order for illustrative purposes. In alternative embodiments, it should be understood that the method may be performed in an order different from the described order without departing from the scope of the embodiments. It should also be understood that the methods described above may be implemented as algorithms executed by hardware components (e.g., circuits) intentionally constructed to execute one or more of the algorithms or portions thereof described herein. In another embodiment, the hardware components may include a general-purpose microprocessor (e.g., a CPU, GPU) that is first converted to a dedicated microprocessor. Then, since the encoded signal is loaded, the dedicated microprocessor here maintains machine-readable instructions so that the microprocessor can read and execute a set of machine-readable instructions derived from the algorithms and / or other instructions described herein. The machine-readable instructions used to execute the algorithm or a portion thereof may be limited but utilize a finite set of instructions recognized by the microprocessor. The machine-readable instructions may, in one or more embodiments, be encoded as signals or values of signal generating components in the microprocessor by the voltage of a memory circuit, the configuration of a switching circuit, and / or the selective use of specific logic gate circuits. Additionally or alternatively, the machine-readable instructions may be accessible from the microprocessor and encoded in a medium or device as magnetic fields, voltage values, charge values, reflective / non-reflective portions, and / or physical markers.

[0157] In another embodiment, the microprocessor may further include one or more of a single microprocessor, a multi-core processor, multiple microprocessors, a distributed processing system (e.g., an array, blade, server farm, "cloud", multi-purpose processor array, cluster, etc.), and / or may be located in the same position as a microprocessor that performs other processing operations. Any one or more microprocessors may be integrated into a single processing appliance (e.g., a computer, server, blade, etc.) or may be disposed in whole or in part in separate components and connected via communication links (e.g., a bus, network, backplane, etc., or a plurality thereof).

[0158] Examples of general-purpose microprocessors can include a central processing unit (CPU) having a data value that includes a memory location containing a data value or a value utilized as an instruction encoded in an instruction register (or other circuitry that maintains instructions). The memory location may further include a memory location external to the CPU. Such external components of the CPU may be embodied as one or more of a field programmable gate array (FPGA), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), random access memory (RAM), bus-accessible storage, network-accessible storage, etc.

[0159] These machine-executable instructions may be stored on one or more machine-readable media such as a CD-ROM or other type of optical disk, a floppy disk, ROM, RAM, EPROM, EEPROM, magnetic or optical card, flash memory or other type of machine-readable medium suitable for storing electronic instructions. Alternatively, the method may be implemented by a combination of hardware and software.

[0160] In another embodiment, the microprocessor can be a system or collection of processing hardware components such as a microprocessor on a client device and a microprocessor on a server, a collection of devices each having a respective microprocessor, or a shared or remote processing service (e.g., a "cloud"-based microprocessor). The system of microprocessors can include task-specific assignments of processing tasks and / or shared or distributed processing tasks. In yet another embodiment, the microprocessor can execute software to provide services that emulate different microprocessors or microprocessors. As a result, a first microprocessor composed of a first set of hardware components can virtually provide the services of a second microprocessor, whereby the hardware associated with the first microprocessor can operate using the instruction set associated with the second microprocessor.

[0161] Computer-executable instructions can be stored locally and executed on a particular computer (e.g., a personal computer, a mobile computing device, a laptop, etc.), but the storage of data and / or instructions and / or the execution of at least a portion of the instructions are generally provided by connection to a remote data storage device and / or processing device or collection of devices that may be known as a "cloud" but can include public, private, dedicated, shared, other service bureaus, computing services, and / or "server farms".

[0162] Examples of microprocessors described in this specification include, but are not limited to, Qualcomm® Snapdragon® 800 and 801, Qualcomm® Snapdragon® 610 and 615 with 4G LTE integration and 64-bit computing, Apple® A7 microprocessor with 64-bit architecture, Apple® M7 motion microprocessor, Samsung® Exynos® series, Intel® Core™ family of microprocessors, Intel® Xeon® family of microprocessors, Intel® Atom™ family of microprocessors, Intel® Itanium® family of microprocessors, Intel® Core™ i5-4670K and i7-4770K 22nm Haswell, Intel® Core™ i5-3570K 22nm Ivy Bridge, AMD® FX™ family of microprocessors, AMD® FX-4300, FX-6300 and FX-8350 32nm Vishera, AMD® Kaveri microprocessor, Texas Instruments® Jacinto C6000™ automotive infotainment microprocessor, Texas Instruments® OMAP™ automotive grade mobile microprocessor, ARM® Cortex-M™ microprocessor, ARM® Cortex-A and ARM926EJ-S™ microprocessor, and at least one equivalent microprocessor in other industries, and such microprocessors can perform computer functions using any known or future-developed standards, instruction sets, libraries, and / or architectures.

[0163] An embodiment includes a method for evaluating a patient's response to cancer treatment in a patient having cancer, the method comprising receiving a first image associated with the patient before administering cancer treatment and automatically determining the level of the microbiome in the first image using a neural network; receiving a second image associated with the patient after administering cancer treatment and automatically determining the level of the microbiome in the second image using a neural network; comparing the level of the microbiome in the first image with the level of the microbiome in the second image, wherein a decrease in the level of the microbiome in the first image compared to the level of the microbiome in the second image indicates that the patient is responding to cancer treatment, and an increase in the level of the microbiome in the first image compared to the level of the microbiome in the second image indicates that the patient is not responding to cancer treatment; and continuing the patient's treatment with cancer treatment if the level of the microbiome in the first image is decreased compared to the level of the microbiome in the second image, or adjusting the cancer treatment if the level of the microbiome in the first image is increased compared to the level of the microbiome in the second image.

[0164] The embodiment further includes a computer system for evaluating a patient's response to cancer treatment in a cancer patient, the system including a processor and a computer-readable storage medium storing computer-readable instructions, the computer-readable instructions, when executed by the processor, cause the processor to receive a first image associated with the patient before administering cancer treatment and automatically determine the level of the microbiome in the first image using a neural network; receive a second image associated with the patient after administering cancer treatment and automatically determine the level of the microbiome in the second image using a neural network; compare the level of the microbiome in the first image with the level of the microbiome in the second image, wherein a decrease in the level of the microbiome in the first image compared to the level of the microbiome in the second image indicates that the patient is responding to cancer treatment, and an increase in the level of the microbiome in the first image compared to the level of the microbiome in the second image indicates that the patient is not responding to cancer treatment; and when the level of the microbiome in the first image is lower compared to the level of the microbiome in the second image, continue the patient's treatment with cancer treatment, or when the level of the microbiome in the first image is higher compared to the level of the microbiome in the second image, adjust the cancer treatment.

[0165] The embodiment further includes a computer system configured to execute a convolutional neural network to detect the level of the microbiome in a whole-slide image associated with the patient, classify the whole-slide image as either microbiome-low or microbiome-high based on the output of the convolutional neural network, and determine the characteristics of cancer based on the classification of the whole-slide image.

[0166] The above method and computer system aspects include that the whole-slide image includes a pathological slide stained with hematoxylin and eosin (H&E).

[0167] The above-described method and aspects of the computer system include that the cancer is selected from the group consisting of head cancer, neck cancer, ovarian cancer, and bladder cancer.

[0168] The above-described method and aspects of the computer system include that the convolutional neural network is trained based on FFPE diagnostic images from The Cancer Genome Atlas bladder cohort.

[0169] The above-described aspects of the computer system include that the output of the convolutional neural network is used to determine the survival statistics of the patient.

[0170] The above-described aspects of the computer system include that a low microbiome is related to a microbiome level of less than about 390,000, and a high microbiome is related to a microbiome level of greater than about 390,000.

[0171] The above-described aspects of the computer system include that the instructions further cause the processor to determine that the patient is responding to cancer treatment based on the classification of the whole slide image, and in response to determining that the patient is responding to cancer treatment, to continue the treatment of the patient with cancer treatment.

[0172] The above-described aspects of the computer system include that the instructions further cause the processor to determine a pathologic complete response (pCR) based on the output of the convolutional neural network.

[0173] An embodiment includes a method for evaluating a patient's response to cancer treatment in a cancer patient, the method comprising the steps of: running a convolutional neural network to detect the level of the microbiome in a whole slide image associated with the patient; classifying the whole slide image as either microbiome low or microbiome high based on the output of the convolutional neural network; and determining the characteristics of the cancer based on the classification of the whole slide image.

[0174] The above-described method and aspects of the computer system include that the level of the microbiome in the image is determined based on the aggregation of the processing results of a plurality of patches from the received first and second images. The above-described method and aspects of the computer system include that the first and second images include whole slide images (WSIs). The above-described method and aspects of the computer system include that the input to one or more neural networks includes a plurality of patches extracted from the WSI. The above-described method and aspects of the computer system include that the first and second images include images of 400×400 pixels with a magnification of 40 times. The above-described method and aspects of the computer system include that each patch of the plurality of patches has a size of 100×100 microns. The above-described method and aspects of the computer system include that each patch of the plurality of patches includes a patch of 100 micrometers square. The above-described method and aspects of the computer system include that the neural network outputs the microbiome label of each of the plurality of patches. The above-described method and aspects of the computer system include that the level of the microbiome in the image includes the aggregation of the respective microbiome labels of the plurality of patches. The above-described method and aspects of the computer system include that the aggregation includes a majority voting method. The above-described method and aspects of the computer system include that the first and second images each include a frozen tissue image. The above-described method and aspects of the computer system include that the neural network is a convolutional neural network. The above-described method and aspects of the computer system include that images from The Cancer Genome Atlas (TCGA) are used to train the convolutional neural network. The above-described method and aspects of the computer system include that the level of the microbiome is related to the tissue and microbial signatures in the blood illustrated in the image. The above-described method and aspects of the computer system include that automatically determining the level of cancer in the image includes calculating the severity of cancer for the patient.

Claims

1. A computer system for evaluating the response of a patient with cancer to cancer treatment, comprising a processor, a computer-readable storage medium storing computer-readable instructions and, when the computer-readable instructions are executed by the processor, causing the processor to execute a convolutional neural network to detect the level of microbiota in a whole slide image associated with the patient, classify the whole slide image as either low microbiota or high microbiota based on the output of the convolutional neural network, and determine the characteristics of the cancer associated with the patient based on the classification of the whole slide image. A computer system.

2. The system according to claim 1, wherein the whole slide image includes a hematoxylin and eosin (H&E) stained pathological slide.

3. The system according to claim 1, wherein the cancer is bladder cancer.

4. The system according to claim 3, wherein the convolutional neural network is trained based on formalin-fixed paraffin-embedded (FFPE) diagnostic images obtained from a bladder cohort.

5. The system according to claim 1, wherein the output of the convolutional neural network is used to determine the survival statistics of the patient.

6. Low microbiota is associated with a microbiota level lower than the median microbiota level of a cohort, and high microbiota is associated with a microbiota level higher than the median microbiota level of the cohort. The system according to claim 1.

7. The command causes the processor to determine whether the patient is responding to the cancer treatment based on the classification of the whole slide image, and in response to determining that the patient is responding to the cancer treatment, continue the treatment of the patient by the cancer treatment The system according to claim 1, further causing the above to be performed.

8. The command further causes the processor to determine pathological complete response (pCR) based on the output of the convolutional neural network. The system according to claim 1.

9. A method for evaluating a patient's response to cancer treatment having cancer, performing a convolutional neural network to detect the level of the microbiome in a whole slide image associated with the patient, classifying the whole slide image as either low microbiome or high microbiome based on the output of the convolutional neural network, determining the characteristics of the cancer associated with the patient based on the classification of the whole slide image, determining that the patient is responding to the cancer treatment based on the classification of the whole slide image, and in response to determining that the patient is responding to the cancer treatment, continuing the treatment of the patient by the cancer treatment A method comprising:

10. The method according to claim 9, wherein the whole slide image includes a pathological slide stained with hematoxylin and eosin (H&E).

11. The method according to claim 9, wherein the cancer is bladder cancer.

12. The method according to claim 11, wherein the convolutional neural network is trained based on formalin-fixed paraffin-embedded (FFPE) diagnostic images obtained from a bladder cohort. **Claim 13** The method according to claim 9, wherein the output of the convolutional neural network is used to determine the survival statistics of a patient. **Claim 14** The method according to claim 9, wherein a low microbiome is associated with a microbiome level lower than the median microbiome level of a cohort, and a high microbiome is associated with a microbiome level higher than the median microbiome level of the cohort. **Claim 15** The method according to claim 9, further comprising determining pathologic complete response (pCR) based on the output of the convolutional neural network. **Claim 16** At least one machine-readable non-transitory medium including a plurality of instructions, which, when executed on a computing device, cause the computing device to: execute a convolutional neural network to detect a level of a microbiome in a whole slide image associated with a patient; classify the whole slide image as either low microbiome or high microbiome based on the output of the convolutional neural network; determine cancer characteristics associated with the patient based on the classification of the whole slide image; and perform. **Claim 17** The machine-readable non-transitory medium according to claim 16, wherein the whole slide image includes a pathologic slide stained with hematoxylin and eosin (H&E). **Claim 18** The machine-readable non-transitory medium according to claim 16, wherein the cancer is bladder cancer. **Claim 19** The machine-readable non-transitory medium according to claim 18, wherein the convolutional neural network is trained based on formalin-fixed paraffin-embedded (FFPE) diagnostic images obtained from a bladder cohort.

20. The machine-readable non-transitory medium according to claim 16, wherein the output of the convolutional neural network is used to determine the survival statistics of the patient.

Images