Systems and methods for high-throughput drug screening

Abstract

The present disclosure provides a method for generating an artificial fluorescence image of a group of tumor organoids, the artificial fluorescence image indicating whether cells contained in the group of tumor organoids are alive or dead. The method includes receiving a bright field image, providing the bright field image to a trained model, receiving an artificial fluorescence image from the trained model, and outputting the artificial fluorescence image to at least one of a memory or a display.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Application No. 62 / 944,292, filed December 5, 2019, and U.S. Application No. 63 / 012,885, filed April 20, 2020, both of which are incorporated by reference herein in their entireties. [Background technology]

[0002] Patient-derived tumor organoid (TO) technology has been used to generate cell models of diverse cancer types, including colon, breast, pancreatic, liver, lung, endometrial, prostate, and esophagogastric, among others. In addition to advancing basic research, TOs have also been adopted in recent years for drug development and precision medicine research.

[0003] Tumor organoids can be used to model cancer growth and evaluate the effectiveness of different therapies to stop cancer growth. To monitor tumor organoid growth before, during, and after the administration of various anticancer drugs, tumor organoids can be imaged to detect cell death and / or viability within the cell culture plate.

[0004] Some methods for detecting dead or live cells can include the use of fluorescent signals, which can be detected by fluorescence microscopy. Fluorescent dyes can be applied to tumor organoids to highlight certain intracellular features and / or make them easier to detect. Cells can then be imaged using techniques such as fluorescence microscopy. However, fluorescence microscopy is very time-consuming, and the fluorescent dyes used can be toxic to cells, so the amount of observed cell death may appear to be increased, which can be mistakenly attributed to the anti-cancer therapy being tested.

[0005] Therefore, there is a need in the art for automated analysis of tumor organoid images without the use of fluorescent dyes and / or fluorescence microscopy. Summary of the Invention [Problem to be solved by the invention]

[0006] Disclosed herein is the system, method and mechanism that is useful for automatically analyzing tumor organoid image.In particular, the present disclosure provides the system, method and mechanism that uses only the raw bright field image of tumor organoid to generate the image of tumor organoid that approximates fluorescent staining technology. [Means for solving the problem]

[0007] According to some embodiments of the disclosed subject matter, there is provided a method for generating the artificial fluorescent image of tumor organoid group, and the artificial fluorescent image of tumor organoid group is represented by the cell that comprises in tumor organoid group being alive or dead.This method includes: receiving bright field image; providing the bright field image to a trained model; receiving artificial fluorescent image from the trained model; and outputting the artificial fluorescent image to at least one of memory or display.

[0008] In some embodiments, the method can further include concatenating the bright field image with the artificial image to generate a concatenated image, and generating a viability value based on the concatenated image.

[0009] In some embodiments, the tumor organoid population may be associated with colorectal cancer, gastric cancer, breast cancer, lung cancer, endometrial cancer, colon cancer, head and neck cancer, ovarian cancer, pancreatic cancer, gastric cancer, hepatobiliary cancer, or genitourinary cancer.

[0010] In some embodiments, the trained model may include an artificial neural network.

[0011] In some embodiments, the artificial neural network may include a generative adversarial network (GAN).

[0012] In some embodiments, the trained model may be trained based on a loss function that includes a structural similarity index (SSIM).

[0013] In some embodiments, the trained model may be trained based on a loss function that includes a classifier loss.

[0014] In some embodiments, the trained model may be trained based on a loss function that includes a generator loss and a discriminator loss.

[0015] In some embodiments, the method may further include pre-processing the bright field image to increase the contrast level.

[0016] In some embodiments, the preprocessing may include converting the bright-field image to an unsigned byte format and stretching and clipping each pixel intensity in the bright-field image to a desired output range of [0 to 255], where the stretching includes uniformly stretching the 2nd and 98th percentiles of pixel intensities in the bright-field image to the desired output range.

[0017] In some embodiments, the trained model can include a generator, where the generator is trained in part by a classifier.

[0018] In some embodiments, the method can further include pre-processing bright field and fluorescent images included in the training data used to train the trained model.

[0019] In some embodiments, the tumor organoid population may be plated onto a well plate containing at least 300 wells.

[0020] In some embodiments, outputting the artificial fluorescent image can include outputting the artificial fluorescent image to a drug screening process to determine the effectiveness of drugs used to treat the tumor organoid population.

[0021] In some embodiments, the method can further include providing a second brightfield image to the trained model and receiving a second artificial fluorescent image from the trained model.

[0022] In some embodiments, the second bright field image can include a second group of tumor organoids, and the second artificial fluorescent image can indicate whether the cells included in the second group of tumor organoids are alive or dead.

[0023] In some embodiments, the second brightfield image can include a view of the tumor organoid group, the second brightfield image is generated a predetermined period of time after the brightfield image is generated, and the second artificial fluorescent image can indicate whether the cells contained in the tumor organoid group are alive or dead.

[0024] In some embodiments, the predetermined period of time may be at least 12 hours.

[0025] In some embodiments, the predetermined period of time may be at least 24 hours.

[0026] In some embodiments, the predetermined period of time may be at least 72 hours.

[0027] In some embodiments, the predetermined period of time may be one week.

[0028] According to some embodiments of the disclosed subject matter, the present disclosure provides a tumor organoid analysis system, comprising at least one processor and at least one memory.System is configured to receive the bright field image of tumor organoid group, provide the bright field image to a trained model, receive the artificial fluorescence image from the trained model, and indicate whether the cells that comprise tumor organoid group are alive or dead, and output the artificial fluorescence image to at least one of memory or display. [Brief explanation of the drawings]

[0029] [Figure 1] FIG. 1 illustrates an example of a system for automatically analyzing tumor organoid images. [Figure 2] FIG. 2 illustrates an example of hardware that may be used in some embodiments of the system. [Figure 3] FIG. 1 illustrates an exemplary flow in which patient-derived organoids grown from tumor specimens can be used to generate brightfield and / or fluorescent images, as well as live / dead assay readouts. [Figure 4] FIG. 1 illustrates an exemplary flow for training a generator to generate artificial fluorescent images based on input brightfield images of organoid cells. [Figure 5] FIG. 1 illustrates an exemplary flow for generating an artificial fluorescence image. [Figure 6] FIG. 1 illustrates an exemplary neural network. [Figure 7] FIG. 2 illustrates an exemplary classifier. [Figure 8] FIG. 1 illustrates an exemplary process by which a model can be trained to generate artificial fluorescent stained images of one or more organoids based on input brightfield images. [Figure 9] FIG. 1 illustrates an exemplary process by which an artificial fluorescent image of one or more organoids can be generated based on a bright field image. [Figure 10] FIG. 1 illustrates an exemplary raw image before and after preprocessing. [Figure 11] 1 shows an exemplary flow chart for culturing tumor organoids. Culturing patient-derived tumor organoids. [Figure 12] FIG. 1 illustrates an exemplary flow for performing a drug screen according to the systems and methods described herein. [Figure 13] FIG. 1 illustrates an exemplary process by which artificial fluorescent images can be generated at multiple time points for at least one organoid. [Figure 14] FIG. 1 shows a table representing an exemplary assay or well plate configuration. [Figure 15] 1A and 1B are diagrams illustrating examples of images generated using a single neural network model and a three-neural network model. [Figure 16] FIG. 1 illustrates a flow for generating an artificial fluorescence image using a first trained model and a second trained model. [Figure 17] FIG. 1 illustrates a process for generating fluorescent images of tumor organoids. [Figure 18] FIG. 1 is a diagram showing a flow for predicting survival rates based on bright-field images. [Figure 19] FIG. 2 illustrates an exemplary generator and an exemplary discriminator. [Figure 20] FIG. 1 illustrates a classifier capable of generating viability predictions based on brightfield and artificial fluorescence images. [Figure 21] FIG. 1 illustrates a process for generating viability values. DETAILED DESCRIPTION OF THE INVENTION

[0030] Various aspects of the present disclosure will now be described with reference to the accompanying drawings. It should be understood, however, that the drawings and the following related detailed description are not intended to limit the claimed subject matter to the particular forms disclosed. Rather, the invention is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the claimed subject matter.

[0031] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific embodiments in which the present disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure. It will be understood, however, that the detailed description and specific examples, while indicating examples of embodiments of the present disclosure, are given by way of illustration only and not limitation. Various substitutions, modifications, additions, rearrangements, or combinations thereof may be made within the scope of the present disclosure, which will be apparent to those skilled in the art.

[0032] According to common practice, the various features illustrated in the drawings may not be drawn to scale. The illustrations presented herein are not intended to be actual views of any particular method, device, or system, but merely idealized representations employed to describe various embodiments of the present disclosure. Thus, for clarity, dimensions of various features may be arbitrarily increased or decreased. Additionally, some of the drawings may be simplified for clarity. Thus, the drawings may not depict all components of a given apparatus (e.g., device) or method. Additionally, like reference numerals may be used to denote like features throughout this specification and the drawings.

[0033] The information and signals described herein may be represented using a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or combinations thereof. Some figures may illustrate signals as single signals for clarity of presentation and explanation. It will be understood by those skilled in the art that the signals represent buses of signals, buses having various bit widths, and that the present disclosure may be implemented with respect to any number of data signals, including a single data signal.

[0034] The operation of the various illustrative logic blocks, modules, circuits, and algorithms described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and operations are described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments described herein.

[0035] Additionally, it should be noted that the embodiments may be described in terms of a process that is depicted as a flowchart, a flow diagram, a structure diagram, or a block diagram. While a flowchart may describe operational acts as a sequential process, many of these acts may be performed in a different order, in parallel, or substantially simultaneously. Additionally, the order of acts may be rearranged. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. Furthermore, the methods disclosed herein may be implemented in hardware, software, or both. If implemented in software, the functions may be stored on or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including a medium that facilitates transfer of a computer program from one place to another.

[0036] It should be understood that referring to elements herein using a designation such as "first," "second," etc. does not limit the quantity or order of those elements unless such limitation is expressly stated. Rather, these designations may be used herein as a convenient method of distinguishing between two or more elements or instances of an element. Thus, a reference to a first and a second element does not imply that only two elements may be used therein, or that the first element must somehow precede the second element. Also, unless otherwise specified, a set of elements may include one or more elements.

[0037] As used herein, the terms "component," "system," and similar terms are intended to refer to a computer-related entity, i.e., either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. For example, both an application running on a computer and the computer may be a component. One or more components may reside within a process and / or thread of execution, and a component may be local to one computer and / or distributed between two or more computers or processors.

[0038] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs.

[0039] Furthermore, the disclosed subject matter may be implemented as a system, method, apparatus, or article of manufacture using standard programming and / or engineering techniques to create software, firmware, hardware, or any combination thereof, to control a computer or processor-based device to implement aspects detailed herein. The term "article of manufacture" (or alternatively, "computer program product") as used herein is intended to encompass a computer program accessible from any computer-readable device, carrier, or medium. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs), etc.), smart cards, and flash memory devices (e.g., cards, sticks).

[0040] In addition, it should be understood that a carrier wave may be employed to carry computer-readable electronic data, such as those used to send and receive email or to access a network such as the Internet or a local area network (LAN). Of course, those skilled in the art will recognize that many modifications can be made to this configuration without departing from the scope or spirit of the claimed subject matter.

[0041] As used herein, the terms "biological specimen," "patient sample," and "sample" refer to specimens collected from patients. Such samples include, but are not limited to, tumors, biopsies, tumor organoids, other tissues, and bodily fluids. Suitable bodily fluids include, for example, blood, serum, plasma, sputum, lavage fluid, cerebrospinal fluid, urine, semen, sweat, tears, saliva, and the like. Samples may be collected, for example, via biopsy, swab, or smear.

[0042] The terms "extracted," "recovered," "isolated," and "isolated" refer to a compound (e.g., a protein, cell, nucleic acid, or amino acid) that has been removed from at least one component with which it is naturally associated and found in nature.

[0043] The phrases "enriched" or "enrichment" herein refer to the process of amplifying nucleic acids contained in a sample. Enrichment can be sequence-specific or non-specific (i.e., involving any of the nucleic acids present in the sample).

[0044] As used herein, "cancer" shall be considered to mean any one or more of a wide range of benign or malignant tumors, including those capable of invasive growth and metastasis throughout a human or animal body or portion thereof, e.g., via the lymphatic system and / or bloodstream. As used herein, the term "tumor" includes both benign and malignant tumors and solid growths. Exemplary cancers include, but are not limited to, carcinomas, lymphomas, or sarcomas, such as, for example, ovarian cancer, colon cancer, breast cancer, pancreatic cancer, lung cancer, prostate cancer, urinary tract cancer, uterine cancer, acute lymphocytic leukemia, Hodgkin's disease, small cell lung cancer, melanoma, neuroblastoma, glioma, and human soft tissue sarcomas.

[0045] Fluorescence microscopy is commonly used to detect the presence of specific molecules in a sample. For example, in cell biology, fluorescence microscopy can be used to highlight specific cellular components (e.g., organelles) or detect molecular markers indicative of specific cellular states (e.g., apoptosis, differentiation, or activation of cell signaling pathways). However, several drawbacks limit the utility of fluorescence microscopy. First, the use of this technique requires increased time, labor, and reagents (e.g., stains) compared to transmitted light microscopy, creating a cost bottleneck in high-throughput screening processes. Second, some fluorescent dyes are toxic to cells, potentially biasing the results of some experiments (e.g., quantification of cell death). Furthermore, cells damaged by these dyes can no longer be used in ongoing experiments, requiring more cells for experiments involving assaying cells at multiple time points. Third, the time a sample can be observed using fluorescence microscopy is limited by photobleaching, a process in which a fluorophore loses its ability to fluoresce when exposed to light.

[0046] Fortunately, methods based on transmitted light microscopy largely avoid these problems because they are relatively fast and inexpensive to use and can capture multiple images of the same biological sample at multiple time points. The term "transmitted light microscopy" is used to refer to any type of microscopy in which light passes from a light source to the other side of a lens. The simplest of these methods is bright-field microscopy, in which the sample is illuminated with white light from below and observed with transmitted light from above. The use of standard bright-field microscopy is somewhat limited for biological samples with low contrast. For example, without the use of stains, the cell membrane and nucleus are the only features of mammalian cells that are discernible in bright-field images. Fortunately, adding optical accessories to standard bright-field microscopy can dramatically improve image contrast, eliminating the need to kill, fix, and stain the sample. One very simple contrast-enhancing method is dark-field microscopy, which works by illuminating the sample with light not focused by the objective lens. For applications requiring more detailed information, phase-contrast and differential interference contrast microscopy may be employed. These complementary techniques produce high-contrast images of transparent biological samples by using optical systems to translate density or thickness variations within the sample into contrast differences in the final image. Importantly, these techniques can be used to reveal small cellular structures, such as nuclei, ribosomes, mitochondria, membranes, spindles, mitotic apparatus, nucleoli, chromosomes, Golgi apparatus, vesicles, pinocytic vesicles, lipid droplets, and cytoplasmic granules. Bright-field microscopy can also be enhanced with polarized light, which creates contrast within samples containing materials with different refractive indices (i.e., birefringent samples). While dark-field, phase-contrast, and differential interference contrast microscopy are well-suited for imaging live, unstained biological samples, polarized light microscopy is well-suited for examining the structure and composition of rocks, minerals, and metals.In particular, any of these contrast enhancement methods may be combined with optical sectioning techniques, such as confocal microscopy and light sheet microscopy, which produce clear images of focal planes deep within thicker samples (e.g., thick tissues, small organisms), reducing or eliminating the need to physically section the sample (e.g., using a microtome).

[0047] In this application, we demonstrate that some cellular states commonly detected using fluorescence microscopy also appear as subtle morphological features in images generated by transmitted light microscopes.In these images, such features may be difficult or impossible to identify using only the human eye, but we show that their identification can be automated using a trained model.In this example, a trained model is designed to predict the percentage of live and dead cells in a sample (i.e., values ​​that would typically be determined using fluorescent staining methods, such as caspase-3 / 7 and TO-PRO-3 staining) using only bright-field images as input.These visualization methods are then used in high-throughput screening of drugs that kill cancer cells in tumor organoids generated from patient tumor samples.

[0048] However, the methods and systems disclosed herein are not limited to such a single application. The ability to relate subtle morphological features present in transmitted light microscope images to cellular states of interest is useful in a myriad of applications across many diverse fields of research. Some exemplary, non-limiting applications are described below.

[0049] First, the systems and methods disclosed herein have clear utility in the field of biology, where they can be used to characterize a variety of samples, ranging from individual cells (e.g., plant cells, animal cells) to tissue slices (e.g., biopsies) to small organisms (e.g., protozoa, bacteria, fungi, embryos, nematodes, insects). Importantly, by avoiding the use of cytotoxic stains, the disclosed systems and methods allow the same sample to be repeatedly imaged over a time course of days or even weeks. Images of the sample are captured using transmitted light microscopy, and a trained system utilizes morphological features (e.g., cell volume, diameter, shape, and topography) to identify cells with specific cellular states. For example, by training a system to distinguish between cells by type, the number or concentration of a particular cell type present in a sample can be estimated, or by training a system to distinguish between live and dead cells, cell viability can be assessed. The trained system can also be used to characterize cells based on behaviors such as proliferation, differentiation, apoptosis, necrosis, motility, migration, cytoskeletal dynamics, cell-cell and cell-substrate adhesion, signal transduction, polarity, and vesicle trafficking. For example, the systems and methods disclosed herein can be used to distinguish between different forms of cell death based on their unique morphology (e.g., shrinkage in apoptosis and swelling in necrosis). Naturally, these systems and methods can also be used to monitor cellular responses to any experimental manipulation, from culture conditions to the effects of the space environment on biological processes. Thus, the disclosed systems and methods provide a means to investigate myriad aspects of biology using a highly efficient platform. While cells, tissues, or organisms can be left unprocessed (e.g., unstained, fixed, or otherwise treated), the systems and methods disclosed herein are also useful for imaging samples that have been stained, fixed, or otherwise treated.For example, but not limited to, immunohistochemically stained, hematoxylin and eosin stained, etc. tissues or cells may be imaged according to the systems and methods disclosed herein.

[0050] Second, the disclosed systems and methods are useful for the development of new and improved therapeutics. For example, the disclosed systems and methods can be used to monitor cellular responses to potential drugs in high-throughput drug screening, as described in the Examples. Additionally, the disclosed systems and methods can be used to monitor the differentiation state of cells during both development and directed differentiation of stem cells. Stem cells can be used to repair tissues damaged by disease or injury, either by direct injection into a patient or by differentiation into replacement cells ex vivo. For example, stem cells can be differentiated into specific blood cell types and thereby used for donor-free transfusions. Other promising stem cell-based therapies include bone marrow cells for blood cancer patients, nerve cells damaged by spinal cord injury, stroke, Alzheimer's disease, or Parkinson's disease, cartilage damaged by arthritis, skin damaged by severe burns, and pancreatic islet cells destroyed by type 1 diabetes. Stem cells can also be used to generate specific cell types, tissues, 3D tissue models, or organoids for use in drug screening. Using a trained system that can monitor the differentiation state of cells would allow for more efficient production of any of these stem cell-based products.

[0051] Third, the disclosed systems and methods are useful for diagnosing medical conditions. For example, the systems and methods disclosed herein can be used to quickly, efficiently, and accurately detect the presence of specific cell types in patient samples that are indicative of a disease or condition, such as tumor cells, blood in urine or stool, thread-like cells in vaginal discharge, or inflammatory cell infiltrates. For example, a system trained on images of a tissue sample (e.g., biopsy) can detect morphological features that can be used to distinguish between benign, non-invasive, and invasive cancer cells. In addition, such systems can be used to identify microorganisms and parasites in patient samples, enabling the diagnosis of a wide range of infectious diseases, including those caused by bacteria (e.g., tuberculosis, urinary tract infections, tetanus, Lyme disease, gonorrhea, syphilis), fungi (e.g., thrush, yeast infections, ringworm), and parasites (e.g., malaria, sleeping sickness, hookworm, scabies). Such methods may be particularly useful for identifying the causative pathogen when a condition can be caused by a variety of microorganisms (eg, infectious keratitis of the cornea).

[0052] Fourth, the systems and methods disclosed herein can be used to identify organisms in environmental samples such as soil, crops, and water. This application can be used in both environmental science, for example, to assess the health of ecosystems based on the number and diversity of organisms, and in epidemiology, for example, to track the spread of contaminants that pose health risks.

[0053] Fifth, the systems and methods disclosed herein can be used to evaluate a wide variety of materials, such as clays, fats, oils, soaps, paints, pigments, foods, drugs, glass, latex, polymer blends, fabrics and other fibers, chemical compounds, crystals, rocks, and minerals. Applications in which such materials are analyzed using microscopy are found across a variety of fields. In industry, the systems and methods disclosed herein can be used in failure analysis, design verification, and quality control of commercial products and building materials. For example, the systems and methods disclosed herein can be used to detect defects or breaks in mechanical parts requiring high precision, such as watches and aircraft engines. In computer science, the systems and methods disclosed herein can be used to inspect integrated circuits and semiconductors. In both archaeology and forensic science, the systems and methods disclosed herein can be used to identify unknown materials and examine wear patterns on artifacts / evidence. In geology, the systems and methods disclosed herein can be used to determine the composition of rocks and minerals and reveal evidence about how they formed. In agriculture, the systems and methods disclosed herein can be used to detect microbial indicators of soil health and to test seeds and grains to assess their purity, quality, and germination ability. In food science, the systems and methods disclosed herein can be used to create in vitro grown meat from animal cells.

[0054] This application provides a non-limiting exemplary system that uses bright-field image as input in the screening of cancer drugs.Typically, drug response is measured through cell viability assay using live / dead fluorescent staining, but this has several drawbacks as described above.Although the use of bright-field microscopes largely avoids these problems, it is not easy to visualize and quantify live / dead cells from bright-field images alone, which is a significant obstacle to the cost-effective high-throughput screening of tumor organoids.Some systems and methods described herein provide artificial fluorescent images that can be generated using only bright-field images.

[0055] In some embodiments, a method for generating an artificial image of a cell or group of cells (e.g., cells in culture) is provided. In some embodiments, the generated image indicates whether the cells contain one or more characteristics indicative of a particular cell state or cell identity (e.g., death, disease, differentiation, bacterial strain, etc.). In some embodiments, the method includes receiving a brightfield image, providing the brightfield image to a trained model, receiving an artificial fluorescent image from the trained model, and outputting the artificial fluorescent image to at least one of a memory or a display. In some embodiments, the cell is a mammalian cell, a plant cell, a eukaryotic cell, or a bacterial cell. In some embodiments, the characteristics indicative of the cell state or cell identity include one or more distinguishing physical or structural characteristics of the cell, which are distinguishable by brightfield microscopy. For example, non-limiting characteristics include size, morphology, intracellular structure, staining value, cell surface structure, etc.

[0056] Drug screening Analyzing drug response data by target can identify important pathways / mutations. For drugs that cause cell death in organoids, the targets of those drugs may be important. Therefore, it is desirable to discover and / or develop additional drugs that modulate these targets. Important cellular pathways and / or mutations may be specific to the cancer type of the organoid. For example, if a CDK inhibitor specifically kills colorectal cancer (CRC) tumor organoid cells, CDKs may be particularly important in CRC.

[0057] FIG. 1 illustrates an example of a system 100 for automatically analyzing tumor organoid images. In some embodiments, the system 100 can include a computing device 104, a secondary computing device 108, and / or a display 116. In some embodiments, the system 100 can include an organoid image database 120, a training data database 124, and / or a trained model database 128. In some embodiments, the trained model database 128 can include one or more trained machine learning models, such as artificial neural networks. In some embodiments, the computing device 104 can communicate with the secondary computing device 108, the display 116, the organoid image database 120, the training data database 124, and / or the trained model database 128 over a communications network 112. As shown in FIG. 1, the computing device 104 can receive tumor organoid images, such as brightfield images of tumor organoids, and generate artificial fluorescent stained images of the tumor organoids. In some embodiments, the computing device 104 can execute at least a portion of the organoid image analysis application 132 to automatically generate artificial fluorescent staining images.

[0058] The organoid image analysis application 132 may be included in a secondary computing device 108, which may be included in the system 100, and / or in the computing device 104. The computing device 104 may communicate with the secondary computing device 108. The computing device 104 and / or the secondary computing device 108 may also communicate over a communications network 112 with a display 116, which may be included in the system 100.

[0059] Communications network 112 may facilitate communications between computing device 104 and secondary computing device 108. In some embodiments, communications network 112 may be any suitable communications network or combination of communications networks. For example, communications network 112 may include a Wi-Fi network (which may include one or more wireless routers, one or more switches, etc.), a peer-to-peer network (e.g., a Bluetooth network), a cellular network (e.g., a 3G network, a 4G network, a 5G network, etc., conforming to any suitable standard, such as CDMA, GSM, LTE, LTE Advanced, WiMAX, etc.), a wired network, etc. In some embodiments, communications network 112 may be a local area network, a wide area network, a public network (e.g., the Internet), a private or semi-private network (e.g., a corporate or university intranet), any other suitable type of network, or any suitable combination of networks. The communications links shown in FIG. 1 may each be any suitable communications link or combination of communications links, such as a wired link, a fiber optic link, a Wi-Fi link, a Bluetooth link, a cellular link, etc.

[0060] The organoid image database 120 can include a large number of raw tumor organoid images, such as bright-field images. In some embodiments, the bright-field images can be generated using a bright-field microscope imaging modality. Exemplary bright-field images are described below. In some embodiments, the organoid image database 120 can include artificial fluorescent staining images generated by the organoid image analysis application 132.

[0061] The training data database 124 can include a large number of images for training the model to generate artificial fluorescent stained images. In some embodiments, the training data image database 124 can include bright-field raw images and corresponding three-channel fluorescent stained images. The trained model database 128 can include a large number of trained models that can receive bright-field raw images of tumor organoids and output artificial fluorescent stained images. In some embodiments, the trained model 136 can be stored in the computing device 104.

[0062] 2 illustrates an example of hardware 200 that may be used in some embodiments of system 100. Computing device 104 may include a processor 204, a display 208, an input 212, a communication system 216, and memory 220. Processor 204 may be any suitable hardware processor or combination of processors, such as a central processing unit ("CPU"), a graphics processing unit ("GPU"), or the like, that is capable of executing programs, which may include processes described below.

[0063] In some embodiments, the display 208 may present a graphical user interface. In some embodiments, the display 208 may be implemented using any suitable display device, such as a computer monitor, a touch screen, a television, etc. In some embodiments, the input 212 of the computing device 104 may include an indicator, a sensor, an actuatable button, a keyboard, a mouse, a graphical user interface, a touch screen display, etc.

[0064] In some embodiments, communications system 216 may comprise any suitable hardware, firmware, and / or software for communicating with other systems over any suitable communications network. For example, communications system 216 may include one or more transceivers, one or more communications chips and / or chipsets, etc. In more specific examples, communications system 216 may include hardware, firmware, and / or software that can be used to establish coaxial connections, fiber optic connections, Ethernet connections, USB connections, Wi-Fi connections, Bluetooth connections, cellular connections, etc. In some embodiments, communications system 216 enables computing device 104 to communicate with secondary computing device 108.

[0065] In some embodiments, memory 220 may include any suitable one or more storage devices that may be used to store instructions, values, etc. that may be used by processor 204, for example, to present content using display 208, to communicate with secondary computing device 108 via communication system 216, etc. Memory 220 may include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 220 may include RAM, ROM, EEPROM, one or more flash drives, one or more hard disks, one or more solid-state drives, one or more optical drives, etc. In some embodiments, memory 220 may have encoded thereon a computer program for controlling the operation of computing device 104 (or secondary computing device 108). In such embodiments, processor 204 may execute at least a portion of the computer program to present content (e.g., a user interface, images, graphics, tables, reports, etc.), receive content from secondary computing device 108, transmit information to secondary computing device 108, etc.

[0066] The secondary computing device 108 may include a processor 224, a display 228, an input 232, a communication system 236, and a memory 240. The processor 224 may be any suitable hardware processor or combination of processors, such as a central processing unit ("CPU"), a graphics processing unit ("GPU"), or the like, that is capable of executing programs, which may include the processes described below.

[0067] In some embodiments, the display 228 may present a graphical user interface. In some embodiments, the display 228 may be implemented using any suitable display device, such as a computer monitor, a touch screen, a television, etc. In some embodiments, the input 232 of the secondary computing device 108 may include an indicator, a sensor, an actuatable button, a keyboard, a mouse, a graphical user interface, a touch screen display, etc.

[0068] In some embodiments, communications system 236 may comprise any suitable hardware, firmware, and / or software for communicating with other systems over any suitable communications network. For example, communications system 236 may include one or more transceivers, one or more communications chips and / or chipsets, etc. In more specific examples, communications system 236 may include hardware, firmware, and / or software that can be used to establish coaxial connections, fiber optic connections, Ethernet connections, USB connections, Wi-Fi connections, Bluetooth connections, cellular connections, etc. In some embodiments, communications system 236 enables secondary computing device 108 to communicate with computing device 104.

[0069] In some embodiments, memory 240 may include any suitable one or more storage devices that may be used to store instructions, values, etc. that may be used by processor 224, for example, to present content using display 228, to communicate with computing device 104 via communication system 236, etc. Memory 240 may include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 240 may include RAM, ROM, EEPROM, one or more flash drives, one or more hard disks, one or more solid-state drives, one or more optical drives, etc. In some embodiments, memory 240 may have encoded thereon a computer program for controlling the operation of secondary computing device 108 (or computing device 104). In such embodiments, processor 224 may execute at least a portion of the computer program to present content (e.g., a user interface, images, graphics, tables, reports, etc.), receive content from computing device 104, transmit information to computing device 104, etc.

[0070] Display 116 may be a computer display, a television monitor, a projector, or other suitable display.

[0071] FIG. 3 illustrates an exemplary flow 300 in which patient-derived organoids grown from tumor specimens can be used to generate brightfield and / or fluorescent images, as well as live / dead assay readouts. In some embodiments, live / dead assay readouts can be generated using brightfield and multiplexed fluorescent imaging. Drug response can be measured via cell viability assays using live / dead fluorescent staining. In some embodiments, flow 300 can be included in a high-throughput drug screening system. An example of high-throughput drug screening is described in U.S. Provisional Patent Application No. 62 / 944,292, entitled "Large Scale Phenotypic Organoid Analysis," filed December 5, 2019, which is incorporated herein by reference in its entirety.

[0072] Flow 300 may include harvesting a tumor specimen 308 from a human patient 304, culturing 312 organoids using the tumor specimen 308, drug screening 316 the organoids, imaging 320 the organoids, and outputting brightfield and fluorescent images 324 of the organoids. After the organoids are cultured, cells from the organoids may be plated in an assay plate (e.g., a 96-well assay plate, a 384-well assay plate, etc.). The assay plate may also be referred to as a plate. Drug screening 316 may include plating the cells and treating the cells with a number of different drugs and / or concentrations. For example, a 384-well plate may contain 14 drugs at seven different concentrations. As another example, a 96-well plate may contain six drugs at five different concentrations. Imaging 320 may include brightfield imaging of the treated cells, or further, applying a fluorescent stain to at least a portion of the cells and fluorescently imaging the cells. In some embodiments, the fluorescent imaging may include generating three channels of data for each cell. The three channels of data can include a blue / total nuclei channel, a green / apoptotic channel, and a red / pink / death channel. Each channel can be used to form a fluorescence image. In addition, imaging 320 can generate a combined three-channel fluorescence image including the blue / total nuclei channel, the green / apoptotic channel, and the red / pink / death channel. In some embodiments, imaging 320 can include generating a bright-field image of the cells using a bright-field microscope and generating a fluorescence image of the cells using a confocal microscope, such as a confocal laser scanning microscope. In some embodiments, instead of using traditional fluorescent staining to generate a fluorescence image, imaging 320 can include generating a bright-field image of at least a portion of the cells and generating an artificial bright-field image of the portion of the cells based on the bright-field image using a process described below (e.g., the process of FIG. 9 ).

[0073] In some embodiments, bright-field images (e.g., 2D bright-field projections) depicting cell culture wells in drug screening assays can be generated using a 10X objective lens of a microscope. In some embodiments, the microscope can be an ImageXPRESS microscope available from Molecular Devices. In some embodiments, the cells can be cancer cell lines or cancer tumor organoids derived from patient specimens.

[0074] FIG. 4 shows an exemplary flow 400 for training a generator 408 to generate an artificial fluorescence image 412 based on an input bright-field image 404 of an organoid cell. In some embodiments, the generator 408 may include a U-Net convolutional neural network. In some embodiments, the generator 408 may include a pix2pix model. In some embodiments, the generator 408 may be a generative adversarial network (GAN). Exemplary neural networks that may be included in the generator 408 are described below in connection with FIG. 6. In some embodiments, the generator may include a neural network that can receive the bright-field image 404 and output a single three-channel fluorescence image (e.g., a 256×256×3 image). In some embodiments, the generator may include three neural networks, each of which can receive the bright-field image 404 and output a one-channel fluorescence image (e.g., a 256×256×1 image). A generator including three neural networks, each of which can receive the bright-field image 404 and output a one-channel fluorescence image, may be referred to as a three-model generator. Each of the neural networks can be trained to output a particular channel of fluorescence. For example, a first neural network can output a blue / total nuclei channel image, a second neural network can output a green / apoptosis channel image, and a third neural network can output a red / death channel image. Flow 400 can include combining the blue / total nuclei channel image, the green / apoptosis channel image, and the red / death channel image into a single three-channel fluorescence image (e.g., a 256x256x3 image, a 1024x1024x3 image, etc.).

[0075] The flow may include providing the bright-field image 404, the artificial fluorescence image 412, and a ground truth fluorescence image 424 associated with the bright-field image to a classifier 416 that can predict whether the image is real or generated by a generator 408 (e.g., the artificial fluorescence image 412). In some embodiments, the generator 408 may receive the image and output a label ranging from 0 to 1, where 0 indicates that the image was generated by the generator 408 and 1 indicates that the image is real (e.g., the ground truth fluorescence image 424 associated with the bright-field image 404). In some embodiments, the classifier 416 may be a PatchGAN classifier, such as a 1×1 PatchGAN classifier. An exemplary classifier is described below in connection with FIG. 7.

[0076] The flow 400 may include an objective function value calculation 420. The objective function value calculation 420 may include calculating an objective function value based on the labels output by the classifier 416 and / or by other metrics calculated based on the bright-field image 404, the artificial fluorescence image 412, and the ground truth fluorescence image 424. The objective function value may capture multiple loss functions (e.g., a weighted sum of multiple loss functions). In this manner, the objective function value may serve as an overall loss value for the generator 408 and the classifier 416. The flow 400 may include transmitting the objective function value and / or other information from the classifier 416 to the generator 408 and the classifier 416 to update both the generator 408 and the classifier 416. Many different suitable objective functions may be used to calculate the objective function value. However, in testing one embodiment of the generator 408, the sum GANLoss + 0.83 SSIM + 0.17 L1 was shown to outperform other tested loss functions, such as GANLoss + L1 as used by the generator 408. GANLoss can be used to determine whether an image is real or generated. L1 loss can be used as an additional objective that is minimized to ensure that generated and real images have a minimum mean absolute error on top of GANLoss. A structural similarity index (SSIM) can be used to improve performance across multiple performance metrics and even reduce artifacts. Objective function value calculation 420 is described below.

[0077] The flow 400 may include receiving multiple pairs of brightfield images and corresponding ground truth fluorescent images, and iteratively training a generator 408 using each pair of images.

[0078] In some embodiments, the flow 400 can include preprocessing the brightfield image 404 and the ground truth fluorescent image 424. The raw brightfield image and the fluorescent image may have the lowest contrast and require enhancement before being used to train the generator 408. For example, during testing, the pixel intensities for the individual channels of the fluorescent image were generally biased toward zero, which could be due to the image being mostly black (i.e., background) except for areas containing organoids and / or cells.

[0079] In some embodiments, the artificial fluorescent image 412 can be used to provide live / dead cell counts. To enhance the contrast of the artificial fluorescent image 412 and improve the ability to count live / dead cells from the artificial fluorescent image 412, both the bright field image 404 and the corresponding ground truth image 424 can undergo contrast enhancement to brighten and clarify the organoids / cells.

[0080] In some embodiments, multiple brightfield images and multiple ground truth fluorescent images can be generated per well, e.g., for a 96-well plate, there can be approximately 9-16 sites per well that are imaged.

[0081] In some embodiments, the bright field raw image and the ground truth fluorescent image are in the range [0, 2 16 [0, 255]. First, a contrast enhancement process that may be included in the organoid image analysis application 132 can convert each image to an unsigned byte format with values ​​in the range of [0, 255]. Next, the contrast enhancement process can stretch and clip each pixel intensity to the desired output range.

[0082] In some embodiments, the desired intensity range of an input image to be stretched can be determined for each image as follows: For three pixel intensities corresponding to the three fluorophores used to generate the fluorescence image, the input range can be rescaled using the mode of the pixel intensity distribution as the lower limit and 1 / 10 of the maximum pixel intensity as the upper limit. The contrast enhancement process can select the upper limit to avoid oversaturated pixels and focus on cellular signals. The contrast enhancement process can normalize each pixel intensity based on the lower and upper limits, which serve as the min / max range, using a min-max norm, and then multiply each pixel by the output range [0, 255]. For the bright-field image 404, the contrast enhancement process can determine the input range by uniformly stretching the 2nd and 98th percentiles of pixel intensities into the output range [0, 255].

[0083] For images with low signals, background noise may be included in the output range. To minimize the remaining background noise, the contrast enhancement process can clip the minimum pixel value by two integer values ​​for the red and green channels and by three integer values ​​for the blue channel, resulting in a wider intensity range on average. The maximum pixel value can be increased accordingly to maintain the intensity range per image.

[0084] In some embodiments, the ground truth image 424 may be a 1024x1024x3 RGB image including a blue channel corresponding to nuclei (Hoecsht), a green channel corresponding to apoptotic cells (Caspase), and a red channel corresponding to dead cells (TO-PRO-3). In some embodiments, the flow 400 may include enhancing the ground truth image 424. In some embodiments, the enhancement may include contrast enhancing the blue, green, and red channels to brighten and sharpen the organoids and / or cells in the ground truth image 424. In some embodiments, the flow may downconvert pixel intensities in the ground truth image 424 (e.g., converting 16-bit pixel intensities to 8-bit intensities). After converting the pixel intensities, the flow 400 may include rescaling the pixel intensities to 1 / 10 of the maximum pixel intensity as an upper limit, and further rescaling to an integer mode of pixel intensity + 2 for the red and green channels and pixel intensity + 3 for the blue channel.

[0085] In some embodiments, the classifier 416 may output a predicted label (e.g., “0” or “1”) to the objective function calculation 420. The predicted label may indicate whether the artificial fluorescent image 412 is fake or real. In some embodiments, the objective function may be calculated as a weighted sum of GAN Loss, SSIM, and L1. In some embodiments, GAN Loss may be calculated based on the predicted label output by the classifier. GAN Loss may be used to determine whether the artificial fluorescent image 412 is real or generated. In some embodiments, L1 loss may be calculated based on the artificial fluorescent image 412 and the corresponding ground truth image. L1 loss may be used as an additional objective to be minimized to ensure that the artificial fluorescent image 412 and the corresponding ground truth image have a minimum mean absolute error in addition to GAN Loss.

[0086] Some machine learning models, such as the pix2pix model, may use only GANLoss and L1 loss in training the generator. As mentioned above, the objective function calculation 420 can include the SSIM metric in addition to the GANLoss and L1 loss, which can improve the performance of the generator 408 compared to a generator trained using only GANLoss and L1 loss.

[0087] In some embodiments, the objective function implemented in the objective function calculation is:

[0088]

number

[0089] can be defined as where λ+β=1 and L L1 is the mean absolute error loss, and 1-L SSIM(G) is the structural similarity index loss between the generated image G (e.g., the fluorescence image 412) and the corresponding ground truth image. In some embodiments, λ may be 0.17 and β may be 0.83. In some embodiments, λ may be selected from 0.1 to 0.3 and β may be selected from 0.7 to 0.9.

[0090] In some embodiments, SSIM can take into account the brightness (l), contrast (c), and structure (s) of the two images and calculate a metric between 0 and 1, where 1 indicates a perfect match between the two images;

[0091]

number

[0092] It is expressed as C1, C2, and C3 are C1=(K1L) 2 , C2=(K2L) 2 , and C3=C2 / 2 (5) is a small constant defined by where K1 and K2 are two scalar constants, each less than 1, and L is the dynamic range of pixel intensities (i.e., 256). SSIM is calculated using the formula SSIM(x,y)=[l(x,y)] α [c(x,y)] β [s(x,y)] γ (6)

[0093]

number

[0094] can be calculated as where l, c, and s are calculated using the mean, variance, and covariance, respectively, of two images of the same size using a fixed window size. α, β, and γ are constants set to 1. In addition to structural similarity, we also evaluated model predictions using the square root of the mean squared error, which is the sum of the squared differences of pixel intensities.

[0095] In some embodiments, the proposed loss function may be: DiscriminatorLoss=MSELoss{Real Prediction,1}+MSELoss{Generated Prediction,0}+MSELoss{Predicted Viability,Viability} GeneratorLoss=MSELoss{Generated Prediction,1}+MAE{Generated Fluorescent,Real Fluorescent}+SSIM{Generated Fluorescent,Real Fluorescent}, where MSE is the mean squared error loss, MAE is the mean absolute error, and SSIM is the structural similarity index. The RCA model was trained for 30 epochs with a learning rate of 2e-3 and the Adam optimizer. Images with a resolution of 1024 x 1024 were captured at a magnification of 10X. These were randomly flipped as a data augmentation step.

[0096] In some embodiments, after the dye is added to a cell culture well, the cells in that well cannot continue to be used in experiments, making it difficult or impossible to measure cell death in that well at subsequent time points. In some embodiments, flow 400 can include generating artificial fluorescent images, which can reduce imaging time by 10-fold compared to using dyes to generate fluorescent images. Standard fluorescent imaging can take up to an hour to perform. In some embodiments, flow 400 can be used in conjunction with a drug screening platform that incorporates patient-derived tumor organoids to uniquely interpret tumor organoids (TOs), which have limited biomass and intratumoral clonal heterogeneity. This platform combines high-content fluorescent confocal image analysis with a robust statistical analysis approach to measure hundreds of discrete data points of TO viability from as few as 10^3 cells.

[0097] 4 and further referring to FIG. 5, an exemplary flow 500 for generating an artificial fluorescent image 512 is illustrated. The flow 500 can include providing an input bright field image 504 of plated cells to a trained model 508. The trained model 508 can include a generator 408, which can be trained using the flow 400. The trained model 508 can output an artificial fluorescent image 512. This fluorescent image 512 can be used to generate a live / dead assay readout for cancer cells in tissue organoids and / or analyze the effectiveness of different drugs and / or dosages.

[0098] Notably, flow 500 can generate fluorescent image 512 without the use of fluorescent dyes, which offers several advantages over traditional fluorescent imaging processes that require the use of fluorescent dyes. Some dyes are cytotoxic and must be added for a certain amount of time before imaging. Additionally, after some dyes are added to cell culture wells, the cells in the wells cannot be subsequently used for reimaging, making it difficult to measure cell death in those wells at a later time point. Therefore, flow 500 can improve the ease of generating fluorescent images because it can require only bright-field imaging, which is not time-dependent like traditional fluorescent imaging. Additionally, flow 500 can increase the speed at which fluorescent images can be acquired because it does not require fluorescent dyes to be applied to the cells, and flow 500 does not require waiting for the fluorescent dye to diffuse before imaging the cells. As another example, flow 500 can enable multiple fluorescent images to be generated for each cell well at multiple different time points. The fluorescent dyes used in traditional fluorescent imaging can damage cells to the point that they prevent reimaging. In contrast, flow 500 can be used to generate multiple fluorescent images over a period of days, weeks, months, or the like. Thus, flow 500 can yield many more data points per cell well than conventional fluorescence imaging.

[0099] In some embodiments, a single trained model (e.g., trained model 508) can be trained with training data including a set of bright-field images and corresponding fluorescent images associated with one of six or more organoid cell lines, each having a different cancer type, whereby each organoid cell line is represented in the training data. In some embodiments, a single trained model (e.g., trained model 508) can be a pan-cancer model trained to generate artificial fluorescent staining images from bright-field images associated with any cancer type. In some embodiments, a trained model can be trained with only training data including images associated with one organoid cell line (e.g., one cancer type).

[0100] 6 shows an exemplary neural network 600. Neural network 600 can be trained to receive an input image 604 and generate an artificial fluorescence image 608 based on the input image 604. In some embodiments, input image 604 can be a raw brightfield image that has been processed to enhance contrast and / or modify other characteristics to enhance the raw brightfield image and potentially generate a better artificial fluorescence image (e.g., fluorescence image 608).

[0101] In some embodiments, neural network 600 can include a Unet architecture. In some embodiments, the Unet architecture can be sized to receive a 256x256x3 input image. The 256x256x3 input image can be a bright-field image. In some embodiments, the input image can be a 256x256x1 image. In some embodiments, generator 408 of FIG. 4 and / or trained model 508 of FIG. 5 can include neural network 600.

[0102] FIG. 7 illustrates an exemplary classifier 700. In some embodiments, the classifier 700 of FIG. 7 may be included as the classifier 416 in the flow 400 shown in FIG. 4. In some embodiments, the classifier 700 may be a 1×1 PatchGAN. In some embodiments, the classifier 700 may receive a bright-field image 704 and a fluorescent image 708. The fluorescent image may be an artificial fluorescent image (e.g., the fluorescent image 608 of FIG. 6) or a ground truth fluorescent image. In some embodiments, the bright-field image 704 and the fluorescent image 708 may each be a 256×256×3 input image. In some embodiments, the bright-field image 704 and the fluorescent image 708 may be concatenated. In some embodiments, the concatenated image may be a 256×256×6 input image.

[0103] In some embodiments, the classifier 700 can receive the bright field image 704 and the fluorescent image 708 and generate a predicted label 712 indicating whether the fluorescent image 708 is real or fake. In some embodiments, the predicted label 712 can be a "0" indicating that the fluorescent image 708 is fake and a "1" indicating that the fluorescent image 708 is real. In some embodiments, the classifier 700 can include a neural network.

[0104] 4-7, in some embodiments, flow 400, flow 500, neural network 600, and classifier 700 can be implemented using Pytorch version 1.0.0. In some embodiments, flow 400 can be used to train generator 408 to generate artificial fluorescent images for a colon cancer organoid cell line. In some embodiments, flow 400 can be used to train generator 408 to generate artificial fluorescent images for a gastric cancer organoid cell line.

[0105] 8 shows an exemplary process 800 for training a model to generate artificial fluorescent staining images of one or more organoids based on an input bright-field image. In some embodiments, the model may be the generator 408 and / or the neural network 600 of FIG. 4. In some embodiments, the model may include a neural network that can receive an input bright-field image and output a single three-channel fluorescent image (e.g., a 256×256×3 image). In some embodiments, the model may include three neural networks that can each receive a bright-field image and output a one-channel fluorescent image (e.g., a 256×256×1 image). The one-channel images may then be combined into a single three-channel fluorescent image.

[0106] In some embodiments, process 800 can be used to train a model to output artificial fluorescent images of objects other than tumor organoids using a large number of non-fluorescent images (e.g., bright field images) and fluorescently stained images (which may have more or fewer than three channels) as training data.

[0107] Process 800 may be implemented as computer-readable instructions in one or more memories or other non-transitory computer-readable media and executed by one or more processors in communication with one or more memories or other media. In some embodiments, process 800 may be implemented as computer-readable instructions in memory 220 and / or memory 240 and executed by processor 204 and / or processor 224.

[0108] At 804, process 800 can receive training data. In some embodiments, the training data can include multiple bright-field images and multiple associated real-fluorescence images of organoids. In some embodiments, the organoids can be from a single tumor organoid cell line. In some embodiments, the bright-field images and real-fluorescence images can be preprocessed to enhance contrast as described above. In some embodiments, the bright-field images and real-fluorescence images can be raw images that have not undergone any preprocessing, such as contrast enhancement.

[0109] At 808, if the training data includes brightfield raw images and / or real fluorescence raw images (i.e., "YES" at 808), process 800 can proceed to 812. If the training data does not include any brightfield raw images or real fluorescence raw images (i.e., "NO" at 808), process 800 can proceed to 816.

[0110] At 812, process 800 can preprocess at least a portion of the brightfield and / or real fluorescence images. In some embodiments, at 812, process 800 can enhance the contrast of any raw brightfield and / or real fluorescence images included in the training data. In some embodiments, the raw brightfield and ground truth fluorescence images are in the range [0, 2 16 ]. In some embodiments, process 800 can convert each image to an unsigned byte format, with values ​​in the range [0, 255]. Process 800 can then stretch and clip each pixel intensity to the desired output range.

[0111] In some embodiments, process 800 can stretch the desired intensity range of the input for each image. For three pixel intensities corresponding to the three fluorophores used to generate the actual fluorescence image, process 800 can rescale the input range using the mode of the pixel intensity distribution as the lower limit and 1 / 10 of the maximum pixel intensity as the upper limit. Process 800 can determine the upper limit to avoid oversaturated pixels and focus on cellular signals. Process 800 can normalize each pixel intensity based on the lower and upper limits, which serve as the min / max range, using a min-max norm, and then multiply each pixel by the output range [0, 255]. For each bright-field image included in the training data, process 800 can determine the input range by uniformly stretching the 2nd and 98th percentiles of pixel intensities into the output range [0, 255].

[0112] For images with low signals, background noise may be included in the output range. In some embodiments, to minimize remaining background noise, process 800 can clip the minimum pixel value by two integer values ​​for the red and green channels and by three integer values ​​for the blue channel, resulting in a wider intensity range on average. In some embodiments, process 800 can increase the maximum pixel value accordingly, preserving the intensity range per image.

[0113] At 816, process 800 can provide the brightfield image to a model. As described above, in some embodiments, the model can be generator 408 of FIG. 4 and / or neural network 600 of FIG. 6. In some embodiments, the model can include three neural networks, each of which can receive a copy of the brightfield image and output a different channel (e.g., red, green, or blue) of the artificial fluorescence image.

[0114] At 820, process 800 can receive an artificial fluorescence image from the model. The model can generate an artificial fluorescence image (e.g., artificial fluorescence image 412) based on a bright-field image (e.g., bright-field image 404) provided to the model. In some embodiments, process 800 can receive three one-channel images from three neural networks included in the model and combine the one-channel images into a single three-channel artificial fluorescence image.

[0115] At 824, process 800 can calculate an objective function value based on the bright-field image, the real fluorescence image associated with the bright-field image, and the artificial fluorescence image. In some embodiments, process 800 can determine a predicted label indicating whether the artificial fluorescence image is authentic by providing the artificial fluorescence image and the real fluorescence image to a classifier (e.g., classifier 416). In some embodiments, the objective function value can be calculated using equation (1) above, where λ is 0.17 and β is 0.83. In some embodiments, λ can be selected from the range of 0.1 to 0.3, and β can be selected from the range of 0.7 to 0.9. In some embodiments, the learning rate can be fixed at 0.0002 for the first number of epochs of training (e.g., 15 epochs) and then linearly decay to zero over the second number of epochs (e.g., 10 epochs).

[0116] At 828, the process 800 can update a model (e.g., the generator 408) and a classifier (e.g., the classifier 416) based on the objective function value. In some embodiments, the model and the classifier can each include a neural network. In some embodiments, the process 800 can update the weights of layers included in the neural networks included in the model and the classifier based on the objective function value.

[0117] At 832, process 800 may determine whether there are bright-field images in the training data that have not been provided to the model. If there are bright-field images in the training data that have not been provided to the model (e.g., "YES" at 832), the process may proceed to 816, where the bright-field images may be provided to the model. If there are no bright-field images in the training data that have not been provided to the model (e.g., "NO" at 832), the process may proceed to 836.

[0118] At 836, process 800 may output the model. At 836, the model has been trained and may be referred to as a trained model. In some embodiments, process 800 may output the trained model to at least one of a memory (e.g., memory 220 and / or memory 240) and / or a database (e.g., trained model database 128). The trained model may be accessed and used in several processes, such as the processes of FIGS. 9 and 13. Process 800 may then end.

[0119] FIG. 9 illustrates an exemplary process 900 for generating artificial fluorescent images of one or more organoids based on bright-field images. More specifically, process 900 can generate the artificial fluorescent images using a trained model. In some embodiments, the model can be generator 408, trained model 508, and / or neural network 600 of FIG. 6 trained using process 800. In some embodiments, the model can include a neural network that can receive an input bright-field image and output a single three-channel fluorescent image (e.g., a 256×256×3 image). In some embodiments, the model can include three neural networks that can each receive a bright-field image and output a one-channel fluorescent image (e.g., a 256×256×1 image). The one-channel images can then be combined into a single three-channel fluorescent image.

[0120] In some embodiments, process 900 can be used to generate artificial fluorescent image (can have 1 channel, 2 channels, 3 channels, etc.) of objects other than tumor organoids using non-fluorescent image (for example, bright field image).In this way, objects other than tumor organoids that require fluorescent staining to be properly imaged can be artificially generated without the use and / or drawbacks of fluorescent dyes.

[0121] Process 900 may be implemented as computer-readable instructions in one or more memories or other non-transitory computer-readable media and executed by one or more processors in communication with one or more memories or other media. In some embodiments, process 900 may be implemented as computer-readable instructions in memory 220 and / or memory 240 and executed by processor 204 and / or processor 224.

[0122] At 904, process 900 can receive a brightfield image of one or more organoids (e.g., brightfield image 404 of FIG. 4 and / or brightfield image 504 of FIG. 5). In some embodiments, the brightfield image can be preprocessed to enhance contrast as described above. In some embodiments, the brightfield image can be a raw image that has not undergone any preprocessing, such as contrast enhancement.

[0123] At 908, process 900 may determine whether the brightfield image is unprocessed (i.e., raw). If the brightfield image is unprocessed (i.e., "YES" at 908), process 900 may proceed to 912. If the brightfield image is not unprocessed (i.e., "NO" at 908), process 900 may proceed to 916.

[0124] At 912, the process 900 can preprocess the brightfield image. In some embodiments, the brightfield image is a pixel in the range [0, 2 16[0, 255]. In some embodiments, process 900 can convert the bright-field image to an unsigned byte format, with values ​​in the range [0, 255]. In some embodiments, process 900 can convert the bright-field image to another format with fewer bits than the original pixel intensities. Process 900 can then stretch and clip each pixel intensity to a desired output range. In some embodiments, process 900 can determine the input range for the bright-field image by uniformly stretching the 2nd and 98th percentiles of pixel intensities in the bright-field image to the output range [0, 255].

[0125] At 916, process 900 can provide the brightfield image to a trained model. In some embodiments, the model can include generator 408 of FIG. 4 trained using process 800 of FIG. 8, trained model 508, and / or neural network 600 trained using process 800 of FIG. 8. In some embodiments, the trained model can include three neural networks, each of which can receive a copy of the brightfield image and output a different channel (e.g., red, green, or blue) of the artificial fluorescence image.

[0126] At 920, the process 900 can receive an artificial fluorescence image from the trained model. In some embodiments, the process 900 can receive three one-channel images from the three neural networks included in the trained model, and combine the one-channel images into a single three-channel artificial fluorescence image. The artificial fluorescence image can indicate whether the cells included in the tumor organoid are alive or dead.

[0127] At 924, process 900 can output the artificial fluorescent image. In some embodiments, process 900 can output the artificial fluorescent image to at least one of a memory (e.g., memory 220 and / or memory 240) and / or a display (e.g., display 116, display 208, and / or display 228). The artificial fluorescent image can be used to provide a live / dead count of cells within the organoid. In some embodiments, process 900 can output the artificial fluorescent image to an automated cell counting process to receive an accurate live / dead count of cells, the percentage of viable (e.g., alive) or dead cells, and / or a cell count report within the artificial fluorescent image. For example, process 900 can output the artificial fluorescent image to CellProfiler, available at https: / / cellprofiler.org. In some embodiments, process 900 can output one or more channels of the artificial fluorescent image to an automated cell counting process to receive a cell count report, a percentage of viable (e.g., alive) or dead cells, and / or an accurate live / dead count of cells in the artificial fluorescent image. In some embodiments, process 900 can output a brightfield image to a trained model to receive a cell count report, a percentage of viable (e.g., alive) or dead cells, and / or an accurate live / dead count of cells in the artificial fluorescent image. In some embodiments, process 900 can output a combination of one, two, or three channels of the brightfield image and the artificial fluorescent image (e.g., image embeddings combined by concatenation) to an automated cell counting process to receive a cell count report, a percentage of viable (e.g., alive) or dead cells, and / or an accurate live / dead count of cells in the artificial fluorescent image.

[0128] In some embodiments, at 924, process 900 can identify cells in the artificial fluorescence image by converting each channel to grayscale, enhancing or suppressing certain features such as specks, rings, neurites, and dark holes, identifying primary objects in the whole cell channel whose typical diameter (in pixels) is set to a value anywhere between 2 and 20 using minimum cross-entropy thresholding at a smoothing scale of 1.3488, and identifying primary objects in the dead cell channel whose typical diameter (in pixels) is set to a value anywhere between 5 and 20. In this way, process 900 can generate a cell count report. In some embodiments, process 924 can determine whether a drug and / or dosage is effective in killing tumor organoid cells based on the live / dead cell count. In some embodiments, at 924, process 900 can extrapolate a dose-response from the distribution of organoid viability at a single concentration.

[0129] In some embodiments, cell counting reports can be analyzed to quantify the effectiveness of drugs in killing specific strains of tumor organoid cells. For example, if a certain drug concentration reduces the number of live cells and / or increases the number of dead cells, the drug can be evaluated as having a higher effect of killing specific strains of tumor organoid cells. For each strain of tumor organoid cells, the characteristics of the tumor organoid cells (e.g., molecular data including detected mutations, RNA expression profiles measured in tumor organoid cells, and / or clinical data associated with the patient from which the tumor organoid was derived) and the results of each drug administration (including efficacy evaluation) can be stored in a drug assay result database. These results can be used to match therapies to patients. For example, if a patient has a cancer with characteristics similar to those of a tumor organoid cell line, drugs that are evaluated as effective in killing these tumor organoid cells can be matched to the patient.

[0130] In some embodiments, process 900 can generate a cell count, a cell count report, and / or a report based on the artificial fluorescent image. In some embodiments, process 900 can cause the report to be output to at least one of a memory (e.g., memory 220 and / or memory 240) and / or a display (e.g., display 116, display 208, and / or display 228). Process 900 can then terminate.

[0131] FIG. 10 shows exemplary raw images before and after preprocessing. The raw image before preprocessing includes a brightfield image 1004, a blue / total nuclei channel fluorescence image 1008, a green / apoptosis channel fluorescence image 1012, a red / pink / death channel fluorescence image 1016, and a combined three-channel fluorescence image 1020. The preprocessed image includes a brightfield image 1024, a blue / total nuclei channel fluorescence image 1028, a green / apoptosis channel fluorescence image 1032, a red / pink / death channel fluorescence image 1036, and a combined three-channel fluorescence image 1040. The organoids and cells are brighter and more distinct in the preprocessed images. In some embodiments, the preprocessed images 1024-1040 can be generated in 812 of process 800 of FIG. 8.

[0132] FIG. 11 illustrates an exemplary flow chart 1100 for culturing tumor organoids. This is the culturing of patient-derived tumor organoids. Flow chart 100 can include obtaining tumor tissue from same-day surgery, dissociating cells from the tumor tissue, and culturing tumor organoids from the cells. An example of a system and method for culturing tumor organoids is described in U.S. Patent Application No. 16 / 693,117, filed November 22, 2019, entitled "Tumor Organoid Culture Compositions, Systems, and Methods," which is incorporated herein by reference in its entirety. Tumor tissue received from a hospital is cultured to form tumor organoids.

[0133] Figure 12 shows an exemplary flow 1200 for performing a drug screening by the system and method described herein. In some embodiments, flow 1200 can include: dissociating tumor organoids into single cells; seeding the cells (e.g., in a well plate, such as a 96-well plate and / or a 384-well plate); allowing the cells to grow into organoids for a predetermined time (e.g., 72 hours); treating the organoids with at least one therapeutic technique; and imaging the tumor organoids after a predetermined time (e.g., 72 hours) since the tumor organoids have been treated. In some embodiments, only bright-field imaging is performed on tumor organoids, and any bright-field image generated can be used to generate an artificial fluorescent image using the process 900 of Figure 9. Then, a live / dead count can be generated based on the artificial fluorescent image. An example of a system and method for using tumor organoids for drug screening is described in U.S. Provisional Patent Application No. 62 / 924,621, entitled "Systems and Methods for Predicting Therapeutic Sensitivity," filed October 22, 2019 (and PCT / US20 / 56930, filed October 22, 2020), which is incorporated by reference in its entirety.

[0134] 13 shows an exemplary process 1300 that can generate artificial fluorescent images of at least one organoid at multiple time points. In particular, process 1300 can provide advantages over standard fluorescent imaging techniques. As mentioned above, the fluorescent dyes used to generate standard fluorescent images can damage (e.g., kill) cells within the organoid, and do not allow fluorescent images to be generated at different time points (e.g., every 12 hours, every 24 hours, every 72 hours, every week, etc.). In contrast, process 1300 allows repeated fluorescent imaging of organoids because only bright-field images (which do not damage the organoid) are required and artificial fluorescent images can be generated based on the bright-field images.

[0135] Process 1300 may be implemented as computer-readable instructions in one or more memories or other non-transitory computer-readable media and executed by one or more processors in communication with one or more memories or other media. In some embodiments, process 1300 may be implemented as computer-readable instructions in memory 220 and / or memory 240 and executed by processor 204 and / or processor 224.

[0136] At 1304, process 1300 can receive instructions to analyze the treated organoids at multiple time points. In some embodiments, the organoids can be plated (e.g., in a well plate, such as a 96-well plate and / or a 384-well plate). In some embodiments, the organoids can be plated on multiple well plates. In some embodiments, the organoids can be plated on one or more Petri dishes. In some embodiments, the organoids can be treated using a variety of different treatment methods, which can vary in drug type, drug concentration, and / or other parameters. In some embodiments, each well in a well plate can be associated with a different treatment.

[0137] In some embodiments, the multiple time points can represent the time after organoid is treated.For example, the 12-hour time point can be 12 hours after organoid is treated.In some embodiments, the multiple time points can be arranged at regular intervals.For example, the multiple time points can appear every 12 hours, every 24 hours, every 72 hours, every week, etc.In some embodiments, the multiple time points can be arranged at irregular intervals.For example, the time points can include the first time point at 6 hours, the second time point at 24 hours, the third time point at 3 days, the fourth time point at 1 week, and the fifth time point at 28 days.

[0138] At 1308, process 1300 can wait until the next time point in the plurality of time points. For example, if 6 hours have passed since the organoid was treated and the next time point is 12 hours, process 1300 can wait 6 hours.

[0139] At 1312, process 1300 can generate at least one brightfield image of the treated organoids. In some embodiments, process 1300 can generate a brightfield image of the treated organoids using a brightfield microscope and generate a fluorescent image of the cells using a confocal microscope, such as a confocal laser scanning microscope. In some embodiments, process 1300 can preprocess at least one brightfield image. For example, process 1300 can perform at least a portion of 912 in process 900 of FIG. 9 for each brightfield image. In some embodiments, multiple brightfield images can be generated for each well. For example, for a 96-well plate, there can be approximately 9-16 sites per well imaged.

[0140] At 1316, process 1300 can generate at least one artificial fluorescence image based on the at least one brightfield image. In some embodiments, process 1300 can provide each brightfield image to a trained model and receive from the trained model an artificial fluorescence image associated with the brightfield image. In some embodiments, the trained model can include generator 408 of FIG. 4 trained using process 800 of FIG. 8, trained model 508, and / or neural network 600 trained using process 800 of FIG. 8. In some embodiments, the trained model can include a neural network that can receive an input brightfield image and output a single three-channel fluorescence image (e.g., a 256×256×3 image). In some embodiments, the trained model can include three neural networks that can each receive a brightfield image and output a one-channel fluorescence image (e.g., a 256×256×1 image). The one-channel images can then be combined into a single three-channel fluorescence image. The at least one artificial fluorescent image can indicate whether the cells contained in the tumor organoid are alive or dead.

[0141] At 1320, process 1300 can output at least one fluorescent image. In some embodiments, process 1300 can output at least one artificial fluorescent image to at least one of a memory (e.g., memory 220 and / or memory 240) and / or a display (e.g., display 116, display 208, and / or display 228). The at least one artificial fluorescent image can be used to provide a live / dead count of cells within the organoid. In some embodiments, process 900 can output the artificial fluorescent image to an automated cell counting process to obtain an accurate live / dead count of cells within the artificial fluorescent image. For example, process 900 can output the artificial fluorescent image to CellProfiler, available at https: / / cellprofiler.org. In this manner, process 1300 can automatically generate live / dead counts for multiple wells at multiple time points, which allows drug therapy experiments to be performed faster and more data to be collected from the same number of wells compared to standard fluorescent dye imaging techniques that kill cells.

[0142] In some embodiments, at 1320, process 1300 can identify cells in the artificial fluorescence image by converting each of the channels to grayscale, enhancing and suppressing certain features such as specks, ring shapes, neurites, and dark holes, identifying primary objects in the whole cell channel whose typical diameter (in pixels) is set to anywhere between 2 and 20 using minimum cross-entropy thresholding at a smoothing scale of 1.3488, and identifying primary objects in the dead cell channel whose typical diameter (in pixels) is anywhere between 5 and 20. In this manner, process 1300 can generate a cell count report. In some embodiments, process 1300 can generate a cell count, a cell count report, and / or a report based on the artificial fluorescence image. In some embodiments, process 1300 may cause the report to be output to at least one of a memory (e.g., memory 220 and / or memory 240) and / or a display (e.g., display 116, display 208, and / or display 228). Process 1300 may then terminate.

[0143] In some embodiments, process 800 of FIG. 8, process 900 of FIG. 9, and / or process 1300 of FIG. 13 may be included in organoid image analysis application 132 of FIG. 1.

[0144] 14 shows a table representing an exemplary assay or well plate configuration. More specifically, the table shows the configuration of treatments by well in a 24×16 well plate.

[0145] In some embodiments, to seed tumor organoids in well plates, single cell suspension of tumor organoid cells can be generated using a predetermined protocol.In some embodiments, to seed in 24x16 well plates, 24 well plate culture can be dissociated into single cells, and then seeded in 384 well plates in a mixture of 30% Matrigel and 70% culture medium.This setup allows tumor organoids to be formed from individual cells for assay, and can maintain the heterogeneity of tumor organoids in each well.Approximately 2000 cells are seeded per well, which can allow tumor organoids (TO) to be sufficiently formed without overcrowding the plate.

[0146] The number of usable wells in each 384-well plate can be 330 wells. There can be two sites imaged within each well. For a 96-well plate, there can be approximately 9-16 sites per well imaged. In some embodiments, each row in the well plate can receive a different drug. In some embodiments, a control (e.g., staurosporine) can be immobilized in row A. The vehicle can be two rows, with DMSO in the first half and staurosporine in the second half. In some embodiments, each row can technically receive a drug concentration in triplicate. [Example]

[0147] In this example, tumor organoids in each well were stained with three fluorophores for high-content fluorescent confocal imaging. To obtain fluorescent readouts, a high-content imager (ImageXpress Confocal, Molecular Devices) was used for data acquisition. Images were acquired at 10X magnification using a 50-micron slit rotating disk aperture. Four channels were acquired using incandescent brightfield and LED light sources, which used the manufacturer's default settings for 4',6-diamidino-2-phenylindole (DAPI), fluorescein isothiocyanate (FITC), and cyanine 5 (CY5) to acquire data from Hoechst 33342 (Thermo), Caspase-3 / 7 Reagent (Essen Bioscience), or TO-PRO-3 (Thermo), respectively.

[0148] In this example, the experimental setup uses a 384-well plate with 330 usable wells within each plate. Each well has two sites imaged, so each plate has a total of 660 pairs of brightfield and fluorescent images. At 10X magnification, two images are imaged per well at two sites, and the stack of images in the Z-plane is 1-100 heights with 15 micron height increments per Z-plane. The Z-stack images are projected in 2D for analysis. Three fluorophores for each brightfield image visualize all nuclei (Hoechst 33342, blue), apoptotic cells (Caspase-3 / 7 Apoptosis Assay Reagent, green), and dead cells (TO-PRO-3, red).

[0149] The final dataset contained registered fluorophores from two patient cell lines: colon cancer patient cell line A, which consisted of 9,900 pairs of brightfield and fluorescent images, and gastric cancer patient cell line B, which consisted of 10,557 pairs of brightfield and fluorescent images.

[0150] The generator 408 and discriminator 416 models were implemented in Pytorch version 1.0.0. Colon cancer organoid patient cell line A was selected to evaluate the performance of all analyzed models. This organoid line contained 8,415 pairs of brightfield and fluorescent images across 15 experimental plates, which underwent an 80-10-10 split for training, testing, and validation, resulting in 6,930 images for training, 742 images for validation, and 743 images for testing. Images were loaded one at a time. Parameter fine-tuning was achieved using only the validation set. The learning rate was fixed at 0.0002 for the first 15 epochs of training, then linearly decayed to 0 for 10 epochs, for a total of 25 epochs for training.

[0151] A fixed set of 743 bright-field images and corresponding fluorescence images randomly sampled from 15 experimental plates was selected as the test set. Evaluation of all experiments was performed on the fixed test set. First, the effect of training three separate models (3 models) was evaluated for each fluorophore channel versus training one single model (1 model) on the combined fluorescence readout. For the three models, predictions for each fluorophore were finally combined and evaluated. Performance was assessed quantitatively using the structural similarity index and the root mean square error, and qualitatively and visually using heat maps.

[0152] No significant improvement in fluorescent dye prediction was observed when using the three-model approach, training a separate generator for each channel. Table 1 reports the average SSIM and root mean square error across predictions for each channel for all 743 test images, and Figure 15 shows example images and organoids. Furthermore, the three-model approach required three times the computational resources with only limited RMSE improvement, rationalizing that a one-model implementation is sufficient and efficient for image-to-image conversion from brightfield images to combined fluorescent readouts. Table 1 indicates that lower RMSE and higher SSIM indicate better performance.

[0153] [Table 1]

[0154] FIG. 15 shows example images generated using a single neural network model (1 model) and a three-neural network model (3 model). The first image 1504 is an example of a ground truth fluorescence image. The second image 1508 is an example of an artificial fluorescence image generated using the 1 model, which has a single neural network that can receive an input brightfield image and output a single three-channel fluorescence image (e.g., a 256×256×3 image). The third image 1512 is an example of an artificial fluorescence image generated using the 3 model, which has three neural networks that can each receive a brightfield image and output a one-channel fluorescence image (e.g., a 256×256×1 image). The fourth image 1516 is an example of a grayscale error map between the second image 1508 and the ground truth image 1504. The fifth image 1520 is an example of a grayscale error map between the third image 1512 and the ground truth image 1504. The sixth image 1524, seventh image 1528, eighth image 1532, ninth image 1536, and tenth image 1540 are examples of enlarged organoids in the first image 1504, second image 1508, third image 1512, fourth image 1516, and fifth image 1520, respectively.

[0155] Next, the effect of adding SSIM loss to the objective function of one model was detailed. The objective function in Equation 1 is a weighted combination of L1 and SSIM losses (λL1 + βSSIM). The impact of SSIM was tested by uniformly evaluating β = {0; 0.25; 0.5; 0.75; 1}. Table 2 highlights the performance on the holdout test set using different β. The combination of β = 0.75 (and λ = 0.25) shows the best performance of the trained model (e.g., trained model 508) in terms of both SSIM and RMSE.

[0156] [Table 2]

[0157] To determine whether the accuracy of the trained model (β = 0.75) was driven by specific improvements in predicting a single fluorophore, such as predicting DAPI (total cells) or FITC (stained / apoptotic cells), the average RMSE and SSIM across each channel were examined. These results are shown in Table 3.

[0158] [Table 3]

[0159] The model trained at β = 0.75 showed consistent RMSE and SSIM scores across all channels. The performance of the model trained at β = 0.75 was evaluated when it was run on two novel patient colorectal cancer organoid lines (organoid lines B and C). Each novel line had a total of 648 brightfield and corresponding fluorescent readouts across different plates. Table 4 demonstrates that the model trained at β = 0.75, trained on a single organoid, can be transferred to other organoid lines. However, the differences between the two lines suggest some limitations. The differences between the two lines suggest that different colorectal cancer organoid lines may present different morphological features that may limit model transfer. In this case, retraining the current best model with some data from organoid line C or employing domain adaptation techniques could facilitate better generalization to organoid line C.

[0160] [Table 4] [Example]

[0161] Experiments were performed to test and improve model performance. Candidate models included the GANLoss+SSIM model, the GANLoss+SSIM+L1 model trained using the GANLoss+0.17L1+0.83SSIM model, the GANLoss+MS-SSIM model, and the GANLoss+0.83MS-SSIM+0.17L1 model.

[0162] First, three separate Pix2Pix models were employed to train individual fluorescence channels. The Avg SSIM and RMSE results across the same 743 blind test images described in Example 1 are shown below. Tables 5-8 show the results of the candidate models implemented in the three-model approach. Table 5 shows the results for the GANLoss+SSIM model. Table 6 shows the results for the GANLoss+0.83SSIM+0.17L1 model. Table 7 shows the results for the GANLoss+MS-SSIM model. Table 8 shows the results for the GANLoss+0.83MS-SSIM+0.17L1 model.

[0163] [Table 5]

[0164] [Table 6]

[0165] [Table 7]

[0166] [Table 8]

[0167] Fluorescence combined 3-channel image results The results in Table 9 below show that for each experiment we take a 3-channel pix2pix model and combine them to form a 3-channel IF counterpart. The individual channels were trained separately and then combined by stacking RGB.

[0168] [Table 9]

[0169] It was observed that GANLoss+SSIM or GANLoss+MS-SSIM alone did not perform as well as the other models. The combination of GANLoss+0.83SSIM+0.17L1 appeared to have the best performance. It was also found that GANLoss+L1 and GANLoss+SSIM did not do a good job of detecting blurry, poor-quality images. The GANLoss+SSIM+L1 model was able to accurately detect blur artifacts. The GANLoss+SSIM+L1 model recognized artifacts and blur better than the other models and completely avoided predictions when blur / artifacts were present in bright-field images. [Example]

[0170] In Example 2, the process of training three separate pix2pix models for different objective functions proved to require several GPUs (three per model) and extra effort in data curation. A similar performance analysis was performed to check whether similar / better RMSE and SSIM values ​​would be observed by training directly from brightfield to three-channel fluorescence using a single Pix2Pix model in an attempt to reduce GPU usage.

[0171] Table 10 below shows the results of directly training the style transfer to IF images for the same set of objective functions on the same test set of 743 images belonging to 10245. The number of GPUs was reduced from 15 to 5, and performance was slightly, but not significantly, better. Therefore, generating artificial fluorescence images using one model may be preferable, as it requires one-third the computational effort while achieving at least the performance of three models. In particular, one model trained using the objective function of the GANLoss+0.83MS-SSIM+0.17L1 model may outperform the other one and / or three models trained on the same training data.

[0172] [Table 10]

[0173] Table 11 below shows the results of a number of 1 models trained using different objective functions. GANLoss+0.75SSIM+0.25L1 has the best RMSE, while GANLoss+0.83SSIM+0.17L1 has the best SSIM performance.

[0174] [Table 11] [Example]

[0175] This example details exemplary Cell Profiler readouts. The Cell Profiler readouts include total cell counts and dead cell counts for real and corresponding artificial fluorescent images. In Table 12, each column indicates a specific site in a well within an experimental plate and whether it is an artificial or real image captured from an ImageXpress Micro microscope.

[0176] [Table 12]

[0177] Table 13 below shows the plate and well information for each image along with the SSIM / RMSE values.

[0178] [Table 13A] [Table 13B]

[0179] Table 13 shows that the fluorescent images can produce similar cell counts compared to the corresponding real fluorescent images. [Example]

[0180] In some embodiments, large-scale drug assay in tumor organoid can increase the throughput of assay.This high-throughput screening can be used to verify or test drug efficacy or discover new therapeutic methods.In some embodiments, 3D TO can be more similar to the tumor that they originate from than the 2D clone-established cell line that originates from that tumor.

[0181] In this example, tumor tissue removed by biopsy is dissociated into single cells and grown into three-dimensional (3D) tumor organoid (TO) cultures containing TO. The TOs are then dissociated into single cells and grown in 384-well tissue culture plates for 72 hours. Each well receives either no treatment (or mock treatment) or a dose (concentration) of a small molecule inhibitor or chemotherapeutic agent, and the effect of the treatment on the cells within the TO is measured. In one example, over 1,000 drugs can be tested. In another example, several concentrations of 140 drugs can be tested.

[0182] In one example, the therapeutic agent is one of 351 small molecule inhibitors, and seven doses of each therapeutic agent are tested on two different organoid types (two organoid cell lines), each derived from a separate patient sample. In this example, one organoid type is a gastric cancer organoid line, and the other is a colorectal cancer organoid line. In one example, the efficacy of the treatment can be measured by counting the number of dead and / or viable cells in a well after treatment. In this example of fluorescent staining, cell nuclei are stained blue with Hoechst 33342, dying (apoptotic) cells are stained green with Caspase-3 / 7 Apoptosis Assay Reagent, and dead cells are stained red with TO-PRO-3.

[0183] In this example, a gastric cancer organoid line has amplification of the HER2 gene. Afatinib (a drug that targets HER2, among other molecules) and two other drugs that target HER2 effectively kill this gastric cancer organoid line.

[0184] In some embodiments, the methods and systems described above can be utilized in conjunction with or as part of a digital and laboratory healthcare platform generally directed to healthcare and medical research. It should be understood that many uses of the methods and systems described above in conjunction with such platforms are possible. One example of such a platform is described in U.S. Patent Application No. 16 / 657,804, filed October 18, 2019, entitled "Data Based Cancer Research and Treatment Systems and Methods," which is incorporated herein by reference in its entirety for all purposes.

[0185] For example, in some embodiments, the methods and systems described above may include microservices that constitute a digital and laboratory healthcare platform supporting artificial fluorescence image generation and analysis. Some embodiments may include a single microservice for performing and distributing artificial fluorescence image generation, or multiple microservices, each with a specific role that together implement one or more of the above embodiments. In one example, a first microservice may perform training data generation to distribute training data to a second microservice to train a model. Similarly, the second microservice may perform model training and distribute the trained model, according to at least some embodiments. A third microservice may receive the trained model from the second microservice and perform artificial fluorescence image generation.

[0186] Some of the above embodiments may be implemented in one or more microservices in conjunction with or as part of a digital and laboratory healthcare platform, and one or more of such microservices may be part of an order management system that orchestrates the sequence of events as needed at the appropriate times and in the appropriate order required to instantiate the above embodiments. A microservices-based order management system is disclosed, for example, in U.S. Provisional Patent Application No. 62 / 873,693, entitled "Adaptive Order Fulfillment and Tracking Methods and Systems," filed July 12, 2019, which is incorporated herein by reference in its entirety for all purposes.

[0187] For example, continuing with the first and second microservices above, the order management system may notify the first microservice that an order for training a model has been received and is ready for processing. The first microservice may execute and notify the order management system after the training data is ready for delivery to the second microservice. Furthermore, the order management system may identify that execution parameters (preconditions) for the second microservice have been met, including that the first microservice has completed, and notify the second microservice that it may continue processing the order to generate a trained model according to some embodiments.

[0188] When the digital and laboratory healthcare platform further includes a report generation engine, the above-described methods and systems can be utilized to generate a summary report of the patient's genetic profile and the results of one or more insight engines for presentation to a physician. For example, the report can provide the physician with information about how much of the specimen was used to harvest organoids. For example, the report can provide a genetic profile for each tissue type, tumor, or organ in the specimen. The genetic profile can represent the gene sequences present in the tissue type, tumor, or organ and can include information about variants, expression levels, gene products, or other information that can be derived from genetic analysis of the tissue, tumor, or organ. The report can include treatments and / or clinical trials matched based on some or all of the genetic profile or insight engine findings and summary. For example, therapies can be matched according to the systems and methods disclosed in U.S. Provisional Patent Application No. 62 / 804,724, filed February 12, 2019, entitled "Therapeutic Suggestion Improvements Gained Through Genomic Biomarker Matching Plus Clinical History," which is incorporated herein by reference in its entirety for all purposes. For example, clinical trials may be matched according to the systems and methods disclosed in U.S. Provisional Patent Application No. 62 / 855,913, filed May 31, 2019, entitled "Systems and Methods of Clinical Trial Evaluation," which is incorporated herein by reference in its entirety for all purposes.

[0189] The report may include a comparison of the results to a database of results from many specimens. One example of a method and system for comparing results to a database of results is disclosed in U.S. Provisional Patent Application No. 62 / 786,739, filed December 31, 2018, entitled "A Method and Process for Predicting and Analyzing Patient Cohort Response, Progression and Survival," which is incorporated herein by reference in its entirety for all purposes. This information, sometimes in conjunction with similar information from additional specimens and / or clinical response information, can be used to discover biomarkers or design clinical trials.

[0190] When the digital and laboratory healthcare platform further includes the application of one or more of the embodiments herein to organoids developed in connection with the platform, the method and system can further evaluate the genetic sequencing data derived from the organoids to provide information regarding the extent to which the sequenced organoids contain a first cell type, a second cell type, a third cell type, etc. For example, the report can provide a genetic profile for each cell type in the specimen. The genetic profile can represent the gene sequences present in a given cell type and can include information about variants, expression levels, gene products, or other information that can be derived from genetic analysis of the cells. The report can include therapies matched based on some or all of the deconvolution information. These therapies can be tested on the organoids, their derivatives, and / or similar organoids to determine the sensitivity of the organoids to those therapies. For example, organoids can be cultured and tested according to the systems and methods disclosed in U.S. Patent Application No. 16 / 693,117, entitled "Tumor Organoid Culture Compositions, Systems, and Methods," filed November 22, 2019, and U.S. Provisional Patent Application No. 62 / 924,621, which are incorporated by reference in their entirety for all purposes.

[0191] When the digital and laboratory healthcare platform further includes one or more of the above applications in combination with or as part of a medical device or laboratory-developed test directed generally to healthcare and medical research, the results of such laboratory-developed test or medical device may be enhanced and personalized through the use of artificial intelligence. One example of a laboratory-developed test, particularly one that may be enhanced by artificial intelligence, is disclosed, for example, in U.S. Provisional Patent Application No. 62 / 924,515, entitled "Artificial Intelligence Assisted Precision Medicine Enhancements to Standardized Laboratory Diagnostic Testing," filed October 22, 2019, which is incorporated herein by reference in its entirety for all purposes.

[0192] It should be understood that the examples given above are illustrative and not limiting of the use of the systems and methods described herein in combination with digital and laboratory healthcare platforms.

[0193] The systems and methods disclosed herein can reduce (1) the time required to image plates, (2) the need for toxic dyes that can cause cell death and skew results, and (3) the amount of manual labor required to add those dyes, allowing for the testing of more drugs or drug concentrations (e.g., 10-fold).The systems and methods reduce the number of images generated to analyze each plate (e.g., by 4- or 5-fold, or from 10,000 images to approximately 2,000-2,500 images) by receiving brightfield images, predicting corresponding fluorescent readouts, and enabling label-free (dye-free) estimation of cell viability (e.g., the percentage of cells in a well or image that are alive at a given time point) at multiple time points.

[0194] The system and method described herein can be used to perform hundreds of measurements in each cell culture well to assess the heterogeneity of organoid drug response (e.g., whether they survive or die after treatment), and this can be performed for each organoid (e.g., analyzing cell death or the percentage of viable cells in each organoid). Multiple measurements include fluorescence intensity, cell proliferation, cell death, cells per organoid, cells per well, dose-response (e.g., graphing cell viability % vs. drug dose, calculating best-fit curves or drug dose-response curves, measuring the area above the curve and below the intercept at 100% viability), etc. These measurements facilitate the determination of the cellular mechanism of drug response and / or the detection of drug-resistant subpopulations of organoids within an organoid line or cell culture well. Drug efficacy (e.g., dose-response) and specificity can be measured. The drug efficacy of all drugs on one or more organoid lines can be plotted. In one example, the x-axis shows drug efficacy against one organoid line, and the y-axis shows drug efficacy against a second organoid line. In this plot, drugs near the top right were effective against both organoid lines. Drugs in the top left or bottom right corners were effective against one of the organoid lines.

[0195] Drugs that kill organoid lines can also be categorized and quantified by their drug targets. For example, the 30 most effective drugs that kill organoid lines can be categorized by target in a bar graph showing the number of effective drugs per target.

[0196] 16, a flow is shown for generating an artificial fluorescence image 1616 using a first trained model 1604 and a second trained model 1612. The first trained model 1604 can generate one or more individual organoids 1608 (e.g., a segmentation of the organoids) based on a brightfield image 1600. The brightfield image 1600 can include one or more organoids, and the trained first model 1604 can identify each organoid by segmenting the organoids 1608 from the brightfield image 1600. In some embodiments, the first trained model can include an artificial neural network. In some embodiments, the artificial neural network can include a Mask-RCNN network.

[0197] In some embodiments, the first trained model 1604 and the second trained model can be used to predict drug response and other characteristics of an organoid line based on the viable cells and / or morphology associated with each individual tumor organoid (TO) within the brightfield image 1600.

[0198] Evaluating each organoid individually may provide better information about treatment than if the organoids were evaluated in place within the brightfield image 1600. Each TO may represent a different tumor clone present in the specimen. Each TO may exhibit a different therapeutic response to different drugs at different dosage levels. Instead of assessing TO viability by aggregating viability across the entire field of view within the image, understanding the distribution of viability at a per-organoid level (e.g., how many cells are surviving in each organoid), and potentially aggregating the viability of TOs belonging to the same tumor clone, may enhance understanding of the organoid's response to the drug, and thus the patient's response to the drug.

[0199] In some embodiments, the first trained model 1604 can be trained to segment organoids from the brightfield image 1600 using a training set of brightfield images that have been annotated with bounding boxes around individual organoids. In some embodiments, the first trained model 1604 can generate masks and bounding boxes around all organoids in the brightfield images. In some embodiments, the first trained model 1604 can generate a model embedding that can be used to generate organoid-based features to assess viability and morphology.

[0200] In some embodiments, the second trained model 1612 can include a generator 408 trained to generate artificial fluorescent images based on input brightfield organoids. The second trained model 1612 can be trained on a training set of individual brightfield organoids and individual fluorescent organoids. Each individual fluorescent organoid can be used to generate a viability. The viability for all organoids can be aggregated. A distribution of viability for each organoid can be generated and / or visualized. In some embodiments, the distribution of live / dead cells per organoid is calculated, and dose-response predictions or extrapolations can be obtained from the distribution of organoid viability at a single drug concentration. In some embodiments, the process can aggregate the viability of different tumor clones among organoids when side information is available to determine which excised TOs belong to which tumor clone.

[0201] In some embodiments, the morphology of all organoids can be visualized.In some embodiments, the morphology of tumor organoids can be visualized by using handcrafted features or model embedding, and supervised or unsupervised clustering.In some embodiments, morphological clusters can be associated with cluster survival rate and therefore with drug response.In some embodiments, TO morphology can be used to predict drug response.

[0202] Next, referring to Figure 16 and further to Figure 17, process 1700 is illustrated for generating the fluorescent image of tumor organoid.Process 1700 can be implemented as computer-readable instructions in one or more memory or other non-transitory computer-readable medium, and can be executed by one or more processors that communicate with one or more memory or other medium.In some embodiments, process 1700 can be implemented as computer-readable instructions in memory 220 and / or memory 240, and can be executed by processor 204 and / or processor 224.

[0203] At 1704, process 1700 can receive a bright-field image of one or more organoids (e.g., bright-field image 1600 of FIG. 16). In some embodiments, the bright-field image can be pre-processed to enhance contrast as described above. In some embodiments, the bright-field image can be a raw image that has not undergone any pre-processing, such as contrast enhancement.

[0204] At 1708, process 1700 may determine whether the brightfield image is unprocessed (i.e., raw). If the brightfield image is unprocessed (i.e., "YES" at 1708), process 1700 may proceed to 1712. If the brightfield image is not unprocessed (i.e., "NO" at 1708), process 1700 may proceed to 1716.

[0205] At 1712, the process 1700 can preprocess the brightfield image. In some embodiments, the brightfield image is a pixel in the range [0, 2 16 [0, 255]. In some embodiments, process 1700 can convert the bright-field image to an unsigned byte format, with values ​​in the range [0, 255]. In some embodiments, process 1700 can convert the bright-field image to another format with fewer bits than the original pixel intensities. Process 1700 can then stretch and clip each pixel intensity to a desired output range. In some embodiments, process 1700 can determine the input range for the bright-field image by uniformly stretching the 2nd and 98th percentiles of pixel intensities in the bright-field image to the output range [0, 255].

[0206] At 1716, process 1700 may provide the brightfield image to a first trained model. In some embodiments, the first trained model may be first trained model 1604 in FIG. 16. In some embodiments, the trained model may be a neural network. In some embodiments, the neural network may include a Mask-RCNN model.

[0207] In 1720, process 1700 can receive at least one individual tumor organoid from the first trained model (for example, in 64x64x1 or 32x32x1 image).Each individual tumor organoid can be part of bright-field image.

[0208] At 1724, process 1700 can provide at least one individual tumor organoid to a second trained model.In some embodiments, the second trained model can comprise the second trained model 1612 in Figure 16.In some embodiments, process 1700 can provide each individual tumor organoid to the second trained model in sequence.

[0209] At 1728, process 1700 can receive at least one artificial fluorescent image from the second trained model.Each artificial fluorescent image can be generated based on individual tumor organoid.The artificial fluorescent image can indicate whether the cells contained in tumor organoid are alive or dead.

[0210] At 1732, process 1700 can output at least one artificial fluorescent image. In some embodiments, process 1700 can output at least one artificial fluorescent image to at least one of a memory (e.g., memory 220 and / or memory 240) and / or a display (e.g., display 116, display 208, and / or display 228). The at least one artificial fluorescent image can be used to provide a live / dead count of cells within each individual organoid. In some embodiments, process 1700 can output the at least one artificial fluorescent image to an automated cell counting process to receive an accurate live / dead count of cells, the percentage of viable cells, and / or a cell count report for each organoid. For example, process 1700 can output the at least one artificial fluorescent image to CellProfiler, available at https: / / cellprofiler.org. In some embodiments, process 1700 can output one or more channels of at least one artificial fluorescent image to an automated cell counting process to receive a cell count report, a percentage of viable cells, and / or an accurate live / dead count of cells in each organoid. In some embodiments, process 1700 can output the artificial fluorescent image to a trained model to receive a cell count report, a percentage of viable cells, and / or an accurate live / dead count of cells in the artificial fluorescent image. In some embodiments, process 1700 can output a combination of one, two, or three channels of the brightfield image and the artificial fluorescent image (e.g., image embeddings combined by concatenation) to an automated cell counting process to receive a cell count report, a percentage of viable cells, and / or an accurate live / dead count of cells in the artificial fluorescent image.

[0211] In some embodiments, at 1732, process 1700 can identify cells in the artificial fluorescence image by converting each channel to grayscale, enhancing or suppressing certain features such as specks, rings, neurites, and dark holes, identifying primary objects in the whole-cell channel whose typical diameter (in pixels) is set to a value anywhere between 2 and 20 using minimum cross-entropy thresholding at a smoothing scale of 1.3488, and identifying primary objects in the dead-cell channel whose typical diameter (in pixels) is set to a value anywhere between 5 and 20. In this way, process 1700 can generate a cell count report. In some embodiments, process 1732 can determine whether a drug and / or dosage is effective in killing tumor organoid cells based on the live / dead cell count or the percentage of viable cells for each organoid. In some embodiments, at 1732, process 1700 can extrapolate a dose-response from the distribution of organoid viability at a single concentration.

[0212] In some embodiments, process 1700 can generate a cell count, a cell count report, and / or a report based on the artificial fluorescent image. In some embodiments, process 1700 can cause the report to be output to at least one of a memory (e.g., memory 220 and / or memory 240) and / or a display (e.g., display 116, display 208, and / or display 228). Process 1700 can then terminate. [Example]

[0213] Neural network-based model for predicting TO drug response, and response prediction from brightfield images in the absence of fluorescent labeling. In some embodiments, the process may output one, two, or three channels of the brightfield and artificial fluorescent images to an automated cell counting process (e.g., a viability estimation process) to receive the percentage of cells in the image that are viable (alive).

[0214] 18 shows a flow 1800 for predicting a survival rate 1820 based on a bright-field image 1804. The bright-field image 1804 may be a three-channel bright-field image of tumor organoids and / or cells. The flow 1800 may include providing the bright-field image 1804 to a generator 1808. In some embodiments, the generator 1808 may generate an artificial fluorescence image 1812 based on the bright-field image 1804. The flow 1800 may include providing the bright-field image 1804 and the artificial fluorescence image 1812 to a classifier 1816. The classifier may generate a survival rate 1820 based on the bright-field image 1804 and the artificial fluorescence image 1812.

[0215] 18 and 19, illustrated is an example generator 1900 and an example discriminator 1902. In some embodiments, the discriminator 1902 may be used to train the generator 1900. In some embodiments, the generator 1900 and the discriminator 1902 may be included in a regularized conditional adversarial (RCA) network.

[0216] In some embodiments, the generator 1900 can include an encoder-decoder U-Net network. In some embodiments, the U-Net can include skip connections. In some embodiments, the generator 1900 can receive a two-dimensional brightfield image (e.g., a 1024x1024 brightfield image). In some embodiments, the generator 1900 can generate a normalized three-channel high-resolution 1024x1024x3 output fluorescence image based on the brightfield image, where the three channels correspond to a Hoechst33342 total nucleus staining readout, a Caspase-3 / 7 apoptosis staining readout, and a TOPRO-3 dead cell staining readout, respectively.

[0217] 18 and 19 , as well as 20 , the classifier 1904 can generate a viability prediction 1924 based on the bright-field image and the artificial fluorescence image. The classifier 1904 can include an encoder branch 1908 and a fully-connected branch 1912. In some embodiments, the encoder branch 1908 can include a 70×70 patch GAN. In some embodiments, the encoder branch 1908 can receive a concatenated bright-field image and a fluorescence image 1916 of size 1024×1024×6. In some embodiments, the encoder branch 1908 can generate an output prediction map 1920 (e.g., an output prediction map of size 126×126×1). The fully-connected branch 1912 can then generate a viability prediction 1924 based on the output prediction map 1920. In some embodiments, the fully-connected branch 1912 can include multiple fully-connected layers (e.g., two fully-connected layers) and a sigmoid activation layer that outputs the viability prediction 1924. The survival rate prediction 1924 may indicate a survival rate. In some embodiments, the survival rate prediction 1924 may be and / or range from 0 (indicating no chance of survival) to 1 (indicating high chance of survival).

[0218] training In testing, generator 1900 and classifier 1902 were trained on 8415 pairs of brightfield and three-channel fluorescent images from a colon adenocarcinoma TO screening experiment, each with an associated calculated drug response based on TO-PRO-3 survival. In some embodiments, the objective function (e.g., the loss function used for training) can include an additional mean squared error loss in the classifier to regress against the branch of the classifier that calculates overall survival for each brightfield image. Exemplary loss functions for classifier 1902 and generator 1900 are provided below: D Loss =MSE Loss {Real Prediction,1}+MSE Loss {Fake Prediction,0}+ MSE Loss {Predicted Viability,Viability} G Loss =MSE Loss {Fake Prediction,1}+MAE Loss {Fake Fluorescent,Real Fluorescent}+ SSIM{Fake Fluorescent,Real Fluorescent} The weight for the classifier 1902 is D Loss and the weights for the generator 1900 are updated by minimizing G Loss can be updated by maximizing

[0219] verification During validation, comparison of representative images of real and generated fluorescence demonstrated nearly indistinguishable visual matching. These results were confirmed using two quantitative metrics: structural similarity index (SSIM) and root mean square error (RMSE). The reported average SSIM and RMSE values ​​across 1,526 samples of colon adenocarcinoma TO used in the screening experiment were 0.90 and 0.13924, respectively. For gastric TO strains, the reported average SSIM and RMSE values ​​across 9,200 samples were 0.898 and 0.136, respectively.

[0220] TO description, image analysis, and image generation for training data TOs were dissociated into single cells and resuspended in a 30:70% mixture of GFR Matrigel and growth medium at a concentration of 100 cells / μl. This solution was added to a 384-well assay plate (Corning) at 20 μl per well for a final concentration of 2,000 cells per well. The assay plate was covered with a Breathe-Easy sealing membrane (Sigma-Aldrich) to prevent evaporation. TOs were grown for 72 h before drug addition. Drugs were prepared in growth medium using 2.5 μM Caspase-3 / 7 Green Apoptosis Assay Reagent (Essen Bioscience). Serial dilutions of each molecule were prepared in 384-well polystyrene plates (Nunc). Seven 10-fold dilutions were made for each compound at a high dose of 10 μM. Selected compounds were limited to a high dose of 1 μM due to maximum solubility. The diluted drug was added to the assay plate using an Integra Viaflo pipette (Integra) attached to an Integra Assist Plus Pipetting Robot (Integra). The assay plate was again covered with a Breathe-Easy sealing membrane, and the TO was exposed to the drug for an additional 72 hours before imaging.

[0221] Prior to imaging, TOs were incubated with 4 μM Hoechst 33342 (Fisher Scientific) and 300 nM TO-PRO-3 Iodide (642 / 661) (Invitrogen) for 1.5–2 h. Assay plates were imaged at 10X magnification using an ImageXpress Micro Confocal (Molecular Devices), resulting in approximately 100–200 TOs per well. Multiplexed fluorescence images were 1024 × 1024 × 3 RGB images, with red corresponding to dead cells (TO-PRO-3), green corresponding to apoptotic cells (caspase-3 / 7), and blue corresponding to nuclei (Hoechst 33342). All wavelength channels were subjected to a simple intensity rescaling contrast enhancement technique to brighten and sharpen TOs / cells and remove background noise.

[0222] Images were acquired as 4 x 15 µm Z-stacks, and 2D projections were analyzed to assess cell viability. Confocal images were analyzed using the custom module editor feature of MetaXpress software (Molecular Devices). An analysis module was designed to identify TOs by clusters of Hoechst33342 staining, individual cells by Hoechst33342 staining, and dead / dying cells by either TO-PRO-3 or Caspase-3 / 7 staining. The result of this analysis module was a spreadsheet detailing the number of live and dead cells for all individual organoids. Viability values ​​were ≥ 0 (0% viable cells) and ≤ 1 (100% viable cells).

[0223] Viability calculation = sum of all live cells in a site / sum of all cells in a site (giving the ratio of live cells per site). More effective drugs have lower viability (more cells die) at higher doses compared to less effective drugs, which have higher viability.

[0224] The average viability of all organoids per site (e.g., per image) was obtained from the MetaXpress software readout. For each image added to the training dataset used to train the viability classifier, the image was saved with the average viability associated with that image as a label or metadata. Images had a resolution of 1024x1024 and were randomly flipped as a data augmentation step before being used as training data.

[0225] The training dataset for this example contained 7000 images representing 15 culture plates.

[0226] The percentage of viable cells per TO was calculated based on the image analysis results described above. TOs with fewer than three cells, TOs larger than the top 1 percent by size, and wells in which fewer than 20 TOs were detected were excluded from the analysis.

[0227] In another example, AUC can be used as metadata or a label to generate training data. The mean survival rate for all TOs at a given drug concentration was used in the dose-response curve to calculate AUC. AUC was calculated using the computeAUC function in the R package PharmacoGx (v1.17.1) using settings for "actual" AUC. Heatmaps of AUC values ​​were generated using the Pheatmap package (v1.0.12) in R. Scatterplots of AUC values ​​were generated using the ggplot2 package (v3.3.0) in R.

[0228] 18-20 and further to FIG. 21, a process 2100 for generating a viability value is illustrated. Process 2100 may be implemented as computer-readable instructions in one or more memories or other non-transitory computer-readable media and executed by one or more processors in communication with one or more memories or other media. In some embodiments, process 2100 may be implemented as computer-readable instructions in memory 220 and / or memory 240 and executed by processor 204 and / or processor 224.

[0229] At 2104, process 2100 can receive a brightfield image of one or more organoids (e.g., brightfield image 1804 of FIG. 18). In some embodiments, the brightfield image can be preprocessed to enhance contrast as described above. In some embodiments, the brightfield image can be a raw image that has not undergone any preprocessing, such as contrast enhancement.

[0230] At 2108, process 2100 may determine whether the brightfield image is unprocessed (i.e., raw). If the brightfield image is unprocessed (i.e., "YES" at 2108), process 2100 may proceed to 2112. If the brightfield image is not unprocessed (i.e., "NO" at 2108), process 2100 may proceed to 2116.

[0231] At 2112, the process 2100 can preprocess the brightfield image. In some embodiments, the brightfield image is a pixel in the range [0, 2 16[0, 255]. In some embodiments, process 2100 can convert the bright-field image to an unsigned byte format, with values ​​in the range [0, 255]. In some embodiments, process 2100 can convert the bright-field image to another format with fewer bits than the original pixel intensities. Process 2100 can then stretch and clip each pixel intensity to a desired output range. In some embodiments, process 2100 can determine the input range for the bright-field image by uniformly stretching the 2nd and 98th percentiles of pixel intensities in the bright-field image to the output range [0, 255].

[0232] At 2116, process 2100 can provide the brightfield image to a trained model. In some embodiments, the trained model can include a generator (e.g., generator 1808 and / or generator 1900) and a classifier (e.g., classifier 1816 and / or classifier 1904). In some embodiments, process 2100 can include providing the brightfield image to the generator, receiving an artificial fluorescence image from the process, concatenating the brightfield image with the artificial fluorescence image to generate a concatenated image, and providing the concatenated image to the classifier.

[0233] At 2120, the process 2100 may receive a survival rate (e.g., a survival rate value) from the trained model. In some embodiments, the process 2100 may receive a survival rate from the classifier 1904. In some embodiments, the survival rate may be a survival rate 1820 and / or a survival rate prediction 1924.

[0234] At 2124, process 2100 may cause the viability rate to be output. In some embodiments, process 2100 may cause the viability rate to be output to at least one of a memory (e.g., memory 220 and / or memory 240) and / or a display (e.g., display 116, display 208, and / or display 228). In some embodiments, process 2100 may generate a report based on the viability rate. In some embodiments, process 2100 may cause the report to be output to at least one of a memory (e.g., memory 220 and / or memory 240) and / or a display (e.g., display 116, display 208, and / or display 228). Process 2100 may then terminate.

[0235] While this disclosure describes one or more preferred embodiments, it should be understood that many equivalents, alternatives, variations, and modifications, except as expressly stated, are possible and are within the scope of the invention. [Explanation of symbols]

[0236] 100 systems 104 Computing Devices 108 Secondary Computing Devices 112 Communication Network 116 Display 120 Organoid Image Database 124 Training Data Database 128 pre-trained model database 132 Organoid Image Analysis Applications 136 pre-trained models 200 examples 204 processors 208 Display 212 input 216 Communication Systems 220 memory 224 processors 228 Display 232 input 236 Communication Systems 240 memory 300 Flow 304 human patients 308 tumor specimens 312 Organoids 316 Drug Screening 320 Organoids 324 Brightfield and Fluorescent Images 400 Flow 404 Input bright-field image 408 Generator 412 Artificial Fluorescence Images 416 Classifier 420 Objective function value calculation 424 ground truth fluorescence images 500 Flow 504 brightfield images 508 pre-trained models 512 Artificial Fluorescence Images 600 Neural Networks 604 input images 608 Artificial Fluorescence Images 700 Classifier 704 brightfield images 708 Fluorescence Images 712 predicted labels 800 processes 900 processes 924 Process 1004 brightfield images 1008 Blue / All-Nucleus Channel Fluorescence Images 1012 Green / Apoptosis Channel Fluorescence Images 1016 Red / Pink / Death Channel Fluorescence Images 1020 combined 3-channel fluorescence images 1024 brightfield images 1028 blue / whole nucleus channel fluorescence images 1032 green / apoptosis channel fluorescence images 1036 Red / Pink / Death Channel Fluorescence Images 1040 combined 3-channel fluorescence images 1100 Flow 1200 flow 1300 processes 1504 First Image 1508 Second Image 1512 Third Image 1516 Fourth Image 1520 5th image 1524 6th image 1528 7th image 1532 8th image 1536 9th image 1540 10th image 1600 brightfield images 1604 First Trained Model 1608 Organoids 1612 Second trained model 1616 Artificial Fluorescence Images 1700 processes 1800 flow 1804 brightfield images 1808 Generator 1812 Artificial Fluorescence Images 1816 Classifier 1820 survival rate 1900 generator 1902 Classifier 1904 Classifier 1908 Encoder Branch 1912 Fully Connected Branch 1920 Output Prediction Map 1924 Survival Prediction 2100 Process

Claims

1. 1. A method for generating an artificial fluorescent image of a cell without fluorescent staining, comprising: receiving a bright field image of at least a portion of cells contained in the specimen; pre-processing the bright field image; providing the brightfield image to a trained model, the trained model being trained to generate the artificial fluorescent image of the cell based on the brightfield image; receiving the artificial fluorescence image from the trained model; outputting the artificial fluorescence image to at least one of a memory or a display; Including, The method, wherein the preprocessing step comprises randomly flipping or rescaling pixel intensities in a desired intensity range of the bright field image.

2. The artificial fluorescent image indicates whether the cells are alive or dead; The method comprises: concatenating the bright field image with the artificial fluorescence image to generate a concatenated image; generating a viability value by assessing live / dead cells based on the linked image; The method of claim 1 further comprising:

3. The method described in claim 1, wherein the cells are included in a group of tumor organoids, and the group of tumor organoids is associated with colorectal cancer, gastric cancer, breast cancer, lung cancer, endometrial cancer, colon cancer, head and neck cancer, ovarian cancer, pancreatic cancer, gastric cancer, hepatobiliary cancer, or genitourinary cancer.

4. The method of claim 1 , wherein the trained model comprises an artificial neural network.

5. The method of claim 4 , wherein the artificial neural network comprises a generative adversarial network (GAN).

6. The method of claim 1 , wherein the trained model is trained based on a loss function comprising a structural similarity index (SSIM).

7. The method of claim 1 , wherein the trained model is trained based on a loss function that includes a classifier loss.

8. The method of claim 1 , wherein the trained model is trained based on a loss function including a generator loss and a discriminator loss.

9. The pre-processing step comprises: converting the bright field image into an unsigned byte format; stretching and clipping each pixel intensity in the bright-field image to a desired output range of [0 to 255]; and stretching comprises uniformly stretching a 2nd and 98th percentile of pixel intensities in the bright field image to the desired output range.

10. 10. The method of claim 1, further comprising preprocessing brightfield and fluorescent images included in training data used to train the trained model.

11. The method of claim 1 , wherein the trained model includes a generator, the generator being trained in part by a discriminator.

12. The method described in claim 1, wherein the cells are included in a group of tumor organoids, and the group of tumor organoids is plated on a well plate containing at least 300 wells.

13. The method described in claim 1, wherein the cells are included in a group of tumor organoids, and the step of outputting the artificial fluorescent image includes a step of outputting the artificial fluorescent image to a drug screening process for determining the effectiveness of drugs used to treat the group of tumor organoids.

14. providing a second brightfield image to the trained model; receiving a second artificial fluorescence image from the trained model; The method of claim 1 further comprising:

15. 15. The method of claim 14, wherein the second bright field image comprises a second group of tumor organoids, and the second artificial fluorescent image indicates whether the cells comprised in the second group of tumor organoids are alive or dead.

16. 15. The method of claim 14, wherein the second brightfield image includes a view of a group of tumor organoids, the second brightfield image is generated a predetermined time period after the brightfield image is generated, and the second artificial fluorescent image indicates whether cells contained in the group of tumor organoids are alive or dead.

17. 17. The method of claim 16, wherein the predetermined period of time is at least 12 hours.

18. 17. The method of claim 16, wherein the predetermined period of time is at least 24 hours.

19. 17. The method of claim 16, wherein the predetermined period of time is at least 72 hours.

20. 17. The method of claim 16, wherein the predetermined period of time is at least one week.

21. A tumor organoid analysis system comprising at least one processor and at least one memory, Receive bright-field images of a group of tumor organoids without fluorescent staining, preprocessing the bright field image; providing the brightfield image to a trained model; generating artificial fluorescent images of cells in the group of tumor organoids based on the bright field images using the trained model; receiving the artificial fluorescent image from the trained model, the image indicating whether cells in the group of tumor organoids are alive or dead; outputting the artificial fluorescence image to at least one of a memory or a display; It is configured as follows: The system, wherein pre-processing includes randomly flipping or rescaling pixel intensities in a desired intensity range of the bright-field image.

Images