Image-based microorganism sphere drug assay

Abstract

The present disclosure relates to methods, systems, and devices for image-based microorganism drug assays. In one aspect, the method includes, in response to acquiring image data of a well plate containing a plurality of microorganisms and applying a machine learning model configured to identify at least some instances of the plurality of microorganisms in the image data, acquiring (i) a representation of each instance of the microorganisms and (ii) attributes of each instance of the microorganisms, and normalizing well-to-well variability within the well plate based on the representation and the attributes.

Description

[Technical Field]

[0001] The present disclosure relates to systems and methods for performing image-based MicroOrganoSphere (MOS) drug assays. Specifically, described herein are systems and methods for identifying instances of MOS using machine learning models and determining drug responses for the instances of MOS. [Background technology]

[0002] Immortalized cell lines and genetically engineered mice have been the foundation of functional assays for the past several decades. Recent advances in the development of in vitro 3D tissue models have provided important scientific benefits in disease modeling, regenerative medicine development, drug discovery and toxicity testing, and functional precision medicine. Among these, organoids derived from induced pluripotent stem cells (iPS cells) and adult stem cells have rapidly emerged as in vitro 3D tissue models for the study of human disease and host-pathogen interactions because organoids can recapitulate the structure and phenotype of host tissues in many important respects, thereby providing new insights and enabling the evaluation of potential therapeutics.

[0003] Microorganospheres (MOS) are droplets of a substrate (e.g., Matrigel or any suitable hydrogel) that are typically spherical (or substantially spherical). MOS can contain one or more three-dimensional cellular aggregates formed from populations or clusters (possibly heterogeneous cell clusters) of dissociated primary cells distributed within the MOS. These cell populations are sometimes referred to as organoids, or in the context of cancer research, "tumor-like masses." The microorganospheres can be derived from patients and are referred to as patient-derived microorganospheres (PMOS). PMOS diameters range from approximately 50 μm to approximately 500 μm. In the emerging field of functional precision drug therapy, PMOS models have been shown to correlate with clinical outcomes.

[0004] Compared to patient-derived xenograft models and large-scale cultured organoids, PMOS offer high establishment rates and throughput, and require a shorter turnaround time. PMOS can indicate responses to treatment, such as drug response, chemotherapy, and radiation therapy, and can serve as avatars for determining treatment strategies. However, challenges in using PMOS in a clinical setting include tissue heterogeneity, excessive well-to-well variability within a plate, and difficulty seeding consistent cell numbers and ensuring uniform growth rates in each PMOS. These challenges complicate interpretation of drug responses. Summary of the Invention

[0005] Described herein is a technology for performing image-based MicroOrganoSphere (MOS) drug assays that overcomes tissue heterogeneity and well-to-well variability. In particular, a technology that can identify MOS by plate imaging and determine different attributes of MOS, such as total surface area and fluorescent activity, such as live and dead cell stain signals, would be useful for improving the reliability of drug responses in personalized medicine.

[0006] In one aspect, a computer-implemented method includes acquiring image data of a well plate including a plurality of microorganisms. The method includes, in response to applying a machine learning model configured to identify at least some instances of the plurality of microorganisms in the image data, acquiring (i) an index for each instance of the microorganisms and (ii) attributes for each instance of the microorganisms. The method includes normalizing well-to-well variability within the well plate based on the index and the attributes.

[0007] In one aspect, a computer-implemented method includes acquiring, for each microorganism of a plurality of microorganisms, a first input including brightfield image data and fluorescent image data, and acquiring a second input indicating a reference-matching label representing each microorganism instance. The method includes training a machine learning model configured to identify (i) each microorganism instance and (ii) attributes of each microorganism instance using the first and second sets of inputs across the plurality of microorganisms.

[0008] Embodiments of the above-described methods may include one or any combination of two or more of the following features.

[0009] The method includes performing a drug assay on a plurality of microorganisms, the drug assay measuring cell viability upon drug treatment in a predetermined well within a well plate, and determining normalized cell viability within the well plate. The cell viability measured by the drug assay can be adjusted using an attribute of MOS, such as total surface area. The microorganisms may be derived from a patient-derived tissue sample, such as a biopsy specimen from a metastatic tumor or a clinical tumor sample containing both cancer cells and stromal cells. The image data includes brightfield and fluorescent images of the well plate. The brightfield and fluorescent images of the well plate are collected using an imaging device, such as an imaging cytometer. The display includes a visual representation of each instance of the microorganisms within the image data. Attributes of each microorganism instance include the total surface area, live cell stain signal, and dead cell stain signal for each instance. In some implementations, acquiring image data of the well plate includes acquiring image data of one or more wells within the well plate at a single focal plane. In some implementations, acquiring image data of the well plate includes using a two-dimensional projection of a three-dimensional confocal microscope Z-stack.

[0010] The method may include displaying a visual representation of each instance of the microorganism on a user interface. Obtaining an index for each instance of the microorganism in the image data may include generating a corresponding mask represented using a Fourier series representation, the Fourier series representation being generated based on coefficients output by the machine learning model. The method may also include determining a fluorescent activity of one or more microorganism instances based on the index output by the machine learning model. The fluorescent activity includes a live cell stain signal and a dead cell stain signal for the one or more microorganism instances. Determining the fluorescent activity of one or more instances of the microorganism includes performing a logical operation between the image data and the instructions.

[0011] The method may include iteratively adjusting an index output from the machine learning model to capture a dead cell dye signal, and outputting the fluorescent activity of one or more instances of the microorganism based on the adjusted index.

[0012] Normalizing for well-to-well variability within a well plate includes obtaining the total surface area of ​​each instance of a microorganism within the well plate, where the total surface area correlates with the adenosine triphosphate (ATP) level of each instance of a microorganism; obtaining cell viability in response to drug treatment for each well of the well plate; and adjusting each cell viability across multiple wells within the well plate based on the total surface area.

[0013] The method can include treating one or more wells in a well plate with a stain (e.g., Invitrogen™ CellTracker™ Deep Red dye) that nonspecifically binds to the substrate of the organic tissue and microorganisms. For example, this treatment can be performed before obtaining initial input including brightfield and fluorescent image data. The method can include treating selected wells in the well plate with a live cell dye and a dead cell dye; performing a drug assay on multiple microorganisms in the selected well; and determining integrated cell viability in the selected well. The drug assay can include a CellTiter-Glo (CTG) luminescent cell viability assay. The live cell indicator can include Calcein-AM (CAM) or MitoTracker Viewer dye. The dead cell indicator can include a fluorescent conjugate of ethidium homodimer (EtH), e.g., EtH-2, or Annexin V. The method can include determining a cytotoxic or cytostatic drug response based on the integrated cell viability and fluorescent image signals. The method can include filtering out stromal cells and non-tumor-like masses in the indicator in response to applying a size filter to the indicator, the size filter removing cells below a predefined size from the indicator.

[0014] The method can include applying binarization to the fluorescence image data in response to determining that a saturation level of the fluorescence image data does not meet a predefined threshold. Training the machine learning model can include providing a first input and a second set of inputs to a Mask-RCNN architecture, where the Mask-RCNN architecture includes a plurality of convolutional neural networks and a feature pyramid network. Alternatively, training the machine learning model can include training a neural network to output coefficients of a Fourier series representation of one or more image segmentation masks (e.g., image instance segmentation masks) corresponding to the microorganisms. In some implementations, the machine learning model can be trained using images including labeled microorganism instances, which are labeled using a pre-trained neural network configured to generate image segmentation masks (e.g., image instance segmentation masks). An image containing labeled microorganism instances can be generated from an unlabeled image, where the unlabeled image has been preprocessed using at least one of the following: (i) a mathematical transformation that emphasizes the fluorescence corresponding to stained microorganisms present in the unlabeled image; (ii) a mathematical morphological process that emphasizes circular objects within a defined size range (e.g., a size range corresponding to the size of potential MOSs of about 50 μm to about 500 μm); and (iii) an image resizing operation (e.g., to adjust the size of MOSs to match the size of objects detectable by a pretrained neural network, i.e., the objects used to train the pretrained neural network). Alternatively, or in addition, the unlabeled image can first be preprocessed using one or more filters (e.g., a median filter) to remove noise signals.

[0015] Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the methods. Embodiments include systems with one or more processors and one or more storage devices that store instructions that, when executed by the processors, cause the processors to perform a process. Embodiments also include non-transitory computer-readable media containing software instructions that, when executed by a computer, cause the computer to perform a process.

[0016] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is an example block diagram of a system for performing an image-based microorganism sphere (MOS) drug assay.

[0018] [Figure 2] 1 is an example of a flow chart of a process for performing an image-based MOS drug assay.

[0019] [Figure 3A] Shown are example images of a whole-well scan (top row) and a zoomed-in view (bottom row) showing MOS instances identified by machine learning methods.

[0020] [Figure 3B] Examples of drug response curves for MOS of CRC436 and CRC406 before and after normalization are shown.

[0021] [Figure 3C] Examples of drug response curves for MOS derived from primary lung, colorectal cancer (CRC), and breast cancer tissues before and after normalization are shown.

[0022] [Figure 3D] Figure 1 shows a comparison of the raw CellTiter-Glo (CTG) luminescent cell viability assay with the adjusted CTG assay in measuring drug response in MOS.

[0023] [Figure 4A] An example of a whole-well scan (left panel) and a zoomed-in view (right panel) showing MOS instances identified by the machine learning method are shown. [Figure 4B] An example of a whole-well scan (left panel) and a zoomed-in view (right panel) showing MOS instances identified by the machine learning method are shown.

[0024] [Figure 4C] Correlation between the total surface area (tSA) of MOS measured by machine learning methods and CTG measurements in relative luminescence units (RLU) is shown. [Figure 4D] Correlation between the total surface area (tSA) of MOS measured by machine learning methods and CTG measurements in relative luminescence units (RLU) is shown.

[0025] [Figure 4E] Example images of MOS from colorectal cancer (CRC), lung, and breast cancer tissues (left panel) are shown, along with a zoomed-in view (right panel) showing MOS instances identified by machine learning methods.

[0026] [Figure 4F] Example images of MOS from primary sarcoma tissue (top row) and zoomed-in views showing MOS instances identified by machine learning methods (bottom row) are shown.

[0027] [Figure 5A] Example images of two CRC models treated with vehicle, erlotinib, or SN38, co-stained with live cell dye (CAM) and dead cell dye (EtH) are shown.

[0028] [Figure 5B]1 shows an example of the difference in drug response of a cytostatic or cytotoxic drug across a range of doses.

[0029] [Figure 5C] 1 shows examples of drug response curves for two CRC MOS models treated with erlotinib or SN38.

[0030] [Figure 5D] 1 shows an example of dose-dependent changes in the integrated intensity ratio of CAM / EtH staining in two CRC models treated with erlotinib or SN38.

[0031] [Figure 5E] Examples of differences in drug response of individual MOS derived from primary lung tumors are shown.

[0032] [Figure 5F] Comparison of drug response curves measured by CTG assay (top panel) and by median live-to-dead cell dye ratios in MOS derived from primary sarcoma tissue (bottom panel).

[0033] [Figure 6] 1 is a flowchart illustrating an example of a process for training a machine learning model used to perform an image-based MOS drug assay. DETAILED DESCRIPTION OF THE INVENTION

[0034] Systems and methods described herein relate to techniques for performing image-based microorganism sphere (MOS) drug assays. For example, the systems and methods described herein aim to identify MOS instances in image data, e.g., brightfield images of a plate, and determine MOS attributes, including total surface area, percentage of live cells, and percentage of dead cells, in each MOS instance. With reference to image data 102 (in FIG. 1 ), this disclosure describes MOS images, but other image data, e.g., structures with similar size, surface-to-volume ratio, and / or properties that allow imaging in a single plane (e.g., wide-field imaging or Z-stack confocal imaging with appropriate two-dimensional projections), can be similarly processed.

[0035] The disclosed systems and methods may have one or more of the following advantages: The machine learning techniques described herein enable the automatic identification of MOS in a computationally efficient manner and the use of such information in drug response analysis. Manually identifying MOS from image data is laborious, time-consuming, and relatively unreliable. This computational efficiency is important in personalized medicine, for example, to provide rapid feedback on which drugs are likely to be effective for a patient. Furthermore, MOS attributes can be used to correct for tissue heterogeneity within a plate and well-to-well variability, enabling interpretable and reliable drug responses. Furthermore, identified resistant clones can be used to develop screens for resistant populations to study mechanisms of drug resistance and recommend treatments when disease recurs. By way of example only, resistant clones can be isolated, and one or more of genetic, transcriptomic, and proteomic analyses can be performed to understand the drug resistance mechanism. Furthermore, MOS responses to one or more tested drugs can be used to determine which drugs or drug combinations should or should not be used to treat a patient based on the test results.

[0036] Identifying MOS from plate images also offers several advantages over directly identifying clusters of cells (e.g., tumor-like masses). First, because MOSs tend to sink to the bottom of the well, images can be acquired using single focal plane imaging or two-dimensional projections, rather than using full three-dimensional Z-stack confocal imaging to acquire images at multiple focal lengths. Second, the presence of multiple MOSs in a single well creates distinct microenvironments that have little effect on each other. Therefore, the presence of multiple MOSs in a single well (e.g., wells treated with a specific set of conditions) allows each MOS to be observed as a separate experiment, effectively increasing the number of experiments researchers can conduct in a given period. Third, because MOSs are made from a hydrogel substrate, different imaging stains (e.g., Invitrogen™ CellTracker™ Deep Red dye) can be used that bind nonspecifically to the MOS substrate rather than specifically to the tumor-like mass (or organoid) tissue. Fourth, because MOSs tend to be more spherical than irregularly shaped tumors (or organoids), machine learning models for identifying MOSs can be adapted to exploit this known knowledge to improve detection capabilities and / or reduce the amount of training data required to achieve similar performance as machine learning models trained to directly identify tumors (or organoids).

[0037] FIG. 1 is an example block diagram of a system 100 for performing an image-based MOS drug assay. The system 100 acquires image data 102 and generates a MOS instance 114 and MOS attributes 116. In some implementations, the system 100 generates a report 120 summarizing drug responses and recommending specific medications for the patient. The system 100 includes an input device 140, a network 128, and one or more computers 130 (e.g., one or more local or cloud-based processors, one or more servers). The computer 130 can include an input processing engine 104, a MOS discriminative machine learning model 108 that can be trained by a training engine 110, and a drug response engine 118. For purposes of this disclosure, an "engine" can include one or more software modules, one or more hardware modules, or a combination of one or more software modules and one or more hardware modules. In some implementations, one or more computers are dedicated to a specific engine. In some implementations, multiple engines may be installed and executed on the same computer or multiple computers.

[0038] The input device 140 may be a device configured to acquire the image data 102, a device configured to provide the image data 102 to another device via the network 128, or any suitable combination thereof. For example, the input device 140 may include a plate scanner 140a configured to scan a plate and output an image of the plate in a format (e.g., tiff format). In some implementations, the image data 102 is a brightfield image of the MOS. In some implementations, the image data 102 is a fluorescent image of the MOS. In some implementations, the image data 102 is a combination of a brightfield image and a fluorescent image of the MOS. In some implementations, the image data 102 includes an image of a structure having a similar size and / or surface-to-volume ratio to the MOS. In some implementations, the structure has special properties that allow it to be imagined in a single plane, such as a structure formed using a droplet-based microfluidic device. Prior to obtaining the image data 102, the MOS may be stained, for example, by treating one or more wells in a well plate containing the MOS with a stain that nonspecifically binds to the substrate (e.g., Matrigel) of the organic tissue and the microorganisms. One example of such a dye is the Invitrogen™ CellTracker™ Deep Red dye, which contributes to the dim emission of MOS in the image data 102 and has been found to bind well to a wide variety of MOS (e.g., exhibiting robustness against substrate variations). While the dim emission of stains such as the Invitrogen™ CellTracker™ Deep Red dye previously made them unsuitable for direct staining of organoids or tumor-like masses, this dim emission has proven highly effective for detecting MOS, especially when combined with image pre-processing steps as described in more detail below.

[0039] The computer 130 can access the image data 102, which may be stored, for example, on a cloud server, via a network 128. The network 128 may include one or more of a wired Ethernet network, a wired optical network, a wireless WiFi network, a LAN, a WAN, a Bluetooth network, a cellular network, the Internet, or other suitable networks, or any combination thereof.

[0040] The computer 130 is configured to obtain the image data 102 from the input device 140. In some implementations, the image data 102 may be data received over the network 128. In some implementations, the computer 130 may store the image data 102 in a database 132 and access the database 132 to obtain the image data 102. The database 132, for example, a local database or a cloud-based database, may store the image data 102 in a standardized format for rapid processing of the images.

[0041] The input processing engine 104 is configured to receive the image data 102 and generate processed image data 106 for input to the MOS identification machine learning model 108. The input processing engine 104 may standardize the format of the image data 102, encrypt the image data 102 to comply with privacy requirements, or perform other data processing. For example, the input processing engine 104 may perform various image preprocessing processes (in any order), including filtering, mathematical morphology, mathematical transformation, and / or image resizing. With regard to filtering, a median filter may be performed to reduce noise signals. With regard to mathematical morphology, the input processing engine 104 may emphasize disk-shaped objects within a predetermined size range in the image data 102 while suppressing non-circular or improperly sized objects (e.g., this may increase the likelihood of specifically identifying approximately spherical MOS droplets in the image data 102). Regarding mathematical transformations, various transformations, including power, square root, cube root, and logarithmic transformations, can be implemented to enhance the fluorescence in the image data 102 corresponding to the dim emission of the stained MOS. In some cases, this fluorescence may be "mid-level fluorescence," which is weaker than the fluorescence corresponding to the dye that specifically binds to the tumor-like mass (or other organoid), but stronger than the image background. In some cases, images in the image data 102 can be scaled up or down to adjust the size of the stained MOS to be on the same order of magnitude as the size of the objects in the images used to train the pre-trained neural network. This image resizing can improve the accuracy of the neural network in generating instance segmentation labels. In some implementations, the input processing engine 104 can perform binarization on the fluorescence images in the training data if they are not fully saturated, or conversely, limit the range of values ​​to a mid-intensity range.For example, the input processing engine 104 can perform binarization on the fluorescence image by calculating a saturation offset and setting each pixel of the fluorescence image so that it is appropriately saturated, e.g., set to maximum if the intensity of a given pixel is greater than a predefined threshold, and set to minimum if the intensity of a given pixel is less than a predefined threshold.

[0042] The processed image data 106 generated by the input processing engine 104 can include data structures representing pixels of a scanned plate image, for example, an image of a whole-well scan as shown in FIG. 3A. The processed image data 106 output by the input processing engine 104 is provided as input to the MOS discrimination machine learning model 108. In some implementations, the MOS discrimination machine learning model 108 acquires the image data 102 without undergoing image processing steps by the input processing engine 104.

[0043] The MOS discrimination machine learning model 108 is configured to take the processed image data 106 and generate MOS instances 114, MOS attributes 116, or both. The MOS instances 114 represent individual instances of MOS within the image data. The MOS attributes 116 can include total surface area and, optionally, live / dead dye signals. The live / dead dye signals are applicable when the image data 106 includes fluorescent images processed with live and dead dyes. The MOS discrimination machine learning model 108 is a model output from the training engine 110.

[0044] The training engine 110 acquires training image data 112 and processes the training image data 112 to train and output the MOS discrimination machine learning model 108. The training image data 112 includes a plurality of paired images, including brightfield image data and fluorescent image data. Each training image data 112 is labeled such that each MOS instance is annotated. The reference-matching labels can be derived by manually annotating discrete active regions from the fluorescent image, or can be derived automatically by a pre-trained neural network, as described in further detail below. In some implementations, the training image data 112 is processed using the input processing engine 104 described with reference to FIG. 1 (or by another engine performing one or more similar operations). Using the training image data 112, the training engine 110 trains a machine learning model configured to identify and segment each instance of MOS and attributes of each instance of MOS by processing the training image data 112 through multiple layers. Additional details regarding training the MOS discrimination machine learning model 108 are described in connection with FIG. 6.

[0045] FIG. 6 is an example flowchart of a process 600 for training a machine learning model (e.g., MOS discrimination machine learning model 108) used to perform an image-based MOS drug assay. The operations of process 600 include receiving (602) an unlabeled image. For example, the unlabeled image can be brightfield and fluorescent image data of stained MOS that have not yet been labeled to indicate the location, size, and shape of the MOS. Process 600 also includes applying one or more filters (e.g., a median filter) to remove noise signals (604), applying a mathematical transformation process to enhance fluorescence corresponding to stained microorganisms present in the unlabeled image (606), applying a mathematical morphological process to enhance disk-shaped objects within a defined size range (608), and applying an image resizing process (610). Processes 604, 606, 608, and 610 are sometimes referred to herein as “preprocessing” and correspond to the image preprocessing described above performed by input processing engine 104. The operations 604, 606, 608, and 610 may be reordered, and in some implementations, only some of the operations 604, 606, 608, and 610 are performed in the process 600.

[0046] After preprocessing the unlabeled images, process 600 includes labeling the images using a pre-trained neural network configured to generate image segmentation masks (612). For example, the neural network can be a pre-trained neural network that outputs coefficients of a Fourier series representation of one or more image segmentation masks corresponding to MOSs (described in further detail below). In this sense, the generated image segmentation masks can be said to be generated "automatically" (e.g., without manual annotation) by the pre-trained neural network, and these masks are then treated as criterion-compliant labels that indicate the location, size, and shape of MOSs in previously unlabeled images. An example of a pre-trained neural network that can be used for process 612 is the “FourierDist” network described in Fully Automatic Cell Segmentation with Fourier Descriptors (December 17, 2021) (www.biorxiv.org / content / 10.1101 / 2021.12.17.472408v1) and Cell segmentation and representation with shape priors, Computational and Structural Biotechnology Journal, Vol 21, 2023, pp 742-740 (doi.org / 10.1016 / j.csbj.2022.12.034), both of which are incorporated by reference in their entirety, with corresponding source code available at www.bitbucket.org / biomag / fourierdist / src / master / .

[0047] Once the images are labeled with "reference-matching labels," process 600 includes training a machine learning model using the labeled images (614). For example, this can correspond to training the MOS-discriminating machine learning model 108 using a supervised learning approach using training engine 110. In some implementations, the MOS-discriminating machine learning model 108 can be a model similar to the pre-trained neural network model described above, except that its weights are fine-tuned for detecting MOS. Furthermore, compared to a pre-trained neural network, the MOS-discriminating machine learning model 108 can be more constrained, outputting a small number of pre-defined Fourier series coefficients, thereby biasing the model toward detecting roughly circular MOS droplets (and reducing the risk of misidentifying irregularly shaped tumor-like masses as MOS).

[0048] Returning to FIG. 1 , in some implementations, the machine learning model 108 uses a semantic segmentation algorithm, an instance segmentation algorithm, or a panoptic segmentation algorithm. In some implementations, the machine learning model is implemented with a Mask-RCNN architecture, which includes multiple convolutional neural networks and a feature pyramid network (FPN). An advantage of the Mask-RCNN architecture is that the trained machine learning model outputs a mask indicating which pixels represent part of a MOS and which pixels do not. This mask can be used to determine the fluorescent activity of a given MOS, for example, by performing a logical operation (e.g., a bitwise "AND") between the mask and the fluorescent image. However, in some cases, Mask-RCNN may require a large amount of training data that may be inaccessible, and other machine learning techniques may be preferable.

[0049] For example, in some implementations, the MOS discriminative machine learning model 108 can be implemented using a different neural network-based architecture that has been trained to perform image segmentation and generate masks similar to the Mask-RCNN architecture. In one implementation, the MOS discriminative machine learning model 108 can be a neural network trained to output the coefficients of a Fourier series representation of one or more image segmentation masks corresponding to the MOS. In particular, because it is known a priori that the MOS tends to be approximately spherical, the neural network can be constrained to output only a small number of predefined Fourier series coefficients (e.g., a perfect circle can be represented by only a single Fourier coefficient, since more complex higher-order representations are unlikely to be necessary). This can reduce the complexity of the neural network, shorten the time required for image analysis, and reduce the amount of training data required to train the machine learning model, compared to the aforementioned Mask-RCNN architecture. An example implementation of instance segmentation using Fourier mapping in an implicit neural network is described in "Fully Automatic Cell Segmentation with Fourier Descriptors." (December 17, 2021) (www.biorxiv.org / content / 10.1101 / 2021.12.17.472408v1) and Cell segmentation and representation with shape priors, Computational and Structural Biotechnology Journal, Vol. 21, 2023, pp. 742-740 (doi.org / 10.1016 / j.csbj.2022.12.034), both of which are incorporated herein by reference in their entireties.

[0050] In some implementations, the machine learning model may be implemented using one or more other architectures (e.g., U-Net, Single Shot Multibox Detector (SSD), YOLO, Detection Transformer (DETR), Vision Transformer (ViT), etc. In some implementations, the machine learning model may utilize a modified face recognition algorithm.

[0051] The drug response engine 118 is configured to retrieve the MOS instances 114 and MOS attributes 116 and generate a report 120. The drug response engine 118 normalizes cell viability measurements, such as CellTiter-Glo (CTG) luminescence cell viability values, across wells. For example, the drug response engine 118 uses the total surface area, MOS count, or pre-dose fluorescence integral values ​​in the MOS instances output from the machine learning model to normalize plating variability, for example, by dividing the cell viability measurements for each well by their surface area ratio. Based on the adjusted cell viability measurements, the drug response engine 118 outputs a drug response curve and summarizes drug recommendations for the patient.

[0052] In some implementations for imaging assays based on live cell dyes and dead cell dyes, the drug response engine 118 determines a live cell dye signal and a dead cell dye signal. In some implementations, the drug response engine 118 determines a ratio (also called integrated intensity) of the live cell dye signal to the dead cell dye signal.

[0053] Computer 130 can generate rendering data that, when rendered by a device having a display, such as user device 150 (e.g., a computer with monitor 150a, server 150b, or other suitable user device), can cause the device to output data including MOS instances 114 and MOS attributes 116. Such rendering data can be transmitted by computer 130 over network 128 to user device 150 and processed by user device 150 or an associated processor to generate output data for display on user device 150. In some implementations, user device 150 can be coupled to computer 130. In such cases, the rendered data is processed by computer 130 to cause computer 130 to display MOS instances, e.g., a visual representation of the identified MOS and a summary of its attributes, on a user interface.

[0054] 2 is a flow chart illustrating an example of a process 200 for performing an image-based MOS drug assay. The process is described as being performed by a system of one or more computers suitably programmed in accordance with the present disclosure. For example, computer 130 of FIG. 1 may perform at least a portion of the example process. In some implementations, various steps of process 200 may be performed in parallel, in combination, in a loop, or in any order.

[0055] The system acquires image data of a well plate containing a plurality of microorganism-associated spheres (MOS) (202). In some implementations, the MOS are derived from patient-derived tissue samples, such as clinical tumor samples and biopsy samples from metastatic tumors. In some implementations, the MOS are stained with a stain (e.g., Invitrogen™ CellTracker™ Deep Red dye) that nonspecifically binds to the organic tissue and substrate of the MOS. The image data includes bright-field and fluorescent images of the well plate. The images are collected using an imaging cytometer.

[0056] In response to applying a machine learning model (e.g., MOS identification machine learning model 108) configured to identify at least some instances of a plurality of microorganisms in the image data (204), the system obtains (i) a mask representing each instance of the microorganisms, and (ii) attributes of each instance of the microorganisms, such as total surface area and, if applicable, live / dead cell signal. In some implementations, the system uses tumor-specific fluorescent dyes to generate an additional mask that can be intersected with the MOS mask to filter stromal cells and non-tumor-like masses within the mask, potentially in addition to size filtering of the resulting intersected mask. This mask includes a visual representation of each instance of the MOS in the image data. In some implementations, the system displays a visual representation of each instance of the MOS on a user interface upon user request. For example, as shown in FIG. 3A, the user interface displays each instance of the MOS with a border around the MOS.

[0057] The system normalizes well-to-well variations in the well plate (206) based on the mask and attributes (e.g., well-to-well variations due to different numbers of MOS in each well, different sizes of MOS in each well, or different initial biomass / tumor cell content of MOS). One approach to normalizing well-to-well variations is based on the total surface area of ​​the MOS. For example, the system obtains the total surface area of ​​each instance of microorganisms in the well plate. As shown in Figures 4C-4D, total surface area has been shown to correlate with the adenosine triphosphate (ATP) levels of each instance of microorganisms. The system obtains cell viability in response to drug treatment for each well in the well plate and adjusts each cell viability across multiple wells in the well plate based on the total surface area. This approach is distinct from relying on cell viability measured by absolute cell counts, which is relatively unreliable due to well-to-well variations.

[0058] In some implementations, the system performs a drug assay on a plurality of microorganism spheres (208). The drug assay, such as the CellTiter-Glo (CTG) luminescent cell viability assay, measures cell viability in response to drug treatment in a given well within a well plate. The system determines normalized cell viability within the well plate (210). In some implementations, the system determines cell viability by considering only fluorescent signals originating within the boundaries of the identified MOS (e.g., eliminating noise from outside the MOS, which should not contain three-dimensional cell aggregates).

[0059] In some implementations, the system treats a given well with a live cell dye, such as calcein-AM or Mitotracker Viewer, and a dead cell dye, such as ethidium homodimer-2 or fluorescently conjugated annexin V. After treating a given well, the system performs a drug assay on the plurality of microorganisms in the given well to determine an integrated cell viability in the given well. This integrated cell viability includes information such as a live cell signal indicating the proportion of live cells in each MOS instance and a dead cell signal indicating the proportion of dead cells in each MOS instance. In some implementations, the system determines a cytotoxic or cytostatic drug response based on the integrated cell viability.

[0060] In some implementations, the system determines the fluorescent activity of one or more instances of the microorganisms based on a mask output by the machine learning model. The fluorescent activity includes live cell dye signals and dead cell dye signals of the one or more instances of the microorganisms. To determine the fluorescent activity, the system performs a bitwise logical operation (e.g., an "AND" operation) between the image data and the mask. In some implementations, the system iteratively adjusts the mask output by the machine learning model to capture the dead cell dye signals, and outputs the fluorescent activity of the one or more instances of the microorganisms based on the adjusted mask.

[0061] The term "configured" is used herein in connection with systems and computer program components. A system consisting of one or more computers configured to perform particular processes or operations means that the system has software, firmware, hardware, or a combination thereof installed on it. One or more computer programs configured to perform particular processes or operations means that one or more programs contain instructions that, when executed by a data processing device, cause the device to perform those processes or operations.

[0062] Embodiments and functional operations of the subject matter described herein can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, or computer hardware, including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., as one or more modules for execution by or control the operation of a data processing apparatus, or as computer program instructions encoded on a non-transitory tangible storage medium. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random-access or serial-access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiving apparatus for execution by a data processing apparatus.

[0063] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. The apparatus may also be or include application-specific logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that establishes an execution environment for computer programs, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations of these.

[0064] A computer program may also be referred to as a program, software, software application, app, module, software module, script, or code, and may be written in any form of programming language, including compiled or interpreted, declarative or procedural. It may also be deployed in any form, such as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subprograms, or code portions). A computer program may be deployed to execute on one or more computers, which may be located at one site or distributed across multiple sites and interconnected by a data communications network.

[0065] The term "database" is used herein broadly to refer to any collection of data. The data need not be structured in any particular way, or may be unstructured at all, and may be stored on storage devices in one or more locations. Thus, for example, an index database may contain multiple collections of data, each organized and accessed in a different way.

[0066] Similarly, the term "engine" is used broadly herein to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers may be dedicated to a particular engine, or multiple engines may be installed and run on the same computer(s).

[0067] The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs that perform functions by processing input data and generating output. These processes and logic flows may also be performed by special purpose logic circuitry (e.g., FPGAs or ASICs), or a combination of special purpose logic circuitry and one or more programmable computers.

[0068] A computer suitable for running a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or may include other types of central processing units. Typically, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are the central processing unit for executing or generating instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by, or integrated with, special-purpose logic circuitry. Typically, a computer also includes one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, and / or is operatively connected to receive data from or transfer data to these devices. However, a computer need not have such devices. Furthermore, a computer may be incorporated in other devices, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive.

[0069] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0070] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer that includes a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) for the user to provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic input, voice input, or tactile input. Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from the web browser. A computer can also interact with a user by sending text messages or other types of messages to a personal device, such as a smartphone running a messaging application, and receiving reply messages from the user in response.

[0071] A data processing device for implementing machine learning models may also include, for example, application-specific hardware accelerator units for handling common and computationally intensive portions of machine learning training or operation, i.e., inference, workloads.

[0072] The machine learning model can be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0073] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components (e.g., a data server), middleware components (e.g., an application server), or front-end components (e.g., a client computer with a graphical user interface, a web browser, or an app through which a user interacts with an implementation of the subject matter described herein). A computer can also be implemented in a system that includes a combination of one or more of these back-end, middleware, and front-end components to interact with a user. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0074] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data, such as HTML pages, to a user device, which acts as a client, displaying the data to the user and receiving input from the user. Data generated at the user device, such as a result of a user interaction, can be received from the device by the server.

[0075] While many specific implementation details are set forth herein, these should not be construed as limiting the scope of the invention or any of the subject matter that may be claimed, but rather as descriptions of features that may be unique to particular embodiments of a particular invention. Certain features that are described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while the features described above may be described as functioning in a particular combination and may initially be claimed as such, one or more features of a claimed combination may, in some cases, be excluded from this combination, and the claimed combination may be directed to a subcombination or variations thereof.

[0076] Similarly, although acts may be shown in a particular order in the figures or recited in the claims in a particular order, it should not be understood that such acts must be performed in the order shown or sequentially, or that all of the acts shown must be performed to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and further, it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. [Example]

[0077] The present disclosure provides the following examples to help explain the systems, methodologies, etc., but not to limit the scope of the claims.

[0078] In the examples below, the following methods were used.

[0079] method

[0080] Bright-field and fluorescence imaging of microorganisms (MOS)

[0081] The plates were scanned using a Celigo image cytometer and its software. Brightfield images were collected on the day of drug administration (day 0) and the day before the CellTiter-Glo (CTG) assay. Once the plate was inserted, the device was configured to acquire images in the brightfield, green fluorescent, and red fluorescent channels. The brightfield channel was used to focus the device and ensure that the MOS boundaries were clear. The exposure time for the green (live cell) fluorescent channel was then adjusted to ensure a pixel intensity of less than 200 across all triplicate wells of the control to avoid overexposure. This process was repeated for the red (dead cell) fluorescent channel in triplicate wells of the positive control cell-killing condition. After scanning the plate, merged images of each well were exported in TIFF format and used for subsequent image analysis.

[0082] Live and dead cell staining

[0083] On the day of the CTG assay, live and dead cell staining was performed prior to the CTG assay. For 96-well plates, a working staining solution containing 20 μM Ethidium Homodimer-2 (Thermo Fisher Scientific, Part Number E3599) and 5 μM Calcein AM (Thermo Fisher Scientific, Part Number C3100MP) was prepared by mixing 1 mM Calcein AM, 1 mM Ethidium Homodimer-2, and CRC medium at a ratio of 1:8:191. For 384-well plates, a working staining solution containing 10 μM Ethidium Homodimer-2 and 2.5 μM Calcein AM was prepared by mixing 1 mM Calcein AM, 1 mM Ethidium Homodimer-2, and colorectal cancer (CRC) medium at a ratio of 1:8:391. After preparation, 11 μL and 10 μL of dye solution were added to each well of the 96-well and 384-well plates, respectively, for final well concentrations of 2 μM Ethidium Homodimer-2 and 0.5 μM Calcein AM, and the plates were then incubated at 37°C for 45 minutes.

[0084] Cell viability assay

[0085] To measure cell viability, a CTG luminescent cell viability assay was used. CellTiter-Glo 3-D reagent (Promega, part number G9681) was added at a 1:1 ratio to the initial well volume (100 μL for 96-well plates and 40 μL for 384-well plates). The plate was placed on a shaker for 30 minutes. The luminescence of each well was measured using a plate reader.

[0086] Machine learning for identifying organoids / tumor-like masses (MOS)

[0087] A machine learning model was trained using training data from established colorectal MOS bright-field images and corresponding Calcein-AM fluorescent images. All images were collected using a Celigo image cytometer. Detectron's Mask-RCNN implementation was used to train the machine learning model. This configuration used a ResNet50 backbone and a feature pyramid network (FPN). Reference-matched labels representing MOS instances were derived from the fluorescent images by identifying each discrete active region as a separate MOS instance. Because some of the fluorescent images in the training data were not fully saturated, binarization was performed using a sliding threshold. First, a saturation offset d was calculated as 255 minus the maximum pixel value in the fluorescent image. Next, the threshold was calculated as max(30,90-d). Finally, each pixel greater than this threshold was set to 255, and each pixel less than this threshold was set to 0. The machine learning model was trained on the resulting labels for 20 epochs with a learning rate of 0.00025.

[0088] For MOSs containing both live and dead cells, the dead cells were often located on the outer surface of the organoid / tumor-like mass or scattered around it. As a result, some of these dead cells were excluded from the predictions made by the machine learning model. Therefore, for studies involving the ratio of live to dead cell stains, the predicted object mask was expanded to capture all the fluorescent signal of the dead cell stain. The size expansion was performed using a single iteration of the OpenCV dilation algorithm with a kernel size of 10 × 10. Any other MOSs detected by the machine learning model that overlapped with this expanded region were removed before calculating the integrated fluorescence.

[0089] Data analysis of image-based drug assay pipelines

[0090] Cell viability was assessed using both metabolic and imaging parameters. For metabolic assays, brightfield images from Day 0 were used to normalize CTG measurements for each well. The total surface area (tSA) output from the machine learning model was used to normalize plating variability. The measured CTG values ​​for each well were then divided by these surface area ratios to determine adjusted CTG values. These adjusted CTG values ​​were used to generate drug response curves, e.g., viability curves for each drug condition.

[0091] In the live- and dead-cell dye-based imaging assay, the integrated live (Calcein AM) and dead (Eth) cell dye signals for each segmented object were calculated. To assess drug responses at the individual MOS level, the integrated intensity or the Calcein AM / Eth ratio was displayed on a scatter plot or histogram. The correlation between the median live / dead dye integrated intensity from each well and the dose-dependent drug response was calculated and plotted.

[0092] Example 1. Combining MOS and machine learning to achieve a normalization method that improves the robustness of large-scale drug assays.

[0093] MOS, such as organoids / tumor-like masses derived from primary tissues in clinical settings, are often limited in number and heterogeneous in size. Therefore, well-to-well and plate-to-plate variability poses significant challenges in traditional drug response assays, such as large-scale assays that measure metabolic activity, such as intracellular ATP levels, at the well level. Furthermore, endpoint assays like CTG cannot assess growth kinetics, characterize heterogeneity, distinguish between cytotoxic and cytostatic effects, or distinguish between tumor and stromal cells. These are important for evaluating drug responses in clinical settings. Unlike the multiple focal planes required for culturing large-scale organoids in BME domes, most MOS settled to the bottom of microplate wells (e.g., 96-well and 384-well plates) and were located in the same focal plane after dispensing (Figures 4A-4B; scale bar 1,000 µm). Therefore, it is possible to rapidly image each MOS without time-consuming multiple Z-stack scans. Using the Celigo image cytometer, we acquired images of individual MOS (3 channels) from an entire 96-well plate in approximately 5 minutes. We developed a machine learning method using Mask-RCNN to accurately segment MOS from acquired bright-field images (Figure 3A) and output their size and fluorescence intensity. The total surface area (tSA) output by the machine learning method showed high correlation with the CTG luminescence signal detected in both the cyst model (Figure 4C) and the dense colorectal cancer (CRC) MOS model (Figure 4D).

[0094] We evaluated how tSA can be used to normalize well-to-well plating variability when performing drug assays. As shown in Figure 3B, after normalizing the raw relative luminescence units (RLU) values ​​of the CTG assay with tSA on Day 0 (red curve), the range error bars and coefficient of determination (R) of the adjusted CTG values ​​are plotted. 2Both the blue and red curves were significantly improved when these two models were treated with SN38, 5-FU, or oxaliplatin compared with the non-normalized blue curve (Figure 3B). This suggests that the image-based Day 0 tSA normalization strategy can overcome well-to-well plating variability. This improvement makes the CTG-based drug assay more robust and sensitive.

[0095] Based on a machine learning model, tSA normalization was applied to MOS models derived from three different types of primary tumors (CRC, lung, and breast) at P0 (Figure 4E), 4–7 days after establishment from primary tissue digestion. A dose-dependent drug response to chemotherapy treatment was observed in all three MOS models tested (blue curve, Figure 3C). However, due to heterogeneity in the establishment rate and size of MOS in each well, there was significant variability for each drug concentration. With tSA normalization, we observed a significant decrease in the range error bars of the adjusted CTG results and an increase in R-squared (red curve, Figure 3C). Furthermore, the sum of squares of the adjusted CTG values ​​significantly decreased in both established PDO and primary tissue-derived MOS, with a p-value of 0.011 in a two-tailed paired t-test (Figure 3D).

[0096] Example 2. Orthogonal approaches combining machine learning methods and fluorescence imaging distinguish between both cytostatic and cytotoxic drug effects and capture heterogeneous drug responses at single tumor / organoid resolution.

[0097] To further enhance the resolution and power of the MOS drug assay pipeline, we developed an orthogonal method by spiking a combination of a live cell dye (calcein AM, CAM) and a dead cell dye (ethidium homodimer II, EtH) into the drug assay plate prior to performing the CTG assay (Figure 5A; scale bar 200 µm). In addition to MOS size, we were able to quantitatively measure the integrated intensity of the CAM and EtH dyes. We observed that treatment with erlotinib (an EGFR inhibitor) and SN38 reduced the size of CRC#5 (top panel). However, the ratio of CAM / EtH dye integrated intensity decreased only in the SN38-treated wells (Figure 5A, top panel), but not in the erlotinib-treated condition. This suggests that erlotinib induced cell activation, whereas SN38 induced cytotoxicity in this model. Similar changes in the integrated intensity ratio of CAM / EtH dye were observed in another model, CRC#6, when treated with erlotinib and SN38. However, treatment with erlotinib did not result in a clear decrease in the size of the CRC#6 model, indicating that CRC#6 is resistant to EGFR inhibition.

[0098] Using a machine learning system, upon drug administration, in addition to the CTG assay, different readouts (size, integrated live cell dye signal, integrated dead cell dye signal) of individual MOSs were automatically obtained (Figure 5B, Figure 5D). In Figure 5B, the size of each dot indicates the relative surface area of ​​individual segmented MOS instances.

[0099] We observed drug response curves for two CRC MOS models treated with erlotinib or SN38. In Figure 5C, the blue curve was plotted based on the CTG assay, and the red curve was plotted based on the median ratio of CAM / Eth dye integrated intensity. Based on the median CAM / Eth dye ratio for each MOS, we observed that the dose-dependent drug response of these two CRC models was comparable when treated with SN38 compared to the CTG-based assay (Figure 5C, lower panel). However, in CRC#5, a discrepancy was observed between the CTG plot and the CAM / Eth ratio plot, thus confirming that erlotinib treatment caused a cytostimulatory effect rather than cytotoxicity in CRC#5. Furthermore, as shown in Figures 5A and 5D, we identified drug-resistant clones in the SN38-treated CRC#6 model. In Figure 5D, the red square indicates a drug-resistant clone identified in CRC#6 treated with SN38. The x-axis indicates the range of drug concentrations. The size of each dot indicates the relative surface area of ​​each segmented object. The grey band indicates 1σ based on the number of objects, not taking into account their size.

[0100] Furthermore, we observed more heterogeneous drug responses in primary tissue-derived MOS (Figure 5E) compared with established models CRC#5 and CRC#6. Therefore, by combining the CAM / EtH cell dye ratio with size measurement and CTG assay, we were able to distinguish between cytotoxic and cytostatic drug effects.

[0101] Furthermore, in MOS of sarcoma cancer cells treated with docetaxel or gemcitabine on day 7 after establishment (Figure 4F), large-scale CTG readouts (raw CTG or adjusted CTG) failed to provide significant dose-dependent drug response curves due to the limited number of tumor-like masses, the presence of prominent resident stromal cells within the MOS droplets, and well-to-well variability. Meanwhile, the median live / dead cell ratios detected and measured within the same wells demonstrated a clear dose-dependent drug response to docetaxel treatment (Figure 5F, lower panel) but a less sensitive response to gemcitabine treatment. Image-based MOS drug assays further clarified the three-dimensional structure of MOS from individual stromal cells by overcoming fundamental limitations of bulk assays, such as cell number, well-to-well variability, heterogeneity, and signal-to-noise ratio. Other embodiments

[0102] While various example embodiments have been described above, it should be understood that the scope of the present disclosure is defined by the scope of the following claims, and that the foregoing description is intended to illustrate, but not limit, the scope of the present disclosure. Other aspects, advantages, and modifications are within the scope of the following claims. For example, the order in which various methods and steps described are performed may often be changed in alternative embodiments, and in other alternative embodiments, one or more method steps may be omitted entirely. Optional features of various device and system embodiments may be included in some embodiments and not in other embodiments. Accordingly, the foregoing description has been provided primarily for illustrative purposes and should not be construed as limiting the scope of the invention, which is set forth in the claims.

[0103] The examples and figures included herein illustrate, by way of illustration, not limitation, specific embodiments in which the subject matter may be practiced. As noted above, other embodiments may be utilized and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Although such embodiments of the inventive subject matter may be individually or collectively referred to herein as the "invention," this is merely for convenience and is not intended to intentionally limit the scope of the present application to any single invention or inventive concept when, in fact, multiple inventions are disclosed. Thus, while specific embodiments have been illustrated and described herein, any configuration calculated to achieve the same purpose may be substituted for the specific embodiment illustrated. The present disclosure is intended to cover all adaptations and modifications of the various embodiments. Combinations of the above embodiments with other embodiments not specifically described herein will be apparent to those skilled in the art upon reviewing the above description.

Claims

1. acquiring image data of a well plate containing a plurality of microscopic organic spheres; In response to applying a machine learning model configured to identify at least some instances of the plurality of microscopic organic spheres within the image data, obtaining (i) a representation of each instance of the microscopic organic sphere, and (ii) attributes of each instance of the microscopic organic sphere; normalizing well-to-well variability within the well plate based on the indication and the attribute.

2. performing a drug assay on the plurality of microorganisms, the drug assay measuring cell viability in response to drug treatment for a given well within the well plate; The method of claim 1 , further comprising measuring the cell viability in the normalized well plate.

3. 10. The method of claim 1, wherein the microorganisms are derived from a tissue sample from a patient.

4. The method of claim 3 , wherein the patient-derived tissue sample comprises a biopsy sample from a metastatic tumor.

5. The method of claim 3 , wherein the patient-derived tissue sample comprises a clinical tumor sample containing both cancer cells and stromal cells.

6. The method of claim 1 , wherein the image data includes bright field and fluorescent images of the well plate.

7. 7. The method of claim 6, wherein the bright field and fluorescent images of the well plate are collected using an imaging cytometer.

8. the display includes a visual representation of each instance of the microscopic organic sphere in the image data; and The method of claim 1 , comprising displaying the visual representation of each instance of the microscopic organic sphere on a user interface.

9. 10. The method of claim 1, further comprising determining fluorescent activity of one or more instances of the microorganisms based on the representation output by the machine learning model, the fluorescent activity comprising a live cell dye signal and a dead cell dye signal of the one or more instances of the microorganisms.

10. 10. The method of claim 9, wherein determining the fluorescent activity of one or more instances of the microscopic organic spheres comprises performing a logical operation between the image data and the representation.

11. iteratively adjusting the representation output from the machine learning model such that the dead cell dye signal is obtained; 10. The method of claim 9, further comprising: outputting the fluorescent activity of one or more instances of the microscopic organic spheres based on the adjusted representation.

12. 2. The method of claim 1, wherein the attributes of each instance of the microorganism include a total surface area of ​​each instance, a live cell dye signal, and a dead cell dye signal.

13. Normalizing for well-to-well variability in the well plate, determining a total surface area of ​​each instance of the microorganisms in the well plate, the total surface area being correlated with the level of adenosine triphosphate (ATP) of each instance of the microorganisms; Obtaining cell viability in response to drug treatment for each well of the well plate; and adjusting the cell viability of each of a plurality of wells within the well plate based on the total surface area.

14. treating predetermined wells in the well plate with a live cell dye and a dead cell dye; performing a drug assay on the plurality of microorganisms in said predetermined wells; 10. The method of claim 1, further comprising determining integrated cell viability in the selected wells.

15. 15. The method of claim 2 or 14, wherein the drug assay comprises a CellTiter-Glo (CTG) luminescent cell viability assay.

16. 15. The method of claim 14, wherein the live cell dyes include calcein-AM and Mitotracker viewer.

17. 15. The method of claim 14, wherein the dead cell dye comprises ethidium homodimer-2 and fluorescently conjugated annexin V.

18. 15. The method of claim 14, further comprising determining a cytotoxic or cytostatic drug response based on the integrated cell viability.

19. 10. The method of claim 1, further comprising filtering stromal cells and non-tumor spheroids in the display in response to applying a size filter to the display, the size filter removing cells below a predetermined size from the display.

20. 10. The method of claim 1, further comprising treating one or more wells in the well plate with a stain that non-specifically binds to organic tissue and the substrate of a plurality of the microorganisms.

21. 2. The method of claim 1, wherein acquiring the image data of the well plate comprises acquiring the image data of one or more wells in the well plate at a single focal plane or using a two-dimensional projection of a Z-stack of a three-dimensional confocal microscope.

22. 2. The method of claim 1, wherein obtaining the representation of each instance of the microscopic organic sphere in the image data comprises generating a corresponding mask represented using a Fourier series representation, the Fourier series representation being generated based on coefficients output by the machine learning model.

23. 23. The method of claim 22, wherein the machine learning model is trained using images including labeled microorganism instances, the labeled microorganism instances being labeled using a pre-trained neural network configured to generate image instance segmentation masks.

24. 24. The method of claim 23, wherein the image containing the labeled microscopic organic sphere instances is generated from an unlabeled image, and the unlabeled image is preprocessed using (i) a mathematical transformation that highlights fluorescence corresponding to stained microscopic organic spheres present in the unlabeled image, (ii) a mathematical morphological process that highlights disk-shaped objects within a defined size range, and (iii) an image resizing operation.

25. 25. The method of claim 24, wherein the unlabeled image is pre-processed with one or more filters to remove noise signals.

26. For each microscopic organic sphere of the plurality of microscopic organic spheres, acquiring a first input comprising brightfield image data and fluorescent image data; obtaining a second input indicating a criteria-matching label representing each microscopic organic sphere instance; and training a machine learning model configured to identify (i) each instance of the microorganism and (ii) attributes of each instance of the microorganism using the first input and the second set of inputs across the plurality of microorganisms.

27. 27. The method of claim 26, further comprising applying binarization to the fluorescence image data in response to determining that a saturation level of the fluorescence image data does not meet a predetermined threshold.

28. training the machine learning model, 27. The method of claim 26, comprising providing the first input and the second set of inputs to a Mask-RCNN architecture, the Mask-RCNN architecture comprising a plurality of convolutional neural networks and a feature pyramid network.

29. 27. The method of claim 26, wherein obtaining the second input indicative of a reference-matching label representing each microscopic organic sphere instance comprises generating the reference-matching label by applying a pre-trained neural network configured to generate image segmentation masks to one or more unlabeled images.

30. 30. The method of claim 29, comprising applying to the one or more unlabeled images: (i) a mathematical transformation that enhances fluorescence corresponding to dyed microscopic organic spheres present in the unlabeled images; (ii) a mathematical morphological process that enhances disc-shaped objects within a predetermined size range; and (iii) an image resizing process.

31. 27. The method of claim 26, wherein training the machine learning model comprises training the machine learning model to output coefficients of a Fourier series representation of one or more image segmentation masks corresponding to the microscopic organic spheres.

32. 27. The method of claim 26, further comprising staining the plurality of microorganisms with a stain that non-specifically binds to organic tissue and to the substrate of the microorganisms prior to acquiring the first input comprising brightfield image data and fluorescent image data.

33. 33. A system comprising one or more processors and one or more storage devices storing instructions operable, when executed by the one or more processors, to cause the one or more processors to perform the method of any one of claims 1 to 32.

34. A non-transitory computer readable medium comprising software instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 32.

Images