Predicting the toxicity of compounds in cellular structures

Abstract

Disclosed are methods, devices, and systems for predicting the toxicity of one or more compounds applied to multiple samples of cellular structures in an in vitro microscopy assay. A set of images associated with the multiple samples is received. Each image in the set of images is input to a first ML model configured to predict a phenotypic feature of the cellular structure in the sample associated with the respective image. Each predicted phenotypic feature associated with each sample is input to a second ML model configured to predict a low-dimensional phenotypic feature embedding for each sample. A distance between the low-dimensional phenotypic feature embedding for each sample and the low-dimensional phenotypic feature embedding of a sample to which a compound with known toxicity has been applied is compared. For each sample, an indication of the toxicity of the sample and the applied compound is output based on the comparison.

Description

[Technical Field]

[0001] The present disclosure relates to devices, systems, and methods for predicting the toxicity of compounds on cellular structures in microscopic assay samples. [Background technology]

[0002] Cellular structures have been developed that can mimic and / or simulate the processes and / or functions of a subject's or patient's organ. These can be used for in vitro testing of the efficacy and / or toxicity of various compounds with respect to the organ, instead of in vivo testing. Such cellular structures include immortalized cell lines developed to mimic or simulate a particular organ of interest. This has resulted in semi-automated toxicity predictive testing systems and methods that can be used to identify compounds that are toxic to the organ of interest using imaging microscopy and observing changes due to toxicity, such as in dose-response (DR) graphs.

[0003] Conventional semi-automated testing systems and methods can be used to identify compounds that produce a detectable signal and evaluate the effects of the compounds on cellular structures in relation to organs. These testing systems are called assays. Once a toxicity predictive assay is developed, researchers can use it to identify compounds with the required activity for toxicity. Typically, compounds are tested at several concentrations and imaged using an imaging microscope, which can generate a DR graph or other metric that is useful for researchers to determine its toxicity. For example, analysis of the DR graph allows researchers to determine whether a compound is active and / or toxic, and at what concentrations.

[0004] High-throughput screening (HTS) is often used to test large numbers of potential compounds. It uses robotics, data processing / control and imaging software, liquid handling devices, and sensitive detectors, allowing researchers to rapidly conduct thousands or even millions of screening tests. However, the large amount of data generated in the imaging and DR steps of an HTS campaign requires careful analysis by researchers to detect artifacts and correct erroneous data points before validating the experiment.

[0005] Unfortunately, even semi-automated toxicity prediction assays using HTS have proven unable to reliably identify the toxicity of all compounds, even those known to be toxic when analyzed on cellular structures that simulate or mimic organs. This has increased the risk of performing in vivo trials on compounds that pass such semi-automated toxicity prediction assays. For example, 20–40% of drug-induced liver injury (DILI) patients present with cholestatic and / or mixed hepatocellular / cholestatic injury patterns. Drug-induced hepatotoxicity, or DILI, is an acute or chronic reaction to natural or manufactured compounds. Traditionally, DILI can be classified based on the clinical presentation (hepatocellular, cholestatic, or mixed), the mechanism of hepatotoxicity, or the histological appearance from liver biopsies. Therefore, reliable in vitro toxicity prediction of compounds is a critical component of drug / compound discovery or research programs.

[0006] Improved methods, devices, systems, and / or architectures that can efficiently and reliably detect or predict the toxicity of compounds on cellular structures, such as using in vitro HTS assays, are desirable. Summary of the Invention [Means for solving the problem]

[0007] According to a first aspect, there is provided a computer-implemented method for predicting toxicity of one or more compounds applied to a plurality of samples of cellular structures in an in vitro microscopy assay, the method comprising: receiving a set of images associated with the plurality of samples; inputting each image of the set of images into a first ML model configured to predict a phenotypic feature of a cellular structure in the sample associated with each image; inputting each of the predicted phenotypic features associated with each sample into a second ML model configured to predict a low-dimensional phenotypic feature embedding for each sample; comparing a distance between the low-dimensional phenotypic feature embedding of each sample and the low-dimensional phenotypic feature embedding of a sample to which a compound with known toxicity has been applied; and outputting, for each sample, an indication of the toxicity of each sample and the applied compound based on the comparison.

[0008] The computer-implemented method of the first aspect, wherein the first ML model is a neural network or a convolutional neural network (CNN) model. Optionally, the neural network or CNN model is trained for classification using cell image training data. Alternatively, the predicted phenotypic features are embedded within all layers of the trained neural network or CNN model, and the method includes outputting the phenotypic features from all layers. Optionally, the final all layers of the neural network or CNN model are used to output the phenotypic feature embeddings.

[0009] The computer-implemented method of the first aspect, wherein the second ML model is based on a Uniform Manifold Approximation and Projection (UMAP) algorithm or a t-SNE algorithm for dimensionality reduction of the phenotypic feature embeddings of the samples, wherein the phenotypic feature embeddings are mapped to a low-dimensional vector space for use in comparing distances between the phenotypic feature embeddings and phenotypic feature embeddings of samples with compounds with known toxicity.

[0010] Optionally, the second ML model is trained with the UMAP technique using unsupervised training based on negative and positive control samples of the plurality of samples to predict a toxicity distance metric associated with a sample to which a compound with known toxicity has been applied. Alternatively, training the second ML model includes iteratively performing a grid search over a set of hyperparameters of the UMAP technique to select hyperparameters that maximize the difference between the negative and positive control samples.

[0011] The computer-implemented method of the first aspect, wherein indicating toxicity of a phenotypic feature embedding of a sample having an applied compound comprises applying the phenotypic feature embedding of the sample having an applied compound to a second ML model to output a low-dimensional embedding of said sample having an applied compound, and determining an indication of toxicity of said sample having an applied compound based on comparing a distance between said low-dimensional embedding and embeddings of one or more samples to which a compound with known toxicity has been applied.

[0012] The computer-implemented method of the first aspect, wherein indicating toxicity of phenotypic feature embeddings of samples having the applied compound further comprises applying the phenotypic feature embeddings of samples having the applied compound to a second ML model to output a low-dimensional embedding of said samples having the applied compound, and applying the low-dimensional embeddings of said samples having the applied compound to a third ML model trained to output an indication of the distance between the low-dimensional embeddings and a set of low-dimensional embeddings associated with negative control samples.

[0013] Train a third ML model based on performing a grid search over a set of hyperparameters of a high-dimensional distance metric algorithm that maximizes the distance between the low-dimensional embeddings of the negative control samples and the positive control samples while minimizing the distance between the low-dimensional embeddings of the negative control samples or minimizing the distance between the low-dimensional embeddings of the positive control samples. The computer-implemented method of the first aspect, further comprising:

[0014] The computer-implemented method of the first aspect, wherein the distance metric is a Wasserstein distance metric and the high-dimensional distance metric algorithm is the Sinkhorn algorithm for estimating the Wasserstein distance between embeddings.

[0015] The computer-implemented method of the first aspect, wherein comparing the distances used to indicate toxicity of the phenotypic feature embeddings of the samples is based on the Wasserstein distance metric.

[0016] The computer-implemented method of the first aspect, wherein receiving the set of images related to a plurality of samples further comprises identifying available samples of cellular structures for analysis in an in vitro microscopy assay based on the steps of: automatically identifying a first set of samples useful for analysis from a plurality of samples in an assay plate; generating a set of two-dimensional (2D) images for each sample in the first set of samples, wherein the set of 2D images for each sample includes a plurality of 2D image slices taken along a z-axis of each sample; identifying a set of available samples from the set of 2D image slices; and outputting data representing the set of samples available for analysis as a set of images.

[0017] 10. The computer-implemented method of the first aspect, wherein automatically identifying a first set of samples further includes: for each sample in the plurality of samples, preprocessing an image of each sample; inputting the preprocessed sample image to a first machine learning (ML) model configured to identify regions of interest in an input sample image comp that include cellular structures; inputting the identified regions of interest in the sample image to a second ML model configured to classify whether the sample is analyzable; and outputting a first set of samples including data representing samples classified as analyzable.

[0018] The computer-implemented method of the first aspect, wherein the first ML model is a convolutional neural network (CNN) or other neural network trained to identify regions of interest that include cellular structures, and the second ML model is a one-class SVM configured to classify whether the regions of interest are analyzable.

[0019] The computer-implemented method of the first aspect, wherein training / constructing the CNN is based on a labeled training dataset comprising a plurality of images, each of the images being annotated with a label comprising data representing, for example, whether a cellular region of interest is present and / or the location of the region of interest within the image.

[0020] Optionally, train / construct a one class SVM to classify whether the region of interest is analyzable or not.

[0021] The computer-implemented method of the first aspect, wherein identifying a set of available samples from the set of 2D image slices further includes: for each sample, identifying a foreground, a background, and multiple uncertain feature regions of the cellular structure in each of the 2D image slices; iteratively combining the foreground, background, and multiple uncertain feature regions of the 2D image slices to generate a single 2D image of the cellular structure; and selecting samples from the set of available samples based on the quality of the single 2D image.

[0022] Optionally, the plurality of uncertain feature regions includes a plurality of uncertain foreground features and a plurality of uncertain background features.

[0023] The computer-implemented method of the first aspect, wherein the cellular structure comprises one or more from the group consisting of a cellular spheroid structure, a vesicle, an organoid, and any other suitable cellular structure.

[0024] The computer-implemented method of the first aspect, wherein the plate comprises a plurality of wells having a sample of a cellular structure within each well.

[0025] According to a second aspect, there is provided an apparatus comprising a processor, a memory unit, and a communication interface, the processor coupled to the memory unit and the communication interface, the processor and the memory configured to perform a computer-implemented method according to the first aspect, combinations thereof, modifications thereof, and / or as described herein.

[0026] According to a third aspect, there is provided a computer-readable medium comprising data or instruction code which, when executed on a processor, causes the processor to perform the computer-implemented method according to the first aspect, combinations thereof, modifications thereof, and / or those described herein.

[0027] According to a fourth aspect, there is provided a tangible computer-readable medium comprising data or instruction codes for predicting the toxicity of one or more compounds applied to a plurality of samples of cellular structures in an in vitro microscopy assay, which, when executed on one or more processors, causes at least one of the one or more processors to perform at least one step of a method consisting of receiving a set of images associated with the plurality of samples; inputting each image of the set of images to a first ML model configured to predict a phenotypic feature of a cellular structure in the sample associated with each image; inputting each of the predicted phenotypic features associated with each sample to a second ML model configured to predict a low-dimensional phenotypic feature embedding for each sample; comparing a distance between the low-dimensional phenotypic feature embedding of each sample and a low-dimensional phenotypic feature embedding of a sample with a compound having known toxicity; and outputting, for each sample, an indication of the toxicity of each sample and the applied compound based on the comparison.

[0028] According to a fifth aspect, there is provided a system comprising: a receiver module configured to receive a set of images associated with a plurality of samples; a first ML model module configured to input each image of the set of images into a first ML model configured to predict a phenotypic feature of a cellular structure within the sample associated with each image; a second ML model module configured to input each of the predicted phenotypic features associated with each sample into a second ML model configured to predict a low-dimensional phenotypic feature embedding for each sample; a distance comparison module configured to compare a distance between the low-dimensional phenotypic feature embedding of each sample and a low-dimensional phenotypic feature embedding with a compound having known toxicity; and an output module configured to output, for each sample, an indication of the toxicity of each sample and an applied compound based on the comparison.

[0029] In various implementations, computer program instructions are provided, optionally stored on a non-transitory computer-readable medium, that, when executed by one or more processors of a data processing apparatus, cause the data processing apparatus to execute the program instructions and cause the one or more processors to perform operations that include one or more aspects of the implementations described above and / or below (including one or more aspects of the accompanying claims).

[0030] In various implementations, an apparatus is disclosed that includes a computer-readable storage medium having embedded program instructions and one or more processors configured to execute the program instructions, causing the apparatus to perform operations including one or more aspects of the implementations described above and / or below (including one or more aspects of the appended claims). The apparatus may include one or more processors or special-purpose computing hardware.

[0031] In order that the present invention may be more easily understood, embodiments thereof will now be described, by way of example only, with reference to the accompanying drawings in which: [Brief explanation of the drawings]

[0032] [Figure 1a] FIG. 1 illustrates an exemplary toxicity prediction system for predicting the toxicity of a compound applied to a cellular structure of a sample in a microscopy assay, according to some embodiments of the present invention. [Figure 1b] FIG. 1B illustrates an exemplary toxicity prediction process for use in the system of FIG. 1A, according to some embodiments of the present invention. [Figure 2] FIG. 1B illustrates an exemplary neural network for predicting the phenotype embedding of cellular structures within a sample of the microscopy assay of FIG. 1A, according to some embodiments of the present invention. [Figure 3a] FIG. 1 shows an exemplary assay plate with negative and positive control groups of samples for training a deep learning toxicity model, and a test group of samples for input into the trained DL toxicity model, according to some embodiments of the present invention. [Figure 3b] FIG. 3b illustrates an exemplary unsupervised training process for a deep learning toxicity model using negative and positive control groups of samples from FIG. 3a, according to some embodiments of the present invention. [Figure 3c] FIG. 3b shows an example of predicting the toxicity of a compound using the trained deep learning toxicity model of FIG. 3b, according to some embodiments of the present invention. [Figure 4a] FIG. 1 shows another exemplary assay plate having negative and positive control groups of samples for training a deep learning toxicity model, and a test group of samples for input into the trained DL toxicity model, according to some embodiments of the present invention. [Figure 4b] FIG. 4b illustrates exemplary distance matrices for negative and positive control samples of the trained DL model of FIG. 4a, according to some embodiments of the present invention. [Figure 4c] FIG. 4c shows another exemplary distance matrix of negative and positive control samples and test samples for predicting the toxicity of a compound in a test sample using the trained DL toxicity model of FIG. 4b, according to some embodiments of the present invention. [Figure 4d]FIG. 1 shows an example of a conventional toxicity prediction method used on a set of eight compounds applied to a sample of cytoarchitectures. [Figure 4e] FIG. 4d shows an example of conventional toxicity prediction results for a set of compounds from the conventional toxicity prediction method of FIG. 4d. [Figure 4f] FIG. 4e shows an example of toxicity prediction results of a DL toxicity model trained on the same set of compounds as in FIG. 4e. [Figure 5a] FIG. 1 illustrates an example of a quality control image analysis process for selecting and pre-processing microscopic images of cellular structures prior to use in training a DL toxicity model and / or as input to a trained DL toxicity model for predicting the toxicity of compounds applied to the cellular structures, according to some embodiments of the present invention. [Figure 5b] FIG. 5B illustrates an example of a first quality control process of a first quality control device for use in the quality control image analysis process of FIG. 5A when selecting a first set of available images of a sample for analysis, according to some embodiments of the present invention. [Figure 5c] FIG. 5B illustrates an example of an image analyzer of a first quality control device for use in the first quality control process of FIG. 5B, according to some embodiments of the present invention. [Figure 5d-1] FIG. 5c illustrates an example of a first quality control device for use in the first quality control process of FIG. 5b, according to some embodiments of the present invention. [Figure 5d-2] Same as above. [Figure 5e] FIG. 5B illustrates an example of a second quality control process of a second quality control device for use in the quality control image analysis process of FIG. 5A in selecting a final set of available images of a sample for pre-processing and analysis, according to some embodiments of the present invention. [Figure 5f] FIG. 5B illustrates another example of a second quality control process of a second quality control device for use in the quality control image analysis process of FIG. 5A in selecting a final set of available images of a sample for pre-processing and analysis, according to some embodiments of the present invention. [Figure 6a]Schematic diagram of a system / apparatus for carrying out the methods described herein; [Figure 6b] FIG. 1 is a schematic diagram of another exemplary system for performing the methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0033] Common reference numbers are used throughout the drawings to denote similar features.

[0034] Various exemplary implementations described herein relate to methods, devices, and systems for automatically, efficiently, and reliably testing and predicting the toxicity of compounds applied to cellular structure samples in an HTS microscopy assay. The toxicity prediction system receives a set of images of cellular structure samples, at least a group of the samples in the assay being perturbed by one or more compounds under test. The received set of images is applied to a deep learning (DL) model trained and configured to predict the toxicity of the compound's effect on the perturbed cellular structure samples. The DL model generates high-dimensional phenotypic representations of the cellular structures represented in each received input image sample. These are further processed into low-dimensional phenotypic embeddings that focus on cellular structure features associated with toxicity. The toxicity prediction for each image sample is output based on a distance or similarity metric to estimate the distance between each low-dimensional phenotypic embedding and a negative control low-dimensional phenotypic embedding, where the negative control is associated with a cellular structure sample that has not been perturbed by the compound under test.

[0035] The cellular structure of the sample can be associated with an organ of the subject or patient and can be designed to mimic or simulate the organ. The cellular structure can be based on at least one of the group consisting of, but not limited to, a cell spheroid, a vesicle, an organoid, a cellular structure of an immortalized cell line, and any other suitable cellular structure that mimics or simulates one or more processes of the organ of the subject or patient.

[0036] The toxicity prediction system is described herein with reference to, but not limited to, drug-induced hepatotoxicity or drug-induced liver injury (DILI), which is the acute or chronic response of the liver to, for example, a natural or manufactured compound. Up to 20-40% of DILI patients exhibit cholestatic and / or mixed hepatocellular / cholestatic injury patterns. The assay samples use liver-associated cellular structures that develop into hepatocytes in vitro following cholestasis. The type of cellular structure used is, but is not limited to, HepaRG® cells, which are terminally differentiated hepatocytes derived from a human hepatic progenitor cell line that retain many characteristics of primary human hepatocytes.

[0037] HepaRG cells are an immortalized cell line with four key characteristics: 1) the full array of functions, responses, and regulatory pathways of primary human hepatocytes, including phase I and II functions, and transporter activity consistent with that found in primary human hepatocyte populations; 2) bile canaliculi; 3) the potential to express key stem cell properties; and 4) high plasticity and complete transdifferentiation capabilities. After 7 days of maturation, cells can be used directly in experimental assay plates. HepaRG cells can form cellular spheroids that mimic or simulate one or more cellular processes in the liver, which can be stained and / or fluorescently labeled and imaged during in vitro microscopy assays for downstream analysis.

[0038] While the toxicity prediction system is described herein with reference to liver cytoarchitectures / spheroids (e.g., HepaRG cell line) and DILI, this is by way of example only, the present invention is not so limited, and one of skill in the art should understand that the toxicity prediction system can be trained and applied to any organ and / or any associated disease, any cell line, cell array, and / or any type of cytoarchitecture associated with a well of cellular samples with an applied compound. For example, the toxicity prediction system can be applied to samples of cytoarchitectures that mimic organs, such as, but not limited to, lungs, skin, kidneys, pancreas, liver, cardiac cytoarchitectures / heart, neuronal cytoarchitectures, and / or any other organ of a subject or patient. Such cytoarchitectures can be used in samples with compounds, such as in in vitro microscopy assays, and automatically analyzed for toxicity by the toxicity prediction system.

[0039] FIG. 1a illustrates an exemplary toxicity prediction pipeline 100 for automatically predicting the toxicity of one or more compounds applied to multiple samples of cellular structures in an in vitro microscopy assay. The toxicity prediction pipeline 100 includes an in vitro HTS microscopy assay system 102, an imaging analysis system 104, and a toxicity prediction system 106. The in vitro HTS assay system 102 is configured to photograph a set of samples of cellular structures 102a and apply one or more compounds or reagents 102b to the set of samples of cellular structures 102a to input them into a set of well samples in a microscopy assay plate 102c for HTS staining and microscopy assay imaging 102d. In HTS staining and microscopy assay imaging 102d, the sample wells may be treated / stained with fluorescent reagents / compounds to highlight the cellular structures in each sample. For example, the treated / stained cellular structures, such as spheroid structures, vesicles, and / or nuclei, may be imaged by the microscopy imaging system. Thus, the in vitro HTS microscopy assay system 102 is configured to output a set of well image samples 103. Each well sample image in the set corresponds to a respective sample of cellular structure and compound applied in the set of well samples of the assay.

[0040] The imaging analysis system 104 may be configured to receive the set of well sample images and perform image preprocessing and / or analysis to identify which samples from the set of well samples are available for further downstream analysis, such as, for example, toxicity prediction of compounds applied to samples in the set of well samples. One or more image processing and / or machine learning algorithms may be applied to identify the availability of each well sample image based on detected image artifacts and / or imaging defects, etc., and / or to enhance or highlight cellular structures of interest within each available well sample image of the set of well samples. As a result, the set of available well sample images 105 may be output by the imaging analysis system 104 for further downstream analysis. Essentially, the set of available well sample images 105 may be any suitable set of images of cellular structures from the set of well samples of an assay that sufficiently describes the cellular structures of the samples for automated analysis.

[0041] The toxicity prediction system 106 is configured to receive a set of images of cellular structures 105 derived from a sample of multiple cellular structures having one or more applied compounds during an in vitro microscopy assay. In this case, the received set of sample images 105 may be a set of available well sample images 105 output from the image analysis system 104. However, any suitable set of images that sufficiently describe the cellular structures of the set of well samples having applied compounds may be received by the toxicity prediction system 106. For example, a set of well image samples 103 output from the in vitro HTS microscopy assay system 102 may be used as long as the images are sufficiently free of artifacts and / or defects and the compounds in the cellular structures of each sample are analyzable.

[0042] In the toxicity prediction system 106, the set of received sample images 105 may each be input to deep learning (DL) toxicity prediction models 106a-106c configured to predict the toxicity of each corresponding compound applied in an assay from the in vitro HTS microscopy assay system 102. The DL toxicity prediction models 106a-106c may be based on any one or more DL modeling techniques / algorithms and / or machine learning (ML) techniques / algorithms used in training the DL toxicity models to identify or predict whether each of the received sample images 105 will exhibit toxicity, even if no compound has been applied to one or more cellular structures in the well sample. The one or more DL / ML techniques / algorithms may be based on supervised ML, unsupervised ML, and / or semi-supervised ML algorithms, etc. However, for the task of training the DL toxicity prediction models 106a-106c, supervised learning has proven challenging due to the limited number of labeled training datasets of cellular structures that indicate toxicity depending on whether a compound has been applied. As a result, the combined supervised / unsupervised DL model training architecture can be used to train one or more component models of the DL toxicity predictive models 106a-106c.

[0043] For example, in the example of FIG. 1a, the DL toxicity prediction models 106a-106c may include a machine learning (ML) phenotypic feature extraction (FE) model 106a trained to extract phenotypic features of cellular structures from each received sample image in the set of received sample images 105. Supervised learning may be used to train the ML phenotypic FE model 106a. The ML phenotypic FE model 106a may be based on a neural network (NN) classifier trained using supervised training on a readily available labeled / annotated training dataset, for example, but not limited to, to classify images of cells, organoids, spheroids, cellular structures, etc. (e.g., classifying images of cellular structures to determine whether the cells are cancerous or tumor cells). The NN classifier may be based on any NN structure, such as, but not limited to, a feedforward NN (FNN), a recurrent NN (RNN), an artificial NN (ANN), a convolutional NN (CNN), any other type of NN, modifications thereto, or combinations thereof. Prior to the output classification layer or softmax output of the NN classifier, the phenotypic representation of the cell structure may be embedded by the high-dimensional output of one of the hidden layers or all layers of the NN classifier. Rather than outputting a classification, the NN classifier is configured to output an embedding of the phenotypic representation of the cell structure from the hidden layer or all layers.

[0044] Typically, the phenotypic representation embedding of an input image sample from the received image sample set 105 is a high-dimensional representation of the phenotypic features (e.g., for a CNN-type NN classifier / model, the dimension may be on the order of 2024 or more). The trained NN classifier may be used to output a high-dimensional phenotypic feature representation 107 of cellular structures for each input well sample having a compound applied in the set of sample images 105. As an example, a neural network classifier may be based on a convolutional neural network (CNN) for classifying images of cellular structures (e.g., classifying images of cellular structures to determine whether the cells are cancer cells or not), where, once trained, one of the final full layers of the CNN may be used as the phenotypic feature representation for each of the set of received image samples 105 being input.

[0045] The ML phenotype embedding model 106a is coupled to a trained ML low-dimensional (LD) embedding model 106b, which is configured to optimally embed high-dimensional phenotype representations 107 of predicted phenotypic features of each received sample image of the received image samples 105 into low-dimensional phenotypic embeddings 108 in a low-dimensional space for further analysis, the low-dimensional phenotypic feature embeddings 108 including those phenotypic features associated with toxicity. Rather than using supervised training on the ML LD embedding model 106b, one or more unsupervised DL / ML techniques / algorithms are used to ensure that the trained ML LD embedding model 106b outputs LD phenotype embeddings 108 that represent phenotypic features associated with toxicity of the cellular structure. The unsupervised DL / ML techniques / algorithms may be based on at least a clustering algorithm or a dimensionality reduction algorithm, such as, but not limited to, a support vector machine (SVM), a uniform manifold approximation and projection (UMAP), or a t-distributed stochastic neighbor embedding (t-SNE) type algorithm, combinations thereof, modifications thereof, etc.

[0046] For example, the ML LD embedding model 106b can be trained using negative and positive control samples contained on an assay plate 102c in an in vitro microscopy assay. That is, the assay plate 102c can include a negative control group of samples, which can be a first group of wells in the assay plate 102c to which the samples were not perturbed or to which no compound was applied, and a positive control group of samples, which can be a second group of wells in the assay plate 102c to which the samples were perturbed with a known compound having known toxicity. The assay plate 102c can also include a third group of wells in the assay plate 102c containing samples of cellular structures with the compound applied. Thus, the negative and control groups of images associated with the negative and positive control samples can be used to train the ML LD embedding model 106b in an unsupervised manner.

[0047] For example, negative and control groups of images are input into an ML phenotype embedding model 106a, which outputs corresponding negative and positive high-dimensional phenotype representations. Iterative optimization of the parameters of a UMAP or t-SNE algorithm can then be performed on the high-dimensional negative and positive control phenotype representations output from the ML phenotype embedding model 106a, where the difference between the resulting LD negative and positive control phenotype representation embeddings is maximized. The resulting optimized parameters can be used with the UMAP or t-SNE algorithm for dimensionality reduction of the high-dimensional phenotype representations 107 corresponding to a third group of well samples. The LD phenotype representations 107 can be output for comparison with the negative control (NC) LD phenotype representations using an appropriate distance or similarity metric used by a trained ML distance model 106c.

[0048] The ML LD embedding model 106b is coupled to a trained ML distance / prediction model 106c / 106d configured to perform a distance or similarity metric comparison / estimation 109 between the LD phenotypic representation of one of the well image samples and the negative control LD phenotypic representation using an appropriate distance or similarity metric for the LD space of the LD phenotypic representation. The ML distance model 106c may output data representing each distance comparison estimate 109, which is input to the ML prediction unit 106d for predicting toxicity. Note that the LD space of the LD phenotypic representations embedding output by the ML LD embedding model 106b may still be considered a high-dimensional space in which a Euclidean distance or similarity metric is broken down and / or cannot be reliably used. Therefore, a high-dimensional distance or similarity metric may be used, for example, but not limited to, based on Wasserstein distance and / or any other high-dimensional distance or similarity metric. The ML distance / prediction model 106c / 106d outputs toxicity predictions as probabilities based on distance comparisons 109 (eg, Wasserstein distance comparisons) between each LD phenotypic representation of the sample and the negative control LD phenotypic representation.

[0049] FIG. 1b illustrates an exemplary toxicity prediction process 110 for use by the toxicity prediction system 106 in predicting the toxicity of one or more compounds applied to multiple samples of cellular structures in the in vitro microscopy assay pipeline 100 of FIG. 1a. The toxicity prediction process 110 includes the following steps: In step 112, a set of images associated with multiple samples is received based on the output of an in vitro microscopy assay. Each sample image includes image data sufficient to describe the cellular structure of the associated sample for automated processing and analysis. In step 114, each image in the set of images is input to a first ML model configured to predict phenotypic features of the cellular structure in the sample associated with each image. In step 116, each predicted phenotypic feature associated with each sample is input to a second ML model configured to predict a low-dimensional phenotypic feature embedding for each sample. In step 118, the distance between the low-dimensional phenotypic feature embedding for each sample and the low-dimensional phenotypic feature embedding for a sample treated with a compound with known toxicity (e.g., a negative control sample) is compared. In step 120, for each sample, an indication (eg, probability) of toxicity of the sample and the applied compound is output based on the comparison.

[0050] FIG. 2 illustrates an exemplary neural network classifier 200 for use in the ML phenotypic FE model 106a of the toxicity prediction system 106 of FIG. 1a. The NN classifier 200 is configured to predict or output phenotypic representations / embeddings from images of cellular structures within a sample of the microscopy assay of FIG. 1a. The NN classifier 200 is based on a CNN-type architecture including a first portion of a CNN-type network 202 followed by one or more fully connected layers 2104a-204n and a classifier output layer 206. One or more of the outputs 205a-205n of the fully connected layers 204a-204n may be tapped and / or selected 208 to output as phenotypic representation embeddings 210 associated with an input sample image representing a cellular structure. As an example, the NN classifier 200 may be a 50-layer RESNET CNN architecture having multiple fully connected layers (e.g., RESNET50® or VGG®).

[0051] The NN classifier 200 is trained on image data including cellular structures for a classification task, and relevant phenotypic information of the cellular structures may be extracted from the output of one of the hidden or fully connected layers of the trained NN classifier 200. The NN classifier 200 may be pre-trained for a classification task using labeled image data related to cell types / structures. For example, the NN classifier 200 may be trained using a labeled training image dataset to classify images, for example, but not limited to, as cancerous or non-cancerous, cell / cellular structure identification, and / or other diseases affecting cellular function / structure.

[0052] Essentially, when training on a classification task, layers within the CNN architecture of NN classifier 200 begin to "recognize" cellular primitives, cell aggregations, how cells form tissue regions, etc., and use this to move toward recognizing larger cellular macrostructures that can form high-dimensional phenotypic representation embeddings of cellular structures present in input images of samples of cellular structures across all layers 204a-204n. The classification task or output 206 is used solely to train NN classifier 200. As another example, NN classifier 200 may be trained on labeled cellular image data from ImageNet and then fine-tuned using a limited number of training image data items related to cellular data and / or stained / fluorescent cellular images, such as from microscopy assays.

[0053] The NN classifier 200 architecture may be configured with multiple fully connected layers 204a-204n prior to the output classification 206. One of the fully connected layers 204n is selected from all of the full layers 204a-204n based on one determined to be focused on a deep representation of the cellular structure. Optionally, the toxicity prediction system 106 may perform automatic optimization of each of the ML models 106a-106c to determine which of the fully connected layers 204a-204n of the trained NN classifier 200 provides the best high-dimensional phenotypic output that highlights the toxic effect or toxicity in the cellular structure.

[0054] FIG. 3a illustrates an exemplary sample assay plate 300 having negative and positive control groups of sample wells 302 and 304 in which samples are imaged during an in vitro microscopy assay for use in training a deep learning (DL) toxicity model 106. Assay plate 300 also includes a test group of sample wells 306, in which samples are imaged during an in vitro microscopy assay for input into the trained DL toxicity model to predict the toxicity of any compound applied to the test group of sample wells 306. The negative control group of sample wells 302 includes samples that are not perturbed by the compound under test or that are perturbed by a non-toxic compound. For example, in an in vitro microscopy assay using liver spheroids, dimethyl sulfoxide (DMSO) can be applied to NC samples (e.g., 0.6% DMSO). The positive control group of sample wells 304 includes samples that are perturbed by a compound with known toxicity for the cellular structure used in the sample. For example, in an in vitro microscopy assay using liver spheroids, PC samples can be perturbed using chlorpromazine (CPZ) at a concentration that imparts a toxic effect (e.g., 400 µM).

[0055] The DL toxicity model 106 is configured to first extract phenotypic features of the cellular structure of each sample in the sample image and then be trained to estimate the phenotypic distance between the extracted phenotypic features and the phenotypic features associated with samples in the negative control group of sample wells 302. Microscopic images of the negative control samples and the positive control samples in the negative control group and positive control group of sample wells 302 and 304, respectively, are used in conjunction with unsupervised DL / ML techniques to train the DL toxicity model 106, which estimates the toxicity of the compound in each sample using the phenotypic distance between the extracted phenotypic features and the phenotypic features associated with samples in the negative control group of sample wells 302. For example, the negative control samples are used to construct an averagely deep phenotypic embedding of the negative control samples (e.g., the first 20 wells). Once the averagely deep phenotypic embedding of the negative control samples is constructed, it can be used as a basis for determining the distances of the lower-dimensional embeddings of subsequent test samples.

[0056] In this example, assay plate 300 has an arrangement of wells in row and column wells (e.g., columns A-P and rows 01-24) that are mapped to groups of negative control sample wells 302, positive control sample wells 304, and test sample wells 306. In this example, assay plate 300 has 16 columns and 24 rows of wells, or a total of 384 wells. While assay plate 300 is shown with the group of negative control wells 302 mapped to columns A-P and rows 1-8, the group of positive control wells 304 mapped to columns A-P and rows 20-24, and the group of test sample wells 306 mapped to columns A-P and rows 9-19, this is by way of example only, the invention is not so limited, and one of skill in the art should understand that any mapping can be defined to assign mappings between the sample wells of assay plate 300 and the groups of negative control sample wells 302, positive control sample wells 304, and test sample wells 306. The mapping of samples to wells in the assay plate 300 can be automatically defined by the software management system of the HTS in vitro microscopy assay system 102 depending on the number of samples needed in each group of negative control sample wells 302, positive control sample wells 304, and test sample wells 306. For example, samples in a test group of sample wells can have one or more compounds at various concentrations applied. The HTS in vitro microscopy assay system 102 can be programmed to map compound types and concentrations onto the assay plate 300, which depends on the number of compounds to be tested, the number of different concentrations, the number of replicates, etc.

[0057] 3b shows an exemplary unsupervised training process 310 for training a deep learning toxicity model including the ML low-dimensional (LD) embedding model 106b and the ML distance model 106c of the toxicity prediction system 106 of FIG. 1a. The training DL toxicity model uses the negative and positive control group sample wells 302 and 304 of the samples of FIG. 3a. The PK model process 310 includes the following steps:

[0058] In step 312, training process 310 receives negative control (NC) and positive control (PC) phenotypic representation embeddings associated with the NC and PC samples corresponding to the group of NC and PC sample wells 302 and 306 of plate 300. Images based on the NC and PC samples corresponding to the group of NC and PC sample wells 302 and 306 of plate 300 may be input to ML phenotypic feature extraction model 106a of FIG. 1a, which may be based on NN classifier 200 of FIG. 2 and has already been trained as described with reference to FIGS. 1a-2. With this in mind, ML phenotypic feature extraction model 106a outputs corresponding NC and PC phenotypic representation embeddings 107. The NC and PC phenotypic representation embeddings 107 are high-dimensional embeddings (e.g., 2024 elements per image).

[0059] In step 313, joint unsupervised training of the ML LD embedding model 106b and the ML distance / prediction model 106c / 106d is performed based on the following steps: In step 314, unsupervised training of the ML LD embedding model 106b is performed to estimate LD phenotype embeddings 108 that represent toxicity features associated with the NC and PC phenotype embeddings. The unsupervised training may be performed by iteratively optimizing the ML LD embedding model 106b over a set of parameter ranges associated with the ML technique (e.g., UMAP or t-SNE) used to construct the ML LD embedding model 106b. The criterion of the ML technique for training or generating the ML LD embedding model 106b is to find a subset of parameters that maximizes the difference between the NC and PC embeddings using input high-dimensional NC and PC phenotype embeddings over a set of parameter ranges. In each iteration, a subset of parameters is selected from the set of parameter ranges and applied to the ML technique (e.g., UMAP or t-SNE) to train or generate the ML LD embedding model 106b to output LD NC and PC phenotype embeddings.

[0060] As an example, the input high-dimensional NC and PC phenotypic representation embeddings 107 output from the ML phenotypic FE model 106a are high-dimensional vectors per image (e.g., each embedding may be 2024 elements per image). The ML phenotypic FE model 106a is not specifically trained for toxicity prediction, but rather is trained to identify the biological structures or phenotypic representations of cellular structures within each image of a sample. Thus, the resulting high-dimensional NC and PC phenotypic representation embeddings describe the phenotypic representation of each corresponding cellular structure as much as possible within the dimensionality (e.g., 2024). The UMAP algorithm is used to generate the ML LD embedding model 106b not only to reduce the dimensionality of the high-dimensional NC and PC phenotypic representations but also to ensure that phenotypic representations related to toxicity are preserved or converged within the resulting LD NC and PC embedding vectors 108 output from the ML LD embedding model 106b (e.g., each embedding may be 64 elements per image). This is performed by the UMAP algorithm, which finds the optimal set of parameters that maximizes the difference between the high-dimensional NC and PC phenotypic embeddings within a set of parameter ranges. The ML LD embedding model 106b is trained to find relevant biological structures associated with toxicity in each image in an unbiased manner, without knowing the cell type or its application. The ML LE embedding model 106b can represent the entire group of negative controls by taking a simple average profile of all negative controls using an LD embedding (e.g., 64 dimensions in the embedding). This can be used by the ML distance / prediction model 106c / 106d to determine a prediction of toxicity based on using a distance or similarity metric to estimate the distance or similarity between the LD embedding of test samples from the group of test sample wells 306 and the average LD NC embedding of NC samples from the group of NC sample wells 302.

[0061] In step 315, the ML LD embedding model 106b outputs NC and PC LD phenotype embeddings 108, each containing phenotypic information of the corresponding cell structure related to toxicity. The NC and PC LD phenotype embeddings are used as inputs to step 316 to train the ML distance model 106c.

[0062] In step 316, unsupervised training of the ML distance model 106c is performed for inference based on an appropriate distance or similarity measure relative to a high-dimensional vector metric (e.g., Wasserstein distance) to maximize the distance between the NC and PC LD phenotype embeddings. This means that the trained ML distance model 106c can determine the distance between the NC LD phenotype embedding and the LD phenotype embedding of the test samples 109 to determine the probability or indication of whether the compound associated with each test sample is toxic or not. Unsupervised training can be performed by iteratively optimizing the ML distance model 106c over a set of parameter ranges associated with the ML distance algorithm / technique (e.g., a Wasserstein distance algorithm such as Earth Mover's distance and / or the Sinkhorn distance algorithm) used to construct the ML distance model 106c. The criterion of the ML distance algorithm / technique for tuning or generating the ML distance model 106c is to find a subset of parameters of the ML distance algorithm that maximizes the distance between the set of NC LD embeddings and the set of PC embeddings, but minimizes the distance between embeddings within the set of NC LD embeddings, and minimizes the distance between embeddings within the set of PC LD embeddings. At each iteration, a subset of parameters is selected from a set of parameter ranges of the ML distance algorithm and applied to the ML distance algorithm (e.g., Earth Mover's distance algorithm or Sinkhorn algorithm), thereby tuning or generating the ML distance model 106c / 106d to output an indication of toxicity between the LD phenotype embeddings and the set of NC LD embeddings.

[0063] In step 317, it is determined whether the optimal distance between the set of NC and PC LD embeddings has been achieved for the set of parameter ranges for the ML distance model 106c. This may also include determining whether the difference between the NC LD and PC LD embeddings has been maximized based on the set of parameters tested so far. If this is the case, or if there are no more parameter combinations from the set of parameters for either the ML LD embedding model 106b or the ML distance model 106c that can be used, the process 310 proceeds to step 319. Otherwise, the process proceeds to step 318 for further parameter selection and tuning / training of either the ML LD embedding model 106b and / or the ML distance model 106c.

[0064] In step 318, further parameters may be selected from the set of parameters associated with the ML LD embedding model 106b for their further training, whereupon process 310 proceeds to step 314. Similarly, further parameters may be selected from the set of parameters associated with the ML distance model 106c for their further training, whereupon process 310 proceeds to step 316.

[0065] In step 319, a subset of parameters on the set of parameters associated with ML LD embedding model 106b that maximizes the difference between the NC and PC high-dimensional phenotype embeddings is selected for use by ML LD embedding model 106b in outputting LD phenotype embeddings for the high-dimensional phenotype embeddings corresponding to the test samples of the group of test sample wells 306. Similarly, a subset of parameters on the set of parameters associated with ML distance model 106c that maximizes the distance (e.g., Wasserstein distance) between the sets of NC and PC LD phenotype embeddings, but also minimizes the distance within each set of NC and PC LD phenotype embeddings, is selected for use by ML distance / prediction model 106c / 106d in outputting an estimate of toxicity based on the distance between the LD phenotype embeddings corresponding to the set of test samples and the NC phenotype embeddings or the set of average NC phenotype embeddings of the group of test sample wells 306.

[0066] FIG. 3c illustrates a toxicity prediction process 320 for predicting the toxicity of a compound using the trained deep learning toxicity model of FIG. 3b. From step 319 of the toxicity training process 310 of FIG. 3b, an ML LD embedding model 106b, an ML distance model 106c, and a prediction model 106d are configured based on the output selected parameter set 310. The DL toxicity model of FIG. 3b includes the ML LD embedding model 106b trained as described with reference to FIG. 3b and the ML distance model 106c trained as described with reference to FIG. 3b. The toxicity prediction system 106 includes the ML phenotypic model 106a as described with reference to FIGS. 1a-2 and the DL toxicity model of FIG. 3b, which are used to predict the toxicity of one or more compounds applied to multiple test samples of cellular structures in an in vitro microscopy assay. The multiple test samples in the group of sample test wells 306 of FIG. 3a can be captured by a microscope imager during an in vitro microscopy assay. During an in vitro microscopy assay, a set of one or more compounds can be applied to the test samples, i.e., one compound per test sample. These may be processed and / or enhanced using the imaging system 104 of Figure la or the quality control system and processes of Figures 5a-5f. The resulting images of the test sample may then be input into a toxicity prediction system 106 to predict the toxicity of compounds applied to the test sample. The toxicity prediction process 320 includes the following steps:

[0067] In step 321, a set of images related to one or more test samples having a compound applied thereto is received from a group of sample test wells 306 of an in vitro microscopy assay. The compound may have known or unknown toxicity to cellular structures within the corresponding test sample. Each image of the test sample includes image data sufficiently describing the cellular structures of the associated test sample having the compound applied thereto for automated processing and analysis. In step 322, each image of the set of images is input to the trained ML phenotypic feature extraction model 106a or 200 of FIG. 1a or 2. For each input image of the test sample, the trained ML phenotypic feature extraction model 106a or 200 outputs a high-dimensional phenotypic representation of the test sample's cellular structures present in the input image of the test sample. The output high-dimensional phenotypic representation of each test sample is then applied to the trained ML LD embedding model 106b. In step 323, each of the high-dimensional phenotypic representations of each test sample is input to the trained ML LD embedding model 106b, which outputs a low-dimensional phenotypic embedding for each test sample. In step 324, the LD phenotype embeddings of each test sample are passed to the trained ML distance model 106c.

[0068] In step 325, the ML distance model 106c receives each LD phenotype embedding 108 for each test sample and outputs a distance estimate 109 between the LD phenotype embedding for each test sample and the set of NC LD phenotype embeddings. In step 325a, the ML distance model 106c may be configured to output a distance or similarity estimate 109 between the LD phenotype embedding for each test sample and the average NC LD embedding for the set of NC LD phenotype embeddings. This may include comparing the distance between the LD phenotype embedding for the test sample and the average NC LD embedding for the set of NC LD embeddings in step 325b. From step 325, the ML distance / prediction model 106c / 106d may output an indication or probability associated with the distance of each of the LD phenotype embeddings to the NC LD embedding (i.e., it has been determined how far the LD phenotype embedding is from the non-toxic NC LD phenotype embedding). In step 326, for each test sample, an indication (eg, probability) of toxicity for the test sample and the applied compound is output based on the comparison of ML distance models 106c.

[0069] FIG. 4a is a schematic diagram showing another exemplary assay plate 400 having negative and positive control groups of samples 402 and 404 for use in training a deep learning (DL) toxicity model of toxicity prediction system 106a and models such as those described with reference to FIGS. 1a, 2, and 3a-3b, as well as a test group of samples 406 for input into the trained DL toxicity model of toxicity prediction system 106.

[0070] In this example, the HepaRG liver cell line is used to determine the toxicity of compounds in the liver. Each sample well of the sample plate 400 is filled with HepaRG cellular structures, and the in vitro microassay system 102 of FIG. 1a is configured to evaluate the cellular bile-reducing effects of compounds using a specific fluorescent substrate (e.g., carboxy-DCFDA (5-(and-6)-carboxy-2',7'-dichlorofluorescein diacetate) or CDFDA). CDFDA is a reagent that passively diffuses into cells. This is the fluorescent substrate used by the imager to capture the accumulation of CDFDA in bile ducts within images of each sample. This allows for the assessment of whether a compound induces cholestasis, which has been observed to occur when canary bile disappears from cellular structures within the images. However, the toxicity prediction system 106 performs additional unbiased processing to take into account other unobserved changes to cellular structures when determining the toxicity of a compound.

[0071] The sample with HepaRG cell structures in the negative control group, sample 402, has only a buffer applied (e.g., DMSO), which has no toxic effect. The sample with HepaRG cell structures in the positive control group, sample 404, has a reference compound (e.g., CPZ @ 60 micromolar) that is known to be toxic to liver cells and has a known toxic effect of triggering cholestasis. The sample with HepaRG cell structures in the test group, sample 406, has a variety of different compounds applied at different concentrations and in replicates. In this example, each compound may be represented at eight doses in assay plate 400, with each dose represented in three replicates in assay plate 400. This is to ensure there are sufficient replicates, so that after quality control is applied to the test sample images, there will then be at least one replicate for every compound and every dose with available images of the test sample for further downstream analysis.

[0072] Images of each sample well in the assay plate 400 are captured and evaluated for usability for further downstream analysis by the toxicity prediction system 106. This may be performed by the image system 104 of FIG. 1a and the image quality control system, and / or the processes of FIGS. 5a-5e. For example, a quality control model may be trained to automatically assess the usability of each well sample by classifying associated images of the test sample in the well. Each well sample is classified with a probability indicating whether the well sample is of good or poor quality. Higher probability values ​​indicate better quality, while lower probability values ​​indicate worse quality. A threshold probability may be used to determine higher-quality sample wells that are usable. In this example, a probability value of approximately 0.1 was determined to obtain usable sample images that can be passed to the toxicity prediction system 106 for toxicity training and / or toxicity prediction. For example, images of samples in wells 402a-402c, 404a, and 406a-406c represent samples in NC, PC, and test sample wells, indicating samples available for further downstream analysis. The darkest gray shaded areas, e.g., images of samples in wells 402d, 406b, and 404d, represent samples in the NC, PC, and test sample wells, indicating unusable samples that have too many artifacts for analysis and may be discarded. Therefore, the set of available sample images from each of the NC, PC, and test sample groups 402, 404, and 406 can be used for further downstream analysis. The set of available images from the NC sample group 402 and the PC sample group 404 are used to train DL toxicity models in an unsupervised manner, including training the ML LD embedding model 106b and the ML distance models 106c / 106d of FIG. 1a, as described with reference to FIGS. 1a-3b. The toxicity prediction system includes the ML phenotypic feature extraction model 106a of FIG. 1a and the DL toxicity models including the ML LD embedding model 106b and the ML distance models 106c / 106d.

[0073] FIG. 4b shows exemplary distance matrices for negative and positive control samples of a trained DL toxicity model trained based on the unsupervised training process of FIGS. 1a and 3a using available NC and PC samples from NC and PC sample groups 402 and 406 on assay plate 400 of FIG. 4a. Once the available images of the NC and PC samples are passed through the ML phenotypic feature extraction model 106a or 200 of FIGS. 1a-2, the output set of NC and PC high-dimensional phenotype embeddings is used as input to the UMAP algorithm to train and generate the ML DL embedding model 106b, as described with reference to FIG. 3b.

[0074] For example, a set of parameters defining the hyperparameters of the UMAP algorithm is iteratively optimized using a grid search, where different hyperparameter values ​​from the set of parameters are iteratively applied to determine the best combination of UMAP parameters that maximizes the difference between the set of NC and PC low-dimensional phenotype embeddings as much as possible. Given that the LD phenotype embeddings are still high-dimensional (e.g., 64 elements), Euclidean, Manhattan, and other standard distance metrics are not applicable, so Wasserstein distance is used instead.

[0075] As described with reference to Figures 1a and 3b, the output set of NC and PC LD phenotype embeddings from the ML LD embedding model 106b based on the UMAP algorithm is applied to the Sinkhorn algorithm to train and generate the ML distance model 106c, as described with reference to Figure 3b. The Wasserstein distance is optimized by iteratively performing a grid search over a set of parameters to find the hyperparameters of the Sinkhorn algorithm that maximize the estimated Wasserstein distance between the set of NC LD phenotype embeddings and the set of PC LD phenotype embeddings. The optimized hyperparameters output by the Sinkhorn algorithm can then be used by the ML distance model 106c on other test sample LD embeddings to enable comparison of distances between them and negative control LD embeddings.

[0076] The matrix distance plot in FIG. 4b is a mapping of distances when the toxicity prediction model 106 is calibrated to use available input sample images from the assay plate 400 in FIG. 4a. Columns 0-9 and rows 0-9 of the matrix plot represent the distances between the NC LD embeddings associated with the available image samples for the NC sample wells 402 in FIG. 4a. Columns 10-19 and rows 10-19 of the matrix plot represent the distances between the PC LD embeddings associated with the available image samples for the PC sample wells 404 in FIG. 4a. It is clear that the DL toxicity models, i.e., the ML LD embedding model 106b and the ML distance model 106c / d, have been trained because the distances between the NC LD embeddings have a dark gray shaded region 412 indicating the minimum distance between them. Similarly, the PC LD embeddings also have a dark gray shaded region 414 indicating the minimum distance between them. Similarly, the light gray shaded region 416 indicates that the distance between the set of NC LD embeddings and the set of PC LD embeddings is maximized (as much as possible) in this example. Considering that the phenotypic distance is maximized in region 416, it indicates that the corresponding PC sample has a toxic effect on the HepaRG cell line sample, which clearly indicates that the DL toxicity model of the toxicity prediction system 106 is calibrated and can be used to test the toxicity of various compounds on the HepaRG cell line.

[0077] FIG. 4c shows another exemplary distance matrix 420 for negative and positive control samples, and a test sample for predicting the toxicity of compounds in the test sample using the trained DL toxicity models 106b-106d of FIG. 4b of the toxicity prediction system 106. In addition to the available NC and PC sample images output from the in vitro microassay represented by the assay plate 400 of FIG. 4a, multiple available images of the test sample are also processed to predict the toxicity of compounds applied to the test sample. In matrix 420, columns 0-95 and 288-311 and rows 0-95 and 288-311 represent the available images of the NC sample used from plate 400, columns 312-335 and rows 312-335 represent the available images of the PC sample used from plate 400 to train the DL toxicity models 106b-106d of FIG. 4b, and columns 96-287 represent the available images of the test sample used from plate 400 for testing. As can be seen, the dark gray area 422 in columns 0-95 and rows 0-95 of the distance matrix represents the distance between the NC LD embeddings, which have the smallest distance between them. There are some false positives, which give a lighter shade of gray, that do not affect the performance of toxicity prediction. Similarly, the dark gray area 424 in columns 312-335 and columns 312-335 of the distance matrix 420 represents the distance between the PC LD embeddings, which also have the smallest distance between them. Similarly, the light gray shaded area 426 of the distance matrix does not indicate the smallest distance from the NC LD embedding, but rather indicates the distance between the set of NC LD embeddings and the set of test sample LD embeddings, which shows a larger distance than would indicate a higher toxic effect on the HepaRG cell line samples used in the majority of test samples.

[0078] Figure 4d shows an example of a conventional toxicity prediction method used on 14 compounds (e.g., compounds A, B, C, D, E, F, G, H, I, J, K, L, M, and N) applied to samples of HepaRG cellular structures. In this example, the assay plate had 14 compounds, 8 doses, with three replicates for each dose. Eight compounds were known to have toxic effects on HepaRG cellular structures and thus the liver. These compounds were tested using in vitro microscopy hardware and commercially available software accompanying conventional toxicity analysis. This conventional toxicity analysis is based on the use of standard image analysis tools to analyze the data (e.g., step 1), while standard toxicity characterization is based on the researcher's analysis of vesicle counts (e.g., step 2) and dose-response (EC50) graphs (e.g., step 3). In stage 4, eight compounds (e.g., compounds A, E, F, I, J, K, M, and N) were found to have toxic effects, while the conventional toxicity analysis method determined six toxic compounds (e.g., compounds B, C, D, G, H, and L) to have no toxic effects. However, it is clear that the conventional toxicity analysis workflow missed compounds with toxic effects due to the six compounds that could not be detected. When these compounds were applied to the toxicity prediction system 106 with the DL toxicity models 106b / 106c trained as described with reference to Figures 4a-4c, all of the compounds were found to have toxic effects. Figure 4e is a schematic diagram showing an example of conventional toxicity prediction results for compounds A, B, C, D, E, F, G, and I using a conventional approach to predict toxic compounds. The bile vesicle and cell counts for the positive control compound, i.e., chlorpromazine, are shown in dashed boxes, and the bile vesicle and cell counts for the negative control compound, i.e., DMSO, are shown by solid boxes. As can be seen, when considering only bile vesicle and cell numbers, it is very difficult to predict whether compounds B, C, D, and G have toxic effects. Figure 4F is a schematic diagram showing an example of toxicity prediction results of the trained DL toxicity model for compounds A, B, C, D, E, F, G, and I using the trained DL toxicity model 106b / 106c.The schematic in Figure 4f is a graphical representation of a novel deep learning-based phenotypic distance metric (y-axis) extracted using a cell-based imaging classification framework (e.g., Resent50). The phenotypic distance metric for the positive control compound, i.e., chlorpromazine (dashed box), and the negative control compound, i.e., DMSO (solid box), is shown. The phenotypic distance metric compares all compounds to the negative control compound (mean phenotypic phenotype), including the tested compounds (e.g., compounds A, B, C, D, E, F, G, and I) and the mean negative DMSO phenotype profile. As can be seen, the distance between the compound phenotype and the reference negative profile (mean DMSO) is higher for each tested compound than the 75th percentile of the DMSO distribution. This difference, in addition to the similarity to the positive control compound (chlorpromazine), can qualify the phenotypic effect as toxic (though the difference from DMSO alone does not qualify as toxic). As can be seen, using the trained DL toxicity models 106b / 106c, it is easier to predict that compounds A, B, C, D, E, F, G, and I are toxic, and the results can be used to more effectively predict the toxicity of compounds compared to traditional toxicity prediction methods.

[0079] FIG. 5a illustrates an example of a quality control process 500 for use in the image analysis and feature extraction system 104 of the toxicology pipeline 100 of FIG. 1a to automatically identify available, analyzable images among images of samples 103 captured from the HTS in vitro microscopy system 102 of FIG. 1a. The QC process 500 is used to determine which samples or groups of samples from a set of wells on an assay plate are analyzable and which are not. The QC process 500 uses ML techniques to assist in determining whether image captures of wells can be retained or discarded for use in downstream analysis. The set of available images is analyzable in that the cellular structure is well-defined for downstream image analysis processing, such as, but not limited to, extracting phenotypic feature representations from the cellular structure of the images for use in training and / or as input to the DL / ML models 106b-106c of the toxicity prediction system 106, as described with reference to FIGS. 1a-4e. For example, the set of available images 105 output from the QC process 500 improves the robustness and reliability of the trained DL models 106a-106c, thereby further improving the robustness and accuracy of the toxicity predictions output by the toxicity prediction system 106 for test samples having compounds applied for toxicity prediction.

[0080] While the set of available images 105 output by the QC process 500 is described with reference to a toxicity prediction system 106, this is by way of example only, and one skilled in the art should understand that the set of available images 105 output by the QC process 500 may be input to any other downstream process related to analyzing, classifying, and / or predicting / estimating one or more aspects or properties of said images depending on the type of HTS in vitro microscopy assay being performed. For example, an HTS in vitro microscopy assay may be used in a drug / compound research program for use in determining the properties or effects of one or more compounds on a cytoarchitectural sample in an in vitro microscopy assay (such as, but not limited to, the toxicity of a compound applied to a cytoarchitectural sample, the effectiveness of the compound's effect on a cytoarchitectural sample, and / or any other type of property of a compound when applied to a cytoarchitectural sample in an in vitro microscopy assay).

[0081] In this example, the QC process 500 identifies a first set of available images of the sample from images 103 captured from the HTS in vitro microscopy system 102 and pre-processes the first set of available images into a final set of available images 105 for input to, for example, but not limited to, a toxicity prediction system 106 and / or other compound analysis systems or downstream workflow processes / analysis systems. With reference to the toxicity prediction system 106, the QC process 500 outputs the set of available images of the sample 105 for use in training the DL toxicity models 106a / 106b / 106c of the toxicity prediction system 106 of FIG. 1a and / or as input to the trained DL models 106a / 106b / 106c of the toxicity prediction system 106, as described with reference to FIGS. 1a-4e. The toxicity prediction system 106 receives a set of available images associated with the plurality of samples, at least one group of which has a compound applied for testing and / or serves as a positive control group for training the DL models 106b-106c of the toxicity prediction system 106.

[0082] Upon receiving a set of images captured from multiple samples in an assay plate (e.g., assay plate 102c, 300, or 400 in Figures 1a, 3a, or 4a), QC process 500 identifies available samples of cellular structures from the in vitro microscopy assay for further downstream analysis. QC process 500 includes the following steps:

[0083] In step 502, a first set of sample images useful for analysis is automatically identified from the received set of sample images captured from multiple samples in the assay plate. For example, the received set of sample images is automatically analyzed to determine whether spheroid / cellular structure features of the sample are present and whether these features highlight possible artifacts on the image, or to determine out-of-focus areas and / or other imaging defects. Artifacts may bias predictions for downstream analysis, such as using ML models. For example, the toxicity prediction system 106 uses the received images of the sample 105 to predict whether a compound is toxic or non-toxic; therefore, any artifacts may mask the effect of the compound applied to the cellular structure and thus be detrimental to toxicity training and / or prediction. For example, these artifacts may mask the dose at which the compound is actually toxic (e.g., the dose with a 50% toxic effect).

[0084] The automatic identification uses one or more ML models to estimate and predict whether the sample in each well is analyzable. For example, automatically identifying a first set of sample images may include inputting a received set of sample images from an assay into one or more ML models configured to identify one or more regions of each image in the set that are likely to have cellular structures therein. The set of images having identified regions may be input into one or more additional ML models to identify whether cellular structures in the identified regions of each image are suitable for further downstream analysis. Those images from the received set of images having regions containing cellular structures determined to be analyzable are selected to form a first set of sample images available for further downstream analysis.

[0085] In step 504, for each image of a sample in the first set of images of the sample, generating a set of two-dimensional (2D) images of the sample includes capturing multiple 2D image slices of the sample in an assay plate, where multiple 2D image slices are taken along a z-axis of the sample. The set of 2D images for each sample in the first set of images of the sample are taken at different z-axis positions such that a 3D representation of each sample is formed. The multiple 2D images for each sample form a 3D representation of each well of each sample, which may be further processed and represented as a fused / compressed 2D representation.

[0086] In step 506, a set of available samples is identified from the set of 2D image slices. For example, for each sample in the first set of images of the samples, the set of 2D image slices may be fused, compressed, or combined to form a 2D representation of the sample's 3D structure, which enhances the cellular structure within the sample. The cellular structure of each fused 2D representation may be further analyzed to identify whether it is sufficiently distinct or well-defined for use in downstream analysis (e.g., for input into the toxicity prediction system 106). For example, a fused 2D representation is sufficiently distinct or well-defined when the quality of the image is determined to be adequate for extracting one or more metrics or phenotypic representations of the cellular structure from the image. Those samples from the first set of images of the samples having sets of 2D image slices identified as sufficiently distinct or suitable for further analysis form the set of samples available for analysis.

[0087] In step 508, data representing the set of samples available for analysis is output as a set of images 105. For example, each fused 2D representation of the set of available samples may form a set of images 105 of FIG. 1 a that may be input to a downstream process for analysis, such as a toxicity prediction system 106.

[0088] FIG. 5b illustrates an example of a first quality control (QC) process 510 for use in step 502 of the QC process 500 of FIG. 5a to select a first set of available images of the sample suitable for further downstream analysis. The first QC process 510 is configured to automatically identify a first set of images of the sample available for analysis, e.g., for input into the toxicity prediction system 106, e.g., as described with reference to FIGS. 1a-5a. The first QC process 510 may include the following steps: In step 512, preprocessing each image of the sample from the received set of images of the sample 103 from the HTS in vitro assay system 102 to form a set of preprocessed images of the sample. Preprocessing may include performing image analysis and / or processing based on, for example, but not limited to, illumination correction, illumination field correction, artifact reduction / interpolation, or any other image processing algorithm to process the images to further enhance cellular structures that may be contained within the images. In step 514, the set of preprocessed images of the sample is input to a machine learning (ML) region-of-interest (ROI) model trained and configured to identify, for each input image of the sample, one or more regions of interest in the input image of the sample, where the regions of interest include cellular structures of the corresponding sample. The ML region-of-interest (ROI) model may be based on any type of neural network (NN) structure, such as, but not limited to, a feedforward neural network (FNN), a recurrent neural network (RNN), an artificial neural network (ANN), a convolutional neural network (CNN), any other type of NN suitable for identifying regions of interest in images related to cellular structures, or a combination thereof. For example, the ML ROI model may be based on a CNN trained to identify ROIs related to cellular structures in input images and / or output images that focus on identified ROIs related to cellular structures in the input images. The ML ROI model may be trained using a CNN structure and a labeled or annotated training image dataset, where each image in the training image dataset is labeled or annotated with information related to whether cellular structures are present and, if so, the region of interest in the image.

[0089] At this stage, those images in the input set of sample images for which the ML ROI model does not detect a region of interest may be discarded from the set of sample images, as they are more likely to be unanalyzable. Alternatively, for each image for which the ML ROI model does not detect a region of interest, the region of interest may then be defaulted to the entire input image for further analysis in step 516. That is, the region of interest may be identified for the entire input image, which may be input to step 516.

[0090] In step 516, each image having an identified region of interest containing the sample's cellular structure is input to an ML image availability model trained and configured to classify whether the cellular structure in the input image is analyzable. Thus, each input image can be classified with a label or probability value representing the availability of the input image as analyzable. If the label or probability value indicates that the input image is analyzable (e.g., the probability value may be greater than or equal to a predetermined availability probability threshold, or the label indicates that the image is available), the input image is placed in a first set of images of the sample that are considered available. The remaining input images are discarded with a label indicating that the image is not available or with a probability value lower than the predetermined availability probability threshold.

[0091] The image-enabled model may be based on, but is not limited to, any ML algorithm or technique that can be used to classify whether a cellular structure within the ROI in the image is analyzable. This may be based on identifying, from each input image, whether phenotypic features associated with the cellular structure (e.g., spheroids, nuclei, vesicles, etc.) are present in the input image, thereby evaluating whether sufficient phenotypic features are present in the cellular structure of the sample captured in the image, thereby enabling further analysis of the cellular structure of the sample in the image. For example, the ML algorithm used to train the image-enabled model may be based on, but is not limited to, a support vector machine (SVM) or a neural network classifier / structure. As an example, the image-enabled model may be based on a set of SVMs, each associated with the classification of a specific phenotypic feature or a cluster of phenotypic features of the cellular structure of the sample captured in the input image, with each SVM outputting a positive or negative classification indicating whether the specific phenotypic feature is present. If enough SVMs output a positive classification indicating the presence of the corresponding specific phenotypic feature, the input image may be considered an image usable for analysis.

[0092] Step 518 outputs a first set of images of samples, including data representing those images of samples classified as analyzable in step 516. The output of the first set of images of samples may be further processed, such as in steps 504-508 of QC process 500, for example, but not limited to, to further enhance cellular structures contained in each image and / or for further selection regarding the usability of the first set of images of samples. Alternatively or optionally, the first set of images of samples deemed usable output from first QC process 510 may be output from image analysis system 104 as a set of images 105 for input to toxicity prediction system 106 or other downstream processes, such as analysis.

[0093] FIG. 5c shows an example of an image analyzer 520 for performing steps 512 and 514 of the first QC process 510 of FIG. 5b, in which each image of the sample is preprocessed and a region of interest containing cellular structures is identified within each image of the sample. In this example, each image in the set of images 103 of the sample output from the HTS in vitro microscopy assay system 102 is preprocessed by an image preprocessing unit 522. In this example, each input image 522a to the image preprocessing unit 522 has an illumination function 522b applied to it, and the resulting illumination-corrected image 522c output from the first image preprocessing unit 522 is passed to an image ROI unit 524. The illumination function 522b may be configured to compensate for uneven fluorescent illumination due to the image capture device within the HTS in vitro microscopy system 102. While image pre-processing unit 522 performs illumination field correction on each image, this is by way of example only, and the present invention is not so limited, and it should be understood by those skilled in the art that image pre-processing unit 522 may perform one or more other image processing functions, such as image focusing, illumination field correction, sharpening, saturation, artifact reduction, correction and / or interpolation, any other image processing algorithm, combination thereof, modification thereof, etc., which further enhances the cellular structure of the sample that captured each input image for use by image ROI unit 524.

[0094] Each of the set of preprocessed images of the sample output from the image preprocessing unit 522 is input to the image ROI unit 524. The image ROI unit 524 is configured to include an ML region of interest (ROI) model based on a CNN architecture (e.g., a CNN-based (CellPose®) segmentation) that is trained and configured to identify each segment of the image, where the segments containing cellular structures form the ROI of the image. CNN-based segmentation (e.g., CellPose) architectures can be used on microscopy images for segmentation of cell bodies, membranes, and nuclei. For example, CellPose is a deep learning CNN architecture trained on a dataset of highly diverse images of cells containing over 70,000 segmented objects. These can be retrained and / or fine-tuned on a training dataset of images focused on cellular structures from microscopy assay images expected to be used in downstream analysis. While a CNN-based CellPose segmentation architecture has been described, this is by way of example only, and the present invention is not so limited. Those skilled in the art should understand that any other suitable ML algorithm / model may be used to identify a ROI containing cellular structures relative to an input image of a sample. The ML ROI model may be trained using a CNN architecture that uses a labeled training dataset including multiple images, each of which is labeled or annotated with data representing whether a cellular region of interest is present and / or the location of the region of interest within the image. For example, the training image dataset may include images labeled or annotated with information related to whether cellular structures (e.g., phenotypic features of cellular and / or macrocellular structures, cellular spheroids, cell nuclei, cellular mutations, perturbations, cellular cholestasis, associated biological features, etc.) are present, and, if cellular structures, perturbed cellular structures, or residues of cellular structures are present, information related to the region of interest containing cellular structures within the image.Thus, the ML ROI model is trained and configured to identify / predict segments / ROIs of an input image that contain remaining / mutated / perturbed cellular structures within the sample image and output this information for use in downstream processing.

[0095] For example, the input image of sample 524a may include cellular structures, such as cellular spheroid objects containing nuclei, where only those segments of the image containing nuclei are identified to form the image region of interest. In another example, the input image of sample 524b may include cellular spheroid objects representing cellular structures undergoing cholestasis, i.e., the cellular structures have been perturbed / mutated by a compound, such as, but not limited to, missing nuclei. In such cases, the ML ROI model is trained to identify image segments / ROIs containing the remaining / mutated / perturbed cellular structures in the sample image. The image ROI unit 524 may be configured to output only those portions of image 524a that contain the ROI. Alternatively or additionally, the image ROI unit 524 may annotate the input image 524a with data representing the ROI for use by other processing units / downstream processes in focusing on the cellular structures contained in the image ROI. In either case, the image ROI unit 524 outputs a set of image data representing each of the set of preprocessed input images of the sample, and for each image in the set, outputs an ROI containing the cellular structure of the sample in the respective input image. The output image and ROI data of the image analyzer 520 may be used in steps 516 and 518 of the first QC process 500, which may be implemented as an image usability system configured to extract features within the ROI of each image of the sample and determine whether each image is of good or bad quality, i.e., available or unavailable for further analysis, or whether each image is analyzable or unanalyzable. The output of the image usability system includes a set of images of the sample that are analyzable / available, which may form a set of images 105 that may be input to further downstream processes, such as, but not limited to, the toxicity prediction system 106.

[0096] FIG. 5d illustrates an example of an image availability system 530 as part of a QC pipeline for implementing steps 516 and 518 of the first QC process 510 of FIG. 5b for use in classifying whether cellular structures are analyzable (e.g., available for further downstream analysis / processing) in the ROI of each input image of the sample. The image availability system 530 may include the image analyzer 520 of FIG. 5c, which receives a set of microscopic images of the sample 103 output from the HTS microscopy assay system 102 of FIG. 1a and outputs, for each image in the set of image 103 data, a ROI that includes cellular structure features for each image. The set of images and corresponding ROIs are input to a phenotypic sampler 532, which identifies and / or classifies, for each image in the set of images, whether the ROI of each image has n phenotypic features (e.g., P1, P2, P3, ..., Pn). Clustering is performed on each of the n phenotypic features detected in the ROI of each image of the sample. Each of the n phenotypic features has one class SVM 536a-536n trained to classify an image ROI into one of the n phenotypic feature clusters. Each of the SVMs 536a-536n outputs a classification of whether the image ROI has the corresponding phenotypic feature P1, P2, ..., Pn, respectively. Each cluster of n phenotypic features P1, P2, P3, ..., Pn is input to a corresponding SVM 536a-536n, each of which outputs a probability or likelihood that the corresponding phenotypic feature is within the ROI of the image of the sample. These n output probabilities are combined to provide an output classification value 538 indicating whether the image is analyzable (e.g., usable). Those images whose classification value 538 indicates analyzable or usable images (e.g., classification value 538 greater than an usability threshold) are selected to form a first set of usable images of the sample.

[0097] FIG. 5e illustrates an example of a second quality control (QC) process 540 for use in steps 504-508 of QC process 500 to augment and select a final set of available images of a sample for downstream analysis, such as, but not limited to, the input set of images 105 to toxicity prediction system 106 of FIG. 1a. Once a first set of available images of a sample is output from step 502 of QC process 500 of FIG. 5a or from image availability system 530 of FIG. 5c, step 504 of QC process 500 can then be executed, in which, for each image of the sample from the first set of available images, a 3D representation of each well sample corresponding to the respective image of the sample is captured by a microscopic imaging device. The 3D representation of each image of the sample is generated by a microscopic imaging device (e.g., ImageXprs®) capturing a set of two-dimensional (2D) images of the sample, the set of 2D images of the sample comprising capturing multiple 2D image slices of the sample within a well sample within an assay plate. Multiple 2D image slices are taken at different z-focal points or at different foci along the longitudinal z-axis of the sample well. A set of 2D images for each sample in the first set of available images is taken at different z-axis positions to form a 3D representation of each sample. The set of 2D images for each sample forms a 3D representation of each well corresponding to the image of the sample from the set of available images. Each set of 2D images for each sample is enhanced to highlight cellular structure in each 2D image by fusing or compressing / combining the sets of 2D images into a single 2D image representation of the image of the sample. A second QC process 540 for enhancing and selecting a final set of available images of a sample for downstream analysis includes the following steps:

[0098] In step 542, for each set of 2D images of the sample from the set of available images of the sample, foreground, background, and multiple uncertain feature regions or classes of cellular structures in each of the 2D image slices from the set of 2D images are identified. The multiple uncertain feature regions or classes include multiple uncertain foreground feature regions or classes and multiple uncertain background feature regions or classes. For example, for each image in the set of 2D images, each x / y pixel value is separated into multiple classes, including, but not limited to, a foreground class, a background class, and multiple uncertain foreground classes and multiple uncertain background classes.

[0099] In step 544, for each set of 2D images of the sample, the identified foreground, background, and multiple uncertain feature regions or classes of the 2D image slices are iteratively combined within the set of 2D images of the sample to generate a single 2D image of the cellular structure of the sample. For example, the iterative process attempts to optimize the projection between the two classes by taking each foreground class, blending it with the uncertain foreground class, and smoothing this projection in local regions. The iterative optimization process may have two criteria: the final projection must be locally smooth with respect to Z-projection in local regions around the foreground class, and the local intensity in the local region is the maximum intensity possible. The iterative combination may be based on an ML smoothing model that iteratively smooths the foreground and uncertain foreground classes.

[0100] In step 546, for each single 2D image of the sample, an iteratively generated single 2D image of the sample is selected for the final set of usable images based on the quality of the single 2D image of the sample. For example, the quality of the single 2D image sample may be evaluated based on whether there are any sharp deviations between adjacent foreground feature regions and adjacent undetermined foreground regions, etc., or between adjacent background feature regions and adjacent undetermined background regions, etc. For example, if, at the end of the iterative procedure, there are too many "jumps" between adjacent foreground regions / classes and adjacent undetermined foreground regions / classes, or if the resulting single 2D image still does not meet the criteria for image intensity of local regions of the z-projection, or if the iterative procedure did not converge within an appropriate error threshold, the single 2D image is not usable and may be selected for discarding. The remaining single 2D images of each sample may be used to form the final set of usable images.

[0101] Step 548 outputs a final set of available images, including image data representing a single 2D image for each sample selected in step 546. The final set of available images may, for example, but not limited to, form the set of images 105 that is input to a downstream process.

[0102] 5f illustrates another example of a smooth manifold extraction (SME) system 550 for implementing steps 504-508 of the QC process 500 of FIG. 5a and / or steps 542-548 of the second QC process 540 of FIG. 5e for use in further enhancing and selecting a final set of available images 567a-567p for output from the image analysis system 104. The SME system 550 includes a chain of processing units, including a 3D image representation unit 552, a profiling unit 554, an FFT unit 556, pixel classification units 558-560 for foreground / background classification / clustering, an iterative SME unit 562, a final image selection unit 564, and an output unit 566. The output unit 566 outputs the final set of available images 567a-567p, which may be used as input as a set of images 105 for further downstream analysis processes, such as, but not limited to, input to a toxicity prediction system 106.

[0103] 5a, the 3D image representation unit 552 is configured to perform step 504, whereby the 3D image representation unit 552 receives a first set of available images of the sample output from step 502 of the QC process 500 of FIG. 5a or from the image availability system 530 of FIG. 5c. The 3D image representation unit 552 captures a 3D representation of the sample in each sample well 551 corresponding to the sample in each available image of the sample. For example, the microscope imaging device 553 (e.g., ImageXpress®) is instructed to capture a set of two-dimensional (2D) images 555 corresponding to the sample wells 551 in the available images of the sample. The set of 2D images 555 of the sample includes a plurality of 2D image slices 555a-555k of the sample from the corresponding sample wells 551 of the assay plate. The sample wells 551 have a longitudinal z-axis 551a, an x-axis 551b, and a y-axis 551c, where the x-axis 551b and the y-axis 551c are orthogonal to each other and to the longitudinal z-axis. 2D image slices 555a-555k are taken at different Z focal points, or at different focal points along the longitudinal z-axis 551a of the sample wells 551. A set of 2D images 555 for each available image of a sample are taken at different z-axis positions such that a 3D representation of each sample is formed. The set of 2D images 555 for each sample forms a 3D representation of each well 551 corresponding to the available image of the sample from the set of received available images.

[0104] The remaining processing units 554-566 of the SME system 550 are configured to process each set of 2D images 555 for each sample to enhance / highlight the cellular structure of the sample represented in each 2D image. This is done by fusing or compressing / combining each set of 2D images 555 for each sample into a single 2D representation 565 of said sample. Essentially, these remaining processing units 554-566 are configured to find the Z focus of each pixel in each set of 2D images 555 for each sample that corresponds to the associated structure of the cell / cellular structure of said each pixel.

[0105] In the profiling unit 554, a profile is extracted for each 2D image slice 555a in the set of 2D images 555, where any (x, y) position corresponds to a profile of focus values ​​in the Z direction 551 of the sample well 551, which is composed of direct intensity values ​​in the case of confocal images, or SML values ​​in the case of wide-field epifluorescence images. Each of the 2D image slices 555a-555k (e.g., profiles) contains low-frequency components, passing through some foreground signal. The profile of each of the 2D image slices 555a-55k is passed to the FFT unit 556. The FFT unit 556 performs a fast Fourier transform (FFT) on each profile of the 2D image slices in the set of 2D images 555 of the sample to determine the power frequency spectrum of each profile of the 2D image slices 555a-555k for each set of 2D images of the sample. The FFT profiles of the 2D image slices 555a-555k are passed to pixel classification units 558-560 for foreground / background classification and clustering.

[0106] The pixel classification units 558-560 perform multi-class classification on the FFT profiles of the 2D image slices 555a-555k using an ML classification algorithm / model to classify each (x,y) location or pixel in each profile of the 2D image slice into one of a plurality of labels associated with a foreground class 559a, a background class 559b, and / or a plurality of undetermined foreground / background classes 559c-559l. The ML classification algorithm may be based on, for example, but not limited to, a multi-class k-means algorithm, a multi-class UMAP algorithm, and / or any other suitable multi-class classification algorithm or model capable of classifying the (x,y) location of each pixel in each of the FFT profiles of the 2D image slices 555a-555k of the set of 2D images 555 into a foreground class 559a, a background class 559b, or a label associated with a plurality of foreground / background classes 559c-559l. The pixel classification unit 556 outputs a labeled / annotated FFT profile 561 representing the 2D image slices 555a-555k to the SME unit 562, where each (x,y) pixel in each 2D image slice is labeled with a classification associated with a foreground class 559a, a background class 559b, or one of multiple undetermined foreground / background classes 559c-559l.

[0107] The SME unit 562 is based on an ML SME model / algorithm 563 (or iterative SME optimization process) that iteratively processes data representing the set of 2D images for each sample based on the determined foreground, background, and uncertain foreground and background classes assigned to pixels of 2D image slices in the set of 2D images for each sample. The ML SME model 563 uses an iterative SME algorithm. The ML SME model 563 is configured to perform an iterative SME optimization procedure using a cost function that balances local smoothness and proximity to a maximum of focus values, which is minimized to combine data representing the 2D image slices of the sample to obtain a final smoothed index map 563m for the sample. Note that the first index map 563a is highly discontinuous at the start of the iterative SME optimization procedure, but should be smoothed as the iterations of the SME optimization procedure converge to the final smoothed index map 563m while preserving fine foreground details. The final smoothed index map 565m is sent to the extraction and selection unit 564.

[0108] For example, the SME algorithm 563 of the SME unit 562 examines foreground pixels mixed with uncertain foreground classes, optimizes the projections of these types of foreground classes in the 2D image slices 555a-555k, and attempts to fuse / compress them into a final single 2D image 565a (or final index map 563m) using an iterative SME optimization procedure. The iterative SME optimization procedure smooths this projection in each iteration from the first iteration (e.g., iteration 001) to the final iteration (e.g., iteration 571) when a final error threshold is reached / met and / or a maximum number of iterations is reached / met. The iterative SME optimization procedure has two criteria to satisfy: the first criterion is that the final projection (e.g., final index map 563m or final single 2D image) needs to be locally smooth with respect to the Z projection, and the second criterion is that the local intensity is the maximum intensity possible (e.g., using a Gaussian energy function). These criteria are used in each iteration from the first to the final iteration, as the 2D image slices are fused / compressed together to form a final single 2D image. Once the iterative SME optimization procedure of the SME algorithm 563 converges to a final index map 563m and / or the maximum number of iterations is met, the resulting final index map 563m is used to form the final single 2D image 565a. The resulting final index map 563m or the final single 2D image 565a is passed to the extraction and selection unit 564 to determine whether the final single 2D image 565a should be included in the final set of available images.

[0109] In an extraction and selection unit 564, the final smoothed index map 563m output from the iterative SME algorithm 563 is analyzed, and multiple voxels corresponding to this index map 563m are extracted from the original stack to generate a final single 2D image 565a. The extraction and selection unit 564 using a selection unit 565b selects whether to include the final single 2D image 565a in the final set of sample available images for sending to an output unit 566. The selection unit 565b determines whether the final single 2D image 565a or the index map 563m corresponding to the final single 2D image 565a is of sufficient quality or sufficient smoothness for output from the extraction and selection unit 564. For example, the selection unit 565b determines whether two criteria are satisfied by the iterative SME algorithm 563 or the final single 2D image 565a, the first criterion being based on determining whether the projection of the final single 2D image 565a or the final index map 563m of the iterative SME algorithm 563 is locally smooth with respect to the Z projection, and the second criterion being based on whether the local intensity is the maximum intensity possible.

[0110] For example, a first criterion is based on examining the smoothness of each local region of pixels in the final single 2D image 565a (or final index map 563m). In this case, for each local region of pixels in the final single 2D image 565a (or final index map 563m), for example and without limitation, for each 5x5 pixel neighborhood in the final 2D image 565a (or final index map 563m), the Z level in each local region (e.g., a 5x5 pixel neighborhood) should be smooth with respect to Z level, and there should be no deviations or "jumps" in intensity (e.g., not a direct move from intensity level 1 to 17 when moving from one pixel to another within the local region). A second criterion is based on the intensity of the Z protrusion of each local region of pixels in the final single 2D image 565a (or final index map 563m). This second criterion is met when, within each local region (e.g., a 5x5 pixel neighborhood), the Z projection captured over the local region (e.g., a 5x5 pixel neighborhood) has an intensity value based on the local region (e.g., a 5x5 pixel neighborhood) that is the maximum Z intensity. If these two main criteria are not met in the selection unit 565b, the selection unit 565b discards the final single 2D image 565a (or final index map 563m), thereby not forming part of the final set of available images 567. Otherwise, the final single 2D image 565a of the sample (or final index map 563m) is output by the extraction and selection unit 564 for transmission to the output unit 566, and the output final single 2D image 565a is added as the final available image 567a of the final set of available images 567.

[0111] While the above first and second criteria are described herein, this is by way of example only, and one skilled in the art will appreciate that modifications to these two criteria and / or other criteria may be applied for use in selecting / discarding the final single 2D image 565a when sending to the output unit 566. For example, these modifications and / or other criteria may be based on, but are not limited to, the first criterion, which may be further modified to count the number of "jumps" in intensity deviations for use in determining whether to discard / select the final single 2D image 565a (e.g., the final single 2D image 565a may be selected for output if the number of "jumps" or intensity deviations is below a minimum "jump" threshold / count, otherwise it may be discarded), and / or the selection unit 565b, which may further consider whether the iterative SME algorithm has converged within a predetermined error threshold (e.g., if the convergence error of the iterative SME algorithm is less than or equal to a predetermined error threshold, the final single 2D image 565a may be selected for output, otherwise it may be discarded), and / or any other criteria for assessing whether the final single 2D image 565a of each sample is of sufficient quality and / or can be analyzed by further downstream processes, etc.

[0112] The output unit 566 stores each of the output final single 2D images 565 a selected for output by the extraction / selection unit 564 as a plurality of images forming a final set of usable images of the sample. The output unit 566 may then transmit the final set of usable images of the sample for further downstream processing, such as, but not limited to, as an input set of images 105 to the toxicity prediction system 106.

[0113] 6a is a schematic diagram of a system / apparatus for performing the methods described herein. The illustrated system / apparatus is an example of a computing device. Those skilled in the art will recognize that other types of computing devices / systems, such as, for example, distributed computing systems, may alternatively be used to perform the methods described herein.

[0114] The device (or system) 600 includes one or more processors 602. The one or more processors control the operation of the other components of the system / device 600. The one or more processors 602 may include, for example, a general-purpose processor. The one or more processors 602 may be single-core or multi-core devices. The one or more processors 602 may include a central processing unit (CPU) or a graphical processing unit (GPU). Alternatively, the one or more processors 602 may include specialized processing hardware, such as a RISC processor or programmable hardware with embedded firmware. Multiple processors may be included.

[0115] The system / device includes a working or volatile memory 604. One or more processors may access the volatile memory 604 to process data and may control data storage in the memory. The volatile memory 604 may include any type of RAM, such as static RAM (SRAM), dynamic RAM (DRAM), or it may include flash memory, such as an SD-card.

[0116] The system / device includes a non-volatile memory 606. The non-volatile memory 606 stores a set of operating instructions 608 in the form of computer-readable instructions for controlling the operation of the processor 602. The non-volatile memory 606 can be any type of memory, such as read-only memory (ROM), flash memory, or magnetic drive memory.

[0117] The one or more processors 602 are configured to execute operational instructions 608 to cause the system / device to perform any of the methods described herein. The operational instructions 608 may include code for the basic operation of the system / device 600, as well as code (i.e., drivers) for hardware components of the system / device 600. Generally, the one or more processors 602 execute one or more instructions of the operational instructions 608, which are stored permanently or semi-permanently in non-volatile memory 606, while the volatile memory 604 is used to temporarily store data generated during execution of the operational instructions 608.

[0118] 6b is a schematic diagram of a system 610 for performing the methods described herein. The illustrated system 610 is an example of a computing device, system, and / or cloud computing system, etc. Those skilled in the art will appreciate that other types of computing devices / systems, such as distributed computing systems, may alternatively be used to perform the methods described herein. The system 610 includes a receiver module / unit 612, a first ML model module / unit 614, a second ML model module / unit 616, a distance comparison module / unit 618, and an output module / unit 620, which may be connected to or in communication with each other as needed to implement the methods and / or apparatuses / systems described herein.

[0119] For example, the receiver module 612 may be configured to receive a set of images associated with a plurality of samples. The first ML model module 614 may be configured to input each image of the set of images to a first ML model 106a configured to predict a phenotypic feature 107 of a cellular structure within the sample associated with the each image. The second ML model module 616 may be configured to input each predicted phenotypic feature 107 associated with each sample to a second ML model 106b configured to predict a low-dimensional phenotypic feature embedding 108 for the each sample. The distance comparison module 618 may be configured to compare the distance between the low-dimensional phenotypic feature embedding 108 of each sample and a low-dimensional phenotypic feature embedding of a sample with a compound having known toxicity, which may output data representing the comparison 109. The output module 620 may be configured to output, for each sample, an indication of the toxicity of the each sample and the applied compound based on the comparison 109.

[0120] Implementations of the methods described herein may be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof, which may include a computer program product (e.g., software stored on a magnetic or optical disk, memory, programmable logic device, etc.) containing computer-readable instructions that, when executed by a computer, cause the computer to perform one or more of the methods described herein, such as those described with respect to Figure 6a or 6b.

[0121] System features described herein may also be provided as method features, and vice versa. As used herein, "means-plus-function" features may be expressed in terms of their corresponding structure. In particular, method aspects may also apply to system aspects, and vice versa.

[0122] Furthermore, any, some, and / or all features of one aspect may be applied to any, some, and / or all features of any other aspect in any appropriate combination. It should also be understood that specific combinations of various features described and defined in any aspect of the invention may be implemented and / or provided and / or used separately.

[0123] While several embodiments have been shown and described, those skilled in the art will understand that changes can be made to those embodiments without departing from the principles of the present disclosure, the scope of which is defined in the claims.

Claims

1. 1. A computer-implemented method for predicting toxicity of one or more compounds applied to a plurality of samples of cellular structures (102a) in an in vitro microscopic assay (102), comprising: receiving (112) a set of images (105) associated with the plurality of samples; inputting (114) each image of the set of images (105) into a first ML model (106a) configured to predict a phenotypic characteristic (107) of the cellular structure (102a) in the sample associated with each image; inputting (116) each of the predicted phenotypic features (107) associated with each sample into a second ML model (106b) configured to predict a low-dimensional phenotypic feature embedding (108) for each of the samples; comparing (118) the distance between the low-dimensional phenotypic feature embedding of each sample and the low-dimensional phenotypic feature embedding of a sample to which a compound with known toxicity has been applied; outputting (120) for each sample an indication of the toxicity of each sample and the applied compound based on the comparison; 20. A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, wherein the first ML model (106a) is a neural network or convolutional neural network (CNN) model (200), the neural network or CNN model (200) is trained for classification using cellular image training data, the predicted phenotypic features are embedded within all layers of the trained neural network or CNN model (200), and the method includes outputting the phenotypic feature embeddings (107) from all layers.

3. 3. The computer-implemented method of claim 1, wherein the second ML model (106b) is based on a Uniform Manifold Approximation and Projection (UMAP) algorithm or a t-SNE algorithm for dimensionality reduction of the phenotypic feature embeddings (107) of samples, whereby the phenotypic feature embeddings (107) are mapped to a low-dimensional vector space for use in comparing distances between the phenotypic feature embeddings and the phenotypic feature embeddings of samples with compounds having known toxicity.

4. training (313) the second ML model (108b) on the UMAP technique using unsupervised training based on negative and positive control samples of the plurality of samples to predict toxicity distance metrics associated with the samples to which a compound with known toxicity has been applied; or training the second ML model (106b), which comprises iteratively performing a grid search over a set of hyperparameters of the UMAP technique to select hyperparameters that maximize the difference between the negative control sample and the positive control sample; The computer-implemented method of claim 3 further comprising:

5. Indicating the toxicity of the phenotypic signature of the sample having the applied compound; applying (323) the phenotypic feature embeddings (107) of the samples with applied compounds to the second ML model (106b) to output a low-dimensional embedding (108) of the samples with applied compounds; determining (325, 325a, 325b, 326) an indication of the toxicity of the sample having a compound applied thereto based on comparing the distance between the low-dimensional embedding (108) and the embedding of one or more samples to which a compound having a known toxicity has been applied; 5. The computer-implemented method of claim 3 or 4, comprising:

6. Indicating the toxicity of the phenotypic signature of the sample having the applied compound; applying (323) the phenotypic feature embeddings (107) of the samples with applied compounds to the second ML model (106b) to output a low-dimensional embedding (108) of the samples with applied compounds; applying (324, 325) the low-dimensional embeddings (108) of the samples with the applied compound to a third ML model (106c) trained to output an indication of the distances (109) between the low-dimensional embeddings (108) and the set of low-dimensional embeddings associated with the negative control samples; The computer-implemented method of claim 3 or 4, further comprising:

7. 7. The computer-implemented method of claim 6, further comprising: training the third ML model (106c) based on performing a grid search over a set of hyperparameters of a high-dimensional distance metric algorithm that maximizes the distance between the low-dimensional embeddings of the negative control samples and the positive control samples while minimizing the distance between the low-dimensional embeddings of the negative control samples or minimizing the distance between the low-dimensional embeddings of the positive control samples.

8. 8. The computer-implemented method of claim 1, wherein comparing the distances used to indicate the toxicity of the phenotypic feature embeddings of samples is based on a Wasserstein distance metric.

9. receiving the set of images (105) associated with the plurality of samples; automatically identifying (502) a first set of samples useful for analysis from a plurality of samples on an assay plate; generating (504) a set of two-dimensional (2D) images for each sample in the first set of samples, the set of 2D images for each sample including a plurality of 2D image slices taken along a z-axis of each sample; Identifying a set of available samples from the set of 2D image slices (506); outputting (508) data representing the set of available samples for analysis as the set of images (105); identifying an available sample of cellular structures for analysis in an in vitro microscopy assay (500) based on the steps consisting of: The computer-implemented method of any one of claims 1 to 8, further comprising:

10. The automatically identifying (502) a first set of samples includes, for each sample in the plurality of samples: Pre-processing (512) an image of each sample; inputting 514 the pre-processed sample image to a first machine learning (ML) model configured to identify regions of interest in the input sample image comp that include cellular structures; inputting the identified region of interest of the sample image into a second ML model configured to classify whether the sample is analyzable (516); outputting (518) the first set of samples including data representative of the samples classified as analyzable; The computer-implemented method of claim 9 further comprising:

11. the first ML model (520) for identifying the region of interest is a convolutional neural network (CNN) or other neural network trained to identify regions of interest that include cellular structures; the second ML model (530) for classifying whether the sample is analyzable is a one-class SVM configured to classify whether the region of interest is analyzable; training / configuring the CNN based on a labeled training dataset comprising a plurality of images, each of the images annotated with a label comprising data representing whether a cellular region of interest is present and / or the location of the region of interest within the image; The computer-implemented method of claim 10 , further comprising training / configuring the one-class SVM to classify whether the region of interest is analyzable or not.

12. Identifying the set of available samples from the set of 2D image slices comprises, for each sample: identifying (542) foreground, background, and a plurality of uncertain feature regions of the cellular structure in each of the 2D image slices; iteratively combining the foreground, background, and uncertain feature regions of the 2D image slices to generate a single 2D image of the cellular structure (544); selecting (546) the samples of the set of available samples based on the quality of the single 2D image; outputting, for each selected sample, a set of available images including image data representing the single 2D image associated with each selected sample; Further comprising: the plurality of uncertain feature regions including a plurality of uncertain foreground features and a plurality of uncertain background features; A computer-implemented method according to any one of claims 9 to 11.

13. An apparatus (600) comprising a processor (602), a memory unit (606), and a communication interface (604), wherein the processor (602) is connected to the memory unit (606) and the communication interface (604), and the processor (602) and memory (606) are configured to perform the computer-implemented method of any one of claims 1 to 12.

14. A computer readable medium containing data or instruction code that, when executed on a processor (602), causes the processor (602) to perform the computer-implemented method of any one of claims 1 to 12.

15. A system (610), comprising: a receiver module (612) configured to receive a set of images associated with a plurality of samples; a first ML model module (614) configured to input each image of the set of images into a first ML model (106a) configured to predict a phenotypic characteristic (107) of a cellular structure in the sample associated with each image; a second ML model module (616) configured to input each of the predicted phenotypic features (107) associated with each sample into a second ML model (106b) configured to predict a low-dimensional phenotypic feature embedding (108) for each sample; a distance comparison module (618) configured to compare the distance between the low-dimensional phenotypic feature embeddings (108) of each sample and the low-dimensional phenotypic feature embeddings of samples with compounds having known toxicity; an output module (620) configured to output, for each sample, an indication of the toxicity of each sample and the applied compound based on the comparison (109); A system (610) comprising:

Images