Systems and methods for computational measurements and learning- based randomized classification

Abstract

An apparatus can include a computational imaging system configured to output a plurality of features. The apparatus can include one or more processors configured to receive the plurality of features, apply the plurality of features as input to a classifier to cause the classifier to output a first performance metric, apply the plurality of features as input to the classifier to cause the classifier to output a plurality of performance metrics, and / or calculate a p-value based on the first performance metric and / or the plurality of performance metrics.

Description

SYSTEMS AND METHODS FOR COMPUTATIONAL MEASUREMENTS AND LEARNING-BASED RANDOMIZED CLASSIFICATIONCROSS-REFERENCE TO RELATED PATENT APPLICATION

[0001] This application claims the benefit and priority of U.S. Provisional Patent Application No. 63 / 537,998, filed on September 12, 2023, the entirety of which is incorporated by reference herein.TECHNICAL FIELD

[0002] The present application relates generally to classification.BACKGROUND

[0003] Scientific experiments can have a goal of establishing effects of a drug on biological samples.SUMMARY

[0004] At least one aspect of the present disclosure is directed to an apparatus. The apparatus can include a computational imaging system configured to output a plurality of features. The apparatus can include one or more processors. The one or more processors can be configured to receive the plurality of features. The one or more processors can be configured to apply the plurality of features as input to a classifier (e.g., to train the classifier, when the plurality of units has ground truth labels) to cause the classifier to output a first performance metric. The classifier can be configured based on a plurality of units including a first subset of a first plurality of units and a first subset of a second plurality of units, when the plurality of units has (e.g., is applied / coupled / used / analyzed / tagged with) ground truth labels. The one or more processors can be configured to apply the plurality of features as input to the classifier e.g., to train the classifier) to cause the classifier to output a plurality of performance metrics, when the plurality of units has a plurality of randomized labels. The plurality of randomized labels can be generated through randomization of the ground truth labels. The one or more processors can be configured to calculate / determine a p-value based on (e.g., using or according to) the first performance metric and / or the plurality of performance metrics.

[0005] Another aspect of the present disclosure is directed to a method. The method can include organizing an initial dataset into at least a first class including a first plurality of units and a second class including a second plurality of units. The method can includeassigning ground truth labels to the first plurality of units and the second plurality of units. The method can include dividing the first plurality of units and the second plurality of units into (1) a training dataset including a plurality of units including a first subset of the first plurality of units and a first subset of the second plurality of units, the training dataset having (e.g., applied / coupled / used / analyzed / tagged with) the ground truth labels, and (2) a test dataset including a third plurality of units including a second subset of the first plurality of units and a second subset of the second plurality of units, the test dataset having the ground truth labels. The method can include training a classifier using the training dataset when the training dataset has the ground truth labels. The method can include evaluating the classifier using the test dataset when the test dataset has the ground truth labels to generate a first metric. The method can include assigning a plurality of randomized labels to the first plurality of units and / or the second plurality of units, the plurality of randomized labels generated through randomization of the ground truth labels. The method can include training the classifier when the training dataset has the plurality of randomized labels. The method can include evaluating (e.g., determining performance of) the classifier using the test dataset when the training dataset has the plurality of randomized labels to generate a plurality of metrics. The method can include calculating a p-value based on the first metric and / or the plurality of metrics.

[0006] Those skilled in the art will appreciate that the summary is illustrative only and is not intended to be in any way limiting. Other aspects, inventive features, and advantages of the devices and / or processes described herein, as defined solely by the claims, will become apparent in the detailed description set forth herein and taken in conjunction with the accompanying drawings.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0008] FIGS. 1 A and IB illustrate systems / apparatuses for randomized classification, according to an embodiment.

[0009] FIG. 2 illustrates a procedure for randomized classification, according to an embodiment.

[0010] FIGS. 3A and 3B illustrate causal effects of NDMA on yH2AX and DAPI, according to an embodiment.

[0011] FIG. 4 illustrates a schematic diagram of a computational imaging system, according to an embodiment.

[0012] FIGS. 5A-5D illustrate causal effects of NDMA on the nuclei morphology established using ultra-fast computational imaging, according to an embodiment.

[0013] FIG. 6 illustrates a schematic diagram of a computational imaging system, according to an embodiment.

[0014] FIGS. 7A-7E illustrate causal effects of NDMA on refractive index maps established using ultra-fast label-free computational imaging, according to an embodiment.

[0015] FIGS. 8A-8D illustrate a simulated example on MNIST data, according to an embodiment.

[0016] FIG. 9 illustrates a method for randomized classification, according to an embodiment.

[0017] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0018] Following below are more detailed descriptions of various concepts related to, and implementations of methods, systems, and apparatuses for computational measurements and learning-based randomized classification. The various concepts introduced above and discussed in greater detail below may be implemented in any of a number of ways, as the described concepts are not limited to any particular manner of implementation. Examples of specific implementations and applications are provided primarily for illustrative purposes.

[0019] It can cost hundreds of millions of dollars to discover a new drug through multiple levels of experiments. Most, if not all, of these experiments can share the goal of establishing the causal effects of a drug on biological samples such as cells, tissue from animal models, and tissue from humans treated with the drug. Often, the potential effect is unknown. The problem known as “bio-marker discovery” can include the process of identifying a measurable effect. An expert biologist can qualitatively examine a set of measurements from a treatment (e.g., with the drug) group and a control (e.g., without the drug) group to identify differential features (e.g., the change of morphology of cells or tissuein images). The features can then be quantified and examined for statistically significant causal differences between the groups. This human-in-the-loop process, while interpretable, is labor-intensive, subjective, and slow.

[0020] An alternative approach can be based on machine learning algorithms. The two sets of measurements from the treatment and control groups can be assigned two class labels. Then a classification model can be trained to predict the class from the measurements. The performance of the classifier on unseen test data is used as a metric to predict the causal effect of the drug. A classifier, such as a deep neural network, may outperform expert biologists by identifying subtle effects indistinguishable to humans. The classifier may also be trained on uninterpretable measurements such as partial images, or new imaging modalities unfamiliar to experts. However, such an algorithm does not establish direct causal effects. Therefore, the entire process, while intriguing, may result in many false positives, risking late-stage failures that can cost millions. Thus, this method has limited adaptability in the drug discovery process. The present disclosure is directed to systems, methods, and apparatuses to reliably establish causal effects through a fully automated classification framework that overcomes this limitation.

[0021] FIGS. 1A and IB illustrate apparatuses 100 (e.g., framework, system) for randomized classification. The framework can include a contrastive learning-based deep learning framework to unify image-based machine learning experiments with hypothesis driven science. The apparatus 100 can include one or more computational imaging systems 105 (e.g., imaging systems). The computational imaging system 105 can receive a sample (e.g., physical sample). For example, the computational imaging system 105 can acquire an image of the sample. The computational imaging system 105 can output a plurality of features. The computational imaging system 105 can output measurements (e.g., encoded measurements). The computational imaging system 105 can be designed to extract a plurality of features using a pre-existing training dataset. The computational imaging system 105 can be designed to extract a plurality of features using a pre-existing training dataset when the training dataset has no labels. The pre-existing training dataset can include the plurality of units (e.g., cells).

[0022] The computational imaging system 105 can include one or more measurement systems 115 (e.g., arbitrary measurement systems). The measurement system 115 can include an imaging device. For example, the measurement system 115 can include a microscope. The measurement system 115 can include an optical system. The measurement system 115 canreceive the sample. For example, the measurement system 115 can acquire an image of the sample. The measurement system 115 can perform, acquire, generate and / or output one or more measurements. For example, the measurement system 115 can output one or more measurements of the sample. The measurement system 115 can output one or more images of the sample. For example, the measurement system 115 can output one or more digital images of the sample. The measurement system 115 can output a plurality of images. The measurement system 115 can be designed or learned independent of the class assignments of the treatment group and the control group. The measurement system 115 can capture spectral information. For example, the measurement system 115 can capture spectral information for each pixel of the plurality of images.

[0023] The computational imaging system 105 can include one or more encoders 120. The encoder 120 can extract information out of (e.g., from) the one or more measurements. The encoder 120 can be designed or learned independent of the class assignments of the treatment group and the control group. The encoder 120 can output the encoded measurements. The encoder 120 can output the plurality of features. For example, the encoder 120 can output a plurality of features for the images output by the measurement system 115. The encoder 120 can receive the plurality of images. For example, the encoder 120 can receive the plurality of images from the measurement system 115. The encoder 120 and the measurement system 115 can interoperate or be combined / coupled to form the computational imaging system 105.

[0024] In some embodiments, the plurality of features includes a plurality of three- dimensional (3D) features. The computational imaging system 105 can include the measurement system 115. The measurement system 115 can output a plurality of images. The computational imaging system 105 can include the encoder 120. The encoder 120 can receive the plurality of images. For example, the encoder 120 can receive the plurality of images from the measurement system 115. The encoder 120 can output the plurality of 3D features.

[0025] In some embodiments, the plurality of features includes a plurality of temporal features. The computational imaging system 105 can include the measurement system 115. The measurement system 115 can output a plurality of images. The computational imaging system 105 can include the encoder 120. The encoder 120 can receive the plurality of images. For example, the encoder 120 can receive the plurality of images from the measurement system 115. The encoder 120 can output the plurality of temporal features.

[0026] The apparatus 100 can include one or more classifiers 110 (e.g., classification model, trainable classification model, classification framework, binary image classifier). The classifier 110 can receive the encoded measurements from the computational imaging system 105. The classifier 110 can receive the encoded measurements from the encoder 120. The classifier 110 can undergo supervised training. The output of the classifier 110 can include identifying the sample as belonging to a certain group (e.g., treatment group or control group). The classifier 110 can classify two or more classes. For example, the classifier 110 can identify the sample as belonging to a first class or a second class. The classifier 110 can identify the sample as belonging to the first class, the second class, or a third class. The classifier 110 can predict if an input is from a treatment group or a control group. The classifier 110 can be evaluated on a group of unseen test data. The performance of the classifier 110 can be recorded. Classification accuracy can be used as a performance metric of the classifier 110. A quantitative property (e.g., model parameter) of the classifier 110 can be used to measure the performance of the classifier 110. The classifier 110 can include a software-based classifier.

[0027] The apparatus 100 can include one or more processors. The one or more processors can receive the plurality of features. The one or more processors can apply the plurality of features as input to the classifier 110. For example, the one or more processors can apply the plurality of features as input to the classifier 110 to cause the classifier 110 to output a first performance metric. The classifier 110 can be configured based on a plurality of units (e.g., cells). For example, the classifier 110 can be configured based on the plurality of units when the plurality of units has (e.g., is applied with, is coupled with, is used with, is analyzed with, is tagged with, is associated with) ground truth labels. The one or more processors can apply the plurality of features as input to the classifier 110 when the plurality of units has ground truth labels. The plurality of units can include a first subset of a first plurality of units and a first subset of a second plurality of units.

[0028] The one or more processors can apply the plurality of features as input to the classifier 110, or to a plurality of classifiers different from the classifier 110. For example, the one or more processors can apply the plurality of features as input to the classifier 110 to cause the classifier 110 to output a plurality of performance metrics. The one or more processors can apply the plurality of features as input to the classifier 110 to cause the classifier 110 to output the plurality of performance metrics when the plurality of units has a plurality of randomized labels. The one or more processors can apply the plurality of featuresas input to the classifier 110 to train the classifier 110. The one or more processors can apply the plurality of features as input to the classifier 110 when the plurality of units has a plurality of randomized labels. The plurality of randomized labels can be generated through randomization of the ground truth labels.

[0029] The plurality of randomized labels can be generated through randomization of the ground truth labels. The plurality of randomized labels can have a structure (e.g., data structure or ratio of control vs treatment datasets) that corresponds to a structure of the initial dataset. The randomization of the ground truth labels can be performed based on a causal structure of an experiment. For example, the randomization of the ground truth labels can be performed based on a causal structure of the experiment such that the units that share the same cause also share the same label during randomization. The randomization of the ground truth labels can be performed based on a hierarchy of a property of interest of the plurality of units. For example, the randomization of the ground truth labels can be performed based on a hierarchy of a property of interest of the plurality of units such that the units that share the same property also share the same label during randomization. The randomization of the ground truth labels can be performed based on external stimuli to the plurality of units. For example, the randomization of the ground truth labels can be performed based on external stimuli to the plurality of units such that the units that share the same external stimuli also share the same label during randomization.

[0030] The one or more processors can calculate a p-value. For example, the one or more processors can calculate (e.g., determine) the p-value based on (e.g., using or according to) the first performance metric (e.g., as a reference or threshold) and / or the plurality of performance metrics (e.g., by comparing / evaluating these against the first performance metric). The p-value can be defined by a ratio of the plurality of performance metrics greater than or equal to the first performance metric. The p-value can be defined by a ratio of plurality of performance metrics equal to or more extreme than the first performance metric.

[0031] The one or more processors can divide / group / arrange the first plurality of units and the second plurality of units into a training dataset and a test dataset. The training dataset can include the plurality of units. The training dataset can have ground truth labels. The test dataset can include a third plurality of units. The third plurality of units can include a second subset of the first plurality of units and a second subset of the second plurality of units. The test dataset can have the ground truth labels.

[0032] The first plurality of units and the second plurality of units can be obtained from an initial dataset. The initial dataset can include a plurality of images The initial dataset can include a first class and a second class. The first class can include the first plurality of units. The second class can include the second plurality of units. The initial dataset can include at least the first class and the second class. The first class can include a treatment class (e.g., of treatment datasets). The second class can include a control class (e.g., of control datasets).

[0033] In some embodiments, the one or more processors can apply the plurality of features as input to the classifier 110 to cause the classifier 110 to output a first quantifiable property of the classifier 110. The one or more processors can apply the plurality of features as input to the classifier 110 to cause the classifier 110 to output a plurality of quantifiable properties of the classifier 110. The one or more processors can apply the plurality of features as input to the classifier 110 to cause the classifier 110 to output the plurality of quantifiable properties of the classifier 110 when the plurality of units has a plurality of randomized labels. The one or more processors can calculate the p-value based on the first quantifiable property of the classifier 110 and the plurality of quantifiable properties of the classifier 110. The p-value can be defined by a ratio of the plurality of quantifiable properties of the classifier 110 greater than or equal to the first quantifiable property of the classifier 110.

[0034] In some embodiments, the one or more processors can apply the plurality of features as input to the classifier 110 to cause the classifier 110 to output a first parameter of the classifier 110. The one or more processors can apply the plurality of features as input to the classifier 110 to cause the classifier 110 to output a plurality of parameters of the classifier 110. The one or more processors can apply the plurality of features as input to the classifier 110 to cause the classifier 110 to output the plurality of parameters of the classifier 110 when the plurality of units has a plurality of randomized labels (e.g., as input to the classifier or for supervised training / learning of the classifier). The one or more processors can calculate the p-value based on the first parameter of the classifier 110 and the plurality of parameters of the classifier 110. The p-value can be defined by a ratio of the plurality of parameters of the classifier 110 greater than or equal to the first parameter of the classifier 110.

[0035] In some embodiments, the one or more processors can apply the plurality of features as input to the classifier 110 to cause the classifier 110 to output a first output of a function of parameters of the classifier 110. The one or more processors can apply theplurality of features as input to the classifier 110 to cause the classifier 110 to output a plurality of outputs of a function of parameters of the classifier 110. The one or more processors can apply the plurality of features as input to the classifier 110 to cause the classifier 110 to output the plurality of outputs of the function of parameters of the classifier 110 when the plurality of units has a plurality of randomized labels (e.g., as input to the classifier or for supervised training / learning of the classifier). The one or more processors can calculate the p-value based on the first output of the function of parameters of the classifier 110 and the plurality of outputs of the function of parameters of the classifier 110. The p- value can be defined by a ratio of the plurality of outputs of a function of parameters of the classifier 110 greater than or equal to the first output of the function of parameters of the classifier 110.

[0036] FIG. 2 illustrates a method 200 (e.g., procedure) for randomized classification. The method 200 can use the apparatus 100. The method 200 can use the computational imaging system 105, the classifier 110, the measurement system 115, and / or the encoder 120. FIG. 2 illustrates the procedure to train and use the performance metric to establish causal effects. The dataset (e.g., set of encoded measurements) can be divided into a training set and a test set. The training set can be used to train the classifier 110. The test set can be used to evaluate the accuracy of the classifier 110. First, the classifier 110 can be trained using the observed true class labels (Wobs). These can include the treatment class (class “1”) and the control class (class “0”). Then another set of similar classifiers can be trained with randomly assigned class labels (e.g., W1, W2, W3, W4. . . ). The test accuracies can be recorded for these randomly permuted ground truths. Then the histogram of the test accuracy values can be calculated. The ratio of randomizations that performed equal to or better than the original Wobscan be calculated as the p-value. The p-value can be directly representative of the causal effect of the treatment (e.g., drug) on the sample (e.g., biological sample) that originated the encoded measurements. The performance of the classifier 110 can be used as a test statistic. For example, the performance of the classifier 110 can be used as a test statistic to numerically generate a null distribution of the no-causal effect of the drug, followed by the calculation of the Fisher-exact p-values. The framework and procedure can be applicable to a wide range of measurement systems including computational imaging systems, and measurement systems with unknown measurands. No-causal effect can be due to random chance, a difference between animal s / subjects used to prepare the samples, how the imaging is performed and / or a difference between how sample slides were prepared, etc.

[0037] FIG. 2 illustrates an example that shows the classifier 110 used in the potential outcomes framework to evaluate the causal effects of a treatment on mice. In the experiment, half the mice are randomly assigned to receive the treatment. Tissue samples are prepared from each mouse, and a large number of cells (e.g., units) are imaged from each sample. Part of the cells are used to train a classification network and the rest are used to test. The observed test accuracy,is recorded. To evaluate if the accuracy is significant, a null distribution of test accuracies (e.g., the histogram) is generated by repeating the training and testing process with random class allocations on the samples.

[0038] The method 200 can include a deep-learning-based image classification in scientific experiments. Consider a high-content microscopy experiment with m independent samples. Let j E {1, 2, ... , m} be the sample identifier. Each sample can belong to an observed class w°bsE {0, 1, 2, . . ., k — 1}. For instance, in a binary classification problem such as treated samples vs. control samples w°bsE {0, 1], 0 represents the control class and 1 represents the treated class. Next, let rij be the number of cells imaged from the jthsample. Let YjObsE RL2be the image of the ithcell from the jthsample. Each image of a cell can be assumed to be of the same size of L x L pixels. While the cells from each sample are likely correlated, the approach may treat each cell as an independent unit in the experiment and train a deep neural network to classify them.

[0039] In that case, the image dataset {can be divided into a training set and a test set by randomly allocating a portion of cells from each sample to the training set. Using the training set, a classifier, 0('F(. )) (e.g., classifier 110), can be trained to predict the observed classes {wz’s} for each input cell according to Equation 1.

[0040] For ease of later formulation, the classifier 110 can be divided into two trainable networks (e.g., an encoder1P(. ) and a classification head 0(. )). The accuracy of the model can be evaluated on the test dataset and treated as evidence to support the existence of a discriminating signal between the classes.

[0041] Such an experiment can result in high accuracy due to two reasons: first, due to an actual signal that makes cells from each class significantly different from other classes and second, due to the classifier 110 finding a decision boundary based on other variationsbetween the samples. The second reason can pose a risk of making false conclusions despite having “high” accuracy.

[0042] The method 200 can include a potential outcomes framework for deeplearning-based image classification. The method 200 can estimate the probability of the accuracy being “high” due to reasons other than the causal effect of a sample belonging to its corresponding (or observed) class. To do this, the problem can be formulated in the potential outcomes framework. The potential outcomes framework can assume that there exists a definitive outcome Yj ^Wj), for the same cell (j, i) belonging to any class Wj, but only one out of these potential outcomes are observed (e.g., the acquired or observed image, Yji(w°bs^ = YjObsfrom the observed class w°bs). All other outcomes can be treated as missing outcomes. For the experiment, a science table can be formulated in the potential outcomes as shown in Table 1.

[0043] A sharp null hypothesis can be defined in which for each cell i from each sample j has no effect of being associated with the observed class w°bs. The hypothesis can be defined by Equation 2.HQ: y.i(o) = ^(i) =... = ^(fc - i) = ^sv( / ,i) (2)

[0044] Under Ho, there is no signal in the observed image Yj°bsand all potential outcomes would have been the same had they have been able to be observed. Therefore, if the classifier 110 is trained by assigning a random class to each sample, under the null hypothesis that classifier 110 would have a similar test accuracy as the original classifier typically trained for the actual (or observed) classes for each sample.

[0045] The method 200 can include a permutation test. The associated p-value can be termed the Fisher Exact P-value (FEP). In the FEP approach, the distribution of a test statistic, T generated by the complete randomization of classes assigned to each sample, under the null hypothesis Hocan be numerically calculated. The actual observed value of the same test statistic, Tobs(generated from the actually observed classes for each sample) can be compared against the distribution of T. A test statistic can be defined as any real-valued scalar function T(W, Yobs) where W is a random assignment vector, and Yobsis the vector of observed outcomes (e.g., the Yj°bscolumn in the Table 1). An assignment can include a random allocation of classes to each sample j and the vector of assigned classes for each cell.For instance, the actual (or the observed) assignment, Wobs, is the column w°bsin the Table 1.Table 1 An example science table in the potential outcome framework, for the high- content microscopy experiment for a two-class problem. Question marks, “?”, can represent missing outcomes.

[0046] Any test performance metric of the classifier 110 trained under a random assignment W can fall under the definition of a test statistic. Thus, any loss function L(. ) can be used as a test statistic, as shown in Equation 3.

[0047] Here, PF is therandom assignment of classes, and w- is the class assigned to the image Y°bsunder IVf. T estSet is the set of cells in the test set and Ntest is the cardinality of TestSet. T (Yobs, IV) is a function of the test images. T (Yobs, IV) is also afunction of the training images. This is because the weights of the classifier can be a function of the training images.

[0048] This experiment can be repeated for multiple random assignments and a distribution of Ts under the null hypothesis can be generated. A class can be randomly assigned for each sample. A classifier can be trained for that assignment. The performance of the classifier can be calculated. This can be repeated many times with many random assignments. This way the exact null distribution of the test statistic T can be generated and can be used to calculate the p-value of the observed, T°bsunder the null hypothesis. An example for this approach is shown in FIG. 2.

[0049] A deep neural network may need to be trained for each assignment. Each network can take hours to days to train even on multiple GPUs. A method to efficiently train this ensemble of networks can be described herein.

[0050] To efficiently train the ensemble of networks (e.g., multiple networks), the network can be divided into two parts: a deep encoder, ¥(.), that is trained self-supervised without using class labels, and a shallow classification head, 0( ), that can be trained fast with class labels. The deep-image-encoder (.) trained self-supervised can encode each image to a low-dimensional representation aj,i G Rlwith I (« / ?) features, as shown in Equation 4.

[0051] Then (.) can be used to generate embeddings for each cell, and a classification head, 0(.) can be trained for each randomization, as shown in Equation 5.

[0052] In this approach, the deep network can be trained only once and may not need to be trained multiple times.

[0053] FIGS. 3 A and 3B illustrate causal effects of nitrosodimethylamine (NDMA) on yH2AX and DAPI. The apparatus 100 and / or method 200 can be used for morphological biomarker discovery on mouse tissue using immunohistochemistry and imaging. The apparatus 100 and / or method 200 can be used to establish causal effects of mice treated with a toxicological profile for NDMA. Mice treated with NDMA and saline were sacrificed.Their liver tissue was sliced and stained with immunohistochemistry labels for the nuclei morphology (DAPI) and DNA damage and repair markers (yH2AX). yH2AX is a knownbiomarker for DNA damage events caused by NDMA. Both yH2AX and DAPI were imaged with a fluorescence microscope. The images were patched around the nuclei and used in the apparatus 100. FIGS. 3A-3D show the results on a cohort of 8 mice, 4 treated with NDMA (treatment class) and 4 treated with Saline (control class). FIG. 3 A shows that the method 200 predicts a causal effect with p = 0.057 on the known biomarker yH2AX. FIG. 3B shows that the method 200 predicts the existence of a new biomarker (e.g., DAPI) related to the nuclei morphology with p = 0.02.

[0054] FIG. 4 illustrates a schematic diagram of the computational imaging system 105. The apparatus 100 and / or method 200 can be used for morphological biomarker discovery on subject / animal (e.g., mouse) tissue using ultra-fast computational imaging. The method 200 can be used with a computational imaging approach.

[0055] The computational imaging system 105 can include a fluorescence imaging system. The computational imaging system 105 can include a coded aperture 405 (e.g., coding element). The coded aperture 405 can receive light from a light source 402 and output structured light. The structured light can pass through a sample 412 (e.g., sample of interest). The coded aperture 405 can encode two-dimensional (2D) features from the focal plane. The computational imaging system 105 can capture 3D features. The coded aperture 405 can be replaced with a coded element designed to project 3D coded illumination patterns that capture 3D morphological information from the sample under investigation. The computational imaging system 105 can capture temporal features. The coded aperture 405 may be replaced with a coding element (e.g., 2D or 3D) that changes with time while making measurements. The coding element can capture dynamic behavior of the sample under consideration. The coding element may also capture the response of the sample to external temporal stimuli.

[0056] The computational imaging system 105 can include a camera 410. The camera can receive the structured light from the coded aperture 405. The structured light can pass through the sample 412 before being received by the camera 410. The camera 410 can receive the structured light after the structured light has passed through the sample 412. The camera 410 can receive emission light from the sample 412. For example, the camera 410 can receive emission light from the sample of interest, illuminated by the encoded light from the coded aperture 405. After the coded aperture, the images can be acquired at low resolution. The coded aperture 405 can encode high-resolution features in the low-resolution images.

[0057] The computational imaging system 105 can include a de-magnifier 415. The de-magnifier can include a focusing device (e.g., electrostatic focusing device) configured to reduce a size of an image. The de-magnifier can receive the structured light from the coded aperture 405. The de-magnifier can output structured light to the camera 410. The camera 410 can receive structured light from the de-magnifier 415.

[0058] The computational imaging system 105 can include a beam expander 420. The beam expander 420 can receive the light from the light source 402. The beam expander 420 can output light with an expanded width. For example, the beam expander 420 can output light with the expanded width to the coded aperture 405.

[0059] The computational imaging system 105 can include a first tube lens 425. The computational imaging system 105 can include a second tube lens 430. The computational imaging system 105 can include a mirror 435 (e.g., dichroic mirror). The computational imaging system 105 can include an objective 440.

[0060] Light from the light source 402 can be received by the beam expander 420. Light from the beam expander 420 can be received by the coded aperture 405. Light from the coded aperture 405 can be received by the first tube lens 425. Light from the first tube lens 425 can be received by the mirror 435. Light from the mirror 435 can be received by the second tube lens 430. Light from the second tube lens 430 can be received by the de- magnifier 415. Light from the de-magnifier 415 can be received by the camera 410. Light from the sample 412 can be received by the objective 440. Light from the objective 440 can be received by the mirror 435.

[0061] FIGS. 5A-5D illustrate causal effects of NDMA on the nuclei morphology established using ultra-fast computational imaging. The low-resolution images can be acquired much faster compared with high-resolution images. The existence of the new biomarker (e.g., DAPI) in the nuclei morphology can be established despite uninterpretable low-resolution computational images at even three orders of magnitude higher imaging speeds. This demonstration shows the use of the method 200 in combination with ultra-highspeed computational imaging for high-speed screening.

[0062] FIG. 5 A illustrates representative measurements of a simulated computational imaging system with ultrafast imaging capability. 4x, 16x, 64x, 256x, and 1024x show possible speed enhancement over fluorescence imaging (e.g., a no-compression case). FIG. 5B illustrates a null histogram and p-value given by imaging with no compression. FIG. 5Cillustrates the null histogram and p-value given by 265x faster computational imaging scheme compared to the no-compression case. FIG. 5D illustrates a plot of the p-value and accuracy for different configurations of the computational imaging system. The p-value of the randomization can be a rank metric.

[0063] FIG. 6 illustrates a schematic diagram of a computational imaging system 105. The apparatus 100 and / or method 200 can be used for morphological biomarker discovery on mouse tissue using label-free computational imaging. The method 200 can discover the existence of a new biomarker using a label-free imaging technique without the need for immunohistochemistry. The computational imaging system 105 can measure the spatial refractive index information of the specimen (e.g., sample). The computational imaging system 105 can be used without staining to measure intrinsic properties of the sample rather than chemical properties of the stain.

[0064] The computational imaging system 105 can include the light source 402. The computational imaging system 105 can include the beam expander 420. The computational imaging system 105 can include the sample 412. The computational imaging system 105 can include the objective 440. The computational imaging system 105 can include the first tube lens 425. The computational imaging system 105 can include a first lens 610. The computational imaging system 105 can include a second lens 615. The computational imaging system 105 can include the camera 410.

[0065] The computational imaging system 105 can include a computational phase imaging system. The computational phase imaging system can include a Fourier filter 605. The Fourier filter 605 can be configured to modulate the phase of a light field. The Fourier filter 605 can encode phase information of the light field. The Fourier filter 605 can receive a light field from the light source 402. The light source 402 can include a laser. The light source 402 can include a coherent, partially-coherent, or an incoherent light source. The Fourier filter 605 can modulate a phase of the light field. The Fourier filter 605 can output a modulated light field. The Fourier filter 605 can be placed at a Fourier plane after the first tube lens 425. The intensity of the modulated light field can be recorded at the next image plane using the camera 410. The camera 410 can receive the modulated light field from the Fourier filter 605.

[0066] Light from the light source 402 can be received by the beam expander 420. Light from the beam expander 420 can be received by the sample 412. Light from the sample412 can be received by the objective 440. Light from the objective 440 can be received by the first tube lens 425. Light from the first tube lens 425 can be received by the first lens 610. Light from the first lens 610 can be received by the Fourier filter 605. Light from the Fourier filter 605 can be received by the second lens 615. Light from the second lens 615 can be received by the camera 410.

[0067] FIGS. 7A-7E illustrate causal effects of NDMA on refractive index maps established using ultra-fast label-free computational imaging. The apparatus 100 and / or method 200 can use images and / or measurements to establish causal effects. Two different sets of treated / treatment and control specimens were imaged / measured using the computational imaging system 105.

[0068] FIG. 7A illustrates representative images of liver tissue specimens using the label-free computational imaging system 105. Image 1 shows the image at the original resolution and images 2-4 show images down-sampled for low-resolution but enhanced throughput. The images can be down-sampled to emulate an ultra-fast label-free screening scenario. Image 2 shows the image at 16x lower resolution. Image 3 shows the image at 32x lower resolution. Image 4 shows the image at 128x lower resolution.

[0069] FIG. 7B shows representative image patches from each resolution level (columns) for samples from each mouse (rows). These images may not be interpretable, especially at low resolutions. The method 200 can establish the existence of a new biomarker at up to 32x lower resolution.

[0070] FIG. 7C illustrates a plot of resulting p-values from the apparatus 100 and / or method 200 for multiple replications of the scheme. The training and test sets can be randomly selected with different seeds. This can demonstrate the use of the method 200 to measure causal effects for new imaging modalities. FIG. 7D illustrates a representative null distribution of the test accuracy at the original resolution. FIG. 7E illustrates a representative null distribution of the test accuracy at 16x lower resolution.

[0071] FIGS. 8A-8D illustrate a simulated example on MNIST data. This example shows two simulated experiments with a signal (FIGS. 8 A and 8B) and without a signal (FIGS. 8C and 8D). FIG. 8 A illustrates representative examples from ten samples belonging to two groups. Each sample (a row) includes variations of a handwritten digit from the MNIST dataset. The first group includes digits that are rotated 45 degrees and serve as the causal signal. FIG. 8B illustrates the observed and the null distribution of classificationaccuracy for the two classes in FIG. 8 A. FIG. 8C illustrates the equivalent of FIG. 8 A but without rotating the digits to emulate an experiment without a causal signal. FIG. 8D illustrates the observed and the null distribution of classification accuracy for the two classes in FIG. 8C. Despite no signal, the classifier 110 can be trained to a “high” (91%) accuracy; without the null distribution, showing there is no way of evaluating whether the signal is real. Therefore, “high” classification accuracy may not be evidence of a causal signal.

[0072] The two simulated experiments can show that high classification accuracy, by itself alone, can be misleading as a metric of a signal. Data from the MNIST dataset can be used to simulate ten samples divided into two groups. A sample was defined as a variation of a handwritten digit from the MNIST dataset. The first group can include samples from digits 0 to 4, and the second group can include samples from digits 5 to 9. In the first experiment, a causal signal can be introduced by rotating the digits from the first group by 45 degrees (FIG. 8A). The potential outcomes framework can be applied and the Fisher-exact p-values (FEPs) can be calculated on the test accuracy. When accuracy is used as the test statistic, right-tailed p-values should be calculated. In this experiment, a 99% test accuracy and a p-value of 0.004 can be observed.

[0073] The same experiment can be repeated but without the causal signal (e.g., no rotations). FIG. 8C shows representative units from each sample from the two groups. Similar to the previous case, the potential outcomes framework can be applied and the FEP can be calculated on the test accuracy. An accuracy of 91% can be observed. This may be considered a “high” accuracy value and can potentially lead to the wrong conclusion that the two groups are statistically different. A p-value of 0.9683 can be observed, confirming no significant difference between the two groups.

[0074] These two experiments show that the test accuracy of the classifier 110 may not be a reliable metric to measure causal effects. When there was a causal signal (e.g., in the first experiment), the classification accuracy was significantly higher than when there was no signal (e.g., in the second experiment). The two experiments are counterfactuals of each other. There is no way to conduct both experiments and compare the two accuracies.

[0075] FIG. 9 illustrates the method 200 for randomized classification. The method 200 can be used for diagnostics. The method 200 can be used for biomarker discovery. In brief summary, the method 200 can include organizing an initial dataset (BLOCK 905). The method 200 can include assigning labels (BLOCK 910). The method 200 can includedividing units into a training dataset and a test dataset (BLOCK 915). The method 200 can include training a classifier (BLOCK 920). The method 200 can include evaluating the classifier (BLOCK 925). The method 200 can include calculating a p-value (BLOCK 930).

[0076] The method 200 can include organizing (e.g., dividing) an initial dataset (BLOCK 905). For example, the method 200 can include organizing the initial dataset into two or more classes (e.g., multiple classes). The initial dataset can include a plurality of images. The method 200 can include organizing the initial dataset into at least a first class and a second class. The first class can include a first plurality of units. The first class can include a treatment class. The second class can include a second plurality of units. The second class can include a control class.

[0077] The method 200 can include assigning labels (BLOCK 910). For example, the method 200 can include assigning ground truth labels (e.g., for supervised training of the classifier). The method 200 can include assigning ground truth labels to the first plurality of units. The method 200 can include assigning ground truth labels to the second plurality of units. The ground truth labels can be indicative of the class that each unit of the first plurality of units belongs. The ground truth labels can be indicative of the class that each unit of the second plurality of units belongs. For example, a unit of the first plurality of units can belong to a treated class and can be assigned a ground truth label of “1.” The ground truth label of “1” can correspond to the treatment class. A unit of the second plurality of units can belong to a control class and can be assigned a ground truth label of “0.” The ground truth label of “0” can correspond to the control class.

[0078] The method 200 can include assigning a plurality of randomized labels. For example, the method 200 can include assigning a plurality of randomized labels to the first plurality of units (e.g., for supervised training of the classifier). A unit of the first plurality of units can be assigned randomized label of “1” or “0.” For example, a first set of randomly assigned labels can include a first unit of the first plurality of units having a randomized label of “0.” The first set of randomly assigned labels can include a second unit of the first plurality of units having a randomized label of “0.” The first set of randomly assigned labels can include a third unit of the first plurality of units having a randomized label of “0.”

[0079] A second set of randomly assigned labels can include the first unit of the first plurality of units having a randomized label of “0.” The second set of randomly assigned labels can include the second unit of the first plurality of units having a randomized label of“0.” The second set of randomly assigned labels can include the third unit of the first plurality of units having a randomized label of “1.”

[0080] A third set of randomly assigned labels can include the first unit of the first plurality of units having a randomized label of “0.” The third set of randomly assigned labels can include the second unit of the first plurality of units having a randomized label of “1.” The third set of randomly assigned labels can include the third unit of the first plurality of units having a randomized label of “1.”

[0081] A fourth set of randomly assigned labels can include the first unit of the first plurality of units having a randomized label of “1.” The fourth set of randomly assigned labels can include the second unit of the first plurality of units having a randomized label of “0.” The fourth set of randomly assigned labels can include the third unit of the first plurality of units having a randomized label of “0.”

[0082] The method 200 can include assigning a plurality of randomized labels to the second plurality of units (e.g., for supervised training of the classifier). A unit of the second plurality of units can be assigned a randomized label of “1” or “0.” The first set of randomly assigned labels can include a first unit of the second plurality of units having a randomized label of “1.” The first set of randomly assigned labels can include a second unit of the second plurality of units having a randomized label of “1.” The first set of randomly assigned labels can include a third unit of the second plurality of units having a randomized label of “1.”

[0083] The second set of randomly assigned labels can include the first unit of the second plurality of units having a randomized label of “0.” The second set of randomly assigned labels can include the second unit of the second plurality of units having a randomized label of “1.” The second set of randomly assigned labels can include the third unit of the second plurality of units having a randomized label of “1.”

[0084] The third set of randomly assigned labels can include the first unit of the second plurality of units having a randomized label of “0.” The third set of randomly assigned labels can include the second unit of the second plurality of units having a randomized label of “0.” The third set of randomly assigned labels can include the third unit of the second plurality of units having a randomized label of “1.”

[0085] The fourth set of randomly assigned labels can include the first unit of the second plurality of units having a randomized label of “0.” The fourth set of randomly assigned labels can include the second unit of the second plurality of units having arandomized label of “1.” The fourth set of randomly assigned labels can include the third unit of the second plurality of units having a randomized label of “1.”

[0086] The plurality of randomized labels can be generated through randomization of the ground truth labels. The plurality of randomized labels can have a structure that corresponds to a structure of the initial dataset. The randomization of the ground truth labels can be performed based on a causal structure of an experiment. For example, the randomization of the ground truth labels can be performed based on a causal structure of the experiment such that the units that share the same cause also share the same label during randomization. The randomization of the ground truth labels can be performed based on a hierarchy of a property of interest of the plurality of units. For example, the randomization of the ground truth labels can be performed based on a hierarchy of a property of interest of the plurality of units such that the units that share the same property also share the same label during randomization. The randomization of the ground truth labels can be performed based on external stimuli to the plurality of units. For example, the randomization of the ground truth labels can be performed based on external stimuli to the plurality of units such that the units that share the same external stimuli also share the same label during randomization.

[0087] The method 200 can include dividing units into a training dataset and a test dataset (BLOCK 915). For example, the method 200 can include dividing the first plurality of units and the second plurality of units into the training dataset and the test dataset. The training dataset can include a plurality of units. The plurality of units can include a first subset of the first plurality of units and a first subset of the second plurality of units. The training dataset can have (e.g., can be used to train the classifier via supervised learning using) the ground truth labels. For example, each unit of the first subset of the first plurality of units can have a ground truth label of “1.” Each unit of the first subset of the second plurality of units can have a ground truth label of “0.”

[0088] The test dataset can include a third plurality of units. The third plurality of units can include a second subset of the first plurality of units and a second subset of the second plurality of units. The test dataset can have the ground truth labels. For example, each unit of the second subset of the first plurality of units can have a ground truth label of “1.” Each unit of the second subset of the second plurality of units can have a ground truth label of

[0089] The method 200 can include training a classifier (BLOCK 920). For example, the method 200 can include training the classifier using the training dataset. The training dataset can include the plurality of units. The method 200 can include training the classifier using the training dataset when the training dataset has the ground truth labels. This classifier can be trained using the observed true class labels (Wobs).

[0090] The method 200 can include training the classifier when the training dataset has the plurality of randomized labels. For example, a first classifier can be trained with the first set of randomly assigned labels (W1). A second classifier can be trained (e.g., via supervised learning) with the second set of randomly assigned labels (W2). The second classifier can be different from the first classifier. A third classifier can be trained with the third set of randomly assigned labels (W3). The third classifier can be different from the first classifier. The third classifier can be different from the second classifier. A fourth classifier can be trained with the fourth set of randomly assigned labels (W3). An zzl11classifier can be trained with an zzl11set of randomly assigned labels. The classifier trained using the observed true class labels can be different from each of the classifiers trained with randomly assigned labels.

[0091] The method 200 can include evaluating the classifier (BLOCK 925). For example, the method 200 can include evaluating the classifier using the test dataset. The test dataset can include the third plurality of units. The method 200 can include determining performance of the classifier. The method 200 can include evaluating the classifier using the test dataset when the test dataset has the ground truth labels. For example, the method 200 can include evaluating the classifier using the test dataset, when the test dataset has the ground truth labels, to generate a first metric. The first metric can include a first performance metric. The first metric can include a first quantifiable property of the classifier. The first metric can include a first parameter of the classifier. The first metric can include a first output of a function of parameters of the classifier. The first metric can be indicative of an accuracy of the classifier.

[0092] The method 200 can include evaluating the classifier using the test dataset when the test dataset has the plurality of randomized labels. For example, the method 200 can include evaluating the classifier using the test dataset when the test dataset has the plurality of randomized labels, to generate a plurality of metrics. The plurality of metrics can include a plurality of performance metrics. The plurality of metrics can include a plurality of quantifiable properties of the classifier. The plurality of metrics can include a plurality ofparameters of the classifier. The plurality of metrics can include a plurality of outputs of the function of parameters of the classifier.

[0093] The method 200 can include evaluating the first classifier using the test dataset when the test dataset has the first set of randomly assigned labels. For example, the method 200 can include evaluating the first classifier using the test dataset when the test dataset has the first set of randomly assigned labels to generate a second metric of the plurality of metrics. The method 200 can include evaluating the second classifier using the test dataset when the test dataset has the second set of randomly assigned labels (e.g., for used in supervised training of the classifier). For example, the method 200 can include evaluating the (e.g., trained) second classifier using the test dataset when the test dataset has the second set of randomly assigned labels to generate a third metric of the plurality of metrics. The method 200 can include evaluating the third classifier using the test dataset when the test dataset has the third set of randomly assigned labels. For example, the method 200 can include evaluating the third classifier using the test dataset when the test dataset has the third set of randomly assigned labels to generate a fourth metric of the plurality of metrics. The method 200 can include evaluating the fourth classifier using the test dataset when the test dataset has the fourth set of randomly assigned labels. For example, the method 200 can include evaluating the fourth classifier using the test dataset when the test dataset has the fourth set of randomly assigned labels to generate a fifth metric of the plurality of metrics.

[0094] The method 200 can include calculating a p-value (BLOCK 930). For example, the method 200 can include calculating the p-value based on the first metric and the plurality of metrics. The method 200 can include calculating the p-value based on the first metric, the second metric of the plurality of metrics, the third metric of the plurality of metrics, the fourth metric of the plurality of metrics, and the fifth metric of the plurality of metrics. The p-value can be defined by a ratio of the plurality of metrics greater than or equal to the performance metric. The p-value can be defined by a ratio of the plurality of performance metrics greater than or equal to the first performance metric. The p-value can be defined by a ratio of the plurality of quantifiable properties of the classifier 110 greater than or equal to the first quantifiable property of the classifier 110. The p-value can be defined by a ratio of the plurality of parameters of the classifier 110 greater than or equal to the first parameter of the classifier 110. The p-value can be defined by a ratio of the plurality of outputs of a function of parameters of the classifier 110 greater than or equal to the first output of the function of parameters of the classifier 110.

[0095] The method 200 can include establishing a causal relationship between a treatment provided to the first class and a feature of the first class. The method 200 can include representing the first metric and the plurality of metrics in a histogram (e.g., the histograms of FIGS. 2, 3 A, 3B, 5B, 5C, 7D, 7E, or 8). The histogram can have a plot of a plurality of accuracy values. The method 200 can include extracting a plurality of features using a pre-existing training dataset when the training dataset has no labels. The pre-existing training dataset can include the plurality of units.

[0096] The method 200 can include training the classifier to extract a plurality of features using the training dataset when the training dataset has no labels. This can correspond to training the network using a deep encoder. The deep encoder can be trained without using class labels. The deep encoder can include training a representation network. The method 200 can include training the classifier using the plurality of features and the training dataset when the training dataset has the plurality of randomized labels. This can correspond to training the network using a shallow classification head. The deep encoder can be trained using class labels.

[0097] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more circuits of computer program instructions, encoded on one or more computer storage media for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices).

[0098] The operations described in this specification can be performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources. The term “data processing apparatus” or “computing device” encompasses various apparatuses, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0099] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a circuit, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more circuits, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0100] Processors suitable for the execution of a computer program include, by way of example, microprocessors, and any one or more processors of a digital computer. A processor can receive instructions and data from a read only memory or a random-access memory or both. The elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. A computer can include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto opticaldisks, or optical disks. A computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a personal digital assistant (PDA), a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0101] To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0102] The implementations described herein can be implemented in any of numerous ways including, for example, using hardware, software or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers.

[0103] Also, a computer may have one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that can be used to provide a user interface include printers or display screens for visual presentation of output and speakers or other sound generating devices for audible presentation of output. Examples of input devices that can be used for a user interface include keyboards, and pointing devices, such as mice, touch pads, and digitizing tablets. As another example, a computer may receive input information through speech recognition or in other audible format.

[0104] Such computers may be interconnected by one or more networks in any suitable form, including a local area network or a wide area network, such as an enterprise network, and intelligent network (IN) or the Internet. Such networks may be based on any suitable technology and may operate according to any suitable protocol and may include wireless networks, wired networks or fiber optic networks.

[0105] A computer employed to implement at least a portion of the functionality described herein may comprise a memory, one or more processing units (also referred to herein simply as “processors”), one or more communication interfaces, one or more display units, and one or more user input devices. The memory may comprise any computer-readable media, and may store computer instructions (also referred to herein as “processor-executable instructions”) for implementing the various functionalities described herein. The processing unit(s) may be used to execute the instructions. The communication interface(s) may be coupled to a wired or wireless network, bus, or other communication means and may therefore allow the computer to transmit communications to or receive communications from other devices. The display unit(s) may be provided, for example, to allow a user to view various information in connection with execution of the instructions. The user input device(s) may be provided, for example, to allow the user to make manual adjustments, make selections, enter data or various other information, or interact in any of a variety of manners with the processor during execution of the instructions.

[0106] The various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine.

[0107] In this respect, various inventive concepts may be embodied as a computer readable storage medium (or multiple computer readable storage media) (e.g., a computer memory, one or more floppy discs, compact discs, optical discs, magnetic tapes, flash memories, circuit configurations in Field Programmable Gate Arrays or other semiconductor devices, or other non-transitory medium or tangible computer storage medium) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement the various embodiments of the solution discussed above. The computer readable medium or media can be transportable, such that the program orprograms stored thereon can be loaded onto one or more different computers or other processors to implement various aspects of the present solution as discussed above.

[0108] The terms “program” or “software” are used herein to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects of embodiments as discussed above. One or more computer programs that when executed perform methods of the present solution need not reside on a single computer or processor, but may be distributed in a modular fashion amongst a number of different computers or processors to implement various aspects of the present solution.

[0109] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Program modules can include routines, programs, objects, components, data structures, or other components that perform particular tasks or implement particular abstract data types. The functionality of the program modules can be combined or distributed as desired in various embodiments.

[0110] Also, data structures may be stored in computer-readable media in any suitable form. For simplicity of illustration, data structures may be shown to have fields that are related through location in the data structure. Such relationships may likewise be achieved by assigning storage for the fields with locations in a computer-readable medium that convey relationship between the fields. However, any suitable mechanism may be used to establish a relationship between information in fields of a data structure, including through the use of pointers, tags or other mechanisms that establish relationship between data elements.

[0111] Any references to implementations or elements or acts of the systems and methods herein referred to in the singular can include implementations including a plurality of these elements, and any references in plural to any implementation or element or act herein can include implementations including only a single element. References in the singular or plural form are not intended to limit the presently disclosed systems or methods, their components, acts, or elements to single or plural configurations. References to any act or element being based on any information, act or element may include implementations where the act or element is based at least in part on any information, act, or element.

[0112] Any implementation disclosed herein may be combined with any other implementation, and references to “an implementation,” “some implementations,” “an alternate implementation,” “various implementations,” “one implementation” or the like arenot necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the implementation may be included in at least one implementation. Such terms as used herein are not necessarily all referring to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

[0113] References to “or” may be construed as inclusive so that any terms described using “or” may indicate any of a single, more than one, and all of the described terms. References to at least one of a conjunctive list of terms may be construed as an inclusive OR to indicate any of a single, more than one, and all of the described terms. For example, a reference to “at least one of ‘A’ and ‘B’” can include only ‘A’, only ‘B’, as well as both ‘A’ and ‘B’. Elements other than ‘A’ and ‘B’ can also be included.

[0114] As used herein, the singular terms “a,” “an,” and “the” may include plural referents unless the context clearly dictates otherwise. Spatial descriptions, such as “above,” “below,” “up,” “left,” “right,” “down,” “top,” “bottom,” “vertical,” “horizontal,” “side,” “higher,” “lower,” “upper,” “over,” “under,” and so forth, are indicated with respect to the orientation shown in the figures unless otherwise specified. It should be understood that the spatial descriptions used herein are for purposes of illustration only, and that practical implementations of the structures described herein can be spatially arranged in any orientation or manner, provided that the merits of embodiments of this disclosure are not deviated by such arrangement.

[0115] As used herein, the terms “approximately,” “substantially,” “substantial” and “about” are used to describe and account for small variations. When used in conjunction with an event or circumstance, the terms can refer to instances in which the event or circumstance occurs precisely as well as instances in which the event or circumstance occurs to a close approximation. For example, when used in conjunction with a numerical value, the terms can refer to a range of variation less than or equal to ±10% of that numerical value, such as less than or equal to ±5%, less than or equal to ±4%, less than or equal to ±3%, less than or equal to ±2%, less than or equal to ±1%, less than or equal to ±0.5%, less than or equal to ±0.1%, or less than or equal to ±0.05%. For example, two numerical values can be deemed to be “substantially” the same if a difference between the values is less than or equal to ±10% of an average of the values, such as less than or equal to ±5%, less than or equal to ±4%, less thanor equal to ±3%, less than or equal to ±2%, less than or equal to ±1%, less than or equal to ±0.5%, less than or equal to ±0.1%, or less than or equal to ±0.05%.

[0116] Additionally, amounts, ratios, and other numerical values are sometimes presented herein in a range format. It is to be understood that such range format is used for convenience and brevity and should be understood flexibly to include numerical values explicitly specified as limits of a range, but also to include all individual numerical values or sub-ranges encompassed within that range as if each numerical value and sub-range is explicitly specified.

[0117] The systems and methods described herein may be embodied in other specific forms without departing from the characteristics thereof. The foregoing implementations are illustrative rather than limiting of the described systems and methods.

[0118] Where technical features in the drawings, detailed description or any claim are followed by reference signs, the reference signs have been included to increase the intelligibility of the drawings, detailed description, and claims. Accordingly, neither the reference signs nor their absence have any limiting effect on the scope of any claim elements.

[0119] The systems and methods described herein may be embodied in other specific forms without departing from the characteristics thereof. The foregoing implementations are illustrative rather than limiting of the described systems and methods. Scope of the systems and methods described herein is thus indicated by the appended claims, rather than the foregoing description, and changes that come within the meaning and range of equivalency of the claims are embraced therein.

[0120] While the present disclosure has been described and illustrated with reference to specific embodiments thereof, these descriptions and illustrations do not limit the present disclosure. It should be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the true spirit and scope of the present disclosure as defined by the appended claims. The illustrations may not be necessarily drawn to scale. There may be distinctions between the artistic renditions in the present disclosure and the actual apparatus due to manufacturing processes and tolerances. There may be other embodiments of the present disclosure which are not specifically illustrated. The specification and drawings are to be regarded as illustrative rather than restrictive. Modifications may be made to adapt a particular situation, material, compositionof matter, method, or process to the objective, spirit and scope of the present disclosure. All such modifications are intended to be within the scope of the claims appended hereto. While the methods disclosed herein have been described with reference to particular operations performed in a particular order, it will be understood that these operations may be combined, sub-divided, or re-ordered to form an equivalent method without departing from the teachings of the present disclosure. Accordingly, unless specifically indicated herein, the order and grouping of the operations are not limitations of the present disclosure.

Claims

WHAT IS CLAIMED IS:

1. An apparatus, comprising: a computational imaging system configured to output a plurality of features; one or more processors configured to: receive the plurality of features; apply the plurality of features as input to a classifier to cause the classifier to output a first performance metric, the classifier configured based on a plurality of units comprising a first subset of a first plurality of units and a first subset of a second plurality of units, when the plurality of units has ground truth labels; apply the plurality of features as input to the classifier to cause the classifier to output a plurality of performance metrics, when the plurality of units has a plurality of randomized labels, the plurality of randomized labels generated through randomization of the ground truth labels; and calculate a p-value based on the first performance metric and the plurality of performance metrics.

2. The apparatus of claim 1, wherein the computational imaging system comprises: a measurement system configured to output a plurality of images; and an encoder configured to: receive the plurality of images from the measurement system; and output the plurality of features.

3. The apparatus of claim 1, wherein the computational imaging system comprises: a measurement system configured to capture spectral information for each pixel of a plurality of images.

4. The apparatus of claim 3, wherein the computational imaging system comprises: an encoder configured to: receive the plurality of images from the measurement system; and output the plurality of features.

5. The apparatus of claim 1, wherein the computational imaging system comprises:a coded aperture configured to receive light from a light source and output structured light; and a camera configured to receive the structured light from the coded aperture.

6. The apparatus of claim 5, wherein the structured light is configured to pass through a sample before being received by the camera.

7. The apparatus of claim 5, wherein the structured light is configured to illuminate a sample and the camera is configured to receive emission light from the sample.

8. The apparatus of claim 5, wherein the computational imaging system comprises: a de-magnifier configured to receive the structured light from the coded aperture and output de-magnified structured light to the camera.

9. The apparatus of claim 5, wherein the computational imaging system comprises: a beam expander configured to receive the light from the light source and output light with an expanded width to the coded aperture.

10. The apparatus of claim 1, wherein the p-value is defined by a ratio of the plurality of performance metrics greater than or equal to the first performance metric.

11. The apparatus of claim 1, wherein the plurality of features comprises a plurality of three- dimensional (3D) features and the computational imaging system comprises: a measurement system configured to output a plurality of images; and an encoder configured to: receive the plurality of images from the measurement system; and output the plurality of 3D features.

12. The apparatus of claim 1, wherein the plurality of features comprises a plurality of temporal features and the computational imaging system comprises: a measurement system configured to output a plurality of images; and an encoder configured to: receive the plurality of images from the measurement system; and output the plurality of temporal features.

13. The apparatus of claim 1, wherein the computational imaging system comprises: a Fourier filter configured to receive a light field from a light source, modulate a phase of the light field, and output a modulated light field; and a camera configured to receive the modulated light field from the Fourier filter.

14. The apparatus of claim 1, wherein the one or more processors are configured to: divide the first plurality of units and the second plurality of units into: a training dataset comprising the plurality of units, the training dataset having ground truth labels; and a test dataset comprising a third plurality of units comprising a second subset of the first plurality of units and a second subset of the second plurality of units, the test dataset having the ground truth labels.

15. The apparatus of claim 1, wherein: the first plurality of units and the second plurality of units are obtained from an initial dataset; and the initial dataset comprises a first class comprising the first plurality of units and a second class comprising the second plurality of units.

16. The apparatus of claim 15, wherein the first class comprises a treatment class and the second class comprises a control class.

17. The apparatus of claim 15, wherein the initial dataset comprises a plurality of images.

18. The apparatus of claim 15, wherein the plurality of randomized labels has a structure that corresponds to a structure of the initial dataset.

19. The apparatus of claim 1, wherein the randomization of the ground truth labels is performed based on a causal structure of an experiment.

20. The apparatus of claim 1, wherein the randomization of the ground truth labels is performed based on a hierarchy of a property of interest of the plurality of units.

21. The apparatus of claim 1, wherein the randomization of the ground truth labels is performed based on external stimuli to the plurality of units.

22. A method, comprising: organizing an initial dataset into at least a first class comprising a first plurality of units and a second class comprising a second plurality of units; assigning ground truth labels to the first plurality of units and the second plurality of units; dividing the first plurality of units and the second plurality of units into: a training dataset comprising a plurality of units comprising a first subset of the first plurality of units and a first subset of the second plurality of units, the training dataset having the ground truth labels; and a test dataset comprising a third plurality of units comprising a second subset of the first plurality of units and a second subset of the second plurality of units, the test dataset having the ground truth labels; training a classifier using the training dataset when the training dataset has the ground truth labels; evaluating the classifier using the test dataset when the test dataset has the ground truth labels, to generate a first metric; assigning a plurality of randomized labels to the first plurality of units and the second plurality of units, the plurality of randomized labels generated through randomization of the ground truth labels; training the classifier when the training dataset has the plurality of randomized labels; evaluating the classifier using the test dataset when the test dataset has the plurality of randomized labels, to generate a plurality of metrics; and calculating a p-value based on the first metric and the plurality of metrics.

23. The method of claim 22, wherein the first metric is a first performance metric and the plurality of metrics is a plurality of performance metrics.

24. The method of claim 22, wherein the first metric is a first quantifiable property of the classifier and the plurality of metrics is a plurality of quantifiable properties of the classifier.

25. The method of claim 22, wherein the first metric is a first parameter of the classifier and the plurality of metrics is a plurality of parameters of the classifier.

26. The method of claim 22, wherein the first metric is a first output of a function of parameters of the classifier and the plurality of metrics is a plurality of outputs of the function of parameters of the classifier.

27. The method of claim 22, wherein the first class comprises a treatment class and the second class comprises a control class.

28. The method of claim 22, wherein the initial dataset comprises a plurality of images.

29. The method of claim 22, comprising: establishing a causal relationship between a treatment provided to the first class and a feature of the first class.

30. The method of claim 22, wherein the first metric is indicative of an accuracy of the classifier.

31. The method of claim 22, comprising: representing the first metric and the plurality of metrics in a histogram.

32. The method of claim 22, comprising: training the classifier to extract a plurality of features using the training dataset when the training dataset has no labels; and training the classifier using the plurality of features and the training dataset when the training dataset has the plurality of randomized labels.

33. The method of claim 22, comprising: extracting a plurality of features using a pre-existing training dataset when the training dataset has no labels.

Images