Biological Image Conversion Using a Machine Learning Model

Abstract

To provide biological image translation using suitable machine learning models.SOLUTION: A system and method trains a machine learning model and generates images of biological samples, and the system and method generates enhanced images of the biological samples. The method for training the machine learning model and generating the images of the biological sample may include a step for obtaining a plurality of training images comprising a first type of training images and a second type of training images. The method may also include steps for generating a plurality of wavelet coefficients using the machine learning model based on the first type of training images, generating a second type of synthetic image based on the plurality of wavelet coefficients, comparing the second type of synthetic image to the second type of training images, and updating the machine learning model based on the comparison.SELECTED DRAWING: Figure 3B

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of priority of U.S. Provisional Application No. 63 / 075,751, filed on September 8, 2020, and U.S. Provisional Application No. 63 / 143,707, filed on January 29, 2021, the entire contents of which are incorporated herein by reference in their entirety.

[0002] The present disclosure generally relates to machine learning techniques, and more specifically, to low - cost machine - learning - based generation of image data at scale. The generated image data (e.g., image data of biological samples) can provide sufficient richness and depth for optimized downstream processing (e.g., phenotyping). Some embodiments of the system include a programmable spatial light modulator ("SLM") that produces, without mechanical modification to the system, data optimized for downstream processing (e.g., phenotyping) at high speed. Some embodiments of the system include a machine - learning model with an attention layer that includes a plurality of weights corresponding to a plurality of illumination settings (e.g., different illumination emitters of an illumination source) for identifying an optimal illumination pattern for capturing image data. Some embodiments of the system include techniques for evaluating candidate treatments for a disease of interest.

Background Art

[0003] Brightfield images of biological samples can be acquired on a large scale and at low cost due to inexpensive equipment, ease of clinical deployment, and low processing and memory resource requirements for the captured images. Acquiring brightfield images is generally non-invasive and involves low phototoxicity. However, low-contrast images lack rich visual details and are thus not suitable for many downstream analyses (e.g., phenotype exploration of microscopic samples). In comparison, other imaging modalities (e.g., fluorescence images) can provide rich visual information of the captured samples. However, acquiring fluorescence images requires additional equipment and materials, can be time-consuming, and can be computationally resource-intensive. Thus, fluorescence images can be difficult to acquire in a large-scale and low-cost manner.

[0004] Converting brightfield images of biological samples into enhanced high-quality images can be difficult for several reasons. First, brightfield images suffer from an inherent class imbalance problem (i.e., rich low-frequency signals but fewer high-frequency signals). Further, the overall geometric shape of the brightfield image needs to be extracted and maintained through the conversion. Additionally, many factors such as the illumination pattern under which the brightfield image is captured can affect the effectiveness of the conversion. Moreover, the robustness of the conversion in supporting downstream analysis needs to be quantified and verified. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0005] What is described is a system and method for training a machine learning model and generating an image of a biological sample. Also described is a system and method for generating an enhanced image of a biological sample. The present system and method can be used, for example, to obtain a first type of image, such as a bright-field image type. The acquired image of the first image type can then be used by the present system and method to generate a second type of synthetic image, such as a fluorescence image.

[0006] In some embodiments, a method for training a machine learning model and generating an image of a biological sample includes obtaining a plurality of training images. The plurality of images comprises a first type of training image and a second type of training image. The method also includes generating a plurality of wavelet coefficients using a machine learning model based on the first type of training image, generating a second type of synthetic image based on the plurality of wavelet coefficients, comparing the second type of synthetic image with the second type of training image, and updating the machine learning model based on the comparison.

[0007] In some embodiments, a method for generating an enhanced image of a biological sample includes obtaining an image of the biological sample using a microscope and generating an enhanced image of the biological sample using a machine learning model based on the image. The machine learning model may be trained by obtaining a plurality of training images comprising a first type of training image and a second type of training image, generating a plurality of wavelet coefficients using the machine learning model based on the first type of training image, generating a second type of synthetic image based on the plurality of wavelet coefficients, comparing the second type of synthetic image with the second type of training image, and updating the machine learning model based on the comparison.

[0008] In some embodiments, a system for training a machine learning model and generating an image of a biological sample comprises a computing system having one or more processors and one or more memories storing the machine learning model, the computing system being configured to receive a plurality of training images of a first type and one training image of a second type. The computing system is configured to generate a plurality of wavelet coefficients using the machine learning model based on the training images of the first type, generate a synthetic image of the second type based on the plurality of wavelet coefficients, compare the synthetic image of the second type with the training image of the second type, and update the machine learning model based on the comparison.

[0009] In some embodiments, a system for generating an enhanced image of a biological sample comprises a computing system having one or more processors and one or more memories storing the machine learning model. The computing system is configured to receive an image of the biological sample obtained from a microscope and generate an enhanced image of the biological sample using the machine learning model based on the image. The machine learning model may be trained by steps including obtaining a plurality of training images comprising a training image of a first type and a training image of a second type, generating a plurality of wavelet coefficients using the machine learning model based on the training image of the first type, generating a synthetic image of the second type based on the plurality of wavelet coefficients, comparing the synthetic image of the second type with the training image of the second type, and updating the machine learning model based on the comparison.

[0010] A method for training an exemplary machine learning model and generating an image of a biological sample includes obtaining a plurality of training images including a first type of training image and a second type of training image, generating a plurality of wavelet coefficients using the machine learning model based on the first type of training image, generating a second type of synthetic image based on the plurality of wavelet coefficients, comparing the second type of synthetic image with the second type of training image, and updating the machine learning model based on the comparison.

[0011] In some embodiments, the first type of training image is a bright-field image of a biological sample.

[0012] In some embodiments, the second type of training image is a fluorescence image of a biological sample.

[0013] In some embodiments, the machine learning model includes a generator and a discriminator.

[0014] In some embodiments, the machine learning model includes a conditional GAN model.

[0015] In some embodiments, the generator includes a plurality of neural networks corresponding to a plurality of frequency groups.

[0016] In some embodiments, each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group.

[0017] In some embodiments, the plurality of neural networks includes a plurality of U-Net neural networks.

[0018] In some embodiments, the discriminator is a PatchGAN neural network.

[0019] In some embodiments, the method further includes generating a third type of image based on the first type of training image.

[0020] In some embodiments, the third type of image is a phase shift image.

[0021] In some embodiments, the method further includes generating a fourth type of image based on the first type of training image.

[0022] In some embodiments, the fourth type of image comprises segmented data.

[0023] In some embodiments, the first type of training image is captured using a microscope according to a first illumination scheme.

[0024] In some embodiments, the first illumination scheme comprises one or more illumination patterns.

[0025] In some embodiments, the first type of training image is part of a brightfield image array.

[0026] In some embodiments, the plurality of training images are a first plurality of training images, and the method further includes identifying a second illumination scheme based on a comparison, and obtaining a second plurality of training images comprising one or more images of the first type, wherein the one or more images of the first type are obtained based on the second illumination scheme, and training a machine learning model based on the second plurality of training images.

[0027] In some embodiments, the method further includes obtaining, using a microscope, a plurality of images of the first type, and generating, using a machine learning model based on the obtained plurality of images, a plurality of composite images of the second type.

[0028] In some embodiments, the method further includes training a classifier based on a plurality of synthetic images of a second type.

[0029] In some embodiments, the microscope is a first microscope, the classifier is a first classifier, and the method further includes obtaining, using a second microscope, a plurality of images of a second type; training a second classifier based on the plurality of images of the second type; and comparing the performance of the first classifier and the second classifier.

[0030] In some embodiments, the second microscope is a fluorescence microscope.

[0031] An exemplary method for generating an enhanced image of a biological sample includes obtaining an image of the biological sample using a microscope; and generating, based on the image and using a machine learning model, an enhanced image of the biological sample, wherein the machine learning model is trained by: obtaining a plurality of training images including a first type of training image and a second type of training image; generating, based on the first type of training image and using the machine learning model, a plurality of wavelet coefficients; generating, based on the plurality of wavelet coefficients, a synthetic image of the second type; comparing the synthetic image of the second type with the second type of training image; and updating the machine learning model based on the comparison.

[0032] In some embodiments, the first type of training image is a bright-field image of the biological sample.

[0033] In some embodiments, the second type of training image is a fluorescence image of the biological sample.

[0034] In some embodiments, the machine learning model includes a generator and a discriminator.

[0035] In some embodiments, the machine learning model includes a conditional GAN model.

[0036] In some embodiments, the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.

[0037] In some embodiments, each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group.

[0038] In some embodiments, the plurality of neural networks comprises a plurality of U-Net neural networks.

[0039] In some embodiments, the discriminator is a PatchGAN neural network.

[0040] In some embodiments, the method further comprises generating a third type of image based on the first type of training image.

[0041] In some embodiments, the third type of image is a phase shift image.

[0042] In some embodiments, the method further comprises generating a fourth type of image based on the first type of training image.

[0043] In some embodiments, the fourth type of image comprises segmented data.

[0044] In some embodiments, the first type of training image is captured using a microscope according to a first illumination scheme.

[0045] In some embodiments, the first illumination scheme comprises one or more illumination patterns.

[0046] In some embodiments, the first type of training image is part of a bright field image array.

[0047] In some embodiments, the plurality of training images are a first plurality of training images, and the method further includes identifying, based on a comparison, a second illumination scheme; obtaining a second plurality of training images comprising one or more images of a first type, wherein the one or more images of the first type are obtained based on the second illumination scheme; and training a machine learning model based on the second plurality of training images.

[0048] In some embodiments, the method further includes obtaining, using a microscope, a plurality of images of a first type; and generating, using a machine learning model based on the plurality of obtained images, a plurality of synthetic images of a second type.

[0049] In some embodiments, the method further includes training a classifier based on the plurality of synthetic images of the second type.

[0050] In some embodiments, the microscope is a first microscope, the classifier is a first classifier, and the method further includes obtaining, using a second microscope, a plurality of images of a second type; training a second classifier based on the plurality of images of the second type; and comparing the performance of the first classifier and the second classifier.

[0051] In some embodiments, the second microscope is a fluorescence microscope.

[0052] A system for training an exemplary machine learning model and generating images of biological samples is a computing system comprising one or more processors and one or more memories storing the machine learning model, the computing system being configured to receive a plurality of training images of a first type and one training image of a second type, generate a plurality of wavelet coefficients using the machine learning model based on the training images of the first type, generate a composite image of the second type based on the plurality of wavelet coefficients, compare the composite image of the second type with the training image of the second type, and update the machine learning model based on the comparison.

[0053] In some embodiments, the training images of the first type are bright-field images of biological samples.

[0054] In some embodiments, the training images of the second type are fluorescence images of biological samples.

[0055] In some embodiments, the machine learning model comprises a generator and a discriminator.

[0056] In some embodiments, the machine learning model comprises a conditional GAN model.

[0057] In some embodiments, the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.

[0058] In some embodiments, each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group.

[0059] In some embodiments, the plurality of neural networks comprises a plurality of U-Net neural networks.

[0060] In some embodiments, the discriminator is a PatchGAN neural network.

[0061] In some embodiments, the computing system is further configured to generate a third type of image based on the first type of training image.

[0062] In some embodiments, the third type of image is a phase-shifted image.

[0063] In some embodiments, the computing system is further configured to generate a fourth type of image based on the first type of training image.

[0064] In some embodiments, the fourth type of image comprises segmented data.

[0065] In some embodiments, the first type of training image is captured using a microscope according to a first illumination scheme.

[0066] In some embodiments, the first illumination scheme comprises one or more illumination patterns.

[0067] In some embodiments, the first type of training image is part of a brightfield image array.

[0068] In some embodiments, the plurality of training images are a first plurality of training images, and the computing system is further configured to identify a second illumination scheme based on a comparison, obtain a second plurality of training images comprising one or more images of the first type, wherein the one or more images of the first type are obtained based on the second illumination scheme, and train a machine learning model based on the second plurality of training images.

[0069] In some embodiments, the computing system is further configured to use a microscope to acquire a plurality of images of a first type and, based on the acquired plurality of images, use a machine learning model to generate a plurality of synthetic images of a second type.

[0070] In some embodiments, the computing system is further configured to train a classifier based on the plurality of synthetic images of the second type.

[0071] In some embodiments, the microscope is a first microscope, the classifier is a first classifier, and the computing system is further configured to use a second microscope to acquire a plurality of images of the second type, train a second classifier based on the plurality of images of the second type, and compare the performance of the first classifier and the second classifier.

[0072] In some embodiments, the second microscope is a fluorescence microscope.

[0073] A system for generating enhanced images of exemplary biological samples includes a computing system comprising one or more processors and one or more memories storing a machine learning model, the computing system being configured to receive an image of a biological sample acquired from a microscope and, based on the image, use the machine learning model to generate an enhanced image of the biological sample, wherein the machine learning model is trained by steps of: acquiring a plurality of training images comprising a first type of training image and a second type of training image; using the machine learning model to generate a plurality of wavelet coefficients based on the first type of training image; generating a second type of synthetic image based on the plurality of wavelet coefficients; comparing the second type of synthetic image with the second type of training image; and updating the machine learning model based on the comparison.

[0074] In some embodiments, the first type of training image is a bright-field image of a biological sample.

[0075] In some embodiments, the second type of training image is a fluorescence image of a biological sample.

[0076] In some embodiments, the machine learning model includes a generator and a discriminator.

[0077] In some embodiments, the machine learning model includes a conditional GAN model.

[0078] In some embodiments, the generator includes a plurality of neural networks corresponding to a plurality of frequency groups.

[0079] In some embodiments, each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group.

[0080] In some embodiments, the plurality of neural networks includes a plurality of U-Net neural networks.

[0081] In some embodiments, the discriminator is a PatchGAN neural network.

[0082] In some embodiments, the machine learning model is further trained by a step of generating a third type of image based on the first type of training image.

[0083] In some embodiments, the third type of image is a phase-shift image.

[0084] In some embodiments, the machine learning model is trained by a step of generating a fourth type of image based on the first type of training image.

[0085] In some embodiments, the fourth type of image comprises partitioned data.

[0086] In some embodiments, the first type of training image is captured using a microscope according to a first illumination scheme.

[0087] In some embodiments, the first illumination scheme comprises one or more illumination patterns.

[0088] In some embodiments, the first type of training image is part of a brightfield image array.

[0089] In some embodiments, the plurality of training images are a first plurality of training images, and the machine learning model comprises identifying a second illumination scheme based on a comparison, and obtaining a second plurality of training images comprising one or more images of the first type, wherein the one or more images of the first type are obtained based on the second illumination scheme, and training the machine learning model based on the second plurality of training images.

[0090] In some embodiments, the machine learning model is trained by obtaining, using a microscope, a plurality of images of the first type, and generating, using the machine learning model based on the plurality of obtained images, a plurality of composite images of the second type.

[0091] In some embodiments, the machine learning model is trained by training a classifier based on the plurality of composite images of the second type.

[0092] In some embodiments, the microscope is a first microscope, the classifier is a first classifier, and the machine learning model is trained by: using a second microscope to acquire a plurality of images of a second type; training a second classifier based on the plurality of images of the second type; and comparing the performance of the first classifier and the second classifier.

[0093] In some embodiments, the second microscope is a fluorescence microscope.

[0094] A method of processing an image of an exemplary biological sample and obtaining one or more output images includes: using a plurality of configurations of a spatial light modulator (SLM) of an optical system to acquire a plurality of images of the biological sample, wherein the SLM is positioned in an optical path between the biological sample and an image recording device; and inputting the plurality of images of the biological sample into a trained machine learning model to obtain one or more output images.

[0095] In some embodiments, at least one of the plurality of configurations of the SLM is for generating one or more optical aberrations.

[0096] In some embodiments, the step of generating one or more optical aberrations includes spherical aberration, astigmatism, defocus, distortion, tilt, or any combination thereof.

[0097] In some embodiments, at least one of the plurality of configurations of the SLM is for enhancing one or more features.

[0098] In some embodiments, the one or more features include cell boundaries, actin filaments, nuclear shape, cytoplasmic compartmentalization, or any combination thereof.

[0099] In some embodiments, at least one of the plurality of configurations of the SLM is for reducing optical aberrations.

[0100] In some embodiments, the plurality of SLM configurations are for acquiring images of a biological sample at different depths.

[0101] In some embodiments, the machine learning model is configured to generate a second type of image based on a first type of image.

[0102] In some embodiments, the first type of image is a brightfield image.

[0103] In some embodiments, the second type of image is a fluorescence image.

[0104] In some embodiments, the second type of image is an enhanced version of the first type of image.

[0105] In some embodiments, the machine learning model is a GAN model or a self-supervised model.

[0106] In some embodiments, the plurality of images are acquired using a plurality of configurations of a light source of an optical system.

[0107] In some embodiments, the light source is an LED array of the optical system.

[0108] In some embodiments, at least one of the plurality of SLM configurations is obtained by a step of training a machine learning model, a step of evaluating the trained machine learning model, and a step of identifying at least one configuration based on the evaluation.

[0109] In some embodiments, the trained machine learning model is configured to receive an input image and output an enhanced version of the input image.

[0110] In some embodiments, the enhanced version of the input image comprises one or more enhanced cell phenotypes.

[0111] An electronic device for processing an image of an exemplary biological sample and obtaining one or more output images comprises one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for using a plurality of configurations of an SLM of an optical system to obtain a plurality of images of the biological sample, wherein the SLM is located in an optical path between the biological sample and an image recording device, and inputting the plurality of images of the biological sample into a trained machine learning model to obtain one or more output images.

[0112] An exemplary non-transitory computer-readable storage medium stores one or more programs for processing an image of a biological sample and obtaining one or more output images, and when the one or more programs are executed by one or more processors of an electronic device, the electronic device is caused to use a plurality of configurations of an SLM of an optical system to obtain a plurality of images of the biological sample, wherein the SLM is located in an optical path between the biological sample and an image recording device, and input the plurality of images of the biological sample into a trained machine learning model to obtain one or more output images.

[0113] An exemplary method of classifying an image of a biological sample includes using a plurality of configurations of an SLM of an optical system to obtain a plurality of images of the biological sample, wherein the SLM is located in an optical path between the biological sample and an image recording device, and inputting the plurality of images of the biological sample into a trained machine learning model to obtain one or more classification outputs.

[0114] In some embodiments, at least one of the plurality of configurations of the SLM is for generating one or more optical aberrations.

[0115] In some embodiments, the step of generating one or more optical aberrations includes spherical aberration, astigmatism, defocus, distortion, tilt, or any combination thereof.

[0116] In some embodiments, at least one of the plurality of configurations of the SLM is for enhancing one or more features.

[0117] In some embodiments, one or more features include cell boundaries, actin filaments, nuclear shape, cytoplasmic compartmentalization, or any combination thereof.

[0118] In some embodiments, at least one of the plurality of configurations of the SLM is for reducing optical aberrations.

[0119] In some embodiments, the plurality of SLM configurations are for acquiring images of a biological sample at different depths.

[0120] In some embodiments, the plurality of images are acquired using a plurality of configurations of a light source of an optical system.

[0121] In some embodiments, the light source is an LED array of the optical system.

[0122] In some embodiments, at least one of the plurality of configurations of the SLM is obtained by training a machine learning model, evaluating the trained machine learning model, and identifying at least one configuration based on the evaluation.

[0123] In some embodiments, the trained machine learning model is configured to receive an input image and detect one or more pre-defined objects within the input image.

[0124] In some embodiments, the pre-defined objects include diseased tissue.

[0125] An electronic device for classifying images of exemplary biological samples includes one or more processors, a memory, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors, and include instructions for using a plurality of configurations of an SLM of an optical system to obtain a plurality of images of a biological sample, where the SLM is located in an optical path between the biological sample and an image recording device, and for inputting the plurality of images of the biological sample into a trained machine learning model to obtain one or more classification outputs.

[0126] An exemplary non-transitory computer-readable storage medium stores one or more programs for classifying images of biological samples. When the one or more programs are executed by one or more processors of an electronic device, the electronic device is caused to use a plurality of configurations of an SLM of an optical system to obtain a plurality of images of a biological sample, where the SLM is located in an optical path between the biological sample and an image recording device, and to input the plurality of images of the biological sample into a trained machine learning model to obtain one or more classification outputs.

[0127] An exemplary method for training a machine learning model includes using a plurality of configurations of an SLM of an optical system to obtain a plurality of images of a biological sample, where the SLM is located in an optical path between the biological sample and an image recording device, and training the machine learning model using the plurality of images.

[0128] In some embodiments, at least one of the plurality of configurations of the SLM is for generating one or more optical aberrations.

[0129] In some embodiments, the step of generating one or more optical aberrations includes spherical aberration, astigmatism, defocus, distortion, tilt, or any combination thereof.

[0130] In some embodiments, at least one of the plurality of configurations of the SLM is for enhancing one or more features.

[0131] In some embodiments, one or more features include cell boundaries, actin filaments, nuclear shape, cytoplasmic compartmentalization, or any combination thereof.

[0132] In some embodiments, at least one of the plurality of configurations of the SLM is for reducing optical aberrations.

[0133] In some embodiments, at least one of the plurality of configurations of the SLM is for acquiring images of a biological sample at different depths.

[0134] In some embodiments, the machine learning model is configured to generate a second type of image based on a first type of image.

[0135] In some embodiments, the first type of image is a bright-field image.

[0136] In some embodiments, the second type of image is a fluorescence image.

[0137] In some embodiments, the machine learning model is a GAN model or a self-supervised model.

[0138] In some embodiments, the machine learning model is a classification model.

[0139] In some embodiments, the plurality of images are acquired using a plurality of configurations of a light source of an optical system.

[0140] In some embodiments, the light source is an LED array of the optical system.

[0141] In some embodiments, the step of training the machine learning model includes: (a) training the machine learning model using a first image, where the first image is acquired using a first configuration of an SLM of the optical system; (b) evaluating the trained machine learning model; (c) identifying a second configuration of the SLM based on the evaluation; and (d) training the machine learning model using a second image, where the second image is acquired using the second configuration of the SLM of the optical system.

[0142] In some embodiments, the evaluation is based on a loss function of the machine learning model.

[0143] In some embodiments, the method further includes repeating steps (a)-(d) until a threshold is met.

[0144] In some embodiments, the threshold indicates convergence of the training.

[0145] In some embodiments, the trained machine learning model is configured to receive an input image and output an enhanced version of the input image.

[0146] In some embodiments, the enhanced version of the input image comprises one or more enhanced cell phenotypes.

[0147] In some embodiments, the trained machine learning model is configured to receive an input image and detect one or more pre - defined objects within the input image.

[0148] In some embodiments, the pre - defined objects include diseased tissue.

[0149] An electronic device for training an exemplary machine learning model comprises one or more processors, a memory, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors, and include instructions for using a plurality of configurations of an SLM of an optical system to acquire a plurality of images of a biological sample, wherein the SLM is located in an optical path between the biological sample and an image recording device, and for training a machine learning model using the plurality of images.

[0150] An exemplary non - transitory computer - readable storage medium stores one or more programs for training a machine learning model. When the one or more programs are executed by one or more processors of an electronic device, the electronic device is caused to use a plurality of configurations of an SLM of an optical system to acquire a plurality of images of a biological sample, wherein the SLM is located in an optical path between the biological sample and an image recording device, and to train a machine learning model using the plurality of images.

[0151] An exemplary method for generating an enhanced image of a biological sample includes obtaining an image of the biological sample illuminated using an illumination pattern of an illumination source using a microscope, the illumination pattern being determined by: training a classification model configured to receive an input image and output a classification result; training a machine learning model having a plurality of weights corresponding to a plurality of illumination settings using the trained classification model; identifying the illumination pattern based on the plurality of weights of the trained machine learning model; and generating an enhanced image of the biological sample by inputting the obtained image of the biological sample into the trained machine learning model.

[0152] In some embodiments, the obtained image is a brightfield image.

[0153] In some embodiments, the enhanced image is a fluorescence image, a phase image, or a combination thereof.

[0154] In some embodiments, the illumination source comprises an array of illumination emitters.

[0155] In some embodiments, the illumination source is an LED array.

[0156] In some embodiments, the illumination pattern indicates whether each illumination emitter is turned on or off and the intensity of each illumination emitter.

[0157] In some embodiments, each illumination setting of the plurality of illumination settings corresponds to an individual illumination emitter of the illumination source, and each weight corresponds to the intensity of the individual illumination emitter.

[0158] In some embodiments, the classification model is configured to receive an input phase image or an input fluorescence image and output a classification result indicating one of a plurality of pre-defined classes.

[0159] In some embodiments, the plurality of pre-defined classes include a healthy class and an affected class.

[0160] In some embodiments, the machine learning model is a GAN model comprising an attention layer with a plurality of weights, a discriminator, and a generator.

[0161] In some embodiments, the machine learning model is a conditional GAN model.

[0162] In some embodiments, the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.

[0163] In some embodiments, each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group.

[0164] In some embodiments, the plurality of neural networks comprises a plurality of U-Net neural networks.

[0165] In some embodiments, the discriminator is a PatchGAN neural network.

[0166] In some embodiments, the step of training the machine learning model using a trained classification model includes applying a plurality of weights to a plurality of bright-field training images, aggregating the plurality of weighted bright-field training images into an aggregated bright-field image, inputting the aggregated bright-field training image into the machine learning model to obtain an enhanced training image and a generator loss, inputting the enhanced training image into the trained classifier to obtain a classifier loss, expanding the generator loss based on the classifier loss, and updating the plurality of weights based on the expanded generator loss.

[0167] In some embodiments, the method further includes classifying the enhanced image using the trained classifier.

[0168] In some embodiments, the method further includes the step of displaying an enhanced image.

[0169] A system for generating an enhanced image of an exemplary biological sample includes one or more processors, a memory, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors, and include instructions for: using a microscope to obtain an image of a biological sample illuminated using an illumination pattern of an illumination source, wherein the illumination pattern is determined by: training a classification model configured to receive an input image and output a classification result; training a machine learning model having a plurality of weights corresponding to a plurality of illumination settings using the trained classification model; identifying the illumination pattern based on the plurality of weights of the trained machine learning model; and generating an enhanced image of the biological sample by inputting the obtained image of the biological sample into the trained machine learning model.

[0170] An exemplary non-transitory computer-readable storage medium stores one or more programs for generating an enhanced image of a biological sample. When the one or more programs are executed by one or more processors of an electronic device, the electronic device is caused to: use a microscope to obtain an image of a biological sample illuminated using an illumination pattern of an illumination source, wherein the illumination pattern is determined by: training a classification model configured to receive an input image and output a classification result; training a machine learning model having a plurality of weights corresponding to a plurality of illumination settings using the trained classification model; identifying the illumination pattern based on the plurality of weights of the trained machine learning model; and generating an enhanced image of the biological sample by inputting the obtained image of the biological sample into the trained machine learning model.

[0171] An exemplary method for evaluating treatment for a target disease includes receiving a first plurality of images depicting a first set of healthy biological samples not affected by the target disease, receiving a second plurality of images depicting a second set of untreated biological samples affected by the target disease, receiving a third plurality of images depicting a third set of treated biological samples affected by the target disease and treated by treatment, inputting the first plurality of images into a trained machine learning model to obtain a first plurality of enhanced images, inputting the second plurality of images into a trained machine learning model to obtain a second plurality of enhanced images, inputting the third plurality of images into a trained machine learning model to obtain a third plurality of enhanced images, and comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment.

[0172] In some embodiments, the first plurality of images, the second plurality of images, and the third plurality of images are brightfield images.

[0173] In some embodiments, the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images are fluorescence images.

[0174] In some embodiments, the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images are phase images.

[0175] In some embodiments, the step of comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment includes identifying signals associated with biomarkers within each image.

[0176] In some embodiments, the step of comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images and evaluating the treatment further includes determining a first distribution based on the signals of the biomarkers in the first plurality of enhanced images, determining a second distribution based on the signals of the biomarkers in the second plurality of enhanced images, and determining a third distribution based on the signals of the biomarkers in the third plurality of enhanced images.

[0177] In some embodiments, the step of comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images and evaluating the treatment further includes comparing the first distribution, the second distribution, and the third distribution and evaluating the treatment.

[0178] In some embodiments, the step of comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images and evaluating the treatment includes, for each image, determining a score indicating the state of the disease of interest.

[0179] In some embodiments, the step of comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images and evaluating the treatment further includes determining a first distribution based on the scores of the first plurality of enhanced images, determining a second distribution based on the scores of the second plurality of enhanced images, and determining a third distribution based on the scores of the third plurality of enhanced images.

[0180] In some embodiments, the step of comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images and evaluating the treatment further includes comparing the first distribution, the second distribution, and the third distribution and evaluating the treatment.

[0181] In some embodiments, the treatment is a first treatment, and the method further includes receiving a fourth plurality of images depicting a set of fourth treated biological samples that are affected by the disease of interest and are being treated by a second treatment, inputting the fourth plurality of images into a trained machine learning model to obtain a fourth plurality of enhanced images, and comparing the first plurality of enhanced images, the second plurality of enhanced images, the third plurality of enhanced images, and the fourth plurality of enhanced images to compare the first treatment and the second treatment.

[0182] In some embodiments, the method further includes selecting a treatment from the first treatment and the second treatment based on the comparison.

[0183] In some embodiments, the method further includes administering the selected treatment.

[0184] In some embodiments, the method further includes providing a medical recommendation based on the selected treatment.

[0185] In some embodiments, the trained machine learning model is a GAN model comprising a discriminator and a generator.

[0186] In some embodiments, the machine learning model is a conditional GAN model.

[0187] In some embodiments, the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.

[0188] In some embodiments, each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group.

[0189] In some embodiments, the discriminator is a PatchGAN neural network.

[0190] A system for evaluating treatment for a target disease comprises one or more processors, a memory, and one or more programs. The one or more programs are stored in the memory and configured to be executed by the one or more processors, and include instructions for: receiving a first plurality of images depicting a first set of healthy biological samples not affected by the target disease; receiving a second plurality of images depicting a second set of untreated biological samples affected by the target disease; receiving a third plurality of images depicting a third set of treated biological samples affected by the target disease and treated by treatment; inputting the first plurality of images into a trained machine learning model to obtain a first plurality of enhanced images; inputting the second plurality of images into the trained machine learning model to obtain a second plurality of enhanced images; inputting the third plurality of images into the trained machine learning model to obtain a third plurality of enhanced images; comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images; and evaluating the treatment.

[0191] An exemplary non-transitory computer-readable storage medium stores one or more programs for evaluating a treatment for a target disease, and when the one or more programs are executed by one or more processors of an electronic device, the electronic device is caused to receive a first plurality of images depicting a first set of healthy biological samples not affected by the target disease, receive a second plurality of images depicting a second set of untreated biological samples affected by the target disease, receive a third plurality of images depicting a third set of treated biological samples affected by the target disease and treated by a treatment, input the first plurality of images into a trained machine learning model to obtain a first plurality of enhanced images, input the second plurality of images into the trained machine learning model to obtain a second plurality of enhanced images, input the third plurality of images into the trained machine learning model to obtain a third plurality of enhanced images, compare the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images, and evaluate the treatment, and includes instructions. The present invention provides, for example, the following. (Item 1) A method for training a machine learning model and generating an image of a biological sample, comprising: obtaining a plurality of training images, the plurality of training images comprising: a first type of training image; and a second type of training image; generating, using the machine learning model, a plurality of wavelet coefficients based on the first type of training image; generating a second type of composite image based on the plurality of wavelet coefficients; comparing the second type of composite image with the second type of training image; updating the machine learning model based on the comparison; and including. (Item 2) The method according to item 1, wherein the training image of the first type is a bright-field image of a biological sample. (Item 3) The method according to item 2, wherein the training image of the second type is a fluorescence image of the biological sample. (Item 4) The method according to any one of items 1-3, wherein the machine learning model includes a generator and a discriminator. (Item 5) The method according to item 4, wherein the machine learning model includes a conditional GAN model. (Item 6) The method according to any one of items 4-5, wherein the generator includes a plurality of neural networks corresponding to a plurality of frequency groups. (Item 7) The method according to item 6, wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group. (Item 8) The method according to any one of items 6-7, wherein the plurality of neural networks includes a plurality of U-Net neural networks. (Item 9) The method according to any one of items 5-8, wherein the discriminator is a PatchGAN neural network. (Item 10) The method according to any one of items 1-9, further including generating a third type of image based on the training image of the first type. (Item 11) The method according to item 10, wherein the third type of image is a phase shift image. (Item 12) The method according to any one of items 1-11, further including generating a fourth type of image based on the training image of the first type. (Item 13) The method according to item 12, wherein the fourth type of image includes segmented data. (Item 14) The training images of the first type are captured using a microscope according to a first illumination scheme, according to the method described in any of items 1-13. (Item 15) The first illumination scheme comprises one or more illumination patterns, according to the method described in item 14. (Item 16) The training images of the first type are part of a bright-field image array, according to the method described in any of items 14-15. (Item 17) The plurality of training images are a first plurality of training images, and the method further comprises identifying a second illumination scheme based on the comparison; and acquiring a second plurality of training images comprising one or more images of the first type, wherein the one or more images of the first type are acquired based on the second illumination scheme; and training the machine learning model based on the second plurality of training images The method according to any of items 14-16, comprising. (Item 18) acquiring a plurality of images of the first type using a microscope; and generating a plurality of synthetic images of the second type using the machine learning model based on the plurality of acquired images The method according to any of items 1-16, further comprising. (Item 19) The method according to item 18, further comprising training a classifier based on the plurality of synthetic images of the second type. (Item 20) The microscope is a first microscope, and the classifier is a first classifier. acquiring a plurality of images of the second type using a second microscope; and training a second classifier based on the plurality of images of the second type; and comparing the performance of the first classifier and the second classifier The method according to item 19, further comprising (Item 21) The method according to item 20, wherein the second microscope is a fluorescence microscope. (Item 22) A method for generating an enhanced image of a biological sample, comprising: acquiring an image of a biological sample using a microscope; generating an enhanced image of the biological sample using a machine learning model based on the image, wherein the machine learning model acquires a plurality of training images, the plurality of training images comprise a first type of training image and a second type of training image; generating a plurality of wavelet coefficients using the machine learning model based on the first type of training image; generating a second type of composite image based on the plurality of wavelet coefficients; comparing the second type of composite image with the second type of training image; updating the machine learning model based on the comparison; and being trained by ; and including (Item 23) The method according to item 22, wherein the first type of training image is a bright-field image of a biological sample. (Item 24) The method according to item 22, wherein the second type of training image is a fluorescence image of the biological sample. (Item 25) The method according to any one of items 22-24, wherein the machine learning model comprises a generator and a discriminator. (Item 26) The method according to item 25, wherein the machine learning model comprises a conditional GAN model. (Item 27) The method according to any one of items 25-26, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups. (Item 28) The method according to item 27, wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group. (Item 29) The method according to any one of items 27-28, wherein the plurality of neural networks comprises a plurality of U-Net neural networks. (Item 30) The method according to any one of items 26-29, wherein the discriminator is a PatchGAN neural network. (Item 31) The method according to any one of items 23-30, further comprising generating a third type of image based on the first type of training image. (Item 32) The method according to item 31, wherein the third type of image is a phase-shifted image. (Item 33) The method according to any one of items 23-32, further comprising generating a fourth type of image based on the first type of training image. (Item 34) The method according to item 33, wherein the fourth type of image comprises segmented data. (Item 35) The method according to any one of items 23-34, wherein the first type of training image is captured using a microscope according to a first illumination scheme. (Item 36) The method according to item 35, wherein the first illumination scheme comprises one or more illumination patterns. (Item 37) The method according to any one of items 35-36, wherein the first type of training image is part of a bright-field image array. (Item 38) The plurality of training images are a first plurality of training images, and the method further comprises Identifying a second illumination scheme based on the comparison; Obtaining a second plurality of training images comprising one or more images of the first type, wherein the one or more images of the first type are obtained based on the second illumination scheme; Training the machine learning model based on the second plurality of training images; The method according to any one of items 35 - 37, comprising. (Item 39) Obtaining a plurality of images of the first type using a microscope; Generating a plurality of synthetic images of the second type using the machine learning model based on the plurality of obtained images; The method according to any one of items 35 - 38, further comprising. (Item 40) The method according to item 39, further comprising training a classifier based on the plurality of synthetic images of the second type. (Item 41) The microscope is a first microscope, and the classifier is a first classifier. Obtaining a plurality of images of the second type using a second microscope; Training a second classifier based on the plurality of images of the second type; Comparing the performance of the first classifier and the second classifier; The method according to item 40, further comprising. (Item 42) The method according to item 41, wherein the second microscope is a fluorescence microscope. (Item 43) A system for training a machine learning model and generating images of biological samples, A computing system comprising one or more processors and one or more memories storing a machine learning model, the computing system being configured to receive a plurality of training images of a first type and one training image of a second type, the computing system generating, using the machine learning model, a plurality of wavelet coefficients based on the training images of the first type; generating a composite image of the second type based on the plurality of wavelet coefficients; comparing the composite image of the second type with the training image of the second type; updating the machine learning model based on the comparison; and being configured to perform the above operations. A system comprising the above. (Item 44) The system according to item 43, wherein the training images of the first type are bright-field images of biological samples. (Item 45) The system according to any one of items 43-44, wherein the training image of the second type is a fluorescence image of the biological sample. (Item 46) The system according to any one of items 43-45, wherein the machine learning model comprises a generator and a discriminator. (Item 47) The system according to item 46, wherein the machine learning model comprises a conditional GAN model. (Item 48) The system according to any one of items 46-47, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups. (Item 49) The system according to item 48, wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group. (Item 50) The plurality of neural networks includes a plurality of U-Net neural networks, and the system according to any one of items 48-49. (Item 51) The discriminator is a PatchGAN neural network, and the system according to any one of items 46-50. (Item 52) The computing system is further configured to generate a third type of image based on the first type of training image, and the system according to any one of items 43-51. (Item 53) The third type of image is a phase shift image, and the system according to item 52. (Item 54) The computing system is further configured to generate a fourth type of image based on the first type of training image, and the system according to any one of items 43-53. (Item 55) The fourth type of image includes segmented data, and the system according to item 54. (Item 56) The first type of training image is captured using a microscope according to a first illumination scheme, and the system according to any one of items 43-55. (Item 57) The first illumination scheme includes one or more illumination patterns, and the system according to item 56. (Item 58) The first type of training image is part of a bright field image array, and the system according to any one of items 56-57. (Item 59) The plurality of training images are a first plurality of training images, and the computing system is further identifying a second illumination scheme based on the comparison, and acquiring a second plurality of training images including one or more images of the first type, wherein the one or more images of the first type are acquired based on the second illumination scheme. training the machine learning model based on the second plurality of training images The system according to any one of items 56-58, configured to perform the above. (Item 60) The computing system further comprises acquiring a plurality of images of the first type using a microscope, and generating a plurality of composite images of the second type using the machine learning model based on the acquired plurality of images The system according to any one of items 43-59, configured to perform the above. (Item 61) The computing system further comprises a classifier trained based on the plurality of composite images of the second type, as described in item 59. (Item 62) The microscope is a first microscope, the classifier is a first classifier, and the computing system further comprises acquiring a plurality of images of the second type using a second microscope, and training a second classifier based on the plurality of images of the second type, and comparing the performance of the first classifier and the second classifier The system according to item 61, configured to perform the above. (Item 63) The system according to item 62, wherein the second microscope is a fluorescence microscope. (Item 64) A system for generating an enhanced image of a biological sample, comprising a computing system comprising one or more processors and one or more memories storing a machine learning model, the computing system receiving an image of a biological sample obtained from a microscope and configured to generate an enhanced image of the biological sample using the machine learning model based on the image, wherein the machine learning model Obtaining a plurality of training images, wherein the plurality of training images include a first type of training image, and a second type of training image ; generating a plurality of wavelet coefficients using the machine learning model based on the first type of training image; generating a synthetic image of the second type based on the plurality of wavelet coefficients; comparing the synthetic image of the second type with the training image of the second type; updating the machine learning model based on the comparison; and a computing system trained by ; A system comprising. (Item 65) The system according to item 64, wherein the first type of training image is a bright-field image of a biological sample. (Item 66) The system according to item 64, wherein the second type of training image is a fluorescence image of the biological sample. (Item 67) The system according to any one of items 64-66, wherein the machine learning model comprises a generator and a discriminator. (Item 68) The system according to item 67, wherein the machine learning model comprises a conditional GAN model. (Item 69) The system according to any one of items 67-68, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups. (Item 70) The system according to item 69, wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group. (Item 71) The system according to any one of items 69-70, wherein the plurality of neural networks comprises a plurality of U-Net neural networks. (Item 72) The discriminator is the system according to any one of items 67-71, which is a PatchGAN neural network. (Item 73) The machine learning model is further trained by generating a third type of image based on the first type of training image, for the system according to any one of items 64-72. (Item 74) The third type of image is a phase-shifted image, for the system according to item 73. (Item 75) The machine learning model is trained by generating a fourth type of image based on the first type of training image, for the system according to any one of items 64-74. (Item 76) The fourth type of image includes segmented data, for the system according to item 75. (Item 77) The first type of training image is captured using a microscope according to a first illumination scheme, for the system according to any one of items 64-76. (Item 78) The first illumination scheme includes one or more illumination patterns, for the system according to item 77. (Item 79) The first type of training image is part of a bright-field image array, for the system according to any one of items 77-78. (Item 80) The plurality of training images are a first plurality of training images, and the machine learning model identifies a second illumination scheme based on the comparison, and obtains a second plurality of training images including one or more images of the first type, where the one or more images of the first type are obtained based on the second illumination scheme, and trains the machine learning model based on the second plurality of training images and is trained thereby, for the system according to any one of items 77-79. (Item 81) The machine learning model uses a microscope to acquire a plurality of images of the first type, and uses the machine learning model to generate a plurality of synthetic images of the second type based on the acquired plurality of images The system according to any one of items 77 - 80, which is trained by the above. (Item 82) The system according to item 81, wherein the machine learning model is trained by training a classifier based on a plurality of synthetic images of the second type. (Item 83) The microscope is a first microscope, the classifier is a first classifier, and the machine learning model uses a second microscope to acquire a plurality of images of the second type, and trains a second classifier based on the plurality of images of the second type, and compares the performance of the first classifier and the second classifier The system according to item 82, which is trained by the above. (Item 84) The system according to item 83, wherein the second microscope is a fluorescence microscope. (Item 85) A method for processing an image of a biological sample to obtain one or more output images, comprising: using a plurality of configurations of an SLM of an optical system to acquire a plurality of images of the biological sample, wherein the SLM is located in an optical path between the biological sample and an image recording device, and inputting the plurality of images of the biological sample into a trained machine learning model to obtain the one or more output images The method comprising the above. (Item 86) The method according to item 85, wherein at least one of the plurality of configurations of the SLM is for generating one or more optical aberrations. (Item 87) Generating one or more optical aberrations, including spherical aberration, astigmatism, field curvature, distortion, tilt, or any combination thereof, according to the method of Item 86. (Item 88) At least one of the plurality of configurations of the SLM is for emphasizing one or more features, according to the method of any of Items 85-86. (Item 89) The one or more features include cell boundaries, actin filaments, nuclear shape, cytoplasmic compartmentalization, or any combination thereof, according to the method of Item 88. (Item 90) At least one of the plurality of configurations of the SLM is for reducing optical aberrations, according to the method of any of Items 85-89. (Item 91) The plurality of SLM configurations are for acquiring images of the biological sample at different depths, according to the method of any of Items 85-90. (Item 92) The machine learning model is configured to generate a second type of image based on a first type of image, according to the method of any of Items 85-91. (Item 93) The first type of image is a bright-field image, according to the method of Item 92. (Item 94) The second type of image is a fluorescence image, according to the method of Item 92. (Item 95) The second type of image is an enhanced version of the first type of image, according to the method of Item 92. (Item 96) The machine learning model is a GAN model or a self-supervised model, according to the method of any of Items 92-95. (Item 97) The plurality of images are acquired using a plurality of configurations of the light source of the optical system, according to the method of any of Items 85-96. (Item 98) The method according to item 97, wherein the light source is an LED array of the optical system. (Item 99) At least one of the plurality of SLM configurations training the machine learning model; evaluating the trained machine learning model; identifying the at least one configuration based on the evaluation The method according to any one of items 85-98, obtained by (Item 100) The method according to any one of items 85-99, wherein the trained machine learning model is configured to receive an input image and output an enhanced version of the input image. (Item 101) The method according to item 100, wherein the enhanced version of the input image comprises one or more enhanced cell phenotypes. (Item 102) An electronic device for processing an image of a biological sample and obtaining one or more output images, comprising one or more processors; a memory; one or more programs, the one or more programs being stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising obtaining a plurality of images of the biological sample using a plurality of configurations of an SLM of an optical system, the SLM being located in an optical path between the biological sample and an image recording device; inputting the plurality of images of the biological sample into a trained machine learning model and obtaining the one or more output images and one or more programs including instructions for performing An electronic device comprising (Item 103) A non-transitory computer-readable storage medium storing one or more programs for processing an image of a biological sample and obtaining one or more output images, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to obtain a plurality of images of the biological sample using a plurality of configurations of an SLM of an optical system, the SLM being located in an optical path between the biological sample and an image recording device, and input the plurality of images of the biological sample into a trained machine learning model and obtain the one or more output images thereby. A non-transitory computer-readable storage medium. (Item 104) A method for classifying an image of a biological sample, comprising: obtaining a plurality of images of the biological sample using a plurality of configurations of an SLM of an optical system, the SLM being located in an optical path between the biological sample and an image recording device, and inputting the plurality of images of the biological sample into a trained machine learning model and obtaining one or more classification outputs thereby. A method. (Item 105) The method according to item 104, wherein at least one of the plurality of configurations of the SLM is for generating one or more optical aberrations. (Item 106) The method according to item 105, wherein generating one or more optical aberrations includes spherical aberration, astigmatism, defocus, distortion, tilt, or any combination thereof. (Item 107) The method according to any one of items 104-106, wherein at least one of the plurality of configurations of the SLM is for enhancing one or more features. (Item 108) The method according to item 107, wherein the one or more features include cell boundaries, actin filaments, nuclear shape, cytoplasmic compartmentalization, or any combination thereof. (Item 109) The method according to any one of items 104 - 108, wherein at least one of the plurality of configurations of the SLM is for reducing optical aberration. (Item 110) The method according to any one of items 104 - 109, wherein the plurality of SLM configurations are for acquiring images of the biological sample at different depths. (Item 111) The method according to any one of items 104 - 110, wherein the plurality of images are acquired using a plurality of configurations of the light source of the optical system. (Item 112) The method according to item 111, wherein the light source is an LED array of the optical system. (Item 113) At least one of the plurality of SLM configurations is training the machine learning model, evaluating the trained machine learning model, identifying the at least one configuration based on the evaluation The method according to any one of items 104 - 112, obtained by (Item 114) The method according to any one of items 104 - 113, wherein the trained machine learning model is configured to receive an input image and detect one or more predefined objects within the input image. (Item 115) The method according to item 114, wherein the predefined object includes diseased tissue. (Item 116) An electronic device for classifying images of biological samples, comprising one or more processors, a memory, One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are To obtain a plurality of images of the biological sample using a plurality of configurations of the SLM of the optical system, wherein the SLM is located in the optical path between the biological sample and the image recording device, and Inputting the plurality of images of the biological sample into a trained machine learning model to obtain one or more classification outputs, and One or more programs including instructions for performing the above, and An electronic device comprising the above. (Item 117) A non-transitory computer-readable storage medium storing one or more programs for classifying images of a biological sample, the one or more programs comprising instructions which, when executed by one or more processors of an electronic device, cause the electronic device to To obtain a plurality of images of the biological sample using a plurality of configurations of the SLM of the optical system, wherein the SLM is located in the optical path between the biological sample and the image recording device, and Inputting the plurality of images of the biological sample into a trained machine learning model to obtain one or more classification outputs, and A non-transitory computer-readable storage medium for causing the above to be performed. (Item 118) A method for training a machine learning model, comprising: To obtain a plurality of images of a biological sample using a plurality of configurations of the SLM of the optical system, wherein the SLM is located in the optical path between the biological sample and the image recording device, and Training the machine learning model using the plurality of images, and A method including the above. (Item 119) The method according to item 118, wherein at least one of the plurality of configurations of the SLM is for generating one or more optical aberrations. (Item 120) Generating one or more optical aberrations includes spherical aberration, astigmatism, defocus, distortion, tilt, or any combination thereof, according to the method described in item 119. (Item 121) The method according to any one of items 118 - 120, wherein at least one of the plurality of configurations of the SLM is for enhancing one or more features. (Item 122) The one or more features include cell boundaries, actin filaments, nuclear shape, cytoplasmic compartmentalization, or any combination thereof, according to the method described in item 121. (Item 123) The method according to any one of items 118 - 122, wherein at least one of the plurality of configurations of the SLM is for reducing optical aberrations. (Item 124) The method according to any one of items 118 - 123, wherein at least one of the plurality of configurations of the SLM is for acquiring images of the biological sample at different depths. (Item 125) The method according to any one of items 118 - 124, wherein the machine learning model is configured to generate a second type of image based on a first type of image. (Item 126) The first type of image is a bright - field image, according to the method described in item 125. (Item 127) The second type of image is a fluorescence image, according to the method described in item 125. (Item 128) The method according to any one of items 118 - 127, wherein the machine learning model is a GAN model or a self - supervised model. (Item 129) The machine learning model is the method according to any one of items 118-128, which is a classification model. (Item 130) The plurality of images are obtained using a plurality of configurations of the light source of the optical system, and the method is according to any one of items 118-129. (Item 131) The light source is an LED array of the optical system, and the method is according to item 130. (Item 132) Training the machine learning model includes: (a) Training the machine learning model using a first image, where the first image is obtained using a first configuration of the SLM of the optical system; (b) Evaluating the trained machine learning model; (c) Identifying a second configuration of the SLM based on the evaluation; (d) Training the machine learning model using a second image, where the second image is obtained using the second configuration of the SLM of the optical system. The method is according to any one of items 118-131. (Item 133) The evaluation is based on the loss function of the machine learning model, and the method is according to item 112. (Item 134) The method further includes repeating steps (a)-(d) until a threshold is satisfied, and the method is according to any one of items 112-113. (Item 135) The threshold indicates the convergence of the training, and the method is according to item 114. (Item 136) The trained machine learning model is configured to receive an input image and output an enhanced version of the input image, and the method is according to any one of items 118-135. (Item 137) The enhanced version of the input image has one or more enhanced cell phenotypes, and the method is according to item 136. (Item 138) The method according to any one of items 118 - 135, wherein the trained machine learning model is configured to receive an input image and detect one or more pre - defined objects within the input image. (Item 139) The method according to item 138, wherein the pre - defined object includes diseased tissue. (Item 140) An electronic device for training a machine learning model, One or more processors, A memory, One or more programs, which are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include Using a plurality of configurations of the SLM of the optical system to acquire a plurality of images of a biological sample, wherein the SLM is located in the optical path between the biological sample and the image recording device, Training the machine learning model using the plurality of images One or more programs including instructions for performing the above, An electronic device comprising the above. (Item 141) A non - transitory computer - readable storage medium storing one or more programs for training a machine learning model, the one or more programs comprising instructions which, when executed by one or more processors of an electronic device, cause the electronic device to Using a plurality of configurations of the SLM of the optical system to acquire a plurality of images of a biological sample, wherein the SLM is located in the optical path between the biological sample and the image recording device, Train the machine learning model using the plurality of images A non - transitory computer - readable storage medium for causing the above to be performed. (Item 142) A method for generating an enhanced image of a biological sample, comprising: obtaining an image of a biological sample illuminated using an illumination pattern of an illumination source using a microscope, wherein the illumination pattern is training a classification model configured to receive an input image and output a classification result; training a machine learning model having a plurality of weights corresponding to a plurality of illumination settings using the trained classification model; identifying the illumination pattern based on the plurality of weights of the trained machine learning model; as determined by; generating an enhanced image of the biological sample by inputting the obtained image of the biological sample into the trained machine learning model; and a method comprising: (Item 143) The method according to item 142, wherein the obtained image is a bright-field image. (Item 144) The method according to any one of items 142-143, wherein the enhanced image is a fluorescence image, a phase image, or a combination thereof. (Item 145) The method according to any one of items 142-144, wherein the illumination source comprises an array of illumination emitters. (Item 146) The method according to item 145, wherein the illumination source is an LED array. (Item 147) The method according to any one of items 142-146, wherein the illumination pattern indicates whether each illumination emitter is turned on or off and the intensity of each illumination emitter. (Item 148) For each illumination setting of the plurality of illumination settings, each illumination setting corresponds to an individual illumination emitter of the illumination source, and each weight corresponds to the intensity of the individual illumination emitter. The method according to any one of items 145-147. (Item 149) The classification model is configured to receive an input phase image or an input fluorescence image and output a classification result indicating one of a plurality of pre-defined classes, according to the method described in any one of items 142-148. (Item 150) The plurality of pre-defined classes include a healthy class and a diseased class, according to the method described in item 149. (Item 151) The machine learning model is a GAN model including an attention layer with the plurality of weights, a discriminator, and a generator, according to the method described in item 150. (Item 152) The machine learning model is a conditional GAN model, according to the method described in item 151. (Item 153) The generator includes a plurality of neural networks corresponding to a plurality of frequency groups, according to the method described in any one of items 151-152. (Item 154) Each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group, according to the method described in item 153. (Item 155) The plurality of neural networks include a plurality of U-Net neural networks, according to the method described in any one of items 153-154. (Item 156) The discriminator is a PatchGAN neural network, according to the method described in any one of items 151-155. (Item 157) Training the machine learning model using the trained classification model includes applying the plurality of weights to a plurality of bright-field training images, aggregating the plurality of weighted bright-field training images into an aggregated bright-field image, inputting the aggregated bright-field training image into the machine learning model to obtain an enhanced training image and a generator loss, Inputting the emphasized training image into the trained classifier to obtain a classifier loss; Based on the classifier loss, expanding the generator loss; Updating the plurality of weights based on the expanded generator loss The method according to any one of items 151 - 156, comprising: (Item 158) The method according to any one of items 142 - 157, further comprising classifying the emphasized image using the trained classifier. (Item 159) The method according to any one of items 142 - 158, further comprising displaying the emphasized image. (Item 160) A system for generating an emphasized image of a biological sample, comprising: One or more processors; A memory; One or more programs, which are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include: Obtaining an image of a biological sample illuminated using an illumination pattern of an illumination source by using a microscope, wherein the illumination pattern: Training a classification model configured to receive an input image and output a classification result; Training a machine learning model having a plurality of weights corresponding to a plurality of illumination settings by using the trained classification model; Identifying the illumination pattern based on the plurality of weights of the trained machine learning model; as determined by; Generating an emphasized image of the biological sample by inputting the obtained image of the biological sample into the trained machine learning model; One or more programs including instructions for performing; A system comprising. (Item 161) A non-transitory computer-readable storage medium storing one or more programs for generating an enhanced image of a biological sample, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to acquire an image of a biological sample illuminated using an illumination pattern of an illumination source using a microscope, the illumination pattern train a classification model configured to receive an input image and output a classification result; train a machine learning model having a plurality of weights corresponding to a plurality of illumination settings using the trained classification model; identify the illumination pattern based on the plurality of weights of the trained machine learning model; as determined by; generate an enhanced image of the biological sample by inputting the acquired image of the biological sample into the trained machine learning model; A non-transitory computer-readable storage medium that causes the above to be performed. (Item 162) A method for evaluating a treatment for a target disease, receiving a first plurality of images depicting a first set of healthy biological samples not affected by the target disease; receiving a second plurality of images depicting a second set of untreated biological samples affected by the target disease; receiving a third plurality of images depicting a third set of treated biological samples affected by the target disease and treated by the treatment; inputting the first plurality of images into a trained machine learning model to obtain a first plurality of enhanced images; inputting the second plurality of images into the trained machine learning model to obtain a second plurality of enhanced images; Inputting the third plurality of images into the trained machine learning model to obtain a third plurality of enhanced images; comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment; A method comprising the above. (Item 163) The method according to item 162, wherein the first plurality of images, the second plurality of images, and the third plurality of images are bright-field images. (Item 164) The method according to any one of items 162-163, wherein the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images are fluorescence images. (Item 165) The method according to any one of items 162-163, wherein the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images are phase images. (Item 166) The method according to any one of items 162-165, wherein comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment includes identifying signals associated with biomarkers within each image. (Item 167) Comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment further includes: determining a first distribution based on the signals of the biomarkers in the first plurality of enhanced images; determining a second distribution based on the signals of the biomarkers in the second plurality of enhanced images; determining a third distribution based on the signals of the biomarkers in the third plurality of enhanced images. The method according to item 166, comprising the above. (Item 168) Comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment further includes: The method according to item 167, comprising comparing the first distribution, the second distribution, and the third distribution to evaluate the treatment. (Item 169) Comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment includes, for each image, determining a score indicating the state of the disease of interest, according to the method described in any one of items 162-165. (Item 170) Comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment further includes determining a first distribution based on the scores of the first plurality of enhanced images; determining a second distribution based on the scores of the second plurality of enhanced images; determining a third distribution based on the scores of the third plurality of enhanced images and including the method according to item 169. (Item 171) Comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment further includes comparing the first distribution, the second distribution, and the third distribution to evaluate the treatment, including the method according to item 170. (Item 172) The treatment is a first treatment, and the method further includes receiving a fourth plurality of images depicting a set of fourth treated biological samples affected by the disease of interest and treated by a second treatment; inputting the fourth plurality of images into the trained machine learning model to obtain a fourth plurality of enhanced images; comparing the first plurality of enhanced images, the second plurality of enhanced images, the third plurality of enhanced images, and the fourth plurality of enhanced images to compare the first treatment and the second treatment and including the method according to any one of items 162-171. (Item 173) The method according to item 172, further comprising selecting a treatment from the first treatment and the second treatment based on the comparison. (Item 174) The method according to item 173, further comprising administering the selected treatment. (Item 175) The method according to item 173, further comprising providing a medical recommendation based on the selected treatment. (Item 176) The method according to any one of items 162-175, wherein the trained machine learning model is a GAN model comprising a discriminator and a generator. (Item 177) The method according to item 176, wherein the machine learning model is a conditional GAN model. (Item 178) The method according to any one of items 176-177, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups. (Item 179) The method according to item 178, wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group. (Item 180) The method according to any one of items 176-179, wherein the discriminator is a PatchGAN neural network. (Item 181) A system for evaluating a treatment for a disease of interest, comprising: One or more processors; A memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs Receiving a first plurality of images depicting a set of first healthy biological samples not affected by the disease of interest; Receiving a second plurality of images depicting a second set of untreated biological samples affected by the disease of interest; Receiving a third plurality of images depicting a third set of treated biological samples affected by the disease of interest and treated by the treatment; Inputting the first plurality of images into a trained machine learning model to obtain a first plurality of enhanced images; Inputting the second plurality of images into the trained machine learning model to obtain a second plurality of enhanced images; Inputting the third plurality of images into the trained machine learning model to obtain a third plurality of enhanced images; Comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment; A program or programs including instructions for performing the above; A system comprising the above. (Item 182) A non-transitory computer-readable storage medium storing one or more programs for evaluating a treatment for a disease of interest, the one or more programs comprising instructions which, when executed by one or more processors of an electronic device, cause the electronic device to: Receive a first plurality of images depicting a first set of healthy biological samples not affected by the disease of interest; Receive a second plurality of images depicting a second set of untreated biological samples affected by the disease of interest; Receive a third plurality of images depicting a third set of treated biological samples affected by the disease of interest and treated by the treatment; Input the first plurality of images into a trained machine learning model to obtain a first plurality of enhanced images; Input the second plurality of images into the trained machine learning model to obtain a second plurality of enhanced images; Inputting the third plurality of images into the trained machine learning model to obtain a third plurality of enhanced images; comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment; A non-transitory computer-readable storage medium for causing the above to be performed.

Brief Description of the Drawings

[0192]

Figure 1

[0193]

Figure 2A

[0194]

Figure 2B

[0195]

Figure 3A

[0196]

Figure 3B

[0197]

Figure 3C

[0198]

Figure 3D

[0199]

Figure 4

[0200]

Figure 5

[0201]

Figure 6A

[0202]

Figure 6B

[0203]

Figure 7A

[0204]

Figure 7B

[0205]

Figure 8

[0206]

Figure 9

[0207]

Figure 10

[0208]

Figure 11

[0209]

Figure 12-1

[0210]

Figure 12-2

[0211]

Figure 13A

[0212]

Figure 13B

[0213]

Figure 13C

[0214]

Figure 14A

Figure 14B

[0215]

Figure 15

[0216]

Figure 16-1

[0217]

Figure 16-2

[0218]

Figure 16-3

[0219]

Figure 17A

[0220]

Figure 17B

DETAILED DESCRIPTION OF THE INVENTION

[0221] DETAILED DESCRIPTION OF THE INVENTION The present disclosure includes a method, a system, an electronic device, a non-transitory storage medium, and an apparatus for performing ML-based generation of image data at scale. The generated image data (e.g., image data of a biological sample) can provide sufficient richness and depth for downstream processing (e.g., phenotyping). Further, embodiments of the present disclosure comprise a set of computer and hardware optimization methods that extend the current dimensions of classical microscopy techniques.

[0222] Embodiments of the present disclosure can process a plurality of brightfield images of a biological sample to produce an enhanced image of the biological sample. The enhanced images include, but are not limited to, fluorescence images, phase shift images, semantic maps, polarization maps, refractive maps (2D and 3D), absorbance maps, and other image modalities. Brightfield images of biological samples can be acquired at scale and at low cost due to (e.g., relative to fluorescence microscopes) inexpensive equipment, ease of clinical deployment, and low processing and memory resource requirements. Acquiring brightfield images is generally non-invasive and involves low phototoxicity. Thus, brightfield images can be acquired efficiently and at scale. The enhanced images provide sufficient richness and depth for downstream processing (e.g., phenotype discovery).

[0223] Embodiments of the present disclosure include a machine learning model that receives a first type of image and is trained to convert the input image into another imaging modality. An exemplary machine learning model can receive a first type of image and convert the input image into a second type of image (e.g., an enhanced image). In some embodiments, different image types refer to different imaging modalities. In some embodiments, the first type of image is a brightfield image. For example, brightfield images can be captured by illuminating in vitro (or biopsy) cell samples using an inexpensive LED array. The second type of image includes fluorescence images. The generated fluorescence images exhibit high-contrast features that are not directly visible in the brightfield image and can be used for downstream processing (e.g., phenotyping).

[0224] Embodiments of the present disclosure reduce or eliminate the need to capture actual fluorescence images (or other special image modalities) of biological samples for downstream analysis, allowing brightfield images to be widely used for various purposes. This is particularly beneficial for live cell imaging. For example, the disclosed methods can be used for monitoring and optimizing cell differentiation experimental protocols. In the context of chemical or genetic perturbations, time-consuming activities associated with cell staining and fixation can be avoided. In some embodiments, the dosing time, which is the incubation time of the drug with the cells under observation, can also be optimized by software. Since the software will be able to notify the researcher of the optimized incubation time, the researcher will no longer need to arbitrarily determine the best incubation time. Furthermore, machine learning techniques used to convert brightfield images to other modalities require lower processing and memory resource utilization. Thus, embodiments of the present disclosure present technological improvements to the field of medical imaging while highlighting the operability and functionality of computing systems.

[0225] Embodiments of the present disclosure further include a machine learning model trained to receive a first type of image and convert the input image into a third type of image. In some embodiments, the third type of image includes image data indicative of various optical properties (e.g., phase shift) of the captured biological sample.

[0226] Embodiments of the present disclosure further include a machine learning model trained to receive a first type of image and convert the input image into a fourth type of image. In some embodiments, the fourth type of image includes image data indicative of segmentation data (e.g., cell boundaries).

[0227] One of ordinary skill in the art should understand that embodiments of the present disclosure can further convert the input image into a number of other types of images that capture various imaging characteristics, such as semantic maps, polarization maps, refractive maps (2D or 3D), absorbance maps, and the like.

[0228] In some embodiments, a single machine learning model is trained to perform multiple conversion tasks simultaneously. For example, the same machine learning model can receive a first type of image and generate multiple types of images (e.g., second type, third type, fourth type of images). The machine learning model can be an adversarial generation network ("GAN") model. For example, the GAN network can be a conditional GAN ("cGAN") model.

[0229] In some embodiments, the machine learning model converts an input image into its corresponding wavelet coefficients and generates one or more output images within the wavelet coefficients. The compact and multi-scale representation of the image, combined with the inherent non-linear options of the neural network, achieves multiple goals at once. First, the wavelet-based representation solves the inherent class imbalance problem that most generative models face. Specifically, most input image data has rich low-frequency signals but fewer high-frequency signals. Second, the wavelet-based representation extracts and maintains the overall geometric shape of the input image data. The discriminator of the model ensures that real and generated images (e.g., real vs. generated fluorescence images) are indistinguishable. The generated fluorescence image or other enhanced image modalities correspond to virtual staining of the sample. Due to the low phototoxicity of the brightfield imaging modality and its availability in clinical settings, the virtual staining according to the embodiments of the present disclosure can be performed on live cells as well as biopsy samples.

[0230] Embodiments of the present disclosure further include a hardware optimization method. For example, embodiments of the present disclosure can further dynamically optimize the illumination scheme of a microscope (e.g., a microscope that acquires a first type of image). In some embodiments, the microscope used to capture the first type of image can be adjusted or programmed to provide different illumination schemes during the training process. During the training of the machine learning model, an optimal illumination scheme for capturing the first type of image can be identified. The optimal illumination scheme can be used to capture the first type of image (e.g., brightfield image) so as to extract the best representation of the biological sample for wavelet-based image conversion (e.g., for downstream phenotype exploration).

[0231] Embodiments of the present disclosure further include the step of evaluating the robustness of the generated images of the machine learning model. In some embodiments, the first downstream classifier is trained using actual images (e.g., actual fluorescence images), and the second downstream classifier is trained using generated images (e.g., generated fluorescence images). The performance of the two classifiers can be compared to evaluate the robustness of the generated images as training data in the downstream task.

[0232] Accordingly, embodiments of the present disclosure provide an integrated platform that simultaneously addresses many problems, namely, image enhancement, phase recovery, low phototoxicity, realistic virtual staining of brightfield images, and robustness in downstream tasks. Embodiments of the present disclosure can evaluate the robustness of generated images via a downstream classification task. These tasks are integrated into the platform to close the loop of data generation from non-invasive brightfield images to fluorescent images. For example, the system can optimize the parameters of the brightfield microscope acquisition system during use. Other parameters of the brightfield microscope acquisition system, including the illumination pattern of the LED array and, for example, the focusing position of the microscope objective lens and the activation timing of the spatial light modulator (SLM), can be optimized by backpropagation during the downstream classification task.

[0233] Furthermore, in some embodiments, the platform learns to perform a series of perturbations on the cells, optimize the illumination scheme, and extract the best representation of the cells for phenotypic discovery.

[0234] Some embodiments of the present disclosure can identify one or more optimal illumination patterns for capturing image data. In some embodiments, the illumination pattern can indicate whether each illumination emitter (e.g., each LED on an LED array) of the illumination source should be turned on or off, and the intensity of each illumination emitter. The system can determine the optimal illumination pattern by training a machine learning model having an attention layer with a plurality of weights (e.g., a plurality of weights corresponding to the intensities of a plurality of LEDs on an LED array) corresponding to the intensities of the plurality of illumination emitters. During the training of the machine learning model, the plurality of weights are applied to a plurality of training images (e.g., brightfield images) illuminated by different illumination emitters. The aggregated image is input into the machine learning model, and a loss can be determined. The model, which includes weights within the attention layer, can be updated as appropriate based on the loss. After training, the illumination pattern can be determined based on the weights within the attention layer of the trained machine learning model, since each weight can correspond to the desired intensity level of the corresponding illumination emitter. Thus, the process involves only the step of capturing an image using a limited number of illumination settings (e.g., turning on a single illumination emitter at a time and capturing an image), and does not require physically adjusting the intensity of the illumination emitters to identify the optimal illumination pattern.

[0235] Some embodiments of the present disclosure can train a machine learning model such that synthetic image data generated by the machine learning model can provide the same performance as actual images in downstream analysis. In some embodiments, the step of training the machine learning model includes first training a classifier corresponding to a downstream task (e.g., classifying healthy versus diseased tissue based on an image), and then using the output of the classifier to guide the training of the machine learning model.

[0236] Some embodiments of the present disclosure can evaluate candidate treatments for a disease of interest. In some embodiments, the system receives a first plurality of images depicting a first set of healthy biological samples that are not affected by the disease of interest, a second plurality of images depicting a second set of untreated biological samples that are affected by the disease of interest, and a third plurality of images depicting a third set of treated biological samples that are affected by the disease of interest and are being treated with a candidate treatment. The images are input into a machine learning model to obtain enhanced images, which are compared to evaluate the treatment (e.g., by analyzing the distribution of the images).

[0237] The following description sets forth exemplary methods, parameters, and equivalents. However, such description is not intended as a limitation on the scope of the present disclosure, but rather is to be recognized as being provided as an explanation of exemplary embodiments.

[0238] In some embodiments, an exemplary optical system includes a programmable spatial light modulator ("SLM"). The SLM of the optical system can improve the performance of a machine learning model through a training phase (e.g., by providing a rich training dataset) and / or through an inference phase (e.g., by providing input data under various optical settings or optimal settings). The SLM is programmed without requiring any mechanical movement or mechanical modification to the optical system.

[0239] The SLM provides additional degrees of freedom and sources of contrast and controls the microscope in a programmable manner. For example, the SLM can be programmed to generate optical aberrations that enhance important phenotypes. As another example, the SLM can be programmed to produce various images that provide different modulations and thus enable fast exploration of deep samples. The SLM can also infer cell phenotypes and enable identification of optimal imaging settings for reconstructing alternative image modalities in a supervised manner. Multifocal acquisition is possible without any mechanical movement and thus accelerates and improves downstream tasks. Three-dimensional phase tomography and reconstruction are thus accelerated and improved.

[0240] The following description uses the terms "first", "second", etc. to describe various elements, but these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, a first graphical representation could be referred to as a second graphical representation without departing from the scope of the various described embodiments, and similarly, a second graphical representation could be referred to as a first graphical representation. The first graphical representation and the second graphical representation are both graphical representations; they are not the same graphical representation.

[0241] The terminology used in the description of the various described embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Also, the term "and / or" as used herein refers to and is to be taken to include any and all possible combinations of one or more of the associated listed items, and is to be understood to encompass them. Further, the terms "includes", "including", "comprises" and / or "comprising", when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0242] The term "if (in the case of ~)" is optionally interpreted, depending on the context, to mean "when (at the time of ~)", "upon (in response to ~)", "in response to determining (in response to determining ~)", or "in response to detecting (in response to detecting ~)". Similarly, the phrases "if it is determined (if it is determined that ~)" or "if [a stated condition or event] is detected (if [a stated condition or event] is detected)" are optionally interpreted, depending on the context, to mean "upon determining (in response to determining ~)", "in response to determining (in response to determining ~)", "upon detecting [the stated condition or event] (in response to detecting [the stated condition or event])", or "in response to detecting [the stated condition or event] (in response to detecting [the stated condition or event])".

[0243] FIG. 1 illustrates an exemplary process for training a machine learning model configured to generate enhanced images of biological samples, according to some embodiments.

[0244] Referring to FIG. 1, the training data 120 includes a first type of image 122 and a second type of image 124. In some embodiments, the images are images of biological samples, which can include one or more than one set of stained, unstained, perturbed, and / or unperturbed biological samples.

[0245] In some embodiments, the first type of image data 122 comprises a set of brightfield images, and the second type of image data 124 comprises images in a different modality (e.g., fluorescence images). The first type of image 122 can be acquired using a brightfield microscope, while the second type of image 124 can be acquired using a fluorescence microscope.

[0246] As shown in FIG. 1, the first type of image 122 is acquired based on one or more illumination patterns 110. The illumination pattern indicates the settings by which an object is illuminated. In some embodiments, the illumination pattern can be defined by one or more parameters indicating the spatial relationship (e.g., distance, angle) between the object and the illumination source, one or more parameters indicating the settings of the illumination source, or a combination thereof. For example, the illumination pattern can indicate a set of LED light sources to be activated, specific out-of-focus / polarization settings, etc.

[0247] In some embodiments, the microscope that captures the first type of image can be a microscope that supports multiple illumination patterns. For example, the microscope can provide a programmable illumination source (e.g., an LED array, a laser), an adaptive optics system (SLM, micromirror), or a combination thereof. By updating the illumination pattern (e.g., controlling the illumination source and / or the pupil function of the optical system), multiple representations of a biological sample corresponding to multiple illumination patterns can be obtained.

[0248] In some embodiments, the training data 120 can be organized as three-dimensional image data (e.g., an image array). For example, the training data 120 can have dimensions (B, C, H, W), where B indicates the batch size, C indicates the number of channels (i.e., the illumination patterns), H indicates the height, and W indicates the width. C is equal to 1 if only a single brightfield image exists, and C is greater than 1 if a stack of brightfield images exists.

[0249] In some embodiments, one or more images within the training data 120 can be normalized before they are used to train the machine learning model 100. For example, fluorescence images can be normalized based on illumination or intensity parameters.

[0250] Referring to FIG. 1, the training data 120 is used to train the machine learning model 100. In some embodiments, the model 100 is an adversarial generative network (“GAN”) model. For example, the GAN network can be a conditional GAN (“cGAN”) model. As will be described in detail below, the GAN network includes a generator and a discriminator. The generator is trained to receive a first type of image (e.g., brightfield image) and convert the input image into a second type of image (e.g., fluorescence image). The training step includes comparing the actual image 122 of the first type with the actual image 124 of the second type and determining a ground truth baseline for the wavelet-based transformation that the first type of image 122 should seek to perform. During training, the generator output (i.e., the generated fluorescence image) can be directly connected to the discriminator input. The discriminator is trained to distinguish a second type of generated image (e.g., generated fluorescence image) from a second type of actual image (e.g., actual fluorescence image). Through backpropagation, the output of the discriminator can be used by the generator to update the weights of the generator so that the generator learns to generate an image that the discriminator would classify as an actual image.

[0251] In some embodiments, the illumination parameters can be updated during the training of the model 100. Thus, during the training of the model 100, the illumination scheme can be continuously updated, and the training data can be obtained according to the updated illumination scheme as described in detail below, and the model 100 can be further trained.

[0252] Figures 2A and 2B illustrate exemplary processes 200 and 250 for training and applying a machine learning model (e.g., model 100) according to some embodiments. Figure 2A illustrates the process when microscope parameters cannot be changed during the acquisition of training data. Figure 2B illustrates the process when microscope parameters can be changed during the acquisition of training data, and thus an optimal illumination pattern can be identified.

[0253] Each process can be implemented, at least in part, using one or more electronic devices. In some embodiments, the blocks of each process step depicted in Figures 2A and 2B can be divided among multiple electronic devices. In each process, some blocks can optionally be combined, the order of some blocks can optionally be changed, and some blocks can optionally be omitted. In some examples, additional steps can be implemented in combination with the process. Thus, the operations as illustrated (and described in more detail below) are exemplary in nature and should not be considered limiting.

[0254] In Figure 2A, microscope parameters cannot be changed during the acquisition of brightfield images. Thus, all brightfield images in the training data (e.g., training data 120) are acquired based on the same illumination pattern.

[0255] In block 204, training data (e.g., training data 120 in FIG. 1) is acquired, at least in part, according to a default illumination pattern. As described above with reference to FIG. 1, the training data comprises a first type of image (e.g., a brightfield image) acquired by a brightfield microscope according to a default illumination pattern and a second type of image (e.g., a fluorescence image) acquired by a fluorescence microscope.

[0256] In block 206, a machine learning model (e.g., model 100 in FIG. 1) is trained based on training data. The model can be a GAN model. For example, the GAN network can be a conditional GAN (“cGAN”) model.

[0257] FIGS. 3A, 3C, and 3D illustrate a training process of a machine learning model according to some embodiments. The GAN model includes a generator 302 and a discriminator 304. The generator 302 is trained to receive a first type of image 310 (e.g., a bright-field image) and generate a second type of image 312 (e.g., a fluorescence image). In some embodiments, the first type of image 310 includes the bright-field image array described above.

[0258] During training, the generator output (i.e., the generated fluorescence image) can be directly connected to the discriminator input. The discriminator 304 is trained to distinguish a second type of generated image 312 (e.g., the generated fluorescence image) from a second type of actual image 314 (e.g., the actual fluorescence image). In some embodiments, the second type of actual image 314 includes the fluorescence image array described above.

[0259] Through backpropagation, the output of the discriminator is used by the generator, as described in detail below, to update the weights of the generator so that the generator learns to generate an image that the discriminator would classify as an actual image. In some embodiments, the generator 302 and the discriminator 304 are neural networks.

[0260] Figure 3B illustrates the operation of an exemplary generator according to some embodiments. The input image 352 can be a single image from the brightfield image array or the entire brightfield image array. For example, the input image 352 can be of dimensions (B, C, H, W), where B indicates the batch size, C indicates the number of channels, H indicates the height, and W indicates the width. C is equal to 1 if only a single brightfield image exists, and C is greater than 1 if a stack of brightfield images exists.

[0261] Referring to FIG. 3B, the generator includes a series of convolutional layers for downsampling the input image 352 to obtain a downsampled input image 354. In some embodiments, the input image 353 can be downsampled to half its original size. For example, if the input is 256×256 within the spatial size, it will be reduced to 128×128.

[0262] The downsampled image 354 is then passed through a plurality of neural networks. In the illustrated embodiment, the plurality of neural networks includes four U-Net neural networks. The U-Net network is a convolutional network for image-to-image transformation. Details of the design and implementation of the U-Net network can be found, for example, in Ronneberger et al., "U-Net: Convolutional Networks for Biomedical Image Segmentation" (incorporated herein by reference in its entirety).

[0263] Multiple neural networks can correspond to different groups of frequencies within the wavelet domain. In the depicted embodiments, four U-Net networks are involved with low frequency, high frequency (horizontal), high frequency (vertical), and high frequency (diagonal), respectively. In signal processing, low frequency signals correspond to features that are very large relative to the size of the image (e.g., when imaging cells, having about the size of the cytoplasm or nucleus). High frequency information is very fine small image features (e.g., having about the size of mitochondria, microtubules). Low frequency signals correspond to the first scale of wavelet coefficients. High frequencies are encoded with higher scale wavelet coefficients. In some embodiments, the multiple neural networks operate independently and do not share weights.

[0264] As shown, three of the four U-Net branches correspond to high frequency blocks. Low frequency information is relatively easy to recover. Thus, having more computing power dedicated to high frequencies guarantees the reconstruction of fine details within the image. Thus, the loss function, operating within the wavelet domain, benefits from the signal's composition (three times more high frequency information than low frequency information).

[0265] Each neural network is configured to output (or predict) wavelet coefficients for an individual group of frequencies. The loss function is applied to the predicted wavelet coefficients 356 and the true wavelet coefficients of the actual fluorescence image. The loss function is further described below with reference to FIGS. 3C and 3D.

[0266] The generated fluorescence image 358 within the image domain can be obtained by applying the inverse wavelet transform to the predicted coefficients 356.

[0267] FIG. 3C illustrates the backpropagation process of discriminator 304 according to some embodiments. Discriminator 304 is a model (e.g., a neural network) trained to provide an output based on a given image. The training data for discriminator 304 comprises a second type of actual image 314 (e.g., an actual fluorescence image) and a second type of synthetic image 312 (e.g., a generated fluorescence image) generated by generator 302.

[0268] In some embodiments, discriminator 304 is a PatchGAN network. Details of the design and implementation of the PatchGAN network can be found, for example, in Isola et al., "Image-to-Image Translation with Conditional Adversarial Networks" (incorporated by reference in its entirety).

[0269] During the training of discriminator 304, a discriminator loss 322 can be calculated based on the output of the generator (i.e., the predicted wavelet coefficients). In some embodiments, the discriminator loss function is a Wasserstein discriminator loss and is calculated as follows.

[0270]

Chemical formula

[0271] Where f(x) is the output of the discriminator based on the wavelet coefficients of the actual fluorescence image, w is the model weight of the discriminator, m is the mini-batch size, f is the discriminator model, x is the actual image, z is the input (brightfield), G is the generator model, and f(G(z)) is the output of the discriminator based on the predicted wavelet coefficients corresponding to the synthetic fluorescence image.

[0272] The discriminator 304 is configured to maximize this function. In other words, it attempts to maximize the difference between its output based on the actual image and its output based on the synthesized image. As depicted in FIG. 3C, the discriminator updates its weights through backpropagation based on the discriminator loss 332 through the discriminator network.

[0273] FIG. 3D illustrates the backpropagation process of the generator 302 according to some embodiments. The generator 302 is a neural network configured to receive a first type of image 310 and generate a second type of image 312 as described with reference to FIGS. 3A and 3B. The wavelet coefficients predicted by the generator are input into the discriminator 304. Depending on the output of the discriminator, a generator loss 324 can be calculated. In some embodiments, the generator loss function is a Wasserstein generator loss and is calculated as follows.

[0274]

Chemical formula

[0275] where f(x) is the output of the discriminator based on the wavelet coefficients of the actual fluorescence image, m is the size of the mini-batch, f is the discriminator model, z is the input (bright field), G is the generator model, and f(G(z)) is the output of the discriminator based on the predicted wavelet coefficients.

[0276] The reconstruction loss, which operates within the wavelet domain, inherently has the property of balancing the contributions of low and high frequencies. As shown in Figure 3B, the wavelet coefficients can be split into two categories, namely, low (one block) and high frequency (three blocks). Three of the four UNet branches can be dedicated to the high-frequency blocks. Low-frequency information can be recovered more easily. Therefore, dedicating more computing power to high frequencies serves to reconstruct the fine details within the image. The loss function, which operates within the wavelet domain, benefits from the signal's main composition (three times as much high-frequency information as low-frequency information). In some embodiments, the wavelet coefficients can be split into more than two categories, such as three categories (high, medium, and low frequencies), four categories, or more than four categories.

[0277] The generator 302 is configured to maximize this function. In other words, it attempts to maximize the output of the discriminator based on its synthetic image. In some embodiments, the generator loss is backpropagated through both the discriminator 304 and the generator 302 to obtain gradients, which are ultimately used to adjust the generator weights.

[0278] In some embodiments, the generator 302 and the discriminator 304 are trained in alternating cycles. In each cycle, the discriminator is trained with respect to one or more reference points, and the generator is trained with respect to one or more reference points. During discriminator training, the generator can remain constant. Similarly, during generator training, the discriminator can remain constant.

[0279] In some embodiments, the generator can convert the input image into a third type of image (e.g., a phase-shifted image). For example, in addition to generating a fluorescence image, the generator may also output a phase-shifted image, in which each pixel represents a local value of the phase within the image that can be transformed (e.g., -5 to 5 phase information). A physics-based image formation model can be used to generate the true phase-shifted image (i.e., the ground truth phase-shifted image). The image formation model generates an image based on the absolute knowledge of the microscope (e.g., the aberrations of the optical system) as well as the optical properties of the captured sample (e.g., refractive index, phase). Since the optical properties of the samples can be compared between samples, the risk of batch effects in downstream tasks is almost nil. The physics-based model allows for the incorporation of certain prior knowledge in the generation process.

[0280] Referring to FIG. 4, during training, the illumination source S is optimized to read X with high fidelity. The microscope PSF (or pupil function) can also be optimized using a set of spatial light modulators or micromirrors. In addition, the polarization of S can also be modulated to inject more contrast into the collected image.

[0281]

Chemical formula

[0282] In the above formula, S(f) refers to a partially coherent illumination source (LED array). X(r) refers to the complex electric field of the sample. P(r) refers to the point spread function (PSF) of the microscope. RI(r) refers to the refractive index. I refers to the image. The forward model is applied outside the training loop to obtain the ground truth phase.

[0283] In some embodiments, the generator can convert the input image into a fourth type of image. In some embodiments, the fourth type of image includes image data that indicates segmented data (e.g., cell boundaries). In some embodiments, the loss function depends on the supported image modalities. For semantic segmentation, the L1 norm can be used to infer discrete labels within the image. For another image modality, another branch can be added to the generator to output the new modality.

[0284] Returning to FIG. 2A, at block 208, metadata can be associated with the trained machine learning model. The metadata can include a default illumination pattern (e.g., parameters of a brightfield microscope).

[0285] At block 210, one or more images of the first type (e.g., brightfield images) are acquired. In some embodiments, the images are acquired using the same illumination pattern (e.g., as indicated by the metadata at block 208).

[0286] At block 212, one or more images are input into the generator (e.g., generator 302) of the trained machine learning model. The generator is trained to convert images of the first type into images of the second type (e.g., fluorescence images). At block 214, one or more images of the second type are acquired. As described below, the generated fluorescence images can then be used as training data for other machine learning models, thus eliminating the need to acquire actual fluorescence images as training data.

[0287] FIG. 2B illustrates the process when the microscope parameters can be changed during the acquisition of training data, and thus the optimal illumination pattern can be identified. For example, the microscope parameters can be programmable such that multiple illumination patterns can be provided by the microscope.

[0288] In block 252, the illumination pattern is loaded onto the microscope. In block 254, a brightfield training image is captured according to the illumination pattern (e.g., a brightfield image), and a fluorescence training image is also captured by the fluorescence microscope. Blocks 252 and 254 can be repeatedly performed, as indicated by arrow 253. In other words, a sequence of illumination patterns can be loaded onto the microscope, and training data corresponding to the sequence of illumination patterns can be acquired. The sequence of illumination patterns is also referred to herein as an illumination scheme.

[0289] In block 256, a machine learning model is trained based on the training data. The model can be a GAN model that operates as described with reference to FIGS. 3A-D. For example, the GAN network can be a conditional GAN (“cGAN”) model.

[0290] During training, the model iteratively updates the illumination pattern. The illumination pattern is updated by backpropagating the gradient of a loss (e.g., a generator loss) to the parameters of the microscope. The training procedure of the model minimizes the overall imaging time and the loss function associated with the conversion or classification task.

[0291] During training, the system determines the illumination pattern leading to the minimum loss (e.g., generator loss). In some embodiments, the model produces a first generator loss when converting a first image, a second generator loss when converting a second image, and so on. The losses can be compared and the minimum loss can be identified. At block 260, the illumination pattern that produced the minimum loss can be identified and a new illumination scheme can be identified as appropriate. For example, the new illumination scheme can include the best illumination pattern and / or one or more new illumination patterns similar to the best illumination pattern. The new illumination scheme can also exclude previously included illumination patterns that have resulted in the maximum loss. As shown by arrow 258, the identified illumination scheme can be loaded into the microscope to acquire additional training data. This process can be repeated until no further improvement (e.g., with respect to generator loss) is observed. The optimal illumination scheme can be stored at block 261.

[0292] At block 262, one or more images of the first type (e.g., brightfield images) are acquired. In some embodiments, the images are acquired using the optimal illumination scheme stored at block 261.

[0293] At block 264, one or more images are input into the generator (e.g., generator 302) of the trained machine learning model. The generator is trained to convert an image of the first type into an image of the second type (e.g., a fluorescence image). At block 266, one or more images of the second type are acquired. As described below, the generated fluorescence images can then be used as training data for other machine learning models, thus eliminating the need to acquire actual fluorescence images as training data.

[0294] FIG. 5 illustrates the output of an exemplary trained generator using a static microscope setting. The trained generator receives an input brightfield image and generates a phase image and a fluorescence image 504. For comparison, FIG. 5 also shows the corresponding actual phase image and fluorescence image 506. In FIG. 5, each input image is a composite of four different illuminations. In some embodiments, the input image is a false-color image representing different illumination patterns.

[0295] FIG. 6A illustrates an exemplary process for determining the robustness of a generated image according to some embodiments. Two classifiers 612 and 614 are trained to receive a fluorescence image and provide an output (e.g., a phenotypic classification result). Classifier 614 is trained based on an actual fluorescence image 604 (e.g., an actual fluorescence image), while classifier 612 is trained based on a generated fluorescence image 610. The generated fluorescence image is generated by a trained generator 608 (e.g., generator 102, 302) based on a brightfield image 506.

[0296] In some embodiments, classifier 612 is verified using the generated image, while classifier 614 is verified using the actual image.

[0297] The performance of classifiers 612 and 614 can be compared to determine the robustness of the generated image. FIG. 6B illustrates an exemplary comparison between two such classifiers. In this particular setting, the dataset is created from fluorescence images and 3D brightfield images of primary cultured hepatocytes from 12 healthy donors and 12 NASH donors. The GAN model is trained to generate fluorescence images from a stack of out-of-focus brightfield images.

[0298] A comparison is made to determine whether a classifier trained on the generated image and verified on the actual image performs equally well as when trained on the actual image and verified on the generated image. As shown in FIG. 6B, no decrease in classification accuracy is observed, and more importantly, the geometry of the embedding space of the actual image is preserved on the generated image.

[0299] Some embodiments include a backpropagation module 616. The backpropagation module 616 can be used, for example, to improve the image acquisition parameters used to obtain a brightfield image to be emphasized. For example, Xi can include a set of parameters for an illumination pattern applied to the LED illumination array of a brightfield microscope. Xi can also include imaging parameters related to the focal position of the microscope objective lens and / or activation parameters of a spatial light modulator (SLM). All elements of Xi can be variables of the optimization procedure. Thus, in 616, the gradient of the loss function estimated on the downstream task

[0612] (e.g., classification, image conversion, regression) may be backpropagated to optimize the parameter Xi. For each update to the value of Xi, a new set of images can be obtained and a new dataset of images can be generated.

[0300] In some embodiments, the data used to train the generator 608, the data used to train the first classifier 614 or the second classifier 612, and the data used to evaluate the performance of the classifier can include overlapping images. The images used in any of the processes described above can be annotated based on the captured biological sample (e.g., cell type, diseased or healthy) and perturbation. These annotations can be used for downstream classification tasks (e.g., as training data for training classifiers 612 and 614 in FIG. 6A) and for validation of the prediction model (e.g., to evaluate the performance of the trained classifiers 612 and 614).

[0301] FIG. 7A illustrates an example of a method 700 for training a machine learning model and generating an image of a biological sample according to one embodiment. The method includes, at 702, obtaining a plurality of training images including a first type of training image and a second type of training image. The method further includes, at 704, generating a plurality of wavelet coefficients using a machine learning model based on the first type of training image; at 706, generating a second type of synthetic image based on the plurality of wavelet coefficients; at 708, comparing the second type of synthetic image with the second type of training image; and at 710, updating the machine learning model based on the comparison.

[0302] FIG. 7B illustrates an example of a method 750 for generating an enhanced image of a biological sample according to one embodiment. The method includes, at 752, obtaining an image of a biological sample using a microscope. The method also includes, at 754, generating an enhanced image of the biological sample using the machine learning model of FIG. 7A based on the image.

[0303] FIG. 8 illustrates an example of a computing device according to one embodiment. Device 800 can be a host computer connected to a network. Device 800 can be a client computer or a server. As shown in FIG. 8, device 800 can be any suitable type of microprocessor-based device, such as a personal computer, a workstation, a server, or a handheld computing device (portable electronic device), such as a phone or a tablet. The device can include, for example, one or more of processor 810, input device 820, output device 830, storage device 840, and communication device 860. Input device 820 and output device 830 can generally correspond to those described above and can either be connectable to or integrated with the computer.

[0304] The input device 820 can be any suitable device that provides input, such as a touch screen, keyboard or keypad, mouse, or voice recognition device. The output device 830 can be any suitable device that provides output, such as a touch screen, tactile device, or speaker.

[0305] The storage device 840 can be any suitable device that provides storage, such as electrical, magnetic, or optical memory, including RAM, cache, hard drive, or removable storage disk. The communication device 860 can include any suitable device that is capable of transmitting and receiving signals via a network, such as a network interface chip or device. The components of the computer can be connected in any suitable manner, such as via a physical bus or wirelessly.

[0306] The software 850 stored in the storage device 840 and executable by the processor 810 can include, for example, programming that embodies the functionality of the present disclosure (e.g., as embodied within the device as described above).

[0307] The software 850 can also be stored and / or conveyed within any non-transitory computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch and execute instructions associated with the software. In the context of the present disclosure, a computer-readable storage medium can be any medium that can contain or store programming for use by or in connection with an instruction execution system, apparatus, or device, such as the storage device 840.

[0308] Software 850 can also be propagated in any carrier medium that can be fetched from an instruction execution system, apparatus, or device associated with the software and used by or connected to an instruction execution system, apparatus, or device such as those described above to execute the instructions. In the context of the present disclosure, a carrier medium can be any medium that can communicate, propagate, or transport programming for use by or connected to an instruction execution system, apparatus, or device. A carrier-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared wired or wireless propagation media.

[0309] Device 800 may be connected to a network, which can be any suitable type of interconnected communication system. The network can implement any suitable communication protocol and can be secured by any suitable security protocol. The network can comprise any suitable arrangement of network links capable of transmitting and receiving network signals, such as, for example, a wireless network connection, a T1 or T3 line, a cable network, DSL, or a telephone line.

[0310] Device 800 can implement any operating system suitable for operating on a network. Software 850 can be written in any suitable programming language, such as C, C++, Java®, or Python. In various embodiments, the application software embodying the functionality of the present disclosure can be deployed in different configurations, such as in a client / server arrangement or, for example, through a web browser as a web-based application or web service.

[0311] Optimization Using a Spatial Light Modulator

[0312] In some embodiments, an exemplary optical system includes a programmable SLM. The exemplary SLM includes a high-resolution liquid crystal panel with micron-sized individually addressable pixels, which can be used to shape the wavefront of an optical beam. The gray level values on the panel are converted into phase shifts. The SLM can be used, in some embodiments, as a programmable Fourier filter for generating contrast enhancement or as a programmable diffractive optical element for quantitative phase microscopy.

[0313] As described herein, the SLM can be programmed to generate various images during the training phase and / or the inference phase of a machine learning model to improve the performance of the machine learning model.

[0314] During the inference phase of a trained machine learning model, the SLM can be programmed to generate different input images for the trained machine learning model. For example, the SLM can be programmed to generate an input image so that a trained image conversion model can obtain an enhanced version of the input image. The enhanced image can then be used for downstream operations. As another example, the SLM can be programmed to generate an input image so that a trained classification model can obtain a more accurate classification result.

[0315] Furthermore, the SLM can be programmed to generate different images as training data during the training phase of a machine learning model. Additionally, the SLM can be iteratively programmed to identify the optimal settings for capturing images that lead to the best performance of a given machine learning model, either during the training phase or the inference phase.

[0316] Therefore, the SLM of the optical system can improve the performance of the machine learning model through the training stage (e.g., by providing a rich training dataset) and / or through the inference stage (e.g., by providing input data under various settings or optimal settings). The SLM is programmed without requiring any mechanical movement or modification to the optical system (e.g., a microscope). The SLM provides additional degrees of freedom for controlling the microscope. Multi-focus acquisition is possible without any mechanical movement and thus accelerates and improves downstream tasks in an efficient manner.

[0317] FIG. 9 illustrates an exemplary optical system according to some embodiments. The exemplary optical system includes a light source 902 (e.g., an LED array), an objective lens 904, an SLM 906, and a camera 908. The optical system has a reflection mode. In the reflection mode, the light source 902 provides illumination to the biological sample 910, which generates reflected light. The reflected light travels through the objective lens 904 and is captured by the camera 908.

[0318] Referring to FIG. 9, the dashed line 912 indicates the intermediate image plane. As shown, the SLM is placed within the imaging path between the biological sample 910 and the camera 908. The SLM is configured to impose spatially varying modulation on the reflected light. For example, the SLM enables shaping of the alternative Fourier plane before the reflected light is focused onto the camera 908.

[0319] In some embodiments, both the light source 902 and the SLM 906 are programmable, thus enabling additional degrees of freedom for controlling and optimizing the optical system without mechanically moving the components of the optical system.

[0320] The configuration of the optical system in FIG. 9 is merely exemplary. Those skilled in the art will understand that other configurations of the optical system can also be used to install the SLM within the imaging path of the optical system and apply the optimization techniques described herein.

[0321] FIG. 10 illustrates an exemplary method of optics for analyzing an image of a biological sample using a programmable SLM of a system according to some embodiments. Process 1000 is implemented using, for example, one or more electronic devices implementing a software platform. In some examples, process 1000 is implemented using a client-server system, and the blocks of process 1000 are divided between the server and one or more client devices in any manner. In other examples, process 1000 is implemented using only one or more client devices. In process 1000, some blocks may optionally be combined, the order of some blocks may optionally be changed, and some blocks may optionally be omitted. In some examples, additional steps may be implemented in combination with process 1000. Thus, the operations as illustrated (and described in more detail below) are exemplary in nature and should not be considered limiting.

[0322] In block 1002, an exemplary system (e.g., one or more electronic devices) acquires a plurality of images of a biological sample (e.g., sample 910 in FIG. 9). The plurality of images are generated using a plurality of configurations of the SLM of the optical system (e.g., SLM 906 in FIG. 9). The SLM is located within the optical path between the biological sample and the image recording device (e.g., camera 908).

[0323] In some embodiments, at least one configuration of the SLM is for generating within an image that results in one or more optical aberrations. The optical aberrations can include spherical aberration, astigmatism, defocus, tilt, or any combination thereof. In some embodiments, more information can be captured within and / or derived from an image with optical aberrations. As an example, astigmatism enables the collection of multi-focal plane information within one image. As another example, defocus enables the system to scan a sample without any mechanical movement. An exemplary method for generating optical aberrations is provided in "Quantitative Phase Imaging and Complex Field Reconstruction by Pupil Modulation Differential Phase Contrast" by Lu et al., the content of which is incorporated herein by reference in its entirety.

[0324] In some embodiments, at least one configuration of the SLM is for emphasizing one or more features. The one or more features can comprise cell boundaries, actin filaments, nuclear shape, cytoplasmic compartmentalization, or any combination thereof. In some embodiments, the system provides neurite enhancement with a specific convolution kernel encoded within the transfer function of the microscope. In some embodiments, the system provides enhancement of the response function of the microscope for a particular liquid-phase separation biological object (protein, RNA, lipid). This enhancement will provide the ability to detect and characterize the composition of these objects. For example, using a spiral phase pattern as a phase filter in the Fourier plane results in a donut-shaped point spread function. Convolution with an extended amplitude or phase object leads to the enhancement of strong isotropic edges in the image. In a homogeneous sample (its homogeneous region), destructive interference occurs due to a π phase shift across the donut (for any angle along the ring). Structures within the sample can result in less incomplete cancellation and thus local brightening in the image. As a result, light is redistributed into the edges and boundaries within the sample. Exemplary methods for emphasizing features are provided in "Spiral phase contrast imaging in microscopy" by Furhapter et al., "Shadow effects in spiral phase contrast microscopy" by Furhapter et al., "Quantitative imaging of complex samples in spiral phase contrast microscopy" by Bernet et al., "Upgrading a microscope with a spiral phase plate" (the contents of which are incorporated herein by reference in their entirety).

[0325] In some embodiments, at least one configuration of the SLM is for reducing optical aberrations. For example, for live cell imaging and continuous monitoring of cells, it is important to reduce the variability of the sample arising from debris within the plate or well. The SLM can enable these aberrations to be corrected. In some embodiments, the SLM can be used for iterative correction of phase aberrations, for example, based on the Gerchberg-Saxton algorithm. For example, the dark center of a single optical vortex can be used as a critical sensor for residual phase aberrations. Exemplary methods for highlighting features are provided in "Wavefront correction of spatial light modulators using an optical vortex image" by Jesacher et al. and "Phase contrast microscopy with full numerical aperture illumination" by Maurer et al. (the contents of which are incorporated herein by reference in their entirety).

[0326] In some embodiments, the plurality of SLM configurations are for acquiring images of a biological sample at different depths. For example, the SLM enables, for example, flexible image multiplexing for combining images from different depths of the sample, or different settings of imaging parameters within one recorded image. Image multiplexing can facilitate quantitative phase microscopy. Imaging a live cell sample needs to be as fast as possible to minimize stress on the cells. Mechanical movement for 3D scanning of the sample is prohibitively expensive. The system enables Fourier ptychography to be implemented and the reconstruction of large 3D volumes from low-resolution images of the overall well. Exemplary methods of multiplexing are provided in "Depth-of-field-multiplexing in microscopy" by Maurer et al., "Differential interference contrast imaging using a spatial light modulator" by McIntyre et al., and "Quantitative SLM-based differential interference contrast Imaging" by McIntyre et al. (the contents of which are incorporated herein by reference in their entirety).

[0327] In some embodiments, the system enables the reconstruction of high-resolution images through optimized Fourier ptychography. Fourier ptychography is a computer imaging technique based on optical microscopy that involves the synthesis of a larger numerical aperture from a set of full-field images obtained using different optical settings, resulting in increased resolution compared to conventional microscopes. The images within the image set can be obtained using different configurations of the LED and / or SLM, and the obtained image set can then be combined into a final high-resolution image containing up to one billion pixels (gigapixels) with diffraction-limited resolution and a high spatial bandwidth product using an iterative phase retrieval algorithm.

[0328] The light source of the optical system can also be programmed. In some embodiments, multiple images are acquired at block 1002 using multiple configurations of the light source (e.g., light source 902) of the optical system. Exemplary methods for programming the light source are described herein with reference to FIGS. 1 and 2B, for example.

[0329] Returning to FIG. 10, at block 1004, the system inputs multiple images of the biological sample into a trained machine learning model and obtains one or more outputs.

[0330] In some embodiments, the trained model is an image conversion model configured to generate a second type of output image (e.g., an enhanced version of the input image) based on a first type of input image. In some embodiments, the enhanced version of the input image comprises enhanced cell phenotypes. In some embodiments, the trained model is a GAN model or a self-supervised model. For example, the trained model can be model 100 configured to receive a brightfield image and generate a fluorescence image.

[0331] In some embodiments, the trained model is a classification model configured to provide a classification output. For example, the model receives an input image and detects one or more pre-defined objects within the input image, such as diseased tissue.

[0332] FIG. 11 illustrates an exemplary method for training a machine learning model according to some embodiments. Process 1100 is implemented using one or more electronic devices that implement, for example, a software platform. In some examples, process 1100 is implemented using a client-server system, and the blocks of process 1100 are divided between a server and one or more client devices in any manner. In other examples, process 1100 is implemented using only one or more client devices. In process 1100, some blocks may optionally be combined, the order of some blocks may optionally be changed, and some blocks may optionally be omitted. In some examples, additional steps may be implemented in combination with process 1100. Thus, the operations as illustrated (and described in more detail below) are exemplary in nature and should not be considered limiting.

[0333] In block 1102, an exemplary system (e.g., one or more electronic devices) obtains a plurality of images of a biological sample. The plurality of images are generated using a plurality of configurations of an SLM of an optical system (e.g., SLM 906 in FIG. 9). The SLM is located within the optical path between the biological sample and an image recording device (e.g., camera 908).

[0334] In some embodiments, the SLM configurations are configured to create in the image resulting in a desired effect. As discussed above, some of the plurality of configurations of the SLM are for generating one or more optical aberrations (e.g., spherical aberration, astigmatism, defocus, distortion, tilt, or any combination thereof), for enhancing one or more features (e.g., cell boundaries, actin filaments, nuclear shape, cytoplasmic compartmentalization, or any combination thereof), for reducing optical aberrations, for obtaining images of the biological sample at different depths, and the like.

[0335] In block 1104, the system trains a machine learning model using multiple images. In some embodiments, the trained model is an image conversion model configured to generate a second type of output image (e.g., an enhanced version of the input image) based on a first type of input image. In some embodiments, the enhanced version of the input image comprises enhanced cell phenotypes. In some embodiments, the trained model is a GAN model or a self-supervised model. For example, the trained model can be model 100 configured to receive a brightfield image and generate a fluorescence image. An exemplary method of training the model is described herein with reference to FIGS. 2A-B and 3A-3D.

[0336] In some embodiments, the trained model is a classification model configured to provide a classification output. For example, the model can receive an input image and detect one or more predefined objects within the input image, such as a diseased tissue. Training of the classification model can be performed using multiple images and associated labels.

[0337] In some embodiments, in block 1104, the SLM can be iteratively programmed to identify an optimal SLM configuration for capturing images that leads to improved performance of a given machine learning model. Specifically, in block 1106, the system trains a machine learning model using a first image, which is acquired using a first configuration of the SLM of the optical system. In block 1108, the system evaluates the trained machine learning model. In block 1110, the system identifies a second configuration of the SLM based on the evaluation. In block 1112, the system trains the machine learning model using a second image, which is acquired using the second configuration of the SLM of the optical system.

[0338] For example, in block 1106, the system trains a model using images corresponding to a first set of SLM configurations. Each image results in a corresponding loss based on the model's loss function. In block 1108, the system determines the SLM configuration among the first set of SLM configurations that results in the minimum loss (e.g., generator loss). In block 1110, the SLM configuration that produced the minimum loss can be identified, and a new second set of SLM configurations can be identified as appropriate. For example, the new set of SLM configurations can include the best SLM configuration from the first set (i.e., the configuration that produced the minimum loss) and / or one or more new SLM configurations similar to the best SLM configuration. The new set of SLM configurations can also exclude from the first set the SLM configurations that resulted in the maximum loss. The new set of SLM configurations is loaded onto the optical system and can acquire additional training data. This step can be repeated until a threshold is met, e.g., until no further improvement (e.g., with respect to generator loss) is observed. The optimal SLM configuration is stored and can be used to acquire input images.

[0339] Steps 1106 - 1112 are described as part of a training process, but they can be performed to identify the optimal SLM configuration for generating input images at other stages of the pipeline (e.g., the inference stage). In some embodiments, the SLMs of the light source and the optical system can be programmed iteratively together to identify the best combination of illumination patterns and SLM configurations for generating input images.

[0340] Figures 9 - 11 illustrate optimization techniques using the SLM components of an optical system, but it should be understood that the SLM can be replaced with another hardware component that can modify the optical function (e.g., pupil function) of the system, such as a micromirror, without departing from the spirit of the invention.

[0341] Figures 12A and 12B illustrate a side-by-side comparison of the classification results of two classification models according to some embodiments. The classification model in FIG. 12A is trained using actual images (e.g., actual fluorescence images) captured by a microscope, while the classification model in FIG. 12B is trained using synthetic images (e.g., generated fluorescence images from brightfield images) generated using the techniques described herein. The classification model determines the chemical compound to which the tissue depicted in the input image responded. Specifically, each model is configured to receive an input image and output a classification result indicating one of 150 pre-defined chemical compounds. In the illustrated example, FIGS. 12A and 12B each show uniform manifold approximation and projection (UMAP), in which each input image is represented as a point within the UMAP. The color of the point represents the chemical compound to which the image is classified by the classification model. In some embodiments, the images input into the model in FIG. 12A are actual images, while the images input into the model in FIG. 12B are generated images.

[0342] Figures 12C and 12D illustrate a juxtaposed comparison of the same classification results in FIGS. 12A and 12B using different color schemes to demonstrate the resilience of two models to batch effects, respectively. A batch effect refers to a situation where subsets of data (i.e., batches) are significantly different in distribution due to non-related instrument-related factors. Batch effects are undesirable because they can introduce systematic errors, produce spurious results in downstream statistical analyses, and / or obscure the signal of interest. In each of FIGS. 12C and 12D, the input images belong to three different batches (e.g., from different plates or experiments), as indicated by different gray levels. As shown, FIG. 12D shows a greater overlap between points corresponding to the three batches, indicating that the generated images are more resilient to batch effects. FIG. 12E illustrates Euclidean distance measurements corresponding to an actual image set (e.g., input images as shown in FIGS. 12A and 12C), a generated image set (e.g., input images as shown in FIGS. 12B and 12D), and a true batch-invariant image set. The Euclidean distance measurements for each image set measure the Euclidean distance between image embeddings from two separate batches (e.g., actual image batch 1 and actual image batch 3 in FIG. 12C, generated image batch 1 and generated image batch 3 in FIG. 12D). For the true batch-invariant image set, the average score should be zero (i.e., no batch effect). As shown, the average score for the generated image set is lower than the average score for the actual images, thus demonstrating superior batch invariance.

[0343] Figures 13A and 13B illustrate exemplary generated phase images and exemplary generated fluorescence images according to some embodiments. FIG. 13A is a phase image generated from, for example, a brightfield image by the GAN model described herein. FIG. 13B is a fluorescence / body pai image generated from the same brightfield image by the GAN. The body pai image includes a virtual green hue and highlights the presence of biomarkers. FIG. 13C shows FIG. 13B overlaid on the phase image in FIG. 13A. Thus, the GAN model can be used for disease modeling. For example, a plurality of samples can be obtained and imaged from a subject. The resulting series of brightfield images can be analyzed by the GAN model to generate synthetic phase and fluorescence images and to consider disease perturbations.

[0344] FIGS. 14A-B illustrate an exemplary process for training a machine learning model (e.g., a GAN model) configured to generate synthetic data (e.g., image data) and identify an optimal illumination scheme for obtaining input data for the machine learning model. Process 1400 is implemented using one or more electronic devices implementing a software platform. In some examples, process 1400 is implemented using a client-server system and the blocks of process 1400 are divided between the server and one or more client devices in any manner. In other examples, process 1400 is implemented using only one or more client devices. In process 1400, some blocks may optionally be combined, the order of some blocks may optionally be changed, and some blocks may optionally be omitted. In some examples, additional steps may be implemented in combination with process 1400. Thus, the operations as illustrated (and described in more detail below) are exemplary in nature and should not be considered limiting.

[0345] In block 1402, an exemplary system (e.g., one or more electronic devices) receives a plurality of training images. The plurality of training images are actual images of biological samples, also referred to as ground truth data. The plurality of training images comprises a type of image data configured to be received by the GAN model and a type of image data configured to be output by the GAN model. For example, if the GAN model is configured to receive brightfield images and output fluorescence and phase images, the plurality of received images will include a plurality of brightfield training images 1402a (i.e., GAN input data type), a plurality of fluorescence training images 1402b (i.e., GAN output data type), and a plurality of phase training images (i.e., GAN output data type).

[0346] The plurality of brightfield training images 1402a can be captured by illuminating in vitro (or biopsy) cell samples with an inexpensive LED array using different illumination settings. The plurality of fluorescence training images 1402b can be captured after a dye has been applied to the biological sample (e.g., to enhance the visibility of biomarkers). The phase training images can be obtained using a physical or optical system-based model. It will be understood by those skilled in the art that the plurality of training images used in process 1400 can vary depending on the type of image data configured to be received and output by the GAN. In some embodiments, the plurality of training images comprises paired image data. For example, brightfield, fluorescence, and phase images of the same biological sample can be included within sets 1402a, 1402b, and 1402c, respectively.

[0347] In some embodiments, a plurality of training images are obtained to enable training of a GAN model such that synthetic images (e.g., synthetic fluorescence images, synthetic phase images) generated by the GAN model will provide the same performance as actual images (e.g., actual fluorescence images, actual phase images) in downstream analysis. In some embodiments, the downstream analysis includes a classification task of classifying an image to correspond to one of M classes. For example, the classification task can involve steps of classifying an image to correspond to a particular cell state from a plurality of cell state classes (e.g., healthy state, diseased state). As another example, the classification task can involve steps of classifying an image to correspond to a particular perturbation from a plurality of perturbation classes. To train the GAN model to generate synthetic images that can be classified as accurately as actual images, the training images include images corresponding to M classes (or conditions). For example, if the M classes include a healthy cell state class and a diseased cell state class, the plurality of brightfield training images 1402a can include a brightfield image depicting healthy cells and a brightfield image depicting diseased cells, the plurality of fluorescence training images 1402b can include a fluorescence image depicting healthy cells and a fluorescence image depicting diseased cells, and the plurality of phase training images 1402c can include a phase image depicting healthy cells and a phase image depicting diseased cells. For example, if the M classes include M perturbations, the plurality of brightfield training images 1402a can include brightfield images depicting the M perturbations, etc. Each training image can be labeled with a corresponding condition. For example, a phase image depicting a diseased cell state can be associated with a diseased label.

[0348] In an exemplary implementation, the plurality of training images include X fields of view per condition, i.e., a total of M×X fields of view. Specifically, in the plurality of bright-field images 1402a, each field of view includes N bright-field images captured using N illumination settings, thus resulting in a total of M×X×N bright-field images. In the plurality of fluorescence images 1402b, each field of view includes one fluorescence image, thus resulting in a total of M×X fluorescence images. In the plurality of phase images 1402c, each field of view includes one phase image, thus resulting in a total of M×X phase images. In some embodiments, the bright-field images have a magnification of m in whereas the fluorescence images have a magnification of m out ≥m in and the phase images have a magnification of m out ≥m in as well.

[0349] In block 1404, the system trains a classifier configured to receive an input image and output a classification result indicating one of the M conditions. For example, if the M conditions include a healthy condition and a diseased condition, the classifier is configured to receive an input image and output a classification result indicating either the healthy condition or the diseased condition. After being trained, the classifier is used during the training of the GAN model to ensure that the GAN model can generate synthetic image data that can be classified to the same or a similar level of accuracy as the actual image data, as described below.

[0350] In some embodiments, the classifier is trained using the same type of image data configured to be output by the GAN model. In the example depicted in FIGS. 14A and 14B, the GAN model is configured to output fluorescence images and phase images. Accordingly, the classifier trained at block 1404 is trained using a plurality of fluorescence training images 1402b and a plurality of phase training images 1402c. During training, each fluorescence image or phase image is input into the classifier to obtain a predicted classification result (e.g., healthy or diseased). The predicted classification result is then compared against the actual class associated with the training image (e.g., whether the training image actually depicts a healthy cell state or a diseased cell state), and based on the comparison, the classifier can be updated as appropriate. The classifier can be implemented using any classification algorithm such as a logistic regression model, a naive Bayes model, a decision tree model, a random forest model, a support vector machine model, etc.

[0351] At block 1406, the system trains the GAN model based on the training images. Block 1406 can include steps 1408a - 1408e, which can be repeated until training is complete (e.g., when convergence is reached). Steps 1408a - e are described below with reference to FIG. 15, which is a schematic diagram illustrating the steps according to some embodiments.

[0352] As shown in FIG. 15, the GAN model includes a multi - head attention layer 1502 comprising K sets of weighted matrices, specifically, weighted w 1 -w n a generator 1504, a discriminator 1508, and a classifier 1506 trained from block 1404. During the training of the GAN model, the classifier remains fixed while the generator, discriminator, and attention layer are updated as described below.

[0353] In block 1408a, the system applies each of the K sets of weights within the attention layer of the GAN model to a set of brightfield training images. The set of brightfield training images is obtained from a plurality of brightfield images 1402a. In some embodiments, the set of brightfield training images corresponds to the same field of view and depicts the same biological sample, but is captured using different illumination settings. For example, if the LED array comprises N illumination emitters (e.g., LEDs), each illumination emitter can be turned on one by one, and a brightfield image of the biological sample illuminated by each illumination emitter is captured, thus resulting in a set of N brightfield training images.

[0354] In the example depicted in FIG. 15, the system receives a set of N brightfield training images corresponding to illumination settings 1 - N. For example, the first brightfield image depicts a biological sample illuminated using illumination setting 1 (e.g., only the first LED in the array is turned on), the second brightfield image depicts a biological sample illuminated using illumination setting 2 (e.g., only the second LED in the array is turned on),... the Nth brightfield image depicts a biological sample illuminated using illumination setting N (e.g., only the Nth LED in the array is turned on).

[0355] The attention layer generates K sets of weights, each set comprising N weights. The weights w 1 -w n of each set are applied to the N images to generate a pooled image. For each set of weights, the attention layer 1502 assigns the sequential weights (e.g., normalized scalar weights) within the set to each of the set of brightfield training images. These weights correspond to the intensity values of the corresponding illumination settings (e.g., the corresponding LEDs). As shown, w 1 is applied to the first brightfield image (e.g., multiplied therewith), w 2 is applied to the second brightfield image, w nIt is applied to the Nth bright-field image. After the weighting is applied, the weighted image can be aggregated (e.g., summed) to obtain one aggregated bright-field image. Since there are K sets of weightings, K aggregated images 1512 can be generated. In some embodiments, the attention layer is a adapted multi-head attention layer. The attention mechanism enables the natural generation of K linear combinations (i.e., aggregated images) of the bright-field images. These aggregated images are fed to the rest of the network as described herein.

[0356] In block 1408b, the system inputs each of the aggregated bright-field images into the GAN model. Referring to FIG. 15, each of the aggregated bright-field images 1512 is input into the generator 1504, which outputs a synthetic fluorescence image 1514a and a synthetic phase image 1514b. The generator can be implemented in a similar manner to the generator described above with reference to FIGS. 3A-D.

[0357] During training, the generator outputs (i.e., the generated fluorescence image 1514a and the generated phase image 1514b) can be directly connected to the discriminator input. The discriminator 1508 is trained to distinguish between the generated images and the actual images. During training, the generated images are input into the discriminator 1508 as described above, and discriminator loss and generator loss can be obtained. Further, the actual images are also input into the discriminator 1508, and discriminator loss and generator loss can be generated. The actual images can be the actual fluorescence image 1516a (from the plurality of fluorescence training images 1402b in FIG. 14A) corresponding to the same field of view as the input bright-field image, or the actual phase image 1516b (from the plurality of phase training images 1402c in FIG. 14A) corresponding to the same field of view as the input bright-field image.

[0358] In some embodiments, the discriminator loss function is the Wasserstein discriminator loss and is calculated as follows.

[0359] [Chemical formula]

[0360] In the formula, f(x) is the output of the discriminator based on the wavelet coefficients of the real fluorescence or phase image, w is the model weight of the discriminator, m is the size of the mini-batch, f is the discriminator model, x is the actual image, z is the input (bright-field image 1512), G is the generator model, and f(G(z)) is the output of the discriminator based on the predicted wavelet coefficients corresponding to the synthetic fluorescence or phase image.

[0361] In some embodiments, the generator loss function is the Wasserstein generator loss and is calculated as follows.

[0362] [Chemical formula]

[0363] In the formula, f(x) is the output of the discriminator based on the wavelet coefficients of the real fluorescence or phase image, m is the size of the mini-batch, f is the discriminator model, z is the input (bright-field image 1512), G is the generator model, and f(G(z)) is the output of the discriminator based on the predicted wavelet coefficients.

[0364] In block 1408c, the system inputs each generated image and the actual image corresponding to the generated image into the trained classifier and obtains a classifier loss. For example, the generated fluorescence image 1514a is input into the classifier 1506 to obtain a first classification result, the actual fluorescence image 1516a is input into the classifier 1506 to obtain a second classification result, and the classifier loss can be calculated based on the difference between the first classification result and the second classification result. As another example, the generated phase image 1514b is input into the classifier 1506 to obtain a third classification result, the actual phase image 1516b is input into the classifier 1506 to obtain a fourth classification result, and the classifier loss can be calculated based on the difference between the third classification result and the fourth classification result.

[0365] In block 1408d, the system extends the generator loss based on the classifier loss. For example, the generator loss can be extended using the L2 norm of the classification scores from the actual images. In some embodiments, if the classifier is not available or the classification is not desired, the classifier loss is replaced by a constant (e.g., 0).

[0366] In block 1408e, the system updates the GAN model based on the extended generator loss. Backpropagation follows the same procedure as described above. The discriminator updates its weights through backpropagation based on the discriminator loss through the discriminator network. Further, the extended generator loss is backpropagated to update the weights in the attention layer (e.g., the weights used to calculate the aggregated image) and the generator. For example, the generated loss calculated based on the aggregated image corresponding to the K-th set of weights can be used to update the K-th set of weights in the attention layer.

[0367] In block 1410, the system obtains one or more optimal illumination patterns based on the weights in the attention layer of the trained GAN model. As described above, the weights in the attention layer (e.g., w 1 -wn ) can indicate the intensity value of the corresponding lighting setting (e.g., the LEDs in the corresponding LED array). As discussed above, the attention layer can be a multi-head attention layer that provides a set of K weights (i.e., K linear combinations) and thus results in K lighting patterns.

[0368] In some embodiments, before block 1406, the generator and discriminator of the GAN model are pre-trained using brightfield images, in which all the LEDs in the LED array are turned on. After pre-training, while block 1406 is implemented to update the attention weights, the generator and discriminator remain fixed. The optimal combination of lighting can be obtained based on the updated weights.

[0369] FIG. 16A illustrates a synthetic image generated by an exemplary GAN model to consider NASH disease according to some embodiments. In the depicted example, Hepg2 cells with an NASH genetic background are illuminated using the optimal lighting pattern identified using the techniques described herein, and a brightfield image is captured. The brightfield image is input into the GAN model, which outputs the generated phase image and the generated fluorescence / body pie image in FIG. 16A. The GAN model used in FIG. 16A can be configured to include a trained classifier as described with reference to FIGS. 14A - B and 15. The classifier has two conditions, namely, a healthy condition (e.g., without perturbation) and an affected condition (e.g., inflammatory reaction mixture + fatty acid). As described, the GAN model can be trained such that the generated image can be classified by the classifier to an accuracy where it is the same or similar to the actual image.

[0370] The synthetic images generated by the GAN model can be used to create a disease model. FIG. 16B illustrates the downstream analysis of the generated images according to some embodiments. As shown, the generated images can be used to perform nuclear compartmentalization and speckle detection (e.g., using an image processing algorithm). Thus, the generated images can be used to create a disease model for NASH disease and to consider chemical perturbations in NASH.

[0371] The synthetic images generated by the GAN model can also be used to evaluate the effectiveness of treatment. The system processes three groups of generated images, namely, a first group of images depicting healthy tissue without disease, a second group of images depicting untreated diseased tissue, and a third group of images depicting diseased tissue that has been treated (e.g., using a specific drug). In the example depicted in FIG. 16C, the first group of images (labeled "untreated") comprises images of healthy tissue without non-alcoholic steatohepatitis (NASH) disease, the second group of images (labeled "NASH 2X") comprises images of untreated tissue with NASH disease, and the third group of images (labeled "NASH 2X + ACC inhibitor") comprises images of tissue with NASH that has been treated with a drug (e.g., 50 μM ACC inhibitor or filsocostat). In some embodiments, the three groups of images capture tissue of the same subject at different times. In some embodiments, the three groups of images capture tissue of different subjects. The images can be phase images or fluorescence images generated using the techniques described herein (e.g., from brightfield images). Each phase or fluorescence image is generated based on a plurality of brightfield images and can thus have more information such as phase shift information. Therefore, the generated phase or fluorescence images can enable higher accuracy in downstream analysis, as described below.

[0372] Specifically, to evaluate the effectiveness of a drug, the system generates a distribution for each group of images and determines whether the distribution reflects the effect of the drug on the disease state. In FIG. 16C, three probability distributions are generated using a classifier trained on the generated images. The X-axis indicates the probability output by the classifier in response to receiving an input image (e.g., a generated phase image). As shown by the three distributions, the generated image of healthy tissue (i.e., distribution 1620) is generally classified as having a lower probability of disease than the generated image of diseased tissue (i.e., distribution 1624). Further, the generated image of treated tissue (i.e., distribution 1622) is generally classified as having a lower probability of disease than the generated image of diseased tissue. Comparison of these distributions can indicate that the drug is effective in regressing or reducing the disease state because the treatment has made the diseased tissue contain features more similar to the healthy state and less similar to the diseased state. Thus, the synthetic images can enable a biophysical understanding of the distribution of lipids within cells and provide insights for treatment.

[0373] In some embodiments, rather than using distributions, the system can identify image clusters within an embedding space (e.g., UMAP), as shown in FIG. 16D. In UMAP, each point represents an image embedding of an image (e.g., a generated phase image). As shown, the generated images of treated tissue form clusters that move away from the cluster of generated diseased images and towards the cluster of generated healthy images, which can indicate that the drug is effective in regressing or reducing the disease state.

[0374] The analysis in FIGS. 16C and 16D can be applied to evaluate multiple drug candidates for a target disease. For example, the tissue treated by each drug candidate can be imaged by a microscope, for example, to obtain a bright-field image. The bright-field image can be converted into a phase image using the techniques described herein. A distribution or cluster can be generated for the phase image of each treatment. Thus, the resulting plot can comprise a distribution or cluster representing the diseased state, a distribution or cluster representing the healthy state, and a plurality of distributions or clusters representing the candidate drugs, respectively. The system can then identify the distribution or cluster that is closest to the healthy state and identify the most effective candidate drug candidate.

[0375] FIG. 17A illustrates a synthetic image generated by an exemplary GAN model to consider tuberous sclerosis complex (“TSC”) according to some embodiments. In the illustrated example, NGN2 neurons are illuminated using the optimal illumination pattern identified using the techniques described herein, and a bright-field image is captured. The bright-field image is input into the GAN model, which outputs the generated phase image and the generated fluorescence / body pie image in FIG. 17A. The GAN model used in FIG. 17A can be configured to include a trained classifier as described with reference to FIGS. 14A-B and 15. The classifier has two conditions, namely, a healthy condition (e.g., wild type) and a diseased condition (e.g., TSC KO). As described, the GAN model can be trained such that the generated image can be classified by the classifier to the same or similar accuracy as the actual image.

[0376] The synthetic images generated by the GAN model can also be used to evaluate the effectiveness of treatment. The system processes three groups of generated images, namely, a first group of images depicting healthy tissue without disease, a second group of images depicting untreated diseased tissue, and a third group of images depicting diseased tissue that has been treated (e.g., using a specific drug). In the embodiment depicted in FIG. 17B, the first group of images (labeled as "wild type") comprises images of healthy tissue without the TSC disease, the second group of images (labeled as "TSC") comprises images of untreated tissue with the TSC disease, and the third group of images (labeled as "TSC + rapamycin") comprises images of tissue with TSC that has been treated with a drug. In some embodiments, the three groups of images capture tissue of the same subject at different times. In some embodiments, the three groups of images capture tissue of different subjects. The images are phase images generated using the techniques described herein (e.g., from brightfield images).

[0377] Specifically, to evaluate the effectiveness of a drug, the system generates a distribution for each group of images and determines whether the distribution reflects the effect of the drug on the disease. In FIG. 17B, three biomarker distributions are generated. As shown by these distributions, the drug is thought to be effective in regressing or reducing the disease state because the treatment causes the diseased tissue to contain features more similar to the healthy state and less similar to the diseased state. The analysis in FIG. 17B can be applied to evaluate multiple drug candidates for the disease of interest, as described above.

[0378] Exemplary methods, non-transitory computer-readable storage media, systems, and electronic devices are described in the following items. 1. A method for training a machine learning model and generating an image of a biological sample, the method comprising: acquiring a plurality of training images, wherein the plurality of training images comprise: a first type of training image, and a second type of training image and comprising, generating, using the machine learning model, a plurality of wavelet coefficients based on the first type of training image; generating a second type of synthetic image based on the plurality of wavelet coefficients; comparing the second type of synthetic image with the second type of training image; updating the machine learning model based on the comparison; A method comprising. 2. The method according to item 1, wherein the first type of training image is a bright field image of a biological sample. 3. The method according to item 2, wherein the second type of training image is a fluorescence image of the biological sample. 4. The method according to any one of items 1-3, wherein the machine learning model comprises a generator and a discriminator. 5. The method according to item 4, wherein the machine learning model comprises a conditional GAN model. 6. The method according to any one of items 4-5, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups. 7. The method according to item 6, wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group. 8. The method according to any one of items 6-7, wherein the plurality of neural networks comprises a plurality of U-Net neural networks. 9. The method according to any one of items 5-8, wherein the discriminator is a PatchGAN neural network. 10. The method according to any one of items 1-9, further comprising generating a third type of image based on the first type of training image. 11. The method according to item 10, wherein the third type of image is a phase-shifted image. 12. The method according to any one of items 1-11, further comprising generating a fourth type of image based on the first type of training image. 13. The method according to item 12, wherein the fourth type of image comprises segmented data. 14. The method according to any one of items 1-13, wherein the first type of training image is captured using a microscope according to a first illumination scheme. 15. The method according to item 14, wherein the first illumination scheme comprises one or more illumination patterns. 16. The method according to any one of items 14-15, wherein the first type of training image is part of a bright-field image array. 17. The plurality of training images are a first plurality of training images, and the method further comprises: identifying a second illumination scheme based on the comparison; and obtaining a second plurality of training images comprising one or more images of the first type, wherein the one or more images of the first type are obtained based on the second illumination scheme; and training the machine learning model based on the second plurality of training images. The method according to any one of items 14-16, comprising: 18. obtaining a plurality of images of the first type using a microscope; and generating a plurality of synthetic images of the second type using the machine learning model based on the plurality of obtained images. The method according to any one of items 1-16, further comprising: 19. The method according to item 18, further comprising training a classifier based on the plurality of synthetic images of the second type. 20. The microscope is the first microscope, and the classifier is the first classifier, using a second microscope to obtain a plurality of images of the second type; training a second classifier based on the plurality of images of the second type; comparing the performance of the first classifier and the second classifier; The method according to item 19, further comprising. 21. The method according to item 20, wherein the second microscope is a fluorescence microscope. 22. A method for generating an enhanced image of a biological sample, comprising: obtaining an image of a biological sample using a microscope; generating an enhanced image of the biological sample using a machine learning model based on the image, wherein the machine learning model: obtaining a plurality of training images, the plurality of training images comprising: a first type of training image; and a second type of training image; generating a plurality of wavelet coefficients using the machine learning model based on the first type of training image; generating a composite image of the second type based on the plurality of wavelet coefficients; comparing the composite image of the second type with the training images of the second type; updating the machine learning model based on the comparison; and being trained by; including. A method. 23. The method according to item 22, wherein the first type of training image is a bright-field image of a biological sample. 24. The method according to item 22, wherein the second type of training image is a fluorescence image of the biological sample. 25. The machine learning model includes a generator and a discriminator, and is the method according to any one of items 22-24. 26. The machine learning model includes a conditional GAN model, and is the method according to item 25. 27. The generator includes a plurality of neural networks corresponding to a plurality of frequency groups, and is the method according to any one of items 25-26. 28. Each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group, and is the method according to item 27. 29. The plurality of neural networks includes a plurality of U-Net neural networks, and is the method according to any one of items 27-28. 30. The discriminator is a PatchGAN neural network, and is the method according to any one of items 26-29. 31. The method according to any one of items 23-30, further including generating a third type of image based on the first type of training image. 32. The third type of image is a phase shift image, and is the method according to item 31. 33. The method according to any one of items 23-32, further including generating a fourth type of image based on the first type of training image. 34. The fourth type of image includes segmented data, and is the method according to item 33. 35. The first type of training image is captured using a microscope according to a first illumination scheme, and is the method according to any one of items 23-34. 36. The first illumination scheme includes one or more illumination patterns, and is the method according to item 35. 37. The training image of the first type is the method according to any one of items 35-36, which is part of the bright-field image array. 38. The plurality of training images are a first plurality of training images, and the method further includes identifying a second illumination scheme based on the comparison; and obtaining a second plurality of training images including one or more images of the first type, wherein the one or more images of the first type are obtained based on the second illumination scheme; and training the machine learning model based on the second plurality of training images The method according to any one of items 35-37, including. 39. obtaining a plurality of images of the first type using a microscope; and generating a plurality of synthetic images of the second type using the machine learning model based on the plurality of obtained images The method according to any one of items 35-38, further including. 40. The method according to item 39, further including training a classifier based on the plurality of synthetic images of the second type. 41. The microscope is a first microscope, the classifier is a first classifier, obtaining a plurality of images of the second type using a second microscope; and training a second classifier based on the plurality of images of the second type; and comparing the performance of the first classifier and the second classifier The method according to item 40, further including. 42. The method according to item 41, wherein the second microscope is a fluorescence microscope. 43. A system for training a machine learning model and generating an image of a biological sample, comprising A computing system comprising one or more processors and one or more memories storing a machine learning model, the computing system being configured to receive a plurality of training images of a first type and one training image of a second type, the computing system generating, using the machine learning model, a plurality of wavelet coefficients based on the training images of the first type; generating a synthetic image of the second type based on the plurality of wavelet coefficients; comparing the synthetic image of the second type with the training image of the second type; updating the machine learning model based on the comparison; and being configured to perform. A system comprising. 44. The system according to item 43, wherein the training images of the first type are bright-field images of biological samples. 45. The system according to any one of items 43-44, wherein the training image of the second type is a fluorescence image of the biological sample. 46. The system according to any one of items 43-45, wherein the machine learning model comprises a generator and a discriminator. 47. The system according to item 46, wherein the machine learning model comprises a conditional GAN model. 48. The system according to any one of items 46-47, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups. 49. The system according to item 48, wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group. 50. The above-mentioned plurality of neural networks is the system according to any one of items 48-49, comprising a plurality of U-Net neural networks. 51. The above-mentioned discriminator is the PatchGAN neural network, and the system according to any one of items 46-50. 52. The above-mentioned computing system is further the system according to any one of items 43-51, configured to generate a third type of image based on the above-mentioned first type of training image. 53. In the system according to item 52, the above-mentioned third type of image is a phase-shifted image. 54. The above-mentioned computing system is further the system according to any one of items 43-53, configured to generate a fourth type of image based on the above-mentioned first type of training image. 55. In the system according to item 54, the above-mentioned fourth type of image includes segmented data. 56. In the system according to any one of items 43-55, the above-mentioned first type of training image is captured using a microscope according to a first illumination scheme. 57. In the system according to item 56, the above-mentioned first illumination scheme includes one or more illumination patterns. 58. In the system according to any one of items 56-57, the above-mentioned first type of training image is part of a bright-field image array. 59. The above-mentioned plurality of training images is a first plurality of training images, and the above-mentioned computing system is further identifying a second illumination scheme based on the above-mentioned comparison; and acquiring a second plurality of training images including one or more images of the above-mentioned first type, where the one or more images of the above-mentioned first type are acquired based on the above-mentioned second illumination scheme. Training the machine learning model based on the second plurality of training images The system according to any one of items 56 - 58, configured to perform the above. 60. The computing system further Using a microscope to obtain a plurality of images of the first type, and Based on the plurality of acquired images, using the machine learning model to generate a plurality of composite images of the second type The system according to any one of items 43 - 59, configured to perform the above. 61. The computing system is further configured to train a classifier based on the plurality of composite images of the second type, as described in item 59 of the system. 62. The microscope is a first microscope, the classifier is a first classifier, and the computing system further Using a second microscope to obtain a plurality of images of the second type, and Based on the plurality of images of the second type, training a second classifier, and Comparing the performance of the first classifier and the second classifier The system according to item 61, configured to perform the above. 63. The system according to item 62, wherein the second microscope is a fluorescence microscope. 64. A system for generating an enhanced image of a biological sample, comprising A computing system comprising one or more processors and one or more memories storing a machine learning model, the computing system receiving an image of a biological sample acquired from a microscope and configured to generate an enhanced image of the biological sample using the machine learning model based on the image, wherein the machine learning model Is to obtain a plurality of training images, and the plurality of training images a first type of training image, a second type of training image, and, generating a plurality of wavelet coefficients using the machine learning model based on the first type of training image; generating a synthetic image of the second type based on the plurality of wavelet coefficients; comparing the synthetic image of the second type with the training image of the second type; updating the machine learning model based on the comparison; and a computing system trained by comprising. 65. The system according to item 64, wherein the training image of the first type is a bright-field image of a biological sample. 66. The system according to item 64, wherein the training image of the second type is a fluorescence image of the biological sample. 67. The system according to any one of items 64-66, wherein the machine learning model comprises a generator and a discriminator. 68. The system according to item 67, wherein the machine learning model comprises a conditional GAN model. 69. The system according to any one of items 67-68, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups. 70. The system according to item 69, wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group. 71. The system according to any one of items 69-70, wherein the plurality of neural networks comprises a plurality of U-Net neural networks. 72. The discriminator is the system according to any one of items 67-71, which is a PatchGAN neural network. 73. The machine learning model is further trained by generating a third type of image based on the first type of training images, and is the system according to any one of items 64-72. 74. In the system according to item 73, the third type of image is a phase shift image. 75. The machine learning model is trained by generating a fourth type of image based on the first type of training images, and is the system according to any one of items 64-74. 76. In the system according to item 75, the fourth type of image includes segmented data. 77. The first type of training images are captured using a microscope according to a first illumination scheme, and is the system according to any one of items 64-76. 78. In the system according to item 77, the first illumination scheme includes one or more illumination patterns. 79. The first type of training images are part of a bright field image array, and is the system according to any one of items 77-78. 80. The plurality of training images are a first plurality of training images, and the machine learning model identifies a second illumination scheme based on the comparison, and obtains a second plurality of training images including one or more images of the first type, where the one or more images of the first type are obtained based on the second illumination scheme, and trains the machine learning model based on the second plurality of training images and is trained by the above, and is the system according to any one of items 77-79. 81. The machine learning model Using a microscope, obtaining a plurality of images of the first type, Based on the plurality of obtained images, using the machine learning model to generate a plurality of composite images of the second type, The system according to any one of items 77-80, which is trained by the above. 82. The system according to item 81, wherein the machine learning model is trained by training a classifier based on a plurality of composite images of the second type. 83. The microscope is a first microscope, the classifier is a first classifier, and the machine learning model is Using a second microscope to obtain a plurality of images of the second type, Based on the plurality of images of the second type, training a second classifier, Comparing the performance of the first classifier and the second classifier, The system according to item 82, which is trained by the above. 84. The system according to item 83, wherein the second microscope is a fluorescence microscope. 85. A method for processing an image of a biological sample and obtaining one or more output images, comprising: Using a plurality of configurations of an SLM of an optical system to obtain a plurality of images of the biological sample, wherein the SLM is located in an optical path between the biological sample and an image recording device; Inputting the plurality of images of the biological sample into a trained machine learning model and obtaining the one or more output images. A method comprising the above. 86. The method according to item 85, wherein at least one of the plurality of configurations of the SLM is for generating one or more optical aberrations. 87. Generating one or more optical aberrations includes the method according to item 86, including spherical aberration, astigmatism, field curvature, distortion, tilt, or any combination thereof. 88. Among the plurality of configurations of the SLM, at least one configuration is for emphasizing one or more features, according to the method described in any of items 85 - 86. 89. The one or more features include cell boundaries, actin filaments, nuclear shape, cytoplasmic compartmentalization, or any combination thereof, according to the method described in item 88. 90. Among the plurality of configurations of the SLM, at least one configuration is for reducing optical aberrations, according to the method described in any of items 85 - 89. 91. The plurality of SLM configurations are for acquiring images of the biological sample at different depths, according to the method described in any of items 85 - 90. 92. The machine learning model is configured to generate a second type of image based on a first type of image, according to the method described in any of items 85 - 91. 93. The first type of image is a bright - field image, according to the method described in item 92. 94. The second type of image is a fluorescence image, according to the method described in item 92. 95. The second type of image is an enhanced version of the first type of image, according to the method described in item 92. 96. The machine learning model is a GAN model or a self - supervised model, according to the method described in any of items 92 - 95. 97. The plurality of images are acquired using a plurality of configurations of the light source of the optical system, according to the method described in any of items 85 - 96. 98. The light source is an LED array of the optical system, according to the method described in item 97. 99. At least one of the plurality of SLM configurations is training the machine learning model, evaluating the trained machine learning model, and identifying the at least one configuration based on the evaluation The method according to any one of items 85-98, obtained by 100. The method according to any one of items 85-99, wherein the trained machine learning model is configured to receive an input image and output an enhanced version of the input image. 101. The method according to item 100, wherein the enhanced version of the input image comprises one or more enhanced cell phenotypes. 102. An electronic device for processing an image of a biological sample and obtaining one or more output images, comprising one or more processors, a memory, one or more programs, which are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are obtaining a plurality of images of the biological sample using a plurality of configurations of an SLM of an optical system, the SLM being located in an optical path between the biological sample and an image recording device, inputting the plurality of images of the biological sample into a trained machine learning model and obtaining the one or more output images one or more programs including instructions for performing An electronic device comprising 103. A non-transitory computer-readable storage medium storing one or more programs for processing an image of a biological sample and obtaining one or more output images, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to acquire a plurality of images of the biological sample using a plurality of configurations of an SLM of an optical system, the SLM being located in an optical path between the biological sample and an image recording device, and input the plurality of images of the biological sample into a trained machine learning model and obtain the one or more output images and perform. A non-transitory computer-readable storage medium. 104. A method for classifying an image of a biological sample, comprising: acquire a plurality of images of the biological sample using a plurality of configurations of an SLM of an optical system, the SLM being located in an optical path between the biological sample and an image recording device, and input the plurality of images of the biological sample into a trained machine learning model and obtain one or more classification outputs and include. A method. 105. The method according to item 104, wherein at least one of the plurality of configurations of the SLM is for generating one or more optical aberrations. 106. The method according to item 105, wherein generating one or more optical aberrations includes spherical aberration, astigmatism, defocus, distortion, tilt, or any combination thereof. 107. The method according to any one of items 104-106, wherein at least one of the plurality of configurations of the SLM is for enhancing one or more features. 108. The method according to item 107, wherein the one or more features include cell boundaries, actin filaments, nuclear shape, cytoplasmic compartmentalization, or any combination thereof. 109. The method according to any one of items 104-108, wherein at least one of the plurality of configurations of the SLM is for reducing optical aberration. 110. The method according to any one of items 104-109, wherein the plurality of SLM configurations are for acquiring images of the biological sample at different depths. 111. The method according to any one of items 104-110, wherein the plurality of images are acquired using a plurality of configurations of a light source of the optical system. 112. The method according to item 111, wherein the light source is an LED array of the optical system. 113. At least one of the plurality of SLM configurations is training the machine learning model, evaluating the trained machine learning model, identifying the at least one configuration based on the evaluation and is obtained by the method according to any one of items 104-112. 114. The method according to any one of items 104-113, wherein the trained machine learning model is configured to receive an input image and detect one or more predefined objects within the input image. 115. The method according to item 114, wherein the predefined object includes diseased tissue. 116. An electronic device for classifying images of a biological sample, one or more processors, a memory, One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs are to obtain a plurality of images of the biological sample using a plurality of configurations of the SLM of the optical system, wherein the SLM is located in the optical path between the biological sample and the image recording device, and inputting the plurality of images of the biological sample into a trained machine learning model to obtain one or more classification outputs One or more programs including instructions for performing, and An electronic device comprising. 117. A non-transitory computer-readable storage medium storing one or more programs for classifying images of a biological sample, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to obtain a plurality of images of the biological sample using a plurality of configurations of the SLM of the optical system, wherein the SLM is located in the optical path between the biological sample and the image recording device, and input the plurality of images of the biological sample into a trained machine learning model to obtain one or more classification outputs A non-transitory computer-readable storage medium for causing the operation. 118. A method for training a machine learning model, comprising obtaining a plurality of images of a biological sample using a plurality of configurations of the SLM of the optical system, wherein the SLM is located in the optical path between the biological sample and the image recording device, and training the machine learning model using the plurality of images A method including. 119. The method according to item 118, wherein at least one of the plurality of configurations of the SLM is for generating one or more optical aberrations. 120. The method according to item 119, wherein generating one or more optical aberrations includes spherical aberration, astigmatism, defocus, distortion, tilt, or any combination thereof. 121. The method according to any one of items 118 - 120, wherein at least one of the plurality of configurations of the SLM is for enhancing one or more features. 122. The method according to item 121, wherein the one or more features include cell boundaries, actin filaments, nuclear shape, cytoplasmic compartmentalization, or any combination thereof. 123. The method according to any one of items 118 - 122, wherein at least one of the plurality of configurations of the SLM is for reducing optical aberrations. 124. The method according to any one of items 118 - 123, wherein at least one of the plurality of configurations of the SLM is for acquiring images of the biological sample at different depths. 125. The method according to any one of items 118 - 124, wherein the machine learning model is configured to generate a second type of image based on a first type of image. 126. The method according to item 125, wherein the first type of image is a bright - field image. 127. The method according to item 125, wherein the second type of image is a fluorescence image. 128. The method according to any one of items 118 - 127, wherein the machine learning model is a GAN model or a self - supervised model. 129. The method according to any one of items 118 - 128, wherein the machine learning model is a classification model. 130. The plurality of images are obtained using a plurality of configurations of a light source of the optical system, the method according to any one of items 118 - 129. 131. The light source is an LED array of the optical system, the method according to item 130. 132. Training the machine learning model comprises: (a) training the machine learning model using a first image, the first image being obtained using a first configuration of an SLM of the optical system; (b) evaluating the trained machine learning model; (c) identifying a second configuration of the SLM based on the evaluation; (d) training the machine learning model using a second image, the second image being obtained using the second configuration of the SLM of the optical system. The method according to any one of items 118 - 131. 133. The evaluation is based on a loss function of the machine learning model, the method according to item 112. 134. The method according to any one of items 112 - 113, further comprising repeating steps (a) - (d) until a threshold is met. 135. The threshold indicates convergence of the training, the method according to item 114. 136. The trained machine learning model is configured to receive an input image and output an enhanced version of the input image, the method according to any one of items 118 - 135. 137. The enhanced version of the input image comprises one or more enhanced cell phenotypes, the method according to item 136. 138. The method according to any one of items 118-135, wherein the trained machine learning model is configured to receive an input image and detect one or more pre-defined objects within the input image. 139. The method according to item 138, wherein the pre-defined object includes diseased tissue. 140. An electronic device for training a machine learning model, One or more processors, A memory, One or more programs, which are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include Acquiring a plurality of images of a biological sample using a plurality of configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device, Training the machine learning model using the plurality of images One or more programs including instructions for performing the above An electronic device comprising the above. 141. A non-transitory computer-readable storage medium storing one or more programs for training a machine learning model, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to Acquire a plurality of images of a biological sample using a plurality of configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device, Train the machine learning model using the plurality of images A non-transitory computer-readable storage medium for causing the above to be performed. 142. A method for generating an enhanced image of a biological sample, obtaining an image of a biological sample illuminated using an illumination pattern of an illumination source using a microscope, wherein the illumination pattern is training a classification model configured to receive an input image and output a classification result, training a machine learning model having a plurality of weights corresponding to a plurality of illumination settings using the trained classification model, identifying the illumination pattern based on the plurality of weights of the trained machine learning model, and as determined by generating an enhanced image of the biological sample by inputting the obtained image of the biological sample into the trained machine learning model, and a method comprising 143. The method according to item 142, wherein the obtained image is a bright field image. 144. The method according to any one of items 142-143, wherein the enhanced image is a fluorescence image, a phase image, or a combination thereof. 145. The method according to any one of items 142-144, wherein the illumination source comprises an array of illumination emitters. 146. The method according to item 145, wherein the illumination source is an LED array. 147. The method according to any one of items 142-146, wherein the illumination pattern indicates whether each illumination emitter is turned on or off and the intensity of each illumination emitter. 148. For each illumination setting of the plurality of illumination settings, the illumination setting corresponds to an individual illumination emitter of the illumination source, and each weight corresponds to the intensity of the individual illumination emitter. The method according to any one of items 145-147. 149. The method according to any one of items 142-148, wherein the classification model is configured to receive an input phase image or an input fluorescence image and output a classification result indicating one of a plurality of pre-defined classes. 150. The method according to item 149, wherein the plurality of pre-defined classes includes a healthy class and an affected class. 151. The method according to item 150, wherein the machine learning model is a GAN model including an attention layer with the plurality of weights, a discriminator, and a generator. 152. The method according to item 151, wherein the machine learning model is a conditional GAN model. 153. The method according to any one of items 151-152, wherein the generator includes a plurality of neural networks corresponding to a plurality of frequency groups. 154. The method according to item 153, wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group. 155. The method according to any one of items 153-154, wherein the plurality of neural networks includes a plurality of U-Net neural networks. 156. The method according to any one of items 151-155, wherein the discriminator is a PatchGAN neural network. 157. Training the machine learning model using the trained classification model includes applying the plurality of weights to a plurality of bright-field training images, aggregating the plurality of weighted bright-field training images into an aggregated bright-field image, inputting the aggregated bright-field training image into the machine learning model to obtain an enhanced training image and a generator loss, inputting the enhanced training image into the trained classifier to obtain a classifier loss, extending the generator loss based on the classifier loss, updating the plurality of weights based on the extended generator loss The method according to any one of items 151 - 156, including 158. The method according to any one of items 142 - 157, further including classifying the emphasized image using the trained classifier. 159. The method according to any one of items 142 - 158, further including displaying the emphasized image. 160. A system for generating an emphasized image of a biological sample, One or more processors, A memory, One or more programs, which are stored in the memory and configured to be executed by the one or more processors, and the one or more programs Acquiring an image of a biological sample illuminated using an illumination pattern of an illumination source using a microscope, where the illumination pattern Training a classification model configured to receive an input image and output a classification result, Training a machine learning model having a plurality of weights corresponding to a plurality of illumination settings using the trained classification model, Identifying the illumination pattern based on the plurality of weights of the trained machine learning model as determined by Generating an emphasized image of the biological sample by inputting the acquired image of the biological sample into the trained machine learning model and including instructions for performing A system comprising 161. A non-transitory computer-readable storage medium storing one or more programs for generating an enhanced image of a biological sample, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to acquire an image of a biological sample illuminated using an illumination pattern of an illumination source using a microscope, the illumination pattern train a classification model configured to receive an input image and output a classification result; using the trained classification model, train a machine learning model having a plurality of weights corresponding to a plurality of illumination settings; identify the illumination pattern based on the plurality of weights of the trained machine learning model; as determined by; generate an enhanced image of the biological sample by inputting the acquired image of the biological sample into the trained machine learning model; A non-transitory computer-readable storage medium that causes the above to be performed. 162. A method for evaluating a treatment for a target disease, receiving a first plurality of images depicting a first set of healthy biological samples not affected by the target disease; receiving a second plurality of images depicting a second set of untreated biological samples affected by the target disease; receiving a third plurality of images depicting a third set of treated biological samples affected by the target disease and treated by the treatment; inputting the first plurality of images into a trained machine learning model to obtain a first plurality of enhanced images; inputting the second plurality of images into the trained machine learning model to obtain a second plurality of enhanced images; Input the third plurality of images into the trained machine learning model to obtain a third plurality of enhanced images; Compare the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment; A method comprising the above. 163. The method according to item 162, wherein the first plurality of images, the second plurality of images, and the third plurality of images are bright-field images. 164. The method according to any one of items 162-163, wherein the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images are fluorescence images. 165. The method according to any one of items 162-163, wherein the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images are phase images. 166. The method according to any one of items 162-165, wherein comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment includes identifying signals associated with biomarkers in each image. 167. Comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment further includes: Determining a first distribution based on the biomarker signals in the first plurality of enhanced images; Determining a second distribution based on the biomarker signals in the second plurality of enhanced images; Determining a third distribution based on the biomarker signals in the third plurality of enhanced images; The method according to item 166, comprising the above. 168. Comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment further includes: The method according to item 167, comprising comparing the first distribution, the second distribution, and the third distribution and evaluating the treatment. 169. Comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images and evaluating the treatment includes, for each image, determining a score indicating the state of the disease of interest, according to the method described in any of items 162-165. 170. Comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images and evaluating the treatment further includes determining a first distribution based on the scores of the first plurality of enhanced images, determining a second distribution based on the scores of the second plurality of enhanced images, determining a third distribution based on the scores of the third plurality of enhanced images, according to the method described in item 169. 171. Comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images and evaluating the treatment further includes comparing the first distribution, the second distribution, and the third distribution and evaluating the treatment, according to the method described in item 170. 172. The treatment is a first treatment, and the method further includes receiving a fourth plurality of images depicting a set of fourth treated biological samples affected by the disease of interest and treated by a second treatment, inputting the fourth plurality of images into the trained machine learning model to obtain a fourth plurality of enhanced images, comparing the first plurality of enhanced images, the second plurality of enhanced images, the third plurality of enhanced images, and the fourth plurality of enhanced images and comparing the first treatment and the second treatment, according to the method described in any of items 162-171. 173. The method according to item 172, further comprising selecting a treatment from the first treatment and the second treatment based on the above comparison. 174. The method according to item 173, further comprising administering the selected treatment. 175. The method according to item 173, further comprising providing a medical recommendation based on the selected treatment. 176. The method according to any one of items 162 - 175, wherein the trained machine learning model is a GAN model comprising a discriminator and a generator. 177. The method according to item 176, wherein the machine learning model is a conditional GAN model. 178. The method according to any one of items 176 - 177, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups. 179. The method according to item 178, wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for an individual frequency group. 180. The method according to any one of items 176 - 179, wherein the discriminator is a PatchGAN neural network. 181. A system for evaluating a treatment for a disease of interest, comprising one or more processors, a memory, one or more programs, which are stored in the memory and configured to be executed by the one or more processors, and the one or more programs receive a first plurality of images depicting a first set of healthy biological samples not affected by the disease of interest, receive a second plurality of images depicting a second set of untreated biological samples affected by the disease of interest, Receiving a third plurality of images depicting a set of third treated biological samples that are affected by the target disease and are being treated by the treatment; Inputting the first plurality of images into a trained machine learning model to obtain a first plurality of enhanced images; Inputting the second plurality of images into the trained machine learning model to obtain a second plurality of enhanced images; Inputting the third plurality of images into the trained machine learning model to obtain a third plurality of enhanced images; Comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment; and one or more programs including instructions for performing the above; A system comprising the above. 182. A non-transitory computer-readable storage medium storing one or more programs for evaluating a treatment for a target disease, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: Receive a first plurality of images depicting a set of first healthy biological samples that are not affected by the target disease; Receive a second plurality of images depicting a set of second untreated biological samples that are affected by the target disease; Receive a third plurality of images depicting a set of third treated biological samples that are affected by the target disease and are being treated by the treatment; Input the first plurality of images into a trained machine learning model to obtain a first plurality of enhanced images; Input the second plurality of images into the trained machine learning model to obtain a second plurality of enhanced images; Input the third plurality of images into the trained machine learning model to obtain a third plurality of enhanced images; Comparing the first plurality of emphasized images, the second plurality of emphasized images, and the third plurality of emphasized images to evaluate the treatment; and A non-transitory computer-readable storage medium for causing the comparison and evaluation to be performed.

[0379] Although the present disclosure and the examples have been described with reference to the accompanying drawings in detail, it should be noted that various changes and modifications will be apparent to those skilled in the art. Such changes and modifications should be understood to be included within the scope of the present disclosure and the examples as defined by the claims.

[0380] The foregoing description has been presented for purposes of illustration and description with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to best explain the principles of the technique and its practical application. Those skilled in the art will thereby be enabled to best utilize the technique and various embodiments with various modifications as are suited to the particular use being considered.

Claims

1. A system for evaluating a treatment for a disease of interest, comprising: one or more processors; Memory, one or more programs stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising: receiving a first one or more images depicting one or more untreated biological samples afflicted by the disease of interest; receiving a second one or more images depicting one or more treated biological samples afflicted by the disease of interest and being treated with the treatment; inputting the first one or more images into a trained machine learning model to obtain a first one or more transformed images; inputting the second one or more images into the trained machine learning model to obtain a second one or more transformed images; comparing the first one or more translated images and the second one or more translated images to evaluate the treatment; one or more programs containing instructions for carrying out the steps of: A system comprising:

2. The one or more programs, receiving a third one or more images depicting one or more healthy biological samples not affected by the disease of interest; inputting the third one or more images into the trained machine learning model to obtain a third one or more transformed images; comparing the first one or more transformed images, the second one or more transformed images, and the third one or more transformed images to evaluate the treatment; The system of claim 1 , further comprising instructions for:

3. The system described in claim 2, wherein the first one or more images, the second one or more images, and the third one or more images are bright field images.

4. The system described in claim 2, wherein the first one or more converted images, the second one or more converted images, and the third one or more converted images are fluorescent images.

5. The system described in claim 2, wherein the first one or more transformed images, the second one or more transformed images, and the third one or more transformed images are phase images.

6. The system of claim 2, wherein comparing the first one or more transformed images, the second one or more transformed images, and the third one or more transformed images and evaluating the treatment includes identifying a signal in each image that is associated with a biomarker.

7. Comparing the first one or more converted images, the second one or more converted images, and the third one or more converted images to evaluate the treatment, determining a first distribution based on signals of the biomarkers in the first one or more transformed images; determining a second distribution based on signals of the biomarkers in the second one or more transformed images; determining a third distribution based on signals of the biomarkers in the third one or more transformed images; The system of claim 6 further comprising:

8. The system of claim 7, wherein comparing the first one or more transformed images, the second one or more transformed images, and the third one or more transformed images and evaluating the treatment further includes comparing the first distribution, the second distribution, and the third distribution and evaluating the treatment.

9. The system of claim 2, wherein comparing the first one or more converted images, the second one or more converted images, and the third one or more converted images and evaluating the treatment includes determining a score for each image indicative of a status of the disease of interest. Comparing the first one or more converted images, the second one or more converted images, and the third one or more converted images to evaluate the treatment includes: determining a first distribution based on the scores of the first one or more transformed images; determining a second distribution based on the scores of the second one or more transformed images; and determining a third distribution based on the scores of the third one or more transformed images; and The system of claim 9 further comprising:

11. The system of claim 10, wherein comparing the first one or more transformed images, the second one or more transformed images, and the third one or more transformed images and evaluating the treatment further includes comparing the first distribution, the second distribution, and the third distribution and evaluating the treatment.

12. The treatment is a first treatment, and the one or more programs include: receiving a fourth one or more images depicting a fourth one or more treated biological samples afflicted by the disease of interest and being treated with a second treatment; inputting the fourth one or more images into the trained machine learning model to obtain a fourth one or more transformed images; comparing the first one or more transformed images, the second one or more transformed images, the third one or more transformed images, and the fourth one or more transformed images; and comparing the first treatment and the second treatment. The system of claim 2 , further comprising instructions for:

13. The system of claim 12, wherein the one or more programs include instructions for selecting a treatment from the first treatment and the second treatment based on the comparison.

14. The system of claim 13, wherein the one or more programs include instructions for administering the selected treatment.

15. The system of claim 13, wherein the one or more programs include instructions for providing medical recommendations based on the selected treatment.

16. The system of claim 1, wherein the trained machine learning model is a GAN model comprising a discriminator and a generator.

17. The system described in claim 16, wherein the trained machine learning model is a conditional GAN ​​model.

18. The system of claim 16, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.

19. The system of claim 18, wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for a distinct set of frequencies.

20. The system of claim 16, wherein the discriminator is a PatchGAN neural network.

21. A method for evaluating a treatment for a disease of interest, the method comprising: receiving a first one or more images depicting one or more untreated biological samples afflicted by the disease of interest; receiving a second one or more images depicting one or more treated biological samples afflicted by the disease of interest and being treated with the treatment; inputting the first one or more images into a trained machine learning model to obtain a first one or more transformed images; inputting the second one or more images into the trained machine learning model to obtain a second one or more transformed images; comparing the first one or more translated images and the second one or more translated images to evaluate the treatment; A method comprising:

22. A non-transitory computer readable storage medium storing one or more programs for evaluating a treatment for a disease of interest, the one or more programs including instructions that, when executed by one or more processors of an electronic device, perform the following: receiving a first one or more images depicting one or more untreated biological samples afflicted by the disease of interest; receiving a second one or more images depicting one or more treated biological samples afflicted by the disease of interest and being treated with the treatment; inputting the first one or more images into a trained machine learning model to obtain a first one or more transformed images; inputting the second one or more images into the trained machine learning model to obtain a second one or more transformed images; comparing the first one or more translated images and the second one or more translated images to evaluate the treatment; A non-transitory computer-readable storage medium that causes the electronic device to perform the steps of:

23. The system of claim 1, wherein the first one or more images and the second one or more images are bright field images.

24. The system of claim 1, wherein the first one or more converted images and the second one or more converted images are fluorescent images.

25. The system of claim 1, wherein the first one or more transformed images and the second one or more transformed images are phase images.

26. The system of claim 1, wherein comparing the first one or more converted images and the second one or more converted images and evaluating the treatment includes identifying a signal in each image that is associated with a biomarker.

27. Comparing the first one or more converted images and the second one or more converted images to evaluate the treatment, comprising: determining a first distribution based on signals of the biomarkers in the first one or more transformed images; determining a second distribution based on signals of the biomarkers in the second one or more transformed images; 27. The system of claim 26, further comprising:

28. The system of claim 27, wherein comparing the first one or more transformed images and the second one or more transformed images and evaluating the treatment further comprises comparing the first distribution and the second distribution and evaluating the treatment.

29. The system of claim 1, wherein comparing the first one or more converted images and the second one or more converted images and evaluating the treatment includes determining a score for each image indicative of a status of the disease of interest.

30. The system of claim 1, wherein the first one or more images include a plurality of images.

Images