Biological image transformation based on machine learning model

Abstract

Systems and methods for training a machine learning model to generate an image of a biological sample, and systems and methods for generating an enhanced image of a biological sample are described. The method for training a machine learning model to generate an image of a biological sample may include: obtaining a plurality of training images including a first type of training image and a second type of training image. The method may also include: generating a plurality of wavelet coefficients using the machine learning model based on the first type of training image; generating a second type of composite image based on the plurality of wavelet coefficients; comparing the second type of composite image to the second type of training image; and updating the machine learning model based on the comparison.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 075,751, filed on September 8, 2020, and U.S. Provisional Application No. 63 / 143,707, filed on January 29, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to machine learning techniques, and more specifically to low-cost, machine learning-based generation of large-scale image data. The generated image data (e.g., image data of a biological sample) can provide sufficient richness and depth to optimize downstream processing (e.g., phenotype). Some embodiments of the system include a programmable spatial light modulator ("SLM") to generate data optimized for downstream processing (e.g., phenotype) at high speed without mechanical modification to the system. Some embodiments of the system include a machine learning model with an attention layer, the attention layer including multiple weights corresponding to multiple illumination settings (e.g., different illumination emitters of an illumination source) for identifying the optimal illumination pattern for capturing image data. Some embodiments of the system include techniques for evaluating candidate treatments for a disease of interest. Background Art

[0004] Bright field images of biological samples can be obtained on a large scale at low cost due to inexpensive equipment, easy clinical deployment, and low requirements for processing and storage resources for captured images. Obtaining bright field images is generally non-invasive and involves low phototoxicity. However, low-contrast images lack rich visual details, making them unsuitable for many downstream analyses (e.g., phenotypic exploration of microscopic samples). In contrast, other image modalities (e.g., fluorescent images) can provide rich visual information of captured samples. However, obtaining fluorescent images requires additional equipment and materials, and can be time-consuming and computationally resource-intensive. Therefore, fluorescent images may be difficult to obtain on a large scale in a low-cost manner.

[0005] Converting bright field images of biological samples into enhanced high-quality images can be difficult for a number of reasons. First, bright field images have an inherent class imbalance problem (i.e., low-frequency signals are abundant and high-frequency signals are scarce). In addition, the overall geometry of the bright field image needs to be extracted and maintained by the transformation. In addition, many factors, such as the illumination mode in which the bright field image was taken, can affect the effectiveness of the transformation. In addition, the robustness of the transformation in supporting downstream analysis also needs to be quantified and verified. Summary of the invention

[0006] Systems and methods for training a machine learning model to generate images of biological samples are described. Systems and methods for generating enhanced images of biological samples are also described. The systems and methods can be used, for example, to obtain an image of a first type, such as an image of a bright field image type. The obtained image of the first image type can then be used by the system and method to generate a second type of composite image, such as a fluorescent image.

[0007] In some embodiments, a method for training a machine learning model to generate an image of a biological sample includes obtaining a plurality of training images. The plurality of images include a first type of training image and a second type of training image. The method may also include: generating a plurality of wavelet coefficients using the machine learning model based on the first type of training image; generating a second type of composite image based on the plurality of wavelet coefficients; comparing the second type of composite image with the second type of training image; and updating the machine learning model based on the comparison.

[0008] In some embodiments, a method for generating an enhanced image of a biological sample includes: obtaining an image of the biological sample using a microscope; and generating an enhanced image of the biological sample using a machine learning model based on the image. The machine learning model can be trained in the following manner: obtaining a plurality of training images including a first type of training image and a second type of training image; based on the first type of training image, generating a plurality of wavelet coefficients using the machine learning model; generating a second type of composite image based on the plurality of wavelet coefficients; comparing the second type of composite image with the second type of training image; and updating the machine learning model based on the comparison.

[0009] In some embodiments, a system for training a machine learning model to generate an image of a biological sample includes: a computing system including one or more processors, and one or more memories storing the machine learning model, wherein the computing system is configured to receive a plurality of training images of a first type and a training image of a second type. The computing system may also be configured to: generate a plurality of wavelet coefficients using the machine learning model based on the training images of the first type; generate a composite image of the second type based on the plurality of wavelet coefficients; compare the composite image of the second type with the training image of the second type; and update the machine learning model based on the comparison.

[0010] In some embodiments, a system for generating an enhanced image of a biological sample includes: a computing system including one or more processors, and one or more memories storing a machine learning model. The computing system can be configured to receive an image of the biological sample obtained from a microscope, and generate an enhanced image of the biological sample based on the image using the machine learning model. The machine learning model may have been trained in the following manner: obtaining a plurality of training images including a first type of training image and a second type of training image; based on the first type of training image, generating a plurality of wavelet coefficients using the machine learning model; generating a second type of composite image based on the plurality of wavelet coefficients; comparing the second type of composite image with the second type of training image; and updating the machine learning model based on the comparison.

[0011] An exemplary method for training a machine learning model to generate images of biological samples includes: obtaining multiple training images including a first type of training images and a second type of training images; generating multiple wavelet coefficients using a machine learning model based on the first type of training images; generating a second type of composite image based on the multiple wavelet coefficients; comparing the second type of composite image with the second type of training images; and updating the machine learning model based on the comparison.

[0012] In some embodiments, the first type of training images are bright field images of biological samples.

[0013] In some embodiments, the second type of training images are fluorescent images of biological samples.

[0014] In some embodiments, the machine learning model includes a generator and a discriminator.

[0015] In some embodiments, the machine learning model includes a conditional GAN ​​model.

[0016] In some embodiments, the generator includes a plurality of neural networks corresponding to the plurality of frequency groups.

[0017] In some embodiments, each neural network in the plurality of neural networks is configured to generate wavelet coefficients for a corresponding frequency group.

[0018] In some embodiments, the plurality of neural networks comprises a plurality of U-Net neural networks.

[0019] In some embodiments, the discriminator is a PatchGAN neural network.

[0020] In some embodiments, the method further comprises generating images of a third type based on training images of the first type.

[0021] In some embodiments, the third type of image is a phase-shifted image.

[0022] In some embodiments, the method further comprises generating images of a fourth type based on the training images of the first type.

[0023] In some embodiments, the fourth type of image includes segmentation data.

[0024] In some embodiments, a first type of training images is captured using a microscope according to a first illumination scheme.

[0025] In some embodiments, the first lighting scheme includes one or more lighting modes.

[0026] In some embodiments, the training images of the first type are part of an array of bright field images.

[0027] In some embodiments, the plurality of training images is a first plurality of training images, and the method further includes: based on the comparison, identifying a second lighting scheme; obtaining a second plurality of training images including one or more images of the first type, wherein the one or more images of the first type are obtained based on the second lighting scheme; and training the machine learning model based on the second plurality of training images.

[0028] In some embodiments, the method further includes: obtaining a plurality of images of a first type using a microscope; and generating a plurality of composite images of a second type using a machine learning model based on the obtained plurality of images.

[0029] In some embodiments, the method further comprises: training a classifier based on a plurality of synthetic images of the second type.

[0030] In some embodiments, the microscope is a first microscope, wherein the classifier is a first classifier, and the method further includes: obtaining a plurality of images of a second type using a second microscope; training a second classifier based on the plurality of images of the second type; and comparing the performance of the first classifier and the second classifier.

[0031] In some embodiments, the second microscope is a fluorescence microscope.

[0032] An exemplary method for generating an enhanced image of a biological sample includes: obtaining an image of the biological sample using a microscope; and generating an enhanced image of the biological sample based on the image using a machine learning model, wherein the machine learning model may have been trained in the following manner: obtaining multiple training images including a first type of training images and a second type of training images; generating multiple wavelet coefficients based on the first type of training images using the machine learning model; generating a second type of composite image based on the multiple wavelet coefficients; comparing the second type of composite image with the second type of training images; and updating the machine learning model based on the comparison.

[0033] In some embodiments, the first type of training images are bright field images of biological samples.

[0034] In some embodiments, the second type of training images are fluorescent images of biological samples.

[0035] In some embodiments, the machine learning model includes a generator and a discriminator.

[0036] In some embodiments, the machine learning model includes a conditional GAN ​​model.

[0037] In some embodiments, the generator includes a plurality of neural networks corresponding to the plurality of frequency groups.

[0038] In some embodiments, each neural network in the plurality of neural networks is configured to generate wavelet coefficients for a corresponding frequency group.

[0039] In some embodiments, the plurality of neural networks comprises a plurality of U-Net neural networks.

[0040] In some embodiments, the discriminator is a PatchGAN neural network.

[0041] In some embodiments, the method further comprises generating images of a third type based on the training images of the first type.

[0042] In some embodiments, the third type of image is a phase-shifted image.

[0043] In some embodiments, the method further comprises generating images of a fourth type based on the training images of the first type.

[0044] In some embodiments, the fourth type of image includes segmentation data.

[0045] In some embodiments, a first type of training images is captured using a microscope according to a first illumination scheme.

[0046] In some embodiments, the first lighting scheme includes one or more lighting modes.

[0047] In some embodiments, the training images of the first type are part of an array of bright field images.

[0048] In some embodiments, the plurality of training images is a first plurality of training images, and the method further includes: based on the comparison, identifying a second lighting scheme; obtaining a second plurality of training images including one or more images of the first type, wherein the one or more images of the first type are obtained based on the second lighting scheme; and training the machine learning model based on the second plurality of training images.

[0049] In some embodiments, the method further includes: obtaining a plurality of images of a first type using a microscope; and generating a plurality of composite images of a second type using a machine learning model based on the obtained plurality of images.

[0050] In some embodiments, the method further comprises: training a classifier based on a plurality of synthetic images of the second type.

[0051] In some embodiments, the microscope is a first microscope, wherein the classifier is a first classifier, and the method further includes: obtaining a plurality of images of a second type using a second microscope; training a second classifier based on the plurality of images of the second type; and comparing the performance of the first classifier and the second classifier.

[0052] In some embodiments, the second microscope is a fluorescence microscope.

[0053] An exemplary system for training a machine learning model to generate an image of a biological sample includes: a computing system comprising one or more processors, and one or more memories storing the machine learning model, wherein the computing system is configured to receive a plurality of training images of a first type and a training image of a second type, and wherein the computing system can be configured to: generate a plurality of wavelet coefficients using the machine learning model based on the training images of the first type; generate a composite image of the second type based on the plurality of wavelet coefficients; compare the composite image of the second type to the training images of the second type; and update the machine learning model based on the comparison.

[0054] In some embodiments, the first type of training images are bright field images of biological samples.

[0055] In some embodiments, the second type of training images are fluorescent images of biological samples.

[0056] In some embodiments, the machine learning model includes a generator and a discriminator.

[0057] In some embodiments, the machine learning model includes a conditional GAN ​​model.

[0058] In some embodiments, the generator includes a plurality of neural networks corresponding to the plurality of frequency groups.

[0059] In some embodiments, each neural network in the plurality of neural networks is configured to generate wavelet coefficients for a corresponding frequency group.

[0060] In some embodiments, the plurality of neural networks comprises a plurality of U-Net neural networks.

[0061] In some embodiments, the discriminator is a PatchGAN neural network.

[0062] In some embodiments, the computing system is further configured to generate images of a third type based on the training images of the first type.

[0063] In some embodiments, the third type of image is a phase-shifted image.

[0064] In some embodiments, the computing system is further configured to generate images of a fourth type based on the training images of the first type.

[0065] In some embodiments, the fourth type of image includes segmentation data.

[0066] In some embodiments, a first type of training images is captured using a microscope according to a first illumination scheme.

[0067] In some embodiments, the first lighting scheme includes one or more lighting modes.

[0068] In some embodiments, the training images of the first type are part of an array of bright field images.

[0069] In some embodiments, the plurality of training images is a first plurality of training images, and wherein the computing system is further configured to: based on the comparison, identify a second lighting scheme; obtain a second plurality of training images comprising one or more images of the first type, wherein the one or more images of the first type are obtained based on the second lighting scheme; and train the machine learning model based on the second plurality of training images.

[0070] In some embodiments, the computing system is further configured to: obtain a plurality of images of a first type using a microscope; and generate a plurality of composite images of a second type using a machine learning model based on the obtained plurality of images.

[0071] In some embodiments, the computing system is further configured to: train the classifier based on the plurality of synthetic images of the second type.

[0072] In some embodiments, the microscope is a first microscope, wherein the classifier is a first classifier, wherein the computing system is further configured to: obtain multiple images of a second type using a second microscope; train a second classifier based on the multiple images of the second type; and compare the performance of the first classifier and the second classifier.

[0073] In some embodiments, the second microscope is a fluorescence microscope.

[0074] An exemplary system for generating an enhanced image of a biological sample includes: a computing system comprising one or more processors, and one or more memories storing a machine learning model, wherein the computing system is configured to receive an image of the biological sample obtained from a microscope, and generate an enhanced image of the biological sample based on the image using the machine learning model, wherein the machine learning model may have been trained in the following manner: obtaining a plurality of training images including a first type of training images and a second type of training images; based on the first type of training images, generating a plurality of wavelet coefficients using the machine learning model; generating a second type of composite image based on the plurality of wavelet coefficients; comparing the second type of composite image to the second type of training images; and updating the machine learning model based on the comparison.

[0075] In some embodiments, the first type of training images are bright field images of biological samples.

[0076] In some embodiments, the second type of training images are fluorescent images of biological samples.

[0077] In some embodiments, the machine learning model includes a generator and a discriminator.

[0078] In some embodiments, the machine learning model includes a conditional GAN ​​model.

[0079] In some embodiments, the generator includes a plurality of neural networks corresponding to the plurality of frequency groups.

[0080] In some embodiments, each neural network in the plurality of neural networks is configured to generate wavelet coefficients for a corresponding frequency group.

[0081] In some embodiments, the plurality of neural networks comprises a plurality of U-Net neural networks.

[0082] In some embodiments, the discriminator is a PatchGAN neural network.

[0083] In some embodiments, the machine learning model is further trained by generating a third type of image based on the first type of training images.

[0084] In some embodiments, the third type of image is a phase-shifted image.

[0085] In some embodiments, the machine learning model is further trained by generating images of a fourth type based on training images of the first type.

[0086] In some embodiments, the fourth type of image includes segmentation data.

[0087] In some embodiments, a first type of training images is captured using a microscope according to a first illumination scheme.

[0088] In some embodiments, the first lighting scheme includes one or more lighting modes.

[0089] In some embodiments, the training images of the first type are part of an array of bright field images.

[0090] In some embodiments, the plurality of training images is a first plurality of training images, wherein the machine learning model has been trained by: identifying a second lighting scheme based on comparison; obtaining a second plurality of training images comprising one or more images of the first type, wherein the one or more images of the first type are obtained based on the second lighting scheme; and training the machine learning model based on the second plurality of training images.

[0091] In some embodiments, the machine learning model has been trained by: obtaining a plurality of images of a first type using a microscope; and generating a plurality of composite images of a second type using the machine learning model based on the obtained plurality of images.

[0092] In some embodiments, the machine learning model has been trained by training a classifier based on a plurality of synthetic images of the second type.

[0093] In some embodiments, the microscope is a first microscope, wherein the classifier is a first classifier, and wherein the machine learning model has been trained by: obtaining a plurality of images of a second type using a second microscope; training a second classifier based on the plurality of images of the second type; and comparing the performance of the first classifier and the second classifier.

[0094] In some embodiments, the second microscope is a fluorescence microscope.

[0095] An exemplary method for processing an image of a biological sample to obtain one or more output images includes: obtaining multiple images of the biological sample using multiple configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and inputting the multiple images of the biological sample into a trained machine learning model to obtain one or more output images.

[0096] In some embodiments, at least one of the multiple configurations of the SLM is used to generate one or more optical aberrations.

[0097] In some embodiments, generating one or more optical aberrations includes spherical aberration, astigmatism, defocus, distortion, tilt, or any combination thereof.

[0098] In some embodiments, at least one of the multiple configurations of the SLM is used to enhance one or more features.

[0099] In some embodiments, the one or more features include cell borders, actin filaments, nuclear shape, cytoplasmic segmentation, or any combination thereof.

[0100] In some embodiments, at least one of the multiple configurations of the SLM is configured to reduce optical aberrations.

[0101] In some embodiments, multiple SLMs are configured to obtain images of a biological sample at different depths.

[0102] In some embodiments, the machine learning model is configured to generate an image of the second type based on an image of the first type.

[0103] In some embodiments, the first type of image is a bright field image.

[0104] In some embodiments, the second type of image is a fluorescence image.

[0105] In some embodiments, the second type of image is an enhanced version of the first type of image.

[0106] In some embodiments, the machine learning model is a GAN model or a self-supervised model.

[0107] In some embodiments, multiple images are obtained using multiple configurations of the light source of the optical system.

[0108] In some embodiments, the light source is an LED array of the optical system.

[0109] In some embodiments, at least one of the plurality of SLM configurations is obtained by: training a machine learning model; evaluating the trained machine learning model; and identifying at least one configuration based on the evaluation.

[0110] In some embodiments, the trained machine learning model is configured to receive an input image and output an enhanced version of the input image.

[0111] In some embodiments the enhanced version of the input image includes one or more enhanced cellular phenotypes.

[0112] An exemplary electronic device for processing an image of a biological sample to obtain one or more output images comprises: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for: obtaining multiple images of the biological sample using multiple configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and inputting the multiple images of the biological sample into a trained machine learning model to obtain one or more output images.

[0113] An exemplary non-transitory computer-readable storage medium stores one or more programs for processing an image of a biological sample to obtain one or more output images, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: obtain multiple images of the biological sample using multiple configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and input the multiple images of the biological sample into a trained machine learning model to obtain one or more output images.

[0114] An exemplary method for classifying an image of a biological sample includes: obtaining multiple images of the biological sample using multiple configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and inputting the multiple images of the biological sample into a trained machine learning model to obtain one or more classification outputs.

[0115] In some embodiments, at least one of the multiple configurations of the SLM is used to generate one or more optical aberrations.

[0116] In some embodiments, generating one or more optical aberrations includes spherical aberration, astigmatism, defocus, distortion, tilt, or any combination thereof.

[0117] In some embodiments, at least one of the multiple configurations of the SLM is used to enhance one or more features.

[0118] In some embodiments, the one or more features include cell borders, actin filaments, nuclear shape, cytoplasmic segmentation, or any combination thereof.

[0119] In some embodiments, at least one of the multiple configurations of the SLM is configured to reduce optical aberrations.

[0120] In some embodiments, multiple SLMs are configured to obtain images of a biological sample at different depths.

[0121] In some embodiments, multiple images are obtained using multiple configurations of the light source of the optical system.

[0122] In some embodiments, the light source is an LED array of the optical system.

[0123] In some embodiments, at least one of the plurality of SLM configurations is obtained by: training a machine learning model; evaluating the trained machine learning model; and identifying at least one configuration based on the evaluation.

[0124] In some embodiments, the trained machine learning model is configured to receive an input image and detect one or more predefined objects in the input image.

[0125] In some embodiments, the predefined object comprises diseased tissue.

[0126] An exemplary electronic device for classifying images of biological samples comprises: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for: obtaining multiple images of the biological sample using multiple configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and inputting the multiple images of the biological sample into a trained machine learning model to obtain one or more output images.

[0127] An exemplary non-transitory computer-readable storage medium stores one or more programs for classifying images of biological samples, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: obtain multiple images of the biological sample using multiple configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and input the multiple images of the biological sample into a trained machine learning model to obtain one or more output images.

[0128] An exemplary method for training a machine learning model includes: obtaining multiple images of a biological sample using multiple configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and training the machine learning model using the multiple images.

[0129] In some embodiments, at least one of the multiple configurations of the SLM is used to generate one or more optical aberrations.

[0130] In some embodiments, generating one or more optical aberrations includes spherical aberration, astigmatism, defocus, distortion, tilt, or any combination thereof.

[0131] In some embodiments, at least one of the multiple configurations of the SLM is used to enhance one or more features.

[0132] In some embodiments, the one or more features include cell borders, actin filaments, nuclear shape, cytoplasmic segmentation, or any combination thereof.

[0133] In some embodiments, at least one of the multiple configurations of the SLM is configured to reduce optical aberrations.

[0134] In some embodiments, at least one of the multiple configurations of the SLM is configured to obtain images of the biological sample at different depths.

[0135] In some embodiments, the machine learning model is configured to generate an image of the second type based on an image of the first type.

[0136] In some embodiments, the first type of image is a bright field image.

[0137] In some embodiments, the second type of image is a fluorescence image.

[0138] In some embodiments, the machine learning model is a GAN model or a self-supervised model.

[0139] In some embodiments, the machine learning model is a classification model.

[0140] In some embodiments, multiple images are obtained using multiple configurations of the light source of the optical system.

[0141] In some embodiments, the light source is an LED array of the optical system.

[0142] In some embodiments, training the machine learning model includes: (a) training the machine learning model using a first image, wherein the first image is obtained using a first configuration of the SLM of the optical system; (b) evaluating the trained machine learning model; (c) based on the evaluation, identifying a second configuration of the SLM; and (d) training the machine learning model using a second image, wherein the second image is obtained using a second configuration of the SLM of the optical system.

[0143] In some embodiments, a loss function based on a machine learning model is evaluated.

[0144] In some embodiments, the method further comprises: repeating steps (a) to (d) until a threshold value is reached.

[0145] In some embodiments, the threshold indicates convergence of training.

[0146] In some embodiments, the trained machine learning model is configured to receive an input image and output an enhanced version of the input image.

[0147] In some embodiments the enhanced version of the input image includes one or more enhanced cellular phenotypes.

[0148] In some embodiments, the trained machine learning model is configured to receive an input image and detect one or more predefined objects in the input image.

[0149] In some embodiments, the predefined object comprises diseased tissue.

[0150] An exemplary electronic device for training a machine learning model comprises: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for: obtaining multiple images of a biological sample using multiple configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and training the machine learning model using the multiple images.

[0151] An exemplary non-transitory computer-readable storage medium stores one or more programs for training a machine learning model, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: obtain multiple images of a biological sample using multiple configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and use the multiple images to train the machine learning model.

[0152] An exemplary method for generating an enhanced image of a biological sample includes: obtaining, using a microscope, an image of the biological sample illuminated using an illumination pattern of an illumination source, wherein the illumination pattern is determined by: training a classification model configured to receive an input image and output a classification result, using the trained classification model to train a machine learning model having multiple weights corresponding to multiple illumination settings, and identifying the illumination pattern based on the multiple weights of the trained machine learning model; and generating an enhanced image of the biological sample by inputting the obtained image of the biological sample into the trained machine learning model.

[0153] In some embodiments, the image obtained is a bright field image.

[0154] In some embodiments, the enhanced image is a fluorescence image, a phase image, or a combination thereof.

[0155] In some embodiments, the illumination source comprises an array of illumination emitters.

[0156] In some embodiments, the illumination source is an LED array.

[0157] In some embodiments, the lighting mode indicates whether each lighting emitter is on or off and the intensity of each lighting emitter.

[0158] In some embodiments, each lighting setting of the plurality of lighting settings corresponds to a respective lighting emitter of the lighting source; and wherein each weight corresponds to an intensity of the respective lighting emitter.

[0159] In some embodiments, the classification model is configured to receive an input phase image or an input fluorescence image and output a classification result indicative of one of a plurality of predefined classes.

[0160] In some embodiments, the plurality of predefined categories includes a healthy category and a diseased category.

[0161] In some embodiments, the machine learning model is a GAN model, which includes an attention layer, a discriminator, and a generator, and the attention layer includes multiple weights.

[0162] In some embodiments, the machine learning model is a conditional GAN ​​model.

[0163] In some embodiments, the generator includes a plurality of neural networks corresponding to the plurality of frequency groups.

[0164] In some embodiments, each neural network in the plurality of neural networks is configured to generate wavelet coefficients for a corresponding frequency group.

[0165] In some embodiments, the plurality of neural networks comprises a plurality of U-Net neural networks.

[0166] In some embodiments, the discriminator is a PatchGAN neural network.

[0167] In some embodiments, training a machine learning model using a trained classification model includes: applying multiple weights to multiple bright field training images; aggregating multiple weighted bright field training images into a clustered bright field image; inputting the clustered bright field training images into the machine learning model to obtain enhanced training images and generator loss; inputting the enhanced training images into the trained classifier to obtain the classifier loss; increasing the generator loss based on the classifier loss; and updating the multiple weights based on the increased generator loss.

[0168] In some embodiments, the method further comprises: classifying the enhanced image using the trained classifier.

[0169] In some embodiments, the method further comprises: displaying the enhanced image.

[0170] An exemplary system for generating an enhanced image of a biological sample includes: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the following operations: obtaining, using a microscope, an image of the biological sample illuminated using an illumination pattern of an illumination source, wherein the illumination pattern is determined by the following steps: training a classification model configured to receive an input image and output a classification result, using the trained classification model to train a machine learning model having multiple weights corresponding to multiple illumination settings, and identifying the illumination pattern based on the multiple weights of the trained machine learning model; and generating an enhanced image of the biological sample by inputting the obtained image of the biological sample into the trained machine learning model.

[0171] An exemplary non-transitory computer-readable storage medium stores one or more programs for generating an enhanced image of a biological sample, the one or more programs including instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: obtain, using a microscope, an image of the biological sample illuminated using an illumination pattern of an illumination source, wherein the illumination pattern is determined by the following steps: training a classification model configured to receive an input image and output a classification result, using the trained classification model to train a machine learning model having multiple weights corresponding to multiple illumination settings, and identifying the illumination pattern based on the multiple weights of the trained machine learning model; and generating an enhanced image of the biological sample by inputting the obtained image of the biological sample into the trained machine learning model.

[0172] An exemplary method for evaluating a treatment for a disease of interest includes: receiving a first plurality of images depicting a first group of healthy biological samples not affected by the disease of interest; receiving a second plurality of images depicting a second group of untreated biological samples affected by the disease of interest; receiving a third plurality of images depicting a third group of treated biological samples affected by the disease of interest and treated with treatment; inputting the first plurality of images into a trained machine learning model to obtain a first plurality of enhanced images; inputting the second plurality of images into the trained machine learning model to obtain a second plurality of enhanced images; inputting the third plurality of images into the trained machine learning model to obtain a third plurality of enhanced images; and comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment.

[0173] In some embodiments, the first plurality of images, the second plurality of images, and the third plurality of images are bright field images.

[0174] In some embodiments, the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images are fluorescence images.

[0175] In some embodiments, the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images are phase images.

[0176] In some embodiments, comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate treatment includes identifying a signal associated with a biomarker in each image.

[0177] In some embodiments, comparing a first plurality of enhanced images, a second plurality of enhanced images, and a third plurality of enhanced images to evaluate treatment further includes: determining a first distribution based on signals of biomarkers in the first plurality of enhanced images; determining a second distribution based on signals of biomarkers in the second plurality of enhanced images; and determining a third distribution based on signals of biomarkers in the third plurality of enhanced images.

[0178] In some embodiments, comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment further comprises comparing the first distribution, the second distribution, and the third distribution to evaluate the treatment.

[0179] In some embodiments, comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate treatment includes determining a score for each image indicating an evaluation statement of the disease of interest.

[0180] In some embodiments, comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate treatment further includes: determining a first distribution based on scores in the first plurality of enhanced images; determining a second distribution based on scores in the second plurality of enhanced images; and determining a third distribution based on scores in the third plurality of enhanced images.

[0181] In some embodiments, comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment further comprises comparing the first distribution, the second distribution, and the third distribution to evaluate the treatment.

[0182] In some embodiments, the treatment is a first treatment, and the method further includes: receiving a fourth plurality of images depicting a fourth group of treated biological samples affected by the disease of interest and treated with the second treatment; inputting the fourth plurality of images into the trained machine learning model to obtain a fourth plurality of enhanced images; and comparing the first plurality of enhanced images, the second plurality of enhanced images, the third plurality of enhanced images, and the fourth plurality of enhanced images to compare the first treatment and the second treatment.

[0183] In some embodiments, the method further comprises: selecting a treatment from the first treatment and the second treatment based on the comparison.

[0184] In some embodiments, the method further comprises: administering the selected treatment.

[0185] In some embodiments, the method further comprises providing medical advice based on the selected treatment.

[0186] In some embodiments, the trained machine learning model is a GAN model that includes a discriminator and a generator.

[0187] In some embodiments, the machine learning model is a conditional GAN ​​model.

[0188] In some embodiments, the generator includes a plurality of neural networks corresponding to the plurality of frequency groups.

[0189] In some embodiments, each neural network in the plurality of neural networks is configured to generate wavelet coefficients for a corresponding frequency group.

[0190] In some embodiments, the discriminator is a PatchGAN neural network.

[0191] An exemplary system for evaluating a treatment for a disease of interest comprises: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for performing the following operations: receiving a first plurality of images depicting a first group of healthy biological samples not affected by the disease of interest; receiving a second plurality of images depicting a second group of untreated biological samples affected by the disease of interest; receiving a third plurality of images depicting a third group of treated biological samples affected by the disease of interest and treated with the treatment; inputting the first plurality of images into a trained machine learning model to obtain a first plurality of enhanced images; inputting the second plurality of images into the trained machine learning model to obtain a second plurality of enhanced images; inputting the third plurality of images into the trained machine learning model to obtain a third plurality of enhanced images; and comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment.

[0192] An exemplary non-transitory computer-readable storage medium stores one or more programs for evaluating a treatment for a disease of interest, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: receive a first plurality of images depicting a first group of healthy biological samples not affected by the disease of interest; receive a second plurality of images depicting a second group of untreated biological samples affected by the disease of interest; receive a third plurality of images depicting a third group of treated biological samples affected by the disease of interest and treated with treatment; input the first plurality of images into a trained machine learning model to obtain a first plurality of enhanced images; input the second plurality of images into the trained machine learning model to obtain a second plurality of enhanced images; input the third plurality of images into the trained machine learning model to obtain a third plurality of enhanced images; and compare the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment. BRIEF DESCRIPTION OF THE DRAWINGS

[0193] Figure 1 An exemplary process for training a machine learning model configured to generate synthetic data (e.g., image data) at scale and in a cost-effective manner is shown in accordance with some embodiments.

[0194] FIG2A illustrates an exemplary process for training and applying a machine learning model according to some embodiments.

[0195] FIG2B illustrates an exemplary process for training and applying a machine learning model according to some embodiments.

[0196] Figure 3A The training process of a machine learning model according to some embodiments is shown.

[0197] Figure 3B Operation of an exemplary generator according to some embodiments is shown.

[0198] Figure 3C The training process of a machine learning model according to some embodiments is shown.

[0199] Figure 3D The training process of a machine learning model according to some embodiments is shown.

[0200] Figure 4 An exemplary image formation model according to some embodiments is shown.

[0201] Figure 5 Shown is the output of an exemplary trained generator using a static microscopy setup in accordance with some embodiments.

[0202] Fig. 6AAn exemplary process for determining the robustness of a generated image is shown in accordance with some embodiments.

[0203] Figure 6B An exemplary comparison between two classifiers according to some embodiments is shown.

[0204] Fig. 7A An exemplary method for training a machine learning model to generate images of biological samples is shown.

[0205] Figure 7B An exemplary method for generating an enhanced image of a biological sample is shown.

[0206] Figure 8 An exemplary electronic device is shown in accordance with some embodiments.

[0207] Fig. 9 An exemplary optical system is shown in accordance with some embodiments.

[0208] Fig.10 An exemplary method for analyzing an image of a biological sample using a programmable SLM of an optical system is shown in accordance with some embodiments.

[0209] Fig.11 An exemplary method for training a machine learning model according to some embodiments is shown.

[0210] Fig. 12A and Fig. 12B Shown is a side-by-side comparison of classification results of two classification models based on the same input image according to some embodiments.

[0211] Fig. 12C and Fig.12D The following are respectively shown according to some embodiments Fig. 9 A and Fig. 9 Side-by-side comparison of the same classification results in B, using different color schemes to illustrate the two models’ resistance to batch effects.

[0212] Fig.12E The Euclidean distance metrics corresponding to a set of real images, a set of generated images, and a set of true batch-invariant images are shown in accordance with some embodiments.

[0213] Fig.13A An exemplary generated image according to some embodiments is shown.

[0214] Fig. 13B An exemplary generated image according to some embodiments is shown.

[0215] Fig. 13C An exemplary generated image according to some embodiments is shown.

[0216] Fig.14A and Fig. 14B An exemplary process for training a machine learning model (e.g., a GAN model) configured to generate synthetic data (e.g., image data) and identifying an optimal lighting pattern for capturing input data for the machine learning model is shown in accordance with some embodiments.

[0217] Fig.15 An exemplary method for training a machine learning model according to some embodiments is shown.

[0218] Fig.16A Shown are synthetic images generated by an exemplary GAN model for studying NASH disease, according to some embodiments.

[0219] Fig. 16B Downstream analysis of generated images according to some embodiments is shown.

[0220] Fig. 16C An exemplary process for evaluating candidate treatments according to some embodiments is shown.

[0221] Fig.16D An exemplary process for evaluating candidate treatments according to some embodiments is shown.

[0222] Fig.17A Shown are synthetic images generated by an exemplary GAN model for studying tuberous sclerosis complex (“TSC”) in accordance with some embodiments.

[0223] Fig. 17B An exemplary process for evaluating candidate treatments according to some embodiments is shown. DETAILED DESCRIPTION

[0224] The present disclosure includes methods, systems, electronic devices, non-transitory storage media, and apparatus for performing ML-based generation of large-scale image data. The generated image data (e.g., image data of biological samples) can provide sufficient richness and depth for downstream processing (e.g., phenotyping). In addition, embodiments of the present disclosure include a set of computational and hardware optimization methods that expand the current dimensions of classical microscopy techniques.

[0225] Embodiments of the present disclosure can process multiple bright field images of biological samples and generate enhanced images of the biological samples. Enhanced images include, but are not limited to, fluorescence images, phase shift images, semantic maps, polarization maps, refraction maps (two-dimensional and three-dimensional), absorbance maps, and other image modalities. Due to the inexpensive equipment (e.g., relative to fluorescence microscopy), easy clinical deployment, and low processing and storage resource requirements for captured images, bright field images of biological samples can be obtained on a large scale at low cost. Obtaining bright field images is generally non-invasive and involves low phototoxicity. Therefore, bright field images can be effectively obtained on a large scale. Enhanced images provide sufficient richness and depth for downstream processing (e.g., phenotypic exploration).

[0226] Embodiments of the present disclosure include a machine learning model that is trained to receive an image of a first type and convert the input image into other imaging modalities. An exemplary machine learning model can receive an image of a first type and convert the input image into an image of a second type (e.g., an enhanced image). In some embodiments, different image types refer to different imaging modalities. In some embodiments, the first type of image is a bright field image. For example, a bright field image can be captured from illuminating an in vitro (or biopsy) cell sample using an inexpensive LED array. The second type of image includes a fluorescent image. The generated fluorescent image exhibits high contrast features that are not directly visible in the bright field image and can be used for downstream processing (e.g., phenotypic analysis).

[0227] Embodiments of the present disclosure reduce or eliminate the need to capture true fluorescent images (or other special image modalities) of biological samples for downstream analysis, and allow bright field images to be widely used for various purposes. This is particularly beneficial for live cell imaging. For example, the disclosed method can be used to monitor and optimize cell differentiation protocols. In the case of chemical or genetic perturbations, time-consuming activities associated with cell staining and fixation can be avoided. In some embodiments, the dosing time, that is, the incubation time of the drug with the observed cells, can also be optimized by software. Researchers will no longer need to arbitrarily determine the optimal incubation time because the software will be able to inform researchers of the optimized incubation time. In addition, machine learning techniques for converting bright field images into other modalities require lower processing and storage resource utilization. Therefore, embodiments of the present disclosure present technical improvements in the field of medical imaging while enhancing the operability and functionality of computing systems.

[0228] Embodiments of the present disclosure also include a machine learning model that is trained to receive an image of the first type and convert the input image into an image of a third type. In some embodiments, the third type of image includes image data indicating various optical properties (e.g., phase shift) of the captured biological sample.

[0229] Embodiments of the present disclosure also include a machine learning model that is trained to receive an image of the first type and convert the input image into an image of a fourth type. In some embodiments, the image of the fourth type includes image data indicating segmentation data (e.g., cell boundaries).

[0230] A person skilled in the art should understand that embodiments of the present disclosure may further convert the input image into many other types of images that capture various imaging features, such as semantic maps, polarization maps, refraction maps (two-dimensional or three-dimensional), absorbance maps, etc.

[0231] In some embodiments, a single machine learning model is trained to perform multiple transformation tasks simultaneously. For example, the same machine learning model can receive a first type of image and generate multiple types of images (e.g., a second type of image, a third type of image, a fourth type of image). The machine learning model can be a generative adversarial network ("GAN") model. For example, the GAN network can be a conditional GAN ​​("cGAN") model.

[0232] In some embodiments, the machine learning model converts an input image into its corresponding wavelet coefficients and generates one or more output images with the wavelet coefficients. The compact and multi-scale representation of the image is combined with the inherent nonlinear options of the neural network to achieve multiple goals at the same time. First, the wavelet-based representation solves the inherent class imbalance problem faced by most generative models. Specifically, most input image data include rich low-frequency signals but fewer high-frequency signals. Second, the wavelet-based representation extracts and maintains the overall geometry of the input image data. The discriminator of the model ensures that the real image and the generated image (for example, the real fluorescence image and the generated fluorescence image) are indistinguishable. The generated fluorescence image or other enhanced image modality corresponds to the virtual staining of the sample. Due to the low phototoxicity of the bright field imaging mode and its availability in clinical settings, the virtual staining of the embodiments of the present disclosure can be performed on living cells as well as biopsy samples.

[0233] Embodiments of the present disclosure further include hardware optimization methods. For example, embodiments of the present disclosure may further dynamically optimize the illumination mode of a microscope (e.g., a microscope that obtains a first type of image). In some embodiments, a microscope used to capture a first type of image may be tuned or programmed to provide different illumination schemes during training. During training of a machine learning model, an optimal illumination scheme for capturing a first type of image may be identified. The optimal illumination scheme may be used to capture a first type of image (e.g., a bright field image) in order to extract an optimal representation of a biological sample for a wavelet-based image transform (e.g., for downstream phenotypic exploration).

[0234] Embodiments of the present disclosure also include evaluating the robustness of the generated images of the machine learning model. In some embodiments, a first downstream classifier is trained using real images (e.g., real fluorescence images), and a second downstream classifier is trained using generated images (e.g., generated fluorescence images). The performance of the two classifiers can be compared to evaluate the robustness of the generated images as training data in downstream tasks.

[0235] Thus, embodiments of the present disclosure include an integrated platform that simultaneously solves many problems, including: image enhancement, phase retrieval, low phototoxicity, realistic virtual rendering of bright field images, and robustness in downstream tasks. Embodiments of the present disclosure can evaluate the robustness of the generated images through downstream classification tasks. These tasks are integrated into the platform and complete the data generation cycle from non-invasive bright field images to fluorescent images. For example, the system can optimize the parameters of the bright field microscope acquisition system during use. The illumination pattern of the LED array and other parameters of the bright field microscope acquisition system, including, for example, the focus position of the microscope objective and the activation timing of the spatial light modulator (SLM), can be optimized by back propagation during downstream classification tasks.

[0236] Additionally, in some embodiments, the platform performs a series of perturbations on the cells and learns to optimize the illumination scheme to extract the best representation of the cells for phenotypic exploration.

[0237] Some embodiments of the present disclosure may identify one or more optimal lighting modes for capturing image data. In some embodiments, the lighting mode may indicate whether each lighting emitter of the lighting source (e.g., each LED on an LED array) is to be turned on or off and the intensity of each lighting emitter. The system may determine the optimal lighting mode by training a machine learning model with an attention layer, the attention layer including multiple weights corresponding to the intensities of multiple lighting emitters (e.g., multiple weights corresponding to the intensities of multiple LEDs on an LED array). During the training of the machine learning model, multiple weights are applied to multiple training images (e.g., bright field images) illuminated by different lighting emitters. The gathered images may be input into the machine learning model to determine the loss, and the model including the weights in the attention layer may be updated accordingly based on the loss. After training, the lighting mode may be determined based on the weights in the attention layer of the trained machine learning model, because each weight may correspond to the desired intensity level of the corresponding lighting emitter. Therefore, the process only involves capturing images using a limited number of lighting settings (e.g., turning on a single lighting emitter at a time to capture an image), and there is no need to physically adjust the intensity of the lighting emitter in order to identify the optimal lighting mode.

[0238] Some embodiments of the present disclosure can train a machine learning model so that the synthetic image data generated by the machine learning model can provide the same performance as real images in downstream analysis. In some embodiments, training the machine learning model includes first training a classifier corresponding to a downstream task (e.g., classifying healthy tissue from diseased tissue based on an image), and then using the output of the classifier to guide the training of the machine learning model.

[0239] Some embodiments of the present disclosure can evaluate candidate treatments for a disease of interest. In some embodiments, a system receives a first plurality of images depicting a first set of healthy biological samples not affected by the disease of interest; receives a second plurality of images depicting a second set of untreated biological samples affected by the disease of interest; and receives a third plurality of images depicting a third set of treated biological samples affected by the disease of interest and treated by the candidate treatment. The images are input into a machine learning model to obtain enhanced images, and the enhanced images are compared to evaluate the treatment (e.g., by analyzing the distribution of the images).

[0240] The following description sets forth exemplary methods, parameters, etc. However, it should be appreciated that such description is not intended as a limitation on the scope of the present disclosure, but is provided as a description of exemplary embodiments.

[0241] In some embodiments, the exemplary optical system includes a programmable spatial light modulator ("SLM"). The SLM of the optical system can improve the performance of the machine learning model via a training phase (e.g., by providing a rich training data set) and / or via an inference phase (e.g., by providing input data under various optical settings or optimal settings). The programming of the SLM does not require any mechanical movement or mechanical modification of the optical system.

[0242] SLM provides additional degrees of freedom and contrast sources to control the microscope in a programmable manner. For example, the SLM can be programmed to generate optical aberrations that enhance key phenotypes. As another example, the SLM can be programmed to provide different modulations, resulting in a variety of images that allow high-speed probing of deep samples. SLM also allows identification of optimal imaging settings to infer cell phenotypes and reconstruct alternative image modalities in a supervised manner. Multifocal acquisition is possible without any mechanical movement, thereby speeding up and improving downstream tasks. Thus, three-dimensional phase tomography and reconstruction are accelerated and improved.

[0243] Although the following description uses the terms "first," "second," etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various described embodiments, a first graphical representation may be referred to as a second graphical representation, and similarly, a second graphical representation may be referred to as a first graphical representation. Both the first graphical representation and the second graphical representation are graphical representations, but they are not the same graphical representations.

[0244] The terms used in the description of the various embodiments described herein are only used for the purpose of describing specific embodiments, and are not intended to be limited. As used in the description of the various described embodiments and the appended claims, the singular forms "one", "one" and "the / said" are also intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any and all possible combinations of one or more of the items listed in association. It should be further understood that when used in this specification, the terms "includes", "including", "comprises" and / or "comprising" specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups.

[0245] The term "if" is optionally interpreted to mean "when" or "in" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [the condition or event] is detected" are optionally interpreted to mean "upon determination" or "in response to determining" or "upon detection of [the condition or event]" or "in response to detecting [the condition or event]," depending on the context.

[0246] Figure 1 An exemplary process for training a machine learning model configured to generate enhanced images of biological samples is shown in accordance with some embodiments.

[0247] refer to Figure 1 , the training data 120 includes a first type of image 122 and a second type of image 124. In some embodiments, the images are images of biological samples, and the biological samples may include one or more sets of stained, unstained, perturbed and / or unperturbed biological samples.

[0248] In some embodiments, the first type of image data 122 includes a set of bright field images, and the second type of image data 124 includes images of a different modality (e.g., fluorescence images). The first type of images 122 can be obtained using a bright field microscope, and the second type of images 124 can be obtained using a fluorescence microscope.

[0249] like Figure 1 As shown, the first type of image 122 is obtained based on one or more illumination modes 110. The illumination mode indicates the settings used to illuminate the object. In some embodiments, the illumination mode can be defined by one or more parameters indicating the spatial relationship (e.g., distance, angle) between the object and the illumination source, one or more parameters indicating the settings of the illumination source, or a combination thereof. For example, the illumination mode can indicate a set of activated LED light sources, a specific defocus / polarization setting, etc.

[0250] In some embodiments, the microscope that captures the first type of image can be a microscope that supports multiple illumination modes. For example, the microscope can provide a programmable illumination source (e.g., LED array, laser), an adaptive optical system (SLM, micromirror), or a combination thereof. By updating the illumination mode (e.g., controlling the illumination source and / or the pupil function of the optical system), many representations of the biological sample corresponding to multiple illumination modes can be obtained.

[0251] In some embodiments, training data 120 may be organized as three-dimensional image data (e.g., an array of images). For example, training data 120 may have dimensions (B, C, H, W), where B indicates the batch size, C indicates the number of channels (i.e., illumination mode), H indicates the height, and W indicates the width. If there is only a single bright field image, then C is equal to 1, and if there is a stack of bright field images, then C is greater than 1.

[0252] In some embodiments, one or more images in training data 120 may be normalized before they are used to train machine learning model 100. For example, fluorescence images may be normalized based on an illumination or intensity parameter.

[0253] refer to Figure 1, the training data 120 is used to train the machine learning model 100. In some embodiments, the model 100 is a generative adversarial network ("GAN") model. For example, the GAN network can be a conditional GAN ​​("cGAN") model. As described in detail below, the GAN network includes a generator and a discriminator. The generator is trained to receive an image of a first type (e.g., a bright field image) and convert the input image into an image of a second type (e.g., a fluorescent image). The training step includes comparing a real image of the first type 122 with a real image of the second type 124 to determine a real-world baseline that the wavelet-based transform of the first type image 122 should seek to achieve. During training, the generator output (i.e., the generated fluorescent image) can be directly connected to the discriminator input. The discriminator is trained to distinguish the generated image of the second type (e.g., the generated fluorescent image) from the real image of the second type (e.g., the real fluorescent image). Through back propagation, the generator can use the output of the discriminator to update the weights of the generator so that the generator learns to generate images that the discriminator will classify as real images.

[0254] In some embodiments, lighting parameters may be updated during training of model 100. Thus, during training of model 100, lighting schemes may be continuously updated, and training data may be obtained based on the updated lighting schemes to further train model 100, as described in detail below.

[0255] 2A and 2B illustrate exemplary processes 200 and 250 for training and applying a machine learning model (e.g., model 100) according to some embodiments. FIG2A illustrates the process when the parameters of the microscope cannot be changed during the acquisition of training data. FIG2B illustrates the process when the parameters of the microscope can be changed during the acquisition of training data and thus the optimal illumination mode can be identified.

[0256] Each of these processes may be performed at least in part using one or more electronic devices. In some embodiments, the blocks of each processing step shown in FIG. 2A and FIG. 2B may be divided between multiple electronic devices. In each process, some blocks may be optionally combined, the order of some blocks may be optionally changed, and some blocks may be optionally omitted. In some examples, additional steps may be performed in conjunction with the processes. Therefore, the operations shown (and described in more detail below) are exemplary in nature and should not be considered restrictive.

[0257] 2A , during the acquisition of the bright field images, the parameters of the microscope cannot be changed. Therefore, all bright field images in the training data (eg, training data 120 ) are obtained based on the same illumination mode.

[0258] At block 204, training data (e.g., Figure 1 The training data 120 is obtained at least in part according to the default lighting mode. Figure 1 The training data includes a first type of images (eg, bright field images) obtained by a bright field microscope according to a default illumination mode and a second type of images (eg, fluorescent images) obtained by a fluorescent microscope.

[0259] At block 206, the machine learning model (e.g., Figure 1 The model 100 is trained based on the training data. The model can be a GAN model. For example, the GAN network can be a conditional GAN ​​("cGAN") model.

[0260] Figure 3A , Figure 3C and Figure 3D The training process of a machine learning model according to some embodiments is shown. The GAN model includes a generator 302 and a discriminator 304. The generator 302 is trained to receive a first type of image 310 (e.g., a bright field image) and generate a second type of image 312 (e.g., a fluorescent image). In some embodiments, the first type of image 310 includes the above-mentioned bright field image array.

[0261] During training, the generator output (i.e., the generated fluorescence image) can be directly connected to the discriminator input. The discriminator 304 is trained to distinguish the second type of generated images 312 (e.g., generated fluorescence images) from the second type of real images 314 (e.g., real fluorescence images). In some embodiments, the second type of real images 314 includes the above-mentioned fluorescence image array.

[0262] Through back-propagation, the generator can use the output of the discriminator to update the weights of the generator so that the generator learns to generate images that the discriminator will classify as real images, as described in detail below. In some embodiments, the generator 302 and the discriminator 304 are neural networks.

[0263] Figure 3B Operation of an exemplary generator according to some embodiments is shown. Input image 352 may be a single image from an array of bright field images or an entire array of bright field images. For example, input image 352 may have dimensions (B, C, H, W), where B indicates the batch size, C indicates the number of channels, H indicates the height, and W indicates the width. If there is only a single bright field image, then C is equal to 1, and if there is a stack of bright field images, then C is greater than 1.

[0264] refer to Figure 3B, the generator includes a series of convolutional layers for downsampling the input image 352 to obtain a downsampled input image 354. In some embodiments, the input image 353 can be downsampled to half of its original size. For example, if the spatial size of the input is 256*256, the input will be reduced to 128*128.

[0265] The downsampled image 354 is then passed to a plurality of neural networks. In the depicted example, the plurality of neural networks includes four U-Net neural networks. The U-Net network is a convolutional network for image-to-image translation. Details of the design and implementation of the U-Net network can be found, for example, in Ronneberger et al., U-Net: Convolutional Networks for Biomedical Image Segmentation, the entire contents of which are incorporated herein by reference.

[0266] Multiple neural networks can correspond to different frequency groups in the wavelet domain. In the depicted example, four U-Net networks are responsible for low frequency, high frequency (horizontal), high frequency (vertical), and high frequency (diagonal), respectively. In signal processing, low frequency signals correspond to very large features relative to the image size (for example, when imaging cells, they have a size of the same order of magnitude as the cytoplasm or nucleus). High frequency information is very fine small image features (for example, having a size of the same order of magnitude as mitochondria, microtubules). Low frequency signals correspond to the first scale of wavelet coefficients. High frequencies are encoded in wavelet coefficients of higher scales. In some embodiments, multiple neural networks operate independently and do not share weights.

[0267] As shown in the figure, three of the four U-Net branches correspond to high-frequency boxes. Low-frequency information is relatively easy to recover; therefore, having more computational power dedicated to high frequencies ensures the reconstruction of fine details in the image. Therefore, the loss function operating in the wavelet domain benefits from this organization of the signal (high-frequency information is 3 times more than low-frequency information)

[0268] Each neural network is configured to output (or predict) the wavelet coefficients of the corresponding frequency group. The loss function is applied to the predicted wavelet coefficients 356 and the real wavelet coefficients of the real fluorescence image. Figure 3C and Figure 3D The loss function is further described.

[0269] The generated fluorescence image 358 in the image domain can be obtained by applying an inverse wavelet transform to the prediction coefficients 356 .

[0270] Figure 3CThe back propagation process of the discriminator 304 according to some embodiments is shown. The discriminator 304 is a model (e.g., a neural network) that is trained to provide an output based on a given image. The training data of the discriminator 304 includes a second type of real image 314 (e.g., a real fluorescence image) and a second type of synthetic image 312 (e.g., a generated fluorescence image) generated by the generator 302.

[0271] In some embodiments, the discriminator 304 is a PatchGAN network. Details of the design and implementation of the PatchGAN network can be found, for example, in Isola et al., "Image-to-Image Translation with Conditional Adversarial Networks", the entire contents of which are incorporated by reference.

[0272] During the training of the discriminator 304, the discriminator loss 322 can be calculated based on the output of the generator (i.e., the predicted wavelet coefficients). In some embodiments, the discriminator loss function is the Wasserstein discriminator loss and is calculated as follows:

[0273]

[0274] where f(x) is the output of the discriminator based on the wavelet coefficients of the real fluorescence image, w is the model weight of the discriminator, m is the size of the mini-batch, f is the discriminator model, x is the real image, z is the input (bright field), G is the generator model, and f(G(z)) is the output of the discriminator based on the predicted wavelet coefficients corresponding to the synthetic fluorescence image.

[0275] Discriminator 304 is configured to maximize this function. In other words, it attempts to maximize the difference between the output based on the real image and the output based on the synthesized image. Figure 3C As depicted, the discriminator updates its weights based on the discriminator loss 332 via back-propagation through the discriminator network.

[0276] Figure 3D 3 shows the back propagation process of the generator 302 according to some embodiments. The generator 302 is a neural network configured to receive a first type of image 310 and generate a second type of image 312, as shown in FIG. Figure 3A and Figure 3B The wavelet coefficients predicted by the generator are input into the discriminator 304. Based on the output of the discriminator, the generator loss 324 can be calculated. In some embodiments, the generator loss function is the Wasserstein generator loss and is calculated as follows:

[0277]

[0278] where f(x) is the discriminator output based on the wavelet coefficients of the real fluorescence image, m is the size of the mini-batch, f is the discriminator model, z is the input (bright field), G is the generator model, and f(G(z)) is the output of the discriminator based on the predicted wavelet coefficients.

[0279] The reconstruction loss working in the wavelet domain has the property of naturally balancing low-frequency and high-frequency contributions. Figure 3B As shown, the wavelet coefficients can be divided into two categories: low frequency (one box) and high frequency (3 boxes). Three of the four UNet branches can be dedicated to high frequency blocks. Low frequency information is easier to recover; therefore, having more computing power dedicated to high frequencies helps to reconstruct fine details in the image. The loss function operating in the wavelet domain benefits from this organization of the signal (there is three times more high frequency information than low frequency information). In some embodiments, the wavelet coefficients can be divided into more than two categories, for example, three categories (high frequency, medium frequency, low frequency), four categories, or more than four categories.

[0280] The generator 302 is configured to maximize this function. In other words, it attempts to maximize the output of the discriminator based on the synthesized image. In some embodiments, the generator loss is back-propagated through both the discriminator 304 and the generator 302 to obtain the gradient, which is in turn used to adjust the generator weights.

[0281] In some embodiments, the generator 302 and the discriminator 304 are trained in alternating cycles. In each cycle, the discriminator is trained for one or more epochs, and then the generator is trained for one or more epochs. During the training of the discriminator, the generator can be kept constant. Similarly, during the training of the generator, the discriminator can be kept constant.

[0282] In some embodiments, the generator can convert the input image into a third type of image (e.g., a phase-shifted image). For example, in addition to the generated fluorescence image, the generator can also output a phase-shifted image in which each pixel indicates a local value of the phase that can be converted in the image (e.g., -5 to 5 phase information). The physics-based image formation model can be used to generate a true phase-shifted image (i.e., a true phase-shifted image). The image formation model generates an image given the absolute knowledge of the microscope (e.g., aberrations of the optical system) and the optical properties of the captured sample (e.g., refractive index, phase). Because the optical properties of the sample can be compared between samples, the risk of batch effects in downstream tasks is almost null. The physics-based model allows reliable prior knowledge to be incorporated into the generation process.

[0283] refer to Figure 4During the training process, the illumination source S is optimized to obtain a high-fidelity X. The PSF (or pupil function) of the microscope can also be optimized using a spatial light modulator or a set of micromirrors. In addition, the polarization of S can be modulated to inject more contrast in the collected image.

[0284] I=∫|∫X(r)e fr P(fr)dr| 2 S(f)df

[0285] In the above formula, S(f) refers to the partially coherent illumination source (LED array). X(r) refers to the complex electron field of the specimen. P(r) refers to the point spread function (PSF) of the microscope. RI(r) refers to the refractive index. I refers to the image. The forward model is applied outside the training loop to obtain the ground truth phase.

[0286] In some embodiments, the generator can convert the input image into a fourth type of image. In some embodiments, the fourth type of image includes image data indicating segmentation data (e.g., cell boundaries). In some embodiments, the loss function depends on the supported image modality. For semantic segmentation, the L1 norm can be used to infer discrete labels in the image. For another image modality, another branch can be added to the generator to output the new modality.

[0287] 2A , metadata may be associated with the trained machine learning model at block 208. The metadata may include a default illumination mode (e.g., parameters for a bright field microscope).

[0288] At block 210, one or more images of a first type (eg, bright field images) are acquired. In some embodiments, the images are acquired using the same illumination mode (eg, as indicated in the metadata at block 208).

[0289] At block 212, one or more images are input into a generator (e.g., generator 302) of a trained machine learning model. The generator has been trained to convert images of a first type into images of a second type (e.g., fluorescence images). At block 214, one or more images of the second type are obtained. As described below, the generated fluorescence images can in turn be used as training data for other machine learning models, thereby eliminating the need to obtain real fluorescence images as training data.

[0290] Figure 2B shows the process when the parameters of the microscope can be changed during the acquisition of training data and thus the optimal illumination mode can be identified.For example, the parameters of the microscope can be programmable so that multiple illumination modes can be provided by the microscope.

[0291] At block 252, an illumination pattern is loaded onto the microscope. At block 254, a bright field training image (e.g., a bright field image) is captured according to the illumination pattern, and a fluorescence training image is also captured by the fluorescence microscope. Blocks 252 and 254 may be repeated, as indicated by arrow 253. In other words, a series of illumination patterns may be loaded onto the microscope, and training data corresponding to the series of illumination patterns may be obtained. The series of illumination patterns is also referred to herein as an illumination scheme.

[0292] At block 256, a machine learning model is trained based on the training data. The model may be as shown in FIG. FIG. 3A to FIG. 3D A GAN model that operates as described. For example, the GAN network can be a conditional GAN ​​("cGAN") model.

[0293] During training, the model iteratively updates the illumination pattern. The illumination pattern is updated by back-propagating the gradient of the loss (e.g., the generator loss) to the parameters of the microscope. The training procedure of the model minimizes the total imaging time and minimizes the loss function associated with the conversion or classification task.

[0294] During training, the system determines which lighting mode results in the smallest loss (e.g., generator loss). In some embodiments, the model produces a first generator loss when converting a first image, a second generator loss when converting a second image, and so on. These losses can be compared and the smallest loss can be identified. At box 260, the lighting mode that has produced the smallest loss can be identified, and a new lighting scheme can be identified accordingly. For example, the new lighting scheme can include the optimal lighting mode and / or one or more new lighting modes similar to the optimal lighting mode. The new lighting scheme can also exclude the previously included lighting mode that caused the largest loss. As shown by arrow 258, the identified lighting scheme can be loaded into the microscope to obtain additional training data. This process can be repeated until no more improvements (e.g., with respect to generator loss) are observed. The optimal lighting scheme can be stored at box 261.

[0295] At block 262 , one or more images of a first type (eg, bright field images) are obtained. In some embodiments, the images are obtained using the optimal illumination scheme stored in block 261 .

[0296] At block 264, one or more images are input into a generator (e.g., generator 302) of a trained machine learning model. The generator has been trained to convert images of a first type into images of a second type (e.g., fluorescence images). At block 266, one or more images of the second type are obtained. As described below, the generated fluorescence images can in turn be used as training data for other machine learning models, thereby eliminating the need to obtain real fluorescence images as training data.

[0297] Figure 5 The output of an exemplary trained generator using a static microscope setup is shown. The trained generator receives an input bright field image and generates a phase image and a fluorescence image 504. For comparison, Figure 5 The corresponding real phase image and fluorescence image 506 are also shown. Figure 5 In some embodiments, each input image is a composite of four different illuminations. In some embodiments, the input image is a false color image representing different illumination modes.

[0298] Fig. 6A An exemplary process for determining the robustness of a generated image according to some embodiments is shown. Two classifiers 612 and 614 are trained to receive a fluorescence image and provide an output (e.g., a phenotype classification result). Classifier 614 is trained based on a real fluorescence image 604 (e.g., a real fluorescence image), while classifier 612 is trained based on a generated fluorescence image 610. The generated fluorescence image is generated by a training generator 608 (e.g., generator 102, 302) based on a bright field image 506.

[0299] In some embodiments, the classifier 612 is validated using the generated images, while the classifier 614 is validated using the real images.

[0300] The performance of classifier 612 and classifier 614 may be compared to determine the robustness of the generated image. Figure 6B An exemplary comparison between two such classifiers is shown. In this particular setting, the dataset consists of fluorescence images and 3D bright field images of primary hepatocytes from 12 healthy donors and 12 NASH donors. The GAN model is trained to generate fluorescence images from a stack of out-of-focus bright field images.

[0301] Comparisons were made to determine whether classifiers trained on generated images and validated on real images performed equally well when trained on real images and validated on generated images. Figure 6B As shown, there is no drop in classification accuracy and, more importantly, the geometry of the embedding space of real images remains unchanged on the generated images.

[0302] Some embodiments include a back propagation module 616. The back propagation module 616 can be used, for example, to improve image acquisition parameters used to acquire the bright field image to be enhanced. For example, Xi can include setting parameters for the illumination mode of the LED illumination array applied to the bright field microscope. Xi can also include imaging parameters related to the focus position of the microscope objective and / or activation parameters of the spatial light modulator (SLM). All elements of Xi can be variables of the optimization process. Therefore, in 616, the gradient of the loss function estimated on the downstream task

[612] (e.g., classification, image-to-image conversion, regression) can be back propagated to optimize the parameters Xi. For each update of the value of Xi, a new set of images can be acquired, generating a new set of images of data.

[0303] In some embodiments, the data used to train the generator 608, the data used to train the first classifier 614 or the second classifier 612, and the data used to evaluate the performance of the classifiers may include overlapping images. The images used in any of the above processes may be annotated based on the captured biological sample (e.g., the type of cell, whether it is diseased or healthy) and the perturbation. These annotations can be used for downstream classification tasks (e.g., as training data to train Fig. 6A ) and validation of the prediction model (e.g., evaluating the performance of the trained classifiers 612 and 614).

[0304] Fig. 7A An example of a method 700 for training a machine learning model to generate an image of a biological sample according to one embodiment is shown. The method includes obtaining a plurality of training images including a first type of training images and a second type of training images at 702. The method further includes: generating a plurality of wavelet coefficients using the machine learning model based on the first type of training images at 704; generating a second type of composite image based on the plurality of wavelet coefficients at 706; comparing the second type of composite image to the second type of training images at 708; and updating the machine learning model based on the comparison at 710.

[0305] Figure 7B An example of a method 750 for generating an enhanced image of a biological sample according to one embodiment is shown. The method includes obtaining an image of the biological sample using a microscope at 752. The method also includes using at 754 Fig. 7A The machine learning model generates enhanced images of biological samples based on images.

[0306] Figure 8 800 is a host computer connected to a network. The device 800 may be a client computer or a server. Figure 8As shown, device 800 may be any suitable type of microprocessor-based device, such as a personal computer, workstation, server, or handheld computing device (portable electronic device) such as a phone or tablet computer. The device may include, for example, one or more of a processor 810, an input device 820, an output device 830, a storage device 840, and a communication device 860. The input device 820 and the output device 830 may generally correspond to those described above, and may be connected or integrated with the computer.

[0307] Input device 820 may be any suitable device that provides input, such as a touch screen, a keyboard or keypad, a mouse, or a voice recognition device. Output device 830 may be any suitable device that provides output, such as a touch screen, a tactile device, or a speaker.

[0308] Storage device 840 can be any suitable device that provides storage, such as electrical, magnetic or optical storage including RAM, cache, hard drive or removable storage disk. Communication device 860 can include any suitable device that can send and receive signals over a network, such as a network interface chip or device. The components of the computer can be connected in any suitable manner, such as through a physical bus or a wireless connection.

[0309] Software 850 , which may be stored in storage device 840 and executed by processor 810 , may include, for example, programming that embodies the functionality of the present disclosure (eg, embodied in a device as described above).

[0310] The software 850 may also be stored in and / or transmitted in any non-transitory computer-readable storage medium for use by or in conjunction with an instruction execution system, apparatus, or device (such as those described above), from which the software 850 may obtain instructions associated with the software and execute the instructions. In the context of the present disclosure, a computer-readable storage medium may be any medium, such as storage device 840, that may contain or store programming for use by or in conjunction with an instruction execution system, apparatus, or device.

[0311] The software 850 may also be propagated within any transmission medium for use by or in conjunction with an instruction execution system, device, or apparatus (such as those described above), from which the software 850 may obtain instructions associated with the software and execute the instructions. In the context of the present disclosure, a transmission medium may be any medium that can communicate, propagate, or transmit programming for use by or in conjunction with an instruction execution system, device, or apparatus. Transmission-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, or infrared wired or wireless propagation media.

[0312] Device 800 can be connected to a network, which can be any suitable type of interconnected communication system. The network can implement any suitable communication protocol and can be protected by any suitable security protocol. The network can include any suitable arrangement of network links that can implement the transmission and reception of network signals, such as a wireless network connection, a T1 or T3 line, a cable network, DSL, or a telephone line.

[0313] Device 800 may implement any operating system suitable for operating on a network. Software 850 may be written in any suitable programming language, such as C, C++, Java, or Python. In various embodiments, application software embodying the functionality of the present disclosure may be deployed in different configurations, such as in a client / server arrangement or as a network-based application or network service through a network browser.

[0314] Optimizing with Spatial Light Modulators

[0315] In some embodiments, an exemplary optical system includes a programmable SLM. The exemplary SLM includes a high-resolution liquid crystal panel with micron-sized individually addressable pixels, which can be used to shape the wavefront of a light beam. The grayscale values ​​on the panel can be converted into phase shifts. In some embodiments, the SLM can be used as a programmable Fourier filter for generating contrast enhancement, or as a programmable diffractive optical element for quantitative phase microscopy.

[0316] The SLM can be programmed to generate various images during the training phase and / or inference phase of a machine learning model to improve the performance of the machine learning model, as described herein.

[0317] During the inference phase of a trained machine learning model, the SLM can be programmed to generate different input images for the trained machine learning model. For example, the SLM can be programmed to generate input images for a trained image transformation model to obtain an enhanced version of the input image. The enhanced image can in turn be used for downstream operations. As another example, the SLM can be programmed to generate input images for a trained classification model to obtain more accurate classification results.

[0318] In addition, the SLM can be programmed to generate different images as training data during the training phase of the machine learning model. In addition, the SLM can be iteratively programmed to identify the best settings for capturing images that result in the best performance of a given machine learning model, either during the training phase or during the inference phase.

[0319] Accordingly, the SLM of the optical system can improve the performance of the machine learning model via a training phase (e.g., by providing a rich training data set) and / or via an inference phase (e.g., by providing input data under various optical settings or optimal settings). The programming of the SLM does not require any mechanical movement or modification of the optical system (such as a microscope). The SLM provides additional degrees of freedom for controlling the microscope. Multifocal acquisition can be performed without any mechanical movement, thereby accelerating and improving downstream tasks in an efficient manner.

[0320] Fig. 9 An exemplary optical system according to some embodiments is shown. The exemplary optical system includes a light source 902 (e.g., an LED array), an objective lens 904, an SLM 906, and a camera 908. The optical system has a reflection mode. In the reflection mode, the light source 902 provides illumination to the biological sample 910, generating reflected light. The reflected light passes through the objective lens 904 and is captured by the camera 908.

[0321] refer to Fig. 9 , dotted line 912 represents an intermediate image plane. As shown, the SLM is placed in the imaging path between the biological sample 910 and the camera 908. The SLM is configured to apply a spatially varying modulation to the reflected light. For example, the SLM allows the shaping of an alternative Fourier plane before the reflected light is focused on the camera 908.

[0322] In some embodiments, both the light source 902 and the SLM 906 may be programmable, thus allowing additional degrees of freedom to control and optimize the optical system without mechanically moving components of the optical system.

[0323] Fig. 9The configuration of the optical system in is merely exemplary. One of ordinary skill in the art will appreciate that other configurations of the optical system may be used to place the SLM in the imaging path of the optical system to apply the optimization techniques described herein.

[0324] Fig.10 An exemplary method for analyzing an image of a biological sample using a programmable SLM of an optical system according to some embodiments is shown. For example, process 1000 is performed using one or more electronic devices that implement a software platform. In some examples, process 1000 is performed using a client-server system, and the blocks of process 1000 are divided between a server and one or more client devices in any manner. In other examples, process 1000 is performed using only one or more client devices. In process 1000, some blocks may be optionally combined, the order of some blocks may be optionally changed, and some blocks may be optionally omitted. In some examples, additional steps may be performed in conjunction with process 1000. Therefore, the operations shown (and described in more detail below) are exemplary in nature and should not be considered limiting.

[0325] At block 1002, an exemplary system (e.g., one or more electronic devices) obtains a biological sample (e.g., Fig. 9 A plurality of images of a sample 910 in FIG. 9A are obtained by using an optical system SLM (e.g., Fig. 9 The SLM 906 is located in the optical path between the biological sample and the image recording device (e.g., camera 908).

[0326] In some embodiments, at least one configuration of the SLM is to generate one or more optical aberrations in the resulting image. Optical aberrations may include spherical aberration, astigmatism, defocus, tilt, or any combination thereof. In some embodiments, more information can be captured in and / or derived from an image with optical aberrations. For example, astigmatism allows multiple focal plane information to be collected in one image. As another example, defocus allows the system to scan a sample without any mechanical movement. An exemplary method for generating optical aberrations is provided in "Quantitative Phase Imaging and Complex Field Reconstruction by Pupil Modulation Differential Phase Contrast" by Lu et al., the entire contents of which are incorporated herein by reference.

[0327] In some embodiments, at least one configuration of the SLM is to enhance one or more features. The one or more features may include cell boundaries, actin filaments, nuclear shape, cytoplasm segmentation, or any combination thereof. In some embodiments, the system provides enhancement of neurites with a specific convolution kernel encoded in the transfer function of the microscope. In some embodiments, the system enhances the response function of the microscope to certain liquid-phase-separated biological objects (proteins, RNA, lipids). This enhancement will provide the ability to detect and characterize the composition of these objects. For example, using a spiral phase pattern as a phase filter in the Fourier plane will produce a point spread function similar to a ring. Convolution with an extended amplitude or phase object leads to strong isotropic edge enhancement in the image. In a homogeneous sample (homogeneous region of the sample), destructive interference occurs due to the π phase shift on the ring (for any angle on the ring). The structure in the sample can produce less imperfect cancellation, thereby producing local brightening in the image. Therefore, light is redistributed to the edges and borderlines within the sample. Exemplary methods for enhancing features are provided in Fürhapter et al., “Spiral phase contrast imaging in microscopy”, Fürhapter et al., “Shadow effects inspiral phase contrast microscopy”, Bernet et al., “Quantitative imaging of complex samples in spiral phase contrast microscopy”, and Maurer et al., “Upgrading a microscope with a spiral phase plate”, the entire contents of which are incorporated herein by reference.

[0328] In some embodiments, at least one configuration of the SLM is to reduce optical aberrations. For example, for real-time imaging and continuous monitoring of cells, it is crucial to reduce the variability of samples from plates or fragments in the wells. The SLM can correct these aberrations. In some embodiments, the SLM can be used for iterative correction of phase aberrations, for example based on the Gerchberg-Saxton algorithm. For example, the dark center of a single optical vortex can be used as a key sensor for residual phase aberrations. Jesacher et al. "Wavefront correction ofspatial light modulators using an optical vortex image" and Maurer et al. "Phase contrast microscopy with full numerical aperture illumination" provide exemplary methods for enhancing features, the entire contents of which are incorporated herein by reference.

[0329] In some embodiments, multiple SLM configurations are used to obtain images of biological samples at different depths. For example, SLM allows flexible image multiplexing, for example, to combine images from different depths of the sample in one recorded image, or for different settings of imaging parameters. Image multiplexing can facilitate quantitative phase microscopy. Imaging of living samples must be as fast as possible to minimize stress on cells. Mechanical movement of three-dimensional scanning of samples is prohibited. This system allows Fourier thermal imaging and reconstruction of large three-dimensional volumes from low-resolution images of the entire hole. Exemplary methods of multiplexing are provided in "Depth-of-field-multiplexing in microscopy" by Maurer et al., "Differential interference contrast imaging using a spatial light modulator" by McIntyre et al., and "Quantitative SLM-based differential interference contrast imaging" by McIntyre et al., the entire contents of which are incorporated herein by reference.

[0330] In some embodiments, the system allows reconstruction of high-resolution images by optimized Fourier thermography. Fourier thermography is a computational imaging technique based on optical microscopy that involves assembling a wider numerical aperture from full-field image sets acquired using different optical settings, resulting in higher resolution than conventional microscopy. Images in an image set can be acquired using different configurations of LEDs and / or SLMs; then, an iterative phase retrieval algorithm can be used to combine the acquired image sets into a final high-resolution image that can contain up to 1 billion pixels (giga-pixels) with a diffraction-limited resolution, resulting in a high spatial-bandwidth product.

[0331] The light source of the optical system may also be programmed. In some embodiments, multiple configurations of the light source of the optical system (e.g., light source 902) are used at block 1002 to obtain multiple images. For example, reference is made herein to Figure 1 2B depicts an exemplary method for programming a light source.

[0332] Back to Fig.10 , at box 1004, the system inputs multiple images of the biological sample into the trained machine learning model to obtain one or more outputs.

[0333] In some embodiments, the trained model is an image transformation model configured to generate an output image of a second type (e.g., an enhanced version of the input image) based on an input image of a first type. In some embodiments, the enhanced version of the input image includes an enhanced cell phenotype. In some embodiments, the trained model is a GAN model or a self-supervised model. For example, the trained model can be a model 100 configured to receive a bright field image and generate a fluorescence image.

[0334] In some embodiments, the trained model is a classification model configured to provide a classification output.For example, the model can receive an input image and detect one or more predefined objects in the input image, such as diseased tissue.

[0335] Fig.11An exemplary method for training a machine learning model according to some embodiments is shown. For example, process 1100 is performed using one or more electronic devices that implement a software platform. In some examples, process 1100 is performed using a client-server system, and the blocks of process 1100 are divided between a server and one or more client devices in any manner. In other examples, process 1100 is performed using only one or more client devices. In process 1100, some blocks may be optionally combined, the order of some blocks may be optionally changed, and some blocks may be optionally omitted. In some examples, additional steps may be performed in conjunction with process 1100. Therefore, the operations shown (and described in more detail below) are exemplary in nature and should not be considered restrictive.

[0336] At block 1102, an exemplary system (e.g., one or more electronic devices) obtains a plurality of images of a biological sample. Using an SLM of an optical system (e.g., Fig. 9 The plurality of images are generated by using a plurality of configurations of an SLM 906 in the biomedical apparatus. The SLM is located in the optical path between the biological sample and an image recording device (eg, a camera 908).

[0337] In some embodiments, the SLM configuration is configured to create a desired effect in the resulting image. As described above, some of the multiple configurations of the SLM can be configured to generate one or more optical aberrations (e.g., spherical aberration, astigmatism, defocus, distortion, tilt, or any combination thereof), enhance one or more features (e.g., cell boundaries, actin filaments, nuclear shape, cytoplasm segmentation, or any combination thereof), reduce optical aberrations, obtain images of biological samples at different depths, etc.

[0338] At block 1104, the system trains a machine learning model using the plurality of images. In some embodiments, the trained model is an image transformation model configured to generate an output image of a second type (e.g., an enhanced version of the input image) based on an input image of a first type. In some embodiments, the enhanced version of the input image includes an enhanced cellular phenotype. In some embodiments, the trained model is a GAN model or a self-supervised model. For example, the trained model can be a model 100 configured to receive a bright field image and generate a fluorescent image. Figure 3A-3D Describe an exemplary method for training a model.

[0339] In some embodiments, the trained model is a classification model configured to provide a classification output. For example, the model can receive an input image and detect one or more predefined objects in the input image, such as diseased tissue. Training of the classification model can be performed using multiple images and associated labels.

[0340] In some embodiments, at block 1104, the SLM can be iteratively programmed to identify an optimal SLM configuration for capturing an image that results in improved performance of a given machine learning model. Specifically, at block 1106, the system trains the machine learning model using a first image, where the first image is obtained using a first configuration of the SLM of the optical system. At block 1108, the system evaluates the trained machine learning model. At block 1110, the system identifies a second configuration of the SLM based on the evaluation. At block 1112, the system trains the machine learning model using a second image, where the second image is obtained using a second configuration of the SLM of the optical system.

[0341] For example, at block 1106, the system trains the model using images corresponding to a first set of SLM configurations. Based on the loss function of the model, each image results in a corresponding loss. At block 1108, the system determines which SLM configuration in the first set of SLM configurations results in a minimum loss (e.g., a generator loss). At block 1110, the SLM configuration that has produced the minimum loss can be identified, and a new second set of SLM configurations can be identified accordingly. For example, the new set of SLM configurations can include the best SLM configuration from the first set (i.e., the configuration that produces the minimum loss) and / or one or more new SLM configurations that are similar to the best SLM configuration. The new set of SLM configurations can also exclude the SLM configuration that results in the largest loss of the first set. The new set of SLM configurations can be loaded onto the optical system to obtain additional training data. This step can be repeated until a threshold is reached, for example, when no more improvements (e.g., improvements in generator loss) are observed. The optimal SLM configuration can be stored and used to obtain an input image.

[0342] Although steps 1106 to 1112 are described as part of the training process, they can be performed in other stages of the pipeline (e.g., the inference stage) to identify the optimal SLM configuration for generating the input image. In some embodiments, the light source and SLM of the optical system can be iteratively programmed together to identify the optimal combination of illumination mode and SLM configuration for generating the input image.

[0343] although Figures 9 to 11 Optimization techniques using the SLM component of an optical system are described, but it should be understood that the SLM may be replaced with another hardware component (such as a micromirror) that can change the optical function (e.g., pupil function) of the system without departing from the spirit of the invention.

[0344] Fig. 12A and Fig. 12B Shown is a side-by-side comparison of classification results of two classification models according to some embodiments. Fig. 12AThe classification model in is trained using real images captured by a microscope (e.g., real fluorescence images), while Fig. 12B The classification models in are trained using synthetic images generated using the techniques described herein (e.g., fluorescent images generated from bright field images). The classification models determine to which compound the tissue depicted in the input image reacts. Specifically, each model is configured to receive an input image and output a classification result indicating one of 150 predefined compounds. In the example shown, Fig. 12A and Fig. 12B Each of the 2D images shows a uniform manifold approximation and projection (UMAP), where each input image is represented as a point in the UMAP. The color of the point represents the compound that the image is classified by the classification model. In some embodiments, Fig. 12A The input image in the model is a real image, and Fig. 12B The input image in the model is the generated image.

[0345] Fig. 12C and Fig.12D They are shown respectively Fig. 12A and Fig. 12B A side-by-side comparison of the same classification results in , with different color schemes to show how resistant the two models are to batch effects. Batch effects are situations where the distribution of a subset of the data (i.e., batches) differs significantly due to unrelated instrument-related factors. Batch effects are undesirable because they introduce systematic error, which can lead to spurious results and / or confound the signal of interest in downstream statistical analyses. Fig. 12C and Fig.12D In each of the three images, the input images belong to three different batches (e.g., from different plates or experiments), as indicated by different gray levels. Fig.12D A larger overlap between the points corresponding to the three batches is shown, indicating that the generated images are more resistant to batch effects. Fig.12E The corresponding image set (e.g. Fig. 12A and Fig. 12C The input image shown in Fig. 12B and Fig.12D The Euclidean distance metric for each image set measures the distance between the input images (shown in Figure 2) and the true batch-invariant image set. ... Fig. 12C The real image batch 1 and the real image batch 3 in Fig.12D Euclidean distance between image embeddings of generated image batch 1 and generated image batch 3 in . For a truly batch-invariant set of images, the mean score should be 0 (i.e., no batch effect). As shown in the figure, the mean score of the generated image set is lower than the mean score of the true images, thus demonstrating superior batch invariance.

[0346] Fig.13A and Fig. 13B An exemplary generated phase image and an exemplary generated fluorescence image according to some embodiments are shown. Fig.13A is a phase image generated by the GAN model described in this paper, for example from a bright field image. Fig. 13B Figure 2. Fluorescence / bodipy images generated by GAN from the same bright field image. The bodipy image includes a virtual green tint to highlight the presence of the biomarker. Fig. 13C Shown superimposed on Fig.13A The phase image of Fig. 13B Therefore, GAN models can be used for disease modeling. For example, multiple samples can be obtained from a subject and imaged. The resulting series of bright field images can be analyzed by the GAN model to generate synthetic phase and fluorescence images to study disease perturbations.

[0347] FIG. 14A to FIG. 14B An exemplary process for training a machine learning model (e.g., a GAN model) configured to generate synthetic data (e.g., image data) and identifying an optimal lighting scheme for obtaining input data for the machine learning model according to some embodiments is shown. For example, process 1400 is performed using one or more electronic devices that implement a software platform. In some examples, process 1400 is performed using a client-server system, and the blocks of process 1400 are divided between the server and one or more client devices in any manner. In other examples, process 1400 is performed using only one or more client devices. In process 1400, some blocks can be optionally combined, the order of some blocks can be optionally changed, and some blocks can be optionally omitted. In some examples, additional steps can be performed in conjunction with process 1400. Therefore, the operations shown (and described in more detail below) are exemplary in nature and should not be considered limiting.

[0348] At block 1402, an exemplary system (e.g., one or more electronic devices) receives a plurality of training images. The plurality of training images are real images of biological samples and are also referred to as ground truth data. The plurality of training images include image data types that the GAN model is configured to receive and image data types that the GAN model is configured to output. For example, if the GAN model is configured to receive bright field images and output fluorescence images and phase images, the received plurality of images will include a plurality of bright field training images 1402a (i.e., GAN input data type), a plurality of fluorescence training images 1402b (i.e., GAN output data type), and a plurality of phase training images (i.e., GAN output data type).

[0349] The bright field training images in the plurality of 1402a can be captured by illuminating an in vitro (or biopsy) cell sample with an inexpensive LED array using different illumination settings. The fluorescence training images in the plurality of 1402b can be captured after the dye is applied to the biological sample (e.g., to enhance the visibility of the biomarker). Phase training images can be obtained using a physics-based or optical model. It should be understood by a person of ordinary skill in the art that the plurality of training images used in process 1400 may differ depending on the type of image data that the GAN is configured to receive and output. In some embodiments, the plurality of training images include paired image data. For example, a bright field image, a fluorescence image, and a phase image of the same biological sample may be included in sets 1402a, 1402b, and 1402c, respectively.

[0350] In some embodiments, a plurality of training images are acquired to enable training of the GAN model so that the synthetic images (e.g., synthetic fluorescence images, synthetic phase images) generated by the GAN model will provide the same performance as real images (e.g., real fluorescence images, real phase images) in downstream analysis. In some embodiments, the downstream analysis includes a classification task that classifies the image into a category corresponding to one of M categories. For example, the classification task may involve classifying the image into a specific cell state corresponding to a plurality of cell state categories (e.g., healthy state, diseased state). As another example, the classification task may involve classifying the image into a specific perturbation corresponding to a plurality of perturbation categories. In order to train the GAN model to generate synthetic images that can be classified as accurately as real images, the training images include images corresponding to M categories (or conditions). For example, if the M categories include a healthy cell state category and a diseased cell state category, the plurality of bright field training images 1402a may include a bright field image depicting a healthy cell and a bright field image depicting a diseased cell, the plurality of fluorescence training images 1402b may include a fluorescence image depicting a healthy cell and a fluorescence image depicting a diseased cell, and the plurality of phase training images 1402c may include a phase image depicting a healthy cell and a phase image depicting a diseased cell. For example, if the M categories include M perturbations, the plurality of bright field training images 1402a may include a bright field image depicting the M perturbations, and so on. Each training image may be labeled with a corresponding condition. For example, a phase image depicting a diseased cell state may be associated with a diseased label.

[0351] In an exemplary implementation, the plurality of training images includes X fields of view per condition, i.e., a total of M x X fields of view. Specifically, in the plurality of bright field images 1402a, each field of view includes N bright field images captured using N illumination settings, thus generating a total of M x X x N bright field images. In the plurality of fluorescent images 1402b, each field of view includes a fluorescent image, thus generating a total of M x X fluorescent images. In the plurality of phase images 1402c, each field of view includes a phase image, thus generating a total of M x X phase images. In some embodiments, the magnification of the bright field image = m in , and the magnification of the fluorescence image is m out >=m in , and the magnification of the phase image is m out >=m in .

[0352] At block 1404, the system trains a classifier configured to receive an input image and output a classification result indicating one of the M conditions. For example, if the M conditions include a healthy condition and a diseased condition, the classifier is configured to receive an input image and output a classification result indicating the healthy condition or the diseased condition. After the classifier is trained, the classifier is used during training of the GAN model to ensure that the GAN model can generate synthetic image data that can be classified to the same or similar level of accuracy as real image data, as described below.

[0353] In some embodiments, the classifier is trained using the same type of image data that the GAN model is configured to output. Fig.14A and Fig. 14B In the example shown, the GAN model is configured to output a fluorescence image and a phase image; therefore, a plurality of fluorescence training images 1402b and a plurality of phase training images 1402c are used to train the classifier trained in block 1404. During training, each fluorescence image or phase image is input into the classifier to obtain a predicted classification result (e.g., healthy or diseased). The predicted classification result is then compared with the actual category associated with the training image (e.g., whether the training image actually depicts a healthy cell state or a diseased cell state), and based on the comparison, the classifier can be updated accordingly. The classifier can be implemented using any classification algorithm, such as a logistic regression model, a naive Bayes model, a decision tree model, a random forest model, a support vector machine model, and the like.

[0354] At block 1406, the system trains a GAN model based on the training images. Block 1406 may include steps 1408a to 1408e, which may be repeated until the training is complete (e.g., when convergence is reached). Fig.15 Describing steps 1408a-e, Fig.15 is a schematic diagram illustrating these steps, according to some embodiments.

[0355] like Fig.15 As shown, the GAN model includes a multi-head attention layer 1502, which includes a weight matrix, specifically, K sets of weights w 1 tow n , generator 1504, discriminator 1508, and the trained classifier 1506 from block 1404. During training of the GAN model, the classifier remains fixed while the generator, discriminator, and attention layers are updated as described below.

[0356] At block 1408a, the system applies each of the K sets of weights in the attention layer of the GAN model to the set of bright field training images. The set of bright field training images is obtained from the plurality of bright field images 1402a. In some embodiments, the set of bright field training images corresponds to the same field of view and depicts the same biological sample, but is captured using different illumination settings. For example, if the LED array includes N illumination emitters (e.g., LEDs), each illumination emitter can be turned on one at a time, and a bright field image of the biological sample illuminated by each illumination emitter can be captured, thereby generating a set of N bright field training images.

[0357] exist Fig.15 In the example shown, the system receives a set of N bright field training images corresponding to illumination settings 1 - N. For example, the first bright field image depicts a biological sample illuminated using illumination setting 1 (e.g., only the first LED in the array is turned on), the second bright field image depicts a biological sample illuminated using illumination setting 2 (e.g., only the second LED in the array is turned on), ... and the Nth bright field image depicts a biological sample illuminated using illumination setting N (e.g., only the Nth LED in the array is turned on).

[0358] The attention layer generates K sets of weights, and each set of weights includes N weights. 1 tow n is applied to N images to generate an aggregate image. For each set of weights, the attention layer 1502 assigns a continuous weight (e.g., a normalized scalar weight) in the set to each bright field training image in the set of bright field training images. These weights correspond to the intensity values ​​of the corresponding lighting settings (e.g., the corresponding LEDs). As shown in the figure, w 1 applied to (e.g., multiplied by) the first bright field image, w 2 Applied to the second bright field image, w nApplied to the Nth bright field image. After applying the weights, the weighted images can be aggregated (e.g., summed) to obtain a clustered bright field image. Because there are K sets of weights, K clustered images 1512 can be generated. In some embodiments, the attention layer is an adapted multi-head attention layer. The attention mechanism allows K linear combinations of bright field images (i.e., clustered images) to be naturally generated. As described herein, these clustered images are fed to the rest of the network.

[0359] At block 1408b, the system inputs each of the gathered bright field images into the GAN model. Fig.15 Each of the bright field images 1512 is input to the generator 1504, and the generator 1504 outputs a synthetic fluorescence image 1514a and a synthetic phase image 1514b. The generator can be used in the same manner as in the above reference. FIG. 3A to FIG. 3D The generator described is implemented in a similar way.

[0360] During training, the generator outputs (i.e., the generated fluorescence image 1514a and the generated phase image 1514b) can be directly connected to the discriminator inputs. The discriminator 1508 is trained to distinguish generated images from real images. During training, the generated images can be input into the discriminator 1508 to obtain the discriminator loss and the generator loss as described above. In addition, the real image can also be input into the discriminator 1508 to generate the discriminator loss and the generator loss. The real image can be a real fluorescence image 1516a (from the same field of view as the input bright field image) corresponding to the same field of view as the input bright field image. Fig.14A or a plurality of fluorescent training images 1402b in FIG. 1 , or a real phase image 1516b (from FIG. 1 ) corresponding to the same field of view as the input bright field image. Fig.14A Multiple phase training images 1402c).

[0361] In some embodiments, the discriminator loss function is the Wasserstein discriminator loss and is calculated as follows:

[0362]

[0363] Where f(x) is the output of the discriminator based on the wavelet coefficients of the real fluorescence or phase image, w is the model weight of the discriminator, m is the size of the mini-batch, f is the discriminator model, x is the real image, z is the input (bright field image 1512), G is the generator model, and f(G(z)) is the output of the discriminator based on the predicted wavelet coefficients corresponding to the synthetic fluorescence or phase image.

[0364] In some embodiments, the generator loss function is a Wasserstein generator loss and is calculated as follows:

[0365]

[0366] where f(x) is the discriminator output based on the wavelet coefficients of the real fluorescence image or phase image, m is the size of the mini-batch, f is the discriminator model, z is the input (bright field), G is the generator model, and f(G(z)) is the output of the discriminator based on the predicted wavelet coefficients.

[0367] At block 1408c, the system inputs each generated image and the real image corresponding to the generated image into the trained classifier to obtain the classifier loss. For example, the generated fluorescence image 1514a is input into the classifier 1506 to obtain a first classification result; the real fluorescence image 1516a is input into the classifier 1506 to obtain a second classification result; the classifier loss can be calculated based on the difference between the first classification result and the second classification result. As another example, the generated phase image 1514b is input into the classifier 1506 to obtain a third classification result; the real phase image 1516b is input into the classifier 1506 to obtain a fourth classification result; the classifier loss can be calculated based on the difference between the third classification result and the fourth classification result.

[0368] At block 1408d, the system increases the generator loss based on the classifier loss. For example, the generator loss can be increased by the L2 norm of the classification score from the real image. In some embodiments, if no classifier is available or classification is not required, the classifier loss is replaced by a constant (e.g., 0).

[0369] At block 1408e, the system updates the GAN model based on the increased generator loss. Back propagation follows the same process as described above. The discriminator updates its weights through the discriminator network based on the discriminator loss by back propagation. In addition, the increased generator loss is back propagated to update the weights in the attention layer (e.g., the weights used to calculate the clustered images) and the generator. For example, the generation loss calculated based on the clustered images corresponding to the Kth set of weights can be used to update the Kth set of weights in the attention layer.

[0370] At block 1410, the system obtains one or more optimal lighting patterns based on the weights in the attention layer of the trained GAN model. As described above, the weights in the attention layer (e.g., w 1 tow n ) may indicate an intensity value of a corresponding lighting setting (e.g., a corresponding LED in an LED array). As described above, the attention layer may be a multi-headed attention layer that provides K sets of weights (i.e., K linear combinations), thereby generating K lighting patterns.

[0371] In some embodiments, before block 1406, the generator and discriminator of the GAN model are pre-trained using bright field images where all LEDs in the LED array are turned on. After pre-training, block 1406 is performed to update the attention weights while the generator and discriminator remain fixed. An optimal lighting combination can be obtained based on the updated weights.

[0372] Fig.16A A synthetic image generated by an exemplary GAN model for studying NASH disease according to some embodiments is shown. In the depicted embodiment, Hepg2 cells with a NASH genetic background are illuminated using the optimal illumination pattern identified using the techniques described herein, and a bright field image is captured. The bright field image is input into the GAN model, and the GAN model outputs Fig.16A The generated phase image and the generated fluorescence / bodipy image. Fig.16A The GAN model used in can be configured to include the following reference FIG. 14A to FIG. 14B and Fig.15 A trained classifier as described. The classifier has two conditions: a healthy condition (e.g., no perturbation) and a diseased condition (e.g., inflammatory mixture plus fatty acids). As described above, the GAN model can be trained so that the classifier can classify generated images to the same or similar accuracy as real images.

[0373] Synthetic images generated by GAN models can be used to create disease models. Fig. 16B Downstream analysis of the generated images according to some embodiments is shown. As shown, the generated images can be used to perform cell nucleus segmentation and spot detection (e.g., using image processing algorithms). Therefore, the generated images can be used to create a disease model of NASH disease and study chemical perturbations in NASH disease.

[0374] The synthetic images generated by the GAN model can also be used to evaluate the efficacy of treatments. The system processes three sets of generated images: a first set of images depicting healthy tissue without disease, a second set of images depicting untreated diseased tissue, and a third set of images depicting diseased tissue that has been treated (for example, with a specific drug). Fig. 16CIn the example shown, a first set of images (labeled "untreated") includes images of healthy tissue without nonalcoholic steatohepatitis (NASH) disease, a second set of images (labeled "NASH 2X") includes images of untreated tissue with NASH disease, and a third set of images (labeled "NASH 2X+ACC inhibitor") includes images of tissue with NASH that has been treated with a drug (e.g., 50 micromolar ACC inhibitor or acetyl-CoA carboxylase inhibitor (firsocostate)). In some embodiments, the three sets of images capture tissue of the same object at different times. In some embodiments, the three sets of images capture tissue of different objects. The images can be phase images or fluorescence images generated using the techniques described herein (e.g., from bright field images). Each phase or fluorescence image can be generated based on multiple bright field images and therefore includes richer information, such as phase shift information. Therefore, the generated phase or fluorescence images can achieve higher accuracy in downstream analysis, as described below.

[0375] Specifically, to evaluate the efficacy of a drug, the system generates a distribution for each set of images to determine whether the distribution reflects the effect of the drug on the disease state. Fig. 16C , three probability distributions are generated using a classifier trained on the generated images. The X-axis indicates the probability output by the classifier when receiving an input image (e.g., a generated phase image). As shown in the three distributions, the generated images of healthy tissue (i.e., distribution 1620) are generally classified as having a lower probability of disease than the generated images of diseased tissue (i.e., distribution 1624). In addition, the generated images of treated tissue (i.e., distribution 1622) are generally classified as having a lower probability of disease than the generated images of diseased tissue. Comparison of these distributions can indicate that the drug is effective in reversing or reducing the disease state because the treatment has caused the diseased tissue to include features that are more similar to the healthy state and less similar to the diseased state. Therefore, the synthetic images can achieve a biophysical understanding of lipid distribution in cells and provide insights into treatment.

[0376] In some embodiments, instead of using distributions, the system can identify image clusters in an embedding space (e.g., UMAP), such as Fig.16D As shown. In UMAP, each point represents an image embedding of an image (e.g., a generated phase image). As shown, the clusters formed by the generated images of the treated tissue are away from the clusters of the generated diseased images and toward the clusters of the generated healthy images, which can indicate that the drug is effective in reversing or reducing the disease state.

[0377] Fig. 16C and Fig.16DThe analysis in can be applied to evaluate multiple candidate drugs for a disease of interest. For example, tissue treated by each candidate drug can be imaged, for example, by a microscope, to obtain a bright field image. The bright field image can be converted into a phase image using the techniques described herein. A distribution or cluster can be generated for each phase image of the treatment. Thus, the resulting graph can include a distribution or cluster representing a disease state, a distribution or cluster representing a healthy state, and multiple distributions or clusters each representing a candidate drug. The system can then identify the distribution or cluster that is closest to the healthy state to identify the most effective candidate drug.

[0378] Fig.17A A synthetic image generated by an exemplary GAN model for studying tuberous sclerosis complex ("TSC") in accordance with some embodiments is shown. In the depicted example, the NGN2 neuron is illuminated using the optimal illumination pattern identified using the techniques described herein, and a bright field image is captured. The bright field image is input into the GAN model, and the GAN model outputs Fig.17A The generated phase image and the generated fluorescence / bodipy image. Fig.17A The GAN model used in can be configured to include the following reference FIG. 14A to FIG. 14B as well as Fig.15 The trained classifier described. The classifier has 2 conditions: healthy condition (e.g., wild type) and diseased condition (e.g., TSC KO). As described above, the GAN model can be trained so that the classifier can classify the generated images to the same or similar accuracy as the real images.

[0379] The synthetic images generated by the GAN model can also be used to evaluate the efficacy of treatments. The system processes three sets of generated images: a first set of images depicting healthy tissue without disease, a second set of images depicting untreated diseased tissue, and a third set of images depicting diseased tissue that has been treated (for example, with a specific drug). Fig. 17B In the example shown, a first set of images (labeled "Wild Type") includes images of healthy tissue without TSC disease, a second set of images (labeled "TSC") includes images of untreated tissue with TSC disease, and a third set of images (labeled "TSC+Rapamycin") includes images of tissue with TSC that has been treated with a drug. In some embodiments, the three sets of images capture tissue of the same subject at different times. In some embodiments, the three sets of images capture tissue of different subjects. The images are phase images generated using the techniques described herein (e.g., from bright field images).

[0380] Specifically, to evaluate the efficacy of a drug, the system generates a distribution for each set of images to determine whether the distribution reflects the effect of the drug on the disease. Fig. 17BIn , three biomarker profiles are generated. As shown in these profiles, the drug appears to be effective in reversing or reducing the disease state because the treatment has caused the diseased tissue to include features that are more similar to the healthy state and less similar to the diseased state. Fig. 17B The analysis in can be applied to evaluate a variety of drug candidates for a disease of interest, as described above.

[0381] Exemplary methods, non-transitory computer-readable storage media, systems, and electronic devices are set forth in the following:

[0382] 1. A method for training a machine learning model to generate an image of a biological sample, the method comprising:

[0383] A plurality of training images are obtained, wherein the plurality of training images include:

[0384] training images of the first type, and

[0385] The second type of training images;

[0386] Based on the first type of training images, generating a plurality of wavelet coefficients using the machine learning model;

[0387] generating a composite image of the second type based on the plurality of wavelet coefficients;

[0388] comparing the second type of synthesized image to training images of the second type; and

[0389] The machine learning model is updated based on the comparison.

[0390] 2. A method according to claim 1, wherein the first type of training images are bright field images of biological samples.

[0391] 3. A method according to item 2, wherein the second type of training image is a fluorescent image of the biological sample.

[0392] 4. A method according to any one of items 1-3, wherein the machine learning model includes a generator and a discriminator.

[0393] 5. A method according to item 4, wherein the machine learning model includes a conditional GAN ​​model.

[0394] 6. A method according to any one of items 4-5, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.

[0395] 7. A method according to item 6, wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for a corresponding frequency group.

[0396] 8. A method according to any one of items 6-7, wherein the multiple neural networks include multiple U-Net neural networks.

[0397] 9. A method according to any one of items 5-8, wherein the discriminator is a PatchGAN neural network.

[0398] 10. The method according to any one of items 1 to 9 further comprises: generating a third type of image based on the first type of training images.

[0399] 11. A method according to item 10, wherein the third type of image is a phase-shifted image.

[0400] 12. The method according to any one of items 1-11 further comprises: generating a fourth type of image based on the first type of training images.

[0401] 13. A method according to clause 12, wherein the fourth type of image comprises segmentation data.

[0402] 14. A method according to any one of items 1-13, wherein the first type of training images is captured using a microscope according to a first illumination scheme.

[0403] 15. A method according to clause 14, wherein the first lighting scheme comprises one or more lighting modes.

[0404] 16. A method according to any of items 14-15, wherein the first type of training images is part of a bright field image array.

[0405] 17. A method according to any one of clauses 14 to 16, wherein the plurality of training images is a first plurality of training images, the method further comprising:

[0406] Based on the comparing, identifying a second lighting scheme;

[0407] obtaining a second plurality of training images including one or more images of the first type, wherein the one or more images of the first type are obtained based on the second lighting scheme;

[0408] The machine learning model is trained based on the second plurality of training images.

[0409] 18. The method according to any one of items 1 to 16, further comprising:

[0410] obtaining a plurality of images of the first type using a microscope; and

[0411] Based on the obtained multiple images, the second type of multiple synthetic images are generated using the machine learning model.

[0412] 19. The method of claim 18, further comprising: training a classifier based on a plurality of synthetic images of the second type.

[0413] 20. The method of clause 19, wherein the microscope is a first microscope, wherein the classifier is a first classifier, the method further comprising:

[0414] obtaining a plurality of images of said second type using a second microscope;

[0415] training a second classifier based on the plurality of images of the second type;

[0416] The performance of the first classifier and the second classifier are compared.

[0417] 21. A system according to claim 20, wherein the second microscope is a fluorescence microscope.

[0418] 22. A method for generating an enhanced image of a biological sample, comprising:

[0419] Obtain images of biological samples using a microscope; and

[0420] Based on the image, an enhanced image of the biological sample is generated using a machine learning model, wherein the machine learning model has been trained by:

[0421] A plurality of training images are obtained, wherein the plurality of training images include:

[0422] training images of the first type, and

[0423] The second type of training images;

[0424] Based on the first type of training images, generating a plurality of wavelet coefficients using the machine learning model;

[0425] generating a composite image of the second type based on the plurality of wavelet coefficients;

[0426] comparing the second type of synthesized image to training images of the second type; and

[0427] The machine learning model is updated based on the comparison.

[0428] 23. A method according to claim 22, wherein the first type of training images are bright field images of biological samples.

[0429] 24. A method according to item 22, wherein the second type of training images are fluorescent images of the biological sample.

[0430] 25. A method according to any one of items 22-24, wherein the machine learning model includes a generator and a discriminator.

[0431] 26. A method according to claim 25, wherein the machine learning model includes a conditional GAN ​​model.

[0432] 27. A method according to any one of items 25-26, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.

[0433] 28. A method according to item 27, wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for a corresponding frequency group.

[0434] 29. A method according to any one of items 27-28, wherein the multiple neural networks include multiple U-Net neural networks.

[0435] 30. A method according to any one of items 26-29, wherein the discriminator is a PatchGAN neural network.

[0436] 31. A method according to any one of items 23-30, further comprising: generating a third type of image based on the first type of training image.

[0437] 32. A method according to item 31, wherein the third type of image is a phase-shifted image.

[0438] 33. A method according to any one of items 23-32, further comprising: generating a fourth type of image based on the first type of training image.

[0439] 34. A method according to item 33, wherein the fourth type of image includes segmentation data.

[0440] 35. A method according to any one of items 23-34, wherein the first type of training images is captured using a microscope according to a first illumination scheme.

[0441] 36. A method according to clause 35, wherein the first lighting scheme comprises one or more lighting modes.

[0442] 37. A method according to any of items 35-36, wherein the first type of training images is part of a bright field image array.

[0443] 38. A method according to any one of clauses 35 to 37, wherein the plurality of training images is a first plurality of training images, the method further comprising:

[0444] Based on the comparing, identifying a second lighting scheme;

[0445] obtaining a second plurality of training images including one or more images of the first type, wherein the one or more images of the first type are obtained based on the second lighting scheme;

[0446] The machine learning model is trained based on the second plurality of training images.

[0447] 39. The method according to any one of items 35-38, further comprising:

[0448] obtaining a plurality of images of the first type using a microscope; and

[0449] Based on the obtained multiple images, the second type of multiple synthetic images are generated using the machine learning model.

[0450] 40. The method of claim 39, further comprising: training a classifier based on a plurality of synthetic images of the second type.

[0451] 41. The method of clause 40, wherein the microscope is a first microscope, wherein the classifier is a first classifier, the method further comprising:

[0452] obtaining a plurality of images of said second type using a second microscope;

[0453] training a second classifier based on the plurality of images of the second type;

[0454] The performance of the first classifier and the second classifier are compared.

[0455] 42. A system according to claim 41, wherein the second microscope is a fluorescence microscope.

[0456] 43. A system for training a machine learning model to generate an image of a biological sample, the system comprising:

[0457] A computing system comprising one or more processors and one or more memories storing a machine learning model, wherein the computing system is configured to receive a plurality of training images of a first type and one training image of a second type, and wherein the computing system is configured to:

[0458] Based on the first type of training images, generating a plurality of wavelet coefficients using the machine learning model;

[0459] generating a composite image of the second type based on the plurality of wavelet coefficients;

[0460] comparing the second type of synthesized image to training images of the second type; and

[0461] The machine learning model is updated based on the comparison.

[0462] 44. A system according to claim 43, wherein the first type of training images are bright field images of biological samples.

[0463] 45. A system according to items 43-44, wherein the second type of training images are fluorescent images of the biological sample.

[0464] 46. ​​A system according to any of items 43-45, wherein the machine learning model includes a generator and a discriminator.

[0465] 47. A system according to claim 46, wherein the machine learning model includes a conditional GAN ​​model.

[0466] 48. A method according to any one of items 46-47, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.

[0467] 49. A system according to item 48, wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for a corresponding frequency group.

[0468] 50. A system according to any of items 48-49, wherein the multiple neural networks include multiple U-Net neural networks.

[0469] 51. A system according to any of items 46-50, wherein the discriminator is a PatchGAN neural network.

[0470] 52. A system according to any one of items 43-51, wherein the computing system is further configured to: generate a third type of image based on the first type of training images.

[0471] 53. A system according to item 52, wherein the third type of image is a phase-shifted image.

[0472] 54. A system according to any of items 43-53, wherein the computing system is further configured to: generate a fourth type of image based on the first type of training images.

[0473] 55. A system according to item 54, wherein the fourth type of image includes segmentation data.

[0474] 56. A system according to any of items 43-55, wherein the first type of training images is captured using a microscope according to a first illumination scheme.

[0475] 57. A system according to item 56, wherein the first lighting scheme includes one or more lighting modes.

[0476] 58. A system according to any of items 56-57, wherein the first type of training image is part of a bright field image array.

[0477] 59. A system according to any one of clauses 56-58, wherein the plurality of training images is a first plurality of training images, and wherein the computing system is further configured to:

[0478] Based on the comparing, identifying a second lighting scheme;

[0479] obtaining a second plurality of training images including one or more images of the first type, wherein the one or more images of the first type are obtained based on the second lighting scheme;

[0480] The machine learning model is trained based on the second plurality of training images.

[0481] 60. The system of any one of clauses 43-59, wherein the computing system is further configured to:

[0482] obtaining a plurality of images of the first type using a microscope; and

[0483] Based on the obtained multiple images, the second type of multiple synthetic images are generated using the machine learning model.

[0484] 61. A system according to item 59, wherein the computing system is further configured to: train a classifier based on multiple synthetic images of the second type.

[0485] 62. The system of clause 61, wherein the microscope is a first microscope, wherein the classifier is a first classifier, wherein the computing system is further configured to:

[0486] obtaining a plurality of images of said second type using a second microscope;

[0487] training a second classifier based on the plurality of images of the second type;

[0488] The performance of the first classifier and the second classifier are compared.

[0489] 63. A system according to claim 62, wherein the second microscope is a fluorescence microscope.

[0490] 64. A system for generating an enhanced image of a biological sample, the system comprising:

[0491] A computing system comprising one or more processors and one or more memories storing a machine learning model, wherein the computing system is configured to receive an image of a biological sample obtained from a microscope and generate an enhanced image of the biological sample based on the image using the machine learning model, wherein the machine learning model has been trained by:

[0492] A plurality of training images are obtained, wherein the plurality of training images include:

[0493] training images of the first type, and

[0494] The second type of training images;

[0495] Based on the first type of training images, generating a plurality of wavelet coefficients using the machine learning model;

[0496] generating a composite image of the second type based on the plurality of wavelet coefficients;

[0497] comparing the second type of synthesized image to training images of the second type; and

[0498] The machine learning model is updated based on the comparison.

[0499] 65. A system according to claim 64, wherein the first type of training images are bright field images of biological samples.

[0500] 66. A system according to item 64, wherein the second type of training images are fluorescent images of the biological sample.

[0501] 67. A system according to any one of items 64-66, wherein the machine learning model includes a generator and a discriminator.

[0502] 68. A system according to item 67, wherein the machine learning model includes a conditional GAN ​​model.

[0503] 69. A method according to any one of items 67-68, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.

[0504] 70. A system according to item 69, wherein each neural network of the plurality of neural networks is configured to generate wavelet coefficients for a corresponding frequency group.

[0505] 71. A system according to any of items 69-70, wherein the multiple neural networks include multiple U-Net neural networks.

[0506] 72. A system according to any of items 67-71, wherein the discriminator is a PatchGAN neural network.

[0507] 73. A system according to any one of items 64-72, wherein the machine learning model is further trained by generating a third type of image based on the first type of training images.

[0508] 74. A system according to item 73, wherein the third type of image is a phase-shifted image.

[0509] 75. A system according to any of items 64-74, wherein the machine learning model has been trained by generating a fourth type of image based on the first type of training images.

[0510] 76. A system according to item 75, wherein the fourth type of image includes segmentation data.

[0511] 77. A system according to any of items 64-76, wherein the first type of training images is captured using a microscope according to a first illumination scheme.

[0512] 78. A system according to item 77, wherein the first lighting scheme includes one or more lighting modes.

[0513] 79. A system according to any of items 77-78, wherein the first type of training image is part of a bright field image array.

[0514] 80. The system of any one of clauses 77-79, wherein the plurality of training images is a first plurality of training images, wherein the machine learning model has been trained by:

[0515] Based on the comparing, identifying a second lighting scheme;

[0516] obtaining a second plurality of training images including one or more images of the first type, wherein the one or more images of the first type are obtained based on the second lighting scheme;

[0517] The machine learning model is trained based on the second plurality of training images.

[0518] 81. A system according to any one of clauses 77-80, wherein the machine learning model has been trained by:

[0519] obtaining a plurality of images of the first type using a microscope; and

[0520] Based on the obtained multiple images, the second type of multiple synthetic images are generated using the machine learning model.

[0521] 82. A system according to item 81, wherein the machine learning model has been trained by training a classifier based on multiple synthetic images of the second type.

[0522] 83. The system of clause 82, wherein the microscope is a first microscope, wherein the classifier is a first classifier, and wherein the machine learning model has been trained by:

[0523] obtaining a plurality of images of said second type using a second microscope;

[0524] training a second classifier based on the plurality of images of the second type;

[0525] The performance of the first classifier and the second classifier are compared.

[0526] 84. A system according to claim 83, wherein the second microscope is a fluorescence microscope.

[0527] 85. A method of processing an image of a biological sample to obtain one or more output images, the method comprising:

[0528] obtaining a plurality of images of the biological sample using a plurality of configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and

[0529] The plurality of images of the biological sample are input into a trained machine learning model to obtain the one or more output images.

[0530] 86. A method according to item 85, wherein at least one of the multiple configurations of the SLM is used to generate one or more optical aberrations.

[0531] 87. A method according to item 86, wherein generating one or more optical aberrations includes spherical aberration, astigmatism, defocus, distortion, tilt, or any combination thereof.

[0532] 88. A method according to any of items 85-86, wherein at least one of the multiple configurations of the SLM is used to enhance one or more features.

[0533] 89. A method according to claim 88, wherein the one or more features include cell boundaries, actin filaments, nuclear shape, cytoplasmic segmentation, or any combination thereof.

[0534] 90. A method according to any one of items 85-89, wherein at least one of the multiple configurations of the SLM is configured to reduce optical aberrations.

[0535] 91. A method according to any one of items 85-90, wherein the multiple SLMs are configured to obtain images of the biological sample at different depths.

[0536] 92. A method according to any one of items 85-91, wherein the machine learning model is configured to generate a second type of image based on a first type of image.

[0537] 93. A method according to item 92, wherein the first type of image is a bright field image.

[0538] 94. A method according to item 92, wherein the second type of image is a fluorescent image.

[0539] 95. A method according to item 92, wherein the second type of image is an enhanced version of the first type of image.

[0540] 96. A method according to any one of items 92-95, wherein the machine learning model is a GAN model or a self-supervised model.

[0541] 97. A method according to any one of items 85-96, wherein the multiple images are obtained using multiple configurations of the light source of the optical system.

[0542] 98. A method according to item 97, wherein the light source is an LED array of the optical system.

[0543] 99. A method according to any one of clauses 85 to 98, wherein at least one of the plurality of SLM configurations is obtained by:

[0544] training the machine learning model;

[0545] Evaluate trained machine learning models; and

[0546] The at least one configuration is identified based on the evaluating.

[0547] 100. A method according to any one of items 85-99, wherein the trained machine learning model is configured to receive an input image and output an enhanced version of the input image.

[0548] 101. A method according to item 100, wherein the enhanced version of the input image includes one or more enhanced cellular phenotypes.

[0549] 102. An electronic device for processing an image of a biological sample to obtain one or more output images, the electronic device comprising:

[0550] one or more processors;

[0551] Memory; and

[0552] one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for:

[0553] obtaining a plurality of images of the biological sample using a plurality of configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and

[0554] The plurality of images of the biological sample are input into a trained machine learning model to obtain the one or more output images.

[0555] 103. A non-transitory computer-readable storage medium storing one or more programs for processing an image of a biological sample to obtain one or more output images, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to:

[0556] obtaining a plurality of images of the biological sample using a plurality of configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and

[0557] The plurality of images of the biological sample are input into a trained machine learning model to obtain the one or more output images.

[0558] 104. A method for classifying an image of a biological sample, the method comprising:

[0559] obtaining a plurality of images of the biological sample using a plurality of configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and

[0560] The plurality of images of the biological sample are input into a trained machine learning model to obtain one or more classification outputs.

[0561] 105. A method according to item 104, wherein at least one of the multiple configurations of the SLM is used to generate one or more optical aberrations.

[0562] 106. A method according to item 105, wherein generating one or more optical aberrations includes spherical aberration, astigmatism, defocus, distortion, tilt, or any combination thereof.

[0563] 107. A method according to any of items 104-106, wherein at least one of the multiple configurations of the SLM is used to enhance one or more features.

[0564] 108. A method according to claim 107, wherein the one or more features include cell boundaries, actin filaments, nuclear shape, cytoplasmic segmentation, or any combination thereof.

[0565] 109. A method according to any one of items 104-108, wherein at least one of the multiple configurations of the SLM is configured to reduce optical aberrations.

[0566] 110. A method according to any one of items 104-109, wherein the multiple SLMs are configured to obtain images of the biological sample at different depths.

[0567] 111. A method according to any one of items 104-110, wherein the multiple images are obtained using multiple configurations of the light source of the optical system.

[0568] 112. A method according to item 111, wherein the light source is an LED array of the optical system.

[0569] 113. A method according to any one of clauses 104 to 112, wherein at least one of the plurality of SLM configurations is obtained by:

[0570] training the machine learning model;

[0571] Evaluate trained machine learning models; and

[0572] The at least one configuration is identified based on the evaluating.

[0573] 114. A method according to any one of items 104-113, wherein the trained machine learning model is configured to receive an input image and detect one or more predefined objects in the input image.

[0574] 115. A method according to item 114, wherein the predefined object includes diseased tissue.

[0575] 116. An electronic device for classifying an image of a biological sample, the electronic device comprising:

[0576] one or more processors;

[0577] Memory; and

[0578] one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for:

[0579] obtaining a plurality of images of the biological sample using a plurality of configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and

[0580] The plurality of images of the biological sample are input into a trained machine learning model to obtain one or more classification outputs.

[0581] 117. A non-transitory computer-readable storage medium storing one or more programs for classifying an image of a biological sample, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to:

[0582] obtaining a plurality of images of the biological sample using a plurality of configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and

[0583] The plurality of images of the biological sample are input into a trained machine learning model to obtain one or more classification outputs.

[0584] 118. A method for training a machine learning model, the method comprising:

[0585] obtaining a plurality of images of a biological sample using a plurality of configurations of an SLM of the optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and

[0586] The machine learning model is trained using the plurality of images.

[0587] 119. A method according to item 118, wherein at least one of the multiple configurations of the SLM is used to generate one or more optical aberrations.

[0588] 120. A method according to item 119, wherein generating one or more optical aberrations includes spherical aberration, astigmatism, defocus, distortion, tilt, or any combination thereof.

[0589] 121. A method according to any of items 118-120, wherein at least one of the multiple configurations of the SLM is used to enhance one or more features.

[0590] 122. A method according to claim 121, wherein the one or more features include cell boundaries, actin filaments, nuclear shape, cytoplasmic segmentation, or any combination thereof.

[0591] 123. A method according to any one of items 118-122, wherein at least one of the multiple configurations of the SLM is configured to reduce optical aberrations.

[0592] 124. A method according to any one of items 118-123, wherein at least one of the multiple configurations of the SLM is configured to obtain images of the biological sample at different depths.

[0593] 125. A method according to any one of items 118-124, wherein the machine learning model is configured to generate a second type of image based on a first type of image.

[0594] 126. A method according to item 125, wherein the first type of image is a bright field image.

[0595] 127. A method according to item 125, wherein the second type of image is a fluorescent image.

[0596] 128. A method according to any one of items 118-127, wherein the machine learning model is a GAN model or a self-supervised model.

[0597] 129. A method according to any one of items 118-128, wherein the machine learning model is a classification model.

[0598] 130. A method according to any one of items 118-129, wherein the multiple images are obtained using multiple configurations of the light source of the optical system.

[0599] 131. A method according to item 130, wherein the light source is an LED array of the optical system.

[0600] 132. The method of any one of clauses 118-131, wherein training the machine learning model comprises:

[0601] (a) training the machine learning model using a first image, wherein the first image is obtained using a first configuration of the SLM of the optical system;

[0602] (b) Evaluate trained machine learning models;

[0603] (c) based on the evaluation, identifying a second configuration of the SLM; and

[0604] (d) training the machine learning model using a second image, wherein the second image is obtained using a second configuration of the SLM of the optical system.

[0605] 133. A method according to item 112, wherein the evaluation is based on a loss function of the machine learning model.

[0606] 134. According to any one of items 112-113, the method further comprises: repeating steps (a) to (d) until a threshold value is reached.

[0607] 135. A method according to item 114, wherein the threshold indicates convergence of training.

[0608] 136. A method according to any one of items 118-135, wherein the trained machine learning model is configured to receive an input image and output an enhanced version of the input image.

[0609] 137. A method according to item 136, wherein the enhanced version of the input image includes one or more enhanced cellular phenotypes.

[0610] 138. A method according to any one of items 118-135, wherein the trained machine learning model is configured to receive an input image and detect one or more predefined objects in the input image.

[0611] 139. A method according to claim 138, wherein the predefined object includes diseased tissue.

[0612] 140. An electronic device for training a machine learning model, the electronic device comprising:

[0613] one or more processors;

[0614] Memory; and

[0615] one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for:

[0616] obtaining a plurality of images of a biological sample using a plurality of configurations of an SLM of the optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and

[0617] The machine learning model is trained using the plurality of images.

[0618] 141. A non-transitory computer-readable storage medium storing one or more programs for training a machine learning model, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to:

[0619] Obtaining multiple images of the biological sample using multiple configurations of an SLM of an optical system, wherein the SLM is located in an optical path between the biological sample and an image recording device; and training the machine learning model using the multiple images.

[0620] 142. A method for generating an enhanced image of a biological sample, comprising:

[0621] Acquire, using a microscope, an image of a biological sample illuminated using an illumination pattern of an illumination source, wherein the illumination pattern is determined by the following steps:

[0622] training a classification model configured to receive an input image and output a classification result,

[0623] using the trained classification model to train a machine learning model having a plurality of weights corresponding to a plurality of lighting settings, and

[0624] identifying the lighting pattern based on a plurality of weights of a trained machine learning model; and

[0625] By inputting the obtained image of the biological sample into the trained machine learning model, an enhanced image of the biological sample is generated.

[0626] 143. A method according to item 142, wherein the image obtained is a bright field image.

[0627] 144. A method according to any one of items 142-143, wherein the enhanced image is a fluorescence image, a phase image, or a combination thereof.

[0628] 145. A method according to any one of items 142-144, wherein the illumination source comprises an array of illumination emitters.

[0629] 146. A method according to item 145, wherein the illumination source is an LED array.

[0630] 147. A method according to any of items 142-146, wherein the lighting mode indicates whether each lighting emitter is on or off and the intensity of each lighting emitter.

[0631] 148. A method according to any of items 145-147, wherein each of the multiple lighting settings corresponds to a corresponding lighting emitter of the lighting source; and wherein each weight corresponds to an intensity of the corresponding lighting emitter.

[0632] 149. A method according to any one of items 142-148, wherein the classification model is configured to receive an input phase image or an input fluorescence image and output a classification result indicating one of a plurality of predefined categories.

[0633] 150. A method according to item 149, wherein the plurality of predefined categories include a healthy category and a diseased category.

[0634] 151. A method according to item 150, wherein the machine learning model is a GAN model, the GAN model includes an attention layer, a discriminator and a generator, and the attention layer includes the multiple weights.

[0635] 152. A method according to item 151, wherein the machine learning model includes a conditional GAN ​​model.

[0636] 153. A method according to any one of items 151-152, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.

[0637] 154. A method according to item 153, wherein each of the plurality of neural networks is configured to generate wavelet coefficients for a corresponding frequency group.

[0638] 155. A method according to any one of items 153-154, wherein the multiple neural networks include multiple U-Net neural networks.

[0639] 156. A method according to any one of items 151-155, wherein the discriminator is a PatchGAN neural network.

[0640] 157. The method of any one of clauses 151-156, wherein training the machine learning model using a trained classification model comprises:

[0641] applying the plurality of weights to a plurality of bright field training images;

[0642] aggregating the plurality of weighted bright field training images into an aggregated bright field image;

[0643] inputting the aggregated bright field training images into the machine learning model to obtain an enhanced training image and a generator loss;

[0644] Inputting the enhanced training image into a trained classifier to obtain a classifier loss;

[0645] increasing the generator loss based on the classifier loss; and

[0646] The plurality of weights are updated based on the increased generator loss.

[0647] 158. According to any one of items 142-157, the method further comprises: classifying the enhanced image using the trained classifier.

[0648] 159. According to the method described in any one of items 142-158, the method further comprises: displaying the enhanced image.

[0649] 160. A system for generating an enhanced image of a biological sample, the system comprising:

[0650] one or more processors;

[0651] Memory; and

[0652] one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for:

[0653] Acquire, using a microscope, an image of a biological sample illuminated using an illumination pattern of an illumination source, wherein the illumination pattern is determined by the following steps:

[0654] training a classification model configured to receive an input image and output a classification result,

[0655] using the trained classification model to train a machine learning model having a plurality of weights corresponding to a plurality of lighting settings, and

[0656] identifying the lighting pattern based on a plurality of weights of a trained machine learning model; and

[0657] By inputting the obtained image of the biological sample into the trained machine learning model, an enhanced image of the biological sample is generated.

[0658] 161. A non-transitory computer-readable storage medium storing one or more programs for generating an enhanced image of a biological sample, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to:

[0659] Acquire, using a microscope, an image of a biological sample illuminated using an illumination pattern of an illumination source, wherein the illumination pattern is determined by the following steps:

[0660] training a classification model configured to receive an input image and output a classification result,

[0661] using the trained classification model to train a machine learning model having a plurality of weights corresponding to a plurality of lighting settings, and

[0662] identifying the lighting pattern based on a plurality of weights of a trained machine learning model; and

[0663] By inputting the obtained image of the biological sample into the trained machine learning model, an enhanced image of the biological sample is generated.

[0664] 162. A method of evaluating a treatment for a disease of interest, the method comprising:

[0665] receiving a first plurality of images depicting a first set of healthy biological samples not affected by the disease of interest;

[0666] receiving a second plurality of images depicting a second set of untreated biological samples affected by the disease of interest;

[0667] receiving a third plurality of images depicting a third set of treated biological samples affected by the disease of interest and treated by the therapy;

[0668] inputting the first plurality of images into a trained machine learning model to obtain a first plurality of enhanced images;

[0669] inputting the second plurality of images into the trained machine learning model to obtain a second plurality of enhanced images;

[0670] Inputting the third plurality of images into the trained machine learning model to obtain a third plurality of enhanced images;

[0671] The first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images are compared to evaluate the treatment.

[0672] 163. A method according to item 162, wherein the first plurality of images, the second plurality of images and the third plurality of images are bright field images.

[0673] 164. A method according to any one of items 162-163, wherein the first plurality of enhanced images, the second plurality of enhanced images and the third plurality of enhanced images are fluorescence images.

[0674] 165. A method according to any one of items 162-163, wherein the first plurality of enhanced images, the second plurality of enhanced images and the third plurality of enhanced images are phase images.

[0675] 166. A method according to any one of items 162-165, wherein comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment includes: identifying a signal associated with a biomarker in each image.

[0676] 167. The method of clause 166, wherein comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment further comprises:

[0677] determining a first distribution based on the signals of the biomarkers in the first plurality of enhanced images;

[0678] determining a second distribution based on the signals of the biomarkers in the second plurality of enhanced images; and

[0679] A third distribution is determined based on the signals of the biomarkers in the third plurality of enhanced images.

[0680] 168. The method of clause 167, wherein comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment further comprises:

[0681] The first distribution, the second distribution, and the third distribution are compared to evaluate the treatment.

[0682] 169. A method according to any one of items 162-165, wherein comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment includes: determining a score for each image indicating an assessment opinion of the disease of interest.

[0683] 170. The method of clause 169, wherein comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment further comprises:

[0684] determining a first distribution based on the scores of the first plurality of enhanced images;

[0685] determining a second distribution based on the scores of the second plurality of enhanced images; and

[0686] A third distribution is determined based on the scores of the third plurality of enhanced images.

[0687] 171. The method of clause 170, wherein comparing the first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images to evaluate the treatment further comprises:

[0688] The first distribution, the second distribution, and the third distribution are compared to evaluate the treatment.

[0689] 172. The method according to any one of items 162-171, wherein the treatment is a first treatment, the method further comprising:

[0690] receiving a fourth plurality of images depicting a fourth set of treated biological samples affected by the disease of interest and treated with a second therapy;

[0691] Inputting the fourth plurality of images into the trained machine learning model to obtain a fourth plurality of enhanced images;

[0692] The first plurality of enhanced images, the second plurality of enhanced images, the third plurality of enhanced images, and the fourth plurality of enhanced images are compared to compare the first treatment and the second treatment.

[0693] 173. According to the method of item 172, the method further comprises: selecting a treatment from the first treatment and the second treatment based on comparison.

[0694] 174. The method of claim 173, further comprising: administering the selected treatment.

[0695] 175. The method of claim 173, further comprising: providing medical advice based on the selected treatment.

[0696] 176. A method according to any one of items 162-175, wherein the trained machine learning model is a GAN model including a discriminator and a generator.

[0697] 177. A method according to item 176, wherein the machine learning model is a conditional GAN ​​model.

[0698] 178. A method according to any of items 176-177, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.

[0699] 179. A method according to item 178, wherein each of the multiple neural networks is configured to generate wavelet coefficients for a corresponding frequency group.

[0700] 180. A method according to any one of items 176-179, wherein the discriminator is a PatchGAN neural network.

[0701] 181. A method of evaluating a treatment for a disease of interest, the method comprising:

[0702] one or more processors;

[0703] Memory; and

[0704] one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for:

[0705] receiving a first plurality of images depicting a first set of healthy biological samples not affected by the disease of interest;

[0706] receiving a second plurality of images depicting a second set of untreated biological samples affected by the disease of interest;

[0707] receiving a third plurality of images depicting a third set of treated biological samples affected by the disease of interest and treated by the therapy;

[0708] inputting the first plurality of images into a trained machine learning model to obtain a first plurality of enhanced images;

[0709] inputting the second plurality of images into the trained machine learning model to obtain a second plurality of enhanced images;

[0710] Inputting the third plurality of images into the trained machine learning model to obtain a third plurality of enhanced images;

[0711] The first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images are compared to evaluate the treatment.

[0712] 182. A non-transitory computer-readable storage medium storing one or more programs for evaluating treatment for a disease of interest, the one or more programs comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to:

[0713] receiving a first plurality of images depicting a first set of healthy biological samples not affected by the disease of interest;

[0714] receiving a second plurality of images depicting a second set of untreated biological samples affected by the disease of interest;

[0715] receiving a third plurality of images depicting a third set of treated biological samples affected by the disease of interest and treated by the therapy;

[0716] inputting the first plurality of images into a trained machine learning model to obtain a first plurality of enhanced images;

[0717] inputting the second plurality of images into the trained machine learning model to obtain a second plurality of enhanced images;

[0718] Inputting the third plurality of images into the trained machine learning model to obtain a third plurality of enhanced images;

[0719] The first plurality of enhanced images, the second plurality of enhanced images, and the third plurality of enhanced images are compared to evaluate the treatment.

[0720] Although the present disclosure and examples have been fully described with reference to the accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications should be understood to be included within the scope of the present disclosure and examples defined by the claims.

[0721] For purposes of explanation, the foregoing description has been described with reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise forms disclosed. In view of the above teachings, many modifications and variations are possible. The embodiments are selected and described in order to best explain the principles of the technology and its practical application. Therefore, others skilled in the art can best utilize the technology and various embodiments with various modifications suitable for the specific use intended.

Claims

1. A method for generating an enhanced image of a biological sample, the method comprising: Obtain images of biological samples using a microscope; as well as Based on the image, an enhanced image of the biological sample is generated using a machine learning model, wherein the machine learning model has been trained by: A plurality of training images are obtained, wherein the plurality of training images include: training images of the first type, and The second type of training images; Based on the first type of training images, generating a plurality of wavelet coefficients using the machine learning model; generating a composite image of the second type based on the plurality of wavelet coefficients; comparing the second type of synthesized image to training images of the second type; and The machine learning model is updated based on the comparison. 2 . The method of claim 1 , wherein the first type of training images are bright field images of biological samples.

3. The method according to claim 2, further comprising: Images of a third type are generated based on the training images of the first type. The method of claim 3 , wherein the third type of image is a phase-shifted image.

5. The method according to claim 2, further comprising: Images of a fourth type are generated based on the training images of the first type. The method of claim 5 , wherein the fourth type of image comprises segmentation data. The method of claim 2 , wherein the first type of training images is captured using a microscope according to a first illumination scheme. The method of claim 7 , wherein the first lighting scheme comprises one or more lighting modes.

9. The method of claim 7, wherein the first type of training images is part of an array of bright field images.

10. The method of claim 7, wherein the plurality of training images is a first plurality of training images, the method further comprising: based on the comparing, identifying a second lighting scheme; obtaining a second plurality of training images including one or more images of the first type, wherein the one or more images of the first type are obtained based on the second lighting scheme; The machine learning model is trained based on the second plurality of training images.

11. The method according to claim 7, further comprising: obtaining a plurality of images of the first type using a microscope; as well as Based on the obtained multiple images, the second type of multiple synthetic images are generated using the machine learning model.

12. The method according to claim 11, further comprising: A classifier is trained based on the plurality of synthetic images of the second type.

13. The method of claim 12, wherein the microscope is a first microscope, wherein the classifier is a first classifier, the method further comprising: obtaining a plurality of images of said second type using a second microscope; training a second classifier based on the plurality of images of the second type; The performance of the first classifier and the second classifier are compared. The method of claim 13 , wherein the second microscope is a fluorescence microscope.

15. The method of claim 1, wherein the second type of training images are fluorescent images of the biological sample.

16. The method of claim 1, wherein the machine learning model comprises a generator and a discriminator.

17. The method of claim 16, wherein the machine learning model comprises a conditional GAN ​​model.

18. The method of claim 17, wherein the discriminator is a PatchGAN neural network.

19. The method of claim 16, wherein the generator comprises a plurality of neural networks corresponding to a plurality of frequency groups.

20. The method of claim 19, wherein each of the plurality of neural networks is configured to generate wavelet coefficients for a corresponding frequency group.

21. The method of claim 19, wherein the plurality of neural networks comprises a plurality of U-Net neural networks.

22. The method according to claim 1, wherein: The image of the biological sample is illuminated using an illumination pattern of an illumination source, wherein the illumination pattern is determined by: training a classification model configured to receive an input image and output a classification result, using the trained classification model to train the machine learning model having a plurality of weights corresponding to a plurality of lighting settings, and The lighting pattern is identified based on a plurality of weights of a trained machine learning model.

23. The method of claim 22, wherein the illumination source comprises an array of illumination emitters.

24. The method of claim 23, wherein the illumination source is an LED array.

25. The method of claim 23, wherein each lighting setting of the plurality of lighting settings corresponds to a respective lighting emitter of the lighting source; and wherein each weight corresponds to an intensity of the respective lighting emitter.

26. The method of claim 22, wherein the lighting mode indicates whether each lighting emitter is on or off and the intensity of each lighting emitter.

27. The method of claim 22, wherein the classification model is configured to receive an input phase image or an input fluorescence image and output a classification result indicating one of a plurality of predefined categories.

28. A method of training a machine learning model to generate an image of a biological sample, the method comprising: A plurality of training images are obtained, wherein the plurality of training images include: a first type of microscopic training image, and The second type of microscopic training images; Based on the first type of microscopic training image, generating a plurality of wavelet coefficients using the machine learning model; generating a composite image of the second type based on the plurality of wavelet coefficients; comparing the second type of synthetic image to the second type of microscopy training image; and The machine learning model is updated based on the comparison.

Images