Multi-model prediction of stained cellular structures

Abstract

One embodiment of the present invention sets forth a technique for training machine learning models to convert label-free images into stained cell images. The technique includes executing a first machine learning model to convert a first set of label-free images into a first set of training output images and training the first machine learning model using a first loss computed between a first set of stained cell images and the first set of training output images to generate a first trained machine learning model. The technique also includes executing the first trained machine learning model to convert a second set of label-free images into a second set of training output images and retraining the first trained machine learning model using a second loss associated with a second set of stained cell images and the second set of training output images to generate a second trained machine learning model.

Description

MULTI-MODEL PREDICTION OF STAINED CELLULAR STRUCTURESCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority benefit of United States Provisional Patent Application titled “MULTI-MODEL PREDICTION OF STAINED CELLULAR STRUCTURES,” serial number 63 / 625,728, filed January 26, 2024. The subject matter of this related application is hereby incorporated herein by reference.BACKGROUNDField of the Various Embodiments

[0002] Embodiments of the present disclosure relate generally to machine learning and computer vision and, more specifically, to multi-model prediction of stained cellular structures.Description of the Related Art

[0003] Optical microscopy refers to the use of microscopes with various light modalities to perform generate images of cells. One of the simplest forms of optical microscopy includes bright field microscopy, which involves imaging a biological sample through a microscope while the biological sample is uniformly illuminated (e.g., using white light). However, label-free microscopy typically generates images with low contrast and information content, as relatively few biological samples absorb light well. Consequently, it can be difficult to distinguish and / or separate specific structures or molecules within a label-free microscopy image.

[0004] To improve the level of contrast and detail in biological samples that are imaged using optical microscopy, various cell staining techniques are used to manually apply stains and / or dyes to biological samples before the biological samples are imaged under a microscope. These stains and / or dyes are preferentially absorbed by certain components and / or structures within the biological samples, thereby enhancing visualization of these components and / or structures under the microscope. For example, specific organelle staining can be used to enhance the observation, identification, and study of the structure and behavior of cells, organelles, and / or other types of shapes and / or structures within the cells. Cell staining can thus be used for applications that range from determining cell types, health, morphology,and / or functionality to identifying the presence and / or accumulation of specific molecules and / or pathogens within the cells.

[0005] However, cell staining is associated with a number of drawbacks. First, staining and imaging a biological sample tends to be a manual and time-consuming process. For example, conventional cell staining techniques can involve multiple procedures to prepare a biological sample for imaging. These procedures can include (but are not limited to) fixing a biological sample using heat and / or chemicals to preserve the shape and / or structure of the underlying cells and / or tissue, immersing the biological sample in a solution of a chosen dye or stain to allow the stain to bind to specific cellular components, washing the biological sample to remove any excess stain, dehydrating the biological sample, embedding the biological sample in paraffin and / or a resin, and / or cutting the biological sample into sections.

[0006] Second, most light microscopes are associated with a small number of imaging channels, which limits the number of variables that can be measured simultaneously within a given biological sample. For example, a typical image-based assay could have a maximum of six stains that can be applied simultaneously across five imaging channels. Given these limitations, a conventional optical microscopy approach would be unable to capture additional morphological information from any unstained structures and / or components. Further, because at least some of the imaging channels are typically used with generic fluorescent stains to image standard cellular components, the conventional optical microscopy approach would also be limited in the ability to use specific imaging channels for experiments that involve nongeneric fluorescent stains and / or perturbations of cells or molecules within the cells.

[0007] More recently, stained microscopy images have been paired with computer vision and deep-learning-based image processing techniques with the aim of extracting quantitative repeatable features related to cell state. These techniques require a higher image quality than prior techniques that involve visual inspection of stained cell images by the scientists. For example, a reduction in image quality of a stained microscopy image can translate to errors in quantitative imaging biomarkers extractions of nuclear shape and / or other attributes.

[0008] As the foregoing illustrates, what is needed in the art are more effective techniques for identifying biological structures and / or components using optical microscopy.SUMMARY

[0009] One embodiment of the present invention sets forth a technique for training one or more machine learning models to convert label-free images into stained cell images. The technique includes executing a first machine learning model to convert a first set of label-free images into a first set of training output images and training the first machine learning model using a first loss computed between a first set of stained cell images corresponding to the first set of label-free images and the first set of training output images to generate a first trained machine learning model. The technique also includes executing the first trained machine learning model to convert a second set of label-free images into a second set of training output images and retraining the first trained machine learning model using a second loss associated with a second set of stained cell images and the second set of training output images to generate a second trained machine learning model.

[0010] One technical advantage of the disclosed techniques relative to the prior art is the ability to convert label-free images of biological samples into stained cell images that accurately depict various quantitative attributes of the same biological samples. In this regard, the disclosed techniques can be used to simulate and / or impute the quantitative attributes of the biological samples within the stained cell images instead of requiring the biological samples to be manually stained and subsequently imaged. Additionally, because the generated stained cell images can be used to determine and / or measure the quantitative attributes within the biological samples without staining the biological samples, the disclosed techniques allow available imaging channels and / or stains to be used to determine and / or control additional attributes of the biological samples. Accordingly, the disclosed techniques provide a more complete view of a biological sample relative to conventional optical microscopy techniques that are limited in the numbers of stains and / or imaging channels that can be used with a given biological sample. These technical advantages provide one or more technological improvements over prior art approaches.BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0012] So that the manner in which the above recited features of the various embodiments can be understood in detail, a more particular description of the inventive concepts, briefly summarized above, may be had by reference to various embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of the inventive concepts and are therefore not to be considered limiting of scope in any way, and that there are other equally effective embodiments.

[0013] Figure 1 illustrates a system configured to implement one or more aspects of various embodiments.

[0014] Figure 2 is a more detailed illustration of the training engine and execution engine of Figure 1 , according to various embodiments.

[0015] Figure 3 illustrates an example set of training label-free images, training stained cell images, and virtual stained cell images associated with the machine learning models of Figure 2, according to various embodiments.

[0016] Figure 4 is a flow diagram of method steps for training a set of machine learning models to convert label-free images of biological samples into stained cell images of the same biological samples, according to various embodiments.

[0017] Figure 5 is a flow diagram of method steps for determining visual attributes of cellular structures depicted in a label-free image, according to various embodiments.DETAILED DESCRIPTION

[0018] In the following description, numerous specific details are set forth to provide a more thorough understanding of the various embodiments. However, it will be apparent to one of skill in the art that the inventive concepts may be practiced without one or more of these specific details.

[0019] As discussed above, staining and imaging a biological sample tends to be a manual and time-consuming process. Additionally, most optical microscopes are associated with a small number (e.q., 2-5) of imaging channels, which limits the number of variables that can be measured simultaneously within a given biological sample.

[0020] To reduce overhead associated with cell staining and / or understanding of various structures and / or components within a biological sample, the disclosed techniques train and execute multiple machine learning models to convert label-free image of biological samples into stained cell images of the same biological samples. A first machine learning model is initially trained to generate predictions of stained cell images of biological samples, given input that includes label-free images of the same biological samples. For example, the first machine learning model could be pretrained using a reconstruction loss that is computed between the stained cell images and output images generated by the first machine learning model from the corresponding label-free images. After initial training of the model is complete, the first machine learning model is subsequently retrained and / or fine-tuned using different losses and / or additional pairs of label-free images and stained cell images. This retraining and / or fine-tuning is used to generate one or more additional trained machine learning models that are optimized for various attributes of structures of components depicted within the images. For example, the pretrained first machine learning model could be retrained using a segmentation-based loss, structural similarity loss, and / or other types of losses to generate additional fine-tuned machine learning models that are capable of accurately reconstructing shapes, textures, sizes, numbers, distributions, and / or other visual attributes of organelles and / or other types of structures within cells depicted in the stained cell images, given input that includes the corresponding label-free images.

[0021] After training of the machine learning models is complete, one or more trained machine learning models can be used to convert additional label-free images into corresponding stained cell images. For example, one or more types of attributes to be determined from a biological sample could be matched to one or more trained machine learning models that are fine-tuned and / or optimized for those types of attributes. One or more label-free images of the biological sample could be inputted into the trained machine learning model(s), and the trained machine learning model(s)could be executed to convert the inputted label-free image(s) into one or more corresponding stained cell images. Quantitative analyses of the generated stained cell image(s) could then be performed to measure and / or characterize these attributes for the structures and / or components depicted in the generated stained cell image(s).System Overview

[0022] Figure 1 is a block diagram illustrating a computer system 100 configured to implement one or more aspects of various embodiments. In one embodiment, computer system 100 includes a desktop computer, a laptop computer, a smart phone, a personal digital assistant (PDA), tablet computer, or any other type of computing device configured to receive input, process data, and optionally display images, and is suitable for practicing one or more embodiments. Computer system 100 also, or instead, includes a machine or processing node operating in a data center, cluster, or cloud computing environment that provides scalable computing resources (optionally as a service) over a network.

[0023] As shown, computer system 100 includes, without limitation, a central processing unit (CPU) 102 and a system memory 104 coupled to a parallel processing subsystem 112 via a memory bridge 105 and a communication path 113. Memory bridge 105 is further coupled to an I / O (input / output) bridge 107 via a communication path 106, and I / O bridge 107 is, in turn, coupled to a switch 116.

[0024] I / O bridge 107 is configured to receive user input information from optional input devices 108, such as a keyboard or a mouse, and forward the input information to CPU 102 for processing via communication path 106 and memory bridge 105. In some embodiments, computer system 100 may be a server machine in a cloud computing environment. In such embodiments, computer system 100 may not have input devices 108. Instead, computer system 100 may receive equivalent input information by receiving commands in the form of messages transmitted over a network and received via the network adapter 118. In one embodiment, switch 116 is configured to provide connections between I / O bridge 107 and other components of the computer system 100, such as a network adapter 118 and various add-in cards 120 and 121.

[0025] In one embodiment, I / O bridge 107 is coupled to a system disk 114 that may be configured to store content and applications and data for use by CPU 102 andparallel processing subsystem 112. In one embodiment, system disk 114 provides non-volatile storage for applications and data and may include fixed or removable hard disk drives, flash memory devices, and CD-ROM (compact disc read-only- memory), DVD-ROM (digital versatile disc-ROM), Blu-ray, HD-DVD (high definition DVD), or other magnetic, optical, or solid state storage devices. In various embodiments, other components, such as universal serial bus or other port connections, compact disc drives, digital versatile disc drives, film recording devices, and the like, may be connected to I / O bridge 107 as well.

[0026] In various embodiments, memory bridge 105 may be a Northbridge chip, and I / O bridge 107 may be a Southbridge chip. In addition, communication paths 106 and 113, as well as other communication paths within computer system 100, may be implemented using any technically suitable protocols, including, without limitation, AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol known in the art.

[0027] In some embodiments, parallel processing subsystem 112 includes a graphics subsystem that delivers pixels to an optional display device 110 that may be any conventional cathode ray tube, liquid crystal display, light-emitting diode display, or the like. In such embodiments, the parallel processing subsystem 112 incorporates circuitry optimized for graphics and video processing, including, for example, video output circuitry. Such circuitry may be incorporated across one or more parallel processing units (PPUs), also referred to herein as parallel processors, included within parallel processing subsystem 112. In other embodiments, the parallel processing subsystem 112 incorporates circuitry optimized for general purpose and / or compute processing. Again, such circuitry may be incorporated across one or more PPUs included within parallel processing subsystem 112 that are configured to perform such general purpose and / or compute operations. In yet other embodiments, the one or more PPUs included within parallel processing subsystem 112 may be configured to perform graphics processing, general purpose processing, and compute processing operations. System memory 104 includes at least one device driver configured to manage the processing operations of the one or more PPUs within parallel processing subsystem 112.

[0028] Parallel processing subsystem 112 may be integrated with one or more of the other elements of Figure 1 to form a single system. For example, parallelprocessing subsystem 112 may be integrated with CPU 102 and other connection circuitry on a single chip to form a system on chip (SoC).

[0029] In one embodiment, CPU 102 is the master processor of computer system 100, controlling and coordinating operations of other system components. In one embodiment, CPU 102 issues commands that control the operation of PPUs. In some embodiments, communication path 113 is a PCI Express link, in which dedicated lanes are allocated to each PPU, as is known in the art. Other communication paths may also be used. PPU advantageously implements a highly parallel processing architecture. A PPU may be provided with any amount of local parallel processing memory (PP memory).

[0030] It will be appreciated that the system shown herein is illustrative and that variations and modifications are possible. First, the functionality of the system can be distributed across multiple nodes of a distributed and / or cloud computing system. Second, the connection topology, including the number and arrangement of bridges, the number of CPUs 102, and the number of parallel processing subsystems 112, can be modified as desired. For example, in some embodiments, system memory 104 could be connected to CPU 102 directly rather than through memory bridge 105, and other devices would communicate with system memory 104 via memory bridge 105 and CPU 102. In another example, parallel processing subsystem 112 may be connected to I / O bridge 107 or directly to CPU 102, rather than to memory bridge 105. In a third example, I / O bridge 107 and memory bridge 105 may be integrated into a single chip instead of existing as one or more discrete devices. Third one or more components shown in Figure 1 may not be present. For example, switch 116 could be eliminated, and network adapter 118 and add-in cards 120, 121 would connect directly to I / O bridge 107.

[0031] In one or more embodiments, computer system 100 is configured to execute a training engine 122 and an execution engine 124 that reside in system memory 104. Training engine 122 and execution engine 124 may be stored in system disk 114 and / or other storage and loaded into system memory 104 when executed.

[0032] More specifically, training engine 122 and execution engine 124 include functionality to train and execute multiple machine learning models to convert label-free images of biological samples into stained cell images of the same biological samples. In a label-free image of a biological sample, one or more features of the biological sample being visualized is typically not labeled or altered in any way. Training engine 122 pretrains a first machine learning model to generate predictions of stained cell images of biological samples, given input that includes label-free images of the same biological samples. For example, the first machine learning model could be pretrained using a reconstruction loss that is computed between the stained cell images and output images generated by the first machine learning model from the corresponding label-free images. After pretraining of the first machine learning model is complete, training engine 122 performs retraining and / or fine-tuning of the first machine learning model using different losses and / or additional pairs of label-free images and stained cell images to generate one or more additional trained machine learning models that are optimized for various attributes of structures and / or components depicted within the images. For example, the pretrained first machine learning model could be retrained using a shape-aware loss, structural similarity loss, and / or other types of losses to generate additional fine-tuned machine learning models that are capable of accurately reconstructing cells, shapes, textures, sizes, numbers, distributions, and / or other visual features of organelles and / or other types of cellular or subcellular structures depicted in the stained cell images, given input that includes the corresponding label-free images.

[0033] After training of the machine learning models is complete, execution engine 124 can use one or more trained machine learning models to convert additional label- free images into corresponding stained cell images. For example, execution engine 124 could match one or more types of attributes to be determined from a biological sample to one or more trained machine learning models that are optimized for those types of attributes. Execution engine 124 could input one or more label-free images of the biological sample into the trained machine learning model(s) and execute the trained machine learning model(s) to convert the inputted label-free image(s) into one or more corresponding stained cell images. After the stained cell image(s) are generated, execution engine 124 could perform quantitative analyses of the generated stained cell image(s) to measure and / or characterize these attributes for structures and / or components depicted in the generated stained cell image(s). The operation of training engine 122 and execution engine 124 is described in further detail below.Multi-Model Prediction of Stained Cellular Structures

[0034] Figure 2 is a more detailed illustration of training engine 122 and execution engine 124 of Figure 1 , according to various embodiments. As mentioned above, training engine 122 and execution engine 124 operate to train and execute multiple machine learning models 224 to convert a label-free image 222 of a biological sample into one or more stained cell images 230 of the same biological sample.

[0035] In some embodiments, label-free image 222 includes an image of a biological sample that is captured using an imaging technology, such as (but not limited to) optical microscopy, brightfield microscopy, quantitative label-free imaging with phase and polarization (QLIPP), fluorescence imaging, autofluorescence imaging, phase-contrast microscopy, and / or differential interference contrast microscopy. In some embodiments, the biological sample is a cell sample and / or tissue sample, such as (but not limited to) cell cultures, tissue cultures, organoid culture, multicellular organism culture (e.q., cultures of C elegans or Drosophila). A cell culture refers to a bioprocess whereby live cells are maintained in an artificial environment such as a cell culture dish. A cell culture dish may include a plate, flask, and / or any other type of container that is compatible with the acquisition of images of a cell culture. For example, the biological sample may include eukaryotic (e.q., mammalian) or prokaryotic cells. In some examples, the cells may include human, mouse, and / or rat cells. In general, any human or non-human animal may be a source of cells or tissues that can be analyzed according to the disclosure. In some embodiments, the cell may be a skin, muscle, blood, liver, heart, spleen, thymus, brain, myoblast, fibroblast, and / or keratinocyte cell. The cells may be living or may be dead (e.q., fixed cells). In general, label-free image 222 could depict cells, tissues, organelles, and / or other structures, substructures, and / or components within the biological sample while these structures or components that are illuminated and magnified using one or more lenses within the optical microscope.

[0036] Stained cell images 230 can include one or more images that depict the biological sample in the presence of one or more stains and / or dyes that are preferentially absorbed by certain components and / or structures within the biological sample. For example, stained cell images 230 could depict cells, tissues, organelles, and / or other structures and / or components within the biological sample as thesestructures and / or components would appear after one or more staining techniques have been applied to the biological sample.

[0037] In one or more embodiments, machine learning models 224 include fully convolutional neural networks with encoder-decoder architectures that are capable of processing inputs of varying resolution. For example, one or more machine learning models 224 could include a ll-Net model that performs image translation via an encoder and a decoder. The encoder could include convolutional layers and max pooling layers that generate downsampled feature maps from a given input image (e.q., label-free image 222) and / or one or more subsets of that input image. The decoder could use upsampling, concatenation, and transpose convolution operations to generate upsampled feature maps from the downsampled feature maps. The ll- Net model could also include skip connections that connect various layers of the encoder with corresponding layers of decoder associated with the same feature map resolution.

[0038] In another example, one or more machine learning models 224 could include a V-Net architecture. Like the ll-Net, the V-Net also includes an encoder that generates downsampled feature maps from a given input image (e.q., label-free image 222), a decoder that generates upsampled feature maps from the downsampled feature maps, and skip connections that connect various layers of the encoder with corresponding layers of decoder associated with the same feature map resolution. The encoder of the V-Net uses strided convolutions to perform learned downsampling of the feature maps, and the decoder of the V-Net uses deconvolution operations to perform upsampling of the feature maps. The convolution and / or deconvolution operations performed by the V-Net model can reduce memory overhead when compared with a standard ll-Net model. These deconvolution operations can include dilated convolutions in the upsampling path of the V-Net. In various embodiments, each resolution at which the V-Net operates includes a residual function that performs nonlinear processing of the input into that resolution and adds the result to the output of the last layer associated with that resolution.

[0039] Training engine 122 trains machine learning models 224 using training data 202 that includes a set of training label-free images 240 and a corresponding set of training stained cell images 242. Each of training label-free images 240 depicts a label-free image of a biological sample that is captured using label-free microscopy.Each of training label-free images 240 is also paired with a stained cell image that is included in training stained cell images 242 and depicts the same biological sample as the training label-free image. For example, training label-free images 240 could include images of tissues, cells, cellular structures, and / or other types of biological specimens that are captured using imaging technology and without staining the biological specimens. Training stained cell images 242 could include images of the same tissues, cells, cellular structures, subcellular structures, and / or biological specimens as those depicted in training label-free images 240. However, unlike training label-free images 240, training stained cell images 242 could be generated by applying specific stains and / or dyes to the biological specimens depicted in training label-free images 240 and subsequently using brightfield microscopy, quantitative label-free imaging with phase and polarization (QLIPP), fluorescence imaging, autofluorescence imaging, phase-contrast microscopy, differential interference contrast microscopy, and / or other techniques to image the stained and / or dyed biological specimens.

[0040] When training label-free images 240 are acquired in three dimensions, n consecutive slices in the z direction can be concatenated channel-wise and provided to machine learning models 224 as input. The corresponding output can include one channel corresponding to the virtual staining of the central slice of the input z-stack. This two-and-a-half dimensional (2.5D) approach reduces the memory footprint of machine learning models 224, which allows machine learning models 224 to have deeper architectures and higher parameter spaces that leverage more spatial information in training label-free images 240.

[0041] In one or more embodiments, at least some of training stained cell images 242 include images of specifically stained organelles and / or other structures or components within the corresponding biological samples. For example, training stained cell images 242 could include multiple groups of images. Each group of images could include a visualization of a different cellular component. These cellular components could include (but are not limited to) nuclei, endoplasmic reticula, nucleoli, cytoplasmic ribonucleic acid (RNA), actin, Golgi apparatuses, plasma membranes, mitochondria, and / or other types of organelles and / or sub-cellular structures. Each group of images could further serve as a set of labels to bepredicted by one or more machine learning models 224, given input that includes corresponding training label-free images 240 of the same biological samples.

[0042] A data-generation component 204 in training engine 122 generates image crops 254 of training label-free images 240 and training stained cell images 242. For example, data-generation component 204 could generate fixed- and / or variable-sized image crops 254 of training label-free images 240. Data-generation component 204 could also generate the same image crops 254 of the corresponding training stained cell images 242, so that a given training label-free image crop is paired with a corresponding training stained cell image crop and both image crops depict the same region of the same biological sample. A given image crop could include a randomized position, height, and / or width. A given image crop could also, or instead, include a position, height, and / or width that is selected so that the image crop captures one or more objects (e.g., a specific organelle or cellular structure) within the corresponding training label-free image and / or training stained cell image.

[0043] Data-generation component 204 also generates image segmentations 252 of some or all training stained cell images 242 and / or the corresponding image crops 254. For example, data-generation component 204 could input training stained cell images 242 and / or the corresponding image crops 254 into a pre-trained segmentation model. Data-generation component 204 could also execute the segmentation model to convert each inputted image and / or image crop into a corresponding image segmentation of the same size as the inputted image and / or image crop. Within the image segmentation, a pixel at a given location could be assigned a label that indicates the type of object (e.g., a specific type of organelle and / or cellular structure) found at the location. When no specific object is detected at a given location, the corresponding pixel could be assigned to a “background” or “unknown” label.

[0044] An update component 206 in training engine 122 uses training data 202, image crops 254, and / or image segmentations 252 to train machine learning models 224 over a number of training stages 212 and 214. During a first training stage 212, update component 206 inputs some or all training label-free images 240 and / or image crops 254 of these training label-free images 240 into a first machine learning model included in machine learning models 224. For each inputted training label-free image (or inputted training label-free image crop), update component 206 executes the firstmachine learning model to generate a corresponding training output 210. This training output 210 includes a prediction of a stained cell image (or stained cell image crop) that depicts the same objects as the inputted training label-free image (or training label-free image crop). Update component 206 also computes one or more losses 216 between training output 210 and a training stained cell image (or training stained cell image crop) that is paired with the inputted training label-free image (or inputted training label-free image crop). Update component 206 additionally uses a training technique (e.q., gradient descent and backpropagation) to update base model parameters 208 of the first machine learning model in a way that reduces losses 216. Update component 206 can repeat this process until losses 216 fall below a threshold, convergence is reached by base model parameters 208, and / or another condition is met.

[0045] In one or more embodiments, losses 216 used to train the first machine learning model include a reconstruction loss between training output 210 generated by the first machine learning model from training label-free images 240 and / or image crops 254 of training label-free images 240 and the corresponding training stained cell images 242 and / or images crops 254 of training stained cell images 242. For example, losses 216 could include a mean squared error (MSE), mean absolute error (MAE), and / or another measure of error that is computed between pixel values in training output 210 and the corresponding “ground truth” training stained cell images 242 and / or images crops 254. This reconstruction loss allows the first machine learning model to learn a general and / or overall mapping between objects in training label-free images 240 (or training label-free image crops) and the corresponding training stained cell images 242 (or training stained cell image crops).

[0046] After the first training stage 212 is complete, update component 206 performs one or more additional training stages 214 to train additional machine learning models 224, using the first machine learning model trained in training stage 212 as a starting point. During each of training stages 214, update component 206 inputs some or all training label-free images 240 and / or image crops 254 of these training label-free images 240 into an additional machine learning model included in machine learning models 224. For each inputted training label-free image (or inputted training label-free image crop), update component 206 executes the additional machine learning model to generate a corresponding training output 220. Thistraining output 220 includes a prediction of a stained cell image (or stained cell image crop) that depicts the same objects as the inputted training label-free image (or inputted training label-free image crop). Update component 206 also computes one or more losses 226 associated with training output 220 and a training stained cell image (or training stained cell image crop) that is paired with the inputted training label-free cell image (or inputted training label-free image crop). Update component 206 additionally uses a training technique (e.q., gradient descent and backpropagation) to update fine-tuned model parameters 218 of the additional machine learning model in a way that reduces losses 226. Update component 206 can repeat this process until losses 226 fall below a threshold, convergence is reached by fine-tuned model parameters 218, and / or another condition is met.

[0047] In some embodiments, training stages 214 include a training stage that optimizes and / or fine-tunes one or more machine learning models 224 to accurately generate shapes of objects in training stained cell images 242. During this training stage, update component 206 updates fine-tuned model parameters 218 of the machine learning model(s) using one or more losses 226 that are computed between a first set of image segmentations 252 of training output 220 (e.q., as generated by data-generation component 204) and a second set of image segmentations 252 of the corresponding “ground truth” training stained cell images 242 (or training stained cell image crops 254). For example, losses 226 could include a cross-entropy loss, weighted cross-entropy loss, Dice loss, and / or another measure of error between a first image segmentation of a predicted stained cell image (or stained cell image crop) generated by a machine learning model from an inputted training label-free image (or training label-free image crop) and a second image segmentation of a training stained cell image (or training stained cell image crop) that is paired with the inputted training label-free image (or training label-free image crop). This segmentation-based loss can be used to increase the accuracy with which the machine learning model predicts shapes in training stained cell images 242 (or training stained cell image crops 254), given input that includes the corresponding training label-free images 240 (or training label-free image crops 254).

[0048] Training stages 214 also, or instead, include a training stage that optimizes and / or fine-tunes one or more machine learning models 224 to accurately generate textures of objects in training stained cell images 242. During this training stage,update component 206 updates fine-tuned model parameters 218 of the machine learning model(s) using one or more losses 226 associated with a structural similarity between training output 220 and the corresponding “ground truth” training stained cell images 242 (or training stained cell image crops 254). For example, losses 226 could include a structural similarity index measure (SSIM), a perceptual loss that is computed between a first set of features generated by a pre-trained feature extractor from training output 220 and a second set of features generated by the pre-trained feature extractor from the corresponding ground truth images or image crops, and / or another measure of similarity between a prediction of a stained cell image (or stained cell image crop) generated by the machine learning model from an inputted training label-free image (or training label-free image crop) and a training stained cell image (or training stained cell image crop) that is paired with the inputted training label-free image (or training label-free image crop). This similarity-based loss can be used to increase the accuracy with which the machine learning model predicts textures in training stained cell images 242 (or training stained cell image crops 254), given input that includes the corresponding training label-free images 240 (or training label-free image crops 254).

[0049] While the operation of training engine 122 has been described with respect to training stages 214 that optimize and / or fine-tune machine learning models 224 to accurately generate shapes and / or textures associated with objects depicted in training stained cell images 242 (or training stained cell image crops 254), it will be appreciated that training stages 212 and / or 214 and the corresponding losses 216 and / or 226 can be adapted to optimize and / or fine-tune the corresponding machine learning models 224 for various types of tasks and / or output. For example, training engine 122 could perform one or more training stages 212 and / or 214 that train one or more machine learning models 224 using a combination of reconstruction, similarity, segmentation, and / or other types of losses 216 and / or 226. This combination could include a linear combination, weighted combination, and / or another combination of multiple losses 216 and / or 226 that is computed and used to update parameters (e.g., base model parameters 208, fine-tuned model parameters 218, etc.) of the corresponding machine learning model(s) over a number of training steps, iterations, batches, and / or epochs. This combination could also, or instead, involve training the machine learning model(s) over multiple training stages 212 and / or 214 using a series of different losses 216 and / or 226, alternating between or among twoor more losses 216 and / or 226 during training of the machine learning model(s), and / or updating different subsets of parameters of the machine learning model(s) using different losses 216 and / or 226.

[0050] In another example, training engine 122 could perform one or more training stages 212 and / or 214 that train one or more machine learning models 224 using one or more losses 216 and / or 226 that optimize for the generation of various attributes of cells, organelles, and / or sub-cellular structures in training stained cell images 242 (or training stained cell image crops 254), given input that includes the corresponding training label-free images 240 (or training label-free image crops 254). These losses 216 and / or 226 could include an adversarial loss that is computed based on additional output and / or intermediate activations generated by one or more discriminator models from training output 210 and / or 220. These losses 216 and / or 226 could also, or instead, include a loss that is computed based on the count of a certain type of object (e.g., lysosome, ribosome, mitochondria, etc.) within training stained cell images 242 (or training stained cell image crops 254). These losses 216 and / or 226 could also, or instead, include a loss that is computed based on the distribution of sizes of a certain type of object within training stained cell images 242 (or training stained cell image crops 254). These losses 216 and / or 226 could also, or instead, include one or more losses that are computed based on the distribution, orientation, and / or shape of a certain type of object within training stained cell images 242 (or training stained cell image crops 254).

[0051] In a third example, training engine 122 could perform one or more initial training stages (e.g., training stage 212) that pretrain a machine learning model to convert inputted training label-free images 240 (or training label-free image crops 254) of a certain type of object (e.g., cell, organelle, sub-cellular structure, protein, etc.) into corresponding training stained cell images 242 (or training stained cell image crops 254). Training engine 122 could then perform one or more additional training stages (e.g., training stages 214) that fine-tune the machine learning model to convert inputted training label-free images 240 (or training label-free image crops 254) of a different type of object (e.g., cell, organelle, sub-cellular structure, protein, etc.) into corresponding training stained cell images 242 (or training stained cell image crops 254). The initial training stage(s) could be performed on pairs of training label-free images 240 (or training label-free image crops 254) and training stained cellimages 242 (or training stained cell image crops 254) of objects that are relatively easy to image, such as relatively abundant cell types and / or cell lines. The additional training stage(s) could be performed on pairs of training label-free images 240 (or training label-free image crops 254) and training stained cell images 242 (or training stained cell image crops 254) of objects that are relatively more difficult to image, such as rare cell types and / or cell lines.

[0052] In a fourth example, training engine 122 could perform one or more initial training stages (e.g., training stage 212) that pretrain a machine learning model to convert inputted training label-free images 240 (or training label-free image crops 254) into training stained cell images 242 (or training stained cell image crops 254) that are captured using a certain type of cell staining and / or fluorescence microscopy technique. Training engine 122 could then perform one or more additional training stages (e.g., training stages 214) that fine-tune the machine learning model to convert inputted training label-free images 240 (or training label-free image crops 254) into training stained cell images 242 (or training stained cell image crops 254) that are captured using a different type of cell staining and / or fluorescence microscopy technique.

[0053] After training of machine learning models 224 is complete, execution engine 124 executes one or more trained machine learning models 224 to convert a given label-free image 222 into one or more stained cell images 230. As shown in Figure 2, execution engine 124 selects one or more machine learning models 224 to be used in converting label-free image 222 into one or more stained cell images 230 based on attribute types 228 of attributes to be determined from the generated stained cell image(s). For example, execution engine 124 could select one or more machine learning models 224 that have been trained to accurately reconstruct the shape, texture, count, distribution, orientation, and / or other visual attributes of objects (e.g., cells, organelles, sub-cellular structures, etc.) depicted in label-free image 222. In another example, execution engine 124 could select one or more machine learning models 224 that have been trained and / or fine-tuned to convert between training label-free images 240 and training stained cell images 242 of a certain cell type and / or cell line that is depicted in label-free image 222.

[0054] Next, execution engine 124 inputs label-free image 222 into the selected machine learning model(s) and uses each machine learning model to convert label-free image 222 into a corresponding stained cell image. For example, execution engine 124 could input the entire label-free image 222 into a given machine learning model and execute the machine learning model to generate a corresponding stained cell image. Execution engine 124 could also, or instead, extract one or more image crops from label-free image 222 and input each label-free image 222 crop into the machine learning model. Execution engine 124 could use the machine learning model to convert each inputted label-free image 222 crop into a corresponding stained cell image crop.

[0055] In some embodiments, execution engine 124 combines multiple stained cell image crops generated by the selected machine learning model(s) from multiple inputted label-free image 222 crops into a corresponding stained cell image. For example, execution engine 124 could use a layout of label-free image 222 crops within label-free image 222 to form a corresponding layout of stained cell image crops within the stained cell image. Execution engine 124 could also use the layout of stained cell image crops to stitch and / or otherwise combine the stained cell image crops into the stained cell image. Alternatively, execution engine 124 can use individual stained cell image crops outputted by the machine learning model as discrete predictions of objects depicted within these stained cell image crops without combining the stained cell image crops into the stained cell image.

[0056] In some embodiments, execution engine 124 uses one or more machine learning models 224 to generate multiple sets of stained cell image crops within a given stained cell image from multiple sets of label-free image 222 crops of different sizes, dimensions, or locations within the same label-free image 222. For example, execution engine 124 could divide label-free image 222 into X different sets of label- free image 222 crops, where each set of label-free image 222 crops includes multiple image crops of the same size and different sets of label-free image 222 crops are associated with different crop sizes. Execution engine 124 could also use a given machine learning model included in machine learning models 224 to convert each set of label-free image 222 crops into a corresponding set of stained cell image crops.

[0057] Execution engine 124 can further combine different levels or types of detail associated with multiple stained cell images 230 and / or multiple sets of stained cell image crops into one or more stained cell images 230. For example, execution engine 124 could generate a given stained cell image as an average and / or anotheraggregation of multiple sets of stained cell image crops generated by the same machine learning model and / or different machine learning models 224 from corresponding sets of label-free image 222 crops. In another example, execution engine 124 could generate a separate stained cell image from each set of stained cell image crops produced by a given machine learning model from a corresponding set of label-free image 222 crops. Each stained cell image could capture a different level and / or type of detail associated with objects in label-free image 222 and could be used to perform a different task (e.g., edge detection, shape detection, object detection, instance detection, texture evaluation, etc.) associated with the objects.

[0058] After label-free image 222 is converted into one or more stained cell images 230, execution engine 124 analyzes label-free image 222 and / or stained cell images 230 to determine attributes 232 of cells, organelles, sub-cellular structures, and / or other objects depicted in label-free image 222 and / or stained cell images 230. For example, execution engine 124 could use additional machine learning models, imageprocessing techniques, and / or other techniques to measure one or more quantitative attributes 232 of the objects in label-free image 222 and / or stained cell images 230. These quantitative attributes 232 could include (but are not limited to) shapes, sizes, textures, counts, size distributions, spatial distributions, and / or other visual attributes 232 of the objects. Execution engine 124 could provide the measured attributes to downstream components that perform further analyses of label-free image 222, stained cell images 230, and / or objects depicted in label-free image 222 and / or stained cell images 230. Execution engine 124 could also, or instead, use additional machine learning models and / or analyses to generate predictions of labels (e.g., phenotypes, diseases, health, age, effects of perturbations, toxicity, etc.) associated with label-free image 222, stained cell images 230, and / or objects depicted in label- free image 222 and / or stained cell images 230.

[0059] Figure 3 illustrates an example set of training label-free images 302 and 304, training stained cell images 312 and 314, and virtual stained cell images 322 and 324 associated with machine learning models 224 of Figure 2, according to various embodiments. Training label-free images 302 and 304 include label-free images of different biological samples. For example, training label-free images 302 could include images of different tissues, cells, cellular structures, and / or other types of biological specimens.

[0060] Training label-free image 302 is paired with a corresponding training stained cell image 312, and training label-free image 304 is paired with a different corresponding training stained cell image 314. For example, training stained cell images 312 and 314 could include images of the same tissues, cells, cellular structures, and / or biological specimens as those depicted in training label-free images 302 and 304, respectively. Unlike training label-free images 302 and 304, training stained cell images 312 and 314 can be generated by applying specific stains and / or dyes to the biological specimens depicted in training label-free images 302 and 304 and using fluorescence microscopy, epifluorescence microscopy, and / or other techniques to image the stained and / or dyed biological specimens. As shown in Figure 3, training stained cell images 312 and 314 include a magenta-colored background with similar objects as those of the corresponding training label-free images 302 and 304. Each of training stained cell images 312 and 314 also includes a bright green foreground object depicting a cellular structure (e.q., nucleus, organelle, etc.) that has been stained with a fluorescent dye.

[0061] Each training label-free image 302 and 304 can be inputted into one or more machine learning models 224, and the machine learning model(s) can be trained to generate a corresponding virtual stained cell image 322 and 324, respectively, that represents a reconstruction of the training stained cell image 312 and 314 with which the training label-free image 302 and 304 is paired. More specifically, the machine learning model(s) can be trained over a number of training steps, iterations, batches, and / or epochs to generate each virtual stained cell image 322 and 324 based on input that includes a corresponding training label-free image 302 and 304, respectively. Each virtual stained cell image 322 and 324 includes a magenta background with objects that are similar to those depicted in the corresponding training stained cell image 312 and 314, respectively. Each virtual stained cell image 322 and 324 also includes a bright green foreground object that is similar in shape and size to the foreground object in the corresponding training stained cell image 312 and 314, respectively.

[0062] As discussed above, the machine learning model(s) can be trained and / or fine-tuned so that various quantitative attributes of the foreground objects in training stained cell images 312 and 314 are accurately depicted in the corresponding virtual stained cell images 322 and 324. For example, a given machine learning modelcould be trained and / or fine-tuned to reconstruct shapes, textures, shapes, sizes, and / or other attributes of the foreground objects in training stained cell images 312 and 314 based on input that includes the corresponding training label-free images 302 and 304.

[0063] Figure 4 is a flow diagram of method steps for training a set of machine learning models to convert label-free images of biological samples into stained cell images of the same biological samples, according to various embodiments. Although the method steps are described in conjunction with the systems of Figures 1-2, persons skilled in the art will understand that any system configured to perform some or all of the method steps in any order falls within the scope of the present disclosure.

[0064] As shown, in step 402, training engine 122 generates image crops of a set of label-free images and a corresponding set of stained cell images. For example, training engine 122 could receive a set of label-free images that are captured using label-free microscopy and depict one or more cell types, cell lines, organelles, sub- cellular structures, tissues, and / or other types of biological samples. Each label-free image of a biological sample could be paired with one or more stained cell images that depict the same biological sample using one or more cell staining and / or fluorescence microscopy techniques. Training engine 122 could extract, from a given label-free image, one or more crops that include a subset of pixels from the label-free image. Each crop could include a predefined and / or randomized location, size, and / or dimensions and could depict a subset of the objects found in the label-free images. For each crop extracted from the label-free image, training engine 122 could also extract, from one or more corresponding stained cell images, one or more crops with the same location, size, and / or dimensions and / or one or more crops that depict the same subset of objects as the label-free image.

[0065] In step 404, training engine 122 executes a machine learning model to convert at least a first portion of the label-free images and / or label-free image crops into training output. For example, training engine 122 could select the first portion of the label-free images and / or label-free image crops as depicting certain types of objects (e.q., cell types, cell lines, organelles, sub-cellular structures, tissues, etc.) within the corresponding biological samples. Training engine 122 could also, or instead, select the first portion of the label-free images and / or label-free image crops as being paired with a certain type of stained cell image and / or stained cell imagecrop. Training engine 122 could input each label-free image and / or label-free image crop into a ll-Net, V-Net, and / or another type of deep neural network corresponding to the machine learning model. Training engine 122 could execute the deep neural network to generate downsampled and / or upsampled feature maps from the input. Training engine 122 could also use the deep neural network to convert some or all of the feature maps into training output that includes a prediction of a stained cell image (or stained cell image crop) corresponding to the inputted label-free image (or label- free image crop).

[0066] In step 406, training engine 122 trains the machine learning model using a first set of losses computed between the training output and a corresponding portion of stained cell images and / or stained cell image crops. For example, training engine 122 could compute an MSE and / or another measure of error between each training output generated by the machine learning model from an inputted label-free image (or label-free image crop) and a stained cell image (or stained cell image crop) that is paired with the input. Training engine 122 could aggregate the error across multiple training outputs and use gradient descent and backpropagation to update weights in the machine learning model in a way that reduces the error. Training engine 122 could continue training the machine learning model until one or more conditions are met. These condition(s) include (but are not limited to) convergence in the parameters of the machine learning models; the lowering of the loss(es) to below a threshold; or a certain number of training steps, iterations, batches, and / or epochs.

[0067] In step 408, training engine 122 executes the trained machine learning model to convert at least an additional portion of the label-free images and / or label- free image crops into training output. For example, training engine 122 could select the additional portion of label-free images and / or label-free image crops as label-free images and / or label-free image crops that depict the same types of objects (e.g., cell types, cell lines, organelles, sub-cellular structures, tissues, etc.) as the initial portion used to train the machine learning model in steps 404 and 406. Training engine 122 could also, or instead, select the additional portion of label-free images and / or label- free image crops as label-free images and / or label-free image crops that depict different types of objects from the initial portion used to train the machine learning model in steps 404 and 406. After the additional portion is selected and / or determined, training engine 122 inputs each label-free image and / or label-free imagecrop in the additional portion into a ll-Net, V-Net, and / or another type of deep neural network corresponding to the machine learning model that was trained in step 406. Training engine 122 could execute the deep neural network to generate downsampled and / or upsampled feature maps from the input. Training engine 122 could also use the deep neural network to convert some or all of the feature maps into training output that includes a prediction of a stained cell image (or stained cell image crop) corresponding to the inputted label-free image (or label-free image crop).

[0068] In step 410, training engine 122 retrains the trained machine learning model using an additional set of losses computed between the training output and the corresponding stained cell images and / or stained cell image crops. For example, training engine 122 could select one or more loss functions used to retrain the trained machine learning model based on a task and / or type of output with which the trained machine learning model is to be optimized. If the task and / or type of output involves accurately reconstructing shapes in the stained cell images and / or stained cell image crops, training engine 122 could select a cross entropy loss, weighted cross entropy loss, Dice loss, and / or another measure of classification error between the training output and the corresponding stained cell images and / or stained cell image crops. If the task and / or type of output involves accurately reconstructing textures in the stained cell images and / or stained cell image crops, training engine 122 could select an SSIM loss, feature-based loss, and / or another measure of similarity between the training output and the corresponding stained cell images and / or stained cell image crops.

[0069] Continuing with the above example, training engine 122 could compute the selected loss(es) using each training output generated by the machine learning model from an inputted label-free image (or label-free image crop) and a stained cell image (or stained cell image crop) that is paired with the input. Training engine 122 could aggregate the computed loss(es) across multiple training outputs and use gradient descent and backpropagation to update weights in the machine learning model in a way that reduces the error. Training engine 122 could continue retraining the trained machine learning model until one or more conditions are met. These condition(s) include (but are not limited to) convergence in the parameters of the machine learning models; the lowering of the loss(es) to below a threshold; or a certain number of training steps, iterations, batches, and / or epochs. After retraining of the trainedmachine learning model is complete, training engine 122 could save the final weights from the retrained machine learning model as a new trained machine learning model that is optimized for the corresponding task and / or type of output.

[0070] In step 412, training engine 122 determines whether or not to continue retraining the trained machine learning model. For example, training engine 122 could determine that the trained machine learning model should continue to be retrained while the trained machine learning model is to be optimized and / or finetuned to perform additional tasks and / or generate additional types of output. If training engine 122 determines in step 412 that retraining is to continue, training engine 122 repeats steps 408 and 410 to generate a different trained machine learning model for each task and / or type of output with which the trained machine learning model is to be optimized and / or fine-tuned. Training engine 122 ends the process of retraining the trained machine learning model after the trained machine learning model has been optimized and / or fine-tuned for all identified tasks and / or types of output.

[0071] Figure 5 is a flow diagram of method steps for determining visual attributes of cellular structures depicted in a label-free image, according to various embodiments. Although the method steps are described in conjunction with the systems of Figures 1-2, persons skilled in the art will understand that any system configured to perform some or all of the method steps in any order falls within the scope of the present disclosure.

[0072] As shown, in step 502, execution engine 124 matches one or more attributes to be measured from one or more objects depicted in a label-free image to a machine learning model. For example, execution engine 124 could match one or more object types (e.g., cell lines, cell types, organelle types, sub-cellular structure types, tissue types, etc.), visual attributes (e.g., shapes, textures, counts, sizes, distributions, etc.) of the object(s), and / or other attributes associated with the object(s) to a machine learning model that has been trained, fine-tuned, and / or optimized to generate these attributes in a stained cell image, given a corresponding label-free image.

[0073] In step 504, execution engine 124 executes the machine learning model to convert the label-free image into an output image corresponding to a prediction of astained cell image depicting the object(s). For example, execution engine 124 could use a ll-Net, V-Net, and / or another type of fully convolutional neural network to convert the label-free image into the output image.

[0074] In step 506, execution engine 124 extracts the attribute(s) from the output image. For example, execution engine 124 could use additional machine learning models, image-processing techniques, and / or other techniques to quantitatively measure the attribute(s) from the output image. Execution engine 124 could also perform additional analyses of the measured attribute(s) to evaluate phenotypes, diseases, health, age, effects of perturbations, toxicity, and / or other characteristics associated with the object(s) depicted in the label-free image.

[0075] In sum, the disclosed techniques train and execute multiple machine learning models to convert label-free images of biological samples into stained cell images of the same biological samples. A first machine learning model is initially trained to generate predictions of stained cell images of biological samples, given input that includes label-free images of the same biological samples. For example, the first machine learning model could be pretrained using a reconstruction loss that is computed between the stained cell images and output images generated by the first machine learning model from the corresponding label-free images. After initial training of the model is complete, the first machine learning model is subsequently retrained and / or fine-tuned using different losses and / or additional pairs of label-free images and stained cell images. This retraining and / or fine-tuning is used to generate one or more additional trained machine learning models that are optimized for various attributes of structures and / or components depicted within the images. For example, the pretrained first machine learning model could be retrained using a segmentationbased loss, structural similarity loss, and / or other types of losses to generate additional fine-tuned machine learning models that are capable of accurately reconstructing shapes, textures, sizes, numbers, distributions, and / or other visual attributes of organelles and / or other types of structures within cells depicted in the stained cell images, given input that includes the corresponding label-free images.

[0076] After training of the machine learning models is complete, one or more trained machine learning models can be used to convert additional label-free images into corresponding stained cell images. For example, one or more types of attributes to be determined from a biological sample could be matched to one or more trainedmachine learning models that are fine-tuned and / or optimized for those types of attributes. One or more label-free images of the biological sample could be inputted into the trained machine learning model(s), and the trained machine learning model(s) could be executed to convert the inputted label-free image(s) into one or more corresponding stained cell images. Quantitative analyses of the generated stained cell image(s) could then be performed to measure and / or characterize these attributes for the structures and / or components depicted in the generated stained cell image(s).

[0077] Fully automatic single cell and lineage tracking on live transmitted light microscopy pose significant challenges especially in longer experiments (days or weeks). However the task of nuclear detection and tracking is trivial using images acquired with DNA staining. Virtual staining is an effective and fully computational method to use models trained on the simplest scenario (stained cell) to solve the tracking problems on time lapse acquired exclusively with transmitted light microscopy. This solution, as it is fully computational, is the less invasive following longer and more complex experiments which are essential to observe biological processes that evolve over the course of weeks as cellular reprogramming. Similarly, virtual staining can be used to define landmarks for multimodal registration.

[0078] One technical advantage of the disclosed techniques relative to the prior art is the ability to convert label-free images of biological samples into stained cell images that accurately depict various quantitative attributes of the same biological samples. In this regard, the disclosed techniques can be used to simulate and / or impute the quantitative attributes of the biological samples within the stained cell images instead of requiring the biological samples to be manually stained and subsequently imaged. Additionally, because the generated stained cell images can be used to determine and / or measure the quantitative attributes within the biological samples without staining the biological samples, the disclosed techniques allow available imaging channels and / or stains to be used to determine and / or control additional attributes of the biological samples. Accordingly, the disclosed techniques provide a more complete view of a biological sample relative to conventional optical microscopy techniques that are limited in the numbers of stains and / or imaging channels that can be used with a given biological sample. These technical advantages provide one or more technological improvements over prior art approaches.

[0079] 1 . In some embodiments, a computer-implemented method for training one or more machine learning models to convert label-free images into stained cell images comprises executing a first machine learning model to convert a first set of label-free images into a first set of training output images; training the first machine learning model using a first loss computed between a first set of stained cell images corresponding to the first set of label-free images and the first set of training output images to generate a first trained machine learning model; executing the first trained machine learning model to convert a second set of label-free images into a second set of training output images; and retraining the first trained machine learning model using a second loss computed based on semantic segmentations of a second set of stained cell images and the second set of training output images to generate a second trained machine learning model.

[0080] 2. The computer-implemented method of clause 1 , further comprising retraining the first trained machine learning model using a third loss associated with a similarity between the second set of stained cell images and the second set of training output images to generate a third trained machine learning model.

[0081] 3. The computer-implemented method of any of clauses 1-2, further comprising executing the third trained machine learning model to convert a first label- free image into a first stained cell image; and extracting, from the first stained cell image, one or more texture-based attributes associated with one or more cellular structures depicted in the first label-free image.

[0082] 4. The computer-implemented method of any of clauses 1-3, wherein the third loss comprises a structural similarity index measure.

[0083] 5. The computer-implemented method of any of clauses 1-4, further comprising executing the second trained machine learning model to convert a first label-free image into a first stained cell image; and extracting, from the first stained cell image, one or more quantitative shape-based attributes associated with one or more cellular structures depicted in the first label-free image.

[0084] 6. The computer-implemented method of any of clauses 1-5, further comprising extracting a set of label-free image crops from the first set of label-free images; executing the first machine learning model to convert the set of label-freeimage crops into a set of training output image crops; and computing the first loss based on a set of stained cell image crops corresponding to the set of label-free image crops and the set of training output image crops.

[0085] 7. The computer-implemented method of any of clauses 1-6, wherein the second loss comprises a weighted cross entropy loss.

[0086] 8. The computer-implemented method of any of clauses 1-7, wherein the first loss comprises a mean squared error.

[0087] 9. The computer-implemented method of any of clauses 1-8, wherein the first set of label-free images and the first set of stained cell images depict a first cell type and the second set of label-free images and the second set of stained cell images depict a second cell type.

[0088] 10. The computer-implemented method of any of clauses 1 -9, wherein the first machine learning model comprises a V-Net architecture and one or more dilated convolutions.

[0089] 11 . In some embodiments, one or more non-transitory computer-readable media store instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of executing a first machine learning model to convert a first set of label-free images into a first set of training output images; training the first machine learning model using a first loss computed between a first set of stained cell images corresponding to the first set of label-free images and the first set of training output images to generate a first trained machine learning model; executing the first trained machine learning model to convert a second set of label-free images into a second set of training output images; and retraining the first trained machine learning model using a second loss associated with a structural similarity between a second set of stained cell images and the second set of training output images to generate a second trained machine learning model.

[0090] 12. The one or more non-transitory computer-readable media of clause 11 , wherein the instructions further cause the one or more processors to perform the steps of executing the first trained machine learning model to convert a third set of label-free images into a third set of training output images; and retraining the first machine learning model using a third loss associated with semantic segmentations ofa third set of stained cell images and semantic segmentations of the third set of training output images to generate a third trained machine learning model.

[0091] 13. The one or more non-transitory computer-readable media of any of clauses 11-12, wherein the instructions further cause the one or more processors to perform the steps of executing the second trained machine learning model to convert a first label-free image into a first stained cell image; executing the third trained machine learning model to convert the first label-free image into a second stained cell image; extracting, from the first stained cell image, one or more texture-based attributes associated with one or more cellular structures depicted in the first label- free image; and extracting, from the second stained cell image, one or more shapebased attributes associated with the one or more cellular structures depicted in the first label-free image.

[0092] 14. The one or more non-transitory computer-readable media of any of clauses 11-13, wherein the third loss comprises a weighted cross entropy loss.

[0093] 15. The one or more non-transitory computer-readable media of any of clauses 11-14, wherein the instructions further cause the one or more processors to perform the steps of extracting a set of label-free image crops from the second set of label-free images; executing the first trained machine learning model to convert the set of label-free image crops into a set of training output image crops; and computing the second loss based on a set of stained cell image crops corresponding to the set of label-free image crops and the set of training output image crops.

[0094] 16. The one or more non-transitory computer-readable media of any of clauses 11-15, wherein the first set of label-free images and the first set of stained cell images depict a first type of object and the second set of label-free images and the second set of stained cell images depict a second type of object.

[0095] 17. The one or more non-transitory computer-readable media of any of clauses 11-16, wherein the first type of object and the second type of object correspond to at least one of different cell types, different cell lines, different organelle types, or different types of subcellular structures.

[0096] 18. The one or more non-transitory computer-readable media of any of clauses 11-17, wherein the first loss comprises a mean squared error and the second loss comprises a structural similarity index measure.

[0097] 19. The one or more non-transitory computer-readable media of any of clauses 11-18, wherein the first machine learning model comprises one or more dilated convolutions included in an upsampling path of a V-Net architecture.

[0098] 20. In some embodiments, a system comprises one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform the steps of matching one or more attributes to be measured from one or more objects depicted within a label-free image to a machine learning model, wherein the machine learning model is trained using a first loss associated with a reconstruction of a set of training stained cell images and a second loss associated with the one or more attributes; executing the machine learning model to convert the label-free image into an output image corresponding to a prediction of a stained cell image depicting the one or more objects; and extracting the one or more attributes from the output image.

[0099] Any and all combinations of any of the claim elements recited in any of the claims and / or any elements described in this application, in any fashion, fall within the contemplated scope of the present invention and protection.

[0100] The descriptions of the various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0101] Aspects of the present embodiments may be embodied as a system, method or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “module,” a “system,” or a “computer.” In addition, any hardware and / or software technique, process, function, component, engine, module,or system described in the present disclosure may be implemented as a circuit or set of circuits. Furthermore, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.

[0102] Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc readonly memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0103] Aspects of the present disclosure are described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine. The instructions, when executed via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / acts specified in the flowchart and / or block diagram block or blocks. Such processors may be, without limitation, general purpose processors, special-purpose processors, application-specific processors, or field-programmable gate arrays.

[0104] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0105] While the preceding is directed to embodiments of the present disclosure, other and further embodiments of the disclosure may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.

Claims

WHAT IS CLAIMED IS:1 . A computer-implemented method for training one or more machine learning models to convert label-free images into stained cell images, the method comprising: executing a first machine learning model to convert a first set of label-free images into a first set of training output images; training the first machine learning model using a first loss computed between a first set of stained cell images corresponding to the first set of label-free images and the first set of training output images to generate a first trained machine learning model; executing the first trained machine learning model to convert a second set of label-free images into a second set of training output images; and retraining the first trained machine learning model using a second loss computed based on semantic segmentations of a second set of stained cell images and the second set of training output images to generate a second trained machine learning model.

2. The computer-implemented method of claim 1 , further comprising retraining the first trained machine learning model using a third loss associated with a similarity between the second set of stained cell images and the second set of training output images to generate a third trained machine learning model.

3. The computer-implemented method of claim 2, further comprising: executing the third trained machine learning model to convert a first label-free image into a first stained cell image; and extracting, from the first stained cell image, one or more texture-based attributes associated with one or more cellular structures depicted in the first label-free image.

4. The computer-implemented method of claim 2, wherein the third loss comprises a structural similarity index measure.

5. The computer-implemented method of claim 1 , further comprising: executing the second trained machine learning model to convert a first label- free image into a first stained cell image; andextracting, from the first stained cell image, one or more quantitative shapebased attributes associated with one or more cellular structures depicted in the first label-free image.

6. The computer-implemented method of claim 1 , further comprising: extracting a set of label-free image crops from the first set of label-free images; executing the first machine learning model to convert the set of label-free image crops into a set of training output image crops; and computing the first loss based on a set of stained cell image crops corresponding to the set of label-free image crops and the set of training output image crops.

7. The computer-implemented method of claim 1 , wherein the second loss comprises a weighted cross entropy loss.

8. The computer-implemented method of claim 1 , wherein the first loss comprises a mean squared error.

9. The computer-implemented method of claim 1 , wherein the first set of label- free images and the first set of stained cell images depict a first cell type and the second set of label-free images and the second set of stained cell images depict a second cell type.

10. The computer-implemented method of claim 1 , wherein the first machine learning model comprises a V-Net architecture and one or more dilated convolutions.11 . One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of: executing a first machine learning model to convert a first set of label-free images into a first set of training output images; training the first machine learning model using a first loss computed between a first set of stained cell images corresponding to the first set of label-freeimages and the first set of training output images to generate a first trained machine learning model; executing the first trained machine learning model to convert a second set of label-free images into a second set of training output images; and retraining the first trained machine learning model using a second loss associated with a structural similarity between a second set of stained cell images and the second set of training output images to generate a second trained machine learning model.

12. The one or more non-transitory computer-readable media of claim 11 , wherein the instructions further cause the one or more processors to perform the steps of: executing the first trained machine learning model to convert a third set of label-free images into a third set of training output images; and retraining the first machine learning model using a third loss associated with semantic segmentations of a third set of stained cell images and semantic segmentations of the third set of training output images to generate a third trained machine learning model.

13. The one or more non-transitory computer-readable media of claim 12, wherein the instructions further cause the one or more processors to perform the steps of: executing the second trained machine learning model to convert a first label- free image into a first stained cell image; executing the third trained machine learning model to convert the first label- free image into a second stained cell image; extracting, from the first stained cell image, one or more texture-based attributes associated with one or more cellular structures depicted in the first label-free image; and extracting, from the second stained cell image, one or more shape-based attributes associated with the one or more cellular structures depicted in the first label-free image.

14. The one or more non-transitory computer-readable media of claim 12, wherein the third loss comprises a weighted cross entropy loss.

15. The one or more non-transitory computer-readable media of claim 11 , wherein the instructions further cause the one or more processors to perform the steps of: extracting a set of label-free image crops from the second set of label-free images; executing the first trained machine learning model to convert the set of label- free image crops into a set of training output image crops; and computing the second loss based on a set of stained cell image crops corresponding to the set of label-free image crops and the set of training output image crops.

16. The one or more non-transitory computer-readable media of claim 11 , wherein the first set of label-free images and the first set of stained cell images depict a first type of object and the second set of label-free images and the second set of stained cell images depict a second type of object.

17. The one or more non-transitory computer-readable media of claim 16, wherein the first type of object and the second type of object correspond to at least one of different cell types, different cell lines, different organelle types, or different types of subcellular structures.

18. The one or more non-transitory computer-readable media of claim 11 , wherein the first loss comprises a mean squared error and the second loss comprises a structural similarity index measure.

19. The one or more non-transitory computer-readable media of claim 11 , wherein the first machine learning model comprises one or more dilated convolutions included in an upsampling path of a V-Net architecture.

20. A system, comprising: one or more memories that store instructions, and one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform the steps of: matching one or more attributes to be measured from one or more objects depicted within a label-free image to a machine learningmodel, wherein the machine learning model is trained using a first loss associated with a reconstruction of a set of training stained cell images and a second loss associated with the one or more attributes; executing the machine learning model to convert the label-free image into an output image corresponding to a prediction of a stained cell image depicting the one or more objects; and extracting the one or more attributes from the output image.