Multi-model prediction of stained cell structures

Abstract

One embodiment of the invention sets forth a technique for training a machine learning model to convert unlabelled images to stained cell images. The technique includes executing a first machine learning model to convert a first set of unlabelled images to a first set of training output images, and training the first machine learning model using a first loss computed between a first set of stained cell images and the first set of training output images to generate a first trained machine learning model. The technique also includes executing the first trained machine learning model to convert a second set of unlabelled images to a second set of training output images, and retraining the first trained machine learning model using a second loss associated with a second set of stained cell images and the second set of training output images to generate a second trained machine learning model.

Description

[0001] Cross-reference to related applications This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 625,728, filed January 26, 2024, entitled "Multi-model prediction of stained cell structures". The subject matter of this related application is incorporated herein by reference.

[0002] background Example Field The embodiments of this disclosure generally relate to machine learning and computer vision, and more specifically, to multi-model prediction of stained cell structures.

[0003] Related technical descriptions Optical microscopy refers to the use of microscopes with various optical modalities to create images of cells. One of the simplest forms of optical microscopy is bright-field microscopy, which involves uniformly illuminating the biological sample (…). For example Label-free microscopy (using white light) images biological samples through a microscope. However, label-free microscopy often produces images with low contrast and limited information content because relatively few biological samples absorb light well. Therefore, it is difficult to distinguish and / or separate specific structures or molecules in label-free microscopic images.

[0004] To enhance the contrast and level of detail in biological samples imaged using optical microscopy, various cell staining techniques are employed to manually apply staining agents and / or dyes to the biological samples before imaging them under a microscope. These staining agents and / or dyes are preferentially absorbed by certain components and / or structures within the biological sample, thereby enhancing the visualization of these components and / or structures under the microscope. For example, specific organelle staining can be used to enhance the observation, identification, and study of the structure and behavior of cells, organelles, and / or other types of shapes and / or structures within cells. Therefore, cell staining can be used in a variety of applications, from determining cell type, health status, morphology, and / or function, to identifying the presence and / or accumulation of specific molecules and / or pathogens within cells.

[0005] However, cell staining has many drawbacks. First, staining and imaging biological samples is often a manual and time-consuming process. For example, conventional cell staining techniques may involve multiple steps to prepare biological samples for imaging. These steps may include (but are not limited to): fixing the biological sample with heat and / or chemicals to maintain the shape and / or structure of the underlying cells and / or tissues; immersing the biological sample in a solution of a selected dye or staining agent to allow the staining agent to bind to specific cellular components; washing the biological sample to remove any excess staining agent; dehydrating the biological sample; embedding the biological sample in paraffin and / or resin; and / or slicing the biological sample into sections.

[0006] Secondly, most optical microscopes have a limited number of imaging channels, which restricts the number of variables that can be measured simultaneously in a given biological sample. For example, a typical image-based assay can apply up to six stains simultaneously on five imaging channels. Given these limitations, conventional optical microscopy methods cannot capture additional morphological information from any unstained structures and / or components. Furthermore, since at least some imaging channels are typically used with universal fluorescent stains to image standard cellular components, the ability of conventional optical microscopy methods to use specific imaging channels is also limited in experiments involving non-universal fluorescent stains and / or perturbations of intracellular cells or molecules.

[0007] Recently, staining microscopy images have been combined with computer vision and deep learning-based image processing techniques to extract quantitatively reproducible features related to cell state. These techniques require higher image quality compared to prior techniques involving visual examination of stained cell images by scientists. For example, reduced image quality in staining microscopy images can lead to errors in the extraction of quantitative imaging biomarkers for cell nucleus shape and / or other properties.

[0008] As mentioned above, there is a need in the art for more effective techniques for identifying biological structures and / or components using optical microscopy. Summary of the Invention

[0009] One embodiment of the present invention describes a technique for training one or more machine learning models to convert unlabeled images into stained cell images. The technique includes: executing a first machine learning model to convert a first set of unlabeled images into a first set of training output images; and training the first machine learning model using a first loss calculated between the first set of training output images and a first set of stained cell images corresponding to the first set of unlabeled images to generate a first trained machine learning model. The technique further includes: executing the first trained machine learning model to convert a second set of unlabeled images into a second set of training output images; and retraining the first trained machine learning model using a second loss associated with the second set of stained cell images and the second set of training output images to generate a second trained machine learning model.

[0010] One advantage of the disclosed technique compared to prior art is its ability to convert label-free images of biological samples into stained cell images that accurately depict various quantitative properties of the same biological sample. In this respect, the disclosed technique can be used to simulate and / or impute quantitative properties of biological samples in stained cell images without requiring manual staining and subsequent imaging of the biological sample. Furthermore, since the generated stained cell images can be used to determine and / or measure quantitative properties within the biological sample without staining, the disclosed technique allows for the determination and / or control of additional properties of the biological sample using available imaging channels and / or staining agents. Accordingly, compared to conventional optical microscopy techniques with limited availability of staining agents and / or imaging channels for a given biological sample, the disclosed technique provides a more comprehensive view of the biological sample. These advantages represent one or more technical improvements over prior art methods. Attached Figure Description

[0011] This patent or application document contains at least one color drawing. A copy of the published text of this patent or patent application, with one or more color drawings, will be provided by the Patent Office upon request and payment of the necessary fees.

[0012] To enable a detailed understanding of the features described above in the various embodiments, the inventive concept briefly outlined above can be described in more detail by referring to various embodiments, some of which are shown in the accompanying drawings. However, it should be noted that the accompanying drawings illustrate only typical embodiments of the inventive concept and should not be considered in any way as a limitation of the scope, and that other equally effective embodiments exist.

[0013] Figure 1 A system configured to implement one or more aspects of the various embodiments is shown.

[0014] Figure 2 Based on the various embodiments Figure 1 A more detailed illustration of the training engine and execution engine.

[0015] Figure 3 The diagram illustrates the relationship between various embodiments and Figure 2 The machine learning model shown is associated with an exemplary set of unlabeled training images, stained cell images for training, and virtual stained cell images.

[0016] Figure 4 This is a flowchart of method steps for training a set of machine learning models to convert unlabeled images of biological samples into stained cell images of the same biological sample, according to various embodiments.

[0017] Figure 5 This is a flowchart of method steps for determining visual properties of cellular structures depicted in a label-free image, according to various embodiments. Detailed Implementation

[0018] In the following description, numerous specific details are set forth to provide a more thorough understanding of the various embodiments. However, it will be apparent to those skilled in the art that the inventive concept can be practiced without one or more of these specific details.

[0019] As discussed above, staining and imaging biological samples is often a manual and time-consuming process. Additionally, most optical microscopes have a limited number of ( For example The imaging channels (2-5) limit the number of variables that can be measured simultaneously in a given biological sample.

[0020] To reduce the overhead associated with cell staining and / or understanding various structures and / or components within biological samples, the disclosed technique trains and executes multiple machine learning models to convert label-free images of biological samples into stained cell images of the same biological sample. Initially, a first machine learning model is trained to generate predictions of stained cell images of the same biological sample given an input including a label-free image of the biological sample. For example, the first machine learning model can be pre-trained using a reconstruction loss computed between the stained cell image and the output image generated by the first machine learning model from the corresponding label-free image. After the initial training of the model is complete, the first machine learning model is subsequently retrained and / or tuned using different losses and / or additional pairs of label-free and stained cell images. This retraining and / or tuning is used to generate one or more additional trained machine learning models optimized for various properties of the structures of the components depicted in the images. For example, a pre-trained first machine learning model can be retrained using segmentation-based loss, structural similarity loss, and / or other types of loss to generate an additional, tuned machine learning model that, given an input including the corresponding unlabeled image, can accurately reconstruct the shape, texture, size, number, distribution, and / or other visual attributes of organelles and / or other types of structures depicted within a cell in a stained cell image.

[0021] After the machine learning models have been trained, one or more trained machine learning models can be used to convert additional label-free images into corresponding stained cell images. For example, one or more types of attributes to be determined from a biological sample can be matched to one or more trained machine learning models tuned and / or optimized for those types of attributes. One or more label-free images of the biological sample can be input into one or more trained machine learning models, and these models can be executed to convert the input label-free images into one or more corresponding stained cell images. Quantitative analysis can then be performed on one or more of the generated stained cell images to measure and / or characterize these attributes of the structures and / or components depicted in the generated stained cell images.

[0022] System Overview Figure 1This is a block diagram illustrating a computer system 100 configured to implement one or more aspects of various embodiments. In one embodiment, the computer system 100 includes a desktop computer, laptop computer, smartphone, personal digital assistant (PDA), tablet computer, or any other type of computing device configured to receive input, process data, and optionally display images, and suitable for practicing one or more embodiments. The computer system 100 also, or alternatively, includes machines or processing nodes operating in a data center, cluster, or cloud computing environment that provide scalable computing resources (optionally as a service) over a network.

[0023] As shown in the figure, the computer system 100 includes, but is not limited to, a central processing unit (CPU) 102 and a system memory 104, which are coupled to a parallel processing subsystem 112 via a memory bridge 105 and a communication path 113. The memory bridge 105 is further coupled to an I / O (input / output) bridge 107 via a communication path 106, and the I / O bridge 107 is coupled to a switch 116.

[0024] I / O bridge 107 is configured to receive user input from an optional input device 108, such as a keyboard or mouse, and forward that input to CPU 102 for processing via communication path 106 and memory bridge 105. In some embodiments, computer system 100 may be a server machine in a cloud computing environment. In such embodiments, computer system 100 may not have input device 108. Instead, computer system 100 may receive equivalent input by receiving commands, which take the form of messages transmitted over a network and received via network adapter 118. In one embodiment, switch 116 is configured to provide connectivity between I / O bridge 107 and other components of computer system 100, such as network adapter 118 and various add-on cards 120 and 121.

[0025] In one embodiment, I / O bridge 107 is coupled to system disk 114, which may be configured to store content, applications, and data for use by CPU 102 and parallel processing subsystem 112. In one embodiment, system disk 114 provides non-volatile storage for applications and data and may include fixed or removable hard disk drives, flash memory devices, and CD-ROMs (CD-ROMs), DVD-ROMs (Digital Versatile Optical Discs), Blu-ray discs, HD-DVDs (High Definition DVDs), or other magnetic, optical, or solid-state storage devices. In various embodiments, other components such as universal serial buses or other port connections, optical disc drives, digital versatile optical disc drives, film recording devices, etc., may also be connected to I / O bridge 107.

[0026] In various embodiments, memory bridge 105 may be a northbridge chip, while I / O bridge 107 may be a southbridge chip. Furthermore, communication paths 106 and 113, as well as other communication paths within computer system 100, may be implemented using any technically suitable protocol, including but not limited to AGP (Accelerated Graphics Port), HyperTransport, or any other bus or point-to-point communication protocol known in the art.

[0027] In some embodiments, the parallel processing subsystem 112 includes a graphics subsystem that transmits pixels to an optional display device 110, which may be any conventional cathode ray tube, liquid crystal display, light-emitting diode display, or similar device. In such embodiments, the parallel processing subsystem 112 incorporates circuitry optimized for graphics and video processing, including, for example, video output circuitry. This circuitry may be incorporated into one or more parallel processing units (PPUs), also referred to herein as parallel processors, and included within the parallel processing subsystem 112. In other embodiments, the parallel processing subsystem 112 incorporates circuitry optimized for general and / or computational processing. Again, this circuitry may be incorporated into one or more PPUs included in the parallel processing subsystem 112 and configured to perform such general and / or computational operations. In other embodiments, one or more PPUs included in the parallel processing subsystem 112 may be configured to perform graphics processing, general processing, and computational processing operations. System memory 104 includes at least one device driver configured to manage the processing operations of one or more PPUs within parallel processing subsystem 112.

[0028] Parallel processing subsystem 112 can be with Figure 1 One or more of the other components can be integrated to form a single system. For example, the parallel processing subsystem 112 can be integrated with the CPU 102 and other interconnect circuitry on a single chip to form a system-on-a-chip (SoC).

[0029] In one embodiment, CPU 102 is the main processor of computer system 100, controlling and coordinating the operation of other system components. In one embodiment, CPU 102 issues commands to control the operation of the PPU. In some embodiments, communication path 113 is a PCI high-speed link, wherein, as is known in the art, a dedicated channel is allocated to each PPU. Other communication paths may also be used. The PPU advantageously implements a highly parallel processing architecture. Any number of local parallel processing memories (PP memories) can be provided for the PPU.

[0030] It should be understood that the system illustrated herein is illustrative, and variations and modifications are possible. First, the system's functionality can be distributed across multiple nodes in a distributed and / or cloud computing system. Second, the connection topology can be modified as needed, including the number and arrangement of bridges, the number of CPUs 102, and the number of parallel processing subsystems 112. For example, in some embodiments, system memory 104 can be directly connected to CPU 102 instead of via memory bridge 105, and other devices will communicate with system memory 104 via memory bridge 105 and CPU 102. In another example, parallel processing subsystem 112 can be connected to I / O bridge 107 or directly to CPU 102 instead of via memory bridge 105. In a third example, I / O bridge 107 and memory bridge 105 can be integrated into a single chip instead of existing as one or more discrete devices. Third, Figure 1 One or more of the components shown may be absent. For example, switch 116 may be eliminated, and network adapter 118 and add-on cards 120, 121 will be directly connected to I / O bridge 107.

[0031] In one or more embodiments, the computer system 100 is configured to execute a training engine 122 and an execution engine 124 residing in system memory 104. The training engine 122 and the execution engine 124 may be stored in system disk 114 and / or other storage devices and are loaded into system memory 104 during execution.

[0032] More specifically, training engine 122 and execution engine 124 include functions for training and executing multiple machine learning models to convert label-free images of biological samples into stained cell images of the same biological sample. In the label-free images of biological samples, one or more features of the biological sample being visualized are typically unlabeled or altered in any way. Training engine 122 pre-trains a first machine learning model to generate predictions of stained cell images of the same biological sample given an input consisting of a label-free image of the biological sample. For example, the first machine learning model can be pre-trained using a reconstruction loss computed between the stained cell image and the output image generated by the first machine learning model from the corresponding label-free image. After the pre-training of the first machine learning model is complete, training engine 122 retrains and / or fine-tunes the first machine learning model using different losses and / or additional pairs of label-free and stained cell images to generate one or more additional trained machine learning models optimized for various properties of the structures and / or components depicted in the images. For example, a pre-trained first machine learning model can be retrained using shape-aware loss, structural similarity loss, and / or other types of loss to generate an additional, optimized machine learning model that, given an input including a corresponding unlabeled image, can accurately reconstruct the cell, shape, texture, size, number, distribution, and / or other visual features of organelles and / or other types of cells or subcellular structures depicted in a stained cell image.

[0033] After the machine learning model has been trained, the execution engine 124 can use one or more trained machine learning models to convert additional label-free images into corresponding stained cell images. For example, the execution engine 124 can match one or more types of attributes to be determined from a biological sample to one or more trained machine learning models optimized for those types of attributes. The execution engine 124 can input one or more label-free images of the biological sample into one or more trained machine learning models and execute those models to convert the one or more input label-free images into one or more corresponding stained cell images. After one or more stained cell images are generated, the execution engine 124 can perform quantitative analysis on the one or more generated stained cell images to measure and / or characterize these attributes of the structures and / or components depicted in the one or more generated stained cell images. The operation of the training engine 122 and the execution engine 124 will be described in further detail below.

[0034] Multi-model prediction of stained cell structure Figure 2 Based on the various embodiments Figure 1A more detailed illustration of the training engine 122 and the execution engine 124 is provided. As described above, the training engine 122 and the execution engine 124 run to train and execute multiple machine learning models 224 to convert unlabeled images 222 of biological samples into one or more stained cell images 230 of the same biological sample.

[0035] In some embodiments, the label-free image 222 includes an image of a biological sample captured using imaging techniques such as (but not limited to) optical microscopy, bright-field microscopy, quantitative label-free imaging with phase and polarization (QLIPP), fluorescence imaging, autofluorescence imaging, phase-contrast microscopy, and / or differential interferometry contrast microscopy. In some embodiments, the biological sample is a cell sample and / or a tissue sample, such as (but not limited to) cell cultures, tissue cultures, organoid cultures, and multicellular organism cultures. For example (e.g., cultures of Caenorhabditis elegans or Drosophila melanogaster). Cell cultures refer to biological processes that maintain living cells in an artificial environment, such as cell culture dishes. Cell culture dishes can include plates, flasks, and / or any other type of container compatible with image acquisition of cell cultures. For example, biological samples can include eukaryotic (e.g., nematodes, vesicles, or fruit flies). For example, (Mammalian) or prokaryotic cells. In some examples, cells may include human, mouse, and / or rat cells. Generally, any human or non-human animal can be a source of cells or tissues that can be analyzed according to this disclosure. In some embodiments, cells may be skin cells, muscle cells, blood cells, liver cells, heart cells, spleen cells, thymocytes, brain cells, myoblasts, fibroblasts, and / or keratinocytes. Cells may be alive or dead ( For example (Immobilized cells). Generally, label-free images 222 can depict cells, tissues, organelles and / or other structures, substructures and / or components within a biological sample, while these structures or components are illuminated and magnified by one or more lenses within an optical microscope.

[0036] The stained cell image 230 may include one or more images depicting a biological sample in the presence of one or more staining agents and / or dyes that are preferentially absorbed by certain components and / or structures within the biological sample. For example, the stained cell image 230 may depict cells, tissues, organelles, and / or other structures and / or components within the biological sample as would appear after one or more staining techniques have been applied to the biological sample.

[0037] In one or more embodiments, the machine learning model 224 includes a fully convolutional neural network with an encoder-decoder architecture capable of processing inputs of different resolutions. For example, one or more machine learning models 224 may include a U-Net model that performs image transformation via an encoder and a decoder. The encoder may include convolutional layers and max-pooling layers that transform a given input image (…). For example The U-Net model generates downsampled feature maps from the unlabeled image (222) and / or one or more subsets of the input image. The decoder can generate upsampled feature maps from the downsampled feature maps using upsampling, concatenation, and transposed convolution operations. The U-Net model may also include skip connections that connect layers in the encoder to corresponding layers in the decoder that are associated with the same feature map resolution.

[0038] In another example, one or more machine learning models 224 may include a V-Net architecture. Similar to U-Net, V-Net also includes an encoder, a decoder, and skip connections, the encoder taking data from a given input image (…). For example The V-Net encoder generates downsampled feature maps from unlabeled images (222), and the decoder generates upsampled feature maps from these downsampled feature maps. Skip connections connect layers in the encoder to corresponding layers in the decoder associated with the same feature map resolution. The V-Net encoder uses strided convolutions to perform learned downsampling of the feature maps, while the V-Net decoder uses deconvolution operations to perform upsampling. Compared to the standard U-Net model, the convolutional and / or deconvolution operations performed by the V-Net model can reduce memory overhead. These deconvolution operations can include dilated convolutions in the V-Net upsampling path. In various embodiments, each resolution in which V-Net operates includes a residual function that performs non-linear processing on the input to that resolution and adds the result to the output of the last layer associated with that resolution.

[0039] Training engine 122 uses training data 202 to train machine learning model 224, which includes a set of training label-free images 240 and a corresponding set of training stained cell images 242. Each training label-free image 240 depicts a label-free image of a biological sample captured using label-free microscopy. Each training label-free image 240 is also paired with a stained cell image included in the training stained cell images 242 that depicts the same biological sample as the training label-free image. For example, the training label-free images 240 may include images of tissues, cells, cellular structures, and / or other types of biological specimens captured using imaging techniques and not stained. The training stained cell images 242 may include images of the same tissues, cells, cellular structures, subcellular structures, and / or biological specimens depicted in the training label-free images 240. However, unlike the training label-free image 240, the training stained cell image 242 can be generated by applying a specific staining agent and / or dye to the biological specimen depicted in the training label-free image 240 and then imaging the stained and / or stained biological specimen using bright-field microscopy, quantitative label-free imaging with phase and polarization (QLIPP), fluorescence imaging, autofluorescence imaging, phase contrast microscopy, differential interference contrast microscopy and / or other techniques.

[0040] When the training unlabeled image 240 is acquired in three dimensions, n consecutive slices in the z-direction can be stitched together along the channel dimension and fed as input to the machine learning model 224. The corresponding output can include a channel that corresponds to a virtual coloring of the center slice of the input z-stack. This 2.5D approach reduces the memory footprint of the machine learning model 224, allowing it to have a deeper architecture and a higher-dimensional parameter space, thereby utilizing more spatial information from the training unlabeled image 240.

[0041] In one or more embodiments, at least a portion of the training stained cell images 242 includes images of organelles and / or other structures or components specifically stained within the corresponding biological sample. For example, the training stained cell images 242 may include multiple sets of images. Each set of images may include visualizations of different cellular components. These cellular components may include (but are not limited to) the nucleus, endoplasmic reticulum, nucleolus, cytoplasmic ribonucleic acid (RNA), actin, Golgi apparatus, plasma membrane, mitochondria, and / or other types of organelles and / or subcellular structures. When the input includes corresponding unlabeled training images 240 of the same biological sample, each set of images may further be used as a set of labels predicted by one or more machine learning models 224.

[0042] The data generation component 204 in the training engine 122 generates image crop 254 of the label-free training image 240 and the stained cell training image 242. For example, the data generation component 204 can generate fixed-size and / or variable-size image crop 254 of the label-free training image 240. The data generation component 204 can also generate the same image crop 254 corresponding to the stained cell training image 242, such that a given label-free training image crop is paired with a corresponding stained cell training image crop, and the two image crop slices depict the same region of the same biological sample. A given image crop may include a random position, height, and / or width. A given image crop may also, or alternatively, include a selected position, height, and / or width so that the image crop captures one or more objects in the corresponding label-free training image and / or stained cell training image. For example (specific organelles or cellular structures).

[0043] The data generation component 204 also generates image segments 252, some or all of the training stained cell images 242 and / or corresponding image crop patches 254. For example, the data generation component 204 can input the training stained cell images 242 and / or corresponding image crop patches 254 into a pre-trained segmentation model. The data generation component 204 can also execute the segmentation model to convert each input image and / or image crop patch into a corresponding image segment of the same size as the input image and / or image crop patch. In image segmentation, pixels at a given location can be assigned a label indicating the type of object found at that location (…). For example (Specific types of organelles and / or cellular structures). When a specific object is not detected at a given location, the corresponding pixel can be assigned the label "background" or "unknown".

[0044] The update component 206 in the training engine 122 trains the machine learning model 224 in several training phases 212 and 214 using training data 202, image crop patches 254, and / or image segmentation 252. During the first training phase 212, the update component 206 inputs some or all of the training unlabeled images 240 and / or image crop patches 254 of these training unlabeled images 240 into the first machine learning model contained in the machine learning model 224. For each input training unlabeled image (or input training unlabeled image crop patch), the update component 206 executes the first machine learning model to generate a corresponding training output 210. The training output 210 includes a prediction of a stained cell image (or stained cell image crop patch) that depicts the same object as the input training unlabeled image (or training unlabeled image crop patch). The update component 206 also calculates one or more losses 216 between the training output 210 and the training stained cell image (or a cropped patch of the training stained cell image), which is paired with the input training unlabeled image (or a cropped patch of the input training unlabeled image). The update component 206 additionally uses training techniques ( For example (gradient descent and backpropagation) are used to update the base model parameters 208 of the first machine learning model to reduce the loss 216. The updating component 206 can repeat this process until the loss 216 falls below a threshold, the base model parameters 208 converge and / or another condition is met.

[0045] In one or more embodiments, the loss 216 used to train the first machine learning model includes a reconstruction loss between the training output 210 generated by the first machine learning model from the training unlabeled image 240 and / or image cropped patches 254 of the training unlabeled image 240 and the corresponding training stained cell image 242 and / or image cropped patches 254 of the training stained cell image. For example, the loss 216 may include mean squared error (MSE), mean absolute error (MAE), and / or another error metric calculated between pixel values ​​in the training output 210 and the corresponding “ground truth” training stained cell image 242 and / or image cropped patch 254. This reconstruction loss allows the first machine learning model to learn a general and / or global mapping between the training unlabeled image 240 (or training unlabeled image cropped patches) and the objects in the corresponding training stained cell image 242 (or training stained cell image cropped patches).

[0046] After the first training phase 212 is completed, the update component 206 implements one or more additional training phases 214 to train additional machine learning models 224, using the first machine learning model trained in training phase 212 as a starting point. During each training phase 214, the update component 206 inputs some or all of the training unlabeled images 240 and / or image cropped patches 254 of these training unlabeled images 240 into the additional machine learning model contained in the machine learning model 224. For each input training unlabeled image (or input training unlabeled image cropped patch), the update component 206 executes the additional machine learning model to generate a corresponding training output 220. This training output 220 includes predictions for stained cell images (or stained cell image cropped patches) that depict the same objects as the input training unlabeled images (or input training unlabeled image cropped patches). The update component 206 also calculates one or more losses 226 associated with the training output 220 and the training stained cell image (or a cropped patch of the training stained cell image), which is paired with the input training unlabeled cell image (or a cropped patch of the input training unlabeled image). The update component 206 additionally uses training techniques ( For example (gradient descent and backpropagation) are used to update the tuned model parameters 218 of the additional machine learning model to reduce the loss 226. The updating component 206 can repeat this process until the loss 226 falls below a threshold, the tuned model parameters 218 converge and / or another condition is met.

[0047] In some embodiments, training phase 214 includes optimizing and / or tuning one or more machine learning models 224 to accurately generate the shapes of objects in training stained cell images 242. During this training phase, update component 206 updates the tuned model parameters 218 of one or more machine learning models using one or more losses 226, which are based on the first set of image segmentation 252 of the training output 220. For exampleThe loss 226 is calculated between a second set of image segments 252 (generated by the data generation component 204) and the corresponding "ground truth" training stained cell image 242 (or training stained cell image crop 254). For example, the loss 226 may include a first image segment of the predicted stained cell image (or stained cell image crop 254) generated by the machine learning model from the input training unlabeled image (or training unlabeled image crop 254) and a second image segment of the training stained cell image (or training stained cell image crop 254) paired with the input training unlabeled image (or training unlabeled image crop 254), a cross-entropy loss, a weighted cross-entropy loss, a Dice loss, and / or another error metric. This segmentation-based loss can be used to improve the accuracy of the machine learning model in predicting shapes in the training stained cell image 242 (or training stained cell image crop 254) when the input includes the corresponding training unlabeled image 240 (or training unlabeled image crop 254).

[0048] Training phase 214 also, or alternatively, includes a training phase that optimizes and / or tunes one or more machine learning models 224 to accurately generate textures of objects in training stained cell images 242. During this training phase, update component 206 updates the tuned model parameters 218 of one or more machine learning models using one or more losses 226, which are associated with the structural similarity between the training output 220 and the corresponding “ground” training stained cell image 242 (or training stained cell image cropped patch 254). For example, loss 226 may include a structural similarity index measure (SSIM), a perceptual loss computed between a first set of features generated by a pre-trained feature extractor from training output 220 and a second set of features generated by a pre-trained feature extractor from the corresponding ground truth image or image crop patch, and / or another measure of the similarity between the prediction of a stained cell image (or stained cell image crop patch) generated by the machine learning model from the input training unlabeled image (or training unlabeled image crop patch) and a paired training stained cell image (or training stained cell image crop patch). This similarity-based loss can be used to improve the accuracy of the machine learning model in predicting texture in training stained cell image 242 (or training stained cell image crop patch 254) when the input includes the corresponding training unlabeled image 240 (or training unlabeled image crop patch 254).

[0049] Although the operation of the training engine 122 has been described above in conjunction with training phase 214, which optimizes and / or tunes the machine learning model 224 to accurately generate shapes and / or textures associated with objects depicted in the training stained cell image 242 (or the cropped patch 254 of the training stained cell image), it should be understood that training phases 212 and / or 214, and the corresponding losses 216 and / or 226, can be adapted to optimize and / or tune the corresponding machine learning model 224 for various types of tasks and / or outputs. For example, the training engine 122 may implement one or more training phases 212 and / or 214 to train one or more machine learning models 224 using combinations of reconstruction, similarity, segmentation, and / or other types of losses 216 and / or 226. This combination may include linear combinations, weighted combinations, and / or other combinations of multiple losses 216 and / or 226, which are computed and used to update the parameters of the corresponding one or more machine learning models over several training steps, iterations, batches, and / or epochs. For example (e.g., base model parameters 208, tuned model parameters 218, etc.). This combination may also, or alternatively, involve training one or more machine learning models using a series of different losses 216 and / or 226 in multiple training phases 212 and / or 214, alternating between two or more losses 216 and / or 226 during the training of one or more machine learning models, and / or using different losses 216 and / or 226 to update different subsets of the parameters of one or more machine learning models.

[0050] In another example, training engine 122 may implement one or more training phases 212 and / or 214 to train one or more machine learning models 224 using one or more losses 216 and / or 226, which optimize for the generation of various attributes of cells, organelles, and / or subcellular structures in the training stained cell image 242 (or the training stained cell image crop 254) given that the input includes the corresponding training unlabeled image 240 (or the training unlabeled image crop 254). These losses 216 and / or 226 may include adversarial losses calculated based on additional outputs and / or intermediate activations generated by one or more discriminator models from the training outputs 210 and / or 220. These losses 216 and / or 226 may also, or alternatively, include losses based on a certain type of object within the training stained cell image 242 (or the training stained cell image crop 254). For exampleThe losses 216 and / or 226 are calculated based on the number of lysosomes, ribosomes, mitochondria, etc. These losses 216 and / or 226 may also, or alternatively, include losses calculated based on the size distribution of a certain type of object within the training stained cell image 242 (or the cropped block 254 of the training stained cell image). These losses 216 and / or 226 may also, or alternatively, include one or more losses calculated based on the distribution, orientation, and / or shape of a certain type of object within the training stained cell image 242 (or the cropped block 254 of the training stained cell image).

[0051] In the third example, training engine 122 can implement one or more initial training phases ( For example Training phase 212), to pre-train the machine learning model, inputting a certain type of object ( For example The training engine 122 converts unlabeled training images 240 (or cropped patches of unlabeled training images 254) of cells, organelles, subcellular structures, proteins, etc., into corresponding stained cell training images 242 (or cropped patches of stained cell training images 254). Subsequently, the training engine 122 can implement one or more additional training phases. For example Training phase 214) is used to fine-tune the machine learning model by inputting different types of objects ( For example The training label-free image 240 (or training label-free image crop 254) of cells, organelles, subcellular structures, proteins, etc., is converted into a corresponding training stained cell image 242 (or training stained cell image crop 254). One or more initial training phases can be implemented on training label-free images 240 (or training label-free image crop 254) and training stained cell images 242 (or training stained cell image crop 254) of relatively easy-to-image objects such as relatively abundant cell types and / or cell lines. One or more additional training phases can be implemented on training label-free images 240 (or training label-free image crop 254) and training stained cell images 242 (or training stained cell image crop 254) of relatively difficult-to-image objects such as rare cell types and / or cell lines.

[0052] In the fourth example, training engine 122 can implement one or more initial training phases ( For example Training phase 212, to pre-train the machine learning model, converts the input training unlabeled image 240 (or training unlabeled image cropped patch 254) into training stained cell image 242 (or training stained cell image cropped patch 254) captured using some type of cell staining and / or fluorescence microscopy technique. Subsequently, training engine 122 can implement one or more additional training phases (…). For exampleIn the training phase 214, the input unlabeled training image 240 (or cropped patch 254 of unlabeled training image) is converted into stained cell images 242 (or cropped patch 254 of stained cell image) of training captured using different types of cell staining and / or fluorescence microscopy techniques.

[0053] After training the machine learning model 224, the execution engine 124 executes one or more trained machine learning models 224 to convert a given unlabeled image 222 into one or more stained cell images 230. Figure 2 As shown, execution engine 124 selects one or more machine learning models 224 based on attribute type 228 of attributes to be determined from one or more generated stained cell images, to be used in converting unlabeled image 222 into one or more stained cell images 230. For example, execution engine 124 may select one or more machine learning models 224 that have been trained to accurately reconstruct the objects depicted in unlabeled image 222 (…). For example The shape, texture, quantity, distribution, orientation, and / or other visual attributes of cells, organelles, subcellular structures, etc. In another example, the execution engine 124 may select one or more machine learning models 224 that have been trained and / or tuned to translate between training unlabeled images 240 and training stained cell images 242 depicting a certain cell type and / or cell line in unlabeled images 222.

[0054] Next, execution engine 124 inputs the label-free image 222 into one or more selected machine learning models and uses each machine learning model to convert the label-free image 222 into a corresponding stained cell image. For example, execution engine 124 can input the entire label-free image 222 into a given machine learning model and execute that machine learning model to generate the corresponding stained cell image. Execution engine 124 may also, or alternatively, extract one or more image cropping blocks from the label-free image 222 and input each label-free image 222 cropping block into a machine learning model. Execution engine 124 can use the machine learning model to convert each input label-free image 222 cropping block into a corresponding stained cell image cropping block.

[0055] In some embodiments, execution engine 124 combines multiple chromocyte image crop blocks generated by one or more selected machine learning models from multiple input unlabeled image 222 crop blocks into a corresponding chromocyte image. For example, execution engine 124 may utilize the layout of the unlabeled image crop blocks within the unlabeled image 222 to form a corresponding layout of the chromocyte image crop blocks within the chromocyte image. Execution engine 124 may also utilize the layout of the chromocyte image crop blocks to stitch and / or otherwise combine the chromocyte image crop blocks into a chromocyte image. Alternatively, execution engine 124 may use the individual chromocyte image crop blocks output by the machine learning model as discrete predictions of the objects depicted within these chromocyte image crop blocks, without combining the chromocyte image crop blocks into a chromocyte image.

[0056] In some embodiments, execution engine 124 uses one or more machine learning models 224 to generate multiple sets of stained cell image crop blocks within a given stained cell image from multiple sets of unlabeled image 222 crop blocks with different sizes, dimensions, or positions within the same unlabeled image 222. For example, execution engine 124 may divide the unlabeled image 222 into X different sets of unlabeled image 222 crop blocks, wherein each set of unlabeled image 222 crop blocks includes multiple image crop blocks of the same size, and different sets of unlabeled image 222 crop blocks are associated with different crop sizes. Execution engine 124 may also use a given machine learning model included in machine learning model 224 to convert each set of unlabeled image 222 crop blocks into stained cell image crop blocks for the corresponding set.

[0057] Execution engine 124 can further combine different levels or types of detail associated with multiple stained cell images 230 and / or multiple sets of stained cell image crop blocks into one or more stained cell images 230. For example, execution engine 124 can generate a given stained cell image as the average and / or another aggregation of multiple sets of stained cell image crop blocks generated by the same machine learning model and / or different machine learning models 224 from corresponding sets of label-free image 222 crop blocks. In another example, execution engine 124 can generate individual stained cell images for each set of stained cell image crop blocks generated by a given machine learning model from corresponding sets of label-free image 222 crop blocks. Each stained cell image can capture different levels and / or types of detail associated with an object in the label-free image 222 and can be used to perform different tasks associated with that object. For example (Edge detection, shape detection, object detection, instance detection, texture evaluation, etc.)

[0058] After the label-free image 222 is converted into one or more stained cell images 230, the execution engine 124 analyzes the label-free image 222 and / or the stained cell image 230 to determine the properties 232 of the cells, organelles, subcellular structures, and / or other objects depicted in the label-free image 222 and / or the stained cell image 230. For example, the execution engine 124 may use additional machine learning models, image processing techniques, and / or other techniques to measure one or more quantitative properties 232 of the objects in the label-free image 222 and / or the stained cell image 230. These quantitative properties 232 may include (but are not limited to) the object's shape, size, texture, quantity, size distribution, spatial distribution, and / or other visual properties 232. The execution engine 124 may provide the measured properties to downstream components, which may then perform further analysis on the label-free image 222, the stained cell image 230, and / or the objects depicted in the label-free image 222 and / or the stained cell image 230. The execution engine 124 may also, or alternatively, use additional machine learning models and / or analyses to generate labels associated with the objects depicted in the label-free image 222, the stained cell image 230, and / or the label-free image 222 and / or the stained cell image 230. For example Predictions of phenotype, disease, health status, age, the effects of disturbances, toxicity, etc.

[0059] Figure 3 The diagram illustrates the relationship between various embodiments and Figure 2 The machine learning model 224 is associated with a set of exemplary training label-free images 302 and 304, training stained cell images 312 and 314, and virtual stained cell images 322 and 324. Training label-free images 302 and 304 include label-free images of different biological samples. For example, training label-free image 302 may include images of different tissues, cells, cell structures, and / or other types of biological specimens.

[0060] Training label-free image 302 is paired with a corresponding training stained cell image 312, and training label-free image 304 is paired with a different corresponding training stained cell image 314. For example, training stained cell images 312 and 314 may each include images of the same tissues, cells, cell structures, and / or biological specimens depicted in training label-free images 302 and 304. Unlike training label-free images 302 and 304, training stained cell images 312 and 314 can be generated by applying specific staining agents and / or dyes to the biological specimens depicted in training label-free images 302 and 304 and imaging the stained and / or stained biological specimens using fluorescence microscopy, epifluorescence microscopy, and / or other techniques. Figure 3As shown, training stained cell images 312 and 314 include a magenta-colored background containing objects similar to those in the corresponding unlabeled training images 302 and 304. Each training stained cell image 312 and 314 also includes depictions of cell structures stained with fluorescent dyes. For example A bright green foreground object (such as the cell nucleus and organelles).

[0061] Each training unlabeled image 302 and 304 can be input into one or more machine learning models 224, and these models can be trained to generate corresponding virtual stained cell images 322 and 324, respectively, representing reconstructions of training stained cell images 312 and 314 paired with the training unlabeled images 302 and 304. More specifically, the one or more machine learning models can be trained through several training steps, iterations, batches, and / or rounds to generate each virtual stained cell image 322 and 324 based on inputs including the corresponding training unlabeled images 302 and 304. Each virtual stained cell image 322 and 324 includes a magenta background with objects resembling those depicted in the corresponding training stained cell images 312 and 314, respectively. Each virtual stained cell image 322 and 324 also includes bright green foreground objects, the shapes and sizes of which are similar to the foreground objects in the corresponding training stained cell images 312 and 314, respectively.

[0062] As described above, the one or more machine learning models can be trained and / or tuned such that various quantitative properties of the foreground objects in the training stained cell images 312 and 314 are accurately depicted in the corresponding virtual stained cell images 322 and 324. For example, a given machine learning model can be trained and / or tuned to reconstruct the shape, texture, size, and / or other properties of the foreground objects in the training stained cell images 312 and 314 based on inputs including the corresponding unlabeled training images 302 and 304.

[0063] Figure 4 This is a flowchart of method steps for training a set of machine learning models to convert label-free images of biological samples into stained cell images of the same biological sample, according to various embodiments. Although the method steps are combined... Figure 1-2 This disclosure describes a system, but those skilled in the art will understand that any system configured to perform some or all of the method steps in any order is within the scope of this disclosure.

[0064] As shown in the figure, in step 402, the training engine 122 generates a set of label-free images and corresponding image cropping blocks of stained cell images. For example, the training engine 122 may receive a set of label-free images captured using label-free microscopy techniques and depicting one or more cell types, cell lines, organelles, subcellular structures, tissues, and / or other types of biological samples. Each label-free image of a biological sample may be paired with one or more stained cell images depicting the same biological sample using one or more cell staining and / or fluorescence microscopy techniques. The training engine 122 may extract one or more cropping blocks from a given label-free image, the cropping blocks comprising a subset of pixels from the label-free image. Each cropping block may include a predefined and / or random location, size, and / or dimension, and may depict a subset of objects found in the label-free image. For each cropped patch extracted from the unlabeled image, the training engine 122 may also extract one or more cropped patches with the same location, size and / or dimensions from one or more corresponding stained cell images, and / or one or more cropped patches depicting a subset of objects identical to those in the unlabeled image.

[0065] In step 404, training engine 122 executes a machine learning model to convert at least a first portion of the unlabeled image and / or a cropped patch of the unlabeled image into training output. For example, training engine 122 may be selected as a model depicting certain types of objects within a corresponding biological sample. For example The training engine 122 can also, or alternatively, select a first portion of a label-free image and / or a label-free image crop block paired with a certain type of stained cell image and / or stained cell image crop block. The training engine 122 can input each label-free image and / or label-free image crop block into a U-Net, V-Net, and / or another type of deep neural network corresponding to the machine learning model. The training engine 122 can execute the deep neural network to generate downsampled and / or upsampled feature maps from the input. The training engine 122 can also use the deep neural network to convert some or all of the feature maps into training outputs, which include predictions of stained cell images (or stained cell image crop blocks) corresponding to the input label-free image (or label-free image crop block).

[0066] In step 406, training engine 122 trains the machine learning model using a first set of losses computed between the training outputs and corresponding portions of the stained cell images and / or stained cell image cropped patches. For example, training engine 122 may compute the MSE and / or another error metric between each training output generated by the machine learning model from the input unlabeled image (or unlabeled image cropped patch) and the paired stained cell image (or stained cell image cropped patch). Training engine 122 may aggregate this error across multiple training outputs and use gradient descent and backpropagation to update the weights in the machine learning model to reduce the error. Training engine 122 may continue training the machine learning model until one or more conditions are met. These one or more conditions include (but are not limited to) convergence of the machine learning model's parameters; reduction of one or more losses below a threshold; or a certain number of training steps, iterations, batches, and / or rounds.

[0067] In step 408, training engine 122 executes a trained machine learning model to convert at least an additional portion of the unlabeled image and / or the cropped unlabeled image patch into a training output. For example, training engine 122 may select an additional portion of the unlabeled image and / or the cropped unlabeled image patch as an object depicting the same type of object as the initial portion used to train the machine learning model in steps 404 and 406. For example The training engine 122 may select additional portions of the label-free images and / or label-free image cropped blocks (cell types, cell lines, organelles, subcellular structures, tissues, etc.) as label-free images and / or label-free image cropped blocks depicting objects of a different type than the initial portions used to train the machine learning model in steps 404 and 406. After the additional portions are selected and / or determined, the training engine 122 inputs each label-free image and / or label-free image cropped block in the additional portions into a U-Net, V-Net, and / or another type of deep neural network corresponding to the machine learning model trained in step 406. The training engine 122 may execute the deep neural network to generate downsampled and / or upsampled feature maps from the input. The training engine 122 may also use the deep neural network to convert some or all of the feature maps into training outputs, which include predictions of stained cell images (or stained cell image cropped blocks) corresponding to the input label-free images (or label-free image cropped blocks).

[0068] In step 410, training engine 122 retrains the trained machine learning model using additional sets of losses calculated between the training output and the corresponding chromocyte image and / or chromocyte image cropped patch. For example, training engine 122 may select one or more loss functions for retraining the trained machine learning model based on the task and / or output type for which it is to be optimized. If the task and / or output type involves accurately reconstructing the shape in the chromocyte image and / or chromocyte image cropped patch, training engine 122 may select cross-entropy loss, weighted cross-entropy loss, Dice loss, and / or another measure of classification error between the training output and the corresponding chromocyte image and / or chromocyte image cropped patch. If the task and / or output type involves accurately reconstructing the texture in the chromocyte image and / or chromocyte image cropped patch, training engine 122 may select SSIM loss, feature-based loss, and / or another similarity measure between the training output and the corresponding chromocyte image and / or chromocyte image cropped patch.

[0069] Continuing the example above, training engine 122 can compute one or more selected losses using each training output generated by the machine learning model from the input unlabeled image (or unlabeled image cropping) and the paired stained cell image (or stained cell image cropping). Training engine 122 can aggregate one or more computed losses across multiple training outputs and use gradient descent and backpropagation to update the weights in the machine learning model to reduce the error. Training engine 122 can continue to retrain the trained machine learning model until one or more conditions are met. These one or more conditions include (but are not limited to) convergence of the machine learning model's parameters; reduction of one or more losses below a threshold; or reaching a certain number of training steps, iterations, batches, and / or epochs. After the retraining of the trained machine learning model is complete, training engine 122 can save the final weights of the retrained machine learning model as a new trained machine learning model optimized for the corresponding task and / or output type.

[0070] In step 412, training engine 122 determines whether to continue retraining the trained machine learning model. For example, training engine 122 may determine that the trained machine learning model should continue to be retrained while it is optimized and / or tuned to perform additional tasks and / or generate additional types of output. If training engine 122 determines in step 412 that retraining should continue, it repeats steps 408 and 410 to generate different trained machine learning models for each task and / or output type for which the trained machine learning model needs to be optimized and / or tuned. After the trained machine learning model has been optimized and / or tuned for all identified tasks and / or output types, training engine 122 terminates the process of retraining the trained machine learning model.

[0071] Figure 5 This is a flowchart of method steps for determining visual properties of cellular structures depicted in a label-free image, according to various embodiments. Although the method steps are combined... Figure 1-2 This disclosure describes a system, but those skilled in the art will understand that any system configured to perform some or all of the method steps in any order is within the scope of this disclosure.

[0072] As shown in the figure, in step 502, execution engine 124 matches one or more attributes to be measured from one or more objects depicted in the unlabeled image with a machine learning model. For example, execution engine 124 can match one or more object types ( For example (cell line, cell type, organelle type, subcellular structure type, tissue type, etc.), visual attributes of one or more objects ( For example The properties (shape, texture, quantity, size, distribution, etc.) and / or other attributes associated with one or more objects are matched with a machine learning model that has been trained, tuned and / or optimized to generate these properties in an image of stained cells given a corresponding unlabeled image.

[0073] In step 504, execution engine 124 executes the machine learning model to convert the unlabeled image into a predicted output image corresponding to an image of stained cells depicting the one or more objects. For example, execution engine 124 may use U-Net, V-Net, and / or another type of fully convolutional neural network to convert the unlabeled image into an output image.

[0074] In step 506, execution engine 124 extracts the one or more attributes from the output image. For example, execution engine 124 may use additional machine learning models, image processing techniques, and / or other techniques to quantitatively measure the one or more attributes from the output image. Execution engine 124 may also perform additional analyses on the one or more measured attributes to assess phenotypes, diseases, health conditions, age, effects of perturbations, toxicity, and / or other characteristics associated with the one or more objects depicted in the unlabeled image.

[0075] In summary, the disclosed technique trains and executes multiple machine learning models to convert label-free images of biological samples into stained cell images of the same biological sample. Initially, a first machine learning model is trained to generate a prediction of a stained cell image of the same biological sample given an input including a label-free image of the biological sample. For example, the first machine learning model can be pre-trained using a reconstruction loss calculated between the stained cell image and the output image generated by the first machine learning model from the corresponding label-free image. After the initial training of the model is complete, the first machine learning model is subsequently retrained and / or tuned using different losses and / or additional pairs of label-free and stained cell images. This retraining and / or tuning is used to generate one or more additional trained machine learning models optimized for various properties of the structures or components depicted in the image. For example, a pre-trained first machine learning model can be retrained using segmentation-based loss, structural similarity loss, and / or other types of loss to generate an additional, tuned machine learning model that, given an input including the corresponding unlabeled image, can accurately reconstruct the shape, texture, size, number, distribution, and / or other visual attributes of organelles and / or other types of structures depicted within a cell in a stained cell image.

[0076] After training, one or more trained machine learning models can be used to convert additional label-free images into corresponding stained cell images. For example, one or more types of attributes to be determined from a biological sample can be matched to one or more trained machine learning models tuned and / or optimized for those attributes. One or more label-free images of a biological sample can be input into one or more trained machine learning models, which are then executed to convert the input label-free images into one or more corresponding stained cell images. Quantitative analysis can then be performed on one or more of the generated stained cell images to measure and / or characterize these attributes of the structures and / or components depicted in the generated stained cell images.

[0077] Automated single-cell and lineage tracking in in vivo transmission light microscopy presents significant challenges, especially in long-duration experiments (days or weeks). However, tasks involving cell nucleus detection and tracking using images acquired through DNA staining are simpler. Virtual staining is an efficient and entirely computationally-based approach that uses models trained in the simplest scenario (stained cells) to address the tracking problem in timelapse images acquired solely through transmission light microscopy. Because this solution is entirely computationally based, it is a less invasive method for observing longer and more complex experiments necessary to observe biological processes that evolve over weeks, such as cell reprogramming. Similarly, virtual staining can also be used to define landmarks for multimodal registration.

[0078] One advantage of the disclosed technique compared to prior art is its ability to convert label-free images of biological samples into stained cell images that accurately depict various quantitative properties of the same biological sample. In this respect, the disclosed technique can be used to simulate and / or assess the quantitative properties of biological samples in stained cell images without requiring manual staining and subsequent imaging of the biological sample. Furthermore, since the generated stained cell images can be used to determine and / or measure quantitative properties within the biological sample without staining, the disclosed technique allows for the determination and / or control of additional properties of the biological sample using available imaging channels and / or staining agents. Accordingly, the disclosed technique provides a more comprehensive view of the biological sample compared to conventional optical microscopy techniques with limited available staining agents and / or imaging channels for a given biological sample. These technical advantages provide one or more technical improvements over prior art methods.

[0079] 1. In some embodiments, a computer-implemented method for training one or more machine learning models to convert unlabeled images into stained cell images includes: executing a first machine learning model to convert a first set of unlabeled images into a first set of training output images; training the first machine learning model using a first loss computed between the first set of training output images and a first set of stained cell images corresponding to the first set of unlabeled images to generate a first trained machine learning model; executing the first trained machine learning model to convert a second set of unlabeled images into a second set of training output images; and retraining the first trained machine learning model using a second loss computed based on semantic segmentation of the second set of stained cell images and the second set of training output images to generate a second trained machine learning model.

[0080] 2. The computer implementation method as described in Clause 1 further includes: retraining the first trained machine learning model using a third loss associated with the similarity between the second set of stained cell images and the second set of training output images to generate a third trained machine learning model.

[0081] 3. The computer-implemented method as described in any one of Clauses 1-2, further comprising: executing the third trained machine learning model to convert the first unlabeled image into a first stained cell image; and extracting from the first stained cell image one or more texture-based attributes associated with one or more cellular structures depicted in the first unlabeled image.

[0082] 4. The computer implementation method as described in any one of Clauses 1-3, wherein the third loss includes a structural similarity index measure.

[0083] 5. The computer-implemented method of any one of Clauses 1-4, further comprising: executing the second trained machine learning model to convert the first unlabeled image into a first stained cell image; and extracting from the first stained cell image one or more quantitative shape-based attributes associated with one or more cellular structures depicted in the first unlabeled image.

[0084] 6. The computer-implemented method of any one of claims 1-5, further comprising: extracting a set of unlabeled image crop blocks from the first set of unlabeled images; executing the first machine learning model to convert the set of unlabeled image crop blocks into a set of training output image crop blocks; and calculating the first loss based on the set of training output image crop blocks and a set of stained cell image crop blocks corresponding to the set of unlabeled image crop blocks.

[0085] 7. The computer implementation method as described in any one of Clauses 1-6, wherein the second loss includes a weighted cross-entropy loss.

[0086] 8. The computer-implemented method as described in any one of Clauses 1-7, wherein the first loss includes mean square error.

[0087] 9. The computer-implemented method of any one of Clauses 1-8, wherein the first set of unlabeled images and the first set of stained cell images depict a first cell type, and the second set of unlabeled images and the second set of stained cell images depict a second cell type.

[0088] 10. The computer implementation method as described in any one of Clauses 1-9, wherein the first machine learning model comprises a V-Net architecture and one or more dilated convolutions.

[0089] 11. In some embodiments, one or more non-transitory computer-readable media store instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps: executing a first machine learning model to convert a first set of unlabeled images into a first set of training output images; training the first machine learning model using a first loss calculated between cropped blocks of the set of training output images and a first set of stained cell images corresponding to the first set of unlabeled images to generate a first trained machine learning model; executing the first trained machine learning model to convert a second set of unlabeled images into a second set of training output images; and retraining the first trained machine learning model using a second loss associated with structural similarity between the second set of stained cell images and the second set of training output images to generate a second trained machine learning model.

[0090] 12. One or more non-transitory computer-readable media as described in Clause 11, wherein the instructions further cause the one or more processors to perform the steps of: executing the first trained machine learning model to convert a third set of unlabeled images into a third set of training output images; and retraining the first machine learning model using a third loss associated with semantic segmentation of the third set of stained cell images and semantic segmentation of the third set of training output images to generate a third trained machine learning model.

[0091] 13. One or more non-transitory computer-readable media as described in any one of Clauses 11-12, wherein the instructions further cause the one or more processors to perform the steps of: executing the second trained machine learning model to convert a first unlabeled image into a first stained cell image; executing the third trained machine learning model to convert the first unlabeled image into a second stained cell image; extracting from the first stained cell image one or more texture-based attributes associated with one or more cell structures depicted in the first unlabeled image; and extracting from the second stained cell image one or more shape-based attributes associated with the one or more cell structures depicted in the first unlabeled image.

[0092] 14. One or more non-transitory computer-readable media as described in any one of Clauses 11-13, wherein the third loss includes weighted cross-entropy loss.

[0093] 15. One or more non-transitory computer-readable media as described in any one of Clauses 11-14, wherein the instructions further cause the one or more processors to perform the steps of: extracting a set of unlabeled image crop blocks from the second set of unlabeled images; executing the first trained machine learning model to convert the set of unlabeled image crop blocks into a set of training output image crop blocks; and calculating a second loss based on the set of training output image crop blocks and a set of stained cell image crop blocks corresponding to the set of unlabeled image crop blocks.

[0094] 16. One or more non-transitory computer-readable media as described in any one of Clauses 11-15, wherein the first set of unlabeled images and the first set of stained cell images depict objects of a first type, and the second set of unlabeled images and the second set of stained cell images depict objects of a second type.

[0095] 17. One or more non-transitory computer-readable media as described in any one of Clauses 11-16, wherein the first type of object and the second type of object correspond to at least one of different cell types, different cell lines, different organelle types, or different types of subcellular structures.

[0096] 18. One or more non-transitory computer-readable media as described in any one of Clauses 11-17, wherein the first loss includes mean squared error and the second loss includes a structural similarity index measure.

[0097] 19. One or more non-transitory computer-readable media as described in any one of Clauses 11-18, wherein the first machine learning model includes one or more dilated convolutions contained in the upsampling path of the V-Net architecture.

[0098] 20. In some embodiments, a system includes: one or more memories storing instructions; and one or more processors coupled to the one or more memories, wherein, when the instructions are executed, the one or more processors are configured to perform the following steps: matching one or more attributes measured from one or more objects depicted in an unlabeled image with a machine learning model, wherein the machine learning model is trained using a first loss associated with the reconstruction of a set of training stained cell images and a second loss associated with the one or more attributes; executing the machine learning model to convert the unlabeled image into an output image corresponding to a prediction of stained cell images depicting the one or more objects; and extracting the one or more attributes from the output image.

[0099] Any and all combinations of any claim element recited in any claim and / or any element described in this application, in any manner, fall within the scope of the invention and protection contemplated.

[0100] Various embodiments have been presented for illustrative purposes but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

[0101] Aspects of this embodiment may be embodied as a system, method, or computer program product. Accordingly, aspects of this disclosure may take the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which are generally referred to herein as a “module,” a “system,” or a “computer.” Furthermore, any hardware and / or software technology, process, function, component, engine, module, or system described in this disclosure may be implemented as a circuit or set of circuits. Moreover, aspects of this disclosure may take the form of a computer program product embodied in one or more computer-readable media having computer-readable program code embodied thereon.

[0102] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may, for example, be, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media will include the following: an electrical connection having one or more wires, a portable computer floppy disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable optical disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the context of this document, a computer-readable storage medium can be any tangible medium capable of containing or storing a program for use by or in connection with an instruction execution system, apparatus, or device.

[0103] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine. When executed via the processor of a computer or other programmable data processing apparatus, these instructions enable the implementation of the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, an application-specific processor, or a field-programmable gate array (FPGA).

[0104] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, code segment, or portion of code, containing one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in some alternative implementations, the order in which the functions marked in the blocks are executed may differ from the order shown in the figures. For example, two blocks shown consecutively in the figures may actually execute substantially simultaneously, or, depending on the functions involved, these blocks may sometimes execute in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system or a combination of dedicated hardware and computer instructions that implements the specified functions or actions.

[0105] Although the foregoing relates to embodiments of the present disclosure, other and further embodiments of the present disclosure may be devised without departing from its essential scope, and the scope of the present disclosure is defined by the appended claims.

Claims

1. A computer-implemented method for training one or more machine learning models to convert unlabeled images into stained cell images, the method comprising: Execute the first machine learning model to convert the first set of unlabeled images into the first set of training output images; The first machine learning model is trained using a first loss calculated between the first set of training output images and a first set of stained cell images corresponding to the first set of unlabeled images, to generate a first trained machine learning model. The first trained machine learning model is executed to convert the second set of unlabeled images into the second set of training output images; as well as The first trained machine learning model is retrained using a second loss calculated based on semantic segmentation of the second set of stained cell images and the second set of training output images to generate a second trained machine learning model.

2. The computer implementation method as described in claim 1, further comprising: The first trained machine learning model is retrained using a third loss associated with the similarity between the second set of stained cell images and the second set of training output images to generate a third trained machine learning model.

3. The computer implementation method as described in claim 2, further comprising: The third trained machine learning model is executed to convert the first unlabeled image into a first stained cell image; as well as Extract one or more texture-based attributes from the first stained cell image that are associated with one or more cell structures depicted in the first unlabeled image.

4. The computer implementation method as described in claim 2, wherein, The third loss includes a structural similarity index measure.

5. The computer implementation method as described in claim 1, further comprising: The second trained machine learning model is executed to convert the first unlabeled image into a first stained cell image; as well as Extract one or more quantitative shape-based attributes from the first stained cell image that are associated with one or more cell structures depicted in the first unlabeled image.

6. The computer implementation method as described in claim 1, further comprising: Extract a set of unlabeled image cropping blocks from the first set of unlabeled images; The first machine learning model is executed to convert the set of unlabeled image crop blocks into a set of output image crop blocks for training. as well as The first loss is calculated based on the set of cropped output images used for training and a set of cropped stained cell images corresponding to the set of cropped unlabeled images.

7. The computer implementation method as described in claim 1, wherein, The second loss includes the weighted cross-entropy loss.

8. The computer implementation method as described in claim 1, wherein, The first loss includes mean square error.

9. The computer implementation method as described in claim 1, wherein, The first set of unlabeled images and the first set of stained cell images depict a first cell type, and the second set of unlabeled images and the second set of stained cell images depict a second cell type.

10. The computer implementation method as described in claim 1, wherein, The first machine learning model includes a V-Net architecture and one or more dilated convolutions.

11. One or more non-transitory computer-readable media storing instructions, which, when executed by one or more processors, cause the one or more processors to perform the following steps: Execute the first machine learning model to convert the first set of unlabeled images into the first set of training output images; The first machine learning model is trained using a first loss calculated between the first set of training output images and a first set of stained cell images corresponding to the first set of unlabeled images, to generate a first trained machine learning model. The first trained machine learning model is executed to convert the second set of unlabeled images into the second set of training output images; as well as The first trained machine learning model is retrained using a second loss associated with the structural similarity between the second set of stained cell images and the second set of training output images to generate a second trained machine learning model.

12. One or more non-transitory computer-readable media as claimed in claim 11, wherein, The instructions further cause the one or more processors to perform the following steps: Execute the first trained machine learning model to convert the third set of unlabeled images into the third set of training output images; and The first machine learning model is retrained using a third loss associated with semantic segmentation of the third set of stained cell images and semantic segmentation of the third set of training output images to generate a third trained machine learning model.

13. One or more non-transitory computer-readable media as claimed in claim 12, wherein, The instructions further cause the one or more processors to perform the following steps: The second trained machine learning model is executed to convert the first unlabeled image into a first stained cell image; The third trained machine learning model is executed to convert the first unlabeled image into a second stained cell image; Extract one or more texture-based attributes from the first stained cell image that are associated with one or more cell structures depicted in the first unlabeled image; as well as Extract one or more shape-based attributes from the second stained cell image that are associated with the one or more cell structures depicted in the first unlabeled image.

14. One or more non-transitory computer-readable media as claimed in claim 12, wherein, The third loss includes weighted cross-entropy loss.

15. One or more non-transitory computer-readable media as claimed in claim 11, wherein, The instructions further cause the one or more processors to perform the following steps: Extract a set of unlabeled image cropping blocks from the second set of unlabeled images; The first trained machine learning model is executed to convert the set of unlabeled image crop blocks into a set of training output image crop blocks; as well as The second loss is calculated based on the set of output image cropping blocks used for training and a set of stained cell image cropping blocks corresponding to the set of unlabeled image cropping blocks.

16. The one or more non-transitory computer-readable media as claimed in claim 11, wherein, The first set of unlabeled images and the first set of stained cell images depict a first type of object, and the second set of unlabeled images and the second set of stained cell images depict a second type of object.

17. One or more non-transitory computer-readable media as claimed in claim 16, wherein, The first type of object and the second type of object correspond to at least one of different cell types, different cell lines, different organelle types, or different types of subcellular structures.

18. One or more non-transitory computer-readable media as claimed in claim 11, wherein, The first loss includes mean squared error, and the second loss includes a structural similarity index measure.

19. One or more non-transitory computer-readable media as claimed in claim 11, wherein, The first machine learning model includes one or more dilated convolutions contained in the upsampling path of the V-Net architecture.

20. A system comprising: One or more memories that store instructions; as well as One or more processors, coupled to the one or more memories, are configured to perform the following steps when the instructions are executed: One or more attributes measured from one or more objects depicted within an unlabeled image are matched with a machine learning model, wherein the machine learning model is trained using a first loss associated with the reconstruction of a set of training stained cell images and a second loss associated with the one or more attributes. The machine learning model is executed to convert the label-free image into an output image corresponding to a prediction of an image of stained cells depicting the one or more objects; and Extract one or more attributes from the output image.

Images