Method and system for acquiring fluorescence images of living cell biological samples

Abstract

A method is disclosed for acquiring a single, focused, two-dimensional projection image of a living, three-dimensional cell culture sample using a fluorescence microscope. To create one or more two-dimensional images of fluorescence intensity integrated across the Z dimension, one or more long-exposure "Z-sweep" images are acquired while moving the camera's Z focal plane through the sample, i.e., via a single or series of consecutive acquisitions. This acquisition method is much faster than Z-stack methods, allowing for higher throughput and reducing the risk of overexposing the sample to fluorescent light. The long-exposure Z-sweep images are then input into a neural network trained to generate high-quality (focused) two-dimensional projection images of the sample. These high-quality projection images allow for biologically relevant analytical metrics to be obtained to describe the fluorescent signal using standard image analysis techniques, such as fluorescent object count and other fluorescence intensity metrics (e.g., mean intensity, texture, etc.).

Description

Technical Field

[0002] ,

[0005] , , , , , ,

[0004] ,

[0003] , , ,

[0006] , , ,

[0001] Cross - reference to related applications This application claims priority to U.S. Non - Provisional Patent Application No. 16 / 935,326, filed July 22, 2020, which is incorporated herein by reference.

[0002] The present disclosure relates to the field of methods and systems for acquiring fluorescence images of live cell biological samples, such as three - dimensional (3D) cell cultures including organoids and tumor spheroids.

Background Art

[0003] The use of live cell samples extends across a wide range of investigative fields, including immunooncology, oncology, metabolism, neuroscience, immunology, infectious diseases, toxicology, stem cells, cardiology, and inflammation. In these investigative fields, studies are conducted on cell health and proliferation, cell function, cell movement and morphology, including studies of complex immune - tumor cell interactions, synaptic activity, metabolism in cancer cells, etc.

[0004] Fluorescence microscopy is a method for observing the photoelectron emission spectrum from a sample containing a live cell sample. To observe the fluorescence from a sample, an optical system including an incident light source and an optical filter with out - of - band rejection parameters is required. This setup enables researchers to perform real - time analysis of the fluorescence quality of the sample. A fluorescence microscope including imaging functionality (by a camera or imaging devices of various designs) is today part of current live cell investigation laboratories.

[0005] In fluorescence microscopy, it is common for the sample to be treated, stained, or chemically formulated with one or more fluorophores. A fluorophore is a microscopic molecule that can be a protein, a small organic compound, an organic dye, or a synthetic polymer that absorbs light of a specific wavelength and emits light of a longer wavelength. Certain semiconducting metal nanoparticles are also eligible as fluorophores and emit light related to their geometric configuration.

[0006] Living cell biological samples can be microscopically imaged with a fluorescence microscope in various ways, typically at some magnification such as 10x or 20x, to evaluate the growth, metabolism, morphology, or other properties of the sample at one or more time points in time. This microscopic imaging can include fluorescence imaging, in which a phosphor in the sample is excited by light at the phosphor's excitation wavelength and emits fluorescence at the phosphor's emission wavelength. In epifluorescence imaging, the excitation light is provided through the same objective lens used to collect the emitted light.

[0007] The overall trend in biological research is shifting towards the importance of studying 3D models (e.g., tumor spheroids, organoids, 3D cell cultures) rather than their two-dimensional relatives (e.g., a single image at a given focal plane), because 3D models are believed to better replicate the physiological conditions present in true in vivo systems.

[0008] A standard method for obtaining 3D information of a sample using a fluorescence microscope or confocal microscope is the stepwise "Z-stack" method, in which a series of images of the sample are each captured at a specific depth or Z-coordinate of the sample in a three-dimensional coordinate system. This methodology is shown in Figure 2C of this disclosure and will be described in more detail later. The resulting series of images can then be manipulated using specialized software to project or combine the images into a single 2D image. One method for performing this Z-projection is described in U.S. Patent Application No. 16 / 854,710, “Image Processing and Segmentation of Sets of Z-Stacked Images of Three-Dimensional Biological Samples,” by Timothy Jackson et al., which is incorporated herein by reference, assigned to the assignee of the present invention, and filed on April 21, 2020. An example of an open-source image analysis software package that projects Z-stack images onto 2D images is ImageJ, described in Schneider, CA, Rasband, WS, and Eliceiri, KW (2012), NIH Image to ImageJ: 25 years of image analysis, Nature Methods, 9(7), pp. 671-675. Several Z projection schemes are available, including maximum projection, mean projection, Sobel filter-based projection, and wavelet-based projection. [Prior art documents] [Patent Documents]

[0009] [Patent Document 1] U.S. Patent Application No. 16 / 854,710 [Patent Document 2] U.S. Patent Application No. 16 / 854,756 [Non-patent literature]

[0010] [Non-Patent Document 1] Schneider, CA, Rasband, WS, and Eliceiri, KW (2012), NIH Image to ImageJ: 25 years of image analysis, Nature methods, 9(7), pp. 671–675. [Non-Patent Document 2] "Generative Adversarial Nets" by Goodfellow et al., in Advances in Neural Information Processing Systems 27, edited by Z. Ghahramani, M. Welling, C. Cortes, ND Lawrence, and KQ Weinberger, Curran Associates, Inc., 2014, pp. 2672-2680. [Non-Patent Document 3] Ronneberger et al., "U-net: Convolutional networks for biomedical image segmentation," published in Springer and at the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 234-241, also published as arXiv:1505.04597 (2015). [Non-Patent Document 4] Zhu, Jun-Yan, et al., "Unpaired image-to-image translation using cycle-consistent adversarial networks," Proceedings of the IEEE international conference on computer vision (2017), also published as arXiv:1703.10593 (2017). [Non-Patent Document 5] https: / / www.essenbioscience.com / en / applications / cell-health-viability / spheroids / [Non-Patent Document 6] "Image-to-Image Translation with Conditional Adversarial Networks" by Philip Isola et al., arXiv.1611.07004 [cv.cs] (2016) and "Least Squares Generative Adversarial Networks" by Xudong Mao et al., arXiv.1611.04076 [cv.cs] (2016) [Overview of the project] [Problems that the invention aims to solve]

[0011] However, the Z-stack method has several inherent challenges, which can be particularly problematic for live cell samples. Firstly, the requirement to fine-tune different acquisition parameters, including exposure time, start Z position, end Z position, and Z step (increment ΔZ between image capture positions), requires prior knowledge of the sample and is only possible for expert-level users. Secondly, the time required to acquire such images is extremely slow, limiting throughput, and the sample is exposed to a large amount of fluorescence from the light source, which can lead to photobleaching or phototoxicity, which is highly undesirable in live cell research. Finally, acquiring 3D Z-stacks in this case requires advanced software for visualizing and analyzing the data. [Means for solving the problem]

[0012] This document describes methods and systems, including image acquisition schemes and image processing procedures, that utilize deep learning to address the imaging challenges described above. In particular, the methods described herein are for acquiring a single, focused 2D projection of a 3D living cell biological sample at high throughput, leveraging recent advances in deep learning and convolutional neural networks.

[0013] Instead of using the known Z-stack method, one or more long-exposure "Z-sweep" images are obtained. This involves acquiring a single or a series of consecutive long-exposure images, where the camera in the fluorescence microscope is exposed to the sample while the Z-focal plane is moved across the sample, thereby integrating the fluorescence intensity from the sample over the Z dimension. This acquisition method is much faster than Z-stacking, thus enabling higher throughput and reducing the risk of the sample being exposed to too much fluorescence, thereby avoiding problems with phototoxicity and photobleaching. The long-exposure images are then fed into a trained neural network, trained to produce high-quality, focused 2D projection images representing the projection of the 3D sample. These high-quality 2D projection images allow for the acquisition of biologically relevant analytical metrics describing the fluorescence signal using standard image analysis techniques, such as the number of fluorescent objects and other fluorescence intensity metrics (e.g., average intensity, texture, etc.).

[0014] In one particular embodiment, a method is provided for generating a focused two-dimensional projection of a fluorescence image of a three-dimensional living cell sample. The method includes the steps of: acquiring one or more long-exposure images of a sample with a camera by moving the focal plane of the camera through the sample in the Z direction, thereby causing the camera to integrate the fluorescence intensity from the sample over the Z dimension; feeding the one or more long-exposure images to a neural network model trained with a plurality of training images; and generating a focused two-dimensional projection image by the neural network model.

[0015] Several different types of neural network models are possible. In one embodiment, the neural network model is a convolutional neural network, and in another embodiment, it is an encoder-decoder-based supervised model. Encoder-decoder-based supervised models can take the form of U-nets, as described in literature such as those described below. Alternatively, the neural network model is trained using adversarial techniques, such as in the case of generative adversarial networks (GANs). See “Generative Adversarial Nets” by Goodfellow et al., Advances in Neural Information Processing Systems 27, edited by Z. Ghahramani, M. Welling, C. Cortes, ND Lawrence, and KQ Weinberger, Curran Associates, Inc., 2014, pp. 2672–2680. This adversarial technique means that two neural networks are trained simultaneously, with the generator predicting a high-quality, in-focus projection image and the discriminator attempting to distinguish between the predicted in-focus projection image and the true or actually acquired image. The advantage of this method is that, in adversarial training, it can produce more realistic outputs compared to regular supervised training, due to the generator attempting to deceive the discriminator. Conditional GANs are an example that offers another possibility, although the generator is conditioned on one or more long-exposure images. Neural network models can also be trained using cycle-consistency loss methods, such as GANs or CycleGANs (GANs with cycle-consistency loss), which are described in the literature, allowing them to be trained on non-paired data. Non-paired data means that there are Z-sweep images and in-focus projection images, but not all pairs necessarily have one-to-one pixel consistency. The advantage of cycle-consistency methods is that they do not require perfect superposition between the Z-sweep image and the in-focus projection image. Imperfect superposition can occur, for example, when there is slight movement in the XY plane as the camera moves along the Z axis.

[0016] Three different methods or paradigms are assumed for generating one or more long-exposure Z-sweep images. One configuration yields a single long-exposure Z-sweep image, which is referred to below as "paradigm 1". Alternatively, a series of long-exposure sequential Z-sweep images are obtained and then added together, which is referred to below as "paradigm 2". Another alternative form yields a series of long-exposure sequential Z-sweep images that are not added together, which is referred to below as "paradigm 3".

[0017] In another embodiment, a method for training a neural network model to generate a focused 2D projection image of a living cell sample is disclosed. The method includes the step of obtaining a training set in the form of multiple images. Such images may be pairs of images or unpaired images. In practice, having pairs is advantageous, especially when the neural network model uses supervised training. However, it is possible to use unpaired images, for example, in cycle-consistency loss training or generative adversarial networks.

[0018] The training images may include (1) one or more long-exposure Z-sweep images of a living cell sample obtained by moving the focal plane of a camera through the sample in the Z direction, thereby causing the camera to integrate the fluorescence intensity from the sample over the Z dimension, and (2) associated 2D Z-stack projected ground truth images, wherein the ground truth image is obtained from a set of Z-stack images of the same living cell sample, each of the images in the Z-stack images is obtained at a different Z-focal plane position of the sample, and the Z-stack images are combined using a Z-projection algorithm. The method further includes the step of performing a neural network training procedure from the training set, thereby generating a model that is trained to ultimately generate 2D projected images of a living cell sample from input in the form of one or more long-exposure Z-sweep images.

[0019] As described above, the long-exposure Z-sweep images in the training set can take the form of a single Z-sweep image or a set of consecutive Z-sweep images optionally added together, or some combination of any of the three acquisition paradigms.

[0020] In yet another aspect, a live-cell imaging system for use with a sample holding device adapted to hold a live-cell sample, such as a microwell plate, is described. The system includes a fluorescence microscope having one or more excitation light sources, one or more objective lenses, and a camera operable to obtain fluorescence images from a live-cell sample held within the sample holding device. The fluorescence microscope includes a motor system configured to move the fluorescence microscope in the Z-direction with respect to the sample holding device such that the camera acquires one or more long-exposure Z-sweep images of the sample, wherein the one or more long-exposure Z-sweep images are obtained by moving the focal plane of the camera in the Z-direction through the sample, and the system further includes a processing unit including a neural network model trained to generate a focused two-dimensional projection of the fluorescence image of the live-cell sample from the one or more long-exposure Z-sweep images.

[0021] In yet another aspect, a method for generating a training set for training a neural network is provided. The method includes: (a) obtaining, with a camera, one or more long-exposure fluorescence images of a three-dimensional sample by moving the focal plane of the camera in the Z-direction through the sample, whereby the camera integrates the fluorescence intensity from the sample over the Z-dimension; (b) generating a ground truth image of the same sample from one or more different images of the sample obtained by the camera; (c) repeating steps (a) and (b) for a number of different samples; and (d) supplying, as a training set for training a neural network, the images obtained by performing steps (a), (b), and (c).

[0022] The method disclosed herein has several advantages. (1) This method makes it possible to obtain biologically relevant analytical measurements from high-quality projection images using standard image analysis techniques such as fluorescence object count values ​​or other fluorescence intensity measurements. (2) One or more long exposure images are a true representation of fluorescence in the sample integrated over the Z dimension, resulting in the generation of true and accurate data. (3) The method of this disclosure can be implemented without any changes to the hardware of a conventional motorized fluorescence microscope. (4) The workflow of this disclosure is designed to be compatible with conventional wide-field fluorescence microscopy, but can be similarly applied to a rotating-plate confocal microscope to similarly improve the throughput of its imaging means. (5) This method provides a single, high-quality 2D projection image as output, thereby eliminating the burden on users of having complex software and analysis tools to handle 3D datasets. The 2D projection image can be input into standard 2D image visualization, segmentation, and analysis pipelines, such as those currently implemented in the latest fluorescence microscopy platforms on the market today.

[0023] In yet another embodiment, a computer-readable medium is provided for storing non-transient instructions for a living cell imaging system including a camera and a processing unit implementing a neural network model, the instructions causing the system to perform a combination of the methods of the present disclosure, for example, the steps of capturing one or more long-exposure Z-sweep images of a sample, supplying the images to a trained neural network model, and generating a two-dimensional projection image by the trained neural network model. [Brief explanation of the drawing]

[0024] [Figure 1] This is a schematic diagram of a method and system for acquiring 3D fluorescence images of living cell biological samples. [Figure 2A]This diagram illustrates a Z-sweep image acquisition method that can be used with the motorized fluorescence microscope shown in Figure 1. [Figure 2B] This is a diagram illustrating a method for acquiring continuous Z-sweep images. [Figure 2C] This is a diagram illustrating a conventional method for acquiring Z-stack images. [Figure 3] Figure 3 illustrates a method for generating a training dataset for training a neural network model. Figure 3 also illustrates how to obtain a new input image for inference using the trained neural network model and an output in the form of a high-quality 2D projection image. [Figure 4] Figure 3 is a flowchart of the model training procedure for training the neural network model. [Figure 5] This diagram illustrates an embodiment of a network where a trained neural network model is implemented on a remote computer and communicates with a remote workstation that receives fluorescence images from a connected fluorescence microscope. [Figure 6] This figure shows a high-quality, in-focus 2D projection image derived from the long-exposure image acquisition method shown in Figure 2A or Figure 2B, using a trained neural network model. [Figure 7] This is a diagram of a living cell imaging system for biological samples filled in sample wells of a microwell plate. The system includes a fluorescence optics module for acquiring fluorescence images of the sample in the microwell plate using the features of the present disclosure. [Figure 8] Figure 7 is a schematic diagram of the fluorescence optics module positioned within the living cell imaging system. [Figure 9] Figure 9 shows a set of three examples of input images and model outputs compared to ground truth projection, demonstrating the usefulness of this method. In Figure 9, the neural network model is a trained conditional generative adversarial network (GAN), and the input images were generated under the second image acquisition paradigm in Figure 3. [Figure 10]Figure 9 shows a high-level architecture of the conditional GAN ​​used to generate the model output (predicted projection). [Modes for carrying out the invention]

[0025] Referring here to Figure 1, a living cell biological sample (which can take the form of any type of living cell sample used in current investigations, such as living three-dimensional cell cultures, as described in the background art) is placed on or in a suitable holding structure 12, such as a microwell plate or a glass slide. One or more fluorescent reagents (phosphors) can be added to the living cell sample 10, as shown in 14, and the type or properties of the fluorescent reagents depend, of course, on the details of the sample 10 and the nature of the investigation being performed. The holding structure 12 containing the sample 10 and reagents is then fed into an electric fluorescence microscope 16 having a camera, which can take the form of any conventional fluorescence microscope with imaging capabilities known in the art or available from many manufacturers, including the assignee of the present invention.

[0026] A fluorescence microscope 16 is used to generate one or more long-exposure Z-sweep images 18 (or a set of such images) according to the method shown in Figure 2A or Figure 2B. These long-exposure Z-sweep images of a 3D living cell biological sample 10 are obtained by moving the focal plane of the camera in the microscope through the sample in the Z direction, as shown in 25 in Figure 2A and 25 in Figure 6, where the camera integrates the fluorescence intensity from the sample over the Z dimension. The long-exposure Z-sweep images yield a dataset 30 as shown in Figures 2A and 6. This dataset 30 is then fed as input to a computer or processor implementing a trained neural network model 22 (Figure 1). This model is trained on a large number of training images, as described below, to produce high-quality, in-focus 2D projection images, as shown in 140 in Figures 3 and 6. This 2D projection image 140 (Figures 3 and 6) contains 3D fluorescence information (because the image was obtained from the integration of fluorescence intensity over the Z dimension), but in the form of a projection onto a 2D focused image 140. The resulting focused 2D projection image 140 (Figures 3 and 6) is then used in any of the standard, known image analysis techniques, such as fluorescence object count values ​​or other fluorescence intensity measurements.

[0027] In Figure 2A, as indicated by arrow 25, a single long-exposure Z-sweep image is acquired by continuously exposing the camera in the fluorescence microscope as the focal plane is moved in the Z direction through the sample, resulting in a two-dimensional image dataset 30. This is "paradigm 1" in Figure 3. The image dataset 30 has limited usefulness in image analysis techniques due to the problem of being out of focus. Therefore, as shown in Figures 3 and 6 at 140, a trained neural network model is used to transform the out-of-focus images into more useful focused projection images. Further examples are shown in Figure 9 and discussed later in this document.

[0028] An alternative method for acquiring long-exposure Z-sweep images is shown in Figure 2B. In this alternative form, image acquisition consists of the step of acquiring a series of consecutive long-exposure Z-sweep images numbered 1, 2, 3, and 4 in Figure 2B, where, while the focal plane moves through increments of the Z dimension, the camera in the microscope integrates the fluorescence intensity over the exposed Z dimension and captures a series of four corresponding images or image datasets shown as 1, 2, 3, and 4 in Figure 2B. The series of images or image datasets are then added together to produce a single resulting long-exposure Z-sweep dataset or image 30. This is “Paradigm 2” in Figure 3. Compared to the method in Figure 2A, this image acquisition technique may have certain advantages, including avoiding image saturation, which can remove out-of-focus fluorescence and ultimately result in more accurate measurements of fluorescence intensity, and allowing the application of conventional algorithms or deep learning-based 2D fluorescence deconvolution algorithms to each smaller sweep (1, 2, 3, 4) before addition.

[0029] A third embodiment for acquiring long-exposure Z-sweep images, indicated as "Paradigm 3" in Figure 3, is also possible, which is a variation of the procedure in Figure 2B. Images 1-4 are acquired in Figure 2B as described above, but no addition is performed. The four images are passed to a trained neural network, from which a single corresponding in-focus image is predicted. In this embodiment, the four images can be combined into a 4-channel image in this case. The advantages listed above also apply to this embodiment. Furthermore, this embodiment may have the advantage that it is easier for the neural network model to predict a well-focused projection image from the images acquired with this paradigm.

[0030] Figure 2C illustrates a conventional Z-stack image acquisition method, in which images of sample 10 are taken at different focal planes such as positions A, B, C, and D, resulting in the corresponding image datasets A, B, C, and D shown in the figure. Note that in this method, the Z-dimensional coordinates are fixed or do not move during the camera exposure time.

[0031] Figure 3 shows in more detail the training of the convolutional neural network model used to generate the 2D projected image 140 in Figure 6 from input in the form of one or more long-exposure Z-sweep images 30.

[0032] The training dataset for model input is prepared by generating a Z-stack 104 (procedure in Figure 2C) and a Z-sweep (100) (procedure in either Figure 2A or Figure 2B, one of the three paradigms discussed above) of the same 3D live cell training sample. In addition, a ground truth image 108 of the same training sample is obtained from the Z-stack 104 and the Z-projection algorithm 106. Thus, the Z-stack 104 is used to generate a high-quality 2D projection 108 that will act as the "ground truth" for model training, using one of the Z-projection algorithms, such as the ImageJ algorithm mentioned earlier. The Z-sweep images 102, 102A, or 105 are used as input for model training along with the associated ground truth 108. Using these pairs of data (actually thousands of such pairs), we train a convolutional neural network (CNN) model, such as the one shown in 110, to enable the network to learn how to generate high-quality 2D projection images (140, Figure 6) from long-exposure Z-sweep images (30, Figure 6). The trained neural network model is shown in 22 in Figures 1 and 3. As mentioned above, the input to training the model 110 may be pairs of images, as precisely stated in the supervised model training exercise, for example, but as explained above, the model input does not have to be pairs, as in the case of cycle-consistency loss or generative adversarial network models, using just a set of images from two regions (ground truth Z-projection images and Z-sweep).

[0033] The neural network model 22 may be a supervised model, such as an encoder-decoder based model, such as U-net. See Ronneberger et al., "U-net: Convolutional networks for biomedical image segmentation," published in Springer and at the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 234-241, also published as arXiv:1505.04597 (2015), the contents of which are incorporated herein by reference. This supervised model is trained to predict high-quality projection images directly from corresponding long-exposure Z-sweep images.

[0034] Neural network models 22 can also be designed and trained using adversarial methods, such as GANs. See "Generative Adversarial Nets" by Goodfellow et al. in Advances in Neural Information Processing Systems 27, edited by Z. Ghahramani, M. Welling, C. Cortes, ND Lawrence, and KQ Weinberger, Curran Associates, Inc., 2014, pp. 2672-2680. Here, a generator in one network is trained against a discriminator in another network. The discriminator is tasked with distinguishing between a true high-quality projection and the output of the generator, while the generator is tasked with making its output indistinguishable from the true projection (i.e., ground truth).

[0035] Another alternative model architecture is a conditional GAN, which will be described in more detail in conjunction with Figure 10.

[0036] The trained neural network model 22 can also be trained using a cycle-consistent loss method, such as CycleGAN. This means that unpaired Z-sweep images are transformed into high-quality projections and then back into Z-sweep images, and the network is trained to minimize the cycle-back reconstruction error. See also Zhu, Jun-Yan et al., "Unpaired image-to-image translation using cycle-consistent adversarial networks," Proceedings of the IEEE international conference on computer vision (2017), arXiv:1703.10593 (2017), the contents of which are incorporated herein by reference. The advantage of using cycle consistency is that such training does not require a perfect superposition between the Z-sweep images and the high-quality projections. Cycle consistency also opens up the possibility of training with unpaired data from two regions (in this case, the Z-sweep images and the Z-stacked projection images).

[0037] Once a neural network model is trained on a set of training samples (perhaps hundreds or thousands of such 3D living cell samples), it is no longer necessary to collect Z-stacked images according to Figure 2C, which is a slower process and damages the samples during inference. Rather, one or more high-throughput, long-exposure Z-sweep images are collected using the techniques of Figure 2A or Figure 2B, which are then input into the trained model 22 to generate high-quality 2D projection images 140. In one embodiment shown in 120 ("paradigm 1"), only one Z-sweep is performed, and the resulting image dataset 132 is input into the trained model 22. In another embodiment shown in Figure 3 as "paradigm 2", a series of sequential Z-sweeps are performed as shown in 134 (technique of Figure 2B), generating images 136 which are added to generate an image dataset 132A. The sequential Z-sweep images 136 have several advantages without sacrificing acquisition speed: (1) they avoid image saturation, and (2) they allow the application of conventional or deep learning-based 2D fluorescence deconvolution algorithms to each smaller sweep before addition, thereby removing out-of-focus fluorescence and ultimately resulting in more accurate measurements of fluorescence intensity. The resulting images 136 are optionally subjected to fluorescence deconvolution and then to addition as shown in 138, and the resulting image dataset 132A is then fed to the trained model 22.

[0038] In the third alternative form, "Paradigm 3," shown in Figure 3, a set of 135 consecutive Z-sweeps are performed (procedure in Figure 2B), yielding a set of image datasets 132B from each sweep. The image datasets 132B are fed into the trained model 22. In this alternative form, the neural network output can be improved because the image datasets, when combined with a single multi-channel input image as previously described, realize complementary features.

[0039] For model inference, a model trained on input images acquired according to image acquisition paradigm 1, 2, or 3 is used, so that the model inference matches the type of paradigm used to obtain the input images. In other words, for example, if the input images are acquired under paradigm 2 at the time of inference, the inference is performed by a model trained on images acquired under paradigm 2 as well.

[0040] As a result of model inference, the trained model 22 generates model output 140, which is a single, high-quality, focused two-dimensional projection image.

[0041] Model training is described in more detail in Figure 4. The model training process 200 includes the steps of: performing a long-exposure Z-sweep on a living cell 3D training sample (for example, using either the image acquisition technique in Figure 2A or Figure 2B); acquiring a Z-stack image of the same training sample using the technique in Figure 2C; and projecting the Z-stack acquired in step 206 onto a 2D image using a Z-stack projection technique such as ImageJ. This process is repeated for a large number of different training samples, e.g., thousands or tens of thousands of living cell 3D samples, across all expected types of samples used in fluorescence microscopy in current living cell investigations, as shown in 208. A large number of images (from steps 202 and 206) from two regions (Z-sweep images and ground truth images), which are either paired or unpaired data, are then provided as training set 210 and used to train a neural network as shown in 212, either in a supervised learning model training exercise or a self-managed model training exercise, according to the CycleGAN procedure or a generative adversarial network method. The result of this exercise is the trained neural network model 22.

[0042] One possible variant is having several individual models trained using this general technique, each designed for different types of living cell samples, such as those for stem cells, oncology, or brain cells. Each model is trained with hundreds or thousands of (paired or unpaired) images obtained using the procedures described in Figures 3 and 4.

[0043] Referring to Figures 1 and 5, the trained neural network model 22 can be implemented on a processor associated with the workstation 24 or the fluorescence microscope / imaging device 16 itself, or as part of the fluorescence microscope / imaging device 16 itself. Alternatively, the trained model can be implemented on a remote server, as shown by 300 in Figure 5. In this configuration, the fluorescence microscope / imaging device 16 is connected to the workstation 24 via connectivity over the computer network 26 to the server 300. The workstation sends long-exposure Z-sweep images to the server 300. The trained neural network model in the server then performs inference as shown in Figure 3, generates a 2D projection image (140, Figure 6), and sends this 2D projection image back to the workstation 24 via the network 26, where the 2D projection image is displayed to the user or provided as input to an image analysis and evaluation software pipeline running on the workstation 24.

[0044] Fluorescence microscope system One possible implementation of the features of this disclosure is a live cell imaging system 400, which includes a fluorescence microscope for acquiring fluorescence images in a three-dimensional live cell investigation application. Figure 7 shows the live cell imaging system 400 having a housing 410, the whole of which can be placed inside a temperature and humidity controlled incubator (not shown) during use. The live cell imaging system 400 is designed to accept a microwell plate 12, each containing a number of sample holding wells 404, each of which accepts a 3D live cell sample. The system includes a set of fluorescence reagents 406, one or more of which are added to each of the sample wells 404 to enable fluorescence measurements from the live cell sample to be acquired. The system includes an associated workstation 24, which implements image analysis software and includes display features to allow investigators to view the results of live cell experiments performed on the sample. The live cell imaging system 400 includes a tray 408 that slides out of the system, allowing the microwell plate 12 to be inserted onto the tray 408, and then retracts and closes so that the microwell plate 12 is placed inside the live cell imaging system housing 410. The microwell plate 12 remains stationary within the housing, while the fluorescence optics module 402 (see Figure 8) moves relative to the plate 12, acquiring a series of fluorescence images throughout the experiment. The fluorescence optics module 402 implements a long-exposure Z-sweep image acquisition technique (paradigm 1, 2, or 3 as previously described) as shown in Figure 2A and / or Figure 2B, generating an image dataset that is input to a trained neural network model as shown in the inference section of Figure 3.

[0045] Figure 8 is a more detailed optical view of the fluorescent optical module 402 shown in Figure 7. Further details of the fluorescent optical module 402 shown in Figure 8 can be found in U.S. Patent Application No. 16 / 854,756, entitled "Optical module with three or more color fluorescent light sources and methods for use thereof," by Brad Neagle et al., filed on April 21, 2020, which is allocated to the assignee of the present invention and whose description is incorporated herein by reference.

[0046] Module 402 includes LED excitation light sources 450A and 450B that emit light at different wavelengths, such as 453–486 nm and 546–568 nm, respectively. Optical module 402 can also be configured with a third LED excitation light source (not shown) that emits light at a third wavelength, such as 648–674 nm, or even a fourth LED excitation light source that emits light at a fourth different wavelength. Light from LEDs 450A and 450B passes through narrow-band-pass filters 452A and 452B, respectively, which allow light to pass through at specific wavelengths designed to excite phosphors in the sample. Light passing through filter 452A is reflected by dichroic 454A, then by dichroic mirror 454B, and directed towards an objective lens 460, such as a 20x magnifying lens. Light from LED 450B also passes through filter 452B and dichroic mirror 454B, and is directed towards the objective lens 460. The excitation light passing through lens 460 then enters the bottom of sample plate 12 and passes into sample 10. Next, the emission from the phosphor in the sample passes through lens 460, is reflected by mirror 454B, passes through dichroic 454A, passes through narrowband emission filter 462 (filtering out non-fluorescent light), and enters digital camera 464. Digital camera 464 can take the form of a charge-coupled device (CCD) or other types of cameras currently known and used in fluorescence microscopes. Then, while light source 450A or 450B is ON, motor system 418 operates to move all-optical module 402 in the Z dimension, thereby obtaining a long-exposure Z-sweep image (Figure 2A or Figure 2B). It is generally understood that only one optical channel is activated at a time; for example, LED450A is turned on to capture a long-exposure Z-sweep image, then LED450A is turned off and LED450B is activated to capture a second long-exposure Z-sweep image.

[0047] It is understood that the objective lens 460 can be mounted on a turret that can rotate around a vertical axis to obtain a second long-exposure Z-sweep image at a different magnification by placing a second objective lens of a different magnification in the optical path. Furthermore, the motor system 418 can be configured to move in the X and Y directions below the sample plate 12 so that the optical paths of the fluorescence optics module 402 and the objective lens 460 are positioned directly below each of the wells 404 of the sample plate 12, and fluorescence measurements as described can be obtained from each of the retained wells in the plate 12 (and thus each of the living cell samples).

[0048] Details of the motor system 418 for the fluorescent optical module 402 can vary widely and are known to those skilled in the art.

[0049] The operation of the living cell imaging system, which includes a motor system and a camera that cooperate to acquire images of the sample in the sample well, is under program control by a conventional computer or processing unit. This processing unit may implement the trained neural network model shown in Figure 3. In this case, as one possible embodiment, the computer includes memory for storing non-temporary instructions (code) for implementing the method of the present disclosure, including image acquisition of paradigm 1, 2, and / or 3, as well as the previously trained neural network model as described above in relation to Figures 3 and 4. Similarly, the computer or processing unit includes memory for instructions for generating all training images from a set of training samples used for model training, and model training can be performed in the computer of the living cell imaging system or in a remote computing system that implements and trains the model.

[0050] Application examples The methods described herein are useful for generating two-dimensional projection images of other three-dimensional structures found in biological samples such as organoids, tumor spheroids, and cell cultures. As previously mentioned, the use of living cell samples extends to a wide range of research areas, including immuno-oncology, oncology, metabolism, neuroscience, immunology, infectious diseases, toxicology, stem cells, cardiology, and inflammation. These research areas involve studies of cell health and proliferation, cellular function, cell migration, and morphology, including complex immune-tumor cell interactions, synaptic activity, and metabolism in cancer cells. The methods described herein are relevant to all of these applications.

[0051] In particular, the method of this disclosure is relevant to the above-mentioned applications, as it enables high-throughput fluorescence image capture of samples and generates high-quality fluorescence 2D projection images that can be segmented and analyzed to measure how experimental conditions (e.g., drug treatment) affect the health of organoids, tumor spheroids, or other three-dimensional biological structures. Organoids (e.g., pancreatic cell organoids, liver cell organoids, intestinal cell organoids) and tumor spheroids are of particular interest, as their three-dimensional structures more closely mimic the "natural" three-dimensional environment of the cultured cells. Thus, the response of organoids, tumor spheroids, or other such three-dimensional multicellular structures to drugs or other imposed experimental conditions may more closely mimic the response of a corresponding sample in the human body or any other environment of interest. [Examples]

[0052] The method described herein was tested on three-dimensional living cell biological samples using a trained conditional generative adversarial network (GAN) model. The trained model generated two-dimensional focused projection images, three examples of which are shown in Figure 9. The images were obtained from samples filled in the wells of a microwell plate using the apparatus shown in Figures 7 and 8. The total fluorescence imaging system and samples were positioned in an incubator culturing the samples under temperature and humidity conditions suitable for living cell culture evaluation.

[0053] In particular, Figure 9 is a set of three example input images (901, 906, 912) and model outputs (902, 908, 914) compared with ground truth projections (904, 910, 916) obtained by the Z-stack projection algorithm, which has been previously described and demonstrates the usefulness of this method. In Figure 9, the input images (901, 906, 912) were generated under the second image acquisition paradigm of Figures 2B and 3, which are described in detail above. The ground truth projection images 904, 910, 916 were obtained using the method and Z-stack projection algorithm of Figure 2C.

[0054] The subject depicted in the image in Figure 9 is an example of a glioblastoma spheroid, a typical example of a single tumor spheroid evaluation. This evaluation is one of the evaluations available with the assignee's IncuCyte® live-cell imaging system, as depicted in Figures 7 and 8 of this disclosure. Further details of this evaluation can be found at https: / / www.essenbioscience.com / en / applications / cell-health-viability / spheroids / . Each of the images shown in Figure 9 represents all the wells of a microwell plate. Spheroids have a radius of approximately 400 μm, and there is only one spheroid per well. Spheroids are 3D structures consisting of thousands of cells. Fluorescent labels were supplied to the wells to label the cell nuclei. Therefore, one of the bright spots that can be seen in the model output images (902, 908, 914) and the ground truth projections (904, 910, 916) is a single cell.

[0055] The trained model that generated the model output images (902, 908, 914) in Figure 9 was a conditional generative adversarial network (GAN). Detailed information about such networks is provided in Philip Isola et al., "Image-to-Image Translation with Conditional Adversarial Networks", arXiv.1611.07004 [cv.cs] (2016) and Xudong Mao et al., "Least Squares Generative Adversarial Networks", arXiv.1611.04076 [cv.cs] (2016), both of which are incorporated herein by reference.

[0056] The high-level architecture of the conditional GAN ​​used is shown in Figure 10. Like other GANs, this model includes a generator 1004 and a discriminator 1010. The generator 1004 takes one of the images acquired according to one of the three paradigms in Figure 3 as input 1002. A “fake” dataset 1006 is generated, which is a concatenation of the input image and the predicted projection image generated by the generator 1004. A “true” dataset 1008 is generated as a concatenation of the input image and the ground truth projection image. These two datasets 1006 and 1008 are then fed to the discriminator, which calculates the loss. With each training iteration, updates to the loss functions of the generator and discriminator make the generator 1004 better at creating “fake” images (i.e., predicted projection images) and the discriminator 1010 better at distinguishing between true (ground truth) images and fake (predicted projection) images. After model training, this enables the generator 1004 to produce high-quality projection images. In the architecture of Figure 10, the generator 1004 and the discriminator 1010 are individual neural networks, which may vary in the architecture on which they are based. In this example, the generator 1004 is a U-net (see Ronneberger et al., "U-net: Convolutional networks for biomedical image segmentation," published by Springer, and the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 234-241), and the discriminator 1010 is a PatchGAN with the architecture of the paper by Isola et al. mentioned above. Since these networks are described in detail in the scientific literature, a more detailed discussion is omitted for the sake of simplification and to avoid confusing the inventive details of this disclosure. ResNet50 is another example of a network that can be used as a discriminator. Further details about GANs are described in the paper by Goodfellow et al. mentioned previously.

[0057] In a conditional GAN, the generator is conditioned with some data, such as a long-exposure (Z-sweep) input image in this case. Conditioning of the generator with one or more long-exposure images is applied to both training and inference. Only the generator is needed at the end of training, because it is used to generate the projected image during inference. Minimizing the loss function and iterating through the discriminator-generator are only relevant in the training phase.

[0058] GAN model training is described as follows: As explained above, a GAN has two models, namely a generator (G) and a discriminator (D). The G generator outputs from noise, and the output mimics the distribution from the training dataset. D attempts to distinguish between true data and generated data. Essentially, G attempts to fool D. With each training iteration, the loss function for both D and G is updated. Model training can be carried out so that either G or D is updated more or less frequently. As the model training progresses, G becomes better at generating data that mimics true data from the training dataset, and after model training, G can make inferences on input images and directly generate the predicted projection. Three examples of this are shown in Figure 9.

[0059] A few other notes on the architecture and model training in Figure 10 are as follows: (1) Soft labels. Instead of the true label being 1, the label can be set to a value between [0.9, 1]. (2) Loss function. The binary cross-entropy loss is the original loss function for GANs. It can be made more stable by using mean squared error (MSE) loss without an activation function (making it a least squares (LS)GAN). (3) No fully connected layers. When Resnet is used as a discriminator, the least fully connected layers can be removed and a flat layer input instead (i.e., it is sent to a list of labels for the loss function). (4) PatchGAN discriminator. In this embodiment of GANs, a fully convolutional discriminator ending with multiple neurons for the output layer can be used, each corresponding to a patch of the input image.

[0060] conclusion The method described herein overcomes many of the shortcomings of conventional methods for obtaining 3D fluorescence information from living three-dimensional cell cultures. Conventional stepwise "Z-stack" fluorescence imaging techniques for 3D samples are slow, require user input, and ultimately expose the sample to excessive amounts of fluorescence, resulting in phototoxicity and photobleaching of the sample, both of which are highly undesirable. Other techniques require dedicated hardware (e.g., rotating disks) or advanced optical setups (e.g., light-sheet microscopes). Alternative deep learning techniques utilize methods that often compromise data integrity, including generating 3D data from very short exposure times or a single focal plane.

[0061] Conversely, the method of this disclosure does not require specialized hardware and only requires a simple fluorescence microscope with an axis motor. Note that the technique described above can be easily applied to other acquisition systems. Image acquisition from live cell samples is fast, and the risk of phototoxicity and photobleaching is reduced compared to conventional methods. Furthermore, the raw images collected from the camera represent true 3D fluorescence, as they are derived from fluorescence integrated over the Z dimension. Finally, the output of a single, high-quality 2D projection image eliminates the burden on the user of having complex software and analysis tools to handle 3D datasets. This high-quality 2D projection (Figures 3 and 6, 140; 902, 908, 914) can be input into standard 2D image visualization, segmentation, and analysis pipelines. [Explanation of Symbols]

[0062] 1 Image 2 images 3 images 4th Grade 10. Live cell samples, 3D live cell biological samples 12. Retention structure, microwell plate, sample plate 14. Fluorescent reagents (phosphors) 16. Motorized fluorescence microscope, fluorescence microscope, imaging device 18 Z-sweep images 22 Pre-trained Neural Network Models 24 Workstations 25 Arrows 26 Arrows, Computer Networks 30 datasets, 2D image dataset, Z-sweep image 100 Z Sweep 102 Z Sweep Image 102A Z Sweep Image 104 Z-stack 105 Z Sweep Image 106 Z-projection algorithms 108 Ground Truth Images, 2D Projection 110 Convolutional Neural Network (CNN) Models, Model Training 120 Paradigm 1 132 image dataset 132A Image Dataset 132B image dataset 134 consecutive Z-sweeps 136 consecutive Z-sweep images 138 Addition 140 2D projection images, model output 200 Model Training Process 210 Training Sets 212 Neural Networks 300 remote servers 400 Living Cell Imaging System 402 Fluorescence Optics Module 404 Sample holding well, sample well 406 Fluorescent Reagents 408 Tray 410 cabinets 418 Motor System 450A LED, light source 450B LED, light source 452A Narrowband Pass Filter 452B Narrowband Pass Filter 454A Dichroic 454B Dichroic Mirror 460 objective lens 462 Narrowband Emission Filter 464 Digital Cameras 901 Input image 902 Model Output Image 904 Ground Truth Projection 906 Input Images 908 Model Output Image 910 Ground Truth Projection 912 Input Images 914 Model Output Image 916 Ground Truth Projection 1002 Input 1004 Generator 1006 "fake" dataset 1008 "True" dataset 1010 Identifier

Claims

1. A method for generating a focused two-dimensional projection image of a fluorescence image of a three-dimensional living cell sample, The steps include: acquiring a long-exposure image of the sample with the camera by continuously exposing the camera while moving the focal plane of the camera through the sample in the Z direction; The steps include supplying one or more long-exposure images to a neural network model trained with multiple training images, The steps include generating the focused two-dimensional projection image using the trained neural network model, and Includes, The aforementioned multiple training images A long-exposure image obtained by continuously exposing the camera when the focal plane of the camera is moved in the Z direction through a 3D living cell training sample, The relevant ground truth image of the aforementioned 3D living cell training sample and including, method.

2. The method according to claim 1, wherein the neural network model includes a convolutional neural network (CNN) model.

3. The method according to claim 2, wherein the CNN model includes an encoder-decoder based model.

4. The method according to claim 1, wherein the neural network model is trained using supervised learning.

5. The method according to claim 1, wherein the neural network model is trained using a generative adversarial network (GAN) method.

6. The method according to claim 5, wherein the GAN method includes a conditional GAN ​​having a generator and a discriminator.

7. The method according to claim 6, wherein the generator of the conditional GAN ​​is conditioned by one or more long-exposure images.

8. The method according to claim 1, wherein the neural network model is trained using a cycle consistency loss method.

9. The method according to claim 8, wherein the cycle consistency loss method includes CycleGAN.

10. The method according to any one of claims 1 to 9, wherein the one or more long-exposure images include a set of consecutive long-exposure images.

11. The steps include performing fluorescence deconvolution on each of the aforementioned consecutive images, An addition step is performed to add the continuous long-exposure images after the fluorescence deconvolution. The method according to claim 10, further comprising:

12. The method according to any one of claims 1 to 11, wherein the three-dimensional living cell sample is contained within the wells of a microwell plate.

13. The method according to any one of claims 1 to 12, wherein the three-dimensional living cell sample comprises an organoid, a tumor spheroid, or a 3D cell culture.

14. The aforementioned camera is incorporated into the living cell imaging system. The method according to any one of claims 1 to 12, wherein the trained neural network model is implemented in a computing platform that is remote from the living cell imaging system and communicates with the living cell imaging system via a computer network.

15. A method for training a neural network to generate a two-dimensional projection image of a fluorescence image of a three-dimensional living cell sample, (a) A step of obtaining a training set in the form of multiple images, wherein the images are (1) Long exposure images of the three-dimensional living cell sample obtained by continuously exposing the camera when the focal plane of the camera is moved in the Z direction through the three-dimensional living cell training sample, and (2) A ground truth image obtained from a set of images obtained at different Z-focal plane positions of the three-dimensional living cell training sample, which is combined into a two-dimensional projected image using a projection algorithm. Steps including, (b) The step of performing a model training procedure using the training set to generate a trained neural network. Methods that include...

16. The method according to claim 15, wherein image (1) and image (2) include a plurality of pairs of images.

17. The aforementioned image (1) and image (2) include multiple non-paired images, The method according to claim 15, wherein the model training procedure includes a cycle consistency loss or a generative adversarial network model training procedure.

18. The method according to claim 17, wherein the cycle consistency loss model training procedure includes CycleGAN.

19. The method according to claim 15, wherein the neural network includes a convolutional neural network (CNN).

20. The method according to claim 19, wherein the CNN includes an encoder-decoder based neural network.

21. The method according to any one of claims 15, 16, 19, or 20, wherein the neural network is trained using supervised learning.

22. The method according to claim 15, wherein the trained neural network includes a generative adversarial network (GAN).

23. The method according to claim 22, wherein the GAN includes a conditional GAN ​​having a generator and a discriminator.

24. The method according to claim 23, wherein the generator of the conditional GAN ​​is conditioned by one or more long-exposure images.

25. A living cell imaging system that generates a two-dimensional projection image of a sample, used in conjunction with a sample holding device adapted to hold a three-dimensional sample, A fluorescence microscope comprising one or more excitation light sources, one or more objective lenses, and a camera capable of obtaining one or more fluorescence images from the three-dimensional sample held in the sample holding device, A fluorescence microscope comprising a motor system configured to move the fluorescence microscope in the Z direction relative to the sample holding device so that the camera acquires a long exposure image of the sample, wherein the image is obtained by continuously exposing the camera as the focal plane of the camera moves through the sample in the Z direction, A processing unit including a trained neural network model for generating the two-dimensional projection image of the three-dimensional sample from one or more long-exposure images. Equipped with, Multiple training images used to train the aforementioned neural network model, A long-exposure image is obtained by continuously exposing the camera while moving the focal plane of the camera through a three-dimensional training sample in the Z direction, The associated ground truth image of the aforementioned 3D training sample and including, A system for imaging living cells.

26. The system according to claim 25, wherein the neural network model is trained according to the method described in any one of claims 15 to 24.

27. The system according to claim 25 or 26, wherein the one or more long-exposure images include a set of consecutive images.

28. The system according to any one of claims 25 to 27, wherein the sample holding device comprises a microwell plate having a plurality of wells.

29. The system according to any one of claims 25 to 28, wherein the three-dimensional sample comprises an organoid, a tumor spheroid, or a 3D cell culture.

30. The system according to any one of claims 25 to 29, further comprising a remotely installed computing platform that implements the trained neural network model and communicates with the living cell imaging system via a computer network.

31. A method for generating a training set for training a neural network, (a) The step of acquiring a long-exposure fluorescence image of the three-dimensional training sample with the camera by continuously exposing the camera while moving the focal plane of the camera through the three-dimensional training sample in the Z direction, (b) A step of generating a ground truth image of the same 3D training sample from one or more different images of the 3D training sample obtained by the camera, (c) Repeat steps (a) and (b) for multiple different 3D training samples, (d) A step of supplying the images obtained by performing steps (a), (b), and (c) as a training set for training the neural network. Methods that include...

32. The method according to claim 31, wherein the ground truth image is generated by projecting the set of images onto a two-dimensional projection image.

33. The method according to claim 31 or 32, wherein the three-dimensional training sample comprises an organoid, a tumor spheroid, or a 3D cell culture.

34. The method according to any one of claims 31 to 33, further comprising the step of repeating steps (a) to (d) for different types of three-dimensional training samples, thereby generating different types of training sets.

35. A computer-readable recording medium storing non-temporary instructions for a living cell imaging system including a camera and a processing unit implementing a neural network model, wherein the instructions cause the system to perform the method according to any one of claims 1 to 12.

Images