A fluorescence labeling prediction method based on label-free transmission cell microscopy images
Through deep learning technology, a conditioned generative adversarial network (cGAN) is constructed based on non-labeled transmission cell microscopy images to predict cell fluorescent labeling, solving the time, cost and safety issues in the prior art and achieving accurate cell labeling and health assessment.
Patent Information
- Application Number
- CN202111134370.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-27
- Publication Date
- 2025-05-30
- Estimated Expiration
- 2041-09-27
AI Technical Summary
The existing cell fluorescent labeling technology consumes a lot of time and effort, is complex in labeling process, is expensive in reagents, is limited by spectral overlap, is toxic to fluorescent probes, and photobleaching will interfere with fluorescence imaging.
Using a deep learning cell fluorescent labeling prediction method based on non-labeled transmission cell microscopy images, the fluorescent labeling is directly predicted from the non-labeled transmitted light images by constructing conditional generation adversarial networks (cGANs) and specific loss functions, bypassing the traditional fluorescent labeling process.
This method can accurately predict the location and intensity of the nucleus and cell membranes and the health of the cells, saving time and cost, and avoiding the risk of toxicity of fluorescent probes and interference from photobleaching.
Smart Images
Figure CN114092391B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a prediction method, and in particular to a fluorescence labeling prediction method based on unlabeled transmission cell microscopy images. Background Art
[0002] In the fields of biology and medicine, microscopy techniques can provide researchers with detailed information that is invisible to the naked eye. Among them, transmission light microscopy is easy to use, but it is difficult to judge the cell condition based on the obtained images. In contrast, the images obtained by fluorescence microscopy are easier to analyze. By fluorescently labeling cell samples, researchers can observe the subtle state of cells. Fluorescence microscopy plays a crucial role in understanding the structure of cells. However, fluorescence microscopy also has many limitations. For example: 1) Sample preparation requires a large amount of time and effort, the fluorescence labeling process is relatively complex, and the reagents used are relatively expensive; 2) The number of fluorescence labels is limited by spectral overlap. When there are many different fluorescence labels in the sample, spectral overlap makes it difficult to distinguish which color belongs to which label; 3) Fluorescent probes may be toxic and sometimes even directly kill cells; 4) The phenomenon of photobleaching will interfere with fluorescence imaging. Summary of the Invention
[0003] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is that the existing cell fluorescence labeling requires a large amount of time and effort, the fluorescence labeling process is complex, the reagents are expensive, the number of fluorescences is limited by spectral overlap, the fluorescent probes are toxic, and the phenomenon of photobleaching will interfere with fluorescence imaging. The present invention provides a fluorescence labeling prediction method based on unlabeled transmission cell microscopy images, which can predict fluorescence labels from unlabeled transmission light images, accurately predict the positions and intensities of cell nuclei and cell membranes, as well as the health status of cells. This method bypasses the traditional fluorescence labeling process, saving time and cost.
[0004] To achieve the above object, the present invention provides a fluorescence labeling prediction method based on unlabeled transmission cell microscopy images, including the following steps:
[0005] Construct a deep learning cell fluorescence labeling prediction model;
[0006] According to the constructed deep learning cell fluorescence labeling prediction model, train the constructed deep learning cell fluorescence labeling prediction model;
[0007] Input the data to be labeled, and perform fluorescence labeling on the data to be labeled based on the trained deep learning cell fluorescence labeling prediction model;
[0008] Establish calculation performance indicators, and evaluate the similarity and peak signal-to-noise ratio of the labeled data.
[0009] Furthermore, a deep learning cell fluorescence labeling prediction model is constructed, which specifically includes the following steps:
[0010] Construct a conditional generative adversarial network architecture, and use cGAN to convert transmitted light images into their corresponding fluorescence images;
[0011] Construct a loss function, and add the L1 loss function and the MS-SSIM loss function on the basis of the adversarial loss function.
[0012] Furthermore, cGAN includes two deep neural networks, namely the generator G and the discriminator D; among them, the generator G adopts the U-net architecture, including four "down blocks" and four subsequent "up blocks"; the discriminator D includes five discriminator blocks, an average pooling layer and two fully connected layers.
[0013] Furthermore, the adversarial loss function:
[0014]
[0015] where x is the transmitted light image, y is the real fluorescence image, and p data (x, y) is the joint probability distribution of the input image x and the real fluorescence image y, is the expectation of the log-likelihood of (x, y); the generator G attempts to minimize the objective, thereby minimizing the difference between the generated fluorescence image and the real fluorescence image, while the discriminator D attempts to maximize the objective.
[0016] L1 loss function:
[0017]
[0018] MS-SSIM loss function:
[0019]
[0020] The loss function of the final network is:
[0021]
[0022] Furthermore, according to the constructed deep learning cell fluorescence labeling prediction model, the constructed deep learning cell fluorescence labeling prediction model is trained, specifically including: using the Adam optimization algorithm to train the data set, training for a total of 200 epochs, where the batch size N for each optimization traversal of the network is 4. The regularization parameters λ 1 and λ 2All are set to 100; the initial learning rate is 0.0002; in order to make the error converge to a smaller value, we use a learning rate decay strategy. For the first 100 epochs, the same learning rate is maintained, and for the last 100 epochs, the learning rate linearly decays to 0.
[0023] Furthermore, before training the constructed deep learning cell fluorescence labeling prediction model, data preprocessing is also included according to the constructed deep learning cell fluorescence labeling prediction model.
[0024] Furthermore, the data preprocessing includes the following steps:
[0025] Image denoising, and after image denoising, the image is normalized;
[0026] Enhance the contrast of the normalized image, and finally crop the image.
[0027] Furthermore, establishing calculation performance metrics includes the peak signal-to-noise ratio and structural similarity of the actual image and the image generated by the network.
[0028] Technical effects
[0029] A fluorescence labeling prediction method based on unlabeled transmission cell microscopy images of the present invention can predict fluorescence labeling from unlabeled transmission light images, and can accurately predict the positions and intensities of cell nuclei and cell membranes as well as the health status of cells. Compared with traditional fluorescence microscopy imaging techniques, the fluorescence labeling prediction method based on unlabeled transmission cell microscopy images of the present invention does not require a fluorescence labeling process, saving time and cost.
[0030] The following will further illustrate the concept, specific structure and technical effects generated by the present invention in conjunction with the drawings to fully understand the purpose, features and effects of the present invention. Brief description of the drawings
[0031] Figure 1 is an operation schematic diagram of a fluorescence labeling prediction method based on unlabeled transmission cell microscopy images of a preferred embodiment of the present invention;
[0032] Figure 2 is a network structure diagram of a generator and a discriminator within the cGAN of a fluorescence labeling prediction method based on unlabeled transmission cell microscopy images of a preferred embodiment of the present invention;
[0033] Figure 3 is a schematic diagram of the training and testing process of the cGAN of a fluorescence labeling prediction method based on unlabeled transmission cell microscopy images of a preferred embodiment of the present invention;
[0034] Figure 4It is a comparison diagram of converting the transmitted light image of rat cortical neurons in a fluorescence labeling prediction method based on unlabeled transmitted cell microscopic images in a preferred embodiment of the present invention into a DAPI-labeled image;
[0035] Figure 5 It is a comparison diagram of converting the transmitted light image of rat cortical neurons in a fluorescence labeling prediction method based on unlabeled transmitted cell microscopic images in a preferred embodiment of the present invention into a PI-labeled image;
[0036] Figure 6 It is a comparison diagram of converting the transmitted light image of human motor neurons in a fluorescence labeling prediction method based on unlabeled transmitted cell microscopic images in a preferred embodiment of the present invention into a DAPI-labeled image;
[0037] Figure 7 It is a comparison diagram of converting the transmitted light image of rat cardiomyocytes in a fluorescence labeling prediction method based on unlabeled transmitted cell microscopic images in a preferred embodiment of the present invention into a CellMask-labeled image;
[0038] Figure 8 It is a schematic diagram of the architecture of a fluorescence labeling prediction method based on unlabeled transmitted cell microscopic images in a preferred embodiment of the present invention. Detailed implementation manners
[0039] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0040] In the following description, specific details such as specific internal programs and technologies are proposed for the purpose of illustration rather than limitation, so as to thoroughly understand the embodiments of the present invention. However, those skilled in the art should clearly understand that the present invention can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details from interfering with the description of the present invention.
[0041] As Figure 8 shown, the embodiment of the present invention provides a fluorescence labeling prediction method based on unlabeled transmitted cell microscopic images, including the following steps:
[0042] Step 1, construct a deep learning cell fluorescence labeling prediction model; specifically including the following steps:
[0043] Step 101: Construct a conditional generative adversarial network architecture and apply cGAN to convert transmitted light images into their corresponding fluorescence images. cGAN includes two deep neural networks, namely the generator G and the discriminator D. The generator G adopts a U-net architecture, which includes four "down blocks" and four subsequent "up blocks". The discriminator D includes five discriminator blocks, one average pooling layer, and two fully connected layers. The structural diagrams of the generator and the discriminator within the cGAN framework are as shown in Appendix Figure 1 . The training objective of the generator is that when the generated G(x) and x are used as inputs to the discriminator, the probability value output by the discriminator is as large as possible, that is, the generator attempts to generate output images with the same statistical characteristics as the real images. The training objective of the discriminator is to output a small probability value when the input is not a pair of real images (x and G(x)), and to output a large probability value when the input is a pair of real images (x and y), that is, the discriminator attempts to distinguish between the target and the output images of the generator. Through mutual learning, the cGAN network is improved, thereby improving the quality of the generated fluorescence-labeled images. The training process and testing process of cGAN are as shown in Appendix Figure 3 .
[0044] Step 102: Construct a loss function and add the L1 loss function and the MS-SSIM loss function on the basis of the adversarial loss function. The loss function has a great influence on network training. The L1 loss function can better maintain the brightness and color unchanged, and can ensure the similarity between the input image and the output image. However, the L1 loss function assumes that the influence of noise and the local characteristics of the image are independent. However, the human visual system's perception of noise is affected by local brightness, contrast, and structure. The MS-SSIM (Multi-Scale SSIM) loss function is an index that synthesizes human subjective perception, considering subjective factors such as brightness, contrast, structure, and resolution. However, the MS-SSIM loss function is prone to cause changes in brightness and color deviation. Therefore, the present invention adds the L1 loss function and the MS-SSIM loss function on the basis of the adversarial loss function. Among them, the adversarial loss function of the network:
[0045]
[0046] where x is the transmitted light image, y is the real fluorescence image, and p data (x, y) is the joint probability distribution of the input image x and the real fluorescence image y, is the expectation of the log-likelihood of (x, y). The generator G attempts to minimize the objective, thereby minimizing the difference between the generated fluorescence image and the real fluorescence image, while the discriminator D attempts to maximize the objective.
[0047] The L1 loss function:
[0048]
[0049] MS - SSIM loss function:
[0050]
[0051] The loss function of the final network is:
[0052]
[0053] Step 2: According to the constructed deep - learning cell fluorescence labeling prediction model, train the constructed deep - learning cell fluorescence labeling prediction model; specifically, according to the constructed deep - learning cell fluorescence labeling prediction model, train the constructed deep - learning cell fluorescence labeling prediction model, which specifically includes: Use the Adam optimization algorithm to train the dataset for a total of 200 epochs, where the batch size N for each optimization pass of the network is 4. The regularization parameters λ 1 and λ 2 are both set to 100. The initial learning rate is 0.0002. To make the error converge to a smaller value, we use a learning rate decay strategy. For the first 100 epochs, keep the same learning rate, and for the last 100 epochs, the learning rate linearly decays to 0. The prediction fluorescence labeling network of the present invention is implemented using Python version 3.8.3. cGAN is implemented using Pytorch version 1.6.0. Other python libraries used include torchvision, dominate, visdom, os, time, Python Imaging Library (PIL), and numpy. The network is implemented on a desktop computer with an Intel(R) Xeon(R) Gold 6248R CPU@3.00GHz 3.00GHz (two processors) and 256G RAM, running the Windows 10 operating system.
[0054] In addition, before training the constructed deep - learning cell fluorescence labeling prediction model according to the constructed deep - learning cell fluorescence labeling prediction model, data pre - processing is also included.
[0055] The data pre - processing includes the following steps:
[0056] Image denoising, and after image denoising, standardize the image;
[0057] Enhance the contrast of the standardized image, and finally crop the image.
[0058] Specifically, the present invention relates to three different types of cells, namely rat cerebral cortical neurons, human motor neurons, and rat cardiomyocytes; and three different staining solutions. DAPI (4′,6-Diamidino-2-Phenylindole) is used to label cell nuclei, CellMask is used to label cell membranes, and Propidium Iodide (PI) is used to label dead cells. The datasets used in Experiment 1 and Experiment 2 are from public databases, and the dataset used in Experiment 3 is from the Medical Image Laboratory of Shanghai University. The detailed information of the datasets is shown in Table 1.
[0059] For the datasets used in Experiment 1 and Experiment 2, the original images need to be cropped. The original images are randomly cropped into images of size 256×256. The number of cropped training set images is shown in Table 1.
[0060] The specific acquisition process of the dataset in Experiment 3 is described in steps (3) and (4), and the obtained data is used for training and testing. The ratio of the training set to the test set in each experiment is 4∶1.
[0061] Table 1 Specific information of experimental data
[0062]
[0063] (3) Data acquisition
[0064] (3-1) After cell resuscitation, culture, and passage, observe the cell state under a microscope. The experimental cells need to meet the following conditions: moderate cell density, spindle-shaped connection, good cell adhesion, and more than 50% of the cells are active.
[0065] (3-2) Prepare a formaldehyde fixative. Add 1 ml of 16% formaldehyde solution to 3 ml of PBS solution and mix well to obtain 4 ml of 4% formaldehyde solution.
[0066] (3-3) Wash the cells with PBS solution multiple times.
[0067] (3-4) Fix the cells with the fixative and place the cell culture dish in an environment at 4°C for 30 minutes.
[0068] (3-5) Wash the cells with PBS solution multiple times. During the operation, pay attention to washing the fixative clean, otherwise it will affect subsequent staining.
[0069] (3-6) Prepare the staining solution. The original concentration of the CellMask solution is 5 mg / ml. Take 0.2 μL of the CellMask solution and add it to 0.2 ml of PBS solution to prepare a staining solution with a concentration of one-thousandth.
[0070] (3-7) Stain the cells with CellMask stain solution and place the cell culture dish in an environment at 4 °C for 30 minutes.
[0071] (3-8) Data acquisition. Use a total internal reflection fluorescence microscope to capture the bright-field images of the cells after fluorescence staining. In the same field of view, the cell samples after fluorescence staining are excited with a 647 nm laser and imaged using a Cy3 filter with a 10× / 0.4NA objective lens.
[0072] (4) Data preprocessing. Before using the dataset to train the deep learning network, it needs to be preprocessed. The specific data preprocessing steps are as follows:
[0073] (4-1) Image denoising. Use a 5×5 median filter to remove salt-and-pepper noise.
[0074] (4-2) Image normalization. The average of the transmitted light pixel intensities is normalized to 0.5, the standard deviation is 0.125, the average of the fluorescence pixel intensities is 0.25, the standard deviation is 0.125, and all pixels are clipped to fall within [0.0, 1.0].
[0075] (4-3) Image contrast enhancement.
[0076] (4-4) Image cropping. Randomly crop the image with a size of 2048×2048 into images with a size of 256×256.
[0077] Step 3: Input the data to be labeled and perform fluorescence labeling on the data to be labeled based on the trained deep learning cell fluorescence labeling prediction model.
[0078] Step 4: To test the network performance, establish calculation performance metrics, evaluate the similarity and peak signal-to-noise ratio of the labeled data, and establish calculation performance metrics including the peak signal-to-noise ratio and structural similarity between the actual image and the image generated by the network. SSIM is a metric for measuring the similarity between two images, and the calculation method is as follows:
[0079] SSIM is a metric for measuring the similarity between two images, and the calculation method is as follows:
[0080]
[0081] where μ x is the average of x, μ y is the average of y, is the variance of x, is the variance of y, σ xy is the covariance of x and y. c 1 =(k 1 L) 2 c 2 =(k 2L) 2 is a constant used to maintain stability. L is the dynamic range of pixel values. k 1 = 0.01, k 2 = 0.03.
[0082] PSNR is the most common and widely used objective image evaluation metric, and its calculation method is as follows:
[0083]
[0084]
[0085] Among them, MSE (Mean Square Error) represents the mean square error between the current image x and the reference image y, h and w are the height and width of the image respectively, and n is the number of bits per pixel.
[0086] The performance metrics of the network-predicted fluorescence images are shown in Table 2. Example figures of the experimental results are shown in the appendix Figure 4 -Appendix Figure 7 .
[0087] Table 2 Performance Metrics of Network-Predicted Fluorescence Images
[0088]
[0089]
[0090] From the performance metric table of the finally obtained predicted fluorescence images and the experimental result figures, it can be seen that the method proposed in the present invention constructs a general image conversion network architecture, which can perform image conversion on images of different cell types and different fluorescence labels; at the same time, the method can accurately predict the positions and intensities of cell nuclei and cell membranes as well as the health status of cells; in addition, based on this method, there is no need for a fluorescence labeling process, saving time and cost. Therefore, this method improves the performance of the existing fluorescence labeling prediction methods.
[0091] The present invention constructs a deep learning cell fluorescence labeling prediction model, including a conditional generative adversarial network architecture (cGAN) and a loss function; uses a fluorescent probe to label cells, and uses a total internal reflection fluorescence microscope (TIRFM) to collect transmitted light and fluorescence images; based on publicly available data and experimentally collected data, trains the above-constructed deep learning model on paired transmitted light images and fluorescence images; based on the trained deep learning model, predicts the fluorescence labeling of other unlabeled transmitted light images; finally, uses structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) to evaluate the quality of the fluorescence labeling prediction images. After constructing a reasonable deep learning model and undergoing good training, during the testing process, as long as the transmitted light image is input into the model, the output fluorescence image can be obtained without the fluorescence labeling process, saving time and cost.
[0092] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations according to the concept of the present invention without creative labor. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field of the present invention based on the concept of the present invention through logical analysis, reasoning, or limited experiments on the basis of the prior art should be within the protection scope determined by the claims.
Claims
1. A fluorescence labeling prediction method based on label-free transmission cell microscopy images, characterized in that, it includes the following steps: Construct a deep learning cell fluorescence labeling prediction model; According to the constructed deep learning cell fluorescence labeling prediction model, train the constructed deep learning cell fluorescence labeling prediction model; Input the data to be labeled, and perform fluorescence labeling on the data to be labeled based on the trained deep learning cell fluorescence labeling prediction model; Establish calculation performance indicators, and evaluate the similarity and peak signal-to-noise ratio of the labeled data; Among them, constructing a deep learning cell fluorescence labeling prediction model specifically includes the following steps: Construct a conditional generative adversarial network architecture, and use cGAN to convert the transmitted light image into its corresponding fluorescence image; Construct a loss function, and add an L1 loss function and an MS-SSIM loss function on the basis of the adversarial loss function; The cGAN includes two deep neural networks, namely a generator G and a discriminator D; wherein the generator G adopts a U-net architecture, including four "downsampling blocks" and four subsequent "upsampling blocks"; the discriminator D includes five discriminator blocks, an average pooling layer and two fully connected layers; The adversarial loss function: where x is the transmitted light image, y is the true fluorescence image, and p data (x, y) is the joint probability distribution of the input image x and the true fluorescence image y, is the expectation of the log-likelihood of (x, y); the generator G attempts to minimize the objective, thereby minimizing the difference between the generated fluorescence image and the true fluorescence image, while the discriminator D attempts to maximize the objective; L1 loss function: MS-SSIM loss function: The loss function of the final network is:
2. A fluorescence labeling prediction method based on label-free transmission cell microscopy images according to claim 1, characterized in that, According to the constructed deep learning cell fluorescence labeling prediction model, training the constructed deep learning cell fluorescence labeling prediction model specifically includes: using the Adam optimization algorithm to train the data set, training for a total of 200 epochs, where the batch size N for each optimization traversal of the network is 4; regularization parameters λ 1 and λ 2 are both set to 100; the initial learning rate is 0.0002; in order to make the error converge to a smaller value, we use a learning rate decay strategy. For the first 100 epochs, the same learning rate is maintained, and for the last 100 epochs, the learning rate linearly decays to 0.
3. A fluorescence labeling prediction method based on label-free transmission cell microscopy images according to claim 2, characterized in that, Before training the constructed deep learning cell fluorescence labeling prediction model according to the constructed deep learning cell fluorescence labeling prediction model, data preprocessing is also included.
4. A fluorescence labeling prediction method based on label-free transmission cell microscopy images according to claim 3, characterized in that, The data preprocessing includes the following steps: Image denoising, and normalizing the image after image denoising; Enhance the contrast of the normalized image, and finally crop the image.
5. A fluorescence labeling prediction method based on label-free transmission cell microscopy images according to claim 1, characterized in that, Establishing calculation performance indicators includes the peak signal-to-noise ratio and structural similarity of the actual image and the image generated by the network.