A label-free virtual staining method for cell transmitted light microscopy images based on automatic optimization using deep learning

Abstract

This invention discloses a label-free virtual staining method for cell transmitted light microscopy images based on deep learning and automatic optimization. The method collects data from cell transmitted light and fluorescence images, augments the dataset using a data augmentation method based on a Poisson fusion algorithm, and uses pre-training on reference cells and fine-tuning based on target cells. This significantly reduces the amount of target cell image data required to generate the virtual staining network model. Experimental results demonstrate that the method can effectively achieve virtual staining of the cytoskeleton and nucleus in transmitted light microscopy images of target cells using only four sets of training images.

Description

Technical Field

[0001] The present invention belongs to the field of cell imaging and relates to a method for virtual staining of cell transmitted light microscopic images through deep learning. Background Art

[0002] In biomedicine, bioimaging is a crucial research tool for understanding the structure of biological tissues and elucidating their various physiological functions. Cells are often considered the smallest unit of research. Their subcellular structure, protein synthesis, size, morphology, spatial distribution, interactions, tracking, and other characteristics provide crucial clues and insights for exploring the internal laws of life. Furthermore, the diagnosis and grading of most diseases rely on information from cellular and histopathological levels.

[0003] Fluorescence imaging (FI) is a commonly used observational method for obtaining morphological and functional images of cells and subcellular structures (such as the skeleton, nucleus, mitochondria, and Golgi apparatus). However, cellular fluorescence imaging has significant drawbacks: First, it requires a complex hardware system, including a fluorescence signal excitation system (excitation light source, optical transmission components), signal reception, and detection components. Second, the fluorescence signal decays with prolonged illumination, which can interfere with quantitative analysis and even produce erroneous results. Third, the addition of chemical fluorescent dyes to the cellular environment is an invasive and even toxic process. Fourth, when an experiment uses two or more fluorescent dyes, the light emitted by one dye may contaminate (diffuse) into the detection channel of another fluorescent dye, resulting in interference from light cross-talk and background fluorescence. Fifth, fluorescent staining experiments are costly, time-consuming, and inefficient.

[0004] Transmitted light microscopic images are easy to obtain, but lack the specificity, clarity, and contrast of fluorescence images, which limits the specific display and quantitative analysis of biological information. Some studies have combined transmitted light imaging, fluorescence imaging, and deep learning, using deep learning's ability to extract implicit, high-level information from images and reconstruct clear fluorescent markers from transmitted light imaging images. Currently, in the field of fluorescence reconstruction based on transmitted light imaging, the segmentation of subcellular structures, the determination of cell states (such as survival / death, proliferation, etc.), and the analysis of living cell dynamics have been achieved. However, in order to obtain accurately reconstructed fluorescence images of target cells, more than 60 groups of high-quality target cell training data are required. Therefore, there is an urgent need for a universal (such as using low-cost data acquisition) and reliable virtual staining method for cell microscopic images.

[0005] Chinese patent CN109410196A involves using a segmentation algorithm to cluster and segment preprocessed digital cervical cancer histopathology images to obtain multiple small image blocks. However, the Poisson conditional random field used therein is a classification algorithm and is not suitable for data enhancement. Chinese patent CN114119386A involves cropping pothole defect areas from a road damage dataset and using them as input to a deep convolutional neural network (DCGAN) after normalization. Poisson fusion can be used to smoothly insert the pothole defect images generated by the DCGAN into the road image, thereby meeting the requirements for the amount of training sample data in pothole defect image recognition technology. However, in the technology of reconstructing fluorescent markers based on cell transmitted light imaging images, it is necessary to process and utilize both transmitted light imaging and fluorescence imaging image information in data enhancement and model training. Compared with the image reconstruction technology of pothole defects, the use of the Poisson fusion algorithm for image generation for data enhancement is more complicated. Summary of the Invention

[0006] The purpose of the present invention is to provide a label-free cell transmitted light microscopy image virtual staining method based on deep learning automatic optimization, which is suitable for visualizing the subcellular structures of different cells through a very small amount of cell transmitted light microscopy image data.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A method for virtual staining of a label-free cell transmitted light microscopic image, comprising the following steps:

[0009] 1) Using multiple sets of reference cell transmitted light imaging images with different cell types and / or magnifications and labeled subcellular structures to construct source datasets, and using target cell transmitted light imaging images with labeled subcellular structures to construct target datasets;

[0010] 2) Pre-training the candidate deep convolutional neural network using each source dataset to obtain a pre-trained model with network pre-trained weight parameters based on the corresponding deep convolutional neural network, where the network pre-trained weight parameters correspond one-to-one with each source dataset; classifying the target dataset using the classification network trained on the source dataset to obtain a similarity matching result between the target dataset and the source dataset (i.e., selecting the source dataset with the highest similarity to the target dataset);

[0011] 3) Based on the test results of the virtual dyeing performance of the pre-trained model, select two or more deep convolutional neural networks (for example, select two networks from multiple pre-trained candidate deep convolutional neural networks), use the target dataset and load the network pre-trained weight parameters that match the dataset to fine-tune the selected deep convolutional neural networks respectively, and obtain virtual dyeing network models based on the corresponding deep convolutional neural networks (a total of more than two virtual dyeing network models are obtained);

[0012] 4) Use the virtual staining network model to virtually stain the transmitted light imaging images of the target cells.

[0013] Preferably, the construction of the source data set specifically includes the following steps: staining reference cells of a certain cell type, then collecting transmitted light imaging images of the reference cells at a certain magnification and fluorescence images of reference cells labeled with different subcellular structures at a corresponding magnification, and contrast enhancing the transmitted light imaging images of the reference cells of the cell type and magnification to obtain reference cell training set images (i.e., a source data set corresponding to the cell type and magnification); the construction of the target data set specifically includes the following steps: staining target cells, then collecting transmitted light imaging images of the target cells at any magnification and fluorescence images of the target cells labeled with different subcellular structures at a corresponding magnification, and contrast enhancing the transmitted light imaging images of the target cells at the magnification to obtain target cell training set images (i.e., target data set).

[0014] Preferably, the target data set includes more than 4 transmitted light imaging images of target cells.

[0015] Preferably, the contrast enhancement refers to enhancing the contrast between the cell image blocks and the image background in the reference cell transmitted light imaging image or the target cell transmitted light imaging image by adopting contrast-limited adaptive histogram equalization processing.

[0016] Preferably, in the pre-training and fine-tuning, the reference cell training set images and the target cell training set images are respectively data enhanced and the sizes of the data-enhanced images are adaptively adjusted (for example, the images are cropped to meet the input requirements of the deep convolutional neural network) and then input into the corresponding deep convolutional neural network; the data enhancement refers to cropping a certain proportion of cell image blocks from the reference cell training set images based on the cytoskeleton marked in the fluorescence image and applying image enhancement operation to these cell image blocks, and then adding them to the background image (from the reference cell training set) by Poisson fusion algorithm (for the cell image blocks cropped from the transmitted light imaging image) or pasting method (for the cell image blocks cropped from the fluorescence image). The method comprises the following steps: a) randomly selecting the target cell training set images and cutting out a certain proportion of cell image blocks from the target cell training set images and applying image enhancement operations to these cell image blocks, and then adding them to the background image (randomly selected from the target cell training set images) through a Poisson fusion algorithm (for the cell image blocks cut out from the transmitted light imaging images) or pasting (for the cell image blocks cut out from the fluorescence images) to generate a synthetic image that is several times larger than the target cell training set images, thereby expanding the number of training set images input into the deep convolutional neural network (compared to the training set images before data enhancement, the number can be expanded to any multiple).

[0017] Preferably, the deep convolutional neural network is selected from Res-UNet, Attention-UNet, Deeplab v3+, Enhanced-UNet (E-UNet), and Enhanced CGAN (E-CGAN) based on the virtual coloring accuracy of the corresponding pre-trained model.

[0018] Preferably, the enhanced UNet adopts a U-shaped network structure containing an encoder and a decoder, wherein a jump connection is used between the encoder and the decoder, dropout is added between convolutions, and the number of output feature layers is 1; the enhanced CGAN includes a discriminator and a generator of a U-shaped network structure, the generator includes an encoder and a decoder, a jump connection is used between the encoder and the decoder, the discriminator is connected to the decoder, and residual connections are added between the convolutions of the generator.

[0019] Preferably, step 4 specifically includes the following steps: fusing the virtual staining results output by the two virtual staining network models, and then obtaining a target cell virtual staining image corresponding to the target cell transmitted light imaging image through visualization processing.

[0020] A label-free cell transmitted light microscopy image virtual staining system, the system comprising a virtual staining network pre-training module, a similarity matching module, a virtual staining network model training module and a virtual staining image output module;

[0021] The virtual staining network pre-training module is used to pre-train the candidate deep convolutional neural network using a source data set; the source data set is constructed using corresponding groups of reference cell transmitted light imaging images marked with subcellular structures selected one by one from multiple groups of reference cell transmitted light imaging images marked with subcellular structures of different cell types and / or magnifications;

[0022] The similarity matching module is used to classify a target dataset using a classification network trained on a source dataset; the target dataset is constructed using transmitted light imaging images of target cells labeled with subcellular structures;

[0023] The virtual dyeing network model training module is used to select two or more deep convolutional neural networks based on the virtual dyeing performance test results of the pre-trained model with network pre-trained weight parameters obtained by the virtual dyeing network pre-training module, match the corresponding network pre-trained weight parameters of a set of source data sets with the highest similarity to the target data set to the target data set based on the classification results of the similarity matching module (i.e., the implementation process of the matching of the above-mentioned target data set with the network pre-trained weight parameters), and use the target data set and load the network pre-trained weight parameters that match the data set to fine-tune the selected deep convolutional neural network;

[0024] The virtual staining image output module is used to virtually stain the target cell transmitted light imaging image using the virtual staining network model obtained by the virtual staining network model training module.

[0025] Preferably, the system further comprises a contrast enhancement module and a data enhancement module;

[0026] The contrast enhancement module is used to perform contrast-limited adaptive histogram equalization processing on the acquired target cell transmitted light imaging images and reference cell transmitted light imaging images of different cell types and magnifications during the process of constructing the source data set and the target data set and before the target cell transmitted light imaging images are virtually stained;

[0027] The data enhancement module is used to crop a certain proportion of cell image blocks from a reference cell transmitted light imaging image of a certain magnification in the source data set and a reference cell fluorescence image of a corresponding magnification marking different subcellular structures according to the cytoskeleton marked in the fluorescence image during the pre-training process of the candidate deep convolutional neural network. These cell image blocks are processed by image enhancement operation and then added to the background image by Poisson fusion algorithm or pasting (randomly selecting one from the reference cell transmitted light imaging image in the source data set for Poisson fusion, and randomly selecting one from the reference cell fluorescence image in the source data set for pasting), thereby expanding the pre-trained training set images (adaptively adjusting the size of the image before inputting the deep convolutional neural network). , and for, in the process of fine-tuning the selected deep convolutional neural network, cropping a certain proportion of cell image blocks from the target cell transmitted light imaging images of a certain magnification in the target data set and the target cell fluorescence images of the corresponding magnification marking different subcellular structures according to the cytoskeleton marked in the fluorescence image, and adding these cell image blocks to the background image through the Poisson fusion algorithm or pasting after applying image enhancement operation (randomly selecting one from the target cell transmitted light imaging images in the target data set for Poisson fusion, and randomly selecting one from the target cell fluorescence images in the target data set for pasting), thereby expanding the fine-tuning training set images (adaptively adjusting the size of the image before inputting the deep convolutional neural network).

[0028] The beneficial effects of the present invention are embodied in:

[0029] The present invention proposes a virtual staining method for label-free cell transmitted light microscopy images based on deep learning automatic optimization. Pre-training weights of different types of source data sets are provided through pre-training. The target data set is used for fine-tuning to cope with the overfitting problem caused by insufficient training data in actual use, which greatly improves the model accuracy. Only a small amount of data set needs to be provided in model training to achieve considerable results. Experimental results show that the present invention can predict the target cell structure and subcellular structure from transmitted light imaging images (ordinary bright field, phase contrast or DIC). Only a small amount of target cell data set needs to be provided to predict the visualization results of the subcellular structure (cytoskeleton, DNA) of the target cell. Therefore, after the model is established, there is no need to chemically stain the target cells again, and the impact on the target cell sample to be tested is extremely low, so it is possible to observe and track the target cells on a temporal and spatial scale.

[0030] Furthermore, the present invention is not limited to the visualization of subcellular structures such as the cytoskeleton and DNA. By providing transmitted light imaging images and fluorescent images of other subcellular structures, visualization results of more channels can be obtained simultaneously, overcoming the limitations of limited chemical staining observation channels, reducing time and operating costs, and reducing resource consumption.

[0031] Furthermore, the present invention proposes a cell image block data enhancement process based on the Poisson fusion algorithm (cell image blocks are pasted on the fluorescent image that provides labeling information), which can highlight the cell area to reduce inter-class imbalance, expand the limited data set, and reduce overfitting during training.

[0032] Furthermore, the present invention fully utilizes the advantages of different models through model fusion to improve the virtual dyeing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Schematic diagram of the virtual staining method for cell transmitted light microscopy images.

[0034] Figure 2 Schematic diagram of the data enhancement method and effect based on Poisson fusion.

[0035] Figure 3 Schematic diagram of the improved model network structure based on U-Net and CGAN.

[0036] Figure 4 Visualization results of predictions for 5 network models.

[0037] Figure 5 This is the visualization result after fusing the models based on Enhanced-UNet and Enhanced-CGAN.

[0038] Figure 6 Comparison of the accuracy of virtual coloring results after the fusion of models based on Enhanced-UNet, Enhanced-CGAN, and models based on other networks.

[0039] Figure 7 Morphological parameter evaluation of virtual staining results and ground truth (GT) at the cellular level. DETAILED DESCRIPTION

[0040] The present invention will be described in further detail below with reference to the accompanying drawings and examples, which are intended only to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0041] (1) Virtual staining method for label-free cell transmitted light microscopy images based on deep learning automatic optimization

[0042] like Figure 1 As shown, the method includes the following steps:

[0043] Step 1: Preparation of different cell types and acquisition of transmitted light and fluorescence images at different magnifications.

[0044] 1.1 Cell Preparation

[0045] Three types of cells (reference cells) including NIH 3T3, 231, and HUVEC were prepared and cultured at a concentration of 1×10 5 The cells were inoculated at a density of 100 / mL and stained with phalloidin (actin for staining the cytoskeleton) and 4',6-diamidino-2-phenylindole (DAPI, for staining DNA, which is mainly distributed in the cell nucleus).

[0046] 1.2 DIC and fluorescence image acquisition

[0047] In order to provide labeling information for the DIC images in the training set, validation set, and test set, namely, the visualized cytoskeleton and cell nucleus formed after fluorescent staining, DIC images and fluorescence images were collected simultaneously under an Olympus FV3000 laser scanning confocal microscope.

[0048] During image acquisition, five images (all with a resolution of 1024 × 1024 pixels) were acquired per field of view at 2.5 μm intervals. The magnifications for the images were 10X / 0.4 DIC objective, 20X / 0.75 DIC objective, and 40X / 0.6 DIC objective, respectively, resulting in spatial resolutions of 1.243 μm / pixel, 0.621 μm / pixel, and 0.311 μm / pixel. Fluorescence image acquisition also included the following laser parameters:

[0049] The excitation light and emission light are: cell nucleus (ex: 405nm, em: 461nm), cytoskeleton (ex: 561nm, em: 586nm).

[0050] The collected image is an RGB image with three channels. One of the channels is extracted to obtain a grayscale image.

[0051] In step 2, different data sets (e.g., 4 sets of data sets) constructed from transmitted light imaging images of different cell types (e.g., NIH 3T3, HUVEC) and magnifications (e.g., 10X, 40X) and fluorescent images of the cytoskeleton and cell nucleus fluorescently labeled collected under the same field of view are preprocessed (limited contrast adaptive histogram equalization, data enhancement, image cropping).

[0052] 2.1 Contrast-limited adaptive histogram equalization (CLAHE) of DIC images

[0053] Histogram equalization can transform the histogram of an image into an approximately uniform distribution, thereby enhancing the image contrast, which is very helpful for distinguishing cells from the background.

[0054] The formula for histogram equalization is:

[0055]

[0056] Among them, h(v) is the gray value of the pixel after equalization, cdf(v) is the cumulative distribution function of different gray values, and cdf min is the minimum non-zero value of the cumulative distribution function, M and N are the number of pixels of the width and height of the image, L is the number of gray levels (here 256), and round is the rounding operation.

[0057] However, the commonly used histogram equalization is global and can sometimes reduce image detail. Therefore, the present invention employs contrast-limited adaptive histogram equalization, performing localized histogram equalization on DIC images directly acquired from a microscope, and employing contrast clipping to enhance local contrast while reducing noise amplification.

[0058] 2.2 Image Data Augmentation Method Based on Poisson Fusion Algorithm

[0059] First, the construction of the cell image block database (Data library construction)

[0060] by Figure 2 Taking the training set shown in as an example, the cytoskeleton fluorescence image is used to determine the cell positioning anchor point. At the same time, in order to avoid cropping to the image boundary and aggregated cell clusters, position and bounding box constraints need to be imposed on the cell positioning anchor point. The cell image block on the cytoskeleton fluorescence image is cropped according to the cell positioning anchor point, and the cell image block is cropped at the same position on the DIC image and the cell nucleus fluorescence image (both of which are collected at the same field of view and magnification as the above-mentioned cytoskeleton fluorescence image). The cell image block database can be described as:

[0061] L={(a0,b0,c0),(a1,b1,c1)…(a n ,b n ,c n )}

[0062] Among them, a i 、b i 、c iThey refer to the i-th cell image block cropped from the DIC image, cytoskeleton fluorescence image, and cell nucleus fluorescence image, respectively, where i = 1, 2, …, n, and n represents the number of cell positioning anchor points on the image.

[0063] Second, the selection of the fusion target area

[0064] Newly selected cell locations (possible locations in the image outside the cell image block) must avoid overlapping with existing cell locations (i.e., already selected locations) and image boundaries. To prevent excessive cell density in the synthesized image, the fusion density (the number of fusion target regions per unit area of ​​the background image) is controlled to 20% of the original image's cell density.

[0065] Third, image synthesis for DIC images

[0066] The present invention uses a Poisson blending algorithm to generate a composite image that naturally transitions between the brightness and texture of the cell image block and the background image. The core mathematical principle of this algorithm is to utilize the Poisson partial differential equation with Dirichlet boundary conditions and, under the Laplace operator of the unknown function in a specified region of interest (i.e., the fusion target region), construct a coefficient matrix to solve for the pixel grayscale values ​​of the fusion target region. The solution process is as follows:

[0067] Assume that there is a closed subregion Ω on the two-dimensional image plane S (for the synthetic image, it refers to its fusion target area), and its boundary is V is the gradient vector field defined on Ω, f and f * is the unknown and known scalar function defined on Ω, and V is the guided vector field. Then f adopts the interpolation function guided by V in Ω, that is, solving the extreme value problem:

[0068]

[0069] The solution to the above extreme value problem is the unique solution to the Poisson equation with Dirichlet boundary conditions:

[0070]

[0071] Here, Δ is the Laplace operator and div is the divergence operator.

[0072] For digital images, the above extreme value problem can be expressed as:

[0073]

[0074] in,<p,q> is a pair of 4-connected adjacent pixels, f p and f qare the values ​​of p and q on f.

[0075] To generate a synthetic image close to a natural distribution, the present invention randomly selects cell image blocks from the cell image block database (the number of blocks selected is determined by the fusion density described above) and processes them in one or more ways: flipping, rotating, and scaling. The blocks are then pasted onto the background image using the Poisson fusion algorithm described above. This process can be described as follows:

[0076]

[0077] In the above formula, O′ and O are the synthesized image and background image respectively, PB is the Poisson fusion algorithm, a i is L(a i ,b i ,c i ), (x c ,y c ) is the coordinate of the fusion target area, is the position constraint added when selecting coordinates, and T is the training dataset (the background image is randomly selected from the DIC image of the training dataset).

[0078] Fourth, image synthesis for fluorescence images

[0079] Instead of using the Poisson fusion algorithm, a cell image block with the same cell positioning anchor point as the cell image block selected in the image synthesis for the DIC image and cropped from the fluorescence image (cytoskeleton fluorescence image and cell nucleus fluorescence image collected at the same field of view and magnification as the DIC image) was used. i Cell positioning anchor point is the same b i 、c i ), directly perform pixel replacement on the fusion target area corresponding to the background image (the cytoskeleton fluorescence image and cell nucleus fluorescence image collected at the same field of view and magnification as the randomly selected DIC image above), that is, paste the cell image block on the background image.

[0080] 2.3 Crop the image to a fixed size

[0081] Because the image resolution obtained from acquisition and synthesis is relatively high, to accommodate feature extraction in the training network, a 1024×1024 pixel image is uniformly cropped into four 512×512 pixel images. If an image cannot be cropped to an integer multiple of 512×512 pixels, bilinear interpolation and edge padding are used for scaling.

[0082] Step 3: Task adaptation and selection of models.

[0083] 3.1 Network Structure

[0084] We use various deep convolutional neural network-based models to perform virtual cell staining of DIC images, including Enhanced-UNet, Res-UNet, Attention-UNet, Deeplab v3+, and Enhanced-CGAN.

[0085] 3.1.1 Enhanced-UNet

[0086] See also Figure 3 , the Enhanced-UNet adopted in the present invention has a U-shaped network structure and jump connections. The first half of the network structure is the feature extraction network (encoder), which is used to obtain local features and obtain abstract semantic features. The feature extraction network consists of four downsampling modules (Encoder convolutional blocks), each of which contains two convolutional layers (Conv 3×3.ReLu) and a pooling layer (Max Pooling 2×2). The second half of the network structure is the feature fusion network (decoder, Decoder convolutional blocks), which is composed of an upsampling layer (bilinear interpolation) and two convolutional layers and a dropout layer (Dropout) (the last convolutional layer of the output layer is Conv 1×1) and is repeatedly constructed. A total of four such operations are performed, which corresponds exactly to the feature extraction network. Finally, a 1×1 convolutional layer is connected to map the high-dimensional semantic features to the image space step by step. The completely symmetrical U-shaped network structure makes the fusion of the front and back features more thorough, so that high-resolution information and low-resolution information are increased in the target image. It combines low-resolution information during downsampling and high-resolution information during upsampling (providing a basis for accurate segmentation and positioning), and also fills in the underlying information through jump connections to improve segmentation accuracy.

[0087] 3.1.2 Res-UNet

[0088] In order to effectively solve the network degradation problem of the U-Net-based model, a residual module is added to the U-Net. Let X be the input of the network and F(X) be the output of the feature extraction module. The output of the residual module is:

[0089] H(X)=F(X)+X

[0090] As can be seen, the residual module adds the input feature X to the feature extraction module F(X) to obtain H(X), ensuring that the network incorporates the input feature information during forward propagation. The addition of the residual module enables Res-UNet to address the network degradation problem caused by excessive hidden layers in deep neural networks, facilitating the restoration of details in fluorescently stained images.

[0091] 3.1.3 Attention-UNet

[0092] The attention mechanism has a remarkable ability to focus on feature information, addressing the problem of information overload. By selecting the correct feature information for subsequent neural network calculations, it can effectively improve image segmentation accuracy. Adding an attention module to the contraction path of the U-Net network structure learns the multi-scale features of the encoder, resolving the U-Net's reliance on fixed weights during image feature extraction and distributing the feature extraction weights across all stages of the contraction path.

[0093] 3.1.4 Deeplab v3+

[0094] Deeplab v3+ is a classic semantic segmentation network that utilizes an encoder-decoder architecture. During the encoding process, ordinary convolutions in the feature extraction network are replaced with dilated convolutions with a sampling rate of 2, increasing the receptive field while maintaining the size of the feature map. During decoding, the spatial information of the feature map is restored through upsampling and fusion of shallow features, thereby capturing clearer object boundaries. Deeplab v3+ utilizes a fully convolutional neural network that accepts multi-scale image inputs, effectively preserving the semantic information and dimensionality of the image, achieving excellent prediction results in the field of semantic segmentation. The encoder architecture utilizes the "Xception" module for segmentation tasks, and applies depthwise separable convolution to the encoder-decoder module.

[0095] 3.1.5 Enhanced-CGAN

[0096] See also Figure 3The network structure of the enhanced CGAN (Conditional Generative Adversarial Network) adopted in the present invention consists of a generator, a discriminator and residual modules. The generator adopts a structure similar to U-Net. The first half of the generator is a feature extraction network (encoder), which is used to obtain local features and obtain abstract semantic features. The feature extraction network consists of 5 downsampling modules, each of which consists of two convolutional layers (Conv 3×3.ReLu) and a pooling layer (MaxPooling 2×2). The second half of the generator is a feature fusion network (decoder), which consists of an upsampling layer (bilinear interpolation) and two convolutional layers (the last convolutional layer of the output layer is Conv 1×1) repeatedly constructed, and a total of 4 such operations are performed. Residual modules are added between the convolutional layers to better retain shallow information. The discriminator is based on the structure of PatchGAN, which helps to generate the high-frequency part of fluorescent staining. PatchGAN consists of a convolutional block (Convolutional blocks: Conv 4×4+LRelu+InstanceNorm) and a downsampling block (Conv 4×4 (stride of 2)+LRelu+InstanceNorm). It divides the output feature map of the generator into patches, judges the authenticity of each patch separately, and then takes the average.

[0097] 3.2 Model Training

[0098] The computer core devices used in this invention are Intel(R) Core(TM) i7-9750H CPU and NVIDIARTX 3090GPU, and the software environment uses Python 3.8, Tensorflow 2.4.0 and Pytorch 1.11.0 frameworks.

[0099] The pre-training datasets are:

[0100] NIH 3T3 cells at 10x magnification (10X), including 71 DIC images.

[0101] NIH 3T3 cells at 40x magnification (40X), including 120 DIC images.

[0102] HUVEC cells at 10x magnification (10X), including 80 DIC images.

[0103] HUVEC cells at 40x magnification (40X), including 120 DIC images.

[0104] The DIC images in each dataset were divided into a training set, validation set, and test set ratio of 7:1.5:1.5. The training set was expanded fivefold through data augmentation. The training, validation, and test sets were all cropped from 1024×1024 to 512×512 resolution before being fed into the network for pre-training.

[0105] The pixel size of the network input images was 512×512, and the batch size was set to 2 or 4. Adadelta was used as the optimizer during training. To prevent overfitting, early stopping was used during training. Model training was terminated when the accuracy on the validation set did not improve at a patience of 50.

[0106] 3.3 Accuracy Evaluation Indicators and Model Selection

[0107] For the four data sets, the five networks mentioned above were used to build models for training and obtain the network weight parameters, and predictions were made based on the network weight parameters. Figure 4 ), and conduct evaluation based on image statistics. Specifically, the Pearson Correlation Coefficient (PCC) can be used to evaluate the performance of the pre-trained model prediction. PCC quantifies the pixel-level similarity between the true value and the predicted result, and is defined as:

[0108]

[0109] In the above formula, X i and Y i They represent the pixel-by-pixel grayscale values ​​of the predicted results (i.e., the virtual staining results output by the pre-trained model) and the real results (i.e., the collected fluorescence staining results, Ground-truth). Represents the grayscale mean of the prediction result, represents the grayscale mean of the true result, and N is the number of pixels. The Pearson correlation coefficient (PCC) on the four data sets is shown in Table 1:

[0110] Table 1. Test results (PCC)

[0111]

[0112] From the evaluation indicators in Table 1, it can be seen that the pre-trained models obtained after pre-training of Enhanced-UNet and Enhanced-CGAN have the best comprehensive performance on different datasets.

[0113] Step 4: Adaptively select pre-trained weight parameters and perform fine tuning on a small batch of target cell datasets.

[0114] 4.1 Adaptive Selection of Pre-training Weight Parameters

[0115] In order to adapt to the diverse data sets in actual tasks, the present invention trains a four-category convolutional classification network model (i.e., classifier) ​​using DIC images from four different data sets. Then, DIC images from a new data set collected from other types of cells are input into the model for prediction. Based on the prediction results, the source data set (i.e., one of the four data sets mentioned above) that is most similar to the new data set is found, and the corresponding pre-trained weight parameters are selected according to the prediction results, thereby achieving adaptive selection of pre-trained weight parameters.

[0116] The classification network model above uses a VGG16 network. After feature extraction through five downsampling modules, a fully connected layer is connected to the classifier. Each downsampling module consists of two or three convolutional layers (Conv 3×3, ReLu) and a pooling layer (MaxPooling 2×2). The convolutional layers are followed by two 4096-dimensional fully connected layers, and finally, the softmax activation function is used for classification.

[0117] 4.2 Fine-tuning to adapt to new datasets

[0118] Typically, training a deep network requires a large amount of training data. However, the present invention uses a data augmentation method and fine-tunes the pre-trained weight parameters with a high degree of matching determined by adaptive selection, achieving better training results using only a training set of DIC images containing a small number of labels (i.e., formed by preprocessing 4-6 DIC images of target cells and corresponding fluorescence images).

[0119] The fine-tuning strategy depends on the size of the new dataset and the similarity between the new dataset and the source dataset. Since the jump connection of U-Net can fuse low-scale and high-scale features, freezing certain layers of the encoder will lead to the loss of detailed texture features. Therefore, all layers of Enhanced-UNet are fine-tuned, the batch size is set to 2, the optimizer is AdaDelta, and the initial learning rate is reduced to one tenth; for Enhanced-CGAN, the discriminator can be regarded as a classifier, so the classification layer needs to be reinitialized to adapt to the new dataset. The present invention fine-tunes the weights of all layers except the last layer of the discriminator, with a batch size of 2, the optimizer of the generator is AdamW, and its initial learning rate is 0.0001, and the optimizer of the discriminator is also AdamW, and its initial learning rate is 0.00001.

[0120] The datasets used for fine-tuning were: 231 cells at 10x and 40x magnifications, with four DIC images in the training, two in the validation, and ten in the test sets, respectively; and HMN iPSCs with four, two, and four in the training, validation, and test sets, respectively (all DIC images were from a public dataset). The image resolution was 1024×1024 (scaled to 512×512 before inputting the network for training). The training set was expanded to 20 images through data augmentation, with two 512×512 images fed into each iteration. The network was trained for 40 generations using the Enhanced-UNet and 150 generations using the Enhanced-CGAN.

[0121] 4.3 Fine-tuning Results

[0122] After fine-tuning the three new datasets based on the Enhanced-UNet and Enhanced-CGAN models, the visualization results of the two final models (virtual dyeing results after restoring RGB) are as follows: Figure 5 As shown in the figure, the visualization results are highly consistent with the fluorescence images of the real cytoskeleton and cell nucleus. The prediction accuracy indicators of the three new data sets are shown in Figure 6 .

[0123] Step 5: Visualize the model fusion results.

[0124] The present invention adopts a model fusion method that directly performs weighted summation of the output results of the two final models, as follows:

[0125] The cell virtual staining results predicted by the Enhanced-UNet and Enhanced-CGAN models obtained in step 4 are weighted and summed pixel by pixel, where the weight sum is 1 and the weights of the individual model prediction results are 0.5 and 0.5 respectively. Finally, the single-channel weighted grayscale image is converted to an RGB image. The visualization result of the model fusion ( Figure 5 Weight-add) and accuracy ( Figure 6 It can be seen from Weight-add that compared with the prediction results of a single model, model fusion further improves the accuracy.

[0126] Based on the above description of the virtual staining method for label-free cell transmitted light microscopy imaging based on deep learning automatic optimization, the present invention can deploy data preprocessing, data enhancement, fine-tuning and model fusion on a single platform, achieving virtual staining and visualization output of the subcellular structure of the target cell by providing only a dataset containing 4-6 groups of labeled images. It has the potential to replace the complex and high-cost fluorescence staining process and greatly facilitates use by non-professionals.

[0127] (II) Parameter evaluation at the cellular level (Cell profiler)

[0128] Quantitative measurement of cell structure is crucial for biomedical and drug research and clinical decision-making. The above-mentioned virtual staining method for label-free cell transmitted light microscopy imaging based on deep learning automatic optimization is used to segment the fluorescence images (including Actin and DNA) of the real results (i.e., the collected fluorescence staining results, Ground-truth) and the predicted results (i.e., the virtual staining results of model fusion) into single cells using a thresholding method. For each single cell, the area (Size), perimeter (Perimeter), density (Compactness, C), major axis (Majoraxis), minor axis (Minor axis), eccentricity (E) and other parameters of Actin and DNA (Nuclear) are calculated respectively ( Figure 7 ).

[0129] The values ​​of all pixels in the target area (including the boundary) are set to 1, and the pixel values ​​of the background area are set to 0. The area is defined as the number of points with all pixel values ​​1 in the area, and the perimeter is defined as the number of pixels in the closed outline outside the target area. The compactness (C) is defined as:

[0130] C=4πA / P 2

[0131] Where A and P represent the area and perimeter respectively.

[0132] The major axis is defined as the maximum distance between any two points on the target boundary; the minor axis is defined as the sum of the distances from the two points on either side of the major axis to the major axis. The eccentricity of the target is defined by the ratio of the major axis to the minor axis, as follows:

[0133] E=r c / r i

[0134] In the above formula, r c and r i are the values ​​of the minor and major axes, respectively.

[0135] The calculation results at the cellular level show that the predicted results are consistent with the cell parameter size and distribution levels calculated from the real image. Therefore, the prediction results of the present invention are reliable and meet the requirements of parameter evaluation.

[0136] The above results show that the present invention can be expanded to characterize or predict biological information, including morphology and function, cell type, cell health, cell phenotype, differentiation status of induced pluripotent stem cells, etc., and serve clinical and research fields with corresponding needs. The details are as follows:

[0137] Morphology and function: used for cell counting (Count); calculation of morphological features of cell structure and organelles; localization, qualitative and quantitative analysis of proteins, such as prediction of cell microfilament distribution, distribution of focal adhesion albumin, prediction of YAP / TAZ activation status, etc.

[0138] Cell typing: used to identify and count cell types, such as in a complete blood count.

[0139] Cell health: used to identify cell health status, such as cell viability, proliferation, differentiation, and apoptosis, and thereby determine the levels of substances such as cell metabolites, apoptosis markers, stress markers, and toxicity markers.

[0140] Cell phenotype: used for cell phenotype detection, such as cell cycle, proliferation, apoptosis, migration, invasion, clone formation, autophagy, EMT (Epithelial-Mesenchymal Transition), angiogenesis, etc.

[0141] Differentiation states of induced pluripotent stem cells: Classify and count different cells after differentiation.

Claims

1. A method for virtual staining of label-free cell transmitted light microscopic images, characterized by: The following steps are involved: 1) Use multiple sets of transmitted light imaging images of reference cells with different cell types and / or magnifications and labeled subcellular structures to construct source datasets, and use transmitted light imaging images of target cells with labeled subcellular structures to construct target datasets; 2) Pre-train the candidate deep convolutional neural network using each source dataset to obtain a pre-trained model based on the corresponding deep convolutional neural network. The network pre-training weight parameters correspond one-to-one with each source dataset. The classification network trained on the source dataset is used to classify the target dataset to obtain the similarity matching result between the target dataset and the source dataset. 3) Based on the test results of the virtual dyeing performance of the pre-trained model, select two or more deep convolutional neural networks, use the target dataset and load the network pre-trained weight parameters that match the dataset to fine-tune the selected deep convolutional neural networks, and obtain the virtual dyeing network model based on the corresponding deep convolutional neural network; 4) Use the virtual staining network model to virtually stain the transmitted light imaging images of target cells; The construction of the source data set specifically includes the following steps: staining reference cells of a certain cell type, then collecting transmitted light imaging images of the reference cells at a certain magnification and fluorescence images of reference cells with different subcellular structures marked at a corresponding magnification, and performing contrast enhancement on the transmitted light imaging images of the reference cells of the cell type and magnification to obtain reference cell training set images; the construction of the target data set specifically includes the following steps: staining target cells, then collecting transmitted light imaging images of the target cells at any magnification and fluorescence images of the target cells with different subcellular structures marked at a corresponding magnification, and performing contrast enhancement on the transmitted light imaging images of the target cells at the magnification to obtain target cell training set images; In the pre-training and fine-tuning, data enhancement is performed on the reference cell training set images and the target cell training set images, and the sizes of the data-enhanced images are adaptively adjusted before being input into the corresponding deep convolutional neural network; the data enhancement refers to cropping a certain proportion of cell image blocks from the reference cell training set images and the target cell training set images based on the cytoskeleton marked in the fluorescence image, and then applying image enhancement operations to these cell image blocks and adding them to the background image through a Poisson fusion algorithm or pasting method to generate a synthetic image that is several times larger than the reference cell training set images and the target cell training set images, thereby expanding the number of training set images input into the deep convolutional neural network, and the background image is randomly selected from the reference cell training set images and the target cell training set images; The step 4 specifically includes the following steps: fusing the virtual staining results output by the two virtual staining network models, and then obtaining a target cell virtual staining image corresponding to the target cell transmitted light imaging image through visualization processing.

2. The method for virtual staining of label-free cell transmitted light microscopic images according to claim 1, characterized in that: The target data set includes more than 4 transmitted light imaging images of target cells.

3. The method for virtual staining of label-free cell transmitted light microscopic images according to claim 1, characterized in that: The contrast enhancement refers to enhancing the contrast between the cell image block and the image background in the reference cell transmitted light imaging image or the target cell transmitted light imaging image by adopting contrast-limited adaptive histogram equalization processing.

4. The method for virtual staining of label-free cell transmitted light microscopic images according to claim 1, characterized in that: The deep convolutional neural network is selected from Res-UNet, Attention-UNet, Deeplab v3+, enhanced UNet, and enhanced CGAN based on the virtual coloring accuracy of the corresponding pre-trained models.

5. The method for virtual staining of a label-free cell transmitted light microscopic image according to claim 4, characterized in that: The enhanced UNet adopts a U-shaped network structure containing an encoder and a decoder, wherein a skip connection is used between the encoder and the decoder, dropout is added between convolutions, and the number of output feature layers is 1; the enhanced CGAN includes a discriminator and a generator of a U-shaped network structure, the generator includes an encoder and a decoder, a skip connection is used between the encoder and the decoder, the discriminator is connected to the decoder, and residual connections are added between the convolutions of the generator.

6. A label-free cell transmitted light microscopic image virtual staining system for executing the label-free cell transmitted light microscopic image virtual staining method according to any one of claims 1 to 5, characterized in that: The system includes a virtual dyeing network pre-training module, a similarity matching module, a virtual dyeing network model training module and a virtual dyeing image output module; The virtual coloring network pre-training module is used to pre-train the candidate deep convolutional neural network using the source dataset; The source data set is constructed by using corresponding groups of reference cell transmitted light imaging images marked with subcellular structures selected one by one from a plurality of groups of reference cell transmitted light imaging images marked with subcellular structures of different cell types and / or magnifications; The similarity matching module is used to classify the target dataset using the classification network trained on the source dataset; This target dataset is constructed using transmitted light imaging images of target cells labeled with subcellular structures; The virtual dyeing network model training module is used to select two or more deep convolutional neural networks based on the virtual dyeing performance test results of the pre-trained model obtained by the virtual dyeing network pre-training module, match the corresponding network pre-trained weight parameters of a group of source data sets with the highest similarity to the target data set to the target data set based on the classification results of the similarity matching module, and fine-tune the selected deep convolutional neural network using the target data set and loading the network pre-trained weight parameters matching the data set; The virtual staining image output module is used to virtually stain the target cell transmitted light imaging image using the virtual staining network model obtained by the virtual staining network model training module.

7. The system according to claim 6, characterized in that: The system also includes a contrast enhancement module and a data enhancement module; The contrast enhancement module is used to perform contrast-limited adaptive histogram equalization processing on the acquired target cell transmitted light imaging images and reference cell transmitted light imaging images of different cell types and magnifications during the process of constructing the source data set and the target data set and before the target cell transmitted light imaging images are virtually stained; The data enhancement module is used to, during the pre-training process of the alternative deep convolutional neural network, crop a certain proportion of cell image blocks from a reference cell transmitted light imaging image at a certain magnification in the source data set and a reference cell fluorescence image at a corresponding magnification with different subcellular structures marked, based on the cytoskeleton marked in the fluorescence image, and then add these cell image blocks to the background image through a Poisson fusion algorithm or a pasting method after performing image enhancement operations on these cell image blocks, thereby expanding the pre-trained training set images; and is used to, during the fine-tuning process of the selected deep convolutional neural network, crop a certain proportion of cell image blocks from a target cell transmitted light imaging image at a certain magnification in the target data set and a target cell fluorescence image at a corresponding magnification with different subcellular structures marked, based on the cytoskeleton marked in the fluorescence image, and then add these cell image blocks to the background image through a Poisson fusion algorithm or a pasting method after performing image enhancement operations on these cell image blocks, thereby expanding the fine-tuning training set images; The background image is randomly selected from the source dataset and the target dataset.

Images