Fluorescence microscopic data reconstruction method based on simulation data and model deep learning

Abstract

The invention discloses a low signal-to-noise ratio fluorescence microscopic data reconstruction method based on simulation data and model-guided deep learning, which comprises the following steps of: generating a series of simulation data by using an imaging equation by taking real biological sample data collected under a low-light dose imaging condition as reference; and a deep learning network structure driven by a guided filtering cascade Ricderson-deconvolution model is designed, and the network structure trained by using simulation data can be directly applied to actually acquired low signal-to-noise ratio data, so that high signal-to-noise ratio and high-resolution reconstruction of the low signal-to-noise ratio data is realized. The method provided by the invention can simply and effectively improve the quality of the acquired image under the condition of low signal-to-noise ratio imaging, can relieve the dependence of a deep learning method on real acquired data, greatly reduces the difficulty of a deep learning data collection process, and is beneficial to the acquisition of intelligent living cell fluorescence long-time-history imaging information.

Description

Technical Field

[0001] The present invention belongs to the technical field of fluorescence microscopy imaging, and in particular relates to a low signal-to-noise ratio fluorescence microscopy data reconstruction method based on simulation data and model-guided deep learning. Background Art

[0002] Live cell dynamic long-term imaging technology is crucial for understanding the complex dynamic processes inside cells. It can not only reveal the basic mechanisms of cell life activities, such as cell division, migration, metabolic processes and signal transduction, but also provide understanding for the development of diseases and cell responses to drugs. Fluorescence microscopy can achieve high-resolution, high-contrast, and high-specificity imaging at the subcellular level, which provides a powerful tool for the dynamics of living cells. However, long-term exposure to strong light will cause phototoxicity to cells, affect the normal physiological activities of cells, and even cause cell death; in addition, fluorescent molecules will gradually lose fluorescence under continuous light and photobleaching will occur, which limits the imaging time. In order to extend the duration of dynamic observation of fluorescent living cells, it is necessary to reduce the light dose of each light excitation during imaging, that is, to reduce the intensity of the excitation light or shorten the exposure time, so as to minimize phototoxicity and photobleaching.

[0003] Reducing the imaging light dose inevitably exacerbates the image degradation problem in fluorescence microscopy. The weakened imaging signal intensity makes the fluorescence signal in the image not bright enough, thus affecting the visibility of the target structure or molecule. Weaker fluorescence signals will lead to a decrease in the signal-to-noise ratio, and the background noise will become relatively more significant, resulting in a decrease in image quality, and the resolution and details of the image may be affected. Therefore, image processing algorithms are very important for the reconstruction of images collected under low light dose imaging conditions. It can improve imaging quality and maintain low phototoxicity and photobleaching, thereby promoting the development of dynamic long-term imaging of living cells.

[0004] Image degradation under low-light-dose imaging conditions includes extremely high noise pollution and blur caused by diffraction, so the corresponding image processing methods need to consider the effects of noise and blur respectively. At present, there are some algorithms that can reconstruct blurred and noisy images. For example, the Chinese patent application with publication number CN118096568A provides an alternating iterative decoupling deconvolution image processing method based on Poisson process, the Chinese patent application with publication number CN115293981A provides a structured light illumination fluorescence microscopy image denoising and super-resolution reconstruction method and system, and the Chinese patent application with publication number CN109636733A provides a fluorescence image deconvolution method and system based on deep neural network. In these patent technologies, the iterative processing process requires careful setting of parameters, and its processing real-time performance is poor compared with the deep learning-based method; the deep learning-based method relies on a large amount of data, but for dynamic imaging of living cells, it is very difficult to obtain a large amount of paired high-quality / low-quality data. In order to obtain efficient reconstruction of low-light-dose imaging data using deep learning methods, it is necessary to improve the generalization ability of the deep learning network and reduce its data dependence as much as possible; therefore, it is very important to study generalizable deep learning network training methods based on simulation data training. Summary of the invention

[0005] In view of the above, the present invention provides a method for reconstructing low signal-to-noise ratio fluorescence microscopy data based on simulation data and model-guided deep learning. By constructing a model-driven deep learning network, the generalization ability of the network is improved, and corresponding simulation training data pairs are generated according to the imaging equation. The model-driven deep learning network is trained by simulation data, thereby realizing fast and efficient reconstruction of low signal-to-noise ratio data collected under low light dose imaging conditions.

[0006] A method for reconstructing low signal-to-noise ratio fluorescence microscopy data based on simulation data and model-guided deep learning, comprising the following steps:

[0007] (1) fluorescently labeling a biological sample, acquiring images of the biological sample using a fluorescence microscope under extremely low light dose imaging conditions, and obtaining a three-dimensional fluorescence microscopic image of the biological sample by stacking a series of acquired two-dimensional images;

[0008] (2) obtaining the point spread function of the fluorescence microscope by measuring or simulating the imaging parameters, and making the axial pixel size consistent with the lateral pixel size by interpolation;

[0009] (3) generating a three-dimensional all-zero matrix, using a position random function to set a certain density of random-sized ball structures, circular ring structures, and three-dimensional space line structures within a limited size range at random positions in the matrix, each structure having a random intensity within a limited range, and then adding a constant background to the entire matrix to obtain simulation data;

[0010] (4) blurring the simulation data using a three-dimensional Gaussian function to smooth its sharp boundaries, and downsampling the axial direction so that the ratio of its lateral pixel size to its axial pixel size is the same as the corresponding ratio of the three-dimensional fluorescence microscopy image, thereby obtaining clean and noise-free reference data;

[0011] (5) degenerating the simulated data by using a point spread function through an imaging equation so that the simulated data has a blur similar to that of a three-dimensional fluorescence microscopy image, and then downsampling the simulated blurred data in the axial direction to obtain downsampled blurred data of the simulated blur;

[0012] (6) introducing random noise (noise intensity is close to or slightly stronger than the noise in the acquired two-dimensional image) into the downsampled fuzzy data through a Poisson random process and a Gaussian random process to obtain simulated degradation data;

[0013] (7) Repeat steps (3) to (6) to obtain a large number of data pairs composed of simulated degradation data and their corresponding reference data, and divide all data pairs into training sets and validation sets;

[0014] (8) Constructing a deep learning network driven by a guided filter cascade Richardson-Lucy deconvolution model, using the simulated degradation data in the training set data pair as the input of the network, and the corresponding reference data as the label of the network output, and then training the network through the loss function, and testing the network performance with the validation set;

[0015] (9) By inputting the three-dimensional fluorescence microscopy image of the biological sample into the trained deep learning network, a high-quality reconstructed image can be output.

[0016] Furthermore, the extremely low light dose imaging condition is low excitation light intensity or short exposure time, and the acquired two-dimensional image contains a large amount of noise (Poisson noise and Gaussian noise) and diffraction blur, low pixel intensity, and low signal-to-noise ratio.

[0017] Furthermore, the point spread function is a three-dimensional matrix structure with Gaussian distribution, and its variation with space is not large.

[0018] Furthermore, the processing expression of step (4) is as follows:

[0019] GT=[Gau(S,σ1)]↓ t

[0020] Among them: GT is the reference data, S is the simulation data, Gau(S,σ1) represents the three-dimensional Gaussian smoothing of the simulation data S with a variance of σ1, Indicates downsampling the target axis at a factor of t.

[0021] Furthermore, the processing expression of step (5) is as follows:

[0022]

[0023] Where: i1 is the downsampled fuzzy data, S is the simulation data, represents the convolution operator, h is the point spread function, Indicates downsampling the target axis at a factor of t.

[0024] Furthermore, the processing expression of step (6) is as follows:

[0025] i2=P(i1)+n

[0026] Among them: i1 is the downsampled fuzzy data, i2 is the simulated degradation data, represents the Poisson random process of photon noise introduced in the imaging process, and n represents the Gaussian random noise introduced by the detection device in the imaging process.

[0027] Furthermore, the simulation data, downsampled fuzzy data, simulated degraded data and reference data are all three-dimensional volume data, and the ratio of the lateral pixel size to the axial pixel size of the downsampled fuzzy data, simulated degraded data and reference data is consistent with the corresponding ratio of the three-dimensional fluorescence microscopy image.

[0028] Furthermore, the computational reasoning process of the deep learning network driven by the guided filter cascade Richardson-Lucy deconvolution model is expressed as follows:

[0029]

[0030] G=Gau(i2,σ2); N_est=i2-G

[0031] Where: e j+1 and e j They represent the reconstructed images of biological samples by the network during the j+1th and jth iterations, respectively. i2 is the simulated degradation data. G is the guided image generated by network learning. q is the preliminary guided filtering result. a and b are the linear coefficients of the guided filtering. N_est is the estimated noise. Gau(i2,σ2) represents the three-dimensional Gaussian smoothing of i2 with a variance of σ2. represents the convolution operator, represents the covariance calculation, represents variance calculation, ε is the adjustment coefficient, It corresponds to the mean of i2, N_est, G, a, and b in the specified window. N is the maximum number of iterations. The initial reconstructed image e0 is q. F and B represent the forward projection operator and back projection operator of Richardson-Lucy deconvolution, respectively.

[0032] Further, the deep learning network driven by the guided filtering cascade Richardson-Lucy deconvolution model includes a guided filtering module and a Richardson-Lucy deconvolution module, and the guided filtering module includes:

[0033] A Gaussian mixture filter module based on Gaussian filter convolution and attention mechanism is used to generate the guidance image G;

[0034] A guided filter calculation module based on linear coefficient solution is used to calculate linear coefficients a and b and apply their average value to the guided image G to obtain a preliminary guided filter result q;

[0035] The residual compensation module based on convolution residual feature extraction is used to calculate the residual of the simulated degraded data i2 and G, q, and extracts features and nonlinearly activates the residual through convolution to obtain the final guided filtering result and output it to the Richardson-Lucy deconvolution module.

[0036] Furthermore, the Richardson-Lucy deconvolution module includes a 1 / 2 size deconvolution module H1, an original size deconvolution module H2 and a feature fusion module H3. The feature transfer process of H1 and H2 conforms to the Richardson deconvolution algorithm, including: ① fitting the forward projection process of the Richardson-Lucy deconvolution through a convolution layer based on a Gaussian blur convolution kernel (i.e. );② Calculate the ratio of input to forward projection result (i.e. );③Fitting the ratio back projection process through the ordinary convolution layer to obtain the update factor (i.e. );④ Apply the update factor to the previous iteration result to obtain a new iteration result (i.e. ); H1 and H2 have the same structure and their inputs are the output results of the guided filter module. The difference is that H1 needs to perform average pooling on the input so that its length, width and height are all reduced to 1 / 2 of the original size. The update factor obtained in H1 acts on the output result of the guided filter module, and the update factor obtained in H2 acts on the output result of H1. The output results of H1 and H2 are connected in parallel and input to H3. H3 uses multiple full convolution modules to fine-tune the input feature map to obtain the output result of the entire network.

[0037] Furthermore, the loss function is expressed as follows:

[0038]

[0039] L2=MSE(O,GT)+(1-SSIM(GT,O))+λ3×(1-SSIM(GT,Ef))

[0040] Where: L1 is the loss function of the guided filter module, L2 is the loss function of the Richardson-Lucy deconvolution module, λ1, λ2 and λ3 are weight factors, represents the root mean square error, represents the structural similarity index, GFo is the output result of the guided filtering module, O is the output result of H3, and Ef is the output result of H1.

[0041] In addition to using simulation data for training, the method of the present invention can also be trained using registered data pairs of low-quality data and high-quality data collected by a real fluorescence microscope.

[0042] The low signal-to-noise ratio fluorescence microscopy data reconstruction method based on simulation data and model-guided deep learning of the present invention uses simple structures such as simple balls, three-dimensional rings, and three-dimensional line structures to generate random simulation data, uses imaging equations to degrade the simulation data, simulates the imaging state of real low light doses, and generates simulated degraded images / high-quality noise-free true value data pairs. In addition, the present invention designs a lightweight deep learning network in a model-driven manner to improve the interpretability and generalization ability of the network. Through the training of simulation data pairs, the model-driven network structure has the ability to denoise and deblur low signal-to-noise ratio fluorescence microscopy data and reconstruct it at the same time. The restoration of signal-to-noise ratio and resolution is achieved through a set of processes, which reduces the dependence of deep learning methods on real collected data, and can achieve high-quality image reconstruction, which is conducive to the realization of dynamic long-term imaging of living cells. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 (a) shows low signal-to-noise ratio data obtained by imaging mEmerald-Tomm20-C-10 labeled mitochondria in living U2OS cells under low light dose conditions using transient structured light illumination microscopy.

[0044] Figure 1 (b) shows high signal-to-noise ratio and high-resolution data obtained by imaging mEmerald-Tomm20-C-10 labeled mitochondria in living U2OS cells under high light dose conditions using transient structured light illumination microscopy.

[0045] Figure 2 Schematic diagram of the process of the present invention for reconstructing low signal-to-noise ratio fluorescence microscopy data based on simulation data and model-guided deep learning.

[0046] Figure 3 (a) is an example diagram of simulation data S.

[0047] Figure 3 (b) is an example of simulated clean and noise-free reference data GT.

[0048] Figure 3(c) is an example diagram of simulated degradation data i2.

[0049] Figure 4 Schematic diagram of the deep learning network structure driven by the guided filter cascade Richardson-Lucy deconvolution model.

[0050] Figure 5 The results of reconstructing images of real low-light dose acquisition data using the method of the present invention and the existing reconstruction method RCAN are compared. DETAILED DESCRIPTION

[0051] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0052] In three-dimensional fluorescence microscopy, fluorescently labeled biological samples are excited to emit fluorescence through the excitation light path, and the fluorescence signal is collected by the camera through the detection light path. A series of two-dimensional images can be obtained by moving the imaging focal plane, and these two-dimensional images can be stacked to obtain a volumetric image of the three-dimensional sample. Due to the influence of out-of-focus light, there is a defocus signal in each two-dimensional plane; in addition, due to the diffraction of light, the resolution of the collected image will be affected, so blur is a major component of fluorescence microscopy image degradation. In the process of low-light-dose fluorescence microscopy, due to the use of lower excitation light intensity and shorter exposure time, the number of photons emitted by fluorescent molecules is reduced, resulting in a weaker captured fluorescence signal. In this case, the image is easily affected by reduced signal-to-noise ratio, photon statistical noise (Poisson noise) and detector quantization noise, which manifests as weakened signal intensity, decreased contrast and reduced resolution. In this case, extremely low signal-to-noise ratio and a large amount of noise are also one of the main components of image degradation. Figure 1 As shown, we used instantaneous structured light illumination microscopy to image mEmerald-Tomm20-C-10 labeled mitochondrial samples in U2OS cells under low light dose and high light dose imaging conditions, respectively, showing the difference between images acquired under low light dose imaging conditions and high light dose imaging conditions. Under low light dose imaging conditions, the acquired data contains a lot of noise and blur; therefore, in order to achieve reconstruction of images acquired under low light dose, both noise and blur need to be considered simultaneously.

[0053] In recent years, deep learning technology has played an important role in the field of image reconstruction; however, this technology usually requires a large amount of paired low-quality data and notes to form data pairs to achieve network training. The acquisition of such data pairs is difficult to achieve in dynamic imaging of living cells because cells are constantly changing; in addition, the acquisition of data pairs also requires a lot of labor. Using simulation data to train the network can greatly reduce the difficulty of obtaining training data; however, this puts forward requirements on the generalization ability of the network. Although traditional iterative methods are not smart enough, they often have good generalization capabilities; therefore, the present invention proposes a low signal-to-noise ratio fluorescence microscopy data reconstruction method based on simulation data and model-guided deep learning, which uses deep learning to fit the parts of the traditional iterative model that require manual design, which retains the generalization ability and interpretability of the traditional method and maintains the intelligence of deep learning; in addition, the present invention also proposes a simulation data generation process, which uses a deep learning network guided by a simulation data training model to reconstruct data collected under low light doses with high quality. As Figure 2 As shown, the low signal-to-noise ratio fluorescence microscopy data reconstruction method based on simulation data and model-guided deep learning of the present invention includes the following steps:

[0054] Step S1: Real data acquisition. Under low excitation intensity or short exposure time imaging conditions, biological sample data is collected, and according to experimental conditions, the corresponding point spread function is obtained through simulation or experimental means, and the compensation size represented by the lateral and axial pixels of the collected data is recorded, where the lateral pixel size is recorded as s x , the axial pixel size is denoted as s z , the corresponding axial downsampling ratio relative to the lateral direction is t = s z / s x ; Through data preprocessing, the background intensity of the collected data is removed.

[0055] Step S2: Simulation data generation. First, a three-dimensional matrix is ​​generated (taking a matrix size of 128×128×128 as an example), and a position random function is used to set a certain density of balls, donut-shaped objects, and three-dimensional space line structures with a limited size range (4 to 10 pixels) and random radius or random length at random positions in the matrix. Each structure has a random intensity within a limited range. The specific intensity range is based on the actual collected data intensity as a reference, and the corresponding number of each structure is based on the distribution density of the actual sample as a reference (or can be set to 100 to 200 each). A constant background is added to the entire matrix as a whole to obtain simulation data S.

[0056] Then, a three-dimensional Gaussian function is used to slightly blur S to smooth its sharp boundaries, and its axial direction is downsampled by a factor of t to serve as the clean and noise-free reference data GT for simulation.

[0057] Then, the point spread function is used to degenerate S through the imaging equation, so that it has a blur similar to the real collected data, and its axial direction is downsampled by t times to obtain the simulated blurred downsampled blurred data i1; finally, random noise is introduced into i1 through Poisson random process and Gaussian random process to obtain simulated degraded data i2, and the noise intensity is close to or slightly stronger than the noise of the real collected data. The process of obtaining GT and i2 from S is described by the following process:

[0058] GT=[Gau(S,σ1)]↓ t

[0059]

[0060] i2=P(i1)+n

[0061] Where: Gau() is a three-dimensional Gaussian blur function with a mean of 0 and a variance of σ1, []↓ t represents the downsampling of the axial direction at a magnification of t, P() represents the Poisson random process of photon noise introduced in the simulation imaging process, n represents the Gaussian random process of noise introduced by the detection equipment in the imaging process, represents the convolution operator, and + represents the addition operation. Figure 3 The simulated data S, the simulated clean and noise-free reference data GT, and the simulated degraded data i2 are shown.

[0062] Repeat the above GT and i2 data generation process to generate a large number of {GT, i2} data pairs.

[0063] Step S3: Dataset preprocessing: The {GT, i2} data pairs obtained by simulation are divided into training sets and test sets according to a ratio of 8:2. Taking the generation of 500 data as an example, 400 data pairs are used as training sets and 100 data pairs are used as validation sets.

[0064] Then, GT and i2 in each data pair are normalized. The normalization method is: arrange all pixel values ​​from small to large. According to the needs, the minimum value in the normalization process is selected from the 3% to 5% value, and the maximum value is selected from the 100% value, as follows:

[0065]

[0066] Where: u represents the value to be normalized, percentile(u,p) represents the value of the p%th position, and p high and p low are the high and low values ​​corresponding to the normalized standard, respectively.

[0067] Step S4: Network construction. The deep learning network driven by the guided filter cascade Richardson-Lucy deconvolution model includes two core modules, namely a learnable guided filter module and a learnable Richardson-Lucy deconvolution module. Figure 4 shown.

[0068] The learnable guided filter module includes three submodules, namely, the Gaussian mixture filter module, the guided filter calculation module and the residual compensation module. The Gaussian mixture filter module is used to generate the guided image G, including the Gaussian filter convolution layer and the attention mechanism layer. The Gaussian filter convolution layer uses the convolution kernel initialized by the Gaussian filter for feature extraction. Each channel of the convolution kernel is a Gaussian kernel with different standard deviations. The standard deviations of each channel are σ=0.5, 1, 1.5, 2, respectively, and are defined as:

[0069]

[0070] Where: (c i ,c j ,c k ) is the center coordinate of each convolution kernel, and m is a random number to increase the randomness of each channel.

[0071] The attention mechanism layer enhances the focus on important features by assigning different weights to different channels of the feature map obtained by the Gaussian filter convolution layer. The operation process includes: first, global average pooling or maximum pooling is performed on the input feature map to extract the global description of each channel; then, these global descriptions are processed through a shared multi-layer perceptron network to generate the attention weight of each channel; these weights are then multiplied element-wise with each channel of the original feature map to adjust the importance of each channel in the feature map and enhance the model's ability to focus on key features. Finally, the guide image G is obtained by averaging the data of each channel.

[0072] The noise map N_est is obtained by the difference between the input data i2 and the guide image G; i2, G and N_est are input into the guide filter calculation module, and the guide filter calculation module first calculates the filter linear coefficients a and b, and the calculation formula is:

[0073]

[0074] Among them: cov() is the calculated covariance, var() is the calculated variance, ε is the adjustment coefficient, are the mean values ​​of i2, N_est, and G in the specified window, respectively. Here, a typical window size is [3×3×3]. ε is learned through a single convolutional layer. A convolution kernel of size [3×3×3] with a channel number of 1 is used to extract features from var(G) with a step size of [1×1×1]. After batch normalization and nonlinear activation, the value of ε is obtained. By taking the sliding average of a and b in the [3×3×3] window, we get and , acting on the guidance image G, and obtaining the preliminary guidance filtering result:

[0075]

[0076] The residual compensation module is used to calculate the residual of i2 with G and GF to obtain two residual values ​​Res1 and Res2, connect their channel dimensions in parallel, and use a convolution kernel of size [3×3×3] with a channel number of 4 and a step size of [1×1×1] for feature extraction. After batch normalization and nonlinear activation, some useful information is obtained, and the obtained feature map is averaged in the channel dimension and summed with GF to obtain the final guided filtering result GFo.

[0077] The learnable Richardson-Lucy deconvolution module consists of three submodules, namely the 1 / 2 size deconvolution module H1, the original size deconvolution module H2 and the feature fusion module H3. The feature transfer process of modules H1 and H2 conforms to the Richardson deconvolution formula. In the H1 module, GFo is first average-pooled using a kernel of size [2×2×2] with a step size of [2×2×2] to make its length, width and height all become 1 / 2 of the original; then three consecutive convolutional layers are used for feature extraction. Each convolutional layer uses a convolution kernel of size [3×3×3] with a channel number of 4 and a step size of [1×1×1] for convolution. After batch normalization and nonlinear activation, the obtained feature maps are averaged in the channel dimension to simulate This process is simply called FP1. Then, by dividing the 1 / 2 size pooling result and FP1, we get something like The effect is recorded as DV1; then three consecutive convolutional layers are used to extract features from the results of DV1. Each convolutional layer uses a [3×3×3] convolution kernel with a channel number of 4 and a step size of [1×1×1] for convolution. After batch normalization and nonlinear activation, the feature maps obtained are averaged in the channel dimension to simulate Steps, this process is simply recorded as BP1. Then, transposed convolution is used with a convolution kernel of [2×2×2] and a step size of [2×2×2]. BP1 is upsampled after batch normalization and nonlinear activation to restore it to its original size. The feature map restored to its original size is multiplied with the guided filtering result GFo to simulate The result of one iteration is obtained, and this process is recorded as update1.

[0078] The H2 module is similar to the H1 module in structure, except that the H2 module does not require average pooling and transposed convolution upsampling operations. Each step is recorded as FP2, DV2, BP2, and update2. The results of update1 and update2 are concatenated in the channel dimension, and two consecutive convolutional layers are used to fuse and fine-tune the features. Each convolutional layer uses a [3×3×3] convolution kernel with a channel number of 8 and a step size of [1×1×1] for convolution. After batch normalization and nonlinear activation, and then a summation and averaging of the channel dimension, the overall output O of the network can be obtained. The activation function used in the network is softplus, which can ensure non-negativity and avoid the "dead zone" problem.

[0079] Step S5: Network training. Input i2 in the training set into the above network for training to obtain the corresponding result O. Use GT in the training set to supervise O and the intermediate results. The loss function used is:

[0080]

[0081] L2=MSE(O,GT)+(1-SSIM(GT,O))+λ3×(1-SSIM(GT,Ef))

[0082] Among them: L1 constrains the learnable guided filter module, L2 constrains the learnable Richardson-Lucy deconvolution module, Ef is the output result of the H1 module, λ1, λ2 and λ3 are weight factors (all set to 0.5 in this experiment), MSE is the root mean square error indicator, and SSIM is the structural similarity index:

[0083]

[0084] Where: x and y correspond to the specified parameters, N is the total number of pixels in the image, μ x ,μ y is the average of x and y, σ x ,σ y is the standard deviation of x and y, σ x,y is the correlation coefficient between x and y. To avoid the denominator being 0, set C1 = 1e -4 and C2=9e -4 .

[0085] During the training process, the guided filter module is first preheated for 1,000 iterations, and then the guided filter module and the Richardson-Lucy deconvolution module are mixed for 20,000 iterations. In the mixed training stage, after every 200 iterations, the network model is evaluated using the validation set, and the best model parameters for the validation set are saved.

[0086] Step S6: Reconstruction of real low signal-to-noise ratio data. Select the network model with the best performance in the validation set, and feed the real low signal-to-noise ratio biological data collected in S1 into the network to obtain high-quality reconstruction results.

[0087] The following experiments use low signal-to-noise ratio data obtained by imaging mEmerald-Tomm20-C-10 labeled mitochondria in living U2OS cells under low light dose conditions to verify the effectiveness of the present invention. We first generate a series of Figure 3 The {GT, i2} data pair shown in the figure sends i2 to Figure 4 The network structure shown in the figure is supervised by GT for training to obtain appropriate network parameters, and then the collected real mitochondrial low signal-to-noise ratio data is put into the trained network for reconstruction. The comparison method selected in the experiment is the three-dimensional residual attention network (RCAN) commonly used in fluorescence microscopy image reconstruction. RCAN is also trained using simulated data. The experimental environment is: Tensorflow1.14.0 framework, using Python3.7 as the programming language, and executed in the Ubuntu 18.04LTS operating system; the running environment for the training and testing process is: 32GB memory, Intel(R) Core(TM) i7-8700, 3.70GHz CPU, Nvidia TITANRTX GPUs with 24GB video memory.

[0088] The simulation data generation process proposed in the present invention greatly reduces the difficulty of data acquisition in the process of fluorescence microscopy image reconstruction. The proposed network structure has excellent generalization ability and can simply and effectively realize the reconstruction of low signal-to-noise ratio data. Figure 5 As shown, we can see that compared with the collected low signal-to-noise ratio mitochondrial data, the reconstruction result of the present invention can effectively remove noise and effectively restore mitochondrial details; the simulation data generation method proposed in the present invention can effectively promote the network to restore real biological samples, and both RCAN and the present invention method have improved the image reconstruction quality based on the simulation data. In addition, compared with RCAN, the network structure proposed in the present invention can better maintain the structure of mitochondria, and the reconstructed structure is closer to the reference data, which proves the reliability of its reconstruction.

[0089] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments. Improvements and modifications made by those skilled in the art to the present invention based on the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. A method for reconstructing low signal-to-noise ratio fluorescence microscopy data based on simulation data and model-guided deep learning, comprising the following steps: (1) fluorescently labeling a biological sample, acquiring images of the biological sample using a fluorescence microscope under extremely low light dose imaging conditions, and obtaining a three-dimensional fluorescence microscopic image of the biological sample by stacking a series of acquired two-dimensional images; (2) obtaining the point spread function of the fluorescence microscope by measuring or simulating the imaging parameters, and making the axial pixel size consistent with the lateral pixel size by interpolation; (3) generating a three-dimensional all-zero matrix, using a position random function to set a certain density of random-sized ball structures, circular ring structures, and three-dimensional space line structures within a limited size range at random positions in the matrix, each structure having a random intensity within a limited range, and then adding a constant background to the entire matrix to obtain simulation data; (4) blurring the simulation data using a three-dimensional Gaussian function to smooth its sharp boundaries, and downsampling the axial direction so that the ratio of its lateral pixel size to its axial pixel size is the same as the corresponding ratio of the three-dimensional fluorescence microscopy image, thereby obtaining clean and noise-free reference data; (5) degenerating the simulated data by using a point spread function through an imaging equation so that the simulated data has a blur similar to that of a three-dimensional fluorescence microscopy image, and then downsampling the simulated blurred data in the axial direction to obtain downsampled blurred data of the simulated blur; (6) introducing random noise into the downsampled fuzzy data through a Poisson random process and a Gaussian random process to obtain simulated degradation data; (7) Repeat steps (3) to (6) to obtain a large number of data pairs composed of simulated degradation data and their corresponding reference data, and divide all data pairs into training sets and validation sets; (8) Constructing a deep learning network driven by a guided filter cascade Richardson-Lucy deconvolution model, using the simulated degradation data in the training set data pair as the input of the network, and the corresponding reference data as the label of the network output, and then training the network through the loss function, and testing the network performance with the validation set; (9) By inputting the three-dimensional fluorescence microscopy image of the biological sample into the trained deep learning network, a high-quality reconstructed image can be output.

2. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 1, characterized in that: The extremely low light dose imaging condition is low excitation light intensity or short exposure time, and the collected two-dimensional image contains a lot of noise and diffraction blur; The point spread function is a three-dimensional matrix structure with Gaussian distribution, and its variation with space is not large.

3. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 1, characterized in that: The processing expression of step (4) is as follows: GT=[Gau(S,σ1)]↓ t Among them: GT is the reference data, S is the simulation data, Gau(S,σ1) represents the three-dimensional Gaussian smoothing of the simulation data S with a variance of σ1, Indicates downsampling the target axis at a factor of t.

4. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 1, characterized in that: The processing expression of step (5) is as follows: Where: i1 is the downsampled fuzzy data, S is the simulation data, represents the convolution operator, h is the point spread function, Indicates downsampling the target axis at a factor of t.

5. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 1, characterized in that: The processing expression of step (6) is as follows: i2=P(i1)+n Among them: i1 is the downsampled fuzzy data, i2 is the simulated degradation data, represents the Poisson random process of photon noise introduced in the imaging process, and n represents the Gaussian random noise introduced by the detection device in the imaging process.

6. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 1, characterized in that: The simulation data, downsampled fuzzy data, simulated degraded data and reference data are all three-dimensional volume data, and the ratio of the lateral pixel size to the axial pixel size of the downsampled fuzzy data, simulated degraded data and reference data is consistent with the corresponding ratio of the three-dimensional fluorescence microscopy image.

7. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 1, characterized in that: The deep learning network driven by the guided filter cascade Richardson-Lucy deconvolution model has the following expression for the computational reasoning process: G=Gau(i2,σ2); N_est=i2-G Where: e j+1 and e j They represent the reconstructed images of biological samples by the network during the j+1th and jth iterations, respectively. i2 is the simulated degradation data. G is the guided image generated by network learning. q is the preliminary guided filtering output result. a and b are the linear coefficients of the guided filtering. N_est is the estimated noise. Gau(i2,σ2) represents the three-dimensional Gaussian smoothing of i2 with a variance of σ2. represents the convolution operator, represents the covariance calculation, represents variance calculation, ε is the adjustment coefficient, It corresponds to the mean of i2, N_est, G, a, and b in the specified window. N is the maximum number of iterations. The initial reconstructed image e0 is q. F and B represent the forward projection operator and back projection operator of Richardson-Lucy deconvolution, respectively.

8. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 7, characterized in that: The deep learning network driven by the guided filtering cascade Richardson-Lucy deconvolution model includes a guided filtering module and a Richardson-Lucy deconvolution module, and the guided filtering module includes: A Gaussian mixture filter module based on Gaussian filter convolution and attention mechanism is used to generate the guidance image G; A guided filter calculation module based on linear coefficient solution is used to calculate linear coefficients a and b and apply their average value to the guided image G to obtain a preliminary guided filter result q; The residual compensation module based on convolution residual feature extraction is used to calculate the residual of the simulated degraded data i2 and G, q, and extracts features and nonlinearly activates the residual through convolution to obtain the final guided filtering result and output it to the Richardson-Lucy deconvolution module.

9. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 7, characterized in that: The Richardson-Lucy deconvolution module includes a 1 / 2 size deconvolution module H1, an original size deconvolution module H2 and a feature fusion module H3. The feature transfer process of H1 and H2 conforms to the Richardson deconvolution algorithm, including: ① fitting the forward projection process of Richardson-Lucy deconvolution through a convolution layer based on a Gaussian blur convolution kernel; ② calculating the ratio of the input to the forward projection result; ③ fitting the reverse projection process of the ratio through a common convolution layer to obtain an update factor; ④ applying the update factor to the previous iteration result to obtain a new iteration result; H1 and H2 have the same structure and the inputs are both the output results of the guided filtering module, the difference is that H1 needs to perform average pooling on the input so that its length, width and height all become 1 / 2 of the original size, the update factor obtained in H1 acts on the output result of the guided filtering module, the update factor obtained in H2 acts on the output result of H1, the output results of H1 and H2 are connected in parallel and input to H3, and H3 uses multiple full convolution modules to fine-tune the input feature map to obtain the output result of the entire network.

10. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 9, characterized in that: The expression of the loss function is as follows: L2=MSE(O,GT)+(1-SSIM(GT,O))+λ3×(1-SSIM(GT,Ef)) Where: L1 is the loss function of the guided filter module, L2 is the loss function of the Richardson-Lucy deconvolution module, λ1, λ2 and λ3 are weight factors, represents the root mean square error, represents the structural similarity index, GFo is the output result of the guided filtering module, GT is the reference data, O is the output result of H3, and Ef is the output result of H1.

Images