Fluorescence microscopy data reconstruction method based on simulation data and model deep learning

Abstract

This paper discloses a method for reconstructing low-SNR fluorescence microscopy data based on simulation data and model-guided deep learning. This method uses real biological sample data collected under low-light-dose imaging conditions as a reference, generates a series of simulated data using imaging equations, and designs a deep learning network structure driven by a guided filter cascade Richardson-Lucy deconvolution model. This network structure, trained using simulation data, can be directly applied to real low-SNR data collected, achieving high-SNR, high-resolution reconstruction of low-SNR data. This method can simply and effectively improve the quality of images collected under low-SNR imaging conditions and alleviate the deep learning method's dependence on real-world data. This significantly reduces the difficulty of the deep learning data collection process and facilitates the acquisition of intelligent long-term fluorescence imaging information for living cells.

Description

Technical Field

[0001] The present invention belongs to the technical field of fluorescence microscopy imaging, and in particular relates to a low signal-to-noise ratio fluorescence microscopy data reconstruction method based on simulation data and model-guided deep learning. Background Art

[0002] Live cell dynamic long-term imaging technology is crucial for understanding the complex dynamic processes within cells. It can not only reveal the basic mechanisms of cellular life activities, such as cell division, migration, metabolic processes and signal transduction, but also provide an understanding of the development of diseases and the response of cells to drugs. Fluorescence microscopy can achieve high-resolution, high-contrast, and high-specificity imaging at the subcellular level, providing a powerful tool for the dynamics of living cells. However, long-term exposure to strong light can cause phototoxicity to cells, affecting their normal physiological activities and even leading to cell death. In addition, fluorescent molecules will gradually lose fluorescence under continuous illumination and undergo photobleaching, which limits the imaging time. In order to extend the duration of dynamic observation of fluorescent live cells, it is necessary to reduce the light dose of each light excitation during imaging, that is, to reduce the excitation light intensity or shorten the exposure time, so as to minimize phototoxicity and photobleaching.

[0003] Reducing the imaging light dose inevitably exacerbates the image degradation problem in fluorescence microscopy. The weakened imaging signal intensity makes the fluorescent signal in the image not bright enough, thus affecting the visibility of the target structure or molecule. Weaker fluorescence signals lead to a decrease in the signal-to-noise ratio, and the background noise becomes relatively more significant, resulting in a decrease in image quality, and the resolution and detail of the image may be affected. Therefore, image processing algorithms are very important for the reconstruction of images collected under low light dose imaging conditions. They can improve imaging quality and maintain low phototoxicity and photobleaching, thereby promoting the development of dynamic long-term imaging of living cells.

[0004] Image degradation under low-light-dose imaging conditions includes extremely high noise contamination and blur caused by diffraction. Therefore, corresponding image processing methods need to consider the effects of noise and blur separately. Currently, several algorithms exist for reconstructing blurred and noisy images. For example, Chinese patent application publication number CN118096568A provides an alternating iterative decoupling deconvolution image processing method based on a Poisson process, Chinese patent application publication number CN115293981A provides a method and system for denoising and super-resolution reconstruction of structured light illumination fluorescence microscopy images, and Chinese patent application publication number CN109636733A provides a method and system for fluorescence image deconvolution based on a deep neural network. These iterative processing techniques require careful parameter setting and suffer from poor real-time performance compared to deep learning-based methods. Deep learning-based methods rely on large amounts of data, but acquiring large amounts of paired high-quality / low-quality data is extremely difficult for dynamic imaging of living cells. In order to obtain efficient reconstruction of low-light-dose imaging data using deep learning methods, it is necessary to improve the generalization ability of the deep learning network and reduce its data dependence as much as possible; therefore, it is very important to study generalizable deep learning network training methods based on simulation data training. Summary of the Invention

[0005] In view of the above, the present invention provides a low signal-to-noise ratio fluorescence microscopy data reconstruction method based on simulation data and model-guided deep learning. By constructing a model-driven deep learning network, the generalization ability of the network is improved, and corresponding simulation training data pairs are generated according to the imaging equation. The model-driven deep learning network is trained by the simulation data, thereby realizing fast and efficient reconstruction of low signal-to-noise ratio data collected under low light dose imaging conditions.

[0006] A method for reconstructing low signal-to-noise ratio fluorescence microscopy data based on simulation data and model-guided deep learning, comprising the following steps:

[0007] (1) Fluorescently labeling a biological sample, acquiring images of the biological sample using a fluorescence microscope under extremely low light dose imaging conditions, and obtaining a three-dimensional fluorescence microscopic image of the biological sample by stacking a series of acquired two-dimensional images;

[0008] (2) obtaining the point spread function of the fluorescence microscope by measurement or simulation based on the imaging parameters, and making its axial pixel size consistent with the lateral pixel size by interpolation;

[0009] (3) Generate a three-dimensional all-zero matrix, use a position random function to set a certain density of random-sized spherical structures, circular ring structures, and three-dimensional spatial line structures within a limited size range at random positions in the matrix, and each structure has a random intensity within a limited range. Then, add a constant background to the entire matrix to obtain simulation data;

[0010] (4) blurring the simulated data using a three-dimensional Gaussian function to smooth its sharp edges, and downsampling the axial direction so that the ratio of its lateral pixel size to its axial pixel size is the same as the corresponding ratio of the three-dimensional fluorescence microscopy image, thereby obtaining clean and noise-free reference data;

[0011] (5) degenerating the simulated data using a point spread function through an imaging equation so that the simulated data has a blur similar to that of a three-dimensional fluorescence microscopy image, and then downsampling the simulated blurred data in the axial direction to obtain downsampled blurred data of the simulated blur;

[0012] (6) introducing random noise (noise intensity is close to or slightly stronger than the noise in the acquired two-dimensional image) into the downsampled fuzzy data through a Poisson random process and a Gaussian random process to obtain simulated degradation data;

[0013] (7) Repeat steps (3) to (6) to obtain a large number of data pairs composed of simulated degradation data and their corresponding reference data, and divide all data pairs into training sets and validation sets;

[0014] (8) Construct a deep learning network driven by a guided filter cascade Richardson-Lucy deconvolution model, using the simulated degradation data in the training set data pair as the network input and the corresponding reference data as the network output label. The network is then trained using a loss function and the network performance is tested using a validation set.

[0015] (9) By inputting the three-dimensional fluorescence microscopy image of the biological sample into the trained deep learning network, a high-quality reconstructed image can be output.

[0016] Furthermore, the extremely low light dose imaging condition is low excitation light intensity or short exposure time, and the collected two-dimensional image contains a large amount of noise (Poisson noise and Gaussian noise) and diffraction blur, low pixel intensity, and low signal-to-noise ratio.

[0017] Furthermore, the point spread function is a three-dimensional matrix structure with a Gaussian distribution, and its variation with space is not significant.

[0018] Furthermore, the processing expression of step (4) is as follows:

[0019] GT=[Gau(S,σ1)]↓ t

[0020] Where: GT is the reference data, S is the simulation data, Gau(S,σ1) represents the three-dimensional Gaussian smoothing of the simulation data S with a variance of σ1, Indicates downsampling the target axis at a rate of t.

[0021] Furthermore, the processing expression of step (5) is as follows:

[0022]

[0023] Where: i1 is the downsampled fuzzy data, S is the simulation data, represents the convolution operator, h is the point spread function, Indicates downsampling the target axis at a rate of t.

[0024] Furthermore, the processing expression of step (6) is as follows:

[0025] i2=P(i1)+n

[0026] Among them: i1 is the downsampled fuzzy data, i2 is the simulated degradation data, represents the Poisson random process of photon noise introduced during the imaging process, and n represents the Gaussian random noise introduced by the detection equipment during the imaging process.

[0027] Furthermore, the simulation data, downsampled fuzzy data, simulated degraded data and reference data are all three-dimensional volume data, and the ratio of the lateral pixel size to the axial pixel size of the downsampled fuzzy data, simulated degraded data and reference data is consistent with the corresponding ratio of the three-dimensional fluorescence microscopy image.

[0028] Furthermore, the computational reasoning process of the deep learning network driven by the guided filter cascade Richardson-Lucy deconvolution model is expressed as follows:

[0029]

[0030] G=Gau(i2,σ2); N_est=i2-G

[0031] Among them: e j+1 and e j They represent the reconstructed images of the biological sample by the network during the j+1th and jth iterations, respectively. i2 is the simulated degradation data, G is the guided image generated by network learning, q is the preliminary guided filtering result, a and b are the linear coefficients of the guided filtering, N_est is the estimated noise, and Gau(i2,σ2) represents the three-dimensional Gaussian smoothing of i2 with a variance of σ2. represents the convolution operator, represents the covariance calculation, represents variance calculation, ε is the adjustment coefficient, corresponds to the mean of i2, N_est, G, a, and b within the specified window, N is the maximum number of iterations, the initial reconstructed image e0 is q, F and B represent the forward projection operator and back projection operator of Richardson-Lucy deconvolution, respectively.

[0032] Furthermore, the deep learning network driven by the guided filtering cascade Richardson-Lucy deconvolution model includes a guided filtering module and a Richardson-Lucy deconvolution module, and the guided filtering module includes:

[0033] A Gaussian mixture filter module based on Gaussian filter convolution and attention mechanism is used to generate the guidance image G;

[0034] The guided filtering calculation module based on linear coefficient solution is used to calculate the linear coefficients a and b and apply their average value to the guidance image G to obtain the preliminary guided filtering result q;

[0035] The residual compensation module based on convolution residual feature extraction is used to calculate the residual of the simulated degradation data i2 and G, q. The residual is extracted and nonlinearly activated through convolution to obtain the final guided filtering result and output it to the Richardson-Lucy deconvolution module.

[0036] Furthermore, the Richardson-Lucy deconvolution module includes a 1 / 2 size deconvolution module H1, an original size deconvolution module H2 and a feature fusion module H3. The feature transfer process of H1 and H2 conforms to the Richardson deconvolution algorithm, including: ① fitting the forward projection process of the Richardson-Lucy deconvolution through a convolution layer based on a Gaussian blur convolution kernel (i.e. ); ② Calculate the ratio of input to forward projection result (i.e. ); ③ The update factor is obtained by fitting the ratio back projection process through the ordinary convolution layer (i.e. );④ Apply the update factor to the previous iteration result to obtain a new iteration result (i.e. ); H1 and H2 have the same structure and their inputs are both the output results of the guided filter module. The difference is that H1 needs to perform average pooling on the input so that its length, width and height are all reduced to 1 / 2 of the original size. The update factor obtained in H1 acts on the output result of the guided filter module, and the update factor obtained in H2 acts on the output result of H1. The output results of H1 and H2 are connected in parallel and input to H3. H3 uses multiple full convolution modules to fine-tune the input feature map to obtain the output result of the entire network.

[0037] Furthermore, the loss function is expressed as follows:

[0038]

[0039] L2=MSE(O,GT)+(1-SSIM(GT,O))+λ3×(1-SSIM(GT,Ef))

[0040] Where: L1 is the loss function of the guided filtering module, L2 is the loss function of the Richardson-Lucy deconvolution module, λ1, λ2 and λ3 are weight factors, represents the root mean square error, represents the structural similarity index, GFo is the output of the guided filtering module, O is the output of H3, and Ef is the output of H1.

[0041] In addition to using simulation data for training, the method of the present invention can also be trained using registered data pairs of low-quality data and high-quality data collected by a real fluorescence microscope.

[0042] The present invention uses a low-SNR fluorescence microscopy data reconstruction method based on simulation data and model-guided deep learning to generate random simulation data using simple structures such as spheres, three-dimensional rings, and three-dimensional line structures. The simulation data is degraded using imaging equations to simulate the imaging state of real low-light doses, and a simulated degraded image / high-quality noise-free true data pair is generated. Furthermore, the present invention designs a lightweight deep learning network through a model-driven approach to improve the network's interpretability and generalization capabilities. Through training on simulation data pairs, the model-driven network structure is equipped with the ability to simultaneously denoise and deblur low-SNR fluorescence microscopy data. The restoration of the signal-to-noise ratio and resolution is achieved through a set of processes, reducing the deep learning method's dependence on real-world acquired data and enabling high-quality image reconstruction, which is beneficial for the realization of dynamic, long-term imaging of living cells. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 (a) shows low signal-to-noise ratio data obtained by imaging mEmerald-Tomm20-C-10 labeled mitochondria in living U2OS cells using transient structured light illumination microscopy under low light dose conditions.

[0044] Figure 1 (b) shows high signal-to-noise ratio and high-resolution data obtained by imaging mEmerald-Tomm20-C-10 labeled mitochondria in living U2OS cells under high light dose conditions using transient structured light illumination microscopy.

[0045] Figure 2 Schematic diagram of the process of the present invention for reconstructing low signal-to-noise ratio fluorescence microscopy data based on simulation data and model-guided deep learning.

[0046] Figure 3 (a) is an example diagram of simulation data S.

[0047] Figure 3 (b) is an example of the simulated clean and noise-free reference data GT.

[0048] Figure 3(c) is an example diagram of simulated degradation data i2.

[0049] Figure 4 Schematic diagram of the deep learning network structure driven by the guided filter cascade Richardson-Lucy deconvolution model.

[0050] Figure 5 Comparison results of reconstructed images of real low-light-dose acquisition data using the method of the present invention and the existing reconstruction method RCAN. DETAILED DESCRIPTION

[0051] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] In three-dimensional fluorescence microscopy, fluorescently labeled biological samples are excited to emit fluorescence through the excitation light path, and the fluorescence signal is collected by the camera through the detection light path. A series of two-dimensional images can be obtained by moving the imaging focal plane, and these two-dimensional images can be stacked to obtain a volumetric image of the three-dimensional sample. Due to the influence of out-of-focus light, there is a defocus signal in each two-dimensional plane; in addition, due to the diffraction phenomenon of light, the resolution of the collected image will be affected, so blur is a major component of fluorescence microscopy image degradation. In the process of low-light-dose fluorescence microscopy, due to the use of lower excitation light intensity and shorter exposure time, the number of photons emitted by fluorescent molecules is reduced, resulting in a weaker captured fluorescence signal. In this case, the image is easily affected by reduced signal-to-noise ratio, photon statistical noise (Poisson noise) and detector quantization noise, which manifests as weakened signal intensity, decreased contrast and reduced resolution. In this case, extremely low signal-to-noise ratio and a large amount of noise are also one of the main components of image degradation. Figure 1 As shown in the figure, we used transient structured light illumination microscopy to image mEmerald-Tomm20-C-10 labeled mitochondrial samples in U2OS cells under low light dose and high light dose imaging conditions, respectively, showing the difference between images obtained under low light dose imaging conditions and high light dose imaging conditions. Under low light dose imaging conditions, the collected data contains a lot of noise and blur; therefore, in order to achieve the reconstruction of images collected under low light dose, it is necessary to consider both noise and blur at the same time.

[0053] In recent years, deep learning technology has played an important role in the field of image reconstruction; however, this technology usually requires a large amount of paired low-quality data and notes to form data pairs to achieve network training. The acquisition of such data pairs is difficult to achieve in dynamic imaging of living cells because cells are constantly changing; in addition, the acquisition of data pairs also requires a lot of labor. Using simulation data to train the network can greatly reduce the difficulty of obtaining training data; however, this puts forward requirements for the generalization ability of the network. Although traditional iterative methods are not smart enough, they often have good generalization capabilities; therefore, the present invention proposes a low signal-to-noise ratio fluorescence microscopy data reconstruction method based on simulation data and model-guided deep learning, which uses deep learning to fit the parts of the traditional iterative model that require manual design, which retains the generalization ability and interpretability of the traditional method and maintains the intelligence of deep learning; in addition, the present invention also proposes a simulation data generation process, which uses a deep learning network guided by a simulation data training model to perform high-quality reconstruction of data collected under low light doses. As Figure 2 As shown, the low signal-to-noise ratio fluorescence microscopy data reconstruction method based on simulation data and model-guided deep learning of the present invention includes the following steps:

[0054] Step S1: Real data acquisition. Under low excitation intensity or short exposure time imaging conditions, biological sample data is collected, and according to experimental conditions, the corresponding point spread function is obtained through simulation or experimental means, and the compensation size represented by the horizontal and axial pixels of the collected data is recorded, where the horizontal pixel size is recorded as s x , the axial pixel size is recorded as s z , the corresponding axial downsampling ratio relative to the transverse direction is t = s z / s x ; Through data preprocessing, the background intensity of the collected data is removed.

[0055] Step S2: Simulation data generation. First, a three-dimensional matrix is ​​generated (taking a matrix size of 128×128×128 as an example). A position randomization function is used to place spheres, rings, and three-dimensional line structures of random radius or random length within a limited size range (4-10 pixels) at random positions in the matrix. Each structure has a random intensity within a limited range. The specific intensity range is based on the intensity of the actual collected data. The corresponding number of each structure is based on the distribution density of the actual sample (or can be set to 100-200 each). A constant background is added to the entire matrix to obtain simulation data S.

[0056] Then, a three-dimensional Gaussian function is used to slightly blur S to smooth its sharp boundaries, and its axial direction is downsampled by a factor of t to serve as the clean and noise-free reference data GT for simulation.

[0057] Then, using the point spread function, S is degraded through the imaging equation to have a blur similar to that of the real collected data. The axial direction is downsampled by a factor of t to obtain the simulated blurred downsampled blurred data i1. Finally, random noise is introduced into i1 through Poisson random process and Gaussian random process to obtain the simulated degraded data i2. The noise intensity is close to or slightly stronger than that of the real collected data. The process of obtaining GT and i2 from S is described as follows:

[0058] GT=[Gau(S,σ1)]↓ t

[0059]

[0060] i2=P(i1)+n

[0061] Where: Gau() is a three-dimensional Gaussian blur function with a mean of 0 and a variance of σ1, []↓ t represents the downsampling of the axial direction at a magnification of t, P() represents the Poisson random process of photon noise introduced during the simulation imaging process, and n represents the Gaussian random process of noise introduced by the detection equipment during the imaging process. represents the convolution operator, and + represents the addition operation. Figure 3 The simulated data S, the simulated clean and noise-free reference data GT, and the simulated degraded data i2 are shown.

[0062] Repeat the above GT and i2 data generation process to generate a large number of {GT, i2} data pairs.

[0063] Step S3: Dataset preprocessing: The {GT, i2} data pairs obtained by simulation are divided into training and test sets in a ratio of 8:2. For example, when generating 500 data, 400 data pairs are used as training sets and 100 data pairs are used as validation sets.

[0064] Then, GT and i2 in each data pair are normalized. The normalization method is: arrange all pixel values ​​from small to large. According to the needs, the minimum value in the normalization process is selected from the 3% to 5% value, and the maximum value is selected from 100%. The details are as follows:

[0065]

[0066] Where: u represents the value to be normalized, percentile(u,p) represents the value of the p% position, and p high and p low are the high and low values ​​corresponding to the normalized standard, respectively.

[0067] Step S4: Network construction. The deep learning network driven by the guided filter cascade Richardson-Lucy deconvolution model includes two core modules: a learnable guided filter module and a learnable Richardson-Lucy deconvolution module. Figure 4 shown.

[0068] The learnable guided filter module consists of three submodules: the Gaussian mixture filter module, the guided filter calculation module, and the residual compensation module. The Gaussian mixture filter module is used to generate the guided image G and includes a Gaussian filter convolution layer and an attention mechanism layer. The Gaussian filter convolution layer uses a convolution kernel initialized with a Gaussian filter for feature extraction. Each channel of the convolution kernel is a Gaussian kernel with a different standard deviation. The standard deviations for each channel are σ = 0.5, 1, 1.5, and 2, respectively, and are defined as:

[0069]

[0070] Where: (c i ,c j ,c k ) is the center coordinate of each convolution kernel, and m is a random number to increase the randomness of each channel.

[0071] The attention mechanism layer enhances focus on important features by assigning different weights to different channels of the feature map obtained by the Gaussian filter convolution layer. Its operation process includes: first, global average pooling or maximum pooling is performed on the input feature map to extract a global description of each channel; then, these global descriptions are processed through a shared multi-layer perceptron network to generate attention weights for each channel; these weights are then element-wise multiplied by each channel of the original feature map to adjust the importance of each channel in the feature map and improve the model's ability to focus on key features. Finally, the guidance image G is obtained by averaging the data of each channel.

[0072] The noise map N_est is obtained by the difference between the input data i2 and the guided image G. i2, G and N_est are input into the guided filter calculation module, which first calculates the filter linear coefficients a and b. The calculation formula is:

[0073]

[0074] Among them: cov() is the calculated covariance, var() is the calculated variance, ε is the adjustment coefficient, are the mean values ​​of i2, N_est, and G in the specified window, respectively. Here, a typical window size is [3×3×3]. ε is learned through a single convolutional layer. A convolution kernel of size [3×3×3] with a channel number of 1 is used to extract features from var(G) with a step size of [1×1×1]. After batch normalization and nonlinear activation, the value of ε is obtained. By taking the sliding average of a and b in the [3×3×3] window, we get and , acting on the guidance image G, we get the preliminary guided filtering result:

[0075]

[0076] The residual compensation module is used to calculate the residual of i2 with G and GF, and obtain two residual values ​​Res1 and Res2. Their channel dimensions are connected in parallel, and a convolution kernel of size [3×3×3] with a channel number of 4 is used with a step size of [1×1×1] for feature extraction. After batch normalization and nonlinear activation, some useful information is obtained, and the obtained feature map is averaged in the channel dimension and summed with GF to obtain the final guided filtering result GFo.

[0077] The learnable Richardson-Lucy deconvolution module consists of three submodules, namely the 1 / 2 size deconvolution module H1, the original size deconvolution module H2 and the feature fusion module H3. The feature transfer process of modules H1 and H2 conforms to the Richardson deconvolution formula. In the H1 module, GFo is first average pooled using a kernel of size [2×2×2] with a step size of [2×2×2] to make its length, width and height all become 1 / 2 of the original; then three consecutive convolution layers are used for feature extraction. Each convolution layer uses a convolution kernel of size [3×3×3] with a channel number of 4 and a step size of [1×1×1] for convolution. After batch normalization and nonlinear activation, the obtained feature maps are averaged in the channel dimension to simulate This process is simply called FP1. Then, by dividing the 1 / 2 size pooling result and FP1, we get something like The effect is recorded as DV1; then three consecutive convolutional layers are used to extract features from the results of DV1. Each convolutional layer uses a [3×3×3] convolution kernel with a channel number of 4 and a step size of [1×1×1] for convolution. After batch normalization and nonlinear activation, the feature maps obtained are averaged in the channel dimension to simulate Step, this process is simply recorded as BP1. Then, transposed convolution is used with a convolution kernel of [2×2×2] and a step size of [2×2×2]. BP1 is upsampled after batch normalization and nonlinear activation to restore it to its original size. The feature map restored to its original size is multiplied with the guided filtering result GFo to simulate The result of one iteration is obtained, and this process is recorded as update1.

[0078] The H2 module has a similar structure to the H1 module, except that it does not require average pooling and transposed convolution upsampling operations. Each step is denoted as FP2, DV2, BP2, and update2. The results of update1 and update2 are concatenated in the channel dimension, and two consecutive convolutional layers are used to fuse and fine-tune the features. Each convolutional layer uses a [3×3×3] convolution kernel with 8 channels and a [1×1×1] stride. After batch normalization and nonlinear activation, and a summed average in the channel dimension, the network's overall output O is obtained. The activation function used in the network is softplus, which ensures non-negativity and avoids the "dead zone" problem.

[0079] Step S5: Network training. Input i2 in the training set into the above network for training to obtain the corresponding result O. Use the GT in the training set to supervise O and the intermediate results. The loss function used is:

[0080]

[0081] L2=MSE(O,GT)+(1-SSIM(GT,O))+λ3×(1-SSIM(GT,Ef))

[0082] Where: L1 constrains the learnable guided filter module, L2 constrains the learnable Richardson-Lucy deconvolution module, Ef is the output of the H1 module, λ1, λ2 and λ3 are weight factors (all set to 0.5 in this experiment), MSE is the root mean square error indicator, and SSIM is the structural similarity index:

[0083]

[0084] Where: x and y correspond to the specified parameters, N is the total number of pixels in the image, μ x ,μ y is the mean of x and y, σ x ,σ y is the standard deviation of x and y, σ x,y is the correlation coefficient between x and y. To avoid the denominator being 0, set C1 = 1e -4 and C2=9e -4 .

[0085] During the training process, the guided filter module is first preheated for 1,000 iterations, and then the guided filter module and the Richardson-Lucy deconvolution module are mixed for 20,000 iterations. During the mixed training phase, the network model is evaluated using the validation set after every 200 iterations, and the best model parameters for the validation set are saved.

[0086] Step S6: Reconstruction of low signal-to-noise ratio data collected in real life. The network model with the best performance in the validation set is selected and the low signal-to-noise ratio biological data collected in S1 is fed into the network to obtain high-quality reconstruction results.

[0087] The following experiments use low signal-to-noise ratio data obtained by imaging mEmerald-Tomm20-C-10 labeled mitochondria in living U2OS cells under low light dose conditions to verify the effectiveness of the present invention. We first generate a series of Figure 3 The {GT, i2} data pair shown is fed into Figure 4 The network structure shown in the figure was trained using GT supervision to obtain appropriate network parameters. Real mitochondrial low-signal-to-noise ratio data was then inserted into the trained network for reconstruction. The comparison method used in the experiment was the 3D Residual Attention Network (RCAN), a commonly used method for fluorescence microscopy image reconstruction. RCAN was also trained using simulated data. The experimental environment used the TensorFlow 1.14.0 framework, Python 3.7 as the programming language, and was executed on the Ubuntu 18.04 LTS operating system. The training and testing processes were performed on an Intel(R) Core(TM) i7-8700, 3.70 GHz CPU, and Nvidia TITANRTX GPUs with 24 GB of video memory, with 32 GB of memory.

[0088] The simulation data generation process proposed in this invention greatly reduces the difficulty of data acquisition in the process of fluorescence microscopy image reconstruction. The proposed network structure has excellent generalization ability and can simply and effectively realize the reconstruction of low signal-to-noise ratio data. Figure 5 As shown in the figure, we can see that compared to the collected low-signal-to-noise ratio mitochondrial data, the reconstruction results of the present invention can effectively remove noise and effectively restore mitochondrial details. The simulated data generation method proposed in the present invention can effectively promote the network's recovery of real biological samples. Both RCAN and the present invention's method achieved improved image reconstruction quality based on simulated data. In addition, compared with RCAN, the network structure proposed in the present invention can better preserve the mitochondrial structure, and the reconstructed structure is closer to the reference data, demonstrating the reliability of its reconstruction.

[0089] The above description of the embodiments is intended to facilitate understanding and application of the present invention by those skilled in the art. It is apparent that those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without requiring creative effort. Therefore, the present invention is not limited to the above embodiments. Any improvements or modifications made by those skilled in the art based on the disclosure of the present invention should fall within the scope of protection of the present invention.

Claims

1. A method for reconstructing low signal-to-noise ratio fluorescence microscopy data based on simulation data and model-guided deep learning, comprising the following steps: (1) Fluorescently labeling a biological sample, acquiring images of the biological sample using a fluorescence microscope under extremely low light dose imaging conditions, and obtaining a three-dimensional fluorescence microscopic image of the biological sample by stacking a series of acquired two-dimensional images; (2) obtaining the point spread function of the fluorescence microscope by measurement or simulation based on the imaging parameters, and making its axial pixel size consistent with the lateral pixel size by interpolation; (3) Generate a three-dimensional all-zero matrix, use a position random function to set a certain density of random-sized spherical structures, circular ring structures, and three-dimensional spatial line structures within a limited size range at random positions in the matrix, and each structure has a random intensity within a limited range. Then, add a constant background to the entire matrix to obtain simulation data; (4) blurring the simulated data using a three-dimensional Gaussian function to smooth its sharp edges, and downsampling the axial direction so that the ratio of its lateral pixel size to its axial pixel size is the same as the corresponding ratio of the three-dimensional fluorescence microscopy image, thereby obtaining clean and noise-free reference data; (5) degenerating the simulated data using a point spread function through an imaging equation so that the simulated data has a blur similar to that of a three-dimensional fluorescence microscopy image, and then downsampling the simulated blurred data in the axial direction to obtain downsampled blurred data of the simulated blur; (6) introducing random noise into the downsampled fuzzy data through a Poisson random process and a Gaussian random process to obtain simulated degradation data; (7) Repeat steps (3) to (6) to obtain a large number of data pairs composed of simulated degradation data and their corresponding reference data, and divide all data pairs into training sets and validation sets; (8) Construct a deep learning network driven by a guided filter cascade Richardson-Lucy deconvolution model, using the simulated degradation data in the training set data pair as the network input and the corresponding reference data as the network output label. The network is then trained using a loss function and the network performance is tested using a validation set. (9) By inputting the three-dimensional fluorescence microscopy image of the biological sample into the trained deep learning network, a high-quality reconstructed image can be output.

2. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 1, characterized in that: The extremely low light dose imaging condition is low excitation light intensity or short exposure time, and the collected two-dimensional image contains a lot of noise and diffraction blur; The point spread function is a three-dimensional matrix structure with a Gaussian distribution and has little variation in space.

3. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 1, characterized in that: The processing expression of step (4) is as follows: GT=[Gau(S,σ1)]↓ t Where: GT is the reference data, S is the simulation data, Gau(S,σ1) represents the three-dimensional Gaussian smoothing of the simulation data S with a variance of σ1, Indicates downsampling the target axis at a rate of t.

4. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 1, characterized in that: The processing expression of step (5) is as follows: Where: i1 is the downsampled fuzzy data, S is the simulation data, represents the convolution operator, h is the point spread function, Indicates downsampling the target axis at a rate of t.

5. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 1, characterized in that: The processing expression of step (6) is as follows: i2=P(i1)+n Among them: i1 is the downsampled fuzzy data, i2 is the simulated degradation data, represents the Poisson random process of photon noise introduced during the imaging process, and n represents the Gaussian random noise introduced by the detection equipment during the imaging process.

6. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 1, characterized in that: The simulation data, downsampled fuzzy data, simulated degraded data and reference data are all three-dimensional volume data, and the ratio of the lateral pixel size to the axial pixel size of the downsampled fuzzy data, simulated degraded data and reference data is consistent with the corresponding ratio of the three-dimensional fluorescence microscopy image.

7. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 1, characterized in that: The computational reasoning process of the deep learning network driven by the guided filter cascade Richardson-Lucy deconvolution model is expressed as follows: G=Gau(i2,σ2); N_est=i2-G Among them: e j+1 and e j They represent the reconstructed images of the biological sample by the network during the j+1th and jth iterations, respectively. i2 is the simulated degradation data, G is the guided image generated by network learning, q is the preliminary guided filtering output result, a and b are the linear coefficients of the guided filtering, N_est is the estimated noise, and Gau(i2,σ2) represents the three-dimensional Gaussian smoothing of i2 with a variance of σ2. represents the convolution operator, represents the covariance calculation, represents variance calculation, ε is the adjustment coefficient, corresponds to the mean of i2, N_est, G, a, and b within the specified window, N is the maximum number of iterations, the initial reconstructed image e0 is q, F and B represent the forward projection operator and back projection operator of Richardson-Lucy deconvolution, respectively.

8. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 7, characterized in that: The deep learning network driven by the guided filtering cascade Richardson-Lucy deconvolution model includes a guided filtering module and a Richardson-Lucy deconvolution module, wherein the guided filtering module includes: A Gaussian mixture filter module based on Gaussian filter convolution and attention mechanism is used to generate the guidance image G; The guided filtering calculation module based on linear coefficient solution is used to calculate the linear coefficients a and b and apply their average value to the guidance image G to obtain the preliminary guided filtering result q; The residual compensation module based on convolution residual feature extraction is used to calculate the residual of the simulated degradation data i2 and G, q. The residual is extracted and nonlinearly activated through convolution to obtain the final guided filtering result and output it to the Richardson-Lucy deconvolution module.

9. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 7, characterized in that: The Richardson-Lucy deconvolution module includes a 1 / 2 size deconvolution module H1, an original size deconvolution module H2 and a feature fusion module H3. The feature transfer process of H1 and H2 conforms to the Richardson deconvolution algorithm, including: ① fitting the forward projection process of Richardson-Lucy deconvolution through a convolution layer based on a Gaussian blur convolution kernel; ② calculating the ratio of the input and the forward projection result; ③ fitting the reverse projection process of the ratio through a normal convolution layer to obtain an update factor; ④ applying the update factor to the previous iteration result to obtain a new iteration result; H1 and H2 have the same structure and the inputs are both the output results of the guided filter module, the difference is that H1 needs to perform average pooling on the input so that its length, width and height are all reduced to 1 / 2 of the original size, the update factor obtained in H1 acts on the output result of the guided filter module, and the update factor obtained in H2 acts on the output result of H1, the output results of H1 and H2 are connected in parallel and input to H3, and H3 uses multiple full convolution modules to fine-tune the input feature map to obtain the output result of the entire network.

10. The low signal-to-noise ratio fluorescence microscopy data reconstruction method according to claim 9, characterized in that: The expression of the loss function is as follows: L2=MSE(O,GT)+(1-SSIM(GT,O))+λ3×(1-SSIM(GT,Ef)) Where: L1 is the loss function of the guided filtering module, L2 is the loss function of the Richardson-Lucy deconvolution module, λ1, λ2 and λ3 are weight factors, represents the root mean square error, represents the structural similarity index, GFo is the output result of the guided filtering module, GT is the reference data, O is the output result of H3, and Ef is the output result of H1.

Images