Fluorescent microscopic image reconstruction method based on noise compensation guided filtering and mismatched projection deconvolution

Abstract

The invention discloses a low-quality fluorescent microscopic image reconstruction method based on noise compensation guided filtering and unmatched projection deconvolution, which efficiently realizes pre-denoising of detail preservation and background noise suppression through guided filtering of noise compensation, and takes the pre-denoising as the input of an unmatched projection deconvolution method. The blurring caused by diffraction is removed, the pre-denoising step can effectively suppress noise amplification in the deconvolution process, and the whole process realizes the improvement of the signal-to-noise ratio of the image and the restoration of the resolution. Therefore, the method disclosed by the invention can be used for simply and effectively improving the quality of the image acquired under the low-light-dose imaging condition, and is beneficial to information acquisition of long-time fluorescent imaging of the living cells.

Description

Technical Field

[0001] The invention belongs to the technical field of fluorescence microscopy imaging, and in particular relates to a low-quality fluorescence microscopy image reconstruction method based on noise compensation guided filtering and mismatched projection deconvolution. Background Art

[0002] Fluorescence microscopy is widely used in cell biology research. However, biological cells are very sensitive to light. Prolonged or high-intensity illumination can cause phototoxicity and damage cell structure and function. In addition, fluorescent dyes or fluorescent proteins will gradually lose their fluorescence ability under long-term or high-intensity illumination, limiting the imaging time of fluorescence imaging experiments. In order to achieve long-term fluorescence imaging of living cells, researchers need to reduce the intensity of the excitation light or shorten the exposure time, resulting in a decrease in the number of photons per imaging, forming low-light-dose imaging conditions. Low light doses result in weak signal intensity and reduced signal-to-noise ratio. For diffraction-limited fluorescence microscopy systems, due to the diffraction properties of light, the collected data will also be affected by blurring degradation. Blurring and low signal-to-noise ratio are the main factors that cause image quality degradation under low-light-dose imaging conditions.

[0003] In order to improve the quality of the collected images, algorithms can be used to reconstruct low-quality images. For data that follows a Poisson distribution, the Richardson-Lucy algorithm based on maximum likelihood estimation is a common method for image blur restoration. By generalizing the back-projection process in the Richardson-Lucy algorithm, a mismatched projection deconvolution method (such as the patented technology with application number CN202010263899.1) can be constructed, which greatly improves the deconvolution efficiency. However, this iterative algorithm will cause noise amplification in the case of low signal-to-noise ratio.

[0004] In order to achieve denoising and deblurring at the same time, a regularization term can be introduced into the mismatched projection deconvolution method to suppress noise. However, since the mismatched projection deconvolution method requires very few iterations, the regularization term is difficult to play an effective role, and the regularization term is usually heavy in calculation, which will reduce the calculation efficiency. Therefore, in order to retain the iterative efficiency of the mismatched projection deconvolution method, low-quality data can be pre-denoised, and then the denoised data can be deconvolved; conventional denoising operations, such as Gaussian filters and wavelet filters, are equivalent to a low-pass filter, which makes the image details too blurred.

[0005] At present, there are some advanced filters that can preserve image details, such as block matching denoising (such as the patent technology with application number CN202111077118.0), bilateral filters (such as the patent technology with application number CN202311436747.7), etc., but these advanced filters usually have a high computational burden and low efficiency. Therefore, how to balance the effect and efficiency of the pre-denoising step and combine it with the mismatched projection deconvolution method to achieve low-quality image reconstruction for low-light dose imaging conditions is of great value to the development of fluorescent living cell imaging. Summary of the invention

[0006] In view of the above, the present invention provides a low-quality fluorescence microscopy image reconstruction method based on noise compensation guided filtering and mismatched projection deconvolution, which achieves high-quality restoration of low-quality images under low-light dose imaging conditions by combining a pre-denoising step and a deconvolution algorithm.

[0007] A low-quality fluorescence microscopic image reconstruction method based on noise compensation guided filtering and mismatched projection deconvolution comprises the following steps:

[0008] (1) fluorescently labeling the biological sample, using a fluorescence microscope to collect images of the biological sample under extremely low light dose imaging conditions, and stacking a series of collected two-dimensional images to obtain a three-dimensional fluorescence microscopic image of the biological sample;

[0009] (2) performing axial interpolation processing on the three-dimensional fluorescence microscopy image so that the step lengths represented by each pixel in the axial and lateral directions are the same, and then performing variance stabilization transformation on the three-dimensional fluorescence microscopy image to obtain a low-quality image that conforms to the additive noise characteristics; at the same time, the point spread function of the fluorescence microscope is obtained by measurement or simulation based on the imaging parameters and normalized;

[0010] (3) Use a filter to smooth the low-quality image to obtain a guided image and a pre-estimated noise image;

[0011] (4) According to the guide image and the pre-estimated noise image, a guided filtering algorithm based on noise compensation is used to denoise the low-quality image to obtain a blurred image with a high signal-to-noise ratio;

[0012] (5) The normalized point spread function is used as the forward projection operator and its corresponding Wiener-Butterworth back projection operator is calculated to construct a mismatched projection deconvolution algorithm. The blurred image is used as the input of the algorithm and a high signal-to-noise ratio and high-resolution reconstructed image is obtained through iteration.

[0013] Furthermore, the extremely low light dose imaging condition in step (1) is low excitation light intensity or short exposure time, and the acquired two-dimensional image contains a large amount of noise (Poisson noise and Gaussian noise) and has an extremely low signal-to-noise ratio; the point spread function is a three-dimensional matrix structure with a Gaussian distribution.

[0014] Furthermore, the degradation equation of the three-dimensional fluorescence microscopic image obtained by image acquisition in step (1) is expressed as follows:

[0015]

[0016] Where: o(x,y,z) represents the biological sample with coordinate point (x,y,z), p0(x,y,z) represents the pixel value of coordinate point (x,y,z) in the three-dimensional fluorescence microscopy image, h(x,y,z) represents the point spread function of coordinate point (x,y,z), n(x,y,z) represents the additive Gaussian noise of coordinate point (x,y,z) during imaging, and P() represents the Poisson random process that introduces photon noise during imaging. Represents the convolution operator.

[0017] Furthermore, in step (2), the three-dimensional fluorescence microscopy image is subjected to variance stabilization transformation by the following formula:

[0018]

[0019] Among them: p is a low-quality image, and p0 is a three-dimensional fluorescence microscopy image.

[0020] Furthermore, the calculation expressions of the guide image and the pre-estimated noise image in step (3) are as follows:

[0021] I=Gau(p,σ)

[0022] N_est=pI

[0023] Where: I is the guide image, Gau(p,σ) represents the three-dimensional Gaussian smoothing of the low-quality image p with variance σ, and N_est is the pre-estimated noise image.

[0024] Furthermore, the specific implementation method of step (4) is as follows: first, the low-quality image is denoised using the guided image and the pre-estimated noise image to obtain a smoothed image of the guided filter, then the difference between the guided image and the smoothed image is calculated as the residual noise, and high-frequency detail information is extracted from the residual noise; then, the smoothed image is summed and compensated using the high-frequency detail information to obtain a denoising result, and finally, the denoising result is subjected to a variance-stabilized inverse transform to obtain a blurred image with a high signal-to-noise ratio.

[0025] Furthermore, the calculation expression of the smoothed image is as follows:

[0026]

[0027] Where: k represents the filter window centered on pixel k, and m is the filter window ω k For any pixel in m Represents the value of pixel m in the smoothed image, I m represents the value of pixel m in the guide image, and The corresponding value is p k 、N_est k ,I k 、a k 、b k In the filter window ω k The mean value of pixels within, p k Represents the value of pixel k in the low-quality image, N_est k Represents the value of the pixel k in the estimated noise image, I k represents the value of pixel k in the guide image, a k and b k is the linear coefficient graph of guided filtering, represents the covariance calculation, represents variance calculation, and ε is the adjustment coefficient.

[0028] Furthermore, the calculation expression of the denoising result is as follows:

[0029] q=q′+ΔN

[0030] ΔN=γF(Iq′)

[0031] Where: q is the denoising result, q′ is the smoothed image, ΔN is the high-frequency detail information, I is the guide image, F() represents the smoothing function with the feature extraction mask, and γ is the gain coefficient of the residual noise.

[0032] Furthermore, the calculation expression of the Wiener-Butterworth back-projection operator in step (5) is as follows:

[0033] B=DFT -1 (B WB (k x ,k y ,k z ))

[0034]

[0035] Where: DFT() represents discrete Fourier transform, conj[] represents the complex conjugate operator, α is the parameter of the Wiener filter, k x , ky , k z Corresponding to the components of spatial frequency in the x, y, and z directions, k cx , k cy , k cz Corresponding to the components of the cutoff frequency in the x, y, and z axis directions, F is the forward projection operator, B is the Wiener-Butterworth back projection operator, ∈ is the maximum bandwidth gain of the Butterworth filter, n is the order of the Butterworth filter, and B WB (k x ,k y ,k z ) is the frequency domain representation of the Wiener-Butterworth back-projection operator.

[0036] Furthermore, the iterative process of the mismatched projection deconvolution algorithm in step (5) is expressed as follows:

[0037]

[0038] Where: F is the forward projection operator, B is the Wiener-Butterworth back-projection operator, represents the convolution operator, q1 is the blurred image, e j+1 and e j They represent the reconstructed images of the biological sample by the algorithm after the j+1th and jth iteration processes respectively.

[0039] The present invention provides an efficient reconstruction method for low-quality images acquired by extremely low signal-to-noise ratio imaging. By optimizing the denoising process of guided filtering, the pre-denoising results of detail preservation and background noise suppression are efficiently achieved, and the pre-denoising results are used as the input of the deconvolution method based on the mismatch operator to achieve the removal of blur caused by diffraction. The signal-to-noise ratio and contrast are restored through a set of processes. The reconstruction of low-quality fluorescence microscopy images under low light dose imaging conditions is achieved quickly and simply under low computational burden conditions, thereby assisting the realization of long-term fluorescence microscopy imaging of living cells. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Image data obtained by imaging ERmoxGFP-labeled endoplasmic reticulum in living U2OS cells using transient structured light illumination microscopy under low light dose conditions.

[0041] Figure 2 It is a schematic flow chart of the low-quality fluorescence microscopy image reconstruction method based on noise compensation guided filtering and mismatched projection deconvolution of the present invention.

[0042] Figure 3 The images reconstructed by the present invention for low-quality data are compared with the images reconstructed by other methods.

[0043] Figure 4Schematic diagram of the comparison of related indicators between the reconstructed images of the present invention and other methods and the reference data, where (a) is the structural similarity comparison, (b) is the peak signal-to-noise ratio comparison, and (c) is the running time comparison. DETAILED DESCRIPTION

[0044] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0045] In three-dimensional fluorescence microscopy, the excitation light path of the fluorescence microscope emits laser light to illuminate biological samples that have been fluorescently labeled or have autofluorescence, causing them to emit fluorescence and be collected by the detector. By continuously moving the imaging focal plane and traversing the entire biological sample, a volume image of the three-dimensional sample can be generated by stacking the two-dimensional images collected each time. Biological samples are usually sensitive to light intensity, and excessive light exposure can cause damage to cell structure and function; in addition, excessive light exposure can cause photobleaching of fluorescent markers. Therefore, in order to extend the imaging time as much as possible and image living cells for a long time, it is necessary to reduce the intensity of the excitation light collected each time or reduce the exposure time, which will reduce the number of photons per imaging. Combined with the diffraction of light during the imaging process, the image degradation caused by the imaging acquisition includes not only the inherent blur caused by the microscope system, but also a lot of noise.

[0046] like Figure 1 The image data shown is obtained by imaging the endoplasmic reticulum labeled with ERmoxGFP in living U2OS cells under low light dose conditions using an instantaneous structured light illumination microscope, in which the endoplasmic reticulum signal is submerged in a large amount of noise and its resolution is not high. In order to achieve the reconstruction of such low-quality images, denoising and deblurring are required. The Richardson-Lucy deconvolution method is a commonly used method for deblurring fluorescence microscopy images. It is based on the maximum likelihood function that obeys the Poisson distribution and restores the resolution in an iterative manner. The mismatched projection deconvolution is obtained through optimization design, and the efficiency of the algorithm is greatly improved after optimization. However, for data with extremely low signal-to-noise ratio, the mismatched projection deconvolution method will inevitably amplify the induced noise; in order to retain the high efficiency of the deconvolution method based on the mismatched projection operator, the present invention designs a low-quality fluorescence microscopy image reconstruction method based on noise compensation guided filtering and mismatched projection deconvolution. The noise compensation guided filtering is used to efficiently and easily suppress noise and retain details, and the mismatched projection deconvolution is used to improve the resolution. The entire process is as follows. Figure 2 As shown:

[0047] Step S1: Data acquisition. Under low light dose imaging conditions, biological sample data is collected, and under known imaging conditions, the point spread function of the system under the current imaging conditions is obtained by simulation or experimental means; through data preprocessing, the background intensity of the collected data is removed, and the collected data and the point spread function are interpolated to make the horizontal and axial pixel step sizes consistent.

[0048] Step S2: VST transformation. Under low light dose conditions, the noise in the collected data is mainly Poisson noise, which is a noise that changes with the signal intensity and is difficult to process. The present invention transforms the additive noise into additive noise through variance stabilizing transformations (VST), and the formula used is as follows:

[0049]

[0050] Step S3: Noise compensation guided filtering. Assuming the total noise is the difference between the image q′ after guided filtering and the image p before filtering, we can get:

[0051] N=pq′

[0052] Taking the guide image I as the intermediate parameter, the noise can be calculated as:

[0053] N=(pI)+(Iq′)=N_est+ΔN

[0054] q′+ΔN=p-N_est

[0055] The present invention performs preliminary denoising on the image p after VST transformation by a smoothing filter, obtains a guiding image I and calculates the estimated noise N_est; a typical smoothing filter is a Gaussian filter, and the corresponding guiding filter can be obtained by minimizing the loss function using I and N_est, and the minimization loss function is:

[0056]

[0057] By minimizing the loss function, the linear coefficients of the guided filter can be obtained:

[0058]

[0059] Where: ε is the adjustment coefficient. If ε is too small, the background noise cannot be effectively suppressed. If ε is too large, the result of the guided filtering will become blurred. It needs to be set according to the specific experiment. The result after guided filtering is expressed as:

[0060]

[0061] Noise compensation is obtained through Iq′. Since unnecessary background noise needs to be removed, the present invention uses a smoothing function with a feature extraction mask to process the compensation noise. The specific formula is as follows:

[0062] ΔN=γF(Iq′)

[0063] Since the guided filter linear coefficients have the ability to extract details, a typical smoothing function with a feature extraction mask can be selected as:

[0064]

[0065] Therefore, the final result of noise compensation guided filtering is:

[0066] q=q′+ΔN

[0067] In the process of generating the guidance image, the typical Gaussian blur variance is selected as 1.0. The lower the signal-to-noise ratio of the low-quality image, the larger the variance is selected. In the process of noise compensation, the typical Gaussian blur variance is selected as 1.5.

[0068] Step S4: Inverse VST transformation: Perform inverse variance stabilization transformation (inverse VST) on the result of noise compensation guided filtering to obtain high signal-to-noise ratio blurred data q1.

[0069] Step S5: mismatched projection deconvolution. Generate a forward projection operator and a corresponding reverse projection operator according to the system point spread function. The formula of the reverse projection operator is as follows:

[0070] B=DFT -1 (B WB (k x ,k y ,k z ))

[0071]

[0072] Among them: α is the parameter of the Wiener filter. If α is too large, the convergence speed will be slow, and if α is too small, it will easily lead to noise amplification; n is the order of the Butterworth filter, and ∈ is the maximum bandwidth gain of the filter. Through research, α is between 0.001 and 0.05, n is between 10 and 12, and ∈ is between 500 and 100.

[0073] The forward / backward projection operator is introduced into the iteration of the mismatched projection deconvolution algorithm, and the high signal-to-noise ratio blurred data q1 is substituted into the iteration. The specific iteration formula is as follows:

[0074]

[0075] Step S6: Determine whether the iteration stop condition is met, that is, whether the expected number of iterations has been reached. If the condition is not met, continue to execute step S5. If it is met, the iteration stops and the image estimate obtained by the last iteration is used as the estimate of the final sample. The number of iterations is usually set to 1 to 5.

[0076] Below, we use instantaneous structured light illumination microscopy to image mEmerald-Tomm20-C-10 labeled mitochondria in living U2OS cells under low light dose conditions to verify the effectiveness of the present invention. The data has an extremely low signal-to-noise ratio. This experiment compares different pre-denoising algorithms (Gaussian filtering, wavelet filtering, block matching, bilateral filtering, noise-free compensation guided filtering) combined with mismatched projection deconvolution, and compares it with direct mismatched projection deconvolution. The deconvolution results of images collected under high light dose conditions are used as a reference, and the structural similarity, peak signal-to-noise ratio and running time of different methods are compared.

[0077] pass Figure 3 and Figure 4 From the results, we can see that whether it is visual effect or quantitative numerical analysis, the low-quality image reconstruction method using the noise compensation guided filtering of the present invention and the mismatch operator deconvolution can achieve the best effect. It effectively suppresses background noise, and its results retain better details, and the mitochondrial shape is well maintained. Compared with other methods, the present invention is closer to the reference high-quality data, and the background is cleaner. Its running time is much better than that of advanced filters with detail preservation capabilities (block matching and bilateral filtering), and the use effect is better than fast simple filters (Gaussian filtering and wavelet filtering). This proves the high efficiency of the present invention, and the operation is simple. Usually, only the adjustment parameter ε of the guided filter needs to be experimentally adjusted, and other parameters can be fixed and used according to experience. It can easily and quickly restore low-quality images under low light dose conditions with high quality.

[0078] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments. Improvements and modifications made by those skilled in the art to the present invention based on the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. A method for reconstructing low-quality fluorescence microscopic images based on noise compensation guided filtering and mismatched projection deconvolution, comprising the following steps: (1) fluorescently labeling the biological sample, using a fluorescence microscope to collect images of the biological sample under extremely low light dose imaging conditions, and stacking a series of collected two-dimensional images to obtain a three-dimensional fluorescence microscopic image of the biological sample; (2) performing axial interpolation processing on the three-dimensional fluorescence microscopy image so that the step lengths represented by each pixel in the axial and lateral directions are the same, and then performing variance stabilization transformation on the three-dimensional fluorescence microscopy image to obtain a low-quality image that conforms to the additive noise characteristics; at the same time, the point spread function of the fluorescence microscope is obtained by measurement or simulation based on the imaging parameters and normalized; (3) Use a filter to smooth the low-quality image to obtain a guided image and a pre-estimated noise image; (4) According to the guide image and the pre-estimated noise image, a guided filtering algorithm based on noise compensation is used to denoise the low-quality image to obtain a blurred image with a high signal-to-noise ratio; (5) The normalized point spread function is used as the forward projection operator and its corresponding Wiener-Butterworth back projection operator is calculated to construct a mismatched projection deconvolution algorithm. The blurred image is used as the input of the algorithm and a high signal-to-noise ratio and high-resolution reconstructed image is obtained through iteration.

2. The low-quality fluorescence microscopic image reconstruction method according to claim 1, characterized in that: The extremely low light dose imaging condition in step (1) is low excitation light intensity or short exposure time, and the collected two-dimensional image contains a large amount of noise and has an extremely low signal-to-noise ratio; The point spread function is a three-dimensional matrix structure with Gaussian distribution.

3. The low-quality fluorescence microscopic image reconstruction method according to claim 1, characterized in that: The degradation equation of the three-dimensional fluorescence microscopic image obtained by image acquisition in step (1) is expressed as follows: Where: o(x,y,z) represents the biological sample with coordinate point (x,y,z), p0(x,y,z) represents the pixel value of coordinate point (x,y,z) in the three-dimensional fluorescence microscopy image, h(x,y,z) represents the point spread function of coordinate point (x,y,z), n(x,y,z) represents the additive Gaussian noise of coordinate point (x,y,z) during imaging, and P() represents the Poisson random process that introduces photon noise during imaging. Represents the convolution operator.

4. The low-quality fluorescence microscopic image reconstruction method according to claim 1, characterized in that: In the step (2), the three-dimensional fluorescence microscopy image is subjected to variance stabilization transformation by the following formula: Among them: p is a low-quality image, and p0 is a three-dimensional fluorescence microscopy image.

5. The low-quality fluorescence microscopic image reconstruction method according to claim 1, characterized in that: The calculation expressions of the guide image and the pre-estimated noise image in step (3) are as follows: I=Gau(p,σ) N_est=pI Where: I is the guide image, Gau(p,σ) represents the three-dimensional Gaussian smoothing of the low-quality image p with variance σ, and N_est is the pre-estimated noise image.

6. The low-quality fluorescence microscopic image reconstruction method according to claim 1, characterized in that: The specific implementation method of step (4) is as follows: first, the low-quality image is denoised using the guided image and the pre-estimated noise image to obtain a smoothed image of the guided filter, then the difference between the guided image and the smoothed image is calculated as the residual noise, and high-frequency detail information is extracted from the residual noise; then, the smoothed image is summed and compensated using the high-frequency detail information to obtain a denoising result, and finally, the denoising result is subjected to a variance-stabilized inverse transform to obtain a blurred image with a high signal-to-noise ratio.

7. The low-quality fluorescence microscopic image reconstruction method according to claim 6, characterized in that: The calculation expression of the smoothed image is as follows: Where: k represents the filter window centered on pixel k, and m is the filter window ω k For any pixel in m Represents the value of pixel m in the smoothed image, I m represents the value of pixel m in the guide image, and Corresponding to p k 、N_est k ,I k 、a k 、b k In the filter window ω k The mean value of pixels within, p k Represents the value of pixel k in the low-quality image, N_est k Represents the value of the pixel k in the estimated noise image, I k represents the value of pixel k in the guide image, a k and b k is the linear coefficient graph of guided filtering, represents the covariance calculation, represents variance calculation, and ε is the adjustment coefficient.

8. The low-quality fluorescence microscopic image reconstruction method according to claim 6, characterized in that: The calculation expression of the denoising result is as follows: q=q′+ΔN ΔN=γF(Iq′) Where: q is the denoising result, q′ is the smoothed image, ΔN is the high-frequency detail information, I is the guide image, F() represents the smoothing function with the feature extraction mask, and γ is the gain coefficient of the residual noise.

9. The low-quality fluorescence microscopic image reconstruction method according to claim 1, characterized in that: The calculation expression of the Wiener-Butterworth back-projection operator in step (5) is as follows: B=DFT -1 (B WB (k x ,k y ,k z )) Where: DFT() represents discrete Fourier transform, conj[] represents the complex conjugate operator, α is the parameter of the Wiener filter, k x , k y , k z Corresponding to the components of spatial frequency in the x, y, and z directions, k cx , k cy , k cz Corresponding to the components of the cutoff frequency in the x, y, and z axis directions, F is the forward projection operator, B is the Wiener-Butterworth back projection operator, ∈ is the maximum bandwidth gain of the Butterworth filter, n is the order of the Butterworth filter, and B WB (k x ,k y ,k z ) is the frequency domain representation of the Wiener-Butterworth back-projection operator.

10. The low-quality fluorescence microscopic image reconstruction method according to claim 1, characterized in that: The iterative process of the mismatched projection deconvolution algorithm in step (5) is expressed as follows: Where: F is the forward projection operator, B is the Wiener-Butterworth back-projection operator, represents the convolution operator, q1 is the blurred image, e j+1 and e j They represent the reconstructed images of the biological sample by the algorithm after the j+1th and jth iteration processes respectively.

Images