A deep learning-based aberration blur correction method for fluorescence microscopy images

Abstract

This invention discloses a method for correcting aberration blur in fluorescence microscopy images based on deep learning. This method proposes a forward model for synthesizing simulated blurry images, uses Zernike aberration patterns to characterize the aberrations that cause image blur, trains a deblurring model using a clearer image and the synthesized simulated blurry image, and then uses the deblurring model to restore the entire three-dimensional image. Theoretically, the blurry image in the three-dimensional image can be restored to a clarity comparable to that of the clearer image, thereby improving the image resolution. Compared to traditional methods, this method improves fluorescence microscopy technology by neither sacrificing temporal resolution nor increasing irradiation dose.

Description

Technical Field

[0001] The present invention belongs to the technical field of fluorescence microscopy, and in particular relates to a method for correcting aberration blur of fluorescence microscopy images based on deep learning. Background Art

[0002] Fluorescence microscopy can provide diffraction-limited imaging only in the absence of optical aberrations, which can arise from optical path differences introduced anywhere in the imaging path, including instrument misalignment, errors in the instrument's own optics, and refractive index inhomogeneities in the specimen. Specimen-induced imperfections are often the primary cause of optical aberrations and are often the reason why three-dimensional fluorescence images exhibit a significant deterioration in image signal-to-noise ratio, contrast, and resolution at greater depths. This effect is commonly observed in confocal or light-sheet microscopy, and much work has been done to compensate for or eliminate these aberrations, thereby removing image blur and providing sharper imaging deep within the specimen.

[0003] One approach to compensating for or even eliminating these aberrations is through adaptive optics (Ji, N. Adaptive optical fluorescence microscopy. Nature Methods 14, 374 (2017)). This is a widely used technique that first measures the aberrated wavefront and then applies an equal or opposite corrective wavefront to minimize the aberrations, helping to restore diffraction-limited imaging of the entire three-dimensional image. Once the distorted wavefront is determined, adaptive elements such as deformable mirrors or spatial light modulators are used to perform wavefront correction. Although this approach is effective in reducing aberrations, wavefront measurement is often difficult, requiring expensive hardware or bright fluorescent “guide stars” in the sample. Typically, the wavefront determination process is time-consuming, sacrificing temporal resolution during imaging and requiring a significantly higher radiation dose to the sample than would be achieved without adaptive optics correction. Therefore, a method for correcting aberrations without sacrificing temporal resolution or increasing radiation dose is needed.

[0004] In addition, some super-resolution methods based on deep learning can also be used to correct aberrations. For example, the paper [Gao Yuan, Liu Zhi, Qin Pinle, Wang Lifang. Medical image super-resolution algorithm based on deep residual generative adversarial network [J]. Computer Applications, 2018, 38(09): 2689-2695] proposed a medical image super-resolution method based on deep learning. This method blurs clear medical images and then uses the paired clear and blurred images to train a deblurring model. The trained deblurring model can be used to deblur medical images. However, in the process of generating the dataset, it only uses downsampling to perform blurring. In the real fluorescence microscopy imaging process, blur is usually caused by aberrations, which is different from the downsampling process. Therefore, this blurring method lacks persuasiveness in medical images. Summary of the Invention

[0005] In view of the above, the present invention provides a method for correcting aberration blur of fluorescence microscopy images based on deep learning, which can correct aberration blur of fluorescence microscopy images without sacrificing temporal resolution and increasing irradiation dose.

[0006] A method for correcting aberration blur of fluorescence microscopy images based on deep learning, comprising the following steps:

[0007] (1) Generate a forward model for simulating blurred images using Zernike aberration patterns;

[0008] (2) For the clearer image in the 3D image, the forward model is used to simulate and synthesize the blurred image;

[0009] (3) The clearer image and the corresponding synthesized blurred image are combined into training samples, and then the training samples are used to perform deep learning training on the deblurring model;

[0010] (4) The more blurred image in the three-dimensional image can be input into the trained deblurring model to restore the entire clear three-dimensional image.

[0011] Furthermore, the forward model is expressed as follows:

[0012] d(x,y,z)=s(x,y,z)*f(x,y,z)

[0013] Where: f(x,y,z) represents the target with horizontal coordinates (x,y) and vertical coordinates z, * represents the convolution operator, d(x,y,z) represents the three-dimensional image of the target, and s(x,y,z) represents the incoherent PSF (Point Spread Function) response function.

[0014] Furthermore, the incoherent PSF response function s(x, y, z) is expressed as follows:

[0015] s(x,y,z)=|h(x,y,z)| 2

[0016] Where: h(x,y,z) represents the three-dimensional point spread function.

[0017] Furthermore, the expression of the three-dimensional point spread function h(x, y, z) is as follows:

[0018]

[0019]

[0020] in: represents the inverse Fourier transform, H(u) is the pupil function of the imaging system, λ is the imaging wavelength, γ(u) is the axial Fourier space coordinate, u is the two-dimensional Fourier space coordinate, n is the refractive index of the medium in which the objective is immersed, and i is the imaginary unit.

[0021] Furthermore, in step (2), the blurred image is simulated and synthesized by the following expression:

[0022]

[0023] in: represents the inverse Fourier transform, represents Fourier transform, d0 represents the unblurred image, d represents the blurred image, is the optical transfer function of the blurred image, is the optical transfer function of the unblurred image, s represents the incoherent PSF response function, s0 represents the ideal PSF response function, noise represents noise, and α is the correction coefficient.

[0024] Furthermore, the optical transfer function and The ratio reflects the size of the added aberration. By adjusting the Zernike aberration coefficient, different degrees of aberration can be added to a clearer image.

[0025] Furthermore, the deblurring model may adopt a residual channel attention network (RCAN), a content-aware image restoration network (CARE), and the like.

[0026] Furthermore, compared with the clearer images in the three-dimensional fluorescence imaging, the blurriness of the blurry images is caused by the aberration in the imaging process, rather than being generated in the sensor or data processing process.

[0027] Furthermore, in step (3), the deblurring model is trained by deep learning, i.e., the simulated synthesized blurred image is used as the model input, and the deblurred image is output, and the clearer image is used as the true value label during training.

[0028] This paper proposes a forward model for synthesizing simulated blurry images. It uses Zernike aberration patterns to characterize the aberrations that cause image blur. A deblurring model is trained using both the sharper image and the synthesized simulated blurry image. This model is then used to restore the entire three-dimensional image. This theoretically restores the blurry image within the 3D image to a clarity comparable to that of the sharper image, improving image resolution. Compared to traditional methods, this method improves fluorescence microscopy by neither sacrificing temporal resolution nor increasing irradiation dose. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Light-sheet microscopy images of Caenorhabditis elegans embryos expressing histone GFP (labeling the cell nucleus).

[0030] Figure 2 Schematic diagram of the process of the fluorescence microscopy image aberration blur correction method of the present invention.

[0031] Figure 3 The figure shows the comparison of experimental results of the true clear image and the deblurred image at shallow and deep layers.

[0032] Figure 4 The following is a comparison of the experimental results of the true clear image and the deblurred image in the horizontal and vertical directions. DETAILED DESCRIPTION

[0033] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] In three-dimensional fluorescence microscopy, the image is blurred due to the presence of aberrations. One of the situations is that the aberrations of images at different depths in a three-dimensional fluorescence microscopy image are different, and the degree of image blur increases with increasing depth. Figure 1As shown, the upper image represents a lateral image 20 μm away from the surface, and the cell nucleus becomes blurred due to the presence of aberrations; the bottom image represents an axial image, and it can be seen that with increasing depth, the cell nucleus appears darker and more blurred, which indicates that with increasing depth, the aberration is also gradually increasing. Traditional aberration correction methods such as adaptive optics will sacrifice temporal resolution or increase the irradiation dose. In recent years, with the rise and development of deep learning, it has become possible to use deep learning to solve such problems. However, the use of deep learning often requires a data set as support. For image deblurring, we need a combination of blurred images and their corresponding clear images as a training set, but the process of obtaining clear images requires a lot of effort, making this work difficult to advance. Based on this, the present invention innovatively proposes a "forward model" for using clearer images to simulate and synthesize blurrier images, so that the simulated synthesized blurrier images are close to the real blurrier images in terms of aberration patterns, and then used for deep learning to finally complete the task of deblurring blurrier images. The entire process is as follows. Figure 2 shown.

[0035] The pupil function of the imaging system is given by:

[0036] H(u)=|P(u)|e iφ(u)

[0037] where φ(u) is the phase aberration in the data or synthesis process, and P(u) is the binary pupil mask covering the wavefront / spatial frequencies transmitted by the imaging system, expressed as follows:

[0038] P(u)=0,foru>NA / λ

[0039] Where: NA is the object-space numerical aperture, and λ is the imaging wavelength.

[0040] Phase aberration can be calculated using the Zernike basis function φ m (u) and its correlation coefficient c m To express:

[0041]

[0042] The three-dimensional point spread function related to the pupil function is given by the following formula. For detailed derivation, please refer to the literature [Hanser, BM, Gustafsson, MGL, Agard, DA & Sedat, JW Phase retrieval for high-numerical-aperture optical systems. Optics Letters 28, 801-803 (2003)].

[0043]

[0044]

[0045] Where: x, y represent the transverse coordinates, z represents the axial coordinate, and n represents the refractive index.

[0046] The incoherent PSF (suitable for widefield fluorescence microscopy) can be given by:

[0047] s(x,y,z)=|h(x,y,z)| 2

[0048] Given an object f(x,y,z) we can describe the forward model as follows:

[0049] d(x,y,z)=s(x,y,z)*f(x,y,z)

[0050] Where: * represents the convolution operator, and d(x, y, z) represents the resulting three-dimensional image.

[0051] We will then use this forward model to generate a blurred image. For convenience, the forward model omitting the coordinates (x, y, z) can be expressed as:

[0052] d=s*f

[0053] The following form is obtained through Fourier transform:

[0054]

[0055] in: stands for Fourier transform, This is the optical transfer function. When φ(u) = 0, there is no aberration. We can obtain a blur-free image d0 close to the diffraction limit based on the ideal PSF function s0, as shown in the following formula:

[0056]

[0057] According to the above two formulas, Eliminate and we get:

[0058]

[0059] To avoid division by zero, we add a small value α to correct the denominator, and finally add the noise term to get:

[0060]

[0061] The above equation provides a method to derive the blurred image d from the unblurred image d0. The optical transfer function of the blurred image is and the optical transfer function of a clear image The ratio of can be obtained through the Zernike aberration function and its correlation coefficient c m To make adjustments, this reflects the size of the aberration added to the image, that is, the degree of blur of the image.

[0062] Through this forward model, we can add aberrations to the clearer image by adjusting the Zernike aberration coefficients, and by comparing the actual blurry image with the aberration size basically the same as that at the required depth, we can obtain a simulated blurry image; repeating the above process, we can generate a sufficient amount of data set for training deep learning models.

[0063] There are many deep learning models used to restore clear images from blurred images, such as the Residual Channel Attention Network (RCAN) and the Content-Aware Image Restoration Network (CARE). After training the deep learning model, the trained model can be used to restore the blurred image to obtain a complete, unblurred three-dimensional image, thereby improving the image resolution.

[0064] The following experiment verifies the effectiveness of the present invention. In this experiment, we processed light sheet fluorescence microscopy images of Caenorhabditis elegans embryos expressing histone GFP (labeling the cell nucleus). Figure 3 This is a comparison of the experimental results of deep and shallow images. The first and second images represent lateral images at depths of 5μm and 28μm, respectively. The third image represents the simulated deep image generated by the forward model of the 5μm shallow image. It can be seen that its blurriness is roughly equivalent to that of the second image. The fourth image represents the restored effect of the 28μm deep image after deblurring using the trained neural network model. It can be seen that the clarity and brightness of the restored image have been significantly improved. Figure 4 This is a comparison of the experimental results of horizontal and longitudinal images. The two figures on the top are the true value and deblurred images of the horizontal image at 20μm, and the two figures on the bottom are axial slices of the entire three-dimensional image. It can be seen that after deblurring and restoring the entire three-dimensional image, both the shallow and deep images become clearer and brighter, and the degree of blur between the deep and shallow images is also roughly the same, which proves that the method of the present invention is effective.

[0065] The above description of the embodiments is intended to facilitate understanding and application of the present invention by those skilled in the art. It is apparent that those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without requiring creative effort. Therefore, the present invention is not limited to the above embodiments. Any improvements or modifications made by those skilled in the art based on the disclosure of the present invention should fall within the scope of protection of the present invention.

Claims

1. A method for correcting aberration blur in fluorescence microscopy images based on deep learning, comprising the following steps: (1) A forward model for simulating a blurred image is generated using the Zernike aberration model. The forward model is expressed as follows: d(x,y,z)=s(x,y,z)*f(x,y,z) s(x,y,z)=|h(x,y,z)| 2 in: f(x,y,z) represents the target with horizontal coordinates (x,y) and vertical coordinates z, * represents the convolution operator, d(x,y,z) represents the three-dimensional image of the target, s(x,y,z) represents the incoherent PSF response function, h(x,y,z) represents the three-dimensional point spread function, represents the inverse Fourier transform, H(u) is the pupil function of the imaging system, λ is the imaging wavelength, γ(u) is the axial Fourier space coordinate, u is the two-dimensional Fourier space coordinate, n is the refractive index of the medium in which the objective lens is immersed, and i is the imaginary unit; (2) For the clearer image in the three-dimensional image, the forward model is used to simulate and synthesize the blurred image through the following expression; in: represents Fourier transform, d0 represents the unblurred image, d represents the blurred image, is the optical transfer function of the blurred image, is the optical transfer function of the unblurred image, s represents the incoherent PSF response function, s0 represents the ideal PSF response function, noise represents noise, and α is the correction coefficient; (3) The clearer image and the corresponding synthesized blurred image are combined into training samples, and then the training samples are used to perform deep learning training on the deblurring model; (4) The more blurred image in the three-dimensional image can be input into the trained deblurring model to restore the entire clear three-dimensional image.

2. The method for correcting aberration blur of a fluorescence microscopic image according to claim 1, wherein: The optical transfer function and The ratio reflects the size of the added aberration. By adjusting the Zernike aberration coefficient, different degrees of aberration can be added to a clearer image.

3. The method for correcting aberration blur of a fluorescence microscopy image according to claim 1, wherein: The deblurring model may adopt a network model architecture including a residual channel attention network and a content-aware image restoration network.

4. The method for correcting aberration blur of a fluorescence microscopy image according to claim 1, wherein: Compared with clear images, the blurriness of the three-dimensional images of fluorescence imaging is caused by aberrations in the imaging process, rather than by the sensor or data processing.

5. The method for correcting aberration blur of a fluorescence microscopy image according to claim 1, wherein: In the step (3), the deblurring model is trained through deep learning, i.e., the simulated synthesized blurred image is used as the model input, and the deblurred image is output, and the clearer image is used as the true value label during training.

Images