Computational adaptive optical microscopic imaging method based on sparse blind deconvolution

Abstract

The invention provides a computational adaptive optical microscopic imaging method based on sparse blind deconvolution. According to the method, sparse prior of a fluorescence sample and an aberration correction algorithm based on a Zernike polynomial are combined, compared with traditional Richardson-Lucy deconvolution, a point spread function and sample information can be reconstructed from a single blurred image at the same time, the requirement for accurate point spread function calibration is avoided, and the method is suitable for large-scale popularization and application. And the method is better in the aspects of improving the robustness to noise and stabilizing the deconvolution process. According to the invention, the image resolution and contrast of the wide-field and confocal fluorescence microscope system are obviously improved, and the wide-field and confocal fluorescence microscope system is suitable for urgent demands in the fields of dynamic life science research, medical imaging and the like, and has a wide application prospect.

Description

Technical Field

[0001] The present invention belongs to the technical field of optical microscopy imaging, and particularly relates to a computational adaptive optical microscopy imaging method based on sparse blind deconvolution. Background Art

[0002] In the field of life sciences, optical microscopes are recognized as indispensable tools for exploring the complexity of the microscopic world, and play a crucial role in the exploration of subcellular dynamics, neuroscience research, tumor diagnosis, and many other fields [C. Zuo, J. Li, J. Sun, Y. Fan, J. Zhang, L. Lu, R. Zhang, B. Wang, L. Huang, Q. Chen, Optics and Lasers in Engineering 2020, 135, 106187]. The historical evolution of microscopes is characterized by a continuous pursuit of enhanced imaging capabilities and reliance on complex optical systems. This progress has helped to reveal fine details in cell structures and capture the complex dynamics of life processes. However, the presence of optical aberrations poses a significant challenge to achieving perfect imaging [S. F. Gibson, F. Lanni, JOSA A 1991, 8, 101601]. The performance of optical microscopes is fundamentally limited by the design parameters of the optical system, and the presence of aberrations will reduce the resolution and contrast of the microscope. Therefore, a deep understanding of these aberrations and the development of effective correction methods are crucial for promoting high-resolution imaging.

[0003] To address these issues, Adaptive Optics (AO) is integrated into microscopes to correct aberrations caused by non-uniform sample refractive index, component defects, microscope misalignment, etc. AO typically uses a Deformable Mirror (DM) array or a Spatial Light Modulator (SLM) to actively correct wavefront aberrations. By guiding light rays from a single light source to a common focus on the sensor at different angles, high-resolution imaging is achieved [T. Furieri, D. Ancora, G. Calisesi, S. Morara, A. Bassi, S. Bonora, Biomedical Optics Express 2021, 13, 1262.]. However, the integration of this hardware with various microscope optical paths limits its general applicability. To overcome this, techniques such as Fourier Ptychographic Microscopy (FPM) and Differential Phase Contrast (DPC) microscopy [R. W. Gerchberg, Optik 1972, 35, 2237.] do not rely on additional compensation hardware and use phase retrieval algorithms for aberration correction. Utilizing the inherent data redundancy, they simultaneously recover the complex field of the specimen and the spatially varying aberrations of the system to reconstruct high-quality images. However, these methods usually require acquiring multiple images to achieve aberration correction.

[0004] Although traditional adaptive optics methods perform well in aberration correction, they usually require complex optical components and are cumbersome to operate. And in practical applications, they may be restricted by various factors such as the need to acquire multiple images or additional hardware.

[0005] In summary, there is currently a lack of a computational adaptive optics microscopy imaging method that is fast, has high resolution and imaging contrast, and can achieve high-quality aberration correction. Summary of the Invention

[0006] To overcome the above-mentioned deficiencies of the prior art, the present invention provides a computational adaptive optics microscopy imaging method based on sparse blind deconvolution.

[0007] The technical solution for achieving the object of the present invention is: A computational adaptive optics microscopy imaging method based on sparse blind deconvolution, comprising:

[0008] Step 1: Take a fluorescence image using the fluorescence incoherent microscopy imaging mode of the imaging system;

[0009] [[ID=,20]]Step 2: Initialize the point spread function;

[0010] Step 3: Iteratively update the ideal image based on sparse prior estimation and the current point spread function;

[0011] Step 4: Iteratively update the point spread function based on the current ideal image;

[0012] Step 5: Loop through Steps 3 and 4 until the change in the point spread function is less than a set threshold, and use Zernike polynomials to perform aberration fitting on the estimated point spread function to solve for the point spread function with aberration;

[0013] Step 6: Perform deconvolution iteration on the fluorescence image using the point spread function with aberration obtained in Step 5 to achieve image deblurring.

[0014] Preferably, the imaging system includes an incoherent illumination light source, an excitation / emission filter assembly that matches the sample band, an objective imaging optical path, and a camera. The sample is excited by spatially incoherent light generated by the incoherent illumination light source, and the fluorescence signal is separated by the excitation / emission filter assembly and then reaches the camera through the objective imaging optical path to achieve low-noise wide-field fluorescence imaging.

[0015] Preferably, the specific method for initializing the point spread function is as follows:

[0016] Initialize the point spread function h( 0 ) as the ideal point spread function, that is, let:

[0017]

[0018] where (ξ,η) are the pupil plane coordinates, P0(ξ,η) is the circular aperture of the imaging system, denotes the Fourier transform, and * denotes the correlation operation.

[0019] Preferably, the specific method for iteratively updating the ideal image based on sparse prior estimation and the current point spread function is as follows:

[0020] Assume that the probability density p(x (n) ) of each pixel value in the ideal image x (n) satisfies a Gaussian distribution, that is, let:

[0021]

[0022] where n is the number of iterations, σ represents the standard deviation of the gradient image, f γ is the γ-th gradient filter, f γ *x (n) represents the gradient image of x (n) , and Π represents the product operation;

[0023] Assume that the noise satisfies a normal distribution, and the noise probability density p(y|x (n) ,h (n-1) ) is:

[0024]

[0025] Where ζ is the standard deviation of the noise, and y is the actual fluorescence image captured in Step 1;

[0026] Maximize the probability density of the image x (n-1) given the known point spread function h (n) and the actual image y, that is, maximize the probability density of the following formula:

[0027] p(y,x (n) |h (n-1) = p(y|x (n) ,h (n-1) )·p(x (n ))

[0028] When the above formula is maximized, the corresponding ideal image x (n) is:

[0029]

[0030] Where and is the Toeplitz matrix corresponding to the point spread function h (n-1) , and T γ is the Toeplitz matrix corresponding to the γ-th derivative filter.

[0031] Preferably, the point spread function is iteratively updated based on the current ideal image, and the specific formula is:

[0032]

[0033] Where and is the Toeplitz matrix corresponding to x (n) .

[0034] Preferably, the aberration fitting of the estimated point spread function is performed using Zernike polynomials, and the specific method for solving the point spread function with aberration is:

[0035] Determine the pupil function, specifically:

[0036]

[0037] Where ξ and η are spatial frequencies, M is the highest order of the Zernike polynomial, c m is the coefficient of each orthogonal Zernike polynomial Z m , and c is a matrix containing M such coefficients;

[0038] For the estimated point spread function h NThe coefficient matrix is solved by the least squares method as follows:

[0039]

[0040] Thus, the point spread function h with aberration is solved:

[0041]

[0042] Preferably, Richardson-Lucy deconvolution is used to restore the fluorescence image, and the specific iteration formula is as follows:

[0043]

[0044] where (u, v) represents the image coordinates, and f (k) (u, v) is the k-th estimate of the deblurred image, and f (k+1) (u, v) is the (k + 1)-th estimate, y(u, v) is the fluorescence real-shot image acquired in step 1, and h(u, v) is the point spread function solved in step 5.

[0045] Compared with the prior art, the significant advantages of the present invention are as follows: (1) The present invention does not require precise point spread function calibration. Through the sparse blind deconvolution method, the point spread function and the original image can be estimated simultaneously from a single blurred image, simplifying the imaging process. (2) The present invention improves the noise robustness. Compared with the traditional method, the present invention uses the sparsity of the fluorescence image and the point spread function constrained by Zernike polynomials as prior knowledge to suppress noise and extract high-frequency information simultaneously. Even in the case of a low signal-to-noise ratio, a high-quality image can still be restored. (3) The present invention realizes high-quality aberration correction. By combining the sparse prior and the Zernike polynomial constraint, the point spread function and aberration can be estimated more accurately, enhancing the image resolution and contrast. (4) It has universality. The present invention not only performs excellently on wide-field fluorescence microscopes and confocal laser scanning microscopes, but also applies to various imaging systems.

[0046] The following further describes the present invention in detail with reference to the accompanying drawings and specific embodiments. Description of the Drawings

[0047] Figure 1 It is a flowchart of the computational adaptive optical microscopy imaging method based on sparse blind deconvolution of the present invention.

[0048] Figure 2Simulation results for the USAF resolution target. Figure a shows the USAF target image under ideal conditions and the blurred results considering the influence of aberrations; Figure b shows the comparison of image reconstruction results when using different point spread functions (ideal point spread function, the spread function estimated by only adding sparse prior, the spread function obtained by jointly applying sparse prior and Zernike constraint to the point spread function, and the true value of the point spread function); Figure c1 shows the simulated pupil aberration, point spread function, and the comparison with the estimated values of our blind deconvolution method; Figure c2 shows the comparison of pupil aberration, point spread function estimated by different methods, and the corresponding deconvolution reconstruction results; Figure c3 shows the line profiles of the line segments in the deconvolved image to compare the imaging resolution and contrast.

[0049] Figure 3 Results of wide-field fluorescence experiments on BPAE cells. Figure a shows the observation results and image reconstruction effect comparison of F-actin in bovine pulmonary artery endothelial cells (BPAEC) labeled with phalloidin-AF488 under a wide-field fluorescence microscope; Figures b1 - b3 magnify and show the detail comparison of the labeled area in Figure a, presenting the reconstruction results of the original wide-field fluorescence image, the traditional RL deconvolution method, and the blind deconvolution method we proposed; Figures c1 - c3 show the cross-sectional views of different colored line segments in Figures b1 - b3; Figure d shows the observation results and image reconstruction effect comparison of mitochondria in bovine pulmonary artery endothelial cells labeled with the mitochondrial red fluorescent probe (MitoTracker TM Red CMXRos) under wide-field fluorescence; Figures e1 - e3 magnify and show the mitochondrial structure of the labeled area in Figure d, comparing the reconstruction results of the original wide-field fluorescence image, the traditional RL deconvolution method, and the blind deconvolution method we proposed. Figures f1 - f3 show the cross-sectional views of different colored line segments in Figures e1 - e3.

[0050] Figure 4 ]>Results of experiments on multi-labeled bovine pulmonary artery endothelial cells under a confocal laser scanning microscope. Figure a shows the comparison of multi-channel fluorescence fusion results, the confocal observation image, and the deconvolution reconstruction results, including F-actin, mitochondria, and nuclei; Figures b1 - b3 magnify and show the mitochondrial structure within the rectangular area in Figure a, highlighting the characteristics of mitochondrial structures in different regions; Figures c1 - c3 show the cross-sectional views of different colored line segments in Figures b1 - b3. Detailed implementation manners

[0051] For the convenience of understanding and implementing the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0052] As an inventive concept, a computational adaptive optical microscopy imaging method based on sparse blind deconvolution of the present invention includes: estimating a point spread function under a sparse prior and performing aberration modeling on the estimated point spread function based on Zernike polynomials to attempt to reconstruct an original image without any system knowledge. A regularizer is used to solve an approximation of this problem, and prior knowledge of natural image statistics, particularly the sparsity of the derivative distribution of fluorescence imaging, is utilized to identify a point spread function kernel that maximizes the posterior probability, and the obtained point spread function is further refined through aberration estimation. To achieve single-image aberration correction, the present invention introduces prior knowledge of common aberrations into the reconstruction process, describes pupil distortion with Zernike polynomials as basis functions, simplifies the degrees of freedom of the solution space from a two-dimensional matrix to a minimum coefficient set of Zernike dominant modes, thereby optimizing the reconstruction of a limited original image.

[0053] As an embodiment, the specific steps of a computational adaptive optical microscopy imaging method based on sparse blind deconvolution are illustrated in combination with simulations:

[0054] Step 1: Take a fluorescence image using the fluorescence incoherent microscopy imaging mode of the imaging system. The imaging system includes an incoherent illumination light source, an excitation / emission filter component matching the sample band, an objective imaging optical path, and a camera. Spatial incoherent light is generated by the incoherent illumination light source to excite the sample, and the fluorescence signal is separated by the excitation / emission filter component and then reaches the camera through the objective imaging optical path to achieve low-noise wide-field fluorescence imaging. To facilitate the analysis of contrast and resolution, Figure 2 the USAF resolution target shown in Fig. a is used as the observation target, and the point spread function shown in this figure is used as the blur kernel, and its corresponding aberration is also plotted in Figure 2 Fig. a.

[0055] Step 2: Initialize the point spread function. Initialize the point spread function h( 0 ) as an ideal point spread function, that is, let:

[0056] [[ID= / / ]]

[0057] where (ξ, η) are the pupil plane coordinates, P0(ξ, η) is the circular aperture of the imaging system, denotes performing a Fourier transform, and ★ denotes a correlation operation.

[0058] Step 3: Iteratively update the ideal image x (n-1) based on the sparse prior estimation and the current point spread function h (n) . Since the fluorescence intensity of the sample in fluorescence imaging is mainly concentrated in a few pixels, the probability of brighter pixels in its gradient image is relatively low. It can be assumed that the probability density p(x (n) ) of each pixel value in the ideal image x (n) ) satisfies a Gaussian distribution, that is, let:

[0059]

[0060] Wherein, n is the iteration number starting from 1, σ represents the standard deviation of the gradient image, and f γ is the γ-th gradient filter, and f γ *x (n) represents the gradient image of x (n) , and Π represents the cumulative multiplication operation. Assuming that the noise follows a normal distribution, its probability density p(y|x (n) ,h (n-1) ) is:

[0061]

[0062] Wherein ζ is the standard deviation of the noise, and y is the actual fluorescence image taken in Step 1.

[0063] Maximize the probability density of the image x (n-1) under the condition of the known point spread function h (n) and the actual image y, that is, it is equivalent to maximizing the probability density of the following formula:

[0064] p(y,x (n) |h (n-1) ) = p(y|x (n) ,h (n-1) )·p(x (n) )

[0065] When the above formula is maximized, the corresponding ideal image x (n) is:

[0066]

[0067] Wherein and is the Toeplitz matrix corresponding to the point spread function h (n-1) , and T γ is the Toeplitz matrix corresponding to the γ-th derivative filter.

[0068] Step 4: Iteratively update the point spread function h (n) based on the current ideal image x (n) . The optimal point spread function kernel h (n) can be expressed as:

[0069]

[0070] Wherein and is the Toeplitz matrix corresponding to x (n) .

[0071] Step 5: Loop steps 3 and 4 until the change in the point spread function h (n) is less than the set threshold. Use Zernike polynomials to perform aberration fitting on the estimated point spread function, thereby solving the point spread function h with aberration. Here, considering that Zernike polynomials are often used to express pupil aberration, by simplifying the aberration modeling to solving a set of coefficients corresponding to Zernike modes, the dimension of the solution space can be reduced, making it possible to achieve aberration correction from a single original image. Based on this, the pupil function can be expressed as:

[0072]

[0073] where ξ, η are spatial frequencies, M is the highest order of the Zernike polynomial, and c m is the coefficient of each orthogonal Zernike polynomial Z m , and c is a matrix containing M such coefficients. Perform the least squares method on h N to solve the coefficient matrix as:

[0074]

[0075] Thereby solving the point spread function h with aberration:

[0076]

[0077] Step 6: Perform deconvolution iteration on the fluorescence image using the point spread function h obtained in step 5 to achieve image deblurring. Use R-L deconvolution to restore the fluorescence image, and its iteration formula is as follows:

[0078]

[0079] where (u, v) represents the image coordinates, f (k) (u, v) is the k-th estimate of the deblurred image, f (k+1) (u, v) is the (k + 1)-th estimate, y(u, v) is the fluorescence actual image collected in step 1, and h(u, v) is the point spread function solved in step 5. The comparison of the aberration and point spread function obtained by this method with the true values is as Figure 2 shown in c1. The root mean square error of the aberration is about 0.0453, and the root mean square error of the point spread is about 0.0074.

[0080] To facilitate the comparison and illustrate the superiority of this method, the deconvolution of the blurred image is performed using the ideal point spread function, the point spread function based on sparse prior estimation, the point spread function based on sparse prior estimation and aberration modeling, and the true point spread function respectively. The obtained results are as Figure 2 shown in b, and the cross-sectional view of the line segment part in the figure is as Figure 2As shown by c3 in the figure. It can be clearly seen from the figure that there is still a lot of blur in the result of deconvolution using the first two point spread functions, while the results of deconvolution using the last two point spread functions are similar and clearer, proving that the proposed method can achieve higher resolution. Then, the ideal point spread function, the point spread function estimated based on sparse prior, the point spread function estimated based on sparse prior and aberration modeling, and their respective aberration terms are compared with the real point spread function and aberration, and the results are as Figure 2 shown in c2. The figure shows that both the aberration and the point spread function estimated using the proposed method are closest to the real values.

[0081] In addition, the proposed method was also used to image the mitochondria in bovine pulmonary artery scleroderma cells labeled with phalloidin-AF488 and bovine pulmonary artery scleroderma cells labeled with mitochondrial red fluorescent probe under a wide-field fluorescence microscope, and the comparison with the blurred image and the result of traditional RL deconvolution is as Figure 3 shown. The cross-section in the figure proves that the present invention can provide higher resolution compared with the traditional method.

[0082] Finally, in order to demonstrate the scalability of the method, multi-labeled bovine artery scleroderma cells were observed using a confocal laser scanning microscope, and blind deconvolution was performed on the captured images using the traditional method and the present method respectively, and the results are as Figure 4 shown. This figure shows that in this application scenario, the present method still has the advantage of high resolution, proving the excellent scalability of the present method.

Claims

1. A computational adaptive optical microscopy imaging method based on sparse blind deconvolution, characterized in that, Including: Step 1: Use the fluorescence incoherent microscopy mode of the imaging system to capture fluorescence images; Step 2: Initialize the point spread function; Step 3: Iteratively update the ideal image based on sparse prior estimation and the current point spread function; Step 4: Iteratively update the point spread function based on the current ideal image; Step 5: Loop Steps 3 and 4 until the change amount of the point spread function is less than the set threshold, and use Zernike polynomials to perform aberration fitting on the estimated point spread function, thereby solving the point spread function with aberration; Step 6: Perform deconvolution iteration on the fluorescence image using the point spread function with aberration obtained in Step 5 to achieve image deblurring.

2. The computational adaptive optical microscopy imaging method based on sparse blind deconvolution according to claim 1, wherein The imaging system includes an incoherent illumination light source, an excitation / emission filter component that matches the sample band, an objective imaging optical path, and a camera. The incoherent illumination light source generates spatially incoherent light to excite the sample, and the fluorescence signal is separated by the excitation / emission filter component and then reaches the camera through the objective imaging optical path to achieve low-noise wide-field fluorescence imaging.

3. The computational adaptive optical microscopy imaging method based on sparse blind deconvolution according to claim 1, wherein The specific method for initializing the point spread function is: Initialize the point spread function h( 0 ) to the ideal point spread function, i.e., let: Wherein, (ξ, η) are the pupil plane coordinates, and P0(ξ, η) is the circular aperture of the imaging system. denotes the Fourier transform, and ★ denotes the correlation operation.

4. The computational adaptive optical microscopy imaging method based on sparse blind deconvolution according to claim 1, wherein The specific method for iteratively updating the ideal image based on sparse prior estimation and the current point spread function is: Assume the ideal image \(x\) (n) The probability density \(p(x\) (n) ) of each pixel value in it satisfies the Gaussian distribution, that is, let: where n is the number of iterations, σ represents the standard deviation of the gradient image, and f γ is the γ-th gradient filter, and f γ *x (n) represents the gradient image of x (n) , and Π represents the product operation; Assume that the noise follows a normal distribution, and the noise probability density p(y|x (n) ,h (n-1) ) is as follows: In the formula, ζ is the standard deviation of the noise, and y is the actual fluorescence image captured in Step 1; Maximize the probability density of the image x (n-1) given the known point spread function h (n) and the captured image y, that is, maximize the probability density of the following formula: p(y,x (n) |h (n-1) )=p ( y|x (n) ,h (n-1) )·p(x (n) ) When the above formula is at its maximum, the corresponding ideal image x (n) is as follows: wherein, and is the point spread function h (n-1) corresponding Toeplitz matrix, T γ is the Toeplitz matrix corresponding to the γ-th derivative filter.

5. The computational adaptive optical microscopy imaging method based on sparse blind deconvolution according to claim 1, wherein Iteratively update the point spread function based on the current ideal image, and the specific formula is: In the formula, and is the Toeplitz matrix corresponding to x (n) .

6. The computational adaptive optical microscopy imaging method based on sparse blind deconvolution according to claim 1, wherein The specific method for using Zernike polynomials to perform aberration fitting on the estimated point spread function, thereby solving the point spread function with aberration is: Determine the pupil function, specifically: where ξ and η are spatial frequencies, M is the highest order of the Zernike polynomial, and c m is the coefficient of each orthogonal Zernike polynomial Z m , and c is a matrix containing M such coefficients; For the estimated point spread function h N Performing the least squares method to solve, the coefficient matrix is as follows: Thereby solving the point spread function h with aberration:

7. The computational adaptive optical microscopy imaging method based on sparse blind deconvolution according to claim 1, wherein Use the Richardson-Lucy deconvolution to restore the fluorescence image, and the specific iteration formula is as follows: where (u, v) represents the image coordinates, f (k) (u, v) is the k-th estimate of the deblurred image, f (k+1) (u, v) is the (k + 1)-th estimate, y(u, v) is the fluorescence real-shot image acquired in Step 1, and h(u, v) is the point spread function solved in Step 5.

Images