Nonlinear super-resolution structured light microscopy reconstruction method and system with spatial domain reconstruction

Abstract

The application discloses a method and system for reconstructing a nonlinear super-resolution structured light microscope with spatial reconstruction. The disclosed scheme comprises calculating a weight image w d,i (r), optimizing a filter function, and a pre-filtered original image D' d,i (r); multiplying the pre-filtered original image D' d,i (r) and the corresponding weight image w d,i (r), and superimposing all the multiplied results to obtain an un-deconvoluted nonlinear super-resolution image C d,i (r); multiplying the obtained un-deconvoluted nonlinear super-resolution image C d,i (r) and the optimized filter function to obtain a final nonlinear super-resolution image. By multiplying the weight image calculated in advance and the original fluorescence image, the super-resolution reconstruction of the NL-SIM can be completed, and the image reconstruction time is maximally increased by 205 times. Meanwhile, the background noise and reconstruction artifacts of the super-resolution image can be eliminated by using two optimized functions.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of optical technology, and relates to a structured light illumination microscopic imaging method, in particular to a fast nonlinear super-resolution structured light microscopic method based on space reconstruction and a related system which can be widely applied to biological, medical, microelectronic and material scientific fields for microscopic research. BACKGROUND

[0002] The spatial resolution of a traditional optical microscope is limited by the optical diffraction limit, and the highest resolution is about half a wavelength, which cannot observe the details of the sample within the diffraction limit range. As a kind of wide-field microscopic technology, structured illumination microscopy (SIM) has a wide application in live cell imaging. However, compared with other fluorescence super-resolution technologies, such as stimulated emission depletion microscopy (STED), photoactivated localization microscopy (PALM), stochastic optical reconstruction microscopy (STORM) and the like, the traditional linear SIM can only improve the resolution to twice the diffraction limit.

[0003] In order to further improve the resolution of the linear SIM, the nonlinear SIM technology emerges as the times require. In 2015, Li Dong et al. successfully realized the observation of biological cells with a resolution of 45 nanometers at a frequency of 20-40 frames by using Skylan-NS fluorescent protein. However, the price to improve the resolution of the nonlinear SIM is to collect more original images. For example, if one higher-order harmonic SIM is excited, five original image sets of different phases need to be collected at five angles respectively, so as to obtain a spatial resolution three times the diffraction limit. If two higher-order harmonic SIM is excited, seven original image sets of different phases need to be collected at nine angles respectively, so as to obtain a spatial resolution four times the diffraction limit. Since the traditional nonlinear SIM needs to complete the super-resolution image reconstruction in the frequency domain, this process involves a large number of complex matrix operations, accompanied by a large amount of original data processing, so that the reconstruction process of the nonlinear SIM is more time-consuming, and the advantage of fast reconstruction of the SIM is lost.

[0004] To solve the problem of slow reconstruction speed in linear SIM reconstruction, Wang et al. proposed a hybrid spatial-spectral reconstruction algorithm (JSFR, patent number: ZL202110985056.7) in 2021. This method simplifies most of the frequency domain space calculation steps in the traditional reconstruction process, such as Fourier transform, spectral separation, spectral shift, spectral splicing, and inverse Fourier transform, to simple multiplication and summation operations in real space, greatly simplifying the reconstruction workflow. Without sacrificing resolution, the reconstruction speed is improved to 80 times that of the traditional method. However, there is currently no report on nonlinear fast reconstruction. SUMMARY

[0005] To overcome the defects or deficiencies of the prior art, the present application provides a nonlinear super-resolution structured light microscope reconstruction method based on spatial domain reconstruction.

[0006] Therefore, the nonlinear super-resolution structured light microscope reconstruction method provided by the present application is used to process the original fluorescence image D d,i (r) taken by a nonlinear structured light super-resolution microscopy system to obtain a super-resolution image, where d is the fringe direction, d = 1, 2, 3, 4, 5; i is the number of phase shifts, i = 1, 2, 3, 4, 5; r is the two-dimensional plane coordinate; the method comprises the following steps:

[0007] Step 1, calculate the weight image w d,i (r), optimize the filter function and the pre-filtered original image D' d,i (r):

[0008] The weight image w d,i (r) of the original fluorescence image D d,i (r) is calculated by formula (1).

[0009]

[0010] In formula (1),

[0011] m1 and m2 are the modulation depths of the first and second harmonics, respectively;

[0012] k d is the wave vector in the fringe direction d;

[0013] is the initial phase in the fringe direction d;

[0014] is the phase shift in the fringe direction d;

[0015] I d is the average light intensity of the structured light field in the fringe direction d;

[0016] The optimized filter function is calculated by formula (2)

[0017]

[0018] In formula (2), is the complex conjugate of the optical transfer function of the nonlinear structured light super-resolution microscopic system, and k is a frequency domain coordinate;

[0019] is the power spectrum corresponding to the m-order high-frequency information, k m is the frequency domain coordinate corresponding to the m-order high-frequency information, m = -2, -1, 0, 1, N; N = 2;

[0020] is the attenuation amplitude parameter corresponding to the m-order high-frequency information, and the value range is 0.1-0.99;

[0021] Apo is an apodization function;

[0022] k σ , w1, w2 are adjustable empirical parameters; k σ , and the value range of w1 and w2 is 0.1-2.0;

[0023] The original image D' after pre-filtering is calculated by formula (3) d,i (r):

[0024]

[0025] In formula (3), is the complex conjugate of the system optical transfer function; is the inverse Fourier transform operator; k is the frequency domain coordinate corresponding to r; is the Fourier transform of the original fluorescence image D d,i (r);

[0026] G(k) is a Wiener-like filter,

[0027]

[0028] In formula (4):

[0029] a att is an attenuation amplitude parameter, and the value range is 0.1-0.99;

[0030] k σ is an adjustable empirical parameter, and the theoretical value is 0.9-1.1;

[0031] w is a Wiener parameter, and the value range is 0.1-2.0;

[0032] Step 2, multiply the pre-filtered original image D' d,i (r) with the corresponding weight image w d,i (r) and superimpose all the multiplied results to obtain the non-deconvolution nonlinear super-resolution image C d,i (r) by using formula (5)

[0033]

[0034] Step 3, multiply the calculated non-deconvolution nonlinear super-resolution image C d,i (r) with the optimized filter function to obtain the final nonlinear super-resolution reconstructed image.

[0035] An optional solution is that m=0, m=±1, m=±2 order The values are 0.87, 0.75, 0.25 respectively. k σ Take 1. w1 and w2 take 0.4 and 0.1 respectively. The a att Take the value range of 0.9. w takes 0.1.

[0036] Correspondingly, the application also provides a related structured light microscope system, which obtains the reconstructed image of the sample by using the above method.

[0037] A storage medium related to the method of the application, which stores computer programs / instructions, and the computer programs / instructions are executed by a processor to realize the steps of the above method. And a software product related to the application, which includes computer programs / instructions, and the computer programs / instructions are executed by a processor to realize the steps of the above method.

[0038] Compared with the traditional frequency domain reconstruction method, the reconstruction method proposed in the application can complete millisecond-level reconstruction, and the reconstruction speed is improved by about 200 times, which lays a foundation for real-time reconstruction of nonlinear SIM, and further expands the application range of nonlinear SIM, and has a wide application prospect in the fields of biology and cell physiology.

[0039] Compared to traditional frequency domain reconstruction algorithms, the nonlinear structured light illumination microscopy reconstruction method based on joint spatial-frequency domain reconstruction has significant advantages: First, traditional frequency domain algorithms require three steps in the frequency domain: spectral decomposition, spectral shifting, and spectral fusion, resulting in high computational complexity. In contrast, the nonlinear structured light illumination microscopy reconstruction method based on joint spatial-frequency domain reconstruction only requires multiplying and summing the pre-calculated weighted image with the original image to obtain the super-resolution reconstruction result, significantly reducing computational complexity. Second, for nonlinear SIM, obtaining higher spatial resolution requires more original images, and the computational cost of traditional frequency domain reconstruction algorithms increases significantly with the number of original images. However, for the nonlinear structured light illumination microscopy reconstruction method based on joint spatial-frequency domain reconstruction, this computational cost increases only linearly. Attached Figure Description

[0040] Figure 1 This invention presents a comparison of the combined effects of classical Wiener nonlinear SIM, high-fidelity nonlinear SIM, and nonlinear SIM with joint spatial-frequency domain reconstruction in the embodiments of the present invention. Figure 1 (a) is a wide-field image; Figure 1 (b) shows the classical Wiener nonlinear SIM reconstruction result; Figure 1 (c) shows the high-fidelity nonlinear SIM reconstruction results; Figure 1 (d) is the nonlinear SIM for joint spatial-frequency domain reconstruction proposed in this invention; Figure 1 (e) is a magnified view of the local information; Figure 1 (f) shows the normalized intensity curves for different methods. Detailed Implementation

[0041] Unless otherwise specified, the scientific and technical terms used herein are for the understanding of one of ordinary skill in the art. The following are specific embodiments provided by the inventors to further explain the solutions of the present invention.

[0042] Example 1:

[0043] In order to verify the accuracy of the joint spatial-frequency domain reconstruction nonlinear structured illumination microscopy reconstruction method, the nonlinear data F-actin actin filament in the open source data set BioSR of Tsinghua University is selected as the verification sample. The illumination wavelength is 488 nm, the objective numerical value is 1.41, and the original pixel size is 62.6 nm. As a comparison, the classical Wiener nonlinear SIM (see Gustafsson et al. in Biophysical Journal, 2008 for a Wiener deconvolution-based SIM reconstruction scheme), the high-fidelity nonlinear SIM (see Wen et al. in Light: Science & Applications, 2021 for a high-fidelity SIM reconstruction scheme based on point spread engineering) and the method of the present application (in this embodiment, the values of m=0, m=±1, m=±2 order are 0.87, 0.75, 0.25 respectively; k σ 1; w1, w2 are 0.4 and 0.1 respectively; a att 0.9; w is 0.1) are used for comparison.

[0044] The results are shown in Figure 1 , wherein Figure 1 (a) is a wide-field image, Figure 1 (b) is a classical Wiener nonlinear SIM reconstruction result, Figure 1 (c) is a high-fidelity nonlinear SIM reconstruction result, Figure 1 (d) is a joint spatial-frequency domain reconstruction nonlinear SIM proposed by the present application. Obviously, for the classical Wiener nonlinear SIM, there are more background fluorescence signals in the reconstruction result, while the robustness of the high-fidelity nonlinear SIM and the joint spatial-frequency domain reconstruction nonlinear SIM is higher, and the resolution is better. However, at the same time, the high-fidelity nonlinear SIM reconstruction time is 570 ms, while the joint spatial-frequency domain reconstruction nonlinear SIM only needs 4.7 ms to complete the reconstruction, and the reconstruction speed is significantly improved.

[0045] The above conclusion can be further verified by the local information magnification graph of Figure 1 (e). At the same time, by measuring the corresponding intensity curve along the white line in Figure 1 (e), the normalized intensity curve graph of Figure 1 (f) is obtained (in the graph, the abscissa is Figure 1 ​The white line in (e) corresponds to a distance of 1.5 microns, and the vertical coordinate is the normalized light intensity, a.u. is arbitrary unit; in the figure, Wiener-NL-SIM is a SIM reconstruction scheme based on Wiener deconvolution published by Gustafsson et al. in 2008 in Biophysical Journal; HiFi-NL-SIM is a high-fidelity SIM reconstruction scheme based on point spread engineering published by Wen et al. in 2021 in Light: Science & Applications; JSFR-NL-SIM is the method proposed in the present application; it can be found that the half-width of the nonlinear SIM of the joint space-frequency domain reconstruction of the present application is comparable to that of the high-fidelity nonlinear SIM, and is superior to that of the classical Wiener nonlinear SIM.

Claims

1. A nonlinear super-resolution structured light microscopy reconstruction method for spatial domain reconstruction, characterized in that, The method is used to analyze raw fluorescence images D acquired by a nonlinear structured light super-resolution microscopy system. d,i (r) is processed to obtain a super-resolution image, where d is the fringe direction, d = 1, 2, 3, 4, 5; i is the phase shift step number, i = 1, 2, 3, 4, 5; and r is the two-dimensional plane coordinate; the method includes the following steps: Step 1, calculate the weighted image w d,i (r) Optimize the filtering function and the pre-filtered original image D' d,i (r): The original fluorescence image D is calculated using equation (1). d,i The weighted image w of (r) d,i (r); In equation (1), m1 and m2 are the modulation indices of the first and second harmonics, respectively; k d The wave vector is located in the fringe direction d. The initial phase along the fringe direction d; I d Let d be the average light intensity of the structured light field along the fringe direction d; The optimized filter function is calculated using equation (2). In equation (2), Let be the complex conjugate of the optical transfer function of the nonlinear structured light super-resolution microscopy system, and k be the frequency domain coordinates; Let k be the power spectrum corresponding to the m-th order high-frequency information. m The frequency domain coordinates corresponding to the m-th order high-frequency information are given, where m = -2, -1, 0, 1, N; and N = 2. The attenuation amplitude parameter corresponding to the m-th order high-frequency information has a value range of 0.1-0.

99. Apo is the apocentric function; k σ w1 and w2 are adjustable empirical parameters; k σ The value ranges from 0.9 to 1.1, while the values ​​of w1 and w2 both range from 0.1 to 2.

0. The pre-filtered original image D' is calculated using equation (3). d,i (r): In equation (3), This is the inverse Fourier transform operator; k is the frequency domain coordinate corresponding to r; Original fluorescence image D d,i Fourier transform of (r); G(k) is a Wiener-like filter. In equation (4): a att This is the attenuation amplitude parameter, with a value range of 0.1-0.99; k σ This is an adjustable empirical parameter, with a theoretical value of 0.9 to 1.1; w is the Wiener parameter, with a value range of 0.1-2.0; Step 2, pre-filter the original image D′ d,i (r) and the corresponding weighted image w d,i (r) perform dot product and sum the results of all multiplications. Then, use equation (5) to calculate the nonlinear super-resolution image C without deconvolution. d,i (r); Step 3, calculate the undeconvolutioned nonlinear super-resolution image C d,i (r) and optimized filtering function Multiplying the results yields the final nonlinear super-resolution reconstructed image.

2. The nonlinear super-resolution structured light microscopy reconstruction method for spatial domain reconstruction according to claim 1, characterized in that, When m = 0, m = ±1, m = ±2, etc. The values ​​are 0.87, 0.75, and 0.25, respectively.

3. The nonlinear super-resolution structured light microscopy reconstruction method for spatial domain reconstruction according to claim 1, characterized in that, k σ Take 1.

4. The nonlinear super-resolution structured light microscopy reconstruction method for spatial domain reconstruction according to claim 1, characterized in that, w1 and w2 are set to 0.4 and 0.1 respectively.

5. The nonlinear super-resolution structured light microscopy reconstruction method for spatial domain reconstruction according to claim 1, characterized in that, The a att The value is 0.

9.

6. The nonlinear super-resolution structured light microscopy reconstruction method for spatial domain reconstruction according to claim 1, characterized in that, w is set to 0.

1.

7. A nonlinear super-resolution structured light microscope system for spatial reconstruction, characterized in that, The system obtains a reconstructed image of the sample using the method described in any one of claims 1 to 6.

8. A storage medium, characterized in that, It stores a computer program / instruction thereon, characterized in that when the computer program / instruction is executed by a processor, it implements the steps of the method described in any one of claims 1 to 6.

9. A software product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the method described in any one of claims 1 to 6.

Images