Non-linear super-resolution structured light microscope reconstruction method and system for spatial domain reconstruction

Abstract

The invention discloses a non-linear super-resolution structured light microscope reconstruction method and a non-linear super-resolution structured light microscope reconstruction system for airspace reconstruction. The disclosed scheme comprises the following steps: calculating a weight image wd, i (r), and optimizing a filtering function # imgabs0 # and a pre-filtered original image D'd, i (r); the pre-filtered original image D'd, i (r) and the corresponding weight image wd, i (r) are subjected to point multiplication, all multiplication results are superposed, and a non-linear super-resolution image Cd, i (r) which is not subjected to deconvolution is obtained through calculation; and multiplying the nonlinear super-resolution image Cd, i (r) which is obtained through calculation and is not subjected to deconvolution by the optimized filtering function # imgabs1 # to obtain a final nonlinear super-resolution image. According to the method, the weight image calculated in advance is multiplied by the original fluorescence image, so that super-resolution reconstruction of the NL-SIM can be completed, and the image reconstruction time is maximally increased by 205 times; meanwhile, background noise and reconstruction artifacts of the super-resolution image can be eliminated by using two optimization functions.

Description

Technical Field

[0001] The present invention belongs to the technical field of optics, and relates to a structured light microscopic imaging method, in particular to a fast non-linear super-resolution structured light microscopy method and related system based on spatial domain reconstruction, which can be widely used in the fields of biology, medicine, microelectronics, materials science, etc. for microscopic research. Background Art

[0002] The spatial resolution of traditional optical microscopes is limited by the optical diffraction limit, and the highest resolution is about half a wavelength, making it impossible to observe the details of samples within the diffraction limit. Structured Illumination Microscopy (SIM), as a wide-field microscopy technique, has been widely used in live cell imaging. However, compared with other fluorescence super-resolution techniques, such as Stimulated Emission Depletion Microscopy (STED), Photoactivated Localization Microscopy (PALM), Stochastic Optical Reconstruction Microscopy (STORM), etc., traditional linear SIM can only improve the resolution to twice the diffraction limit.

[0003] In order to further improve the resolution of linear SIM, non-linear SIM technology emerged. In 2015, Li Dong et al. successfully achieved the observation of biological cells with a resolution of 45 nanometers at a frame rate of 20-40 frames per second using Skylan-NS fluorescent protein. However, the cost of improving the resolution of non-linear SIM is to collect more original images. For example, to excite one higher-order harmonics non-linear SIM, five sets of original images with different phases need to be collected at five angles respectively to obtain a spatial resolution three times that of the diffraction limit. If two higher-order harmonics non-linear SIM is excited, seven sets of original images with different phases need to be collected at nine angles respectively to obtain a spatial resolution four times that of the diffraction limit. Since traditional non-linear SIM needs to complete super-resolution image reconstruction in the frequency domain, this process involves a large number of complex matrix operations and is accompanied by the processing of a large amount of original data, making the reconstruction process of non-linear SIM more time-consuming and losing the advantage of fast reconstruction of SIM.

[0004] In response to the slow reconstruction speed and other problems of linear SIM reconstruction, Wang et al. proposed a hybrid reconstruction algorithm based on space-frequency domain (JSFR, patent number: ZL202110985056.7) in 2021. This method simplifies most of the frequency domain calculation steps in the traditional reconstruction process, such as Fourier transform, spectral separation, spectral shift, spectral splicing and inverse Fourier transform, into simple multiplication and summation operations in real space, greatly simplifying the reconstruction workflow. Without sacrificing resolution, the reconstruction speed is increased to 80 times that of traditional methods. However, there are currently no reports on nonlinear fast reconstruction. Summary of the invention

[0005] In view of the defects or shortcomings of the prior art, the present invention provides a nonlinear super-resolution structured light microscope reconstruction method for spatial domain reconstruction.

[0006] To this end, the nonlinear super-resolution structured light microscope reconstruction method provided by the present invention is used to reconstruct the original fluorescence image D taken by the nonlinear structured light super-resolution microscope system. d,i (r) processing to obtain a super-resolution image, wherein d is the stripe direction, d = 1, 2, 3, 4, 5; i is the phase shift step number, i = 1, 2, 3, 4, 5; r is the two-dimensional plane coordinate; the method comprises the following steps:

[0007] Step 1: Calculate the weight image w d,i (r), optimize the filter function And the original image D′ after pre-filtering d,i (r):

[0008] The original fluorescence image D is calculated using formula (1): d,i The weight image w of (r) d,i (r);

[0009]

[0010] In formula (1),

[0011] m 1 and m 2 are the modulation degrees of the first-order harmonic and the second-order harmonic respectively;

[0012] k d is the wave vector in the stripe direction d;

[0013] is the initial phase in the fringe direction d;

[0014] is the phase shift in the stripe direction d;

[0015] I d is the average light intensity of the structured light field in the stripe direction d;

[0016] The optimized filtering function is calculated using Equation (2).

[0017]

[0018] In Equation (2), is the complex conjugate of the optical transfer function of the non-linear structured light super-resolution microscopy system, and k is the frequency domain coordinate;

[0019] is the power spectrum corresponding to the m-th order high-frequency information, and k m is the frequency domain coordinate corresponding to the m-th order high-frequency information, m = -2, -1, 0, 1, N; N = 2;

[0020] is the attenuation amplitude parameter corresponding to the m-th order high-frequency information, and its value range is 0.1 - 0.99;

[0021] Apo is the apodization function;

[0022] k σ , w 1 , w 2 are adjustable empirical parameters; k σ takes values from 0.9 to 1.1, and w 1 , w 2 both take values in the range of 0.1 - 2.0;

[0023] The pre-filtered original image D′ d,i (r) is calculated using Equation (3):

[0024]

[0025] In Equation (3), is the complex conjugate of the system optical transfer function; is the inverse Fourier transform operator; k is the frequency domain coordinate corresponding to r; is the Fourier transform of the original fluorescence image D d,i (r);

[0026] G(k) is a Wiener-like filter,

[0027]

[0028] In Equation (4):

[0029] a att is the attenuation amplitude parameter, and its value range is 0.1 - 0.99;

[0030] k σ is an adjustable empirical parameter, and its theoretical value is from 0.9 to 1.1;

[0031] w is the Wiener parameter, and its value range is 0.1 - 2.0;

[0032] Step 2: Multiply the pre-filtered original image D′ d,i (r) by the corresponding weight image w d,i (r), add up all the multiplication results, and calculate the non-deconvolved non-linear super-resolution image C d,i (r) using Equation (5);

[0033]

[0034] Step 3: Multiply the calculated non-deconvolved non-linear super-resolution image C d,i (r) by the optimized filtering function to obtain the final non-linear super-resolution reconstructed image.

[0035] An optional solution is that when m = 0, m = ±1, m = ±2, the values are 0.87, 0.75, 0.25 respectively. k σ takes 1. w 1 and w 2 take 0.4 and 0.1 respectively. The a att value range is 0.9. w takes 0.1.

[0036] Correspondingly, the present invention also provides a related structured light microscope system, and the system uses the method described in the above claims to obtain the reconstructed image of the sample.

[0037] A storage medium related to the method of the present invention, on which a computer program / instructions are stored, and when the computer program / instructions are executed by a processor, the steps of the above method are implemented. In addition, a software product related to the present invention, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps of the above method are implemented.

[0038] Compared with the traditional frequency domain reconstruction method, the reconstruction method proposed by the present invention can complete the reconstruction in milliseconds, and the reconstruction speed is increased by about 200 times, laying a foundation for the real-time reconstruction of non-linear SIM, and will further expand the application range of non-linear SIM, and has broad application prospects in the fields of biology, cell physiology, etc.

[0039] Compared with traditional frequency-domain reconstruction algorithms, the non-linear structured illumination microscopy reconstruction method that combines spatial and frequency-domain reconstruction has obvious advantages: First, traditional frequency-domain algorithms need to complete three steps in the frequency domain: spectral decomposition, spectral shift, and spectral fusion, with a relatively high computational complexity. However, the non-linear structured illumination microscopy reconstruction method that combines spatial and frequency-domain reconstruction only needs to multiply the pre-computed weight image by the original image and accumulate them to obtain the super-resolution reconstruction result, significantly reducing the computational complexity. Second, for non-linear SIM, to obtain a higher spatial resolution, more original images are required. Based on traditional frequency-domain reconstruction algorithms, the required computational amount will increase significantly with the increase in the number of original images. However, for the non-linear structured illumination microscopy reconstruction method based on combined spatial and frequency-domain reconstruction, this computational amount only increases linearly. Description of the Drawings

[0040] Figure 1 Comparison of the comprehensive effects of the classical Wiener non-linear SIM, high-fidelity non-linear SIM, and non-linear SIM with combined spatial and frequency-domain reconstruction in the embodiments of the present invention; Figure 1 (a) is the wide-field image; Figure 1 (b) is the reconstruction result of the classical Wiener non-linear SIM; Figure 1 (c) is the reconstruction result of the high-fidelity non-linear SIM; Figure 1 (d) is the non-linear SIM with combined spatial and frequency-domain reconstruction proposed by the present invention; Figure 1 (e) is the enlarged view of local information; Figure 1 (f) is the normalized intensity curve graph of different methods. Detailed Embodiments

[0041] Unless otherwise specified, scientific and technical terms in this article are understood according to the understanding of those of ordinary skill in the relevant field. The following are specific embodiments provided by the inventors to further explain the solutions of the present invention.

[0042] Embodiment 1:

[0043] In order to verify the accuracy of the nonlinear structured light illumination microscopy reconstruction method of joint spatial-frequency domain reconstruction, the nonlinear data F-actin actin filaments in the open source data set BioSR of Tsinghua University were selected as the verification sample. The illumination wavelength is 488nm, the objective lens value is 1.41, and the original pixel size is 62.6nm. For comparison, the classical Wiener nonlinear SIM (see Gustafsson et al. in 2008 in "Biophysical Journal" for a SIM reconstruction scheme based on Wiener deconvolution), high-fidelity nonlinear SIM (see Wen et al. in 2021 in "Light: Science & Applications" for a high-fidelity SIM reconstruction scheme based on point diffusion engineering) and the method of the present invention (in this embodiment, m=0, m=±1, m=±2 orders) are used respectively. The values ​​of k are 0.87, 0.75, and 0.25 respectively; σ The value is 1; w 1 , w 2 Take 0.4 and 0.1 respectively; a att The value is 0.9; w is 0.1).

[0044] The results are as follows Figure 1 As shown, Figure 1 (a) is a wide-field image. Figure 1 (b) is the classical Wiener nonlinear SIM reconstruction result. Figure 1 (c) is the high-fidelity nonlinear SIM reconstruction result. Figure 1 (d) is the nonlinear SIM with joint space-frequency domain reconstruction proposed by the present invention. Obviously, for the classic Wiener nonlinear SIM, there are more background fluorescence signals in its reconstruction result, while the high-fidelity nonlinear SIM and the nonlinear SIM with joint space-frequency domain reconstruction are more robust and have better resolution. But at the same time, the reconstruction time of high-fidelity nonlinear SIM is 570ms, while the nonlinear SIM with joint space-frequency domain reconstruction can complete the reconstruction in only 4.7ms, which significantly improves the reconstruction speed.

[0045] pass Figure 1 The local information magnification diagram of (e) can further confirm the above conclusion. Figure 1 The intensity curve corresponding to the white line measurement in (e) is obtained Figure 1 (f) Normalized intensity curve diagram (in this figure, the horizontal axis is Figure 1(e) The distance corresponding to the white line, in micrometers, with the normalized light intensity on the vertical axis and a.u. being an arbitrary unit; in the figure, Wiener-NL-SIM is a SIM reconstruction scheme based on Wiener deconvolution published by Gustafsson et al. in *Biophysical Journal* in 2008; HiFi-NL-SIM is a high-fidelity SIM reconstruction scheme based on point spread engineering published by Wen et al. in *Light: Science & Applications* in 2021; JSFR-NL-SIM is the method proposed in the present invention); it can be found that the full width at half maximum of the non-linear SIM with joint spatio-frequency domain reconstruction of the present invention is comparable to that of the high-fidelity non-linear SIM and is superior to the classical Wiener non-linear SIM.

Claims

1. A nonlinear super-resolution structured light microscopy reconstruction method based on spatial domain reconstruction, characterized in that: The method is used to analyze the original fluorescence image D taken by the nonlinear structured light super-resolution microscopy system. d,i (r) processing to obtain a super-resolution image, wherein d is the stripe direction, d = 1, 2, 3, 4, 5; i is the phase shift step number, i = 1, 2, 3, 4, 5; r is the two-dimensional plane coordinate; the method comprises the following steps: Step 1: Calculate the weight image w d,i (r), optimize the filter function And the original image D' after pre-filtering d,i (r): The original fluorescence image D is calculated using formula (1): d,i The weight image w of (r) d,i (r); In formula (1), m1 and m2 are the modulation degrees of the first-order harmonic and the second-order harmonic respectively; k d is the wave vector in the stripe direction d; is the initial phase in the fringe direction d; is the phase shift in the stripe direction d; I d is the average light intensity of the structured light field in the stripe direction d; Formula (2) is used to calculate the optimized filter function In formula (2), is the complex conjugate of the optical transfer function of the nonlinear structured light super-resolution microscopy system, k is the frequency domain coordinate; is the power spectrum corresponding to the m-order high-frequency information, k m is the frequency domain coordinate corresponding to the m-order high frequency information, m = -2, -1, 0, 1, N; N = 2; is the attenuation amplitude parameter corresponding to the m-order high-frequency information, and its value range is 0.1-0.99; Apo is the apodization function; k σ , w1, w2 are adjustable empirical parameters; k σ The value range is 0.9 to 1.1, and the value range of w1 and w2 is 0.1-2.0; The original image D' after pre-filtering is calculated using formula (3): d,i (r): In formula (3), The complex conjugate of the system's optical transfer function; is the inverse Fourier transform operator; k is the frequency domain coordinate corresponding to r; is the original fluorescence image D d,i Fourier transform of (r); G(k) is a Wiener-like filter, In formula (4): a att is the attenuation amplitude parameter, the value range is 0.1-0.99; k σ It is an adjustable empirical parameter, and its theoretical value is 0.9 to 1.1; w is the Wiener parameter, ranging from 0.1 to 2.0; Step 2: The pre-filtered original image D′ d,i (r) and the corresponding weight image w d,i (r) and superimpose all the multiplication results, and use formula (5) to calculate the nonlinear super-resolution image C without deconvolution: d,i (r); Step 3: The calculated nonlinear super-resolution image C without deconvolution d,i (r) and optimized filter function Multiply them together to get the final nonlinear super-resolution reconstructed image.

2. The nonlinear super-resolution structured light microscope reconstruction method of spatial domain reconstruction according to claim 1, characterized in that: When m=0,m=±1,m=±2 The values ​​are 0.87, 0.75, and 0.25 respectively.

3. The nonlinear super-resolution structured light microscope reconstruction method of spatial domain reconstruction according to claim 1, characterized in that: k σ Take 1.

4. The nonlinear super-resolution structured light microscope reconstruction method of spatial domain reconstruction according to claim 1, characterized in that: w1 and w2 are 0.4 and 0.1 respectively.

5. The nonlinear super-resolution structured light microscope reconstruction method of spatial domain reconstruction according to claim 1, characterized in that: The a att The value range is 0.

9.

6. The nonlinear super-resolution structured light microscope reconstruction method of spatial domain reconstruction according to claim 1, characterized in that: w is taken as 0.

1.

7. A nonlinear super-resolution structured light microscope system with spatial domain reconstruction, characterized in that: The system adopts the method described in any one of claims 1 to 6 to obtain a reconstructed image of the sample.

8. A storage medium, characterized in that: A computer program / instruction is stored thereon, wherein the computer program / instruction, when executed by a processor, implements the steps of the method described in any one of claims 1 to 6.

9. A software product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

Images