Structured light imaging device and method based on signal fluctuation expansion resolution

Abstract

The invention discloses a structured light imaging device and method based on signal fluctuation expansion resolution, belongs to the technical field of super-resolution microscopic imaging and computational imaging, and aims to overcome the defects that an existing microscopic imaging system is limited by optical diffraction limit, the resolution is insufficient and a fine structure cannot be formed. The method comprises the following steps: firstly, imaging a sample for multiple times by using illumination of sine stripe structured light at different angles to obtain nine groups of image sequences at three angles, and then analyzing and reconstructing a time domain image sequence by using a signal fluctuation imaging theory to obtain resolution expansion images at different angles and phases; and finally, reconstructing the whole image by using a structured light reconstruction principle to obtain a required super-resolution image. According to the structured light imaging method based on the signal fluctuation expansion resolution, the signal fluctuation imaging and the structured light illumination imaging resolution improvement capability are combined, so that the resolution improvement effect of the image is achieved to a greater extent, and the super-resolution purpose is finally achieved.

Description

Technical Field

[0001] The present invention belongs to the field of super-resolution imaging and computational imaging, and relates to a structured light imaging device and method for expanding resolution based on signal fluctuations. Background Art

[0002] The fluorescence super-resolution method based on the molecular intensity signal fluctuation model only utilizes the physical model of the random fluctuation of the fluorescence molecular intensity and does not rely on any hardware modulation. It is a flexible and highly cost-effective super-resolution means. Due to its characteristic of not being restricted by the hardware system, it can be flexibly coupled to various different imaging modalities. On the other hand, structured light microscopy is a super-high-resolution imaging means that utilizes photons most efficiently. A grating is formed by using a high numerical aperture objective lens and a special spatial light modulator to modulate the sample information, so that the high-frequency information enters the numerical aperture of the objective lens, and then the obtained high-frequency information is frequency-domain mosaicked and fused to obtain a super-resolution image. This method has the advantages of high speed, low phototoxicity, and high photon utilization rate. Summary of the Invention

[0003] A brief overview of the present invention is given below to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify the key or important parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is only to present certain concepts in a simplified form as a prelude to the more detailed description discussed later.

[0004] In view of this, the present invention provides a structured light imaging device and method for expanding resolution based on signal fluctuations, which combines the resolution improvement capabilities of signal fluctuation imaging and structured light illumination imaging to enable the image to obtain a greater degree of resolution improvement effect and ultimately achieve the purpose of super-resolution.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] Solution 1: The present invention provides a fluorescence super-resolution device based on the molecular intensity signal fluctuation model, including a laser, a lens, a polarization beam splitter, a spatial light modulator, a mask, a polarization rotator, a storage camera, a dichroic mirror, a tube lens, and an objective lens;

[0007] The propagation direction of the system illumination light is successively: laser, lens three, polarization beam splitter, spatial light modulator, lens two, mask, polarization rotator, lens one, dichroic mirror, tube lens, objective lens, sample;

[0008] The propagation direction of the system imaging light is successively: sample, objective lens, tube lens, dichroic mirror, and storage camera;

[0009] The polarization beam splitter, spatial light modulator, lens two, mask, polarization rotator, and lens one are located in the same vertical direction.

[0010] Preferably, the microscopic imaging process includes the following steps:

[0011] Step a: The laser emits illumination light, which forms structured light through the spatial light modulator and irradiates the sample through the illumination optical path. The fluorescence signal generated is imaged by the camera through the imaging optical path.

[0012] Step b: Group the obtained image sequences, with a total size of 9×M×N×L. Since structured light at 3 different angles is used for irradiation and each angle contains 3 different phases, the image sequences will be divided into 9 groups. Each group contains a complete time sequence, and the size of each image sequence is M×N×L.

[0013] Step c: For each image sequence, perform single-pixel decomposition on the sequence along the first and second dimensions, and decompose the image sequence into single-pixel intensity time sequences with a length of L, totaling M×N.

[0014] Step d: Perform signal fluctuation analysis and reconstruction on each decomposed time sequence to obtain the reconstructed intensity results, totaling M×N.

[0015] Step e: Combine and reconstruct the M×N intensity results to obtain the signal fluctuation reconstructed image.

[0016] Step f: Repeat steps c to e until the reconstruction of all 9 groups of image sequences is completed, and obtain 9 resolution-enhanced images at different angles and phases.

[0017] Step g: Use the structured light illumination reconstruction principle to fuse the images in the frequency domain to obtain the final super-resolution image.

[0018] Preferably, in step a, the sample needs to be stained with a dye with fluorescence intermittency. These fluorescent molecule groups have their own independent molecular brightness that changes with time. The fluorescence signal at position r and time t can be expressed as:

[0019] F(r,t) = h(r - r k )·c k ·ω k (t)

[0020] Where h, c, and ω represent the point spread function of the corresponding microscope, the molecular brightness constant, and the function of the molecular brightness fluctuating with time, respectively.

[0021] Preferably, in step a, the structured light is a sine stripe, which is generated by the interference of two plane waves and can be expressed as:

[0022]

[0023] Preferably, in the imaging process described in step a, structured light at three different angles is used for repeated imaging.

[0024] Preferably, in the imaging process described in step a, for the structured light at each angle, imaging is performed three times with different phases, and the imaging results can be expressed as:

[0025]

[0026] D(k) is the frequency-domain result corresponding to ordinary microscope imaging, and E(k) is the frequency-domain result corresponding to structured light microscope imaging.

[0027] Preferably, in the imaging process described in step a, multi-angle and multi-phase imaging is performed more than 100 times.

[0028] Preferably, in the reconstruction process described in step d, quadratic temporal cumulants are generally used for reconstruction, and the reconstruction process can be expressed as:

[0029] G2(r) = 〈δF(r,t)·δF(r,t)> t

[0030] δF(r,t) = F(r,t) - <F(r,t)> t

[0031] where, <·> t is the time-averaging function.

[0032] Preferably, in the reconstruction process described in step d, when the expression of the second-order temporal cumulant is expanded, when meeting the preset conditions, the cross-correlation terms in the expansion of the second-order temporal cumulant are regarded as zero, and the second-order temporal cumulant is expressed as the sum of the squares of the point spread functions weighted by the corresponding luminance constants. When the expression of the second-order temporal cumulant G2 is expanded, the following formula is obtained:

[0033]

[0034] Preferably, in the reconstruction process described in step d, assuming that the emission intensities of each fluorescent molecule are independently fluctuating and uncorrelated, when i≠k (preset condition), the cross-correlation terms in the expansion are regarded as zero, and the second-order temporal cumulant G2 is expressed as the sum of the squares of the point spread functions weighted by the corresponding luminance constant γ, as shown in the following formula:

[0035]

[0036] Preferably, in the reconstruction process described in step f, different frequency-domain components need to be separated, which can be expressed as:

[0037]

[0038] Preferably, in the reconstruction process described in step f, it is necessary to displace the high-frequency frequency domain components, and the displacement operator can be expressed as:

[0039]

[0040] Preferably, in the reconstruction process described in step f, it is necessary to perform apodization function and inverse Fourier transform processing on the reconstructed frequency domain structure.

[0041] Beneficial effects:

[0042] The structured light imaging device and method based on the present invention have the advantages of high spatial resolution, high temporal resolution, low phototoxicity, high photon utilization efficiency, etc. On this basis, the present invention innovatively uses the signal fluctuation imaging principle to expand the resolution. In this scheme, the signal fluctuation principle is first used to reconstruct the sample signal, improving the resolution and signal-to-noise ratio, and then combined with the structured light imaging system to further improve the resolution and obtain the required super-resolution image. This scheme does not require additional hardware and is compatible with ordinary structured light microscopes. Brief description of the drawings

[0043] Figure 1 It is a schematic structural diagram of the structured light imaging device based on signal fluctuation expansion resolution of the present invention.

[0044] In the figure: laser 1, lens three 2, polarization beam splitter 3, spatial light modulator 4, lens two 5, mask 6, polarization rotator 7, lens one 8, storage camera 9, dichroic mirror 10, tube lens 11, objective lens 12 and sample 13;

[0045] Figure 2 It is a flow chart of the structured light imaging method based on signal fluctuation expansion resolution of the present invention. Detailed implementation manners

[0046] Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings. For clarity and conciseness, not all features of the actual implementation manners are described in the specification. However, it should be understood that many implementation-specific decisions must be made in the process of developing any such actual implementation to achieve the specific goals of the developer, for example, to comply with those system- and business-related constraints, and these constraints may vary with different implementation manners. In addition, it should also be understood that although the development work may be very complex and time-consuming, for those skilled in the art who benefit from the disclosure of the present invention, such development work is merely a routine task.

[0047] Here, it should also be noted that in order to avoid obscuring the present invention with unnecessary details, only the device structures and / or processing steps closely related to the solution according to the present invention are shown in the drawings, while other details less related to the present invention are omitted.

[0048] Embodiment 1: As shown in the attached Figure 1 figures, this embodiment provides a structured light imaging device based on signal fluctuation to expand resolution, which is composed of a laser 1, a lens III 2, a polarization beam splitter 3, a spatial light modulator 4, a lens II 5, a mask 6, a polarization rotator 7, a lens I 8, a storage camera 9, a dichroic mirror 10, a tube lens 11, an objective lens 12, and a sample 13.

[0049] The propagation direction of the illumination light of the system is successively: laser 1, lens III 2, polarization beam splitter 3, spatial light modulator 4, lens II 5, mask 6, polarization rotator 7, lens I 8, dichroic mirror 10, tube lens 11, objective lens 12, sample 13.

[0050] The propagation direction of the imaging light of the system is successively: sample 13, objective lens 12, tube lens 11, dichroic mirror 10, and storage camera 9.

[0051] The polarization beam splitter 3, spatial light modulator 4, lens II 5, mask 6, polarization rotator 7, and lens I 8 are located in the same vertical direction.

[0052] More specifically, the microscopic imaging process includes the following steps:

[0053] Step a: The laser emits illumination light, which forms structured light through the spatial light modulator, irradiates the sample through the illumination optical path, and the generated fluorescence signal reaches the camera through the imaging optical path for imaging.

[0054] Step b: Group the obtained image sequences, with a total size of 9×M×N×L. Since structured light at 3 different angles is used for irradiation and each angle contains 3 different phases, the image sequences will be divided into 9 groups, each group containing a complete time sequence, and the size of each image sequence is M×N×L.

[0055] Step c: For each image sequence, perform single-pixel decomposition on the sequence along the first and second dimensions, and decompose the image sequence into single-pixel intensity time sequences with a length of L, totaling M×N.

[0056] Step d: Perform signal fluctuation analysis and reconstruction on each decomposed time sequence to obtain the reconstructed intensity results, totaling M×N.

[0057] Step e: Combine and reconstruct the M×N intensity results to obtain a signal fluctuation reconstructed image.

[0058] Step f: Repeat steps c to e until the reconstruction of all 9 groups of image sequences is completed, and 9 resolution-enhanced images with different angles and phases are obtained.

[0059] Step g: Use the principle of structured-light illumination reconstruction to fuse the images in the frequency domain to obtain the final super-resolution image.

[0060] More specifically: In step a, the sample needs to be stained with a dye with fluorescence intermittency. These groups of fluorescent molecules have their own independent molecular brightness that varies with time. The fluorescence signal at position r and time t can be expressed as:

[0061] F(r,t) = h(r - r k )·c k ·ω k (t)

[0062] where h, c, and ω represent the point spread function of the corresponding microscope, the molecular brightness constant, and the function of the molecular brightness fluctuating with time, respectively.

[0063] More specifically: The structured light in step a is a sine stripe, which is generated by the interference of two plane waves and can be expressed as:

[0064]

[0065] More specifically: In the imaging process in step a, the imaging needs to be repeated using structured light at 3 different angles.

[0066] More specifically: In the imaging process in step a, for the structured light at each angle, imaging needs to be performed 3 times with different phases, and the imaging results can be expressed as:

[0067]

[0068] D(k) is the frequency domain result corresponding to the imaging of an ordinary microscope, and E(k) is the frequency domain result corresponding to the imaging of a structured-light microscope.

[0069] More specifically: In the imaging process in step a, multi-angle and multi-phase imaging needs to be performed more than 100 times.

[0070] More specifically: In the reconstruction process in step d, generally, the second-order time cumulant is used for reconstruction, and the reconstruction process can be expressed as:

[0071] G2(r) = <δF(r,t)·δF(r,t)> t

[0072] δF(r,t) = F(r,t) - <F(r,t)> t

[0073] wherein, <·> t is the time average function.

[0074] More specifically, in the reconstruction process described in step d, the expression of the second-order time cumulant is expanded. When meeting the preset conditions, the cross-correlation terms in the expansion of the second-order time cumulant are regarded as zero, and the second-order time cumulant is expressed as the sum of squares of the point spread functions weighted by the corresponding luminance constants. The expression of the second-order time cumulant G2 is expanded to obtain the following formula:

[0075]

[0076] More specifically, in the reconstruction process described in step d, it is assumed that the emission intensities of each fluorescent molecule are uncorrelated individual fluctuations. When i≠k (preset condition), the cross-correlation terms in the expansion are regarded as zero, and the second-order time cumulant G2 is expressed as the sum of squares of the point spread functions weighted by the corresponding luminance constant γ, as shown in the following formula:

[0077]

[0078] More specifically, in the reconstruction process described in step f, different frequency domain components need to be separated, which can be expressed as:

[0079]

[0080] More specifically, in the reconstruction process described in step f, the high-frequency frequency domain components need to be displaced, and the displacement operator can be expressed as:

[0081]

[0082] More specifically, in the reconstruction process described in step f, the apodization function and inverse Fourier transform processing need to be performed on the reconstructed frequency domain structure.

Claims

1. A structured light imaging device based on signal fluctuation to expand resolution, characterized in that The hardware system consists of a laser (1), a third lens (2), a polarization beam splitter (3), a spatial light modulator (4), a second lens (5), a mask (6), a polarization rotator (7), a first lens (8), a memory camera (9), a dichroic mirror (10), a tube lens (11), an objective lens (12), and a sample (13). The propagation direction of the illumination light of the system is successively: laser (1), third lens (2), polarization beam splitter (3), spatial light modulator (4), second lens (5), mask (6), polarization rotator (7), first lens (8), dichroic mirror (10), tube lens (11), objective lens (12), sample (13). The propagation direction of the imaging light of the system is successively: sample (13), objective lens (12), tube lens (11), dichroic mirror (10), and memory camera (9). The polarization beam splitter (3), spatial light modulator (4), second lens (5), mask (6), polarization rotator (7), and first lens (8) are located in the same vertical direction.

2. The structured light imaging method based on signal fluctuation to expand resolution according to claim 1, wherein: The microscopic imaging process includes the following steps: Step a: The laser emits illumination light, which forms structured light through the spatial light modulator, irradiates the sample through the illumination optical path, generates a fluorescence signal, and reaches the camera for imaging through the imaging optical path. Step b: Group the obtained image sequences, with a total size of 9×M×N×L. Since structured light at 3 different angles is used for illumination and each angle contains 3 different phases, the image sequences will be divided into 9 groups, each group containing a complete time series sequence, and the size of each image sequence is M×N×L. Step c: For each image sequence, perform single-pixel decomposition on the sequence along the first and second dimensions, and decompose the image sequence into single-pixel intensity time series sequences with a length of L, totaling M×N. Step d: Perform signal fluctuation analysis and reconstruction on each decomposed time series sequence to obtain the reconstructed intensity results, totaling M×N. Step e: Combine and reconstruct the M×N intensity results to obtain a signal fluctuation reconstructed image. Step f: Repeat steps c to e until the reconstruction of all 9 groups of image sequences is completed, and obtain 9 resolution-enhanced images with different angles and phases. Step g: Use the structured light illumination reconstruction principle to fuse the images in the frequency domain to obtain the final super-resolution image.

3. The structured light imaging method based on signal fluctuation to expand resolution according to claim 2, wherein: In step a, the sample needs to be stained with a dye with fluorescence intermittency. These fluorescent molecule groups have their own independent molecular brightness that changes over time. The fluorescence signal at position r and time t can be expressed as: F(r,t) = h(r - r k )·c k ·ω k (t) where h, c, and ω respectively represent the point spread function of the corresponding microscope, the molecular brightness constant, and the function of the molecular brightness fluctuating over time.

4. The structured light imaging method based on signal fluctuation to expand resolution according to claim 2, wherein: In step a, the structured light is a sine stripe, which is generated by the interference of two plane waves and can be expressed as:

5. The structured light imaging method based on signal fluctuation to expand resolution according to claim 1, characterized in that: In step a, the imaging process needs to use structured light at 3 different angles for repeated imaging.

6. The structured light imaging method based on signal fluctuation to expand resolution according to claim 1, wherein: In the imaging process in step a, for the structured light at each angle, imaging is performed 3 times using different phases, and the imaging results can be expressed as: D(k) is the frequency domain result corresponding to the imaging of an ordinary microscope, and E(k) is the frequency domain result corresponding to the imaging of a structured light microscope.

7. The structured light imaging method based on signal fluctuation to expand resolution according to claim 2, wherein: In the imaging process described in step a, multi-angle and multi-phase imaging needs to be performed more than 100 times.

8. The structured light imaging method based on signal fluctuation to expand resolution according to claim 2, wherein: In the reconstruction process described in step d, the second-order time cumulant is generally used for reconstruction, and its reconstruction process can be expressed as: G2(r) = <δF(r,t)·δF(r,t)> t δF(r,t) = F(r,t) - 〈F(r,t)〉 t Among them, <·〉 t is the time-averaging function.

9. The structured light imaging method based on signal fluctuation to expand resolution according to claim 2, wherein: In the reconstruction process described in step d, when the expression of the second-order time cumulant is expanded, and when it meets the preset conditions, the cross-correlation term in the expansion of the second-order time cumulant is regarded as zero, so that the second-order time cumulant is expressed as the sum of the squares of the corresponding brightness constant weighted point spread functions. When the expression of the second-order time cumulant G2 is expanded, the following formula is obtained:

10. The structured light imaging method based on signal fluctuation to expand resolution according to claim 2, wherein: In the reconstruction process described in step d, assuming that the luminescence intensities of each fluorescent molecule are independent fluctuations that are not correlated, when i≠k (preset condition), the cross-correlation term in the expansion is regarded as zero, and the second-order time cumulant G2 is expressed as the sum of the squares of the corresponding brightness constant γ weighted point spread functions, as shown in the following formula:

Images