A deep learning based structured light super-resolution microscopic imaging method and system
By combining wide-field imaging with structured light illumination and employing deep learning methods to switch illumination modes with keyframe assistance, the problems of phototoxicity and photobleaching in structured light illumination micro-technology were solved, enabling high-quality super-resolution dynamic observation of live cells.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUZHOU INST OF BIOMEDICAL ENG & TECH CHINESE ACADEMY OF SCI
- Filing Date
- 2023-04-28
- Publication Date
- 2026-05-12
AI Technical Summary
Existing structured illumination microscopy techniques have high requirements for the signal-to-noise ratio of the original fluorescence image during reconstruction, resulting in severe artifacts that affect the quality of super-resolution images. Furthermore, the problems of phototoxicity and photobleaching have not been effectively solved, limiting the long-term dynamic observation of live cells.
By combining wide-field imaging and structured light illumination, and employing deep learning methods, high-quality super-resolution images are directly obtained from wide-field images through keyframe assistance. Illumination modes are switched using a porous mask and polarization adjustment device, and super-resolution images are reconstructed using neural networks.
It reduces phototoxicity and photobleaching, improves the quality and reliability of super-resolution images, and enables long-term dynamic observation of the submicroscopic structure of living cells.
Smart Images
Figure CN116559128B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of fluorescence microscopy imaging technology, and in particular, it is a method and system for performing dynamic long-term super-resolution microscopy imaging of live cells by fusing a wide-field illumination mode and a structured light illumination mode based on deep learning. Background Technology
[0002] In life science research, fluorescence microscopy is widely used due to its advantages such as specific labelability and real-time imaging of living cells. However, its application in biomedical research is greatly limited by the diffraction limit. In recent years, researchers have proposed super-resolution microscopy techniques that can overcome the diffraction limit, including stimulated emission depletion microscopy, photoactivated localization microscopy, stochastic optical reconstruction microscopy, and structured illumination microscopy. Among these, structured illumination microscopy illuminates the sample with modulated striped structured light and then reconstructs the image to achieve a resolution twice that of the diffraction limit. Compared to other super-resolution imaging techniques, structured illumination microscopy has a high temporal resolution (approximately 80 Hz), does not have special requirements for the fluorescent dyes used to label the sample, and has low phototoxicity (illumination intensity approximately 10 W / cm²). These advantages are crucial for the dynamic observation of living cells. Therefore, structured illumination microscopy is mainly used to observe subcellular level in vivo observations, including dynamic changes in mitochondria, cytoskeleton, chromosomes, intracellular vesicle movement, and viral movement within cells. However, structured light microscopy requires multiple measurements to reconstruct a super-resolution image, and its reconstruction algorithm involves complex operations such as frequency domain information separation and stitching. This reconstruction algorithm, however, demands a high signal-to-noise ratio (SNR) from the original fluorescence image. Otherwise, if a low SNR original fluorescence image is used for super-resolution reconstruction, the final super-resolution image will contain significant artifacts. These artifacts severely impact the quality of the final super-resolution reconstructed image, making it impossible to distinguish between the real sample information and the artifacts generated during reconstruction, thus affecting the microscopic observation results.
[0003] To address the aforementioned shortcomings, deep learning-based super-resolution microscopy methods using structured illumination have emerged. Deep learning-based methods have achieved significant success in learning end-to-end image transformation relationships from a large amount of example data. Since deep learning was first applied to structured illumination super-resolution microscopy in 2019, numerous researchers have proposed corresponding super-resolution reconstruction algorithms based on U-Net, Generative Adversarial Networks (GANs), and Residual Channel Attention Networks (RCANs). These algorithms primarily address two types of problems: 1) improving imaging quality under low signal-to-noise ratio conditions and reducing phototoxicity; 2) reducing the number of original images required for reconstruction. Solving these problems can effectively improve the temporal resolution of structured illumination microscopy, reduce photobleaching and phototoxicity during the imaging process, and make long-term real-time dynamic imaging of live cells possible.
[0004] However, obtaining super-resolution images directly from wide-field images using deep learning methods under structured light modulation still falls short of ideal results, and the reconstructed super-resolution structures are unreliable. Wide-field imaging can further reduce phototoxicity and photobleaching during live-cell dynamic imaging, extending the live-cell dynamic imaging time. Since diffraction-limited wide-field images do not contain resolvable super-resolution information, traditional algorithms based on analytical models such as Wiener deconvolution cannot obtain super-resolution images from wide-field images. Deep learning-based methods, however, do not require explicit analytical models; these data-driven methods can approximate not only the pseudo-inverse function of the image degradation process but also the stochastic characteristics of the super-resolution solution. Therefore, researching how to directly obtain high-quality, reliable super-resolution images from wide-field images using deep learning methods is crucial for long-term dynamic observation of submicroscopic structures in live cells. Summary of the Invention
[0005] The purpose of this invention is to address the problems existing in the prior art by providing a super-resolution microscopy imaging technology and system that integrates wide-field imaging and structured light illumination, so as to reduce phototoxicity and photobleaching during the structured light illumination super-resolution microscopy imaging process and realize long-term super-resolution dynamic observation of the submicroscopic structure of living cells.
[0006] The technical solution to achieve the purpose of this invention is as follows: On the one hand, a structured light super-resolution microscopic imaging system based on deep learning is provided. The system includes a light source, a first lens, a polarizing beam splitter, a spatial light modulator (SLM), a second lens, a photographic mode switching module, a third lens, a microscope objective, a three-dimensional motorized stage, a dichroic mirror, a tube mirror, a color filter, and a camera, all arranged sequentially along the optical path.
[0007] The light emitted from the light source passes through the first lens and the polarizing beam splitter, and is incident perpendicularly on the spatial light modulator (SLM). The diffracted light from the SLM returns along the original path, is reflected by the polarizing beam splitter, enters the second lens, and is collimated by the lens before entering the illumination mode switching module. The illumination mode switching module is used to switch between different illumination modes according to specific imaging requirements. The light emitted from the illumination mode switching module passes through the third lens and is reflected by the dichroic mirror to the microscope objective. The microscope objective focuses the light and excites the sample placed on the three-dimensional motorized stage to produce fluorescence. The fluorescence signal is collected by the microscope objective, transmitted through the dichroic mirror, and then through a combination of tube lens and color filter before being acquired and imaged by the camera.
[0008] Furthermore, the photographic mode switching module includes a porous mask, a polarization adjustment device, and a fourth lens arranged sequentially along the optical path.
[0009] Furthermore, the porous mask is used for spatial filtering and includes 2N rotationally symmetric pinholes, or a combination of these pinhole distribution patterns, where N is an integer.
[0010] Furthermore, the polarization adjustment device includes, but is not limited to, a combination of a polarization rotator and a liquid crystal phase compensator, or a combination of two half-wave plates.
[0011] Furthermore, the camera mode switching module includes three modes:
[0012] 1) 3D-SIM mode: This mode allows 0th order and positive and negative 1st order diffracted light to pass through;
[0013] 2) 2D-SIM mode: This mode only allows the positive and negative first-order diffracted light to pass through;
[0014] 3) Wide-field mode: This mode allows only one of the three diffracted beams (0th order, positive 1st order, or negative 1st order) to pass through, or it can be achieved by disrupting the beam interference conditions, or by switching the structured light stripe illumination mode within a single camera exposure time.
[0015] On the other hand, a structured light super-resolution microscopy method based on deep learning is provided, the method comprising the following steps:
[0016] Step 1: Switch the structured light super-resolution microscopy system to 3D-SIM mode or 2D-SIM mode, and acquire N frames of original images modulated by the structured light illumination mode.
[0017] Step 2: Based on the original image, a super-resolution image is reconstructed using an algorithm, which serves as a keyframe;
[0018] Step 3: Switch the structured light super-resolution microscopy system to wide-field mode and acquire the wide-field image at the current moment;
[0019] Step 4: Input the keyframes and wide-field images into the neural network to obtain the super-resolution image at the current moment;
[0020] Step 5: Acquire the wide-field image at the next moment.
[0021] Step 6: Input the keyframe and the wide-field image of the next moment into the neural network to obtain the super-resolution image of the current moment;
[0022] Step 7: Repeat steps 5 to 6 until super-resolution imaging is complete.
[0023] Furthermore, steps 5 to 7 can be replaced with:
[0024] Step 5: Determine whether the morphological changes of the biological structure exceed the preset change threshold. If so, execute steps 1 and 2 to obtain another keyframe, then acquire the wide-field image at the next moment and execute the next step; otherwise, acquire the wide-field image at the next moment and execute the next step.
[0025] Step 6: If the current number of keyframes is one, then execute: input the keyframe and the wide-field image of the next moment into the neural network to obtain the super-resolution image of the current moment; otherwise, execute: input any one or more keyframes and the wide-field image of the next moment into the neural network to obtain the super-resolution image of the current moment.
[0026] Step 7: Repeat steps 5 and 6 until super-resolution imaging is completed;
[0027] Furthermore, the judgment condition in step 5 can be replaced with: whether a new biological structure appears.
[0028] Furthermore, the judgment condition in step 5 can be replaced by: whether the time difference between the current time and the initial time reaches the preset time interval threshold.
[0029] Furthermore, in step 6, the input to the neural network also includes one or more super-resolution images output by the neural network at previous time steps, and their relationship with the keyframes is a sum-or-or relationship.
[0030] This invention leverages the temporal continuity of biological samples, the high temporal resolution and low phototoxicity of structured illumination super-resolution microscopy with keyframe assistance, and the strong correlation between adjacent frames to propose a deep learning-based method and system for structured illumination super-resolution microscopy with keyframe assistance. Compared with existing technologies, its significant advantages are:
[0031] 1) With the help of the keyframes containing real super-resolution structural information, the reconstruction results from wide field to super resolution based on deep learning are of better quality and more realistic and reliable.
[0032] 2) The light dose during the entire imaging process is comparable to that of wide-field imaging, which effectively reduces phototoxicity and photobleaching during structured light super-resolution microscopy, making it more conducive to long-term dynamic super-resolution observation of submicroscopic structures of living cells.
[0033] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description
[0034] Figure 1 This is a schematic diagram of a 3D-SIM imaging system that combines deep learning-based wide-field and structured light illumination super-resolution microscopy in one embodiment.
[0035] Figure 2 for Figure 1 The optical path diagrams for the three lighting modes included in the lighting mode switching module are shown below. Figure 2 (a) in the image represents the 3D-SIM mode. Figure 2 (b) in the diagram represents the 2D-SIM mode. Figure 2 (c) to (f) represent four wide-field modes.
[0036] Figure 3 This is a schematic diagram of a 2D-SIM imaging system that combines deep learning-based wide-field and structured light illumination super-resolution microscopy in one embodiment.
[0037] Figure 4 for Figure 3 The optical path diagrams for the three lighting modes included in the lighting mode switching module are shown below. Figure 4 (a) in the image represents the 2D-SIM mode. Figure 4 (b) to (d) represent three wide-field modes.
[0038] Figure 5 This is a schematic diagram of a reconstruction method that acquires only one keyframe in one embodiment.
[0039] Figure 6 This is a schematic diagram illustrating the principle of a reconstruction method for acquiring multiple keyframes in one embodiment.
[0040] Figure 7(a) shows a wide-field fluorescence image in one embodiment.
[0041] Figure 7(b) shows a super-resolution image reconstructed by an algorithm based on the Wiener deconvolution model in one embodiment.
[0042] Figure 7(c) shows a super-resolution image reconstructed without keyframe assistance based on deep learning in one embodiment.
[0043] Figure 7(d) shows a super-resolution image reconstructed using keyframe-assisted deep learning in one embodiment. Detailed Implementation
[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0045] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.
[0046] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.
[0047] In one embodiment, combined Figure 1 and Figure 3 This paper presents a structured light super-resolution microscopy imaging system based on deep learning, comprising, sequentially arranged along the optical path, a light source, a first lens 1, a polarizing beam splitter 2, a spatial light modulator (SLM), a second lens 3, a porous mask 4, a polarization adjustment device 5, a fourth lens 6, a third lens 7, a microscope objective 8, a three-dimensional motorized stage 9, a dichroic mirror 10, a tube mirror 11, a color filter 12, and a camera. Detailed description follows:
[0048] Light emitted from the light source passes through the first lens 1 and the polarizing beam splitter 2, and is incident perpendicularly on the spatial light modulator (SLM). The SLM is loaded with a pre-set mode, and the diffracted light exiting the SLM returns along its original path, is reflected by the polarizing beam splitter 2, and enters the second lens 3 (focal length 300mm). After being collimated by the second lens 3, it passes through the illumination mode switching module. The illumination mode switching module switches between different illumination modes according to specific imaging requirements. The light exiting the illumination mode switching module passes through the third lens 7 and finally reaches the inverted microscope objective 8. The microscope objective 8 focuses the light and excites the sample placed on the three-dimensional motorized stage 9. In structured light illumination mode, two or three diffracted beams form interference fringes on the sample surface, exciting the sample to produce fluorescence. The fluorescence signal is collected by the microscope objective 8 and transmitted through the dichroic mirror 10, then through the tube lens 11 and the color filters 12 configured in the microscope system. Finally, these signals are acquired by the camera and imaged.
[0049] The specific description of the lighting mode switching module is as follows:
[0050] The illumination mode switching module includes a porous mask 4, a polarization adjustment device 5, and a third lens 6 arranged sequentially along the optical path. The porous mask consists of 2N (where N is an integer) rotationally symmetrically distributed pinholes, or a combination of these distribution patterns, and its function is spatial filtering, allowing only light rays meeting a specific incident angle to pass through. The polarization adjustment device includes, but is not limited to, a combination of a polarization rotator and a liquid crystal phase compensator. The polarization rotator ensures that the polarization direction of the positive and negative first-order beams is always perpendicular to the plane common to both beams. The fast axis of the liquid crystal phase compensator is parallel to the polarization direction of the transmitted light from the polarizing beam splitter 2. The phase compensation amount is set by a computer to compensate for laser phase drift caused by optical elements such as dichroic mirrors, ensuring that the laser illuminating the sample surface remains linearly polarized and its polarization direction is perpendicular to the plane common to both beams, thus achieving maximum structured light modulation. The phase compensation amount needs to be adjusted when the laser wavelength changes. Another commonly used polarization adjustment device is a combination of two half-wave plates.
[0051] The lighting mode switching module in this invention includes three modes, combined with... Figure 2 and Figure 4 The specific description is as follows:
[0052] a) 3D-SIM mode: This mode will allow 0th order and positive and negative 1st order diffraction light to pass through;
[0053] b) 2D-SIM mode: This mode only allows the positive and negative first order diffraction light to pass through;
[0054] c) Wide-field mode: This mode allows only one of the three diffracted beams (0th order, positive 1st order, or negative 1st order) to pass through, or it can be achieved by disrupting the beam interference conditions, or by switching the structured light stripe illumination mode during a single camera exposure.
[0055] The optical paths for the three lighting modes will be described in detail below:
[0056] a) 3D-SIM mode
[0057] Light emitted from the light source passes through the first lens 1 and the polarizing beam splitter 2, and is incident perpendicularly on the spatial light modulator. A pre-set pattern is applied to the spatial light modulator, and the diffracted light exiting the modulator returns along its original path, reflected by the polarizing beam splitter 2, and enters the second lens 3. After being collimated by the second lens 3, it passes through the porous mask 4, allowing the 0th order and positive and negative 1st order diffracted light to pass through. These three orders of diffracted light pass through the polarization adjustment device 5, then through the fourth lens 6 and the third lens 7, finally reaching the inverted microscope; focused by the microscope objective 8, it excites the sample placed on the three-dimensional motorized stage 9. The three diffracted beams form interference fringes on the sample surface, exciting the sample to produce fluorescence. The fluorescence signal is collected by the microscope objective 8 and transmitted through the dichroic mirror 10, then through the tube lens 11 and the color filters 12 configured in the microscope system. Finally, these signals are acquired by the camera and imaged.
[0058] b) 2D-SIM mode
[0059] Light emitted from the light source passes through the first lens 1 and the polarizing beam splitter 2, and is incident perpendicularly on the spatial light modulator. A pre-set pattern is applied to the spatial light modulator, and the diffracted light exiting the modulator returns along its original path, reflected by the polarizing beam splitter 2, and enters the second lens 3. After being collimated by the second lens 3, it passes through the porous mask 4, at which point only the positive and negative first-order diffracted light is allowed to pass. These two orders of diffracted light pass through the polarization adjustment device 5, then through the fourth lens 6 and the third lens 7, finally reaching the inverted microscope; it is focused by the microscope objective 8 and excites the sample placed on the three-dimensional motorized stage 9. The three diffracted beams form interference fringes on the sample surface, exciting the sample to produce fluorescence. The fluorescence signal is collected by the objective 8 and transmitted through the dichroic mirror 10, then through the tube lens 11 and the color filters 12 configured in the microscope system. Finally, these signals are acquired by the camera and imaged.
[0060] c) Wide field mode
[0061] Light emitted from the light source passes through the first lens 1 and the polarizing beam splitter 2, and is incident perpendicularly on the spatial light modulator. A pre-set pattern is applied to the spatial light modulator, and the diffracted light exiting the modulator returns along its original path, reflected by the polarizing beam splitter and entering the second lens 3. After being collimated by the second lens 3, it passes through a porous mask, allowing only one beam of diffracted light from the 0th, positive 1st, and negative 1st orders to pass through. This beam of diffracted light passes through the polarization adjustment device 5, then through the fourth lens 6 and the third lens 7, finally reaching the inverted microscope. It is then focused by the microscope objective 8 and excites the sample placed on the three-dimensional motorized stage 9. The fluorescence signal is collected by the objective 8 and transmitted through the dichroic mirror 10, passing through the tube lens 11 and the color filters 12 configured in the microscope system. Finally, these signals are acquired by the camera and imaged.
[0062] Alternatively, positive and negative beams can pass through, and the interference conditions can be disrupted by adjusting the polarization adjustment device. After passing through the polarization adjustment device 5, these two beams of light pass through the fourth lens 6 and the third lens 7, finally reaching the inverted microscope. After being focused by the microscope objective 8, they excite the sample placed on the three-dimensional motorized stage 9. The fluorescence signal is collected by the objective 8 and transmitted through the dichroic mirror 10. After passing through the tube lens 11 and the color filter 12 configured in the microscope system, these signals are finally acquired by the camera and imaged.
[0063] Alternatively, positive and negative beams can pass through, and by rapidly switching illumination modes (different azimuth angles and phases) within a single camera exposure time, the two beams of light pass through the polarization adjustment device 5, then through the fourth lens 6 and the third lens 7, finally reaching the inverted microscope; after being focused by the microscope objective 8, they excite the sample placed on the three-dimensional motorized stage 9. The fluorescence signal is collected by the objective 8 and transmitted through the dichroic mirror 10, then through the combination of the tube lens 11 and the color filters 12 configured in the microscope system, and finally, these signals are acquired by the camera and imaged.
[0064] Alternatively, the 0th order and positive and negative orders can be allowed to pass through, and the interference conditions can be disrupted by adjusting the polarization adjustment device. These three beams of light pass through the polarization adjustment device 5, then through the fourth lens 6 and the third lens 7, and finally reach the inverted microscope. After being focused by the microscope objective 8, they excite the sample placed on the three-dimensional motorized stage 9. The fluorescence signal is collected by the objective 8 and transmitted through the dichroic mirror 10. After passing through the tube lens 11 and the color filter 12 configured in the microscope system, these signals are finally acquired by the camera and imaged.
[0065] Alternatively, it may allow level 0 and positive / negative levels to pass through, and by rapidly switching illumination modes (different azimuth angles and phases) within a single camera exposure time, the three beams of light pass through the polarization adjustment device 5, then through the fourth lens 6 and the third lens 7, finally reaching the inverted microscope; after being focused by the microscope objective 8, they excite the sample placed on the three-dimensional motorized stage 9. The fluorescence signal is collected by the objective 8 and transmitted through the dichroic mirror 10, then through the combination of the tube lens 11 and the color filters 12 configured in the microscope system, and finally, these signals are acquired by the camera and imaged.
[0066] In one embodiment, a deep learning-based keyframe-assisted structured illumination micro-super-resolution imaging method is provided. A keyframe refers to a real super-resolution image obtained by reconstructing N frames (N is an integer, typically 9 or 15) of the original image modulated by structured illumination patterns under high signal-to-noise ratio conditions using traditional algorithms based on analysis models such as Wiener deconvolution.
[0067] Specifically, a deep learning-based structured light super-resolution microscopy imaging method is provided, the method comprising the following steps:
[0068] Step 1: Switch the illumination mode of the structured light super-resolution microscopy system to 3D-SIM mode or 2D-SIM mode, and acquire N frames of original images modulated by the structured light illumination mode.
[0069] Step 2: Based on the original image, a super-resolution image is reconstructed using an algorithm, which serves as a keyframe;
[0070] Step 3: Switch the illumination mode of the structured light super-resolution microscopy system to wide-field mode and acquire the wide-field image at the current moment;
[0071] Step 4: Input the keyframes and wide-field images into the neural network to obtain the super-resolution image at the current moment;
[0072] Step 5: Acquire the wide-field image at the next moment.
[0073] Step 6: Input the keyframe and the wide-field image of the next moment into the neural network to obtain the super-resolution image of the current moment;
[0074] Step 7: Repeat steps 5 to 6 until super-resolution imaging is completed.
[0075] Furthermore, in one embodiment, steps 5 to 7 can be replaced with:
[0076] Step 5: Determine whether the morphological changes of the biological structure exceed the preset change threshold (with the same total acquisition time, the number of keyframes acquired for samples with faster changes will be more, and the number of keyframes acquired for samples with slower changes will be less). If yes, execute steps 1 and 2 to obtain another keyframe, then acquire the wide-field image at the next moment and execute the next step; otherwise, acquire the wide-field image at the next moment and execute the next step.
[0077] Step 6: If the current number of keyframes is one, then execute: input the keyframe and the wide-field image of the next moment into the neural network to obtain the super-resolution image of the current moment; otherwise, execute: input any one or more keyframes and the wide-field image of the next moment into the neural network to obtain the super-resolution image of the current moment.
[0078] Step 7: Repeat steps 5 to 6 until super-resolution imaging is completed.
[0079] Furthermore, in one embodiment, the judgment condition in step 5 can be replaced by: whether a new structure appears.
[0080] Furthermore, in one embodiment, the judgment condition in step 5 can be replaced by: whether the time difference between the current time and the initial time reaches a preset time interval threshold.
[0081] Furthermore, in one embodiment, the input of the neural network in step 6 also includes the super-resolution image output by the neural network at any time, and its relationship with the keyframe is an AND-OR relationship.
[0082] As shown above, keyframe-assisted methods include the following:
[0083] 1) Assisted reconstruction mode that acquires only one keyframe:
[0084] In this mode, a single keyframe is acquired at the very beginning of imaging, and subsequent fluorescence images are acquired only in wide-field mode. The utilization of this single keyframe in this mode can be further divided into the following two categories:
[0085] a) Super-resolution images without utilizing the output of the neural network: keyframes are input into the network to participate in the reconstruction each time a super-resolution image is reconstructed from the current wide-field image through the neural network;
[0086] b) Utilizing super-resolution images output by neural networks: Each time a super-resolution image is reconstructed from the current wide-field image using a neural network, firstly, except for the first frame (where the keyframe is defined as the 0th frame), which must be included in the reconstruction, keyframes for the remaining frames can be selectively input into the network for reconstruction. Secondly, the super-resolution images output by the network from previous time steps will also be selectively input into the network for reconstruction. One example is shown below:
[0087] The first super-resolution image is obtained by using keyframes and the first wide-field image. Then, the second super-resolution image is obtained by using the first super-resolution image output by the network and the second wide-field image. This process is repeated to obtain all the super-resolution images.
[0088] 2) Assisted reconstruction mode with multiple keyframes:
[0089] In this mode, the approach remains the same: acquire a keyframe at the very beginning of the imaging process, followed by acquiring fluorescence images only under wide-field illumination for a period of time. Then, as needed, keyframes can be acquired again at intervals. The utilization of keyframes in this mode can also be categorized into the following two types:
[0090] a) Super-resolution images without utilizing the output of the neural network: keyframes are selectively input into the network to participate in the reconstruction each time a super-resolution image is reconstructed from the current wide-field image through the neural network;
[0091] b) Using the super-resolution image output by the neural network: each time a super-resolution image is reconstructed from the current wide-field image through the neural network, keyframes and the super-resolution images output by the network at previous times are selectively input into the network to participate in the reconstruction.
[0092] The following is a detailed explanation.
[0093] Figure 5 The principle of the reconstruction method that acquires only one keyframe is illustrated. It will be described in detail below:
[0094] a) Acquisition of time series data: At the beginning of the imaging experiment, the illumination mode was switched to 3D-SIM or 2D-SIM mode and N frames of raw images with stripes were acquired. Then the illumination mode was switched to wide field mode and wide field fluorescence images of the biological sample were acquired until the end of the experiment.
[0095] b) Reconstruction Strategy 1: This reconstruction strategy does not utilize the super-resolution image output by the neural network.
[0096] Step 1: Reconstruct the keyframes from the initial N frames of original images using traditional reconstruction algorithms based on Wiener deconvolution and other analysis models;
[0097] Step 2: Input the first frame wide-field image and the keyframe into the neural network, and the network outputs the first frame super-resolution image;
[0098] Step 3: Input the second frame wide-field image along with the keyframe into the neural network, and the network outputs the second frame super-resolution image;
[0099] This process is repeated to obtain all the super-resolution images.
[0100] c) Reconstruction Strategy Two: This reconstruction strategy utilizes the super-resolution image output by the neural network. (The following is just one example to illustrate the idea behind this type of reconstruction strategy.)
[0101] Step 1: Reconstruct the keyframes from the initial N frames of original images using traditional reconstruction algorithms based on Wiener deconvolution and other analysis models;
[0102] Step 2: Input the first frame wide-field image and the keyframe into the neural network, and the network outputs the first frame super-resolution image;
[0103] Step 3: Input the second frame wide-field image and the first frame super-resolution image into the neural network, and the network outputs the second frame super-resolution image;
[0104] Step 4: Input the third frame wide-field image and the second frame super-resolution image into the neural network, and the network outputs the third frame super-resolution image;
[0105] This process is repeated to obtain all the super-resolution images.
[0106] Figure 6 The principle of a reconstruction method that acquires multiple keyframes is illustrated. It will be described in detail below:
[0107] a) Methods for obtaining time series data:
[0108] Step 1: At the very beginning of the imaging experiment, switch the illumination mode to 3D-SIM or 2D-SIM mode and acquire N frames of raw images with stripes;
[0109] Step 2: Switch the illumination mode to wide field mode and continue to acquire wide field fluorescence images of the biological sample;
[0110] Step 3: After a period of time, switch the lighting mode back to 3D-SIM or 2D-SIM mode as needed, and acquire N frames of original images with stripes.
[0111] Step 4: Switch the illumination mode back to wide-field mode and continue acquiring wide-field fluorescence images of the biological sample;
[0112] Repeat steps three and four as needed until the experiment is finished.
[0113] b) Reconstruction Strategy 1: This reconstruction strategy does not utilize the super-resolution image output by the neural network. (Only two examples are given below to illustrate the idea behind this type of reconstruction strategy.)
[0114] Method 1:
[0115] Step 1: Reconstruct K keyframes from the K sets (K is an integer and greater than or equal to 2) of original images (each set contains N original images) using traditional reconstruction algorithms based on Wiener deconvolution and other analysis models.
[0116] Step 2: Input the first frame wide field image and M key frames (M is greater than or equal to 1 and less than K) with the closest time interval (key frame 1 and key frame 2 at this time) into the neural network. Do not input other key frames into the network. The network outputs the first frame super-resolution image.
[0117] Step 3: Input the second frame wide-field image and M keyframes (M is greater than or equal to 1 and less than K) with the closest time interval into the neural network. Do not input other keyframes into the network. The network outputs the second frame super-resolution image.
[0118] This process continues, selecting M keyframes (M ≥ 1 and < K) that are closest in time to the current wide-field image and inputting them into the network together with the current wide-field image to obtain the current super-resolution image. Finally, super-resolution images for all time points are obtained.
[0119] Method 2:
[0120] Step 1: Reconstruct K keyframes from the K sets (K is an integer and greater than or equal to 2) of original images (each set contains N original images) using traditional reconstruction algorithms based on Wiener deconvolution and other analysis models.
[0121] Step 2: Input the first frame wide field image, the key frame with the closest time interval to it, and M randomly selected key frames (M is greater than or equal to 1 and less than K-1) into the neural network. Do not input other key frames into the network. The network outputs the first frame super-resolution image.
[0122] Step 3: Input the second frame wide field image, a key frame with the closest time interval to it, and M randomly selected key frames (M is greater than or equal to 1 and less than K-1) into the neural network. Do not input other key frames into the network. The network outputs the second frame super-resolution image.
[0123] By selecting the corresponding keyframes according to this rule, all super-resolution images can be reconstructed.
[0124] c) Reconstruction Strategy Two: This reconstruction strategy utilizes the super-resolution image output by the neural network. (Only two examples are given below to illustrate the idea behind this type of reconstruction strategy.)
[0125] Method 1:
[0126] Step 1: Reconstruct K keyframes from the K sets (K is an integer and greater than or equal to 2) of original images (each set contains N original images) using traditional reconstruction algorithms based on Wiener deconvolution and other analysis models.
[0127] Step 2: Input the first frame wide-field image and M keyframes (M is greater than or equal to 1 and less than K) with the closest time interval into the neural network. Do not input other keyframes into the network. The network outputs the first frame super-resolution image.
[0128] Step 3: Input the second frame wide field image, along with M keyframes (M is greater than or equal to 1 and less than K) that are closest to it in time interval, and the first frame super-resolution image into the neural network. Do not input other keyframes into the network. The network outputs the second frame super-resolution image.
[0129] Step 4: Input the 3rd frame wide-field image, along with M keyframes (M is greater than or equal to 1 and less than K) that are closest to it in time interval, and the 2nd frame super-resolution image into the neural network. Do not input other keyframes into the network. The network outputs the 3rd frame super-resolution image.
[0130] This process continues, selecting M keyframes (M ≥ 1 and < K) that are closest in time to the current wide-field image, along with the super-resolution image from the previous time step output by the network, and inputting them together with the current wide-field image to obtain the current super-resolution image. Finally, super-resolution images from all time steps are obtained.
[0131] Method 2:
[0132] Step 1: Reconstruct K keyframes from the K sets (K is an integer and greater than or equal to 2) of original images (each set contains N original images) using traditional reconstruction algorithms based on Wiener deconvolution and other analysis models.
[0133] Step 2: Input the first frame wide field image, the key frame with the closest time interval to it, and M randomly selected key frames (M is greater than or equal to 1 and less than K-1) into the neural network. Do not input other key frames into the network. The network outputs the first frame super-resolution image.
[0134] Step 3: Input the second frame wide field image, the key frame with the closest time interval to it, M randomly selected key frames (M is greater than or equal to 1 and less than K-1), and the first frame super-resolution image into the neural network. Do not input other key frames into the network. The network outputs the second frame super-resolution image.
[0135] Step 4: Input the 3rd frame wide-field image, the key frame with the closest time interval to it, M randomly selected key frames (M is greater than or equal to 1 and less than K-1), and the 1st and 2nd frame super-resolution images into the neural network. Do not input other key frames into the network. The network outputs the 3rd frame super-resolution image.
[0136] Step 5: Input the 4th frame wide-field image, the key frame with the closest time interval to it, M randomly selected key frames (M is greater than or equal to 1 and less than K-1), and the 2nd and 3rd frame super-resolution images into the neural network. Do not input other key frames into the network. The network outputs the 4th frame super-resolution image.
[0137] This process continues, each time selecting one keyframe (the one closest in time to the current wide-field image), M randomly selected keyframes (M greater than or equal to 1 and less than K-1), and two adjacent super-resolution images output by the network, and inputting them into the network along with the current wide-field image to obtain the super-resolution image at the current moment. Finally, the super-resolution images at all moments are obtained.
[0138] For example, Figures 7(a) to 7(d) The study demonstrates that with keyframe assistance, the network reconstructs super-resolution images of higher quality than those without keyframe assistance, revealing clear inner ridges.
[0139] In summary, the proposed solution can reduce phototoxicity and photobleaching during structured light illumination super-resolution microscopy, enabling long-term super-resolution dynamic observation of submicroscopic structures of living cells.
[0140] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:
[0141] Step 1: Switch the illumination mode of the structured light super-resolution microscopy system to 3D-SIM mode or 2D-SIM mode, and acquire N frames of original images modulated by the structured light illumination mode.
[0142] Step 2: Based on the original image, a super-resolution image is reconstructed using an algorithm, which serves as a keyframe;
[0143] Step 3: Switch the illumination mode of the structured light super-resolution microscopy system to wide-field mode and acquire the wide-field image at the current moment;
[0144] Step 4: Input the keyframes and wide-field images into the neural network to obtain the super-resolution image at the current moment;
[0145] Step 5: Acquire the wide-field image at the next moment.
[0146] Step 6: Input the keyframe and the wide-field image of the next moment into the neural network to obtain the super-resolution image of the current moment;
[0147] Step 7: Repeat steps 5 to 6 until super-resolution imaging is completed.
[0148] or:
[0149] Step 1: Switch the illumination mode of the structured light super-resolution microscopy system to 3D-SIM mode or 2D-SIM mode, and acquire N frames of original images modulated by the structured light illumination mode.
[0150] Step 2: Based on the original image, a super-resolution image is reconstructed using an algorithm, which serves as a keyframe;
[0151] Step 3: Switch the illumination mode of the structured light super-resolution microscopy system to wide-field mode and acquire the wide-field image at the current moment;
[0152] Step 4: Input the keyframes and wide-field images into the neural network to obtain the super-resolution image at the current moment;
[0153] Step 5: Determine whether the time difference between the current moment and the initial moment reaches the preset time interval threshold. If yes, execute steps 1 and 2 to obtain another keyframe, then acquire the wide-field image of the next moment and execute the next step; otherwise, acquire the wide-field image of the next moment and execute the next step.
[0154] Step 6: If the current number of keyframes is one, then execute: input the keyframe and the wide-field image of the next moment into the neural network to obtain the super-resolution image of the current moment; otherwise, execute: input any one or more keyframes and the wide-field image of the next moment into the neural network to obtain the super-resolution image of the current moment.
[0155] Step 7: Repeat steps 5 to 6 until super-resolution imaging is completed.
[0156] In step 6, the input to the neural network also includes one or more super-resolution images output by the neural network at previous time steps, and their relationship with the keyframes is a sum-or-or relationship.
[0157] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0158] Step 1: Switch the illumination mode of the structured light super-resolution microscopy system to 3D-SIM mode or 2D-SIM mode, and acquire N frames of original images modulated by the structured light illumination mode.
[0159] Step 2: Based on the original image, a super-resolution image is reconstructed using an algorithm, which serves as a keyframe;
[0160] Step 3: Switch the illumination mode of the structured light super-resolution microscopy system to wide-field mode and acquire the wide-field image at the current moment;
[0161] Step 4: Input the keyframes and wide-field images into the neural network to obtain the super-resolution image at the current moment;
[0162] Step 5: Acquire the wide-field image at the next moment.
[0163] Step 6: Input the keyframe and the wide-field image of the next moment into the neural network to obtain the super-resolution image of the current moment;
[0164] Step 7: Repeat steps 5 to 6 until super-resolution imaging is completed.
[0165] or:
[0166] Step 1: Switch the illumination mode of the structured light super-resolution microscopy system to 3D-SIM mode or 2D-SIM mode, and acquire N frames of original images modulated by the structured light illumination mode.
[0167] Step 2: Based on the original image, a super-resolution image is reconstructed using an algorithm, which serves as a keyframe;
[0168] Step 3: Switch the illumination mode of the structured light super-resolution microscopy system to wide-field mode and acquire the wide-field image at the current moment;
[0169] Step 4: Input the keyframes and wide-field images into the neural network to obtain the super-resolution image at the current moment;
[0170] Step 5: Determine whether the time difference between the current moment and the initial moment reaches the preset time interval threshold. If yes, execute steps 1 and 2 to obtain another keyframe, then acquire the wide-field image of the next moment and execute the next step; otherwise, acquire the wide-field image of the next moment and execute the next step.
[0171] Step 6: If the current number of keyframes is one, then execute: input the keyframe and the wide-field image of the next moment into the neural network to obtain the super-resolution image of the current moment; otherwise, execute: input any one or more keyframes and the wide-field image of the next moment into the neural network to obtain the super-resolution image of the current moment.
[0172] Step 7: Repeat steps 5 to 6 until super-resolution imaging is completed.
[0173] In step 6, the input to the neural network also includes the super-resolution image output by the neural network at any time, and its relationship with the keyframe is a sum-or-or relationship.
[0174] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.
Claims
1. A deep learning-based structured light super-resolution microscopy imaging method for achieving long-term super-resolution dynamic observation of submicroscopic structures of living cells, characterized in that... This is achieved based on a deep learning-based structured light super-resolution microscopy imaging system, which includes a light source, a first lens, a polarizing beam splitter, a spatial light modulator (SLM), a second lens, a photographic mode switching module, a third lens, a microscope objective, a three-dimensional motorized stage, a dichroic mirror, a tube mirror, a color filter, and a camera, all arranged sequentially along the optical path. The light emitted from the light source passes through the first lens and the polarizing beam splitter, and is incident perpendicularly on the spatial light modulator (SLM). The diffracted light from the SLM returns along the original path, is reflected by the polarizing beam splitter, enters the second lens, and is collimated by the lens before entering the illumination mode switching module. The illumination mode switching module is used to switch between different illumination modes according to specific imaging requirements. The light emitted from the illumination mode switching module passes through the third lens and is reflected by the dichroic mirror to the microscope objective. The microscope objective focuses the light and excites the sample placed on the three-dimensional motorized stage to produce fluorescence. The fluorescence signal is collected by the microscope objective, transmitted through the dichroic mirror, and then through a combination of a tube lens and a color filter before being acquired and imaged by the camera. The camera mode switching module includes three modes: 1) 3D-SIM mode: This mode allows 0th order and positive and negative 1st order diffracted light to pass through; 2) 2D-SIM mode: This mode only allows the positive and negative first-order diffracted light to pass through; 3) Wide-field mode: This mode allows only one of the three diffracted beams (0th order, positive 1st order, or negative 1st order) to pass through, or it can be achieved by breaking the beam interference conditions, or by switching the structured light stripe illumination mode during a single camera exposure. The method includes the following steps: Step 1: Switch the illumination mode of the structured light super-resolution microscopy system to 3D-SIM mode or 2D-SIM mode, and acquire N frames of original images modulated by the structured light illumination mode. Step 2: Based on the original image, a super-resolution image is reconstructed using an algorithm, which serves as a keyframe; Step 3: Switch the illumination mode of the structured light super-resolution microscopy system to wide-field mode and acquire the wide-field image at the current moment. Step 4: Input the keyframes and wide-field images into the neural network to obtain the super-resolution image at the current moment; Step 5: Acquire the wide-field image at the next moment. Step 6: Input the keyframe and the wide-field image of the next moment into the neural network to obtain the super-resolution image of the current moment; Step 7: Repeat steps 5 to 6 until super-resolution imaging is complete.
2. The deep learning-based structured light super-resolution microscopy imaging method according to claim 1, characterized in that, Steps 5 to 7 can be replaced with: Step 5: Determine whether the morphological changes of the biological structure exceed the preset change threshold. If so, execute steps 1 and 2 to obtain another keyframe, then acquire the wide-field image at the next moment and execute the next step; otherwise, acquire the wide-field image at the next moment and execute the next step. Step 6: If the current number of keyframes is one, then execute: input the keyframe and the wide-field image of the next moment into the neural network to obtain the super-resolution image of the current moment; otherwise, execute: input any one or more keyframes and the wide-field image of the next moment into the neural network to obtain the super-resolution image of the current moment. Step 7: Repeat steps 5 to 6 until super-resolution imaging is complete.
3. The deep learning-based structured light super-resolution microscopy imaging method according to claim 2, characterized in that, The judgment condition in step 5 can be replaced with: whether a new biological structure appears.
4. The deep learning-based structured light super-resolution microscopy method according to claim 3, characterized in that, The judgment condition in step 5 can be replaced by: whether the time difference between the current time and the initial time reaches the preset time interval threshold.
5. The deep learning-based structured light super-resolution microscopy imaging method according to claim 2, characterized in that, In step 6, the input to the neural network also includes one or more super-resolution images output by the neural network at previous time steps, and their relationship with the keyframes is a sum-and-sum relationship.
6. The deep learning-based structured light super-resolution microscopy imaging method according to claim 1, characterized in that, The photographic mode switching module includes a porous mask, a polarization adjustment device, and a fourth lens arranged sequentially along the optical path.
7. The deep learning-based structured light super-resolution microscopy method according to claim 6, characterized in that, The porous mask template is used for spatial filtering and includes 2N rotationally symmetric pinholes, or a combination of these pinhole distribution patterns, where N is an integer.
8. The deep learning-based structured light super-resolution microscopy method according to claim 6, characterized in that, The polarization adjustment device includes a combination of a polarization rotator and a liquid crystal phase compensator, or a combination of two half-wave plates.