A fast frequency-shifting super-resolution microscopy imaging method based on annular illumination

Abstract

The present invention discloses a fast frequency-shifting super-resolution microscopic imaging method based on annular illumination, comprising the following steps: constructing an annular illumination light source composed of multiple circles of annular light sources with the same center; acquiring a single wide-field image when a sample is illuminated at normal incidence, using each circle of annular light sources to illuminate the sample at an angle and acquiring a corresponding single mixed-frequency-shifted low-resolution image to obtain multiple low-resolution images; using the single wide-field image and the multiple low-resolution images to perform fast frequency-shifting super-resolution microscopic imaging through a super-resolution imaging model constructed based on deep learning to obtain a super-resolution image. Compared with the existing method of sequentially lighting LEDs for oblique-incident illumination / sequentially illuminating with different illumination stripes, the annular illumination method can greatly reduce the number of acquired images, thereby improving the imaging speed, and is expected to be applied to video-level super-resolution microscopic imaging.

Description

Technical Field

[0001] The invention belongs to the technical field of super-resolution microscopy, and in particular relates to a fast frequency-shifting super-resolution microscopy imaging method based on annular illumination. Background Art

[0002] Super-resolution microscopy technology breaks the diffraction limit of traditional optical microscopes, directly and profoundly reveals the details of the microscopic world of organisms and materials, as well as the internal mechanisms and interrelationships of related substances. It has become an important tool for achieving key breakthroughs in biology, materials science and basic medical research.

[0003] The super-resolution microscopy technology based on spatial spectrum frequency shift is based on the wide-field imaging mechanism and can obtain a large imaging field of view, which makes the super-resolution microscopy technology have broad application prospects in the fields of biology, medicine, industrial detection, etc. The super-resolution microscopy technology based on spatial spectrum frequency shift needs to collect a large number of low-resolution images, which carry different spatial high-frequency information, and then splice these low-resolution images in the frequency domain to finally reconstruct the super-resolution image.

[0004] The existing super-resolution microscopy technology based on spatial spectrum frequency shift mainly includes Fourier stack microscopy (FPM), which uses programmable LED light sources to provide illumination at different distances and directions, and reconstructs super-resolution images by collecting low-resolution images under each illumination. The FPM technology requires that the low-resolution images collected each time have a certain overlap rate in the spectrum, and the overlap rate generally needs to be at least 35%. This results in the FPM technology needing to collect a large number of original low-resolution images, and the imaging speed is slow.

[0005] The patent application with application publication number CN110658195A provides a frequency-shifted label-free super-resolution microscopy chip and an imaging method thereof, which performs super-resolution imaging based on FPM technology. Specifically: first, an ordinary microscope is used to illuminate the sample, and an optical camera is used to collect low-frequency spatial information of the sample; then the light spot is projected onto different gratings to collect high-frequency spatial information of the sample in different directions and frequencies; the low-frequency and high-frequency spatial information are frequency-shifted, and then iteratively spliced ​​in the frequency domain space to obtain an expanded spectrum, and finally an inverse Fourier transform is performed to reconstruct a super-resolution sample image. This technology still has the technical defect of slow imaging speed.

[0006] In addition, there is also structured illumination microscopy (SIM) technology based on spatial spectrum frequency shifting. This SIM technology uses spatial structured light (illumination stripes) to excite fluorescence, shifts high-frequency information to the low-pass band of the microscope, and obtains more information by changing the direction and phase of the illumination stripes, thereby restoring the super-resolution image. This SIM technology requires the acquisition of at least 9 frames of original low-resolution images, and the imaging speed is still limited. Summary of the invention

[0007] In view of the above, an object of the present invention is to provide a fast frequency shift super-resolution microscopy imaging method based on annular illumination, which reduces the number of original low-resolution images collected through annular illumination, thereby improving the imaging speed of super-resolution imaging.

[0008] To achieve the above-mentioned purpose of the invention, an embodiment provides a method for fast frequency shifting super-resolution microscopic imaging based on annular illumination, comprising the following steps:

[0009] Constructing a circular lighting source composed of multiple circular ring light sources with the same center;

[0010] A single wide-field image is acquired when the sample is illuminated at normal incidence, and each ring light source is used to illuminate the sample obliquely and acquire a corresponding single frequency-mixed and frequency-shifted low-resolution image to obtain multiple low-resolution images;

[0011] A single wide-field image and multiple low-resolution images are combined into a super-resolution microscopy image using a super-resolution imaging model built based on deep learning to obtain a super-resolution image.

[0012] Preferably, in the circular lighting light source, each circle of the circular lighting light source adopts the same light source, including a discrete light source or a continuous light source.

[0013] When a discrete light source is used, each ring of light source is formed by a series of discrete light sources arranged in a circle;

[0014] When a continuous light source is used, each circle of circular light source is formed by arranging the continuous light sources in a circle.

[0015] Preferably, the discrete light source includes an LED light source, a laser light source, a super luminescent diode light source (SLD) and the like.

[0016] The continuous light source includes a fiber light source, a tungsten filament light source, a light spot with an annular amplitude generated by phase modulation (for example: vortex phase, bottle-shaped phase modulation), etc.

[0017] Preferably, when constructing the annular illumination light source, the goal is to ensure that the corresponding spectrum overlap rate between the low-resolution images of two adjacent annular light sources is greater than 0, and the distance from each annular light source to the center of the circle is arbitrarily determined, and the highest resolution of the imaging is calculated based on the annular illumination light source with the determined distance.

[0018] Preferably, when constructing the circular ring lighting source, the required highest resolution is known, and with the goal of ensuring that the corresponding spectral overlap rate between the low-resolution images of two adjacent circles of circular ring light sources is greater than 0, the lateral wave vector that needs to be regulated for each circle of circular ring lighting is calculated, and then the distance from each circle of circular ring light source to the center of the circle is determined.

[0019] Resolution of frequency-shift super-resolution microscopy Calculation formula Determine the maximum transverse wave vector that can be provided by oblique incident light corresponding to the highest resolution , based on the fact that the corresponding spectrum overlap rate between the low-resolution images of two adjacent ring light sources is greater than 0 and the radius of each spectrum is , design the transverse wave vector of each ring lighting k i , and based on the formula Determining the angle of incidence for oblique lighting , then based on the formula Calculate the distance from the center of the circle to the lighting source of each circle ,in, λ is the wavelength of the illumination light, NA is the numerical aperture of the objective lens, is the wave vector of the incident light in vacuum, is the refractive index of the waveguide material, is the thickness of the waveguide.

[0020] Preferably, the super-resolution imaging model is constructed through deep learning based on a generative adversarial network with joint spatial Fourier channel attention, including:

[0021] Construct a generative adversarial network including a generator and a discriminator, wherein the generator is combined with a Fourier channel attention module to perform super-resolution imaging on a single wide-field image and multiple low-resolution images input to obtain a reconstructed super-resolution image, and the discriminator is used to distinguish the authenticity of a real super-resolution image and a reconstructed super-resolution image;

[0022] The generative adversarial network is trained adversarially, and the generator after adversarial training is used as a super-resolution imaging model.

[0023] Preferably, the generator adopts a convolutional neural network, and introduces multiple groups of residual groups connected in sequence between any two convolutional layers of the convolutional neural network, each residual group includes multiple Fourier channel attention modules connected in sequence, and a jump connection is constructed between the input of the first Fourier channel attention module and the output of the last Fourier channel attention module to form a residual structure;

[0024] Each Fourier channel attention module includes multiple convolutional layers, and a fast Fourier transform layer is introduced between any two convolutional layers. The input of the fast Fourier transform layer and the output of the last convolutional layer are channel-multiplied, and the channel product result is skip-connected with the input of the first convolutional layer and then output.

[0025] Preferably, the discriminator adopts a convolutional neural network.

[0026] Preferably, the super-resolution imaging model may also be a deep Fourier channel attention network, a U-net super-resolution model based on a spatial domain channel attention mechanism, or the like.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] First, a single wide-field image of the sample is captured when it is illuminated at normal incidence. Then, a circular illumination light source composed of multiple circular light sources with the same center is used to illuminate the sample at an angle to obtain multiple low-resolution images. The single wide-field image and the multiple low-resolution images are combined through a super-resolution imaging model built based on deep learning to achieve fast frequency-shift super-resolution microscopy imaging to obtain a super-resolution image. Compared with the existing method of sequentially lighting LED oblique-incident illumination / sequentially illuminating with different illumination stripes, the multi-circle circular illumination method can greatly reduce the number of multiple low-resolution images that need to be collected while keeping the amount of information unchanged, thereby improving the imaging speed, and is expected to be applied to video-level super-resolution microscopy. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 is a flow chart of a fast frequency shift super-resolution microscopic imaging method based on annular illumination provided in an embodiment;

[0031] Figure 2 is a schematic diagram of a structure of a generative adversarial network provided by an embodiment;

[0032] Figure 3 is a schematic diagram of an experimental device for lighting using a circular lighting source provided in an embodiment;

[0033] Figure 4 is a schematic diagram of calculating the oblique incident angle of each circle of optical fiber illuminating a sample in the experimental device provided in the embodiment;

[0034] Figure 5 is a schematic diagram of the spectrum corresponding to the circular lighting provided in the embodiment;

[0035] Figure 6 The embodiments provide a low-resolution image, a wide-field image, a reconstructed super-resolution image, and a real super-resolution image of each circle of circular illumination of a point-shaped sample;

[0036] Figure 7 The embodiments provide a low-resolution image, a wide-field image, a reconstructed super-resolution image, and a real super-resolution image of each circle of circular illumination of a linear sample;

[0037] Figure 8 The embodiments provide a low-resolution image, a wide-field image, a reconstructed super-resolution image and a real super-resolution image of each circle of circular illumination of the curved sample;

[0038] Fig. 9 The embodiments provide low-resolution images, wide-field images, reconstructed super-resolution images and real super-resolution images of each circle of circular illumination of a complex sample. DETAILED DESCRIPTION

[0039] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0040] The technical concept of the present invention is as follows: the present invention provides a fast frequency shift super-resolution microscopic imaging method based on annular illumination, which can greatly reduce the number of original low-resolution images required for reconstruction while keeping the amount of information unchanged, thereby greatly speeding up the imaging process, and is expected to be applied to dynamic imaging of living cells and living tissues. Compared with total internal reflection fluorescence microscopy (TIRF), TIRF can only have one annular illumination, so the resolution can only be doubled. The present invention can have multiple annular illuminations, and theoretically can achieve infinitely high resolution.

[0041] like Figure 1 As shown, the fast frequency shift super-resolution microscopic imaging method based on annular illumination provided in the embodiment comprises the following steps:

[0042] S1, construct a circular lighting source composed of multiple circular ring light sources with the same center.

[0043] In the embodiment, in the same circular lighting light source, each circle of circular light sources uses the same light source, including but not limited to discrete light sources such as LED light sources or continuous light sources such as optical fiber light sources. When LED light sources are used, each circle of circular light sources is formed by a series of discrete light sources arranged in a circle; when optical fiber light sources are used, each circle of circular light sources is formed by optical fiber light sources arranged in a circle.

[0044] In the embodiment, when constructing the circular lighting light source, it is also necessary to determine the distance from each circle of circular light source to the center of the circle, that is, to determine the position of each circle of circular light source, including two methods:

[0045] The first method is to first determine the position of each ring light source, and then determine the highest resolution of the imaging. Specifically, first, the distance from each ring light source to the center of the circle is arbitrarily determined with the goal of ensuring that the spectrum overlap rate between the low-resolution images of two adjacent ring light sources is greater than 0. Then, the highest resolution of the imaging is calculated based on the ring illumination light source determined by the distance, that is, the maximum oblique incidence angle is calculated based on the maximum distance. , according to the maximum oblique incidence angle Calculate the maximum transverse wave vector , according to the maximum transverse wave vector Calculate the highest resolution , where λ is the wavelength of the illumination light, NA is the numerical aperture of the objective lens, is the wave vector of the incident light in vacuum, is the refractive index, is the thickness of the waveguide.

[0046] The second method is to determine the position of each ring light source when the highest resolution is known. Specifically, , and with the goal of ensuring that the corresponding spectrum overlap rate between the low-resolution images of two adjacent ring light sources is greater than 0, the transverse wave vector that needs to be regulated for each ring lighting is calculated, and then the distance from each ring light source to the center of the circle is determined, specifically:

[0047] Resolution of frequency-shift super-resolution microscopy Calculation formula Determine the maximum transverse wave vector that can be provided by oblique incident light corresponding to the highest resolution , based on the fact that the corresponding spectrum overlap rate between the low-resolution images of two adjacent ring light sources is greater than 0 and the radius of each spectrum is , rationally design the transverse wave vector of each ring lighting k i , and based on the formula Determining the angle of incidence for oblique lighting , then based on the formula Calculate the distance from the center of the circle to the lighting source of each circle ,in, λ is the wavelength of the illumination light, NA is the numerical aperture of the objective lens, is the wave vector of the incident light in vacuum, is the refractive index of the waveguide material, is the thickness of the waveguide.

[0048] S2, collecting a single wide-field image when the sample is illuminated at normal incidence, using each circle of the annular light source to illuminate the sample obliquely and collecting a corresponding single frequency-mixed and frequency-shifted low-resolution image to obtain multiple low-resolution images.

[0049] In the embodiment, when collecting images, the sample is placed directly above the center of the circular illumination light source, and a single wide-field image is collected when other light sources illuminate the sample at normal incidence, and then each circle of circular light sources is sequentially lit to illuminate the sample obliquely. It should be noted that the light sources in the same circle should be lit together. A camera is used to sequentially collect a single mixed-frequency-shifted low-resolution image illuminated by each circle of circular light sources, thereby obtaining multiple low-resolution images.

[0050] S3, a single wide-field image and multiple low-resolution images are combined into a super-resolution microscopy image using a super-resolution imaging model built based on deep learning to obtain a super-resolution image.

[0051] In an embodiment, a super-resolution imaging model can be constructed through deep learning based on a generative adversarial network with joint spatial Fourier channel attention, including:

[0052] First, we build a generative adversarial network consisting of a generator and a discriminator, such as Figure 2 As shown, the generator is combined with a Fourier channel attention module to perform super-resolution imaging on a single wide-field image and multiple low-resolution images as input to obtain a reconstructed super-resolution image, and the discriminator is used to distinguish the authenticity of the real super-resolution image and the reconstructed super-resolution image; then the generative adversarial network is adversarially trained, and the generator after adversarial training is used as a super-resolution imaging model.

[0053] The generator adopts a convolutional neural network, and introduces multiple groups of residual groups connected in sequence between any two convolutional layers of the convolutional neural network. Each residual group includes multiple Fourier channel attention modules connected in sequence, and a jump connection is constructed between the input of the first Fourier channel attention module and the output of the last Fourier channel attention module to form a residual structure; wherein each Fourier channel attention module includes multiple convolutional layers, and a fast Fourier transform layer is introduced between any two convolutional layers, and the input of the fast Fourier transform layer and the output of the last convolutional layer are channel-multiplied, and the channel product result is jump-connected with the input of the first convolutional layer and then output. The specific structure is as follows Figure 2 shown.

[0054] The discriminator can use a convolutional neural network, and the specific results are as follows Figure 2 shown.

[0055] Of course, the super-resolution imaging model can also adopt the deep Fourier channel attention network, the U-net super-resolution model based on the spatial domain channel attention mechanism, etc.

[0056] After the super-resolution imaging model is obtained, the process of performing super-resolution imaging includes: inputting the obtained single wide-field image and multiple low-resolution images into the super-resolution imaging model to realize fast frequency-shifting super-resolution microscopic imaging to obtain a super-resolution image.

[0057] The above-mentioned fast frequency shifting super-resolution microscopy imaging method is now explained by taking a continuous light source as an example to form a circular illumination light source. Specifically, a 10x objective lens with an NA of 0.25 is selected, and K9 glass is used as a waveguide. Under a green light with a wavelength of 510nm, the refractive index of K9 glass is n is 1.53. Figure 3 As shown in the figure, four optical fibers are placed at the bottom of the waveguide and arranged into four circles of circular light sources. The center above the waveguide is used as the sample area. Figure 4 As shown, the thickness of the waveguide is H , the distance between each circle of optical fiber and the center is L i , i represents the index of each ring light source, and the oblique incident angle of each ring of optical fiber illumination sample is , the transverse wave vector provided by the oblique incident light is: ,in, is the wave vector of the incident light in vacuum. The fiber arrangement positions, the transverse wave vector provided by each ring illumination, and the corresponding spectrum overlap rate are shown in Table 1. The spectrum diagram is shown in Figure 5 , the solid line range is the spectrum corresponding to the first circle of circular lighting frequency shift, the dotted line range is the spectrum corresponding to the second circle of circular lighting frequency shift, the dotted line range is the spectrum corresponding to the third circle of circular lighting frequency shift, and the dotted line range is the spectrum corresponding to the fourth circle of circular lighting frequency shift.

[0058] Table 1 Specific parameters of circular lighting

[0059]

[0060] The specific imaging process is as follows: First, a beam of normally incident light is injected from directly below the waveguide to collect a wide-field image. Then, the light is coupled into the 1st, 2nd, 3rd, and 4th turns of optical fiber in sequence, that is, the 1st, 2nd, 3rd, and 4th turns of circular illumination are performed in sequence, and 4 low-resolution images of frequency mixing and frequency shifting are collected. Next, these 5 images are put into the super-resolution imaging model for reconstructing to obtain a super-resolution image. Specifically, super-resolution images can be reconstructed for point samples, line samples, curve samples, and complex samples. Low-resolution images, wide-field images, reconstructed super-resolution images, and real super-resolution images are shown in Figure 2. Figure 6-9 shown.

[0061] The maximum resolution of this example can reach 364 nm, which is about 5.6 times higher than the original resolution at 0.25NA. When achieving the same resolution, the imaging speed is increased by 10 times compared to FPM (it is expected to collect 50 original low-resolution images). In addition, under the light source of the same brightness, since the circular illumination is a circle of light sources that are lit at the same time, rather than a single direction, the exposure time required to collect a single low-resolution original image under the circular illumination is much shorter than the exposure time required for a single original low-resolution image of FPM. Therefore, considering all factors, the imaging speed of super-resolution microscopy based on circular illumination can be increased by at least 20 times compared to FPM. Compared with single-circle circular illumination microscopy techniques such as TIRF, the multi-circle circular illumination of the present invention can improve the resolution of the imaging system to a higher level.

[0062] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A fast frequency shift super-resolution microscopic imaging method based on annular illumination, characterized in that: The following steps are involved: Constructing a circular lighting source composed of multiple circular ring light sources with the same center; A single wide-field image is acquired when the sample is illuminated at normal incidence, and each ring light source is used to illuminate the sample obliquely and acquire a corresponding single frequency-mixed and frequency-shifted low-resolution image to obtain multiple low-resolution images; A single wide-field image and multiple low-resolution images are combined into a super-resolution microscopy image using a super-resolution imaging model built based on deep learning to obtain a super-resolution image.

2. The method for fast frequency shift super-resolution microscopy based on annular illumination according to claim 1, characterized in that: In the circular lighting source, each circle of the circular lighting source adopts the same light source, including a discrete light source or a continuous light source. When a discrete light source is used, each ring of circular light sources is formed by a series of discrete light sources arranged in a circle; When a continuous light source is used, each circle of circular light source is formed by arranging the continuous light sources in a circle.

3. The method for fast frequency shift super-resolution microscopy based on annular illumination according to claim 2, characterized in that: The discrete light source includes at least one of an LED light source, a laser light source, and a superluminescent diode light source.

4. The method for fast frequency shift super-resolution microscopy based on annular illumination according to claim 2, characterized in that: The continuous light source includes at least one of an optical fiber light source, a tungsten filament light source, and a light spot with a circular amplitude generated by phase regulation.

5. The method for fast frequency shift super-resolution microscopy based on annular illumination according to claim 1, characterized in that: When constructing a circular ring illumination source, the goal is to ensure that the corresponding spectrum overlap rate between the low-resolution images of two adjacent circular ring light sources is greater than 0, and the distance from each circular ring light source to the center of the circle is arbitrarily determined. The highest resolution of the imaging is calculated based on the circular illumination source with the determined distance.

6. The method for fast frequency shift super-resolution microscopic imaging based on annular illumination according to claim 1, characterized in that: When constructing a circular ring lighting source, the highest required resolution is known, and the goal is to ensure that the corresponding spectral overlap rate between the low-resolution images of two adjacent circular ring light sources is greater than 0. The transverse wave vector that needs to be regulated for each circle of circular lighting is calculated, and then the distance from each circle of circular light source to the center of the circle is determined.

7. The method for fast frequency shift super-resolution microscopic imaging based on annular illumination according to claim 6, characterized in that: Resolution of frequency-shift super-resolution microscopy Calculation formula Determine the maximum transverse wave vector that can be provided by oblique incident light corresponding to the highest resolution , based on the fact that the corresponding spectrum overlap rate between the low-resolution images of two adjacent ring light sources is greater than 0 and the radius of each spectrum is , design the transverse wave vector of each ring lighting k i , and based on the formula Determining the angle of incidence for oblique lighting , then based on the formula Calculate the distance from the center of the circle to the lighting source of each circle ,in, λ is the wavelength of the illumination light, NA is the numerical aperture of the objective lens, is the wave vector of the incident light in vacuum, is the refractive index of the waveguide material, is the thickness of the waveguide.

8. The method for fast frequency shift super-resolution microscopy based on annular illumination according to claim 1, characterized in that: The super-resolution imaging model is constructed through deep learning based on a generative adversarial network with joint spatial Fourier channel attention, including: Construct a generative adversarial network including a generator and a discriminator, wherein the generator is combined with a Fourier channel attention module to perform super-resolution imaging on a single wide-field image and multiple low-resolution images input to obtain a reconstructed super-resolution image, and the discriminator is used to distinguish the authenticity of a real super-resolution image and a reconstructed super-resolution image; The generative adversarial network is trained adversarially, and the generator after adversarial training is used as a super-resolution imaging model.

9. The method for fast frequency shift super-resolution microscopic imaging based on annular illumination according to claim 8, characterized in that: The generator adopts a convolutional neural network, and introduces multiple groups of residual groups connected in sequence between any two convolutional layers of the convolutional neural network, each residual group includes multiple Fourier channel attention modules connected in sequence, and a jump connection is constructed between the input of the first Fourier channel attention module and the output of the last Fourier channel attention module to form a residual structure; Each Fourier channel attention module includes multiple convolutional layers, and a fast Fourier transform layer is introduced between any two convolutional layers. The input of the fast Fourier transform layer and the output of the last convolutional layer are channel-multiplied, and the channel product result is skip-connected with the input of the first convolutional layer and then output.

10. The method for fast frequency shift super-resolution microscopic imaging based on annular illumination according to claim 8, characterized in that: The discriminator adopts a convolutional neural network.

11. The method for fast frequency shift super-resolution microscopy based on annular illumination according to claim 1, characterized in that: The super-resolution imaging model is also a deep Fourier channel attention network, or a U-net super-resolution model based on the spatial domain channel attention mechanism.

Images