Hexagonal lattice illumination super-resolution microscopy system and image reconstruction method

Abstract

The present application belongs to the field of microscope technology, and provides a hexagonal lattice illumination super-resolution microscopy system and image reconstruction method, which comprises: a digital micromirror device containing a plurality of micromirrors, the micromirrors being set as a hexagonal array, reflecting the illumination light to modulate the illumination light into a hexagonal structured light; and a central control unit configured to control the digital micromirror device to adjust the micromirror reflection angle in a specific time sequence to generate a plurality of phase shift images, and to reconstruct the image stack after the camera detection module completes image acquisition to output a super-resolution image. The advantage of the present application is that by using hexagonal structured light, it is not necessary to take pictures in three directions as in traditional super-resolution structured light illumination, which reduces the number of single-phase images required for one super-resolution image in principle, and performs parallel operation under GPU, thereby improving the time required for super-resolution image reconstruction, and making the reconstruction frame rate reach the standard required for real-time reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of microscopy, in particular to a hexagonal lattice illumination super-resolution microscopy system and image reconstruction method. BACKGROUND

[0002] In traditional wide-field fluorescence microscopy, unless the sample is very flat, all images will be affected by fluorescence emission from a certain error within the longitudinal depth of the focal plane of the objective lens. Structured illumination microscopy provides a better way to eliminate this unnecessary image effect by projecting a linear grid onto the sample. This method requires taking images of different phases at different positions of the linear grid, and using a matching reconstruction algorithm, a super-resolution image can be obtained from a certain number of single-phase images.

[0003] In recent years, most traditional methods use a modulation pattern of linear grid in a single direction for structured illumination. In the two-dimensional SIM super-resolution reconstruction algorithm proposed by Gustafsson, at least 9 original images are needed to input a super-resolution image, and the 9 images are divided into 3 groups, each group has three corresponding phase shift images in three illumination directions, and the super-resolution reconstruction method uses 3 linear equations to solve and shift frequency, and finally obtains the resolution improvement in each direction. The frame rate of the super-resolution image reconstruction is low due to the limitation of the system response speed or the processing speed of the reconstruction algorithm, and the reconstruction speed of the video frame is low, which is difficult to meet the requirement of real-time reconstruction. In order to improve the speed of the super-resolution reconstruction algorithm, some new methods can use as low as 4 original single-phase images to reconstruct, but when constructing the structured light, the robustness of the initial phase is low, and the phase value needs to be set with higher accuracy when the system is built. SUMMARY

[0004] The purpose of the present application is to provide a microscope technology based on DMD for structured light modulation and image stack super-resolution reconstruction, which can reconstruct a super-resolution image from only 7 single-phase images in the mode of hexagonal structured light, thereby solving the above problems.

[0005] In order to achieve the above purpose, the technical scheme adopted by the present application is:

[0006] A hexagonal lattice illumination super-resolution microscopy system, the system at least includes an illumination light path module and a camera detection module, the illumination light emitted by a multi-color laser arranged in the illumination light path generates fluorescence on a sample to be tested, which is collected by the camera detection module, comprising:

[0007] A digital micromirror device, wherein a plurality of micromirrors are arranged in a hexagonal array, and the micromirrors are configured to reflect illumination light to modulate the illumination light into a hexagonal structured light; and a central control unit configured to control the digital micromirror device to adjust the reflection angle of the micromirrors in a specific timing sequence to generate a plurality of phase-shifted images, and to reconstruct the image stack after the camera detection module acquires the images to output a super-resolution image.

[0008] Further, a dichroic mirror is arranged between the camera detection module and the digital micromirror device, and is configured to reflect the emitted light from the digital micromirror device to the sample to be measured to excite fluorescence, and the excited fluorescence is transmitted by the dichroic mirror to the camera detection module for image acquisition.

[0009] Further, a tube lens and a first filter are arranged in sequence along the light transmission path between the digital micromirror device and the dichroic mirror, the structured light generated by the digital micromirror device is transmitted to the dichroic mirror after passing through the tube lens and the first filter to filter out stray light.

[0010] Further, a second filter and a converging lens are arranged in sequence along the light transmission path between the dichroic mirror and the camera detection module, and the fluorescence transmitted by the dichroic mirror passes through the second filter to filter out light of wavelengths other than the excitation light, and is focused by the converging lens to the camera detection module for collection.

[0011] Further, the illumination light path module further comprises, in sequence along the light source transmission path, an objective lens, a pentagonal prism, a double-cemented lens, and a mirror; the objective lens expands the illumination light excited by the multi-color laser, the pentagonal prism folds the space size, the double-cemented lens converges the parallel light beams emitted after passing through the objective lens, and the mirror adjusts the angle of the light incident on the digital micromirror device, so that the excited illumination light is incident on the hexagonal array in the digital micromirror device.

[0012] Further, the wavelength of the illumination light emitted by the multi-color laser is adjusted by the central control unit.

[0013] The present application also provides a reconstruction method of output images of a hexagonal lattice illumination super-resolution microscopy system, comprising the steps of:

[0014] S1, a multi-color laser controlled by a central control unit emits illumination light of a predetermined wavelength along a light transmission path to the reflection surface of a digital micromirror device, and after reflection modulation by a hexagonal array of the digital micromirror device, forms a hexagonal lattice of illumination light on a sample to be measured;

[0015] S2, the central control unit controls the tilting direction of the hexagonal array on the digital micro-mirror device on the CPU platform, so that at least 7 original image stacks of the hexagonal lattice projection modulation monophase image collected by the camera detection module in a certain timing sequence are obtained, and the original image stacks are transmitted from the CPU platform to the GPU;

[0016] S3, the central control unit re-samples the obtained original image stacks in the GPU, and processes the original image stacks by a pre-filter generated by combining an optical transfer function and an attenuation coefficient of an out-of-focus signal;

[0017] S4, the initial parameters of the lattice structure light modulation image are estimated from the original image information, so that the spatial frequency vector, the spatial phase size and the modulation amplitude size are determined by using a cross-correlation algorithm;

[0018] S5, the reconstruction factor is calculated in the GPU using parallel operation, and the reconstruction factor and the pre-filter are used together on the original image stacks processed by the pre-filter to perform frequency shift and reconstruction processing on the image;

[0019] S6, the reconstruction factor is substituted into the Wiener filter constructed by the optical transfer function, the attenuation coefficient and the pre-filter to post-filter the reconstructed image;

[0020] S7, the data information in the GPU is transmitted back to the CPU platform, the super-resolution image data information is arranged, and the reconstructed super-resolution image is output.

[0021] Compared with the prior art, the present application has at least the following beneficial effects:

[0022] (1) By using hexagonal structured light, it is not necessary to take pictures in three directions as in the traditional super-resolution structured light illumination, the hexagonal structured light itself can uniformly modulate the sample in three directions of the spatial plane, thereby reducing the number of monophase images required for one super-resolution image from the principle, improving the speed of the super-resolution image reconstruction algorithm, and greatly reducing the number of original image shooting times required for the super-resolution image. At the same time, the three-direction illumination mode is not required, which further reduces the complexity of the system;

[0023] (2) By linearly converting the 7 monophase images in hex-SIM to 9 images in the conventional demodulation algorithm, the hexagonal lattice algorithm is processed in parallel in the GPU framework, so that the new positions of them in the Fourier space are transferred, the resolution of the system is increased, and the time required for super-resolution image reconstruction is greatly reduced from the program and algorithm, so that the reconstruction frame rate reaches the standard required for real-time reconstruction;

[0024] (3) By using DMD as a spatial modulator to modulate the light incident on the DMD surface to generate structured light, the high frame rate is realized to realize high-speed capture of images, which is beneficial to record super-resolution 3D data set, greatly reduces the switching time between adjacent two single-phase images, and reduces the artifact effect generated by the sample to be tested in the shooting process;

[0025] (4) By independently controlling the binary of the millions of micromirror array on the DMD, the opening number of the DMD micromirror pixel unit with reflection modulation function can be set on the micron level as needed, which not only can enhance the accuracy of the structured light, but also can freely adjust the period size of the structured light, enhance the flexibility of the sample object modulation, and realize the adjustable contrast of the sample modulation;

[0026] (5) By the opening and closing state of the DMD specific period pixel, the required light in the incident light is reflected, and other light is efficiently filtered out, avoiding the mechanical disturbance caused by the traditional use of mechanical motor to adjust the mask, and improving the imaging accuracy; BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a structural schematic diagram of a hexagonal lattice illumination super-resolution microscopy system in the embodiment of the present application;

[0028] Figure 2 is a hexagonal structured light construction schematic diagram in the embodiment of the present application;

[0029] Figure 3 is a principle diagram of DMD modulation to form structured light illumination in the embodiment of the present application;

[0030] Figure 4 is a flow chart of hexagonal lattice illumination reconstruction to generate super-resolution image in the embodiment of the present application;

[0031] In the figure, 1 is a multi-color laser, 2 is an optical fiber, 3 is an objective lens, 4 is a five-corner prism, 5 is a double-cemented lens, 6 is a mirror, 7 is a digital micromirror device, 8 is a tube lens, 9 is a first filter, 10 is a dichroic mirror, 11 is a high-power objective lens, 12 is a sample, 13 is a second filter, 14 is a converging lens; 15 is a camera detection module, 16 is a central control unit. DETAILED DESCRIPTION

[0032] It should be noted that all directionality indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present application are only used to explain the relative position relationship, motion condition, etc. between components in a certain specific posture (as shown in the drawings), if the specific posture changes, the directionality indications also change accordingly.

[0033] In addition, the descriptions such as "first", "second", "one", etc. in the present application are only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can be explicitly or implicitly included at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified and limited.

[0034] In the present application, unless otherwise explicitly specified and limited, the terms "connection", "fixing", etc. should be understood broadly, for example, "fixing" can be fixed connection, or detachable connection, or integral; can be mechanical connection, or electrical connection; can be direct connection, or indirect connection through intermediate medium, can be internal communication of two elements or interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0035] In addition, the technical solutions of various embodiments of the present application can be combined with each other, but it must be based on the fact that a person skilled in the art can realize it, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the scope of protection required by the present application.

[0036] The following are specific embodiments of the present application, and the technical solutions of the present application are further described in conjunction with the accompanying drawings, but the present application is not limited to these embodiments.

[0037] As shown in Figure 1 The present application provides a hexagonal lattice illumination super-resolution microscopy system, which includes an illumination light path module composed of a multi-color laser 1, an optical fiber 2, an objective lens 3, a five-corner prism 4, a double-cemented lens 5, and a mirror 6.

[0038] The optical fiber 2 is connected to the outlet of the multi-color laser 1, and the objective lens 3 is located behind the optical fiber 2, and the end face of the optical fiber 2 is located at the front focal plane of the objective lens 3. The multi-color laser 1 emits illumination light which is expanded by the objective lens 3 in turn, the expanded light path is folded by the five-corner prism 4, and the folded light path is focused by the double-cemented lens 5, and then deflected by the mirror 6 to be collected on the receiving surface of the digital micromirror device (DMD) 7.

[0039] The illumination light is modulated by the digital micromirror device 7 to generate structured light, which enters the microscope frame through the tube lens 8, is filtered by the first filter 9 to remove stray light, is reflected by the dichroic mirror 10, and is projected on the sample 12 to be measured through the high-power objective 11. The structured light excites the sample 12 coated with fluorescent dye to generate excitation light, which returns along the sample 12 and the high-power objective 11, is transmitted by the dichroic mirror 10, and is filtered by the second filter 13 to remove light of wavelengths other than the excitation light. Finally, the light is focused by the converging lens 14 on the sensor of the camera detection module 15.

[0040] The central control unit 16 can control the multi-color laser 1 to emit illumination light of different wavelengths. Before the illumination light is modulated by the digital micromirror device 7, the digital micromirror device 7 is loaded with different modulation images required for generating structured light with different phase shifts according to a specific timing sequence by the central control unit 16.

[0041] The image generated by the excitation light is recorded by the CCD camera in the camera detection module 15, and the exposure timing of the CCD camera in the camera detection module 15 is also controlled by the central control unit 16.

[0042] The digital micromirror device 7 has millions of micromirror units, each of which can be independently controlled by the central control unit 16. The size of the micromirror unit is on the order of microns, and high-precision micromirror units can construct a hexagonal lattice pattern with higher resolution.

[0043] As shown in Figure 2 , P is the period length of the hexagonal unit in the structured light, which is related to the period rule when converted to the frequency spectrum. The higher the precision of the digital micromirror device 7, the greater the space for adjusting the hexagonal spacing P in the figure, and the easier the construction of the hexagonal lattice.

[0044] As shown in Figure 3 , the micromirror units in the digital micromirror device 7 can be adjusted to two states of "0" and "1". The "0" state represents the "off" state, and the "1" state represents the "on" state. The binary input of the central control unit 16 can determine the state of each micromirror reflecting surface in the digital micromirror device 7.

[0045] In the "on" state, the square micromirror unit will be deflected by +12 degrees around the 45-degree diagonal symmetry axis, and the reflected light will be received by the subsequent system; in the "off" state, the square micromirror unit will be deflected by -12 degrees around the 45-degree diagonal symmetry axis, and the reflected light will overflow to the outside of the system as stray light.

[0046] In a specific timing sequence, different modulation images loaded on the digital micromirror device 7 are as shown in Figure 4As shown in the bottom, the moving direction and size of each image are the same, and after 7 movements, the 8th modulation image is the same as the 1st modulation image. At different T values, the central control unit 16 controls the array binarization according to the input image, so that the reflected light is received as different phase shift modulation effects, thereby generating a specific structured light. The central control unit controls the modulation image timing of the digital micromirror device 7 and the timing of the CCD camera shooting single phase image, and generates 7 images of continuous phase shift, which is a period, thereby outputting the super-resolution image through the reconstruction algorithm in the central control unit.

[0047] According to the above embodiment of the present application, the present application also provides a reconstruction method of output image of a hexagonal lattice illumination super-resolution microscopic system, as shown in Figure 3 The method comprises the steps of:

[0048] S1, a multi-color laser controlled by a central control unit emits a preset wavelength illumination light along an optical transmission path to the reflection surface of a digital micromirror device, and after reflection modulation by the hexagonal array of the digital micromirror device, a hexagonal lattice illumination light is projected on a sample to be measured;

[0049] S2, the central control unit controls the tilting direction of the hexagonal array on the digital micromirror device on the CPU platform, thereby obtaining at least 7 original image stacks of the hexagonal lattice projection modulation single phase image collected by the camera detection module in a specific timing, and transmitting the original image stack from the CPU platform to the GPU;

[0050] The size of the periodic structure of the hexagonal lattice image is determined by the parameter of the depth of the structured light finally projected on the sample surface. The periodic structure of the hexagonal lattice illumination field satisfies the following equation:

[0051]

[0052] Wherein, x and y represent the coordinate positions of the two-dimensional space plane, and p is the periodic characteristic of the hexagonal periodic structure (the periodic distance of the hexagonal unit).

[0053] After simplification, it can be arranged in proportion to the exponential form:

[0054]

[0055] The projection of the hexagonal structured light on the image can be determined by the following expression:

[0056]

[0057] There are seven unknown quantities in the above formula, which are I0, I s1 , I s2 , I s3、 and Therefore, at least 7 single-phase images are needed to obtain a super-resolution image to solve all unknown parameters.

[0058] The central control unit controls the multi-color laser to open and close according to a specific time sequence on the CPU platform, controls the digital micro-mirror device to load the image required for hexagonal lattice illumination modulation according to a specific time sequence, and controls the digital micro-mirror device to output a signal to the CCD camera according to a specific time sequence, and controls the CCD camera to perform image acquisition operation under the time sequence output by the digital micro-mirror device.

[0059] In order to ensure that the system has a relatively long image exposure time when continuously imaging, the central control unit adopts a synchronous readout trigger mode, and the formula for calculating the frame rate is as follows:

[0060]

[0061] Where M is the frame rate, unit fps, m is the number of rows set by the camera, and H is the length of the camera system error delay.

[0062] Then record at least 7 original image stacks of hexagonal lattice projection modulation single-phase images in order, and transfer the image stacks from the CPU platform to the GPU to prepare for subsequent parallel operation.

[0063] S3, the central control unit resamples the obtained original image stack in the GPU, and processes the original image stack through a pre-filter generated by combining the optical transfer function and the attenuation coefficient of the out-of-focus signal suppression;

[0064] In the CUDA (Compute Unified Device Architecture) library in the GPU framework, the input original image stack is resampled, and the image data unit is stored one by one with the GPU storage unit. The control trigger part of the hardware is completed in the CPU platform, and the pure operation part required in the image reconstruction in the GPU, and the pre-filter generated by combining the system optical transfer function and the attenuation coefficient of the out-of-focus signal suppression in the GPU acts on the original image stack.

[0065] S4, estimate the initial parameters of the lattice structure light modulation image from the original image information, so as to calculate and determine the spatial frequency vector, the spatial phase size and the modulation amplitude size by using the cross-correlation algorithm;

[0066] The initial intensity information of the image stack is:

[0067]

[0068] Estimating the system initial parameters of the lattice structure light modulation image from the initial intensity information of the image stack, so as to obtain the spatial frequency vector The spatial phase size And the modulation amplitude size m The spatial phase size In the estimation, the autocorrelation reconstruction method is used for parallel operation, and the relationship is:

[0069]

[0070] Where a is the intensity coefficient of the component.

[0071] S5, in the GPU, using the parallel operation mode, according to the obtained spatial frequency vector The spatial phase size And the modulation amplitude size m , the parameters are substituted into the equation of the four frequency bands:

[0072]

[0073] The reconstruction factor of the system is obtained:

[0074]

[0075] The reconstruction factor and the pre-filter jointly act on the original image stack processed by the pre-filter, and the image is frequency shifted and reconstructed.

[0076] S6, the reconstruction factor is substituted into the Wiener filter constructed by the optical transfer function, the attenuation coefficient and the pre-filter, and the reconstructed image is post-filtered, which is used to eliminate the artifacts caused by sample movement;

[0077] S7, the data information in the GPU is transmitted back to the CPU platform, the super-resolution image data information is arranged, and the reconstructed super-resolution image is output.

[0078] The present application converts 7 images into 9 images of conventional demodulation algorithm by linear conversion, parallel processes the hexagonal lattice demodulation reconstruction algorithm in the GPU, so as to transfer them to the new position in the Fourier space, increase the resolution of the system, and through experiment and simulation, it is proved that when the hexagonal lattice illumination is used to reconstruct 7 original images, the time required for image processing can be greatly reduced in the reconstruction process, the image reconstruction frame rate is improved, and the reconstruction frame rate reaches the standard required by real-time reconstruction.

[0079] The specific embodiments described herein are merely illustrative of the spirit of the application. Various modifications or changes in the specific embodiments described herein can occur to those skilled in the art to which the application pertains without departing from the spirit of the application, and it is understood that such modifications or changes are to be considered as within the scope of the application as defined by the appended claims.

Claims

1. A hexagonal lattice illumination super-resolution microscopy system, the system comprising at least an illumination light path module and a camera detection module, wherein the illumination light emitted by a polychromatic laser arranged in the illumination light path generates fluorescence on a sample to be measured and is collected by the camera detection module, characterized in that, The application relates to a method for outputting a super-resolution image, comprising the following steps: S1. Emitting preset wavelength illumination light from a polychromatic laser controlled by a central control unit along an optical transmission path to a reflecting surface of a digital micromirror device, and projecting the illumination light in a hexagonal lattice on a sample to be detected after reflection modulation of the hexagonal array of the digital micromirror device; S2. Controlling the tilting direction of the hexagonal array on the digital micromirror device on a CPU platform by the central control unit, thereby obtaining at least seven original image stacks of the hexagonal lattice projection modulation single-phase images collected by a camera detection module in a specific time sequence, and transmitting the original image stacks from the CPU platform to a GPU; S3. Resampling the obtained original image stacks in the GPU by a pre-filter generated by combining an optical transfer function and an attenuation coefficient of an out-of-focus signal; S4. Estimating initial parameters of the lattice structure light modulation image from the original image information, thereby calculating and determining a spatial frequency vector, a spatial phase size and a modulation amplitude size by using a cross-correlation algorithm; S5. Calculating a reconstruction factor in the GPU by using a parallel operation mode, and jointly acting the reconstruction factor and the pre-filter on the original image stacks processed by the pre-filter to perform frequency shift and reconstruction processing on the image; S6. Substituting the reconstruction factor into a Wiener filter constructed by the optical transfer function, the attenuation coefficient and the pre-filter to perform post-filtering on the reconstructed image; S7. Transmitting the data information in the GPU back to the CPU platform, arranging the super-resolution image data information, and thereby outputting the reconstructed super-resolution image. A dichroic mirror is arranged between the camera detection module and the digital micromirror device, and is configured to reflect the emitted light in the digital micromirror device to the sample to be detected to excite fluorescence, and the excited fluorescence is transmitted to the camera detection module to collect images. A tube mirror and a first filter are arranged in sequence along the optical transmission path between the digital micromirror device and the dichroic mirror, the tube mirror enters a microscope frame, and the first filter filters stray light, and the structure light generated by the digital micromirror device is propagated to the dichroic mirror. A second filter and a converging lens are arranged in sequence along the optical transmission path between the dichroic mirror and the camera detection module, the fluorescence transmitted through the dichroic mirror filters light of wavelengths other than the excitation light through the second filter, and the converging lens focuses the fluorescence on the camera detection module for collection.

2. A hexagonal-lattice illuminated super-resolution microscopy system according to claim 1, wherein, ​ 3. A hexagonal-lattice illuminated super-resolution microscopy system according to claim 2, wherein, ​ 4. The hexagonal-lattice illuminated super-resolution microscopy system of claim 2, wherein, ​ 5. The hexagonal-lattice illuminated super-resolution microscopy system of claim 1, wherein, The illumination light path module further comprises an objective lens, a five-cornered prism, a double-cemented lens and a mirror arranged in sequence according to the front and rear order of the light source transmission path in addition to the multi-color laser; the illumination light excited by the multi-color laser is expanded by the objective lens, the parallel light beams emitted after the objective lens are folded in size by the five-cornered prism and converged at the double-cemented lens, the light angle of incidence into the digital micro-mirror device is adjusted by the mirror, so that the excited illumination light is incident into the hexagonal array in the digital micro-mirror device.

6. The hexagonal-lattice illuminated super-resolution microscopy system of claim 1, wherein, The wavelength of the illumination light emitted by the multi-color laser is adjusted by the central control unit.

7. A method of reconstruction of an output image of a hexagonal lattice illuminated super-resolution microscopy system, characterized in that, The method comprises the steps of: S1, the multi-color laser controlled by the central control unit emits preset wavelength illumination light along the light transmission path and is incident onto the reflection surface of the digital micro-mirror device, and after reflection modulation by the hexagonal array of the digital micro-mirror device, the illumination light forms a hexagonal lattice on the sample to be measured; S2, the central control unit controls the tilt direction of the hexagonal array on the digital micro-mirror device on the CPU platform, so as to obtain at least 7 original image stacks of the hexagonal lattice projection modulation monophase image collected by the camera detection module in a specific time sequence, and transmit the original image stacks from the CPU platform into the GPU; S3, the central control unit re-samples the obtained original image stack in the GPU, processes the original image stack by a pre-filter generated by combining the optical transfer function and the attenuation coefficient of the out-of-focus signal; S4, estimate the initial parameters of the lattice structure light modulation image from the original image information, so as to calculate and determine the spatial frequency vector, the spatial phase size and the modulation amplitude size by using the cross-correlation algorithm; S5, calculate the reconstruction factor in the GPU using the parallel operation mode, and apply the reconstruction factor and the pre-filter to the original image stack processed by the pre-filter, and perform frequency shift and reconstruction processing on the image; S6, substitute the reconstruction factor into the Wiener filter constructed by the optical transfer function, the attenuation coefficient and the pre-filter to perform post-filtering on the reconstructed image; S7, transmit the data information in the GPU back to the CPU platform, arrange the super-resolution image data information, and output the reconstructed super-resolution image.

Images