A three-dimensional high-precision real-time drift correction method and device

Abstract

The present invention discloses a three-dimensional high-precision real-time drift correction method and device. The method of the present invention first uses to obtain an initial image and establish an axial stack of images; secondly, the axial stack is analyzed to determine the initial position of the image; then the image is collected in real time, and the relative displacement between images is calculated using a cross-correlation algorithm; finally, according to the relative displacement, the system is fine-tuned to achieve sub-nanometer precision positioning and drift correction. The device of the present invention includes two near-infrared laser light sources for illuminating samples and markers; a high frame rate camera for capturing images; a piezoelectric nano-displacement stage for fine-tuning the position of the sample or optical system; a processor, configured with a linear fitting algorithm and a cross-correlation algorithm, for calculating the relative displacement between images and controlling the piezoelectric nano-displacement stage for fine-tuning. The present invention realizes real-time monitoring and correction of the sample position to compensate for sample drift that may occur during long-term imaging.

Description

Technical Field

[0001] The present invention belongs to the technical field of super-resolution, and relates to a drift correction method and device for performing three-dimensional high-precision real-time correction of a sample position in sub-ten-nanometer super-resolution imaging. Background Art

[0002] All microscopes are subject to factors such as temperature or mechanical instabilities that can cause undesired deviations in the setup. For microscopy, the most critical is the displacement of the sample relative to the microscope objective, commonly referred to as sample drift. Well-made microscopes typically experience sample drift of a few nanometers per minute. For most microscopy systems with resolution at the diffraction-limited size, as well as for fast super-resolution microscopy methods such as stimulated emission depletion microscopy (STED), this level of drift is negligible; but for single-molecule localization microscopy (SMLM) and the recently proposed minimum light flux microscopy (MINFLUX), as well as STED-based single-molecule localization technology (MINSTED), which aim to achieve resolutions below 10 nanometers, sample drift can greatly affect the image quality and imaging results.

[0003] There are basically two approaches to correct sample drift: post-processing correction and active correction during the measurement. Post-processing drift correction, which is often used in SMLM, consists in determining sample motion by correlation between subsequent images. This correction can only compensate for long-term drift, as multiple frames must be used to obtain partial images of sufficient quality to allow for a reasonable correlation. More precise corrections can be obtained by introducing markers (usually micro- or nanoparticles) into the sample, which are then used to correct sample position during post-acquisition analysis, either by scattering, photoluminescence, fluorescence detection or transmission detection. In general, the more markers used, the more precise the correction. Advanced post-processing drift correction methods using DNA origami as markers and particle averaging algorithms have enabled resolutions below 5 nm in DNA-PAINT, but this approach requires hundreds of DNA origami per field of view in addition to the biological structure. Post-processing drift correction is limited to in-plane lateral drift (XY) and should be supplemented by active correction of the axial position (Z). In contrast, the ability to achieve sub-10 nm axial resolution with modulated excitation localization microscopy (ModLoc) or optical selective exposure-based localization microscopy (ROSE) depends critically on the stability of the microscope and usually requires active (XYZ) drift correction. Summary of the invention

[0004] The present invention proposes an innovative three-dimensional sub-nanometer precision drift correction method, which precisely controls the movement of the piezoelectric nano-platform in the X, Y, and Z directions, utilizes advanced cross-correlation algorithms and fast Fourier transform technology, monitors and analyzes tiny changes in image sequences in real time, and adopts efficient computational models and feedback mechanisms to calculate the instantaneous position deviation of the platform; until the precise correction of the platform position is achieved, ensuring the maintenance of sub-nanometer stability and repeatability during long-term operation, thereby significantly improving the performance and reliability of the precision positioning system.

[0005] In a first aspect, the present application provides a three-dimensional high-precision real-time drift correction method, the method comprising the following steps:

[0006] S1. Use the camera to obtain the initial image;

[0007] S2. Create an axial stack of initial images;

[0008] S3. The axial stack was analyzed using a linear fitting algorithm to determine the initial position of the image;

[0009] S4. collect each subsequent image in real time and calculate the relative displacement between images using a cross-correlation algorithm;

[0010] S5. Based on the relative displacement, the sample or the optical system is fine-tuned by a piezoelectric nano-translation stage to achieve sub-nanometer precision positioning and drift correction.

[0011] In a second aspect, the present application provides an electronic device, comprising: a memory on which a program code is stored; and a processor connected to the memory, and when the program code is executed by the processor, the above method is implemented.

[0012] In a third aspect, the present application provides a computer-readable storage medium having program instructions stored thereon, wherein the program instructions implement the above method when executed.

[0013] In a fourth aspect, the present application provides a computer program product, comprising a computer program / instructions, which implement the above method when executed by a processor.

[0014] Compared with the prior art, the present invention has the following technical advantages:

[0015] 1) Sub-nanometer drift correction capability: This application uses advanced cross-correlation algorithms and fast Fourier transform (FFT) technology to achieve sub-nanometer drift correction in three-dimensional space. This accuracy significantly improves the stability and reliability of precision instruments and equipment, especially in demanding scientific and industrial applications such as biomedical imaging and precision engineering. By reducing or eliminating drift caused by factors such as environmental changes, this method ensures continued stability and data accuracy during long-term operation.

[0016] 2) Real-time performance and efficient processing: This application uses real-time data processing and fast algorithm execution to enable instant response and correction of drift in high-speed imaging and dynamic monitoring applications. By utilizing graphics processing units (GPUs) to accelerate computing, this method significantly shortens data processing time and improves correction efficiency, making it possible to achieve sub-nanometer real-time drift correction in data acquisition at video frame rates or even higher frame rates.

[0017] 3) System compatibility and flexibility: This application is designed to be compatible with existing equipment and can be easily integrated into a variety of different piezoelectric nanostages and imaging systems without complex hardware modifications. Its flexibility and modular design allow users to adjust parameters and settings according to specific application requirements, such as correction accuracy, response speed, etc., to achieve optimal performance in different operating environments and conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A diagram of a specific implementation device for three-dimensional high-precision real-time drift correction;

[0019] Figure 2 The imaging results and selection diagram of the markers in this embodiment;

[0020] Figure 3 This is a graph of the long-term standard deviation data of the XYZ three axes obtained in this embodiment. DETAILED DESCRIPTION

[0021] The present invention will be described in detail below with reference to the embodiments and the accompanying drawings, but the present invention is not limited thereto.

[0022] A three-dimensional high-precision real-time drift correction method according to an embodiment of the present application includes the following steps:

[0023] S1. Use the camera to obtain the initial image;

[0024] S2. Create an axial stack (Z-stack) of images;

[0025] S3. Analyze the Z-stack using a linear fitting algorithm to determine the initial position of the image;

[0026] S4. Each subsequent image is acquired in real time, and the relative displacement between images is calculated using fast Fourier transform (FFT) and cross-correlation algorithm;

[0027] S5. Based on the calculated displacement, the sample or optical system is fine-tuned by a piezoelectric nanostage to achieve sub-nanometer precision positioning and drift correction.

[0028] Furthermore, in step S3, the linear fitting algorithm is a least squares fitting, which is used to determine the position of each image in the Z-stack.

[0029] Further, in step S4, the cross-correlation algorithm comprises the following steps:

[0030] S101. Performing FFT on the reference image and the image to be corrected;

[0031] Fast Fourier Transform is an efficient algorithm for calculating Discrete Fourier Transform (DFT). In image processing, the image is converted from the spatial domain to the frequency domain. Through FFT, the pixel information in the image can be converted into frequency components. For example, for a grayscale image, its pixel values ​​are distributed in two dimensions in the spatial domain. After FFT, it becomes the amplitude and phase information of different frequency components. The purpose of this is to make it easier to calculate the correlation between images in the frequency domain, because frequency domain operations are more intuitive and efficient when dealing with problems such as image displacement.

[0032] S102. Calculating the conjugate product of the Fourier transforms of the two images;

[0033] In the frequency domain, the Fourier transform result of the reference image is conjugate multiplied with the Fourier transform result of the image to be corrected. The conjugate product is a complex number operation. For two complex numbers a + bi and c + di, their conjugate product is (a + bi) (c - di) = ac + bd + (bc - ad)i. In image processing, this operation can be regarded as fusing the information of the two images in the frequency domain. Through the conjugate product, the similar parts of the two images in the frequency domain can be highlighted to prepare for the subsequent finding of the cross-correlation peak. For example, if the two images have similar amplitude and phase on a certain frequency component, then after the conjugate product, the amplitude of this frequency component will be larger, making it easier to be identified in the subsequent steps.

[0034] S103. Find the cross-correlation peak value through inverse Fourier transform (iFFT) to determine the relative displacement between the images;

[0035] The inverse Fourier transform converts the information in the frequency domain back to the spatial domain. Performing iFFT on the result after the conjugate product can obtain a cross-correlation image. In this cross-correlation image, there will be an obvious peak. The position of this peak corresponds to the relative displacement between the reference image and the image to be corrected. For example, if the peak appears at the (x, y) position, it means that the image to be corrected has moved x pixels horizontally and y pixels vertically relative to the reference image. This is because the essence of the cross-correlation algorithm is to find the most similar part between the two images. When the two images are aligned in space, their correlation is the strongest, thus forming a peak in the cross-correlation image.

[0036] S104. Calculate the row and column displacements between images based on the peak positions to achieve sub-nanometer displacement estimation.

[0037] According to the precise position of the peak in the cross-correlation image, the row (vertical direction) and column (horizontal direction) displacement between images can be calculated. Due to the high precision of FFT and iFFT, as well as the development of modern image processing technology, this displacement estimation can achieve sub-nanometer accuracy. For example, in some high-precision optical measurements or microscopic image analysis, the cross-correlation algorithm can accurately measure the displacement changes of objects at the microscopic scale. This is of great significance for studying the microstructural changes of materials and the movement of biological cells.

[0038] Furthermore, in step S5, the drift correction process is performed in real time and can be synchronized with the imaging process.

[0039] Furthermore, the method also includes a system for storing and recording drift correction data to facilitate historical data analysis and performance monitoring.

[0040] Furthermore, the method also includes a user interface for displaying the correction status and adjusting the correction parameters in real time.

[0041] Based on the same concept as the above method, the present application also provides an electronic device, comprising: a memory on which program code is stored; a processor, which is connected to the memory and implements the above method when the program code is executed by the processor.

[0042] Based on the same concept as the above method, the present application also provides a computer-readable storage medium on which program instructions are stored, and the above method is implemented when the program instructions are executed.

[0043] Based on the same concept as the above method, the present application also provides a computer program product, including a computer program / instruction, which implements the above method when executed by a processor.

[0044] The present application also provides a real-time drift correction microscopic imaging device constructed based on the above method, such as Figure 1 As shown, the device includes: a first laser light source 1, a first reflector 2, a first polarization beam splitter 3, a dichroic mirror 4, a quarter wave plate 5, a field lens 6, a second reflector 7, an objective lens 8, a sample 9, a first filter 10, a high frame rate camera 11, a second laser light source 12, a second polarization beam splitter 13, a relay lens 14, a filter 15, an imaging lens 16, a second filter 17 and a high frame rate camera 18; in addition, it also includes a piezoelectric nano-displacement stage 19, a first laser fiber 20, a second laser fiber 21, a first collimator 22 and a second collimator 23.

[0045] Among them, the first laser light source 1 and the second laser light source 12 are near-infrared lasers of different wavelengths, the first filter 10 and the second filter 17 are filters with different central bandwidths, the second reflector 7 can be a dichroic mirror, which is responsible for reflecting infrared laser signals and transmitting laser excitation and fluorescence collection for super-resolution imaging; the objective lens 8 is an objective lens with a high numerical aperture, the upper layer of the sample 9 is a coverslip, and the lower layer is biological tissue or cells; the entire sample is fixed to the piezoelectric nano-displacement stage 19 by mechanical support.

[0046] The drift correction idea of ​​the device in the XYZ direction is as follows: through off-axis incidence, the signal returned by total internal reflection (TIR) ​​illumination is collected, and the image changes at the interface between the coverslip and the biological tissue are analyzed through the centroid fitting algorithm to achieve real-time drift correction on the Z axis; through central incidence, the central light is blocked, the scattered signal is collected, and the shape changes of the gold nanoparticle marker are analyzed through the cross-correlation algorithm to achieve real-time drift correction on the XY axis.

[0047] Furthermore, real-time drift correction of the Z axis is achieved by:

[0048] The first laser light source 1 is output through the first laser fiber 20 and collimated by the first collimator 22. The optimal beam diameter is 2-5 mm. The specific size is based on the sensor size of the high frame rate camera 11. Then, it is reflected by the first reflector 2 and the first polarization beam splitter 3, and transmitted by the dichroic mirror 4. At this time, it is vertical linear polarized light, which is adjusted to circular polarized light by the quarter wave plate 5, converged by the field lens 6, and focused on the edge of the rear focal plane of the objective lens 8. The second reflector 7 is responsible for folding the converged beam.

[0049] After entering the objective lens, the light beam presents a large-angle collimated light illumination on the sample surface. Since there is objective lens oil between the objective lens and the sample, and its refractive index is similar to that of the cover glass, the light beam will not have much reflection when passing through the interface between the objective lens oil and the cover glass; when the objective lens passes through the cover glass and enters the interface of the biological tissue, since it enters the light-sparse medium from the light-dense medium, and the angle is greater than the critical angle, total reflection occurs, and most of the light beam is reflected back to the objective lens, returning from the other edge of the rear focal plane of the objective lens, passing through the field lens 6, the quarter-wave plate 5, the dichroic mirror 4, the first polarization beam splitter 3, and then passing through the first filter 10, and is collected by the high frame rate camera 11. Since the infrared light beam passes through the quarter-wave plate 5 twice, the polarization direction of the returning light beam is rotated by 90°, so it is adjusted to horizontal linear polarized light, which can be transmitted through the first polarization beam splitter 3; the first filter 10 is responsible for filtering stray light. The high frame rate camera 11 transmits the received signal to the computer processing module in real time, and realizes real-time drift correction in the Z-axis direction through the centroid fitting algorithm.

[0050] Among them, the centroid fitting algorithm is a commonly used image processing technique to determine the centroid position of an object in an image. In this case, it is used to measure the position of the sample in the axial direction by measuring the image change of the total internal reflection (TIR) ​​beam at the cover glass-biological tissue interface. The algorithm principle is as follows:

[0051] 1) Calculate the center of mass of each image: For each image in the image sequence, calculate its center of mass position. Center of mass position , ) The central moment formula is as follows:

[0052] ,

[0053] in, is the image at position The intensity of , N is the total number of all pixels in the image.

[0054] 2) Track the changes in the center of mass position: record the center of mass position of each image in the image sequence to form a center of mass trajectory.

[0055] 3) Analyze the centroid trajectory: The centroid trajectory can reflect the movement of the sample on the XY plane. If the sample drifts in the Z-axis direction, the projection position of the image on the XY plane will change. By analyzing the law of the centroid trajectory, the drift in the Z-axis direction can be inferred.

[0056] 4) Establish a z-axis drift model: According to the change law of the center of mass trajectory, establish a mathematical model to describe the drift in the z-axis direction. This model can be linear, nonlinear or based on physical processes.

[0057] 5) Correction of Z-axis drift: Use the established Z-axis drift model to correct the image sequence. The specific method can be to translate each image according to the Z-axis drift amount predicted by the model, or to consider the influence of Z-axis drift during image reconstruction.

[0058] Furthermore, real-time drift correction of the XY axis is achieved by:

[0059] The second laser light source 12 is outputted through the second laser fiber 21, which is a light beam with a wavelength different from the near-infrared light outputted by the first laser light source 1. It is collimated by the second collimator 23, and its beam diameter is preferably 2-5 mm. The specific size is subject to the sensor size of the high frame rate camera 18. It is then reflected by the second polarization beam splitter 13 and the dichroic mirror 4, and is vertically linearly polarized light at this time. It is adjusted to circularly polarized light by the quarter-wave plate 5, converged by the field lens 6, and focused to the center of the rear focal plane of the objective lens 8. The second reflector 7 is responsible for folding the converged beam.

[0060] The light beam entering the objective lens presents collimated light illumination at the center of the sample surface and irradiates the gold nanoparticles (target) in the sample, generating a strong zero-angle signal and weaker scattered signals at other angles, which return from the rear focal plane of the objective lens, pass through the field lens 6, the quarter-wave plate 5, the dichroic mirror 4, the second polarization beam splitter 13, the relay lens 14, the filter 15, the imaging lens 16, and the second filter 17 in sequence, and are collected by the high-frame rate camera 18. Since the infrared light beam passes through the quarter-wave plate 5 twice, the polarization direction of the returning light beam rotates 90°, so it is adjusted to horizontal linear polarized light and can be transmitted through the second polarization beam splitter 13; the second filter 17 is responsible for filtering out stray light. The focal point is conjugated by the relay lens 14 and the field lens 6, and the focal point on the pupil plane of the objective lens is conjugated to the filter 15; since the filter is annularly distributed, the center is solid and the edge is hollow, the zero-angle signal can be filtered out, while the weaker scattered signals at other angles can pass through the filter. The weaker scattered signals at other angles are contour information with high-frequency signals, which provide a higher-precision calculation basis for the drift correction in the XY direction. It is worth noting that the ratio of the ring of the filter 15 needs to be selected through experiments to obtain the best value. The high-frame rate camera 18 transmits the received signal to the computer processing module in real time, and realizes real-time drift correction in the XY axis direction through the cross-correlation algorithm.

[0061] The Redundant Cross-Correlation (RCC) algorithm in this application is a technique used for signal processing, especially for drift correction in optical super-resolution imaging. Its principle is based on calculating the cross-correlation between two signals to determine their similarity and relative displacement. In optical super-resolution imaging, the position of the image changes over time due to the drift of the sample. RCC can detect and correct this drift to improve the accuracy of imaging.

[0062] After achieving accurate estimation of sample drift during imaging in single-molecule localization microscopy experiments, active drift correction can be achieved by using a high-speed movable piezoelectric stage with nanometer positioning accuracy to compensate for the drift. It should be emphasized that the positioning accuracy and movement speed of the piezoelectric stage directly limit the performance of the drift correction system.

[0063] According to the above records, the device in the present application realizes real-time monitoring and correction of the sample position by combining an infrared (IR) beam to compensate for the sample drift that may occur during long-term imaging. Gold nanoparticles (NPs) are used as reference markers (targets) to perform active drift correction through scattered light signals. The excitation / fluorescence is combined with the IR light through a spectroscope, and the system is divided into an excitation branch and a detection branch through a non-polarizing spectroscope. In the detection branch, the system forms an intermediate image of the scattered NPs by an objective lens and a lens, and blocks the reflected excitation light through an aperture, thereby realizing a dark field imaging system. It communicates with the instrument (high frame rate camera and piezoelectric nano-displacement stage) digitally and is controlled by software integration, providing a graphical user interface to monitor stabilization and perform measurement interaction.

[0064] In one embodiment, the gold nanoparticle sample preparation process is as follows: first, the culture dish is treated with polylysine to prevent shedding, 400 μL of 0.1 mg / mL polylysine aqueous solution is added to the culture dish and allowed to stand for 3 hours, then the solution is aspirated and the culture dish is dried. Polylysine can enhance the adsorption capacity of the culture dish glass slide and prevent the gold nanoparticles from falling off. The polystyrene-wrapped fluorescent beads are diluted 1000 times with distilled water, then added to the culture dish that has been treated with anti-shedding and allowed to stand for 2 hours. During the standing process, the gold nanoparticles will sink under the action of gravity and adsorb on the glass slide at the bottom of the culture dish.

[0065] Figure 2 This is an image of the gold nanoparticles captured in this embodiment. By selecting an area (white box) and performing calculations on it, long-term XY drift correction can be achieved. Figure 3This is the experimental result of the drift correction module working for 2 hours. For the 200nm fluorescent particle sample (mixed with gold nanoparticles as anti-drift benchmark), the drift correction module runs at a speed of 12Hz (i.e. 1 / 12 millisecond), including: taking a frame and acquiring it in 10 milliseconds (100 frames / second), three-dimensional drift calculation in 38.5 milliseconds (26 frames / second), and moving the piezoelectric station in 30 milliseconds, which can achieve stabilization performance better than 1 nanometer (in the XY direction) and 1.4 nanometers (in the Z direction).

[0066] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A three-dimensional high-precision real-time drift correction method for the imaging process of single-molecule localization microscopy experiments, characterized in that: The method comprises the following steps: S1. Use the camera to obtain the initial image; S2. Create an axial stack of initial images; S3. The axial stack was analyzed using a linear fitting algorithm to determine the initial position of the image; S4. Collect each subsequent image in real time and use the cross-correlation algorithm to calculate the relative displacement between images; specifically: Through off-axis incidence, the signal from total internal reflection illumination is collected; through the centroid fitting algorithm, the image changes at the interface between the coverslip and the biological tissue are analyzed to achieve real-time Z-axis drift correction; Through central incidence, the central light is blocked and scattered signals are collected; through the cross-correlation algorithm, the shape changes of the gold nanoparticle markers are analyzed to achieve real-time drift correction of the XY axis; S5. Based on the relative displacement, the sample or the optical system is fine-tuned by a piezoelectric nano-translation stage to achieve sub-nanometer precision positioning and drift correction.

2. The method according to claim 1, characterized in that The linear fitting algorithm described is a least squares fit and is used to determine the position of each image in the axial stack.

3. The method according to claim 1 or 2, characterized in that The cross-correlation algorithm comprises the following steps: S101. Performing a fast Fourier transform on the reference image and the image to be corrected; S102. Calculating the conjugate product of the Fourier transforms of the two images; S103. Find the cross-correlation peak value through inverse Fourier transform and determine the relative displacement between images; S104. Calculate the row and column displacements between images based on the peak positions to achieve sub-nanometer displacement estimation.

4. The method according to claim 3, characterized in that The drift correction process is performed in real time and synchronously with the imaging process.

5. The method according to claim 1 or 4, characterized in that The method also includes a system for storing and logging drift correction data to facilitate historical data analysis and performance monitoring.

6. The method according to claim 1 or 4, characterized in that: The method also includes a user interface for displaying the correction status in real time and adjusting the correction parameters.

7. An electronic device, characterized in that: include: A memory having program code stored therein; A processor is coupled to the memory and implements the method according to any one of claims 1 to 6 when the program code is executed by the processor.

8. A computer-readable storage medium, characterized in that: Program instructions are stored thereon, and when the program instructions are executed, the method according to any one of claims 1 to 6 is implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images