Few-photon structure illumination obvious micro-imaging method based on timestamp principle

Abstract

A few-photon structure illumination obvious micro-imaging method based on the timestamp principle comprises the steps that photons reflected by a laser irradiation sample are collected through an area array detector, the arrival time and space information of the photons are recorded, and the geometric center of a cluster is determined according to the measured arrival time (TOT) and the time when photon energy exceeds a threshold value (TOA); the estimation of the space coordinates of the incident photons is obtained; and after estimation of the space coordinates of the incident photons is obtained through statistical analysis, a clear image with a few photon level is obtained through timestamp reconstruction and structured light reconstruction. According to the invention, clear microscopic imaging is realized by using information of a small number of photons received by the detector, and technical support is provided for observation of biological samples and processes with extremely low light dose.

Description

Technical Field

[0001] The present invention relates to a technology in the field of optical microscopic imaging, specifically a few-photon structured illumination microscopic imaging method based on the time stamp principle. Background Art During microscopic imaging, illumination light can cause the stained fluorescent molecules to lose their ability to emit light, affecting the observation of fluorescent biological samples. For biological structures and processes that are extremely sensitive to light doses, excessive illumination light may inactivate the biological structures and affect biological dynamic processes at the microscopic scale, making it impossible to conduct normal observations. Summary of the invention

[0002] In view of the shortcomings of the prior art that biological samples cannot be observed under low-photon conditions and high-speed dynamic imaging cannot be achieved, and that living cell imaging will cause the defect of smearing, the present invention proposes a low-photon structured illumination microscopic imaging method based on the timestamp principle, which uses the information of the few photons received by the detector to achieve clear microscopic imaging, providing technical support for the observation of biological samples and processes with extremely low light doses.

[0003] The present invention is achieved through the following technical solutions:

[0004] The present invention relates to a few-photon structured light illumination microscopy imaging method based on the timestamp principle. A planar array detector is used to collect photons reflected from a laser-irradiated sample and record their arrival time and spatial information. The geometric center of the cluster is determined by using the measured arrival time (TOT) and the time (TOA) when the photon energy exceeds a threshold value, and an estimate of the spatial coordinates of the incident photons is obtained. After the estimate of the spatial coordinates of the incident photons is obtained through statistical analysis, a clear image at the few-photon level is obtained through timestamp reconstruction and structured light reconstruction.

[0005] The few photons mentioned above refers to that the detected photon signals are extremely few, with an average of 10-20 photon signals per pixel.

[0006] The acquisition described above uses but is not limited to a planar array pixel detection device with a single-photon accuracy level and nanosecond time resolution, which can detect the time information of the photon signal arriving at the detector and the initial image with extremely high accuracy. The collected data is numerical data including the signal arrival time, spatial coordinates and the time when the photon energy exceeds the threshold.

[0007] The laser is preferably generated by a single-mode fiber-coupled laser.

[0008] The sample is a fixed sample or a living cell sample.

[0009] The irradiation means: after dripping water on the objective lens, placing the sample on the stage, moving the coarse focusing screw so that the upper surface of the water on the objective lens just contacts the sample, then adjusting the fine focusing screw, and observing the image on the detector until the focusing is completed.

[0010] The irradiation is preferably performed in a light-free state to prevent damage to the sample and the detector caused by external light.

[0011] The statistical analysis is as follows: using a recursive algorithm to identify clusters, screening photon information of adjacent pixels no more than 300ns to divide clusters; using the measured arrival time (TOT) and the time when the photon energy exceeds the threshold (TOA) to determine the geometric center of the cluster, and obtain an estimate of the spatial coordinates of the incident photon.

[0012] The timestamp reconstruction refers to: sorting the data containing photon coordinates and time information according to pixel position and time sequence; reconstructing the time intervals of the first 10-20 photon signals of the time series according to the timestamp principle to obtain a normalized structured fringe image of structured light illumination, specifically: rewriting the photon arrival time information into a numerical file in the form of cells according to spatial coordinates, and each row of the cell data represents the photon arrival time information of this spatial coordinate. The average time interval information can be obtained using the arrival time series in the cell data, and the intensity value of the coordinate can be obtained using the inverse of the average time interval, and finally the entire image is normalized.

[0013] The structured light reconstruction refers to: using a number of structured fringe images with different directional phases to perform structured light illumination microscopy reconstruction to obtain a clear few-photon microscopy image, which is achieved by using but not limited to the technology recorded in "High-fidelity structured illumination microscopy by point-spread-function engineering" by Gang Wen et al. Technical Effects

[0014] The present invention uses area array detection combined with super-resolution microscopy technology to perform few-photon time-stamp imaging. Compared with the prior art, super-resolution imaging can be achieved at a lower light dose than conventional light doses. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the present invention;

[0016] Figure 2 is the optical path diagram of the system;

[0017] Figure 3 This is the schematic diagram of micro-reconstruction under structured light illumination;

[0018] Figure 4 and Figure 5 It is a schematic diagram of the effect of the embodiment. DETAILED DESCRIPTION

[0019] The present embodiment relates to a few-photon structured light illumination microscopy imaging system based on the timestamp principle, comprising: an illumination module, a data receiving and processing module, a data processing and sorting module, and a timestamp reconstruction module, wherein: after the illumination module illuminates the sample with coherent light, it is modulated by a spatial light modulator (DMD) to generate structured light, thereby obtaining structured stripe illumination light; the data receiving and processing module receives the fluorescence information reflected by the sample, performs spatial centroid fitting and time correction processing, and obtains the arrival time information and spatial coordinates of the photons; the data processing and sorting module writes the arrival time information and spatial coordinates of the photons into the cellular data according to the arrival time information and spatial coordinates of the photons, and obtains the order according to the spatial coordinates and time sequence; the timestamp reconstruction module obtains the average time interval information according to the obtained time series information, and obtains the intensity value by calculating the inverse, and obtains a clear image by processing each pixel.

[0020] like Figure 2 As shown, the lighting module includes: a light source 1, lenses 14, 15, an aperture 5, a reflector 11, an aperture 6, a digital micromirror device 2, a lens 16, an aperture 7, a lens 17, an excitation plate 8, a dichroic mirror 9, an objective lens 13 and a sample 4 arranged in sequence; the data receiving and processing module includes a transmitting plate 10, a reflector 12, a tube lens 18 and a camera 3 arranged in sequence on the other side of the dichroic mirror 9.

[0021] like Figure 1 As shown, this embodiment relates to a few-photon structured illumination microscopic imaging method based on the timestamp principle of the above system, including:

[0022] Step 1: Determine the measurement variables. The selected biological sample is thick and difficult to penetrate. Therefore, after the laser irradiates the sample, the reflected photon data is collected to obtain the initial image.

[0023] Step 2: Build a few-photon structured light microscopy imaging system. Based on the characteristics of structured light microscopy imaging, adopt an array detection method. The photons emitted by the laser light source are modulated into structural stripes through a spatial light modulator (DMD) to illuminate the target object, and a nanosecond time resolution array detection device (Timepix3) is used for recording.

[0024] Step 3: Collect and sort preliminary data: Asynchronous reading that may occur under high-rate detection will disrupt the time order, and the detected photon data will be re-sorted by pixel in time.

[0025] Step 4: Cluster analysis and centroid processing: Use a recursive algorithm to identify clusters, and filter the photon information of adjacent pixels no more than 300ns to divide the clusters; use the measured photon arrival time (TOA) and the time when the photon energy exceeds the threshold (TOT) to determine the geometric center of the cluster and obtain an estimate of the spatial coordinates of the incident photon.

[0026] Step 5: Correcting TOA and TOT, specifically including: calculating the time difference for each pixel in the cluster and performing time difference correction, specifically: dTOA(TOT)=TOA(TOT)-TOA centroid (TOT centroid ), where dTOA refers to the photon arrival time corresponding to the pixel with the largest TOT in the cluster after statistical calculation (zero time difference TOA centroid ) and the arrival time (TOA) of a photon in a certain pixel of the cluster; dTOT refers to the maximum time when the energy of the clustered photon exceeds the threshold (zero time difference TOT centroid ) is the difference between the time when the photon energy of a certain pixel in the cluster exceeds the threshold (TOT).

[0027] Preferably, the dTOA(TOT) lookup table is obtained by repeating the calculation of step 5 in large quantities, and the TOT in the lookup table is centroid When the value is greater than 1500ns, dTOA(TOT) remains unchanged, and the value is set to centroid If it is greater than 1500ns, perform unified correction, otherwise perform the time difference correction in step 5.

[0028] Step 6, sorting and timestamp reconstruction: Sort the data containing photon coordinates and time information according to pixel position and time sequence. The sorted photon event sequence is in chronological order and stored in the cell data; reconstruct the time intervals of the first 10-20 photon signals in the time series according to the timestamp principle to obtain a normalized structural fringe image of structured light illumination.

[0029] Step 7: Structured light illumination micro-reconstruction: Use the 9 structural fringe images with different phases in different directions obtained in step 6 to perform the following Figure 3 The structured light illumination microscopic reconstruction shown in the figure ultimately results in a clear few-photon microscopic image.

[0030] After specific actual experiments, under the condition of constant environment of 20 degrees and no ambient light, coherent light source illumination was used, and detectors that can detect photon signals with extremely high accuracy were used for detection, and the time and space information of the obtained photons were processed. Finally, the Poisson statistical distribution law of photons was used to reconstruct the timestamp and obtain a clear few-photon microscopic image. The experimental data obtained by data acquisition in the above steps are as follows Figure 5As shown in the figure, the comparison between the reconstruction result of 20-photon timestamp and the cumulative and total photons is shown, as well as the magnified image of the key area and the corresponding intensity-distance image for resolution comparison. From the magnified image of the key area, it can be seen intuitively that the 20-photon timestamp reconstructed image and the total photon reconstructed image are significantly better than the wide-field image and the 20-photon cumulative image, and can distinguish the two line structures; in the intensity-distance image, only the 20-photon timestamp reconstructed image and the total photon reconstructed image can distinguish the two line structures, proving that the image comparable to strong light can be obtained by using the 20-photon timestamp reconstruction, which can also demonstrate the superiority of the method.

[0031] like Figure 4 As shown in the figure, a comparison between the 20-photon timestamp reconstruction result and the cumulative and total photons and an enlarged image of the key area are shown. In the figure, a is a strong-light image directly taken without structured light reconstruction and timestamp reconstruction; b is a strong-light image reconstructed by structured light without timestamp reconstruction; c is a weak-light image reconstructed by structured light without timestamp reconstruction; d is a weak-light image reconstructed by timestamp reconstruction and structured light reconstruction.

[0032] Compared with the prior art, the present invention achieves clear images that meet the needs of biological research under low-photon conditions of 5-20 photons per pixel through timestamp reconstruction processing. The photon dose is 20 times lower than that of ordinary intensity accumulation imaging schemes, and the same resolution is maintained as the reconstructed image of all photons.

[0033] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A few-photon structured illumination microscopic imaging method based on the time stamp principle, characterized in that: The photons reflected from the laser-irradiated sample are collected using a planar array detector and their arrival time and spatial information are recorded. The geometric center of the cluster is determined using the measured arrival time (TOT) and the time when the photon energy exceeds the threshold (TOA), and the spatial coordinates of the incident photons are estimated. After the spatial coordinates of the incident photons are estimated through statistical analysis, a clear image at the low-photon level is obtained through timestamp reconstruction and structured light reconstruction. The timestamp reconstruction refers to: sorting the data containing photon coordinates and time information according to pixel position and time sequence; reconstructing the time intervals of the first 10-20 photon signals of the time series according to the timestamp principle to obtain a normalized structured fringe image of structured light illumination.

2. The method for microscopic imaging using few-photon structured illumination based on the time stamp principle according to claim 1, characterized in that: The acquisition described above uses a planar array pixel detection device with a single-photon accuracy level and nanosecond time resolution to detect the time information of the photon signal arriving at the detector and the initial image. The collected data is numerical data including the signal arrival time, spatial coordinates and the time when the photon energy exceeds the threshold.

3. The method for microscopic imaging using few-photon structured illumination based on the time stamp principle according to claim 1, characterized in that: The irradiation means: after dripping water on the objective lens, placing the sample on the stage, moving the coarse focusing screw so that the upper surface of the water on the objective lens just contacts the sample, then adjusting the fine focusing screw, and observing the image on the detector until the focusing is completed.

4. The method for microscopic imaging using few-photon structured illumination based on the time stamp principle according to claim 1, characterized in that: The statistical analysis is as follows: using a recursive algorithm to identify clusters, screening photon information of adjacent pixels no more than 300ns to divide clusters; using the measured arrival time (TOT) and the time when the photon energy exceeds the threshold (TOA) to determine the geometric center of the cluster, and obtain an estimate of the spatial coordinates of the incident photon.

5. The method for microscopic imaging using few-photon structured illumination based on the time stamp principle according to claim 1, characterized in that: The timestamp reconstruction is specifically as follows: the photon arrival time information is rewritten into the numerical file in the form of cells according to the spatial coordinates. After each row of the cell data represents the photon arrival time information of this spatial coordinate, the average time interval information can be obtained using the arrival time series in the cell data, and the intensity value of the coordinate can be calculated using the inverse of the average time interval. Finally, the entire image is normalized.

6. The method for microscopic imaging based on the time stamp principle of few-photon structured illumination according to any one of claims 1 to 5, characterized in that: include: Step 1: Determine the measurement variable. The selected biological sample is thick and difficult to penetrate. Therefore, after the laser irradiates the sample, the reflected photon data is collected to obtain the initial image. Step 2: Build an imaging system. According to the characteristics of structured light microscopy, adopt the method of area array detection. The photons emitted by the laser light source are modulated into structural stripes through the spatial light modulator (DMD) to illuminate the target object. The area array detection device (Timepix3) with nanosecond time resolution is used for recording. Step 3: Collect and sort preliminary data: Asynchronous reading that may occur under high-rate detection will disrupt the time sequence, and the detected photon data is re-sorted by time for pixels; Step 4: Cluster analysis and centroid processing: A recursive algorithm is used to identify clusters, and the photon information of adjacent pixels not exceeding 300ns is screened to divide the clusters; the geometric center of the cluster is determined by measuring the photon arrival time (TOA) and the time when the photon energy exceeds the threshold (TOT), and the spatial coordinates of the incident photons are estimated; Step 5: Correcting TOA and TOT, specifically including: calculating the time difference for each pixel in the cluster and performing time difference correction, specifically: dTOA(TOT)=TOA(TOT)-TOA centroid (TOT centroid ), where dTOA refers to the photon arrival time corresponding to the pixel with the largest TOT in the cluster after statistical calculation (zero time difference TOA centroid ) and the arrival time (TOA) of a photon in a certain pixel of the cluster; dTOT refers to the maximum time when the energy of the clustered photon exceeds the threshold (zero time difference TOT centroid ) is the difference between the time when the photon energy of a certain pixel in the cluster exceeds the threshold (TOT); Step 6, sorting and reconstructing timestamps: sorting the data containing photon coordinates and time information according to pixel position and time sequence, the sorted photon event sequence is in chronological order and stored in the cell data; reconstructing the time intervals of the first 10-20 photon signals of the time series according to the timestamp principle to obtain a normalized structured fringe image of structured light illumination; Step 7: Structured light illumination microscopic reconstruction: The nine structured fringe images with different directional phases obtained in step 6 are used to perform structured light illumination microscopic reconstruction, and finally a clear few-photon microscopic image is obtained.

7. The method for microscopic imaging using few-photon structured illumination based on the time stamp principle according to claim 6, characterized in that: When TOT is met centroid If it is greater than 1500ns, perform unified correction, otherwise perform the time difference correction in step 5.

8. A few-photon structured illumination microscopic imaging system based on the time stamp principle for implementing any of the methods described in claims 1-7, characterized in that: include: An illumination module, a data receiving and processing module, a data processing and sorting module, and a timestamp reconstruction module, wherein: the illumination module illuminates the sample with coherent light, and then modulates the sample with a spatial light modulator (DMD) to generate structured light, thereby obtaining structured stripe illumination light; the data receiving and processing module receives the fluorescence information reflected by the sample, performs spatial centroid fitting and time correction processing, and obtains the arrival time information and spatial coordinates of the photons; the data processing and sorting module writes the arrival time information and spatial coordinates of the photons into the cellular data, and obtains the order according to the spatial coordinates and time sequence; the timestamp reconstruction module obtains the average time interval information based on the obtained time series information, and obtains the intensity value by calculating the inverse, and a clear image is obtained by processing each pixel.

9. The few-photon structured illumination microscopic imaging system according to claim 8, characterized in that: The lighting module comprises: a light source, a lens, an aperture, a reflector, an aperture, a digital micromirror device, a lens, an aperture, a lens, an excitation plate, a dichroic mirror, an objective lens and a sample which are arranged in sequence.

10. The few-photon structured illumination microscopic imaging system according to claim 8, characterized in that: The data receiving and processing module comprises a transmitting plate, a reflecting mirror, a tube mirror and a camera which are sequentially arranged on the other side of the dichroic mirror.

Images