STED microscopy with improved signal-to-noise ratio in low-photon-counting imaging conditions

Abstract

The present invention relates to a method for generating a composite image, the method comprising the steps of acquiring STED images of a sample comprising pixels, calculating arrival-time Fourier coefficients for the image pixels to obtain real coefficients representing a first image and imaginary coefficients representing a second image, deriving an intensity image from the STED image, applying a spatial filter to the first image, the second image and the intensity image to obtain respective filtered images, calculating an image G based on the filtered first image and the filtered intensity image, calculating an image S based on the filtered second image and the filtered intensity image, and calculating a composite image based on the two images G and S.

Description

[Technical field]

[0001] In STED (Stimulated Emission Depletion) fluorescence microscopy, as described for example in EP 0801759, at least one image of a sample stained with a fluorescent molecule or dye is obtained by exciting the fluorescent molecule or dye with a light pulse in the focal region of the microscope objective. In addition, a de-excitation light is irradiated onto the sample to de-excite areas outside the focal region or center of the focal area. Thus, such microscopes can generate images with a spatial resolution below the diffraction limit of normal light. This de-excitation can come from a pulsed light source or a continuous wave (CW) light source. The two light sources are usually lasers and are called the excitation laser and the depletion laser. [Background technology]

[0002] US Patent No. 9,551,658 describes a method and apparatus for extracting light pulses originating from a sample with better spatial resolution using time-gated detection. De-excitation light causes a decrease in the fluorescence lifetime. Thus, the time decay of the fluorescence signal is shorter outside the center of the focal area than in the center of the focal area. In this case, the lifetime τ is:

number

[0003] The publication Lanzano et. al. (2015), Encoding and decoding spatio-temporal information for super-resolution microscopy, Nature Communications, 6:6701, 10.1038 / ncomms7701 describes the SPLIT method, which can preserve the signal of the initial photons by applying a phasor approach. In this publication, the lifetime decay for each pixel is converted into a polar coordinate plot (called a phasor plot). In this plot, the coordinate of each pixel (called a phasor) is a linear combination of three components: the high-resolution signal, the low-resolution signal, and the background coordinate. The weight of the high-resolution component is used as the intensity for the resulting image, which contains information from all detected photons. Although the signal of all photons is used, the method has the limitation that a large number of photons must be acquired. Under low photon counting conditions, the distribution in the phasor plot becomes too broad, so that an improvement in the signal-to-noise ratio cannot be achieved.

[0004] To achieve a narrower distribution, the phasor coordinates must be filtered. A book chapter, Digman et. al. (2012), The phasor approach to fluorescence lifetime imaging: exploiting phasor linear properties, in Fluorescence Lifetime Spectroscopy and Imaging: Principles and Applications in Biomedical Diagnostics (ed. L Marcu, PMW French, and DS Elson), CRC Press, describes a filtering method for phasor plots, aiming to preserve spatial information by using a median filter. However, in the context of super-resolution, even this filtering method introduces an undesirable degradation of spatial resolution.

[0005] More advanced filtering methods are known in digital image processing. As an example, the publication Sendur, L. & Selesnick, I., (2003), Bivariate Shrinkage with Local Variance Estimation, Signal Processing Letters, IEEE. 9, 438-441, 10.1109 / LSP.2002.806054. describes an advanced denoising filter based on the wavelet transform that better preserves the spatial resolution when applied to images. Attempts to apply this denoising filter or similar filters to the phasor components in the same way as a median filter do not work due to the strong variations in noise in the phasor component images. Summary of the Invention [Means for solving the problem]

[0006] It is an object of the present invention to provide a method for increasing the signal-to-noise ratio in STED images under low photon counting conditions while maintaining the spatial resolution of the gated STED approach. acquiring a STED image comprising pixels of the sample; calculating Fourier coefficients of the photon arrival times or photon arrival time data for the image pixels to obtain real and imaginary coefficients, the real coefficients representing the first image and the imaginary coefficients representing the second image; deriving an intensity image from the STED image; applying a spatial filter to the first image, the second image and the intensity image to obtain respective filtered images; calculating an image G based on the filtered first image and the filtered intensity image; calculating an image S based on the filtered second image and the filtered intensity image; and A step of calculating a composite image based on the two images G and S. This is achieved by a method including at least some of the steps:

[0007] Preferably, the step of acquiring an STED image comprises: Irradiating the sample with pulsed excitation light, the pulsed excitation light including a focal point when irradiated onto the sample; illuminating the sample with de-excitation light surrounding a central region of the focal spot; Includes.

[0008] Typically, a two-dimensional (or three-dimensional) STED image of a sample or a part of a sample is acquired using a scanning microscope with STED capabilities. For example, the acquired two-dimensional STED image contains pixels (e.g. 512x512 pixels) in the x and y directions, depending on the acquisition parameters of the scanning microscope. The photon arrival time with respect to the laser pulse is recorded for each pixel. The fluorescence lifetime of typical fluorescent dyes is in the range of 1ns to 4ns. Many STED dyes have various lifetime components. In the center of the focal region, there is no or only a small amount of de-excitation light and a natural lifetime is observed. This lifetime decreases / is shorter outside the center of the focal region, but is typically less than 1ns due to de-excitation processes, for example in the area illuminated by the de-excitation light. The decrease in lifetime outside the center of the focal region is inversely proportional to the depletion energy by the depletion laser. For a typical STED image, about 100 photon counts are recorded in the bright pixels. The calculation of the Fourier coefficients of the photon arrival times is performed pixel by pixel, preferably for all image pixels. Because every pixel has a real Fourier coefficient and an imaginary Fourier coefficient, the real (Fourier) coefficient can be considered as a first image, and the imaginary (Fourier) coefficient can be considered as a second image. However, it is not important whether the first image or the second image actually forms, for example, a two-dimensional image. The first image and the second image can be represented or considered as a data structure having, for example, only a linear data storage arrangement or other suitable data storage arrangement. The same applies to at least one of the composite image, the filtered image, the intensity image, the image G, and the image S.

[0009] In a preferred embodiment, acquiring the STED image further comprises recording the arrival time of each of the detected fluorescence photons with respect to the excitation pulse separately for each pixel.

[0010] Alternatively or additionally, the arrival times are recorded by at least two time gates.

[0011] Preferably, the calculation of the Fourier coefficients includes the calculation of the nth order Fourier coefficient, where n is a preselected value (a natural number).

[0012] Preferably, the intensity image is derived from the acquisition of a STED image or from the calculation of an intensity value derived from the number of photons recorded for a pixel.

[0013] In a preferred embodiment, the image G is calculated based on dividing the filtered first image by the filtered intensity image.

[0014] In another preferred embodiment, the image S is calculated based on dividing the filtered second image by the filtered intensity image.

[0015] In a further preferred embodiment, the spatial filter is at least one of anisotropic diffusion, total variation denoising, maximum entropy denoising, non-local means and wavelet denoising.

[0016] The above object is further solved by an apparatus adapted to carry out the above method.

[0017] Moreover, the above object is further solved by a computer program implementing the above method.The first image and / or the second image in the sense of the present application may be represented in the form of a two-dimensional array of respective Fourier coefficients.

[0018] In a particularly preferred embodiment of the invention, the filter / spatial filtering of the first image, the second image and the intensity image is applied before the Fourier coefficients are divided by the filtered intensity. Previously, the filter was applied after the step when the images were divided. The (divided) phasor coordinates of neighboring pixels have similar values, while the raw Fourier coefficients vary with the intensity, especially under low photon count conditions. The term phasor coordinate in the present context is used similarly to the term described in Lanzano et. al. In the case of a lower number of photon counts, the Fourier coefficients of neighboring pixels show a higher degree of variation than in the case of high photon counts. Different variations with respect to a similar average value contradict the prerequisites of advanced filter methods. In the prior art, a median filter was chosen, which works reasonably when there is little variation in the photon counts in the neighboring pixels. However, STED images show a high degree of variation that must be maintained. If the images are filtered before division, the prerequisites for advanced filters are obtained according to the invention. Furthermore, if the filtered image is divided, there is a disadvantage that the filtered image may contain information from different regions with different weights, and therefore the division step may result in a broader distribution of phasor coordinates.

[0019] However, these results achieved by the procedure of a particularly preferred embodiment are surprisingly superior to conventional / previous approaches, to the extent that the objective can actually be achieved: the advanced filtering algorithm turns out to be much more sensitive to strong variations in the statistics of the neighboring pixels. [Brief description of the drawings]

[0020] [Figure 1]The results of STED de-excitation in terms of lifetime decay are shown: in Fig. 1a a phasor plot with typical lifetime decay for data from the central region of the focus, and in Fig. 1b a phasor plot with typical lifetime decay for data from the outer regions of the focus. In Fig. 1c the combined lifetime decay and a typical time gate selection are shown. The location of these components in the phasor plot is shown in Fig. 1d. [Diagram 2] The phasor distributions from 20 image pixels are shown for different filtering methods, i.e., in Fig. 2a no filter is applied, in Fig. 2b a median filter 5x5 is applied, in Fig. 2c a median filter 7x7 is applied, and in Fig. 2d a wavelet filter of a preferred embodiment of the present invention is applied. The data used is simulated for an average of 10 photon counts per pixel. [Diagram 3] The results of filtering on unwanted smooth edges are shown in Fig. 3b for optimal filtering, in Fig. 3c for no filtering, in Fig. 3d for a median filter 7x7, and in Fig. 3e for the filter of a preferred embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] In addition to a standard STED microscope, a means of photon arrival time detection with respect to the excitation pulse of excitation light, usually implemented in a confocal laser scanning microscope, is essential for the method of the invention, which may for example include a time-correlated single photon counting (TCSPC) recording device or a time-gated based recording device.

[0022] The recorded data can be expressed as the number N of photons detected at arrival time t per pixel i, i.e. Ni(t). For each pixel, the Fourier coefficients Freal and Fimag are calculated:

number

[0023] The integer value n is the preselected order of the Fourier coefficients. Typically, the first order (1) is selected, i.e., n=1. f is the pulse repetition rate of the excitation light. Additionally, for each pixel, an intensity I is calculated:

number

[0024] In the next step, the three images Freal, Fimag and I are filtered using spatial filters, in particular denoising filters. Preferably, a wide variety of advanced denoising algorithms can be used in this step, which can preserve the spatial resolution. For example, the following algorithms can be applied: anisotropic diffusion, total variation denoising, maximum entropy denoising, non-local means and wavelet denoising.

[0025] In a preferred embodiment, a wavelet filter is selected. The basic operation method is similar to the Fourier filter for denoising. Gaussian noise causes an additive offset to the amplitude of all Fourier coefficients. The image is transformed into Fourier space. The offset is subtracted and the image is transformed back. This process is also called shrinkage. Wavelet space is distinguished from Fourier space by the fact that wavelet coefficients represent frequency information per image region. For wavelet denoising, the same operation method of shrinkage of coefficients is used. A very simple method is to shrink all coefficients by the same offset. More advanced methods have been developed and can be applied to the present invention (e.g. Sendur, L. & Selesnick, I., (2003), Bivariate Shrinkage with Local Variance Estimation, Signal Processing Letters, IEEE. Vol. 9. 438-441, 10.1109 / LSP.2002.806054.).

[0026] The shrinkage approach assumes a Gaussian distribution of the noise. The noise distribution for an intensity image I is known to be Poisson distributed. A way to use a filter designed for Gaussian distributed noise with Poisson distributed noise is to transform the data, for example by the Anscombe transformation:

number

[0027] The situation for the two images Freal and Fimag is different: after the integral action the noise is not exactly Poisson distributed but depends on the fluorescence lifetime.

[0028] However, this situation is not so spectacular for attempts to filter the phasor components. First, we can assume that the distribution is mostly Poisson and that the noise variance for these images can be evaluated by a statistically robust method, e.g., the median absolute deviation (MAD):

number

[0029] In a preferred embodiment, all three images Freal, Fimag and I are transformed by the Anscombe transform (3) and then the noise for all three images is estimated by the median absolute deviation method (4). The Anscombe transformed images are transformed into wavelet space. The coefficients are deflated by a value derived from the standard deviation of the estimated noise. The deflated images are then inverse transformed from wavelet space and inverse Anscombe transformed. The results are the filtered images Freal_filtered, Fimag_filtered and I_filtered.

[0030] In the next step, the phasor coefficients for every image pixel are calculated:

number

[0031] In the final step, a composite image is calculated from the phasor coefficients, which can be performed similarly to the SPLIT method described in Lanzano et. al. (2015), Encoding and decoding spatio-temporal information for super-resolution microscopy, Nature Communications, 10.1038 / ncomms7701.

[0032] It will now be explained why the method of the present invention works better than the prior art, where the phasor G and S components are filtered. In Fig. 1a, an attenuation diagram is shown with a graph 3 for photons originating from the central region of the focus of the excitation light, with intensity I along ordinate 1 and arrival time t along abscissa 2. In Fig. 1b, a graph 6 is shown for photons originating from the region outside the center of the focus, where the de-excitation light is irradiated. The average arrival time of photons originating from the region outside the center of the focus is shorter than the average arrival time of photons originating from the focus. In Fig. 1c, a graph 7 for all detected photons is shown. In time-gated STED, an arrival time interval 8 is selected (for example by gating) in which photons from the central region dominate. Photons detected before this interval are ignored. Thus, the signal 4 (see Fig. 1a), which carries information about the central region, is lost. Only part 5 of the signal (see Fig. 1a) is used to generate the image. The time-gated approach results in a spatially better resolved, but noisier image compared to conventional STED methods.

[0033] The pixel with only attenuation from the central region is located for example at position 9 in the phasor plot (see FIG. 1d). Photons from outside are located at a separate position 10 in the phasor plot. The idea behind the SPLIT method (prior art) is to apply weights to the intensity of individual pixels depending on their position in the phasor plot. This results in a spatial resolution comparable to the time-gated STED method, but with less noise, since no counted photons are lost.

[0034] Figure 2a visualizes simulated data for an average of 10 photon counts in an image pixel. The simulation assumes a mixture of 50% photons 11 from the central area with a lifetime of 4ns (4 nanoseconds) and 50% photons 12 from the outside with a lifetime of 200ps (200 picoseconds). The phasor coordinates 13 of 20 pixels are shown. For high photon counts, the position of the phasor must be at the midpoint between the positions of the two components 11, 12, i.e. at 14. The low unfiltered photon counts, marked 15, are in a situation where no improvement is possible compared to the gating method due to the wide distribution in which no filter is present.

[0035] In figure 2b the same simulation data is shown when a 5x5 median filter, labelled 16, is applied according to the prior art. In the case of figure 2c a 7x7 median filter, labelled 17, is used. Finally, in figure 2d the results are shown when a wavelet filter according to one embodiment of the invention is used (labelled 18). To achieve a comparable signal to noise ratio a 7x7 median filter is required.

[0036] The filter must also maintain spatial resolution to be useful in STED applications. Simulations with two mixtures show the behavior of the filter along the edges. In the simulation of Fig. 3a, an image 19 was created with a mixture of two-thirds low-resolved and one-third high-resolved components in the left half 20. In the right half 21, one-third low-resolved and two-thirds high-resolved components are simulated. Seven rows of eight pixels in the central region 22 are stacked vertically and weighted with an intensity gradient along a line between the 4 ns and 200 ps positions in the phasor.

[0037] In Fig. 3b the intensity 23 over 8 average pixels is shown for the optimum conditions of very high photon counting. The intensity over 8 average pixels with 10 counts per pixel is shown in Fig. 3c for no filtering, Fig. 3d for using the median filter 7x7, and Fig. 3e for using the wavelet filter of the present invention.

[0038] The wavelet filter of the present invention maintains spatial resolution (FIG. 3e) better than the prior art median filter (FIG. 3d) while maintaining a comparable signal-to-noise ratio (FIGS. 2c and 2d).

[0039] Advanced filters are designed for similar noise characteristics in neighboring pixels. Such filters do not work if applied directly to the phasor components as per the prior art. The noise fluctuates excessively due to the division by magnitude in the phasor equation.

[0040] Some or all of the steps may be performed by (or using) a hardware apparatus, such as, for example, a processor, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, any one or more of the crucial steps may be performed by such an apparatus.

[0041] Depending on certain implementation requirements, the embodiments of the present invention can be implemented in hardware or software. The implementation can be performed by a non-transitory recording medium, such as a digital recording medium, for example a floppy disk, a DVD, a Blu-ray, a CD, a ROM, a PROM and EPROM, an EEPROM or a FLASH memory, on which electronically readable control signals are stored, which cooperate (or can cooperate) with a programmable computer system to implement the respective methods. Thus, the digital recording medium can be computer readable.

[0042] Some embodiments of the present invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to perform any of the methods described herein.

[0043] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code which is operable to perform any of the methods when the computer program product is run on a computer, the program code may for example be stored on a machine readable carrier.

[0044] Another embodiment comprises the computer program for performing any of the methods described herein, stored on a machine readable carrier.

[0045] In other words, an embodiment of the present invention is, therefore, a computer program having a program code for performing any of the methods described herein, when the computer program runs on a computer.

[0046] Therefore, another embodiment of the present invention is a recording medium (or data carrier or computer readable medium) containing a computer program stored thereon for performing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the present invention is an apparatus as described herein, including a processor and a recording medium.

[0047] A further embodiment of the invention is therefore also a data stream or a sequence of signals representing the computer program for performing any of the methods described herein, the data stream or the sequence of signals being for example adapted to be transmitted via a data communication connection, for example the Internet.

[0048] Another embodiment comprises a processing means, for example a computer, or a programmable logic device configured to or adapted to perform any of the methods described herein.

[0049] Another embodiment comprises a computer having the computer program installed thereon for performing any of the methods described herein.

[0050] Another embodiment of the invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for implementing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may, for example, include a file server to transfer the computer program to the receiver.

[0051] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods are advantageously performed by any hardware apparatus.

Claims

1. A method for generating a result image, the method comprising: a. acquiring a STED image comprising pixels of a sample; b. performing a time-resolved measurement of the fluorescence pixel by pixel and performing a Fourier analysis to obtain real coefficients representing a first image and imaginary coefficients representing a second image in order to complexly represent the STED image as a real image and an imaginary image; c. deriving an intensity image from the STED image; d. applying a spatial filter, which is a noise reduction filter, to the first image, the second image and the intensity image to obtain respective filtered images; e. calculating an image G based on the filtered first image and the filtered intensity image; f. calculating an image S based on the filtered second image and the filtered intensity image; g. calculating said result image based on the two images G and S; The method includes:

2. The step of acquiring the STED image includes: irradiating the sample with a pulsed excitation light having a focus; illuminating the sample with de-excitation light surrounding a central region of a focal spot; Including, The method of claim 1.

3. The step of acquiring the STED image includes: recording the arrival time of the detected fluorescence photons relative to the excitation pulse for each pixel separately; and / or Recording arrival times through at least two time gates. Further comprising:

3. The method according to claim 1 or 2.

4. The Fourier analysis includes calculating n-th order Fourier coefficients, where n is a preselected value.

4. The method according to claim 1 ,

5. The intensity image is derived from the acquisition of the STED image or from the calculation of intensity values ​​derived from the number of photons recorded for a pixel.

5. The method according to any one of claims 1 to 4.

6. The image G is calculated based on dividing the filtered first image by the filtered intensity image.

6. The method according to any one of claims 1 to 5.

7. The image S is calculated based on dividing the filtered second image by the filtered intensity image.

7. The method according to any one of claims 1 to 6.

8. Preferably, before step d) of claim 1 is performed, the method further comprises the step of transforming the acquired STED image, the first image, the second image and / or the intensity image having a first noise distribution, e.g. a Poisson noise distribution, into a transformed image having a second noise distribution, e.g. a Gaussian noise distribution, and performing the remaining steps, in particular those of claim 1, on the basis of the transformed image.

8. The method according to any one of claims 1 to 7.

9. The method further comprises the step of inverse transforming the acquired STED image, the filtered first image, the filtered second image and / or the filtered intensity image after step d) of claim 1 has been performed. The method of claim 8.

10. The spatial filter is at least one of anisotropic diffusion, total variation denoising, maximum entropy denoising, non-local means and wavelet denoising.

10. The method according to any one of claims 1 to 9.

11. An apparatus for carrying out the method according to any one of claims 1 to 10, said apparatus being preferably a STED microscope or a STED scanning microscope. Device.

12. comprising program code which, when executed on a processor, performs the method according to any one of claims 1 to 10, Computer program.

Images