Simultaneous multispecies super-resolution imaging with time multiplexing and single-photon detector arrays

Abstract

A laser scanning microscope configured to illuminate a sample with a plurality of pulsed excitation light beams consisting of different excitation spectral components (B1, B2) comprises a single photon detector (40) configured to detect a fluorescence signal emitted by the sample, the fluorescence signal comprising different spectral components, an excitation spectrum encoder (50) configured to impose a respective time delay on each excitation spectral component, an emission spectrum encoder (60) configured to impose a respective time delay on each emission spectral component, and a species decoder (70) configured to decode the excitation spectrum, emission spectrum and fluorescence decay curves of fluorescent species contained in the sample.

Description

[Technical field]

[0001] The present invention generally relates to multi-species fluorescence microscope technology. [Background technology]

[0002] Confocal laser scanning microscopy (CLSM) with fluorescence is a widely used imaging tool in cell and molecular biology. This technique offers the possibility of three-dimensional imaging, deep imaging, live cell imaging and quantitative imaging. Currently, a wide variety of fluorescent dyes (synthetic dye molecules, fluorescent proteins, inorganic nanoparticles, etc.) can be bound to biological targets with high specificity by many coupling strategies, potentially providing a complete portrait of the observed sample. The so-called multispecies (or multilabel) fluorescence imaging allows the detection of multiple targets to observe the interactions between different cellular structures and their relative spatial organization with specific molecular contrast. Moreover, the recent introduction of high-speed (MHz-range) detector arrays allows a transparent and versatile realization of image scanning microscopy (ISM), making CLSM one of the super-resolution techniques. In such an implementation of ISM (for example, as described in WO2019 / 145889A1), a detector array (with a projection size of 1–1.5 Airy units (AU) at the sample plane) replaces the pinhole and single-element detectors commonly used in CLSM, providing an image of the fluorescence volume. This additional spatial information provided by the detector array can be used to reconstruct super-resolution images with twice the resolution of conventional microscopy and a higher signal-to-noise ratio (SNR). Each element of the detector array acts as one physical pinhole generating a series of super-resolution but low-SNR images. As most of the fluorescence signal is acquired from the detector elements in parallel, a high-SNR image can be obtained by combining all the collected images by pixel reallocation or by deconvolution of multiple images.

[0003] In conventional CLSM, simultaneous multispecies imaging is achieved by separating different probes (or rather different fluorescent signals) based on the photophysical properties of each probe. In particular, three different properties are commonly used: excitation spectrum, emission spectrum, and mean fluorescence lifetime (Fig. 1). Strategies that rely on only one of these properties have strengths and limitations. For this reason, a combined approach is favored. However, the resulting implementation always involves a trade-off between the system complexity, the number of separable species, the capacity for simultaneous imaging, and the cost. Implementations of ISM based on arrays of detectors (hereafter simply ISM), multispecies strategies demonstrated so far, explore only excitation and emission spectra and consist of copies of basic CLSM approaches. In fact, the implementation of more sophisticated multispecies CLSM approaches requires expensive and less scalable architectures in terms of the number of separable species, or may not even be possible to implement.

[0004] Multispecies CLSM

[0005] The most widely used method for simultaneous multispecies CLSM imaging separates the signals of different probes by searching for their emission spectral signatures. All probes are illuminated by one or more monochromatic laser beams (λ exc ) are excited simultaneously in a series of spectral windows (λ ), one for each probe. The fluorescence signal is separated from the excitation light and, by a combination of filters, dichroic mirrors, and tunable acousto-optic filters, is focused onto a series of spectral windows (λ ), one for each probe. det , λ det where denotes the center value of the spectral band. Finally, a detector, or probe, for each spectral band records the fluorescence signal. There are two main limitations to the implementation of multi-detector spectra: the number of detectors required increases as the number of probes increases, and crosstalk between the emission spectra of different probes degrades the image.

[0006] The second problem can be partially solved by breaking the synchronicity of excitation to add information of the excitation spectrum of the probe. In this case, the excitation lasers are activated / modulated sequentially to obtain a series of images: detector bands and excitation beam wavelengths (λ exc , λ det ) are recorded. For each probe, the image formed by the optimal excitation and emission wavelengths is selected. A linear unmixing algorithm can then be used to remove any remaining crosstalk. Due to the sequential activation of the laser excitation beams, it is crucial to employ fast modulation for effective simultaneous imaging. For example, frame-by-frame or line-by-line modulation may be too slow for imaging of dynamic structures. Since fast biological processes occur on sub-millisecond time scales (protein shape changes occur on much smaller scales down to microseconds, but imaging is not an option for observing such processes in all cases) and typical dwell times of a pixel are in the range of tens of microseconds or less, modulating the laser beam pixel by pixel ensures simultaneous multi-species imaging.

[0007] Recently, it has been proposed to modulate the excitation beam with a frequency greater than that given by the pixel dwell time (US9231575B2). The multi-species CLSM implementation multiplexes the excitation beam in the frequency domain. Demodulation is therefore achieved by filtering in the frequency domain. Furthermore, the implementation uses a single detector regardless of the number of probes to be separated, thus significantly reducing the implementation cost. In contrast, the use of a single detector does not allow the excitation and emission spectra of the probes to be interrogated.

[0008] Multispecies CLSM, using a single detector but probing the emission spectrum, can be implemented using a linear detector array (US2019 / 0361213A1). Here, the fluorescence signal induced by one or more excitation beams is spatially spectrally resolved using a prism (or grating) and each wavelength is mapped to a specific location / element of the linear detector. The three-dimensional (x, y, λ) image, the so-called emission spectrum image, is processed by a (blind) spectral unmixing algorithm to form a final multispecies image that is potentially free of artifacts due to signal crosstalk. In this class of CLSM implementations, called hyperspectral implementations, it is also possible to probe the excitation spectrum by modulating the excitation beam as described above. The implementation of hyperspectral CLSM can also be achieved by sequentially recording the emission spectrum; i.e., spectrally dispersing the fluorescence signal in space, selectively filtering only a part of the spectrum with an adjustable double slit, and recording the selected signal with a single-element sensor. Nevertheless, sequential recording of the spectra (by definition) precludes simultaneous multispecies imaging.

[0009] All the above mentioned implementations of multi-species CLSM use emission and / or excitation signatures to separate different probes. Another probe signature to explore is the mean fluorescence lifetime, or more generally, the time decay distribution. Similar to excitation and emission spectra, probes can also be characterized by a time spectrum f(t), which represents the probability that an excited molecule / probe will emit a spontaneous luminescence (fluorescence photon) at a given time t after the excitation event. The average time that a molecule spends in the excited state is called the mean fluorescence lifetime τ. For pure organic fluorophores, this distribution is a single exponential:

number

[0010] In CLSM, the mean lifetime decay distribution of a fluorescent probe is obtained by performing a time-correlated single-photon counting (TCSPC) experiment. A pulsed laser beam (typically a pulse length of tens of picoseconds) excites the probe at a specific time (t=0), a single-photon detector with small timing jitter (tens to hundreds of picoseconds) records the fluorescence photons emitted by the probe, and a time-to-digital converter (TDC) or high-frequency digitizer measures the time difference between excitation and recording. The histogram of the photon arrival times is measured by repeating the experiment (typically the repetition rate of the pulsed laser is on the order of tens of MHz).

[0011] In TCSPC-based CLSM, multispecies imaging can be performed by recording the photon arrival histogram for each pixel and decomposing said histogram (according to the attenuation distribution of the different probes) using different computational algorithms based on fitting, deconvolution, phasor decomposition or linear unmixing. An interesting property of said implementation is the possibility to separate species with very similar excitation and emission spectra, thus using a single excitation beam and a single detection band, i.e. a single detector, which not only brings advantages in terms of complexity but also avoids imaging artifacts due to chromatic aberration. With the same TCSPC-based CLSM system, it is also possible to implement the above mentioned excitation laser modulation in pulse-by-pulse mode, which results in the exploration of excitation spectral signatures for multispecies imaging. Pulses from a series of monochromatic lasers are interleaved (with a delay longer than the mean fluorescence lifetime) and the histogram of the photon arrival times is used to implement a time-gated detection and to separate the signals associated with each probe. The pulse interleaving approach can also be combined with an emission spectrum separation approach to increase the number of probes separated, but at the expense of requiring more detectors.

[0012] Multispecies ISM

[0013] One of the most notable advantages of ISM implementations based on high-speed detector arrays is their compatibility with many labeling methods, techniques, and protocols implemented in fluorescence CLSM, including many of the multispecies imaging techniques mentioned above.

[0014] ISM is fully compatible with spectral multi-detector approaches, modulated multi-execution approaches, and combinations of both. As a result, one skilled in the art could easily incorporate such simultaneous multi-species approaches into ISM. However, to date, only slow frame-by-frame modulated multi-excitation ISM implementations have been demonstrated, where a single detector array is used to alternate between different excitation wavelengths and spectral windows through a motorized filter lens. However, true simultaneous imaging as demonstrated in CLSM is not possible.

[0015] Conversely, hyperspectral methods are not fully compatible with ISM. The basic principle of hyperspectral methods is to spatially decode the spectral information, i.e. to spatially resolve the emission spectrum. Such spatial decomposition hampers ISM, since in ISM the spatial information channel is already used for image transfer of the detection volume. Since the spatial and spectral information are fused in the same channel and scale, their decomposition is non-trivial. For completeness, a hyperspectral ISM implementation has been proposed and used to reconstruct two-species ISM images. However, for the same reasons mentioned above, this implementation suffers from significant artifacts in both spatial resolution and species separation.

[0016] Recently introduced single-photon detector arrays (such as single-photon avalanche diode array detectors (SPADs)) have also enabled the realization of multispecies imaging methods based on mean fluorescence lifetime signatures. In fact, it has been fully demonstrated that SPAD array-based ISMs can perform TCSPC measurements. However, in this case, the demonstration of multispecies ISMs based on mean fluorescence lifetimes has not been reported.

[0017] With the introduction of asynchronous SPAD array detectors, many simultaneous multispecies imaging methods demonstrated in CLSM could in principle be extended to ISM, albeit with some practical limitations related to hyperspectral approaches. However, as with CLSM, no single method is capable of simultaneously exploring excitation spectra, emission spectra and mean fluorescence lifetime signatures. This constraint becomes even more important when only one detector is available.

[0018] One objective of the present invention is to propose a new architecture for multispecies ISM imaging that allows simultaneous imaging, is highly scalable in terms of the number of species, and is low-cost. Another objective of the present invention is to propose a new architecture that offers similar advantages to CLSM, and more generally to any fluorescence technique based on laser scanning microscopy (LSM), such as stimulated luminescence attenuation microscopy (STED), two-photon excitation microscopy (TPE), and fluorescence fluctuation spectroscopy (FFS). Summary of the Invention [Means for solving the problem]

[0019] With these objectives in mind, the subject of the present invention is a laser scanning microscope configured to illuminate a sample with a plurality of pulsed excitation light beams containing different spectral components (hereinafter excitation spectral components), the sample containing a plurality of fluorescent species, said microscope comprising: a single-photon detector array configured to detect a fluorescent signal emitted from the sample, the fluorescent signal comprising different spectral components (hereinafter emission spectral components); an excitation spectral encoder configured to impose a respective time delay on each excitation spectral component such that each excitation spectral component irradiates the sample at a different time relative to the other excitation spectral components; an emission spectral encoder configured to impose a respective time delay on each emission spectral component such that each emission spectral component arrives at the single-photon detector array at a different time relative to the other emission spectral components; a data acquisition system configured to collect measurement signals provided by the single photon detector array and provide a time resolved image of the sample; a multi-species decoder configured to decode an excitation spectrum, an emission spectrum, and a fluorescence decay curve for each of the fluorescent species based on the time-resolved image of the sample.

[0020] Another subject of the invention is a laser scanning microscope inspection method. The laser scanning microscope inspection method is illuminating the sample with a plurality of pulsed excitation light beams having different spectral components (hereinafter excitation spectral components), the sample including a plurality of fluorescent species; The step of irradiating the sample comprises: Imposing a respective time delay on each excitation spectral component such that each excitation spectral component irradiates the sample at a different time relative to the other excitation spectral components. The laser scanning microscope inspection method further comprises: detecting with a single photon detector array a fluorescent signal emitted by a fluorescent species in the sample, the fluorescent signal comprising different spectral components (hereinafter emission spectral components); The step of detecting a fluorescent signal comprises: Imposing a respective time delay on each emission spectral component such that each emission spectral component arrives at the single-photon detector array at a different time relative to the other emission spectral components. The laser scanning microscope inspection method further comprises: collecting measurement signals provided by the single photon detector array to provide a time resolved image of the sample; and decoding an excitation spectrum, an emission spectrum, and a fluorescence decay curve for each of the fluorescent species based on the time-resolved images of the sample.

[0021] Single photon detectors offer low photon timing jitter (when an element of the detector array receives a photon, a signal is activated with an accuracy of up to 200 ps), short latency (up to 50 MHz, i.e. the element remains blinded for 20 ns after the photon has been collected) and asynchronous readout (i.e. each detector array element is completely independent of the others), allowing access to a wide range of time scales (with the only lower limit being the sub-nanosecond scale).

[0022] The fastest biological and laser scanning processes occur in the microsecond range, while the mean fluorescence lifetime information is in the sub-nanosecond / nanosecond time scale. Therefore, simultaneous multispecies LSM is feasible even with detectors that can only cover time scales up to microseconds, but to realize multispecies ISM based on the mean fluorescence lifetime of the probes, it is necessary to utilize the nanosecond time scale. In short, the time scales from tens of nanoseconds to microseconds are hardly used and could be adopted to transfer additional information (actually in LSM, to improve the SNR).

[0023] Therefore, the idea behind the present invention is to use this available bandwidth to transfer useful information to separate probes in a simultaneous multi-species ISM. Here, in particular, the available bandwidth is used to encode the excitation and emission spectral signatures of the probes. This idea is reflected in the ISM measurements based on TCSPC, where the photon arrival time histogram is used to encode: (i) the mean fluorescence lifetime signature of the probes, as in a typical TCSPC experiment; (ii) the excitation spectral signature of the probes by phase multiplexing the different excitation beams; (iii) the emission spectral signature of the probes by separating in time the different spectral components of the fluorescence signal.

[0024] It should be noted that hyperspectral methods separate the spectral components of the fluorescence signal spatially, not temporally. Once the photon arrival time histograms are recorded, well-established computational techniques such as (blind) linear unmixing, phaser-based unmixing, fitting, deconvolution, and machine learning can be used to decode the probe signatures, resulting in multispecies ISM images. Since all encoding operations are performed using a temporal channel, the proposed approach can use a single detector. Furthermore, we use sub-microsecond time scales for encoding, thus ensuring a simultaneous multispecies approach. It should be noted that the same strategy can be applied to CLSM with typical single-element single-photon detectors, but in this case super-resolved images are not obtained.

[0025] The main advantages that the proposed solution offers compared to existing multispecies imaging approaches are: - Compatible with standard CLSMs and does not require significant changes to the setup: the single photon detector array and the data acquisition system are the only modifications or upgrades required compared to standard CLSM architectures; - compatibility with ISM approaches: this solution enables multispecies imaging while maintaining the improved spatial resolution offered by ISM; -The solution allows the full exploration of the three main probe signatures (excitation spectrum, emission spectrum, fluorescence lifetime) alone or in combination (color-time signature), enabling multi-species imaging in a scalable and simple manner; - Compatible with known spectral unmixing, deconvolution, phase vector, fitting and machine learning algorithms. [Brief description of the drawings]

[0026] Further features and advantages of the invention will be presented in the following detailed description with reference to the accompanying drawings, given by way of non-limiting example only, in which:

[0027] [Figure 1] FIG. 1 plots the absorption spectrum (I), emission spectrum (II), and fluorescence lifetime decay of the ATTO488 and ATTO565 fluorescent probes from Merck KGaA. [Diagram 2] FIG. 2 is a functional diagram of the device according to the invention. [Diagram 3] FIG. 3 shows (I) the pulse-interleaved excitation pattern, (II-III) the color-temporal signatures of the ATTO488 and ATTO565 probes, and (IV) the color-temporal signature given by the combination of the signatures of boxes II and III. [Figure 4] FIG. 4 shows an experimental prototype of a microscope according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] The method according to the present invention can be implemented using a TCSPC-based ISM configuration (as described in M. Castello et al. “A robust and versatile platform for image scanning microscopy enabling super-resolution FLIM” [1]) or a digital heterodyne-based starting microscope architecture.

[0029] In short, the starting microscope is a multi-beam pulsed laser fluorescence confocal microscope with a fully open pinhole, and the traditional single-element detector commonly used in CLSM is replaced by a single-photon detector array (e.g., SPAD array detector), denoted 40 in the figure. The different spectral components defined by the excitation beam are denoted B1 and B2 in Fig. 2 and Fig. 4, and the fluorescence signal is denoted F. To maintain the optical resolution capacity of the system, a telescope (or magnifying lens) ensures a projected detector array size of 1 to 1.5 Airy Units (AU). To perform time-resolved measurements, each element of the detector array is connected to a multi-channel time-labeled DAQ card (or a multi-channel frequency-domain average fluorescence lifetime digital system, or a multi-channel high-speed digitizer), synchronized to the laser beam pulses. To implement the method according to the invention, it is necessary to introduce three main updates to the architecture described above (see Fig. 2 and Fig. 4): - an excitation spectrum encoder, indicated in the figure as 50. This is a system for encoding the excitation spectrum signature of fluorescent probes / species into the time channels of the microscope system. This system acts on the time distribution of the different pulses of the laser excitation beam. In the following, a technical embodiment of the excitation spectrum encoder is described, based on pulse interleaving; - an emission spectrum encoder, shown as 60 in the figure. This is a system that encodes the emission spectrum signature of fluorescent probes / species into a time channel of the microscope. This system, like a spectrometer, resolves the fluorescence signal into its spectral components. However, instead of being spatially separated (i.e. each component is projected onto a different position on a linear array of detectors), they are resolved in time (i.e. each component experiences a different delay before reaching the detector array). In the following, a technical embodiment based on optical delay lines is described; - A multi-species decoder, denoted as 70 in the diagram. This is a system that decodes the excitation spectrum, emission spectrum, mean fluorescence lifetime decay of each fluorescent probe / species, along with the probe concentration in the sample (or rather at each sample position, since the entire pipeline is implemented pixel-by-pixel / position-by-position by scanning the sample with a laser beam).

[0030] Due to the two encoders and the pulsed excitation scheme, each probe is characterized by a specific time signature, which is referred to below as the color-time signature. The single-photon detector array 40 records a linear combination of said signatures, whose coefficients are proportional to the probe concentration. The main task of the decoder 70 is therefore to computationally retrieve the coefficients of said linear combination (and possibly also the color-time signature). In practice, the decoder 70 is implemented using spectral (blind) unmixing, (blind) deconvolution, phase decomposition, fitting (if a signature model is provided) or machine learning algorithms. In the following, the (blind) spectral unmixing approach is described in detail.

[0031] general theory

[0032] The most important properties and conditions to be met for a practical implementation of the proposed method are: _xThe fluorescence signal generated by (t) (i.e., the histogram of photon arrival times) is expressed as the color-time signatures s of the different probes. l (t) and relative contribution / concentration c l,x The point is that it is a linear combination of

number

[0033] Figure 3 shows an example where the excitation beam is composed of two excitation spectral components, denoted as Excit1 and Excit2, whose pulses alternately illuminate the sample with a 12.5 ns interval (Excitation.Delay). The signal emitted by each of the fluorescent probes ATTO488 and ATTO565 is resolved into two emission spectral components, specifically a first spectral component (Excitation1) and a second spectral component (Excitation2), the wavelength of the first spectral component being shorter than the discrimination wavelength determined by the dichroic filter, and the wavelength of the second spectral component being longer than the aforementioned discrimination wavelength and delayed by 6.25 ns from the first emission spectral component (Emission.Delay). Thus, for the ATTO488 and ATTO565 probes, four windows a, b, c, d, e, f, g, h, determined by the excitation pulses and emission delays, respectively, can be identified in the respective time graphs of Boxes II and III. For example, window a of signature 1 is related to the first emission spectral component of ATTO 488 excited by Excitation 1, and in window b said component is added to the second emission spectral component of ATTO 488 excited by Excitation 1. In windows c and d (following the illumination of the sample by excitation component Excitation 2), these components decay without being excited by Excitation 2. Similarly, window e of signature 2 is related to the first emission spectral component of ATTO 565 excited by Excitation 1, and in window 1 of said component is added to the second emission spectral component of ATTO 565 excited by Excitation 1. Said component is small but not zero, since the wavelength of Excitation 1 is slightly included in the absorption spectrum of ATTO 565. The contribution of the first and second spectral components excited by Excitation 2 is added to windows g, h (following the illumination of the sample by excitation component Excitation 2).

[0034] To perform multi-species blind imaging, the objective of the decoder 70 is to obtain, for each sample position x, the concentration c l,x and color-time signature s l (t) is their "mixture" f xStarting from (t), we estimate (see box IV in Figure 3, where we have the sum of the contributions of ATTO488 and ATTO565 in the four time windows of the example above: a+b, b+f, c+g, d+h).

[0035] Time-resolved measurements after time discretization and the time sampling window of photon arrival times

number

number

number

number

number

number

number

number

number

number

number

number

number

[0036] In fluorescence microscopy, samples are usually artificially labeled so that not only is the number of species L known, but also the color-time signature s l (t) can also be individually calibrated and estimated a priori, but it should be noted that the calibrated signature may be highly dependent on the imaging conditions and labeling protocols (and therefore may vary from sample to sample).

[0037] Therefore, the color-time signature s l The most difficult situation is when (t) is not known and the color-time signature s l Both the simplest case, where (t) is known, are considered: if the color time spectrum S is known, then C can be computed from F by linear unmixing, otherwise the problem becomes a linear blind unmixing problem.

[0038] Here we provide an example of how to solve the blind problem. The non-blind problem is obviously a simplification of a more general problem, and is typically solved iteratively (pixel-by-pixel) using Equation 3 instead of Equation 4. Using the matrix notation of Equation 4, the decoding operation, i.e. the unmixing problem, can be written as a problem of minimizing the difference between the measured "mixture" F and the modeled / predicted "mixture" SC. Different formulations of said minimization problem can be obtained based on alternative assumptions, such as the probability distribution of the noise contaminating the measurements (e.g. Gaussian or Poissonian noise), and other assumptions on the color-temporal signatures and intensity values, which are typically introduced through a regularization term in the minimization problem. Here we provide a formulation using the Frobenius norm [2]:

number

[0039] Typical constraints introduced in the minimization problem are normalization and non-negativity of the color-temporal signatures:

number

number

number

number

number

[0040] The absolute intensity of each species l and each pixel x is the sum of the measured non-normalized initial intensities

number

[0041] A typical solution to the linear blind unmixing problem is obtained by an inverse least squares method that minimizes the difference between the measured mixture and the modeled / predicted mixture [3]. Another example of how to convert between blind and non-blind unmixing problems by considering Poisson noise in the measured mixture can be found in [4] (references). It should be noted that the minimization approach described here is only one of several ways to convert the unmixing problem of Eqs. 4 and 5. The unmixing problem can be solved using other computational approaches, for example, phase transformation, fitting, machine learning, etc. It should also be noted that all the above examples are used to solve the unmixing problem in the context of CLSM and only when typical hyperspectral or mean fluorescence lifetime histogram measurements are used.

[0042] In our approach, we measure a specially defined signature, the color-time signature, which encodes all the photophysical characteristics of a fluorescent probe in a time-resolved measurement. Conventional unmixing algorithms are then used to calculate the probe coefficients / concentrations from the measured color-time signature.

[0043] Embodiments with Discrete Emission Spectrum Encoder

[0044] As mentioned above, a first practical embodiment for effectively encoding the excitation and emission spectra of a probe into a photon arrival time histogram is now described. In the excitation spectrum encoder 50, pulse interleaving is used to encode the excitation spectrum. That is, a sequence of J excitation pulses of different wavelengths (λ j exc , j=1,...,J) is performed, which is repeated periodically at a given frequency (see Fig. 3I). In this way, different species / probes in the sample are excited with different probabilities (e l (λ), l=1, ..., L).

[0045] An effective implementation of pulse interleaving relying solely on electronic components can be achieved by using a set of triggerable pulsed diodes, whose pulses can be electronically synchronized with each other with a specific delay. The excitation beam is focused onto the sample and the fluorescence signal is collected by an objective lens and fed to an emission spectrum encoder 60. There, the fluorescence signal is filtered by a series of dichroic mirrors into a spectral window K([λ j em ,λ j+1 em ], j = 1, ..., K + 1). Each window is divided into 10 windows, each containing the emission spectrum of the probe (m l The color-time signature of the lth probe can contain different portions of the fluorescence of the probe depending on the number of components (λ, l=1,...,L). The different components are then guided through optical paths of different lengths (e.g., optical delay lines) to introduce specific time delays. Finally, all components are recombined by a second set of dichroic mirrors and sent back to the single-photon detector array 40. (A simplified example of what this process accomplishes is shown in Boxes II-IV of FIG. 3.) With that particular encoder, the color-time signature of the lth probe is given by:

number

number

[0046] Actually, since the pulse sequence is repeated at a frequency of 1 / T, T is the maximum value of the measured photon arrival time; t j exc < T, ∀j; t k em < T, ∀j. Furthermore, in order to prevent the signature copies of the average fluorescence lifetimes from overlapping, the excitation delay and the emission delay are separated by a time longer than the average lifetime τ l fl of all the probes.

[0047] The sequence of excitation pulses may be arranged to perform Hadamard coding to improve the SNR of the measurement, where the delay t j exc should also be noted that may be equal.

[0048] Embodiment with a continuous emission spectrum encoder

[0049] In this practical alternative embodiment, the emission spectrum encoder 60 changes, but the excitation spectrum encoder 50 does not change. In the continuous encoder, the emission spectrum is not separated into discrete windows, but rather spatially separated by a prism, grating, or other similar device. Each component of the emission spectrum (continuous) follows a different path (linearly longer) that introduces a different delay as a function of wavelength. All components are then spatially recombined and sent to the single-photon detector array 40.

[0050] In this case, if we define a coordinate variation m(t)=m(λ / a) that linearly delays the emission spectrum in time as a function of the time constant a, the color temporal signature is given by:

number

[0051] A quick calculation gives us:

number

[0052] Step-by-step instructions 1. The excitation spectrum encoder 50 provides a precise and known time delay to each spectral component of the excitation beam. 2. The multicolor excitation beam is focused by an objective lens onto a specific sample location, which has been previously labeled with a series of different fluorescent probes in order to target precise substructures. 3. The fluorescent light emitted from the sample is collected by the same objective lens and the fluorescent beam is sent to the emission spectrum of the encoder 60 which imparts a specific time delay to each spectral component of the fluorescent beam. 4. The fluorescence signal is filtered by the excitation light and imaged by a magnifying lens onto a single-photon detector array 40. 5. The output signal from the single photon detector array 40 is acquired by a time resolved acquisition system (such as a TCSPC card) which produces P images, one for each element of the single photon detector array 40. The images have a size of T×X, where X is the total number of spatial locations (i.e. pixels or voxels in case of 3D imaging) and T is the number of channels in the photon arrival time histogram. 6. Fuse the P images by pixel reallocation (or other restoration algorithm) to obtain a T × X time-multiplexed, high SNR, super-resolution ISM image f x Generate. 7. The multispecies decoder 70 solves the linear unmixing problem by inverting the linear system of Equation 3 sequentially (pixel by pixel) or the linear system of Equation 4 globally. Specifically, l,x is the time-resolved ISM image f x are extracted and used to construct the final multispecies super-resolved ISM image.

[0053] A prototype of the microscope (FIG. 4) has been made by the inventors. The values ​​shown in FIG. 4 are given as examples and are not intended to limit the present invention.

[0054] We modified the detector of a conventional confocal laser scanning microscope (CLSM) installed on an optical bench. The conventional instrument is equipped with two excitation laser sources with wavelengths of 485 nm (LDH-PC-485B, PicoQuant) and 560 nm (LDH-D-TA-560, PicoQuant), respectively.

[0055] A pulse-interleaved excitation scheme (excitation spectrum encoder 50) is implemented by commanding the laser with TTL signals from the FPGA control system (repetition rate of 40 MHz (laser period 25 ns), delay between each pulse 12.5 ns).

[0056] The sample is scanned with a laser beam through a pair of galvanometer mirrors (6215HM40B, CTI-Cambridge) and an objective (CFI Plan Apo VC60x oil, Nikon). Fluorescence photons are collected from the same objective, descanned, and filtered with a multiband dichroic mirror (ZT-488-561-640-775, AHF Analysentechnik). The fluorescence signal is then sent to an emission spectrum encoder 60, which consists of a dichroic mirror (low pass filter 575 nm, Edmund Optics) and four mirrors that define a 6.25 ns delay line (corresponding to an optical path length of approximately 1.87 m) for fluorescence wavelengths above 575 nm.

[0057] Finally, the beam is expanded and projected onto an array of SPAD detectors 40. The detector array is arranged in a 5 × 5 array of 25 elements and mounted on a commercially available stand equipped with micrometer screws for fine alignment in three axes. The spatial and temporal performance of the detector array was evaluated, with a temporal resolution of 200 ps and a temporal jitter of 110–160 ps over the active region.

[0058] The detector array is controlled by a conventional dedicated steering board that provides power and electrical signal conditioning, and provides 25 digital output channels (each channel relating to the arrival of a photon at a particular element of the detector array) that are fed into a data acquisition system.

[0059] The data acquisition system was developed using a commercially available FPGA development board (National Instruments USB-7856R) equipped with a Kintex7 FPGA processor and connected to a personal computer. Standard digital pixel / line / frame clock lines are used to synchronize the acquisition system and the microscope control system.

[0060] To calculate the histogram of photon arrival times, frequency domain digital heterodyning is implemented in an FPGA controlled system.

[0061] References [1] M. Castello et al. A robust and versatile platform for image scanning microscopy enabling super-resolution FLIM. Nat. Methods, 16 (2): 175-178, 2019. [2] O. Gutierrez-Navarro et al. Blind end-member and abundance extraction for multispectral fluorescence lifetime imaging microscopy data. IEEE Journal of Biomedical and Health Informatics, 18 (2): 606-617, Mar 2014. [3] M. Dickinson et al. Multispectral imaging and linear unmixing add a whole new dimension to laser scanning fluorescence microscopy. BioTechniques, 31 (6): 1272-1278, Dec 2001. [4] RA Neher et al. Blind source separation techniques for the decomposition of multiply labeled fluorescence images. Biophysical Journal, 96 (9): 3791-3800, May 2009.

Claims

1. 1. A laser scanning microscope configured to illuminate a sample with a plurality of pulsed excitation light beams having different spectral components (B1, B2) (hereinafter excitation spectral components), the sample comprising a plurality of fluorescent species; The microscope is a single-photon detector array (40) configured to detect a fluorescent signal emitted by a sample, the fluorescent signal comprising different spectral components (hereinafter emission spectral components); an excitation spectral encoder (50) configured to impose a respective time delay on each excitation spectral component such that each excitation spectral component irradiates the sample at a different time relative to the other excitation spectral components; an emission spectral encoder (60) configured to impose a respective time delay on each emission spectral component such that each emission spectral component arrives at the single-photon detector array (40) at a different time relative to the other emission spectral components; a data acquisition system configured to collect measurement signals provided by the single-photon detector array (40) and to provide a time-resolved image of the sample, the time-resolved image including, for each pixel or voxel in the image, a histogram of photon arrival times of emission spectral components at the single-photon detector array (40); a multi-species decoder (70) configured to decode an excitation spectrum, an emission spectrum, and a fluorescence decay curve for each of the fluorescent species based on the time-resolved image of the sample; Includes a microscope.

2. the single photon detector array includes an array of elements responsive to a fluorescent signal emitted by a sample, each of the responsive elements being capable of providing a respective time resolved image of the sample, and the data acquisition system is configured to fuse the time resolved images provided by the responsive elements by a reconstruction algorithm to generate a super resolved image of the sample. The microscope according to claim 1.

3. the data acquisition system is synchronized with the pulsed excitation light beam.

3. A microscope according to claim 1 or 2.

4. the excitation spectrum encoder (50) is configured to execute a sequence of periodically repeated excitation pulses at a predetermined frequency, each of the excitation pulses corresponding to one of the excitation spectrum components; 3. A microscope according to claim 1 or 2.

5. The emission spectrum encoder (60) - a splitting means configured to split the fluorescent signal into a plurality of spectral windows, each of the spectral windows including one of the emission spectral components; delay means configured to impose a respective time delay on each of said emission spectral components; recombination means configured to recombine the emission spectral components and retransmit them to the single-photon detector array (40); The microscope of claim 4 .

6. The emission spectrum encoder (60) splitting means configured to spatially separate emission spectral components from said fluorescence signal and to impose a respective time delay on each of said emission spectral components; recombination means configured to recombine and retransmit said emission spectral components to said single photon detector array (40); The microscope of claim 4 .

7. A laser scanning microscope inspection method, comprising: illuminating the sample with a plurality of pulsed excitation light beams containing different spectral components (B1, B2) (hereinafter excitation spectral components), the sample containing a plurality of fluorescent species; Including, The step of irradiating the sample comprises: imposing a respective time delay on each excitation spectral component such that each excitation spectral component irradiates the sample at a different time relative to the other excitation spectral components. Including, The laser scanning microscope inspection method further comprises: detecting a fluorescence signal emitted by the sample with a single-photon detector array (40), the fluorescence signal comprising different spectral components (hereinafter emission spectral components); The step of detecting a fluorescent signal comprises: imposing a respective time delay on each emission spectral component such that each emission spectral component arrives at the single-photon detector array (40) at a different time relative to the other emission spectral components. Including, The laser scanning microscope inspection method further comprises: collecting measurement signals provided by the single-photon detector array (40) to provide a time-resolved image of the sample, the time-resolved image comprising, for each pixel or voxel in the image, a histogram of photon arrival times of emission spectral components at the single-photon detector array (40); decoding an excitation spectrum, an emission spectrum, and a fluorescence decay curve for each of the fluorescent species based on the time-resolved images of the sample; A method comprising:

Images