Image scanning microscope and method

Abstract

An image scanning microscope (100, 200) comprises a detection arrangement (112, 204) comprising a beam splitting element (122) configured to receive detection light (120), to split the detection light (120) into two parts, to direct a first part (124a) of the detection light (120) into a first beam path (126a), and to direct a second part (124b) of the detection light (120) into a second beam path (126b). The first beam path (126a) comprises a spatially resolved detector (128, 202) configured to detect a spatial distribution of the intensity of the first part (124a) of the detection light (120). The second beam path (126b) comprises a spectrally resolved detector (130) configured to detect a spectral information of the second part (124b) of the detection light (120). The image scanning microscope (100, 200) further comprises a controller (116) configured to determine a spatial distribution of at least one fluorophore species in a sample (104) based on the spatial distribution of the intensity of the first part (124a) of the detection light (120) and the spectral information of the second part (124b) of the detection light (120).

Description

[0001] Image scanning microscope and method Technical field The invention relates to an image scanning microscope. The invention further relates to a method for determining a spatial distribution of at least one fluorophore species in a sample. Background Image Scanning Microscopy (ISM) is an advanced fluorescence microscopy technique that improves the spatial resolution and signal-to-noise ratio beyond the capabilities of traditional confocal microscopy. In conventional confocal microscopy, a single point detector, such as a single photomultiplier tube, is used to detect the fluorescent light emitted from the sample. In the ISM approach, the point detector is replaced by a multi-element photodetector comprising a plurality of photodetector elements(pixels) arranged in a photodetector array. Each photodetector element in the array isconfigured to output a detector signal upon receiving fluorescent light. As the sample is scanned with a laser focus, each photodetector element detects a small part of an image of the illuminated sample at each scan position. Appropriate algorithms are then used to combine multiple parts of scan images to reconstruct a single high- resolution image of the sample. While it is possible to use the information from the different photodetector elements to increase spatial image resolution and signal-to-noise ratio, one of the major limitations of current ISM is the limited usability of these photodetectors for quantifying the emitted fluorescence light in terms of its spectral content. Existing solutions, for example as described in F. Strasser et al., Biomed. Opt. Expr 10 (2019) 2513, are limited to a narrow band of the visible light spectrum and suffer from light loss due to the generation of higher diffraction orders and the use of only one polarization state. Summary It is an object to provide a detection arrangement for an image scanning microscope and a method for determining a spatial distribution of at least one fluorophore speciesin a sample that allow the spatial distribution of at least one fluorophore species in asample to be determined better than with known image scanning microscopes or methods. The aforementioned object is achieved by the subject-matter of the independent claims. Advantageous embodiments are defined in the dependent claims and the following description.The proposed image scanning microscope comprises an excitation unit configured togenerate excitation light, and an objective lens directed at a sample space and configured to direct the excitation light into the sample space and to receive detection light from the sample space. The image scanning microscope also comprises a scanning unit arranged along a beam path between the excitation unit and the objective lens and configured to selectively direct the excitation light into different regions of the sample space via the objective lens. The image scanning microscope further comprises a detection arrangement comprising a beam splitting element configured to receive the detection light, to split the detection light into two parts, todirect a first part of the detection light into a first beam path, and to direct a secondpart of the detection light into a second beam path. The first beam path comprises aspatially resolved detector configured to detect a spatial distribution of the intensityof the first part of the detection light. The second beam path comprises a spectrallyresolved detector configured to detect a spectral information of the second part of thedetection light. The image scanning microscope further comprises a controller configured to determine a spatial distribution of at least one fluorophore species in a sample based on the spatial distribution of the intensity of the first part of thedetection light and the spectral information of the second part of the detection light.The present invention might be embodied in a detection arrangement as such which might be used for an image scanning microscope. The detection arrangementcomprises a beam splitting element, a first beam path, a second beam path. The beamsplitting element is configured to split detection light into two parts, to direct a firstpart of the detection light into the first beam path, and to direct a second part of thedetection light into the second beam path. The first beam path comprises a spatiallyresolved detector configured to detect a spatial distribution of the intensity of a first part of the detection light. The second beam path comprises a spectrally resolved detector configured to detect a spectral information of a second part of the detection light. The detection arrangement may further comprise a controller configured todetermine a spatial distribution of at least one fluorophore species in a sample basedon the spatial distribution of the intensity of the first part of the detection light and aspectral information of the second part of the detection light. The spatial distributionof the at least one fluorophore species in the sample may also be determined by anexternal controller based on the spatial distribution of the intensity of the first part of the detection light and a spectral information of the second part of the detection light. Preferably, the controller is configured to determine the spatial distribution of at leastone, preferably of at least two fluorophore species in the sample based on the spatialdistribution of the intensity of the first part of the detection light and the spectralinformation of the second part of the detection light. In an embodiment, the controlleris configured to determine the spatial distribution of a concentration of the at leastone fluorophore species in the sample. The controller may also be configured to determine the presence of the at least one fluorophore species in the sample. Varying environmental conditions may change the emission and / or excitation characteristics of a fluorophore. For example, varying pH levels can shift the emission wavelength of the fluorophore, and changes in temperature can alter its excitationefficiency and the fluorescence lifetime. Thus, in this document, the term fluorophorespecies is used to refer to a set of fluorophores grouped by their emission and / or excitation characteristics. Two different fluorophore species may be two differentfluorophores, or the same fluorophore found in different regions of the sample, whicheach may have different environmental conditions. The fluorophores may be exogenous fluorophores that have been introduced into the sample, and / or endogenous fluorophores which naturally occur in the sample. The sample is imaged with the image scanning microscope by scanning the sample with excitation light focused by the objective lens using the scanning unit. The excitation light excites the fluorophore species in the sample, which causes them to emit the detection light according to their intrinsic properties and the environmental conditions at the location of the fluorophore species in the sample. This detection light is collected by the objective lens and directed into the detection arrangement via thescanning unit, thereby descanning it. The descanned detection light is split into thefirst and second parts by the beam splitting element. The first part is directed into thefirst beam path, where the spatially resolved detector detects the spatial distributionof the intensity of the detection light. As the sample is scanned with the excitation light, at least one spatial distribution is detected at each scan position by the spatially resolved detector. From the collection of the spatial distributions a single high- resolution image of the sample can be reconstructed using algorithms known from Image Scanning Microscopy (ISM). The second part is directed into the second beam path, where the spectrally resolved detector detects the spectral information of the detection light. The proposed image scanning microscope thereby expands upon the ISM approach by not only reconstructing the single high-resolution image from the spatial distributions, but also determining the spectral information of the detection light at each scan position. Thus,the proposed image scanning microscope not only enables image scanning microscopywith high spatial resolution and a high signal-to-noise ratio by detecting the first partof the detection light using the spatially resolved detector. By detecting the spectralinformation of the second part of the detection light at each scan position, the proposed image scanning microscope makes it possible to distinguish multiple different fluorophore species in the sample more robustly. In combination, thisenables the determination of the spatial distribution of the at least one fluorophorespecies in the sample in the high spatial resolution and signal-to-noise ratio provided by the ISM approach. In an embodiment the controller is configured to determine the spatial distribution of the at least one fluorophore species in the sample based on at least one image formation model that parametrizes the imaging behavior of the image scanningmicroscope. The at least one image formation model describes how the spatiallyresolved detector and / or the spectrally resolved detector detect the sample. Amongthe properties that may be described by the at least one image formation model arethe excitation PSF (Point Spread Function), the emission PSF, and the properties of thedetectors, such as individual detector elements, their size, their spatial arrangement,and their spectral sensitivity. The at least one image formation model may be used incombination with a database of different fluorophore species, for example, to determine how a fluorophore species would be detected by the spatially resolved detector and / or by the spectrally resolved detector. To determine which fluorophorespecies are present in the sample, the controller may then, for example, minimize acost function that characterizes a distance between the measured signal and a signalreconstructed or inferred using the database and the at least one image formationmodel. The database and the at least one image formation model may each be stored in a local or remote memory element or in a cloud service.In another embodiment a first image formation model has an arbitrary spatialdistribution of the at least one fluorophore species in the sample as an input and acorresponding expected spatial distribution of the intensity of the detection light asthe output. The first image formation model describes how any arbitrary spatialdistribution of the at least one fluorophore species in the sample is detected by thespatially resolved detector. The controller may determine the spatial distribution ofthe at least one fluorophore species in the sample using the first image formation model by varying the input of the first image formation model until a cost function that characterizes a distance between the measured data and the output of the firstimage formation model is minimized. The first image formation model may be writtenas where ^^(^⃗) is the spatial distribution of a fluorophore species with index ^ which isused as the input of the first image formation model, ^ is the index of a specificillumination sequence, ^^⃗ is a coordinate of the scan position, ^^⃗ is a coordinate on the spatially resolved detector, ^^^is the emission wavelength, ^^^is the excitation wavelength, ℎ^^^is the excitation point spread function, ℎ^^^is the detection point spread function, ^^^^and ^^^are the excitation and emission spectra, respectively,of the fluorophore species ^, ^^^^ is the spectrum of excitation light in theillumination sequence with index ^, ^^^^^^^^ is the spectral sensitivity of the spatiallyresolved detector, ^ℎ(^^⃗) is the detection pinhole function, and ^ is the splitting ratiobetween the first beam path and the second beam path. The first image formationmodel may comprise any or all of the aforementioned terms.In another embodiment a second image formation model has an arbitrary spatialdistribution of the at least one fluorophore species in the sample as an input and acorresponding expected spectral information of the detection light as the output. Thesecond image formation model describes how any arbitrary spatial distribution of the at least one fluorophore species in the sample is detected by the spectrally resolved detector. The controller may determine the spatial distribution of the at least onefluorophore species in the sample using the second image formation model by varyingthe input of the second image formation model until a cost function that characterizesa distance between the measured data and the output of the second image formationmodel is minimized. The second image formation model may be written as where ^^^^^^^^is the spectral sensitivity of the spectrally resolved detector, ^^⃗ is acoordinate on the spectrally resolved detector and the remaining terms are definedabove with reference to the first image formation model. The second image formation model may comprise any or all of the aforementioned terms. The first and second image formation model may further comprise terms describingother system and / or fluorophore characteristics such as a photon arrival time, noisecontributions, in particular introduced by components of the detectors, a samplingrate of the detectors, and the detection pinhole function. Values, parameters, and / ormathematical expressions for each of the terms may be known from literature, systemdesign data and / or be determined in a calibration. The first image formation modeland the second image formation model can be combined by vectorial techniques into a single composite image formation model, allowing for joint processing of common terms and a joint solution of the reconstruction problem.In another embodiment the controller is configured to determine the spatialdistribution of the at least one fluorophore species in the sample using a maximum-likelihood method. Other optimization methods may be used instead. For example,the controller may be configured to determine the spatial distribution of the at least one fluorophore species in the sample by minimizing a cost function which characterizes a distance between the measured data and reconstructed data. In particular, the controller may be configured to combine the spectral information and the spatial distribution of the intensity of the detection light using multiviewdeconvolution, for example. The controller may also be configured to combine theoutput of the first and second image formation models in a similar fashion to comparethe combined sensor data with the output of the image formation models. Thecontroller may further be configured to determine the spatial distribution of the atleast one fluorophore species in the sample by minimizing a cost function that characterizes a distance between the combined measured data and the combined output of the first and second image formation models.In another embodiment the controller is configured to determine the spatialdistribution of the at least one fluorophore species in the sample based on previouslydetermined calibration data. The calibration data describes what measured data isdetected by the image scanning microscope for a specific fluorophore species, forexample. In an embodiment, the controller may be configured to determine the firstand second image formation models based on the calibration data. The calibrationdata may be stored in a local or remote memory element or in a cloud service.In another embodiment the spatially resolved detector comprises a two-dimensionalarray of photodetector elements. Each photodetector element acts as a single pixeldetector that captures a portion of the first part of the detection light at a differentposition in the two-dimensional array. Such an arrangement makes it possible to detect the two-dimensional spatial distribution of the intensity of the detection light. The two-dimensional array of photodetector elements may be a SPAD-array or SiPM-array, for example. SPAD stands for single-photon avalanche diodes and refers to atype of photodetector element characterized by their high sensitivity, their fast timing resolution, and their ability to detect single photons with high efficiency. An advantageof the SPAD-array is its capability for fast single-photon detection and precise time-resolved measurements, making it possible for the SPAD-array to be used as a timeresolved detector element for detecting photon arrival times, for example. SiPMstands for Silicon Photomultiplier, another type of photodetector element, which are based on SPADs. Advantages of SiPMs include a low signal-to-noise ratio, a high gain, a low operating voltage, their compact size, and their robustness. Like the SPAD-array, the SiPM-array enable fast single-photon detection and may be used as the time resolved detector element.In another embodiment the spatially resolved detector comprises an array ofphotodetector elements, and a fiber-bundle configured to guide the second part ofthe detection light onto the array of photodetector elements. The array ofphotodetector elements may be one-dimensional or linear array of photodetectorelements in particular. The photodetector elements may be PMT, SPAD or SiPM, forexample. A first end of the fiber bundle may comprise a two-dimensional array ofoptical fibers. Each optical fiber may be arranged to receive a portion of the first partof the detection light at a different position in the two-dimensional array. A secondend of the fiber bundle may be arranged at the array of photodetector elements suchthat each optical fiber guides a portion of the first part of the detection light onto oneof the photodetector elements. Such an arrangement also enables detecting the two-dimensional spatial distribution of the intensity of the detection light.In another embodiment the spectrally resolved detector comprises at least onespectral encoding element configured to change the spatial distribution of theintensity of the second part of the detection light. The spectrally resolved detectormay further comprise a photodetector element or an array of photodetector elementsarranged downstream of the spectral encoding element and configured to detect aspatial distribution of the intensity of the second part of the detection light. Thespectral encoding element may comprise at least one of a dispersing prism, a planargrating, a volume grating, a grism, a diffractive optical element, and an array ofwavelength selective filters, for example a Bayer-mask. The photodetector elementsmay be PMT, SPAD, or SiPM, for example. The second part of the detection light ismodulated by the spectral encoding element based on the wavelength of the second part of the detection light. For example, the spectral encoding element may be a diffractive element that deflects different wavelengths by a different amount. Themodulated detection light is then received by the array of photodetector elements,which detects the spatial distribution of the intensity of the modulated second part ofthe detection light. Based on the modulation pattern, the spectral information can bedetermined. In an embodiment, the controller is configured to determine the spectral information based on the spatial distribution of the intensity of the second part of the detection light.In another embodiment the spectrally resolved detector comprises a selection unitarranged downstream of the spectral encoding element and configured to selectively block a part of the spatial distribution of the intensity of the modulated second part ofthe detection light from reaching the photodetector element. By selectively blocking,the selection unit determines light of which wavelength range reaches the photodetector element. The spectral information can then be determined based on asetting of the selection unit. Exemplary detection arrangements comprising a selectionunit are described in WO 95 / 07447 A1 and in WO 99 / 39231 A1.In another embodiment at least one of the spatially resolved detector and thespectrally resolved detector is configured to detect a photon arrival time of a photonof the detection light. This enables the image scanning microscope to determine afluorescence lifetime of the at least one fluorophore species, for example, as an additional information. Based on the additional information, the at least onefluorophore species can be reliably identified. The excitation light source may beconfigured to generate the excitation light from pulsed laser light. By recording thearrival times of individual photons with respect to a laser pulse generated by the excitation light source, it is possible to determine fluorescence lifetime characteristics.In another embodiment the detection arrangement comprises a pinhole arrangedupstream of the beam splitting element in a beam path of the detection light. Thepinhole is a spatial filter that excludes out-of-focus light, thereby enhancing theimaging quality. Other spatial filters may be used instead of or in addition to thepinhole, for example a digital mirror device (DMD) or other spatial light modulators(SLM). The pinhole might be arranged in the beam path in accordance with thearrangement of a pinhole of a conventional confocal laser scanning microscope.In another embodiment the image scanning microscope comprises a main beamsplitter configured to direct the excitation light into the objective lens via the scanningunit, and to direct the detection light into the detection arrangement. The main beamsplitter may comprise at least one of an acousto-optical beam splitter, a dichroic beam splitter, an optical filter (in particular an exchangeable optical filter), and a filter wheelor a filter slider. The excitation light may be reflected in the sample space and bypassing the main beam splitter leak into the detection arrangement. The acousto- optical device can be controlled to selectively deflect certain wavelengths orwavelength bands. This property is used to deflect the leaked excitation light awayfrom the detection arrangement. An exemplary main beam splitter comprising anacousto-optical device is disclosed in WO 99 / 42884 A1. The dichroic beam splitter canbe used in similar fashion to deflect the wavelengths of the excitation light away fromthe detection arrangement. The optical filter and the filter wheel may be used to blockthe excitation light from entering the detection arrangement to the same effect.In another embodiment, the beam splitting element is configured to direct a firstpolarized component of the detection light into the first beam path as the first part,and to direct a second polarized component of the detection light into the secondbeam path as the second part. In this embodiment, the detection light is split into thefirst and second parts based on polarization. Assuming unpolarized light, for examplefluorescence, this means that the detection light is split almost evenly. Such an evensplit may also be achieved by employing a neutral beam splitter having areflection / transmission ratio of 1 or close to 1. In another embodiment, thereflection / transmission ratio is chosen such that the number of photons detected perspatial detector element and spectral detector element is essentially equal. Thenumber of photons per spatial detector element can be inferred by the size anddistribution of the detector elements over the Airy disc. The number of photons perspectral detector element can be inferred from the spectral dispersion and samplingof the spectral arrangement together with the emission spectrum of the fluorophores.This evens out the SNR ratios of both beam paths / models.In another embodiment the excitation light source comprises a super-continuum laserand / or multiple single-wavelength lasers. In this embodiment, it is possible todynamically generate laser light with multiple different wavelengths as the excitation light. This makes it possible to adapt the excitation light to the excitation spectra ofmany different fluorophores, making the image scanning microscope very versatile.The invention further relates to a method for determining a spatial distribution of atleast one fluorophore species in a sample. The method comprises the following steps:a) Generating excitation light using an excitation unit. b) Selectively directing the excitation light into different regions of the sample using a main beam splitter, a scanning unit, and an objective lens. c) Receiving detection light from the sample using the objective lens and directing the detection light into a detection arrangement. d)Directing a first part of the detection light into a first beam path, and a second part ofthe detection light into a second beam path using a beam splitting element. e)Detecting a spatial distribution of the intensity of the first part of the detection lightusing a spatially resolved detector. f) Detecting a spectral information of the secondpart of the detection light using a spectrally resolved detector. h) Determining the spatial distribution of the at least one fluorophore species in the sample based on thespatial distribution of the intensity of the first part of the detection light and thespectral information of the second part of the detection light.The method has the same advantages as the detection arrangement, or the imagescanning microscope described above. In particular, the method may be supplemented with the features described in this document in connection with the detection arrangement or the image scanning microscope. Furthermore, the detection arrangement or the image scanning microscope described above may be supplemented with the features described in this document in connection with the method. Short Description of the Figures Hereinafter, specific embodiments are described referring to the drawings, wherein:Figure 1 is a schematic view of an image scanning microscope according to anembodiment;Figure 2 is a schematic view of the image scanning microscope according toanother embodiment;Figure 3 is a flowchart of the method for determining a spatial distribution of atleast one fluorophore species in a sample; andFigure 4 is a flowchart of a calibration that may be performed as part of themethod according to Figure 3. Detailed DescriptionFigure 1 is a schematic view of an image scanning microscope 100 according to anembodiment. The image scanning microscope 100 exemplary comprises a singleobjective lens 102 directed at a sample 104 arranged in a sample space 106. The imagescanning microscope 100 further comprises an excitation unit 108, a scanning unit 110,a detection arrangement 112, a main beam splitter 114, and a controller 116.The excitation unit 108 is configured to generate excitation light 118, for example laserlight with a single wavelength or a narrow wavelength band. For this, the excitationunit 108 may comprise one or more lasers. Each laser may be configured to generatelaser light having one single wavelength or a narrow wavelength band. In anembodiment, the excitation unit 108 may comprise a continuum laser and an arrangement of exchangeable filters or a tunable laser to selectively generateexcitation light 118 with different wavelengths. In another embodiment, the excitationunit 108 may be configured to generate modulated light, for example pulsed light. Theexcitation unit 108 may comprise further optical elements such as lenses andapertures for forming a light beam from the excitation light 118, which are not shown in Figure 1.The excitation light 118 generated by the excitation unit 108 is directed by the mainbeam splitter 114 towards the scanning unit 110. The scanning unit 110 is configuredto deflect the excitation light 118 to selectively direct the excitation light 118 intodifferent regions of the sample space 106 via the objective lens 102, for example in ameandering fashion. This makes it possible to successively illuminate the sample 104at different scan positions using the excitation light 118 focused by the objective lens102. To deflect the excitation light 118, the scanning unit 110 may comprise one ormore galvanometric mirrors or acousto-optic deflectors, for example. The beam pathof the excitation light 118 is shown in Figure 1 using dashed lines originating at theexcitation unit 108 and ending at the sample 104. Using the excitation unit 108, thesample 104 may be illuminated in multiple illumination sequences. Each illuminationsequence may be distinguished from other illumination sequences in at least one excitation modality. The excitation modality may be the spectral composition of the excitation light 118, the intensity of the excitation light 118, or the modulation of the excitation light 118, for example.Detection light 120 is generated by illuminating the sample 104 using the excitationlight 118. In an embodiment, the excitation light 118 excites fluorophores arranged inthe sample 104, which emit fluorescence light as the detection light 120. Depending on the excitation modality of the excitation light 118, different fluorophore species may be excited. For example, only some of the fluorophore species arranged in thesample 104 are excited due to the excitation light 118 having a specific narrowwavelength band. In an embodiment, the sample 104 is illuminated in successiveillumination sequences. During each illumination sequence another set of fluorophorespecies is excited. The detection light 120 emitted by the fluorophore species iscollected by the objective lens 102 and directed back towards the main beam splitter114 via the scanning unit 110. Due to the arrangement of the scanning unit 110between the main beam splitter 114 and the objective lens 102, the deflection of theexcitation light 118 is reversed for the detection light 120. This directs the detectionlight 120 towards a single point regardless of a deflection angle of the scanning unit110. The detection light 120 has been descanned, so to speak. The descanneddetection light 120 is then directed by the main beam splitter 114 into the detectionarrangement 112. The beam path of the detection light 120 is shown in Figure 1 usinga dotted line originating at the sample 104.The detection arrangement 112 comprises a beam splitting element 122 that isconfigured to split the detection light 120 into two parts 124a, 124b. The splitting ratioα between the first and second parts 124a, 124b is determined by the beam splittingelement 122 and the composition of the detection light 120. For example, fluorescence light is generally unpolarized. A polarizing beam splitter will split fluorescence light almost evenly. In another embodiment, a neutral beam splitterhaving a predetermined reflection / transmission ratio may be employed as the beamsplitting element 122. The reflection / transmission ratio determines the splitting ratioin such an embodiment.The first part 124a of the detection light 120 is directed into a first beam path 126a bythe beam splitting element 122. The first beam path 126a comprises a spatiallyresolved detector 128, which is configured to detect a spatial distribution of the firstpart 124a of the detection light 120. In Figure 1, the spatially resolved detector 128 isexemplary formed as a two-dimensional array of photodetector elements. Each photodetector element acts as a single pixel detector that captures a portion of thefirst part 124a of the detection light 120 at a different position in the array, enablingthe spatially resolved detector 128 to capture the spatial distribution of the first part124a of the detection light 120. As different scan positions are illuminated successivelyusing the excitation light 118, at least one spatial distribution is detected at each scanposition. From the collection of the spatial distributions a single high-resolution imageof the sample 104 can be reconstructed. At least one of the photodetector elementsmay be configured to determine a photon arrival time of a photon of the detectionlight 120. For example, said photodetector element configured to detect a photonarrival time may be a single-photon avalanche diode (SPAD) or Silicon Photomultiplier(SiPM).The second part 124b of the detection light 120 is directed into a second beam path126b by the beam splitting element 122. The second beam path 126b comprises aspectrally resolved detector 130, which is configured to detect a spectral informationof the second part 124b of the detection light 120. For example, the spectrally resolveddetector 130 may be configured to detect the presence of at least one wavelength orwavelength band in the second part 124b of the detection light 120. In Figure 1, thespectrally resolved detector 130 comprises a spectral encoding element 132, which isexemplary formed as a dispersing prism. The spectral encoding element 132 isconfigured to change the spatial distribution of the intensity of the second part 124bof the detection light 120. Thereby, the spatial distribution of the intensity of thesecond part 124b of the detection light 120 is modulated. For example, the dispersingprism deflects longer wavelengths at smaller angles and shorter wavelengths at largerangles. The spectrally resolved detector 130 shown in Figure 1 further comprises anarray 134 of photodetector elements, which is arranged downstream of the spectralencoding element 132. The array 134 is configured to detect the modulated spatialdistribution of the intensity of the second part 124b of the detection light 120. Thespectral information can be determined from the modulation. For example, in the case of monochromatic detection light 120, the wavelength of the monochromaticdetection light 120 may be determined from the point of incidence of the second partdetection light 120 on the array 134. Like with the spatially resolved detector 128, atleast one of the photodetector elements of the array 134 may be configured todetermine a photon arrival time of a photon of the detection light 120.The spectrally resolved detector 130 may comprise a spatially resolved detectorelement, for example the array 134 of photodetector elements. However, thespectrally resolved detector 130 itself is spatially integrating, i.e. the spectralinformation is determined by integrating the modulated spatial distribution of theintensity of the second part 124b of the detection light 120.In Figure 1, the detection arrangement 112 further comprises a pinhole 136 arrangedat the conjugate image plane between the main beam splitter 114 and the beamsplitting element 122. The pinhole 136 is a spatial filter which filters out-of-focus light,thereby only allowing detection light 120 originating from the focus region of theobjective lens 102 to pass on into the detection arrangement 112.The controller 116 is configured to control the excitation unit 108, the scanning unit110, the spatially resolved detector 128, and the spectrally resolved detector 130. Thecontroller 116 is further configured to cause the image scanning microscope 100 toperform a method for determining a spatial distribution of at least one fluorophorespecies in the sample 104 based on the detected spatial distribution of the intensityof the first part 124a of the detection light 120 and the spectral informationdetermined from the second part 124b of the detection light 120. The method isdescribed in more detail below with reference to Figures 3 and 4.Figure 2 is a schematic view of the image scanning microscope 200 according toanother embodiment. The image scanning microscope 200 according to Figure 2 isdistinguished form the image scanning microscope 100 according to Figure 1 in thatthe spatially resolved detector 202 of the detection arrangement 204 comprises alinear array 206 of photodetector elements.The linear array 206 is one-dimensional as opposed to the two-dimensional array ofthe spatially resolved detector 128 shown in Figure 1. In order to be able to detect thespatial distribution of the first part 124a of the detection light 120, the spatiallyresolved detector 128 further comprises a fiber-bundle 208 comprising multiple fibers,each fiber acting as a light guide. The fibers at a first end 210 of the fiber-bundle 208are arranged in a two-dimensional array and positioned such, that each fiber capturesa portion of the first part 124a of the detection light 120 at a different position. Thecaptured portions of the first part 124a of the detection light 120 are then guided bythe fibers to a second end 212 of the fiber-bundle 208. The second end 212 of thefiber-bundle 208 is arranged such at the linear array 206 of photodetector elements,that each fiber is connected to one of the photodetector elements. This arrangementmakes it possible to capture the two-dimensional spatial distribution of the first part124a of the detection light 120 using the linear array 206.Figure 3 is a flowchart of the method for determining a spatial distribution of at least one fluorophore species in a sample 104. The method is described as being performedusing the image scanning microscope 100, 200 according to Figure 1 or 2 as an exampleonly. Before the method is started, the sample 104 may be prepared by introducingfluorophores, such as fluorescent dyes, proteins, or quantum dots, into the sample 104. Alternatively, the method may be performed using endogenous fluorophores only. The method is started in step S300. In the optional step S302, a calibration is performed to generate calibration data. The calibration is described in more detailbelow with reference to Figure 4. In step S304, excitation light 118 is generated. Theexcitation light 118 may be pulsed in order to determine a fluorescence lifetime, forexample. In an embodiment, the controller 116 controls the excitation unit 108 togenerate the excitation light 118. In step 306, the excitation light 118 is then directedinto the sample 104. In an embodiment, the excitation light 118 is focused onto a scanposition in the sample 104 using the scanning unit 110 and the objective lens 102. Theexcitation light 118 then excites the fluorophores present in the sample 104, which inturn generate fluorescence light as the detection light 120. Both the emission and excitation characteristics of the fluorophore may vary based on intrinsic properties of the fluorophore, for example on the molecule used and the molecule’s configuration, and on its environmental conditions, such as pH levels and temperature. For example, the same fluorophore may have a different emission spectrum based on the pH level of its immediate surroundings. Thus, the term fluorophore species is used to distinguish groups of fluorophores with different emission and / or excitation characteristics.In step S308, the detection light 120 emitted by the fluorophore species is receivedand split into a first part 124a and a second part 124b. In an embodiment, thedetection light 120 is received by the objective lens 102 and directed into the detectionarrangement 112. The beam splitting element 122 splits the received detection light120 into the first part 124a and the second part 124b. In step S310, a spatialdistribution of the first part 124a of the detection light 120 is detected. The spatialdistribution is a convolution of a (wavelength dependent) point spread function (PSF) that depends on the optical configuration of the imaging system, for example the image scanning microscope 100, 200, and the source of the detection light 120, which may be assumed to be point-like, integrated over the spectral detection range of the spatially resolved detector (see image formation model above). In an embodiment,the spatially resolved detector 128 detects the spatial distribution of the first part 124aof the detection light 120. In step S312 a spectral information of the second part 124bof the detection light 120 is detected. In an embodiment, the spectrally resolveddetector 130 detects the spectral information of the second part 124b of the detectionlight 120.Additionally, in the optional step S314 at least one photon arrival time of the detectionlight 120 is detected. In an embodiment, the photon arrival time of the detection light120 is detected by at least one photodetector element of the spatially resolveddetector 128 and / or the spectrally resolved detector 130. A fluorescence lifetimemeasurement may be performed based on the photon arrival times detected in the step S314 as well as on a modulation of the excitation light 118. In an embodiment,the excitation light 118 may be pulsed. Delays between individual pulses of theexcitation light 118 and the detected photon arrival times are determined. From thedetermined delays, the fluorescence lifetimes may be determined using exponentialfitting of histogram data. The fluorescence lifetime measurement may be performedby the controller 116, for example.The steps S310 to S314 may be performed concurrently or consecutively in any orderand may be repeated until a region of interest of the sample 104 has been scanned.Each time, the excitation light 118 is directed into a different region of the sample 104.Thereby, different scan positions are illuminated with the excitation light 118. In anembodiment, the sample 104 is scanned with the excitation light 118 in a meanderingfashion to illuminate the region of interest. For example, the excitation light 118 isselectively deflected such that it is incident under different angles at an entrance pupilof the objective lens 102, which then focusses the excitation light 118 into the differentregions of the sample 104. As a result, a collection of spatial distributions of theintensity of the first part 124a detection light 120 is obtained, each spatial distributionbeing associated with a specific scan position. Additionally, a collection of spectral information is obtained, each spectral information as well being associated with aspecific scan position. If step S314 is performed, one also obtains a collection ofphoton arrival times or fluorescence lifetimes, each member of the collection beingassociated with a specific scan position. Further, steps S304 to S314 may be repeated, for example using a different excitation modality each time. Each repetition will be called an illumination sequence in thisdocument. By repeating the steps S304 to S314, one obtains multiple collections ofthe spatial distributions, spectral information, and photon arrival times, if applicable,each being associated with a different illumination sequence. For example, in a firstmeasurement, the region of interest is scanned using excitation light 118 according toa first excitation modality, such as a first wavelength range. In a second measurement,the region of interest is scanned again using excitation light 118 according to a secondexcitation modality, such as a second wavelength range different from the first wavelength range. In step S316 the spatial distribution of the at least one fluorophore species in thesample 104 is determined, which will be denoted by where ^^ is an index ofthe fluorophore species and ^^⃗ is a spatial coordinate in the sample 104. Thedetermination is based on the detected spatial distributions of the intensity of the first part 124a of the detection light 120. A single spatial distribution will be called^^(^^⃗, ^^⃗ , ^)^^^^^^^ in the following, where ^ is an index denoting the illuminationsequence, ^^⃗ is a coordinate of the scan position, ^^⃗ is a coordinate on the spatiallyresolved detector 128, and ^ is a coordinate on the time axis that is used when at leastone photon arrival time has been determined in step S314. The determination isfurther based on the detected spectral information of the second part 124b of thedetection light 120. A single spectral information will be called ^^(^^⃗, ^^^ , ^)^^^^^^^^ ,where ^ is the index denoting the illumination sequence, ^^⃗ is the coordinate of thescan position, ^^^ is the emission wavelength, and ^ is the coordinate on the time axis.Both the spatial distribution of the intensity of the first part 124a of the detection light120 and the spectral information of the second part 124b of the detection light 120may be modelled using the image formation models described in this document. Another exemplary image formation model may be found in Hung, Shih-Te, Kalisvaart, Dylan, and Smith, Carlas. "Image scanning microscopy: a vectorial physical opticsanalysis." Optics Express, 18 July 2023. The image formation models may be used todetermine the spatial distribution of the fluorophore species with index ^^ inthe sample 104. Each of the image formation models describes how an arbitraryspatial distribution of at least one fluorophore species is imaged, for example by the image scanning microscope 100, 200.The output of the first image formation model will be called ^^(^^⃗, ^^⃗, ^)^^^^^^^ inorder to distinguish the output from the spatial distribution ^^(^^⃗, ^^⃗ , ^)^^^^^^^ of thefirst part 124a of the detection light 120 measured by the spatially resolved detector128. The output of the first image formation model may be written as The output of the first image formation model ^^(^^⃗, ^^⃗ , ^)^^^^^^^ may be written as alinear superposition of fingerprints ^^^^^(^^⃗ , ^^⃗ , ^) which is spatially convolved withthe spatial distribution of the respective fluorophore species ^^ in the sample104. The symbol ∗^^⃗denotes the partial convolution for the coordinate ^^⃗ of the scan position, and is defined as where is the set of all variables of the functions ^ and ^except ^^⃗. The output set of variables is the union ^ = ^^ ∪ ^^. A fingerprint^^^^^(^^⃗, ^^⃗ , ^) is a dataset that represents a system response for the respectivefluorophore species ^^and can be obtained by modelling and / or during the calibrationstep S302. Each fingerprint ^^^^^(^^⃗ , ^^⃗, ^) describes the expected spatial distributionof the first part 124a of the detection light 120 for a point-like source of the respectivefluorophore species ^^and includes the optical effects of the imaging system, for example the image scanning microscope 100, 200, assuming an imaging from thesample 104 plane to the detector plane, for example, the effect of the excitation PSF,the emission PSF, and various aberrations introduced by the imaging system. Thefingerprints ^^^^^(^^⃗ , ^^⃗, ^) may further include at least one of the following: effectsregarding the fluorescent lifetime of the fluorophore species ^^, which can be derived from the photon arrival times, and effects regarding the excitation spectrum of the fluorophore species ^^, which can be derived from the response for different excitation modalities if the steps S304 to S314 have been repeated for different excitationmodalities in different illumination sequences ^.For the first image formation model, the fingerprint ^^^^^(^^⃗, ^^⃗ , ^) may be written as using where ^^^^(λ^^, ^) is the spectro temporal distribution of the excitation light in theillumination sequence ^ and ^^^(^) describes the temporal emission behavior of thefluorescent species. The total spectro-temporal emission light distribution is where ^^^^^^^^(λ^^, ^) is a term describing the optical effects of filters or acousto-optical elements arranged between the sample 104 and the spatially resolved detector128 and the effect of potential temporal gating or temporal weighting.A second image formation models describes how an arbitrary spatial distribution of the at least one fluorophore species is detected by the spectrally resolved detector130. The output of the second image formation model will be called^^(^^⃗ , λ^^, ^)^^^^^^^^ in order to distinguish the output from the spectral information^^(^^⃗, λ^^ , ^)^^^^^^^^ of the second part 124b of the detection light 120 measured bythe spectrally resolved detector 130. The output of the second image formation modelmay be written as For the second image formation model, each fingerprint will be called ^^^^^(^^⃗, λ^^ , ^)and may further include the optical effects of the spectral encoding element 132, and effects regarding the emission spectrum of the fluorophore species ^^, which areencoded into the spatial distribution of the modulated second part 124b of thedetection light 120 by the spectral encoding element 132. For the second imageformation model, the fingerprint ^^^^^(^^⃗ , λ^^, ^) may be written as using the total spectro-temporal emission light distribution defined as where ^^^^^^^^(λ^^, ^) is a term describing the optical effects of filters arrangedbetween the sample 104 and the spectrally resolved detector 130 and the effect ofpotential temporal gating, temporal weighting, and / or an arrival time processing likegating at the detector 130.If the fingerprints of the different fluorophore species ^^ in the sample 104 are known,the spatial distribution of the fluorophore species ^^ in the sample 104 may bedetermined using the maximum likelihood estimation (MLE) method or equivalently the negative-log-likelihood (NLL) minimization methods. In either case, the determination may be carried out by minimizing a cost function ^, which characterizes a distance between the measured data and the output of the first image formationmodel or the second image formation model respectively. For example, thedetermination of the spatial distribution of the different fluorophore^^ in the sample 104 may be written as using the first image formation model. Assuming Gaussian noise in the measured data, it is appropriate to take the (squared) ^^-norm as the cost function: In the case of assumed Gaussian noise, the spatial distribution of the fluorophorespecies ^^can be estimated by the least squares method. If instead Poisson noise is assumed in the measured data, the appropriate cost function is the Kullback-Leibler- divergence, which leads to an iterative Richardson-Lucy-like estimation algorithm that is described, for example, in Richardson, William Hadley: “Bayesian-Based Iterative Method of Image Restoration”, 1970-09-15, JOSA. Alternative solutions can be foundin S Bonettini et al: Inverse Problems, 2009, 25 015002, in H. Wang et al, IEEE TransImage Proc, 2014, 23848, M. Guo et al., Nature Biotechnology 2020, 381337, and the references cited therein.Likewise, the spatial distribution of the different fluorophore species ^^ in thesample 104 may be determined using the second image formation model using, for example, the following cost function: The spatial distribution ^^ ^^⃗ , ^)^^^^^^^ of the first part 124a of the detection light120 measured by the spatially resolved detector 128 and the spectral information^^(^^⃗, λ^^ , ^)^^^^^^^^ of the second part 124b of the detection light 120 measured bythe spectrally resolved detector 130 may be combined into a fused signal for ^^ = 2. Likewise, the output ^^(^^⃗ , ^^⃗, ^)^^^^^^^ of the first image formation modeland the output ^^(^^⃗, λ^^ , ^)^^^^^^^^ of the second image formation model may becombined into a fused output ^^^^(^^⃗, ^^⃗ , λ^^ , ^). The spatial distribution ofthe different fluorophore species ^^ in the sample 104 may then be determined byminimizing a cost function ^, which characterizes a distance between the fused signal^^^^(^^⃗ , ^^⃗, λ^^, ^) and the fused output ^^^^(^^⃗, ^^⃗ , λ^^ , ^), for example In case the fingerprints are not known, they have to be modelled and / or determined in the calibration described below with reference to Figure 4. It is also possible toestimate unknown emission spectra ^^^^ and / or unknown temporal responses^^^(^) of the fluorophore species ^^, for example, by minimizing the cost function ^and using the image formation model with the set of all unknown emission spectra ^^ = … , ^^^^(^^^)^ andunknown temporal responses ^ = … , ^^^(^)^.Step S316 may be performed by the controller 116, for example. The method is then ended in step S318. Figure 4 is a flowchart of the calibration that may be performed as part of the method according to Figure 3. Before the calibration and the method according to Figure 3 are performed, one or more reference samples may be prepared such that the spatialdistribution of the fluorophore species ^^ in the reference sample or samplesare known. Such a reference sample is also known as a technical sample, and may comprise beads, lines, a checker-board pattern, or a similar arrangement of thefluorophore species ^^. Alternatively, or additionally, the sample 104 itself may beprepared to contain one or more regions which comprise only one fluorophore species ^^each. Alternatively, two or more reference samples may be prepared, such that each reference sample contains a single fluorophore species with an unknown spatial distribution. The calibration is started in step S400. In step S402 the calibration data is gathered byimaging the reference sample, and / or the regions of the sample 104 which compriseonly one fluorophore species ^^each using the imaging system that will be used for the actual measurement, i.e. steps S304 to S316, or an imaging system that is equivalent with respect to the measurement, for example the same model of imaging system.Then in step S404, the fingerprints of the fluorophore species ^^ are determined, whichare present in the reference sample, or the specially prepared regions of the sample104. If the spatial distributions are known, the fingerprint used in the firstimage formation model ^^^^^(^^⃗, ^^⃗ , ^) can be determined by minimizing the costfunction ^, for example If the spatial distribution is not known, it may be estimated as well Likewise, the fingerprints ^^^^^(^^⃗, λ^^ , ^) of second image formation model may bedetermined. The calibration is then ended in step S306.If the fingerprints can neither be modelled nor determined from the calibration data, a blind reconstruction is also possible. This can, for example, be carried out with an expansion of the methods introduced in Neher, Richard A., Miso Mitkovski, Frank Kirchhoff, Erwin Neher, Fabian J. Theis, and André Zeug. "Blind source separation techniques for the decomposition of multiply labeled fluorescence images." Biophysical Journal, vol.96, no.9, 6 May 2009, pp.3791-3800. Identical or similarly acting elements are designated with the same reference signs in all Figures. As used herein the term “and / or” includes any and all combinations of one or more of the associated listed items and may be abbreviated as Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0002] Reference signs 100 Image scanning microscope102 Objective lens104 Sample106 Sample space108 Excitation unit110 Scanning unit112 Detection arrangement114 Main beam splitter116 Controller118 Excitation light120 Detection light122 Beam splitting element124a, 124b Part of the detection light126a, 126b Beam path128 Spatially resolved detector130 Spectrally resolved detector132 Spectral encoding element134 Array136 Pinhole200 Image scanning microscope202 Spatially resolved detector204 Detection arrangement206 Array208 Fiber-bundle210, 212 End

Claims

Claims1. An image scanning microscope (100, 200), comprisingan excitation unit (108) configured to generate excitation light (118);an objective lens (102) directed at a sample space (106) and configured to directthe excitation light (118) into the sample space (106) and to receive detectionlight (120) from the sample space (106);a scanning unit (110) arranged along a beam path between the excitation unit(108) and the objective lens (102) and configured to selectively direct theexcitation light (118) into different regions of the sample space (106) via theobjective lens (102); adetection arrangement (112, 204) comprising a beam splitting element (122)configured to receive the detection light (120), to split the detection light (120) into two parts, to direct a first part (124a) of the detection light (120) into a firstbeam path (126a), and to direct a second part (124b) of the detection light (120)into a second beam path (126b), wherein the first beam path (126a) comprises a spatially resolved detector (128,202) configured to detect a spatial distribution of the intensity of the first part(124a) of the detection light (120), and the second beam path (126b) comprisesa spectrally resolved detector (130) configured to detect a spectral informationof the second part (124b) of the detection light (120); anda controller (116) configured to determine a spatial distribution of at least onefluorophore species in a sample (104) based on the spatial distribution of theintensity of the first part (124a) of the detection light (120) and the spectralinformation of the second part (124b) of the detection light (120).

2. The image scanning microscope (100, 200) according to claim 1, wherein thecontroller (116) is configured to determine the spatial distribution of the at leastone fluorophore species in the sample (104) based on at least one imageformation model that parametrizes the imaging behavior of the image scanning microscope (100, 200).

3. The image scanning microscope (100, 200) according to claim 2, wherein a firstimage formation model has an arbitrary spatial distribution of the at least onefluorophore species in the sample (104) as an input and a correspondingexpected spatial distribution of the intensity of the detection light (120) as theoutput.

4. The image scanning microscope (100, 200) according to claim 2 or 3, wherein asecond image formation model has an arbitrary spatial distribution of the atleast one fluorophore species in the sample (104) as an input and acorresponding expected spectral information of the detection light (120) as theoutput.

5. The image scanning microscope (100, 200) according to any one of thepreceding claims, wherein the controller (116) is configured to determine thespatial distribution of the at least one fluorophore species in the sample (104)using a maximum-likelihood method.

6. The image scanning microscope (100, 200) according to any one of thepreceding claims, wherein the controller (116) is configured to determine thespatial distribution of the at least one fluorophore species in the sample (104)based on previously determined calibration data.

7. The image scanning microscope (100) according to any one of the precedingclaims, wherein the spatially resolved detector (128) comprises a two-dimensional array of photodetector elements.

8. The image scanning microscope (200) according to any one of the claims 1 to 6,wherein the spatially resolved detector (202) comprises an array (206) ofphotodetector elements, and a fiber-bundle (208) configured to guide thesecond part (124b) of the detection light (120) onto the array (206) ofphotodetector elements.

9. The image scanning microscope (100, 200) according to any one of thepreceding claims, wherein the spectrally resolved detector (130) comprises atleast one spectral encoding element (132) configured to change the spatialdistribution of the intensity of the second part (124b) of the detection light(120), and aphotodetector element or an array (134) of photodetector elements arrangeddownstream of the spectral encoding element (132) and configured to detect aspatial distribution of the intensity of the second part (124b) of the detectionlight (120).

10. The image scanning microscope (100, 200) according to any one of thepreceding claims, wherein at least one of the spatially resolved detector (128, 202) and the spectrally resolved detector (130) is configured to detect a photonarrival time of a photon of the detection light (120).

11. The image scanning microscope (100, 200) according to any one of thepreceding claims, wherein the detection arrangement (112, 204) comprises apinhole (136) arranged upstream of the beam splitting element (122) in a beampath of the detection light (120).

12. The optical scanning microscope according to any one of the preceding claims,comprising a main beam splitter (114) configured to direct the excitation light(118) into the objective lens (102) via the scanning unit (110), and to direct thedetection light (120) into the detection arrangement (112, 204).

13. The optical scanning microscope according to any one of the preceding claims,wherein the excitation unit (108) comprises a super-continuum laser and / ormultiple single-wavelength lasers.

14. A method for determining a spatial distribution of at least one fluorophorespecies in a sample (104), the method comprising: a) generating excitation light (118) using an excitation unit (108);b) selectively directing the excitation light (118) into different regions of thesample (104) using a main beam splitter (114), a scanning unit (110), and anobjective lens (102); c) receiving detection light (120) from the sample (104) using the objective lens(102) and directing the detection light (120) into a detection arrangement (112,204); d) directing a first part (124a) of the detection light (120) into a first beam path(126a), and a second part (124b) of the detection light (120) into a second beampath (126b) using a beam splitting element (122);e) detecting a spatial distribution of the intensity of the first part (124a) of thedetection light (120) using a spatially resolved detector (128, 202);f) detecting a spectral information of the second part (124b) of the detectionlight (120) using a spectrally resolved detector (130); andh) determining the spatial distribution of the at least one fluorophore speciesin the sample (104) based on the spatial distribution of the intensity of the firstpart (124a) of the detection light (120) and the spectral information of thesecond part (124b) of the detection light (120).

Images