Method of detecting emitted light, detection device and laser scanning microscope

Abstract

The invention relates to a method for detecting emitted light, in particular fluorescence from at least one fluorescent dye, in a laser scanning microscope, wherein the emitted light from the sample is guided using imaging optics onto a two-dimensional matrix sensor having a plurality of pixels in an image plane, wherein a distribution function of detection points is detected using the matrix sensor in a spatially oversampled manner. The method is characterized in that the emitted light from the sample is spectrally resolved using a dispersive device, in particular in the dispersion direction, the spectrally resolved emitted light is detected in a spectrally resolved manner using the matrix sensor, and in evaluating the intensities measured by the pixels of a pixel region, the spectral separation is cancelled at least for some of the pixels. In a further aspect, the invention relates to a detection device for detecting emitted light in a spectrally resolved manner in a laser scanning microscope, and to a laser scanning microscope.

Description

Technical Field

[0001] In a first aspect, the present invention relates to a method for detecting emitted light, particularly fluorescence from at least one fluorescent dye, in a laser scanning microscope. In another aspect, the present invention relates to a detection device for detecting emitted light in a laser scanning microscope and a laser scanning microscope. Background Technology

[0002] Methods for detecting emitted light, particularly fluorescence from at least one fluorescent dye, in laser scanning microscopy, according to the relevant category, are disclosed, for example, in M. Castello et al., “Image Scanning Microscopy with Single-Photon Detector Array,” bioRxiv, doi: http: / / dx.doi.org / 10.1101 / 335596 (hereinafter referred to as: [Castell et al., 2019]). Here, the emitted light from the sample is guided onto a two-dimensional matrix sensor located in the image plane using imaging optics. This two-dimensional matrix sensor has multiple pixels, and the distribution function of the detection points is detected using the matrix sensor in a spatially oversampled manner.

[0003] Also disclosed in [Castell et al., 2019] is a genus-based detection device for detecting emitted light in a laser scanning microscope. The genus-based detection device has a two-dimensional matrix sensor with multiple pixels in the image plane for detecting the detection point distribution function of emitted light from the sample in a spatially oversampled manner, and has imaging optics for guiding the emitted light onto the two-dimensional matrix sensor.

[0004] Also described in [Castello et al., 2019] is a type of laser scanning microscope, which has the following components: a light source, in particular a laser, for emitting excitation light; an excitation optical path with microscope objectives for guiding the excitation light onto or into the sample to be examined; a scanning device in the excitation optical path for scanning at least one illumination spot on the sample; a probe optical path for guiding emitted light, in particular fluorescence, emitted from the sample onto a probe unit; a master dichroist for separating the excitation light and the emitted light; a probe unit for detecting the emitted light; and a control and evaluation unit, in particular a PC, for driving the light source and for evaluating the measurement data obtained from the probe unit.

[0005] In biomedical research, confocal laser scanning microscopy (LSM) has established itself as a powerful tool over the past few decades because it supports numerous image-based, correlated, or statistical experiments and analyses, in addition to providing pure imaging of fluorescent biological samples. A key reason for this capability is that LSM can simultaneously measure multiple dyes with relatively modest instrumentation costs. Several solutions for this are known. For example, a dichroic filter can be used to split the light beam into different fractions, which are then fed separately, for example, to a photomultiplier tube acting as a sensor. P hotomultiplier T ube - PMT). Components are more flexible, utilizing gratings or prisms to spectrally disperse the emitted light so that the spectrum can then be detected using a line sensor. Undesired spectral ranges can be flexibly and dye-dependently blocked using a movable aperture in front of the sensor. Furthermore, solutions are known where, for example, specularly reflective apertures are used to define spectral bands, which are then fed to different PMTs. Here and below, the term "dye" includes not only synthetic dyes but also fluorescent proteins. Additionally, intrinsically emitting structures should also be included. For example, many biological structures will emit light when illuminated by laser light of a specific wavelength; this is, for example, referred to as autofluorescence.

[0006] While theoretically described approximately 30 years ago, the innovation of a confocal microscope system commercialized using Zeiss's LSM880 was the so-called image scanning microscopy (ISM; http: / / www.gattaquant.com / files / en_wp_airyscan-detector.pdf). It is based on measuring the point distribution function (PSF) of the probe using a camera-like sensor in an oversampling manner, achieving confocal resolution limits despite a pinhole opening, and significantly improving system sensitivity through inherent parallelization of the probe. However, the solution implemented in the LSM980 today is extremely expensive due to the fiber-based image conversion (from 2D to 1D) combined with the GaAs P-PM-line detector. This severely limits the number of pixels that can be used for oversampling the PSF; in this case, it is exactly 32 pixels. Therefore, these sensors can only be meaningfully used in components that define spectral channels using dichroic filters, and in these components, optical zoom is also required to guide the PSF, which depends on the selected objective (more precisely, the optical conductivity and NA / M ratio) and wavelength, to the sensor. Furthermore, the unit cost of this detector technology makes it nearly impossible to install more than one detector per instrument.

[0007] In terms of sensors, in recent years there has been a rise in so-called SPAD arrays or SPAD cameras (SPAD = S ingle- p hoton a valanche dRapid development has occurred in the field of SPAD (single-photon avalanche diode). In principle, these cameras can achieve unique activation of pixels, where each pixel is represented by a separate single-photon avalanche diode. Furthermore, pixels can operate in so-called Geiger mode, allowing photon counting on a surface sensor. This makes the readout signal instantly digitized, enabling extremely high frame rates in the range of approximately 1 MHz. Moreover, SPAD cameras do not have readout noise. Readout noise occurs particularly in other sensors, such as sCMOS sensors or CCDs, and increases sharply with high readout rates. EM-CCDs employ an amplification mechanism close to the sensor to amplify the signal beyond the readout noise, and thus can theoretically be single-photon sensitive. However, this introduces significant amplification noise (also known as excess noise; multiplication noise), which halves the effective sensitivity of the sensor. Furthermore, it fundamentally limits the achievable speed with higher pixel counts. sCMOS cameras achieve lower readout noise in the range of 0.3e, which in principle allows for photon counting using these cameras.

[0008] Because the frame rate is in the MHz range, SPAD cameras have become very useful as sensors for LSM, with pixel dwell times in the 1μs range. For example, using such a camera, 156 kfps (fps = ...) was achieved with a sensor resolution of 512 × 128 pixels and a data depth of 1 bit. f rames p er s (econd, frames per second). This equates to a data rate of approximately 10 Gbit / s. This means that a line sensor with approximately 1000×6 pixels should even be able to achieve a frame rate of 1 Mfps with a data depth of 1 bit. Currently, these limitations are not fundamentally at their limit, but rather technically determined by the enormous data rates. With a data depth of only one bit, the achievable speed is roughly equivalent to the reciprocal of the dead time, and therefore falls within the 10 MHz range. However, for the example above, this would equate to a data rate of 100 Gbit / s. Summary of the Invention

[0009] One objective of this invention is to provide a method of the type described above, which can be used in a particularly diverse manner. Furthermore, a suitable detection device and a laser scanning microscope should also be described.

[0010] This task is accomplished by the method described in this invention, the detection device described in this invention, and the laser scanning microscope described in this invention.

[0011] Advantageous variations of the method according to the invention, as well as preferred embodiments of the detection device according to the invention and the microscope according to the invention, will be described below, especially in conjunction with the accompanying drawings.

[0012] According to the present invention, the method of the above type is improved by using a dispersive device to perform spectral decomposition of the emitted light from the sample, especially in the dispersive direction, using a matrix sensor to perform spectrally resolved detection of the spectrally decomposed emitted light, and canceling spectral separation for at least some of these pixels when evaluating the intensity measured by the pixels of the pixel region.

[0013] According to the present invention, the detection device of the above type is improved in that it includes a dispersive device for spectrally separating the emitted light, a matrix sensor is provided and positioned for detecting the spectrally separated emitted light in a spectrally resolved manner, and an evaluation electronic device connected to the matrix sensor is provided, which is configured to cancel spectral separation for the pixels when evaluating the intensity measured by the pixels in the pixel region.

[0014] According to the present invention, the laser scanning microscope of the above type is improved in such a way that the detection unit has the detection device according to the present invention.

[0015] The basic idea of ​​this invention can be considered as follows: spectral separation is performed on the emitted light from the sample, so that the spectral portions of different fluorescent dyes, in principle, first strike different, spatially separated regions of the matrix sensor. These regions are identified and each is associated with a fluorescent dye. Then, for at least some pixels in each region, the spectral separation is calculated backwards, thereby allowing the point distribution function to be determined for each fluorescent dye, in principle, as with known ISMs. Therefore, the point distribution function can be measured for different fluorescent dyes. In any case, spectral separation is suitably performed for those pixels where a significant intensity is measured.

[0016] A key advantage of this invention is that it achieves both the advantages of spectral flexibility and the effects in terms of resolution and sensitivity achieved through the oversampling point distribution function.

[0017] Therefore, the method, the detection device, and the microscope according to the invention can also achieve a signal distribution that makes the ISM exhibit spectral dispersion. Furthermore, the detection device and the microscope according to the invention are characterized by achieving particularly high optical efficiency and overall stable arrangement.

[0018] The detection device according to the invention is particularly suitable for performing the method according to the invention.

[0019] The term "detector point distribution function" refers to the intensity distribution produced by a point-emitting object on the detector in the sample plane.

[0020] Pixel regions belonging to different dyes may, in principle, overlap on the detector. Therefore, it is important to ensure at least some spatial separation of the pixel regions. Clearly, the clearer the separation of pixel regions, or the greater the difference in the spectral characteristics of their respective dyes, the easier and better the evaluation will be.

[0021] The laser scanning microscope according to the invention, and especially the control and evaluation unit, can be configured together with the detection device to perform the method according to the invention.

[0022] The following section will explain some examples of how computational backtracking for spectral separation can be performed.

[0023] In a particularly preferred variant of the method, at least one pixel region assigned to the emission of a dye is identified based on the spectrum measured using a matrix sensor.

[0024] Suitablely, the spectral intensity distribution of the emitted light on the matrix sensor is first obtained. Then, based on this intensity distribution, pixel regions can be identified, for example. Furthermore, this intensity distribution can be used for computational backtracking of spectral separation. In a particularly preferred variant of the method according to the invention, to determine the spectral intensity distribution of the emitted light on the matrix sensor, intensity values ​​belonging to a specific wavelength are determined by summing measurement data of multiple pixels in a column of the matrix sensor perpendicular to the dispersion direction, particularly the measurement data of all pixels in that column of the matrix sensor.

[0025] These data concerning the spectral intensity distribution of emitted light can be automatically retrieved, for example, after changes in the measurement environment. Changes in the measurement environment are considered to be changes in the sample under test, and if necessary, changes in the position of the sample under test, which may be prepared using different dyes.

[0026] Additionally, it can be configured that data relating to the spectral distribution are continuously and, in particular, automatically obtained from a specified number of measurements, especially those just preceding them. This variant has the advantage that it eliminates the need for a separate data acquisition initiation.

[0027] It should be understood that for all the evaluations described herein, multiple measurements are summed to achieve a better signal-to-noise ratio. If necessary, the number of measurements averaged to obtain the desired data can vary.

[0028] In another preferred variant of the method according to the invention, then, in order to identify pixel regions, a maximum and minimum quantity are automatically searched in the known spectral distribution, and spectral boundaries for calculating the point distribution function of a determined dye can be suggested to the user based on the found maximum and minimum quantities. Alternatively, the spectral boundaries can be automatically defined based on the found maximum and minimum quantities. Thus, in the microscope according to the invention, the control and evaluation unit can be configured to search for maximum and minimum quantities in the known spectral distribution and to suggest spectral boundaries for calculating the point distribution function of a determined dye based on the found maximum and minimum quantities. Alternatively, the control and evaluation unit can be configured to autonomously define spectral boundaries for calculating the point distribution function of a determined dye based on the found maximum and minimum quantities.

[0029] The method and microscope according to the invention can also be used to image and measure samples of dyes or fluorescent proteins with spectral overlap.

[0030] In a preferred variant of the method according to the invention, pixel regions are overlapped on a matrix sensor, and spectral unmixing is performed on the intensities measured by each pixel. In principle, methods of spectral unmixing are known.

[0031] In addition to the methods for canceling spectral splitting described herein, such methods for spectral unmixing (“spectral unmixing”) may also be performed before or after canceling spectral splitting.

[0032] For example, dispersion can first be canceled for a freely chosen number of spectral bands (typically two regions with pure emission and one region with overlapping emission in the case of two dyes), then an image scanning microscope can be applied, and then the channels (three in this example) can be demixed.

[0033] This method allows the relative proportions of specific spectral components within a pixel to be determined. This means, for example, according to specifications for pixel redistribution (see below), shifting only the weighted portion of the pixel's intensity, without shifting the entire intensity. The components according to the invention can be used to acquire the corresponding reference spectrum. Then, in the next step, for example, row-by-row spectral demixing can be performed, and the respective spectral weights determined, thereby obtaining the weighted portion of the spectrum. Besides the spectral demixing method, other measures for separating spectral components include PCS, SCA, and the use of "deep learning."

[0034] The method according to the invention can be used in particular to determine the detector point distribution function for at least one fluorescent dye. However, a particular advantage of the invention is that it is also possible to determine the detector point distribution function for multiple dyes with different emission spectra from measurement data of a single measurement.

[0035] If the size of the matrix sensor used, specifically the number of pixels in the matrix sensor, allows, then multi-point variations of the method according to the invention are also possible in principle. Here, emitted light from multiple points simultaneously illuminating the sample with excitation light is simultaneously guided to the matrix sensor and evaluated. The excitation and detection optical paths need to be configured for multi-spot excitation and multi-spot detection.

[0036] In a first preferred variant of canceling spectral separation for each pixel in a pixel region, the intensity values ​​measured by these pixels cancel each other out, taking into account the spectral intensity distribution of the emitted light of the dye belonging to that pixel region and the spatial intensity distribution of each spectral portion on the matrix sensor.

[0037] Therefore, the evaluation electronics of the detection device according to the present invention can be configured to cancel spectral separation by causing the intensity values ​​measured by the pixels of the pixel region to cancel each other, taking into account the spectral intensity distribution of the emitted light of the dye belonging to the pixel region and the spatial intensity distribution of each spectral part on the matrix sensor.

[0038] The spatial intensity distribution of each spectral part determines the degree of spatial overlap of the spectral parts of the dye's point distribution function that are relatively displaced on the matrix sensor in the dispersion direction.

[0039] The following intensity distribution can be used here as the spatial intensity distribution of each spectral part: this intensity distribution is measured from a column of pixels perpendicular to the dispersion direction, especially from the column where the highest intensity is measured in its respective pixel region. This is based on the assumption that the distribution function of the detection points on the matrix sensor is rotationally symmetric, i.e., circular. This is a good assumption if rotationally symmetric optics are used.

[0040] In another important set of method variants, spectral separation is canceled for individual pixels within a pixel region by pixel reallocation. This aspect is known from image scanning microscopy (ISM) for non-spectrally resolved methods, see, for example, Castello et al.

[2019] .

[0041] Therefore, the evaluation electronics of the detection device according to the invention can be configured to assign intensity values ​​measured by pixels to positions in the image plane that have been shifted relative to their respective pixels (pixel reallocation) to cancel spectral separation for each pixel in the pixel region, wherein the shift vector depends on the position of the respective pixel and on the wavelength belonging to that position.

[0042] In these variations of the method, intensity values ​​measured by pixels are assigned to positions in the image plane that have been shifted relative to their respective pixels (pixel reallocation) to cancel spectral separation for individual pixels within a pixel region. Similar to known pixel reallocations, the shift vector here depends on the position of the respective pixel, but additionally also on the wavelength belonging to that position.

[0043] Specifically, a shift vector is determined for each pixel for pixel reallocation. This shift vector depends on the position of the relevant pixel and the wavelength belonging to that pixel. Then, the intensity value measured for that relevant pixel is assigned to the position relative to that pixel that has been shifted by the shift vector.

[0044] The wavelength-independent portion of the shift vector for a given pixel can be obtained, for example, by scaling the vector components of the vector from the reference pixel to the relevant pixel using a redistribution factor. Assuming the point distribution functions used for excitation and emission are identical (ignoring Stokes offset), a redistribution factor of -1 / 2 will be obtained, meaning the intensity value measured from the given pixel will be assigned to a position in the image plane exactly midway between the reference pixel and the relevant pixel.

[0045] In particular, pixel redistribution is performed such that the resulting detector point distribution function has essentially the same shape in the dispersion direction as it does perpendicular to the dispersion direction. This is based on the assumption that, when using rotationally symmetric optics, the detector point distribution function must have a circularly symmetric intensity distribution.

[0046] However, there are other methods to obtain the corresponding shift vectors. For example, shift vectors belonging to the wavelength range associated with the sample structure can be determined by evaluating the phase correlation of multiple scan images.

[0047] In the detection device according to the invention, a matrix sensor can be used as an analog integration detector and / or a photon counting detector. Preferably, an sCMOS camera, an EMCCD camera, and / or a charge integration sensor is used. Particularly advantageously, the matrix sensor has a SPAD camera, or a SPAD camera is used as the matrix sensor. Particularly preferably, the matrix sensor, especially the SPAD camera, operates in photon counting mode, wherein individual photons can be counted. This mode, also known as the "Geiger mode," is characterized by a particularly advantageous signal-to-noise ratio.

[0048] Recently, the cost of the sensors used has been greatly reduced, which is particularly advantageous to this invention. It would be difficult to imagine the components described herein being coupled to optical fibers using conventional PMT technology.

[0049] In particular, in the detection device according to the invention, optical diffraction devices and / or optical refraction devices can be used as dispersive devices. Particularly preferred is that the dispersive device has a grating, especially a scribed grating, and / or a prism. In principle, a grating, i.e., a combination of a prism and a grating, can also be used. Adjustable or controllable elements, such as a DMD (DMD = ... D igital M icromirror D device, digital micromirror device), MEMS (MEMS = m icro e electron- m echanical s Microelectromechanical systems (MEMS) or SLM (SLM = ...) S patial L ight M It is also possible to use a modulator (German: Flächenlichtmodulator, spatial light modulator).

[0050] In a particularly preferred variant of the detection device according to the invention, the dispersion direction is aligned with, and in particular parallel to, the coordinate direction of the matrix sensor. For example, pixels along the dispersion direction may be referred to as pixel rows, while pixels perpendicular to the dispersion direction may be referred to as pixel columns. The matrix sensor is oriented such that the dispersion direction is parallel to the direction of the pixel rows. This has the advantage that pixels in a pixel column belong to exactly the same wavelength range, or simply put, to exactly the same wavelength.

[0051] Regarding the specification of the matrix sensor and / or the optical imaging on the matrix sensor, it is preferable that the pixel pitch of the matrix sensor (in principle, a grating constant) is chosen to be greater than the variation of the Airy disk width of the detection point distribution function in the plane of the matrix sensor across the spectral bandwidth of the dye. Furthermore, it is preferable that the matrix sensor and / or the optical imaging on the matrix sensor are specified such that the spectral bandwidth of each pixel of the matrix sensor in the dispersion direction is less than 0.5 nm, preferably less than 0.4 nm, and particularly preferably less than 0.3 nm. These designs enable simplified computational assumptions in calculating the point distribution function, specifically in performing discrete deconvolution. More precisely, it can be well approximated that the width of the Airy disk is substantially independent of wavelength in the spectral direction of the dye. Nevertheless, sufficient spectral resolution is achieved.

[0052] Particularly preferably, the optical imaging on the matrix sensor and / or the matrix sensor can also be sized such that the diameter of the Airy disk of the detection point distribution function on the plane of the matrix sensor is less than 20 times the grating constant of the matrix sensor, particularly preferably less than 7 times the grating constant, and even more preferably less than 5 times the grating constant. In this sizing, the calculation of discrete deconvolution can be advantageously limited to a relatively small number of wavelengths, i.e., limited to the number corresponding to the number of pixels covered by the respective Airy disk.

[0053] Preferably, the diameter of the Airy disk on the matrix sensor can be greater than three times the grating constant, so that the method equivalent to the final stage of ISM can be performed.

[0054] SPAD array sensors are particularly suitable for measuring fluorescence lifetime information (FLIM = Fluorescence Lifetime Imaging Microscopy) using TCSPC (time-correlated single photon counting) or "histogram" or "time window detection." This requires pulsed excitation and a suitably configured sensor. This sensor, and its combination with the Airyscan method, is described, for example, in NatMethods 16, 175-178 (2019). https: / / doi.org / 10.1038 / s41592-018-0291-9.

[0055] Furthermore, this invention enables spectral resolution. This combination is indeed very attractive.

[0056] In a preferred variant of the method according to the invention, a matrix sensor is used, for example, some pixels of the matrix sensor, and in particular each single pixel of the matrix sensor, to perform a time-resolved measurement for determining the fluorescence lifetime of the dye.

[0057] An advantageous improvement to the detection device according to the invention is characterized in that the matrix sensor and evaluation electronics, especially for determining the fluorescence lifetime of the dye, are configured to perform time-resolved measurements, for example, using some pixels of the matrix sensor, preferably using each single pixel of the matrix sensor.

[0058] Therefore, the matrix sensor and electronics are designed for this purpose to enable the determination of fluorescence lifetime, making it possible to perform time-resolved measurements, preferably using each pixel. Thus, these variations of the invention enable the combination of spectrally resolved FLIM with image scanning. Suitably, pulsed lasers are used for these measurements.

[0059] To improve the detection efficiency of the matrix sensor, a microlens can be arranged in front of the matrix sensor, that is, upstream of the matrix sensor.

[0060] In order to change the imaging of the emitted light, especially to scale the spectral bandwidth of each pixel, the imaging optics of the detection device according to the present invention can have a zoom system.

[0061] Finally, in the microscope according to the invention, a device for masking the excitation light can be present in a manner known in principle, particularly at least one emission filter. Preferably, a conversion device with multiple emission filters, such as a filter wheel, can be present. However, the detection device according to the invention can also achieve variations in which undesired components of the emitted light spectrum are not evaluated. For this purpose, for example, the corresponding pixels, especially pixel columns, of the matrix sensor can be made passive or set to inactive. Attached Figure Description

[0062] Further advantages and features of the method according to the invention, the detection device according to the invention, and the laser scanning microscope according to the invention are explained below with reference to the accompanying drawings. Wherein:

[0063] Figure 1 : A schematic view showing a laser scanning microscope according to the present invention;

[0064] Figure 2 : A schematic view showing the detection device according to the present invention;

[0065] Figure 3 : A graph illustrating the correlation between the diameter and wavelength of the Airy disk for different numerical apertures;

[0066] Figure 4 : A graph illustrating the variation of dispersion angle versus wavelength in the case of a diffraction grating;

[0067] Figure 5 : A graph showing the emission spectra of a typical dye;

[0068] Figure 6 : A graph illustrating the variation of the detection point distribution function with respect to multiple pixels of the matrix sensor in a direction perpendicular to the dispersion direction;

[0069] Figure 7 : Showing a simulated image ( Figure 7 a), showing a graph illustrating the relationship between the count rate and the pixel number in the pixel column ( Figure 7 b) and a graph illustrating the relationship between the count rate and the pixel number in the dispersion direction ( Figure 7 c)

[0070] Figure 8 : Shows an image accumulated from many simulated images ( Figure 8 a), showing a graph illustrating the relationship between the averaged count rate and the pixel number in the pixel column ( Figure 8 (b), and a graph illustrating the relationship between the averaged count rate and the pixel number in the dispersion direction ( Figure 8 b)

[0071] Figure 9 : This shows an image accumulated from numerous simulated images in the presence of two dyes in the sample. Figure 9 a), showing a graph illustrating the average count rate versus pixel numbering in the dispersion direction for two dyes ( Figure 9 b). A graph illustrating the relationship between the averaged count rate and the pixel number in the dispersion direction is shown, where the spectrum is limited to the emission region of the dye ( Figure 9 c), and a graph is shown illustrating the relationship between the averaged count rate and the pixel numbers in the pixel columns that have the highest emission levels for each of the two dyes. Figure 9 d);

[0072] Figure 10 This diagram illustrates a matrix sensor without spectral resolution, used to explain the principle of pixel redistribution.

[0073] Figure 11 A schematic diagram of a matrix sensor for a detection device according to the present invention is shown, used to explain the pixel region and the basic objective of the invention; and

[0074] Figure 12: A schematic diagram of a matrix sensor used in a detection device according to the present invention is shown to explain the principle of pixel redistribution under spectral resolution.

[0075] Identical components that have the same function are usually labeled with the same reference numerals in the drawings. Detailed Implementation

[0076] Figure 1 A schematic diagram of a laser scanning microscope 100 according to the present invention is shown. As main components, the microscope 100 firstly includes: a light source 12, particularly a laser, for emitting excitation light 14; an excitation optical path 10 having a microscope objective 24 for guiding the excitation light onto or into the sample S to be examined; and a scanning device 22 in the excitation optical path 10 for scanning at least one illumination spot 27 on the sample S. To guide the emitted light 28, particularly fluorescence, emitted from the sample S due to the application of excitation light 14 to a detection unit 32, a detection optical path 30 is also provided, which has a detection unit 32 for detecting the emitted light 28. A master dichroist 18, present according to the invention, is used to separate the excitation light 14 and the emitted light 28. Finally, a control and evaluation unit 34, particularly a PC, is provided for controlling the light source 12 and evaluating the measurement data obtained by the detection unit 32. According to the invention, the detection unit 32 includes a detection device 200 according to the invention.

[0077] Excitation light 14 emitted by light source 12 reaches the main dichroistor 18 via deflector 16, and is guided towards scanning device 22 at the main dichroistor. Scanning device 22 can be arranged on a plane optically conjugate with the rear pupil of microscope objective 24. Excitation light 14 travels from scanning device 22 through scanning objective and microscope tube lens to microscope objective 24, which focuses the excitation light 14 onto sample S or illumination spot 27 in sample plane 26. The scanning objective and microscope tube lens... Figure 1 The middle part is schematically shown as component 23.

[0078] An excitation light 14 is applied to a region of sample S, which then emits emission light 28, typically from the fluorescence of a dye prepared beforehand for sample S. The emitted light 28 then returns to the main dichroist 18 via a descanned probe path 30, following the same path as the excitation light 14, but is transmitted through the main dichroist and then reaches the detector unit 32, which has a detection device 200 according to the invention. The data measured by the detector unit 32 is evaluated by a control and evaluation unit 34. Furthermore, the control and evaluation unit 34 can be used to drive the light source 12 and the scanning device 22.

[0079] The microscope 100 according to the present invention is particularly a confocal laser scanning microscope.

[0080] Figure 2 a) A schematic diagram of an embodiment of the detection device 200 according to the present invention is shown. As a main component, the detection device 200 includes, for spectrally resolved detection of the emitted light 28 in a laser scanning microscope 100, a dispersive device 40 for spectrally separating the emitted light 28 from the sample under test S, a two-dimensional matrix sensor 50 for position-resolved detection of the spectrally decomposed emitted light, and an imaging optics 48 for guiding the spectrally decomposed emitted light onto the two-dimensional matrix sensor 50.

[0081] exist Figure 2 In the embodiment shown in a), the dispersive device 40 is a grating 43, more precisely a reflective grating, wherein the emitted light 28 to be detected is incident obliquely onto the grating 43 from below. A prism or prism grating may also be used instead of the grating 43. Crucially, the signal is defined on the matrix sensor 50 by a spectral pattern that can, in principle, be linear or non-linear in terms of wavelength distribution with respect to position. It only needs to be monotonic. In the illustrated embodiment, the grating 43's scribe lines are perpendicular to the drawing plane, which results in (in Figure 2 (a) Spectral splitting along the vertically extending dispersion direction 41. Light with zero diffraction order is indicated by reference numeral 29. Light with zero diffraction order can either be omitted or, if necessary, redirected along the incident light onto the grating 43 with rotational polarization to improve detection efficiency.

[0082] The emitted light 28, filtered through a pinhole (not shown) conjugate filter, is collimated by an optical device (also not shown) and directed toward the dispersive device 40.

[0083] Then, each spectral component 42, 44, 46, 47 collided with ( Figure 2 (A) The imaging optics 48 is schematically shown as a single lens. In principle, the imaging optics 48 may also have multiple beam-shaping components, and in particular, a zoom system.

[0084] Imaging optics 48 focuses different spectral components 42, 44, 46, and 47 onto a matrix sensor 50 having multiple pixels 51. The matrix sensor 50 may be, for example, a SPAD multi-line camera, which has, for example, 5 rows, each with 800 pixels, equivalent to having 800 columns.

[0085] Figure 2 b) A schematic diagram of the surface of the matrix sensor 50 is shown. The matrix sensor 50 is positioned relative to the grating 43 such that the dispersion direction 41 extends along the row direction, i.e., parallel to... Figure 2 The coordinate system shown in b) extends along the x-direction. The column direction extends parallel to the y-direction. For example, in Figure 2 In (b), the blue spectral portion may be located at the left edge of the spectrum, while the red spectral portion may correspondingly be located at the right edge. (Footnote) and This represents the columns or rows of the matrix sensor 50, that is, the pixels ( ) is the first In the middle of the line Pixels in the column.

[0086] Figure 2 (a) schematically illustrates an evaluation electronics 60 according to the invention, which is connected to a matrix sensor 50 and configured to evaluate pixels in a pixel region ( The measured intensity Cancel targeting these pixels ( Spectral separation. These functions will be explained in detail below.

[0087] The evaluation electronics 60, which may include a data grabber, can also be implemented, for example, in an FPGA, a microcontroller, or a similar component. It is important to place it as close to the hardware as possible, i.e., as close as possible to the matrix sensor 50, to reduce the amount of data, thereby allowing data to flow as continuously as possible from the evaluation electronics 60 to the control PC, especially to the system according to the invention. Figure 1 The control and evaluation unit 34 of the microscope 100.

[0088] Figure 2 The situation in a) can also be described as follows: the dispersive device 40 produces a large, defined chromatic aberration in the dispersive direction 41, thereby separating different wavelengths and allowing them to be detected separately. In the direction perpendicular to this, i.e., in… Figure 2 In b, the point distribution function remains essentially unchanged in the y-direction. However, the spectral dispersion of the emitted light 28 disrupts the point distribution function, making it unclear how ISM imaging should be achieved using such a system.

[0089] To achieve this, the specifications of the imaging optics 48 relative to the matrix sensor 50, i.e., relative to the SPAD camera in this example, are first determined. For example, the matrix sensor 50 may have a pixel pitch, which may also be referred to as the grating constant of the matrix sensor 50, and the pixel pitch is a = 25 μm.

[0090] As is well known, the diameter of an Airy disk is related to the wavelength of light and the numerical aperture, as follows:

[0091]

[0092] To evaluate the signal for ISM, oversampling of the point distribution function for at least three to four pixels in each spatial direction is required. To achieve this in the relevant wavelength range of 450 nm to 800 nm, a numerical aperture of approximately NA = 0.01 on the probe side is needed when illuminating the sensor. This can be achieved from... Figure 3 The chart shows the correlation between the diameter of the Airy disk and the wavelength at different numerical apertures. Figure 3 The grid on the y-axis in the chart here corresponds to the assumed pixel size of the SPAD camera as 25μm.

[0093] Figure 4 The diagram shows the variation of the dispersion angle in degrees with respect to wavelength in nm for a 1000 lines / mm grating. For example, assuming a focal length of f = 50mm for the imaging optics 48 between the grating 43 and the SPAD camera, this would result in a spectral distribution over approximately 730 pixels within the stated spectral range. Therefore, the spectral bandwidth deposited on pixel 51 is approximately... (pixel) = 0.5nm.

[0094] In principle, numerical integration must be performed on different spectral components in order to recover the point image distribution function from the dispersive and blurred signal distribution. If the pixel intensity is obtained in this way, the next step is to use an image scanning microscope (ISM) method.

[0095] The intensity incident on a specific pixel 51 is used This indicates that, among them, footnotes This indicates the column number, meaning it extends along the x-direction and the dispersion direction 41. (Footnote) This indicates the row number and therefore extends in the y direction.

[0096] strength It can be written as:

[0097]

[0098] Here, It is the emission spectrum of an excited fluorescent dye. It is the Airy function, which is equivalent to the spatial intensity distribution of the point distribution function, where the wavelength in the Airy function is... The following parameters represent the parameters that specifically determine the Airy function. width, This represents the center of the point distribution function. Then, through dispersion... And calibrated to the center of the column by optical components. This defines the center of the point distribution function, i.e., the point of maximum intensity.

[0099] Typically, the spectral bandwidth of common fluorescent dyes (phosphors) is approximately [missing information]. (See Figure 5 ).according to Figure 3 In the graph, the width variation of the Airy function over this bandwidth is significantly less than 10 μm. Therefore, if the width of the spectral emission center is set to [value missing] for the point distribution function... This would result in an error of at most one-quarter of a pixel, especially occurring in the weak emission edge regions of the spectrum, and thus contributing relatively little. Therefore, it can be achieved through... Good approximation Therefore, the PSF term along the y-axis will become independent of... And can be found from about Extracted from the integral of coordinates.

[0100] Now, only a one-dimensional deconvolution problem remains in order to know the spectrum of the excited fluorescent dye. In this case, the spectral tail in the x-direction is calculated back to the undisturbed Airy function. The computational and hardware costs should not be underestimated here, as a signal of at least 4 × 730 pixels must be evaluated.

[0101] To further reduce the computational load on the evaluation of electronic device 60, the following further assumptions can be made. First, it is assumed that the undisturbed point distribution function is radially symmetric. This means that the spatial extent of the point distribution function along the dispersion direction 41 of the spectrum, i.e., in the x-direction, can be derived from the intensity distribution that can be read orthogonally to the x-direction, i.e., in the y-direction. This was learned from [the source]. Furthermore, [it possesses / has / is] The spectral bandwidth of each pixel (< 0.5 nm) is very small, so the dispersion signal distortion due to pixel spread can be neglected well, and therefore it can be assumed that the spectrum of each pixel is piecewise constant. Furthermore, the width of the integration range, i.e., the number of pixels that must be integrated, can be obtained from the intensity distribution determined in the orthogonal direction, i.e., the y-direction. According to... Figure 3 The graph clearly shows that integrating over five pixels is perfectly sufficient. Therefore, the following discretization of the problem is obtained:

[0102]

[0103] Here, It is the intensity of the discretized fluorescent dye at the spectral location of the relevant pixel. What is applicable now is to determine the Airy function centered on adjacent pixels and its relationship to the pixel (...). Intensity measured at ) The proportion.

[0104] Therefore, approximate values ​​can also be obtained from data measured in the orthogonal direction, i.e., in the y-direction.

[0105] Advantageously, the sum of signals from all pixels perpendicular to the dispersion direction (i.e., all pixels in a column) is first calculated. This yields the emitted spectrum, which can be normalized to obtain... Figure 5 The distribution shown is used to determine the discrete values ​​of the spectral distribution needed to calculate the discrete integral. . Figure 5 An example is shown of the first pixel ( ) at the first wavelength The spectrum below The value and equivalent wavelength The adjacent pixels Spectrum The value. If the spectral distribution bulges once, then the adjacent pixels... , , , For pixels The contribution can be determined based on the intensity proportion along the y-direction. Figure 6 In the example of the spatial distribution of the Airy function shown, the spectral values... The five pixels in this case will have the following intensity distribution.

[0106]

[0107] Figure 6 It also shows the application of spectral positioning place That is, the pixel number for SPAD cameras. Adjacent pixels The point distribution function is equivalent to Figure 5 The lower values ​​of the spectrum.

[0108] Using this obtained value, we can also calculate the remaining integral. Discretize. Then, for each pixel ( Intensity measured in) The following expression is obtained:

[0109]

[0110] Here, It is a pixel ( (at the adjacent pixels) The known part is the strength of the Airy function centered at the center. All contributions are included In the middle. The final step is to rearrange the intensity content of the pixels to maintain the shift extended by the summation sign. Therefore, what is ultimately of interest is not a particular pixel ( The signal is not the same as the sum of the correlated sampled parts of the point distribution function of the fluorescent dye. The new foot alignment is physically equivalent to shifting the emission point distribution function of the spectral dispersion to a common reference location, just as it can be achieved, for example, by passing through a dispersive prism a second time in the optical path. Therefore, the following expression for the point distribution function, known only from the measured data, is obtained:

[0111]

[0112] In matrix sensors used for photon counting in Geiger mode The photons of each sensor pixel in the detection matrix are as follows: Sub-pixels of the PSF to be determined are assigned based on the pre-calibrated ratio distribution of the PSF at a given sensor location:

[0113]

[0114] Therefore, the spectral splitting of pixels belonging to the pixel region of the dye under consideration is cancelled according to the present invention.

[0115] If the error caused by signal trailing due to neglecting dispersion seems too large, it can be addressed by using a larger sensor array and imaging optics with a larger focal length. For example, using an array of 4×1400 pixels with a focal length of f = 100 mm, the bandwidth of each pixel is already [missing information]. Therefore, a reasonable operating point can be found under any circumstances. The matrix sensor 50 does not need to be limited to a few rows. If a high frame rate can be achieved, significantly more image rows are also allowed, thereby enabling multi-point excitation measurements. Such parallelization is in line with the method according to the invention because, at a given frame rate, the pixel dwell time can be extended by the factor of parallelization.

[0116] The problem with applying the above principle lies in the limitation on the number of photons in the LSM. The matrix sensor 50 provides a digital signal, i.e., photons for each sensor pixel and the image. The single image from the matrix sensor 50 is later converted into pixels of the entire LSM image. Since the photon flux hitting the matrix sensor 50 can, for example, be on the order of several megahertz, and the pixel dwell time of the LSM should be on the order of μs, only a few photons are allocated to, for example, 4 × 730 pixels per pixel dwell time. Therefore, most pixels of the matrix sensor 50 will provide zero as a reference, while only a few pixels will provide one. Therefore, a direct one-time readout of the matrix sensor 50 cannot yet provide usable results. Furthermore, photon emission and detection are statistical events, making it impossible to infer the long-term average photon distribution from a single image with such a small number of photons. This is in Figure 7 As explained in a), there is an exposure time of 1 μs for an exposure at a photon flux of Iphot = 10 MHz. Figure 5 The dye spectrum was simulated using a single image from a SPAD camera. The spectral bandwidth per pixel was 0.5 nm. Only a few events were observed, grouped around the dye's maximum emission at approximately 520 nm and along the center line of the five-row SPAD camera. Figure 7 b) shows the sum of signals along the image rows (in the x direction) of the SPAD camera, reflecting the point distribution function. Figure 7 c) shows the sum of count events for all pixels in a column (i.e., in the y-direction), consistent with the detected dye spectrum. Clearly, the algorithm explained above cannot yet be meaningfully used... Figure 7 The data.

[0117] Therefore, the system must be calibrated using integrated images. However, this is in principle a very fast process and usually does not require more than one image scan. Figure 8 Section a) describes the averaging of 1000 single images with an average photon flux of 10 MHz. The integration time is only 1 ms, which is roughly equivalent to scanning one image line. Figure 8 The point distribution function determined by the summed signal in b) almost completely reflects the expected intensity distribution on the five pixels mentioned above, and Figure 8 The spectrum in c) can also be reproduced.

[0118] from Figure 8 b) and Figure 8 c) measurements can obtain and The data, for example, can be stored in the memory of the evaluation electronics 60. The SPAD image of each scanned image pixel can then be evaluated according to the specifications (Formula 5). Integer photon values ​​can be converted to floating-point values ​​to obtain the brightness information.

[0119] According to the present invention, a second possibility for canceling spectral separation of pixels in pixel regions assigned to dyes is based on a pixel reassignment method. (See also...) Figures 10 to 12 Describe it.

[0120] First, combined Figure 10 The methods of pixel redistribution, which are known in themselves, are explained, with the expression of technical terms based on [Castello et al., 2019].

[0121] Consider the case of image scanning, where a point light source scans the sample, and the intensity of radiation returning from the sample is measured with resolution using a two-dimensional confocal detector array positioned relative to the sample. A confocal matrix sensor can have... take 1 pixel ( ), its footnotes From 1 to or .

[0122] Since the spectral measurements on the rectangular array are expected to be evaluated later, it is assumed here that the array is rectangular for simplicity. However, this is not a limitation in itself, and pixelated sensors of other geometries can also be considered in principle. In particular, hexagonal arrangements are often used because they can achieve good fill factor.

[0123] Then, for clarity, we assume a magnification of 1 from the object plane to the intermediate image plane. Furthermore, it is meaningfully assumed that each pixel of the sensor is less than 1 airy unit (AU). This is equivalent to the standard assumption in image scanning that the PSF is probed in a spatially oversampled manner, and each pixel represents an effective pinhole of less than 1 AU (or less than 0.8 AU, or even better, less than 0.3 AU), as mentioned above. In principle, this arrangement also works for PSFs smaller than the pixel size. However, in this case, only the spectrum can be measured and spectral confocal images captured, thus image scanning cannot be meaningfully used.

[0124] Image scanning microscopy can be considered a linear and spatially invariant system, meaning that the sample plane is linearly imaged onto the image plane 56 of the matrix sensor. Utilizing variables... This is intended to represent the position projected back into the sample plane from the image plane 56, i.e., the plane of the matrix sensor. If the scan positioning is located at the position within the sample... In other words, if the maximum amount of illumination intensity is located at a certain position in the sample... So, using pixels ( ) measured intensity It can then be described as a convolution as follows:

[0125]

[0126] Here, It is the effective PSF of the sensor pixels considered separately. It is the activated PSF and the launched PSF The product of . Excited PSF In principle, it can be measured and can be assumed to be known. To obtain the correct detection PSF, Even when the size of a single pixel is not negligible, convolution still needs to be performed using an aperture function (Blendenfunktion) that describes the geometry of the pixel. It is a vector located in the plane of the detector array, which corresponds to a reference element, such as the element located at the center of the sensor and the pixel. The offset between ). It can be described as follows:

[0127]

[0128] In a simplified consideration, we can assume that the effective PSF The maximum value is approximately located in the function and The middle of the maximum values, approximately at the location Specifically, if the excitation PSF and the emission PSF are the same, this will apply to aberration-free systems, such as fluorescence without Stokes shift. The basic idea of ​​pixel redistribution is to assume that by utilizing pixels ( Most of the measured intensity originates from locations in the sample where the position coordinates of the corresponding pixels in image plane 56 are inconsistent. In a simplified consideration, where it is assumed that the maximum value of the effective PSF lies within the function... maximum quantity and The maximum value in the middle is then used with pixels ( The measured intensity originates from a location within the sample plane. The location.

[0129] The basic idea behind pixel reconfiguration and image scanning is to shift the signals that have been shifted relative to the reference pixel back towards the reference pixel and then add them together. In principle, this is intuitively clear because in a confocal system operating in this way, each pixel of the matrix sensor ( Both provide shifted images. In addition to shifting back, images can also be simply recorded on top of each other, or other forms of cancellation can be used, such as multi-view deconvolution, to advantageously cancel out the signals of all pixels.

[0130] Figure 10 The diagram illustrates the situation and representation of confocal image scanning using a matrix sensor 54 with 5×5 pixels. The middle pixel ( () is used as a reference pixel. Circle 57 represents pixel () The effective PSF's center of gravity. However, the range of the PSF is usually much larger than this circle 57, which is in Figure 10 The image is shown only to illustrate the location of the maximum PSF value.

[0131] It is well known that image scanning can achieve better optical resolution and improved signal-to-noise ratio (SNR). Color correlation in image scanning can be achieved via filters. Furthermore, it is known to use a grating to measure and evaluate the photometric fraction (PSF) containing two colors.

[0132] However, it is clear that measuring the spectrum directly at multiple points would yield significantly more information. This can also be used for spectral unmixing of the data. Additionally, it is hoped that the positive properties of image scanning can be utilized even in this measurement scenario.

[0133] The measurement can be performed, for example, using the detection device according to the invention described herein, such that the spectral portions of a defined emission band corresponding to a pixel region on the matrix sensor 50 are spectrally combined in such a way that a so-called image scanning (also known as Airy scanning or optical redistribution) can be performed on that spectral band. This combination is referred to as cancellation of spectral separation in the technical terminology of this invention.

[0134] Therefore, in order to achieve a combination of pixels belonging to a single emission band, it is necessary to eliminate the dispersive effects of gratings or other dispersive devices on a coherent number of N pixels, i.e., on pixels within a pixel region.

[0135] This is ultimately analogous to the situation where light is returned through a dispersive element, thus canceling the dispersion. This results in a PSF containing only spatial information, in which the spectral components are again spatially superimposed as a point image function. For example, some are described in US2019258041. However, it is not always possible to cancel dispersion using optical means. For example, gratings are preferred for spectral splitting because they produce linear dispersion. However, it is disadvantageous to use gratings multiple times—in both the path and the return of light—due to limited efficiency. Furthermore, purely optical components used to cancel dispersion are complex, expensive, and require extensive calibration.

[0136] In the case of linear dispersion, such as that produced by a grating, it is suitable for use along the longitudinal direction of the matrix sensor. The wavelength association relationship is a correlation. Here This is a proportionality constant, which is related to the intensity of dispersion, i.e., to the linewidth of the grating. Therefore, its unit is nm / mm. In principle, these considerations apply not only to gratings but also to prisms as dispersive elements. However, the relationship between the position on the sensor and the wavelength can no longer be described by a simple linear relationship. Consequently, the calibration and evaluation of the measurement differ somewhat.

[0137] Furthermore, the use of a grating is preferred because linear dispersion results in optimal scanning of the spectrum across wavelengths (pixels per wavelength), and the relationship between position and wavelength remains linear. Using a prism may be optically more efficient, but it leads to better sampling of the blue portion of the spectrum, while red wavelengths are "compressed" and therefore scanned poorly. However, this is disadvantageous because precisely for imaging live samples, the longer the wavelength chosen, the milder the excitation / probe becomes. Here, the components according to the invention can be used to detect multiple stainings very well.

[0138] An exemplary design scheme should be described here. Figure 11A matrix sensor with two schematically illustrated spectral bands is shown, which correspond, for example, to the spectral characteristics of a sample emitting fluorescence with a blue-green emission range (pixel range 71) and a more extensive orange-red emission range (pixel range 72). The task of data evaluation is, in principle, to combine the signals of the determined spectral bands, and therefore the pixels of pixel regions 71 and / or 72, such that one or two circularly symmetric point distribution functions 80 (PSFs) are generated. The PSFs obtained in this way are then evaluated, either subsequently or simultaneously, using image scanning methods known per se, resulting in an image of the sample in which the advantageous characteristics of image scanning are combined with spectral resolution, i.e., both dyes are represented in the image. To maintain transparency, it is initially assumed that the dyes are spectrally separated to such an extent that they fall on different pixels of the sensor. This is advantageous. However, this arrangement can also be used to image and measure samples with spectrally overlapping dyes or fluorescent proteins. This has been explained in the summary section of the invention.

[0139] Figure 12 A cutaway of a matrix sensor 50 of the detection unit according to the present invention is shown, which has pixels ( These pixels are used to measure the dispersed emitted light 28. The emitted light 28 is along the direction... It is dispersed and split. This means that, given each column... Assign a wavelength. In direction No wavelength dispersion splitting occurs there, meaning that only spatial information exists there. Figure 12 In the situation shown, the pixel ( ) represents the corresponding reference wavelength The reference pixel is used as the reference pixel, and the signals of other pixels are referenced to this reference pixel. This means that the signals of other pixels are shifted back to the reference pixel in a manner similar to that achieved to date in known image scanning microscopy. In this context, "shifting back" refers to digitally assigning the signals of each pixel to specific positions in the matrix sensor plane, and further, in the sample plane. However, compared to known methods, this shifting must be performed in a manner that properly accounts for dispersion.

[0140] The main change is for shift vectors Assume the shift stroke consists of two parts:

[0141]

[0142] This is a function of wavelength. Possible choices for this function include, for example:

[0143]

[0144] proportionality constant Here, the unit is length, and it is determined by the dispersion intensity of the dispersive element, especially by the grating constant of the grating 43 used.

[0145] Therefore, the shift stroke is represented as follows:

[0146]

[0147] In this example, dispersion only involves the x-direction. (In the interpolation...) Figure 12 In the middle, for the drawn pixels ( The shift vector of ) is therefore not altered by dispersion, while for the drawn pixel ( The shift vector will have a dispersive component.

[0148] The center of gravity of an effective PSF can be obtained as follows:

[0149]

[0150] In the direction perpendicular to dispersion, there is no change similar to conventional image scanning (Airy scanning) evaluation, but stretching / compression occurs in the direction of dispersion. This happens in such a way that the portion of the wavelength farther from the reference wavelength is more strongly corrected, thus shifting the center of gravity of the effective PSF closer to the reference pixel. The center of gravity is no longer half the shift length between the pixel and the reference pixel, but rather moved slightly closer to the reference pixel. Therefore, a specification is obtained of how the portion of the pixel must be shifted so that the contribution of all pixels can ultimately be offset. Optionally, further calibration can also be used here to determine the correct... However, especially when the sample is thick, it is often examined with multiphoton microscopy, which may result in the displacement vector being affected by the distortion caused by the sample [Castello et al., 2019; Figure 1 c]. The next paragraph will explain one possibility of obtaining these shift vectors from image data of a single pixel of the sensor.

[0151] The shift vector is determined by phase correlation.

[0152] Another possibility for evaluating the data for the considered situation is based on the assumption that the wavelengths belonging to a band of a dye, evaluated here, are typically characterized by a very specific spatial structure of the sample, and that this structure is ultimately identical across all spectral portions because it is labeled with the corresponding dye. Then, a second color, for example... Figure 11The blue-green spectrum in the image (pixel region 71) is characterized by the differences in the samples and the distinguishable structures (e.g., cell nuclei, which are labeled with DAPI, for example). Therefore, it can be assumed that these images ultimately have largely the same structural content, even if they emit slightly different colors. In this case, phase correlation is provided as another variant [Castello et al., 2019].

[0153] Here, the pixels of the scanned image are first numbered, and then... To be labeled. Therefore, by An image composed of 100 pixels is denoted as ,in, and Furthermore, the so-called correlation diagram (which involves reference pixels in this context) is defined. ):

[0154]

[0155] In a manner that is known in principle, and This refers to the (Fast) Fourier Transform or its reciprocal. The maximum value of this correlation plot.

[0156]

[0157] Then, a corresponding shift vector is provided so that the image content must be shifted back using that shift vector.

[0158] The applicability of this method has also been discussed in [Castello et al., 2019]. Ultimately, this method is similar to providing a so-called record of the image with different pixels. The advantage of this evaluation is that dispersion is essentially unnecessary to know, and the different orientations of dispersion as a function can also be handled algorithmically. Furthermore, this method has less dependence on the distortion of the image on the sensor. However, it is more computationally intensive.

[0159] In principle, other methods known from image scanning can also be used in this invention. For this purpose, see again [Castello et al., 2019]. There, so-called multi-view deconvolution for data evaluation is also discussed, which can also be used in this invention. Furthermore, see the published literature from Zeiss Airyscan.

[0160] This illustrates another approach to how data from a confocal spectral sensor with a spatial oversampling PSF can be evaluated to simultaneously obtain a higher resolution image with a higher signal-to-noise ratio and determine the spectrum.

[0161] As mentioned above, the spectrum itself can always be obtained by considering a series of points, that is, perpendicular to the dispersion direction (in the interpolation). Figure 3The result is obtained by summing the pixels in direction j).

[0162] Applications of multicolor excitation

[0163] Highly advantageously, the method according to the invention can also be used to simultaneously detect multiple dyes. For this purpose, it is advantageous that the integration boundaries, particularly along the dispersion direction 41, can be flexibly defined. The integration boundaries here refer to those boundaries where, for a given dye, the individual spectral contributions of the point distribution function must be summed. Thus, for example, the range of the point distribution function can be calibrated separately for each dye. This aspect will utilize... Figure 9 To provide a more detailed explanation.

[0164] Figure 9 a) shows the sum of 1000 simulated images from a SPAD camera, in which two dyes were exposed at a photon flux of 10 MHz and the exposure time of each image was 1 μs. Figure 9 b) shows the entire spectrum obtained from the image data. Figure 9 c) shows a partial spectrum obtained with limited integration bandwidth. Finally, Figure 9 d) shows that in Figure 9 The point distribution function known within a certain range as shown in c).

[0165] Automation options

[0166] System calibration is always strictly effective only for experiments with fixed presets, because it is particularly dependent on the selected objectives and the dyes being examined. Therefore, it is meaningful after modifying the experiment to allow the system to relearn the calibration in the following way: according to... Figure 8 and Figure 9 The average image data is evaluated and the calibration data is written, for example, into the memory of the evaluation electronics 60.

[0167] Furthermore, the calibration data is continuously updated from the last (several) LSM image scans. This allows the system to autonomously respond to changes in the experimental environment.

[0168] Another advantage is the possibility of automatically adjusting spectral channels by defining integration boundaries. This can be achieved through high-resolution scanning of the spectral space with step sizes of 1 nm or even smaller. For example, according to Figure 9 The integrated signal (b) can be used in algorithms to find the maximum and minimum quantities to determine recommendations for defining integration boundaries or equivalent spectral channels. Therefore, the system can also effectively detect emission from unknown samples and pre-set favorable dye separation. If necessary, the column into which light with the wavelength of excitation light 14 falls in the y-direction of the matrix sensor 50 is not evaluated or is turned off.

[0169] List of reference numerals

[0170] 10 Excitation Optical Path

[0171] 12 Light Sources

[0172] ( Reference pixel

[0173] 14 Excitation Light

[0174] 16 Deflecting Mirrors

[0175] 18. Master color separator

[0176] 22 Scanning device

[0177] 23. Lens tube

[0178] 24 Microscope Objectives

[0179] 26 Sample plane

[0180] 27 Illumination Spots

[0181] 28. Emitting light

[0182] 29. Zero-order diffraction

[0183] 30 Detection optical path

[0184] 32 detection units

[0185] ( Reference pixel

[0186] 34. Control and evaluation units, especially PCs

[0187] 40 Dispersion apparatus

[0188] 41. Dispersion direction

[0189] 42 spectral components of emitted light 28

[0190] 43. Grating

[0191] 44 Spectral components of emitted light 28

[0192] 46 spectral components of emitted light 28

[0193] 47 Spectral components of emitted light 28

[0194] 48 Imaging Optical Elements

[0195] 50 Two-dimensional matrix sensors

[0196] 51-matrix sensor with 50 pixels

[0197] 53 pixels in size

[0198] 54 Matrix Sensor

[0199] 56 Image plane (= plane of matrix sensor 50)

[0200] 57 Functions center of gravity

[0201] 60. Evaluate electronic devices

[0202] 71. Pixel area of ​​dye

[0203] 72. Pixel area of ​​dye

[0204] 80. Circularly Symmetric Point Distribution Function

[0205] 100 Laser Scanning Microscope

[0206] 200 detection devices

[0207] The grating constant of a matrix sensor 50

[0208] Shift vector for pixel redistribution

[0209] shift vector x-component

[0210] shift vector y component

[0211] In positioning Image at location

[0212] From pixels Measured intensity value

[0213] Pixels Position vector in image plane 56

[0214] Launched PSF

[0215] Excited PSF

[0216] For pixels Effective PSF

[0217] Columns of matrix sensor 50

[0218] ( Pixels

[0219] Matrix sensor 50 rows

[0220] m matrix sensor 50 row number

[0221] mi spectral distribution The minimum quantity in

[0222] mx1 spectral distribution The maximum quantity in

[0223] mx2 spectral distribution The maximum quantity in

[0224] n Matrix sensor 50 number of columns

[0225] Position vector to image point

[0226] Position vector to image point x-component

[0227] Position vector to image point The y-component.

[0228] Pixels Related images

[0229] Emission spectrum of s1 dye

[0230] Emission spectrum of s2 dye

[0231] To a valid PSF The position vector of the maximum quantity

[0232] Spectral distribution (continuous)

[0233] Spectral distribution (discrete)

[0234] The position of the image plane 56 projected onto the sample plane

[0235] x-axis coordinate direction of matrix sensor 50 (=dispersion direction)

[0236] The coordinate direction of the y-matrix sensor 50 (perpendicular to the dispersion direction).

[0237] Airy function

[0238] Spatial intensity distribution

[0239] From pixels Measured intensity value

[0240] Number of image points in the x-direction

[0241] Number of image points in the y-direction

[0242] FFT (Fast Fourier Transform)

[0243] FFT -1 Inverse Fast Fourier Transform

[0244] P i-n,j Overlap data regarding the spatial overlap of the spectral portion of the point distribution function of the dye on matrix sensor 50, shifted along the dispersion direction 41.

[0245] PSF point distribution function ( P oint S pread F unction)

[0246] S sample

[0247] SNR (Signal-to-Noise Ratio)

[0248] shift vector Wavelength-dependent portion

[0249] Spectral bandwidth of dyes

[0250] wavelength

[0251] In the column of matrix sensor 50 wavelength

[0252] Wavelength at the position or column of the reference pixel

[0253] Used to simulate shift vectors The constant of the dispersion condition

Claims

1. A method for detecting emitted light (28) in a laser scanning microscope, wherein, The emitted light (28) from the sample (S) is guided by imaging optics (48) onto a two-dimensional matrix sensor (50) with multiple pixels (51) located in the image plane (56). The matrix sensor (50) is used to detect the distribution function of the detection points using spatial oversampling. Its features are, The emitted light (28) from the sample (S) is spectrally decomposed using a dispersive apparatus (40). The matrix sensor (50) is used to perform spectrally resolved detection of the emitted light after spectral decomposition, and In evaluating the pixels in pixel regions (71, 72) The measured intensity ( ; When, at least for these pixels ( Some pixels in the image have their spectral separation disabled. In order to target each pixel in pixel region (71, 72) Cancel spectral separation, and the pixels ( The measured strength value ( ) assigned to the image plane (56) relative to their respective pixels ( The position that has been shifted, where the shift vector ( ) depends on their respective pixels ( The location of the wavelength at that location depends on the wavelength at that location. ).

2. The method according to claim 1, Its features are, At least one pixel region (71, 72) assigned to the emission of dye is identified based on the spectrum measured using the matrix sensor (50).

3. The method according to claim 1, Its features are, In order to determine the spectral intensity distribution of the emitted light on the matrix sensor (50) , The wavelength is determined as follows: The intensity value of ) , That is, the columns of the matrix sensor (50) perpendicular to the dispersion direction (41) are arranged as follows: Multiple pixels of ) Measurement data () ; Summation is performed using the given formulas.

4. The method according to claim 2, Its features are, In order to identify the pixel regions (71, 72), the maximum amount (mx1, mx2) and the minimum amount (mi) are automatically searched in the known spectral distribution. and Based on the found maximum (mx1, mx2) and minimum (mi), suggest to the user the spectral boundary for calculating the point distribution function of a given dye, or The spectral boundaries are automatically defined based on the found maximum (mx1, mx2) and minimum (mi) quantities.

5. The method according to claim 1, Its features are, The pixel regions (71, 72) overlap on the matrix sensor (50), and spectral demixing is performed on the intensity measured by each pixel.

6. The method according to claim 1, Its features are, Determine the detection point distribution function for at least one fluorescent dye.

7. The method according to claim 1, Its features are, The emitted light (28) emitted from multiple points that are simultaneously irradiated on the sample (S) by excitation light (14) is simultaneously guided to the matrix sensor (50) and evaluated.

8. The method according to claim 1, Its features are, The matrix sensor (50) operates in photon counting mode.

9. The method according to claim 1, Its features are, In order to target each pixel in pixel region (71, 72) ) Cancel spectral separation, taking into account the spectral intensity distribution of the emitted light of the dye belonging to pixel regions (71, 72) , In the case of considering the spatial intensity distribution of each spectral portion on the matrix sensor (50), In the case of these pixels, The measured strength value ( They cancel each other out.

10. The method according to claim 9, Its features are, The following intensity distribution ( ) is used as the spatial intensity distribution of each spectral part: the intensity distribution consists of columns perpendicular to the dispersion direction (41) ( ) pixels ( ) was measured.

11. The method according to claim 1, Its features are, The emitted light (28) is fluorescence from at least one fluorescent dye.

12. The method according to claim 1, Its features are, For a given pixel ( ) shift vector ( The wavelength-independent portion of ) is obtained by transferring the data from the reference pixel ( ) to relevant pixels ( ) vector ( The vector components of ) are obtained by scaling with redistribution factors.

13. The method according to claim 1, Its features are, The probe point distribution function obtained after performing pixel redistribution has essentially the same shape in the dispersion direction (x, 41) as it does in the direction perpendicular to the dispersion direction (y).

14. The method according to claim 1, Its features are, The shift vector belonging to the wavelength range that matches the sample structure ( This is determined by evaluating the phase correlation of multiple scanned images.

15. The method according to claim 1, Its features are, The matrix sensor (50) is used to perform time-resolved measurements to determine the fluorescence lifetime of the dye.

16. The method according to claim 1, Its features are, The emitted light (28) from the sample (S) is spectrally decomposed in the dispersion direction (41) using a dispersive device (40).

17. The method according to claim 3, Its features are, The columns of the matrix sensor (50) perpendicular to the dispersion direction (41) are... All pixels of ) The measurement data are summed.

18. The method according to claim 10, Its features are, The intensity distribution ( The column with the highest intensity measured in its respective pixel region (71, 72) ) pixels ( ) was measured.

19. The method according to claim 1, Its features are, For a given pixel ( ) shift vector ( The wavelength-independent portion of ) is obtained by transferring the data from the reference pixel ( ) to relevant pixels ( ) vector ( The vector components are obtained by scaling by -1 / 2.

20. The method according to claim 1, Its features are, Time-resolved measurements are performed using some pixels of the matrix sensor (50) to determine the fluorescence lifetime of the dye.

21. The method according to claim 1, Its features are, Time-resolved measurements are performed using each single pixel of the matrix sensor (50) to determine the fluorescence lifetime of the dye.

22. A detection device for detecting emitted light in a laser scanning microscope, said detection device being used to perform the method according to any one of claims 1 to 21, said detection device A two-dimensional matrix sensor (50) having multiple pixels (51) on an image plane (56) is provided, which is used to detect the detection point distribution function of emitted light (28) from the sample (S) in a spatially oversampled manner. It also includes an imaging optics (48) for guiding the emitted light (28) onto the two-dimensional matrix sensor (50). Its features are, To perform spectral separation of the emitted light (28), a dispersive device (40) is provided, and the matrix sensor (50) is set up and positioned for spectrally resolved detection of the spectrally separated emitted light, and An evaluation electronics device (60) is present and connected to the matrix sensor (50), the evaluation electronics device being configured to evaluate pixels in pixel regions (71, 72). The measured intensity ( ; When targeting these pixels ( Cancel spectral separation. In order to target each pixel in pixel region (71, 72) Cancel spectral separation, and the pixels ( The measured strength value ( ) assigned to the image plane (56) relative to their respective pixels ( The position that has been shifted, where the shift vector ( ) depends on their respective pixels ( The location of the wavelength at that location depends on the wavelength at that location. ).

23. The detection device according to claim 22, Its features are, The evaluation electronics (60) are configured to, in order to eliminate spectral separation, take into account the spectral intensity distribution of the emitted light of the dye belonging to the pixel regions (71, 72). , In the case of considering the spatial intensity distribution of each spectral portion on the matrix sensor (50), In the case of ), the pixels of pixel region (71, 72) are ( The measured strength value ( They cancel each other out.

24. The detection device according to claim 22, Its features are, The evaluation electronics (60) are configured to cancel the evaluation of individual pixels in pixel regions (71, 72). The spectral separation of the pixels will be determined by the spectral separation of the pixels. The measured strength value ( ) assigned to the image plane (56) relative to their respective pixels ( The position that has been shifted, where the shift vector ( ) depends on their respective pixels ( The location of the wavelength at that location depends on the wavelength at that location. ).

25. The detection device according to claim 22, Its features are, The dispersion device (40) has a light diffraction device and / or a light refraction device.

26. The detection device according to claim 22, Its features are, The matrix sensor (50) has an analog integrating detector and / or a photon counting detector.

27. The detection device according to claim 22, Its features are, The dispersion direction (41) is in the direction of the coordinate direction (x) of the matrix sensor (50).

28. The detection device according to claim 22, Its features are, A microlens is arranged in front of the matrix sensor (50) to improve detection efficiency.

29. The detection device according to claim 22, Its features are, The diameter of the Airy disk of the in-plane detection point distribution function of the matrix sensor (50) is less than 20 times the grating constant of the matrix sensor (50).

30. The detection device according to claim 22, Its features are, The matrix sensor (50) has each pixel (51; () in the dispersion direction (41) of each pixel (51; The spectral bandwidth is less than 0.5 nm.

31. The detection device according to claim 22, Its features are, The imaging optics (48) has a zoom system.

32. The detection device according to claim 22, Its features are, The matrix sensor (50) and the evaluation electronics (60) are configured to perform time-resolved measurements.

33. The detection device according to claim 22, Its features are, The dispersive device (40) has a grating (43) and / or a prism.

34. The detection device according to claim 22, Its features are, The matrix sensor (50) has an sCMOS camera, an EMCCD camera and / or a (CMOS) SPAD array.

35. The detection device according to claim 34, Its features are, Microlenses are arranged in front of the SPAD array to improve detection efficiency.

36. The detection device according to claim 22, Its features are, The matrix sensor (50) has each pixel (51; () in the dispersion direction (41) of each pixel (51; The spectral bandwidth is less than 0.4 nm.

37. The detection device according to claim 22, Its features are, The matrix sensor (50) has each pixel (51; () in the dispersion direction (41) of each pixel (51; The spectral bandwidth is less than 0.3 nm.

38. The detection device according to claim 22, Its features are, The matrix sensor (50) and the evaluation electronics (60) are configured to perform time-resolved measurements in order to determine the fluorescence lifetime of the dye.

39. The detection device according to claim 22, Its features are, The matrix sensor (50) and the evaluation electronics (60) are configured to perform time-resolved measurements using some pixels of the matrix sensor (50).

40. The detection device according to claim 22, Its features are, The matrix sensor (50) and the evaluation electronics (60) are configured to perform time-resolved measurements using each single pixel of the matrix sensor (50).

41. Laser scanning microscope The laser scanning microscope is used to perform the method according to any one of claims 1 to 21, the laser scanning microscope having: A light source (12) is used to emit excitation light (14). An excitation optical path (10) with a microscope objective (24) is provided, the excitation optical path being used to guide the excitation light (14) onto or into the sample (S) to be examined. A scanning device (22) in the excitation optical path (10) is used to scan at least one illumination spot (27) on the sample (S). A detection optical path (30) is provided for guiding the emitted light (28) from the sample (S) onto the detection unit (32). Detection unit (32) for detecting the emitted light (28). The main dichroic separator (18) is used to separate the excitation light (14) and the emission light (28), and A control and evaluation unit (34) is used to drive the light source (12) and to evaluate the measurement data obtained by the detection unit (32). Its features are, The detection unit (32) has a detection device (200) according to any one of claims 22 to 40.

42. The microscope according to claim 41, Its features are, The control and evaluation unit (34) is configured as follows: Used to search within the known spectral distribution ( , The maximum (mx1, mx2) and minimum (mi) in ). And based on the found maximum (mx1, mx2) and minimum (mi) values, suggest spectral boundaries for calculating the point distribution function of a given dye, or The spectral boundary is used to autonomously define the point distribution function of the determined dye based on the found maximum (mx1, mx2) and minimum (mi) values.

43. The microscope according to claim 41, Its features are, The control and evaluation unit (34), together with the detection device (200), is configured to implement the method according to any one of claims 1 to 21.

44. The microscope according to claim 41, Its features are, The light source (12) is a laser.

45. The microscope according to claim 41, Its features are, The emitted light (28) is fluorescence.

46. ​​The microscope according to claim 41, Its features are, The control and evaluation unit (34) is a PC.

Images