METHOD, LIGHT MICROSCOPE AND COMPUTER PROGRAM FOR LOCATING OR TRACTING EMISSORS IN A SAMPLE

Abstract

The invention relates to a method for locating or tracking emitters (E) in a sample (2), wherein the sample (2) is illuminated with illumination light (B) at grid points (4) of a search grid (3), detection light from the sample (2) is detected, the detection light is stored as search data, the sample (2) is illuminated with an intensity distribution (5) of light with a local minimum (6), light emissions of the emitter (E) are detected for different positions, position data of the emitter (E) are calculated, and estimated positions of several stationary emitters (E) or trajectories are determined from the position data, wherein reference image data of an environment of the estimated positions or trajectories are determined from the stored search data, as well as a light microscope (1) and a computer program for carrying out the method.

Description

Technical field of the invention

[0001] The invention relates to a light microscopic method for localizing or tracking individual emitters in a sample, e.g. according to the MINFLUX principle, as well as to a light microscope and a computer program for carrying out the method. State of the art

[0002] In contrast to classical imaging techniques of light microscopy, in light microscopy localization methods, the positions of individual emitters (e.g., fluorophores or molecules labeled with fluorophores) are calculated based on detected light emissions. From the positions of a large number of emitters, usually determined one after the other, a localization map can be created that visualizes the distribution of the emitters in the sample. Such a localization map is similar to a wide-field image, in which sample structures are imaged onto a detector, or to a raster image, in which the sample is scanned with focused illumination light and the light emission from the sample is recorded for each raster point. However, depending on the localization precision of the corresponding imaging technique, localization microscopy localization maps can have a resolution well below the diffraction limit.

[0003] The term "individual" here means that at a given time, actively light-emitting emitters whose emission light is indistinguishable are separated by a distance above the diffraction limit, so that their emission light can be separated. This can be achieved, for example, in the case of permanently light-emitting emitters by a labeling density of the sample below a limit value. However, if the emitters flash asynchronously, for example, a sufficient separation can be achieved at any given time even with a higher labeling density. For this purpose, the sample environment (e.g. buffer, embedding medium) can be chemically configured to result in a desired flashing rate. Smaller separations are also tolerable when different types of emitters are present whose emission light can be optically separated, e.g. due to different emission spectra or emission lifetimes.Finally, in exceptional cases, special evaluation methods (e.g., statistical or time-resolved methods) may also make it possible to locate groups of several closely adjacent emitters together. This can determine a specific position for each emitter, or an average position of several emitters can be determined.

[0004] Individual emitters moving within the sample can be tracked by several rapid, consecutive localizations within the sample (tracking). The corresponding measurement data can be displayed, for example, in the form of a trajectory.

[0005] In the so-called MINFLUX technique, a sample containing a single emitter (e.g., a single fluorophore, a fluorophore-labeled molecule, or a light-scattering particle) is illuminated with an intensity distribution of excitation light at illumination positions in a range around a roughly estimated position of a single emitter. The intensity distribution has a central intensity minimum (ideally, an intensity zero). For each illumination position, the light emissions (in particular, photon rates) of the individual emitter are recorded. A new position estimate of the emitter is then calculated from the light emissions and the associated illumination positions. The intensity distribution can, for example, be a so-called 2D donut, a so-called bottle beam, or a superposition of these light distributions, which are known from the field of STED microscopy.

[0006] Various variants of the MINFLUX technique are described, for example, in the publications F. Balzarotti et al. (2017) Nanometer resolution imaging and tracking of fluorescent molecules with minimal photon fluxes, Science 355 (6325), 606-612, KC Gwosch et al. (2020) MINFLUX nanoscopy delivers 3D multicolor nanometer resolution in cells, Nat. Methods, 17 (2), 217-224 and R. Schmidt et al. (2021) MINFLUX nanometer-scale 3D imaging and microsecond-range tracking on a common fluorescence microscope, Nat. Commun. 12 (1), 1478 as well as in the patent documents US 9 719 928 B1, US 10 900 901 B2, US 10 908 089 B2 and US 10 962 479 B2.

[0007] Due to the excitation light distribution with an intensity minimum, the MINFLUX technique allows for localization precision in the single-digit nanometer range with a significantly lower number of emitted photons compared to the so-called PALM / STORM technique. This can be explained by the fact that the emitter is illuminated with less excitation light the closer the minimum of the excitation light distribution is to the actual emitter position.

[0008] Patent publication US 11 255 791 B2 describes, among other things, a variant of the MINFLUX technique in which the sample is illuminated with a combination of an excitation light distribution with a local maximum and a STED (stimulated emission depletion) light distribution with a local minimum. The position of an individual emitter is also calculated from the light emissions recorded for different illumination positions, but the closer the minimum of the STED light distribution (and the maximum of the excitation light distribution superimposed with the minimum) is to the actual emitter position, the more light is emitted. Therefore, the advantage of particularly high photon efficiency is not achieved here.

[0009] The patent specification DE 10 2020 127 320 B3 describes a special MINFLUX method in which a raster image is recorded by means of scanning laser microscopy before scanning a dye molecule, whereby a scanning rule for the MINFLUX method is determined from signals from pixels of the raster image.

[0010] A special localization technique using an excitation light distribution with a local maximum and a STED light distribution with a local minimum is described in the publication M. Weber et al. (2021) MINSTED fluorescence localization and nanoscopy, Nat. Photonics 15, 361-366 and the patent application WO 2023 / 006176 A1 and is known as MINSTED. The combination of the excitation light distribution and the STED light distribution is moved along a continuous trajectory around an estimated emitter position. For each detected photon, the corresponding position of the minimum of the STED light distribution is determined, and the trajectory is adjusted based on the detected photon. The excitation and STED intensities can also be increased.

[0011] The localization and tracking methods described above require first locating a single emitter in the sample and performing an initial position estimation using an independent method. Then, the sample is illuminated near the estimated position with the excitation light distribution. Such a position estimation can also be referred to as pre-localization. It typically exhibits significantly greater position uncertainty, thus significantly lower localization precision, than the subsequent localization method.

[0012] To locate a single emitter, the sample can be illuminated with excitation light at grid points of a search grid, for example, and the detection of emission light above a background intensity or background rate can be interpreted as an indication of the presence of an emitter in the corresponding area of ​​the sample. One possible way of illuminating with excitation light is to scan the sample with focused excitation light at the grid points. A disadvantage of this, however, is that for each localized emitter, it is necessary to switch between a regular focus and a light distribution with a local intensity minimum between pre-localization and subsequent localization with high precision, or that an additional light distribution with a local minimum must be switched on.

[0013] One possibility for performing pre-localization for a MINFLUX method using the same excitation light distribution with a local minimum as used in the main localization is described in patent application US 2023 / 0251479 A1 and is referred to as pinhole orbit scanning. This involves using two scanning devices, one of which is located only in the illumination beam path and the other in the shared illumination and detection beam path, to move the projection of a detection pinhole into the sample along a path (in particular, a circular path), while the excitation light distribution with the local minimum is kept stationary relative to the sample. Since switching between different light distributions is not necessary, pre-localization can be performed more quickly and with less equipment.

[0014] The patent application WO 2022 / 079265 A1 describes, among other things, a microscope capable of examining different structural levels of a sample using different high-resolution methods, e.g. confocal laser scanning microscopy and MINFLUX.

[0015] Patent application WO 2023 / 166065 A1 discloses a light microscope with an adjustable scanner and two different operating modes, wherein in one operating mode a MINFLUX localization can be performed and in the other operating mode, for example, a diffraction-limited light microscopy method such as confocal laser scanning microscopy can be performed.

[0016] Patent application US 2023 / 0003651 A1 discloses a disturbance correction method for a laser scanning microscope, in particular a MINFLUX microscope, in which a disturbance is detected, weighting factors being assigned to the pairs of fluorescence photons and scanning positions based on the detected disturbance in order to correct the localization data, or the illumination of the sample is interrupted if the detected disturbance exceeds a threshold. The disturbance can be, for example, a rapid relative movement between the sample and the objective. A disadvantage of the method is that only external disturbances are detected, but not relative movements between structures in the sample.

[0017] Patent application WO 2022 / 200549 A1 describes a method that can be used, in particular, for drift correction during MINFLUX localization. For this purpose, the light microscopic examination of the sample is interrupted in order to localize reference markers in the sample according to the MINFLUX principle. A disadvantage of this method is that reference markers, e.g., fluorescent or light-scattering nanoparticles, must be introduced into the sample. This method can only detect drift movements in the sample that occur in areas where corresponding reference markers are located. However, specific sample structures, e.g., some organelles of biological cells, cannot be marked with reference markers, or can only be marked with difficulty. Object of the invention

[0018] This gives rise to the task of providing a method for localizing or tracking emitters in a sample, which effectively relates localization data or trajectories to, in particular, larger, sample structures, and thus improves the results of the method through context information. Solution

[0019] This problem is solved by the subject matter of the independent claims (method, light microscope, and computer program). Advantageous further developments are specified in the subclaims and described below. Description of the invention

[0020] A first aspect of the invention relates to a method for locating or tracking emitters in a sample, wherein the sample is illuminated with illuminating light at a grid point of a search grid in at least one search step, wherein the illuminating light induces or modulates light emissions from emitters in the sample, wherein detection light from the sample is detected for the respective grid point, wherein search data is stored on the basis of the detection light detected for the respective grid point, and wherein it is determined on the basis of light emissions detected from the sample whether an individual emitter is located in an area around the respective grid point.In the method according to the invention, the sample is further illuminated in a localization step or a tracking step with an intensity distribution of the illuminating light or another light, wherein the illuminating light or the another light induces or modulates light emissions from the emitters in the sample, and wherein the intensity distribution has a local minimum. Light emissions from the emitter are detected for various positions of the intensity distribution, wherein position data of the emitter are calculated based on the detected light emissions and the positions of the local minimum of the light distribution associated with the light emissions, and wherein respective estimated positions are determined from the position data of several stationary emitters, in particular one after the other, or respective trajectories are determined from the position data of several emitters moving in the sample, in particular one after the other.According to the invention, reference image data of an environment of the estimated positions or trajectories are determined from the stored search data.

[0021] The reference image data provides information about the context of the positions or trajectories of individual emitters estimated by localization, particularly with respect to sample structures in whose area the localized or tracked emitters are located. In contrast to state-of-the-art methods, obtaining this context information does not require a separate overview image acquisition. Instead, the reference image data is advantageously determined from the search data already generated for locating emitters.The at least one search step thus fulfills a dual function: first, based on the light emissions detected in the at least one search step, positions in the sample are determined in the vicinity of which individual emitters are located, which can then be localized or tracked with high precision (particularly with localization inaccuracies in the single-digit nanometer range). Second, additional contextual information about the emitter's surroundings is obtained "free of charge" from the search data. This effectively improves the results of the localization or tracking process.

[0022] Of course, the method according to the invention can also be used to capture a dedicated overview image, particularly at the beginning of the measurement, for example, to allow the user to select a region of interest within the sample. The method according to the invention is also advantageous in this case: "free" context information is obtained at various intermediate points of the measurement (before each localization or recording of a trajectory of an individual emitter), without having to capture another time-consuming and potentially sample-damaging overview image.

[0023] A method for localizing emitters within the meaning of this specification is characterized by the fact that the positions of individual emitters are computationally estimated in the localization or tracking step based on detected light emissions. This distinguishes such methods from conventional light microscopy imaging techniques, such as wide-field or scanning microscopy, which optically image a large number of emitters without knowing the positions of the individual emitters. In the at least one preliminary search step, however, an imaging light microscopy technique can certainly be used.

[0024] In the context of this specification, an emitter tracking method is a method that tracks the position of a moving individual emitter over time. This includes, for example, so-called single-molecule tracking methods. In particular, a trajectory of the emitter is recorded, i.e., a track that connects successive locations of the emitter over time.

[0025] In this application, emitters are understood to be objects that, when illuminated with excitation light, can be considered point light sources with regard to the measurements according to the invention. The light emitted by the object acting as a point light source can, for example, be scattered light resulting from elastic scattering such as Rayleigh scattering or inelastic scattering such as Raman scattering, or it can be luminescent light, in particular fluorescent light. An emitter can therefore be, for example, a light-reflecting nanoparticle, a quantum dot, a fluorescent dye molecule (fluorophore), or a molecule or nanoparticle labeled with one or more fluorescent dye molecules.Depending on the size of the molecule and the distance between the fluorophores, a molecule labeled with multiple fluorophores or a nanoparticle labeled with multiple fluorophores can of course also have multiple emitters in the sense of the definition used here.

[0026] As mentioned at the beginning, "individual" emitters are understood here to be emitters that can be optically separated from one another using light microscopy techniques. This can be achieved by a labeling density of the sample that results in an average distance between the emitters above the diffraction limit. Alternatively, asynchronously flashing emitters can be used if the sample conditions (in particular the composition of the sample buffer and embedding medium) are adjusted so that the average distances between the actively emitting emitters are above the diffraction limit at all times. Finally, different emitters that can be optically distinguished by other means can, under certain circumstances, also be separated if their distance is below the diffraction limit, e.g., based on their characteristic emission spectrum or their emission lifetime.

[0027] In particular, several search steps can be carried out consecutively, in which the sample is illuminated with the illumination light at adjacent grid points of the search grid. However, not all grid points of the search grid have to be illuminated; rather, the at least one search step can be terminated as soon as light emissions indicating the presence of a single emitter have been detected, and the localization or tracking step can be initiated. If only background light is detected at the respective grid point in a search step, the search step can be aborted, in particular after a predetermined period of time, and the next search step can be initiated, for example, at a neighboring grid point of the search grid.

[0028] The detection light recorded during a search step can be identical to the light emissions of a single emitter, i.e. the detection light can originate from the emitter, whereby the detection light can in particular be light scattered or reflected by the emitter or luminescence light, in particular fluorescent light. Alternatively, the detection light can also be a different light than the light emissions of the individual emitter. For example, it can be transmitted light, i.e. illumination light that shines through the sample, or light reflected from the sample. In the latter cases, the light emissions that indicate the presence of a single emitter, e.g. fluorescent light, are also recorded in the search step. For example, transmitted light or reflected light on the one hand and fluorescent light on the other hand can be recorded by two separate detectors.Transmitted or reflected light are suitable for determining contextual information about the surroundings of individual emitters, from which the reference image data can be obtained. Fluorescence or reflection / scattering from individual emitters, on the other hand, are suitable for determining whether an emitter is located in the area of ​​the current grid point, as well as for the subsequent localization or tracking of the individual emitter with high precision in the localization or tracking step. Regardless of the specific implementation and the type of detection, it is advantageous if, in the search step, the information about the individual emitter and the contextual information about the sample's surroundings can be acquired in parallel from the illumination of the sample with one type of illumination light.

[0029] If the detection light and the light emissions of the emitters contained in the sample are identical, context information can sometimes be obtained in a search step by, for example, detecting light from neighboring emitters located outside the focal plane, which represents background light for localization or tracking. Furthermore, it is possible that detection light is obtained from a larger sample area during the search step than in the subsequent localization or tracking step, so that the light of several emitters in the focal plane is detected that are at a distance above the diffraction limit, i.e. represent individual emitters within the meaning of this specification. The context information can then be obtained from the detection light of the neighboring emitters. To obtain the reference image data, the detection light from emitters that non-specifically mark structures in the sample can also be important, for example.

[0030] The search grid used can, in particular, be a regular grid, e.g., a Cartesian grid or a hexagonal grid. The search grid can be two-dimensional, wherein the search grid can, in particular, be arranged in a focal plane, i.e., a plane that intersects the geometric focus of the illuminating light and is perpendicular to the optical axis (incident direction of the illuminating light). Alternatively, the search grid can also be three-dimensional, i.e., it can also extend in an axial direction parallel to the optical axis.

[0031] The illumination light can, in particular, be excitation light, which excites an emitter in the sample to emit light, i.e., induces light emission from the emitter. The light emissions can, in particular, be reflected light, scattered light, or luminescence light (e.g., fluorescent light). Alternatively, the illumination light can also modulate light emissions from the emitter. In this case, the illumination light can, for example, be STED light, which causes emitters to de-excite from an excited state to the ground state, or switching light, which converts emitters from an actively emitting state to a dark state.

[0032] In the localization step, the sample is illuminated with an intensity distribution of the illuminating light or another light which has a local minimum. Intensity increase regions border the local minimum, in particular in at least one spatial direction. The local minimum is in particular a central minimum of the intensity distribution, thus forming a center of the intensity distribution, whereby the center can in particular be arranged at the geometric focus. In this case, it can in particular be a light distribution that is point-symmetric with respect to the geometric focus. The local minimum can in particular be at least approximately an intensity zero. Such intensity distributions include in particular a so-called donut and a so-called bottle beam. Such light distributions can, for example,by phase modulation of the illumination light (e.g. with a phase plate or a spatial light modulator) and focusing it into the sample using an objective lens.

[0033] The localization or tracking method according to the invention is, in particular, a so-called MINFLUX method (when the illumination light with which the sample is illuminated in the localization step is excitation light that induces the light emissions of the emitters) or STED-MINFLUX method (when the illumination light with which the sample is illuminated in the localization step is inhibition light that modulates the light emissions of the emitters). In a MINFLUX method or STED-MINFLUX method, the local minimum of the intensity distribution of the illumination light is placed at illumination positions in a near range of the approximate position of an individual emitter, determined in particular in the at least one search step, and photons emitted by the emitter are detected for each illumination position. The near range can, in particular, have an extent on the order of the optical diffraction limit.The detected photon numbers (or photon rates) and the associated illumination positions can form input values ​​of a position estimator (e.g., a least means square estimator or a maximum likelihood estimator), which is then used to determine an updated position estimate for the emitter. This process can be repeated iteratively by placing the intensity distribution of the illumination light at updated illumination positions in a close range of the updated position estimate and, in turn, detecting photons for each position. In particular, a radius of an illumination pattern formed by the illumination positions around the previously estimated position can be reduced. Optionally, the light intensity of the illumination light can also be increased. The iteration steps can, for example,The tracking process can be continued until the emitter stops emitting light or until a photon limit or a threshold of localization precision is reached. In a MINFLUX or STED-MINFLUX tracking method, one or more iterations can be repeated at short intervals to track the trajectory of a moving emitter. Illumination patterns specifically tailored to a tracking method can be used.

[0034] According to one embodiment, the sample is illuminated in the at least one search step and in the localization or tracking step with the same intensity distribution of the same illumination light, wherein the intensity distribution exhibits the local minimum. This has the advantage that there is no need to switch between different intensity distributions, such as between a donut and a regular (Gaussian) focus, between the at least one search step and the localization or tracking step. The method is thus particularly fast and can also be carried out with less complex equipment.

[0035] According to a further embodiment, the detection light is captured in a location-dependent manner in a detection plane in the at least one search step. The detection plane is optionally confocal to a focal plane of the illumination light in the sample.

[0036] According to a further embodiment, the projection of a detection pinhole into the sample is moved on a, in particular circular, path during a search step for the respective grid point of the search grid, while the illumination light is stationary relative to the sample.

[0037] This method, which can also be referred to as pinhole orbit scanning, can be implemented in particular with two independently controllable scanning devices for the illumination light, wherein a first scanning device is arranged exclusively in the illumination beam path (between the light source and the beam splitter), while the second scanning device is placed in the shared illumination and detection beam path (between the beam splitter and the objective). By controlling the second scanning device, the projection of the detection pinhole can be moved along the path in the sample, while the first scanning device performs an opposite scanning movement of the illumination light such that the intensity distribution of the illumination light remains stationary at the focus in the sample. An electro-optical scanning device, for example, is suitable as the first scanning device, while a galvanometer scanning device, for example, can be used as the second scanning device.

[0038] According to a further embodiment, the detection light is detected with a detector having a plurality of detector elements.

[0039] So-called arrays of avalanche photodiodes (APDs) are suitable for this purpose. This also allows the light emissions to be captured spatially resolved in the detection plane, where the plane of the detector elements corresponds to the detection plane. In particular, the detector elements can be individually readable. In particular, the detector elements can be designed to count individual photons.

[0040] According to a further embodiment, at least one overview image of the sample is generated from the reference image data.

[0041] From the corresponding overview image, context information can be obtained for each emitter localization or trajectory by imaging the sample structures in, on, or near which the emitter was located during localization or during the recording of the trajectory. In particular, the localization or tracking step can be fast compared to the change and / or movement of sample structures, so that it can be assumed that the obtained reference image data correspond to the conditions prevailing in the sample during the localization or tracking step.

[0042] For example, in the case of a biological sample, the method could be used to determine whether a specific emitter or trajectory is located within a biological cell or within a cellular compartment such as an organelle. Such sample structures can move or change shape within the sample, particularly in live-cell experiments. By using the search data to generate overview images, the invention provides overview images that reflect the situation in the sample at the time of the corresponding localization or acquisition without additional effort or additional time delay.

[0043] In particular, prior to a localization or tracking step, a complete overview image can be obtained from the reference image data obtained directly one after the other in a plurality of search steps, provided that reference image data from a sufficient number of grid points is available. However, it is also possible to recalculate only specific areas or specific pixels of an overview image, or in extreme cases even only a single pixel of an overview image, from the reference image data, with the remaining pixels of the overview image originating from previously performed search steps. It is even possible to replace only individual pixels or groups of pixels in an overview image created at the beginning of the measurement using another technique, e.g., confocal scanning microscopy or wide-field fluorescence microscopy.A mosaic-like overview image can therefore be created, particularly from measurements taken at different points in time. It is advantageous to update primarily those pixels of the overview image that show relevant and, in particular, temporally changing sample structures, which provide important contextual information about a region of interest within the sample containing emitters to be localized or tracked.

[0044] According to a further embodiment, the at least one overview image is superimposed with at least one of the estimated positions or at least one of the trajectories and displayed by means of a display unit.

[0045] Such an overlay provides the user with a particularly clear presentation of context information. Particularly when there are a large number of locations or trajectories, and in cases where structural changes in the sample occur on a slower timescale than the locations or the recording of the trajectories, it is advisable to overlay several locations or trajectories with an overview image representative of this selection. For this purpose, for example, corresponding reference data sets can be generated for only all n search data sets, from which an overview image representative of several locations or trajectories can then be generated.

[0046] If the overview image consists of reference image data from search data obtained at different times, e.g. in the case of an overview image in which individual pixels or groups of pixels have been updated on the basis of corresponding reference image data, the user can, for example, select a time interval, whereby in particular an overview image can be displayed which corresponds to the last update in this time interval.

[0047] According to a further embodiment, a temporal sequence of overview images is generated from the stored search data. Optionally, estimated positions or trajectories recorded at corresponding times are overlaid with the respective overview images of the temporal sequence and displayed on the display unit.

[0048] The respective overview images can be created entirely from reference image data that were obtained in a sequence of search steps directly before determining the respective emitter position or recording the respective trajectory, or the respective overview images can be created only in individual pixels or groups of pixels or a partial area of ​​the image from these reference image data, while other pixels, pixel groups or areas can be adopted, for example, from previous overview images.

[0049] The sequence of overview images allows the user to intuitively track structural changes in the sample in which the emitters are located. The temporally ordered overview images can be displayed on the display unit in a spatial arrangement, for example. Alternatively, the overview images can be presented to the user as a movie, similar to time-lapse microscopy data.

[0050] According to a further embodiment, pixel intensities of pixels of the at least one overview image corresponding to sample positions between grid points of the search grid are calculated, in particular interpolated, on the basis of the detection light which was detected in the at least one search step for the respective grid point.

[0051] This makes it possible to obtain a relatively fine overview image despite larger distances between the grid points, which can particularly accelerate the search for a single emitter.

[0052] According to a further embodiment, the estimated positions or trajectories are processed and / or corrected based on the reference image data.

[0053] In this way, it is possible, for example, to correct artifacts in localization maps or trajectory maps caused by movements or changes in the shape of sample structures. A set of localizations or trajectories can thus be related to a reference system moving within the sample, e.g., a cell or organelle, without the need for reference markers, as described in the prior art.

[0054] According to a further embodiment, at least one structure of the sample is identified based on the reference image data, wherein the estimated positions or trajectories are processed and / or corrected with respect to the at least one structure. In particular, the at least one structure can change and / or shift relative to the sample on a timescale of determining the estimated position or recording the trajectories.

[0055] An example of such an application is a MINFLUX tracking experiment, in which a fluorophore-labeled protein (e.g., kinesin or dynein) is tracked as it moves along a filament of the cytoskeleton of a biological cell (e.g., microtubules). Using the reference image data created from the search data, a deformation or movement of such a filament in the cell, or a movement of the cell in the sample, could be detected, and the correction of the position data could involve relating different trajectories of the labeled protein, recorded at different times, to the reference system of the filament. Identifying structures in the sample based on the reference image data can, in particular, be done manually, e.g., by a user marking corresponding structures in an overview image created from the reference image data. Alternatively, the identification can also be performed automatically, e.g.,using known image recognition or image segmentation algorithms, including artificial intelligence methods.

[0056] According to a further embodiment, the grid points are adjusted based on the reference image data.

[0057] This may in particular mean that the entire search grid is shifted, that the relative positions of grid points of the search grid are changed (e.g. by changing the distances between the grid points or by providing denser arrangements of grid points at certain positions than at other positions) or that only certain grid points of the search grid are used for the at least one search step, e.g. those positions that overlap with certain sample structures.

[0058] In the latter case, it is possible, for example, to specifically locate and localize or track emitters in specific organelles of a cell. The entire search grid can be shifted, for example, to follow a moving region of interest in the sample, and to localize individual emitters or record trajectories there.

[0059] According to a further embodiment, in the at least one search step, background light is detected in the event that there are no light-emitting emitters in the vicinity of the respective grid point, wherein a background correction of the position data is carried out on the basis of the detected background light, and wherein a frequency or a duration of the detection of background light is adjusted in order to adjust a frame rate with which reference image data is determined from the search data.

[0060] Background correction for a MINFLUX or STED-MINFLUX method can be performed, for example, as described in US 2022 / 0042914 A1.

[0061] According to a further embodiment, the light emissions from the sample are detected in a first detection channel, and the detection light is detected in a second detection channel that is different from the first detection channel, in particular spectrally. The position data of the emitter are determined based on the light emissions detected in the first detection channel, and the search data used to determine the reference image data are determined based on the detection light detected in the second detection channel. In particular, light emissions from the sample in the first detection channel and the detection light in the second detection channel can be detected temporally parallel to one another (e.g., simultaneously, temporally overlapping, or alternately).

[0062] An additional detection channel has the advantage that sample structures that are in the context of the emitters to be localized or tracked can be made more visible, e.g. by detecting a counterstain of the sample or based on other imaging principles (e.g. reflection, transmitted light or phase contrast for the second detection channel and fluorescence for the first detection channel).

[0063] According to a further embodiment, based on the detection light detected in the second detection channel, it is checked whether an estimated position of an emitter or a trajectory is located in a region of interest of the sample.

[0064] According to a further embodiment, in the event that it is determined on the basis of the detection light detected in the second detection channel that the estimated position of the emitter or the trajectory is not located in the region of interest, grid points of the search grid are adjusted (in particular as described above).

[0065] A second aspect of the invention relates to a light microscope for locating or tracking emitters in a sample, in particular according to a method according to the first aspect.

[0066] The light microscope has an illumination optics which is designed to illuminate the sample in at least one search step with illumination light at a grid point of a search grid, wherein the illumination light induces or modulates light emissions from emitters in the sample, and to illuminate the sample in a localization step or a tracking step with an intensity distribution of the illumination light or a light which induces or modulates light emissions from the emitters in the sample, wherein the intensity distribution has a local minimum.

[0067] The light microscope also has at least one detector which is designed to detect detection light from the sample for the respective grid point in the at least one search step and to detect light emissions from the emitter for different positions of the intensity distribution in the localization or tracking step.

[0068] Furthermore, the light microscope comprises a computing unit which is designed to store search data in the at least one search step on the basis of the detection light detected for the respective grid point and to determine on the basis of the light emissions detected from the sample in the at least one search step whether an individual emitter is located in an area around the respective grid point and to calculate position data of the emitter in the localization or tracking step on the basis of the detected light emissions and the positions of the local minimum of the light distribution associated with the light emissions and to determine respective estimated positions from the position data of several stationary emitters or to determine respective trajectories from the position data of several emitters moving in the sample.The computing unit is designed to determine reference image data of an environment of the estimated positions or trajectories from the stored search data.

[0069] According to one embodiment, the light microscope has a control unit connected to the computing unit, which is configured to control the illumination optics such that the sample is illuminated at illumination positions. These illumination positions can, in particular, be determined automatically by the computing unit based on the light emissions detected by the detector. The control unit and the computing unit can also be the same device or be integrated into the same device.

[0070] The illumination optics may in particular comprise at least one light source (e.g. a laser) for generating the illumination light and a light modulator (e.g. a phase plate or a phase-modulating SLM with controllable pixels) for shaping the illumination light so that the intensity distribution with the local minimum is formed in the sample.

[0071] According to a further embodiment, the light microscope has an objective lens designed to focus the illumination light into the sample.

[0072] According to a further embodiment, the light microscope comprises a first detection device for detecting the light emissions in the first detection channel and a second detection device for detecting the light emissions in the second detection channel.

[0073] According to a further embodiment, the light microscope has a display unit which is designed to display overview images generated on the basis of the reference image data, in particular superimposed with the determined positions of the individual emitters or the trajectories.

[0074] According to a further embodiment, the light microscope has a first scanning device and a second scanning device, wherein the first and second scanning devices are designed to shift the illumination light relative to the sample. In particular, the first scanning device can be arranged in an illumination beam path of the light microscope between a light source and a beam splitter, wherein the beam splitter separates the detected light emissions from the illumination light. In particular, the second scanning device can be arranged in a combined illumination and detection beam path between the beam splitter and an objective of the light microscope. The first scanning device can in particular have at least one electro-optical deflector. The second scanning device can in particular have at least one galvanometer scanner.

[0075] According to a further embodiment, the light microscope has a detection pinhole in front of the detector.

[0076] According to a further embodiment, the control unit is designed to control the first scanning device and the second scanning device, in particular during the at least one search step, such that a projection of the detection pinhole into the sample is moved on a path, in particular a circular path, while the illumination light is kept stationary relative to the sample.

[0077] According to a further embodiment, the detector has a plurality of detector elements arranged in a detection plane. In particular, the detector elements can be individually readable. In particular, the detector elements can be designed to detect individual photons. Such a detector can be, for example, an APD array.

[0078] A third aspect of the invention relates to a computer program comprising instructions that cause the light microscope according to the second aspect to carry out the method according to the first aspect.

[0079] Further, in particular functional, features of the light microscope according to the second aspect and of the computer program according to the third aspect result from the features of the method according to the first aspect.

[0080] Advantageous developments of the invention will become apparent from the patent claims, the description, the drawings, and the accompanying explanations of the drawings. The described advantages of features and / or combinations of features of the invention are merely exemplary and may be effective alternatively or cumulatively.

[0081] With regard to the disclosure content (but not the scope of protection) of the original application documents and the patent, the following applies: Further features can be found in the drawings – in particular the relative arrangements and operative connections shown. The combination of features of different embodiments of the invention or of features of different patent claims is also possible, deviating from the selected references of the patent claims, and is hereby suggested. This also applies to features that are shown in separate drawings or mentioned in their description. These features can also be combined with features of different patent claims. Likewise, features listed in the patent claims can be omitted for further embodiments of the invention; however, this does not apply to the independent patent claims of the granted patent.

[0082] The reference signs contained in the patent claims do not represent a limitation of the scope of the subject-matter protected by the patent claims. They serve solely to make the patent claims easier to understand.

[0083] In the following, exemplary embodiments of the invention are described with reference to figures. These do not limit the subject matter of this disclosure or the scope of protection. Short description of the characters Fig. 1 shows a schematic representation of a sample area on which the method according to the invention is carried out; Fig. 2 shows a light microscope according to the invention. Description of the characters

[0084] Fig. Figure 1 schematically shows a section of a sample 2 with a sample structure 7, which is examined using the method according to the invention according to an exemplary embodiment. Fig. 1 shows an exemplary Cartesian search grid 3 with grid points 4, wherein the sample 2 is illuminated in at least one search step of the method according to the invention at a respective grid point 4 with an intensity distribution 5 of an illumination light B, wherein the intensity distribution 5 has a central, local minimum 6 and adjacent intensity increase regions. The illumination light B is in particular excitation light, which excites individual emitters E in the sample 2 to fluoresce. In this case, the method according to the invention is a MINFLUX method. The intensity distribution 5 is in particular a donut or a bottle beam (3D donut) with a central intensity zero. In other embodiments not shown here, a regular focus of the illumination light B with a central intensity maximum can also be used in the at least one search step.

[0085] In the at least one search step, detection light emanating from the sample 2 is detected for the respective grid point 4 of the search grid 3. The detection light can include light emissions that indicate the presence of an emitter E in the vicinity of the respective grid point 4. Optionally, the light emissions can be detected location-dependently in a detection plane, e.g., by pinhole orbit scanning or by means of a spatially resolving detector with multiple detector elements.

[0086] If light emissions are detected which indicate the presence of an emitter, a localization step, in particular according to the MINFLUX principle, is carried out following the at least one search step, in which the position of the emitter E is determined with high accuracy (in particular in the single-digit nanometer range), or a tracking step is carried out in which a trajectory of an emitter E moving in the sample 2 is recorded by means of several temporally successive localizations.

[0087] For this purpose, for example, an intensity distribution of excitation light with a central intensity zero (e.g., a donut or a bottle beam) can be successively placed at illumination positions that form an illumination pattern around a roughly estimated position of the emitter E based on the light emissions detected in the last search step. The illumination positions can, for example, be arranged on a circle whose center is the roughly estimated position of the emitter E. For each of the illumination positions, light emissions from the emitter E can then be detected, and the value pairs of light emissions and the corresponding position of the minimum of the intensity distribution can form the input values ​​for a position estimation algorithm, e.g., for a maximum likelihood estimator or a least mean square estimator. The position estimation algorithm, calibrated if necessary, then provides an updated, more accurate estimate for the position of the emitter E in sample 2.This process can be repeated iteratively, particularly during a localization step at an emitter E stationary in sample 2. This means that sample 2 is again illuminated at illumination positions with the intensity distribution of the illumination light B, which this time form an illumination pattern around the updated estimated position. In particular, the illumination positions can be arranged on a smaller circle around the estimated position than before. Optionally, the total intensity of the illumination light can be increased to create a steeper intensity gradient near the zero point.

[0088] The roughly estimated position (pre-localization) of the emitter E can be calculated, in particular, by evaluating light emissions recorded at various positions in the detection plane. Alternatively, it is also possible, for example, to use the position of the corresponding grid point 4 as the roughly estimated position.

[0089] According to the invention, the light detected in the at least one search step is not only used to locate a single emitter E in the sample 2 and, if necessary, to roughly estimate its position, but reference image data are also created from the stored search data.

[0090] For example, an overview image can be created from the search data and optionally assigned to the detected emitter E and the position subsequently estimated for this emitter E in the subsequent localization step or the trajectory obtained in the subsequent tracking step.

[0091] The corresponding overview images can, for example, be overlaid with the estimated positions of individual emitters E or the trajectories in order to display them in the context of a temporally changing sample environment.

[0092] Alternatively or in addition to generating overview images, the estimated positions or trajectories can also be corrected using the reference image data. For example, if the reference image data reveals that sample structures 7 are moving within the sample 2 while a plurality of localizations are being performed or a plurality of trajectories are being recorded, this correction can, for example, comprise a transformation of the estimated positions to a reference system that corresponds to or is defined by the corresponding sample structure 7.

[0093] According to a simple example, trajectories of individual emitters E located within a sample structure 7, e.g. an organelle of a biological cell, can be related to a length coordinate of the sample structure 7 when the latter itself moves within the sample 2 or changes its shape.

[0094] Fig. 2 shows a light microscope 1 according to the invention, which is configured to perform a MINFLUX method. The light microscope 1 has an illumination optics 10, which comprises a light source 11, in particular a laser, for generating the illumination light B, and a light modulator 13 for amplitude and / or phase modulation of the illumination light B. The light modulator 13 can be, for example, a phase modulator or an SLM. The illumination light B is, in particular, excitation light, which excites emitter E in the sample 2 to fluoresce. If the light microscope 1 is alternatively configured for a STED-MINFLUX method, the illumination light B can, in particular, be inhibition light (further, in particular, STED light or switching light), and the light microscope 1 can have a further light source for excitation light, which is, for example, combined with the inhibition light at an additional beam splitter (not shown).

[0095] The light modulator 13 can, for example, phase-modulate the illumination light in such a way that an intensity distribution 5 with a, in particular central, local minimum 6 is formed at the focus in the sample 2, to which areas of intensity increase are adjacent.

[0096] Located in the beam path of the illumination light between the light source 11 and the light modulator 13 is a first scanning device 12, which is designed to shift a focus of the illumination light B formed by the objective 16 in the sample 2, in particular in a focal plane that runs perpendicular to the optical axis of the objective 16. The first scanning device 12 can have at least one electro-optical deflector, in particular at least two electro-optical deflectors, which deflect the illumination light successively in two mutually perpendicular directions.

[0097] To shift the focus of the illumination light in the z-direction, ie along the optical axis of the objective 16, the light microscope 1 can optionally additionally have an axial scanning device, e.g. a deformable mirror or a lens with variable focus (not shown).

[0098] The illumination light B passes through a beam splitter 14 and a second scanning device 15, which may, for example, comprise at least one galvanometric scanner. The second scanning device 15 allows, in particular, a slower coarse positioning of the illumination light B in the sample 2 compared to the first scanning device 12, with a larger image field than the first scanning device 12.

[0099] The illumination light B is focused by the objective 16 into the sample 2, where the intensity distribution 5 coincides with the local minimum 6 (see Fig. 1) is formed.

[0100] Light emissions emanating from the sample 2, in particular fluorescent light or reflected light emitted by individual emitters E in the sample 2, are bundled by the objective 16, descanned by the second scanning device 15 and reflected into a detection beam path by the beam splitter 14, which is designed in particular as a dichroic beam splitter.

[0101] There, the emission light passes through an optional confocal detection pinhole 17 and is then detected by the detector 18. The detector 18 can be a point detector (e.g., an APD) or a spatially resolved detector (e.g., an APD array).

[0102] If the light emissions are detected in two different detection channels during at least one search step, the detection light can be spectrally split, for example, between the detection pinhole 17 and the detector 18 and directed to another detector (not shown). Alternatively, different detection channels can also be selectively detected with the same detector 18.

[0103] The detector 18 is connected to a computing unit 19, which is designed to evaluate the detection light received by the detector 18 and the light emissions detected by the detector 18. The computing unit 19 can, in particular, estimate positions of individual emitters E in the sample based on the detected light emissions and the associated positions of the illumination light B, calculate trajectories of individual emitters E, store the search data in a storage unit, generate reference image data from the search data, generate one or more overview images based on the reference image data, and / or correct the estimated positions based on the reference image data.

[0104] The computing unit 19 is also connected to a display unit 21, which can be designed in particular to display the overview images, in particular superimposed with the estimated positions or trajectories.

[0105] Furthermore, the computing unit 19 is connected to a control unit 20, which can control at least the first scanning device 12 and the second scanning device 15 in order to determine the illumination light B, in particular the local minimum 6 of the intensity distribution 5 (see Fig. 1) of the illumination light B, at a desired position in the sample 2. List of reference symbols 1 light microscope 2 Sample 3 search grids 4 Grid point 5 Intensity distribution 6 Local Minimum 7 Sample structure 10 Lighting optics 11 Light source 12 First scanning device 13 Light modulator 14 beam splitters 15 Second scanning device 16 lens 17 Detection pinhole 18 detector 19 Computing unit 20 Control unit 21 Display unit B Illumination light E Emitter

Claims

[1] Method for locating or tracking emitters (E) in a sample (2), wherein - the sample (2) is illuminated with illuminating light (B) in at least one search step at a grid point (4) of a search grid (3), wherein the illuminating light induces or modulates light emissions from emitters (E) in the sample (2), wherein detection light from the sample (2) is detected for the respective grid point (4), wherein search data are stored on the basis of the detection light detected for the respective grid point (4), and wherein in the at least one search step, it is determined on the basis of light emissions detected from the sample (2) whether an individual emitter (E) is located in an area around the respective grid point (4), - the sample (2) is illuminated in a localization step or a tracking step with an intensity distribution (5) of the illuminating light (B) or another light that induces or modulates light emissions from the emitters (E) in the sample (2), wherein the intensity distribution (5) has a local minimum (6), and wherein light emissions from the emitter (E) are detected for different positions of the intensity distribution (5), and wherein position data of the emitter (E) are calculated on the basis of the detected light emissions and the positions of the local minimum (6) of the light distribution associated with the light emissions, and wherein respective estimated positions are determined from the position data of several stationary emitters (E) or respective trajectories are determined from the position data of several emitters (E) moving in the sample (2), characterized bythat reference image data of an environment of the estimated positions or trajectories are determined from the stored search data. [2] Method according to claim 1, characterized by that the sample (2) is illuminated in the at least one search step and in the localization or tracking step with the same intensity distribution (5) of the same illumination light (B), wherein the intensity distribution (5) has the local minimum (6). [3] Method according to claim 2, characterized by that the detection light is detected in a detection plane in a location-dependent manner in the at least one search step, in particular wherein the detection plane is confocal to a focal plane of the illumination light (B) in the sample (2). [4] Method according to claim 3, characterized bythat the projection of a detection pinhole (17) into the sample (2) is moved on a, in particular circular, path, while the illumination light (B) is stationary relative to the sample (2). [5] Method according to claim 3, characterized by that the detection light is detected by a detector (18) having a plurality of detector elements. [6] Method according to one of the preceding claims, characterized by that at least one overview image of the sample (2) is generated from the reference image data. [7] Method according to claim 6, characterized by that the at least one overview image is superimposed with at least one of the estimated positions or at least one of the trajectories and is displayed by means of a display unit (21). [8] Method according to claim 6 or 7, characterized bythat a temporal sequence of overview images is generated from the stored search data, in particular wherein positions estimated at corresponding times or trajectories recorded at corresponding times are superimposed with respective overview images of the temporal sequence and displayed by means of the display unit (21). [9] Method according to one of claims 6 to 8, characterized by that on the basis of the detection light which was detected in the at least one search step for the respective grid point (4), pixel intensities of pixels of the at least one overview image which correspond to sample positions between grid points (4) of the search grid (3) are calculated, in particular interpolated. [10] Method according to one of the preceding claims, characterized by that the estimated positions or trajectories are processed and / or corrected based on the reference image data. [11] Method according to claim 10, characterized by that at least one structure of the sample (2) is identified on the basis of the reference image data, wherein the estimated positions or the trajectories are processed and / or corrected with respect to the at least one structure, in particular wherein the at least one structure changes on a time scale of the determination of the estimated positions or the recording of the trajectories and / or is displaced relative to the sample (2). [12] Method according to one of the preceding claims, characterized by that the grid points (4) are adjusted based on the reference image data. [13] Method according to one of the preceding claims, characterized byin that in the at least one search step, background light is detected in the event that there are no light-emitting emitters (E) in the vicinity of the respective grid point (4), wherein a background correction of the position data is carried out on the basis of the detected background light, and wherein a frequency or a duration of the detection of background light is adjusted in order to adjust a frame rate with which reference image data is determined from the search data. [14] Method according to one of the preceding claims, characterized bythat the light emissions from the sample (2) are detected in a first detection channel and the detection light is detected in a second detection channel different from the first detection channel, wherein the position data of the emitter (E) are determined on the basis of the light emissions detected in the first detection channel, and wherein the search data used to determine the reference image data are determined on the basis of the detection light detected in the second detection channel. [15] Method according to claim 14, characterized bythat on the basis of the detection light detected in the second detection channel, it is checked whether an estimated position of an emitter (E) or a trajectory is located in a region of interest of the sample (2), in particular wherein, in the event that it is determined that the estimated position of the emitter (E) or the trajectory is not located in the region of interest, grid points (4) of the search grid (3) are adjusted. [16] Light microscope (1) for locating or tracking emitters (E) in a sample (2), in particular according to a method according to one of claims 1 to 15, comprising - an illumination optics (10) designed to illuminate the sample (2) in at least one search step at a grid point (4) of a search grid (3) with illumination light (B), wherein the illumination light (B) induces or modulates light emissions from emitters (E) in the sample (2), and to illuminate the sample (2) in a localization step or a tracking step with an intensity distribution (5) of the illumination light (B) or another light which induces or modulates light emissions from the emitters (E) in the sample (2), wherein the intensity distribution (5) has a local minimum (6), - at least one detector (18) which is designed to detect detection light from the sample (2) in the at least one search step for the respective grid point (4) and to detect light emissions from the emitter (E) in the localization or tracking step for different positions of the intensity distribution (5), - a computing unit (19) designed to store search data in the at least one search step based on the detection light detected for the respective grid point (4) and to determine, on the basis of light emissions detected from the sample (2) in the at least one search step, whether an individual emitter (E) is located in an area around the respective grid point (4), and to calculate, in the localization or tracking step, position data of the emitter (E) based on the detected light emissions and the positions of the local minimum (6) of the light distribution associated with the light emissions, and to determine respective estimated positions from the position data of several stationary emitters (E) or to determine respective trajectories from the position data of several emitters (E) moving in the sample (2), characterized bythat the computing unit (19) is designed to determine reference image data of an environment of the estimated positions or trajectories from the stored search data. [17] Computer program comprising instructions which cause the light microscope (1) according to claim 16 to carry out the method according to one of claims 1 to 15.

Images