Method for locating or tracking emitters in a sample, method for training a data processing network, data processing apparatus, optical microscope, and computer program

Abstract

The invention relates to a method for locating or tracking emitters in a sample (2), wherein the sample (2) is illuminated with an intensity distribution of an illuminating light (B) with a local minimum, and wherein light emissions (L) from a measurement emitter in the sample (2) are taken as a basis for producing location data (21), wherein a position of the measurement emitter in the sample (2) is estimated by a trained data processing network (20) by processing the location data (21), wherein the data processing network (20) is trained by means of training data, and to a method for training a data processing network (20), a data processing apparatus (6), an optical microscope (100) and a computer program for carrying out the method.

Description

[0001] Method for locating or tracking emitters in a sample, method for training a data processing network, data processing device, light microscope, and computer program

[0002] Technical field of the invention

[0003] The invention relates to a method for locating or tracking emitters in a sample, in particular a MINFLUX method, using a trained data processing network, in particular an artificial neural network, a method for training the data processing network, a data processing device with a memory for storing the trained data processing network, a light microscope, in particular a MINFLUX microscope, with the data processing device for carrying out the method for locating or tracking emitters in a sample and a computer program comprising program code for carrying out the method for locating or tracking an emitter by means of the data processing device or by means of the light microscope.

[0004] State of the art

[0005] Various artificial intelligence methods are known from the state of the art, with which it is possible, among other things, to automatically classify input data or input signals.

[0006] So-called deep learning methods are particularly well-suited for classifying image data. Deep learning methods belong to the so-called representation learning methods, meaning they are capable of learning from annotated raw data. Another distinguishing feature of deep learning methods is that they create representations of the data in different layers.

[0007] An overview of deep learning methods, especially neural networks, is provided, for example, in the publication “Deep Learning” by Y. LeCan, Y. Bengio and G. Hinton, Nature 521 (2015), 436-444.

[0008] Artificial neural networks (ANNs) are data processing networks that can schematically replicate structures in the animal and human brain. They consist of processing nodes organized into an input layer, an output layer, and typically a multitude of hidden layers arranged between the input and output layers.

[0009] Each node receives input data, processes it with a non-linear function, and outputs output data to subsequent nodes. The nodes of the input layer receive input data (training data or test data). The nodes of the hidden layers and the output layer typically receive the output data from several nodes of the previous layer in the data flow direction. Weights are defined (at least implicitly) for all connections between nodes, i.e., relative proportions with which the input data is considered when processing with the non-linear function. A network can be configured for a specific task, e.g.The segmentation or classification of image data is trained by processing the training data from the network, applying an error function to the result whose value reflects the agreement of the determined result with the correct result, and adjusting the weights between the nodes based on the error function. For this purpose, a gradient of the error function can be determined for each weight using a process called backpropagation, for example, and the weights can be adjusted based on the steepest gradient (gradient descent method).

[0010] Convolutional neural networks (CNNs) are a subgroup of neural networks that contain so-called convolutional layers, which are typically followed by pooling layers. In convolutional layers, the data transfer between two layers can be represented by a convolution matrix (also called a kernel or filter bank), i.e., each input node receives as input data the inner product of the output data of a subset of the nodes of the previous layer and the convolution matrix. In so-called pooling, the output data of one layer is transferred to a layer with a smaller number of nodes, with the output data of several nodes being combined.

[0011] Such convolutional neural networks are particularly advantageous in image processing, as the convolutional layers greatly improve the recognition of local structures and shift invariance.

[0012] Recurrent neural networks or feedback neural networks are neural networks in which nodes of one layer are connected to nodes of the same layer or to nodes of a layer preceding it in the direction of data flow. Such networks are fundamentally capable of storing representations of inputs from the immediate past, thus possessing a short-term memory. A special form of recurrent neural network, the so-called "long short-term memory" (LSTM) networks, and corresponding training procedures were first described by S. Hochreiter and J. Schmidthuber in the article "Long Short-Term Memory" (1997, Neural Computation 9(8), 1735-1780). Such networks are characterized by so-called memory cells, whose inputs and outputs are regulated via multiplicative gate units. The memory cells each contain a self-recurrent connection (feedback with a delay from one memory).According to the aforementioned publication, the network is trained using a variant of a real-time recurrent learning algorithm with truncated error propagation. This allows a constant error flow to be achieved in the network during training. LSTM networks can store representations of inputs from significantly further back in time than conventional recurrent neural networks ("long short-term memory").

[0013] In the field of image processing of microscopic data, artificial intelligence methods, in particular artificial neural networks, have already been used for a variety of tasks, for example for the segmentation of image data (see, for example, “Best Practices in Deep-Learning-Based Segmentation of Microscopy Images” by T. Scherr, A. Bartschat, M. Reischl, J. Stegmaier and R. Mikut (Proc. 28 Workshop Computational Intelligence, Dortmund, November 29-30, 2018, pp. 175-195).

[0014] Localization and tracking techniques for individual emitters are newer and specialized methods within the field of light microscopy. In contrast to classical imaging techniques in light microscopy, these methods calculate the positions of individual emitters (e.g., fluorophores or molecules labeled with fluorophores) based on detected light emissions. A localization map can be created from the positions of a large number of emitters, usually determined one after the other, which visualizes the distribution of the emitters in the sample. Depending on the localization precision of the corresponding imaging technique, localization microscopy localization maps can have a resolution well below the diffraction limit.If positions of the same emitter moving in the sample are determined in rapid succession, the same principle can also be applied to determine trajectories of the emitter in the sample, i.e. to track the emitter (also called tracking).

[0015] The methods of localization microscopy can be applied to individual emitters. The term "individual" means that at a given time, light-emitting emitters whose emission light is indistinguishable are separated by a distance above the diffraction limit, so that their emission light can be separated. This can be achieved, for example, with permanently light-emitting emitters by a labeling density of the sample below a threshold value. However, if the emitters blink asynchronously, for example, a sufficient separation can be achieved at any given time even with a higher labeling density. For this purpose, the sample environment (e.g., buffer, embedding medium) can be chemically configured to result in a desired blinking rate. Smaller distances are also tolerable when different types of emitters are present whose emission light is optically separable, e.g.through different emission spectra or emission lifetimes. Finally, in exceptional cases, special evaluation methods (e.g., statistical or time-resolved methods) may also make it possible to jointly localize groups of several closely adjacent emitters. This can determine a specific position for each emitter, or an average position of several emitters can be determined.

[0016] In the so-called MINFLUX technique, a sample containing a single emitter (e.g., a single fluorophore, a fluorophore-labeled molecule, or a light-scattering particle) is illuminated with an intensity distribution of excitation light at illumination positions in a range around a roughly estimated position of a single emitter. The intensity distribution has a central intensity minimum (ideally, an intensity zero). For each illumination position, the light emissions (in particular, photon rates) of the individual emitter are recorded. A new position estimate of the emitter is then calculated from the light emissions and the associated illumination positions. The intensity distribution can, for example, be a so-called 2D donut, a so-called bottle beam, or a superposition of these light distributions, which are known from the field of STED microscopy.

[0017] Various variants of the MINFLUX technique are described, for example, in the publications F. Balzarotti et al. (2017) Nanometer resolution imaging and tracking of fluorescent molecules with minimal photon fluxes, Science 355 (6325), 606-612 and KC Gwosch et al. (2020) MINFLUX nanoscopy delivers 3D multicolor nanometer resolution in cells, Nat. Methods, 17 (2), 217-224 and R. Schmidt et al. (2021) MINFLUX nanometer-scale 3D imaging and microsecond-range tracking on a common fluorescence microscope, Nat. Commun. 12 (1), 1478 as well as in the patent documents US 9,719,928 B1 , US 10,900,901 B2, US 10,908,089 B2 and US 10,962,479 B2.

[0018] Due to the excitation light distribution with an intensity minimum, the MINFLUX technique allows for localization precision in the single-digit nanometer range with a significantly lower number of emitted photons compared to the so-called PALM / STORM technique. This is primarily due to the fact that the emitter is illuminated with less excitation light the closer the minimum of the excitation light distribution is to the actual emitter position.

[0019] Patent publication US 11,255,791 B2 describes, among other things, a variant of the MINFLUX technique in which the sample is illuminated with a combination of an excitation light distribution with a local maximum and a STED (stimulated emission cfep / et / on) light distribution with a local minimum. The position of an individual emitter is also calculated from the light emissions recorded for various illumination positions, but the closer the minimum of the STED light distribution (and the maximum of the excitation light distribution superimposed with the minimum) is to the actual emitter position, the more light is emitted. Therefore, the advantage of particularly high photon efficiency is not achieved here.

[0020] Patent application US 2023 / 0236401 A1 describes a method for localizing an emitter by switching between different orientations of an excitation light distribution with a local minimum. This light distribution can be generated, for example, using the principle of conical diffraction, which is described, for example, in the publication "Conical diffraction as a versatile building block to implement new imaging modalities for superresolution in fluorescence microscopy" by C. Fallet, J. Caron, S. Oddos, JY. Tinevez, L. Moisan, GY Sirat, PO Braitbart, and SL Shorte (2014, Proc. of SPIE Vol. 9169, 916905-1 - 916905-6).

[0021] A special localization technique using an excitation light distribution with a local maximum and a STED light distribution with a local minimum is described in the publication M. Weber et al. (2021) MINSTED fluorescence localization and nanoscopy, Nat. Photonics 15, pp. 361-366 and the patent application WO 2023 / 006176 A1 and is known as MINSTED. The combination of the excitation light distribution and the STED light distribution is moved along a continuous trajectory around an estimated emitter position. For each detected photon, the corresponding position of the minimum of the STED light distribution is determined, and the trajectory is adjusted based on the detected photon. The excitation and STED intensity can also be increased.

[0022] Position estimation algorithms for the localization and / or tracking of individual emitters in a sample, particularly those based on the MINFLUX principle, suffer from the problem of systematic error (bias) dependent on the actual position. Therefore, position estimators must be calibrated. Such calibration can depend, in particular, on the shape and positioning of the intensity distribution of the illumination light, the light intensity of the illumination light, and the intensity of background light, and therefore sometimes yields results of varying quality under different experimental conditions. Certain iterative MINFLUX methods based on the state of the art can nevertheless produce highly reproducible and highly accurate position estimates. However, these methods are time-consuming, which is particularly disadvantageous for tracking methods of rapidly moving emitters and for creating localization maps of larger sample areas.

[0023] Furthermore, special problems arise in position estimation when multiple emitters are located in the excitation volume simultaneously. Often, only one of these emitters is ultimately localized, because one of the emitters is bleached first, and the position estimate then converges to the actual position of the remaining emitter. This leads to an unsatisfactory density of localizations of different emitters.

[0024] Object of the invention

[0025] Based on the disadvantages of the prior art discussed above, the object of the present invention is to provide a method for localizing or tracking an emitter in which the position estimation is improved, in particular with regard to systematic errors and / or multiple emitters in the excitation volume.

[0026] Solution

[0027] This problem is solved by the subject matter of the independent claims. Advantageous further developments are specified in the subclaims and are described below.

[0028] Description of the invention

[0029] A first aspect of the invention relates to a method for localizing or tracking emitters in a sample, wherein the sample is illuminated with an intensity distribution of an illuminating light having a local minimum, and wherein localization data are created on the basis of light emissions of a, in particular individual, measuring emitter in the sample, wherein a position of the measuring emitter in the sample is estimated by means of processing the localization data by a trained data processing network, wherein the data processing network is trained using training data.

[0030] A trained data processing network can replace a conventional position estimation algorithm, e.g., a MINFLUX method. This allows for highly flexible position estimation from various types of localization data using artificial intelligence, provided appropriate training data is available. Such a data processing network can function as a black-box system, into which localization data is read, and which subsequently outputs a position estimate without requiring knowledge of the exact way in which the data processing network performs the estimation.

[0031] In particular, some systematic errors (bias) can be avoided by training the data processing network to implicitly account for the dependencies underlying these errors. This advantageously eliminates the need to calibrate a position estimator, which is used to account for the systematic error in state-of-the-art estimation methods.

[0032] Furthermore, the data processing network could also be trained for special applications that cannot be handled or cannot be satisfactorily handled using the known position estimation algorithms, e.g. the parallel or simultaneous estimation of positions of several emitters.

[0033] In this application, emitters are understood to be objects that, when illuminated with excitation light, can be considered point light sources with regard to the measurements according to the invention. The light emitted by the object acting as a point light source can, for example, be scattered light resulting from elastic scattering such as Rayleigh scattering or inelastic scattering such as Raman scattering, or it can be luminescent light, in particular fluorescent light. An emitter can therefore be, for example, a light-reflecting nanoparticle, a quantum dot, a fluorescent dye molecule (fluorophore), or a molecule or nanoparticle labeled with one or more fluorescent dye molecules.Depending on the size of the molecule and the distance between the fluorophores, a molecule labeled with multiple fluorophores or a nanoparticle labeled with multiple fluorophores can of course also have multiple emitters in the sense of the definition used here.

[0034] In the context of this specification, "individual" emitters are understood to be emitters that can be optically separated from one another using light microscopy techniques. This can be achieved by a labeling density of the sample that results in an average distance between the emitters above the diffraction limit. Alternatively, asynchronously flashing emitters can be used if the sample conditions (in particular the composition of the sample buffer and embedding medium) are adjusted so that the average distances between the light-emitting emitters are above the diffraction limit at all times. Finally, different emitters that are otherwise optically distinguishable can, under certain circumstances, also be separated if they are separated below the diffraction limit, e.g., based on their characteristic emission spectrum or their emission lifetime.Optionally, a spatially limited area of ​​the sample can be illuminated with activation light in order to convert emitters in this area from an inactive state in which the emitters do not emit light when irradiated with excitation light, to an active state in which the emitters emit light when irradiated with excitation light.

[0035] The terms "measurement emitter" and "training emitter" are used in this application only to distinguish whether an emitter is used for training the data processing network (training emitter) or whether the position of an emitter is to be estimated using a previously trained data processing network. Apart from that, the term "emitter" is not intended to be restricted in any way by this distinction. Both measurement emitters and training emitters can belong to all emitter species described above. An emitter can also be both a measurement emitter and a training emitter, e.g., if the data processing network is further trained during the ongoing measurement.

[0036] In a localization method, a high-resolution image of structures in the sample can be determined by successively determining the positions of a plurality of emitters in the sample, in particular with a spatial resolution, ie a localization precision, of 1 nm to 10 nm, more particularly 1 nm to 5 nm.

[0037] A tracking method is characterized by the fact that light emissions from an emitter moving in the sample are recorded several times in succession in order to estimate the respective positions of the emitter. From the positions of the emitter, a trajectory can then be determined which describes the movement of the emitter in the sample.

[0038] The illumination light can, in particular, be excitation light, which excites an emitter in the sample to emit light, i.e., induces light emission from the emitter. The light emissions can, in particular, be reflected light, scattered light, or luminescence light (e.g., fluorescent light). Alternatively, the illumination light can also modulate light emissions from the emitter. In this case, the illumination light can, for example, be STED light, which causes emitters to de-excite from an excited state to the ground state, or switching light, which converts emitters from an excitable state to a dark state.

[0039] The sample is illuminated with an intensity distribution of the illumination light which has a local minimum. In particular, regions of intensity increase border the local minimum in at least one spatial direction. The local minimum is in particular a central minimum of the intensity distribution, thus forming a center of the intensity distribution, whereby the center can in particular be located at the geometric focus. In this case, the light distribution can in particular be point-symmetric with respect to the geometric focus. The local minimum can in particular be at least approximately an intensity zero. Such intensity distributions include in particular a so-called donut beam and a so-called bottle beam.The intensity distribution can, in particular, also have two intensity maxima separated by a plane of minimum intensity, in particular a zero plane, where the zero plane intersects the geometric focus, and where, in particular, the zero plane can be perpendicular to the focal plane, i.e., parallel to an optical axis of the objective. Such light distributions can be generated, for example, by phase modulation of the illuminating light in or near a pupil of an objective lens (e.g., with a phase plate or a spatial light modulator) and focusing it into the sample using the objective lens. A donut can be generated, for example, by a so-called vortex phase pattern (also called a phase clock), a bottle beam by an annular phase jump (in particular with the phase difference n), and a light distribution with two maxima separated by a zero plane by a linear phase jump (in particular with the phase difference TT).

[0040] The localization or tracking method according to the invention can, in particular, be a so-called MINFLUX method (when the illumination light with which the sample is illuminated in the localization step is excitation light that induces the light emissions of the emitters) or a STED-MINFLUX method (when the illumination light with which the sample is illuminated in the localization step is inhibition light that modulates the light emissions of the emitters). In a MINFLUX method or STED-MINFLUX method, the local minimum of the intensity distribution of the illumination light is placed, in particular, at illumination positions in a near range of the approximate position of an isolated emitter, and photons emitted by the emitter are detected for each illumination position. The near range can, in particular, have an extent on the order of the optical diffraction limit.The position estimation from the light emissions can be repeated iteratively by placing the intensity distribution of the illumination light at updated illumination positions in a range close to the updated position estimate and again detecting photons for each position. In particular, a radius or an extension of the trajectory around the previously estimated position can be reduced. Optionally, the light intensity of the illumination light can also be increased. The iteration steps can be continued, for example, until the emitter stops emitting light or until a photon limit or a threshold of the localization precision is reached. In a MINFLUX or STED-MINFLUX tracking method, one or more iterations can be repeated at short intervals to track the trajectory of a moving emitter.In particular, illumination patterns specifically tailored to a tracking method can be used. The data processing network according to the invention can be stored as a data structure on a storage medium. It is capable of processing input data (localization data or training data) provided to the data processing network using arithmetic operations and outputting one or more position estimates as output data.

[0041] The localization data and the training data can have a variety of data formats, whereby it is advantageous, but not mandatory, that the localization data have the same format as the training data.

[0042] According to one embodiment, the localization data comprises pairs of values ​​of the light emission values ​​of the measuring emitter and the illumination positions of the local minimum of the intensity distribution of the illumination light associated with the light emission values. In the corresponding MINFLUX method, the intensity distribution can, in particular, be shifted stepwise to the corresponding illumination positions (e.g., using a beam scanner), with the corresponding light emissions (e.g., photon numbers or light intensities) being detected by a detector at each illumination position during a certain dwell time.

[0043] In particular, these value pairs can also be represented as vectors, where an index of the vector indicates the illumination positions and the components of the vector represent the light emissions (e.g., photon counts or intensities). Another possible representation would be a 2xk or kx2 matrix, where k is the number of illumination positions.

[0044] As an alternative to stepwise displacement, the local minimum of the intensity distribution can also be moved along a continuous trajectory. Then, for example, specific intervals of the trajectory's locus can be assigned to specific illumination positions, or each light emission (in particular, each photon) can be assigned, based on its detection time, to an illumination position where the local minimum of the intensity distribution was located at that time.

[0045] According to a further embodiment, the localization data comprise value pairs of light emission values ​​of the measuring emitter and shapes and / or arrangements of the intensity distribution associated with the light emission values.

[0046] For example, MINFLUX methods are known in which an intensity distribution of excitation light with two maxima separated by an area of ​​minimum intensity (in particular, an intensity of zero) is rotated around the geometric focus, e.g., by switching the phase modulation of the illumination light, with light emissions from an emitter being recorded for each orientation. Such localization data can also be represented, e.g., in the form of a vector, where the index of the vector can indicate the various shapes and / or arrangements of the intensity distribution.

[0047] According to a further embodiment, the localization data consists of a time series of light emissions (without associated illumination positions).

[0048] In this case, the data processing network can estimate the position solely from analyzing the pattern of light emissions, whereby, for example, any periodic illumination patterns can be implicitly detected from the data. This advantageously simplifies the data acquisition of the localization or tracking process.

[0049] According to a further embodiment, the training data are localization data of at least one training emitter of known position in a training sample illuminated by the illumination light.

[0050] The training emitter can be immobilized in the sample. The position of the training emitter can be estimated in advance using an independent method, or a highly accurate MINFLUX localization of the training emitter can be performed first. Subsequently, the sample can be illuminated with the illumination light in a close range of the emitter to acquire the training data. The local minimum of the intensity distribution of the illumination light is positioned at illumination positions where light emissions are detected.

[0051] To prevent the training emitter from bleaching during the initial position determination, a particularly stable emitter can be used, such as a nanoparticle with coupled fluorophores or a light-reflecting nanoparticle. This also advantageously prevents fluctuations in light emissions and / or transitions between emitting states and dark states during the acquisition of the training data.

[0052] In other applications, however, such fluctuations in light emissions or such transitions during training data acquisition may actually be desirable, allowing the data processing network to be trained for such phenomena occurring in real-world measurements. In this respect, (individual) fluorophores can also be used as training emitters. This also has the advantage that, due to the generally lower photon emission rate, background light plays a greater role and is therefore incorporated into the training process.

[0053] More stable nanoparticles have the further advantage for training that a very large number of illumination positions can be controlled without bleaching the training emitter. In this way, for example, a large number of illumination positions could be controlled, forming a dense network around the actual position of the training emitter. According to another embodiment, the training data is simulation data.

[0054] Simulation data is training data calculated based on a mathematical model. Such a model can, in particular, take into account the shape and orientation of the intensity distribution of the illuminating light, the type (especially wavelength) and total intensity of the illuminating light (in particular represented by the peak intensity value or the integral over the intensity distribution), and the type of emitter (especially its photophysics).

[0055] A simulation of the training data has the particular advantage that a large amount of training data can be obtained with little effort.

[0056] According to a further embodiment, background light is taken into account in the simulation data.

[0057] In this way, the data processing network learns to take background into account when estimating the position, i.e. to distinguish at least implicitly between desired light emissions from the emitter to be located or tracked and background light.

[0058] In real experiments, background light can, for example, originate from other emitters located outside the focal plane of the emitter to be located or tracked. Another source of background light can (depending on the type of detector) be, for example, the dark count rate of the detector.

[0059] According to a further embodiment, the data processing network is an artificial neural network.

[0060] In the context of this specification, the term "artificial neural network" refers to a data processing network comprising a plurality of nodes organized into an input layer, at least one hidden layer, and an output layer, wherein each node converts input data into output data using a non-linear function, and wherein weights are defined (at least implicitly) between the input layer and a hidden layer, between a hidden layer and the output layer, and optionally (in the event that multiple hidden layers are provided) between different hidden layers, wherein the weights specify proportions with which the output data of a respective node are considered as input data of a node downstream of the respective node in a data flow direction. The weights can in particular also be defined by convolution matrices.

[0061] The definition of “neural network” according to this specification includes not only so-called convolutional neural networks (which are characterized by a convolution operation between convolutional layers and by pooling layers that summarize the input data in fewer nodes than the layer upstream in the direction of data flow), but also in particular so-called fully connected networks or multilayer perceptrons with exclusively fully connected layers, in particular of the same dimension.

[0062] A trained neural network is a neural network that has weights adjusted to a specific task by processing training data.

[0063] According to another embodiment, the neural network is a recurrent neural network. A recurrent neural network is a neural network that has at least one node connected to a node of the same layer or to a node of a layer preceding it in the data flow direction.

[0064] Recurrent neural networks can have a short-term memory, which in the context of the present invention has the advantage that the data processing network can remember certain patterns in the training data, which improves the position estimation from the localization data.

[0065] According to a further embodiment, the neural network is an LSTM network.

[0066] An LSTM network is a recurrent neural network comprising at least one memory cell, where the memory cell has a multiplicative input gate and a multiplicative output gate. "Multiplicative" means that the corresponding input or output data is multiplied by a function during the gate processing step.

[0067] According to a further embodiment, the localization data is provided to the data processing network as a time series. For example, the time series may comprise binary light emission values ​​(0 = no photon; 1 = detection of a photon) associated with timestamps, wherein the timestamps describe the time of detection of the photon by a detector.

[0068] According to a further embodiment, the localization data provided to the data processing network for position estimation are determined based on a window function applied to the time series. This means that a window function is applied to the time series to obtain partial data sets, wherein the partial data sets are provided separately to the data processing network, and wherein the data processing network estimates one or more positions of the emitter from each of the partial data sets.

[0069] According to a further embodiment, the time series is divided into sections, wherein the sections are assigned to different measurement steps of a localization measurement. For example, the sections (partial data sets) can be assigned to different iterations of an iterative MINFLUX method. Alternatively, the sections can also be assigned, for example, to an illumination position (position of the local minimum of the intensity distribution), a shape and / or orientation of the intensity distribution, or one of several emitters.

[0070] According to a further embodiment, the training data are or will be obtained with different values ​​of at least one parameter, in particular a measurement parameter or a simulation parameter.

[0071] In this way, the data processing network can learn to estimate the position of emitters from localization data obtained with different parameters.

[0072] According to a further embodiment, the at least one parameter comprises a size measure of an illumination pattern of illumination positions and / or a total intensity of the illumination light.

[0073] The size measure can, for example, be the diameter of a circular illumination pattern on which the illumination positions are located. In some state-of-the-art iterative MINFLUX methods, the size of the illumination pattern is successively reduced in the iteration steps to increase localization precision. In particular, the overall intensity can also be increased to obtain a sufficient signal despite the generally smaller distance between the actual emitter position and the minimum of the intensity distribution.

[0074] According to a further embodiment, the training data is generated based on simulation data or light emissions from multiple emitters, wherein the trained data processing network estimates the positions of multiple measuring emitters simultaneously or in parallel. In particular, the multiple measuring emitters are spaced apart below the diffraction limit.

[0075] Such parallel localization measurements cannot be performed with conventional estimation algorithms, or are difficult to perform. According to this embodiment, the data processing network according to the invention is trained with several simultaneously measured or simulated training emitters.

[0076] According to a further embodiment, the measuring emitters belong to the same emitter species. The positions of several measuring emitters, which are separated by a distance below the diffraction limit, can also be determined simultaneously, since a different spatial probability distribution of the photon emission results from several emitters than from a single, isolated emitter. According to a further embodiment, the measuring emitters belong to different emitter species, whereby the measuring emitters can differ in their excitation spectrum, their emission spectrum and / or their emission lifetime. In particular, a spectrally resolved detection of the light emissions takes place, i.e., detection in several different wavelength ranges or detection channels. For example, the emission light can be split and detected with several detectors of different spectral sensitivities. Then, for example, a ratiometric detection can take place, i.e.A relationship can be established between the intensities or photon counts measured by the various detectors to gather information about which emitter emitted the photon. Alternatively or additionally, an analysis of the emission lifetime can be performed based on time-resolved single-photon detection. This information can optionally be fed into the data processing network, e.g., via separate input data channels corresponding to the spectrally resolved detection channels.

[0077] According to a further embodiment, the position of several measuring emitters, in particular those isolated within the sample, is estimated sequentially using the data processing network, whereby a high-resolution image of structures in the sample is created from the estimated positions. This means that a localization map of the sample is created that resembles a high-resolution image.

[0078] According to a further embodiment, the position of the same measuring emitter moving within the sample is estimated several times in succession using the data processing network, whereby a trajectory of the measuring emitter is created from the estimated positions. Thus, a tracking process is performed.

[0079] According to a further embodiment, the position estimation is performed using the trained data processing network between or during measurement steps of a localization experiment, wherein in each measurement step, the sample is illuminated with the intensity distribution of the illumination light, and light emissions from the measuring emitter are recorded. This means that a live position estimation is performed during an experiment using the trained data processing network. Based on the estimated position, the intensity distribution of the illumination light can then be placed, for example, in the next iteration step. The input data can be fed into the data processing network in sections of appropriate size. A continuous data stream, to which a window function is applied, in particular, can also be used as input data.According to another embodiment, the position estimation is performed using the trained data processing network after completion of a localization experiment in order to post-process an already performed position estimation of the measuring transmitter. Thus, the trained data processing network is used to post-process existing localization data. This can improve the position estimation in some cases. In particular, a larger input data set can be used here, since, on the one hand, the computing time is not as critical as with a live position estimation, and, on the other hand, all raw data is already available.

[0080] According to a further embodiment, the data processing network is further trained based on the localization data from measurement emitters. The measurement data is simultaneously used as training data to further train the data processing network. Nevertheless, a pure training step can be performed initially before the data processing network is applied in order to train the data processing network to the extent that it delivers correct results with a sufficient probability.

[0081] A second aspect of the invention relates to a method for training a data processing network, wherein the data processing network is trained with training data for a position estimate of a measuring emitter in a sample according to the method according to the first aspect.

[0082] A third aspect of the invention relates to a data processing device comprising a data memory in which a data processing network trained with the method according to the second aspect is stored, wherein the data processing device has a computing unit which is designed to carry out the method for locating or tracking emitters in a sample according to the first aspect.

[0083] A fourth aspect of the invention relates to a light microscope comprising illumination optics configured to illuminate a sample with illumination light, wherein the illumination light forms an intensity distribution with a local minimum in the sample, and a detector configured to detect light emissions from a measuring emitter, wherein the light microscope comprises a data processing device according to the third aspect. A fifth aspect of the invention relates to a computer program comprising program code that causes the data processing device according to the third aspect and / or the light microscope according to the fourth aspect to carry out the method according to the first aspect.

[0084] Further embodiments and advantages of the training method according to the second aspect, the data processing device according to the third aspect, the light microscope according to the fourth aspect, and the computer program according to the fifth aspect emerge from the above description of the method according to the first aspect. Advantageous developments of the invention emerge from the patent claims, the description and the drawings, and the associated explanations of the drawings. The described advantages of features and / or combinations of features of the invention are merely exemplary and can be used alternatively or cumulatively.

[0085] With regard to the disclosure content (but not the scope of protection) of the original application documents and the patent, the following applies: Further features can be found in the drawings – in particular the relative arrangements and operative connections shown. The combination of features of different embodiments of the invention or of features of different patent claims is also possible, deviating from the selected references of the patent claims, and is hereby suggested. This also applies to features that are shown in separate drawings or mentioned in their description. These features can also be combined with features of different patent claims. Likewise, features listed in the patent claims can be omitted for further embodiments of the invention; however, this does not apply to the independent patent claims of the granted patent.

[0086] The reference signs contained in the patent claims do not represent a limitation of the scope of the subject-matter protected by the patent claims. They serve solely to make the patent claims easier to understand.

[0087] In the following, exemplary embodiments of the invention are described with reference to figures. These do not limit the subject matter of this disclosure or the scope of protection.

[0088] Short description of the characters

[0089] Fig. 1 shows a first embodiment of an inventive

[0090] data processing network;

[0091] Fig. 2 shows a second embodiment of an inventive

[0092] data processing network;

[0093] Fig. 3 shows an embodiment of a light microscope according to the invention;

[0094] Fig. 4 shows an embodiment of a data processing device according to the invention.

[0095] Description of the characters

[0096] Fig. 1 and Fig. 2 each schematically illustrate an exemplary data processing network 20, in particular an artificial neural network, with which an artificial intelligence method can be carried out within the framework of the method according to the invention. The data processing network 20 consists of nodes 23, which are organized in an input layer 22a (here consisting of a single node), five hidden layers 22b and an output layer 22c (likewise consisting of a single node), and are connected by means of connections 24. For the sake of simplicity, the data processing network 20 illustrated here is shown as a fully connected network. However, this does not restrict the subject matter of the invention in any way. For example, the data processing network 20 can also be a recurrent network, in particular an LSTM network.

[0097] The input layer 22a receives localization data 21 as input data. Node 23 of the input layer 22a applies a nonlinear function to the input data 21, whereby the result of the calculation operation is successively passed on to the nodes 23 of the hidden layers 22b downstream in the data flow direction, according to the example shown. After further calculation operations by the nodes 23 of the hidden layers 22b and the output layer 22c, position data 25 is obtained as output data.

[0098] Although only five hidden layers 22b are shown in Fig. 1 and Fig. 2 for clarity, real data processing networks 20 usually contain significantly more hidden layers 22b, for example hundreds to thousands.

[0099] For each of the connections 24 between the nodes 23 of adjacent layers 22a, 22b, 22c, weights are defined in particular which indicate the proportion of the output of a node 23 to the input of the node 23 downstream in the data flow direction.

[0100] Such a data processing network 20 can be trained according to the invention to calculate position data 25 of one or more emitters in a sample 2, ie a position estimate of the emitter(s), from localization data 21, in particular MINFLUX data.

[0101] For this purpose, experimentally obtained or simulated training data, which are in particular localization data of actual or simulated emitters of known position, are fed into the data processing network 20 and the estimated position (position data 25) is compared with the actual, previously known position. This means that an error function is applied to the result, the value of which reflects the agreement of the determined result with the correct result. Based on the results of the error function, the weights at the connections 24 between the nodes 23 are then adjusted, e.g., with a so-called backpropagation, whereby a gradient of the error function is determined for each weight, and the weights are adjusted based on the steepest gradient. Alternatively, e.g.,A so-called real-time recurrent learning algorithm with truncated error propagation can also be used, especially if the data processing network 20 is an LSTM network.

[0102] Fig. 1 and Fig. 2 show two different examples of localization data 21 according to the invention, which are fed into the data processing network 20. In both cases, this may in particular be MINFLUX data.

[0103] The localization data 21 according to Fig. 1 comprise pairs of values ​​of (here two-dimensional) position vectors of illumination positions of the illumination light B in the sample 2 and photon numbers associated with the position vectors. These photon numbers were recorded by a detector s of a MINFLUX microscope, while the local minimum of the intensity distribution of the illumination light B was located at the corresponding illumination position.

[0104] For example, the local minimum of an excitation light distribution can be successively shifted to illumination positions arranged in a circle around a pre-estimated emitter position in the sample using a fast scanning device (e.g., comprising electro-optical or acousto-optical deflectors). During a dwell time, individual photons emitted by the emitter can be detected at each illumination position, e.g., using an avalanche photodiode (APD). The closer an illumination position is to the actual emitter position, the lower the measured photon count, since the emitter is exposed to less excitation light.

[0105] From such value pairs, the data processing network 20 according to the invention can determine position data 25 (here, a position vector with the x-coordinate Px and the y-coordinate Py) of the emitter. Of course, this principle can also be applied to a three-dimensional MINFLUX measurement. In this case, both the position vectors of the illumination positions and the vector of position data 25 each have three components.

[0106] Fig. 2 shows a further example of localization data 21 that can be processed with the data processing network 20 according to the invention in order to determine position data 25.

[0107] In this case, the localization data 21 comprise a time series of individual light emissions L, i.e., photon detection events that were recorded by the detector s at a specific time. This recording can, for example, take place during a continuous movement of the intensity distribution of the illumination light B, for example on a circle around a previously estimated position of an emitter in the sample 2. The positions of the local minimum of the intensity distribution are not explicitly specified here, as in the example of Fig. 1. These positions can also be part of the localization data 21 fed into the data processing network 20, e.g., as a second time series of the positions or as a locus curve as a function of time. However, it is also possible that the data processing network 20 does not require any explicit information about the positions of the illumination light distribution at all, because this is implicitly contained in the time series of the light emissions L.

[0108] Fig. 2 also outlines the application of a window function 26 to the time series. The window function 26 can, for example, be designed such that only a specific time range of the time series is used for input to the data processing network 20.

[0109] Fig. 3 shows an embodiment of a light microscope 100 according to the invention, in particular a MINFLUX microscope.

[0110] The light microscope 100 has a light source 3 for generating illumination light B. The light source 3 is in particular a laser and generates excitation light that excites emitters in the sample 2 to fluoresce or is scattered or reflected by emitters in the sample 2. The illumination light B passes through a first scanning device 4a (e.g., consisting of two electro-optical or acousto-optical deflectors) for deflecting the illumination light, as well as a light modulator 12, which spatially modulates the phase and / or the amplitude of the illumination light in order to generate an intensity distribution with a local minimum at a focus of the illumination light B in a sample 2. The illumination light B passes through a beam splitter 11, which deflects light emitted by the sample 2, in particular fluorescent light, via a confocal pinhole 10 to a detector s, and then a second scanning device 4b with a scanning mirror 41 and a scanning lens 42, e.g.a galvoscan device, which can be used, for example, for coarse positioning of the intensity distribution of the illumination light B. For clarity, only one scanning mirror 41 is shown in Fig. 3, although xy-beam scanners, in particular, have at least two mirrors. The illumination light beam then passes through a tube lens 8 to an objective 9, which focuses the light beam onto the sample 2.

[0111] The light emissions L emitted by individual emitters in sample 2 are collected by lens 9, descanned by second scanning device 4b, reflected by beam splitter 11, and detected by confocal detector 5. The signals from detector 5 are evaluated by a data processing device 6, whose data memory 63 stores a trained data processing network 20 according to the invention, which determines position data 25 of the emitter to be localized from the localization data 21 obtained from detector s and possibly preprocessed. From the position data 25, for example, a high-resolution image of sample 2 or a trajectory of an emitter moving in sample 2 can then be created.

[0112] The light microscope 100 further comprises a control unit 7 connected to the detector 5, the first scanning device 4a, the second scanning device 4b and the light modulator 12 as well as optionally further components. The control unit 7 can be used to implement, for example, iterative MINFLUX methods in which, in an iteration step, measurement parameters, e.g. the illumination positions at which the minimum of the intensity distribution of the illumination light B is placed, are derived from the position data 25 of previous steps.

[0113] In the MINFLUX method carried out with the light microscope 100, the procedure can be the same as for generating the localization data 21 shown in Fig. 1 and Fig. 2. This means that the minimum of the intensity distribution of the illumination light B can be arranged successively at illumination positions around an estimated position of an individual emitter, for example by means of the first scanning device 4a, wherein light emissions L of the emitter are detected for each illumination position with the detector 5 (Fig. 1), or the minimum of the intensity distribution can be moved continuously over the sample 2 with the first scanning device 4a and / or the second scanning device 4b, e.g. on a circular path around a previously estimated position of the emitter, and a time series of light emissions L can be recorded continuously with the detector 5 (Fig. 2).

[0114] Another possibility is to control the light modulator 12 to change the shape and / or orientation of the intensity distribution in the sample 2, wherein for the different shapes and / or orientations, light emissions L of the emitter are detected by the detector 5. According to the invention, such localization data 21 can also be processed with the data processing network 20 into position data 25 of the emitter.

[0115] The data processing device 6 is shown schematically in Fig. 4. It has a data input 61, a computing unit 62, a data memory 63, and a data output 64. Information that implements an artificial intelligence method can be stored in the data memory 63 by the computing unit 62 performing corresponding computing operations. A data processing network 20, e.g., a trained artificial neural network, with corresponding weights for the connections 24 between nodes 23 (see Fig. 1 and Fig. 2) can be stored in the data memory 63. Via the data input 61, the data processing device 6 can receive localization data 21, which is then converted into position data 25 by the computing unit 62 according to the stored trained data processing network 20. This can be output via the data output 64.Multiple sets of position data 25 can be processed by the computing unit 62 or another computing unit to create a high-resolution image or trajectory. Alternatively or additionally, the output data can be stored in the data memory 63 or a separate storage unit.

[0116] The components shown in Fig. 4 can be implemented at the hardware or software level. Furthermore, the data input 61 and the data output 64 can optionally be combined in a bidirectional data interface.

[0117] The data processing device 6 can be, for example, a computer (in particular a general-purpose computer, a graphical processing unit, an FPGA (field programmable gate array), or an ASICS (application-specific integrated circuit), an electronic control unit, or an embedded system. The data memory 63 stores instructions which, when executed by the computing unit 62, cause the data processing device 6 or the light microscope 100 to carry out the method according to the invention. The stored instructions therefore form a program executable by the computing unit 62 in order to carry out the

[0118] Procedures, in particular artificial intelligence procedures or artificial intelligence algorithms.

[0119] List of reference symbols

[0120] 2 Sample

[0121] 3 Light source

[0122] 4a First scanning device

[0123] 4b Second scanning device

[0124] 5 Detector

[0125] 6 Data processing device

[0126] 7 Control unit

[0127] 8 tube lens

[0128] 9 Lens

[0129] 10 pinhole

[0130] 11 beam splitters

[0131] 12 Light modulator

[0132] 20 Data processing network

[0133] 21 Localization data

[0134] 22a Input layer

[0135] 22b Hidden layer

[0136] 22c Output layer

[0137] 23 knots

[0138] 24 Connection

[0139] 25 Position data

[0140] 41 scanning mirrors

[0141] 42 scan lens

[0142] 61 Data input

[0143] 62 computing unit

[0144] 63 data storage

[0145] 64 Data output

[0146] 100 light microscope

[0147] 101 Illumination optics B Illumination light L Light emissions

Claims

Patent claims 1. A method for localizing or tracking emitters in a sample (2), wherein the sample (2) is illuminated with an intensity distribution of an illuminating light (B) having a local minimum, and wherein localization data (21) are created on the basis of light emissions (L) of a measuring emitter in the sample (2), characterized in that a position of the measuring emitter in the sample (2) is estimated by means of processing the localization data (21) by a trained data processing network (20), wherein the data processing network (20) is trained using training data.

2. Method according to claim 1, characterized in that the localization data (21) comprise pairs of values of light emission values of the measuring emitter and illumination positions of the local minimum of the intensity distribution of the illumination light (B) associated with the light emission values.

3. Method according to claim 1, characterized in that the localization data (21) comprise pairs of values of light emission values of the measuring emitter and shapes and / or arrangements of the intensity distribution associated with the light emission values.

4. Method according to one of claims 1 to 3, characterized in that the training data are localization data (21) of at least one training emitter of known position in a training sample illuminated by the illumination light (B).

5. Method according to one of claims 1 to 3, characterized in that the training data are simulation data.

6. Method according to claim 5, characterized in that background light is taken into account in the simulation data.

7. Method according to one of the preceding claims, characterized in that the data processing network (20) is an artificial neural network.

8. The method according to claim 7, characterized in that the neural network is a recurrent neural network.

9. Method according to claim 7 or 8, characterized in that the neural network is an LSTM network.

10. Method according to one of the preceding claims, characterized in that the localization data (21) are made available to the data processing network (20) as a time series.

11. The method according to claim 10, characterized in that the localization data (21) made available to the data processing network (20) for the position estimation are determined on the basis of a window function (26) applied to the time series.

12. Method according to claim 10 or 11, characterized in that the time series is divided into sections, wherein the sections are assigned to different measuring steps of a localization measurement.

13. Method according to one of the preceding claims, characterized in that the training data are obtained with different values of at least one parameter.

14. The method according to claim 13, characterized in that the at least one parameter comprises a size measure of an illumination pattern of illumination positions and / or a total intensity of the illumination light.

15. Method according to one of the preceding claims, characterized in that the training data are generated on the basis of simulation data or light emissions (L) of several emitters, wherein the positions of several measuring emitters are estimated simultaneously or in parallel by means of the trained data processing network.

16. The method according to claim 15, characterized in that the measuring emitters belong to the same emitter species.

17. The method according to claim 15, characterized in that the measuring emitters belong to different emitter species, wherein the measuring emitters differ in their excitation spectrum, their emission spectrum and / or their emission lifetime.

18. Method according to one of the preceding claims, characterized in that the position of several measuring emitters, in particular isolated in the sample (2), is estimated one after the other by means of the data processing network (20), a high-resolution image of structures in the sample (2) being created from the estimated positions.

19. Method according to one of the preceding claims, characterized in that the position of the same measuring emitter moving in the sample (2) is estimated several times in succession by means of the data processing network (20), a trajectory of the measuring emitter being created from the estimated positions.

20. Method according to one of claims 1 to 19, characterized in that the position estimation is carried out by means of the trained data processing network (20) between or during measuring steps of a localization experiment, wherein in the In each measuring step, the sample (2) is illuminated with the intensity distribution of the illuminating light (B) and light emissions (L) of the measuring emitter are recorded.

21. Method according to one of claims 1 to 19, characterized in that the position estimation is carried out by means of the trained data processing network (20) after completion of a localization experiment in order to reprocess an already carried out position estimation of the measuring transmitter.

22. Method according to one of the preceding claims, characterized in that the data processing network (20) is further trained on the basis of the localization data (21) of measurement emitters.

23. A method for training a data processing network (20), wherein the data processing network (20) is trained with training data for a position estimate of a measuring emitter in a sample (2) according to the method according to one of claims 1 to 22.

24. Data processing device (6) comprising a data memory (63) in which a data processing network (20) trained with the method according to claim 23 is stored, wherein the data processing device (6) has a computing unit (62) which is designed to carry out the method for locating or tracking emitters in a sample (2) according to one of claims 1 to 22.

25. A light microscope (100) comprising an illumination optics (101) designed to illuminate a sample (2) with an illumination light (B), wherein the illumination light (B) forms an intensity distribution with a local minimum in the sample (2), and a detector (5) designed to detect light emissions (L) of a measuring emitter, characterized in that the light microscope (100) has a data processing device (6) according to claim 24.

26. A computer program comprising program code which causes the data processing device (6) according to claim 24 and / or the light microscope (100) according to claim 25 to carry out the method according to one of claims 1 to 22.

Images