Method, computer program and apparatus for adjusting an estimator used in a microscope

Abstract

The invention relates to a method, a computer program and a device for adjusting an estimator used in a microscope for estimating the position of an emitter in a sample based on a method of illuminating the sample with light at one or more sets of probe positions and acquiring fluorescent photons for the set of probe positions. The invention also relates to a microscope utilizing such an adjustment method or device. According to the invention, the sample is illuminated (S1) with light at one or more sets of probe positions and fluorescent photons are acquired (S2) for the set of probe positions. Then, the photon counts of the acquired (S2) photons are added (S3) to a photon count vector or photon count sum determined for the set of probe positions. Subsequently, a value representative of background noise is determined (S4) from the photon count vector or photon count sum. The value representative of background noise is then used to adjust (S5) the estimator in real time.

Description

Technical Field

[0001] This invention relates to a method, computer program, and apparatus for adjusting an estimator used in a microscope to estimate the position of an emitter in a sample. The invention also relates to a microscope utilizing this method or apparatus. Background Technology

[0002] Fluorescence microscopy has undergone a second resolution improvement due to the specific advantages of the coordinate-targeted super-resolution family [1] represented by STED (STimulated Emission Depletion) [2] and RESOLFT (Reversible Saturable Optical Fluorescence Transitions) [3], as well as the synergistic combination of its coordinate random counterparts, including PALM (Photo Activated Localization Microscopy) / STORM (Stochastic Optical Reconstruction Microscopy) [4] and PAINT (Point Accumulation for Imaging in Nanoscale Topography) [5]. The resulting synergistic design known as MINFLUX (Minimal photon Flux) [6] has reduced the general resolution gap from ~20-30 nanometers in STED, PALM / STORM and other fluorescence nanomirrors to the size scale of 1-5 nanometers of the molecules themselves.

[0003] At its core, MINFLUX localization is based on the crucial principle of injecting reference coordinates into a sample using structured beams, such as a ring with a central minimum intensity (i.e., zero). The location of the zero in the sample defines the target sample coordinates. The MINFLUX design also applies to the entire set of reference coordinates, i.e., linear and point zeros, as well as parallel detection in a wide field. Coordinate alignment enables well-controlled and therefore photon-efficient localization of fluorescent molecules because the coordinates of the fluorophore to be determined are no longer found by establishing the center of a weak, diffraction-limited fluorescence spot that appears on the camera. Instead, the fluorophore is localized by actively aligning the zero of the excitation ring to the fluorophore. Specifically, in deliberate iterations, the zero of the excitation intensity is brought as close as possible to the molecule until the detected fluorescence rate roughly matches the fluorescence rate of the background noise. In this nearest neighbor, maximum localization accuracy is achieved with only a minimum number of fluorescent photons because fewer detected photons are needed to establish the remaining distance between the target coordinates of the ring zero and the molecule's position. Therefore, the burden of "injecting" or aligning reference coordinates in the sample, which would otherwise require a large number of fluorescent photons for positioning, is shifted to an infinite number of photons in the annular excitation beam.

[0004] Because MINFLUX positioning is no longer limited by waiting for a large number of fluorescent photons, this nanoscale precision positioning is much faster than the camera-based positioning used in PALM / STORM. The idea of ​​optically injecting coordinates using a circular null point is therefore inherent in the original STED design. For STED microscopy, it is evident that, in the absence of background, a single detected photon is sufficient to demonstrate the presence of a fluorophore at the target coordinates of the circular null point. There, the emitting fluorophore is also perfectly positioned by the photons injected by the STED beam.

[0005] The minimum localization accuracy achievable using an unbiased estimator (i.e., the Cramér-Rao lower bound (CRLB) for locating fluorophores within a diameter L region delineated by the target coordinate pattern) is determined by... Let N be the sum of photons detected at the zero point located at the coordinates of the target coordinate pattern. The dependence on the diameter L of the target coordinate pattern is linear, while the dependence on the number of detected photons N follows only the well-known square root reciprocal relationship. Therefore, bringing the zero point of the excitation ring closer to the fluorophore's location, i.e., controllingly reducing the diameter L of the target coordinate pattern, is more effective in improving localization accuracy than waiting for a larger number N of detected photons. This fundamental fact is central to the iterative MINFLUX method, which leverages the central idea of ​​spatially aligning the zero point of the ring with the detected fluorophore, a process limited only by background noise. For the general case of a target coordinate pattern consisting of a set of outer triangles plus a central probing point, the diameter L is gradually reduced. k Continuous amplification of molecules is a feasible strategy for improving position estimation, where L k Selected as σ (k-1) Three times the uncertainty of the previous iteration (i.e., the uncertainty of the previous iteration). After a small number of iterations of k, this results in N being the combination of photons detected when the number of photons is the same or similar in each iteration. t =k·N, CRLB becomes:

[0006]

[0007] Four steps (i.e., k=4) have been completed. That is, it depends on the number of photons N detected. t It is the reciprocal of a quadratic function, not the reciprocal of a square root function. More iterations can easily produce higher orders, reflecting an exponential relationship. Equally important, the number of photons collected in each iteration does not have to be the same. Instead, it can be individually adjusted for its most efficient overhead during iteration.

[0008] Sample optimization for MINFLUX measurements typically revolves around finding the correct imaging conditions for the sample, such as laser power or buffer composition, which is usually accomplished through successive measurements with adjusted parameters. To facilitate routines and generally improve sample throughput, it is highly desirable to present the final localization data immediately during the measurement, as this allows for assessment of sample quality shortly after the measurement begins.

[0009] One possible mode of super-resolution imaging utilizes iterative MINFLUX imaging with a power ramp. This method derives the location of the emitter in the sample from the photon counts measured at the detector location group by applying a random estimator to the data. This method is complicated by the fact that specific experimental conditions, such as the spatial intensity distribution of the ring and the signal-to-noise ratio, must be considered in order to obtain unbiased results, i.e., results without systematic errors. So far, these conditions have been taken into account by the time-consuming post-processing of the localization data [6], [6a], [7], [8]. Summary of the Invention

[0010] The technical problem to be solved by the present invention is to provide a solution for real-time adjustment of the estimator used in a microscope.

[0011] This technical problem is solved by the method according to the invention, the computer program for implementing the method according to the invention, the apparatus according to the invention, and the microscope according to the invention. The invention includes advantageous further developments and modifications of the principles described below.

[0012] According to a first aspect, the present invention relates to a method for adjusting an estimator used in a microscope for estimating the position of an emitter in a sample based on illuminating a sample with light at one or more sets of detection locations and acquiring fluorescence photons for those sets of detection locations, the method for adjusting the estimator used in the microscope comprising:

[0013] - Illuminate the sample at one or more detection locations;

[0014] - Acquire fluorescent photons at this set of detection locations;

[0015] - Add the photon count of the acquired photons to the photon count vector or the total photon count for that set of detection locations;

[0016] - Determine the value representing background noise from the photon count vector or the sum of photon counts; and

[0017] - Adjust the estimator in real time using a value representing background noise.

[0018] The photon count vector or photon count sum is determined from the acquired photons for that set of detection locations by adding the photon count of the acquired photons to the photon count vector or photon count sum for that set of detection locations.

[0019] Accordingly, the computer program includes instructions that, when executed by at least one processor, cause the processor to perform the following steps to adjust an estimator used in the microscope for estimating the location of emitters in the sample based on a method of illuminating the sample with light at one or more sets of detection locations and acquiring fluorescence photons for those sets of detection locations:

[0020] - Illuminate the sample at one or more detection locations;

[0021] - Acquire fluorescent photons at this set of detection locations;

[0022] - Add the photon count of the acquired photons to the photon count vector or the total photon count for that set of detection locations;

[0023] - Determine the value representing background noise from the photon count vector or the sum of photon counts; and

[0024] - Adjust the estimator in real time using a value representing background noise.

[0025] The term "computer" must be understood in a broad sense. In particular, it also includes embedded devices, workstations, parallel computing devices such as field-programmable gate arrays (FPGAs), and processor-based data processing devices.

[0026] For example, computer program code can be used for electronic retrieval or stored on computer-readable storage media.

[0027] According to another aspect, the present invention relates to an apparatus for adjusting an estimator used in a microscope for estimating the position of an emitter in a sample based on a method of illuminating a sample with light at one or more sets of detection locations and acquiring fluorescence photons for those sets of detection locations, the apparatus comprising:

[0028] - An illumination device configured to illuminate a sample with light at one or more sets of detection locations;

[0029] - Acquisition device, which is configured to acquire fluorescent photons for the set of detection locations;

[0030] - A processing device configured to add the photon counts of the acquired photons to a photon count vector or a photon count sum for the set of detection locations, and to determine a value representing background noise from the photon count vector or the photon count sum; and

[0031] - A control unit configured to adjust the estimator in real time using a value representing background noise.

[0032] The processing device is configured to determine a photon count vector or photon count sum from the acquired photons for the set of detection locations by adding the photon count of the acquired photons to the photon count vector or photon count sum for the set of detection locations.

[0033] According to the present invention, the estimator is adjusted in real time to compensate for background noise. In this way, near-unbiased real-time localization can be achieved. To adjust the estimator, the background is estimated based on the measurement data and used to remove bias from the estimator. Real-time adjustment of the estimator allows the final localization data to be presented during the measurement, i.e., the image is provided to the user during the measurement. This allows for the assessment of sample quality shortly after the start of the measurement, which is beneficial for sample optimization routines and generally increases sample throughput. Furthermore, in the case of the iterative MINFLUX method, available photons can be better utilized because the continuous amplification of the probe points on the molecule is unbiased. Another advantage is that it is also unbiased during molecule tracking.

[0034] Of course, the use of this invention is not limited to iterative MINFLUX. Real-time background compensation also works when simply using patterns. Therefore, the solution according to the invention also works during tracking, where the pattern is tracked. Also during tracking, the radius of the pattern can be adjusted. However, the speed is adjusted here. If multiple locations are close to each other, the radius can be reduced. If the locations shift, the radius increases.

[0035] In an advantageous embodiment, to determine a value representing background noise, only the group of detector locations where no fluorescence from the emitter was detected is evaluated. The advantage of this approach is that the background noise estimate is not distorted by photons that actually originate from fluorescence from the emitter. In this way, a more accurate background estimation is achieved.

[0036] In an advantageous embodiment, to determine whether fluorescence from the emitter is detected, the sum of photon counts for a set of detection locations is compared to a threshold derived from the currently estimated background. For example, the threshold could be the sum of the currently estimated background and a preset threshold, or the product of the currently estimated background and a preset constant greater than 1. In this way, the sum of photon counts for a set of detection locations is only classified as a signal if the sum significantly deviates from the currently estimated background. The currently estimated background can also be calculated as a moving average of background estimates from previous detection locations.

[0037] In an advantageous embodiment, the sum of photon counts is added to the histogram. Using a histogram, for example, running a histogram constructed from a finite number of entries in a first-in-first-out manner, has the advantage of allowing for a simple evaluation of the measurement history.

[0038] In an advantageous embodiment, the histogram is specific for each of the multiple scan iterations. In the iterative MINFLUX method, the size of the probe location group decreases in successive iterations. Simultaneously, the intensity increases because the fluorophore is now closer to the minimum of the excitation ring. Typically, the correlation between background noise and intensity is non-linear. Therefore, it is advantageous to determine a value representing the background noise separately for each intensity used. In the case of a linear correlation, a single estimate is sufficient.

[0039] In an advantageous embodiment, the value representing background noise is derived from the peaks of the histogram. Deriving the value representing background noise from the peaks of the histogram is readily achievable. For example, the absolute maximum value of the histogram can be used, which is a computationally inexpensive option. Alternatively, a peak detection algorithm can be applied. For example, after smoothing the histogram using a Gaussian kernel, the value representing background noise can be derived as the highest peak or as the first peak in a copy of the histogram. Alternatively, the value can be derived from the position of the peak containing the most total entries.

[0040] In an advantageous embodiment, the estimator can be adjusted such that the expected value of the background noise is subtracted from the photon count before estimating the emitter's position. A typical estimator provides an expected value of the emitter's position. This expected value can be affected by background noise. By subtracting the expected value of the background noise from the photon count before estimating the emitter's position, a more reliable expected value of the emitter's position is obtained.

[0041] In an advantageous embodiment, the estimator uses a calibration polynomial that is specific to each scan iteration. For example, a second-order calibration polynomial can be used. The advantage of using a calibration polynomial is that it is computationally inexpensive on the microscope hardware. The optimization parameters of the polynomial can be obtained, for example, through a Monte Carlo simulation that takes into account known experimental parameters. The calibration polynomial can also be provided as a lookup table.

[0042] In an advantageous embodiment, the group of detector locations is illuminated multiple times. This ensures effective detection of the time-varying background signal at all detector locations, mitigating directional distortion that can occur when short bursts of background signal are concentrated in subsets of the detector locations.

[0043] In an advantageous embodiment, a set of detection positions comprises three or more detection positions arranged rotationally symmetrically on a circle. The aim of the measurements is to infer the location of the fluorophore from the set of measurement data for the set of detection positions in the simplest possible manner. At least for real-time background estimation, this set of detection positions does not use the central detection position, but only the detection positions arranged rotationally symmetrically around the center. In other words, each detection position can be mapped to a different detection position by rotating it around the center. Of course, this does not preclude the possibility of measuring additional detection positions, in which case the obtained information is used additionally. For example, measurements can be taken at the center to check the reasonableness of the measurements.

[0044] A pattern comprising three or more probe locations (arranged rotationally symmetrically on a circle) has the following property: any reasonable estimator provides an expected value that depends only on the position of the transmitter and is independent of position-independent background noise. This is because each reasonable estimator essentially determines the centroid of the measurements. If a fixed value is added to all measurements, this centroid does not change. However, the centroid is determined by the vector sum of the individual measurements. In the noise-free case, this vector sum must be normalized relative to the sum of the measurements. If background noise is now added, the expected value of the vector sum in the enumerator does not change, but the expected value of the sum is increased by the expected value of the background noise recorded for the probe locations during the total measurement time. To make the expected value of the position estimate independent of background noise, the expected value of the background noise is subtracted from the denominator.

[0045] In the prior art, a pattern with a detection position also at the center is preferably used for estimation. This estimation causes the expected value to change when background noise is added. However, it is not possible to simply subtract the background noise using prior art estimators. In the case of an estimator with a detection position at the center, at least the value measured for that center position must be corrected relative to the background noise to eliminate it. Unfortunately, at the center, the number of photons expected to produce a signal is minimal, meaning the relationship between the signal and the background is extremely unfavorable. If the emitter is exactly at the center, the signal is practically zero. If the average noise is now subtracted from zero, a negative value is obtained. This causes serious problems. Setting the signal at the center to zero, or negative, is also impossible. Therefore, in the prior art, the optimal coefficients of the correction function are determined for each ratio of signal to background, but the photon count measured in this way is not corrected. In other words, a correction function needs to be determined that takes the ratio of signal to background as a parameter.

[0046] Using the solution according to the invention, the signal-to-background ratio can be determined in real time. To this end, the average signal reduction value estimated by the background in real time is determined, i.e., the sum of the corresponding corrections for each measurement. This ratio is a widely accepted signal / background ratio. Subsequently, an appropriate correction function can be selected, or the determined signal / background ratio can be used as a parameter of the overall correction function.

[0047] In an advantageous embodiment, opposing detector positions within a set of detector positions are illuminated sequentially in pairs. This method further supports preventing directional distortion that can occur when short bursts of background light concentrate into a subset of detector positions. Each pair of detector positions defines an angle together with the center of the set of detector positions, where the detector positions are on the legs and the center is on the apex. For an even number of detector positions, the opposing detector positions are those with an angle of 180°. For an odd number of detector positions, the opposing detector positions are those with an angle closest to 180°.

[0048] In an advantageous embodiment, the estimator is a minimum mean square estimator. The minimum mean square estimator is very robust to background noise and is well-suited for use in MINFLUX localization. Of course, other types of estimators can also be used, such as maximum likelihood estimators or other estimators known to those skilled in the art.

[0049] According to one aspect of the invention, this aspect can be used in conjunction with the aforementioned background estimation, but can also be used independently of the aforementioned background estimation, realizing an improved positioning mode, referred to as pinhole track scanning. Specifically, the method for positioning an emitter in a sample includes:

[0050] - Irradiate the sample with a fixed circular excitation beam;

[0051] - Project the pinholes sequentially onto the detection positions of the target coordinate pattern;

[0052] - Acquiring fluorescent photons at the detection location; and

[0053] - Estimate the location of the emitter in the sample from the acquired fluorescent photons.

[0054] Accordingly, the computer program includes instructions that, when executed by at least one processor, cause the at least one processor to perform the following steps to locate the emitter in the sample:

[0055] - Irradiate the sample with a fixed circular excitation beam;

[0056] - Project the pinholes sequentially onto the detection positions of the target coordinate pattern;

[0057] - Acquiring fluorescent photons at the detection location; and

[0058] - Estimate the location of the emitter in the sample from the acquired fluorescent photons.

[0059] As above, the term "computer" must be understood in a broad sense. In particular, it also includes embedded devices, workstations, parallel computing devices such as field-programmable gate arrays (FPGAs), and processor-based data processing devices.

[0060] For example, computer program code can be used for electronic retrieval or stored on computer-readable storage media.

[0061] Similarly, the device for locating the emitter in the sample includes:

[0062] - An irradiation device configured to irradiate a sample with a fixed annular excitation beam;

[0063] - A projection device configured to sequentially project pinholes onto the detection positions of a target coordinate pattern;

[0064] - Acquisition device, configured to acquire fluorescent photons at a detection location; and

[0065] - A processing device configured to estimate the position of the emitter in the sample from the acquired fluorescent photons.

[0066] The described solution has the advantage of improving the efficiency of the search process. Unlike the previous MINFLUX implementation, this pre-positioning does not rely on a regularly focused beam. This significantly simplifies the setup, as it renders the dedicated hardware used for microsecond-level switching between different focal intensity distributions (e.g., between Gaussian and toroidal shapes) redundant. Spatial light modulators typically used for switching between Gaussian and toroidal shapes are usually too slow to switch between shapes during measurement routines. While the capture range depends on several parameters, the toroidal shape helps extend the capture range beyond that of the Gaussian shape. In other words, every emitter located within the potential well can be found.

[0067] In an advantageous embodiment, a synergistic galvanometer and electro-optic scanner are used to keep the excitation beam stationary. This has the advantage of obtaining a fairly large field of view.

[0068] In an advantageous embodiment, the distance between the scan positions is between 10% and 50% of the excitation wavelength used. This prevents black dots next to the search grid positions from being caused by circular zeros. Attached Figure Description

[0069] Further features of the invention will become apparent from the following description taken in conjunction with the accompanying drawings.

[0070] Figure 1The method for adjusting the estimator used in a microscope to estimate the position of emitters in a sample is illustrated schematically.

[0071] Figure 2 A first embodiment of a device for adjusting an estimator used in a microscope to estimate the position of an emitter in a sample is illustrated schematically.

[0072] Figure 3 A second embodiment of a device for adjusting an estimator used in a microscope to estimate the position of an emitter in a sample is illustrated schematically.

[0073] Figure 4 A sample containing multiple molecules is shown;

[0074] Figure 5 An exemplary intensity curve is shown on a cross-section of the light distribution;

[0075] Figure 6 An exemplary optical setup for a MINFLUX microscope is shown;

[0076] Figure 7 The principle of real-time background compensation is explained;

[0077] Figure 8 This explains the principle of MINFLUX positioning; and

[0078] Figure 9 The target coordinate pattern applied during the MINFLUX positioning process is shown, along with the simulated average position estimate and confidence interval. Detailed Implementation

[0079] This specification illustrates the principles of this disclosure. It will therefore be understood that those skilled in the art will be able to design various arrangements that, while not explicitly described or shown herein, embody the principles of this disclosure.

[0080] All examples and conditional language cited herein are intended for educational purposes to help the reader understand the principles of this disclosure and the designs contributed by the inventors to the field, and should be construed as not being limited to these specifically cited examples and conditions.

[0081] Furthermore, all statements relating to the principles, aspects, and embodiments of this disclosure and their specific examples herein are intended to cover equivalents in terms of their structure and function. Additionally, such equivalents are intended to include both currently known equivalents and those developed in the future, i.e., any element developed that performs the same function regardless of its structure.

[0082] Therefore, for example, those skilled in the art will understand that the line diagrams presented herein represent conceptual views of illustrative circuits that embody the principles of this disclosure.

[0083] The functionality of the various elements shown in the accompanying drawings can be provided using dedicated hardware and hardware capable of executing the software, associated with appropriate software. When provided by a processor, the functionality can be provided by a single dedicated processor, a single shared processor, or multiple separate processors some of which can be shared. Furthermore, the explicit use of the terms "processor" or "controller" should not be construed as specifically referring to hardware capable of executing software, and may implicitly include, but is not limited to, digital signal processor (DSP) hardware, read-only memory (ROM), random access memory (RAM), and non-volatile memory for storing software.

[0084] Other conventional and / or custom hardware may also be included. Similarly, any switches shown in the accompanying drawings are conceptual only. Their functionality can be performed through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the specific technology of which can be selected by the implementer from a more specific understanding of the context.

[0085] In its claims, any element expressed as a means for performing a particular function is intended to cover any manner in which that function is performed, such as including combinations of circuit elements that perform the function or software in any form, and thus including firmware, microcode, etc., combined with appropriate circuitry to perform the software to perform the function. The disclosure as defined by such claims lies in the fact that the functionality provided by the various cited means is combined and arranged together in the manner claimed in the claims. Therefore, any means capable of providing these functionalities is considered equivalent to the means shown herein.

[0086] Figure 1A method according to the invention for adjusting an estimator used in a microscope to estimate the position of emitters in a sample is illustrated. For example, the estimator may be a least mean square estimator. In this method for adjusting the estimator used in a microscope, the sample is illuminated S1 at one or more sets of detection positions, and S2 fluorescent photons are acquired for each set of detection positions. Advantageously, the set of detection positions is illuminated S1 multiple times. Opposite detection positions in a set of detection positions may be illuminated S1 sequentially in pairs. Preferably, a set of detection positions comprises three or more detection positions arranged rotationally symmetrically on a circle. The photon count of the acquired S2 photons is then added S3 to the photon count vector or photon count sum for that set of detection positions. Subsequently, a value representing background noise S4 is determined from the photon count vector or photon count sum. For example, the photon count sum can be added to a histogram, preferably specific to each of multiple scan iterations. The value representing background noise can then be derived from the peak of the histogram. The value representing background noise is used to adjust the S5 estimator in real time. For example, the estimator S5 can be adjusted such that the expected value of the background noise is subtracted from the photon count before estimating the emitter's position. Preferably, the estimator uses a calibration polynomial that is specific to each scan iteration. Advantageously, to determine the value representing the background noise, only the group of detector positions where no fluorescence from the emitter was detected is evaluated. To determine whether fluorescence from the emitter was detected, the sum of the photon counts for a group of detector positions can be compared with a threshold derived from the currently estimated background.

[0087] Figure 2A block diagram schematically illustrates a first embodiment of an apparatus 40 according to the invention for adjusting an estimator 35 used in a microscope 10 to estimate the position of an emitter in a sample. For example, the estimator 35 may be a least mean square estimator. The apparatus 40 has an interface 41 for exchanging data with components of the microscope 10. An illumination device 42 is configured to illuminate the sample with light at one or more sets of detection positions, for example by correspondingly indicating the light source and position control elements of the microscope. Advantageously, the set of detection positions is illuminated multiple times S1. Opposite detection positions in a set of detection positions may be illuminated sequentially in pairs S1. Preferably, a set of detection positions comprises three or more detection positions arranged rotationally symmetrically on a circle. An acquisition device 43 is configured to acquire fluorescence photons for the set of detection positions, for example by evaluating signals from one or more detectors of the microscope. A processing device 44 is configured to add the photon counts of the acquired photons to a photon count vector or photon count sum for that set of detection positions, and to determine a value representing background noise from the photon count vector or photon count sum. For example, the sum of photon counts can be added to a histogram, which is preferably specific to each of the multiple scan iterations. Control device 45 is configured to adjust estimator 35 in real time using a value representing background noise. For example, the estimator can be adjusted such that the expected value of the background noise is subtracted from the photon counts before estimating the emitter's position. Preferably, the estimator uses a calibration polynomial specific to each scan iteration. Advantageously, to determine the value representing background noise, only the group of detection locations where no fluorescence from the emitter was detected is evaluated. To determine whether fluorescence from the emitter was detected, the sum of photon counts for a set of detection locations can be compared to a threshold derived from the currently estimated background.

[0088] The irradiation device 42, acquisition device 43, processing device 44, and control device 45 can be controlled by a controller 46. A local storage unit 47 is provided, for example, for storing data during processing. A user interface 48 can be provided to allow the user to modify the settings of the various elements 42-46 of the device 40. The different elements 42-46 of the device 40 can be embodied as dedicated hardware units. Of course, they can also be combined entirely or partially into a single unit or implemented as software running on a processor (e.g., CPU or GPU).

[0089] Figure 3A block diagram illustrating a second embodiment of an apparatus 50 according to the present invention for adjusting an estimator used in a microscope to estimate the position of an emitter in a sample. Apparatus 50 includes a processing device 51 and a storage device 52. For example, apparatus 50 may be a computer, an electronic control unit, or an embedded system. Storage device 52 stores instructions that, when executed by processing device 51, cause apparatus 50 to perform steps according to any of the described methods. Thus, the instructions stored in storage device 52 tangibly embody a program of instructions executable by processing device 51 to perform the program steps as described herein according to these principles. Apparatus 50 has an input 53 for receiving data. Data generated by processing device 51 is provided via output 54. Additionally, such data may be stored in storage device 52. Input 53 and output 54 may be combined into a single bidirectional interface.

[0090] The processing device 51 used herein may include one or more processing units, such as a microprocessor, a digital signal processor, or a combination thereof.

[0091] Local storage unit 47 and storage device 52 may include volatile and / or non-volatile storage areas and storage devices, such as hard disk drives, optical drives and / or solid-state drives.

[0092] In the following text, reference will be made to Figures 4 to 9 A more detailed explanation of aspects of the invention follows.

[0093] Figure 4 Sample 1 with several emitters 2 is shown, such as molecules labeled with a fluorophore or molecules that act as fluorophores themselves. In this example, five emitters 2 are shown, their positions M1 to M5 to be determined. The fluorophore can be excited with light of a suitable wavelength to emit photons. In a MINFLUX microscope, the fluorophore is excited such that the fluorophore to be located is always placed near or at the minimum of the light distribution used for excitation; therefore, the light distribution must have an intensity increase range adjacent to the minimum. In this way, better utilization of fluorescent photons is achieved in obtaining information about the location of the corresponding emitting fluorophore. Ideally, the minimum of the excitation light distribution is zero.

[0094] Figure 5 An exemplary intensity curve is shown on a cross-section of a light distribution 3 passing through a suitable structured beam (in this case, a ring). Intensity I is plotted relative to a location within the light distribution 3, indicated here along the X-axis. The intensity curve of the light distribution 3 shows a central minimum intensity 4, surrounded by an area of ​​increasing intensity 5.

[0095] Figure 6An exemplary optical setup for the MINFLUX microscope 10 is shown. This optical setup is based on a conventional fluorescence microscope. Typical microscopy workflows also require preliminary steps before recording nanometer-resolution images, particularly eyepiece examination of the sample based on wide-field mode or confocal scanning to select the region of interest. Implementing MINFLUX on top of a standard inverted microscope platform 11 helps combine practicality with final resolution, as conventional fluorescence microscopy provides the features needed daily, such as compatibility with standard stages, probe holders, bright-field and radiofluorescence illumination with filters and eyepieces for rapid examination of large regions of interest in samples. The MINFLUX microscope 10 is controlled by a controller 12.

[0096] The excitation beam 13 emitted by laser 14 is focused into the focal plane of an objective lens, which is part of microscope platform 11. To enable flexible pre-observation of large regions of interest and precise measurements using the MINFLUX scheme in the desired area, beam scanning relative to a stationary sample is achieved through the combined operation of a galvanometer scanning unit 16 and electro-optic beam deflectors 17 and 18. To enable rapid lateral scanning of MINFLUX measurements in the xy-plane, the excitation beam 13 passes through a λ / 2 waveplate 15 and is deflected by two electro-optic beam deflectors 17 and 18, which are connected in series and mounted with a 90° rotation relative to their respective axes. A λ / 2 waveplate 20 between the electro-optic beam deflectors 17 and 18 rotates the laser beam polarization by 90° to accommodate this rotation. The accessible field of view of the deflector array in the focal plane is small and is extended by an additional galvanometer scanning unit 16 mounted at the camera port of the microscope body, which serves as a coarser basis for additional, finer, and faster electro-optic X and Y displacements. To switch between the Gaussian and annular shapes of the excitation beam 13, a phase-modulated spatial light modulator 21 is used, which introduces a helical phase shift into the beam 13. Subsequently, a λ / 4 waveplate 22 forms the circular polarization of the excitation light. The excitation beam 13 is then superimposed on the detection beam path with a beam splitter 23 and enters the microscope 11 via a galvanometer scanning unit 16. The galvanometer scanning unit 16 and electro-optic beam deflectors 17 and 18 work together to position the excitation beam 13 within the microscope 11, either as a normally focused beam for confocal scanning or as an annular beam with a central zero-point intensity for the MINFLUX protocol.

[0097] To activate a single fluorophore, an activation laser 24 is used. The intensity of this laser 24 is reduced to the nanowatt range by a neutral density filter 25. After passing through a λ / 4 waveplate 26, the activation laser beam 27 is superimposed on the detection laser beam path and the excitation laser beam path by a beam splitter 28. The activation laser beam 27 is provided to the microscope 11 without passing through the electro-optic beam deflectors 17 and 18.

[0098] The fluorescence 29 emitted by the sample is collected by the objective lens, scanned by the galvanometer scanning unit 16, transmitted by the aforementioned beam splitters 23 and 28, and passed to the variable pinhole 30 for confocal detection using two avalanche photodiodes 31 and 32, which detect photons in different spectral ranges defined by the dichroic mirror 33.

[0099] To estimate the position of the emitter in the sample, controller 12 includes an estimator 35. The estimator 35 can be adjusted in real time by means 40 according to the invention for adjusting the estimator 35. In this example, means 40 is also included in controller 12. Of course, it can also be provided as a separate device.

[0100] As previously mentioned, one possible mode of super-resolution imaging utilizes iterative MINFLUX imaging with a power ramp. This method derives the position of the emitter in the sample from the photon counts measured at the detector location set by applying a random estimator to the data. To obtain unbiased results, i.e., results without systematic errors, a measurement scheme that provides near-unbiased positioning in real time is used.

[0101] For a circle with radius L, there are m beam positions. The target coordinate pattern, and the corresponding photon counts p collected during equal dwell times. j Used for the (relative) position of the launcher within the target coordinate pattern. The least mean square estimator (LMSE) can be written as the product of a normalized vector sum and a scaling constant c, which takes into account L and the full-width-half-maximum (FWHM) w of the point spread function.

[0102]

[0103]

[0104] According to one aspect of the invention, the proportionality constant c is replaced by an iteration-dependent second-order calibration polynomial P. k It is effortless to compute on the microscope hardware to obtain the unbiased emitter position from the calibrated least mean square estimator:

[0105]

[0106] For each iteration k, for P k The optimization coefficients can be obtained through Monte Carlo simulations, which take into account known experimental parameters (e.g., the precise shape of the torus, the geometry of the target coordinate pattern, and the photon confinement N). kPreferably, the microscope probes the target coordinate pattern multiple times in each positioning step until the actual number of photons collected exceeds a preset N. k The dwell time and excitation power can be adjusted so that the distribution of the actual number of collected photons is sufficiently close to this limit to ensure P k The use of m=3 for the eccentric beam position used in previous MINFLUX implementations resulted in a significant deviation of the estimator from radial symmetry. Therefore, for all iterations, m is preferably increased to 6. Exceptions may be made for iterations where the objective is to achieve the highest positioning rate and where increased overhead should be avoided. The increased angular uniformity of positioning accuracy becomes noteworthy, especially for emitters located outside the target coordinate pattern.

[0107] However, when providing an unbiased emitter position, equation (4) does not take into account the background that is unavoidable in typical applications. A uniform background will introduce a bias toward the center of the target coordinate pattern, as its average contribution to the numerator of u will be equal to 0, and the denominator will now overestimate the number of photons from the local emitter. Based on this concept, MINFLUX control logic with a real-time background estimation algorithm is used to mitigate a sufficiently uniform quasi-static background.

[0108] The algorithm is implemented as follows: Figure 7 As shown in the diagram. The current fluorescence baseline for each iteration is derived from a run histogram of fluorescence intensities detected during localization. Variables with subscript k are iteration-specific, where k is the current iteration, starting from 0. NIT indicates the number of iterations in the MINFLUX sequence. k This refers to the target coordinate pattern, i.e., the relative coordinates of the probe position. EPE is the emitter position estimate relative to POS, i.e., the measured position in the sample, which is the origin of TCP. POS is also the initial emitter position estimate. SIP refers to the last signal from the pattern, which is the photon count vector. ΣSIP is the total photon count of the last signal. SIA refers to the accumulated SIP in the current iteration. Correspondingly, ΣSIA is the total photon count in the estimation background during the current iteration. PHL k It's photon-limited, HST k This is the (running) histogram of the obtained ΣSIP. DKT refers to the dark time counter, and DKL refers to the dark time limit. EBG is the estimated background count for a single pattern in the current iteration. TBG k This refers to the background threshold. STA is the measured state. If the state is "bright," the algorithm is attempting to locate the emitter. If the state is "dark," the algorithm is estimating the background. CLMSE k This refers to the calibrated minimum mean square estimator.

[0109] This implementation uses a vector of photon counts, "signals from the pattern" SIP = (p1, ..., p...). m This photon count is acquired during signal detection of the target coordinate pattern and serves as the smallest unit of data upon which decisions are based. For a single SIP detection, each It needs to be processed at least once. However, typically, during a single SIP acquisition process, the target coordinate pattern is scanned multiple times to ensure that in all cases... Effectively detects background signals with temporal fluctuations to mitigate the concentration of short bursts in the background. Directional distortion that occurs when the subset is selected. For the same reason, it is beneficial to choose the scanning order to make the relative They were detected in pairs in sequence.

[0110] To estimate the background, a baseline estimate of the histogram type is used. For each MINFLUX iteration k, this is achieved by inputting the ΣSIP of each measurement into an iteration-specific running histogram HST. k In this process, the number of photons collected during each individual SIP is kept recorded. The running histogram is a histogram constructed with a finite number of entries in a first-in, first-out manner. While scanning the region of interest to search for fluorescent emitters, the microscope acquires the 0th iteration SIP at different locations in the sample until a fluorescence signal is detected. Imaging conditions, such as activation levels, are chosen so that most non-background emitters are in their dark state during this process. Therefore, most of the SIPs obtained in the 0th iteration contain only background signals. For higher iterations k>0, the corresponding histogram HST... k The histograms are constructed solely from SIP measurements taken after the measurement state transitions to "darkness," meaning no significant signal above the background is detected. Therefore, all histograms will iteratively produce a maximum at the value EBG of ΣSIP, which is an estimate of the current average background level.

[0111] During each MINFLUX iteration, one or more SIPs are acquired until enough photons are acquired for localization. During this process, the currently localized emitter, i.e., the current emitter, may emit fluorescence intermittently in the form of (micro)bursts, a well-known characteristic of fluorescent proteins, and may stop emitting at any time. To allow only those SIPs with a high probability of carrying a signal from the current emitter to contribute to localization, ΣSIP is compared with the current value EBG and a threshold TBG. kThe sum of these values ​​determines whether a single SIP contains a significant signal beyond the background. If the answer is yes, the SIP is added to the vector SIA, which holds the vector signal accumulated for this iteration and will later become the numerator of equation (3). For the denominator of equation (3), ΣSIP is background corrected and then added to the accumulated total signal ΣSIA → ΣSIA + ΣSIP - EBG. If the answer is no, the SIP is discarded because the current emitter has obviously entered a non-fluorescent dark state. The counter DKT measures the number of consecutive SIPs belonging to this situation and is now incremented by one. If the counter DKT exceeds the preset dark time limit DKL, which determines the longest time to wait for the current emitter to re-emit, the measurement state is set to dark. In this state, the microscope skips localization and probes individual SIPs in each subsequent iteration to fill the corresponding histogram HST. k During the initialization of the next positioning event, the measurement state will revert to active, indicating that it expects a signal from the current transmitter.

[0112] While poorly performing background structures may require additional effort, such as multiple instances of background estimation assigned to different regions of interest, it is worth noting that this simple approach has proven effective for the imaging task at hand. As a beneficial side effect, the focusing defect of increasing the center intensity of the ring above zero is also mitigated, as it manifests as an additional background contribution primarily dependent on the average emitter brightness and the applied laser power.

[0113] The above description uses histogram HST k The method of using the location of the absolute maximum value as the value EBG is a computationally inexpensive choice. It requires histogram bin sizes that are chosen to be wide enough to allow the formation of the maximum value, while being narrow enough to prevent quantization effects caused by distorted estimation. In practice, a bin size corresponding to 5%–15% of the expected individual emitter intensity represents an effective trade-off. More sophisticated methods can be applied for higher estimation accuracy and / or less dependence on measurement parameters. For example, the bin size can be chosen to be less than 1% of the expected individual emitter intensity. In this case, peak detection algorithms can be applied to derive the value EBG. For example, after smoothing the histogram using a Gaussian kernel, the value EBG can be derived as the highest peak or as the first peak in a copy of the histogram. Alternatively, the value EBG can be derived from the location of the peak containing the most total entries, rather than the most entries in each bin. Of course, other density estimation methods can be used instead of histograms.

[0114] Regarding the threshold TBG kThe influence and determination are assumed to be based on a simple case where a histogram (HST) is constructed from a SIP containing either background or background plus the signal from the emitter of interest. k Histogram (HST) k Two peaks will be observed, amplified by the Poisson noise in the detected signal and variations in emitter and background brightness. Under the above measurement conditions, the background peak will exhibit a global maximum, and its location will be the value EBG. If the background peak is approximately symmetrical, and the threshold TBG... k If set to zero, approximately half of the measured pure background SIP will be incorrectly interpreted as a signal from the current emitter, i.e., a false positive will occur. If the threshold TBG... k Choosing a larger value will result in more and more emitter peak areas eventually falling below the value EBG+TBG. k The resulting threshold, and the corresponding portion of the measured emitter SIP, is discarded. Typically, the threshold is TBG. k The threshold can be set to 10%–50% of the expected intensity of a single emitter during measurement, which decreases with the iteration index. A higher threshold during the initial iteration k=0 prevents background chasing. In higher iterations, a threshold TBG can be tolerated. k The values ​​are lower and therefore more false positives are tolerated because the emitters of interest have already been identified, and the background compensation scheme mitigates the impact of false positive SIP classification on mislocalization.

[0115] In practice, the usable field of view for a single MINFLUX positioning step is typically limited to less than half the wavelength of the excitation light used. To scan an extended sample within a micrometer-scale region of interest, the microscope spans it using a hexagonal grid of scanning positions. Figure 8The example shown illustrates the principle of iterative MINFLUX localization. In this example, the process begins with pre-positioning via pinhole orbit scanning, where the pinhole is sequentially projected onto points of a hexagonal target coordinate pattern, with no detection position at its center. After pre-positioning via pinhole orbit scanning, the iterative process performs three intermediate subsequent iterations and one final iteration. In the accompanying figures, the projected pinhole positions are indicated by solid black circles. Hollow black circles represent the target coordinate pattern applied to the iteration, i.e., the center position of the ring. The center position of the target coordinate pattern is also marked, although it is not used as a detection position. An asterisk indicates the position of the fluorophore. After each iteration, the target coordinate pattern is re-centered based on a previous emitter position estimate derived from a calibrated minimum mean square estimate based on the recorded photon counts. Thus, the process rapidly magnifies the emitter, reducing the target coordinate pattern to a smaller size L. Uniform accuracy of ~2 nm is provided. Subsequent emitted photons can be used to further refine the position estimate until the emitter ceases to fluoresce.

[0116] Figure 9 The diagram shows the target coordinate pattern applied during the MINFLUX localization process, along with the simulated average position estimate and confidence interval. Solid black circles represent the simulated average position estimate. Ellipses represent confidence intervals separating the emitter's position, indicated by a black cross. Figure 9 a) The pre-position is shown. Figure 9 b) shows the intermediate positioning and Figure 9 c) shows the final positioning. The distance d between the scan positions is best at the excitation wavelength λ used. exc Between 10% and 50%, i.e., 0.1·λ exc ≤d≤0.5·λ exc These locations are repeatedly probed during measurement and serve as the starting point for the activation, search, and localization of fluorescent markers. Since the field of view of a single MINFLUX localization step is typically limited to an area approximately the diffraction limit or even smaller, according to one aspect of the invention, a less precise but wider-range localization mode, named pinhole orbit scanning, is achieved, which improves the efficiency of the search process. In contrast to previous MINFLUX implementations, pre-localization does not use a regularly focused beam. Instead, a galvanometer and an electro-optic scanner work together to keep the annular excitation beam stationary at the grid positions, while the pinholes are sequentially projected onto points in a hexagonal target coordinate pattern, i.e., m = 6, with no probe position at its center and beam spacing of approximately L = 300 nm. The position estimate is then obtained according to equation (4), as in the MINFLUX mode, where 40 photons are matched in one or more steps. kTo prevent black dots that might originate from the zero point of the torus from appearing near the grid location, the usable range of d should ideally be limited to approximately λ. exc / 2. This arrangement provides a detection radius of approximately the entire wavelength and significantly simplifies setup because it makes dedicated hardware for microsecond-level switching between different focal intensity distributions (e.g., between Gaussian and toroidal shapes) redundant.

[0117] References

[0118] [1]SW Hell: "Far-Field Optical Nanoscopy", Science 316, 1153-1158 (2007).

[0119] [2]SW Hell et al., “Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy”, Optics Letters 19, 780-782 (1994).

[0120] [3]SW Hell: "Toward fluorescence nanoscopy" Nat Biotech 21, 1347-1355 (2003).

[0121] [4] E. Betzig et al.: "Imaging Intracellular Fluorescent Proteins at Nanometer Resolution", Science 313, 1642-1645 (2006).

[0122] [5] A. Sharonov et al., “Wide-field subdiffraction imaging by accumulated binding of diffusing probes”, Proceedings of the National Academy of Sciences 103, 18911-18916 (2006).

[0123] [6] F. Balzarotti et al.: “Nanometer resolution imaging and tracking of fluorescent molecules with minimal photon fluxes”, Science 355, 606-612 (2017).

[0124] [6a] F. Balzarotti et al.: “Supplementary Materials for Nanometer Resolution Imaging and Tracking of Fluorescent Molecules with Minimal Photon Fluxes”, Science 355, 606-612 (2017).

[0125] [7] KC Gwosch et al.: “MINFLUX nanoscopy delivers 3D multicolor nanometer resolution in cells”, Nature Methods 17, 217-224 (2020).

[0126] [8] Y. Eilers et al.: “MINFLUX monitors rapid molecular jumps with superior spatiotemporal resolution”, Proceedings of the National Academy of Sciences 115, 6117-6122 (2018).

[0127] List of reference numerals

[0128] 1 sample

[0129] 2. Launcher

[0130] 3. Light distribution

[0131] 4. Minimum Intensity

[0132] 5. Areas with increased intensity

[0133] 10 Microscopes

[0134] 11 Inverted microscope platform

[0135] 12 controllers

[0136] 13 Excitation Beam

[0137] 14 Lasers

[0138] 15 λ / 2 waveplate

[0139] 16 galvanometer scanning units

[0140] 17 Electro-optic beam deflector

[0141] 18 Electro-optic beam deflector

[0142] 20 λ / 2 waveplate

[0143] 21 Spatial Light Modulator

[0144] 22 λ / 4 waveplate

[0145] 23 beam splitters

[0146] 24. Activate the laser

[0147] 25 Neutral Density Filter

[0148] 26 λ / 4 waveplate

[0149] 27. Activate the beam.

[0150] 28 beam splitters

[0151] 29 Fluorescence

[0152] 30 Variable pinhole

[0153] 31 Avalanche Photodiode

[0154] 32 Avalanche Photodiode

[0155] 33 Dichroic mirror

[0156] 35 Estimators

[0157] 40 devices

[0158] 41 Interface

[0159] 42 Irradiation device

[0160] 43 Acquisition Device

[0161] 44 Processing device

[0162] 45 Control device

[0163] 46 Controller

[0164] 47 Local storage units

[0165] 48 User Interface

[0166] 50 devices

[0167] 51 Processing Equipment

[0168] 52 Storage devices

[0169] 53 Input

[0170] 54 Output

[0171] EBG estimated background

[0172] HST k Histogram

[0173] M i Launcher position

[0174] P i Detection location

[0175] SIP photon counting vector

[0176] ΣSIP (Sum of Photons Counted)

[0177] TCP probe location group

[0178] S1 illuminates the sample at the detection location group with light.

[0179] S2 Acquires fluorescent photons

[0180] S3 adds the photon count of the acquired photons to the photon count vector or the total photon count.

[0181] S4 Determine the value representing background noise.

[0182] S5 Real-time Adjustment Estimator

Claims

1. A method for adjusting an estimator (35) used in a microscope (10) for use based on one or more sets (TCP) probe positions (P i The method of estimating the position of emitter (2) in sample (1) by illuminating the sample with light and acquiring fluorescent photons at the probe position (Pi) of the set (TCP) includes: - The sample (1) is illuminated (13) at one or more sets (TCP) detection positions (Pi) (S1); - Acquire (S2) fluorescent photons at the probe location (Pi) for this group (TCP); - Add (S3) the photon count of the acquired (S2) photons to the photon count vector (SIP) or the photon count sum (ΣSIP) for the detection position (Pi) of the group (TCP); - Determine (S4) the value representing background noise (EBG) from the photon count vector (SIP) or the photon count sum (ΣSIP); and - The estimator (35) is adjusted in real time (S5) using the value representing background noise (EBG). In order to determine the value of (S4) representing background noise (EBG), only the group (TCP) where no fluorescence from emitter (2) was detected was evaluated. In order to determine whether fluorescence from the emitter (2) is detected, the sum of photon counts (ΣSIP) for a set of (TCP) detection locations (Pi) is compared with a threshold derived from the currently estimated background.

2. The method according to claim 1, wherein, The total photon count (ΣSIP) is added to the histogram (HST). k ).

3. The method according to claim 2, wherein, The histogram (HST) k () is specific to each scan iteration.

4. The method according to claim 2, wherein, The value representing background noise (EBG) is derived from the histogram (HST). k Peak value of ).

5. The method according to claim 1, wherein, The estimator (35) is adjusted (S5) to subtract the expected value of the background noise from the photon count before estimating the position of the emitter (2).

6. The method according to claim 1, wherein, The estimator (35) uses a calibration polynomial that is specific to each of the multiple scan iterations.

7. The method according to claim 1, wherein, The group (TCP) at the detection location (Pi) is illuminated multiple times (S1).

8. The method according to claim 1, wherein, A set of TCP probe locations (Pi) includes three or more probe locations (P... i ), which are arranged rotationally symmetrically on the circle.

9. The method according to claim 8, wherein, Relative probe positions (P) in a set of TCP probe positions (Pi) i They are irradiated in pairs in sequence (S1).

10. The method according to claim 1, wherein, The estimator (35) is a minimum mean square estimator.

11. A computer program product comprising a computer program, the computer program including instructions that, when executed by a computer, cause the computer to perform a method for adjusting an estimator (35) used in a microscope (10) according to any one of claims 1 to 10, for estimating the position of an emitter (2) in a sample (1) based on illuminating a sample with light at one or more sets (TCP) detection locations (Pi) and acquiring fluorescent photons for the set (TCP) detection locations (Pi).

12. An apparatus (40) for adjusting an estimator (35) used in a microscope (10) for estimating the position of an emitter (2) in a sample (1) based on illuminating the sample with light at one or more sets (TCP) detection locations (Pi) and acquiring fluorescent photons for those sets (TCP) detection locations (Pi), the apparatus (40) comprising: - Irradiation device (42), which is configured to irradiate (S1) the sample (1) with light (13) at one or more sets (TCP) detection positions (Pi); - Acquisition device (43), which is configured to acquire (S2) fluorescent photons for the group (TCP) detection location (Pi); - A processing device (44) configured to add (S3) the photon count of the acquired (S2) photons to the photon count vector (SIP) or the photon count sum (ΣSIP) for the group (TCP) detection position (Pi), and determine (S4) a value representing background noise (EBG) from the photon count vector (SIP) or the photon count sum (ΣSIP); and - Control device (45), which is configured to adjust (S5) the estimator (35) in real time using the value representing background noise (EBG). In order to determine the value of (S4) representing background noise (EBG), only the group (TCP) where no fluorescence from emitter (2) was detected was evaluated. In order to determine whether fluorescence from the emitter (2) is detected, the sum of photon counts (ΣSIP) for a set of (TCP) detection locations (Pi) is compared with a threshold derived from the currently estimated background.

13. A microscope (10), characterized in that, The microscope (10) includes the apparatus (40) according to claim 12, or is configured to perform the method according to any one of claims 1 to 10.

Images