Dark tracking, hybrid method, conical diffraction microscopy, and dark addressing

Abstract

A super-resolution technique, primarily aimed at fluorescence microscopy, obtains the three-dimensional position of an emitter through a hybrid method involving several steps. In the first step, the two-dimensional position of the emitter is obtained using a technique referred to herein as the Abbe loophole technique. In this technique, a doughnut or a combination of distributions with zero intensity at their center of gravity is projected onto a sample containing the emitter, with the doughnut zero moving toward the emitter until it reaches a position where the emitter does not emit light. In the second step, axial measurements are obtained using a 3D shaping method, characterized by the fact that the emitted light shapes an additional optical module that shapes the shape of the light emitted by the emitter. This shape depends on the axial position and the means for obtaining the axial position from the shape.

Description

Technical Field

[0001] (Related Applications) This application claims the benefit of U.S. Provisional Patent Application No. 62 / 942,559, filed December 2, 2019, entitled "Efficient Three-Dimensional Super-Resolution Positioning Method," and for all purposes, incorporates the entire contents of that application herein by reference.

[0002] The present invention mainly relates to methods and apparatuses for optical measurement, quantification, and classification of a living body using an inelastic interaction between an incident beam and a marker, for example, a fluorescent marker, or a marker based on other inelastic interactions such as Raman or multi-photon fluorescence. In the present invention, references to fluorescence or fluorescent substances should be understood as a simplification of inelastic interactions for the sake of brevity and clarity. Embodiments of the present invention can also be applied to methods and apparatuses for optical measurement, quantification, and classification of non-stained living bodies. Embodiments of the present invention can also be applied to methods and apparatuses for optical measurement, quantification, and classification of non-living bodies such as semiconductors.

[0003] (Introduction) The present invention mainly relates to methods and measurement devices. The present invention finds use particularly in microscopy, for example, in the field of biology and for obtaining biological information by optical observation.

[0004] The term "biological" refers to biological entities in the life sciences, regardless of whether the source is a eukaryotic organism, a prokaryotic organism, or a virus, and regardless of whether the purpose of the observation is for research, diagnosis, or therapeutic use. This term includes the use of the described methods and devices for medical, human, animal, plant, virus, and bacteria.

[0005] A microscope is generally an optical instrument used to observe, analyze, or measure objects that are too small to be seen with the naked eye. Microscopy is used, for example, in the field of biology to observe, study, and measure biological entities (objects) and their dynamics.

[0006] (Definitions) As used in this specification and the appended claims, the following terms have the following specific meanings, unless the context dictates otherwise.

[0007] A "set" includes at least one component.

[0008] In the case of incoherent light, "minimum" intensity includes the case where the intensity is zero.

[0009] "Lateral" refers to a plane that is orthogonal or non-parallel to the chief ray of an optical system as represented in a geometric-optics paradigm, and "axial" refers to the direction of propagation of the chief ray in the geometric representation of the optical system.

[0010] In the context of imaging, an "object" refers to the light distribution generated by light impinging on a physical object. For simplicity, assume that this light object is a faithful representation of the physical object, even if it is subject to the same constraints as the concordance between the physical object and the generated light distribution introduced by the inventor (Sirat2016) in 2010 (hereinafter "Sirat2016"). An "object plane" refers to the physical plane on which the object is placed. In the case of a two-dimensional object, assume that the microscope is focused on the object plane. For a three-dimensional object, assume that the operator manually or automatically selects the "best focus" according to some appropriate criterion, and the plane on which the microscope is focused is referred to as the object plane. An "imaging plane" refers to any (combined) plane on which the microscope images the object placed on the object plane at an appropriate magnification, and an "entrance plane" refers to the first intermediate plane, assuming that light propagates backward from the laser through the microscope to the object. The entrance plane is the imaging plane closest to the laser.

[0011] The term "value" as used in this specification and the appended claims shall refer to the actual number characterizing the quantity associated with a parameter. In an actual situation, it should be understood that the quantity associated with a parameter can be characterized within the range that constitutes the measurement accuracy. In that case, the term "value" can be used as an abbreviation for the distribution of values.

[0012] The term "system ruler" is used as a quantitative value indicating a characteristic measure of a system. In the present invention, in both a standard imaging system and a super-resolution system, the diffraction limits in the lateral and axial directions are used as system rulers. If a value is "much smaller" than the system ruler, the value is small and, in many cases, is ignored. "Much smaller" is defined as smaller than one-third or one-tenth, depending on the context.

[0013] "Temporality" is defined as a temporary characteristic. "Simultaneous" refers to events occurring at the same time, and "substantially simultaneous" refers to a temporal pattern in which several events are recorded at high speed such that the measured values obtained are only very slightly different from the measured values obtained from a completely simultaneous measurement of the same event. According to the present invention, for the sake of brevity and clarity, simultaneous refers to both completely simultaneous events and substantially simultaneous events.

[0014] The "Cartesian" axes have their well-known meanings. The three-dimensional position of a point or object can be decomposed into measurements of positions along one of three orthogonal axes. Following the convention of optical systems, in our geometric-optical representation, we distinguish the axis along the chief ray, which is called the z-axis or the axial direction, and the two axes orthogonal to the chief ray, which are called the x- and y-axes or the lateral axes.

[0015] "Dimension" is defined as any one of the three physical or spatial properties of length, area, and volume. Geometrically, a point is said to have zero dimension. A figure having only length, such as a line, has one dimension, a surface has two dimensions, and a figure having volume has three dimensions.

[0016] The "dimension" of a geometric feature refers to the dimension of the corresponding idealized feature within limits such that the size of the geometric feature (such as the "diameter" of a point object, the "width" of a line, or the "thickness" of a coating) tends to be much smaller than the size of any other dimension and approaches zero.

[0017] A "point" is a geometric feature with zero dimension and size in two or three dimensions. This is an over-simplification that omits much information about actual objects, but greatly simplifies estimation and calculation.

[0018] Without distinguishing between the two in two or three dimensions, an object that is small but not negligibly small compared to a system ruler is called a "point object". The expressions "small" or "negligibly small" should be understood in comparison to a system ruler. A point object is determined by its position and size and may or may not be isotropic in two or three dimensions. However, a point object is distinguished from a point. A point object can consist of a structure smaller than the diffraction limit, and its characteristics can be very important. In many cases, this structure can be approximated by a geometric model, and the information obtained is the parameter of the model. Most biological objects are point objects in a diffraction limit or super-resolution optical system, and precluding the information retained by their representation as point objects and point objects is a great loss. The distinction between a point and a point object is also very important in this invention, similar to a past invention (Sirat, 2017#12) by the same inventor whose full text is incorporated herein by reference.

[0019] A "line" (and other similar terms referring to shapes that are theoretically one-dimensional) refers to a geometric feature (i.e., a physical object having length, width, and thickness), where the length is at least five times either the width or the thickness. A line object is defined on the same basis as a point object.

[0020] A "line object" is an analog of a lower-dimensional point object with necessary modifications.

[0021] The "center" of a light distribution or a sequence of superpositions of light distributions is mainly referred to in conjunction with positions having intensities much lower than zero intensity or maximum intensity, and is understood as a non-rigorous expression including positions sufficiently close to the geometric center of the distribution.

[0022] For the "optical diffraction limit", Rayleigh criterion, Airy disk and its radius and diameter, the usual definitions are used. In the context of the present invention, the terms "super-resolution", "super-resolved", "super-resolution imaging", and "super-resolution microscopy" (with or without hyphens) are used to describe optical data acquisition, optical imaging, and microscopy at resolutions higher than the optical diffraction limit. The observed FWHM of a point or line is often used as a practical evaluation of the "diffraction limit", a quantitative term generally used to quantify the smallest resolvable detail. However, in an imaging system, the "Rayleigh criterion" is the generally accepted criterion for the smallest resolvable detail.

[0023] In an optical system, the abbreviation "diffraction size at the incident (intermediate) plane" is used to characterize the geometric range of the diffraction limit at the incident plane or intermediate imaging plane. For example, in the case of a pixelated DMD or SLM used in image projection, additional resolution is blurred by the diffraction phenomenon, so a normal system uses a pixel size close to the diffraction size at the incident plane. We present different strategies in the present invention.

[0024] The "Abbe resolution limit" used in this specification is confirmed in (Schermelleh, Heintzmann et.al.2010) (hereinafter "Schermelleh2010"), which is incorporated herein by reference. Abbe's famous resolution limit is very attractive because it simply depends on the maximum relative angle between different waves that leave the object and are captured by the objective lens and sent to the image. This resolution limit describes the smallest level of detail that can be imaged with this PSF "brush". Details of periodic objects smaller than this shortest wavelength cannot be sent to the image.

[0025] The expression "beyond the Abbe limit" is defined to refer to an object that includes a periodic structure limited by the Abbe limit by including details smaller than the details of the system ruler. The basis for this definition is that such an object includes spatial frequencies that exceed the Abbe frequency circle at the aperture plane.

[0026] In estimation theory and statistics, the "Cramér–Rao bound (CRB)" or equivalently the "Cramér–Rao lower bound (CRLB)" represents a lower bound on the variance of an estimator of a deterministic (unknown but fixed) parameter. The exact definition adopted herein is provided at https: / / en.wikipedia.org / wiki / Cram%C3%A9r%E2%80%93Rao_bound of Wikipedia accessed on November 30, 2020, and is incorporated herein by reference.

[0027] As used herein, the term "localized" light distribution shall refer to a light distribution having energy concentrated in a small range. If the energy outside a radius of 3.5 * half Rayleigh criterion is less than 2.5% of the total energy, the light distribution is localized.

[0028] The description of the present invention assumes that the described optical system is "photon noise (or shot noise) limited" as described at https: / / en.wikipedia.org / wiki / Shot_noise of Wikipedia, or is close to being photon noise limited, i.e., the Gaussian noise component is smaller than the equivalent of half the photon (or shot) noise. The best case is actually a "photon noise limited" optical system as described, and a "Gaussian noise limited" system achieves only some of the advantages of the present invention but still falls within the scope of the present invention.

[0029] The "full width at half maximum" (FWHM) is an expression that refers to the range of a function given by the difference between two extreme values of an independent variable for which the dependent variable is equal to half of the maximum value, as is the case at https: / / en.wikipedia.org / wiki / Full_width_at_half_maximum of Wikipedia accessed on November 30, 2020.

[0030] Regarding "telecentricity", for example, use the ordinary definition provided by Wikipedia at https: / / en.wikipedia.org / wiki / Telecentric_lens accessed on November 30, 2020, and distinguish between telecentricity related to the characteristics of the entrance pupil and telecentricity related to the characteristics of the exit pupil as described therein.

[0031] Regarding "fluorescence", use the ordinary definition described in https: / / en.wikipedia.org / wiki / Fluorescence accessed on November 30, 2020, and regarding "fluorescent substance", use the ordinary definition described in https: / / en.wikipedia.org / wiki / Fluorophore accessed on November 30, 2020.

[0032] "Photobleaching" refers to the photochemical denaturation of dye or fluorescent substance molecules so that they can no longer fluoresce permanently or temporarily, as a modification of https: / / en.wikipedia.org / wiki / Photobleaching accessed on November 30, 2020.

[0033] "Phototoxicity" refers to the action in which excited-state fluorescent molecules tend to react with oxygen molecules to generate free radicals that can damage components below the cell level and damage the entire cell. A second, similar but slightly different physical action, "photo damage", should also be considered to avoid dependence on experimental results regarding the intensity of light projected onto the sample.

[0034] A chip having thousands, tens of thousands, or hundreds of thousands of microscopic mirrors on its surface, as described in https: / / en.wikipedia.org / wiki / Digital_micromirror_device accessed on November 30, 2020, is referred to as a digital micromirror device (abbreviated as DMD). These microscopic mirrors, when used for image projection, are arranged in a rectangular array corresponding to the pixels in the displayed image. The DMD can also be used as a component of an optical system and optical processing, mainly for image projection. The individual mirrors of the DMD are abbreviated as pixels. The DMD pixels can be in an on or off mode. In the on mode, the micromirror reflects light in a direct path, which is the direction of light when the pixelated DMD is replaced by a simple mirror, while in the off mode, the micromirror reflects light in an indirect path and rotates by a fixed angle with respect to the direct path. Both light rays can be used in an optical system.

[0035] A spatial light modulator (SLM) refers to an object that imposes spatially varying modulation of amplitude, intensity, phase, or polarization on a light beam (https: / / en.wikipedia.org / wiki / Spatial_light_modulator accessed on November 30, 2020). The SLM includes liquid crystal on silicon (LCOS) and devices used in LCoS displays that use ferroelectric liquid crystal (FLCoS) or nematic liquid crystal (electrically controlled birefringence effect). The SLM also includes a grating light valve (GLV) as described in https: / / en.wikipedia.org / wiki / Grating_light_valve accessed on December 2, 2020, and all acronyms are defined in past references. The direct path in a transmissive SLM refers to the optical path where the pixelated SLM is replaced by a simple optical transmission element or an absorptive SLM, and in the most common case, unlike the DMD, only one path is available.

[0036] An "acousto-optic deflector" (or acousto-optic deflection system) as described in https: / / en.wikipedia.org / wiki / Acousto-optics accessed on November 30, 2020 refers to a device that can displace or focus a light beam obliquely in one or two dimensions or as a focusing mechanism using the electro-optic effect. The multi-channel electro-optic deflector (Pape, 1992#40) is commercially available and can be used in practical systems.

[0037] An "electro-optic deflector" (or electro-optic deflection system) as described in https: / / www.conophcs.com / electro-optic-deflection-systems / accessed on November 30, 2020 and commercially available from the same company refers to a device that can displace or focus a light beam obliquely in one or two dimensions or as a focusing mechanism. Multi-channel electro-optic modulators can also be developed.

[0038] "DMD or SLM" in the present invention is an abbreviation for a device that can modulate the amplitude, intensity, phase, or polarization of a generally uniform coherent or incoherent beam, in the sense of a variable distribution of the described physical parameters, including but not limited to, for example, acoustic, magnetic, or electro-optic devices.

[0039] "Singular optics" including "optical vortices" as the simplest example is a new field of optics with logical and practical applications today. A detailed explanation can be found in (Nye, 1974#37; Soskin, 2001#38) Nye, et al., and both references are incorporated herein by reference.

[0040] A "wavefront shaper" is a device that can dynamically modify the light distribution. Wavefront shaping in microscopy mainly uses a spatial light modulator (SLM) placed in the pupil of the optical system and has been applied to control multiple scattered lights in biological tissues as described in (Park, 2018#41) or (Ritsch-Marte, 2009#39). In this document, the same technical tool can be used to simultaneously form several light spots or more complex patterns.

[0041] We refer to the singularity distribution with a radial symmetry as a "doughnut", and the position of zero intensity of these distributions as the doughnut zero, or as the zero or center of the doughnut as cited in the text of (Balzarotti, 2017#4) of the present invention.

[0042] "Inelastic optical interaction" refers to the interaction between incident light and an object that generates photons with different wavelengths. Inelastic optical interactions include, but are not limited to, fluorescence, multi-photon interactions, and Raman scattering.

[0043] The "locus of the singularity distribution" is an ensemble of Cartesian positions where the intensity of the singularity distribution is zero. The locus of the singularity distribution has a "nominal parameter" which is an appropriate parameter, and is placed at the "nominal position" which is the correct position, defining a group of basic shapes that do not emit (or reflect or scatter) light. Here, we introduce a new concept and express it as "the singularity light distribution embeds a geometric shape".

[0044] "Conical refraction" is an optical phenomenon predicted by Hamilton (Hamilton 1831) and experimentally confirmed by Lloyd two months later (Lloyd 1883). Both of the above references are incorporated into this document by reference. Conical refraction is an explanation of the propagation of light rays in the direction of the optical axis of a biaxial crystal. Hamilton predicted that light would emerge in the shape of a hollow cone of light rays. The discovery of conical refraction was an epoch-making event in the history of science and played a role in the demonstration of electromagnetic waves.

[0045] However, as early as 1898, the discrepancies between Hamilton's theory and Lloyd's preliminary experiments, as well as more accurate measurements and observations, were pointed out by Poggendorff. These inexplicable results have puzzled scientists for over 150 years and have hindered the use of this powerful effect in practical systems.

[0046] A fully logical analysis is provided in Berry (Berry2004), incorporated herein by reference, by Sir Michael Berry. Berry also changed the name of this physical effect from "conical refraction," used by Sir Hamilton, to "conical diffraction." Conical diffraction is used in the present invention.

[0047] Berry's paper and the availability of synthetic biaxial crystals of high quality and reasonable price have opened the way to the use of conical diffraction as one of the most powerful tools in the optical technology toolbox.

[0048] The inventor is one of the leading scholars who understand the practical effects of this effect. The inventor introduced the thin crystal concept as "the beauty and elegance of the Poggendorff rings and conical diffraction developed by you (Sir Michael Berry) for a monotonic but efficiently controllable beam shaping unit."

[0049] Prior art systems based on conical diffraction for super-resolution microscopy are described in (Caron, 2014#33; Sirat, 2016#36), incorporated herein by reference.

[0050] In this specification, the term "energy law" is defined as follows. Assuming that an object is modeled as the geometric shape of a mathematical abstraction, the "energy law" is the parametric relationship between energies that are functions of shape parameters and position. This law generates a relationship that quantifies the energy dependence in the parametric space. The energy law may include the energy distribution radiated by a luminous object having the same shape as the geometric shape.

[0051] According to the present invention, herein, it is assumed that, as described by Sirat in (Sirat2016), the optical singularity distribution can be controlled by external means to switch from one type of distribution to another type of distribution, and from a distribution of a given group to a distribution of another group, and to modify the parameters of the optical singularity distribution. There are also other solutions that do not require controlling the optical singularity distribution, and although they are actually part of the present invention, in the opinion of the inventors, they can be much more cumbersome.

[0052] "Control means" refers to a set of control hardware that can modify the input, and a "control algorithm" that can predict the next step of the input value necessary to quantify the "law of energy" so as to accurately obtain the parameters directly or by successive approximations. The "inverse law of energy" is a method that can obtain the optimal or non-optimal parameters of position and / or shape from the measured values of the singularity distribution or a set of singularity distributions. It is embedded in the control algorithm. Also, it is selected to optimize the functional parameters of the system, the number of necessary steps, the total energy or power applied to the living body, the measurement speed, or a combination of the above or any other functional parameters of the system.

[0053] Even in the simplest case of a single point, the law of energy depends on the three Cartesian positions of the point. Some of the irradiation options described later can, depending on the region, make it possible to lose the correlation of the dependence of the inverse law of energy from two of the Cartesian positions, greatly simplifying the information collection and accuracy improvement.

[0054] In addition, many inverse laws of energy are quadratic equations and lose information regarding the sign of the position. One solution is proposed below.

[0055] Finally, the inverse law of energy is based on hypotheses such as the object being a point, a line, or a rod. Additional measurements, redundancy, and redundant determination can be used as verification of the hypotheses.

[0056] For the purposes of this specification, assume that a separate mechanism is used to collect the nominal positions of objects. Within the scope of the present invention, this mechanism can use any microscopy method that is a wide-field confocal or super-resolution technique, or any localization method such as PALM, STED, STORM, or prior external knowledge.

[0057] In the context of the singularity distribution, a "value close to zero" is used to quantitatively describe a projection intensity that is much smaller than the maximum intensity achievable on the projection light, or an energy that is much smaller than the energy emitted when the maximum amount of projection light hits this point. When comparing between the projection intensity and the maximum intensity of the distribution, or between the emitted energy and the energy emitted when irradiated with the maximum intensity, the numerical value of the intensity or energy close to zero is one-ninth. Assuming Poisson noise, an energy close to zero is significantly smaller than the maximum energy and is worthy of mention as having a noise value smaller than one-third. Similarly, a geometric parameter value of a shape close to zero has a value smaller than one-third of the full range of the parameter.

[0058] "Conical diffraction microscopy" or "CODIM" refers to the conical diffraction microscopy described in (Sirat, 2016#11) and (Caron, 2014#33).

[0059] (Prior art: Microscopy) Refer to FIG. 1 showing an example of a microscopy 100 in the biological field.

[0060] Microscopy includes irradiating a biological sample 11 with a light source (not shown) using a microscope 10 and a time-dependent measurement of the light emitted from the sample using either visual observation or a detection module 12.

[0061] Biological samples include one or more different biological entities 13 and 14 arranged at different positions. Examples of such objects include, in particular, cells, viruses, proteins, and DNA fragments.

[0062] (Fluorescence microscopy) Fluorescence microscopy is one of the variations of microscopy and, in many biological applications, supersedes other microscopy techniques. A fluorescence microscope is an optical microscope that uses the fluorescence phenomenon, instead of or in addition to other methods such as reflection and absorption, to investigate the properties of organic or inorganic substances.

[0063] In fluorescence microscopy, a sample is irradiated with light of a certain wavelength or a specific plurality of wavelengths absorbed by the fluorescent substance, thereby promoting the emission of light of various higher wavelengths.

[0064] The illumination light is separated from the emitted fluorescence, which is mostly at a higher wavelength, using a spectral emission filter, significantly reducing the background of the acquired image.

[0065] Fluorescence microscopy is the most common in this type of method, but there are also many other microscopy methods that use inelastic interactions, i.e., samples that emit light at a different wavelength from the light source or entities fixed to the sample.

[0066] In the present invention, for ease of reading and brevity, fluorescence microscopy is referred to, but other inelastic interactions including, but not limited to, Raman, two- or multi-photon microscopy are also considered part of the present invention.

[0067] Referring again to FIG. 1, the fluorescence microscope will now be described. In fluorescence microscopy, small point sources 15-18 that are fluorescent substances are fixed to specific positions of predetermined living bodies 13 and 14 based on the physical phenomenon of single-photon fluorescence. Instead of observing the light emitted by the living bodies 13 and 14, the light emitted by the fluorescent substances is observed.

[0068] Fluorescent substances have become an important tool for visualizing living bodies. Biological information including activity and details less than 200-250 nm of the diffraction limit is systematically examined and measured using fluorescence microscopy. This resolution limit is obtained from the Rayleigh criterion and, in the best case, reaches 200-250 nm in specially designed systems.

[0069] In fluorescence microscopy, it should be noted that the information collected and acquired is a map of fluorescent substances, not a direct image of a living body, as pointed out by several authors including Sirat-2016. The relationship between the measured values and the object relies on hypotheses and is reliable in most cases.

[0070] The main implementations of fluorescence microscopy are confocal microscopes, which are often used in scanning configurations, or spinning disk microscopes, and wide-field imaging microscopes, as described in detail in the same literature.

[0071] Next, refer to FIG. 2, which is a schematic diagram of a conventional confocal fluorescence microscope 200. The confocal fluorescence microscope in FIG. 2 is an optical instrument. Its main hardware components are shown in FIG. 2. These components include a light source 20, an optical mechanical frame (not shown), a cube filter 21, a microscope objective lens 22, a detector assembly 23, and a processing unit (not shown).

[0072] The light source 20 may be an arc lamp or a laser and generates the light energy required for fluorescence. The optical mechanical frame (not shown) is a support for all optical components and auxiliary optical systems and includes an alignment function. The frame also includes optical elements (not shown) that can shape the beam to enable a minimum-sized focus by the microscope objective lens. The frame further includes a spatial or angular scanning mechanism (not shown) for changing the position of the point source with respect to the object being measured in confocal scanning fluorescence.

[0073] Alternatively, the scanning mechanism can · For example, mechanically translate the object by using a translation stage, · For example, optically scan the beam over the object by using a set of galvanometer mirrors or acousto-optic translation means, or · Use any combination of these mechanical or optical translation means.

[0074] In confocal scanning fluorescence, information is collected point by point using a scanning mechanism. The scanning mechanism can further include a rotating disk having a plurality of pinholes that enable simultaneous projection of multiple points in a rotating disk type confocal fluorescence. In a confocal fluorescence rotating disk, a set of points corresponding to the pinholes is acquired at any given time, and rotation of the disk enables scanning of the entire surface of the sample with respect to a given longitudinal position.

[0075] The cube filter 21 carries various optical signals and prevents contamination of the fluorescence signal by radiation. The cube consists of filters such as an excitation filter 210, a dichroic mirror 211, and an emission filter 212. The filters and the dichroic mirror are selected according to the excitation wavelength and emission spectral characteristics of the fluorescent substance.

[0076] The microscope objective lens 22 focuses the light generated by the light source at the focal plane of the lens 24, and the optimal light distribution of the small-diameter light distribution pattern consists of an Airy disk. The microscope objective lens 22 also collects the fluorescence emitted by the fluorescent substance.

[0077] In the case of confocal scanning fluorescence, the system can perform descanning, that is, the return light passes through the scanning mechanism and can compensate for the translation due to scanning.

[0078] An enlarged image of the focal plane of the lens 24 is generated at the imaging surface of the detector lens 25 and the detector 26.

[0079] The confocal hole 27 is logically disposed at the imaging surface of the detector 26. In most practical systems, the confocal hole 27 is disposed at an intermediate imaging surface (not shown) and re-imaged onto the imaging surface of the detector 26.

[0080] The detector assembly 23 detects the fluorescence intensity over the entire exposure dose and converts it into a digital signal. In the simplest implementation, in the case of a confocal scanning microscope, the detector assembly comprises a single-element detector such as a photomultiplier tube PMT or a single-photon avalanche diode SPAD. In the case of a confocal microscope using a rotating disk, the detector assembly consists of a matrix of detector elements such as a charge-coupled device CCD, an electron-multiplying CCD EMCCD, a complementary metal-oxide-semiconductor CMOS, or a matrix of SPADs.

[0081] All components from the light source to the dichroic filter are mounted on the illumination path 201. The detection channel 202 represents all components mounted from the dichroic filter to the detector assembly.

[0082] Fluorescence microscopes are available from several manufacturers such as Zeiss, Leica, Nikon, and Olympus. The fluorescence microscope can be either a standard microscope suitable for fluorescence or a microscope specifically optimized for fluorescence. Recent microscopes are multifunctional devices that can operate in many different modes, including but not limited to fluorescence modes that use the same platform and most of the optomechanical components. Most fluorescence microscopes have been developed as open platforms that can implement several additional features with minimal changes. Other fluorescence microscopes are dedicated devices adapted for special tasks such as medical diagnosis or pharmaceuticals.

[0083] (Prior art: super-resolution) For a long time until the emergence of the super-resolution techniques described below, optical techniques including fluorescence microscopy were widely considered unable to visualize details smaller than the Rayleigh criterion, which is about 200 - 250 nm for visible light.

[0084] However, other basic biological activities also occur at scales less than 200 nm within a biological sample. At this level of spatial resolution, important phenomena such as intracellular-scale biological processes, cell signaling, protein folding and unfolding, and DNA and RNA changes can be observed. For example, the measurement of this intracellular information opens up new avenues for understanding biological activities and advances the understanding and monitoring of investigations and medical diagnoses.

[0085] "Super-resolution microscopy is a term that encompasses several techniques that enable the acquisition of images in optical microscopy with a resolution higher than that imposed by the diffraction limit" (Contributors 2019). Wikipedia, for example, accessed on November 30, 2020, at https: / / en.wikipedia.org / wiki / Super-resolution_microscopy, uses the definitions found for STED, photoactivated localization microscopy, PALM, STORM, and many other methods. Additional super-resolution methods known to those skilled in the art are also included among the super-resolutions.

[0086] Conical diffraction microscopy is a super-resolution method developed by the inventors (Sirat 2016), hereinafter referred to as "Sirat 2016".

[0087] Some microscopy methods use a reconstruction process. That is, the direct data physically acquired by the detection system is not the final result, and an algorithmic process is required to obtain the final image.

[0088] The positive constraint, i.e., the physical fact that the light intensity is essentially positive, adds supplementary constraints to the mathematical solution for the optical system in which the algorithm is used to calculate the final object (reconstruction). The positive constraint can be applied to the general case, restricting the possible solutions and somewhat improving the quality of the result. However, in a specific case described later and named by the inventor as Abbe's loophole, this constraint makes it possible to overcome Abbe's resolution limit, which is the case in the present invention.

[0089] (Imaging, localization, and tracking...) In the present invention, in many cases, although they can be evaluated on the same device, imaging and localization, which are two different functions, are mentioned separately.

[0090] During imaging, we observe the object without suggesting any additional information about the object in advance. Localization assumes that the object is a point and can be parameterized by a few descriptors. "Tracking" is a short term for dynamic localization or localization as a function of time, and depending on the context, either localization or tracking is used.

[0091] During imaging, the observed object is different from the original object and can be described in terms of degrees of freedom using a finite number of degrees of freedom restricted by the optical system (Lukosz 1966). The observed object is considered a filtered version of the original object and is restricted by diffraction in the general case, even if some super-resolution schemes can be applied.

[0092] In localization, the object is known or assumed to be a single point. It is a problem of parameter search, quantified by the Cramer-Rao criterion, and is not clearly restricted by the diffraction limit but is restricted by the signal-to-noise ratio.

[0093] Between the two extremes of a complete lack of prior information and the parametric description of an object, there are many cases, some of which contain some partial information. As an example, the assumption that a scene consists of sparse objects can, in some cases, enable additional information or additional information about objects that can be converted into additional information or resolution. In (Sirat2017), with respect to the Abbe loophole technique described below, the inventors extended the problem of locating a single emitter to the case of a simple geometric object that obtains both position and shape descriptors.

[0094] (... and metrology) "Metrology" refers to a third method in addition to imaging and tracking, in which, as described in (Sirat2017) incorporated herein in its entirety, in addition to the position of a single point or a small set of single points, a descriptive geometric parameter set is obtained that describes the observed entity as a simplified geometric and / or temporal object. The inventors consider metrology to have a practical impact similar to imaging and tracking and to be a new and additional beneficial method of the present invention.

[0095] (Alternative routes to localization) Worldwide, there are the following two strategies that can be applied jointly for either or both lateral or axial measurements to determine the position of an object assumed to be a point source of radiation of infinitesimal size (hereinafter abbreviated as emitter) beyond the diffraction limit. · Projection strategy: Project a light distribution of infinitesimal size and record the regression energy · Emission strategy: Analyze the regression light under the assumption of infinitesimal size

[0096] The STED method described in https: / / en.wikipedia.org / wiki / STED microscopy accessed on November 30, 2020, and the references cited therein is a typical example of a localization projection strategy. Through careful non-linear manipulation, the projection light can be focused to a size smaller than the diffraction spot logically infinitely in the case of infinite energy projection. Both two-dimensional and three-dimensional solutions are described and implemented in actual working systems.

[0097] The localization techniques described in https: / / en.wikipedia.wiki / Super-resolution_microscopy#Localization_microscopy_SPDM accessed on November 30, 2020, including techniques called PALM and STORM, are typical examples of a localization emission strategy. Assuming that the emitter is a single point, in the case of two-dimensional localization, if the light distribution generated on the detector is a single spot, the centroid of the spot can be obtained through an appropriate algorithm. The centroid is a measure of the position of the emitter. Assuming infinite energy, the position can be obtained with infinite precision.

[0098] For example, more advanced techniques using appropriate optical means, such as (Pavani, Thompson et.al.2009) or (Fallet, Dubois et.al.2015), mainly targeting three-dimensional localization, shape, and emitted light, generate a more complex and accurate relationship between the localization value of the shape of the detected light and the parameters. In the present invention, these techniques are referred to as three-dimensional shaping of the emitted light. These techniques can measure the axial position of the emitter but require a minimum number of photons to do so.

[0099] In the present invention, a method in which the light projected by an emitter is shaped to form a predetermined shape that depends on the axial position, and the shape is obtained from the axial position, is referred to as the "3D projected light shaping method". The concept of 3D STED is an example of such a method in CODIM, and (Fallet, 2015#15; Fallet, 2016#35; Sirat, 2016#ll; Sirat, 2017#8; Sirat, 2017#12; Sirat, 2017#31; Sirat, 2016#36; Sirat, 2017#34) describes several methods of shaping the projected light to make it dependent on the axial direction.

[0100] In the present invention, a method in which the light emitted by an emitter is shaped by an additional optical module that forms a predetermined shape depending on the axial position, and the shape is obtained from the axial position, is referred to as the "3D emission shaping method". Some considerations of these techniques can also be found in (Martens, Jabermoradi et.al.2020).

Brief Description of the Drawings

[0101] The above features of the present invention will be more easily understood by referring to the following detailed description while referring to the accompanying drawings below.

[0102]

Figure 1

Figure 2

[0103]

Figure 3

[0104] FIG. 4 shows four alternatives for generating a single point from the incident light rays, assuming, although not essential, that the beam is parallel at the entrance of the device.

Figure 4a

Figure 4b

Figure 4c

Figure 4d

[0105] FIG. 5 shows a simplified algorithm and process for obtaining the two-dimensional or three-dimensional position of an emitter.

Figure 5a

Figure 5b

Figure 5c

Figure 5d

Figure 5e

Figure 5f

Figure 5g

Figure 5h

[0106] In one embodiment, the present invention provides a method for determining the three-dimensional position of a luminescent object within a sample. In this embodiment, the method comprises projecting a first set of light radiation distributions onto a portion of the sample containing the luminescent object along a first projection axis, the light radiation distributions having zero intensity at the respective common centers of the set of distributions, measuring the intensity of the emitted light, and moving the set of distributions to a first point on the object where the light radiation is minimal and which characterizes two of the dimensions of the position within a first coordinate system by using Abbe's loophole technique to characterize two of the dimensions of the position within the first coordinate system; and using an optical module to shape the light emitted from the object so that the resulting shape depends on the axial position of the object before detecting the light, recording the shape and the overall intensity, and determining the axial position by using the measured values of the resulting shape; comprises.

[0107] In related embodiments, the present invention projects a second set of distributions of light radiation onto an object along a second projection axis, the distributions of light radiation having zero intensity at each of the three-dimensional positions of the set of distributions determined according to claim 1, and moving the set of distributions to a second point on the object where the light radiation is minimal and which characterizes all three dimensions, thereby characterizing all three dimensions of the position in a second coordinate system with higher precision by using Abbe's loophole technique to further include determining the position of the luminescent object. Optionally, the method further includes obtaining an estimate of the position of the two-dimensional or three-dimensional position of the luminescent object by a standard positioning or imaging method before characterizing two of the dimensions of the position in the first coordinate system. In additional related embodiments, the present invention further includes measuring the intensity of light emitted along a selected Cartesian axis within a segmented region including a point having a minimum intensity during the process of projecting a set of distributions of light radiation onto the luminescent object selected from the group consisting of the first set, the second set of distributions, and combinations thereof, the intensity along the Cartesian axis providing a measure of the position of the luminescent object along the Cartesian axis and losing the correlation from the position of the luminescent object along the other axes. In additional related embodiments, the set of distributions includes at least two distributions. In additional related embodiments, to provide verification of the determined position, the set of distributions of light radiation having zero intensity at each common center is selected such that the values of the set measured from intensity and shape are selected from the group consisting of redundant and overdetermined.

[0108] In another embodiment, the present invention provides a method for determining the positions of a set of separate light-emitting objects in two or three dimensions. In this embodiment, the method uses a scanning device to project a distribution of light radiation of a set having zero intensity at each common center in two or three dimensions onto each of the objects of the set along the projection axis, where the centers are different for each object of the set of objects, measures the intensity of the light, and uses the scanning device to move the center of the set of distributions to a point on the object of the set of objects where the light radiation is minimum for each object of the set of objects, thereby characterizing the two- or three-dimensional position of each of the separate light-emitting objects. Optionally, the scanning device used to move the position is an SLM or DMD positioned on the surface coupled to the sample, and in the scanning device, in the SLM or DMD, only the pixels coupled to the determined positions are in the on mode and the collected light is in a direct path, or, in the DMD, only the pixels coupled to the position of the emitter are in the off mode and the collected light is in an indirect path. In a related embodiment, the size of the spot corresponding to the diffraction limit is 3 to 11 times the size of the pixel size. In an additional related embodiment, an algorithm is applied to determine the appropriate combination of on and off of the pixels to position the centroid of the spot on the sample with a resolution better than that of a single overall pixel of the scanning device projected onto the sample surface. In a further additional related embodiment, a first optical module is used to generate a light distribution including the position of the emitter as part of the illumination region. Optionally, the scanning module is selected from the group consisting of a multi-channel acoustic or electro-optic deflector, a wavefront shaper, and combinations thereof.

Embodiments for Carrying Out the Invention

[0109] (Abbe's loophole) We refer to the Abbe loophole, a term coined by the inventors in past patent applications and publications, to describe a group of projection techniques. However, emitter localization is derived from the absence of light generated by the projection of an optical pattern, including the approximate center zero of the energy on an emitter that is assumed to be of infinitesimal size.

[0110] The basis of this scheme is that the Abbe's law excludes the direct measurement of spatial frequency components beyond the diffraction limit. However, the positive constraint generates a fundamental additional relationship between all frequency components. In the specific case where all frequency components below the diffraction limit and the spatial DC component are zero (black image), this constraint requires that all frequency components beyond the diffraction limit are also zero. In this loophole, it is known that both frequency components below and above the diffraction limit are zero, and this indirect measurement of all frequency components is a practical violation of the diffraction limit.

[0111] In the present invention, the names "White system" for describing a localization system based on the above-described radiation strategy and "Black system" for describing a localization system based on the Abbe loophole are used.

[0112] A group of techniques based on the Abbe loophole use a vortex or another similar distribution having zero intensity exactly at the center or approximately at the center as the light distribution projected onto the sample. This approach was proposed in a divisional application (Sirat2017) with a priority date of October 2010 (Sirat2016) using the name "black fluorophore". This technique is also described in several papers such as those with a priority date of November 2011 (Hell2016), and later (Balzarotti, Eilers et.al.2017, Gwosch, Pape et.al.2019), and is publicly available and commercially available under the name MINFLUX.

[0113] The extension to a three-dimensional case with an appropriate three-dimensional distribution can be found in the cited references.

[0114] Another complementary technique that the inventors named "dark tracking" in the published literature uses multiple light distributions that are sequentially projected onto the sample, have a similar radial function and a common central zero, but different azimuthal dependencies (Sirat2016, Sirat2017). This technique was proposed in a divisional patent (Sirat2017) with a priority date of October 2010 (Sirat2016).

[0115] The difference between these two techniques is that in dark tracking, in addition to the radial direction information, which is the only information available in the "black fluorophore" / MINFLUX technique, the azimuth angle is obtained with the same number of photons. This additional information greatly simplifies the function of reaching the zero - in this case the sum of the distribution - to the position where the emitter is located.

[0116] A derivative technique that the inventors call "metrology" in (Sirat2017) generalizes the "dark tracking" technique to emitters consisting of simple geometric objects as the above - mentioned point objects and line objects, rather than on infinitesimally small points, in order to obtain both the position of the emitter shape and the descriptors of the shape.

[0117] In the present invention, as will be described below, a technique that enables simultaneous or almost simultaneous execution of dark tracking on several identified targets is called "dark addressing".

[0118] These techniques enable theoretically unlimited additional resolution by combining the following. · Abbe's loophole principle · Near - complete absence of noise according to Poisson's law, which in a particular case is called Abbe's loophole · Exception at the Cramer - Rao limit close to zero intensity

[0119] To obtain the best performance from these techniques, it is expected that there are no spurious photons in order to meet the theoretical conditions and reach the optimal infinite resolution. The condition of the absence or at least minimization of spurious photons is more easily met by using inelastic light interactions such as fluorescence, multi-photon interactions, and Raman scattering. In these methods, the incident beam can be completely filtered by spectral means with (almost) no single photons left over.

[0120] Even if background photons limit the highest achievable resolution and form part of the present invention, all other optical techniques can also utilize these principles.

[0121] More precisely, the technique described above as Abbe's loophole technique can be structured as including the following three steps: a detection step, an intermediate step (positioning to a donut zero or a dark tracking coupling zero close to a few nanometers or less than 20 nm to the emitter, and this step is referred to as the "demon step"), and a precision step.

[0122] The oldest description of the step we refer to as the demon step was found in Sirat 2010. However, we will explain the technique we call Abbe's loophole and its limitations in the present invention with reference to the later, explanatory and very accessible revised version called Balzarotti 2017 (Balzarotti, Eilers et.al. 2017) (cited in full). Next, perform a thought experiment that attempts to determine the trajectory of a molecule diffusing in space. Instead of using uniform wide-field excitation and a camera, we excite a reasonably bright focal donut that can be rapidly moved across the focal plane. If we or the demon somehow manage to accurately target the molecule with the zero of the donut-shaped excitation beam and guide it to always overlap with the molecule in space, the donut target device will be able to completely map the molecule without inducing single emission. On the other hand, for example, single emission (due to the smallest misplacement) would be sufficient to know that the molecule is not at the location of the donut zero. Unfortunately, we cannot know the position of the molecule in advance and place the donut at its coordinates with a single shot. Therefore, complete localization without radiation remains a privilege of the demon. However, as this thought experiment suggests, multiple shots exploring the position of the molecule at zero intensity should reduce the radiation required for localization. This is because, in our view, fluorescence radiation is the price paid for not knowing the position, and the closer it gets to zero during the exploration process, the smaller the price. In fact, apart from confirming the presence of the molecule, radiation is very useful for transmitting information about the distance to zero being explored.

[0123] In this application, a technique using a combination or sequence of donuts or distributions in two or three dimensions is referred to as the Abbe loophole technique or the black technique. This technique is unique in the fact that an intensity zero or an intensity close to zero existing at the combination center of the distribution in one, two, or three dimensions is projected onto a sample containing an emitter, and the donut zero moves towards the emitter until it reaches a position where the emitter does not emit light.

[0124] The two-dimensional Abbe loophole technique can use a combination of donuts or distributions as the conical diffraction distribution shown in FIG. 3, which is projected onto the sample sequentially or simultaneously. The three-dimensional Abbe loophole technique adds a distribution that is zero at the axial position of the emitter.

[0125] Some embodiments of the Abbe loophole technique also include means for controlling the system and correcting the relative position between the zero and the emitter.

[0126] In summary, we can reach a nearly infinite theoretical resolution with a minimum photon cost, where the distance between the molecule and the donut zero is on the order of a few nm. However, at the start, the zero can be separated from the molecule by a distance of more than about 100 nm. In many cases, an intermediate step called the "demon process," which is a subsequent set of measurements, is in the low photon regime and must be carefully designed.

[0127] As a starting point, the detection of the presence of the emitter and its uniqueness are used in many cases, using relatively standard techniques. This makes it possible to reach the diffraction limit accuracy (200 - 250 nm), or an accuracy of 90 - 100 nm as exemplified by super-resolution systems developed by the inventors through some additional development (Sirat2016). The cost of its photons and system complexity can be reduced by reducing the accuracy requirements, but there is clearly an additional burden on the second step, the intermediate daemon step.

[0128] The final step where the doughnut zero is positioned at or very near the molecule and reaches an accuracy of a few nanometers is surprisingly conceptually quite simple. This step is (theoretically) a step where single photons carry yes / no information.

[0129] This claim about the simplicity inherent in the final step is debatable and is only correct at the conceptual level. Still, the system requires a carefully designed optical system, single-photon detection, motion control at the nanometer level, and low-noise electronics. These technical specifications need to avoid the generation of fake photons, electrons, or digital counts and control the movement. Nevertheless, all these most extreme specifications are very challenging, but all these requirements are within the scope of existing well-established systems and technologies.

[0130] The actually difficult step both conceptually and practically is the intermediate (daemon) step that bridges the gap between the starting point of 100 nm accuracy and the few nanometers of the final step, at the expense of reasonable photon and system complexity, through a definitive and reliable procedure.

[0131] Another notable point where Abbe's loophole technology is inherently superior to other technologies is that in many biological situations, the issue is the fact that, over long periods of time, the points follow a stationary or predictable path. Sudden and unexpected events change the dynamics of the points. These events can trigger significant events such as apoptosis or necrosis events in cells, for example.

[0132] The cost of waiting time for photons is theoretically zero in Abbe's loophole technology, and the waiting can potentially continue for a long time. The event triggers an immediately recognizable burst of photons, serving as a trigger for the detection of the event. All white technologies need to repeatedly check the position of the point at a considerable photon cost per query.

[0133] Another notable point where Abbe's loophole technology is inherently superior to other technologies is that assuming the system is well-designed and reaches a speed exceeding the standard speed of particle movement, using an expression that intentionally evokes air-to-air terms, after the system aims at the target, the photon cost of tracking the particle can be reduced by appropriate observation and control.

[0134] The main problem is to identify the molecule, which becomes even more important in the following paragraphs. In fluorescence, as is well known to those skilled in the art, the two main recognition mechanisms of a specific fluorescent substance are multi-wavelength systems and emission / transmission wavelength specificity leading to lifetime characteristics, which can also be a tool for distinguishing fluorescent substances.

[0135] When photons are available, it is possible to measure the lifetime of the incident photons. This is very important in both the daemon process and the precision process to recognize whether the incident photons are generated by the target being observed or by spurious fluorescence, ghost images, nearby objects, or any other parasitic light.

[0136] Finally, the presence of the target may need to be evaluated at any time, and the responsiveness of the target to light and the amount of light available for a given projection power are required by some, if not all, algorithms. In a conical diffraction-based implementation, the availability of a controllable amplitude Gaussian beam or a distribution similar to an Airy disk has a significant impact on the practicality of the actual system and forms part of the present invention.

[0137] (Description of Embodiments of the Invention: Hybrid Solution) This specification presents a new method where the measurement procedure is, in some cases, a separate dedicated procedure and architecture initialized by the imaging procedure.

[0138] The invention described hereinafter relates to accurately positioning an emitter with an accuracy beyond the diffraction limit of the light employed in the measurement and with a minimum light beam.

[0139] The present invention is particularly adapted to accurately position features using inelastic light interactions including, but not limited to, fluorescence or Raman scattering, where the emerging light can be separated from the incident light by simple means, but can also be applied to other microscopy modalities.

[0140] The method according to an embodiment of the present invention can advantageously provide measurements with a resolution higher than the diffraction limit imposed by optics and the minimum photon beam.

[0141] This method can be used instead of the Abbe loophole three-dimensional technique where the resolution required in three dimensions is in the range of 10 nanometers, or to simplify the "demon process" and reduce the photon cost.

[0142] We refer to this new method as the "hybrid method". As will be described later, this method uses position identification in two or three dimensions based on the analysis of synchrotron radiation in an intermediate process in order to measure the position of the emitter. The hybrid method can be used for two-dimensional position identification, but a preferred embodiment is to record the three-dimensional position of the emitter using the emitted photons.

[0143] In a preferred embodiment, this technique is called hybrid because it uses a dedicated method, a combination of projection and radiation strategies for position identification, i.e., a projection strategy for obtaining a two-dimensional lateral position and a radiation strategy for obtaining an axial position through three-dimensional shaping.

[0144] Using three-dimensional shaping of synchrotron radiation as a tool to obtain the axial position with a reasonable accuracy of about 10 to 30 nm using three-dimensional beam shaping is an excellent solution that utilizes physical existing data and information that are readily available when properly designed. The three-dimensional shaping of synchrotron radiation uses the photons generated by the two-dimensional donut used to obtain the lateral position using the projection strategy. These photons exist anyway because they are not close to the donut zero. The cost of the photons is significantly reduced and the speed of reaching the donut zero increases.

[0145] In addition, three-dimensional information is the most costly to obtain in terms of photons and system complexity. This information actually exists, but by the basic principle, it requires a three-dimensional equivalent that is much more inefficient than the two-dimensional donut. Furthermore, an approach to reach the three-dimensional position of the emitter by obtaining many different positions in three-dimensional space is also required.

[0146] In Abbe's loophole technology, the addition of three-dimensional requirements significantly complicates the system. The demon must be very clever! The change in energy can be due to the movement in any of the three Cartesian directions or any combination thereof. Obtaining the direction to move the zero of a donut or an equivalent set of distributions in space requires many measurements, leading to a large photon budget.

[0147] The object of the present invention is to avoid discouraging the demon when used as an intermediate step in Abbe's loophole technology. We simplify the task by obtaining the axis information individually using various means. The acquisition of axis information is based on photons already generated during the process by the localization projection strategy used for the localization in the lateral dimension. This individual information is obtained based on Abbe's loophole used for the localization in the lateral dimension and by using the localization emission strategy as a complement to the localization projection strategy.

[0148] In conclusion, the present invention proposes a novel and hitherto unheard-of solution by using a hybrid system consisting of the use of special and efficient information available in a way that has never been proposed before.

[0149] In another configuration, the present invention consists of three subsequent steps.

[0150] Detection step: In the detection step, the presence of the emitter, its uniqueness, and the provisional three-dimensional position of the emitter are detected. Many different techniques, including but not limited to confocal and wide-field microscopy, can be used for the detection step. The position can also be known from other deductive or external knowledge. The choice of technique is partially defined by the low photon requirements.

[0151] Demon step: Obtaining the three-dimensional position of the emitter through the hybrid method described in the present invention

[0152] Precision engineering: Acquisition of the three-dimensional position of an emitter, which consists of obtaining the three-dimensional position with the highest possible accuracy of several nanometers using three-dimensional Abbe loophole technology as described in the present invention, including the zero movement of the distribution towards the particles

[0153] (Description of embodiments of the present invention: Multiple synchronization points) (Hell2016, Sirat2016, Balzarotti, Eilers et.al.2017) The above description presents the concept of measuring the position of a single point within one dimension, two dimensions, or three dimensions, with high precision, high speed, and low photobleaching and / or phototoxicity, and potentially at several wavelengths. However, many diverse biological mechanisms rely on synchronized single events, and the ability to simultaneously (or almost simultaneously) acquire several targets that are optically separated by the distance above a system ruler or several system rulers can have an important impact on the fields of biology and medicine.

[0154] Needless to say, many biological functional events are complex and are the result of signal trafficking and the delicate balance between permissive and inhibitory signals, and the dynamics of these signals are of utmost importance.

[0155] In the present invention, a new paradigm of multi-target tracking scenarios is distinguished and defined in super-resolution microscopy, generally in microscopy, and generally in target high-resolution, such that the same scenario can be applied to white systems and, even if considered part of the present invention, adjusted for all black systems.

[0156] The multi-target tracking scenario is defined as measuring a limited number of small targets simultaneously or almost simultaneously as points, point objects, lines, line objects, or simple geometric substructures.

[0157] The additional complexity of this scenario is more than compensated for by the importance in biology of this particular case. Both the confocal and wide-field geometries are actually simpler and easier to implement compared to scenarios assigned and optimized for multiple synchronous points.

[0158] The inventors state again that such a measurement system has a major impact on the ability to measure, visualize, and quantify the root causes of functional biological activity, which can take the form of natural molecular events. This is extremely important and justifies the development of dedicated optimized solutions.

[0159] The inventors also state that this scenario is well-suited and complements two concepts introduced in their own latest invention (Sirat, 2017 #12) incorporated herein in full, which introduced metrology and deep learning schemes. The metrological features introduced there are different information adapted to the goal of correlating natural events with functional information, and the deep learning capabilities are additional tools for extracting significant information.

[0160] Using aerospace terms again, the multi-target tracking scenario simulates the coordinated attack of several aircraft and missiles with different characteristics, speeds, and kill rates, and the single-target case focuses on a single specific aircraft.

[0161] The solutions presented by (Hell 2016, Sirat 2016, Balzarotti, Eilers et.al. 2017) are based on confocal configurations, single points (and targets), and scanning systems. To replicate points, point-tracking techniques are cumbersome and require overly complex systems, but are not infeasible as described by (Gu, Li et.al. 2019).

[0162] Other solutions based on wide-field configurations, such as SIMFLUX (Cnossen, Hinsdale et.al. 2019), ModLoc (Jouchet, Cabriel et.al. 2020), SIMPLE (Reymond, Ziegler et.al. 2019), and ROSE (Gu, Li et.al. 2019), lack the advantages of Abbe's loophole technology.

[0163] In other words, two existing simplified conceptual geometric shapes that explain the dimensions of light and the dynamics projected onto an object, namely, confocal and wide-field, are not suitable for the problems and challenges described herein of tracking and identifying multiple but a small number of individual targets. Due to the selection of unoptimized configurations, all prior art solutions require some compromise either by sacrificing performance or by requiring enormous technical costs. Primarily, confocal solutions sacrifice simultaneity while wide-field solutions project light everywhere, contaminating the signal.

[0164] To illustrate the new geometric shape of the light interaction, consider again the basic point formation process of Figure 4a. Assuming uniform light rays

[40] , a simple solution to generate a single point

[50] is clearly to focus the beam using a lens or optical system

[43] and move the point

[50] onto the object plane

[49] using mechanical means, which are galvanometer mirrors

[41] and

[42] . Figure 4a shows the basic configuration used in a confocal system. The point position can be controlled using a scanning mechanism such as a galvanometer mirror as shown in Figure 4a or using an acousto-optic deflector as shown in Figure 4b.

[0165] In Figure 4b, the uniform light rays

[40] are deflected by the acousto-optic deflector

[44] and focused using a lens or optical system

[43] to generate a single point

[50] on the object plane

[49] .

[0166] Although there are similar transmission solutions, there is another solution in FIG. 4c where a uniform light beam

[40] is projected onto a controllable DMD or SLM

[48] represented as a reflective DMD device. The DMD is positioned on the imaging surface of a lens or optical system

[43] . Assuming that all pixels are switched off, as shown in FIG. 4c, except for a small area where a geometric range consisting of a single pixel [6] or several adjacent pixels

[47] is smaller than the diffraction limit, in FIG. 4d, a single point

[50] appears on the object plane

[49] . The point position depends on the position of a single pixel in FIG. 4c or the centroid of the area irradiated on the DMD or SLM in FIG. 4d. In addition, in FIG. 4d, for the case of a small area, the light generated, reflected, or transferred from the pixel is coherent, and although other cases with performance degradation can be considered, it is assumed that no additional transfer delay occurs between adjacent pixels.

[0167] The solution shown in FIG. 4c exists, but usually has many drawbacks and is considered an insufficient alternative to the standard solutions as shown in FIGS. 4a and 4b. The reason this solution is rarely used is that in the case of a single point, the energy efficiency of such a solution is very low, so such a solution is almost impractical. In fact, all the light hitting the off pixels is simply lost. In numerical terms, assuming that the area scanned by the system is 10*10μm on the sample, in the case of a diffraction limit of 250nm, the energy loss, without considering additional geometric losses, is approximately 1:1600 roughly speaking.

[0168] In addition, such a solution enables the positioning of points at individual positions corresponding to the entire pixel when a single on-pixel is used, and thus is clearly not suitable for the Abbe loophole system that requires movement of points in nanometers. As will be described later, the use of a small area smaller than the diffraction size at the incident surface first improves the movement process because the pixel is designed to be smaller than the diffraction limit, and also improves the sub-pixel process by carefully handling the on / off of individual pixel characteristics.

[0169] A straightforward way to configure direct imaging is to adjust the pixel size to the diffraction spot, as shown in Fig. 4c. This direct imaging scheme is described, for example, in (Gauthier, Lenton et.al. 2016) as a DMD or SLM.

[0170] We introduce a new scheme in Fig. 4d that uses direct imaging of a DMD or SLM. Configure the dimensions of the DMD such that the size of the diffraction-limited spot is much smaller than the diffraction size (typically on the order of 1:5 to 1:11, represented by the ratio 1:α). m , y m Assuming that light is added coherently, to form a spot centered on a particular pixel located at x it is possible to switch on all the pixels within a region of size α or sub-regions contained therein. More complex combinations of on-pixels within the above-described region can generate an Airy-like spot positioned at a predetermined portion of the pixel, enabling addressing at almost any position on the sample. Deviating further from the sub-region to increase the degree of freedom results in a merely slight modification of the spot shape by controllable and quantifiable differences that can be considered by the system algorithm.

[0171] Such a configuration provides a dual purpose of significantly reducing light loss on the one hand and being able to position points more accurately than the pixel size at the expense of the field of view.

[0172] The configuration shown in Fig. 4d requires a high tolerance for the positioning of adjacent mirrors in the DMD or the phase of adjacent pixels in the SLM to ensure that different beamlets hitting adjacent pixels are added coherently to form a single point. However, existing actual technologies can meet such requirements.

[0173] In this scheme, the spatial bandwidth is traded for the direct positioning of spots at almost any position using simple mathematical techniques in terms of the size of the region that can be addressed by the system.

[0174] In a specific case of biology, technology development enables such a trade-off. Diffraction limit of 200 μm, Availability of a DMD or SLM of 1920 * 1024, Assuming α = 7, The measurement area extends over an area of 55 * 30 μm. This value is typically adapted to single cells sized 25 - 40 μm.

[0175] Also, in such a configuration, several independent points can be irradiated simultaneously and controlled in parallel.

[0176] This solution can be improved as shown in Figure 4e, on the one hand to mitigate energy loss, and on the other hand to move points at a part of the element size.

[0177] Assume that a low-resolution system is based on the galvanometer scanner of Figure 4a, or preferably the acousto-optic deflector of Figure 4b, a multi-channel acousto- or electro-optic deflector placed at the pupil of the system as shown in Figure 4e, or a waveform shaping DMD or SLM, and generates a point on the DMD or SLM

[48] larger than the diffraction limit. Assume that all pixels are switched off except for a small area consisting of a single pixel in a configuration similar to Figure 4c not shown, or multiple adjacent pixels

[51] in a configuration similar to Figures 4d and 4e.

[0178] The additional intermediate level simply reduces the energy loss to the ratio between the larger point generated by the first-level point positioning and the diffraction-limited size on the DMD or SLM. Additionally, as shown in Figures 4c and 4d, several points can be irradiated simultaneously.

[0179] For the sake of simplicity, in this paragraph, the geometric shape of the system has been described assuming that the scanning device is positioned at the entrance of the system. This configuration seems reasonable assuming that the beam shaper follows appropriate optical constraints, mainly telecentricity. The beam shaper can also be placed in front of the scanning device that directly manipulates the distribution. Practically, this solution is simpler, but either solution is appropriate, and the engineering aspects will determine the choice between the two configurations.

[0180] In another embodiment, multiple synchronized point light distributions can be created using a waveform shaping DMD or SLM positioned at the pupil of the system in a configuration similar to that used in, for example, (Ritsch-Marte, 2009#39) or similar publications.

[0181] In another embodiment, multiple synchronized point light distributions can be generated by a multi-channel acousto-optic or electro-optic deflector described as a product of G&H at https: / / gandh.com / nproduct-categpories / multi-channel-modulators-aomc / , accessed on December 2, 2020, for obtaining substantially simultaneous points, or by an ultra-fast acousto-optic or electro-optic deflector.

[0182] (Description of Embodiments of the Invention: Simplified Cartesian Algorithm and Control System) In FIG. 5, a group of simplified Cartesian algorithms and control systems for dark tracking and dark addressing are presented.

[0183] Assuming the horizontal pattern shown in FIG. 5a, the position x of the emitter along the y-axis center reference line, specifically indicated by the star on the drawing, depends on the energy. The theoretical function is parabolic-dependent in the conical diffraction pattern, but the actual dependence can be calibrated using an appropriate procedure to account for small differences. Small dependencies in the orthogonal direction that may occur can also be accounted for by that procedure.

[0184] Assuming the vertical pattern shown in FIG. 5b, the position y of the emitter with respect to the central reference line along the x-axis, specifically indicated by the star on the drawing, is energy-dependent. The theoretical function is parabolic-dependent in the confocal diffraction pattern, but the actual dependence can be calibrated using an appropriate procedure to account for small differences. Small energy dependencies in the orthogonal directions that can occur can also be taken into account by that procedure.

[0185] Assuming the axial dependence pattern described in (Sirat, 2016 #11) shown in FIG. 5c, the energy depends on the position s, the distance between the emitter and the 3D central point of the distribution in 3D. FIG. 5c represents three cuts at different axial positions, namely, (1) the focus, (2) the axial position at 0.1 Rayleigh range, and (3) the axial position at 0.2 Rayleigh range, as FIGS. 5c1, 5c2, and 5c3. At the focus, if the point comes to the focus AND at zero of the x-y coordinates, as described in (Sirat, 2016 #11) and subsequent patents, an energy zero in this distribution will be obtained. The theoretical function is parabolic-dependent either in terms of a known combination of s or z and r in the conical diffraction pattern (FIG. 5d), where r is the lateral projection distance between points (r = √x 2 +y 2 ). The calculations can use the theoretical function or can be calibrated using an appropriate procedure to account for small differences. The energy dependence in the orthogonal direction must also be taken into account by the procedure.

[0186] The same procedure can be carried out using all axial dependence distributions described in the inventors' inventions including, but not limited to, (Sirat, 2016 #11; Sirat, 2017 #8; Sirat, 2017 #12; Sirat, 2017 #31; Sirat, 2016 #36; Sirat, 2017 #34) and other known axial dependence distributions known to those skilled in the art that feature an energy zero.

[0187] The preferred embodiment features a common zero of the lateral and axial dependence distributions that simplifies many algorithms, as shown in FIG. 5. However, other embodiments can be implemented under other conditions, such as acceptance regarding the commonality of the zeros of the distribution, a penalty on the photon cost, or the need for an additional shift of one distribution relative to other distribution(s).

[0188] The same FIG. 5 presents a second simplified Cartesian algorithm.

[0189] Assume a first horizontal pattern of FIG. 5e, which is an asymmetric version of the pattern shown in FIG. 5a, and a second horizontal pattern of FIG. 5f, which is an asymmetric version of the pattern shown in FIG. 5a and is the mirror image of the pattern of FIG. 5e. Both of these patterns can be generated using CODIM - conical diffraction microscopy with appropriate selection of the input and output polarizations. The position x of the emitter along the central reference line on the y - axis, specifically indicated by the star, directly depends on the sum of the energies recovered by two measurements. The direction of the x - position can be measured by comparing the two patterns, and the sign - positive or negative - yields a measurement value that depends on the comparison of the two energies. The theoretical function is parabolic - dependent in the conical diffraction pattern, but the actual dependence can be calibrated using an appropriate procedure to account for small differences. Small dependencies in the orthogonal direction that may occur can also be accounted for by that procedure.

[0190] Assume two vertical patterns (FIG. 5g and FIG. 5h) that behave similarly to the horizontal pattern with the necessary modifications.

[0191] Symmetric or asymmetric pattern diagrams 5a - 5h (excluding FIG. 5d) can be generated at an angle β selected to optimize the measurement or system parameters instead of the horizontal and vertical axes described above.

[0192] The measurements obtained by projecting sequences of these patterns or combinations thereof at the same position of the emitter measured simultaneously or approximately simultaneously, or as a time-dependent sequence, or in combination with a movement imposed by the operator or generated naturally, and reinforcing by simple mathematical procedures known to those skilled in the art, provide simple deterministic lateral, axial, or three-dimensional measurement procedures. These measurements can also, in combination with additional information, provide accurate positioning.

[0193] For example, the measurement of symmetric patterns is simpler and, in some cases, more accurate if the position polarity is known from external or past information. To maximize accuracy and minimize the time and the number of photons required for the measurement, combinations of these measurements, as well as their direct and obvious derivatives, can be (will be) used.

[0194] In addition, a model verification index i mod can be calculated. This index ensures that the measurements are within a reasonable tolerance and fits the established model. This index is premised on the assumptions of the model.

[0195] As an example, if the model assumes that the object is a single light-emitting point, the four measurements of the second simplification algorithm described above have some predetermined relationship. Deviations from this constraint enable the detection of outliers, which is a major problem in many positioning techniques.

[0196] With the lowest level of information, the detection of this raw data is the most reliable way to avoid errors and mistakes.

[0197] Furthermore, the model verification index can be extended, as described for the above metrics, for example, by performing additional measurements that recognize and quantify additional cases. The recognition process is a predetermined relationship based on logical, experimental, or functional grounds and uses deep learning techniques as an extension of (Sirat, 2017#12).

[0198] (Description of Embodiments of the Invention: Electronic Devices) In another embodiment, an electronic design optimized for the present invention, all CODIM systems, and all optical setups using Pockels cells in a fixed polarization state is described. Assuming the use of single or dual Pockels cells to control the polarization state, one of the limitations of a practical device is the requirement to switch from one polarization state, which is fast, to another. Even if high-voltage amplifiers and DC-DC voltage sources are commercially available, for high-speed kilovolt voltages, high currents and expensive devices are required to switch from one state to another. An alternative solution of the present invention that takes advantage of the fact that Pockels cells are electrically low-capacitance in the pF range is to charge relatively large capacitors at a reasonable current level in the range of nF or more in advance and implement a few voltage values by connecting them when required for the Pockels cell. In an additional supplementary embodiment, taking advantage of the fact that the Pockels cell is a two-electrode device and the voltage that produces the optical effect is the difference between the voltages applied to the two electrodes, a much lower voltage amplifier or DC-DC converter is used to compensate for the large voltage drop or to switch between neighboring states such as the states shown in FIGS. 5e and 5f, or the states shown in FIGS. 5g and 5h. Assuming that means known to those skilled in the art are used to monitor the large capacitance and / or the voltage on the Pockels cell, in another embodiment, a control system is added to refresh the voltage on the large capacitance as needed by taking advantage of when other polarization states are applied to the Pockels.

[0199] The references listed below are incorporated herein by reference. (References) Balzarotti, F., Y. Eilers, K. C. Gwosch, A. H. Gynna, V. Westphal, F. D. Stefani, J. Elf, and S. W. Hell (2017), "Nanometer resolution imaging and tracking of fluorescent molecules with minimal photon fluxes", Science 355(6325): 606 - 612 Berry, M. (2004), "Conical diffraction asymptotics: fine structure of Poggendorff rings and axial spike", Journal of Optics A: Pure and Applied Optics 6(4): 289 Cnossen, J., T. Hinsdale, R. O. Thorsen, F. Schueder, R. Jungmann, C. S. Smith, B. Rieger, and S. Stallinga (2019), "Localization microscopy at doubled precision with patterned illumination", bioRxiv: 554337. contributors, S.-r.m.W. (2019). "Super-resolution microscopy. In Wikipedia" Fallet, C., M. Dubois, J.-Y. Tinevez, S. Oddos, J. Caron, R. Persson, S. L. Shorte and G. Y. Sirat (2015), "A new method to achieve tens of nm axial super-localization based on Conical diffraction PSF shaping. Single Molecule Spectroscopy and Superresolution Imaging VIII", International Society for Optics and Photonics. Gauthier, G., I. Lenton, N. M. Parry, M. Baker, M. Davis, H. Rubinsztein-Dunlop, and T. Neely (2016), "Direct imaging of a digital-micromirror device for configurable microscopic optical potentials", Optica 3(10):1136 - 1143 Gu, L., Y. Li, S. Zhang, Y. Xue, W. Li, D. Li, T. Xu, and W. Ji (2019), "Molecular resolution imaging by repetitive optical selective exposure", Nature Methods 16(11):1114 - 1118 Gwosch, K. C., J. K. Pape, F. Balzarotti, P. Hoess, J. Ellenberg, J. Ries, and S. W. Hell (2019), "MINFLUX nanoscopy delivers multicolor nanometer 3D-resolution in (living) cells", bioRxiv:734251 Hamilton, W. R. (1831), "Third supplement to an essay on the theory of systems of rays", The Transactions of the Royal Irish Academy. Hell, S. W. (2016), "Method and apparatus for tracking a particle, particularly a single molecule, in a sample", Google Patents. Jouchet, P., C. Cabriel, N. Bourg, M. Bardou, C. Poiis, E. Fort, and S. Leveque-Fort (2020), "In Depth 3D Single Molecule Localization Microscopy with Time Modulated Excitation", Biophysical Journal 118(3):149a Lloyd, H. (1883), "On the phenomena presented by light in its passage along the axes of biaxial crystals", Philos. Mag. 1, 112 - 120 and 207 - 210 Lukosz, W. (1966), "Optical systems with resolving powers exceeding the classical limit", JOSA 56(11):1463 - 1471. Martens, K. I, A. Jabermoradi, S. Yang and J. Hohlbein (2020), "Integrating engineered point spread functions into the phasor-based single-molecule localization microscopy framework", Methods. Pavani, S. R. P., M. A. Thompson, J. S. Biteen, S. J. Lord, N. Liu, R. J. Twieg, R. Piestun, and W. Moemer (2009), "Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function" Proceedings of the National Academy of Sciences 106(9):2995 - 2999 Reymond, L., J. Ziegler, C. Knapp, F.-C. Wang, T. Huser, V. Ruprecht, and S. Wieser (2019), "SIMPLE: Structured illumination based point localization estimator with enhanced precision", Optics express 27(17):24578-24590 Schermelleh, L., R. Heintzmann, and H. Leonhardt (2010). "A guide to super-resolution fluorescence microscopy", Journal of Cell Biology 190(2):165~175 Sirat, G.Y. (2016), Method and device for superresolution optical measurement using singular optics, Google Patents Sirat, G.Y. (2017), Method and device for superresolution optical measurement using singular optics, Google Patents Sirat, G.Y. (2017), "Superresolution metrology Methods based on singular distribusions and deep learning", WO2019043458A2

Claims

1. A method for determining the three-dimensional position of a luminescent object within a sample, comprising: using a conical diffraction technique to two-dimensionally define the position within a first coordinate system by projecting a first set of distributions of light radiation, each having zero intensity at a common center, along a first projection axis onto a portion of the sample containing the luminescent object; measuring the intensity of the emitted light and moving the first set of distributions to a position on the luminescent object where the light radiation is minimal and which defines two of the dimensions; using an optical module to shape the light emitted from the luminescent object such that the resulting shape depends on the axial position of the luminescent object before detecting the light; recording the shape and intensity of the entire shaped object and using this measurement to determine the axial position of the luminescent object. A method comprising the above steps.

2. The method according to claim 1, further comprising defining the three-dimensional position of the luminescent object using a second set of distributions of light radiation projected along a second projection axis and having zero intensity at the three-dimensional position predetermined according to claim 1.

3. The method according to claim 1 or claim 2, further comprising pre-estimating the two-dimensional or three-dimensional position of the luminescent object using a standard positioning or imaging method before defining the position.

4. The method according to any one of claims 1 to 3, further comprising measuring the intensity of the light emitted along a selected Cartesian axis within a segmented region containing the point of minimum intensity when projecting the set of distributions, wherein the intensity along the Cartesian axis results in determining the position of the luminescent object along the Cartesian axis and losing the correlation from positions along other axes.

5. The method according to claim 4, wherein the set of distributions includes at least two separate distributions.

6. The method according to claim 4 or claim 5, wherein the set of distributions of light radiation having zero intensity at the common center of each of the set is designed to provide a set of measured values that are redundant and / or overdetermined for verifying the determined position.

Images