Dark tracking, hybrid method, conical diffraction microscopy and dark addressing

Abstract

To provide a method for determining a two-dimensional or three-dimensional position of a set of light-emitting objects.SOLUTION: A super-resolution technique, intended mainly for fluorescence microscopy, acquires the three-dimensional position of an emitter through a hybrid method comprising several steps. In a first step, the two-dimensional position of the emitter is acquired using a technique referred to in this application as the Abbe's loophole technique. In this technique, a doughnut, or a combination of distributions having zero intensity at the combined center of the distributions, is projected onto a sample containing the emitter, under conditions in which the doughnut null is moved toward the emitter until reaching a position where the emitter does not emit light. In a second step, an axial measurement is obtained using a 3D shaping method, characterized in that the emitted light is shaped by an additional optical module to form a shape of the light emitted by the emitter, the shape being dependent on the axial position, and means to retrieve the axial position from the shape.SELECTED DRAWING: None

Description

Technical Field

[0001] (Related Applications) This application claims the benefit of U.S. Provisional Patent Application No. 62 / 942,559, filed on December 2, 2019, entitled "Efficient 3D Super-Resolution Positioning Method", and for all purposes, the entire contents of the application are incorporated herein by reference.

[0002] The present invention mainly relates to methods and devices for optical measurement, quantification, and classification of living organisms using inelastic interactions between an incident beam and markers, such as fluorescent markers, and other markers based on other inelastic interactions such as Raman or multiphoton fluorescence. In the present invention, references to fluorescence or fluorescent substances should be understood as simplifications of inelastic interactions for the sake of brevity and clarity. Embodiments of the present invention can also be applied to methods and devices for optical measurement, quantification, and classification of non-stained living organisms. Embodiments of the present invention can also be applied to methods and devices for optical measurement, quantification, and classification of non-living organisms such as semiconductors.

[0003] (Introduction) The present invention mainly relates to methods and measurement devices. The present invention finds use particularly in microscopy, for example, in the field of biology and the acquisition of biological information by optical observation.

[0004] The term "biological" refers to biological entities in the life sciences, regardless of whether the source is a eukaryotic organism, a prokaryotic organism, or a virus, and regardless of whether the purpose of the observation is for research, diagnosis, or therapeutic use. This term includes the use of the described methods and devices for medical, human, animal, plant, virus, and bacteria.

[0005] A microscope is generally an optical instrument used to observe, analyze, or measure objects that are too small to be seen with the naked eye. Microscopy is used, for example, in the field of biology to observe, study, and measure biological entities (objects) and their dynamics.

[0006] (Definitions) As used in this specification and the appended claims, the following terms have the following specific meanings, unless the context clearly dictates otherwise.

[0007] A "set" includes at least one component.

[0008] For incoherent light, "minimum" intensity includes the case where the intensity is zero.

[0009] "Lateral" refers to a plane that is perpendicular or non-parallel to the chief ray of an optical system as represented in a geometric-optics paradigm, and "axial" refers to the direction of propagation of the chief ray in the geometric representation of the optical system.

[0010] In the context of imaging, an "object" refers to the light distribution generated by light impinging on a physical object. For simplicity, assume that this light object is a faithful representation of the physical object, even if it is subject to the same constraints as the concordance between the physical object and the generated light distribution introduced, for example, by the inventor (Sirat2016) in 2010 (hereinafter "Sirat2016"). An "object plane" refers to the physical plane on which the object is placed. In the case of a two-dimensional object, assume that the microscope is focused on the object plane. For a three-dimensional object, assume that the operator manually or automatically selects the "best focus" according to some appropriate criterion, and the plane on which the microscope is focused is referred to as the object plane. An "imaging plane" refers to any (combined) plane on which the microscope images the object placed on the object plane at an appropriate magnification, and an "incident plane" refers to the first intermediate plane, assuming that light propagates backward from the laser through the microscope to the object. The incident plane is the imaging plane closest to the laser.

[0011] The term "value" as used in this specification and the appended claims is intended to refer to the actual number that characterizes the quantity associated with a parameter. In a practical situation, it should be understood that the quantity associated with a parameter can be characterized within the range that constitutes the measurement accuracy. In that case, the term "value" can be used as an abbreviation for the distribution of values.

[0012] The term "system ruler" is used as a quantitative value indicating characteristic measures of a system. In the present invention, in both a standard imaging system and a super-resolution system, the diffraction limits in the lateral and axial directions are used as system rulers. If a value is "much smaller" than the system ruler, the value is small and, in many cases, is ignored. "Much smaller" is defined as smaller than one-third or one-tenth, depending on the context.

[0013] "Temporality" is defined as a temporary characteristic. "Simultaneous" refers to events occurring at the same time, and "substantially simultaneous" refers to a temporal pattern in which several events are recorded at high speed such that the resulting measured values differ only very slightly from the measured values obtained from a completely simultaneous measurement of the same event. According to the present invention, for the sake of brevity and clarity, simultaneous refers to both completely simultaneous events and substantially simultaneous events.

[0014] The "Cartesian" axes have their well-known meanings. The three-dimensional position of a point or object can be decomposed into measurements of positions along one of three orthogonal axes. Following the convention of optical systems, in our geometric optical representation, we distinguish the axis along the chief ray, which is referred to as the z-axis or the axial direction, and the two axes orthogonal to the chief ray, which are referred to as the x- and y-axes or the lateral axes.

[0015] "Dimension" is defined as any one of the three physical or spatial properties of length, area, and volume. Geometrically, a point is said to have zero dimension. A figure having only length, such as a line, has one dimension, a surface has two dimensions, and a figure having volume has three dimensions.

[0016] The "dimension" of a geometric feature refers to the dimension of the corresponding idealized feature within limits such that the size of the geometric feature (such as the "diameter" of a point object, the "width" of a line, or the "thickness" of a coating) tends to be much smaller than the size of any other dimension and approaches zero.

[0017] A "point" is a geometric feature with zero dimension and size in two or three dimensions. This is an over-simplification that omits much information about actual objects, but greatly simplifies estimation and calculation.

[0018] Regardless of whether it is in two or three dimensions, an object that is small but not negligibly small compared to the system ruler is called a "point object". The expressions "small" or "negligibly small" should be understood in comparison to the system ruler. A point object is determined by its position and size and may or may not be isotropic in two or three dimensions. However, a point object is distinguished from a point. A point object can consist of a structure smaller than the diffraction limit, and its characteristics can be very important. In many cases, this structure can be approximated by a geometric model, and the information obtained is the parameter of the model. Most biological entities are point objects in a diffraction-limited or super-resolution optical system, and precluding the information retained by their representation as point objects and point objects is a significant loss. The distinction between a point and a point object is also very important in this invention, similar to a past invention (Sirat, 2017#12) by the same inventors, the full text of which is incorporated herein by reference.

[0019] A "line" (and other similar terms referring to shapes that are theoretically one-dimensional) refers to a geometric feature (i.e., a physical object having length, width, and thickness), where the length is at least five times either the width or the thickness. A line object is defined on the same basis as a point object.

[0020] A "line object" is an analogue of a lower-dimensional point object with the necessary modifications.

[0021] The "center" of a sequence of light distributions or superpositions of light distributions is mainly referred to in conjunction with positions having intensities much lower than zero intensity or maximum intensity, including positions that are sufficiently close to the geometric center of the distribution, and is understood as an imprecise expression.

[0022] For "optical diffraction limit", Rayleigh criterion, Airy disk and its radius and diameter, the usual definitions are used. In the context of the present invention, the terms "super-resolution", "super-resolved", "super-resolution imaging", and "super-resolution microscopy" (with or without hyphens) are used to describe optical data acquisition, optical imaging, and microscopy at resolutions higher than the optical diffraction limit. The observed FWHM of a point or line is often used as a practical evaluation of the "diffraction limit", a quantitative term generally used to quantify the smallest resolvable detail. However, in an imaging system, the "Rayleigh criterion" is the generally accepted criterion for the smallest resolvable detail.

[0023] In an optical system, the abbreviation "diffraction size at the incident (intermediate) plane" is used to characterize the geometric range of the diffraction limit at the incident plane or intermediate imaging plane. For example, in the case of a pixelated DMD or SLM used in image projection, additional resolution is blurred by the diffraction phenomenon, so a normal system uses a pixel size close to the diffraction size at the incident plane. We present different strategies in the present invention.

[0024] The "Abbe resolution limit" used in this specification is confirmed in (Schermelleh, Heintzmann et.al.2010) (hereinafter "Schermelleh2010"), which is incorporated herein by reference. Abbe's famous resolution limit is very attractive because it simply depends on the maximum relative angle between different waves that leave the object and are captured by the objective lens and sent to the image. This resolution limit describes the smallest level of detail that can be imaged with this PSF "brush". Details of periodic objects smaller than this shortest wavelength cannot be sent to the image.

[0025] The expression "beyond the Abbe limit" is defined to refer to an object that includes periodic structures limited by the Abbe limit by including details smaller than the details of the system ruler. The basis for this definition is that such objects contain spatial frequencies that exceed Abbe's frequency circle at the aperture plane.

[0026] In estimation theory and statistics, the "Cramér–Rao bound (CRB)" or equivalently the "Cramér–Rao lower bound (CRLB)" represents a lower bound on the variance of an estimator of a deterministic (unknown but fixed) parameter. The precise definition adopted herein is provided at https: / / en.wikipedia.org / wiki / Cram%C3%A9r%E2%80%93Rao_bound of Wikipedia accessed on November 30, 2020, and is incorporated herein by reference.

[0027] As used herein, the term "localized" light distribution shall refer to a light distribution having energy concentrated in a small area. The light distribution is localized if the energy outside a radius of 3.5 * half Rayleigh criterion is less than 2.5% of the total energy.

[0028] The description of the present invention assumes that the optical system described is "photon noise (or shot noise) limited" as described at https: / / en.wikipedia.org / wiki / Shot_noise of Wikipedia, or is close to being photon noise limited, i.e., the Gaussian noise component is smaller than the equivalent of half the photon (or shot) noise. The best case is actually a "photon noise limited" optical system as described, and a "Gaussian noise limited" system achieves only some of the advantages of the present invention but still falls within the scope of the present invention.

[0029] The "full width at half maximum" (FWHM) is an expression that refers to the range of a function given by the difference between two extreme values of an independent variable for which the dependent variable is equal to half of the maximum value, as is the case at https: / / en.wikipedia.org / wiki / Full_width_at_half_maximum of Wikipedia accessed on November 30, 2020.

[0030] Regarding "telecentricity", for example, use the ordinary definition provided by Wikipedia at https: / / en.wikipedia.org / wiki / Telecentric_lens accessed on November 30, 2020, and distinguish between telecentricity related to the characteristics of the entrance pupil and telecentricity related to the characteristics of the exit pupil as described therein.

[0031] Regarding "fluorescence", use the ordinary definition described at https: / / en.wikipedia.org / wiki / Fluorescence accessed on November 30, 2020, and regarding "fluorophore", use the ordinary definition described at https: / / en.wikipedia.org / wiki / Fluorophore accessed on November 30, 2020.

[0032] "Photobleaching" refers to the photochemical modification of dye or fluorophore molecules so that they can no longer fluoresce permanently or temporarily, as a modification of https: / / en.wikipedia.org / wiki / Photobleaching accessed on November 30, 2020.

[0033] "Phototoxicity" refers to the action in which excited-state fluorescent molecules tend to react with oxygen molecules to generate free radicals that can damage components below the cell level and damage the entire cell. A second, similar but slightly different physical action, "photo damage", should also be considered to avoid dependence on experimental results regarding the intensity of light projected onto the sample.

[0034] A chip having thousands, tens of thousands, or hundreds of thousands of microscopic mirrors on its surface, as described in https: / / en.wikipedia.org / wiki / Digital_micromirror_device accessed on November 30, 2020, is referred to as a digital micromirror device (abbreviated as DMD). These microscopic mirrors are arranged in a rectangular array corresponding to the pixels in the displayed image when used for image projection. The DMD can also be used as a component of an optical system and optical processing, mainly for image projection. The individual mirrors of the DMD are abbreviated as pixels. The DMD pixels can be in an on or off mode. In the on mode, the micromirror reflects light in a direct path, which is the direction of light when the pixelated DMD is replaced by a simple mirror, while in the off mode, the micromirror reflects light in an indirect path and rotates by a fixed angle with respect to the direct path. Both light rays can be used in an optical system.

[0035] A spatial light modulator (SLM) refers to an object that imposes a spatially varying modulation of amplitude, intensity, phase, or polarization on a light beam (https: / / en.wikipedia.org / wiki / Spatial_light_modulator accessed on November 30, 2020). The SLM includes liquid crystal on silicon (LCOS) and devices used in LCoS displays that use ferroelectric liquid crystals (FLCoS) or nematic liquid crystals (electrically controlled birefringence effect). The SLM also includes grating light valves (GLVs) as described in https: / / en.wikipedia.org / wiki / Grating_light_valve accessed on December 2, 2020, and all acronyms are defined in past references. The direct path in a transmissive SLM refers to the optical path where the pixelated SLM is replaced by a simple optical transmission element or an absorptive SLM, and in the most common case, unlike the DMD, only one path is available.

[0036] An "acousto-optic deflector" (or acousto-optic deflection system) as described in https: / / en.wikipedia.org / wiki / Acousto-optics accessed on November 30, 2020 refers to a device that can deflect or focus a light beam obliquely in one dimension, two dimensions, or as a focusing mechanism using the electro-optic effect. A multi-channel electro-optic deflector (Pape, 1992#40) is commercially available and can be used in practical systems.

[0037] An "electro-optic deflector" (or electro-optic deflection system) as described in https: / / www.conophcs.com / electro-optic-deflection-systems / accessed on November 30, 2020 and commercially available from the same company refers to a device that can deflect or focus a light beam obliquely in one dimension, two dimensions, or as a focusing mechanism. A multi-channel electro-optic modulator can also be developed.

[0038] "DMD or SLM" in the present invention is an abbreviation for a device that can modulate the amplitude, intensity, phase, or polarization of a generally uniform coherent or incoherent beam, in the sense of a variable distribution of the described physical parameters, including but not limited to, for example, acoustic, magnetic, or electro-optic devices.

[0039] "Singular optics" including "optical vortices" as the simplest example is a new field of optics with logical and practical applications today. A detailed explanation can be found in (Nye, 1974#37; Soskin, 2001#38) Nye, et al., and both references are incorporated herein by reference.

[0040] A "wavefront shaper" is a device that can dynamically modify the light distribution. Wavefront shaping in microscopy mainly uses a spatial light modulator (SLM) placed at the pupil of the optical system and has been applied to control multiple scattered lights within biological tissues, as described in (Park, 2018#41) or (Ritsch-Marte, 2009#39). In this document, the same technical tool can be used to simultaneously form several light spots or more complex patterns.

[0041] We refer to the singular point distribution with a radiation symmetry as a "donut", and the position of zero intensity of these distributions as the donut zero, or as the zero or center of the donut in the text of (Balzarotti, 2017#4) cited in the present invention.

[0042] "Inelastic optical interaction" refers to the interaction between incident light and an object that generates photons with different wavelengths. Inelastic optical interactions include, but are not limited to, fluorescence, multi-photon interactions, and Raman scattering. [[ID=IO]]

[0043] The "locus of singular point distributions" is an ensemble of Cartesian positions where the intensity of the singular point distribution is zero. The locus of singular point distributions has a "nominal parameter" which is an appropriate parameter, and is placed at the "nominal position" which is the correct position, defining a group of basic shapes that do not emit (or reflect or scatter) light. Here, we introduce a new concept and express it as "the singular point light distribution embeds a geometric shape".

[0044] "Conical refraction" is an optical phenomenon predicted by Hamilton (Hamilton 1831) and experimentally confirmed by Lloyd two months later (Lloyd 1883). Both of the above references are incorporated herein by reference. Conical refraction is an explanation of the propagation of light rays in the direction of the optical axis of a biaxial crystal. Hamilton predicted that light would emerge in the shape of a hollow cone of light rays. The discovery of conical refraction was an epoch-making event in the history of science and played a role in the demonstration of electromagnetic waves.

[0045] However, as early as 1898, the inconsistencies between Hamilton's theory and Lloyd's preliminary experiments, as well as more accurate measurements and observations, were pointed out by Poggendorff. These inexplicable results have puzzled scientists for over 150 years and have hindered the use of this powerful effect in practical systems.

[0046] A fully logical analysis has been provided by Sir Michael Berry in Berry (Berry2004), which is incorporated herein by reference. Berry also changed the name of this physical effect from "conical refraction" used by Sir Hamilton to "conical diffraction". Conical diffraction is used in the present invention.

[0047] Berry's paper and the availability of synthetic biaxial crystals of high quality and at a reasonable price have paved the way for the use of conical diffraction as one of the most powerful tools in the optical technology toolbox.

[0048] The inventor is one of the leading scholars who understand the practical effects of this effect. The inventor introduced the thin crystal concept as "the beauty and elegance of the Poggendorff rings and conical diffraction that you (Sir Michael Berry) developed for a monotonic but efficiently controllable beam shaping unit".

[0049] Prior art systems based on conical diffraction for super-resolution microscopy are described in (Caron, 2014#33; Sirat, 2016#36), which are incorporated herein by reference.

[0050] In this specification, the term "energy law" is defined as follows. Assuming that an object is modeled as a geometric shape of a mathematical abstraction, the "energy law" is a parametric relationship between energies that are functions of shape parameters and position. This law generates a relationship that quantifies the energy dependence in the parametric space. The energy law may include the energy distribution radiated by a luminous object having the same shape as the geometric shape.

[0051] According to the present invention, herein, it is assumed that, as described by Sirat in (Sirat2016), the optical singularity distribution can be controlled by external means to switch from one type of distribution to another type of distribution, and from a distribution of a predetermined group to a distribution of another group, and to modify the parameters of the optical singularity distribution. There are also other solutions that do not require controlling the optical singularity distribution, and although they are actually part of the present invention, in the opinion of the inventors, they can be much more cumbersome.

[0052] "Control means" refers to a set of control hardware that can modify the input, and a "control algorithm" that can predict the next step of the input value necessary to quantify the "law of energy" so as to accurately obtain the parameters directly or by successive approximations. The "inverse law of energy" is a method that can obtain optimal or non-optimal position and / or shape parameters from the measured values of the singularity distribution or a set of singularity distributions. It is embedded in the control algorithm. Also, it is selected to optimize the functional parameters of the system, the number of necessary processes, the total energy or power hitting the living body, the measurement speed, and any combination of the above or any other functional parameters of the system.

[0053] Even in the simplest case of a single point, the law of energy depends on the three Cartesian positions of the point. Some of the irradiation selections described later can, depending on the region, make it possible to lose the correlation of the dependence of the inverse law of energy from two of the Cartesian positions, greatly simplifying the information collection and accuracy improvement.

[0054] In addition, many inverse laws of energy are quadratic equations and lose information regarding the sign of the position. One solution is proposed below.

[0055] Finally, the inverse law of energy is based on hypotheses such as the object being a point, a line, or a rod. Additional measurements, redundancy, and redundant determination can be used as verification of the hypotheses.

[0056] For the purposes of this specification, assume that a separate mechanism is used to collect the nominal positions of objects. Within the scope of the present invention, this mechanism can use any microscopy method that is a wide-field confocal or super-resolution technique, or any localization method such as PALM, STED, STORM, or prior external knowledge.

[0057] In the context of the singularity distribution, a "value close to zero" is used to quantitatively describe a projection intensity that is much smaller than the maximum intensity achievable on the projected light, or an energy that is much smaller than the energy emitted when the maximum amount of projected light hits this point. When comparing between the projection intensity and the maximum intensity of the distribution, or between the emitted energy and the energy emitted when irradiated with the maximum intensity, the numerical value of the intensity or energy close to zero is one-ninth. Assuming Poisson noise, an energy close to zero is significantly smaller than the maximum energy and is worthy of mention as having a noise value smaller than one-third. Similarly, a geometric parameter value of a shape close to zero has a value smaller than one-third of the entire range of the parameter.

[0058] "Conical diffraction microscopy" or "CODIM" refers to the conical diffraction microscopy described in (Sirat, 2016#11) and (Caron, 2014#33).

[0059] (Prior art: Microscopy) Refer to FIG. 1 showing an example of a microscopy 100 in the biological field.

[0060] Microscopy includes irradiating a biological sample 11 with a light source (not shown) using a microscope 10 and a time-dependent measurement of the light emitted from the sample using either visual observation or a detection module 12.

[0061] Biological samples include one or more different biological entities 13 and 14 arranged at different positions. Examples of such objects include, in particular, cells, viruses, proteins, and DNA fragments.

[0062] Fluorescence microscopy Fluorescence microscopy is a variation of microscopy and, in many biological applications, supersedes other microscopy techniques. A fluorescence microscope is an optical microscope that uses the fluorescence phenomenon, instead of or in addition to other methods such as reflection and absorption, to investigate the properties of organic or inorganic substances.

[0063] In fluorescence microscopy, a sample is irradiated with light of a certain wavelength or specific multiple wavelengths absorbed by a fluorescent substance, thereby promoting the emission of light of various higher wavelengths.

[0064] The illumination light is separated from the fluorescent emission, which is mostly at a higher wavelength, using a spectral emission filter, significantly reducing the background of the acquired image.

[0065] Fluorescence microscopy is the most common in this type of method, but there are also many other microscopy methods that use inelastic interactions, i.e., samples that emit light at a different wavelength from the light source or entities fixed to the sample.

[0066] In the present invention, for ease of reading and brevity, fluorescence microscopy is mentioned, but other inelastic interactions including, but not limited to, Raman, two- or multi-photon microscopy are also considered part of the present invention.

[0067] Referring again to FIG. 1, the fluorescence microscope will now be described. In fluorescence microscopy, small point sources 15-18 that are fluorescent substances are fixed at specific positions of predetermined living bodies 13 and 14 based on the physical phenomenon of single-photon fluorescence. Instead of observing the light emitted by the living bodies 13 and 14, the light emitted by the fluorescent substances is observed.

[0068] Fluorescent substances have become an important tool for visualizing living bodies. Biological information including activity and details less than 200-250 nm of the diffraction limit is systematically examined and measured using fluorescence microscopy. This resolution limit is obtained from the Rayleigh criterion and, in the best case, reaches 200-250 nm in specially designed systems.

[0069] In fluorescence microscopy, it should be noted that the information collected and acquired is a map of the fluorescent substance, not a direct image of the living body, as pointed out by several authors including Sirat-2016. The relationship between the measured values and the object relies on hypotheses and is reliable in most cases.

[0070] The main implementations of fluorescence microscopy are often confocal microscopes, or spinning disk microscopes, and wide-field imaging microscopes, which are used in a scanning configuration as described in detail in the same literature.

[0071] Next, refer to FIG. 2, which is a schematic diagram of a conventional confocal fluorescence microscope 200. The confocal fluorescence microscope in FIG. 2 is an optical instrument. Its main hardware components are shown in FIG. 2. These components include a light source 20, an optical mechanical frame (not shown), a cube filter 21, a microscope objective lens 22, a detector assembly 23, and a processing unit (not shown).

[0072] The light source 20 may be an arc lamp or a laser and generates the light energy required for fluorescence. The optical mechanical frame (not shown) is a support for all optical components and auxiliary optical systems and includes an alignment function. The frame also includes optical elements (not shown) that can shape the beam to enable a minimum-sized focus by the microscope objective lens. The frame further includes a spatial or angular scanning mechanism (not shown) for changing the position of the point source with respect to the object being measured in confocal scanning fluorescence.

[0073] Alternatively, the scanning mechanism can · For example, mechanically translate the object by using a translation stage, · Optically scan the beam over the object by using, for example, a set of galvanometer mirrors or acousto-optic translation means, or · Use any combination of these mechanical or optical translation means.

[0074] In confocal scanning fluorescence, information is collected point by point using a scanning mechanism. The scanning mechanism can further include a rotating disk having a plurality of pinholes that enable simultaneous projection of a plurality of points in the case of a rotating disk type confocal fluorescence. In a confocal fluorescence rotating disk, a set of points corresponding to the pinholes is acquired at any given time, and rotation of the disk enables scanning of the entire surface of the sample with respect to a given longitudinal position.

[0075] The cube filter 21 carries various optical signals and prevents contamination of the fluorescence signal by radiation. The cube consists of filters such as an excitation filter 210, a dichroic mirror 211, and an emission filter 212. The filters and the dichroic mirror are selected according to the excitation wavelength and emission spectral characteristics of the fluorescent substance.

[0076] The microscope objective lens 22 focuses the light generated by the light source at the focal plane of the lens 24, and the optimal light distribution of the small-diameter light distribution pattern consists of an airy disk. The microscope objective lens 22 also collects the fluorescence emitted by the fluorescent substance.

[0077] In the case of confocal scanning fluorescence, the system can be descanned, that is, the return light passes through the scanning mechanism and can compensate for translational motion due to scanning.

[0078] At the imaging surface of the detector lens 25 and the detector 26, an enlarged image of the focal plane of the lens 24 is generated.

[0079] The confocal hole 27 is logically located at the imaging surface of the detector 26. In most practical systems, the confocal hole 27 is located at an intermediate imaging surface (not shown) and reimaged onto the imaging surface of the detector 26.

[0080] The detector assembly 23 detects the fluorescence intensity over the entire exposure dose and converts it into a digital signal. In the simplest implementation, in the case of a confocal scanning microscope, the detector assembly comprises a single-element detector such as a photomultiplier tube PMT or a single-photon avalanche diode SPAD. In the case of a confocal microscope using a rotating disk, the detector assembly consists of a matrix of detector elements such as a charge-coupled device CCD, an electron-multiplying CCD EMCCD, a complementary metal-oxide-semiconductor CMOS, or a matrix of SPADs.

[0081] All components from the light source to the dichroic filter are mounted on the illumination path 201. The detection channel 202 represents all components mounted from the dichroic filter to the detector assembly.

[0082] Fluorescence microscopes are available from several manufacturers such as Zeiss, Leica, Nikon, and Olympus. The fluorescence microscope can be either a standard microscope suitable for fluorescence or a microscope specifically optimized for fluorescence. Recent microscopes are multifunctional devices that can operate in many different modes, including but not limited to fluorescence modes that use the same platform and most of the optomechanical components. Most fluorescence microscopes have been developed as open platforms that can implement several additional features with minimal changes. Other fluorescence microscopes are dedicated devices adapted for special tasks such as medical diagnosis or pharmaceuticals.

[0083] (Prior art: Super resolution) For a long time until the emergence of the super-resolution techniques described below, optical techniques including fluorescence microscopy were widely considered unable to visualize details smaller than the Rayleigh criterion, which is about 200 - 250 nm for visible light.

[0084] However, other basic biological activities also occur at scales less than 200 nm within biological samples. At this level of spatial resolution, important phenomena such as intracellular-scale biological processes, cell signaling, protein folding and unfolding, and DNA and RNA changes can be observed. For example, the measurement of this intracellular information opens up new avenues for understanding biological activities and advances the understanding and monitoring of investigations and medical diagnoses.

[0085] "Super-resolution microscopy is a term that encompasses several techniques that enable the acquisition of images in optical microscopy with a resolution higher than that imposed by the diffraction limit" (Contributors 2019). Wikipedia, for example, accessed on November 30, 2020, at https: / / en.wikipedia.org / wiki / Super-resolution_microscopy, uses the definitions found for STED, photoactivated localization microscopy, PALM, STORM, and many other methods. Additional super-resolution methods known to those skilled in the art are also included among the super-resolutions.

[0086] Conical diffraction microscopy is a super-resolution method developed by the inventors (Sirat 2016), hereinafter referred to as "Sirat 2016".

[0087] Some microscopy methods use a reconstruction process. That is, the direct data physically acquired by the detection system is not the final result, and an algorithmic process is required to obtain the final image.

[0088] The positive constraint, i.e., the physical fact that the light intensity is essentially positive, adds supplementary constraints to the mathematical solution for the optical system in which the algorithm is used to calculate the final object (reconstruction). The positive constraint can be applied to the general case, restricting the possible solutions and somewhat improving the quality of the results. However, in a specific case described later and named by the inventor as Abbe's loophole, this constraint makes it possible to overcome Abbe's resolution limit, which is the case in the present invention.

[0089] (Imaging, localization, and tracking...) In the present invention, in many cases, although they can be evaluated on the same device, imaging and localization, which are two different functions, are separately referred to.

[0090] During imaging, we observe the object without suggesting any additional information about the object in advance. Localization assumes that the object is a point and can be parameterized by a few descriptors. "Tracking" is a short term for dynamic localization or localization as a function of time, and depending on the context, either localization or tracking is used.

[0091] During imaging, the observed object is different from the original object and can be described in terms of degrees of freedom using a finite number of degrees of freedom restricted by the optical system (Lukosz 1966). The observed object is considered a filtered version of the original object and is restricted by diffraction in the general case, even if some super-resolution schemes can be applied.

[0092] In localization, the object is known or assumed to be a single point. It is a problem of parameter search, quantified by the Cramer-Rao criterion and not clearly restricted by the diffraction limit but by the signal-to-noise ratio.

[0093] Between the two extremes of a complete lack of prior information and the parametric description of an object, there are many cases, some of which contain some partial information. As an example, the assumption that a scene consists of sparse objects can, in some cases, enable additional information about the objects that can be converted into additional information or resolution. In (Sirat2017), regarding the Abbe loophole technique described below, the inventors extended the problem of locating a single emitter to the case of a simple geometric object that obtains both a position and a shape descriptor.

[0094] (... and metrology) "Metrology" refers to a third approach in addition to imaging and tracking, in which, as described in (Sirat2017) incorporated herein in its entirety, in addition to the position of a single point or a small set of single points, a descriptive geometric parameter set is obtained that describes the observed entity as a simplified geometric and / or temporal object. The inventors consider metrology to have a practical impact similar to imaging and tracking and to be a new and additional beneficial approach of the present invention.

[0095] (Alternative routes to localization) Worldwide, there are the following two strategies that can be applied jointly for either or both lateral or axial measurements to determine the position of an object assumed to be a point source of radiation of infinitesimal size (hereinafter abbreviated as emitter) beyond the diffraction limit. · Projection strategy: Project a light distribution of infinitesimal size and record the regression energy · Emission strategy: Analyze the regression light under the assumption of infinitesimal size

[0096] The STED method described in https: / / en.wikipedia.org / wiki / STED microscopy accessed on November 30, 2020, and the references cited therein is a typical example of a localization projection strategy. Through careful non-linear manipulation, the projection light is focused to a size smaller than the diffraction spot logically without limit in the case of infinite energy projection. Both two-dimensional and three-dimensional solutions are described and implemented in actual working systems.

[0097] The localization techniques described in https: / / en.wikipedia.wiki / Super-resolution_microscopy#Localization_microscopy_SPDM accessed on November 30, 2020, including techniques called PALM and STORM, are typical examples of a localization emission strategy. Assuming that the emitter is a single point, in the case of two-dimensional localization, if the light distribution generated on the detector is a single spot, the centroid of the spot can be obtained through an appropriate algorithm. The centroid is a measure of the position of the emitter. Assuming infinite energy, the position can be obtained with infinite precision.

[0098] For example, more advanced techniques that use appropriate optical means, mainly targeting three-dimensional localization, shape, and emitted light, such as (Pavani, Thompson et.al.2009) or (Fallet, Dubois et.al.2015), generate a more complex and accurate relationship between the localization value of the shape of the detected light and the parameters. In the present invention, these techniques are referred to as three-dimensional shaping of the emitted light. These techniques can measure the axial position of the emitter, but require a minimum number of photons to do so.

[0099] In the present invention, a method in which light projected by an emitter is shaped to form a predetermined shape that depends on the axial position, and the shape is obtained from the axial position, is referred to as a "3D projection light shaping method". The concept of 3D STED is an example of such a method in CODIM, and (Fallet, 2015#15; Fallet, 2016#35; Sirat, 2016#ll; Sirat, 2017#8; Sirat, 2017#12; Sirat, 2017#31; Sirat, 2016#36; Sirat, 2017#34) describes several methods of shaping the projected light to make it dependent on the axial direction.

[0100] In the present invention, a method in which light emitted by an emitter is shaped by an additional optical module that forms a predetermined shape depending on the axial position, and the shape is obtained from the axial position, is referred to as a "3D emission shaping method". Some considerations of these techniques can also be found in (Martens, Jabermoradi et.al.2020).

Brief Description of the Drawings

[0101] The above features of the present invention will be more easily understood by referring to the following detailed description while referring to the accompanying drawings below.

[0102]

Figure 1

Figure 2

[0103]

Figure 3

[0104] FIG. 4 shows four alternatives for generating a single point from the incident light rays, assuming, although not essential, that the beam is parallel at the entrance of the device.

Figure 4a

Figure 4b

Figure 4c

Figure 4d

Figure 4e

[0105] Figure 5 shows a simplified algorithm and process for obtaining the two-dimensional or three-dimensional position of an emitter.

Figure 5a

Figure 5b

Figure 5c

Figure 5d

Figure 5e

Figure 5f

Figure 5g

Figure 5h

SUMMARY OF THE INVENTION

[0106] In one embodiment, the present invention provides a method for determining the three-dimensional position of a luminescent object within a sample. In this embodiment, the method comprises projecting a first set of light radiation distributions onto a portion of the sample containing the luminescent object along a first projection axis, the light radiation distributions having zero intensity at the respective common centers of the set of distributions, measuring the intensity of the emitted light, and moving the set of distributions to a first point on the object where the light radiation is minimal and which characterizes two of the dimensions of the position within a first coordinate system by using the Abbe loophole technique to characterize two of the dimensions of the position within the first coordinate system; and using an optical module to shape the light emitted from the object so that the resulting shape depends on the axial position of the object before detecting the light, recording the shape and the overall intensity, and determining the axial position using the measured values of the resulting shape; and comprises.

[0107] In related embodiments, the present invention projects a second set of distributions of light radiation onto an object along a second projection axis, where the distributions of light radiation have zero intensity at each of the three-dimensional positions of the set of distributions determined according to claim 1, and the set of distributions is moved to a second point on the object where the light radiation is minimal and characterizes all three dimensions of the position within a second coordinate system by using Abbe's loophole technique, and further includes determining the position of the light-emitting object with even higher accuracy. Optionally, the method further includes obtaining an estimate of the position of the two-dimensional or three-dimensional position of the light-emitting object by a standard positioning or imaging method before characterizing two of the dimensions of the position within the first coordinate system. In additional related embodiments, the present invention further includes measuring the intensity of light radiated along a selected Cartesian axis within a segmented region including a point having a minimum intensity during the process of projecting a set of distributions of light radiation selected from the group consisting of a first set, a second set of distributions, and combinations thereof onto the light-emitting object, where the intensity along the Cartesian axis provides a measure of the position of the light-emitting object along the Cartesian axis and loses the correlation from the position of the light-emitting object along the other axes. In additional related embodiments, the set of distributions includes at least two distributions. In additional related embodiments, to provide verification of the determined position, the set of distributions of light radiation having zero intensity at each common center is selected such that the measured values of the set from intensity and shape are selected from the group consisting of redundant and overdetermined.

[0108] In another embodiment, the present invention provides a method for determining the positions of a set of separate light-emitting objects in two or three dimensions. In this embodiment, the method uses a scanning device to project a distribution of light radiation of a set having zero intensity at each common center in two or three dimensions along a projection axis onto each of the set of objects, the center being different for each object of the set of objects, measuring the intensity of the light, and using the scanning device to move the center of the set of distributions to a point on the object of the set of objects where the light radiation is minimum for each object of the set of objects, thereby characterizing the respective two- or three-dimensional positions of the separate light-emitting objects. Optionally, the scanning device used to move the position is an SLM or DMD positioned on a surface coupled to the sample, and the scanning device, in the case of an SLM or DMD, only the pixels coupled to the determined position are in the on mode and the collected light is a direct path, or, in the case of a DMD, only the pixels coupled to the position of the emitter are in the off mode and the collected light is an indirect path. In a related embodiment, the size of the spot corresponding to the diffraction limit is 3 to 11 times the pixel size. In an additional related embodiment, an algorithm is applied to determine the appropriate combination of on and off of the pixels to position the centroid of the spot on the sample with a resolution better than that of a single overall pixel of the scanning device projected onto the sample surface. In a further additional related embodiment, a first optical module is used to generate a light distribution including the position of the emitter as part of the illumination region. Optionally, the scanning module is selected from the group consisting of a multi-channel acoustic or electro-optical deflector, a wavefront shaper, and combinations thereof.

Embodiments for Carrying Out the Invention

[0109] (Abbe's loophole) To explain a group of projection techniques, we refer to the Abbe loophole, a term coined by the inventors in past patent applications and publications. However, emitter localization is derived from the absence of light generated by the projection of an optical pattern, including the approximate center zero of the energy on the emitter, which is assumed to be of infinitesimal size.

[0110] The basis of this scheme is that the Abbe's law excludes the direct measurement of spatial frequency components beyond the diffraction limit. However, the positive constraint generates a fundamental additional relationship between all frequency components. In the specific case where all frequency components below the diffraction limit and the spatial DC component are zero (black image), this constraint requires that all frequency components beyond the diffraction limit are also zero. In this loophole, it is known that both frequency components below and beyond the diffraction limit are zero, and this indirect measurement of all frequency components is a practical violation of the diffraction limit.

[0111] In the present invention, the names "White system" for explaining a localization system based on the above-described radiation strategy, and "Black system" for explaining a localization system based on the Abbe loophole are used.

[0112] A group of techniques based on the Abbe loophole uses a vortex or another similar distribution having zero intensity exactly at the center or approximately at the center as the light distribution projected onto the sample. This technique was proposed in the divisional application (Sirat2017) of (Sirat2016) with a priority date of October 2010, using the name "black fluorophore". This technique is also described in several papers such as (Hell2016) with a priority date of November 2011, and later (Balzarotti, Eilers et.al.2017, Gwosch, Pape et.al.2019), and is publicly available and commercially available under the name MINFLUX.

[0113] The extension to a three-dimensional case with an appropriate three-dimensional distribution can be found in the cited references.

[0114] Another complementary technique that the inventors named "dark tracking" in the published materials uses multiple light distributions that are sequentially projected onto the sample, which have similar radial functions and a common central zero but different azimuthal dependencies (Sirat2016, Sirat2017). This technique was proposed in a divisional patent (Sirat2017) with a priority date of October 2010 (Sirat2016).

[0115] The difference between these two techniques lies in that in dark tracking, in addition to the radial direction information, which is the only information available in the "black fluorophore" / MINFLUX technique, the azimuth angle is obtained with the same number of photons. This additional information greatly simplifies the function of reaching the position where the zero - in this case the sum of the distribution - is placed at the emitter.

[0116] A derivative technique that the inventors called "metrology" in (Sirat2017) generalizes the "dark tracking" technique to emitters consisting of simple geometric objects as the above - mentioned point objects and line objects, rather than at an infinitely small point, in order to obtain both the position of the emitter shape and the descriptors of the shape.

[0117] In the present invention, as will be described below, a technique that enables simultaneous or almost simultaneous execution of dark tracking on several identified targets is called "dark addressing".

[0118] These techniques enable theoretically unlimited additional resolution by combining the following. · Abbe's loophole principle · Near - complete absence of noise according to Poisson's law, which is called Abbe's loophole in a specific case · Exception at the Cramer - Rao limit close to zero intensity

[0119] To obtain the best performance from these techniques, it is expected that there are no spurious photons in order to meet the theoretical conditions and reach the optimal infinite resolution. The condition of the absence or at least minimization of spurious photons is more easily satisfied by using inelastic light interactions such as fluorescence, multi-photon interactions, and Raman scattering. In these methods, the incident beam can be completely filtered by spectral means with (almost) no residual single photons.

[0120] Even if background photons limit the highest achievable resolution and form part of the present invention, all other optical techniques can also utilize these principles.

[0121] More precisely, the technique described above as Abbe's loophole technique can be structured as including the following three steps: a detection step, an intermediate step (positioning to a donut zero or a dark tracking bond zero close to a few nanometers or less than 20 nm to the emitter, this step is called the "demon step"), and a precision step.

[0122] The oldest description of the step we call the demon step was found in Sirat2010. However, we refer to the subsequent, explanatory and very easy-to-understand revised version called Balzarotti2017 (Balzarotti, Eilers et.al.2017) (citing the full text) to explain the technique we call Abbe's loophole and its limitations in the present invention. Next, perform a thought experiment that attempts to determine the trajectory of a molecule diffusing in space. Instead of using uniform wide-field excitation and a camera, we excite a reasonably bright focal donut that can quickly move across the focal plane. If we or the demon somehow guide the zero of the donut-shaped excitation beam to exactly target the molecule and always overlap with the molecule in space, the donut target device will be able to completely map the molecule without inducing single emission. On the other hand, for example, single emission (due to the smallest misplacement) would be sufficient to know that the molecule is not at the location of the donut zero. Unfortunately, we cannot know the position of the molecule in advance and place the donut at its coordinates with a single shot. Therefore, complete positioning without radiation remains the privilege of the demon. However, as this thought experiment suggests, multiple shots exploring the position of the molecule at zero intensity should reduce the radiation required for positioning. This is because, in our view, fluorescence radiation is the price paid for not knowing the position, and the closer it gets to zero during the exploration process, the smaller the price. In fact, apart from confirming the presence of the molecule, radiation is very useful for transmitting information about the distance to zero being explored.

[0123] In the present application, a method using a combination or sequence of donuts or distributions in two or three dimensions is referred to as the Abbe loophole technique or the black technique. This technique is unique in the fact that an intensity of zero or close to zero existing at the center of the combined distribution in one, two, or three dimensions is projected onto a sample containing an emitter, and the donut zero moves towards the emitter until it reaches a position where the emitter does not emit light.

[0124] The two-dimensional Abbe loophole technique can use a combination of donuts or distributions as the conical diffraction distribution shown in FIG. 3, which is projected onto the sample sequentially or simultaneously. The three-dimensional Abbe loophole technique adds a distribution that is zero at the axial position of the emitter.

[0125] Some embodiments of the Abbe loophole technique also include means for controlling the system and modifying the relative position between the zero and the emitter.

[0126] In summary, we can reach a nearly infinite theoretical resolution with a minimum photon cost, where the distance between the molecule and the donut zero is on the order of a few nm. However, at the starting point, the zero can be separated from the molecule by a distance of more than about 100 nm. In many cases, an intermediate step called the "demon process," which is a subsequent set of measurements, is in the low photon regime and must be carefully designed.

[0127] As a starting point, the detection of the presence of the emitter and its uniqueness are used in many cases and relatively standard techniques are employed. This enables reaching a resolution limit accuracy (200 - 250 nm), or an accuracy of 90 - 100 nm as exemplified by a super-resolution system developed by the inventors through some additional development (Sirat2016). The cost of its photons and system complexity can be reduced by reducing the accuracy requirements, but there is clearly an additional burden on the second step, the intermediate daemon step.

[0128] The final step where the donut zero is positioned at or very near the molecule and reaches an accuracy of several nanometers is surprisingly conceptually quite simple. This step is (theoretically) a step where a single photon holds yes / no information.

[0129] This claim about the simplicity inherent in the final step is debatable and is only correct at the conceptual level. Still, the system requires a carefully designed optical system, single-photon detection, motion control at the nanometer level, and low-noise electronics. These technical specifications need to avoid the generation of false photons, electrons, or digital counts and control the movement. Nevertheless, all these most extreme specifications are very challenging, yet all these requirements are within the scope of existing well-established systems and technologies.

[0130] The actually difficult step both conceptually and practically is the intermediate (daemon) step that bridges the gap between a starting point of 100 nm accuracy and the several nanometers of the final step, at the expense of reasonable photon and system complexity, through a definitive and reliable procedure.

[0131] Another notable point where Abbe's loophole technique is inherently superior to other techniques is that in many biological situations, the issue is the fact that, over a long period of time, the points follow a stationary or predictable path. Sudden, unexpected events change the dynamics at that point. These events can trigger significant events such as apoptosis or necrosis events in cells, for example.

[0132] The cost of waiting time for photons is theoretically zero in Abbe's loophole technique, and the waiting can potentially continue for a long time. The event triggers an immediately recognizable burst of photons, serving as a trigger for the detection of the event. All white techniques need to repeatedly check the position of the point at a considerable photon cost per query.

[0133] Another notable point where Abbe's loophole technique is inherently superior to other techniques is that assuming the system is well-designed and reaches a speed exceeding the standard speed of particle movement, using an expression that intentionally evokes air defense terms, after the system aims at the target, the photon cost of tracking the particle can be reduced by appropriate observation and control.

[0134] The main problem is to identify the molecule, which will become even more important in the following paragraphs. In fluorescence, as is well known to those skilled in the art, the two main recognition mechanisms of a specific fluorescent substance are multi-wavelength systems and emission / transmission wavelength specificity leading to lifetime characteristics, which can also be a tool for differentiating fluorescent substances.

[0135] When photons are available, it is possible to measure the lifetime of the incident photons. This is very important in both the daemon process and the precision process to recognize whether the incident photons are generated by the target being observed or by false fluorescence, ghost images, nearby objects, or any other parasitic light.

[0136] Finally, the presence of the target may need to be evaluated at any time, and the responsiveness of the target to light and the amount of light available for a given projection power are required by some, if not all, algorithms. In a conical diffraction-based implementation, the availability of a controllable amplitude Gaussian beam or a distribution similar to an Airy disk has a significant impact on the practicality of the actual system and forms part of the present invention.

[0137] (Description of Embodiments of the Invention: Hybrid Solution) The present specification presents a new method in which the measurement procedure is, in some cases, a separate dedicated procedure and architecture initialized by the imaging procedure.

[0138] The invention described hereinafter relates to accurately positioning the emitter with an accuracy beyond the diffraction limit of the light employed in the measurement and with a minimum light beam.

[0139] The present invention is particularly adapted to accurately localize features using inelastic light interactions including, but not limited to, fluorescence or Raman scattering, in which the emerging light can be separated from the incident light by simple means, but can also be applied to other microscopy modalities.

[0140] The method according to an embodiment of the present invention can advantageously provide measurements at a resolution higher than the diffraction limit imposed by optics and the minimum photon beam.

[0141] This method can be used instead of the Abbe loophole three-dimensional technique where the resolution required in three dimensions is in the range of 10 nanometers, or to simplify the "demon process" and reduce the photon cost.

[0142] We refer to this new method as the "hybrid method". As will be described later, this method utilizes position identification in two or three dimensions based on the analysis of synchrotron radiation during intermediate processes in order to measure the position of the emitter. The hybrid method can be used for two-dimensional position identification, but a preferred embodiment is to record the three-dimensional position of the emitter using the emitted photons.

[0143] In a preferred embodiment, this technique is called hybrid because it uses a dedicated method that combines a projection and a synchrotron radiation strategy for position identification, namely, a projection strategy for obtaining the two-dimensional lateral position and a synchrotron radiation strategy for obtaining the axial position through three-dimensional shaping.

[0144] Using three-dimensional beam shaping to obtain the synchrotron radiation three-dimensional shaping as a tool for obtaining the axial position with a reasonable accuracy of about 10 - 30 nm is an excellent solution that utilizes the physical existing data and information that can be easily obtained when appropriately designed. The synchrotron radiation three-dimensional shaping uses the photons generated by the two-dimensional donut used to obtain the lateral position using the projection strategy. These photons exist anyway because they are not close to the donut zero. The cost of the photons is significantly reduced, and the speed of reaching the donut zero increases.

[0145] In addition, three-dimensional information is the most costly to obtain in terms of photons and system complexity. Although this information actually exists, according to the basic principle, it requires a three-dimensional equivalent that is much more inefficient than the two-dimensional donut. Furthermore, an approach to reaching the three-dimensional position of the emitter by obtaining many different positions in three-dimensional space is also required.

[0146] In Abbe's loophole technology, the addition of three-dimensional requirements significantly complicates the system. The demon has to be very intelligent! Changes in energy can be due to movement in any of the three Cartesian directions or any combination thereof. Obtaining the direction to move the zero of a donut or equivalent set of distributions in space requires many measurements, leading to a large photon budget.

[0147] The object of the present invention is to avoid discouraging the demon when used as an intermediate step in Abbe's loophole technology. We simplify the task by individually obtaining axis information using various means. The acquisition of axis information is based on photons already generated during the process by the positioning projection strategy used for positioning in the lateral dimension. This individual information is obtained by using a positioning emission strategy as a complement to the positioning projection strategy based on Abbe's loophole used for positioning in the lateral dimension.

[0148] In conclusion, the present invention proposes a novel and never-before-heard solution by using a hybrid system consisting of the use of special and efficient information available in a way that has never been proposed before.

[0149] In another configuration, the present invention consists of three subsequent steps.

[0150] Detection step: In the detection step, the presence of the emitter, its uniqueness, and the tentative three-dimensional position of the emitter are detected. Many different techniques can be used for the detection step, including but not limited to confocal and wide-field microscopy. The position can also be known from other deductive or external knowledge. The choice of technique is partially defined by the low photon requirements.

[0151] Demon step: Obtaining the three-dimensional position of the emitter through the hybrid method described in the present invention

[0152] Precision engineering: Acquisition of the three-dimensional position of an emitter, which consists of obtaining the three-dimensional position with the highest possible accuracy of several nanometers using three-dimensional Abbe loophole technology as described in the present invention, including the zero shift of the distribution towards the particles

[0153] (Description of embodiments of the present invention: Multiple synchronization points) (Hell2016, Sirat2016, Balzarotti, Eilers et.al.2017) The above description presents the concept of measuring the position of a single point within one dimension, two dimensions, or three dimensions with high precision, high speed, and low photobleaching and / or phototoxicity, and potentially at several wavelengths. However, many diverse biological mechanisms rely on synchronized single events, and the ability to simultaneously (or almost simultaneously) acquire several targets that are optically separated only by the distance above a system ruler or several system rulers can have important implications in the fields of biology and medicine.

[0154] Needless to say, many biological functional events are complex and are the result of signal trafficking and the delicate balance between permissive and inhibitory signals, and the dynamics of these signals are of utmost importance.

[0155] In the present invention, a new paradigm of multi-target tracking scenarios is distinguished and defined in super-resolution microscopy, generally in microscopy, and generally in target super-resolution, such that the same scenario can be applied to white systems and, even if considered part of the present invention, adjusted for all black systems.

[0156] A multi-target tracking scenario is defined as measuring a limited number of small targets simultaneously or almost simultaneously as points, point objects, lines, line objects, or simple geometric substructures.

[0157] The additional complexity of this scenario is more than compensated for by the importance in biology of this particular case. Both the confocal and wide-field geometries are actually simpler and easier to implement compared to scenarios assigned and optimized for multiple synchronous points.

[0158] The inventors state again that such a measurement system has a major impact on the ability to measure, visualize, and quantify the root causes of functional biological activity, which can take the form of natural molecular events. This is extremely important and is worthy of developing dedicated optimized solutions.

[0159] The inventors also state that this scenario is well-suited and complements two concepts introduced in their own latest invention (Sirat, 2017#12) that incorporates the full text herein and introduced metrology and deep learning schemes. The metrological features introduced there are different information adapted to the goal of correlating natural events with functional information, and the deep learning capabilities are additional tools for extracting significant information.

[0160] Using aerial terms again, the multi-target tracking scenario simulates a coordinated attack by several aircraft and missiles with different characteristics, speeds, and kill rates, and the single-target case focuses on a single specific aircraft.

[0161] The solutions presented by (Hell 2016, Sirat 2016, Balzarotti, Eilers et.al. 2017) are formed based on confocal configurations, single points (and targets), and scanning systems. To replicate points, point-tracking techniques are cumbersome and require overly complex systems, but are not infeasible as described by (Gu, Li et.al. 2019).

[0162] Other solutions based on wide-field configurations such as SIMFLUX (Cnossen, Hinsdale et.al. 2019), ModLoc (Jouchet, Cabriel et.al. 2020), SIMPLE (Reymond, Ziegler et.al. 2019), and ROSE (Gu, Li et.al. 2019) lack the advantages of Abbe's loophole technology.

[0163] In other words, two existing simplified conceptual geometric shapes that explain the dimensions of light and the dynamics projected onto an object, namely, confocal and wide-field, are not suitable for the problems and challenges described herein of tracking and identifying multiple but a small number of individual targets. Due to the selection of unoptimized configurations, all prior art solutions require some compromise, either by sacrificing performance or by requiring enormous technical costs. Primarily, confocal solutions sacrifice simultaneity, while wide-field solutions project light everywhere and contaminate the signal.

[0164] [[ID=,7]] To illustrate a new geometric shape of light interaction, the basic point formation process of Figure 4a is taken up again. Assuming a uniform light beam

[40] , a simple solution to generate a single point

[50] is clearly to focus the beam using a lens or optical system

[43] and to move the point

[50] onto the object plane

[49] using mechanical means which are galvanometer mirrors

[41] and

[42] . Figure 4a is the basic configuration used in a confocal system. The point position can be controlled using a scanning mechanism such as a galvanometer mirror as shown in Figure 4a, or using an acousto-optic deflector as shown in Figure 4b.

[0165] In Figure 4b, a uniform light beam

[40] is deflected by an acousto-optic deflector

[44] and focused using a lens or optical system

[43] to generate a single point

[50] on the object plane

[49] .

[0166] Although there are similar transmission solutions, there is another solution in Figure 4c where a uniform light beam

[40] is projected onto a controllable DMD or SLM

[48] represented as a reflective DMD device. The DMD is positioned on the imaging surface of a lens or optical system

[43] . Assuming that all pixels are switched off, as shown in Figure 4c, except for a small area where a geometric range consisting of a single pixel [6] or several adjacent pixels

[47] is smaller than the diffraction limit, in Figure 4d, a single point

[50] appears on the object plane

[49] . The point position depends on the position of a single pixel in Figure 4c or the centroid of the area irradiated on the DMD or SLM in Figure 4d. In addition, in Figure 4d, in the case of a small area, the light generated, reflected, or transferred from the pixel is coherent, and although other cases with performance degradation can be considered, it is assumed that no additional transfer delay occurs between adjacent pixels.

[0167] The solution shown in Figure 4c exists, but usually has many drawbacks and is considered an insufficient alternative to the standard solutions as shown in Figures 4a and 4b. The reason this solution is rarely used is that in the case of a single point, the energy efficiency of such a solution is very low, so such a solution is almost impractical. In fact, all the light hitting the off pixels is simply lost. In numerical terms, assuming that the area scanned by the system is 10*10μm on the sample, for a diffraction limit of 250nm, the energy loss is approximately 1:1600, roughly speaking, without considering additional geometric losses.

[0168] In addition, such a solution enables the positioning of points at individual positions corresponding to the entire pixel when a single on-pixel is used, and thus is clearly not suitable for the Abbe loophole system that requires movement of points in nanometers. As will be described later, the use of a small area smaller than the diffraction size at the incident surface first improves the movement process because the pixel is designed to be smaller than the diffraction limit, and also improves the sub-pixel process by carefully handling the on / off of individual pixel characteristics.

[0169] A reasonable way to configure direct imaging is to adjust the pixel size to the diffraction spot, as shown in Figure 4c. This direct imaging scheme is described as a DMD or SLM, for example, in (Gauthier, Lenton et.al. 2016).

[0170] We introduce a new scheme that uses direct imaging of a DMD or SLM in Figure 4d. Configure the dimensions of the DMD so that the size of the diffraction-limited spot is much smaller than the diffraction size (typically represented by a ratio of 1:α, 1:5 to 1:11). m , y m Assuming that light is added coherently to form a spot centered on a specific pixel positioned at x, all pixels within a region of size α or sub-regions contained therein can be switched on. More complex combinations of on-pixels within the above-mentioned region can generate an Airy-like spot positioned at a predetermined portion of the pixel, enabling addressing at almost any position on the sample. Deviating further from the sub-region to increase the degree of freedom, the shape of the spot is simply slightly modified by controllable and quantifiable differences that can be considered by the system algorithm.

[0171] Such a configuration provides a dual purpose of significantly reducing light loss on the one hand and being able to position points more accurately than the pixel size at the expense of the field of view.

[0172] The configuration shown in Figure 4d requires a high tolerance for the positioning of adjacent mirrors in the DMD or the phase of adjacent pixels in the SLM to ensure that different beamlets hitting adjacent pixels are added coherently to form a single point. However, existing actual technologies can meet such requirements.

[0173] In this scheme, the spatial bandwidth is traded for direct positioning of spots at almost any position using a simple mathematical approach in the sense of the size of the region that can be addressed by the system.

[0174] In the specific case of biology, technology development enables such a trade-off. Diffraction limit of 200 μm, Availability of a 1920*1024 DMD or SLM, Assuming α = 7, The measurement area extends over an area of 55 * 30 μm. This value is typically adapted to single cells sized 25 - 40 μm.

[0175] Also, in such a configuration, several independent points can be irradiated simultaneously and controlled in parallel.

[0176] This solution can be improved as shown in Figure 4e, on the one hand to mitigate energy loss and on the other hand to move points at a fraction of the element size.

[0177] Assume that a low-resolution system is based on the galvanometer scanner of Figure 4a, or preferably the acousto-optic deflector of Figure 4b, a multi-channel acousto- or electro-optic deflector placed in the pupil of the system as shown in Figure 4e, or a waveform shaping DMD or SLM, generating a point larger than the diffraction limit on the DMD or SLM

[48] . Assume that all pixels are switched off except for a small region consisting of a single pixel in a configuration similar to Figure 4c not shown, or a number of adjacent pixels

[51] in a configuration similar to Figures 4d and 4e.

[0178] The additional intermediate level simply reduces the energy loss to approximately the ratio between the larger point generated by the first-level point positioning and the diffraction-limited size on the DMD or SLM. Additionally, as shown in Figures 4c and 4d, several points can be irradiated simultaneously.

[0179] For the sake of simplicity, in this paragraph, the geometric shape of the system is described assuming that the scanning device is positioned at the entrance of the system. This configuration seems reasonable assuming that the beam shaper follows appropriate optical constraints, mainly telecentricity. The beam shaper can also be placed in front of the scanning device that directly manipulates the distribution. In practice, this solution is simpler, but any solution is appropriate, and the engineering aspects will determine the choice between the two configurations.

[0180] In another embodiment, multiple synchronized point light distributions can be created using a waveform shaping DMD or SLM positioned at the pupil of the system in a configuration similar to that used, for example, in (Ritsch-Marte, 2009#39) or similar publications.

[0181] In another embodiment, multiple synchronized point light distributions can be generated by a multi-channel acousto-optic or electro-optic deflector described as a product of G&H at https: / / gandh.com / nproduct-categpories / multi-channel-modulators-aomc / accessed on December 2, 2020, for obtaining substantially simultaneous points, or by an ultra-fast acousto-optic or electro-optic deflector.

[0182] (Description of Embodiments of the Invention: Simplified Cartesian Algorithm and Control System) In FIG. 5, a group of simplified Cartesian algorithms and control systems for dark tracking and dark addressing are presented.

[0183] Assuming the horizontal pattern shown in FIG. 5a, the position x of the emitter along the y-axis central reference line, specifically indicated by the star on the drawing, is energy-dependent. The theoretical function is parabolic-dependent in the conical diffraction pattern, but the actual dependence can be calibrated using an appropriate procedure to account for small differences. Small dependencies in the possible orthogonal directions can also be considered by that procedure.

[0184] Assuming the vertical pattern shown in Fig. 5b, the position y of the emitter with respect to the central reference line along the x-axis, specifically indicated by the star on the drawing, depends on the energy. The theoretical function is parabolic-dependent in the confocal diffraction pattern, but the actual dependence can be calibrated using an appropriate procedure to account for small differences. A small dependence of the energy in the possible orthogonal direction can also be considered by that procedure.

[0185] Assuming the axial dependence pattern described in (Sirat, 2016#11) shown in Fig. 5c, the energy depends on the position s, the distance between the emitter and the 3D central point of the distribution in three dimensions. Fig. 5c shows three cuts at different axial positions, namely, (1) the focus, (2) the axial position at 0.1 Rayleigh range, and (3) the axial position at 0.2 Rayleigh range, represented as Fig. 5c1, Fig. 5c2, and Fig. 5c3. At the focus, if the point comes to the focus AND at zero of the x-y coordinates, as described in (Sirat, 2016#11) and subsequent patents, the energy zero in this distribution will be obtained. The theoretical function is parabolic-dependent either in terms of a known combination of s or z and r (Fig. 5d) in the conical diffraction pattern, where r is the lateral projection distance between points (r = √x 2 +y 2 ). The calculation can use the theoretical function or can be calibrated using an appropriate procedure to account for small differences. The dependence of the energy in the orthogonal direction must also be considered by the procedure.

[0186] The same procedure can be carried out using all the axial dependence distributions described in the inventors' inventions that feature energy zero, including but not limited to (Sirat, 2016#11; Sirat, 2017#8; Sirat, 2017#12; Sirat, 2017#31; Sirat, 2016#36; Sirat, 2017#34), and other known axial dependence distributions known to those skilled in the art.

[0187] The preferred embodiment features a common zero of the lateral and axial dependent distributions that simplifies many algorithms, as shown in FIG. 5. Other embodiments, however, can be run under other conditions, such as tolerance regarding the commonality of the zeros of the distribution, penalty of the photon cost, or the need for additional shift of one distribution relative to other distribution(s).

[0188] The same FIG. 5 presents a second simplified Cartesian algorithm.

[0189] Assume a first horizontal pattern of FIG. 5e, which is an asymmetric version of the pattern shown in FIG. 5a, and a second horizontal pattern of FIG. 5f, which is an asymmetric version of the pattern shown in FIG. 5a and is a mirror image of the pattern of FIG. 5e. Both of these patterns can be generated using CODIM - conical diffraction microscopy with appropriate selection of input and output polarizations. The position x of the emitter along the central reference line on the y - axis, specifically indicated by the star, directly depends on the sum of the energies recovered by two measurements. The direction of the x - position can be measured by comparing the two patterns, and the sign - positive or negative - yields a measurement value that depends on the comparison of the two energies. The theoretical function is parabolic - dependent in the conical diffraction pattern, but the actual dependence can be calibrated using an appropriate procedure to account for small differences. Small dependencies in the orthogonal directions that may occur can also be accounted for by that procedure.

[0190] Assume two vertical patterns (FIGS. 5g and 5h) that behave similarly to the horizontal pattern with the necessary modifications.

[0191] Symmetric or asymmetric pattern FIGS. 5a - 5h (except FIG. 5d) can be generated at an angle β selected to optimize measurement or system parameters instead of the horizontal and vertical axes described above.

[0192] Sequences of these patterns or combinations thereof are projected at the same position of the emitter measured simultaneously or approximately simultaneously, or as a time-dependent sequence, or in combination with movements imposed by the operator or naturally generated, and the measurements obtained by reinforcing by simple mathematical procedures known to those skilled in the art provide simple deterministic lateral, axial, or three-dimensional measurement procedures. These measurements can also, in combination with additional information, provide accurate positioning.

[0193] For example, the measurement of symmetric patterns is simpler and, in some cases, more accurate if the position polarity is known from external or past information. To maximize accuracy and minimize the time and the number of photons required for the measurement, combinations of these measurements, as well as their direct and obvious derivatives, can be used (will be used).

[0194] In addition, a model verification index i mod can be calculated. This index ensures that the measurements are within a reasonable tolerance and fits the established model. This index is premised on the assumptions of the model.

[0195] As an example, when the model assumes that the object is a single light-emitting point, the four measurements of the second simplified algorithm described above have some predetermined relationship. Deviation from this constraint enables the detection of outliers, which is a major problem in many positioning techniques.

[0196] With the lowest level of information, the detection of this raw data is the most reliable way to avoid errors and mistakes.

[0197] Furthermore, the model verification index can be extended, as described for the above metrics, by, for example, performing additional measurements that recognize and quantify additional cases. The recognition process is a predetermined relationship based on logical, experimental, or functional grounds and uses deep learning techniques as an extension of (Sirat, 2017#12).

[0198] (Description of Embodiments of the Invention: Electronic Devices) In another embodiment, an electronic design optimized for the present invention, all CODIM systems, and all optical setups using a Pockels cell in a fixed polarization state is described. Assuming the use of single or dual Pockels cells to control the polarization state, one of the practical device limitations is the requirement to switch from one fast polarization state to another. Even if high-voltage amplifiers and DC-DC voltage sources are commercially available, for high-speed kilovolt voltages, high currents and expensive devices are required to switch from one state to another. An alternative solution of the present invention that utilizes the fact that the Pockels cell is electrically low-capacitance in the pF range is to charge relatively large capacitors at a reasonable current level in the nF and above range in advance and implement a few voltage values by connecting them when required for the Pockels cell. In an additional supplementary embodiment, leveraging the fact that the Pockels cell is a two-electrode device and the voltage that produces the optical effect is the difference between the voltages applied to the two electrodes, a much lower voltage amplifier or DC-DC converter is used to compensate for the large voltage drop or to switch between neighboring states such as those shown in FIGS. 5e and 5f or FIGS. 5g and 5h. Assuming that known means to those skilled in the art are used to monitor the large capacitance and / or the voltage on the Pockels cell, in another embodiment, a control system is added to refresh the voltage on the large capacitance as needed when other polarization states are applied to the Pockels.

[0199] The references listed below are incorporated herein by reference. (References) Balzarotti, F., Y. Eilers, K. C. Gwosch, A. H. Gynna, V. Westphal, F. D. Stefani, J. Elf, and S. W. Hell (2017), "Nanometer resolution imaging and tracking of fluorescent molecules with minimal photon fluxes", Science 355(6325): 606 - 612 Berry, M. (2004), "Conical diffraction asymptotics: fine structure of Poggendorff rings and axial spike", Journal of Optics A: Pure and Applied Optics 6(4): 289 Cnossen, J., T. Hinsdale, R. O. Thorsen, F. Schueder, R. Jungmann, C. S. Smith, B. Rieger, and S. Stallinga (2019), "Localization microscopy at doubled precision with patterned illumination", bioRxiv: 554337. contributors, S.-r.m.W. (2019). "Super-resolution microscopy. In Wikipedia" Fallet, C., M. Dubois, J.-Y. Tinevez, S. Oddos, J. Caron, R. Persson, S. L. Shorte and G. Y. Sirat (2015), "A new method to achieve tens of nm axial super-localization based on Conical diffraction PSF shaping. Single Molecule Spectroscopy and Superresolution Imaging VIII", International Society for Optics and Photonics. Gauthier, G., I. Lenton, N. M. Parry, M. Baker, M. Davis, H. Rubinsztein-Dunlop, and T. Neely (2016), "Direct imaging of a digital-micromirror device for configurable microscopic optical potentials", Optica 3(10):1136 - 1143 Gu, L., Y. Li, S. Zhang, Y. Xue, W. Li, D. Li, T. Xu, and W. Ji (2019), "Molecular resolution imaging by repetitive optical selective exposure", Nature Methods 16(11):1114 - 1118 Gwosch, K. C., J. K. Pape, F. Balzarotti, P. Hoess, J. Ellenberg, J. Ries, and S. W. Hell (2019), "MINFLUX nanoscopy delivers multicolor nanometer 3D-resolution in (living) cells", bioRxiv:734251 Hamilton, W. R. (1831), "Third supplement to an essay on the theory of systems of rays", The Transactions of the Royal Irish Academy. Hell, S. W. (2016), "Method and apparatus for tracking a particle, particularly a single molecule, in a sample", Google Patents. Jouchet, P., C. Cabriel, N. Bourg, M. Bardou, C. Poiis, E. Fort, and S. Leveque-Fort (2020), "In Depth 3D Single Molecule Localization Microscopy with Time Modulated Excitation", Biophysical Journal 118(3):149a Lloyd, H. (1883), "On the phenomena presented by light in its passage along the axes of biaxial crystals", Philos. Mag. 1, 112 - 120 and 207 - 210 Lukosz, W. (1966), "Optical systems with resolving powers exceeding the classical limit", JOSA 56(11):1463 - 1471. Martens, K. I, A. Jabermoradi, S. Yang and J. Hohlbein (2020), "Integrating engineered point spread functions into the phasor-based single-molecule localization microscopy framework", Methods. Pavani, S. R. P., M. A. Thompson, J. S. Biteen, S. J. Lord, N. Liu, R. J. Twieg, R. Piestun, and W. Moemer (2009), "Three-dimensional, single-molecule fluorescence imaging beyond the diffraction limit by using a double-helix point spread function", Proceedings of the National Academy of Sciences 106(9):2995 - 2999 Reymond, L., J. Ziegler, C. Knapp, F.-C. Wang, T. Huser, V. Ruprecht, and S. Wieser (2019), "SIMPLE: Structured illumination based point localization estimator with enhanced precision", Optics express 27(17):24578-24590 Schermelleh, L., R. Heintzmann, and H. Leonhardt (2010). "A guide to super-resolution fluorescence microscopy", Journal of Cell Biology 190(2):165~175 Sirat, G.Y. (2016), Method and device for superresolution optical measurement using singular optics, Google Patents Sirat, G.Y. (2017), Method and device for superresolution optical measurement using singular optics, Google Patents Sirat, G.Y. (2017), "Superresolution metrology Methods based on singular distribusions and deep learning", WO2019043458A2

Claims

Claim 1 A method for determining the two-dimensional or three-dimensional positions of a set of separate light-emitting objects, comprising: using a scanning device to project, along a projection axis, a set of distributions of light radiation in two or three dimensions, each distribution of light radiation having zero intensity at a common center of each distribution, onto each object of a set of objects with different centers for each object of the set of objects; measuring the intensity of the emitted light; and using the scanning device to move, for each object of the set of objects, to a point on the object of the set of objects where the light radiation is minimum, thereby characterizing two or three of the dimensions of the respective positions of the separate light-emitting objects. Claim 2 The method according to claim 1, wherein the scanning device used to move the position is an SLM or DMD disposed on a surface coupled to the sample, and in the SLM or DMD, only the pixels coupled to the determined position are in the on-mode and the collected light is a direct path, or in the DMD, only the pixels coupled to the position of the emitter are in the off-mode and the collected light is an indirect path. Claim 3 The method according to claim 1 or claim 2, wherein the size of the spot corresponding to the diffraction limit is larger than the pixel size and is between 3 times and 11 times. Claim 4 The method according to claim 3, wherein an algorithm is applied to determine an appropriate combination of on and off pixels to position the centroid of the spot on the sample with a resolution better than that of a single overall pixel of the scanning device projected onto the sample surface. Claim 5 The method according to any one of claims 1 to 4, wherein a first optical module is used to generate a light distribution including the position of the emitter as part of the irradiation region. Claim 6 The method according to claim 1 or claim 5, wherein the scanning module is selected from the group consisting of a multi-channel acousto-optic or electro-optic deflector, a wavefront shaper, and combinations thereof.

Images