Super-resolution metrology method based on single distribution and deep learning

Abstract

Method for determining values ​​of intrinsic geometric parameters characterizing geometric features of a physical object, method for classifying a scene into at least one geometric shape, each geometric shape modeling a light emitting object. A single light distribution characterized by a first wavelength and a singularity position is projected onto the physical object. Light excited by the single light distribution that has interacted with the geometric features and impinged on a detector is detected, and the return energy distribution at one or more locations is identified and quantified. A deep learning or neural network layer is employed, using the detected light as a direct input to the neural network layer, suitable for classifying the scene into multiple shapes, static or dynamic, which are part of a group of shapes predetermined or acquired by learning.

Description

[0001] Related Applications

[0002] This application claims priority to two U.S. provisional applications, serial numbers 62 / 551,906 and 62 / 551,913, filed on August 30, 2017, both of which are incorporated herein by reference. Technical Field

[0003] The present invention relates to a method and apparatus for optically measuring geometric features using machine, representation and deep learning methods, and more particularly, to a method and apparatus for optically measuring geometric features using a projected single light distribution. Background Art

[0004] Metrology is the field of measurement technology. Measurements based on fluorescence, multiphoton imaging, Raman scattering, light transmission, reflection or light scattering are practically limited in resolution to the limit stated by Ernst Abbe in 1873. The Abbe limit arises due to diffraction of a defined aperture of the optical system used to illuminate or collect light from the sample, as described below.

[0005] Some biological and life science objects can be modeled mathematically as basic geometric shapes. Accurate quantification of such model parameters is one of the challenges facing modern biology and life sciences; it will reveal hidden information about biological objects and their functions, either directly or statistically. Sirat defines "luminescent biological objects" as "maps" of physical objects, "in the sense of ... general semantics," in U.S. Patent No. 9,250,185 (hereinafter "Sirat'185 Patent"), which is incorporated herein by reference. (Sirat'185 Patent, column 18, lines 15-22).

[0006] Measuring the shape or other geometric properties of the object includes applying a parameterized shape assumption, referred to in the present invention as a mathematical prior, to the elements and / or reconstruction data from the "luminous biological object" and retrieving the shape parameters from the data. In addition, there is another mathematical constraint on the data, namely positivity due to the inherent positivity of the light intensity.

[0007] In the prior art, both mathematical priors and positivity constraints rely on the traditional imaging paradigm. After image acquisition and reconstruction, mathematical priors and positivity constraints must be applied to the resulting images as a post-processing step. This imaging paradigm dates back to the earliest days of microscopy, where shape measurement was considered an adjunct to imaging and not a discipline in itself.

[0008] The shape assumption is a powerful mathematical prior and once the invention is described below, it will greatly reduce the number of degrees of freedom of the solution and create an embedded parameter space for the solution. As used herein, the term "mathematical prior" refers to any mathematical assumption about the measurement results that reduces the number of degrees of freedom for the problem at hand. As has been done in the past, applying the mathematical prior after the acquisition and reconstruction process reduces the accuracy of the entire metrology process relative to data acquired using general imaging paradigms.

[0009] Historical perspective

[0010] Ernst Abbe, who lived from 1840 to 1905, would have been comfortable using the latest generation of biological microscopes for imaging and measurement, after some training on the newer hardware and software. The persistence of biological imaging and metrology methods is due first to the quality of the concepts developed by our predecessors, but also to the difficulty of modifying ingrained working procedures.

[0011] However, even in biology, some new trends are shaking up embedded microscopic observation prototypes. The main progress is due to the desire to accurately quantify the basic parameters of biological object models in a comprehensive statistical way, rather than making biological observations based only on images or assumptions.

[0012] J. Shamir in "Singular beams in metrology and nanotechnology", Opt. Eng., Vol. 51, 073605 (2012) and US Pat. No. 7,746,469 (both incorporated herein by reference) have suggested using a single light distribution to quantify the distribution of particle sizes.

[0013] To the best of the inventors' knowledge, it has never been suggested to create a null image that uses the absence of photons to achieve a resolution not obtainable where light itself is imaged. In particular, Shamir's teaching uses elastic interactions such as reflection, transmission, and (elastic) scattering. It would be advantageous to obtain information based on the null image, and the present invention described in detail below shows how to perform such imaging.

[0014] Background: Measurement of geometric shape parameters

[0015] Although all measurement systems and methods used in biological microscopy need not be reviewed, in theory the measurement limit of any parameter, including those describing shape, is ultimately determined by the signal-to-noise ratio and can be quantified using the Cramer Rao lower limit (CRLB) defined below. However, for shape-related parameters, the calculations are poorly conditioned and rely on measurements that are imprecise due to practical and experimental conditions.

[0016] For example, Gustafsson, "Nonlinear structured-illumination microscopy: wide-field fluorescence imaging with theoretically unlimited resolution", Proc. Nat. Acad. Sci, Vol. 102, pp. 13081-86 (2005), incorporated herein by reference, presents nonlinear structured illumination microscopy, super-resolution techniques, coupled nonlinear saturation effects, and structured illumination. Gustafsson's technique is considered exemplary by those skilled in the art of microscopy, demonstrating a resolution of 50 nm in the best case. The measurements were made on calibrated, isolated, planar, identically manufactured beads that are capable of sustaining high energy levels, are not susceptible to photobleaching or phototoxicity, and were selected by Gustafsson to have a precise diameter of 51 nm, a bead size nearly equal to the resolution limit.

[0017] Based on the foregoing data, the effect of bead size, the simplest possible measurement of the shape parameter calculated by an exact procedure ("known bead shape was removed from the reconstruction by linear deconvolution"), as defined below, Gustafsson 2004 is less than 10% of the "System Ruler" of his experiment. If the same experiment is performed as a procedure to quantify the unknown diameter value of the bead, the relative accuracy will be 5%, a relatively poor performance in this ideal case. This indicates the need for an improved tool to measure the shape parameter, as described below in accordance with the present invention.

[0018] Deep Learning Background

[0019] A large literature is available on deep learning. The reader is referred to Wikipedia, https: / / en.wikipedia.org / wiki / Deep_learning, and the references cited therein. A synthesis of the concepts of deep learning can be found in LeCun et al., “Deep learning”, Nature, Vol. 521, pp. 436-45 (2015) (hereinafter “LeCun 2015”), which is incorporated herein by reference to define several basic concepts used below. According to LeCun 2015:

[0020] Representation learning is a group of methods that allow machines to be fed raw data and automatically discover the representations needed for detection or classification. Deep learning methods are representation learning methods with multiple levels of representations, obtained by composing simple but nonlinear modules, each of which transforms the representation of one level (starting from the raw input) into a higher, slightly more abstract level representation. With enough combinations of such transformations, very complex functions can be learned. For classification tasks, higher-level representations amplify aspects of the input that are important for distinguishing and suppressing irrelevant variations. For example, images come in the form of arrays of pixel values, and the features learned in the first layer representation typically represent the presence or absence of edges at specific orientations and locations in the image. The second layer typically detects patterns by locating specific arrangements of edges, regardless of small changes in edge locations. The third layer may assemble patterns into larger combinations of parts that correspond to familiar objects, and subsequent layers will detect objects as combinations of these parts. The key aspect of deep learning is that these layers of features are not designed by human engineers: they are learned from data using a general learning process.

[0021] (Lecun 2015, p. 436, bold emphasis added)

[0022] Current prototypes of deep learning networks in practice involve processing acquired images. Figure 6 A deep learning network according to the prior art following the teaching of Lecun 2015 is illustrated; an image 10 shown of a cat is the input to the network. The first layer 11 is generally aware of the presence or absence of edges at specific orientations and locations in the image. Additional layers schematically represented by numbers 12 and 13, learned from data using a general learning procedure, are able to recognize the cat species and that it is not a dog.

[0023] As used herein, “CODIM” refers to the conical diffraction microscopy hardware described by Caron et al. (2014). “Conical diffraction illumination opens the way for low phototoxicity super-resolution imaging.” Cell Adhesion & Migration , Volume 8, Pages 430-439 (2014), hereinafter referred to as "Caron 2014", which is incorporated herein by reference. The use of hardware as described in Caron 2014 is referred to as the "imaging case". The hardware setup and use described below as the "metrology case" is not part of the prior art. In the imaging case, a single light distribution is projected on a regular grid, which is Cartesian, or a single light distribution is projected on a grid that is optimized as a function of the retrieved data.

[0024] The standard way to apply CODIM in deep learning or machine learning networks is to reconstruct the image and apply the known algorithm to the reconstructed image. The limitation of this approach is the time burden and information destruction required in image reconstruction.

[0025] Sirat'185 patent states:

[0026] The measurement system will compute an evaluation, a measurement map of the fluorophore descriptors. This measurement map differs from the original map due to noise, measurement conditions, system limitations or measurement uncertainties. The information map can later be developed to different levels of abstraction.

[0027] (Sirat'185 patent, col. 18, lines 37-42, emphasis added)

[0028] Sirat'185 patent further teaches:

[0029] The raw information, or map in general semantic terms, is the set of descriptor fluorophores and their evolution over time. The biological and geometric information will be just an extrapolation of this raw information. The measurement system will compute an evaluation of the fluorophore descriptors, the measurement map.

[0030] (Sirat'185 patent, column 18, lines 32-37)

[0031] Sirat '185 assumes that "[t]he measurement system will calculate an evaluation of ... the measured map" (as above, the slash emphasis is added), considering the measured map as a prerequisite for any information collection. As Sirat '185 describes in the same paragraph, in requiring the measured map as a prerequisite for information collection, Sirat '185 follows the general consensus: the "measured map" is a prerequisite for further processing, as an abstraction at the geometric and biological levels. Summary of the invention

[0032] According to an embodiment of the present invention, a method for determining the value of an intrinsic geometric parameter of a geometric feature of a specific dimension of a physical object is provided. The method comprises:

[0033] a. projecting a single light distribution characterized by a first wavelength and a singularity position onto a physical object;

[0034] b. detecting light excited by the excited single light distribution, which has interacted with the geometric features and impinged on the detector, the light constituting the detected light;

[0035] c. Identifying and quantifying the return energy distribution at one or more locations of a single light distribution as a quantification of intrinsic geometric parameters; and

[0036] d. Determine the values ​​of intrinsic geometric parameters based on the parameters retrieved from the energy distribution.

[0037] According to other embodiments of the invention, successive applications of the above method are performed in a manner of retrieving a single or group of intrinsic geometrical parameters. The intrinsic geometrical parameter may be one of the size and ellipticity of the point object. The measurement of the displacement may be determined as an offset transverse to the line.

[0038] According to another embodiment of the present invention, the intrinsic geometric parameter may be a width of a line, and the intrinsic geometric parameter may be based on a model shape of the geometric feature. Detecting the light may include employing a pixelated detector.

[0039] According to another aspect of the present invention, there is provided a method for representation learning for classifying a scene into at least one geometric shape, static or dynamic, quantified by a suitable set of parameters, each geometric shape modeling a luminous object. The method comprises:

[0040] a. Project a single distribution of light onto the scene;

[0041] b. detecting light distributions re-emitted by the scene when illuminated by the single light distribution, the single light distribution having interacted with each luminous object and impinging on a detector, the detected light constituting the detected light;

[0042] c. measuring at least one projection of a single distribution at a given location to obtain a set of measurements about the scene; and

[0043] d. Employing a deep learning or neural network layer, using the detected light as a direct input to the neural network layer, suitable for classifying the scene into multiple physical or biological objects or static or dynamic geometric shapes, wherein the objects or shapes are part of a group of objects or shapes predetermined or acquired through learning.

[0044] According to other embodiments of the present invention, the at least one geometric shape may be dynamic or static.The method may further include acquiring data, the data being imaging data acquired by CODIM hardware or metrology data acquired by metrology application hardware.

[0045] According to another embodiment of the present invention, the method may have a further step of acquiring data as imaging data acquired by CODIM hardware, wherein the resulting classification information is used to drive metrology hardware to implement the metrology method. A further step may include providing control hardware and a control process to drive the metrology hardware to implement the measurement method. Another step may also include feeding the data to a neural network that has been acquired at different times, and including the temporal correlation of the luminous object in the neural network recognition process. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The foregoing features of the present invention will be more readily understood by referring to the following detailed description with reference to the accompanying drawings, in which:

[0047] Figure 1 The prior art modulation transfer function (MTF) of the CODIM light distribution compared to the Airy pattern is shown.

[0048] Figures 2A-2D Influenza A virus (IAV) is used as an illustrative example in this specification. For the isotropic and non-isotropic models, the virus is schematically represented in Figure 2A The three-dimensional model is shown in Figure 2B In the figure, the two-dimensional models are represented in Figure 2C and Figure 2D middle.

[0049] Figure 3 Reproduced from U.S. Patent No. 9,250,185 (hereinafter "Sirat '185") and presents some of the different unitary distributions obtainable using conical diffraction; and

[0050] Figure 4 Schematic diagram showing the trajectory of a crescent moon, known as a spiral, and planes A and B intersecting the spiral at two points with different lateral positions according to an embodiment of the present invention.

[0051] Figure 5 is a flowchart showing the steps of a representation learning method according to an embodiment of the present invention;

[0052] Figure 6 An implementation of a deep learning system according to the prior art is schematically shown.

[0053] Figure 7 Schematic diagram of the implementation of the deep learning system on the CODIM imaging system.

[0054] Figure 8 The implementation of a direct deep learning network on a CODIM imaging system according to the present invention is schematically shown.

[0055] Fig. 9The implementation of a direct deep learning network on a composite imaging system according to the present invention is schematically shown.

[0056] Fig.10 Schematically depicted is the implementation of a direct deep learning network for a general object, represented as a cat for heuristic reasons. The fact that the intermediate image is represented as a sub-image of a part of the cat frame is just a graphical trick. The information would be unstructured information, not a partial view of the image.

[0057] Fig.11 The implementation of a controlled direct deep learning network on a CODIM imaging system according to the present invention is described.

[0058] Fig.12 Depicted is the implementation of a controlled direct deep learning network on a general object, represented as a cat for heuristic reasons. The fact that the intermediate image is represented as a sub-image of a part of the cat frame is just a graphical trick. This information would be unstructured information, not a local view of the image. DETAILED DESCRIPTION

[0059] New approaches are proposed here, where the measurement process is a separate dedicated process and architecture, initiated in some cases by the imaging process, but fully embedded in a separate tool (the metrology tool).

[0060] The invention described below is directed to accurately measuring, with minimal throughput, geometric features that are less than the diffraction limit of the light employed in their measurement.

[0061] The invention is particularly suitable for measuring geometric features using inelastic light interactions, such as but not limited to fluorescence, multiphoton imaging or Raman scattering, where the outgoing light can be separated from the incident light in a simple manner.

[0062] Some embodiments of the invention relate to methods for machine learning of geometric and functional features and objects and hardware systems that implement them. These methods require classification and recognition of features and objects; these methods differ from practical solutions based on imaging paradigms and further processing after imaging.

[0063] Although some embodiments of the present invention are described primarily with reference to systems and methods implemented in fluorescence microscopy, it should be understood that extension to reflected or scattered light is straightforward and within the knowledge of one of ordinary skill in the art having comprehended the present specification. In addition, although the present invention is described herein in terms of systems and methods for metrology tools for fluorescent objects encountered in biology, they can be readily applied to other imaging modalities and other disciplines and fields, such as, but not limited to, semiconductor and machine vision.

[0064] It is useful to apply mathematical priors before the reconstruction process and embed them into the acquisition process, however there has never been any teaching on how to do this, which will now be described in detail. According to an embodiment of the present invention, applying mathematical priors before image reconstruction can advantageously improve the measurement accuracy of shape parameters, reduce the number of observations required for measurement, and improve the stability and reproducibility of the entire process.

[0065] The method described according to the embodiment of the present invention relies on the specificity of Poisson's law, specifically, the absence of photon noise - which reaches negligible values ​​due to the theory of quantum physics - in the absence of photons, without any photon incident and emitted on the detector, rather than a posteriori correction. This method requires a mechanism to physically remove the incident light by a filtering operation. It cannot be a mathematical post-processing operation, because if the noise has been introduced into the system, it cannot be filtered out by simple means. This is the advantage and specificity of inelastic light interactions; in the present invention, inelastic light interactions refer to any light interaction that is able to produce (or emit) new photons, which have physical properties that distinguish them from the incident photons, so that the emitted photons can be physically separated from the incident photons. It is assumed that the "filter" allows the separation of the emitted photons from the incident photons. An obvious example of an inelastic light interaction is an interaction that produces photons of different wavelengths; in this case, the filter will be a wavelength-sensitive filter. Examples of inelastic light interactions are fluorescence, multiphoton imaging or Raman scattering. Elastic light interactions, such as reflection, transmission or elastic scattering, do not allow this distinction to be made in a simple way.

[0066] As described below, embodiments of the present invention rely on creating an empty image or an image close to an empty image. Such an image allows additional resolution, theoretically unlimited, by combining the loopholes in Abbe's law with the lack of noise due to Poisson's law, as described below. A prerequisite is the absence of any parasitic photons required to meet the theoretical conditions. This condition can be met almost exclusively using inelastic light interactions (such as fluorescence, multiphoton interactions and Raman scattering), where the incident beam can be completely filtered by spectroscopic means.

[0067] For the sake of convenience, the present invention may be illustrated by a hypothetical example; this example is for illustration only and does not represent a tested experimental situation. Assume that a specific biological object constitutes a luminous object, and for the sake of illustration, assume that the biological object is influenza A virus (IAV), such as Figure 1A. It is known that IAV viruses have a typical size of 80-120 nm, and as described by Badham et al. in “The biological significance of IAV morphology in human clinical infections is a subject of great interest”, Filamentous Influenza Viruses, Current clinical microbiology reports, As described in Volume 3, Pages 155-161 (2016), which is incorporated herein by reference.

[0068] Assuming that the IAV virus is uniformly labeled with a suitable fluorescent protein throughout its volume, three different models can be used to describe the virus:

[0069] a. In the imaging paradigm used by the prior art, due to the small size of viruses, at the diffraction limit, the best model is to represent the viruses as glowing spots that emit fluorescence; it does allow accurate monitoring of viral positions.

[0070] b. Alternatively, using, for example, an electron microscope with nm or sub-nm resolution, various details of the virus, including hemagglutinin 201 and neuraminidase 203, can be characterized.

[0071] c. In accordance with the teachings of the present invention, an intermediate approach is adopted in which the virus is described as a point object defined as follows, which is modeled as a sphere in three dimensions ( Figure 2B ), or in two dimensions it can be modeled as a uniform circle with a given radius ( Figure 2C ), or modeled as an ellipse ( Figure 2D ), where the parameters are the semi-minor axis a, the semi-major axis b, and the angle θ of the major axis relative to the Cartesian coordinate system.

[0072] According to a further embodiment of the present invention, deep learning may be employed, as described in detail below.

[0073] Methods according to embodiments of the present invention may advantageously provide measurements requiring a resolution and a minimum photon flux greater than the diffraction limit imposed by the optics.

[0074] According to an embodiment of the present invention, one or more mathematical priors resulting from shape assumptions are embedded into a group of specific single light distributions, allowing full exploitation of the properties of the single light distributions. The coupling is based on a dedicated metrology tool that relates the mathematical priors to the group of distributions; in other words, the measurement of a specific shape mathematically defines the priors and the solution domain, which itself determines the properties of the acquisition tool and the type of distribution required.

[0075] Definitions: As used herein and in any appended claims, unless the context requires otherwise, the following terms shall have the following designated meanings:

[0076] The term "value" as used herein and in any appended claims refers to a real number that characterizes a quantity associated with a parameter. It should be understood that in practical situations, the quantity associated with a parameter may be characterized within a certain range constituting the precision of the measurement. In such cases, the term "value" may be used as a shorthand for a distribution of values.

[0077] "System gauge" is a value that one of ordinary skill in the imaging arts would consider to characterize the system's ability to discern detail.

[0078] In imaging systems, the "Rayleigh criterion" - the ability to separate two adjacent points - is the generally accepted standard for the smallest resolvable detail, even though in many cases the observed FWHM of a point or line is used as a practical assessment of the "diffraction limit", a qualitative term often used to quantify the smallest resolvable detail. In this invention, we use the FWHM of an infinitely thin line as the system scale. If the value is "much smaller" than the system scale, the value can be ignored, where "much smaller" is defined as a factor of 3 or more smaller than the system scale.

[0079] As used herein, the term "Abbe resolution limit" is defined in Schermellheh et al., "A guide to super-resolution fluorescence microscopy", J.Cell Biology , Vol. 190, pp. 165-75 (2010) (hereinafter "Schermelleh 2010"), which is incorporated herein by reference:

[0080] Abbe's famous resolution limit is so intriguing because it simply depends on the maximum relative angle between the different waves that leave the object and are captured by the objective to be sent to the image. It describes the smallest level of detail that can be imaged with this PSF "brush". It is impossible to transfer periodic object details smaller than this shortest wavelength to the image.

[0081] The expression "above the Abbe limit" is defined to refer to an object containing a periodic structure that contains details smaller than any detail on the scale of the system and is therefore limited by the Abbe limit. The rationale for this definition is that such an object contains spatial frequencies above the Abbe frequency circle in the aperture plane.

[0082] In estimation theory and statistics, the Cramer-Rao bound (CRB) or equivalently the Cramer-Rao lower bound (CRLB) represents a lower bound on the variance of an estimate of a deterministic (although unknown, fixed) parameter. The precise definition used herein is provided at https: / / en.wikipedia.org / wiki / Cram%C3%A9r%E2%80%93Rao_bound, which is incorporated herein by reference.

[0083] As used herein, the term "localized" light distribution refers to a light distribution in which the energy is concentrated over a small domain. A light distribution will be localized if the energy outside a radius of 3.5*half-Rayleigh criterion is essentially zero.

[0084] The invention description assumes that the optical system is close to being "photon noise limited", as described in https: / / en.wikipedia.org / wiki / Shot_noise, or close to being photon noise limited, i.e. the Gaussian noise component is less than half of the photon (or shot) noise. The best case is actually the described "photon noise limited" optical system and a "Gaussian noise limited" or "detector noise limited" system will only collect part of the advantages of the invention, but still be within the scope of the invention.

[0085] “Full width at half maximum” (FWHM) is an expression for the range of a function given by the difference between the two extreme values ​​of the independent variable, where the independent variable is equal to half its maximum value, from Wikipedia, https: / / en.wikipedia.org / wiki / Full_width_at_half_maximum.

[0086] "Dimension" is defined as any of the three physical or spatial properties of length, area, and volume. In geometry, a point is said to have zero dimensions; a figure having only length, such as a line, has one dimension; a plane or surface, two dimensions; and a figure having volume, three dimensions.

[0087] The dimensionality of a geometric feature refers to the dimension of the corresponding idealized feature to the extent that the extent of the geometric feature (e.g., the "diameter" of a point object, or the "width" of a line, or the "thickness" of a coating) is much smaller than any other dimension that tends to zero.

[0088] As used herein, the term "trajectory" shall mean a set of points (typically points, lines, line segments, curves, or surfaces) whose positions satisfy or are determined by one or more specified conditions.

[0089] "Geometric parameter" refers to any measurable quantity that characterizes an extent in physical space. Thus, for example, distance (defined in three-dimensional Euclidean space or along a surface) is an example of a geometric parameter. Area defined on a surface is another example of a geometric parameter.

[0090] A "geometric feature" is a physical object that is geometrically characterized or modeled as a trajectory, quantified by one or more geometric parameters; for example, the thickness of a line describes a trajectory, the line being a trajectory, or the position and thickness of a line segment that is also a trajectory. An "embedding space" represents the parameter space spanned by the geometric parameters.

[0091] A "point" is a two- or three-dimensional geometric feature with zero dimensionality and zero size. It is an outdated simplification that eliminates much information about the real object but greatly simplifies the assumption. We call an object of small but non-negligible size compared to the System scale in all three dimensions a "point object". The terms small or negligible must be understood in contrast to the System scale which is diffraction limited for the optical system. A point object is determined by its position and its size, which may or may not be isotropic in two or three dimensions. In the system scale of diffraction limited or super-resolution optical systems point objects, most biological objects are point objects, and the a priori elimination of the information carried by point objects is a huge loss. The distinction between a point and a point object is important in the present invention.

[0092] "Line" (and similar other terms that refer to a theoretically one-dimensional shape) shall refer to a geometric feature (i.e., a physical object with length, width, and thickness), where the length is at least 10 times the width or thickness. "Line objects" are defined in the same way as point objects.

[0093] The term "intrinsic geometric parameter" as used herein and in any appended claims refers to a parameter that characterizes the properties of a surface (in n-dimensional space) that is independent of the isometric embedding of the surface in Euclidean space. Thus, for example, while the width of a "line" at a given point is an intrinsic parameter of the line, it is independent of any orientation of the line in space.

[0094] With reference to geometric elements in an abstract space, a set of parameters will be called "appropriate" if and only if the representation of the geometric elements described by the set of parameters truly represents the geometric class and parameters of the object within the limits of measurement uncertainty.

[0095] "Single optics", including "optical vortices" as their simplest example, is now an emerging field of optics with both theoretical and practical applications. A detailed description can be found in Nye et al. Dispositions in wave trains , Proceedings of the Royal Society of London A , Vol. 336, pp. 165-90 (1974), and Soskin et al., "Singular optics", Progr.in Optics , Vol. 42, pp. 219-76, both of which are incorporated herein by reference.

[0096] "Inelastic optical interaction" refers to the interaction between light and matter that produces photons with wavelengths different from the incident light beam. Inelastic optical interactions include, but are not limited to, fluorescence, multiphoton interactions, and Raman scattering.

[0097] A "trajectory of a single distribution" is the set of Cartesian positions where the intensity of the single distribution is zero. The trajectory of a single distribution defines a family of basic shapes that have appropriate parameters, "nominal parameters" and are located at the correct position; the "nominal position" will not emit (or reflect or scatter) light. In this case, we will propose a new concept and express "single light distribution embedded shape". For example, for the light distribution depicted as case 10 (or 01) in Figure 2, the vortex, the trajectory is an infinitely small point, and the only family of shapes that can be studied using the vortex are point shapes and point object shapes, which will also not produce energy with a zero-value radius. For the distribution called "vertical half moon" described as case 42 (or 24) in Figure 2, the relevant family of shapes is more diverse, including lines and line objects, as well as multiple point shapes and point objects. Other cases will be introduced below.

[0098] Conical refraction is an optical phenomenon predicted by Hamilton in "Third supplement to an essay on the theory of systems of rays", Trans. Royal Irish Acad., Vol. 17, pp. 1-144 (1837), and confirmed experimentally by Lloyd two months later, see Lloyd, H. (1831) "On the Phenomenon Presented by Light in Its Passage along the Axes of Biaxial Crystals", Philos. Mag., Vol. 1, pp. 112-20 (1831). Both of the above references are incorporated herein by reference. Conical refraction describes the propagation of a light beam in the direction of the optical axis of a biaxial crystal. Hamilton predicted that light would appear in the form of a hollow light cone.

[0099] A description is provided in Berry, "Conical diffraction asymptotics: fine structure of Poggendorff rings and axial spike", J. opt. A: Pure and Applied Optics, Vol. 6, pp. 289-300 (2004), which is incorporated herein by reference. A prior art system based on conical diffraction for super-resolution microscopy is described in Sirat 2016.

[0100] U.S. Patent No. 8,514,685 (Sirat'685) describes the specific properties of conical diffraction in thin crystals and demonstrates the use of conical diffraction to shape light beams. Thin biaxial crystals transform the point spread function (PSF) of a regular incident beam into a wide family of light distributions, the choice of which is controlled by the input and output polarizations. In practice, beam shaping is achieved by encapsulating the biaxial crystal between controllable polarizers; this simple optical device is similar to a polarimeter, with the ability to switch from one mode to another with a different topology in microseconds or faster. Furthermore, these patterns are completely co-localized because they are generated by the same primary beam.

[0101] The CODIM beam shaper can be used as an add-on to the scanning module and scans the profile over the sample, generating several microimages for each scan point. The CODIM system is available from BioAxial SAS, Paris, France.

[0102] The CODIM beam shaper uses the principle of conical diffraction to produce a compact local light distribution. Each micro-image contains a large amount of high frequencies, close to the Abbe limit (compared to the Airy mode 101, reaching a factor of nearly 3, such as Figure 1 ). These light distributions projected onto a sample area called a "scene" are analyzed using algorithms known to those of ordinary skill in the art. For exemplary algorithms suitable for obtaining super-resolved images from microscopic images, the reader is referred to the published application WO2016 / 092161 (published in English as US2017 / 0336326, hereinafter referred to as "Sirat'161 Application"), which is incorporated herein by reference. This allows super-resolution images to be reconstructed for general objects with a resolution improvement of up to 2 times. In addition, these algorithms exploit positivity constraints and sparsity, allowing the resolution to be further improved for sufficient samples.

[0103] Ultimately, the combination of lower distributed peak power, the use of a high quantum yield camera, and longer camera exposure times greatly reduces the peak power and energy of the light sent to the sample. This is why this method has very low photobleaching and phototoxicity. This also avoids the fluorophore saturation problem, making the method linear and quantitative.

[0104] In this specification, the term "energy law" is defined as follows: Assuming that the object has been modeled as a mathematical abstraction, a geometric shape, an "energy law" is a parametric relationship between energy and position as a function of shape parameters. It creates a relationship that quantifies the energy dependence of the parameter space. The energy law can include the distribution of energy emitted by a light-emitting object with the same shape as the geometric shape.

[0105] In this description it is assumed that the optically unitary distribution can be controlled so that it is possible to switch from one type of distribution to another type of distribution, to switch from a predetermined family of distributions, and to modify the parameters of the optically unitary distribution using external means, as described in Sirat '185. Other solutions exist that do not require the control of the optically unitary distribution and are part of the present invention, but they will be more cumbersome.

[0106] The imaging performed using the method according to the invention may be referred to as "compound imaging" and may also be referred to as coded imaging, as defined in Wikipedia. https: / / en.wikipedia.org / wiki / Coded_aperture, or as "indirect imaging", with reference to an imaging process in which the elemental data from the scene are not images or maps in the sense of the Sirat'185 patent describing the object. In a compound imaging process, the data measured by the physical instrument are referred to as "intermediate results" (a term used in the Sirat'161 application), or as elemental (or basic) data used synonymously in this specification, containing information from which a reconstructed image can be created by a reconstruction process. In the present invention, we identify a compound imaging process as an imaging process in which it is not possible to assume the intensity or number of photons measured at a specific time, and the position to a specific pixel of the image or to a specific attribute of an element in a map. In other words, a compound imaging system of the intensity or number of photons does not have a direct translation to pixel data, but contains information related to several of them.

[0107] In the field of super-resolution microscopy, CODIM imaging is an example of a compound imaging process as well as structured illumination microscopy (SIM) or localization microscopy techniques such as PALM / STORM, see Sirat '185 patent.

[0108] The "measurement process" will consist of groups of "elementary single measurements", each consisting of the projection of a predetermined single light distribution with given parameters, positioned so that the trajectory of the single distribution is embedded in the geometric shape with nominal parameters, and a "light measurement" quantifying the light incident from the object, its intensity and optionally its spatial distribution, as recorded on a light detector, the spatial distribution being pixelated or non-pixelated. In some cases, information collection may also be accomplished using "elementary regular measurements" consisting of the projection of a predetermined regular light distribution.

[0109] The "input" of the measurement refers to the location, type and parameters of the optical single distribution. The "output" is the light intensity and / or light distribution recorded in the measurement. We assume that the reader is a person skilled in the art and is familiar with "control theory". Control theory is an interdisciplinary branch of engineering and computational mathematics that deals with the behavior of dynamical systems with inputs, and how their behavior is modified by feedback (from Wikipedia).

[0110] "Control means" refers to a set of control hardware capable of modifying the inputs and a "control algorithm" capable of foreseeing the next step of the input values ​​that quantify the input values ​​required by the "Energy Law" directly or through successive approximations in a manner sufficient to retrieve the parameters accurately. The "Inverse Energy Law" is an optimal or non-optimal recipe capable of retrieving the parameters of the shape from a single single distribution or a set of measurements of a set of single distributions. It is embedded in the control algorithm. It will be chosen to optimize the functional parameters of the system, the number of steps required, the total energy or power that impacts the biological object, the measurement speed, any combination of the above, or any other functional parameter of the system.

[0111] In the context of a singular distribution, a "value close to zero" refers to an energy used to qualitatively describe the projected intensity or emitted energy that is reasonably less than the maximum intensity available on the projected light or the energy emitted when the maximum value of the projected light hits that point. The quantitative value of the intensity or energy close to zero is a factor of 6 between the intensity of the projection and the maximum intensity of the distribution, or between the energy emitted and the energy emitted if illuminated at maximum intensity. It is worth mentioning that, assuming Poisson noise, energies close to zero will have significantly smaller noise values, nearly 2.5 times less at the maximum energy. Similarly, parameter values ​​that are "close to zero" for a geometry will have values ​​that are less than the entire range of the parameter divided by 2.5.

[0112] According to an embodiment of the present invention, the observed object can be modeled as a specific shape that is known or assumed in advance, or as a shape in a list of potential shapes. The geometric shape can be described according to the following aspects:

[0113] a. Spatial position: The “position” of a shape refers to the position of the origin determined in the shape, which is the center of gravity of the light distribution in most cases.

[0114] b. Groups of "structural parameters" that describe "shape parameters", e.g., for an isotropic circle ( Figure 2C ) is its radius, for an object represented by an ellipse ( Figure 2D ) are the semi-axes of the ellipse and the angles.

[0115] As used herein, the term "machine learning" refers to a field of computer science that uses statistical techniques to give computer systems the ability to "learn" with data (e.g., gradually improve performance on a specific task) without being explicitly programmed to do so.

[0116] The terms "representation learning" and "deep learning" are used according to Lecun 2015 as described in paragraph

[0015] above. Lecun 2015 draws a clear boundary between "representation learning" and "deep learning" on the one hand, and standard "machine learning" on the other hand.

[0117] The terms "representation learning" and "deep learning" are used interchangeably herein, as all embodiments of the present invention apply to both. Portions of the present invention may also be applied to machine learning.

[0118] In the metrology case (as defined above), data is acquired at the nominal position of the identified object. "Nominal position" when it refers to an object shall mean the position that is assumed to be the position of the object based on prior knowledge or based on assumptions.

[0119] For the purposes of this description, it is assumed that a separate mechanism has been used to collect the nominal position of the object. Within the scope of the invention, this mechanism may use any localization technique, such as measurement of the centroid of the light distribution produced by the object in another imaging modality or directly on the camera.

[0120] Basic principle of the invention

[0121] The present invention introduces a new method for measuring parameters of biological, life science or physical objects modeled as geometric shapes. According to the method described in detail herein, a system is created to directly access the parameter values ​​of the underlying geometric shapes; the method and system exploit the properties of a single light distribution and are neither limited by diffraction as imaging is, nor by photon flux as parameter evaluation using imaging paradigms and image processing tools is limited.

[0122] The novel measurement methods described here are associated with a physical insight, first described here, to our knowledge in the form of a specific restriction of the direct applicability of the measurements by Abbe's resolution limit.

[0123] Indeed, going back to the Abbe resolution law described in Schermelleh 2010 or Horstmeyer et al., “Standardizing the resolution claims for coherent microscopy”, Nature Photonics 10, pp. 68-71 (2016) (incorporated herein by reference), all “bandwidth extrapolation” techniques to reconstruct information above the Abbe limit rely on prior knowledge, assumptions or even guesses, which in some cases may be wrong. We have only described here the case for which we know – measurements above the Abbe resolution law without prior knowledge, assumptions or guesses, which is the case of … a black (zero) image.

[0124] Due to the positivity constraint, the null image contains no signal, so there are no spatial frequencies below and above the Abbe limit, and all frequency content is completely known... to zero; the null image allows high-frequency components above the Abbe limit to be quantified to zero based on diffraction-limited optical measurements.

[0125] This is contrary to the natural view of Abbe's law, which is understood as that above the Abbe limit no information can be retrieved by optical processes. The situation described in the present invention, even if it is very special, allows the collection of information above the limit, theoretically up to infinity, so that the diffraction is infinite.

[0126] This condition is specific for optics and similar monopole signals and is due to a positivity constraint that is independent of the diffraction limit, which is an additional physical constraint on the solution. These concepts cannot be readily applied to electrical signals or similar bipolar signals. The positivity constraint is known in a sense to allow additional information, but its contribution to information extension is marginal in most practical cases and is not well characterized or quantified. In the present invention, the impact of the positivity constraint is clear and decisive, allowing accurate quantification of spatial frequencies below and above the Abbe limit.

[0127] Several formulations of Abbe's law of analysis exist. The preferred wording of Abbe's law of analysis is that of Schermelleh 2010: "No periodic object details smaller than this shortest wavelength may be transferred to the image" (Schermelleh 2010, p. 166). The results of the present invention constitute a consequence of Abbe's law of analysis, not an exception.

[0128] However, the results of the present invention are indeed an exception and reconstruction of some broad statements of Abbe's law of resolution in the scientific literature, which are not very carefully stated as "no frequency components can be measured above the Abbe resolution limit".

[0129] In most cases, the null image contains no information and this single point or exception is useless and insignificant, and the discussion in these cases may seem meaningless and empty. However, the recorded image of an isolated point of a given size illuminated by an optical vortex located at a nominal position will be equal to zero only if the size is zero. In this specific case, the complete nulling of the image and all its frequency components is a powerful measurement from which not only the position of the point-like object can be retrieved, but also its size, as will be explained later.

[0130] The present invention additionally introduces a new approach that can be applied to any composite imaging system and is exemplified in more detail in the case of CODIM imaging. It consists in applying a direct bridge, in most cases a specific neural network, directly to the elementary data. In the previous paragraphs, we described how to apply natural paths (or bridges) between elemental data in the composite imaging process to guide metrology methods. In this paragraph, we describe how to apply natural paths (or bridges) between elemental data to deep learning in the composite imaging process by applying a supervised or unsupervised deep learning layer directly to the elemental data without the need for any image reconstruction algorithm. In addition, this layer will be able to collect data from multiple (in most cases continuous) measurements. The results of this layer (metrology layer or object layer) will be the direct input for further processing, bypassing the image or mapping step, which can still be used for visualization, identification and monitoring.

[0131] These two processes, direct metrology and deep learning, can also be compounded to create even stronger and more powerful acquisition tools.

[0132] The method described according to the invention differs from all prior art methods ever described or suggested, at least in that all prior art methods consider image acquisition as a mandatory step.

[0133] Sirat'185 patent does not suggest the use of deep learning or neural network layers herein, using the detected light as direct input to a neural network layer, is able to classify a scene into multiple shapes, static or dynamic, that are part of a family of basic shapes that are predetermined or acquired through learning. According to the present invention, a deep learning layer is used to directly structure the raw data information into basic objects with metrological features and properties that are a suitable basis for further generalizations. This approach is developed in the present invention in accordance with the deep learning perspective.

[0134] It should be noted that the Sirat '185 patent relied on physical reality, and in order to define the measured map as a descriptor of the fluorophore, this new orientation is tied to biological reality, defining biological entities as structured concepts, like the influenza A virus in our case.

[0135] Additionally, the approach can be extended to dynamic properties, retrieved as time-dependent data, or preferably as higher-level dynamic parameters. In short, returning to our example, the middle layer is able to determine the presence of a 110nm point object moving along a path at a fixed speed without morphological changes; and we will assume in the next layer that the object classification layer is an influenza A virus.

[0136] Another advantage of the proposed architecture is that reconstruction artifacts are avoided by bypassing the reconstruction step. Even the best reconstruction algorithms can be prone to artifacts, and the possible presence of artifacts is one of the barriers to the adoption of super-resolution techniques by biologists. Even worse, many artifacts can be mistakenly perceived as small dots or objects, and classification and quantification algorithms can be corrupted by these artifacts. Avoiding reconstruction and performing classification and quantification on the raw data completely eliminates this source of error.

[0137] As mentioned above, information above the diffraction limit is present in metrology data and can also be present in imaging data. This priceless information is usually erased by the reconstruction process, which cannot be processed independently of all other information, and therefore is forced to apply general procedures optimized for average data, but not suitable for these specific cases.

[0138] Geometric and biological information through non-imaging processes

[0139] The Sirat '185 patent does not suggest how geometric and biological information can be learned through a non-imaging process. In such a non-imaging process, a general learning process is used to collect higher-level information directly from the measured elemental data, without requiring the generation (or reconstruction) of an image, or in the terms of the Sirat '185 patent, requiring a quantitative map. The Sirat '185 patent does not understand that non-imaging direct learning may be able to retrieve information with much higher accuracy, as described below, with high computational efficiency, and without data loss. In addition, such a learning process may have the ability to use data that exists in the absence of light, as described below and is known as the Abbe's Law Loophole.

[0140] General approach

[0141] According to typical embodiments of the present invention, the following features are notable:

[0142] -Light distribution at a nominal position; the wavelength of the projected light is chosen so that it produces an inelastic light interaction with the photoresponsive material, and the unitary distribution is chosen so that it is embedded in the geometry, so that under the nominal parameters, the energy of the light measurement generated by the inelastic interaction of the projected light and the photoresponsive material will be zero.

[0143] - Apply the algorithm "Inverse Energy Law" to retrieve parameters from energy measurements.

[0144] The method according to the invention can advantageously use metrology hardware and a measurement process that controls the measurement sequence dynamically or not in order to collect the set of measurements required to retrieve the shape parameters.

[0145] The photoresponsive material can be exogenous, based on, for example, binding fluorescence, or endogenous, using, for example, the direct fluorescence properties of some proteins, or using Raman scattering.

[0146] For ease of inspiration, we follow each general statement of the method in the above high-level language by illustrating it with an example; the example is the simultaneous measurement of the position and size of a point-like object; to make it more illustrative, the point object is described in terms of the influenza A virus, as explained in the previous paragraph.

[0147] The general approach can be illustrated using the simplest case of a virus, which is modeled as Figure 2C The biological object is a virus; in this example, the measurement goal is to quantify the virus diameter between 80-120 nm. The virus is assumed to be located on a two-dimensional surface.

[0148] The biological object is modeled by its position and the radius R of the virus as Figure 2C The measurement involves: using metrology hardware to measure the virus at multiple locations (x i , v i ) projects an optical vortex capable of exciting fluorescent molecules located on the virus embedded in a point object geometry, the metrology hardware comprises a galvanometer two-dimensional optomechanical system capable of scanning the vortex on the biological sample, measuring the energy on a photodetector, and converging to the location with the minimum energy, using either an analytical or iterative algorithm. In this case, the inverse energy law is very simple, and the location is the location of the minimum, and the energy at the minimum depends quadratically on the radius (Equation 3).

[0149] In general, the measurement method according to the invention allows the retrieval of zero-dimensional, one-dimensional and two-dimensional trajectories using one or more different single distributions and the simultaneous retrieval of one or more simple parameters and multiple complex or compound parameters.

[0150] Measurement of point object size and position

[0151] A first embodiment of a method for measuring shape parameters is presented: measuring the position and size of a point object in two dimensions. This is one of the most common cases in biological object measurement. Furthermore, it can be extended, mutatis mutandis, to measuring the position and thickness of a line, which is another common task in biological object measurement.

[0152] Assume that a circular luminous object (such as Figure 2C The radius is R, located at the origin, and the density of fluorophores is uniform, denoted by n D, and assume that N E =n D πR 2 is the total number of emitting fluorophores, then an optical system, such as that described in detail in the Sirat '185 patent, is provided that is suitable for moving the position of the vortex, given with high precision in the plane as (v,0). In the polar coordinate notation "(m,n)" employed herein, m refers to a radial vector in a specified transverse plane relative to the z-axis, and n refers to a predetermined axis ( Figure 2C Without loss of generality, due to the symmetry of the problem, we assume that the vortex is located on the x-axis, that is, the angle is zero.

[0153] The strength of the eddy current as a function of position is given by the coefficient i S The quadratic correlation depends on the distance between the eddy current zero point and the location where the intensity is measured. More specifically, for v>R (where R is the radius of the luminous object), the energy law will be given by:

[0154]

[0155] I v =πn D i s R 2 v 2 =N E v 2 , equation (2)

[0156]

[0157] The energy law is the sum of two independent components:

[0158] aI v Independent of the position dependence of the radius, its minimum is at the origin, just as it will be the minimum at an infinitesimal point, N E The fluorophores are at nominal locations, and

[0159] I R Radius dependence, which is independent of position.

[0160] At the nominal position, at v = 0, the energy is given by (3). The inverse energy law scales quadratically with the energy. For a shot noise limited system, the same 5% characteristic obtained by the best imaging experiment can be achieved with only 100 photons due to the quadratic dependence of energy as a function of radius.

[0161] Measurement of point object size, position and ellipticity

[0162] In a second embodiment of the invention, shape parameters can be measured: the position, size and ellipticity of point objects can be measured in two dimensions. This is the least common case in the measurement of biological objects, probably because it is impossible to perform using any existing paradigm.

[0163] Assumptions

[0164] a. An elliptical luminous object, Figure 2D , with a minor semi-axis a and a major semi-axis b, respectively.

[0165] b. The distribution position is given by (0, y0) in Cartesian coordinates, rather than by polar coordinates as in the previous paragraph,

[0166] c. n D represents the uniform density of fluorophores,

[0167] d. Projection Figure 3 The half-moon distribution of

[0168] e. Figure 3 The half-moon distribution of case 24 is already related to the major axis ( Figure 2D The x-axis in the figure is aligned with the direction of the x-axis;

[0169] In a first approximation, this results in a quadratic dependence of the intensity distribution in the y direction perpendicular to the major axis. It can be seen that N E =n D πab is the total number of emitting fluorophores, where πab is the area of ​​the ellipse.

[0170] An optical system, such as that described in detail in the Sirat '185 patent, is provided which is suitable for moving the position of the vortex in a plane with high precision, the position being given as (v, 0), the polar coordinate symbol a referring to the radial vector in a specified transverse plane relative to the z-axis, and b referring to the predetermined axis ( Figure 2C Without loss of generality, due to the symmetry of the problem, we assume that the vortex is located on the x-axis, that is, the angle is zero.

[0171] Under the assumption that the distribution is consistent with the main semi-axis b( Figure 2D The intensity of this distribution as a function of position y is given by the coefficient i S The distance from the y-axis is quadratically related and has nothing to do with x.

[0172] The equation of the ellipse is:

[0173] x=a cos(t) and y=b sin(t) Equation (4)

[0174] At a given Cartesian position (x,y), the intensity is i S(y-y0) 2 , is a quadratic parameter.

[0175] At a given x=A position, the integrated energy along the y-axis is given by:

[0176]

[0177] where A lies on the ellipse, and as mentioned above, A = a sin(t). The integral over x yields two terms, the first of which is I y0 , which depends quadratically on y0:

[0178]

[0179] Again using equation (4) and its derivative dx = b sin (t) dt. The first term I y0 Depends on y0:

[0180]

[0181] This is the energy obtained for an infinitesimal point, and the total number of emitting fluorophores at the origin is N. E Item 2 I a Depends on the semi-axis value via the following equation:

[0182]

[0183] Again using equation (4) and its derivative dx = b sin (t) dt. The second term I y0 It depends on the semi-axis a by:

[0184]

[0185] The above is the energy depending on the semi-axis a of the ellipse. Use the energy corresponding to the minor axis ( Figure 2D The y-axis in the Figure 3 A similar measurement of the half-moon distribution of case 24 will produce a term that depends on the major axis, and comparison of these two terms will allow the measurement of the ellipticity.

[0186] Evaluation of the position of two points located on two different planes

[0187] According to further embodiments of the present invention, the techniques described herein may be used to measure the relative position of two points in space even when the two points are located in two different planes.

[0188] Reference now Figure 4 , the relative position of two points located in two different planes is a difficult problem from a practical point of view and is at the heart of one of the most important metrology steps in semiconductor processing: overlay metrology. Assume that the moon is crescent-shaped, such as Figure 3In case 02, the trajectory of the single distribution is a spiral 401. The geometric shape embedded in the distribution includes the spiral, and also includes multiple points located along the spiral. If two points are in two different planes 403 and 405, but on the same spiral, the relative position of one point with respect to the other point can be monitored along the spiral, and the position of two points on two different planes can be evaluated with theoretically infinite resolution.

[0189] Implementation of Conical Diffraction Method

[0190] Conical diffraction may also be advantageously used to implement methods according to embodiments of the present invention using the teachings of Caron, J. et al., “Conical diffraction illumination opens the way for low phototoxicity super-resolution imaging,” Cell Adhesion & Migration 8(5): 430-439 (2014), which is incorporated herein by reference.

[0191] Now refer to Figure 5 An embodiment of the present invention is described. A method, generally represented by the numeral 500, is provided for determining the value of an intrinsic geometric parameter of a geometric feature that characterizes a specific dimension of a physical object. To perform the claimed method, CODIM or metrology application hardware as described above is provided (step 501), and a single light distribution is projected (503) onto the physical object. When impinging on a detector, light excited by the single light distribution that has interacted with the geometric feature is detected (505). The return energy distribution is identified and quantified (507) at one or more locations of the single light distribution as a quantification of the intrinsic geometric parameter. Finally, the value of the intrinsic geometric parameter is determined (509) based on the value of the intrinsic geometric parameter.

[0192] Now refer to Figure 7 The use of standard deep learning concepts on CODIM imaging is described. A biological object 20, graphically represented as a bacterium, is illuminated by a series of localized single distributions 21, detected by a suitable low-noise imaging detector 22. A series of micro-images 23 (also referred to herein as "μ images") are recorded and processed by a suitable algorithm 25 to produce a reconstructed super-resolution image 10. A deep learning network comprising a plurality of layers 11, 12 and 13 is trained in a manner similar to Figure 6 The process described in is applied to reconstruct the image. A control system 24 is used to control and synchronize acquisition and processing.

[0193] Reference now Figure 8The use of direct deep learning in CODIM imaging is described. A biological object 20, graphically represented as a bacterium, is illuminated by a series of localized unitary distributions 21, which are detected by a suitable low-noise imaging detector, 22 also referred to herein as the "detector". The series of μ images 23 is directly applied without reconstruction to a deep learning layer 26, which may be different from the layers used in the standard process. Additional layers 12 and 13, different from the layers used in the standard process, are applied on the results of the first layer. A control system 24 is used to control and synchronize the acquisition and processing.

[0194] Now refer to Fig. 9 The use of standard deep learning concepts on a composite imaging system is described. A general object 120, illustrated for explanatory convenience as a cat, is illuminated by a series of localized single or regularly distributed images 21 detected by a detector 22. A series of micro-images 23, also referred to herein as "intermediate images", are represented as sub-images of a portion of the cat frame, but it is only a graphical trick. This series of μ images 23 is recorded and processed by a suitable algorithm 25 to produce a reconstructed super-resolved image 10. A deep learning network comprising a plurality of layers 11, 12 and 13 is processed in a manner similar to Figure 6 The process described in is applied to reconstruct the image. A control system 24 is used to control and synchronize acquisition and processing.

[0195] Now refer to Fig.10 The use of direct deep learning in a general composite imaging situation is described. A general object 120, represented by a cat, is illuminated by a series of localized single or regularly distributed 21 detected by a detector 22. A series of μ images 23 or intermediate images are represented as sub-images of a portion of the cat picture, but this is only a graphical trick. The series of μ images is applied directly to a deep learning layer 26 without reconstruction, which can be different from the layers used in the standard process. Additional layers 12 and 13, different from the layers used in the standard process, are applied on the results of the first layer. A control system 24 is used to control and synchronize the acquisition and processing.

[0196] Now refer to Fig.11The use of controlled direct deep learning in CODIM imaging is described. A biological object 20, graphically represented as a bacterium, is illuminated by a series of localized single distributions 21; the sequence of distributions including their position, intensity, type, duration and timing is determined by a control system based on any type of external information 30 and / or partial or final information from processing. The light distributions emitted by the object illuminated by the different light distributions are detected by a suitable low-noise imaging detector 22. The series of μ images 23 is directly applied without reconstruction to a deep learning layer 26, which can be different from the layers used in the standard process. Additional layers 12 and 13, different from the layers used in the standard process, are applied on the results of the first layer. A control system 24 is used to control and synchronize the acquisition and processing, and to determine the sequence of distributions.

[0197] Now refer to Fig.12 The use of controlled direct deep learning in a general composite imaging situation is described. A general object 120, graphically represented as a cat, is illuminated by a series of localized single or regular distributions 21. A series of μ images 23 or intermediate images are represented as sub-images of a portion of the cat picture, but this is only a graphical trick. The sequence of distributions including their position, intensity, type, duration and timing is determined by a control system based on any type of external information 30 and / or partial or final information from processing. The light distributions emitted by the object illuminated by the different light distributions are detected by a detector 22. The series of μ images is applied directly to a deep learning layer 26 without reconstruction, which can be different from the layers used in the standard process. Additional layers 12 and 13, different from the layers used in the standard process, are applied on the results of the first layer. A control system 24 is used to control and synchronize the acquisition and processing, and to determine the distribution sequence.

[0198] Classifying scenes using deep learning

[0199] Features generally associated with methods according to embodiments of the present invention include:

[0200] - a physical or biological scene consisting of multiple luminescent biological objects, each object being modeled as a luminescent geometry quantified by an appropriate set of parameters,

[0201] - Hardware setup, such as the imaging case described in Caron 2014 or the metrology case described above;

[0202] - Groups of measurements, each consisting of the projection of a single distribution at a nominal position, either on a regular or pseudo-regular grid, called the "imaging case", or on a customized grid, called the "metrology case"; in the metrology case, the single distributions are chosen so that they are embedded in the geometry, so that, with nominal parameters, the measured energy will be zero.

[0203] - detecting light constituting the detected light from a single light distribution that has interacted with the geometrical feature and hits the detector (pixelated or non-pixelated);

[0204] - A deep learning or neural network layer, using the detected light directly as input to the neural network layer, is able to classify the scene into multiple shapes, static or dynamic, that are part of a family of basic shapes that are predetermined or acquired through learning.

[0205] According to other embodiments of the present invention, the direct and dynamic methods may also provide control hardware and control programs that dynamically or non-dynamically control the sequence of measurement and detection light to collect sets of improved measurements required to retrieve shape parameters by analyzing element data through standard or deep learning programs.

[0206] Details of Deep Learning Implementation

[0207] According to embodiments of the present invention, methods are provided that use the specificity and advantages of deep learning to analyze imaging and measurement data obtained by CODIM-cone diffraction microscopy (as described in Caron 2014), or hardware for implementing the above-mentioned metrology methods is provided.

[0208] Based on these methods, a new system for optical measurement of geometric features is described herein, and more specifically, methods for measurements with a resolution greater than the diffraction limit imposed by the optics and a minimum photon flux are described.

[0209] Example

[0210] One embodiment of the present invention can be illustrated by the following example; this example is for illustration only and is not intended to represent the experimental conditions tested. Assume that the luminescent object is a biological object, and for illustration, the biological object is the same influenza A virus (IAV), such as Figure 2A As shown; the virus is known to have a typical size of 80-120nm and is also known to be spherical in most cases, but can also be filamentous in some cases. Badham et al., "Filamentous Influenza Viruses", Current clinical microbiology reports, Vol. 3, pp. 155-61 (2016) (incorporated herein by reference) states that "the biological importance of IAV morphology is a subject of great interest in human clinical infection." It is assumed that the IAV virus is uniformly labeled with some sufficient fluorescent protein throughout its volume. Several models can be used to describe the virus; the virus is described as being modeled as an ellipse ( Figure 2D ) with the semi-minor axis a, semi-major axis b, and the angle θ of the major axis relative to the Cartesian reference frame.

[0211] According to an embodiment of the present invention, dynamic changes in virus location and morphology are assessed in order to potentially assess the virulence of the virus.

[0212] CODIM Examples

[0213] For imaging using deep learning, the CODIM system described in Caron 2014 can be applied, while for measurement, the setup described in the measurement application is preferred.

[0214] According to an embodiment of the present invention, methods are provided for analyzing imaging and metrology data obtained by CODIM-cone diffraction microscopy, as described in Caron 2014, using the specificity and advantages of deep learning.

[0215] Neural Network Example

[0216] Spiliotis et al., “Priming for destruction: septins at the crossroads of mitochondrial fission and bacterial autophagy,” EMBO Reports, Vol. 17, pp. 935-37 (2016) (“Spiliotis 2016”), incorporated herein by reference, teaches:

[0217] Mitochondria are essential organelles for cell survival, programmed cell death, and autophagy. They undergo cycles of fission and fusion, are disrupted by infectious pathogens, and are altered in many human diseases. Spiliotis 2016, 935.

[0218] Pagliuso et al., “A role for septin 2 in Drp1-mediated mitochondrial fission,” EMBO Reports, Vol. 17, pp. 857-73 (2016) (“Pagliusso 2016”), (which is incorporated herein by reference) shows that

[0219] Among the multiple mechanisms that appear to regulate mitochondrial fission…

[0220] Septin 2 plays an important role by mediating mitochondrial contraction.

[0221] Pagliuso 2016, Abstract, italic emphasis added.

[0222] In this study, the biological axial CODIM system was used to collect mitochondrial images and reconstruct the images.

[0223] Reconstructed images of mitochondria can be used as input to deep learning networks to gather information about biological states, such as Fig.10 Alternatively, the collected data can be fed directly into a deep learning network, such as Fig.11 shown.

[0224] The embodiments of the present invention described herein are exemplary only; changes and modifications will be apparent to those skilled in the art. All such changes and modifications are within the scope of the present invention as defined by the appended claims.

Claims

1. A method for determining the value of a non-position-intrinsic geometric parameter characterizing a geometric feature of a particular dimension of a physical object, the method comprising: a. projecting a single light distribution characterized by a first wavelength and a singularity position onto a physical object; b. detecting light excited by a single light distribution, the light excited by a single light distribution having interacted with the geometric features and impinging on the detector, the light constituting the detected light; c. Identifying and quantifying the return energy distribution at one or more locations of a single light distribution as a quantification of intrinsic geometric parameters; as well as d. Determining values ​​of intrinsic geometric parameters based on the values ​​of the intrinsic geometric parameters.

2. The method according to claim 1, wherein: The intrinsic geometric parameters are the dimensions of the point objects.

3. The method according to claim 1, wherein: The intrinsic geometric parameter is the ellipticity of the point object.

4. The method of claim 1 further comprising determining a measure of displacement as an offset transverse to the line.

5. The method according to claim 1, wherein: The intrinsic geometric parameter is the width of the line.

6. The method according to claim 1, wherein: The intrinsic geometric parameters are based on a model shape of the geometric feature.

7. The method according to claim 1, wherein: Detecting the light includes using a pixelated detector.

8. A method of performing successive applications of the method of claim 1 in a manner that retrieves individual or groups of intrinsic geometrical parameters.

9. A method for representation learning for classifying a scene into at least one geometric shape, static or dynamic, quantified by a suitable set of parameters, each geometric shape modeling a light emitting object, the method comprising: a. Project a single distribution of light onto the scene; b. detecting a light distribution re-emitted by the scene when illuminated by the single light distribution, the single light distribution having interacted with each luminous object and impinging on a detector, the detected light constituting the detected light; c. measuring at least one projection of a single distribution at a given location to obtain a set of measurements about the scene; as well as d. Employing a deep learning or neural network layer, using the detected light as a direct input to the neural network layer, adapted to classify the scene into multiple shapes, static or dynamic, that are part of a set of shapes that are predetermined or acquired through learning.

10. The method according to claim 9, wherein: The at least one geometric shape is dynamic.

11. The method according to claim 9, wherein: The at least one geometric shape is static.

12. The method of claim 9, further comprising acquiring the data as imaging data acquired by CODIM hardware.

13. The method of claim 9, further comprising acquiring the data as metering data acquired by metering application hardware.

14. The method according to claim 9, wherein: Also included is acquiring data as imaging data acquired by CODIM hardware, wherein the resulting classification information is used to drive metrology hardware to implement the method of claim 1 .

15. The method of claim 9, further comprising providing control hardware and a control process to drive metering hardware to implement the method of claim 1.

16. The method of claim 9, further comprising feeding data to the neural network, the neural network having been acquired at different times, and the method including temporal correlation of luminous objects in the neural network recognition process.

Images