Super-resolution metrology based on singular distribution and deep learning

Abstract

To provide a method that determines values of intrinsic geometric parameters of geometric features characterizing a physical object, and models a light emitting object by each geometric shape.SOLUTION: Singular light distribution characterized by a first wavelength and a singularity location is projected onto a physical object. Light excited by the singular light distribution that interacts with geometric features and collides with a detector is detected, and returned energy distribution at one or more locations is identified and quantified. A deep learning or neural network layer may be used, which uses detected light as a direct input to a neural network layer, and is adapted to classify scenes as a plurality of static or dynamic shapes, where the shape is part of a set of shapes predetermined or obtained by learning.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] This application claims priority to two U.S. Provisional Application Nos. 62 / 551,906 and 62 / 551,913, both of which were filed on August 30, 2017, and both of which are incorporated herein by reference.

[0002] The present invention relates to a method and apparatus for optical measurement of geometric features using machine,representation and deep learning, and more particularly, to optical,measurement using projected singular light distributions. [Background technology]

[0003] Metrology is the technical field of measurement. Measurements based on fluorescence, multiphoton imaging, Raman scattering, optical transmission, reflection, or light scattering are practically limited in resolution to the limit stated by Ernst Abbe in 1873. The Abbe limit arises due to diffraction by the aperture stop of the optical system used to illuminate or collect light from the sample, as explained below.

[0004] Some biological and life science objects can be mathematically modeled as elementary geometric shapes. Accurate quantification of the parameters of such models is one of the challenges of modern biology and life sciences, revealing hidden information about biological objects and their functions, either directly or statistically. Sirat in U.S. Pat. No. 9,250,185 (hereinafter "Sirat '185 patent"), incorporated herein by reference, defines a "luminous biological object" as a "map" of a physical object, "... in the general semantic sense." (Sirat '185 patent, column 18, lines 15-22)

[0005] Measuring the shape or other geometric attributes of an object consists of applying parameterized shape assumptions, which we call mathematical priors, to elemental and / or reconstructed data from a "luminescent biological object" and obtaining the shape parameters from the data. Yet another mathematical constraint exists on the data: positivity, which is the inherent positivity of the light intensity.

[0006] In the prior art, both the mathematical prior constraints and the positivity constraints rely on a traditional imaging paradigm: after image acquisition and reconstruction, the mathematical prior constraints and the positivity constraints need to be applied to the obtained image as a post-processing step. This imaging paradigm can be traced back to the earliest days of microscopy, when shape measurements were considered as epiphenomena of the image, rather than the image itself.

[0007] The shape assumption is a powerful mathematical prior that, as the invention is described below, significantly reduces the number of degrees of freedom of the solution and creates an embedded parametric space for the solution. As used herein, the term "mathematical prior" refers to a mathematical assumption about the measurement results that reduces the number of degrees of freedom of the problem at hand. Applying a mathematical prior after the acquisition and reconstruction process, as has been done in the past, for data acquired using common imaging paradigms, reduces the accuracy of the entire metrology process. Historical perspective

[0008] Ernst Abbe, who lived from 1840 to 1905, would have been comfortable using the latest generation of biological microscopes for imaging and measurements after completing his training in the new hardware and software. The permanence of biological imaging and metrology is due, among other things, to the quality of the concepts developed by our predecessors, but also to the difficulty of changing ingrained working procedures.

[0009] However, several new trends are challenging well-established archetypes of microscopy, even in biology. The main evolutions arise from the desire to precisely quantify fundamental parameters of models of biological objects in comprehensive statistical ways, and not to base biological observations solely on images or assumptions.

[0010] J. Shamir, “Singular beams in metrology and nanotechnology,” Opt. Eng . , vol. 51, 073605 (2012) and U.S. Pat. No. 7,746,469 (both incorporated by reference) propose the use of differential light distributions to quantify particle size distributions.

[0011] To the inventor's knowledge, it has not been proposed to create a null image that does not use photons per se, to achieve a resolution that cannot be achieved where light per se is imaged. In particular, Shamir's teachings use elastic interactions such as reflection, transmission, and (elastic) scattering. It is advantageous to obtain information based on the null image, and the invention described in detail below shows how such imaging can be performed. Background: Measurement of parameters of technical geometries

[0012] It is not necessary to review all measurement systems and methods used in biological microscopy, but in theory, the measurement limit for any parameter, including those describing shape, is ultimately determined by signal-to-noise and may be quantified using the Cramer Rao lower bound, CRLB, defined below. However, for shape-related parameters, the calculations are ill-conditioned and rely on measurements that are not precise due to practical and experimental conditions.

[0013] See, e.g., Gustafsson, “Nonlinear structured-illumination microscopy: wide-field fluorescence imaging with theoretically unlimited resolution,” incorporated herein by reference. Proc. Nat. Acad. Sci., vol. 102, pp. 13081-86 (2005) presented nonlinear structured illumination microscopy, a technique for super-resolution, combining nonlinear saturation effects with structured illumination. Gustafsson's technique is considered exemplary by those skilled in the art of microscopy, and in the best case demonstrated a resolution of 50 nm. The measurements were performed on calibrated, isolated, planar, identically manufactured beads that could sustain high energy levels and were not susceptible to photobleaching or phototoxicity, which were chosen by Gustafsson to have a precise diameter of 51 nm, a bead of about the same size as the limit of resolution.

[0014] Based on the aforementioned data, the effect of bead size and the simplest possible measurement of shape parameters calculated with an accurate procedure ("known bead shape was removed from the reconstruction by linear deconvolution"), Gustafsson 2004 found that the accuracy was less than 10% of the "system ruler" defined in his experiment. If the same experiment were performed as a procedure to quantify the unknown diameter value of the beads, the relative accuracy would be 5%, a relatively poor performance in this ideal case. This indicates the desirability of improved tools for the measurement of shape parameters as described below according to the invention. Deep Learning Background

[0015] A wealth of literature surrounds deep learning. The reader is directed to Wikipedia, https: / / en.wikipedia.org / wiki / Deep_learning, and the references cited therein. A synthesis of deep learning concepts is incorporated herein by reference and defines several fundamental concepts that will be used below: LeCun et al., “Deep learning,” Nature, vol. 521, pp. 436-45 (2015) (hereinafter referred to as “LeCun2015”). According to LeCun2015, Representation learning is a set of methods that allow a machine to be fed raw data and automatically discover the representations required for detection or classification. Deep learning methods are representation learning methods with multiple levels of representation, obtained by composing simple but non-linear modules, each of which transforms the representation of one level (starting from the raw input) to a higher, slightly abstract level of representation. By composing enough such transformations, very complex features can be learned. For classification tasks, higher layers of representations amplify aspects of the input that are important for differentiation and suppress irrelevant variations. For example, an image is provided in the form of an array of pixel values, and the features learned in the first layer of the representation typically represent the presence or absence of edges at specific orientations and positions in the image. Typically, the second layer detects motifs by spotting specific arrangements of edges, regardless of slight variations in edge positions. The third layer is able to assemble motifs into larger combinations that correspond to parts of familiar objects, and subsequent layers detect objects as combinations of these parts. A key aspect of deep learning is that these layers of features are not designed by human engineers, they are learned from the data using generic learning procedures. (LeCun 2015, p. 436, emphasis added)

[0016] Prototypes of deep learning networks currently in practice include processing acquired images. Figure 6 is an example of a prior art deep learning network following the teachings of LeCun 2015. An image 10, shown as a picture of a cat, is the input of the network. The first layer 11 typically learns the presence or absence of edges at specific orientations and locations in the image. Additional layers, represented diagrammatically by numbers 12 and 13, trained from the data using a generic training procedure, may identify cat breeds that are not dogs.

[0017] As used herein, "CODIM" refers to the method described in Caron, et al. (2014). "Conical diffraction illumination opens the way for low phototoxicity super-resolution imaging.", which is incorporated herein by reference. Cell adhesion & migration , vol. 8, pp. 430-439 (2014) (hereafter referred to as "Caron2014"). The use of the hardware described in Caron2014 is referred to as the "imaging case". The setup and use of the hardware described below as the "metrology case" is not part of the prior art. In the imaging case, the singular light distribution is projected onto a regular grid, which is a Cartesian coordinate, or onto a grid optimized as a function of the acquired data.

[0018] The standard way to apply CODIM in deep learning or machine learning networks is to reconstruct an image and then apply a known algorithm to the reconstructed image. The limitations of such an approach are the time burden and information corruption involved in reconstructing the image.

[0019] The Sirat '185 patent states: The measurement system calculates an evaluation of the fluorophore descriptors, a measurement map, which may differ from the original map due to noise, measurement conditions, system limitations, or measurement uncertainties. This information map can later be expanded to different abstraction levels. (Sirat '185 patent, column 18, lines 37-42, emphasis added)

[0020] The Sirat '185 patent further teaches: The primitive information is a map of common semantic terms, a set of descriptors fluorophores and their evolution over time. The biological and geometric information are just extrapolations of this primitive information. The measurement system calculates the evaluation of the descriptors of the fluorophores, the measurement map. (Sirat'185 patent, column 18, lines 32-37)

[0021] Sirat '185 assumes that "the measurement system...computes an estimate of the measurement map" (ibid., emphasis added) and sees the measurement map as a prerequisite for any information gathering. In requiring a measurement map as a prerequisite for information gathering, Sirat '185 follows the general consensus that a "measurement map" is a prerequisite for further processing at the geometric and biological levels of abstraction described in Sirat '185 in the same paragraph. Summary of the Invention [Means for solving the problem]

[0022] According to an embodiment of the present invention, there is provided a method for determining values ​​of intrinsic geometric parameters of a geometric feature of a particular dimension characterizing a physical object, the method comprising: a. projecting a singular light distribution characterized by a first wavelength and a location of a singularity onto a physical object; b. detecting light excited by the excited unique light distribution that interacts with a geometric feature and impinges on a detector, the light comprising detected light; c. identifying and quantifying a return energy distribution at one or more locations of said unique light distribution as a quantification of a characteristic geometric parameter; d. determining values ​​of said intrinsic geometric parameters based on parameters obtained from the energy distribution.

[0023] According to another embodiment of the invention, successive applications of the above method are performed in such a way as to search for a single or set of intrinsic geometric parameters, which are either the size and ellipticity of the point object, and a displacement measure that is an offset across the line may be determined.

[0024] According to further embodiments of the present invention, the intrinsic geometric parameter may be a line width. The intrinsic geometric parameter may be based on a model shape of the geometric feature. Detecting light may include using a pixelated detector.

[0025] According to another aspect of the present invention, there is provided a method for representation learning for classifying a scene into at least one geometric shape, static or dynamic, quantified by a suitable set of parameters, each geometric shape modeling a luminous object. The method comprises: a. projecting a singular distribution of light onto a scene; b. detecting a light distribution re-emitted by said scene upon illumination by a singular light distribution interacting with each light-emitting object and impinging on a detector, said detected light constituting a detected light; c. measuring at least one projection of the singular distribution at a particular location to obtain a set of measurements for the scene; d. using a deep learning or neural network layer adapted to classify the scene as a plurality of physical or biological objects, or as static or dynamic geometric shapes, using the detected light as a direct input of a neural network layer, wherein the objects or shapes are part of a set of objects or shapes pre-determined or obtained by learning.

[0026] According to other embodiments of the invention, the at least one geometric shape may be either dynamic or static.The method may also include acquiring data, the data being imaging data acquired by CODIM hardware or metrology data acquired by metrology application hardware.

[0027] According to a further embodiment of the invention, the method may have the further step of acquiring data, which is imaging data acquired by CODIM hardware, and the acquired classification information is used to drive said metrology hardware to perform the metrology method. A further step may include providing control hardware and control procedures for driving the metrology hardware to perform the metrology method. A further step may further include feeding data acquired at different times to a neural network and including the time dependency of the luminous object in the neural network recognition process. [Brief description of the drawings]

[0028] The foregoing features of the present invention will be more readily understood by reference to the following detailed description taken in conjunction with the accompanying drawings, in which:

[0029] [Figure 1] The prior art Modulation Transfer Function (MTF) of the CODIM light distribution compared to the Airy pattern is shown.

[0030] [Diagram 2] 2A-2D represent influenza type A virus (IAV), which is used as an illustrative example in this description, with FIG. 2A showing a schematic representation of the virus, FIG. 2B showing a three-dimensional model thereof, and FIG. 2C and FIG. 2D showing two-dimensional models for isotropic and anisotropic models, respectively.

[0031] [Diagram 3]It is reproduced from U.S. Pat. No. 9,250,185 (hereinafter "Sirat '185") and shows some of the various singular distributions that can be utilized using conical diffraction.

[0032] [Figure 4] 1 shows a schematic diagram of a crescent trajectory, known to be a spiral, according to an embodiment of the present invention, and planes A and B intersecting the spiral at two points having different lateral positions.

[0033] [Diagram 5] 3 is a flow chart illustrating method steps for representation learning according to one embodiment of the present invention.

[0034] [Figure 6] 1 illustrates a schematic representation of a prior art implementation of a deep learning system.

[0035] [Figure 7] 1 illustrates a schematic of the implementation of a deep learning system on a CODIM imaging system.

[0036] [Figure 8] 1 illustrates a schematic of an implementation of a direct deep learning network on a CODIM imaging system in accordance with the present invention.

[0037] [Figure 9] 1 illustrates a schematic of an implementation of a direct deep learning network on a complex imaging system according to the present invention.

[0038] [Figure 10] We show a schematic implementation of a direct deep learning network on a generic object, represented for heuristic reasons as a cat. The fact that the intermediate image is represented as a sub-image of a part of a picture of a cat is merely a graphical technique: the information is unstructured, not a partial representation of an image.

[0039] [Figure 11]1 illustrates the implementation of a controlled direct deep learning network on a CODIM imaging system in accordance with the present invention.

[0040] [Figure 12] We present the implementation of a controlled direct deep learning network on a generic object, represented for heuristic reasons as a cat. The fact that the intermediate image is represented as a sub-image of a part of a picture of a cat is merely a graphical technique: the information is unstructured, not a partial representation of an image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0041] Here we present a novel methodology where the measurement procedure is a separate dedicated procedure and architecture, possibly initialized by the imaging procedure, but fully integrated into the metrology tool, which is a separate tool.

[0042] The invention described below is directed to accurately measuring geometric features that are smaller than the diffraction limit of the light used to measure them, with minimal light flux.

[0043] The present invention is particularly suited to measuring geometric features using inelastic light interactions such as, but not limited to, fluorescence, multiphoton imaging, or Raman scattering, where the emerging light can be separated from the incident light by simple means.

[0044] Some embodiments of the present invention relate to methods for machine learning of geometric and functional features and objects, and hardware systems for implementing them. These methods require classification and recognition of features and objects. These methods differ from real solutions based on imaging paradigms and further processing following imaging.

[0045] Although some embodiments of the invention are described with reference to systems and methods implemented primarily in fluorescence microscopy, it will be understood that extension to reflected or scattered light is straightforward and within the purview of one of ordinary skill in the art once apprehended by the present description. Furthermore, although the invention is described herein with respect to systems and methods for metrology tools for fluorescent objects encountered in biology, they may be readily applied to other imaging modalities, such as, but not limited to, semiconductor and machine vision, as well as other disciplines and domains.

[0046] It would be useful to apply a mathematical prior before the reconstruction process and embed it in the acquisition process, but there is no teaching as to how this can be done, which is now described in detail. Application of a mathematical prior before image reconstruction, in accordance with embodiments of the present invention, may advantageously increase the accuracy of measurements of shape parameters, reduce the number of observations required for a measurement, and increase the stability and repeatability of the overall process.

[0047] The method described according to the embodiment of the present invention relies on the peculiarities of Poisson's law, the apparent absence of photon noise - to a negligible number, according to the theory of quantum physics - and the absence of photons incident and emitted at the detector, not post-correction. Such a method requires a mechanism to physically remove the incident light, by a filtering operation. It cannot be a mathematical post-processing operation, since noise cannot be filtered by simple means if it is already introduced into the system. This is the advantage and peculiarity of inelastic light interactions, which in the present invention refer to any light interaction that can create (or emit) a new photon with physical properties that distinguish it from the incident photon, such that the emitted photon can be physically separated from the incident photon. It is assumed that a "filter" can separate the emitted photon from the incident photon. An obvious example of an inelastic light interaction is an interaction that produces photons at different wavelengths, in which case the filter would be a wavelength-sensitive filter. Examples of inelastic light interactions are fluorescence, multiphoton imaging or Raman scattering. Elastic light interactions such as reflection, transmission, elastic scattering do not allow such discrimination in a simple way.

[0048] As explained below, embodiments of the invention rely on the creation of a null image, or a near null result. Such an image allows theoretically unlimited additional resolution by combining a loophole in Abbe's law with the absence of noise due to Poisson's law, as explained below. A prerequisite is the absence of spurious photons, which is necessary to satisfy theoretical conditions. Such conditions can be approximately met only when using inelastic light interactions such as fluorescence, multiphoton interactions, and Raman scattering, and the incident beam can be fully filtered by spectral means.

[0049] For heuristic convenience, the present invention may be described herein by a hypothetical example. This example is used merely as an illustration and is not intended to represent a tested experimental case. A specific biological object is assumed to constitute a luminous object, and this biological object is assumed to be an influenza type A virus (IAV) for the purpose of illustration, which is represented diagrammatically in FIG. 1A. IAV viruses are known to have a typical size of 80-120 nm, and are described in “The biological significance of IAV morphology in human clinical infections is a subject of great interest,” in Filamentous Influenza Viruses, by Badham et al., incorporated herein by reference. Current clinical microbiology reports , vol. 3, pp. 155-161, (2016).

[0050] The IAV virus is assumed to be uniformly marked throughout its volume with an appropriate fluorescent protein. Three different models can be used to describe the virus: In the imaging paradigm used in the prior art, due to the small size of viruses, under the diffraction limit, the best model is to represent the viruses as fluorescent light spots, which in fact allows for accurate monitoring of the location of the viruses. b. Alternatively, electron microscopy, for example at nm or sub-nm resolution, can be used to characterize various details of the virus, including hemagglutinin 201 and neuraminidase 203. c. In accordance with the teachings of the present invention, an intermediate approach is adopted in which viruses are modeled in three dimensions as spheres (FIG. 2B) or in two dimensions as uniform circles with a given radius (FIG. 2C), defined as follows:

[0051] According to further embodiments of the present invention, deep learning may be used, as described in more detail below.

[0052] Methods according to embodiments of the present invention may advantageously provide measurements with resolution higher than the diffraction limit imposed by the optical system and minimum photon flux.

[0053] According to an embodiment of the invention, one or more mathematical prior distributions resulting from a shape assumption are embedded in a specific set of singular light distributions, allowing to make the most of the properties of the singular light distributions. This coupling is based on a dedicated metrology tool that associates the mathematical prior with a set of distributions. In other words, the measurement of a specific shape mathematically defines a prior domain and a solution domain that themselves determine the properties of the acquisition tool and the type of distribution required.

[0054] Definitions: As used in this specification and the appended claims, the following terms have the following specified meanings, unless the context requires otherwise.

[0055] The term "value" as used herein and in the appended claims is intended to refer to a real number that characterizes a quantity associated with a parameter. It should be understood that in practical contexts, the quantity associated with a parameter may be characterized within a certain range that constitutes the precision of the measurement. In that case, the term "value" may be used as shorthand for a distribution of values.

[0056] A "system ruler" is a value considered by those skilled in the imaging arts to characterize a system's ability to discern detail.

[0057] In imaging systems, the "Rayleigh criterion" - the ability to separate two adjacent points - is a commonly accepted criterion for the smallest resolvable detail, and is a qualitative term commonly used to quantify the smallest resolvable detail, even though the FWHM of the observed point or line is often used as a practical assessment of the "diffraction limit". In this invention, we use the FWHM of an infinitely thin line as the system ruler. Values ​​can be neglected if they are "much smaller" than the system ruler, with "much smaller" being defined as more than three times smaller.

[0058] As used herein, the term "Abbe resolution limit" refers to the "Abbe limit" of Schermelleh et al., "A guide to super-resolution fluorescence microscopy," which is incorporated herein by reference. J. Cell Biology , vol. 190, pp. 165-75 (2010) (hereinafter referred to as “Schermelleh 2010”). Abbe's famous resolution limit is fascinating because it simply depends on the maximum relative angle between the different waves that emerge from the object and are captured by the objective lens and sent to the image. It describes the smallest possible level of detail that can be imaged with this PSF "brush." ​​Details of periodic objects smaller than this shortest wavelength cannot possibly be transferred to the image.

[0059] The expression "beyond the Abbe limit" is defined to refer to objects that contain periodic structures with details smaller than the details of the system ruler, and are therefore limited by the Abbe limit. The rationale for this definition is that such objects contain spatial frequencies above the Abbe frequency circle of the aperture plane.

[0060] In estimation theory and statistics, the Cramer-Rao bound (CRB), or equivalently the "Cramer-Rao lower bound (CRLB)," represents a lower bound on the variance of an estimator of deterministic (unknown but fixed) parameters. The precise definition used herein is given at https: / / en.wikipedia.org / wiki / Cram%C3%A9r%E2%80%93Rao_bound, which is incorporated herein by reference.

[0061] As used herein, the term "localized" light distribution refers to a light distribution in which the energy is concentrated in a small area. A light distribution is localized if the energy outside a 3.5*radius of half the Rayleigh criterion is essentially zero.

[0062] The present description assumes that the described optical system is close to "photon noise limited" as described at https: / / en.wikipedia.org / wiki / Shot_noise, or is close to photon noise limited, i.e., the Gaussian noise component is less than half the photon (or shot) noise. In fact, the optimal case is the described "photon noise limited" optical system, while "Gaussian noise limited" or "detector noise limited" systems would only capture some of the benefits of the present invention, but would still be within the scope of the present invention.

[0063] "Full width at half maximum" (FWHM), from Wikipedia, https: / / en.wikipedia.org / wiki / Full_width_at_half_maximum, is an expression for the range of a function given by the difference between two extreme values ​​of the independent variable where the dependent variable is equal to half its maximum value.

[0064] "Dimension" is defined as any of three physical or spatial properties: length, area, or volume. In geometry, a point is said to be zero-dimensional, a figure that has only length such as a line has one dimension, a plane or surface has two dimensions, and a figure that has volume has three dimensions.

[0065] The dimensionality of a geometric feature shall refer to the dimensionality of the corresponding idealized feature in the limit much smaller than the range of any other dimension of the geometric feature (such as the "diameter" of a point object, the "width" of a line, the "thickness" of a coating, etc.) whose range tends to zero.

[0066] As used herein, the term "trajectory" is intended to denote a set of points (typically points, lines, line segments, curves, or surfaces) whose locations satisfy or are determined by one or more specified conditions.

[0067] "Geometric parameter" refers to a measurable quantity that characterizes an area in a physical space. Thus, for example, a distance (defined along a three-dimensional Euclidean space or surface) is one example of a geometric parameter. An area defined on a surface is another example of a geometric parameter.

[0068] A "geometric feature" is a physical object that is geometrically characterized or modeled as a locus, quantified by one or more geometric parameters, e.g., the thickness of a line represents a locus, a line is a locus, or the position and thickness of a line segment is also a locus. An "embedded space" refers to the parametric space that the geometric parameters span.

[0069] A "point" is a geometric feature in two or three dimensions that has zero dimension and size. It is an exaggerated simplification and erases a lot of information about the real object, but it is a very simplifying assumption. We call an object a "point object" if it is small in size, but not negligible, in all three dimensions compared to the system ruler. The terms small or negligible must be understood in comparison to the system ruler, which in the case of optical systems is diffraction limited. A point object is determined by its location and size, and size may or may not be isotropic in two or three dimensions. Most biological objects are point objects in the system ruler of diffraction-limited or super-resolution optical systems, and a priori dissolution of the information carried by a point object is a major loss. In this invention, the distinction between points and point objects is very important.

[0070] A "line" (and similarly other terms that refer theoretically to one-dimensional shapes...) refers to a geometric feature (i.e., a physical object that has a length, width, and thickness) whose length is at least 10 times its width or thickness. A "line object" is defined according to the same concepts as a point object.

[0071] The term "intrinsic geometric parameter" as used herein and in the appended claims is intended to refer to a parameter that characterizes a property of a surface (in n-dimensional space) that is independent of the isometric embedding of the surface in Euclidean space. Thus, for example, the width of a "line" through a given point is an intrinsic parameter of the line, regardless of any orientation of the line in space.

[0072] Referring to geometric elements of an abstract space, a set of parameters should be called "suitable" if and only if the representation of the geometric elements described by the set of parameters faithfully represents the geometric class and parameters of the object at the limits of measurement uncertainty.

[0073] "Singular optics," which includes "optical vortices" as its simplest example, is today an emerging area of ​​optics with theoretical and practical applications. For further details, see Nye, et al., "Singular Optics," which includes optical vortices as its simplest example, and which are both incorporated herein by reference. Dislocations in wave trains . Proceedings of the Royal Society of London A, vol. 336 , pp. 165-90 (1974), and Soskin, et al., ''Singular optics,'' Program in Optics , vol. 42, pp. 219-76.

[0074] "Inelastic light interactions" refer to interactions of light with matter that produce photons of a different wavelength than the incident beam. Inelastic light interactions include, but are not limited to, fluorescence, multiphoton interactions, and Raman scattering.

[0075] A "singular distribution locus" is a set of Cartesian locations where the intensity of the singular distribution is zero. A singular distribution locus is "nominal" with appropriate parameters, defining a family of basic shapes in the right locations, where "nominal locations" do not emit (or reflect or scatter) light. In this case, we create a new concept to express that "singular light distributions embed shapes". For example, in the case of a light distribution shown as a vortex, case 10 (or 01) in Figure 2, the locus is a very small point, and the family of shapes that can be explored using the vortex is only point shapes and point object shapes, which do not generate energy at a radius of value zero. In the case of the distribution described as case 42 (or 24) in Figure 2, named "vertical half moon", the family of shapes involved is more diverse, including not only lines and line objects, but also multiple point shapes and point objects. Additional cases will be developed below.

[0076] Conical refraction is described in Hamilton, ''Third supplement to an essay on the theory of systems of rays,'' Trans. Royal Irish Acad., vol.17, pp. 1-144 (1837), and Lloyd, Lloyd, H. (1831). ''On the Phenomena Presented by Light in Its Passage along the Axes of Biaxial Crystals,'' Philos. Mag., vol. 1, pp. 112-20 (1831) experimentally confirmed two months later. Both of the aforementioned references are incorporated herein by reference. Conical refraction describes the propagation of a ray of light in the direction of the optic axis of a biaxial crystal. Hamilton predicted that light would appear in the form of a hollow cone-shaped ray of light.

[0077] The description is given in Berry, "Conical diffraction asymptotics: fine structure of Poggendorff rings and axial spike," which is incorporated herein by reference. J. Opt. A: Pure and Applied Optics , vol. 6, pp. 289-300 (2004). A prior art system based on conical diffraction for super-resolution microscopy is described in Sirat 2016.

[0078] US Patent No. 8,514,685 (Sirat'685) described certain properties of conical diffraction in thin crystals and demonstrated its use to shape conical beams. A thin biaxial crystal transforms the Point Spread Function (PSF) of a normal incident beam into a wide range of light distributions, the choice of distribution being controlled by the input and output polarizations. In practice, beam shaping is achieved by enclosing the biaxial crystal between controllable polarizers. This simple optical setup, similar to a polarimeter, can switch from one pattern to another with different topologies in microseconds or even faster. Moreover, these patterns coexist perfectly, since they are generated by the same primary light beam.

[0079] The CODIM beam shaper can be used as an add-on to the scan module, where the distribution is scanned over the sample and multiple micro-images are generated for each scan point. The CODIM system is available from BioAxial SAS, Paris, France.

[0080] The CODIM beam shaper uses the principle of conical diffraction to generate compact and localized light distributions. Each micro-image contains a large amount of high frequencies close to the Abbe limit (up to a factor of 3 compared to the Airy pattern 101, as shown in Figure 1). These light distributions, projected onto an area of ​​the sample, called the "scene", are analyzed using algorithms well known to those skilled in the art. For examples of algorithms suitable for deriving super-resolution images from micro-images, the reader is referred to published application WO2016 / 092161 (published in English as US2017 / 0336326, hereinafter the "Sirat'161 application"), which is incorporated herein by reference. This allows the reconstruction of super-resolution images with an improvement in resolution of up to 2x for typical objects. Moreover, these algorithms exploiting positivity constraints and sparsity may further improve the resolution for suitable samples.

[0081] Finally, the combination of a much lower distribution peak power, the use of a high quantum yield camera, and longer camera exposure times significantly reduces the peak power - and energy - of the light delivered to the sample. This is the reason for the very low photobleaching and phototoxicity of this method. This also avoids the problem of fluorophore saturation, making the method linear and quantitative.

[0082] For the purposes of this description, the term "energy law" is defined as follows: assuming that an object is modeled as a mathematical abstraction, a geometric shape, an "energy law" is a parametric relationship between energy as a function of shape parameters and position. It creates a relationship that quantifies the energy dependence in parametric space. The energy law may include the energy distribution emitted from a light-emitting object of identical shape to the geometric shape.

[0083] In this description, it is assumed that the optical singular distribution can be controlled from a given distribution family in such a way as to switch from one type of distribution to another, and that the parameters of the optical singular distribution can be changed using external means, as described in Sirat '185. Other solutions exist and do not require control of the optical singular distribution and are part of the present invention, but they would be much more difficult to handle.

[0084] Imaging performed using the method according to the invention may be called "composite imaging", coded imaging as defined in Wikipedia, https: / / en.wikipedia.org / wiki / Coded_aperture, or "indirect imaging" for imaging processes in which elemental data from a scene are not images or maps, in the sense of the Sirat '185 patent describing an object. In a composite imaging process, data measured by a physical instrument is called an "intermediate result" (a term used in the Sirat '161 application), or the elemental (or basic) data, used synonymously in this description, includes information from which a reconstructed image can be created through a reconstruction process. In this invention, we identify a composite imaging process as an imaging process in which it is not possible to assume the intensity or number of photons measured at a particular time and position for a particular pixel of an image or a particular attribute of an element in a map. In simple terms, in a composite imaging system, the intensity or number of photons is not directly converted to pixel data, but some of them contain information related to them.

[0085] In the realm of super-resolution microscopy, CODIM imaging is an example of a combined imaging process referenced in the Sirat '185 patent, Structured Illumination Microscopy (SIM), or localized microscopy techniques such as PALM / STORM.

[0086] A "measurement procedure" consists of a set of "elementary singular measurements" as well as a set of "light measurements" that quantify the light incident from the object, whether pixelated or not, its intensity and optionally its spatial distribution recorded on a photodetector, each elementary singular measurement consisting of a projection of a predetermined singular light distribution, with given parameters, arranged in such a way that the locus of the singular distribution embeds the geometric shape in the nominal parameters. In some cases, "elementary regular measurements" consisting of projections of predetermined regular light distributions can also be used to complete the information collection.

[0087] The "input" of a measurement refers to the location of the optical singular distribution, its type, and parameters. The "output" is the light intensity and / or light distribution recorded in the measurement. We assume that the reader is skilled in the art and familiar with "control theory". Control theory is an interdisciplinary field of engineering and computational mathematics that deals with the behavior of dynamical systems with inputs, and how their behavior can be modified by feedback (from Wikipedia).

[0088] "Control means" refers to a set of control hardware, capable of varying inputs and "control algorithms" and predicting the next step of input values ​​required to quantify the "energy law" directly or by successive approximations in a manner appropriate to obtain the parameters accurately. An "inverse energy law" is an optimal or non-optimal recipe that allows obtaining the parameters of a shape from a set of measurements of a single singular distribution or a set of singular distributions. It is incorporated into the control algorithm. It is selected to optimize the functional parameters of the system, the overall energy - or power - impinging on the biological object, the measurement speed, a combination of the above, or other functional parameters of the system.

[0089] In the context of singular distributions, a "value close to zero" refers to the energy used to qualitatively describe a projected intensity or emitted energy that is significantly less than the maximum intensity available in the projected light or emitted energy when illuminated at maximum intensity. The quantitative value of an intensity or energy close to zero is 6 times between the projected intensity and the maximum intensity of the distribution, or between the emitted energy and the emitted energy, assuming Poisson noise. It is worth mentioning that the noise value of an energy close to zero is significantly less than the maximum energy, approaching less than 2.5 times. Similarly, the parameter value of a geometric shape that is "close to zero" will be less than the full range of the parameter divided by 2.5.

[0090] According to an embodiment of the present invention, an observed object may be modeled as a specific shape that is known or assumed a priori, or as a shape in a list of potential shapes. A geometric shape may be described in terms of: a. Spatial Location,The "location" of a shape refers to the location of the shape's,origin, which is most often determined by the center of,gravity of the light distribution. b. A set of "structural parameters" describing the "shape parameters", e.g., for an isotropic circle (Figure 2C), for an ellipse (Figure 2D), for an object represented by the axis and the angle of the semi-axis of the ellipse.

[0091] As used herein, the term "machine learning" refers to the field of computer science that uses statistical methods to give computer systems the ability to "learn" from data (e.g., gradually improve their performance at a particular task) without being explicitly programmed to do so.

[0092] The terms “representation learning” and “deep learning” are used following LeCun 2015, as explained above in

[0015] . LeCun2015 clearly draws the boundary between "representation learning" and "deep learning", on the one hand, and between "representation learning" and standard "machine learning", on the other.

[0093] The terms "representation learning" and "deep learning" are used interchangeably herein as all embodiments of the present invention apply to both. Portions of the present invention may also be applied to machine learning.

[0094] In the measurement case (defined above), data are obtained at the nominal location of an identified object. "Nominal location", when referring to an object, shall mean the location assumed to be the location of the object from prior knowledge or hypotheses.

[0095] For the purposes of this description, it is assumed that a separate mechanism was used to collect the nominal position of the object. Within the scope of the present invention, this mechanism may use any localization technique, such as, for example, measuring the center of gravity of the light distribution created by the object in another imaging modality or directly on the camera. Basis of the invention

[0096] The present invention introduces a new method for the measurement of parameters of biological, life science or physical objects modeled as geometric shapes. According to the method detailed herein, a system is created for direct access to parameter values ​​of the underlying geometric shapes, the method and system exploiting the properties of singular light distributions and not limited by diffraction as in imaging, nor by photon flux as in imaging paradigms and parameter evaluation using image processing tools.

[0097] The new measurement method described here correlates with physical insights to our knowledge first described herein in the form of specific limitations of the direct application of Abbe's resolution limit to the measurements.

[0098] Indeed, see Schermelleh 2010 or Horstmeyer, et al., "Standardizing the resolution claims for coherent microscopy," both of which are incorporated herein by reference. Nature Photonics Returning to Abbe's law of resolution, as explained by Roberts, 10, pp. 68-71 (2016), all "bandwidth extrapolation" techniques that reconstruct information beyond the Abbe limit rely on prior knowledge, assumptions, or guesses, and in some cases, these may be wrong. Herein, we describe the only case of knowledge of the measurement beyond Abbe's law of resolution, without prior knowledge, assumptions, or guesses: the case of a black (null) image.

[0099] Because of the positivity constraint, the null image contains no signal, so there are no spatial frequencies above or below the Abbe limit, and all frequency content is known to be zero.... The null image allows us to quantify high frequency content from diffraction-limited optical measurements all the way up to zero beyond the Abbe limit.

[0100] This is inimical to the naive view of Abbe's law, which is understood to mean that no information can be obtained by optical processes beyond the Abbe limit. The case described in this invention makes it possible to go beyond the limit, even if it is highly specific, and thus collect information up to infinity, theoretically, so that diffraction is unlimited.

[0101] This condition is inherent to optical systems and similar unipolar signals and is due to the positivity constraint, an additional physical constraint on the solution, independent of the diffraction limit. These concepts do not readily apply to electrical signals or similar bipolar signals. The positivity constraint is known to allow additional information for some reason, but its contribution to the information expansion is small in most practical cases and has not been well characterized or quantified. In the present invention, the impact of the positivity constraint is clear and conclusive, allowing for precise quantification of spatial frequencies above and below the Abbe limit.

[0102] Several formulations of Abbe resolution coexist. The preferred expression of Abbe resolution is that of Schermelleh 2010: "Details of periodic objects smaller than this shortest wavelength cannot possibly be transferred to an image." (Schermelleh 2010, p. 166) The present results constitute a corollary of Abbe's law of resolution, not an exception.

[0103] However, the present results are certainly an exception and a refutation of the widely known formulation of Abbe's resolution law in the scientific literature, which is not a cautious wording: "Frequency components beyond the Abbe resolution limit cannot be measured."

[0104] In most cases the null image contains no information, this singular point or exception is useless and trivial, and this discussion in these cases may seem meaningless and empty. But only in the case of zero size will the recorded image of an isolated point of a certain size, illuminated by an optical vortex placed at a nominal position, be exactly zero. In this particular case, the complete nulling of the image and all its frequency components is a powerful measure, by which we may obtain not only the position of a point-like object, but also its size, as will be described later.

[0105] The invention further introduces a new method that can be applied to any compound imaging system and is exemplified in more detail in the case of CODIM imaging. Direct bridging, which in most cases consists of applying a specific neural network directly to the element data. In the previous paragraph, we described how to apply natural paths (or bridges) between element data in a compound imaging process to direct the metrology approach. In this paragraph, we describe how to apply natural paths (or bridges) between element data in a compound imaging process, or deep learning, without using any image reconstruction algorithm, applying a supervised or unsupervised deep learning layer directly to the element data. Moreover, this layer can collect data from several measurements (in most cases consecutively). The result of this layer, either a measurement layer or an object layer, is a direct input to further processing, avoiding the image or map step that can still be used for visualization, qualification and monitoring.

[0106] Additionally, both direct metrology and deep learning processes may be combined to create even stronger and more powerful acquisition tools.

[0107] The method described in accordance with the present invention differs from all prior art methods heretofore described or suggested in that at least all prior art methods treat image collection as an essential step.

[0108] The Sirat '185 patent does not suggest the use of deep learning or neural network layers that can classify a scene as multiple shapes, static or dynamic, using the detected light as a direct input to the neural network layer, where the shapes are part of a family of basic shapes pre-determined or acquired by learning. According to the present invention, the deep learning layer directly structures the raw data information as basic objects, with metric properties and attributes, these basic objects being a suitable basis for further generalization. This approach, which is consistent with the deep learning perspective, is developed in the present invention.

[0109] The Sirat'185 patent relies on physical reality and defines the measured maps as descriptors of fluorophores, such as in our example, the influenza A virus as a structuring concept, so this new positioning is consistent with biological reality and defines a biological entity.

[0110] Furthermore, this method can be extended to dynamic properties, which can be obtained as time-dependent data or preferably as higher-level dynamic parameters. In short, going back to our example, the intermediate layer may determine the presence of a 110 nm point object moving along a path at a constant speed without morphological changes, and we assume in the next layer, the object classification layer, that it is an influenza A virus.

[0111] An additional advantage of the proposed architecture is that it avoids reconstruction artifacts by avoiding the reconstruction step. Even the best reconstruction algorithms are prone to artifacts, and the possible presence of artifacts is one of the barriers to the adoption of super-resolution techniques by biologists. Even worse, many artifacts are erroneously considered as small points or objects, and classification and quantification algorithms can be impaired by these artifacts. Avoiding reconstruction and performing classification and quantification on the raw data completely eliminates this source of error.

[0112] As mentioned before, information beyond the diffraction limit is present in the metrology data and may also be present in the imaging data. This valuable information is usually erased by the reconstruction process, and since we cannot process this information separately from all the other information, we are forced to apply general procedures, optimized for the average data, that do not fit these specific cases. Geometric and biological information from non-imaging processes

[0113] The Sirat '185 patent does not suggest a way to learn geometric and biological information through a non-imaging process, where higher level information is gleaned directly from measured element data without the need to create (or reconstruct) images to quantify maps, or, in the Sirat '185 patent's terms, using a general purpose learning procedure. The Sirat '185 patent does not understand that non-imaging direct learning obtains information with much higher accuracy, achieves high computational efficiency, and does not lose data, as explained below. Moreover, such learning procedures may have the ability to use data that exists in the absence of light, explained below and referred to as the Abbe's Law loophole. Common methods

[0114] According to exemplary embodiments of the invention, the following features are prominent: -Distribution of light at the nominal position; the wavelength of the projected light is chosen such that it can create an inelastic light interaction with the photoresponsive material, and the singular distribution is chosen to embed the geometric shape, so that when using the nominal parameters, the energy measured from the light created by the inelastic interaction of the projected light with the photoresponsive material will be zero. -Applying an algorithm, the "inverse energy law", to obtain the parameters from the energy measurements.

[0115] The method according to the invention may advantageously use measurement hardware and measurement procedures that dynamically or non-dynamically control the sequence of measurements to collect the set of measurements required to retrieve the parameters of the shape.

[0116] The light-responsive material can be exogenous, for example based on fluorescent binding, or intrinsic, using the direct fluorescent properties of some proteins, or using Raman scattering.

[0117] For heuristic convenience, following the general description of the method in the high-level language above, an example is provided, the simultaneous measurement of position and size of a point-like object, and for clarity, the point object is described in terms of an influenza type A virus, as described in the previous paragraph.

[0118] The general method can be illustrated using the simplest case of a virus modeled as a two-dimensional uniform circle shown in Figure 2C. The biological object is a virus. In this example, the goal of the measurement is to quantify the diameter of the virus, which is between 80 and 120 nm. The virus is assumed to be located on a two-dimensional surface.

[0119] The biological object is a virus modeled as a point object represented by its position and the radius R of the virus, as in Figure 2C. The measurement consists of projecting a vortex of light capable of exciting a fluorescent molecule placed on the virus, using a measurement hardware consisting of a two-dimensional optomechanical system of galvanometers capable of scanning the vortex of the biological sample, which embeds the geometry of the point object at multiple positions (xi, yi) and measures the energy with a photodetector, using analytical or iterative algorithms, converging on the position with the minimum energy. In this case, the inverse energy law is very simple: the position is the minimum and the energy is quadratically dependent on the radius (Equation 3).

[0120] In general, the measurement method according to the invention searches for zero, one-dimensional and two-dimensional loci using one or more different singular distributions, searching simultaneously for one or more simple parameters and more complex or composite parameters. Measuring the size and position of point objects

[0121] A first embodiment of the method is presented for measuring shape parameters: measuring the position and size of point objects in two dimensions, which is one of the most common cases in measuring biological objects. Moreover, it can be extended mutatis mutandis to measuring the position and thickness of lines, which is another widely used task in measuring biological objects.

[0122] A radius R is placed at the origin and n D Assuming a circular light-emitting object (see Figure 2C) with a uniform density of phosphors as shown, N E =n D πR 2 Assuming that v is the total number of fluorophores, an optical system is provided that is adapted to move the position of the vortex, given as (v, 0), in a plane with high precision, as described in detail in the Sirat '185 patent. In the polar coordinate notation "(m, n)" used herein, m refers to the radius vector in a transverse plane specified relative to the z axis, and n refers to the angle between a given axis (x in FIG. 2C) and the radius. Without loss of generality, due to the symmetry of the problem, we assume that the vortex is located on the x axis, i.e., the angle is zero.

[0123] The strength of the vortex as a function of position is determined by the coefficient i s via . More specifically, for v>R, where R is the radius of the luminous object, the energy law is:

number

[0124] The energy law is the sum of two independent components. aI v This minimum is placed at the origin so that it is the position-dependent, radius-independent minimum of the local minimum, N E Fluorophores are placed at nominal positions. bI R, position-independent radial dependence.

[0125] At the nominal position of v=0, the energy is given by equation (3). The inverse energy law is quadratic in energy. The same 5% characteristic obtained by the optimal imaging experiment can be reached in a shot-noise limited system with as few as 100 photons, due to the quadratic dependence of energy as a function of radius. Measuring size, position and ellipticity of point objects

[0126] In a second embodiment of the invention, shape parameters can be measured, and the position, size and ellipticity of a point object may be measured in two dimensions. This is perhaps the least common case in measuring biological objects, as it cannot be performed using existing paradigms.

[0127] Assumptions a. An elliptical luminous object, Fig. 2D, with minor and major axes a and b, respectively; b. The distribution position given by (0, y0) in Cartesian coordinates, rather than polar coordinates as in the previous paragraph, cn D The uniform density of fluorophores, expressed as d. Projecting the half-moon distribution in Figure 3; e. The "half moon" distribution for Case 24 in FIG. 3 is aligned along the major axis (x-axis in FIG. 2D). In a first approximation, a quadratic dependence of the intensity distribution in the direction y perpendicular to the long axis occurs. N E =n D It can be shown that πab is the total number of fluorophores and πab is the area of ​​the ellipse.

[0128] As described in detail in the Sirat '185 patent, an optical system is provided that is adapted to move with high precision in a plane the position of the vortex, given as (v, 0). In polar coordinate notation, a denotes the radius vector of a cross section designated relative to the z-axis, and b denotes the angle between a given axis (x in FIG. 2C) and the radius. Without loss of generality, due to the symmetry of the problem, we assume that the vortex is located on the x-axis, i.e., the angle is zero.

[0129] Under the assumption that the distribution is angularly aligned to the semimajor axis b, the x-axis in Figure 2D, and shifted by y, the intensity of the distribution as a function of position y is given by the coefficient i s via, which depends quadratically on the distance on the axis y and independent of x. The equation of an ellipse is given by:

number

number

number

number

number

number

[0130] According to further embodiments of the present invention, the relative position of two points in space may be measured using the techniques described herein, even if the two points are located in two different planes.

[0131] With reference to Fig. 4, the relative position of two points located in two different planes is a challenging problem from a practical point of view and is at the heart of overlay metrology, one of the most critical metrology steps in semiconductor processing. If we assume a crescent moon as in case 02 in Fig. 3, the locus of this singular distribution is a spiral 401. The geometric shape embedded in this distribution includes not only the spiral shape but also multiple points located along the spiral. If two points are in two different planes 403 and 405, but on the same spiral, we can monitor the relative position of one point with respect to the other along the spiral and evaluate the positions of the two points in the two different planes with theoretically infinite resolution. Implementation of the method using conical diffraction

[0132] Methods according to embodiments of the present invention may also be advantageously implemented using conical diffraction, see Caron, J., et al., “Conical diffraction illumination opens the way for low phototoxicity super-resolution imaging,” incorporated herein by reference. Cell Adhesion & Migration 8(5): 430-439 (2014) may be used.

[0133] An embodiment of the present invention will now be described with reference to FIG. 5. A method, generally designated by the numeral 500, is provided for determining a value of an intrinsic geometric parameter of a geometric feature of a particular dimension characterizing a physical object. To implement the claimed method, CODIM or metrology application hardware is provided (step 501), as described in detail above, and a singular light distribution is projected onto the physical object (503). Light excited by the singular light distribution interacting with the geometric feature is detected when it impinges on a detector (505). Return energy distributions are identified and quantified at one or more locations of the singular energy distribution as a quantification of the intrinsic geometric parameter (507). Finally, a value of the intrinsic geometric parameter is determined based on the value of the intrinsic geometric parameter (509).

[0134] The use of standard deep learning concepts in CODIM imaging is described with reference to FIG. 7. A biological object 20, graphically represented as a bacterium, is illuminated by a series of local singular distributions 21 that are detected by a suitable low-noise imaging detector 22. A series of micro-images 23 (also referred to herein as "μ-images") are recorded and processed by a suitable algorithm 25 to obtain a reconstructed super-resolution image 10. A deep learning network comprising several layers 11, 12 and 13 is applied to the reconstructed images in a process similar to that described in FIG. 6. A control system 24 is used to control and synchronize the acquisition and the processing.

[0135] The use of direct deep learning in CODIM imaging will now be described with reference to FIG. 8. A biological object 20, graphically represented as a bacterium, is illuminated by a set of local singular distributions 21 that are detected by a suitable low-noise imaging detector 22 (also referred to herein as "detector"). The set of micro-images 23 is applied directly, without reconstruction, to a deep learning layer 26, which may differ from the layers used in the standard process. Additional layers 12 and 13, which may differ from the layers used in the standard process, are applied to the results of the first layer. A control system 24 is used to control and synchronize the acquisition and processing.

[0136] The use of standard deep learning concepts in a composite imaging system is described with reference to FIG. 9. A generic object 120, graphically represented as a cat for heuristic convenience, is illuminated by a series of localized singular or regular distributions 21 detected by a detector 22. A series of micro-images 23, also referred to herein as "intermediate images", are represented as sub-images of parts of the cat picture, but this is merely a graphical artifact. The series of micro-images 23 are recorded and processed by a suitable algorithm 25 to generate a reconstructed super-resolution image 10. A deep learning network including several layers 11, 12 and 13 is applied to the reconstructed image, in a process similar to that described in FIG. 6. A control system 24 is used to control and synchronize the acquisition and the processing.

[0137] The use of direct deep learning in the general case of compound imaging is now described with reference to FIG. 10. A general object 120, represented as a cat, is illuminated by a series of localized singular or regular distributions 21 detected by a detector 22. A series of micro-images 23, or intermediate images, are represented as sub-images of parts of the cat picture, which are merely graphical artefacts. The series of micro-images is applied directly, without reconstruction, to a deep learning layer 26, which may differ from the layers used in the standard process. Additional layers 12 and 13, which may differ from the layers used in the standard process, are applied to the results of the first layer. A control system 24 is used to control and synchronize the acquisition and the processing.

[0138] The use of controlled direct deep learning in CODIM imaging is described with reference to FIG. 11. A biological object 20, graphically represented as a bacterium, is illuminated by a series of localized singular distributions 21, the sequence of distributions including their location, intensity, type, duration and timing being determined by a control system based on any type of external information 30, and / or partial or final information from processing. The light distributions emitted by the object illuminated by the different light distributions are detected by a suitable low-noise imaging detector 22. The series of micro-images, 23, are applied directly, without reconstruction, to a deep learning layer 26, which may differ from the layers used in the standard process. Additional layers 12 and 13, which may differ from the layers used in the standard process, are applied to the results of the first layer. A control system 24 is used to control and synchronize the acquisition and the processing and to determine the sequence of distributions.

[0139] The use of controlled direct deep learning in the case of general compound imaging is now described with reference to FIG. 12. A general object 120, graphically represented as a cat, is illuminated by a sequence of localized singular or normal distributions 21. A sequence of micro-images 23 or intermediate images are represented as sub-images of parts of the cat picture, which are merely graphical artefacts. The sequence of distributions, including their position, intensity, type, duration and timing, is determined by a control system based on any type of external information 30, and / or partial or final information from processing. The light distributions emitted by the object illuminated by the different light distributions are detected by a detector 22. The sequence of micro-images is applied directly, without reconstruction, to a deep learning layer, 26, which may differ from the layers used in the standard process. Additional layers 12 and 13, which may differ from the layers used in the standard process, are applied to the results of the first layer. A control system 24 is used to control and synchronize the acquisition and the processing and to determine the sequence of distributions. Classify scenes using deep learning

[0140] Features generally associated with methods according to embodiments of the invention include the following. a physical or biological scene, consisting of a plurality of luminescent biological objects, each object being modeled as a luminescent geometric shape, the geometric shape being quantified by an appropriate set of parameters; -Hardware setup such as the imaging case described in Caron2014 and the measurement case mentioned above, - a set of measurements, each consisting of a projection of a singular distribution at a nominal location on a regular or pseudo-regular grid, called the "imaging case", or on a custom grid, called the "metric case", where in the metric case the singular distributions are chosen such that they embed a geometric shape, so that at nominal parameters the measured energy is zero, - detecting light from a singular light distribution that interacts with geometric features and impinges on the detector, whether or not the light constituting the detected light is pixelated; A deep learning or neural network layer that directly uses the detected light as a direct input to a neural network layer that can classify the scene as multiple shapes, static or dynamic, where the shapes are part of a family of basic shapes pre-determined or obtained by learning.

[0141] According to other embodiments of the present invention, direct and dynamic methods provide control hardware and control procedures that can dynamically or non-dynamically control the sequence of measurements and detection light in order to collect an improved set of measurements required to obtain the shape parameters by analysis of the element data by standard or deep learning procedures. Deep Learning Implementation Details

[0142] According to an embodiment of the present invention, a method is provided that uses the specifics and advantages of deep learning to analyze imaging and metrology data acquired by the CODIM-Cone Analysis Microscopy method described in Caron 2014, or hardware implementing said metrology.

[0143] Based on these methods, a new system for the optical measurement of geometric features is described herein, and more specifically, methods for measurements with resolution higher than the diffraction limit imposed by the optical system and the minimum photon flux are described. EXAMPLES

[0144] One embodiment of the present invention can be illustrated by the following example, which is used for illustration purposes only and is not intended to represent a tested experimental case. The luminous object is assumed to be a biological object, which for illustration purposes is assumed to be the same influenza type A virus (IAV) shown diagrammatically in FIG. 2A. This virus is known to have a typical size of 80-120 nm and is known to be spherical in most cases, but may be filamentous in some cases. Badham et al., “Filamentous Influenza Viruses,” incorporated herein by reference, Current clinical microbiology reports , vol. 3, pp. 155-61 (2016) states that "The biological significance of IAV morphology in human clinical infection is of great interest." IAV viruses are assumed to be uniformly marked all over their volume with some suitable fluorescent protein. Several models can be used to describe the virus. The virus is described as a point object modeled as an ellipse (Figure 2D), with parameters being the semi-minor axis a, the semi-major axis b, and the angle of the major axis θ, with respect to Cartesian coordinates.

[0145] According to embodiments of the present invention, dynamic changes in viral location and morphology are assessed, potentially to assess viral virulence. CODIM embodiment

[0146] For imaging using deep learning the CODIM system described in Caron2014 can be applied, but for metrology the setup described in Metrology Applications is preferred.

[0147] According to an embodiment of the present invention, a method is provided that uses the specificity and advantages of deep learning to analyze imaging and metrology data acquired by CODIM-Cone Analysis Microscopy as described in Caron 2014. Neural Network Implementation

[0148] Spiliotis et al., “Priming for destruction: septins at the crossroads of mitochondrial fission and bacterial autophagy,” EMBO Reports, vol. 17, pp. 935-37 (2016) ("Spiliotis 2016"), incorporated herein by reference, teaches the following: Mitochondria are organelles essential for cell survival, programmed cell death, and autophagy. They undergo cycles of fission and fusion, are disrupted by infectious pathogens, and are altered in many human diseases. 935 in Spiliotis 2016. Pagliuso et al., “A role for septin 2 in Drp1-mediated mitochondrial fission”, EMBO Reports, vol. 17, pp. 857-73 (2016) (“Pagliuso 2016”), which is incorporated herein by reference, states the following: Among the multiple mechanisms that appear to regulate mitochondrial fission, Septin 2 plays a key role by mediating mitochondrial contraction. Pagliuso 2016, abstract, emphasis added. In this study, the BioAxial CODIM system was used to obtain images of mitochondria by acquiring data and reconstructing images.

[0149] The reconstructed image of mitochondria can be used as an input for a deep learning network to gather information about the biological state, as shown in FIG. 10, or the collected data can be fed directly into a deep learning network, as shown in FIG. 11.

[0150] The embodiments of the invention described herein are intended to be merely illustrative; variations and modifications will be apparent to those skilled in the art. All such variations and modifications are intended to be within the scope of the present invention, as defined in the appended claims.

Claims

1. 1. A deep learning method for classifying a scene into at least one geometric feature using imaging data acquired by CODIM hardware, comprising: a. projecting a singular light distribution, characterized by a first wavelength, a singularity location, and a controlled spatial variation, onto a light emitting sample; b. detecting light excited by the specific light distribution that has interacted with the luminescent sample as it strikes an imaging detector; c. identifying a feature of the returned energy distribution in the form of a two-dimensional map representative of a geometric feature at a particular location on the light emitting sample; d. determining values ​​of specific geometric parameters based on analysis of data obtained by deep learning, the data including correlations between the return energy distribution and geometric features of the light emitting sample; The deep learning method in question.

2. The method of claim 1 , wherein the sample is a dynamic sample.

3. The method of claim 1 , wherein the sample is a static sample.

Images