Calibration of light field imaging systems

Abstract

To provide methods of calibrating a light-field imaging system, a light-field imaging system for implementing the calibration methods, a calibration target for the calibration methods, and methods of projecting a light-field image into an object space using a calibrated light-field imaging system.SOLUTION: An exemplary calibration method is implemented using a light-field imaging system comprising a microlens array (72) and an image sensor (76). A z-stack of light-field images of a calibration target may be captured using the image sensor, while the calibration target is on a stage and located at a plurality of different z-positions. A total magnification of the imaging system and a microlens magnification of the microlens array may be determined from each light-field image of the z-stack.SELECTED DRAWING: None

Description

[Technical field]

[0001] (Related Applications) This application claims priority to U.S. Patent Application No. 16 / 265,220, filed February 1, 2019, the contents of which are incorporated by reference in their entirety herein. [Background technology]

[0002] In contrast to conventional microscopes, light field microscopes record multiple views of the same object from different angles in one light field image. These recorded views allow the reconstruction of depth information for different parts of the object. The depth information enables a variety of applications, such as digital refocusing of the calculated two-dimensional projections or three-dimensional reconstruction of the object, neither of which is possible using conventional microscopes. However, such projections and reconstructions performed using light field microscopes generally come with a tradeoff of lower spatial resolution compared to conventional microscopy images.

[0003] A conventional microscope can be converted into a light field microscope by replacing a conventional image sensor and camera array or by inserting a microlens array in front of the image sensor. The use of a microlens array is often the preferred option due to its lower cost and ability to be easily retrofitted to existing conventional microscopes. Two configurations with distinctly different properties can be defined depending on the location of the microlens array relative to the objective lens and image sensor of the microscope. In the first configuration, labeled "LF1.0", the microlens array is at the intermediate image plane between the objective lens and the image sensor, and the image sensor is at the back focal plane of the microlens array. In the second configuration, labeled "LF2.0", these restrictions are removed and the microlens array can have an arbitrary location between the objective lens and the image sensor. Thus, in LF2.0, the location of the microlens array can be selected to optimize the desired performance criteria of the light field microscope (e.g., lateral and axial resolution and depth of focus).

[0004] In order to be able to perform a quantitative reconstruction of the 3D object space (x,y,z) from light field microscopy images, the relationship between the object space and the 2D light field space is determined by the microlens sub-images (x,y,z) of the light field image. k ",y k ") needs to be known. The mapping function is [ka] It is defined as: In principle, the mapping function can be calculated based on values ​​for geometric parameters of the experimental setup, such as the relative distances from the base planes of the relevant optical elements to the image sensor. In practice, the exact locations of these base planes are unknown and the relative distances inside the light field microscope cannot be easily determined to the required accuracy. This lack of precise geometric information is more problematic for the LF2.0 configuration, where there are no clearly defined optical references for the positioning of the microlens array inside the light field microscope (such as the intermediate image plane of the objective lens and the focal plane of the microlens array). In addition, optical effects of the real lens system (e.g., aberrations or errors caused by the lenses of the microlens array or other optical elements of the light field microscope) can lead to deviations in the actual optical geometry compared to the assumed optical geometry measured externally. Beyond these inherent difficulties in determining the relevant geometry from external measurements, user-induced changes to the geometry (e.g., due to refocusing or unintentional misalignment of the microscope) can occur frequently. Any deviation in the assumed geometry relative to the actual geometry will lead to errors in the projection and reconstruction, resulting in incorrect depth information, loss of contrast, or failure to complete the reconstruction.

[0005] Systems, methods and apparatus that enable mapping between light field space and object space are needed for calibration of light field microscopes or cameras. Summary of the Invention [Means for solving the problem]

[0006] The present disclosure provides a method for calibrating a light field imaging system, a light field imaging system for performing the calibration method, a calibration target for the calibration method, and a method for projecting a light field image of an object into an object space using a calibrated light field imaging system. An exemplary calibration method is performed with a light field imaging system including a microlens array and an image sensor. A z-stack of light field images of the calibration target can be captured using the image sensor while the calibration target is located at a plurality of different z-positions. A total magnification of the imaging system and a microlens magnification of the microlens array can be determined from each light field image of the z-stack. The present invention provides, for example, the following items. (Item 1) 1. A method for calibrating a light field imaging system having a microlens array and an image sensor, the method comprising: capturing a z-stack of light field images of the calibration target using the image sensor while the calibration target is positioned at a plurality of different z-positions; determining a total magnification of the light field imaging system and a microlens magnification of the microlens array from each light field image of the z-stack; A method comprising: (Item 2) 2. The method of claim 1, further comprising using the z-stack of light field images to calculate a point spread function as a function of z-position of the calibration target. (Item 3) 2. The method of claim 1, further comprising using a z-stack of the light field images to determine a radial distortion for at least one microlens of the microlens array as a function of z-position of the calibration target. (Item 4) 2. The method of claim 1, wherein determining a microlens magnification ratio comprises calculating, for each z-position, at least one microlens magnification ratio for the microlens array using a pitch of the microlens array. (Item 5) 2. The method of claim 1, wherein the calibration target has at least one type of marker arranged to form a first periodic repetition and a second periodic repetition, the first periodic repetition being used to determine a total magnification at each of the z-locations and the second periodic repetition being used to determine at least one microlens magnification at each of the z-locations. (Item 6) 6. The method of claim 5, wherein the first periodic repetition is formed by a marker repeated a number of times to define a grid, and the second periodic repetition is formed by the periodic absence of the marker from nodes of the grid. (Item 7) 2. The method of claim 1, wherein the calibration target comprises a two-dimensional array of pinholes. (Item 8) 8. The method of claim 7, wherein the pinholes are located at the nodes of a grid, and the pinholes are periodically absent from the nodes of the grid. (Item 9) 2. The method of claim 1, wherein the calibration target has an array of markers each located at a different node of the same grid, each marker being optically transmissive or luminescent, and a subset of the nodes in the grid being non-transmissive or non-luminescent nodes that do not have any optically transmissive or luminescent markers. (Item 10) 10. The method of claim 9, wherein the non-transparent or non-emissive nodes are repeated periodically in two dimensions within the grid. (Item 11) The method further comprises: Capturing a light field image of an object; projecting the light field image into a plane in an object space using a total magnification of the imaging system corresponding to the plane and a magnification of at least one microlens of the microlens array corresponding to the plane; The method according to item 1, comprising: (Item 12) Item 12. The method of item 11, wherein the projecting reduces blurring using at least one point spread function, the at least one point spread function being calculated from one or more of the light field images of the z-stack. (Item 13) Item 12. The method of item 11, wherein the projecting includes correcting radial distortion produced by microlenses of the microlens array. (Item 14) 1. A light field imaging system comprising: A calibration target; a light source for illuminating the calibration target; an objective lens for collecting light from the illuminated calibration target; a microlens array downstream of the objective lens; an image sensor for capturing a light field image of the calibration target; a computer configured to determine a total magnification and a microlens magnification from the light field image; A system comprising: (Item 15) Item 15. The system of item 14, wherein the computer is configured to calculate at least one point spread function from the light field image. (Item 16) Item 15. The system of item 14, wherein the computer is configured to determine a radial distortion for at least one microlens of the microlens array from the light field image. (Item 17) Item 15. The system of item 14, wherein the calibration target includes at least one type of marker arranged to form a first periodic repetition and a second periodic repetition, and the computer is configured to use the first periodic repetition to determine the total magnification ratio, and the computer is configured to use the second periodic repetition to determine the microlens magnification ratio. (Item 18) 20. The system of claim 17, wherein the first periodic repetition is formed by a marker repeated a number of times to define a grid, and the second periodic repetition is formed by the periodic absence of the marker from nodes of the grid. (Item 19) 1. A method for projecting a light field image of an object into an object space using a light field imaging system, the method comprising: capturing a light field image of the object supported in an object space; geometrically mapping pixels of the light field image to a plane in object space using a value for a total magnification of the light field imaging system corresponding to the plane and at least one value for a microlens magnification corresponding to the plane; A method comprising: (Item 20) (i) the mapping uses a point spread function to reduce blurring, and / or (ii) mapping corrects for radial distortion produced in the light field image by an array of microlenses; Item 19. The method according to item 19. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram of an exemplary light field imaging system including a light field microscope supporting an exemplary calibration target as the light field microscope is calibrated. [Diagram 2] FIG. 2 is a ray diagram created using selected components of the light field microscope of FIG. 1 illustrating the propagation of an example ray from a point object located in a plane in object space to a sensor plane in the two-dimensional light field space. [Diagram 3] 3 is a front view of an exemplary microlens array for the light field microscope of FIG. 2 taken generally along line 3-3 of FIG. 2. [Figure 4] FIG. 4 is a ray diagram similar to that of FIG. 2, but with the object plane having a z-position closer to the microscope objective. [Diagram 5] FIG. 5 is another ray diagram similar to that of FIG. 2, but in which the object plane has a z-position farther from the microscope objective. [Figure 6A] FIG. 6A is a schematic diagram of a light field microscope illustrating the imaging of a real inverted image by a microlens array along with representative light rays, and also illustrates the basis of a mapping function that relates a point in object space to multiple points in light field space. [Figure 6B] FIG. 6B is another schematic diagram of a light field microscope illustrating the imaging of a virtual inverted image by a microlens array along with representative light rays, and also illustrates the basis of a mapping function that relates a point in object space to multiple points in light field space. [Figure 7] FIG. 7 is a plan view of an example calibration target for the light field imaging system of FIG. 1, with light-transmissive areas shown in black and light-blocking areas shown in white. [Figure 8A] FIG. 8A is a fragmentary view of the calibration target of FIG. 7 taken generally around the area designated "8A" in FIG. [Figure 8B] FIG. 8B is a fragmentary view of FIG. 8A obtained with a rectangular grid defined by markers superimposed on the calibration target. [Figure 9] FIG. 9 is a cross-sectional view of the calibration target of FIG. 8A taken generally along line 9-9 of FIG. 8A. [Figure 10] FIG. 10 is an example negative light field image of the calibration target of FIG. 7 that may be captured using a light field imaging system. [Figure 11] FIG. 11 is a magnified fragmentary view of the negative light field image of FIG. 10 taken generally around the area indicated at "11" in FIG. [Figure 12]FIG. 12 is another enlarged fragmentary view of the negative light field image of FIG. 10, taken generally from around the area indicated at "12" in FIG. 11, in one of the separate sub-images present within the negative light field image of FIG. 10. [Figure 13] FIG. 13 is a superimposed form of the sub-image of FIG. 12, identifying its features and attributes. [Figure 14] FIG. 14 is a superimposed form of sub-images similar to that of FIG. 12, but in which the sub-images exhibit radial distortion produced by corresponding microlenses of a microlens array of a light field imaging system. [Figure 15] FIG. 15 is a fragmentary plan view of another example calibration target for calibrating light field imaging, with light-transmissive areas shown in black and light-blocking areas shown in white. [Figure 16] FIG. 16 is a plan view of yet another exemplary calibration target for calibrating a light field imaging system. [Figure 17] FIG. 17 is a plan view of yet another example calibration target for calibrating a light field imaging system, with light-transmissive areas shown in black and light-blocking areas shown in white. [Figure 18] FIG. 18 is a fragmentary view of the calibration target of FIG. 17 taken generally around the area designated "18" in FIG. [Figure 19] 19 and 20 are graphs plotting magnification values ​​as a function of z-position measured using a calibration target and a physical embodiment of a light field microscope. [Figure 20] 19 and 20 are graphs plotting magnification values ​​as a function of z-position measured using a calibration target and a physical embodiment of a light field microscope. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The present disclosure provides a method for calibrating a light field imaging system, a light field imaging system for performing the calibration method, a calibration target for the calibration method, and a method for projecting a light field image of an object into object space using the light field imaging system to be calibrated. An exemplary calibration method is performed using a light field imaging system including a microlens array and an image sensor. A z-stack of light field images of the calibration target can be captured using the image sensor while the calibration target is located at a plurality of different z-positions. The total magnification of the imaging system and the microlens magnification of the microlens array can be determined from each light field image of the z-stack. The systems and methods disclosed herein can enable higher sample throughput and low-cost mapping and tracking of microscopic objects in three dimensions.

[0009] Light field (plenoptic) imaging allows the reconstruction of a three-dimensional object, or at least a digital refocusing of the object, from a single light field image. A prerequisite is to obtain precise light field calibration information. The present disclosure provides a procedure for automatically calibrating the light field characteristics of a light field imaging system as an algorithm, even when the imaging system has a "black box" optical system. A mapping function and a simple but accurate calibration of the optical path of the light field imaging system to enable the use of the mapping function are provided. The calibration can be automated and therefore can be repeated, if necessary, to have the correct geometry as a basis for a high-quality reconstruction of the object (e.g., a sample of interest) at all times using the mapping function.

[0010] Although calibration approaches are available for light-field photography, which is performed with an array of separate cameras, the experimental situation and challenges of light-field microscopy are significantly different. First, in contrast to light-field photography, the imaging geometry from the object to the image plane is relatively fixed in microscopy. This geometry can only change within a given range, determined primarily by the mechanical structure of the microscope (mainly through changes in the distance between the object and the objective). Modern microscopes are equipped with high-precision xyz-stages for adjusting the object-objective distance, so that any specific measurement geometry of a given microscope can be repeatedly reproduced with high accuracy for that microscope. Second, the spatial resolution in the object plane needs to be much higher for microscopy than for photography. At the same time, the field of view and the depth of focus are much smaller in microscopy compared to photography. This means that the size of the object volume that needs to be calibrated is much smaller in microscopy, but also the quality of the calibration needs to be much higher. Third, in typical microscope geometries, the object sits closer to the front focal plane of the objective. This leads to a highly non-linear relationship between object magnification and object-object distance (also called z-position of the object). This axial dependence of magnification needs to be taken into account during calibration.

[0011] This disclosure provides a simple yet concise calibration concept primarily aimed at light field microscopy and light field close-ups. The concept enables calibration using measured light field data of a suitable calibration target and values ​​for the pitch of the microlens array. From the light field image of the calibration target, the total magnification of the system and the magnification due to the microlens array alone can be determined for a series of different z-positions of the calibration target. These values ​​for magnification can uniquely determine the mapping geometry between the three-dimensional object space and the two-dimensional light field space. The properties of the microlens array in the imaging path and the interactions of the microlens array with the entire imaging system can be calibrated in three dimensions, allowing the tracing of an optical ray back to its origin in the object space.

[0012] The method of the present disclosure can be used with light field imaging systems having different light field configurations (e.g., types LF1.0 and LF2.0). In particular, the method applies to light field imaging systems having non-telecentric geometry, where the magnification depends on the distance from the lateral object plane to the objective lens (i.e., the z-position of the object plane). The method can be used not only to determine the values ​​of global parameters for the mapping function, but also to correct any local aberrations, such as those caused by lens errors. Since the calibration data can be captured (i.e., recorded) and analyzed in an automated manner for a series of z-positions, the method can be used iteratively to determine and adapt to any changes in the optical geometry of the system, as necessary.

[0013] The methods disclosed herein can be applied to light field imaging systems constructed to perform light field photography of macroscopic objects, provided that the parameters of the calibration target are appropriately adapted and a suitable calibration stage for reproducible z-positioning is employed. A calibration stage is generally required to accurately adjust the distance from the calibration target in the volume of interest in object space to the light field camera. For practical reasons, in addition to use in light field microscopy, the methods may be most suitable for relatively extended close-up photography, where the axial length of the volume of interest in object space is typically less than about 1-100 centimeters.

[0014] The methods disclosed herein can be used for all wavelength regimes for which lens-based light field systems are viable, for example, wavelength regimes can include optical radiation that is ultraviolet, visible, near infrared, or the like.

[0015] This disclosure describes a calibration method that determines values ​​for system parameters required for projection and object reconstruction from light field images. The method may utilize a dedicated calibration target, which may include a superposition of two periodic patterns. Using the calibration target, values ​​for local parameters required for object reconstruction can be determined from a captured light field image of the calibration target. From the local values, values ​​for global parameters can be calculated with high accuracy through averaging of the values ​​for the local parameters and local aberrations due to optical artifacts.

[0016] The calibration method may directly measure relevant parameters of a mapping function that relates the three-dimensional object space to the two-dimensional light field space. The parameters may include the total magnification and the magnification due to the microlens array over a range of z-locations of interest in the object space. The calibration method may also measure the radial distortion for at least one microlens (e.g., the individual radial distortion for each of the multiple microlenses of the microlens array and / or the average radial distortion for the microlens array) and / or the point spread function (e.g., the individual point spread function for each of the multiple microlenses and / or the average point spread function for the microlens array). All this information may be extracted from the captured light field image without any prior assumptions about the explicit configuration of the system's optics (such as the types of optical elements in the system, their optical parameters, and their relative distances to each other) other than the pitch of the microlens array. The information may also be extracted without the need for an iterative optimization process to determine the free parameters of a complex optical model.

[0017] The total system magnification and the magnification of the microlens array can be measured from the same light field image, even if these magnifications have significantly different absolute values. Only the light field image needs to be evaluated; no other images at other positions in the beam path (e.g., intermediate image planes in front of the microlens array) need to be recorded. In other words, the entire light field imaging system can be treated as a black box.

[0018] The method disclosed herein captures a light field image of a calibration target, which is located at several z-positions (e.g., more than two). These z-positions, also called axial positions, can be known relative to each other with sufficient accuracy to determine the nonlinear dependence of the total and microlens magnification as a function of z-position. The accuracy of the z-position is generally better than the desired axial resolution for the reconstruction of the object in object space. In principle, this potential limitation does not constitute a disadvantage for light field microscopy, since microscope systems are usually equipped with high-precision xyz-stages.

[0019] The method and system disclosed herein can enable the structure (e.g., spheroid, organoid) of three-dimensional cell population to be determined quickly in screening applications.Currently, the throughput of these screening applications is limited by the time required to obtain a series of z-images (generally more than 10 images) at each site of interest.The throughput is even more limited when a confocal microscope is employed to image the three-dimensional volume at the site.In contrast, the method and system disclosed herein can generate three-dimensional information about cell population from a single light field image after calibration.

[0020] Further aspects of the present disclosure are described in the following sections: (I) Definitions, (II) Light Field Imaging System, (III) Geometric Mapping Between Object Space and Light Field Space, (IV) Calibration Targets to Enable the Geometric Mapping, (V) Calibration and Mapping Methods, and (VI) Examples.

[0021] I. Definition Technical terms used in this disclosure have meanings commonly recognized by those of ordinary skill in the art. However, the following terms may be further defined as follows:

[0022] Computer - an electronic device for storing and processing data, typically in binary form, according to instructions that may be provided by a variable program. Exemplary computers, also referred to as computing devices, include desktop computers, laptop computers, tablets, smartphones, and the like.

[0023] Test Volume - the volume, intersected by the optical axis of a light field imaging system, in which objects such as calibration targets or samples (e.g., biological cells) may be imaged using an image sensor.

[0024] Feature - A marker or absence of a marker on a calibration target at a location that can be precisely defined by extrapolation and / or interpolation over an organized set of markers on the calibration target.

[0025] Image - an optical or digital representation of an object and / or field of view. If the representation is optical, it may be formed by light at a plane in space (e.g., an intermediate image plane) or at a surface (e.g., of an image sensor), among others. If the representation is digital, it may be a raw light field image captured by an image sensor, a processed form thereof, or a two-dimensional or three-dimensional image in object space created using at least one light field image.

[0026] Image Sensor - an optical sensor capable of detecting spatial variations in light (e.g., variations in intensity) in two dimensions, where the light is incident on a light-sensitive detection area of ​​the sensor. The image sensor may be a two-dimensional array sensor, such as a charge-coupled device (CCD) sensor, an active pixel sensor (e.g., a complementary metal-oxide-semiconductor (CMOS) sensor), a hybrid CCD-CMOS sensor, or the like. The image sensor may create a raster image (i.e., a bitmap) as a rectangular array of pixels, or may be configured to create color images, grayscale (monochrome) images, or both.

[0027] Lens array - a two-dimensional array of lenses. Exemplary lenses that may be suitable in an array for light field microscopy are microlenses, i.e., lenses each having a diameter of less than about 1 millimeter. Exemplary microlens diameters that may be suitable for microlens arrays include, among others, less than about 750, 500, 400, 300, 200, or 100 micrometers per microlens. Exemplary lens diameters for light field close-up include lenses having a diameter of at least 1 millimeter, or lenses that are microlenses as described above in some cases.

[0028] The array may have any suitable properties. The pitch (inter-lens spacing) of the array may (but may not) be similar to the lens diameter, such as being no more than about 50%, 25%, 20%, or 10% greater than the lens diameter. The lenses may form a rectangular or hexagonal grid, among others. The array may contain any suitable number of lenses, generally at least four. In some embodiments, the array may contain at least 25, 50, or 100 lenses, among others. For example, the array may contain at least 2, 3, 4, 5, 6, 7, or 10 lenses along each of a pair of orthogonal axes defined by the array. The lenses of the array may be substantially the same as one another in size and / or shape, or may differ in size and / or shape. Additionally, the array may be an integrated unit and / or the lenses of the array may be integrally formed with one another.

[0029] Light - optical radiation, including ultraviolet radiation, visible radiation (i.e., visible light), and / or infrared radiation.

[0030] Light Field Image - An optical image formed with light by a microlens array on an image sensor, consisting of a two-dimensional array of sub-images, or their captured (digital) forms, that correspond to the microlenses of the array. The sub-images generally, and importantly, do not overlap one another, if at all. The sub-images may or may not be substantially in focus.

[0031] Light source - a device that generates light, optionally as a beam of light, and optionally powered by electricity. A light source includes at least one light emitting element and may also include any associated optical elements that shape, resize, filter, polarize, scatter, direct, and / or otherwise interact with the light emitted by the light emitting element. These optical elements may include any combination of at least one waveguide (e.g., fiber optics or liquid light guides), lenses, mirrors, filters, diffusers, masks, apertures, beam splitters, gratings, prisms, polarizers, and / or the like. Exemplary light emitting elements include semiconductor devices, lasers (e.g., excimer lasers, gas lasers, dye lasers, solid state lasers, semiconductor crystal or diode lasers, free electron lasers, etc.), arc lamps, and / or the like. Exemplary semiconductor light emitting elements include laser diodes, light emitting diodes (LEDs), and superluminescent diodes, among others.

[0032] Objective Lens - a single optical element or combination of optical elements that collects light from an object and / or field of view and focuses the collected light. Objective lens as used herein may include each optical element located along the optical axis between the inspection region and the microlens array of the light field imaging system.

[0033] Optical system - a set of optical elements of an imaging system that may be arranged along an optical path between a light source and an inspection region (illumination optics) and / or along an optical axis between an inspection region and an image sensor (collection optics). An optical element may be any device or structure that interacts with light, such as collecting, directing, focusing, filtering, polarizing, scattering, collimating, and / or partially blocking light. Optical elements may function by any suitable mechanism, such as reflection, refraction, scattering, diffraction, absorption, and / or filtering, among others. Exemplary optical elements include lenses, mirrors, diffusers, gratings, prisms, filters, apertures, masks, beam splitters, waveguides, polarizers, and the like.

[0034] Pinhole - a hole or opening having a diameter or width of less than about 1 millimeter. Exemplary pinholes that may be suitable for creating the features of the calibration targets disclosed herein have diameters or widths of less than about 100, 50, 20, or 10 micrometers, among others. The pinhole may have any suitable shape. For example, the pinhole may be circular, rectangular, or the like.

[0035] Sample - a specimen, having any suitable nature. A sample may be organic and / or inorganic, natural and / or manufactured, and may include any suitable assembly, material, substance, isolate, extract, particle, or the like. In an exemplary embodiment, the sample includes biological cells (such as isolated cells or one or more populations of cells). The biological cells may be eukaryotic (e.g., mammalian cells) or prokaryotic (e.g., bacterial cells). Exemplary biological cells include established cells (cell lines), primary cells, cells of a tissue sample, transfected cells, cells from a clinical sample (e.g., blood sample, fluid aspirate, tissue section, etc.), clones of cells, and / or the like. The cells are contained by a sample holder and may optionally be in contact with (e.g., infiltrated into) any suitable liquid medium. The liquid medium may be an aqueous medium, which may include, among others, water, salts, buffers, glucose, detergents, dyes, proteins, amino acids, or any combination thereof. The liquid medium may be a growth medium for the cells.

[0036] Sample holder - a device for holding at least one sample or any array of spatially isolated samples, and optionally allowing the samples to be imaged through a horizontal transparent wall of the device (e.g., the bottom wall of a well). Exemplary sample holders for light field microscopy include microplates, culture dishes, culture flasks, slides, flow channel chips, and the like.

[0037] z-stack of light field images - a series of light field images of an object (e.g., a calibration target) captured while the object is located at different z-positions in object space, optionally without changing the lateral position of the object in object space.

[0038] z-position - one of the locations along the optical axis of a light field imaging system where an object may be located. The z-position may be defined as the distance between the object and the objective lens of the light field imaging system, measured parallel to the optical axis, and may be changed by moving the object or the objective lens parallel to the optical axis.

[0039] II. Light Field Imaging System This section provides an overview of light field imaging systems of the present disclosure, as exemplified by light field imaging system 50 (see FIGS. 1-5).

[0040] The light field imaging system 50 includes a light field microscope 52 (or a light field camera for light field close-ups) and a calibration target 54. The light field microscope 52 (and / or the light field camera) has a stage 56 for supporting either the calibration target 54 or other objects of interest (e.g., a sample holder, containing one or more samples). The calibration target or other objects are supported within an examination region 58 on an optical axis 60 defined by an objective lens 62. The calibration target 54 may define an object plane 64 (xy plane) that is orthogonal to the optical axis 60 (z-axis) and may be located near a front focal plane of the objective lens 62 (and in particular its collection module 66) when the calibration target is operably supported by the stage 56. The calibration target 54 may be illuminated with optical radiation (e.g., visible light) generated by a light source 68, and only a spatially limited portion of the optical radiation may be transmitted by the calibration target 54 to the collection module 66 of the objective lens 62. The transmitted optical radiation may propagate through a microlens array 72 having, for example, a focusing module 66, an optional tube lens 70, and any suitable number of microlenses 74 arranged in a two-dimensional array. The individual microlenses 74 of the microlens array 72 may at least partially focus the optical radiation onto an image sensor 76 to form individual sub-images, each corresponding to a different one of the microlenses. The image sensor 76 captures a light field image of the sub-images, which is communicated to a computer 78 for processing (see, for example, Sections III-VI, among others). In other embodiments (e.g., for light field close-ups), the microlens array 72 may be replaced by a lens array, composed of lenses, not necessarily microlenses. In other embodiments, the calibration target 54 may reflect or emit light that forms the light field image.

[0041] The light field microscope 52 allows adjustment of the z-position of the calibration target 54 along the optical axis 60 via a drive mechanism 80. More specifically, the drive mechanism can operate to vary the distance between the calibration target 54 and the collection module 66 along the optical axis 60. Thus, the drive mechanism 80 may be coupled to the stage 56 as shown in FIG. 1 and drive the movement (e.g., vertical movement) of the stage while the z-position (e.g., elevation) of the collection module 66 remains fixed. Alternatively, the drive mechanism 80 may be coupled to the collection module 66 (and / or the objective lens 62) and drive the movement (e.g., vertical movement) of the collection module 66 (and / or the objective lens 62) while the z-position (e.g., elevation) of the stage 56 remains fixed. The drive mechanism 80 may include a motor 82 (or may be manually driven) and may be controlled by the computer 78 for z-positioning during implementation of a fully automated calibration method.

[0042] The optical axis 60 may have any suitable orientation and nature. The optical axis is typically vertical at the collection module 66 and may extend vertically upward (as shown here) or vertically downward from the collection module 66 through the inspection region 58 and the object plane 64 (and the calibration target 54 or other object disposed therein). In other words, the light field microscope 52 may have an inverted configuration with the collection module 66 below the inspection region 58, as shown here, or a non-inverted configuration with the collection module 66 above the inspection region 58. However, in other embodiments, the optical axis 60 may be horizontal or oblique (neither substantially horizontal nor substantially vertical) at the collection module 66. The optical axis 60 may not be folded as shown here, or may be folded (e.g., with a mirror) at one or more locations along the optical axis.

[0043] The light source 68 may be arranged to provide any suitable illumination of the inspection region 58 using any suitable light source (see, e.g., Section I). The illumination may be trans-illumination (as shown here), epi-illumination, oblique illumination, or combinations thereof, among others. Any suitable optical elements may be positioned on the illumination path between the light source 68 and the inspection region 58 and operable to direct and / or condition the optical radiation generated by the light source upstream of the inspection region. Exemplary optical elements positioned in the illumination path may include a diffuser, a spectral filter, a collimating lens or mirror, a folding mirror, a focusing module 66, a light guide, and / or the like.

[0044] 2 shows a ray diagram with only selected components of the light field microscope 52. The diagram depicts how optical radiation may propagate from an object point 84 in an object plane 64 in object space to multiple distinct light field image points / spots 86 on the image sensor 76. The optical radiation is represented by upstream marginal rays 88, which extend from the object point 84 to an intermediate point 90 in an intermediate image plane 92, and downstream marginal rays 94, which extend from the intermediate point 90 to a sensor plane 96 defined by the image sensor 76 in the two-dimensional light field space. In this exemplary configuration, the object plane 64, the intermediate image plane 92, and the sensor plane 96 are mutually conjugated such that the optical radiation is precisely focused to the light field image point 86.

[0045] A plan view of an exemplary microlens array 72 is shown in FIG. 3. The microlenses 74 may be arranged in a two-dimensional array. The microlenses may be distributed along a pair of orthogonal axes in the plane of the array as shown to define a rectangular grid, although non-rectangular arrangements of microlenses may be used. In some embodiments, the microlenses 74 may be uniformly spaced from one another along each orthogonal axis of the pair of axes, and may optionally have the same spacing, i.e., pitch d, from one another along both of the axes as shown.

[0046] The distance between image points 86 may exceed the spacing of microlenses 74 (e.g., pitch d in FIG. 3) due to parallax (see FIG. 2). Optical radiation from midpoint 90 may be incident on at least a subset of individual microlenses 74 of microlens array 72 at different angles for each microlens. In other words, each chief ray propagating from midpoint 90 through each microlens of the subset has a different orientation. Optical radiation from midpoint 90 may be incident only on a subset of microlenses of microlens array 72 as shown because the cone size of the optical radiation propagating from midpoint 90 may be limited by an aperture (e.g., stop element 97) located upstream (e.g., in collection module 66) (however, light field microscope 52 generally has a non-telecentric optical configuration such that magnification varies with distance from collection module 66). As an object point 84 is moved sufficiently laterally in the object plane 64, different subsets of microlenses may receive optical radiation from the object point 84. Thus, a set of sufficiently spaced object points 84 in the object plane 64 may be imaged by different subsets of microlenses and may have conjugate points / spots in different subsets of sub-images in the light field image captured by the image sensor 76.

[0047] In other cases, the distance between image points 86 may be less than the physical spacing of the microlenses 74 (e.g., pitch d in FIG. 3). For example, in a hyper-perfect Galileo type setup, the pitch of the projected images of the microlenses on the image sensor may be less than the physical pitch of the microlens array. Similarly, in this setup, the distance between image points 86 is less than the (projected and physical) pitch of the microlens array.

[0048] 2 collimates the optical radiation using an upstream infinity-corrected collection module 66 and focuses the collimated radiation using a downstream tube lens 70. In other embodiments, the tube lens 70 may be eliminated and the optical radiation may be both collected and focused, for example, using the collection module 66.

[0049] In some embodiments, the stage may be replaced by a manual mechanism for holding a calibration target or other object at different z-positions. For example, a user may set and adjust the z-position using one or more spacers of known size manually placed on top of the objective lens. Each spacer may be, for example, about 5, 10, 15, 20, 25, 30, 40, 50, or 100 micrometers thick, among others.

[0050] FIG. 2 shows the object plane 64 located at z-position z1, which is precisely conjugate with the sensor plane 96. However, this precise relationship with respect to the location of the object is not required. More specifically, light field images suitable for the methods of the present disclosure may be captured by the image sensor 76 with the object plane 64 located closer to the light collection module 66 at z-position z2 and farther from the light collection module 66 at z-position z3 (see FIGS. 4 and 5, respectively). In either case, the optical radiation from the intermediate point 90 is not as focused at the sensor plane 96 as in FIG. 2, so that the image point 86 in FIG. 2 becomes a larger image spot in FIGS. 4 and 5.

[0051] Further example aspects of light field imaging system 50, light field microscope 52 (or light field close-up system), and calibration target 54 that may be suitable are described elsewhere herein, such as in Sections I and III-VI.

[0052] III. Geometric Mapping Between Object Space and Light Field Space This section describes the basics of a geometric mapping function that correlates object space and light field space with respect to an object plane at a given z-position along the optical axis (see FIGS. 6A and 6B). The mapping function may be used, among other things, in conjunction with a light field microscope or light field close-up system.

[0053] The light field microscope 52 of Figures 6A and 6B may be constructed as in Figures 1 and 2, but is depicted in a somewhat different form. The objective lens 62 is shown as a single lens instead of a separate collection module and tube lens, and the microlens array 72 is shown in a more stylized form. These simplifications emphasize that the objective lens and microlenses may otherwise have a "black box" structure, except for the relevant parameters explained below. Furthermore, one of the microlenses 74, which has been selected to illustrate the mapping, is labeled as the kth microlens "74k". The object point 84 is located at the object plane 64 as in Figures 2, 4, and 5, but is offset laterally from the optical axis 60 (compared to Figure 2). Marginal rays 88 and chief ray 98 extend from the object point 84, through the objective lens 62, to a midpoint 90. Marginal rays are shown extending downstream of image point 90 to indicate the boundary of the cone of optical radiation incident on microlens array 72 from object point 84. Chief ray 98 is shown in Figure 6A as terminating earlier than usual at midpoint 90 to reduce drawing complexity downstream of midpoint 90.

[0054] A mapping function relating the three-dimensional object space to the two-dimensional light field space can be defined for microlens 74k as follows: [ka] where the subscripts xy indicate the lateral components of the associated vector. This equation holds regardless of whether the imaging system displays an inverted image and whether the microlens array images a real (inverted or non-inverted image) or a virtual image (inverted or non-inverted image). The presence or absence of inversion is contained in the sign of the corresponding magnification value, as explained below.

[0055] A schematic of the mapping geometry for Equation 1 is shown in Figures 6A and 6B, which illustrate the imaging of real and virtual inverted images by a microlens array, respectively. [ka] is a vector representing an object point 84 in the object plane 64 of the three-dimensional object space, [ka] is a vector representing the position of the optical axis of the microlens 74k, and M Total is the total magnification of the system (e.g., produced by the combination of objective lens 62 and microlens array 72), and M ML is the magnification due to the microlens array 72 alone, [ka] is in object space [ka] is a vector that represents a point in the two-dimensional light field space (sensor plane 96) corresponding to (x,y,y,=1,x,y,=1,x,y,=0,x,y,=0,x,y,=1 ...

[0056] Vector [ka] have a tail at optical axis 60. However, the optical axis used for mapping need not be an actual optical axis, but may instead be defined as a virtual global optical axis from the light field image, as described elsewhere herein (e.g., see Section V). The downstream chief rays 100 extend from the intermediate image point 90 through at least a two-dimensional subset of microlenses 74 that corresponds to the number of light field image points on image sensor 76, as described in Figures 2, 4, and 5 above. One of the chief rays 100 passes through microlens 74k, [ka] 4, is incident on the sensor plane 96.

[0057] Equation 1 can be derived as follows: [ka] is the vector [ka] The image is mapped to an intermediate image point 90, represented by [ka] In the formula, M objective is the collective magnification provided by the objective lens 62 of the light field microscope 52 at the intermediate object plane 64, and r xy ' is a vector [ka] In the case of Figures 6A and 6B, M objective will be a negative number.

[0058] Intermediate image point [ka] is expressed as: [ka] is mapped to. [ka] In the case of Figure 6A, M ML In the case of Figure 6B, M would be negative, whereas M would be negative, since it would invert again the real image produced by the microscope. ML will be a positive number.

[0059] Equation 3 can be rearranged and regrouped to generate equations 4 and 5. [ka] According to Equation 2, in Equation 5 [ka] Substituting results in Equation 6. [ka] The magnification within the light field microscope 52 can be described by Equation 7. [ka] Substituting in the right hand term of Equation 6 according to Equation 7 results in Equation 1.

[0060] The mapping geometry is well defined once the total magnification and the microlens magnification are known (e.g., measured). In general, each of these magnifications depends on the distance between the object plane 64 and the collection module 66 (i.e., the z-position of the object plane 64). Local aberrations (e.g., due to the microlens array) can be represented by the total magnification, which varies with each microlens image. This leads to Equations 8 and 9. [ka]

[0061] The magnification due to the microlens array can be determined using the calibration target by evaluating the disparity (also called virtual depth) between nearby views of the same object point of the calibration target using Equation 10. [ka] In the formula, d MLA is the pitch of the microlens array, and x i ”(z) is the (z-dependent) position of the common object point in the image of the i-th microlens. M ML To determine the correct labeling of, it is necessary to know whether the microlens array is placed upstream or downstream of the image plane of the microscope image plane. In the first case, the microlens array images a virtual image, in the second case a real image. In general, this does not limit the applicability of the approach, since the location of the microscope image plane is either located at the native image sensor location or can be easily located by moving the image sensor along the z-axis downstream of the microscope objective relative to the microlens array. Total The sign of is also M objective and M. MLNote that the inversion nature of and associated sign of the total magnification may vary. However, only the sign change of the total magnification leads to an inversion in object space, i.e., is generally not relevant. In other words, it is only necessary to know whether the microlens array is inserted between the original image plane of the microscope and the upstream optics or downstream of the original image plane. It is not necessary to know the exact distance between the microlens array and the original image plane.

[0062] Further aspects of exemplary calibration targets and their use to measure total and microlens magnification values, radial distortion values, and point spread functions are further described below in Sections IV-VI.

[0063] IV. Calibration Targets to Enable Geometric Mapping This section describes an exemplary calibration target, as exemplified by embodiment 102 of calibration target 54 (see FIGS. 7, 8A, 8B, and 9).

[0064] The calibration target 54 may include at least one optically detectable marker located within and / or on a background layer. The at least one marker and the background layer differ from one another with respect to at least one optical property, such as transmittance, reflectance, photoluminescence, and / or the like. For example, each marker may be substantially more transmissive, reflective, and / or photoluminescent than the background layer in response to illumination with optical radiation from a light source of the light field imaging system, allowing the marker to be optically distinguished from the background layer. Each marker or marker-defining feature may be interchangeably referred to as a fiducial.

[0065] In some embodiments, the calibration target may include a plurality of optically detectable markers arranged in a two-dimensional array parallel to a plane defined by the background layer. The markers may be of a single type or of at least two types that are optically distinguishable from one another in a light field imaging system, such as by size, shape, and / or optical properties.

[0066] At least a subset of the markers may be arranged along a number of parallel lines and / or along a number of non-parallel lines (e.g., mutually orthogonal lines) to form a grid. Thus, at least a subset of the markers may define a grid and / or be arranged with their centers at the nodes of the grid. The grid may be rectangular or hexagonal, among others. The period (i.e., spacing) between adjacent nodes of the grid may be uniform along at least one axis of the grid and / or along a pair of non-parallel (e.g., orthogonal) axes defined by the grid. Markers of the same first type (e.g., having the same size and / or shape) may be located at only a subset of the nodes in the grid, creating a first repeating structure with a first period. Markers of a second type or no markers may be present at other nodes in the grid, defining a second repeating structure with a second period. The second period may be an integer multiple of the first period.

[0067] Each marker may have any suitable size and shape, and the markers may collectively define any suitable spacing and / or period. The size of each marker may be defined and known. Exemplary marker sizes include widths and / or diameters of less than about 100, 50, 25, 20, 15, 10, 5, or 2 micrometers, among others. The size of at least one type of marker may be less than the period defined by the marker, e.g., the period is at least 2, 3, or 4 times its size.

[0068] Each marker may have any suitable structure formed by any suitable process. The marker may be an opening, such as a pinhole or slit, in the background layer. The opening may be created in the background layer by material removal, such as laser drilling (e.g., laser induced forward transfer), photolithography, or the like. In other embodiments, the marker may be a chemically altered area, such as a bleached area, of the background layer. In yet other embodiments, the marker may be created by depositing a material onto the background layer. The deposited material may be, for example, photoluminescent or reflective, among others.

[0069] The calibration target 54 may satisfy the first requirement as described in this paragraph. The calibration target may contain at least one optically detectable marker of well-defined size, or an array of at least two markers of well-defined period (or interval), allowing measurement of the total magnification of the light field system for a range of z-positions. The maximum suitable size of the marker or array period still allows the marker or array to be imaged in its entirety in a single microlens sub-image over the entire axial range of z-positions that are explicitly calibrated. The minimum size of the marker or array may be determined by the lateral resolution of the light field system. More specifically, the marker or array should be large enough to be resolved for all z-positions that are explicitly calibrated in object space.

[0070] The calibration target 54 may satisfy the second requirement, as described in this paragraph. The calibration target may contain a specific instance of a marker, or a specific instance of a feature defined by at least two markers that are imaged in at least two microlens sub-images of the light field image and whose location may be determined at or above pixel resolution, allowing measurement of microlens magnification for multiple z-locations in object space. The locations may be, for example, edges, corners, or center of mass of the marker or feature. The type or instance of the marker or feature used to measure the total magnification may be the same or different from that used to measure the microlens magnification.

[0071] A calibration target may meet the first and second requirements described above, inter alia, by containing only one instance of a single type of marker, multiple copies of only one type of marker, or multiple copies of at least two different types of distinguishable markers. For example, a calibration target may have a single periodic sequence, or a superposition of two or more different periodic sequences formed by copies of one or more types of markers. In the case of two superposed periodic sequences, the period of at least one of the sequences should be accurately known, and the structure should meet the first and second requirements. The periods of the different sequences do not need to have a clearly defined relationship to each other, provided that the above first and second requirements are met.

[0072] The calibration target does not need to cover the entire field of view. It is sufficient if a relevant marker or marker array of known size can be imaged in the two microlens images over the entire range of relevant z-positions. It may be advantageous to have the markers of the calibration target distributed or extending across most or all of the field of view in order to be able to determine local aberrations in the magnification parameters and increase accuracy. In other words, at least one marker of the calibration target may be represented in at least 50%, 75%, 80%, 90%, or all of the sub-images of each captured light field image of the calibration target in order to calibrate the light field imaging system.

[0073] An illustrative calibration target 102 is shown in FIGS. 7, 8A, 8B, and 9 with light-transmissive regions represented in black and light-blocking regions represented in white (except for FIG. 9). The calibration target may include a flat substrate 104 having a planar top or bottom surface 106. A background layer 108 may be bonded or otherwise attached to the surface 106. The background layers form or support at least one or more optically detectable markers 110, each having any suitable combination of the properties described above. In the depicted embodiment, the markers 110 are created by pinholes 112, which are optically transmissive with respect to the wavelength of optical radiation being used for illumination and / or imaging (see FIG. 9). Additionally, the substrate 104 may be substantially transmissive or transparent with respect to optical radiation, and the background layer 108 may be substantially non-transmissive (i.e., blocking or opaque) with respect to optical radiation. In some embodiments, the substrate 104 may be omitted from the calibration target 102 such that only the background layer 108 and / or the calibration target 102 are a single layer.

[0074] In order to be able to determine the magnification parameters with high accuracy (by averaging several measurements) and to be able to characterize local aberrations (e.g. due to optical aberrations) from the average, the calibration target 102 may include a superposition of at least two periodic arrays extending over the entire field of view. For example, in FIG. 8A two periodic arrays are visible: a first periodic array of markers 110 (which may be identical along a pair of orthogonal lateral axes) that defines a marker period 114, and a second periodic array of features 116 (generally indicated with dashed boxes) in which the markers 110 are periodically absent. The features 116 define a feature period 118, which may be identical along a pair of orthogonal lateral axes. FIG. 8B shows a rectangular grid 120 defined by the first periodic array of markers 110 and superimposed on the calibration target. The markers 110 may be periodically absent from the nodes 122 of the grid 120, creating a second periodic array, as shown in FIG. 8B. In other embodiments, different types of markers may be present at these nodes (see, e.g., Example 1). In other embodiments, the second periodic array may be offset from the grid 120 and / or may have a characteristic period 118 that is not related to the node spacing of the grid 120.

[0075] The marker period 114 may be utilized to determine the total magnification. The period may be optimized such that multiple nearby pairs of markers 110 are imaged by each microlens for each z-position, which may allow the total magnification, and any local aberrations thereof, to be determined with high accuracy. In some cases, at least 2, 3, 4, or more instances of the marker period 114 may be imaged in each lateral dimension per microlens image.

[0076] The calibration target 102 has a two-dimensional array of markers 110 with a fixed period, which [ka] (i.e., total magnification as a function of z-position and microlens position), satisfying the first requirement described above. In this periodic structure, every nth marker 110 is missing (e.g., n is at least 2, 3, 4, 5, or more). These missing markers 110 constitute a second periodic feature 116, which is used to determine the microlens magnification as a function of z-position, satisfying the second requirement described above. The exact location of each node 122 of the grid 120 where a marker 110 is missing can be determined with high accuracy by interpolation between (or extrapolation from) nearby grid nodes. If the markers 110 are of similar size or smaller than the diffraction-limited resolution, the point spread function of the system can also be extracted.

[0077] The periodic arrangement of features 116 used to determine a value for microlens magnification satisfies the second requirement described above. The period of the features is generally selected such that the correspondence of a particular instance of a feature in neighboring sub-images 134 is uniquely defined. To meet this criterion, it is sufficient if only one instance of a feature 116 is imaged by each microlens. If multiple instances of a feature are imaged by each microlens, the period should be such that the correspondence between instances of a feature in neighboring views is (e.g., M ML The correct value for,M,is extracted from all possible combinations of correspondences between the two sub-images. ML The θ should be chosen so that it can be established through a simple assumption (that out of all possible values ​​for θ, θ is closest to zero).

[0078] A variety of other calibration targets may be suitable for light field imaging systems and may satisfy the first and second requirements described above. For example, a calibration target may be: [ka] The marker 110 may include only a single instance of a marker 110, such as a circle of a defined size, for determining M ML (z). In another embodiment, the calibration target can be used to determine [ka] To allow for the determination of M ML To determine (z), the corner of one line of the grid can be used.

[0079] V. Calibration and Mapping Methods This section describes an exemplary method of calibrating a light field microscope and the mapping between light field space and object space after calibration. The method steps described in this section may be performed in any suitable order and combination using any of the system components, calibration targets, and / or other aspects of the present disclosure. Each of the method steps disclosed herein may be performed by and / or under the control of a computer. The method steps are illustrated with an embodiment of a light field imaging system 50 equipped with a calibration target 102 and a 10×10 array 72 of microlenses 74 (see FIGS. 10-14 (also see FIGS. 1, 2, and 7-9)).

[0080] A calibration target may be placed on the stage 56 of the light field imaging system 50 and located at a relevant z-position in object space. At the relevant z-position, the calibration target intersects a volume of interest in object space where a sample or other object of interest will be placed for imaging. For calibration, a z-stack of light field images of the calibration target may be recorded. More specifically, individual light field images of the calibration target may be captured with the image sensor 76 of the light field imaging system by adjusting the location of the stage 56 or collection module 66 of the light field microscope 52 along the z-axis (i.e., parallel to the optical axis 60) at each of at least two, three, or more relevant z-positions of the calibration target in object space. The lateral position (orthogonal to the optical axis 60) of the calibration target may or may not be the same for the captured light field images, but generally there is no incentive to change this lateral position during calibration with the same calibration target.

[0081] 10-12 show exemplary negative light field images 130 of a calibration target 102 that may be captured by image sensor 76 of light field imaging system 50 while the calibration target is supported by stage 56 at one of the z-positions (see also FIGS. 7-9). Light field image 130 consists of an array 132 of sub-images 134, each sub-image 134 corresponding to and generated by a different microlens 74 of array 72. Sub-images 134 may be distinct from one another (i.e., non-overlapping) or may overlap slightly to facilitate subsequent image processing. The separation distance between neighboring sub-images 134 is generally minimized to maximize the amount of image data contained by each sub-image.

[0082] The sub-images 134 represent overlapping views of the calibration target 102 and are arranged along lateral axes 136, 138 defined by the light field image 130. Thus, a given marker 110 and / or feature 116 of the calibration target 102 may be represented by a corresponding (imaged) marker 140 and / or (imaged) feature 142 in at least two neighboring sub-images 134 along each axis 136, 138. For example, in the depicted embodiment, the markers 110 and features 116 of the calibration target 102, which are more centrally imaged in the light field image 130, may each be presented centered in a 3×3 array of sub-images 134, although in other embodiments, each marker / feature may be represented in a smaller or larger two-dimensional array of the sub-images 134 of the light field image 130.

[0083] The position of a given imaged marker 140 or imaged feature 142 shifts in adjacent sub-images 134 along each axis 136, 138 due to parallax. This range of shifts can be used to calculate the magnification of the microlenses 74, as described elsewhere herein. Figure 11 shows how feature 142a, which represents the same instance of feature 116 in calibration target 102, shifts along axis 136 by approximately two columns of markers 140 between successive sub-images, and along axis 138 by approximately two rows of markers 140 between successive sub-images.

[0084] An optical axis may be defined for each light field image. The actual intersection of the true optical axis of the microlens array with the light field image is generally not known. Instead, a virtual (global) optical axis may be defined during pre-processing of the light field image by any suitable method. For example, the light field image may be centered and cropped so that the two central sub-images in each lateral dimension abut exactly at the center of the sub-image array, i.e., the four central pixels of the sub-image array abut. This point may be defined as the point where the virtual optical axis intersects the orthogonal projection of the microlens array into the light field space (the virtual optical axis so defined may also be assumed to intersect the physical microlens array at a point centered between the four microlenses). This definition of the optical axis may be sufficient for both calibration (where only the distances in the light field image are evaluated) and mapping from the light field space into the object space (e.g., based on Equation 1) (see also Section III). The mapping is sufficiently accurate with this definition of the optical axis because the shift between the true optical axis and the virtual optical axis corresponds to a global shift in object space, which is inconsequential for most applications. Figures 10 and 11 show the virtual optical axis 144 in a light field image 130 defined according to this approach.

[0085] Sub-images of each calibration light field image may be identified. For example, the center point, size, and / or boundary of each sub-image may be determined by processing the light field image. Figure 13 shows one of the sub-images 134, where a center 146 and boundary 148 are identified for the sub-image.

[0086] The location of the markers / features within each sub-image 134 may be detected. For example, the center of mass or point of maximum intensity may be determined for each marker 140 (and / or feature 142, if the feature is created by a different type of marker than marker 140 (see, e.g., Example 1)).

[0087] A mesh may be created for each sub-image using the detected positions of the markers / features by quad- plexing. FIG. 13 shows an example mesh 150 created by quad- plexing. Mesh 150 corresponds to a portion of grid 120 defined by calibration target 102. The mesh may be rectangular, as shown, if no substantial distortion is generated by the system optics (e.g., microlens array). FIG. 14 shows an example sub-image 152 of calibration target 102 in which pincushion radial distortion is generated by corresponding microlenses 74 of microlens array 72. Here, mesh 150 is not rectangular. In other cases, the radial distortion may be a barrel distortion.

[0088] The imaged features of the calibration target may be detected. For example, a node of mesh 150 (with marker 140 missing from its periodic arrangement) that represents imaged feature 142 may be identified and its location determined (see FIG. 13). In other embodiments, imaged features produced by different types of markers (see, e.g., Example 1) may be detected and its location determined.

[0089] The total magnification of the light field imaging system can be determined. The total magnification can be calculated according to Equation 11: [ka] During the ceremony, Dist LF is the distance between two points in a sub-image of a light field image, Dist OBis the distance between the same two points in object space. The two points are defined by the calibration target in object space and the two-dimensional light field space, and may be provided, among other things, by a single marker of the calibration target (e.g., a characteristic dimension (such as diameter or length) of a marker of a defined size), or by a nearby pair of markers (e.g., the center-to-center or edge-to-edge distance between the pair of markers). In an example embodiment, the distance in object space is the known marker period 114 (see, e.g., FIG. 8A), and the corresponding distance in the two-dimensional light field space is the measured period 156 (see FIG. 12).

[0090] In some embodiments, the measured period 156 may be determined by fitting the positions of the imaged markers 140 to a regular grid. The period of the grid is calculated by Dist LF to provide.

[0091] A global total magnification can be determined. For example, the global total magnification can be calculated as the average of the measured period 156 over the known period 114. Local aberrations can be determined with a resolution of about one period and can therefore be incorporated into the reconstruction process (e.g., see Section III), such as by employing locally varying total magnifications or by correcting the captured light field images in a pre-processing step.

[0092] At least one microlens magnification may be determined. For example, an average microlens magnification for the microlens array may be determined and then used for mapping from light field space to object space, or a microlens-specific magnification may be determined and used for mapping. The magnification of the microlens array is determined by measuring the disparity between different views of the same object point of the calibration target. The views do not need to be provided by sub-images that are adjacent to each other, as long as the same object point is imaged. For example, the microlens magnification may be calculated as shown in Equation 12. [ka] where Δ is the disparity of the reference object point, n-1 is the number of microlens sub-images between the two views under consideration, and d is the period of the microlens array. Similarly, the microlens magnification can be determined according to Equation 10. For example, the distance 158 between a given feature 142a in neighboring sub-images 134 can be determined (see FIG. 11 ), and the distance 158 and the microlens pitch d MLA The ratio of can be calculated. A value for the (nominal) pitch of a microlens array may be obtained from the array's manufacturer or measured by the user, among others.

[0093] At a defined z-position, the total and microlens magnification values ​​obtained as described above can then be interpolated or extrapolated to obtain corresponding magnification values ​​at other z-positions that have not been specifically calibrated.

[0094] A vignette distribution may be measured. For example, the vignette distribution may describe the intensity of the imaged markers 140 according to lateral position. The intensity may be the peak intensity, total intensity, or the like, per imaged marker.

[0095] The field distortion of each light field image may be determined. The field distortion may be radial distortion, such as pincushion distortion or barrel distortion, caused by individual microlenses 74 of the microlens array 72 in the corresponding sub-image of the light field image. The radial distortion for a given microlens may be calculated, for example, as the deviation of the measured marker positions for the (i) corresponding sub-image from a least-squares fitted regular grid defined by the (ii) marker positions. The average radial distortion may be calculated, or the microlens specific radial distortion may be used for the mapping. The radial distortion may be taken into account in the mapping function (e.g., Equation 1) when mapping from light field space to object space (i.e., from a captured light field image of a sample or other object of interest). For example, the radial distortion may be taken into account using a correction factor that depends on the radial distance of the sub-image points from the center of the microlens sub-image. The global radial distortion due to the objective lens may also, or alternatively, be evaluated generally with a resolution of about the size of each microlens. The change in value for the average total magnification per microlens across the microlens array can be evaluated to determine the global radial distortion.

[0096] A point spread function may be determined from at least one of the sub-images of each calibration light field image. For example, a deconvolution operation may be performed to calculate a point spread function for the microlenses that created the sub-images using the markers 140 imaged in the sub-images and the corresponding markers 110 of the calibration target. An average point spread function may be calculated for the microlens array for use with all of the captured sub-images in the light field image for mapping into the object space. Alternatively, each microlens may be assigned a different point spread function to be used for mapping.

[0097] A geometric mapping between the object space and the light field image space may be performed based on the first-order optics (collinear transformation) in Equation 1, etc. This mapping may allow the object to be reconstructed, at least in part, by forward projection of the light field image from the light field space onto one or more planes of the object space. The mapping may alternatively or additionally allow the light field image (or a portion thereof) to be reconstructed by backward projection of the object from the object space into the light field space. The forward projection to form a two-dimensional image or to reconstruct the object in three dimensions may correct distortions (e.g., radial distortions) produced by the microlenses and / or other optical elements of the imaging system (e.g., the objective lens). The forward projection may alternatively or additionally utilize at least one point spread function (e.g., the average point spread function for the microlenses) to reduce blurring.

[0098] VI. Working Examples This section describes selected embodiments and aspects of the light field imaging system, calibration target, calibration method, and mapping method of the present disclosure. These embodiments and aspects are intended for illustration purposes only and should not limit the overall scope of the present disclosure.

[0099] Example 1: Further calibration targets This example describes a further exemplary calibration target 54 that may be suitable for the systems and methods of the present disclosure (see FIGS. 15-18).

[0100] 15 shows an example embodiment 170 of a calibration target 54 having two types of periodic markers 110, 172 located at the nodes of the same rectangular grid. Calibration target 170 is similar to calibration target 102 (see FIGS. 7 and 8A), but markers 172 replace features 116 (created by the absence of any marker at the nodes of the grid defined by markers 110). Each copy of marker 172 has a size and / or shape that makes it optically distinguishable from each copy of marker 110, and may be repeated periodically to define a feature period 118 for the markers.

[0101] 16 illustrates another exemplary embodiment 180 of calibration target 54 having only a single type of marker 110. Calibration target 180 is identical to calibration target 102, except that every tenth marker 110 is missing to form a feature 116, instead of every fifth marker.

[0102] 17 and 18 show yet another exemplary embodiment 190 of the calibration target 54. The calibration target 54 has a repeating unit 192 consisting of two sets of gratings 194, 196 aligned perpendicular to one another. In other embodiments, only one unit 192 may be provided by the calibration target.

[0103] Example 2: Calibration Data This example describes a working embodiment of a light field microscope and a calibration target, along with measured total and microlens magnification values ​​as a function of z-position (see FIGS. 19 and 20).

[0104] The measured magnification values ​​for each magnification parameter M(z) may be fitted to a function of the form shown in Equation 13 below, allowing for extrapolation and interpolation to other z-positions that have not been explicitly calibrated. [ka] where a and b are constants. Other fitness functions may alternatively be suitable.

[0105] Example 3: Selected embodiments This example describes selected embodiments of the present disclosure as a series of indexed paragraphs.

[0106] Paragraph 1. A method for calibrating a light field imaging system having a lens array (e.g., a microlens array) and an image sensor, comprising: (a) capturing a z-stack of light field images of the calibration target using the image sensor while the calibration target is positioned at a plurality of different z-positions; and (b) determining a total magnification of the light field imaging system and a lens magnification of the lens array from each light field image in the z-stack, optionally wherein the imaging system has a stage and the calibration target is supported by the stage at each of the z-positions.

[0107] Paragraph 2. The method of paragraph 1, further comprising calculating a point spread function as a function of the z-position of the calibration target using a z-stack of light field images.

[0108] Paragraph 3. The method of paragraph 2, wherein each light field image of the z-stack includes a plurality of sub-images, and the step of calculating the point spread function includes a step of calculating an average point spread function from the plurality of sub-images of the light field image.

[0109] Paragraph 4. The method of paragraph 2 or 3, wherein the step of calculating the point spread function includes a step of performing a deconvolution operation on at least a portion of each light field image in the z-stack.

[0110] Paragraph 5. The method of any of paragraphs 1-4, further comprising determining radial distortion for at least one lens of the lens array as a function of z-position of the calibration target using a z-stack of light field images.

[0111] Paragraph 6. The method of paragraph 5, wherein the radial distortion is the average radial distortion for the lenses of the lens array as a function of the z-position of the calibration target.

[0112] Paragraph 7. A method according to any of paragraphs 1-6, wherein the step of determining the lens magnification uses the pitch of the lens array to calculate at least one lens magnification for the lens array for each z-position.

[0113] Paragraph 8. A method according to any of paragraphs 1-7, wherein the calibration target has at least one type of marker arranged to form a first periodic repetition and a second periodic repetition, the first periodic repetition being used to determine a total magnification at each of the z-positions, and the second periodic repetition being used to determine at least one lens magnification at each of the z-positions.

[0114] Paragraph 9. The method of paragraph 8, wherein each light field image of the z-stack consists of a two-dimensional array of sub-images, and the period of the first periodic repetition within the sub-images is smaller than the diameter of each sub-image.

[0115] Paragraph 10. The method of paragraphs 8 or 9, wherein the first periodic repetition is formed by a marker repeated a number of times to define a grid, and the second periodic repetition is formed by the periodic absence of the marker from the nodes of the grid.

[0116] Paragraph 11. The method of any of paragraphs 1-10, wherein the calibration target comprises a two-dimensional array of pinholes.

[0117] Paragraph 12. The method of paragraph 11, wherein the pinholes are located at the nodes of a grid.

[0118] Paragraph 13. The method of paragraph 12, wherein the pinholes are periodically absent from the nodes of the grid.

[0119] Paragraph 14. A method according to any of paragraphs 1-13, wherein the calibration target has an array of markers, each located at a different node of the same grid, each of which is optically transmissive or luminescent (e.g., photoluminescent), and a subset of the nodes in the grid are non-transmissive or non-luminescent nodes that do not have any optically transmissive or luminescent markers.

[0120] Paragraph 15. A method according to paragraph 14, in which the non-transparent or non-emissive nodes are repeated periodically in two dimensions within a grid.

[0121] Paragraph 16. A method according to any of paragraphs 1-15, further comprising capturing a light field image of the object and projecting the light field image into a plane in the object space using a total magnification of the imaging system corresponding to that plane and a magnification of at least one lens of the lens array corresponding to that plane.

[0122] Paragraph 17. The method of paragraph 16, wherein the projecting step reduces blurring using at least one point spread function, the at least one point spread function being calculated from one or more of the light field images of the z-stack.

[0123] Paragraph 18. The method of paragraph 16 or 17, wherein the projecting step includes a step of correcting radial distortion produced by lenses of the lens array.

[0124] Paragraph 19. A light field imaging system comprising: (a) a calibration target; (b) an optional stage for supporting the calibration target; (c) a light source for illuminating the calibration target; (d) an objective lens for collecting light from the illuminated calibration target; (e) a lens array (e.g., a microlens array) downstream of the objective lens; (f) an image sensor for capturing a light field image of the calibration target; and (g) a computer configured to determine a total magnification and a lens magnification of the lens array from the light field image.

[0125] Paragraph 20. The system of Paragraph 19, wherein the computer is configured to calculate at least one point spread function from the light field image.

[0126] Paragraph 21. The system of Paragraph 20, wherein the computer is configured to perform a deconvolution operation on at least a portion of the light field image and calculate at least one point spread function.

[0127] Paragraph 22. The system of any of paragraphs 19-21, wherein the computer is configured to determine radial distortion for at least one lens of the lens array from the light field image.

[0128] Paragraph 23. The system of Paragraph 22, wherein the computer is configured to determine an average radial distortion for the lenses of the lens array from the light field image.

[0129] Paragraph 24. A system described in any of paragraphs 19-23, wherein the calibration target includes at least one type of marker arranged to form a first periodic repetition and a second periodic repetition, and the computer is configured to determine a total magnification using the first periodic repetition, and the computer is configured to determine a lens magnification using the second periodic repetition.

[0130] Paragraph 25. The system described in Paragraph 24, wherein a first periodic repetition is formed by a marker repeated a number of times to define a grid, and a second periodic repetition is formed by the periodic absence of the marker from nodes of the grid.

[0131] Paragraph 26. The system of paragraphs 24 or 25, wherein the calibration target defines a two-dimensional array of pinholes.

[0132] Paragraph 27. The system of Paragraph 26, wherein the two-dimensional array of pinholes is a rectangular array.

[0133] Paragraph 28. The system of paragraph 27, wherein the pinhole is located at a node of a grid, and the pinhole is periodically absent from other nodes in the grid.

[0134] Paragraph 29. The system of paragraph 28, wherein the pinholes are periodically absent and other nodes are not optically transparent.

[0135] Paragraph 30. A system according to any of Paragraphs 26-29, wherein the pinholes in the two-dimensional array of pinholes have substantially the same size.

[0136] Paragraph 31. A system described in any of paragraphs 19-30, wherein the calibration target has an array of markers located at nodes of an identical rectangular grid, each of the markers being optically transmissive or emissive, and a subset of the nodes in the rectangular grid lacking one of the markers and not optically transmissive or emissive.

[0137] Paragraph 32. A calibration target for a light field imaging system, comprising: a layer that substantially blocks transmission of optical radiation of at least one predetermined wavelength and defines a plurality of apertures that are transparent with respect to the optical radiation, the apertures defining a grid, wherein each node of a periodic two-dimensional subset of nodes in the grid either (i) is devoid of an opening in the layer and is non-transparent with respect to the optical radiation, or (ii) has a different type of aperture distinguishable in size and / or shape from each of the plurality of apertures.

[0138] Paragraph 33. A calibration target according to Paragraph 32, wherein the layer is bonded to a substrate that is transparent for optical radiation.

[0139] Paragraph 34. A calibration target according to paragraph 32 or 33, wherein the at least one predetermined wavelength comprises ultraviolet radiation, visible light, and / or infrared radiation.

[0140] Paragraph 35. A method for projecting a light field image of an object into object space using a light field imaging system, the method comprising: (a) capturing a light field image of an object in object space, optionally supported by a stage; and (b) geometrically mapping pixels of the light field image to a plane in object space using a value for the total microscope magnification corresponding to the plane and at least one value for the lens magnification corresponding to the plane.

[0141] Paragraph 36. The method of Paragraph 35, wherein the mapping step uses a point spread function to reduce blurring.

[0142] Paragraph 37. The method of paragraph 35 or 36, wherein the light field imaging system includes an array of lenses, and the mapping step corrects radial distortion produced in the light field image by the array of lenses.

[0143] Paragraph 38. A method according to any of paragraphs 35-37, wherein the mapping step includes a step of reconstructing the object in three dimensions.

[0144] Paragraph 39. A method according to any of paragraphs 35-38, further comprising a step of determining a position of a virtual global optical axis of the imaging system from a light field image captured by an image sensor of the light field imaging system, wherein the mapping step is performed relative to the virtual global optical axis in light field space and object space.

[0145] Paragraph 40. The method of any of paragraphs 35-39, wherein the object comprises a plurality of biological cells.

[0146] The term "exemplary" as used in this disclosure means "illustrative" or "serving as an example." Similarly, the term "exemplary" means "to illustrate by giving an example." Neither term connotes preference or superiority.

[0147] The disclosure described above may encompass multiple distinct inventions with independent utility. Each of these inventions has been disclosed in its preferred form, but the specific embodiments thereof as disclosed and illustrated herein should not be considered in a limiting sense, since numerous variations are possible. The subject matter of the invention includes all novel and non-obvious combinations and subcombinations of the various elements, features, functions, and / or properties disclosed herein. The following claims specifically point out certain combinations and subcombinations that are considered to be novel and non-obvious. Inventions embodied in other combinations and subcombinations of features, functions, elements, and / or properties may be claimed in applications claiming priority from this or related applications. Such claims are also considered to be included within the subject matter of the invention of the present disclosure, whether directed to different or the same invention, and whether broader, narrower, equal, or different in scope than the original claims. Furthermore, ordinal indicators such as first, second, or third with reference to identified elements are used to distinguish between the elements and do not indicate a particular position or order of such elements, unless specifically stated otherwise.

Claims

1. 1. A method for projecting a light field image of an object into an object space using a light field imaging system, the method comprising: capturing the light field image of the object supported in an object space; geometrically mapping pixels of the light field image to a plane in object space using a value for a total magnification of the light field imaging system corresponding to the plane and at least one value for a microlens magnification corresponding to the plane; the light field imaging system comprising: capturing a z-stack of light field images of the calibration target using an image sensor while the calibration target is positioned at a plurality of different z-positions; determining the total magnification and the microlens magnification of the light field imaging system from each light field image of the z-stack; wherein the calibration target has at least one type of markers arranged to form a first periodic repetition and a second periodic repetition, the first periodic repetition being used to determine the total magnification at each of the z-locations, and the second periodic repetition being used to determine at least one microlens magnification at each of the z-locations.

2. (i) the mapping uses a point spread function to reduce blurring; and / or (ii) mapping corrects for radial distortion produced in the light field image by an array of microlenses; The method of claim 1.

3. 2. The method of claim 1 , wherein the object is arranged at a z-position along an optical axis orthogonal to a plane of the object space, and geometrically mapping pixels of the light field image to a plane of the object space includes correlating object space and light field space with respect to an object plane at the z-position.

4. 2. The method of claim 1 , wherein capturing a z-stack of light field images includes controlling a calibration stage to adjust a distance from the calibration target to the image sensor that corresponds to a volume of interest in object space.

5. The method of claim 4 , wherein the volume of interest in the object space corresponds to an expected location of the object.

6. The method of claim 1 , further comprising illuminating the object with optical radiation using a light source, wherein only a spatially limited portion of the optical radiation from the light source is transmitted to a focusing module of an objective lens.

7. 2. The method of claim 1, wherein the calibration target is illuminated with optical radiation generated by a light source, and only a spatially limited portion of the optical radiation is transmitted through the calibration target to a collection module of an objective lens.

8. The method of claim 1 , further comprising calculating a locally varying global total magnification and determining local aberrations therein.

9. The method of claim 8 , wherein determining local aberrations comprises correcting the captured light field image using pre-processing.

10. The method of claim 1, wherein the value for the microlens magnification is determined based on the disparity between nearby views of the same object point of the calibration target, without any distance between the array of microlenses and the original image plane of the image sensor.

11. The method of claim 1, further comprising determining the structure of the three-dimensional cell population, spheroid, or organoid in screening applications.

Images