Sample imaging via two-stage brightfield reconstruction

Abstract

Method and system for sample imaging via two-stage bright-field reconstruction. In an exemplary method (50), a bright-field image (72) of a sample (74) may be captured at a bright-field plane (76). The bright-field image (72) may be computationally forward projected onto each of a plurality of z-planes in object space (78) to generate a set of forward-projected z-plane images (88a-f). A computational back-projection from each z-plane image onto the bright-field plane (76) of the same xy-region in object space (78) may be compared with the bright-field image (72) to determine a respective degree of correspondence between the back-projected xy-region from each of the z-plane images (88a-f) and the bright-field image (72). For each different xy-region, at least one of the forward-projected z-plane images (88a-f) may be selected to provide data for a different xy-region within the 2D or 3D object space image of the sample (74).

Description

[Background technology]

[0001] (Related Applications) This application claims priority to U.S. Provisional Application No. 62 / 885,735, filed Aug. 12, 2019, the contents of which are incorporated herein by reference in their entirety.

[0002] (introduction) Classical imaging uses an image sensor configured as a focal plane array of light-sensitive elements to obtain the position-dependent radiance of a scene. The image sensor quantifies the incident radiance optically projected by a front lens (objective lens) and captures a flattened spatial image of the scene that lacks any directional information about the incident radiance. In contrast, bright-field imaging preserves this directional information, extending the two-dimensional image into a four-dimensional bright-field, which describes the radiance of the scene with respect to position and direction.

[0003] Two different metrology approaches for bright-field acquisition have been reported, using either (i) a separate array of cameras or (ii) a microlens array in front of the image sensor. For bright-field microscopy, the incorporation of a microlens array is often the preferred option due to its lower cost and ability to be easily retrofitted to conventional microscopes.

[0004] The presence of a microlens array between the front lens and the image sensor forms a bright-field image on the image sensor. The bright-field image consists of an array of separate sub-images, each corresponding to a different microlens of the microlens array. These sub-images describe the directional distribution of light for a given point of the scene. This additional directional information generally allows several advantageous processing operations to obtain a reconstructed "object space" image after bright-field acquisition: (i) line of sight change of the viewpoint, (ii) digital refocusing, (iii) artificial focusing on a selected object in the scene, and (iv) matching-free 3D reconstruction. Two bright field configurations with different characteristics can be defined depending on the location of the microlens array relative to the objective lens and image sensor of the imaging system. In the first configuration, labeled "LF1.0", the microlens array is at an intermediate image plane between the objective lens and the image sensor, and the image sensor is at the back focal plane of the microlens array. In the second configuration, labeled "LF2.0", these restrictions are removed and the microlens array can have an arbitrary location between the objective lens and the image sensor. Thus, in LF2.0, the location of the microlens array can be selected to optimize the desired performance criteria of the imaging system (e.g., lateral and axial resolution and depth of focus).

[0005] One of the most important applications of bright-field imaging is 2D and 3D reconstruction of object space, based on mapping 2D bright-field data into object space. However, existing approaches for reconstructing object space from bright-field images are inadequate for a variety of reasons. For example, the approaches may require iterative mapping and re-mapping, filtering to remove noise, and / or assumptions about the optical configuration, each of which may introduce reconstruction errors and / or cause the reconstruction to fail completely. Furthermore, user inspection of the mapped bright-field data may be required to select candidate reconstructions, which prevents full automation. New methods for sample imaging with bright-field imaging systems are needed. Summary of the Invention [Means for solving the problem]

[0006] (summary) The present disclosure provides a method and system for sample imaging via two-stage bright-field reconstruction. In an exemplary method, a bright-field image of a sample may be captured in a bright-field plane. The bright-field image may be computationally forward projected onto each of a plurality of z-planes in object space to generate a set of forward projected z-plane images. A computational backprojection of the same xy-region in object space from each z-plane image onto the bright-field plane may be compared to the bright-field image to determine a respective degree of correspondence between the backprojected xy-region from each of the z-plane images and the bright-field image. This comparison may be repeated for each of a plurality of different xy-regions in object space. For each different xy-region, at least one of the forward projected z-plane images may be selected to provide data for the different xy-region within a 2D or 3D object space image of the sample. An object space image incorporating the data may be generated. The present invention provides, for example, the following: (Item 1) A method (50) for imaging a sample (74), the method (50) comprising: (a) capturing a brightfield image (72) of the sample (74) at a brightfield plane (76); (b) computationally forward projecting the brightfield image (72) onto each of a plurality of z-planes in object space (78) to generate a set of z-plane images (88a-f); (c) comparing a backprojection onto the bright-field plane (76) of the same xy-region in object space (78) from each z-plane image to the bright-field image (72) to determine a respective degree of correspondence between the backprojected xy-region from each of the z-plane images (88a-f) and the bright-field image (72), the comparing being repeated for each of a plurality of different xy-regions in object space (78); (d) based on the comparison, for each different xy-region, selecting at least one of the z-plane images (88a-f) to provide data for the different xy-region within the object space image of the sample (74); (e) generating an object space image of the sample (74); and A method (50), comprising: (Item 2) 2. The method (50) of claim 1, wherein each of the different xy-regions corresponds to a single pixel in each of the set of z-plane images (88a-f). (Item 3) 2. The method (50) according to item 1, wherein at least one of the different xy-regions corresponds to a group of pixels or to an object. (Item 4) 2. The method (50) of claim 1, wherein the generated object space image is a two-dimensional image. (Item 5) 5. The method (50) according to claim 4, wherein the two-dimensional images are substantially all in focus. (Item 6) 2. The method according to claim 1, wherein the object space image is a three-dimensional image. (Item 7) 7. The method (50) according to claim 6, wherein the three-dimensional images are substantially all in focus. (Item 8) 2. The method (50) of claim 1, wherein the selecting includes selecting only one of the z-plane images (88a-f) for which the degree of correspondence is higher with respect to at least one of the different xy-regions with respect to each other z-plane image of the set. (Item 9) 2. The method (50) of claim 1, wherein the selecting includes selecting at least two of the z-plane images (88a-f) for which, for at least one of the different xy-regions, a corresponding degree of correspondence of the backprojection is higher with respect to each other z-plane image of the set. (Item 10) 10. The method (50) of claim 9, further comprising comparing the respective degrees of correspondence for a backprojection of a given xy-region to a threshold, and wherein the selecting comprises selecting each z-plane image for which the comparing of the respective degrees of correspondence satisfies a predetermined condition. (Item 11) 2. The method (50) of claim 1, wherein the selecting includes selecting a best z-plane in object space (78) for a group of pixels or a single pixel in object space (78), and the generating includes generating a three-dimensional image of the sample (74) substantially all in focus. (Item 12) 2. The method (50) of claim 1, wherein the selecting includes selecting a best z-plane in object space (78) for a group of pixels or a single pixel in object space (78), and the generating includes generating a two-dimensional image of the sample (74) substantially all in focus. (Item 13) 2. The method (50) of claim 1, wherein the selecting includes selecting two or more z-planes in object space (78) for each of a plurality of single pixels or groups of pixels in object space (78), the method further including combining information from the two or more selected z-planes for the single pixel or each of the pixels. (Item 14) Item 14. The method (50) of item 13, wherein combining the information comprises averaging or interpolating values ​​for the single pixel or groups of pixels from the two or more selected z-planes. (Item 15) Item 14. The method (50) of item 13, wherein combining the information is based on the respective degrees of correspondence for the single pixel or groups of pixels from the two or more selected z-planes. (Item 16) 2. The method (50) of claim 1, further comprising computationally backprojecting each of the set of z-plane images (88a-f) onto the bright-field plane (76) to obtain a set of backprojected images, wherein the comparing uses the set of backprojected images. (Item 17) 2. The method (50) of claim 1, wherein the capturing is performed by an image sensor (146) of an imaging system (110), and the forward projection and each back projection uses a known mapping geometry for the imaging system (110). (Item 18) Item 18. The method (50) according to item 17, wherein the known mapping geometry utilizes a value relating to a total magnification of the imaging system (110) and at least one value relating to a magnification produced by a microlens array (142) of the imaging system (110). (Item 19) A system (110) for imaging a sample (74), said system (110) comprising: (a) a stage for holding a sample (74) on an optical axis (128) within an object space (78); (b) a microlens array (142) on the optical axis (128); (c) an image sensor (146) configured to capture a bright field image (72) of the sample (74) formed with light propagating through the microlens array (142); and (d) Processor (156) and Equipped with The processor (156) (i) computationally forward projecting the brightfield image (72) onto each of a plurality of z-planes in object space (78) to generate a set of z-plane images (88a-f); (ii) comparing a backprojection onto the bright-field plane (76) of the same xy-region in object space (78) from each z-plane image (88a-f) against the bright-field image (72) to determine a respective degree of correspondence between the backprojected xy-region from each of the z-plane images (88a-f) and the bright-field image (72), the comparing being repeated for each of a plurality of different xy-regions in object space (78); (iii) for each different xy-region, a front view in the object space image of the sample (74) selecting at least one of the z-plane images (88a-f) to provide data relating to the different xy-regions; (iv) generating an object space image of the sample (74); and The system (110) is configured to: (Item 20) 2. A computer readable medium having instructions for causing a processor to perform the method of claim 1. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a flow chart of exemplary steps that may be performed within a method of sample imaging based on a two-stage bright-field reconstruction involving forward and back projection between bright-field space and object space. [Diagram 2] FIG. 2 is a schematic diagram of some of the steps from the flow chart of FIG. [Diagram 3] FIG. 3 is a simplified illustration of forward projection of example bright field images of three objects (beads) onto multiple z-planes in object space to generate a forward projected z-plane image. [Figure 4] 4 and 5 are exemplary graphs that may be obtained by plotting the degree of correspondence between the backprojections of groups of pixels depicting two respective beads from each of the bright field images of FIG. 3 and the forward projected z-plane images of FIG. 3 as a function of z-position. [Diagram 5] 4 and 5 are exemplary graphs that may be obtained by plotting the degree of correspondence between the backprojections of groups of pixels depicting two respective beads from each of the bright field images of FIG. 3 and the forward projected z-plane images of FIG. 3 as a function of z-position. [Figure 6] FIG. 6 is a schematic diagram of an exemplary bright-field imaging system including a bright-field microscope supporting a microplate holding an array of cell-containing samples in wells. [Figure 7] FIG. 7 is a ray diagram generated using selected components of the bright field microscope of FIG. 6 illustrating the propagation of an exemplary light ray from a point object located in a plane in object space to a sensor plane (i.e., the bright field plane) in the two-dimensional bright field space. [Figure 8] 8 is a front view of an exemplary microlens array for the bright field microscope of FIG. 7 taken generally along line 8-8 of FIG. [Figure 9] FIG. 9 is a ray diagram similar to that of FIG. 7, but with the object plane having a z-position closer to the objective lens of the bright field microscope. [Figure 10] FIG. 10 is another ray diagram similar to that of FIG. 7, but where the object plane has a z-position farther from the objective lens of the bright field microscope. [Figure 11] FIG. 11 is a schematic diagram of a bright-field microscope, illustrating the imaging of an actual inverted image using a microlens array, along with representative light rays, and also illustrating the basis for a mapping function that relates a point in object space to multiple points in bright-field space. [Figure 12] FIG. 12 is another schematic diagram of a bright field microscope, illustrating the imaging of a hypothetical inverted image using a microlens array, along with representative light rays, and also the basis for a mapping function that relates a point in object space to multiple points in bright field space. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] (Detailed Description) The present disclosure provides a method and system for sample imaging via two-stage bright-field reconstruction. In an exemplary method, a bright-field image of a sample may be captured in a bright-field plane. The bright-field image may be computationally forward projected onto each of a plurality of z-planes in object space to generate a set of forward projected z-plane images. The computational backprojection from each z-plane image onto the bright-field plane of the same xy-region in object space may be compared to the bright-field image to determine the respective degree of correspondence between the backprojected xy-region from each of the z-plane images and the bright-field image. This comparison may be repeated for each of a plurality of different xy-regions in object space. For each different xy-region, at least one of the forward projected z-plane images may be selected to provide data for the different xy-region in a 2D or 3D object space image of the sample. An object space image incorporating the data may be generated.

[0009] The method of the present disclosure can be applied to bright-field imaging configurations of type LF1.0 and LF2.0. For example, the method is applied to bright-field imaging systems with non-telecentric geometry, where the magnification depends on the distance of each lateral object plane to the objective lens (i.e., the z-position of the object plane). The method may utilize non-iterative forward and back projections. A threshold-free forward projection may be performed on the bright-field image to the 3D object space using a geometric representation of the optical path (e.g., determined from a stack of calibration images) to obtain an initial z-estimate of the object location. The z-estimate of each object voxel (or pixel) may be refined by using a computational back projection into the bright-field space and ranking the support for each z-hypothesis based on the patterns in the covered sub-images of the bright-field image.

[0010] The reconstruction of the object space corresponds to the spatial positions acquired using a classical image stack. However, an important advantage of acquisition using a bright-field setup and the described reconstruction is that a 3D volume containing the spatial positions (x,y,z) of the sample objects (e.g., beads or cells) can be acquired from a single bright-field image. Thus, the method is much faster than acquiring a classical image stack by scanning the z-axis using an adjustable z-stage.

[0011] To be able to perform a quantitative reconstruction of the three-dimensional object space (x,y,z) from the bright-field image, the relationship between the object space and the two-dimensional bright-field space is determined for each microlens sub-image of the bright-field image.

[0012]

number

[0013] The mapping function needs to be understood in terms of

[0014]

number

[0015] It is defined as: In principle, the mapping function can be determined based on values ​​for the geometric parameters of the experimental setup (such as the relative distances from the base planes of the relevant optical elements to the image sensor). In practice, the exact locations of these base planes are unknown and the relative distances inside the bright-field imaging system (e.g., a bright-field microscope) cannot be easily determined to the required accuracy. This lack of precise geometric information is more problematic for the LF2.0 configuration (in which there are no clearly defined optical references for the positioning of the microlens array inside the bright-field imaging system, such as the intermediate image plane of the objective lens and the focal plane of the microlens array). In addition, optical effects of the real lens system (e.g., aberrations or errors caused by the lenses of the microlens array or other optical elements of the bright-field imaging system) can lead to deviations in the real optical geometry compared to the assumed optical geometry measured from the outside. Besides these inherent difficulties in determining the relevant geometry from external measurements, user-induced changes to the geometry (e.g., due to refocusing or unintentional misalignment) can also occur frequently. Any deviation in the assumed geometry relative to the actual geometry will lead to errors in the projection and reconstruction, resulting in incorrect depth information, reduced contrast, or failure to complete the reconstruction.

[0016] The methods disclosed herein may utilize a mapping function that relates the three-dimensional object space to the two-dimensional light-field space without detailed knowledge of the geometry of the optical layout. The mapping function may employ values ​​for system parameters, such as the total magnification over a range of z-positions of interest in the object space and the magnification due to the microlens array, as described in Example 1 below.

[0017] The methods disclosed herein can be used for all wavelength regimes in which lens-based bright field systems are viable, for example, wavelength regimes can include optical radiation that is ultraviolet, visible, and / or infrared.

[0018] The method and system disclosed herein can enable the structure of three-dimensional cell population (e.g., spheroid, organoid) to be determined quickly in screening applications.Currently, the throughput of these screening applications is limited by the time required to obtain the z-series images (generally more than 10 images) at each location of interest.The throughput is further limited when a confocal microscope is employed to image the three-dimensional volume at the location.In contrast, the method and system disclosed herein can generate three-dimensional information about cell population from a single bright-field image.

[0019] Further aspects of the present disclosure are described in the following sections: (I) Definitions, (II) Method Overview, (III) Bright Field Imaging System, (IV) Exemplary Geometric Mapping Functions, and (V) Selected Aspects.

[0020] (I. Definition) Technical terms used within this disclosure have meanings commonly recognized by those of ordinary skill in the art. However, the following terms may be further defined as follows:

[0021] (Computer): An electronic device for storing and processing data, typically in binary form, according to instructions that may be provided by a variable program. Exemplary computers, also referred to as computing devices, include desktop computers, laptop computers, tablets, smartphones, and the like.

[0022] Inspection Region: The volume intersected by the optical axis of a bright-field imaging system within which a sample object may be located for imaging with an image sensor.

[0023] (Image): An optical or digital representation of an object and / or field of view. If the representation is optical, it may be formed by light, inter alia, at a plane in space (e.g., an intermediate image plane or sensor plane) or on a surface (e.g., of an image sensor). If the representation is digital, it may be a raw bright-field image captured by an image sensor, a processed form thereof, or a two-dimensional (pixel) or three-dimensional (voxel) "object space" image.

[0024] Image Sensor: An optical sensor capable of detecting spatial variations in light (e.g., variations in intensity) in two dimensions, where light is incident on a light-sensitive detection area of ​​the sensor. An image sensor can be a two-dimensional array sensor, such as a charge-coupled device (CCD) sensor, an active pixel sensor (e.g., a complementary metal-oxide-semiconductor (CMOS) sensor), a hybrid CCD-CMOS sensor, etc. An image sensor can generate a raster image (i.e., a bitmap) as a rectangular array of pixels, and can be configured to obtain color images, grayscale (monochrome) images, or both.

[0025] (Lens array): A two-dimensional array of lenses. Exemplary lenses that may be suitable in an array for bright-field microscopy are microlenses, i.e., lenses each having a diameter of less than about 1 millimeter. Exemplary microlens diameters that may be suitable for a microlens array include, inter alia, less than about 750, 500, 400, 300, 200, or 100 micrometers for each microlens. Exemplary lens diameters for bright-field close-up photography include lenses having a diameter of at least 1 millimeter, or in some cases, lenses that are microlenses as described above.

[0026] The lens array may have any suitable characteristics. The pitch (inter-lens spacing) of the array may be similar or dissimilar to the lens diameter (such as about 50%, 25%, 20%, or 10% or less larger than the lens diameter). The lenses may form a rectangular or hexagonal grid, among others. The array may include any suitable number of lenses (generally at least four). In some embodiments, the array may include at least 25, 50, or 100 lenses, among others. For example, the array may include at least 2, 3, 4, 5, 6, 7, or 10 lenses along each of a pair of orthogonal axes defined by the array. The lenses of the array may be substantially identical to one another in size and / or shape, or may differ in size and / or shape. Additionally, the array may be an integrated unit and / or the lenses of the array may be integrally formed with one another.

[0027] (Light): Optical radiation, including ultraviolet radiation, visible radiation (i.e., visible light), and / or infrared radiation.

[0028] (Bright Field Image): An optical image formed of light on an image sensor by a microlens array, the optical image consisting of a two-dimensional array of sub-images corresponding to the microlenses of the array, or the captured (digital) forms of those sub-images. The sub-images generally do not overlap each other significantly, if at all. The sub-images may be substantially in focus or out of focus.

[0029] (Light Source): A device that generates light, optionally powered by electricity, optionally as a beam of light. A light source includes at least one light emitting element and may include any associated optical elements for shaping, sizing, filtering, polarizing, scattering, directing, and / or otherwise interacting with the light emitted by the light emitting element. These optical elements may include any combination of at least one waveguide (e.g., optical fiber or liquid light guide), lenses, mirrors, filters, diffusers, masks, apertures, beam splitters, gratings, prisms, polarizers, and the like. Exemplary light emitting elements include semiconductor devices, lasers (e.g., excimer lasers, gas lasers, dye lasers, solid state lasers, semiconductor crystal or diode lasers, free electron lasers, and the like), arc lamps, and the like. Exemplary semiconductor light emitting elements include laser diodes, light emitting diodes (LEDs), and superluminescent diodes, among others.

[0030] (Object Space Image): A 2D or 3D image that represents a region of physical space (i.e., object space) and is reconstructed from at least one bright-field image.

[0031] Objective Lens: A single optical element or combination of optical elements that collects light from an object and / or field of view and concentrates the collected light. Objective lens as used herein may include each optical element located along an optical axis between the inspection region and the lens array of a brightfield imaging system.

[0032] (Optical System): A set of optical elements of an imaging system, which may be located along the optical path between a light source and an inspection area (illumination optics) and / or along the optical axis between the inspection area and an image sensor (collection optics). Optical elements may be any device or structure that interacts with light, such as collecting, directing, concentrating, filtering, polarizing, scattering, collimating, and / or partially blocking light. Optical elements may function by any suitable mechanism, such as reflection, refraction, scattering, diffraction, absorption, and / or filtering, among others. Exemplary optical elements include lenses, mirrors, diffusers, gratings, prisms, filters, apertures, masks, beam splitters, wave guides, polarizers, and the like.

[0033] (Sample): A specimen having any suitable characteristics. A sample may be organic and / or inorganic, natural and / or man-made, and may include any suitable collection, material, substance, isolate, extract, particle, etc. In an exemplary embodiment, a sample includes biological cells (such as an isolated cell or one or more populations of cells). The biological cells may be eukaryotic (e.g., mammalian cells) or prokaryotic (e.g., bacterial cells). Exemplary biological cells include established cells (cell lines), primary cells, cells of a tissue sample, transfected cells, cells from clinical samples (e.g., blood samples, fluid aspirates, tissue fragments, etc.), clones of cells, etc. The cells may optionally be contained by a sample holder in contact with (e.g., immersed in) any suitable liquid medium. The liquid medium may be an aqueous medium, which may include, among other things, water, salts, buffers, glucose, detergents, dyes, proteins, amino acids, or any combination thereof. The liquid medium may be a growth medium for the cells.

[0034] (Sample Holder): A device for holding at least one sample or any array of spatially separated samples, optionally allowing the samples to be imaged through a horizontal transparent wall of the device (e.g., the bottom wall of a well). Exemplary sample holders for brightfield microscopy include (multi-well) microplates, culture dishes, culture flasks, slides, flow chips, etc.

[0035] (z-position): One of the locations along the optical axis of a brightfield imaging system where an object may be placed. The z-position may be defined as the distance between the object and the objective lens of the brightfield imaging system, measured parallel to the optical axis, and may be changed by moving either the object or the objective lens parallel to the optical axis.

[0036] (II. Method overview) This section provides an overview of an exemplary method 50 of imaging a sample using two-stage bright-field reconstruction (see FIGS. 1-5). The two-stage bright-field reconstruction involves a computational forward projection of the bright-field image from the bright-field planes to the z-planes of object space, and a computational back projection from each z-plane to the bright-field plane.

[0037] Method 50 may be performed with any bright-field imaging system for which the mapping geometry between object space and bright-field space is known. More specifically, the mapping geometry may be sufficiently characterized prior to performing method 50 to enable a two-stage mapping in opposite directions between object space and bright-field space. Pixels of a two-dimensional bright-field image captured in a bright-field plane defined by the imaging system may be mapped (i.e., forward projected via data processing performed with a mapping function) to pixels / voxels in object space. Then, pixels / voxels of the two-dimensional or three-dimensional object space image may be mapped (i.e., back projected via data processing performed with a mapping function) to the bright-field plane.

[0038] The mapping geometry may be characterized by any suitable procedure. In some examples, the mapping geometry may be defined using a calibration target. Exemplary calibration targets and their use for determining the mapping geometry based on the total magnification and microlens magnification of the bright-field imaging system are described in Section IV. In other examples, the mapping geometry may be determined from knowledge of the explicit configuration of the system's optical system (such as the types of optical elements in the system, their optical parameters, and their relative distances from each other). In yet other examples, the mapping geometry may be extracted via one or more iterative optimization processes to determine the free parameters of an optical model for the imaging system.

[0039] Method 50 itself may be performed non-iteratively and without thresholding. Mapping from bright-field space to object space may be performed in a first stage, and then back to bright-field space in a second stage. An iteration of the first and second stages is not required. Furthermore, mapping may be performed without applying any thresholds to filter out noise.

[0040] 1 shows exemplary steps that may be performed in method 50. The steps shown at 52, 54, 56, 58, 60, and 62 may be performed in any suitable order and combination and may be modified as may be described anywhere herein. Illustrative aspects of method 50 are shown diagrammatically in FIGS. 2-5.

[0041] A bright field image 72 of a sample 74 may be captured at a bright field plane 76, as shown at 52 in method 50 (see FIGS. 1 and 2 (panel A)). The sample 74 is located in a three-dimensional object space 78 and may include one or more separate objects 80.1, 80.2, and 80.3, which may be microscopic objects such as cells or beads, among others. Each object 80.1-80.3 has a position relative to the x, y, and z axes in the object space 78. The z-axis is typically parallel to, and may be the same as, the optical axis for imaging. In the object space 78 of FIG. 2, the z-axis 82 is shown as horizontal, the x-axis is vertical, and the y-axis is orthogonal to the plane of the drawing. The spaced planes, each oriented orthogonal to the z-axis, are described herein as z-planes.

[0042] Brightfield image 72 may be formed by propagating light from sample 74, indicated by the arrow at 84 in panel A of Figure 2, through a microlens array. Light incident on the brightfield plane 76 is detected by an image sensor, which results in the capture of a brightfield image.

[0043] Bright field image 72 may be computationally forward projected from bright field plane 76 into object space 78, as shown at 54 in method 50, to generate forward projected images 88a-f, also referred to as z-plane images (see FIGS. 1 and 2 (Panel B)).

[0044] The forward projection may be performed in any suitable manner. For example, each point in the bright-field space (i.e., each point of the bright-field image) may be forward projected into the object space using a mapping geometric shape function (see, e.g., Section IV). As another example, the entire bright-field image may be forward projected at once using a pre-computed look-up table that connects the bright-field space with the object space. As yet another example, the entire bright-field image may be forward projected at once using a derived system matrix (expressed as a two-dimensional or higher dimensional matrix) that connects the bright-field space with the object space.

[0045] Two, three, four, or more z-plane images may be generated, each representing a forward projection onto a different z-plane in object space 78. Each z-plane image 88a-f consists of an xy-region, each corresponding to, among other things, a pixel, a group of pixels, or an object. The z-plane images 88a-f may or may not be uniformly spaced from one another along the z-axis 82.

[0046] The z-plane images 88a-f, or at least their corresponding xy-regions (e.g., xy-regions 90.1, 90.2, and 90.3) of each z-plane image may be computationally backprojected to the bright-field plane 76, as shown at 56 in method 50, to generate a set of backprojections 94a-f, 96a-f, and 98a-f (see FIGS. 1 and 2 (panel C)). The members of each set of backprojections 94a-f, 96a-f, or 98a-f represent the same xy-region in object space 78, but different z-planes. The xy-regions 90.1, 90.2, and 90.3 depicted in FIG. 2 (panel C) correspond to z-candidates (i.e., z-location estimates) for objects 80.1, 80.2, and 80.3, respectively (see FIG. 2 (panel A)). In other embodiments, these xy-regions may correspond to single pixels or groups of pixels of the z-plane images 88a-f, among others.

[0047] The elements of each set of backprojections 94a-f, 96a-f, and 98a-f may be compared to the brightfield image 72, as shown at 58 in method 50 (see FIGS. 1 and 2, panel D). This comparison measures the degree of correspondence between each element (94a-f, 96a-f, or 98a-f) of a given backprojection set and the brightfield image 72. A higher degree of correspondence for a given element of the backprojection set makes the xy-region of the ancestor z-plane image for the given element a better candidate for that same region of the object space image to be constructed.

[0048] As shown at 60 and 62 in method 50, at least one z-plane image may be selected to generate each of the xy-regions in an output object space image (e.g., a displayed object space image), and corresponding regions of the object space image may be generated using the selected z-plane image (see FIG. 1). For example, each of one or more of the xy-regions of the object space image may be generated directly using corresponding xy-regions from a single z-plane image. Alternatively, or in addition, each of one or more of the xy-regions of the object space image may be generated using the same xy-regions from two or more z-plane images based on the degree of correspondence measured for the same xy-regions for the different z-plane images, by averaging, interpolation, maximum intensity operations, etc.

[0049] The generated object space image may be a two-dimensional (2D) or three-dimensional (3D) image. If it is a 2D image, it may be a substantially all-in-focus 2D image made up of pixels. If it is a 3D image, it may be made up of voxels.

[0050] FIG. 3 shows a simplified illustration of the forward projection of an exemplary bright-field image 72 of three objects 80.1-80.3 (e.g., fluorescent beads) onto four z-planes in object space 78 (see also FIG. 2). In this example, the z-planes are located at respective z-locations: −40, −30, −20, and −10 micrometers (μm). Each of the z-plane images 88a-88d generated by the forward projection of the bright-field image 72 includes xy-regions 90.1, 90.2, and 90.3. The contents of these xy-regions in each of the z-plane images 88a-88d are candidate counterparts in object space 78 of the bright-field representations 102, 104, and 106 of the objects 80.1-80.3 in the bright-field image 72.

[0051] The degree of focus for each xy-region 90.1-90.3 in each image 88a-88d is indicated by a solid circle or a diagonal line. The presence of a solid circle for a given xy-region in a given z-plane means that the given xy-region is substantially in focus in that z-plane. The presence of a diagonal line indicates that the given xy-region is less in focus in that z-plane than a solid circle for the same xy-region in a different z-plane. A closer spacing of the diagonal lines for a given xy-region in a given z-plane means better focus than a wider spacing of the diagonal lines for the same xy-region in a different z-plane. Thus, xy-region 90.1 (for object 80.1) has the best focus alignment in forward projected image 88c at -20 μm, while xy-regions 90.2 and 90.3 (for objects 80.2 and 80.3) have the best focus alignment in forward projected image 88a at -40 μm.

[0052] The degree of focus alignment for each xy-region 90.1-90.3 in each z-plane can be estimated by backprojecting the xy-region onto the bright-field plane of the bright-field image 72 and comparing each backprojection with the bright-field image. Figures 4 and 5 show graphs illustrating an exemplary comparison for each xy-region 90.1 of Figure 3 (Figure 4) and for each xy-region 90.2 of Figure 3 (Figure 5). The graphs plot the degree of correspondence with the bright-field image 72 for each backprojected xy-region 90.1 or 90.2 according to the z-position. Consistent with the degree of focus alignment visible in Figure 3, the backprojection (-20 μm) of the xy-region 90.1 from the forward projected image 88c shows the greatest degree of correspondence with the bright-field image 72 for the other z-positions tested (see Figure 4). The back projection (-40 μm) of xy-region 90.2 from forward projected image 88a also shows the greatest degree of correspondence with bright field image 72 for the other z-positions tested (see FIG. 5). Hence, a 2D or 3D object space image can be generated using a greater or exclusive contribution of data from image 88c for xy-region 90.1 and data from image 88a for xy-region 90.2 based on the measured degree of correspondence.

[0053] Any combination of the steps of Figure 1 or other methods disclosed herein may be embodied as a computer method, computer system, or computer program product. Thus, aspects of the methods may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be generally referred to herein as a "circuit," "module," or "system." Additionally, aspects of the methods may take the form of a computer program product embodied in a computer-readable medium (or media) having computer-readable program code / instructions embodied thereon.

[0054] Any combination of computer readable media may be utilized. The computer readable medium may be a computer readable signal medium and / or a computer readable storage medium. The computer readable storage medium may include an electronic, magnetic, optical, electromagnetic, infrared, and / or semiconductor system, apparatus, or device, or any suitable combination thereof. More specifically, the computer readable storage medium may include: a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, any suitable combination thereof, and the like. In the context of this disclosure, the computer readable storage medium may include any suitable non-transitory tangible medium that may contain or store a program for use by or in association with an instruction execution system, apparatus, or device.

[0055] A computer-readable signal medium may include a propagated data signal in which computer-readable program code is embodied, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, and / or any suitable combination thereof. A computer-readable signal medium may include any computer-readable medium capable of communicating, propagating, or transporting a program for use by or in association with an instruction execution system, apparatus, or device, other than a computer-readable storage medium.

[0056] The program code embodied on the computer readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wireline, fiber optic cable, and / or any suitable combination thereof, among others.

[0057] Computer program code for performing operations for aspects of the methods disclosed herein may be written in one or any combination of programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and traditional procedural programming languages ​​such as C. Mobile apps may be developed using any suitable language, including those mentioned above and Objective-C, Swift, C#, HTML5, and the like. The program code may run entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), and / or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider).

[0058] Aspects of the methods are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, systems, and / or computer program products. Each block and / or combination of blocks in the flowchart and / or block diagrams may be implemented by computer program instructions. The computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine that executes the instructions via a processor of the computer or other programmable data processing apparatus to implement the functions / acts specified in the flowchart and / or block diagram blocks. In some embodiments, the machine-readable instructions may be programmed onto a programmable logic device, such as a field programmable gate array (FPGA).

[0059] Computer program instructions may also be stored in a computer-readable medium that may instruct a computer to function in a particular manner to generate an article of manufacture containing instructions, the instructions stored in the computer-readable medium implementing the functions / acts specified in the flowchart and / or block diagram blocks.

[0060] The computer program instructions can be loaded onto a computer such that the instructions, which execute on the computer, provide a process for implementing the functions / acts defined in the flowchart and / or block diagram blocks, causing a series of operational steps to be performed on the device to generate a computer-implemented process.

[0061] Any flowcharts and / or block diagrams in the drawings are intended to illustrate the architecture, functionality, and / or operation of possible implementations of systems, methods, and computer program products according to aspects of the methods disclosed herein. In this regard, each block may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function. In some implementations, the functions described in the blocks may occur out of the order described in the drawings. For example, two blocks shown in succession may in fact be executed substantially in parallel, or the blocks may sometimes be executed in reverse order, depending on the functionality involved. Each block and / or combination of blocks may be implemented by a special purpose hardware-based system (or a combination of special purpose hardware and computer instructions) that performs the specified functions or acts.

[0062] (III. Bright Field Imaging System) This section describes an exemplary brightfield imaging system 110 for implementing the method of Section II (see FIGS. 6-10).

[0063] Brightfield imaging system 110 may include a brightfield microscope 112 (or a brightfield camera for brightfield close-up photography) having a stage 114 for supporting an object of interest, where a sample holder 116 is supported by stage 114. Sample holder 116 may be, for example, a microplate 118 having multiple wells 120 containing samples 122, each of which may include biological cells 124. A drive mechanism 126 may move stage 114 and sample holder 116 relative to one another to position each well 120 on an optical axis 128 of system 110.

[0064] An optical axis 128 may be defined in an inspection region 130 of the object space 78 by an objective lens 132. An object plane 134 (z-plane) in the object space 78 is orthogonal to the optical axis 128 (z-axis) when the sample holder 116 is operably supported by the stage 114 and may be positioned near a front focal plane of the objective lens 132.

[0065] The sample 122 may be illuminated with optical radiation (e.g., visible light) generated by the light source 138. Illumination of the sample 122 may, for example, induce photoluminescence (such as fluorescence) from at least one luminophore in the sample. The photoluminescence may propagate, for example, through the objective lens 132, the optional tube lens 140, and the microlens array 142. The microlens array 142 may have any suitable number of microlenses 144 arranged in a two-dimensional array. The individual microlenses 144 of the microlens array 142 may at least partially focus the photoluminescence, each corresponding to a different one of the microlenses 144, onto the image sensor 146 to form individual sub-images. The image sensor 146 captures brightfield images of the sub-images, which are communicated to the computer 148 for data processing. In other embodiments (e.g., for brightfield close-up photography), the microlens array 142 may be replaced by a lens array composed of lenses that are not necessarily microlenses. In other embodiments, sample 122 may transmit, reflect, or otherwise interact with incident optical radiation that forms the bright field image.

[0066] The bright field microscope 112 allows adjustment of the z-position of the sample 122 along the optical axis 128 via a drive mechanism 126. More specifically, the drive mechanism can operate to vary the distance between the sample 122 and the objective lens 132 along the optical axis 128. Thus, the drive mechanism 126 can be coupled to the stage 114 to drive the movement (e.g., vertical movement) of the stage while the z-position (e.g., altitude) of the objective lens 132 remains fixed, as shown in FIG. 1 . Alternatively, the drive mechanism 126 can be coupled to the objective lens 132 to drive the movement (e.g., vertical movement) of the objective lens 132 while the z-position (e.g., altitude) of the stage 114 remains fixed. The drive mechanism 126 can include a motor 150 (or can be manually driven) and can be controlled by a computer 148.

[0067] The optical axis 128 may have any suitable orientation and characteristics. The optical axis is typically vertical at the objective lens 132 and may extend vertically upward (as shown here) or vertically downward from the objective lens 132 through the inspection region 130 and object plane 134. In other words, the bright field microscope 112 may have an inverted configuration with the objective lens 132 below the inspection region 130 as shown here, or a non-inverted configuration with the objective lens 132 above the inspection region 130. However, in other embodiments, the optical axis 128 may be horizontal or oblique (neither substantially horizontal nor substantially vertical) at the objective lens 132. The optical axis 128 may be unfolded or folded (e.g., with mirrors) at one or more points along the optical axis.

[0068] The light source 138 may be positioned to provide any suitable illumination of the inspection region 130 (and the sample 122 therein) using any suitable light source (see, e.g., Section I). The illumination may be epi-illumination (as shown here), trans-illumination, oblique illumination, or a combination thereof, among others. Any suitable optical elements may be operatively positioned on the illumination path between the light source 138 and the inspection region 130 to direct and / or condition the optical radiation generated by the light source upstream of the inspection region. Exemplary optical elements located in the illumination path may include a beam splitter 152, a diffuser, a spectral filter, a collimating lens or mirror, a folding mirror, a light guide, and the like. The optical radiation from the sample 122 may be collected by the objective lens 132, the beam splitter 152, the tube lens 140 for propagation through the objective lens, and the microlens array 142 for detection by the image sensor 146.

[0069] Computer 148 communicates with image sensor 146 and processes brightfield images captured by image sensor 146. Computer 148 may include a processor 156 for processing instructions, a memory 158 for storing instructions and data, and a user interface 160 for communication between computer 148 and a user. The user interface may include user input devices such as a keyboard, mouse, or touch screen, and a display device such as a monitor.

[0070] 7 shows a ray diagram with only selected components of brightfield microscope 112. The schematic diagram depicts how optical radiation may propagate from an object point 162 in object plane 134 of object space 78 to multiple distinct brightfield image points 164 on image sensor 146. The optical radiation is represented by upstream marginal rays 166 extending from object point 162 to an intermediate point 168 in intermediate image plane 170, and downstream marginal rays 172 extending from intermediate point 168 to brightfield plane 76 defined by image sensor 146. In this exemplary configuration, object plane 134, intermediate image plane 170, and brightfield plane 76 are conjugate to one another such that the optical radiation is precisely focused to brightfield image point 164.

[0071] A plan view of an exemplary microlens array 142 is shown in FIG. The microlenses 144 may be arranged in a two-dimensional array. The microlenses are distributed along a pair of orthogonal axes in the plane of the array and may define a rectangular grid as shown, although non-rectangular arrangements of microlenses may alternatively be used. In some embodiments, the microlenses 144 may be uniformly spaced from one another along each orthogonal axis of the pair of axes, and may optionally have the same spacing, i.e., pitch d, from one another along both axes as shown.

[0072] The distance between bright field image points 164 may exceed the spacing of microlenses 144 (e.g., pitch d in FIG. 8) due to parallax (see FIG. 7). Optical radiation from midpoint 168 may be incident on at least a portion of individual microlenses 144 of microlens array 142 at a different angle for each microlens. In other words, each chief ray propagating from midpoint 168 through each of some microlenses 144 has a different orientation. Optical radiation from midpoint 168 may be incident only on a portion of the microlenses of microlens array 142, as shown, because the cone size of the optical radiation propagating from midpoint 168 may be limited by an aperture (e.g., stop element 174) located upstream (e.g., in objective lens 132). (However, brightfield microscope 112 generally has a non-telecentric optical configuration such that magnification varies with distance from objective lens 132.) When object point 162 is moved sufficiently laterally in object plane 134, different portions of the microlenses may receive optical radiation from object point 162. Thus, a set of object points 162 that are sufficiently spaced in object plane 134 may be imaged by different portions of the microlenses and may have conjugate points / spots within different portions of sub-images in the brightfield image captured by image sensor 146.

[0073] In other cases, the distance between bright field image points 164 may be less than the physical spacing of the microlenses 144 (e.g., pitch d in FIG. 8). For example, in a hyper-perfect Galileo type setup, the pitch of the optically projected images of the microlenses on the image sensor may be less than the physical pitch of the microlens array. Similarly, in this setup, the distance between bright field image points 164 is less than the (optically projected, physical) pitch of the microlens array.

[0074] 7 uses an infinity corrected objective lens 132 to collimate the upstream optical radiation and a downstream tube lens 140 to focus the collimated radiation. In other embodiments, the tube lens 140 may be eliminated and the optical radiation may be both collected and focused using, for example, the objective lens 132.

[0075] FIG. 7 shows an object plane 134 located at z-position z1, which is precisely conjugate with the bright-field plane 76. However, this precise relationship with respect to the object location is not required. More specifically, bright-field images suitable for the methods of the present disclosure may be captured by the image sensor 146 for an object plane 134 located closer to the objective lens 132 at z-position z2, and for an object plane 134 located farther from the objective lens 132 at z-position z3 (see FIGS. 9 and 10, respectively). In either case, the optical radiation from the intermediate point 168 is not as concentrated at the bright-field plane 76 as in FIG. 7, so that the image point 164 in FIG. 7 becomes a larger image spot in FIGS. 9 and 10.

[0076] Further exemplary aspects of brightfield imaging system 110 and brightfield microscope 112 (or brightfield close-up photography system) that may be suitable are described elsewhere herein, such as in Sections I, II, IV, and V.

[0077] IV. EXEMPLARY GEOMETRIC MAPPING FUNCTIONS This section describes the basis for an exemplary geometric mapping function for use in method 50 (see FIGS. 1, 11, and 12). The mapping function relates object space and bright-field space to each other with respect to the object plane at a given z-position along the optical axis. The mapping function may be used with, among other things, bright-field microscopes or bright-field close-up photography systems.

[0078] The bright field microscope 112 of Figs. 11 and 12 may be constructed as in Figs. 6 and 7, but is depicted in a somewhat different form. The objective lens 132 is shown as a single lens, and the microlens array 142 is shown in a more stylized form. These simplifications emphasize that, except for the relevant parameters explained below, the objective lens and the microlenses may otherwise have a "black box" structure. Furthermore, one of the microlenses 144 selected to illustrate the mapping is labeled as the kth microlens "144k". The object point 162 is located in the object plane 134 as in Figs. 7, 9, and 10, but is offset laterally from the optical axis 128 (compare Fig. 7). The marginal ray 166 and the chief ray 176 extend from the object point 162, through the objective lens 132, to the midpoint 168 in Fig. 11. Marginal rays are shown extending downstream of midpoint 168 and indicate the boundary of the cone of optical radiation incident on microlens array 142 from object point 162. Upstream chief ray 176 is shown in FIG. 11 as terminating early at midpoint 168 to reduce drawing complexity downstream of midpoint 168.

[0079] A mapping function relating the three-dimensional object space to the two-dimensional light-field space can be defined for microlens 144k as follows:

[0080]

number

[0081] where the subscripts xy indicate the lateral components of the associated vector. This formula applies whether the imaging system displays an inverted image or not, and whether the microlens array images a real image (inverted or non-inverted image) or a virtual image (inverted or non-inverted image). The presence or absence of inversion is included in the sign of the corresponding magnification value, as explained below.

[0082] A schematic of the mapping geometry for Equation 1 is shown in Figures 11 and 12, which illustrate the imaging of real and virtual inverted images by a microlens array, respectively.

[0083]

number

[0084] is a vector representing an object point 162 in the object plane 134 of the three-dimensional object space 78,

[0085]

number

[0086] is a vector representing the position of the optical axis of the microlens 144k, and M Total is the total magnification of the system (e.g., produced by the combination of objective lens 132 and microlens array 142), and M ML is the magnification due to the microlens array 142 alone,

[0087]

number

[0088] is in the object space 78

[0089]

number

[0090] is a vector representing the point in the bright field plane 76 corresponding to

[0091] Vector

[0092]

number

[0093] , and

[0094]

number

[0095] Each of the downstream chief rays 178 has a tail at the optical axis 128. However, the optical axis used for mapping need not be an actual optical axis, but instead may be defined from the brightfield image as a virtual global optical axis. The downstream chief rays 178 extend from the intermediate image point 168 through at least a portion of two dimensions of the microlenses 144 to a corresponding number of brightfield image points on the image sensor 146, as described in Figures 7, 9, and 10 above. One of the downstream chief rays 178 passes through microlens 144k,

[0096]

number

[0097] 4, is incident on the bright field plane 76.

[0098] Equation 1 can be derived as follows:

[0099]

number

[0100] is the vector according to Eq.

[0101]

number

[0102] The image is mapped to an intermediate image point 168, represented by

[0103]

number

[0104] In the formula, M objective is the collective magnification provided by the objective lens 132 of the bright field microscope 112 at the intermediate image plane 170,

[0105]

number

[0106] is a vector

[0107]

number

[0108] In the cases of Figures 11 and 12, M objective will be a negative number.

[0109] Intermediate image point

[0110]

number

[0111] is expressed as follows in accordance with Equation 3:

[0112]

number

[0113] is mapped to.

[0114]

number

[0115] In the case of Figure 11, M ML would again be negative since it would invert the real image produced by the microscope, whereas in the case of Figure 12, MML will be a positive number.

[0116] Equation 3 is rearranged and regrouped to generate equations 4 and 5.

[0117]

number

[0118]

number

[0119] According to Eq. 2, in Eq. 5

[0120]

number

[0121] Substituting into yields Equation 6.

[0122]

number

[0123] The magnification in bright field microscope 112 can be described by Equation 7:

[0124]

number

[0125] Substituting into the right hand term of Equation 6 according to Equation 7 yields Equation 1.

[0126] The mapping geometry is fully defined once the total magnification and the microlens magnification are known (e.g., measured). In general, each of these magnifications depends on the distance between the object plane 134 and the objective lens 132 (i.e., the z-position of the object plane 134). Local aberrations (e.g., due to the microlens array) can be expressed in terms of the total magnification, which varies with each microlens image. This leads to Equations 8 and 9.

[0127]

number

[0128]

number

[0129] The magnification due to the microlens array can be determined using the calibration target by evaluating the disparity (also called virtual depth) between nearby views of the same object point of the calibration target using Equation 10.

[0130]

number

[0131] In the formula, d MLA is the pitch of the microlens array,

[0132]

number

[0133] is the (z-dependent) location of the common object point in the image of the i-th microlens. M MLTo determine the correct sign of , it is necessary to know whether the microlens array is placed upstream or downstream of the image plane of the microscope image plane. In the first case, the microlens array images a virtual image, and in the second case, a real image. In general, that does not limit the applicability of the approach, since the location of the microscope image plane is either located at the native image sensor position or can be easily known by moving the image sensor relative to the microlens array along the z-axis downstream of the microscope objective. M Total The sign of the inversion characteristic and M Objective and M. ML It should also be noted that the sign of the total magnification can vary depending on the associated sign of . However, a sign change in the total magnification only leads to an inversion in object space, i.e., in general, is not relevant. In other words, it is only necessary to know whether the microlens array is inserted between the original image plane of the microscope and the upstream optics or downstream of the original image plane. It is not necessary to know the exact distance between the microlens array and the original image plane.

[0134] (V. Selected Aspects) This example describes selected aspects of this disclosure as a series of indexed paragraphs.

[0135] Paragraph 1. A method of imaging a sample, the method including: (a) capturing a bright field image of the sample at a bright field plane; (b) computationally forward projecting the bright field image onto each of a plurality of z-planes in object space to generate a set of z-plane images; (c) comparing back projections of the same xy-region in object space from each z-plane image to the bright field plane against the bright field images to determine a respective degree of correspondence between the back projected xy-region from each of the z-plane images and the bright field image, the comparison being repeated for each of a plurality of different xy-regions in object space; (d) based on the comparison, for each different xy-region, selecting at least one of the forward projected images to provide data for a different xy-region in the object space image of the sample; and (e) generating an object space image of the sample.

[0136] Paragraph 2. The method of paragraph 1, wherein each distinct xy-region corresponds to a single pixel in each of the set of z-plane images.

[0137] Paragraph 3. The method of paragraph 1, wherein at least one of the distinct xy-regions corresponds to a group of pixels or an object.

[0138] Paragraph 4. The method of any of paragraphs 1-3, wherein the generated object space image is a two-dimensional image.

[0139] Paragraph 5. The method of paragraph 4, wherein the two-dimensional images are substantially all in focus.

[0140] Paragraph 6. The method of any of paragraphs 1-3, wherein the object space image is a three-dimensional image.

[0141] Paragraph 7. The method of paragraph 6, wherein the three-dimensional images are substantially all in focus.

[0142] Paragraph 8. A method as described in any of paragraphs 1-7, wherein the selecting includes selecting only one of the z-plane images for which the degree of correspondence is higher with respect to at least one of the distinct xy-regions relative to each other z-plane image of the set.

[0143] Paragraph 9. The method of any of paragraphs 1-8, wherein the selecting includes selecting at least two of the z-plane images in which, with respect to at least one of the distinct xy-regions, a corresponding degree of correspondence of backprojection is higher with respect to each other z-plane image of the set.

[0144] Paragraph 10. The method of paragraph 9, further comprising comparing each degree of correspondence for the backprojection of the given xy-region with a threshold, and wherein selecting comprises selecting each z-plane image for which comparing the respective degrees of correspondence satisfies a predetermined condition.

[0145] Paragraph 11. The method of any of paragraphs 1-10, wherein selecting includes selecting a best z-plane in object space for a group of pixels or a single pixel in object space, and generating includes generating a three-dimensional image of the sample substantially all in focus.

[0146] Paragraph 12. The method of any of paragraphs 1-11, wherein selecting includes selecting a best z-plane in object space for a group of pixels or a single pixel in object space, and generating includes generating a two-dimensional image of the sample substantially all in focus.

[0147] Paragraph 13. A method as described in any of paragraphs 1-12, wherein the selecting includes selecting two or more z-planes in object space for each of a plurality of single pixels or pixels in the object space, and further includes combining information from the selected two or more z-planes for each single pixel or group of pixels.

[0148] Paragraph 14. The method of paragraph 13, wherein combining the information includes averaging or interpolating values ​​for a single pixel or groups of pixels from two or more selected z-planes.

[0149] Paragraph 15. The method of paragraph 13, wherein combining the information is based on respective degrees of correspondence for single pixels or groups of pixels from two or more selected z-planes.

[0150] Paragraph 16. The method of any of paragraphs 1-15, further comprising computationally backprojecting each of the set of z-plane images onto a bright field plane to obtain a set of backprojected images, and the comparison uses the set of backprojected images.

[0151] Paragraph 17. A method according to any of paragraphs 1-16, wherein the capture is performed by an image sensor of an imaging system, and the forward projection and each back projection uses a known mapping geometry for the imaging system.

[0152] Paragraph 18. The method of paragraph 17, wherein the mapping geometry utilises a value relating to a total magnification of the imaging system and at least one value relating to a magnification produced by a microlens array of the imaging system.

[0153] Paragraph 19. A system for imaging a sample, the system comprising: (a) a stage for holding the sample on an optical axis in an object space; (b) a microlens array on the optical axis; (c) an image sensor configured to capture a bright field image of the sample formed with light propagated through the microlens array; and (d) a processor, the processor configured to: (i) computationally forward project the bright field image onto each of a plurality of z-planes in the object space to generate a set of z-plane images; (ii) compare back projections of the same xy-region in the object space from each z-plane image to the bright field planes to determine a respective degree of correspondence between the back projected xy-region from each of the z-plane images and the bright field image, the comparing being repeated for each of a plurality of different xy-regions in the object space; (iii) for each different xy-region, select at least one of the z-plane images to provide data for the different xy-region in the object space image of the sample; and (iv) generate an object space image of the sample.

[0154] The term "exemplary" as used within this disclosure means "illustrative" or "serving as an example." Similarly, the term "exemplary" means "to illustrate by giving an example." None of the terms imply desirability or superiority.

[0155] Although the present invention has been described through the above examples and features, it will be understood by those skilled in the art that various modifications, combinations, and alterations of the examples and features may be made without departing from the inventive concept disclosed herein. Moreover, the present invention should not be considered as being limited to any specific object or embodiment described herein, but rather as being applicable to accomplish various objects other than those described herein. This disclosure describes some examples of the present technology with reference to the accompanying drawings, in which only some of the possible examples are shown. However, other aspects may be embodied in many different forms, and even if not explicitly illustrated in combination, should not be construed as being limited to the examples described herein. Rather, these examples are provided so that this disclosure will be thorough and complete, and will fully convey the scope of possible embodiments to those skilled in the art.

Claims

1. A method (50) for imaging a sample (74), the method (50) comprising: (a) capturing a light field image (72) of the sample (74) at a light field plane (76); (b) generating a set of z-plane images (88a-f) by computationally forward projecting the light field image (72) onto each of a plurality of z-planes in object space (78); (c) determining a respective degree of correspondence between the backprojected xy-regions from each of the plurality of z-plane images (88a-f) and the light field image (72) by comparing, for the light field image (72), backprojections onto the light field plane (76) of the same xy-regions in object space (78) from each z-plane image, wherein the comparing is repeated for each of a plurality of different xy-regions in object space (78); (d) selecting, for each distinct xy-region, at least one of the plurality of z-plane images (88a-f) to provide data for the distinct xy-region within an object space image of the sample (74) based on the comparing; (e) generating the object space image of the sample (74); and A method (50).

2. The method (50) of claim 1, wherein each of the plurality of distinct xy-regions corresponds to a single pixel in each of the plurality of z-plane images (88a-f) of the set.

3. The method (50) of claim 1, wherein at least one of the plurality of different xy-regions corresponds to a group of pixels or an object.

4. The method (50) of claim 1, wherein the generated object space image is a two-dimensional image.

5. The method (50) of claim 4, wherein the two-dimensional images are substantially all in focus.

6. The method of claim 1 , wherein the object space image is a three-dimensional image.

7. The method (50) of claim 6, wherein the three-dimensional images are substantially all in focus.

8. 2. The method (50) of claim 1, wherein the selecting comprises selecting only one of the plurality of z-plane images (88a-f) for which the degree of correspondence is higher for at least one of the plurality of different xy-regions than for each other z-plane image of the set.

9. 2. The method (50) of claim 1, wherein the selecting includes selecting at least two of the plurality of z-plane images (88a-f) for which, for at least one of the plurality of distinct xy-regions, a corresponding degree of correspondence of the backprojection is higher than for each other z-plane image of the set.

10. 10. The method of claim 9, further comprising comparing the respective degrees of correspondence for a backprojection of a given xy-region to a threshold, and wherein the selecting comprises selecting each z-plane image for which the comparing of the respective degrees of correspondence satisfies a predetermined condition.

11. 2. The method (50) of claim 1, wherein the selecting includes selecting a best z-plane in object space (78) for a group of pixels or a single pixel in object space (78), and the generating includes generating a substantially all-in-focus three-dimensional image of the sample (74).

12. 2. The method (50) of claim 1, wherein the selecting includes selecting a best z-plane in object space (78) for a group of pixels or a single pixel in object space (78), and the generating includes generating a substantially all-in-focus two-dimensional image of the sample (74).

13. 2. The method (50) of claim 1, wherein the selecting includes selecting two or more z-planes in object space (78) for each of a plurality of single pixels or groups of pixels in object space (78), the method (50) further including combining information from the two or more z-planes selected for each single pixel or group of pixels.

14. The method (50) of claim 13, wherein combining the information comprises averaging or interpolating values ​​for the single pixel or groups of pixels from the two or more selected z-planes.

15. The method (50) of claim 13, wherein combining the information is based on the respective degrees of correspondence for the single pixel or groups of pixels from the two or more selected z-planes.

16. The method (50) of claim 1, further comprising: obtaining a set of backprojections by computationally backprojecting each of the set of z-plane images (88a-f) onto the light field plane (76), the set of backprojections including the backprojections.

17. 2. The method of claim 1, wherein the capturing is performed by an image sensor of an imaging system, and the forward projection and each back projection uses a known mapping geometry for the imaging system.

18. 20. The method (50) of claim 17, wherein the known mapping geometry utilizes a value for a total magnification of the imaging system (110) and at least one value for a magnification produced by a microlens array (142) of the imaging system (110).

19. A system (110) for imaging a sample (74), said system (110) comprising: (a) a stage for holding a sample (74) on an optical axis (128) within an object space (78); (b) a microlens array (142) on said optical axis (128); (c) an image sensor (146) configured to capture a light field image (72) of the sample (74) formed with light propagated through the microlens array (142); and (d) a processor (156); Equipped with The processor (156) (i) generating a set of z-plane images (88a-f) by computationally forward projecting the light field image (72) onto each of a plurality of z-planes in an object space (78); (ii) determining a respective degree of correspondence between the backprojected xy-regions from each of the plurality of z-plane images (88a-f) and the light field image (72) by comparing, for the light field image (72), backprojections onto the light field plane (76) of the same xy-regions in object space (78) from each z-plane image, wherein the comparing is repeated for each of a plurality of different xy-regions in object space (78); (iii) for each different xy-region, selecting at least one of the plurality of z-plane images (88a-f) to provide data relating to the plurality of different xy-regions within an object space image of the sample (74); (iv) generating the object space image of the sample (74); and The system (110) is configured to:

20. A computer-readable storage medium having instructions stored thereon which, when executed by a processor, cause the processor to perform the method of claim 1.

Images