Computational microscopy methods and systems for volumetric imaging.

Abstract

A microscopy method for volumetric imaging of a sample disposed in an object space, the microscopy method comprising: acquiring a set of light field images using a light field detection device, each associated with an illumination slice corresponding to a light sheet projected into the object space from a particular illumination position at an illumination angle that is not parallel to an optical axis of the light field detection device, such that each voxel of a set of voxels associated with the object space, defined by a position in the object space, is illuminated by at least one illumination slice; and determining, for each voxel, the intensity of light emitted from the object space associated with the voxel, using, in part, angular information captured by the light field images, such that the intensity determined for each voxel of the set of voxels defines a volumetric image of emitted light intensity within the object space and associated system.

Description

[Technical Field]

[0001]

[0001] The present invention relates generally to microscope systems and methods for volumetric imaging of object space in a sample. [Background technology]

[0002]

[0002] Various methods of three-dimensional or volumetric imaging are known for imaging biological samples, such as confocal laser scanning microscopy. Subsequently, light sheet microscopy was developed, in which the sample is moved through a plane of light to perform optical sectioning of the sample. Single-objective scanning light sheets using oblique plane (OP) illumination techniques, in which the light sheet is at a non-incident angle, have recently pushed the boundaries of image-based biological research.

[0003] The main drawback of using a single-objective scanning light sheet is that an expensive and complex remote imaging unit containing two complementary objectives (secondary and tertiary objectives) is required to achieve the necessary diffraction-limited imaging and optical sectioning. Furthermore, additional elements such as scanning mirrors, diffraction gratings, or custom prisms are required to deskew and replicate the 3D volumetric image on the 2D imaging sensor. More importantly, these remote focusing units are incompatible with standard imaging and detection schemes, which typically comprise a single tube lens with a 2D camera sensor. Therefore, single-objective scanning light sheet systems (e.g., eSPIM and SCAPE) are limited to specialized microscope setups.

[0004]

[0004] Light field imaging is a special class of single-shot volumetric fluorescence imaging that focuses on computational imaging that performs 3D depth search using a single 2D light field image data. There are various light field computational tools designed to identify 3D information (x, y, z) of an object based on the angular disparity (r, θ) encoded in the light field image generated by a microlens array. However, current light field technology is not sophisticated enough to address the challenges of light sheet systems that use oblique plane illumination techniques with standard detection schemes.

[0005]

[0005] Embodiments of the present invention desirably address, at least in part, one or more of the drawbacks of the above-mentioned methods or systems. Furthermore, embodiments of the present invention desirably provide a method or system for obtaining volumetric information from oblique plane illumination techniques. In particular, embodiments of the present invention desirably provide a method or system for obtaining volumetric information from oblique plane light sheet illumination techniques using standard imaging detection microscope equipment.

[0006]

[0006] Reference to background art in this specification is not an admission that that art forms part of the common general knowledge of persons skilled in the art in Australia or any other country. Summary of the Invention

[0007]

[0007] According to one aspect of the present disclosure, there is provided a microscopy method for volumetric imaging of a sample disposed in an object space, the microscopy method including: acquiring a set of light field images using a light field detection device, each associated with an illumination slice corresponding to a light sheet projected into the object space from a particular illumination position at an illumination angle that is not parallel to the optical axis of the light field detection device, such that each voxel of a set of voxels associated with the object space, defined by a position in the object space, is illuminated by at least one illumination slice; and determining, for each voxel, the intensity of light emitted from the object space associated with the voxel, using in part the angular information captured by the light field images, such that the intensity determined for each voxel of the set of voxels defines a volumetric image of emitted light intensity in the object space.

[0008]

[0008] According to another aspect of the present disclosure, there is provided a microscope system for volumetric imaging of an object space of a sample, the microscope system comprising a light field detection device in communication with a computer having a processor and a memory, the light field detection device comprising optical elements configured to scan a plane of light formed from a laser light source at a plurality of positions at an illumination angle relative to an optical axis for illuminating the object space, such that each voxel of a set of voxels associated with the object space defined by its position in the object space is illuminated by at least one illumination slice, thereby generating a plurality of illumination slices, each associated with a unique position, and to image the object space for each illumination slice, thereby generating a corresponding light field image for each illumination slice; the processor is adapted to execute a plurality of modules stored in memory, each module capable of executing a set of instructions; and the plurality of modules are configured to determine, for each voxel, the intensity of light emitted from the object space associated with said voxel, in part using angular information captured by the light field image, such that the intensity determined for each voxel of the set of voxels defines a volumetric image of emitted light intensity in the object space.

[0009]

[0009] According to another aspect of the present disclosure, there is provided a computer-readable storage medium for generating a volumetric image storing instructions that, when executed by one or more processors of a computer, cause the computer to execute a plurality of modules stored in memory, each module capable of executing a set of instructions, the modules including: acquiring a set of light field images using a light field detection device, each associated with an illumination slice corresponding to a light sheet projected into the object space from a particular illumination position at an illumination angle that is not parallel to the optical axis of the light field detection device, such that each voxel of a set of voxels associated with the object space, defined by a position in the object space, is illuminated by at least one illumination slice; and determining, for each voxel, the intensity of light emitted from the object space associated with said voxel, using in part the angular information captured by the light field images, such that the intensity determined for each voxel of the set of voxels defines a volumetric image of emitted light intensity in the object space.

[0010] Optionally, the method further includes scanning the light sheet to illuminate the object space at a plurality of locations, such that each location is associated with a corresponding illumination slice of the light sheet, and imaging the object space as illuminated by each illumination slice to generate a set of voxels. Each voxel may be associated with a depth within the object space. The depth may be transverse to the optical axis.

[0011]

[0011] The light field detection device preferably includes a microlens array for generating the light field image. Accordingly, the method can include positioning the microlens array at an image plane. Each microlens of the array can have an imaging-side NA that is smaller than the NA of the microlens array.

[0012]

[0012] The scanning direction may be transverse to the angle of the plane of light. The illumination positions may be stepped with a constant step width when generating the set of light field images.

[0013] In one embodiment, the illumination angle is adjustable so that each light field image is associated with a unique combination of illumination position and illumination angle.

[0014] Optionally, determining the intensity of light emitted from the object space associated with said voxel comprises summing adjacent intensities within a predetermined axial distance of the voxel. Optionally, the method comprises correcting for apparent axial shifts in the apparent positions of the voxels.

[0015]

[0015] As used in this specification, the words "comprise," "include," and "having," or variations such as "comprises," "comprising," "includes," and "including," are used in an inclusive sense, i.e., they specify the presence of stated features but do not exclude the presence or addition of further features in various embodiments of the invention.

[0016]

[0016] One or more embodiments of the present invention will now be described with reference to the accompanying drawings. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a schematic diagram of an exemplary system for generating a volumetric image of an object space in a document, in accordance with a preferred embodiment of the present invention; [Figure 2] 1 is a flowchart of a method for generating a volumetric image of an object space in a sample, according to another preferred embodiment of the present invention. [Figure 3] 3 is a flow chart of the steps for processing the signals referred to in FIG. 2; [Figure 4] Ray diagrams of light field imaging under illumination at different tilt angles (α1=0°, α2=30°, α3=60°). [Figure 5] FIG. 1 is a ray diagram illustrating the mapping of excitation voxels of an illumination slice of one angle α in object space to form a light field image at the sensor plane. [Figure 6] FIG. 1 is a schematic diagram illustrating the extraction of excited voxels from the illumination of a slice. [Figure 7] 1. A view of excited voxels as planar images re-arranged in columns according to depth for depths Z1 to Z4 and re-assigned to stacks. [Figure 8a] 10 shows light field images captured of a sub-resolution fluorescent sample under light sheet illumination at different angles. [Figure 8b] 8b is a plot of the lateral (XY) PSF imaging obtained from the light field image of FIG. 8a using a conventional light field depth retrieval tool. [Figure 8c] 8b is a plot of the axial (XZ) PSF imaging obtained from the light field image of FIG. 8a using a conventional light field depth retrieval tool. [Figure 9a] Figure 1 shows plots of XY, XZ, and YZ slices of a 1 μm fluorescent microsphere excited by a light sheet illumination scanned at 60°, acquired by a conventional light field depth acquisition tool (left column) and by a method according to a preferred embodiment of the present invention (right column). [Figure 9b] 10 is a bar graph of the XZ and YZ axial FWHM profiles of a 1 μm fluorescent microsphere acquired by a conventional light field depth acquisition tool (left bar graph) and by a method according to a preferred embodiment of the present invention (right bar graph). [Figure 10a] FIG. 1 is a schematic diagram showing imaging of lithographic microstructures for verification purposes. [Figure 10b]1A-1C are plots of XY slices containing the letters "A" and "U" at z=-2 μm and z=2 μm acquired by a conventional light field depth acquisition tool (left column) and by a method according to a preferred embodiment of the present invention (right column). [Figure 11a] 10A-10C are plots of YZ slices across the center of fine structures acquired by a conventional light field depth acquisition tool (left column) in accordance with a preferred embodiment of the present invention (right column). [Figure 11b] 1 shows normalized axial intensity profiles across the letter "A" acquired by a conventional light field depth acquisition tool (left column) and in accordance with a preferred embodiment of the present invention (right column). [Figure 12a] The resulting axial focus shift of the modeled voxel, which shows the optical path change of the light induced by the refractive index change of the sample, can be compensated for by an iterative method. [Figure 12b] Multi-view excitation and axial reassignment are shown. DETAILED DESCRIPTION OF THE INVENTION

[0018] 1-12b, a light sheet microscope system 2 and method 4 for volumetric imaging of an object space 8 according to a preferred embodiment of the present invention are described. The system 2 and method 4 are suitable for volumetric imaging of the object space 8 and a sample (not shown) disposed therewith.

[0019]

[0018] As shown in FIG. 1, the light sheet microscope system 2 includes an apparatus 6 for directing excitation light from a laser source 10 to illuminate an object space 8 of a sample and then directing the light to a detector 12. The system 2 also includes a computing environment 100 for processing signals from the detector 12, which will be described in more detail in the following paragraphs. The apparatus 6 also includes a scanning device 14 that forms the excitation light from the laser source 4 into a sheet of light, otherwise known as a "light sheet" (i.e., the excitation light is formed in a plane and projected into the object space 8). In the embodiment shown in FIG. 1, the scanning device 14 includes a collimator, an iris, and a lens 16, all of which are mounted on a single-axis translation stage (not shown) that collimates, crops, and focuses the light. In one example, the scanning device 14 is configured to form a light sheet having a thickness of approximately 2.4 μm.

[0020]

[0019] Scanning device 14 directs the light sheet to the back focal plane of objective lens 18 to illuminate object space 8. Referring to FIG. 4, the incidence angle α of the light sheet is shown, which is the angle at which the light sheet enters object space 8 relative to the optical axis of objective lens 18. The incidence angle α can be adjusted by translating a stage on which the laser, collimator, iris, and lens 16 are located, the stage translation being in the direction of double arrow "C" in FIG. 1. The stage translation effectively offsets the light sheet at the back focal plane of objective lens 18. The resulting incidence angle α is conveniently referred to herein as the illumination angle α, which may be adjustable as shown in FIG. 4. In this arrangement, a maximum illumination angle α of approximately 60° is achieved using a 1.3 NA objective lens, although it will be understood that a larger or smaller illumination angle α may be provided depending on the design, requirements, and limitations of system 2.

[0021] The scanning device 14 comprises a pair of scanning galvo mirrors 20, 22 and associated lenses 24, 26. One scanning galvo mirror 20 scans the light beam vertically at a frequency much higher than the detector speed, thereby forming a thin, time-averaged beam that forms a light sheet. The boxes labeled B and A in Figure 1 show the light beam profiles after the scanning galvo mirrors 20 and 22, respectively.

[0022]

[0021] The second scanning galvo mirror 22 scans the laser line horizontally to achieve scan plane illumination, which laterally translates the light sheet across the object space 8. As described herein, the object space 8 is imaged (by the detector 12) as it is illuminated by the light sheet at different positions and, optionally, different illumination angles α, such that each image is essentially associated with an effectively stationary light sheet at a particular position (and optional illumination angle α); such "stationary" light sheets are referred to herein as illumination slices K. An index can be used to distinguish between different illumination slices K, such as the distinction between illumination slices K1, K2, and K3 in FIG. 6. Optionally, the lens 16 comprises an electronically adjustable lens (ETL), thereby enabling control of the focus of the laser line. Advantageously, the use of a lens 16 with an ETL can improve the effectiveness of the light sheet illumination by allowing the focal point to be adjusted to ensure it is optimally positioned within the object space 8.

[0023] In the embodiments described herein, "k" (e.g., as shown in FIGS. 5 and 6) is the lateral translation of the light sheet relative to the optical axis of the objective lens 18, e.g., the translation along the x-axis transverse to the optical axis, which is defined as the z-axis. In the example of FIGS. 5 and 6, each of K1, K2, and K3 is at a different position along the x-axis.

[0024] The apparatus 6 also includes a light field detection apparatus 28, which includes the objective lens 18 and a tube lens 30 for an effective magnification of 111x at the image plane 34 of the detector 12. The detection apparatus 28 also includes a microlens array 32, which in one example has a pitch and focal length of 150 μm and 3700 μm, respectively. In a preferred embodiment, the light field detection apparatus 28 is arranged for unfocused light field detection, with the microlens array 32 located at the image plane 34. In a more preferred embodiment, the microlens array 32 is located at the image plane 34 of a standard epifluorescence microscope.

[0025]

[0024] The detected light (e.g., from fluorescence emitted by a sample placed in object space 8 when illuminated by illumination slice K projected into object space 8) is split into a plurality of partial lenslet images by microlens array 32. This allows microlens array 32 to capture a light field image in object space 8, the light field image containing both intensity information and information about the direction in which the light rays incident on detector 12 are traveling. In a preferred embodiment, system 2 includes underfilling by having each microlens have an imaging side NA smaller than the NA of microlens array 32 (focal length -3700 μm, aperture size -150 μm, NA -0.02), which advantageously ensures light field imaging without overlapping lenslet images.

[0026] For purposes of this disclosure, the "x-axis" and "y-axis" are interpreted as axes parallel to the plane of detector 12 (i.e., parallel to image plane 34), and the "z-axis" is interpreted as an axis perpendicular to detector 12 and image plane 34. Thus, the x-axis and y-axis correspond to the spatial resolution of detector 12 ("lateral" resolution), and the z-axis corresponds to the depth of object space 8 ("axial" resolution).

[0027]

[0026] As shown in Figure 1, the device 6 also has an optional conventional wide-field detection device 36 for comparing the methods and systems of embodiments of the present invention with conventional light-field detection, as described in more detail below in the section entitled "Verification."

[0028] To illustrate the shortcomings of conventional light-field depth acquisition techniques, Fig. 8a shows three light-field images captured when illuminating a single 1 μm fluorescent microsphere 50 with three corresponding illumination angles (α1=0°, α2=30°, α3=60°). Using existing standard light-field depth acquisition tools, a point spread function (PSF) can be calculated from each light-field image in Fig. 8a.

[0029]

[0028] Figure 8b shows the transverse (XY) and Figure 8c shows the axial (XZ) PSFs obtained from the three different illumination conditions of Figure 8a. The results for both the transverse (XY) PSFs (compared to each other) and the axial (XZ) PSFs (compared to each other) at different illumination angles α1, α2, α3 are almost identical. Figure 8c also shows the axial PSFs (PSFs) calculated for each illumination angle α1, α2, α3. imaging ) are overlaid and calculated as follows:

number

[0030] This formula is derived from the light sheet PSF (gradient PSF illumination ) is the detector point spread function (PSF detection ), the point spread function (PSF) imaged from illumination with an oblique beam is imaging ) can be derived.

[0031]

[0030] The obtained axial PSFs in Fig. 8c are based on three different illumination angles α1, α2, and α3. imagingshould present a distorted intensity profile (ellipse). Since light field detection results in a lower spatial resolution at the detector 12 compared to the diffraction-limited resolution of the objective lens 18 (e.g., due to the diameter of the microlenses), we consider the PSF detection is a limited gradient PSF with a small thickness illumination This is because the gradient PSF illumination In the axial direction (Equation 1), PSF imaging This means that the erroneous PSF imaging leads to inaccuracies in depth retrieval and poor Richardson-Lucy deconvolution.

[0032] The inventors have demonstrated that, by use of a preferred embodiment of the present invention, PSF imaging It was decided that it would be desirable to restore

[0033] According to an embodiment of the present invention, the light field detection device 28 utilizes a collection of light field images captured of the object space 8 when illuminated by an illumination slice K from different positions and / or directions. Unless otherwise stated, it is assumed herein that each light field image is uniquely associated with a position of incidence at a common (for all light field images 38) illumination angle α.

[0034] 6, object space 8 is divided into a plurality of 3D imaging units, referred to herein as "voxels 40." Each voxel 40 represents a 3D location within object space 8. The voxels 40 are sized so that each can be independently measured by light field detection device 28 (i.e., voxels 40 must have a dimension greater than the resolving power of light field detection device 28). While the voxels 40 in FIG. 6 are shown as cubes, this is not intended to be limiting; for example, voxels 40 could instead represent spatial coordinates that are separated by a sufficient distance to minimize or eliminate overlap in signal detection between adjacent voxels 40, as depicted in FIG. 5.

[0035]

[0034] For ease of explanation, the term "voxel 40" is used to represent the corresponding location in the "real" object space 8 as well as the corresponding element of the resulting volumetric image (i.e., data structure) of the object space 8.

[0036] 5, there is shown a schematic diagram illustrating how voxels 40 in object space 8 relate to a light field image 38. The diagram shows two separate light sheets K1 and K2 illuminating object space 8. As shown, the first light sheet K1 is incident on the second light sheet K2 from a different position (but at the same illumination angle α). The diagram shows that certain voxels 40a are illuminated by the second light sheet K2, while the remaining voxels 40b are not illuminated.

[0037] According to one embodiment, each illumination slice K i is associated with a particular translation of the light sheet relative to the x-axis. iare spaced a distance k along the x-axis. Essentially, in the illustrated embodiment, as illustrated in FIG. 2, in the first step (S102) of method 4, light sheet K is scanned horizontally across object space 8 (i.e., along the x-axis). The choice to align the light sheet translation direction with the x-axis is arbitrary, but can simplify calculations. In such an arrangement, the other horizontal axis (y-axis) is parallel to the plane of the light sheet.

[0038]

[0037] Using ray matrix analysis (described in more detail below), the intensity of each voxel 40 (i.e., the sample emission from the location in object space 8 associated with the voxel 40) can be determined from the detected light field image obtained when the particular voxel 40 is illuminated. Considering the example of Figure 5, the captured light field image associated with illumination slice K2 is suitable for determining the emission intensity of each illuminated voxel 40a.

[0039] Each voxel 40 has an associated luminous intensity I x、y、z 6 shows three illumination slices K1, K2, and K3, each at the same illumination angle α, exciting diagonally positioned voxels 40a, 40b, 40c in different portions of object space 8. That is, the use of angled illumination slices K allows different depths of object space 8 to be imaged in each light field image, as the different depths are separated in the cross-section.

[0040]

[0039] Accordingly, in step S108, detector 12 images the sample illuminated by the plurality of illumination slices K projected by detector 12 into object space 8 (step S104 in FIG. 2), thereby creating a corresponding plurality of light field images (each light field image associated with a unique one of illumination slices K). In one embodiment, during steps S102-S108, the light sheet is scanned horizontally (i.e., in direction k in FIGS. 5 and 6) at a constant angle α. The resulting plurality of illumination slices K are parallel to one another, as shown in FIGS. 5 and 6. Furthermore, in one embodiment, illumination slices K correspond to uniformly spaced steps of the light sheet along the x-axis (e.g., at intervals δx). Generally, sufficient light field images are captured to ensure that all voxels 40 are illuminated by illumination slices K.

[0041] In one embodiment, the tilt angle α is varied (step S108). Steps S102-S106 can then be repeated to capture light field images as electronic signals by scanning the object space 8 at multiple illumination angles α and multiple illumination positions. In such a case, each illumination slice K and each light field image can be associated (e.g., indexed) with both the light sheet's incident position and incident angle α.

[0042] In steps S112-S124, once sufficient light field images have been captured, computing system 100 can process the light field images to generate a volumetric image. Computing system 100 can include processor 102 and data storage 104 (e.g., volatile and / or non-volatile memory, such as one or more hard drives, optical drives, dynamic memory, or solid-state memory) capable of storing light field images collected by detector 12 and other information, including system parameters, associated with system 2. Detector 12 is in data communication with computing environment 100, and processor 102 is configured to execute instructions in the form of software library routines or modules 106-114 specifically written to process electronic signals in accordance with an embodiment of the present invention, with reference to FIG. 1 .

[0043] FIG. 5 schematically illustrates the physical basis of the computational model embedded by steps S112-S124, according to one embodiment. This figure shows the relationship between the intensity of light emitted by the sample at a particular voxel 40a and the resulting detection signal at detector 12. For illustrative purposes, four rays R(1)-R(4) are traced from the location of a particular voxel 40a in object space 8 to coordinates S(1)-S(4) on detector 12. Rays R(1)-R(4) are then (effectively) mapped to pixels LF(1)-LF(4) of the light field image (i.e., a 2D array of detector pixels associated with a particular microlens that is itself associated with a particular voxel 40a in the particular light field image). The resulting light field pixels are analyzed to obtain a measure of the intensity of light emitted by the sample at the location of the particular voxel 40a.

[0044] According to this embodiment, the computational model uses ray matrix analysis, where individual rays R(N) of a voxel 40 are mapped from the object space 8 to the sensor plane S as shown in the following equation:

number

[0045]

[0044] Although Figure 5 shows the propagation of ray R(N) relative to the XZ plane, in general, the propagation of ray R(N) in both the XZ and YZ planes can be determined, and the results can then be combined to represent the 2D divergence angle of a particular ray from the optical axis. By uniformly distributing the N rays R(N), the angular information of the signal (eg, fluorescence) emitted by the voxel 40 can be calculated.

[0046] Considering the indexing of voxels 40, the focal point of objective lens 18 can be considered as the origin of the volumetric image defined by the set of voxels 40. For example, by using lateral and axial sampling factors δxy and δz (respectively), voxel V x,y,z It may be preferable to use a regular spacing of δxy×δz, i.e. the "size" of each voxel 40 relative to object space 8 is the product δxy×δz.

[0047] Next, we determine the expected light field image. The following equation describes how the sensor plane S is mapped to the light field pixel image LF:

number

[0048] The following formula is used to calculate the expected intensity Ix,y,z of each voxel 40 (ie, a voxel 40 located at coordinates (x,y,z) in object space 8).

number

[0049] 6 shows how the mapping of illumination slice K at angle α in object space 8 forms a volumetric image. In practice, once the intensities I x,y,z have been calculated for each voxel 40, it is possible to construct the volumetric image simply by assigning each voxel 40 to its position in object space 8. Thus, although a single light field image is associated with voxels 40 at different depths for different x-axis positions, the result of processing the set of all light field images is a data structure suitable for 3D volumetric imaging.

[0050] The reallocation process is given by the following equation:

number

[0051] Optionally, the model can take into account the thickness t of the oblique surface illumination during processing. This involves summing a given voxel's neighbors along the z-axis (and itself) to represent the resultant intensity as described in the following equation, where the resulting voxel intensity is I′ and t / δz is the “depth factor” (i.e., D represents the nearest number of voxels covered by the oblique surface thickness). By taking t into account, fluorescence signal collection is maximized while rejecting out-of-focus light.

number

[0052] That is, the total intensity I' is based on the focal intensity I of the voxel in question (ie, Vx,y,z) and the focal intensities of each of its neighboring voxels in the z direction (within a distance ±D / 2).

[0053] 12a, the sample 60 itself can cause a shift between the apparent illumination point 62 and the actual illumination point 64 within the sample 60 due to refractive index effects of the sample 60. For example, refractive index effects of the sample 60 can result in a change in the optical path that can lead to an axial (z-direction) focus shift. This can cause the relative position of the modeled voxel 40 to shift from its corresponding relative position within the sample 60, and as a result, the modeled voxel 60 can exhibit aberrations relative to the sample. This is indicated by 66 (illumination position) and 68 (apparent point of emitted light from the sample 60).

[0054]

[0053] Thus, in one embodiment, Equation 8 is utilized to identify, quantify, and evaluate axial focus shift. This embodiment is based on the known position of a particular illumination slice K, from which the expected location 64 of an excited voxel can be identified, and axial focus shift is expected to shift the apparent (i.e., modeled) location 62 of an excited voxel 40 relative to the expected location 64 in the axial direction (z direction). Thus, in this embodiment, Equation 8 is utilized as a search function over the range of voxels 40 axially relative to the expected location of voxels 40. The search function looks to maximize I'x,y,z with respect to z. The apparent maximum intensity can then be mapped to the excited voxel 40, thereby accounting for focus shift.

[0055]

[0054] Then, by following this procedure and adjusting the parameters in Equation 2, and thus using accurate coordinates, axial reassignment by depth mapping can produce a light field image that advantageously has reduced or no aberrations for a given illumination slice K.

[0056]

[0055] This embodiment can optionally be extended by using multi-view excitation, i.e., illuminating the same voxel 40 by illumination slices K rotated about the z-axis (see K1, K2, K3, K4, which represent rotations in 90° increments, although other angular steps can be utilized), as shown in Figure 12b. This may advantageously allow iterative determination of axial focus shifts between full apertures.

[0057] Exemplary Implementation Using MATLAB Several MATLAB™ modules may be suitable for implementing the steps of Method 4. In particular, the inventors have utilized MATLAB modules such as EXE_rectify 106, EXE_parameter 108, EXE_model, EXE_translate 110, EXE_mapping 112, and EXE_reassign 114, the functions of which are shown in Table 1 below. These modules may be suitable for performing aspects of steps S112-S126. Although modules 106-114, having the functions shown in the table below, are prepared as MATLAB modules, this is not intended to be limiting, as one or more functions of the modules may be implemented via different programming languages ​​or modules, as known to those skilled in the art.

[0058]

[0057] [Table 1]

[0059] In module EXE_rectify 106, light field image processing is performed to ensure that lateral translations are correctly interpreted in the subsequent modules EXE_mapping 114 and EXE_reassign 116. To do this, the light field image must be rotated so that the illumination angle α, i.e., the scanning direction, matches the x-axis of the model used (see steps S114-S116). Depth information can therefore be obtained from the light field image at any tilt angle α by this method 4.

[0060] Module EXE_recify106 includes light field image preprocessing in step S112, which may include a light field image correction or calibration step. In one embodiment, the correction includes capturing an image of a sample with uniform fluorescent emission, such as a fluorescent microscope slide. The light field image of the uniform fluorescent emission is used to generate a microlens array correction setting based on the particular light field detection optics used. The inventors have found that adjusting the tilt angle α does not significantly affect the correction setting.

[0061]

[0060] Module EXE_recify106 may include further pre-processing, including cropping the raw light field image that is scaled to have an integer number N of pixels, since only light field images with a field of view (FOV) smaller than the modeled FOV are acceptable.

[0062]

[0061] The module EXE_parameter 108 outlines the parameters used to determine the volumetric images calculated in the subsequent modules EXE_model 110, EXE_mapping 114, and EXE_reassign 116.

[0063]

[0062] The module EXE_model 110 performs modeling using the parameters defined in the module EXE_parameter 108. To aid in faster calculations, pre-calculation based on previously known parameters from known systems can be performed by extracting a data set from a pre-calculated model. The parameters for the pre-calculated model can be outlined in the module EXE_parameter 108. The following table shows the parameters used in the module EXE_model 110:

[0064]

[0063] [Table 2]

[0065] As illumination slice K is translated laterally, a series of light field images, k, from each other are recorded, with each light field image representing illumination at a unique k assigned to a scan index. To determine the position k of each illumination slice K in step S118, a calibration protocol is performed to determine the lateral translation k as a function of the scan index in module EXE_translate 112. Calibration includes ensuring that galvo mirror 20 is conjugate to the back focal plane of the objective lens, thereby achieving tilt-invariant scanning, and ensuring that microlens array 32 is conjugate to the front focal plane of objective lens 18. Once accurate conjugation is achieved, lateral translation can be determined by imaging a sample with uniform fluorescent emission and plotting the processed light field images to determine the illumination profile with their spatial position, which advantageously ensures the accuracy of k.

[0066] The emission intensity of each of the voxels 40 when excited by the respective illumination slice K is then mathematically reassigned to a 3D position using a different Z slice (see step S124). In the example shown in Figure 7, Z slices Z1, Z2, Z3, Z4 are mapped to the depths of illumination slices K1, K2, and K3. Depth mapping is facilitated by EXE_mapping 114 and can be done on either a single light field image, a sequence of indexed light field images, or light field images streamed live from the imaging sensor / detector 12.

[0067] In module EXE_mapping 114, the light field image is received together with its lateral translation determined in EXE_translate 112 to generate a set of images as a set of voxels according to the position k (step S120). This generates a set of oblique plane images P1, P2 and P3 for a given illumination angle α. Each plane image P1, P2 and P3 can then be repositioned to have voxels 40 (voxels) in the form of a column of pixels associated with one or more of different depths Z1, Z2, Z3, Z4 (see FIGS. 6 and 7), a so-called "depth mapping".

[0068] These columns of voxels 40 are then reassigned to different plots or z-stacks, each associated with a different depth in step S124 of module EXE_reassign 116. In particular, the columns of pixels (voxels) in image P1 relating to depth Z3 are assigned to the Z3 stack, the columns of pixels in image P1 relating to depth Z2 are assigned to the Z2 stack, and the columns of pixels in image P1 relating to depth Z1 are assigned to the Z1 stack (see dashed lines in FIG. 7). Similarly, the columns of pixels or voxels 44, 46 in P2 and P3 are reassigned to corresponding z-stacks Z1, Z2, Z3, Z4 (see dotted and solid lines). Computationally, the pixel reassignment represents only 5% of the required calculations, thus making this step quick to compute. The different z-stacks are then graphically represented by a graphics module in step S126 to provide a volumetric image as desired.

[0069]

[0068] Thus, by the system 2 and method 4 of this preferred embodiment of the present invention, more accurate information about the depth of a section of object space 8 is achieved than was previously possible with standard depth acquisition techniques.

[0070] In addition to obtaining improved depth information from the detected light field image, the method and system of the present invention advantageously allows for obtaining depth information regardless of the light sheet angle α. Specifically, the mapping of illumination slice K includes only a subset of voxels 40 because it models all voxels 40 sampled in object space 8, and S x′′,y′′ The data set is independent of the oblique plane illumination angle α, which can be calculated in advance using predetermined system parameters, for example, using the module EXE_parameter 108, to calculate the illumination α for any angle. x′′,y′′ This means that the vertices can be mapped by

[0071] We have found that the performance (i.e., computational efficiency) of the depth identification step is improved by S x′′,y′′ We found that the computational efficiency depends significantly on both the size of the dataset and the tilt angle α. Specifically, a larger angle α results in more voxels in the x-axis, which are interrogated by the following equation, describing the profile of the tilted plane illumination slice and therefore increasing the depth mapping time. Furthermore, although computational efficiency is related to the lateral sampling δxy, we found that the parameters of the objective lens, tube lens, and microlens array do not significantly affect computational efficiency.

[0072] Importantly, through the development of the disclosed system and method, the inventors have achieved recovery of sufficient depth information to generate high-speed, high-resolution volumetric images from single-objective light-sheet microscopy using conventional microscope imaging equipment, preferably generating volumetric images in real time. Specifically, depth information can be recovered without the need for an additional remote optical imaging unit typically required for conventional light-sheet microscopy. As described in the "Experimental Results, Verification" section below, optical sectioning is also advantageously improved by a factor of two, and volumetric images can be achieved in approximately 0.5 seconds on a standard CPU without multicore parallel processing.

[0073]

[0072] The inventors believe that this method 4 and system 2 are not only useful for conventional laser scanning microscopes, but are also applicable to many applications involving adaptive optics and uses involving structured beams (Airy beams).

[0074]

[0073] Embodiments herein describe a system 2 and method 4 that provide 3D imaging of an object space (more specifically, a sample located within the object space). For example, the sample may be a biological sample that, when illuminated by light of a particular wavelength, may emit light at certain other wavelengths (e.g., via fluorescence).

[0075]

[0074] The object space 8 is illuminated by a light sheet (incident light defining a plane) and imaged by a light field detector 28 having an optical axis whereby the light sheet is directed into the object space at a constant angle α relative to the optical axis. The use of the tilt angle α and a planar light sheet allows the embodiment to effectively "see behind" the front of the sample (i.e., facing the light field detector 28), thereby allowing detection of emission light at different depths in the sample.

[0076]

[0075] The object space 8 can be associated with a set of voxels 40, and each voxel 40 can be illuminated by imaging the object space with light sheets incident from different positions such that the resulting set of images collectively represents each voxel 40 in the object space 8.

[0077]

[0076] Light field techniques are therefore used by the embodiments described herein to enable accurate measurement of the light intensity emitted by voxels 40 at all depths within the object space 8, and therefore at all depths of the sample. In the embodiments described herein, a microlens array 32 is provided in the imaging plane of the light field detection device 28, and a detector is positioned to detect the resulting image due to the microlens array 32. The light field image thereby obtained is suitable for reconstructing depth information, allowing accurate determination of the intensity emitted by each voxel 40, regardless of voxel depth.

[0078]

[0077] The embodiments described herein can advantageously enable 3D imaging of a sample using inclined plane techniques while enabling accurate reconstruction at different sample depths without the need for a remote imaging unit which can include two (or more) complementary objective lenses (secondary and tertiary objective lenses) in addition to the primary objective lens.

[0079] Experimental results and verification As a demonstration and validation of the more accurate volumetric imaging achieved by Method 4 and System 2 according to a preferred embodiment of the present invention, an example is shown of imaging a 1 μm fluorescent microsphere 50 below the resolution limit of the optical system used as shown in FIG. 4. In FIG. 9a, the columns show imaging by standard light-field depth acquisition techniques with significant out-of-focus blur in both the XZ and YZ planes. The columns in FIG. 9b show imaging by the method and system as described above, showing a factor of 2 reduction in out-of-focus signal in the axial direction, compared to the ideal PSF. imaging The effective PSF with axial FWHM in XZ and YZ (see Figure 9b) (1.39 μm and 1.37 μm) is close to the theoretical limit of the system, with a profile nearly identical to that of the axial FWHM of the microsphere, 1.35 μm, thereby achieving accurate depth sectioning of the microsphere. imaging The gray shading indicates the ideal PSF. imaging Shows.

[0080] To further validate the present method and system on densely packed samples, imaging was performed on customized laser-written, hard fluorescent microstructures 52, 54, 56, with the example shown in FIG. 10a, in which the letters "A" 52 and "U" 54 are stacked vertically on a glass coverslip with a support structure 56. The solid line represents the scanning illumination slice K, and the dashed line represents the YZ slice across the center of the microstructures 52, 54, 56. FIG. 10b shows XY slices containing the letters "A" and "U" at z=-2 μm and z=2 μm, respectively, using standard optical depth acquisition techniques in the left column and the imaging technique of the present method and system in the right column. This allows for a small separation of 4 μm along the z-axis between "A" and "U" to test axial sectioning.

[0081] From Figure 10b, it appears that both the standard light-field depth acquisition technique and the imaging technique of the present method and system accurately depict the characters at their specified axial locations, excluding out-of-focus intensity arising from other characters located 4 μm away in z. However, upon closer examination of a YZ slice (see Figure 10b) across the center of the microstructure (dashed line in Figure 10a), it is observed that the standard light-field depth acquisition technique (item 60—left column in Figure 10b) removed only half of the out-of-focus cone, while the imaging method of the present invention (item 62—right column in Figure 10b) exhibits no out-of-focus cone. Figure 11a plots the normalized axial intensity across the letter "A" (white line in Figure 11a) and quantifies the out-of-focus region where the standard light-field depth acquisition technique 58 produces the top image and the method of the present invention produces the bottom image 60. Standard light-field depth acquisition techniques result in out-of-focus regions extending beyond 8 μm (FWHM) in depth (see item 60). Conversely, imaging with the present method achieves a sharp axial fluorescent signal of the letter "A" (see item 62 in FIG. 11b) accurately mapped to z=-2 μm with an axial profile (FWHM) of 1.9 μm. Results of the present imaging method and system for densely packed fluorescent microstructures show a significant improvement in depth sectioning over standard light-field depth acquisition techniques.

[0082]

[0081] Thus, the above-described systems and methods for generating volumetric images advantageously provide more accurate and deeper depth information with sufficiently fast computation to generate volumetric images in real time with a standard laser scanning microscope without the need for a remote optical imaging unit.

[0083] Further modifications can be made without departing from the spirit and scope of this specification.

[0084] References [1]Bouchard,Matthew B.,et al.「Swept confocally-aligned planar excitation(SCAPE)microscopy for high-speed volumetric imaging of behaving organisms.」Nature photonics 9.2(2015):113-119. https: / / doi.org / 10.1038 / nphoton.2014.323 [2]Kumar,Manish,et al.「Integrated one-and two-photon scanned oblique plane illumination(SOPi)microscopy for rapid volumetric imaging.」Optics express 26.10(2018):13027-13041. https: / / doi.org / 10.1364 / OE.26.013027 [3]Madaan,Sara,et al.「Single-objective selective-volume illumination microscopy enables high-contrast light-field imaging.」Optics Letters 46.12(2021):2860-2863. https: / / doi.org / 10.1364 / OL.413849 [4]Yang,Bin,et al.「DaXi-high-resolution,large imaging volume and multi-view single-objective light-sheet microscopy.」Nature methods 19.4(2022):461-469. https: / / doi.org / 10.1038 / s41592-022-01417-2

Claims

1. 1. A microscopy method for volumetric imaging of a sample disposed in an object space, comprising: acquiring a set of light field images using the light field detection device, each associated with an illumination slice corresponding to a light sheet projected into the object space from a particular illumination position at an illumination angle that is not parallel to an optical axis of the light field detection device, such that each voxel of a set of voxels associated with the object space, defined by a position in the object space, is illuminated by at least one illumination slice; determining, for each voxel, an intensity of light emitted from the object space associated with the voxel, using in part angular information captured by the light field image, such that the intensities determined for each voxel of the set of voxels define a volumetric image of emitted light intensities in the object space; Microscopy, including

2. scanning a light sheet to illuminate the object space at a plurality of positions such that each position is associated with an illumination slice corresponding to the light sheet; imaging the object space as illuminated by each illumination slice to generate the set of voxels; The microscopy method of claim 1 further comprising:

3. 3. A microscopy method according to claim 1 or claim 2, wherein each voxel is associated with a depth within the object space transverse to the optical axis.

4. 4. The microscopy method according to claim 1, wherein the light field detection device comprises a microlens array for generating the light field image.

5. The microscopy method of claim 4 , including positioning the microlens array at an image plane.

6. 6. The microscopy method of claim 4 or claim 5, wherein each microlens of the array has an imaging-side NA that is smaller than the NA of the microlens array.

7. Microscopy according to any one of claims 1 to 6, wherein the scanning direction is transverse to the angle of the plane of light.

8. The microscopy method of claim 7 , wherein the illumination positions are stepped with a constant step width when generating the set of light field images.

9. A microscopy method according to any one of claims 1 to 8, wherein the illumination angle is adjustable such that each light field image is associated with a unique combination of illumination position and illumination angle.

10. A microscopy method according to any one of claims 1 to 9, wherein determining the intensity of the light emitted from the object space associated with the voxel comprises summing adjacent intensities within a predetermined axial distance of the voxel.

11. A microscopy method according to any preceding claim, comprising correcting for apparent axial shifts in the apparent positions of said voxels.

12. 1. A microscope system for volumetric imaging of an object space of a sample, comprising: the microscope system comprising a light field detection device in communication with a computer having a processor and a memory, scanning a plane of light formed from a laser light source at an illumination angle relative to an optical axis for illuminating the object space at a plurality of positions such that each voxel of a set of voxels associated with the object space, defined by a position within the object space, is illuminated by at least one illumination slice, thereby generating a plurality of illumination slices, each associated with a unique position; and a light field detection device comprising optical elements configured to image the object space for each illumination slice, thereby generating a corresponding light field image for each illumination slice; the processor is adapted to execute a plurality of modules stored in a memory, each module capable of executing a set of instructions, and the plurality of modules are configured to determine, for each voxel, an intensity of light emitted from the object space associated with the voxel, using in part angular information captured by the light field image, such that the intensity determined for each voxel of the set of voxels defines a volumetric image of emitted light intensities in the object space. Microscope system.

13. The microscope system of claim 12 , wherein each voxel is associated with a depth within the object space that is lateral to the optical axis.

14. 14. The microscope system of claim 12 or claim 13, wherein the light field detection device comprises a microlens array for generating the light field image.

15. The microscope system of claim 14 , wherein the microlens array is positioned at an image plane.

16. 16. The microscope system of claim 14 or claim 15, wherein each microlens of the array has an imaging-side NA that is smaller than the NA of the microlens array.

17. A microscope system according to any one of claims 12 to 16, wherein the scanning direction is transverse to the angle of the plane of light.

18. The microscope system of claim 17 , wherein the illumination positions are stepped with a constant step width when generating the set of light field images.

19. A microscope system according to any one of claims 12 to 18, wherein the illumination angle is adjustable such that each light field image is associated with a unique combination of illumination position and illumination angle.

20. A microscope system according to any one of claims 12 to 19, wherein determining the intensity of the light emitted from the object space associated with the voxel comprises summing adjacent intensities within a predetermined axial distance of the voxel.

21. The microscope system according to any one of claims 12 to 20, wherein the plurality of modules are further configured to correct apparent axial shifts of the apparent positions of the voxels.

22. 1. A computer-readable storage medium for generating volumetric images storing instructions that, when executed by one or more processors of a computer, cause the computer to execute a plurality of modules stored in a memory, each module capable of executing a set of instructions, the modules comprising: acquiring a set of light field images using the light field detection device, each associated with an illumination slice corresponding to a light sheet projected into the object space from a particular illumination position at an illumination angle that is not parallel to an optical axis of the light field detection device, such that each voxel of a set of voxels associated with the object space, defined by a position in the object space, is illuminated by at least one illumination slice; determining, for each voxel, an intensity of light emitted from the object space associated with the voxel, using in part angular information captured by the light field image, such that the intensities determined for each voxel of the set of voxels define a volumetric image of emitted light intensities in the object space; 1. A computer-readable storage medium comprising:

Images