Computational microscope method and system for volumetric imaging

Abstract

A microscopy method for volumetric imaging of a sample located within an object space, comprising: acquiring a set of light field images using a light field detection device, each light field image being associated with an illumination slice, the illumination section corresponds to a light sheet projected into the object space from a specific illumination position and at an illumination angle non-parallel to the optical axis of the light field detection device, so that each voxel in a group of voxels associated with the object space is illuminated by at least one illumination section, and each voxel is defined by a position in the object space; and, for each voxel, determining an intensity of light emitted from an object space associated with the voxel using, in part, angle information captured by the light field image, such that the determined intensity for each voxel in the set of voxels defines a volumetric image of the intensity of emitted light within the object space, and an associated system.

Description

Technical Field

[0001] The present invention generally relates to microscopy systems and methods for volumetric imaging of an object space in a sample. Background Art

[0002] Various three-dimensional or volumetric imaging methods are known, such as confocal laser scanning microscopy and others, which are used to image biological samples. Subsequently, light sheet microscopy was developed, in which the sample is moved through a light plane to achieve optical sectioning of the sample. More recently, single objective scanning light sheets using oblique plane (OP) illumination techniques (in which the light sheet is at a non-incident angle) have pushed the limits of image-based biological studies.

[0003] The main disadvantage of using a single objective to scan light sheets is the need for an expensive and complex telescopic imaging unit containing two complementary objectives (secondary and tertiary objectives) to achieve the necessary diffraction-limited imaging and optical sectioning. Furthermore, additional elements such as scanning mirrors, diffraction gratings or custom prisms are required to tilt-correct and reproduce the 3D volumetric image on the 2D imaging sensor. More importantly, these telescopic focusing units are not compatible with standard imaging detection schemes, which usually contain a single tube mirror with a 2D camera sensor. Therefore, single objective scanning light sheet systems (such as eSPIM and SCAPE) are limited to specialized microscope setups.

[0004] Light field is a special type of single-exposure volumetric fluorescence imaging that focuses on computational imaging using a single 2D light field image data to perform 3D depth retrieval. A range of light field computational tools exist that are designed to identify 3D information (x,y,z) of objects based on the angular parallax (r,θ) encoded in the light field image generated by a microlens array. However, current light field technology is not advanced enough to address the challenges of light sheet systems using oblique plane illumination techniques using standard detection schemes.

[0005] Embodiments of the present invention are intended to at least partially address one or more of the disadvantages of the above-described methods or systems. In addition, embodiments of the present invention are intended to provide a method or system for retrieving volume information from an oblique plane illumination technique. In particular, embodiments of the present invention preferably provide a method or system for retrieving volume information from an oblique plane light sheet illumination technique using a standard imaging detection microscope device setup.

[0006] Reference to the background art herein is not an admission that the art forms part of the common general knowledge of a person skilled in the art in Australia or any other country. Summary of the invention

[0007] According to one aspect of the present disclosure, there is provided a microscopy method for volumetric imaging of a sample located in an object space, comprising: obtaining a set of light field images using a light field detection device, each light field image being associated with an illumination slice corresponding to a light sheet projected into the object space from a specific illumination position and at an illumination angle that is not parallel to an optical axis of the light field detection device, such that each voxel in a set of voxels associated with the object space is illuminated by at least one illumination slice, wherein each voxel is defined by a position in the object space; and for each voxel, determining, in part using angular information captured by the light field image, an intensity of light emitted from the object space associated with the voxel, such that the determined intensity for each voxel in the set of voxels defines a volumetric image of the intensity of the emitted light within the object space.

[0008] According to another aspect of the present disclosure, a microscope system for volumetric imaging of an object space of a sample is provided, the system comprising a light field detection device in communication with a computer having a processor and a memory: the light field detection device comprises an optical element, the optical element being configured to: scan a light plane at a plurality of positions, thereby generating a plurality of illumination slices, each illumination slice being associated with a unique position, wherein the light plane is formed by an illumination angle of a laser light source relative to an optical axis to illuminate the object space, so that each voxel in a set of voxels associated with the object space is illuminated by at least one illumination slice, wherein each voxel is defined by a position in the object space; and image the object space for each illumination slice, thereby generating a corresponding light field image for each illumination slice; and the processor being adapted to execute a plurality of modules stored in the memory, each module being capable of executing a set of instructions, wherein the modules are configured to: for each voxel, determine, in part using angle information captured by the light field image, an intensity of light emitted from the object space associated with the voxel, so that the determined intensity for each voxel in the set of voxels defines a volumetric image of the intensity of the emitted light within the object space.

[0009] According to another aspect of the present disclosure, a computer-readable storage medium for generating a volumetric image is provided, the computer-readable storage medium storing instructions, which, when executed by one or more processors of a computer, causes the computer to execute a plurality of modules stored in the memory, each module being capable of executing a set of instructions, and wherein the modules include: obtaining a set of light field images using a light field detection device, each light field image being associated with an illumination slice, the illumination slice corresponding to a light sheet projected into an object space from a specific illumination position and at an illumination angle that is not parallel to an optical axis of the light field detection device, such that each voxel in a set of voxels associated with the object space is illuminated by at least one illumination slice, wherein each voxel is defined by a position in the object space; and, for each voxel, determining, in part, using angle information captured by the light field image, an intensity of light emitted from the object space associated with the voxel, such that the determined intensity for each voxel in the set of voxels defines a volumetric image of the intensity of emitted light in the object space.

[0010] Optionally, the method further comprises: scanning the light sheet at a plurality of positions to illuminate the object space, such that each position is associated with an illumination slice corresponding to the light sheet; and imaging the object space when illuminated by each illumination slice to generate the set of voxels. Each voxel may be associated with a depth in the object space. The depth may be a direction transverse to the optical axis.

[0011] The light field detection device preferably comprises a microlens array for generating a light field image. Thus, the method may comprise positioning the microlens array at the image plane. Each microlens of the array may have an imaging side NA that is less than the NA of the microlens array.

[0012] The scanning direction may be a direction transverse to the angle of the light plane.When generating the set of light field images, the illumination position may be stepped with a constant step size.

[0013] In one embodiment, the illumination angle is adjustable such that each light field image is associated with a unique combination of illumination position and illumination angle.

[0014] Optionally, determining the intensity of light emitted from object space associated with said voxel comprises summing adjacent intensities within a predefined axial distance of said voxel. Optionally, the method comprises correcting for an apparent axial offset of the apparent position of the voxel.

[0015] As used herein, the words "comprises," "including," and "having," or variations thereof, such as "comprising," "including," "comprising," and "comprising," are used in an inclusive sense, i.e. specifying the presence of stated features but not excluding the presence or addition of further features in various embodiments of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] One or more embodiments of the present invention will be described below with reference to the accompanying drawings, in which:

[0017] Figure 1 is a schematic diagram of an exemplary system for generating a volumetric image of an object space in a sample according to a preferred embodiment of the present invention;

[0018] Figure 2 is a flow chart of a method for generating a volume image of an object space in a sample according to another preferred embodiment of the present invention;

[0019] Figure 3 yes Figure 2 A flowchart of the steps of signal processing referred to in;

[0020] Figure 4 is the light diagram of light field imaging under illumination at different tilt angles (α1 = 0°, α2 = 30°, α3 = 60°);

[0021] Figure 5 is a ray diagram showing mapping of excited voxels of an illumination slice at angle α in object space to form a light field image on a sensor plane;

[0022] Figure 6 is a schematic diagram showing the extraction of excited voxels from slice illumination;

[0023] Figure 7 is a view of the excited voxels as a planar image rearranged in columns with respect to depths Z1 to Z4 and redistributed into the stack according to depth;

[0024] Figure 8a Light-field images of sub-resolution fluorescent samples captured under light-sheet illumination at different angles are shown;

[0025] Figure 8b and 8c is retrieved from the conventional light field depth retrieval tool Figure 8a Figure 2. Transverse (XY) and axial (XZ) PSF images retrieved from the light field image.

[0026] Figure 9a Graphs showing XY, XZ and YZ slices of 1 μm fluorescent microspheres excited by 60° scanning light sheet illumination retrieved by a conventional light field depth retrieval tool (left column) and a method according to a preferred embodiment of the present invention (right column);

[0027] Figure 9b is a bar graph of the XZ and YZ axial FWHM profiles of 1 μm fluorescent microspheres retrieved by a conventional light field depth search tool (left bar graph) and a method according to a preferred embodiment of the present invention (right bar graph);

[0028] Fig.10a is a schematic diagram showing the imaging of photolithographic microstructures for verification purposes;

[0029] Fig.10b is a diagram of XY slices containing letters "A" and "U" at Z=-2μm and Z=2μm retrieved by a conventional light field depth retrieval tool (left column) and a method according to a preferred embodiment of the present invention (right column);

[0030] Fig.11a is a diagram of a YZ slice through the center of a microstructure retrieved by a conventional light field depth retrieval tool (left column) according to a preferred embodiment of the present invention (right column);

[0031] Fig.11b shows the normalized axial intensity distribution of the letter "A" retrieved by the conventional light field depth search tool (left column) according to the preferred embodiment of the present invention (right column);

[0032] Fig.12a shows that the optical path changes of light due to refractive index changes in the sample, and the resulting axial focal shift of the modeled voxel can be compensated by an iterative method; and

[0033] Figure 12b Multiview excitation and axial redistribution are shown. DETAILED DESCRIPTION

[0034] Reference now Figures 1 to 12b , which describes a light sheet microscope system 2 and a method 4 for volume imaging of an object space 8 according to a preferred embodiment of the present invention. The system 2 and the method 4 are suitable for volume imaging of a sample (not shown) located in the object space 8.

[0035] The light sheet microscope system 2 has a device 6 for directing excitation light from a laser light source 10 to illuminate the object space 8 of the sample, and which then directs the light to a detector 12, such as Figure 1 The system 2 also has a computing environment 100 for processing signals from the detector 12, which will be described in more detail in the following paragraphs. The device 6 also includes a scanning device 14, which forms the excitation light into a sheet of light from the laser light source 4 (i.e., the excitation light is formed into a plane and projected into the object space 8), also called a "light sheet". Figure 1 In the illustrated embodiment, the scanning device 14 includes a collimator, an aperture, and a lens 16, all mounted on a single-axis translation stage (not shown) that collimates, crops, and focuses the light. In one example, the scanning device 14 is configured to form a light sheet having a thickness of approximately 2.4 μm.

[0036] The scanning device 14 directs the light sheet to the back focal plane of the objective lens 18 to illuminate the object space 8. Figure 4 , shows the incident angle α of the light sheet, i.e. the angle at which the light sheet enters the object space 8 relative to the optical axis of the objective lens 18. The incident angle α can be adjusted by translating the platform on which the laser, collimator, aperture and lens 16 are located, wherein the translation of the platform is along Figure 1 The translation of the stage effectively shifts the light sheet at the rear focal plane of the objective lens 18. The resulting incident angle α is conveniently referred to herein as the illumination angle α, which is Figure 4 In this arrangement, a maximum illumination angle α of approximately 60° is achieved using a 1.3NA objective, however it will be appreciated that a larger or smaller illumination angle α may be provided depending on the design, requirements and limitations of the system 2 .

[0037] The scanning device 14 comprises a pair of scanning mirrors 20, 22 and associated lenses 24, 26. One scanning mirror 20 scans the beam vertically at a frequency much higher than the detector rate, thereby forming a thin time-averaged beam which forms a light sheet. Figure 1 The boxes marked B and A in FIG. 1 show the profiles of the beams after the scanning mirrors 20 and 22 , respectively.

[0038] The second scanning galvanometer 22 scans the laser line horizontally to achieve scan plane illumination so as to translate the light sheet laterally across the object space 8. As described herein, when the light sheet illuminates the object space 8 at different positions and optionally at different illumination angles α, the object space 8 is imaged (via the detector 12) such that each image is substantially associated with a light sheet that is effectively stationary at a particular position (and optionally illumination angle α) - such a "stationary" light sheet is referred to herein as an illumination slice K. An index may be used to distinguish between different illumination slices K, for example Figure 6 , and the illumination slices K1, K2, and K3 in the object space 8. Optionally, the lens 16 comprises an electronically tunable lens (ETL), thereby enabling control of the focus of the laser line. Advantageously, the use of a lens 16 comprising an ETL can improve the effectiveness of the light sheet illumination by allowing adjustment to ensure that the focus is optimally positioned within the object space 8.

[0039] In the embodiments described herein, "k" (e.g., Figure 5 and Figure 6 ) is a lateral translation of the light sheet relative to the optical axis of the objective lens 18, for example, a translation along the x-axis transverse to the optical axis defined as the z-axis. Figure 5 and Figure 6 In the example of , each of k1, K2 and K3 is located at a different position along the x-axis.

[0040] The device 6 also includes a light field detection device 28, which includes an objective lens 18 and a tube lens 30 for effectively magnifying 111 times at an image plane 34 of the detector 12. The detection device 28 also includes a microlens array 32, which in one example has a pitch of 150 μm and a focal length of 3700 μm, respectively. In a preferred embodiment, the light field detection device 28 is arranged for non-focused light field detection, wherein the microlens array 32 is placed at the image plane 34. In a more preferred embodiment, the microlens array 32 is placed at the image plane 34 of a standard epi-fluorescence microscope.

[0041] The detected light (e.g., fluorescence emitted from a sample located in the object space 8 when illuminated by an illumination slice K projected into the object space 8) is divided into a plurality of partial lenslet images by the microlens array 32. The microlens array 32 thereby enables the capture of a light field image of the object space 8; the light field image comprises both intensity information and information about the direction of travel of light rays incident on the detector 12. In a preferred embodiment, the system 2 comprises having each microlens underfilled with an imaging side NA (focal length - 3700 μm, aperture size - 150 μm, NA - 0.02) less than the NA of the microlens array 32, which advantageously ensures light field imaging without overlapping lenslet images.

[0042] For purposes of this disclosure, the "x-axis" and "y-axis" are considered to be "x-axis" and "y-axis" parallel to the plane of the detector 12 (i.e., parallel to the image plane 34), while the "z-axis" is considered to be perpendicular to the detector 12 and the image plane 34. Thus, the x-axis and y-axis correspond to the spatial resolution of the detector 12 (the "lateral" resolution), while the z-axis corresponds to the depth of the object space 8 (the "axial" resolution).

[0043] like Figure 1 As shown, the device 6 also has an optional conventional wide field detection device 36, however this is for the purpose of comparing the method and system of embodiments of the present invention with conventional light field detection, as discussed in more detail below in the section entitled "Validation".

[0044] To illustrate the shortcomings of traditional light field depth retrieval techniques, Figure 8a Three light field images captured when illuminating a single 1 μm fluorescent microsphere 50 at three corresponding illumination angles (α1 = 0°, α2 = 30°, α3 = 60°) are shown. Using existing standard light field depth retrieval tools, the depth of the image can be obtained from Figure 8a A point spread function (PSF) is calculated for each light field image.

[0045] Figure 8b Shows from Figure 8a The lateral (XY) PSF obtained under three different lighting conditions, Figure 8cThe axial (XZ) PSF is shown. At different illumination angles α1, α2, α3, the results of both the lateral (XY) PSF (when compared to each other) and the axial (XZ) PSF (when compared to each other) are almost the same. Figure 8c Also superimposed is the calculated axial PSF (PSF) calculated according to the following equation for each illumination angle α1, α2, α3 成像 ):

[0046] PSF 成像 = Slant PSF 照明 ×PSF 检测 (Equation 1)

[0047] This equation is based on the observation that the imaging point spread function (PSF) from an oblique beam illumination 成像 ) can be obtained by transforming the light sheet PSF (tilt PSF 照明 ) and the detector point spread function (PSF 检测 ) are multiplied together to arrive at the result.

[0048] Based on three different lighting angles - lighting angle α1, α2, α3, Figure 8c The retrieved axial PSF in 成像 should present a tilted intensity distribution (elliptical). Since the spatial resolution at the detector 12 due to light field detection is low compared to the diffraction-limited resolution of the objective 18 (eg due to the diameter of the microlenses), the inventors expect the PSF 检测 Confined tilt PSF with thin thickness 照明 Wider spatial range. This means that the tilted PSF 照明 May cause PSF 成像 Significant spatial modulation in the axial direction (by Eq. 1). Therefore, the incorrect PSF 成像 This can lead to inaccurate depth retrieval and poor Richardson-Lucy deconvolution.

[0049] The inventors have determined that it is desirable to use the preferred embodiments of the present invention to 成像 Recovery is performed as described more fully in the following paragraphs.

[0050] According to an embodiment of the present invention, the light field detection device 28 utilizes a collection of light field images of the object space 8 captured when the illumination slice K is illuminated from different positions and / or different directions. Unless otherwise stated, it is assumed herein that each light field image is uniquely associated with an incident position at a common (for all light field images 38) illumination angle α.

[0051] refer to Figure 6, object space 8 is divided into a plurality of 3D imaging units, referred to herein as “voxels 40”. Each voxel 40 represents a 3D position within object space 8. The size of voxels 40 is such that each voxel 40 can be independently measured by light field detection device 28 (i.e., voxels 40 should have a size length that is greater than the resolution capability of light field detection device 28). Figure 6 The voxels 40 in FIG. 4 are shown as cubes, but this is not meant to be limiting—for example, the voxels 40 may alternatively represent spatial coordinates that are spaced sufficiently apart to minimize or eliminate overlap in signal detection between adjacent voxels 40, such as Figure 5 shown.

[0052] For ease of description, the term “voxel 40 ” is used to denote a corresponding location in the “real” object space 8 and a corresponding element of a resulting volumetric image (ie, data structure) of the object space 8 .

[0053] refer to Figure 5 , a schematic diagram is shown showing how voxels 40 of object space 8 are related to light field image 38. In the figure, two separate light sheets K1 and K2 are shown to illuminate object space 8. As shown, the first light sheet K1 is incident from a different position than the second light sheet K2 (but the illumination angle α is the same). The figure shows that some voxels 40a are illuminated by the second light sheet K2, while the remaining voxels 40b are not illuminated.

[0054] According to an embodiment, each illumination slice K i is associated with a specific translation of the light sheet relative to the x-axis. i are spaced a certain distance k along the x-axis. Essentially, in the embodiment shown, Figure 2 In the first step (S102) of the illustrated method 4, the light sheet K is scanned horizontally (i.e., along the x-axis) across the object space 8. The choice of aligning the translation direction of the light sheet with the x-axis is arbitrary, but can simplify calculations. In this arrangement, the other transverse axis (y-axis) is parallel to the plane of the light sheet.

[0055] By using a light transformation matrix (described in more detail below), the intensity of each voxel 40 (i.e., the emission of the sample from the location in object space 8 associated with the voxel 40) can be determined from the detected light field image obtained when the particular voxel 40 is illuminated. Figure 5 For example, the captured light field image associated with the illumination slice K2 is suitable for determining the emission intensity of each illuminated voxel 40a.

[0056] Assume that each voxel 40 has an associated emission intensity I x,y,z In this example, Figure 6The schematic diagram of FIG. 8 shows three illumination slices K1, K2, and K3, each with the same illumination angle α, exciting diagonally arranged voxels 40a, 40b, 40c in different parts of the object space 8. That is, the use of angled illumination slices K allows different depths of the object space 8 to be imaged in each light field image, because the different depths are separated in the transverse plane.

[0057] Therefore, in step S108, the detector 12 detects the illumination of the plurality of illumination slices K projected into the object space 8 ( Figure 2 The sample of step S104 of FIG. 104 is imaged to create a corresponding plurality of light field images (such that each light field image is associated with a unique illumination slice in the illumination slices K). In one embodiment, during steps S102 to S108, the light field image is horizontally (i.e., at a constant angle α) at ​​a constant angle α. Figure 5 and Figure 6 The light sheet is scanned in the direction k. Figure 5 and Figure 6 As shown, the resulting multiple illumination slices K are parallel to each other. In addition, in one embodiment, the illumination slices K correspond to constantly spaced steps of the light sheet along the x-axis (e.g., with a spacing δx). Typically, enough light field images are captured to ensure that each voxel 40 is illuminated by the illumination slices K.

[0058] In one embodiment, the tilt angle α is varied (step S108). Steps S102 to S106 may then be repeated to capture light field images as electronic signals by scanning the object space 8 at multiple illumination angles α and at multiple illumination positions. In this case, each illumination slice K and each light field image may be associated with (e.g., indexed by) both the incident position and the incident angle α of the light sheet.

[0059] In steps S112 to S124, once sufficient light field images have been captured, the computing system 100 may process the light field images to generate a volumetric image. The computing system 100 may include a processor 102 and a data storage device 104 (e.g., volatile memory and / or non-volatile memory, such as one or more hard disk drives, optical drives, dynamic memory, or solid-state memory) that may store light field images collected by the detector 12 and other information related to the system 2 (including system parameters). The detector 12 is in data communication with the computing environment 100, and the processor 102 is configured to execute instructions in the form of software library routines or modules 106 to 114, which are specifically written to perform operations according to embodiments of the present invention and with reference to the present invention. Figure 1 to process the electronic signal.

[0060] Figure 5The physical basis of the computational model in which steps S112 to S124 are embedded is schematically illustrated according to one embodiment. The figure illustrates the relationship between the intensity of light emitted by the sample at a particular voxel 40a location and the detection signal generated at the detector 12. For ease of illustration, four light rays R(1) to R(4) are traced from the location of a particular voxel 40a in object space 8 to coordinates S(1) to S(4) on the detector 12. The light rays R(1) to R(4) are then (effectively) mapped to pixels LF(1) to LF(4) of the light field image (i.e., a 2D array of detector pixels associated with a particular microlens, which is itself associated with a particular voxel 40 in the particular light field image). The resulting light field pixels are analyzed in order to retrieve a measurement of the intensity of light emitted by the sample at the particular voxel 40a location.

[0061] According to this embodiment, the computational model uses a ray transformation matrix analysis where each ray R(N) of a voxel 40 is mapped from the object space 8 to the sensor plane S as shown in the following equation, S x"，y" (N)=[P][L MLA ][P][L tube ][P][L obj ]V x，y，z，x′，y′ (N) (Equation 2)

[0062] where the spatial coordinates of the voxel are (x, y, z), the ray index is defined as N, the ray coordinates along the objective aperture are given as (x′, y′), the ray coordinates on the sensor plane are given as (x", y"), the free space transformation term is [P], and the thin lens transformation terms for the objective, tube lens, and microlens array are defined as [L obj ]、[L tube ] and [L MLA ]. V is determined by using the transverse and axial sampling factors δ xy and δ z The object space is discretely sampled to form S(N). By indexing each unique ray with N, S(N) represents the lateral coordinate of the ray R(N) on the sensor plane 12 (relative to the microlens). The above equation also includes the encoding of angle information in light field imaging.

[0063] although Figure 5 The propagation of the ray R(N) relative to the XZ plane is shown, but in general, the propagation of the ray R(N) in both the XZ plane and the YZ plane is determined; the results can then be combined to represent the 2D divergence angle of a particular ray from the optical axis. By evenly distributing the N rays R(N), the angular information of the signal (e.g., fluorescence) emitted by the voxel 40 can be calculated.

[0064] Taking into account the index of the voxel 40, the focus of the objective lens 18 can be used as the origin of the volume image defined by the group of voxels 40. Preferably, the voxel V x，y，z , for example, by using a transverse sampling factor δxy and an axial sampling factor δz (respectively). That is, the "size" of each voxel 40 relative to object space 8 is the product δxy×δz.

[0065] Next, we determine the expected light field image. The following equation describes how to map the sensor plane S to the light field pixel image LF,

[0066] LF l，m (N) = S x，y (N) / U (Equation 3)

[0067] The pixel size of the detector 12 is U, and the pixel coordinates of the light field image are (l, m). For ease of explanation, Figure 4 Four rays R(1) to R(4) are shown, which are focused onto the detector 12 at S(1) to S(4) by specific microlenses in the microlens array 32 and recorded as LF(1) to LF(4). Since the detector 12 samples the image discretely, bilinear interpolation is applied in this step to construct each voxel intensity. The detected intensity I is the sum and average of all rays R(1) to R(4).

[0068] The following equation is used to calculate the expected intensity I of the corresponding voxel 40 (ie, the voxel 40 located at the coordinates (x, y, z) in the object space 8) x，y，z ,

[0069]

[0070] Figure 6 8 shows how the mapping of the illumination slice K at a certain angle α in the object space 8 forms a volume image. In practice, once the intensity I of each voxel 40 is calculated x，y，z , a volumetric image can be constructed by simply assigning each voxel 40 to its position in object space 8. Therefore, although a single light field image is associated with voxels 40 at different depths for different x-axis positions, the result of processing the set of all light field images is a data structure suitable for 3D volumetric imaging.

[0071] The redistribution process is given by the following equation,

[0072] Z z =P k(z) (Equation 5)

[0073] Where P k(z) is the set of light field images, and Z zis a slice at depth z. Both the OP image and the Z slice have pixel coordinates i,(z) as described below, which are used to locate the OP image with index k from which pixel i,j is to be extracted.

[0074] Optionally, the model can take into account the thickness of the tilted plane illumination t during processing. This involves summing the nearby voxels of a given voxel along the z-axis (and itself) to represent the combined intensity, as described in the following equation, where the resulting voxel intensity is I', and D = t / δz is the "depth factor" (i.e., D represents the approximate number of voxels covered by the thickness of the tilted plane). By taking t into account, fluorescence signal collection is maximized while eliminating out-of-focus light.

[0075]

[0076] That is, the total intensity I' is based on the in-focus intensity I (i.e., V x,y,z ) and the in-focus intensity of each neighboring voxel in the z direction (within distance (±D / 2)).

[0077] refer to Fig.12a , the sample 60 itself can introduce deviations between the illumination apparent point 62 within the sample 60 and the actual illumination point 64 due to the refractive index effects of the sample 60. For example, the refractive index effects of the sample 60 can cause changes in the optical path, which can cause an axial (z-direction) focal shift. This can cause the relative position of the modeled voxel 40 to deviate from the corresponding relative position within the sample 60, which can cause the modeled voxel 60 to exhibit aberrations relative to the sample. This is shown by 66 (illumination position) and 68 (apparent point of emission light from the sample 60).

[0078] Therefore, in one embodiment, Equation 8 is utilized in order to identify, quantify, and evaluate axial focal shift. This embodiment is based on the known position of a particular illumination slice K, from which the expected position 64 of the excited voxel can be identified; axial focal shift is expected to cause the apparent (i.e., modeled) position 62 of the excited voxel 40 to shift relative to the expected position 64 in the axial direction (z direction). Therefore, in this embodiment, Equation 8 is utilized as a search function over a series of voxels 40 relative to the expected position of the voxel 40 in the axial direction. The search function seeks the maximum I′ relative to z ,y,z The apparent maximum intensity can then be mapped to the excited voxel 40; thereby taking the focal shift into account.

[0079] Then, by adjusting the parameters in Equation 2 according to this process, and thus using the correct coordinates, axial redistribution via depth mapping can generate a light field image that advantageously reduces or eliminates aberrations for a given illumination slice K.

[0080] This embodiment can optionally be extended by using multi-view excitation, e.g. Figure 12b That is, by illuminating the same voxel 40 with an illumination slice K rotated about the z-axis (see K1, K2, K3, K4, which represent rotations in 90° increments, although other angular steps may also be used). This may advantageously allow iterative determination of the axial focal shift within the full aperture.

[0081] Example Implementation Using MATLAB

[0082] Some MATLAB TM Modules may be suitable for implementing the steps of method 4. Specifically, the inventors utilized the following MATLAB modules: EXE_rectify 106; EXE_parameter 108; EXE_model 110; EXE_translate; EXE_mapping 112; and EXE_reassign 114, and their functionality is set forth in Table 1 below. These modules may be suitable for performing various aspects of steps S112 to S126. Although the modules 106 to 114 having the functionality set forth in the table below are prepared as MATLAB modules, this is not meant to be limiting, as the functionality of one or more of the modules may be implemented via different programming languages ​​or modules, as will be known to those skilled in the art.

[0083] Table 1: MATLAB package modules

[0084]

[0085] In module EXE_Correction 106, light field image processing is performed to ensure that lateral translations are correctly accounted for in subsequent modules EXE_Mapping 114 and EXE_Redistribution 116. To do this, the light field image must be rotated so that the illumination angle α (i.e. the scanning direction) matches the x-axis of the model used (see steps S114 to S116). Therefore, depth information can be retrieved from the light field image at an arbitrary tilt angle α by this method 4.

[0086] Module EXE_Correction 106 involves light field image pre-processing in step S112, which may include a light field image correction or calibration step. In one embodiment, the correction includes capturing an image of a sample with uniform fluorescence emission, such as a fluorescence microscope slide. The light field image of the uniform fluorescence emission is used to generate a microlens array correction setting according to the specific light field detection optics used. The inventors have found that adjusting the tilt angle α has no significant effect on the correction setting.

[0087] Module EXE_Correction 106 may involve another pre-processing including cropping the original light field image to scale to have an integer number of N pixels, since only light field images with a field of view (FOV) smaller than the modeling FOV are acceptable.

[0088] The module EXE_Parameters 108 summarizes the parameters used when determining the volume image, which parameters are calculated in the subsequent modules EXE_Model 110 , EXE_Map 114 and EXE_Redistribute 116 .

[0089] The module EXE_model 110 performs modeling using the parameters defined in the module EXE_parameters 108. To help perform calculations faster, precalculations can be performed based on previously known parameters from a known system by extracting data sets from precalculated models. The parameters used for the precalculated models can be summarized in the module EXE_parameters 108. The following table sets forth the parameters used in the module EXE_model 110.

[0090] Table 2: Parameters involved in the EXE_ model

[0091] N Light density N fov_xy Field of view of the object space to be modeled (μm) fov_z The depth of the object space to be modeled res_xy Horizontal sampling (μm) res_z Axial sampling δxy (μm) obj_f Focal length of objective lens fobj (μm) obj_na Numerical aperture of the objective lens tube_f Focal length of tube lens ftube(μm) mla_pitch Pitch size of MLA lenslets (μm) mla_f Focal length of MLA fMLA (μm) pixel_size Pixel size of imaging sensor U (μm)

[0092] As the illumination slices K are laterally translated k relative to each other, a series of light field images are recorded, each of which represents the illumination at a unique k assigned to the scan index. In order to determine the position k of each illumination slice K in step S118, a calibration protocol is performed to determine the lateral translation k as a function of the scan index in module EXE_Translation 112. This calibration involves ensuring that the galvanometer 20 is conjugated to the back focal plane of the objective so that tilt-invariant scanning can be achieved, and that the microlens array 32 is conjugated to the front focal plane of the objective 18. Once the correct conjugation is achieved, the lateral translation can be determined by imaging the sample with uniform fluorescence emission and plotting the processed light field images to determine the illumination profiles as well as their spatial positions, and this advantageously ensures the accuracy of k.

[0093] The emission intensity of each voxel 40 when excited by the corresponding illumination slice K is then mathematically redistributed back to the 3D position using different Z slices (see step S124). Figure 7 In the example shown, Z slices Z1, Z2, Z3, Z4 are mapped to the depth of illumination slices K1, K2, and K3. Depth mapping is implemented by EXE_MAPPING 114 and can be done on a single light field image, a series of indexed light field images, or a light field image streamed in real time from the imaging sensor / detector 12.

[0094] In module EXE_MAP 114, the light field image and its lateral translation determined in EXE_TRANSLATION 112 are received to generate a set of images as a set of voxels according to the position k (step S120). This generates a set of oblique planar images P1, P2 and P3 for a given illumination angle α. Each planar image P1, P2 and P3 can then be rearranged into voxels 40 in the form of pixel columns (voxels) associated with one or more of the different depths Z1, Z2, Z3, Z4 (see Figure 6 and Figure 7 ), which is called so-called "depth mapping".

[0095] These voxel columns 40 are then reallocated to different images or z-stacks, each associated with a different depth, in step S124 of EXE_Reassign 116. Specifically, pixel columns (voxels) associated with depth Z3 in image P1 are allocated to the Z3 stack, pixel columns associated with depth Z2 in image P1 are allocated to the Z2 stack, and pixel columns associated with depth Z1 in image P1 are allocated to the Z1 stack (see Figure 7 ). Similarly, the pixel columns or voxels 44, 46 in P2 and P3 are reallocated to the corresponding z-stacks Z1, Z2, Z3, Z4 (see the dashed and solid lines). Computationally speaking, the reallocation of pixels only accounts for 5% of the required calculations, so this step can be calculated quickly. The different z-stacks are then graphically represented by a graphics module in step S126 to provide the required volume image.

[0096] Thus, more accurate information about the segment depths of object space 8 is obtained by the system 2 and method 4 of this preferred embodiment of the present invention than was previously possible by standard depth retrieval techniques.

[0097] In addition to obtaining improved depth information from the detected light field image, in the method and system of the present invention, depth information can be advantageously retrieved regardless of the light sheet angle α. Specifically, S x″,y″ The dataset does not depend on the angle α of the oblique plane illumination, because it models all voxels 40 sampled in object space 8, while the mapping of the illumination slice K only involves a subset of the voxels 40. This means that it is possible to x″,y″ Mapping the illumination for any angle α, which can be pre-calculated using system parameters that have been predetermined (eg, using module EXE_Parameters 108).

[0098] The inventors have found that the performance (i.e., computational efficiency) of the depth recognition step depends largely on S x”,y”The size of the data set and the tilt angle α of the illumination slice K. Specifically, a large angle α results in more voxels 40 on the x-axis being interrogated by the equation below, which describes the profile of the oblique plane illumination slice, thus increasing the time for depth mapping. In addition, the computational efficiency is related to the lateral sampling δxy, but the inventors have found that the parameters of the objective lens, the tube lens, and the microlens array have no significant effect on the computational efficiency.

[0099] Importantly, by developing the disclosed systems and methods, the inventors have achieved recovery of depth information sufficient to generate fast, high-resolution volumetric images from a single-objective light sheet microscope with a conventional microscope imaging setup, preferably in real time. Specifically, depth information can be recovered without the need for an additional remote optical imaging unit typically required for conventional light sheet microscopes. As described in the "Experimental Results: Validation" section below, optical sectioning is also advantageously improved by a factor of 2, and volumetric images can be achieved in about 0.5 seconds on a standard CPU without the need for multi-core parallel processing.

[0100] The inventors believe that this method 4 and system 2 are applicable not only to conventional laser scanning microscopy, but also to many applications involving adaptive optics and many uses involving structured beams (Airy beams).

[0101] The embodiments herein describe systems 2 and methods 4 that provide 3D imaging of an object space, more specifically, a sample located within the object space. For example, the sample may be a biological sample that, when illuminated by light of certain wavelengths, may emit certain other wavelengths (e.g., by fluorescence).

[0102] The object space 8 is illuminated by a light sheet (incident light defining a plane) and imaged by a light field detection device 28 having an optical axis such that the light sheet is directed into the object space at an angle α relative to the optical axis. The use of an inclined angle α and a planar light sheet enables the described embodiment to effectively "see behind" the front of the sample (i.e., those portions facing the light field detection device 28), thereby enabling detection of light emissions at different depths in the sample.

[0103] The object space 8 may be associated with a set of voxels 40 , and each voxel 40 may be illuminated by imaging the object space with light sheets incident from different positions, so that the set of images obtained thereby collectively represents each voxel 40 of the object space 8 .

[0104] Therefore, the embodiments described herein employ light field technology so as to be able to accurately measure the light intensity emitted by the voxels 40 at all depths in the object space 8, and therefore also to be able to accurately measure the light intensity emitted by the voxels 40 at all depths of the sample. In the embodiments described herein, a microlens array 32 is provided at the imaging plane of the light field detection device 28, and a detector is arranged to detect the image obtained due to the microlens array 32. The light field image thus obtained is suitable for reconstructing depth information, so that the intensity emitted by each voxel 40 can be accurately determined regardless of the voxel depth.

[0105] The embodiments described herein can advantageously allow 3D imaging of a sample by using an oblique plane technique while enabling accurate reconstruction at different sample depths without the need for a remote imaging unit which may include two (or more) supplemental objectives (secondary and tertiary objectives) in addition to the main objective.

[0106] Experimental results: verification

[0107] As a demonstration and verification of the more accurate volume imaging achieved by the method 4 and the system 2 according to the preferred embodiment of the present invention, an example of imaging a 1 μm fluorescent microsphere 50 is shown, which is lower than Figure 4 The resolution limit of the optical system used is shown. Figure 9a , which shows imaging via standard light-field depth retrieval techniques, with significant out-of-focus blur in both the XZ and YZ planes. Figure 9b The columns show imaging performed by the method and system as described above and illustrate a reduction of the out-of-focus signal in the axial direction by a factor of 2 and show the effective PSF 成像， Its XZ and YZ (see Figure 9b ) are close to the theoretical limit of 1.35 μm for the system, and their profiles are almost the same as the ideal PSF 成像 The gray ellipse shows the ideal PSF. 成像 .

[0108] To further validate the present method and system on densely packed samples, custom laser-written rigid fluorescent microstructures 52, 54, 56 were imaged, which consist of the letters "A" 52 and "U" 54, which are vertically stacked on a glass cover slip and supported by a support structure 56, as shown in Fig.10a In the example shown, the solid line represents the scanned illumination slice K, while the dashed line represents the YZ slice through the center of the microstructures 52 , 54 , 56 . Fig.10bXY slices containing the letters "A" and "U" at z = -2 μm and z = 2 μm, respectively, are shown, with the left column being imaged by the standard optical depth retrieval technique and the right column being the imaging technique of the present method and system. This provides a small interval of 4 μm along the z-axis between "A" and "U" to test axial slices.

[0109] from Fig.10b As can be seen, both the standard light field depth retrieval technique and the imaging technique of the present method and system appear to accurately display the letter at its specified axial position, excluding any out-of-focus intensity due to another letter located 4 μm away in the z direction. However, upon careful inspection of the image through the center of the microstructure ( Fig.10a The YZ slice of Fig.10b ), we observe that the standard light field depth retrieval technique ( Fig.10b ) only eliminates half of the out-of-focus cone, while the imaging method of the present invention ( Fig.10b Item 62 in the right column of shows that there is no out-of-focus cone. Fig.11a The normalized axial intensity across the letter “A” is plotted ( Fig.11a ), and quantified the out-of-focus regions, where the top image was generated by a standard light field depth retrieval technique 58, and the bottom image 60 was generated by the method of the present invention. The standard light field depth retrieval technique resulted in out-of-focus regions extending beyond 8 μm (FWHM) in depth (see item 60). In contrast, imaging using the method of the present invention achieved accurate mapping to z = -2 μm, with an axial profile (FWHM) of 1.9 μm, resulting in a clear axial fluorescence signal for the letter "A" (see Fig.11b 62). Results of the present imaging method and system on densely packed fluorescent microstructures show significant improvements in depth sectioning compared to standard light-field depth retrieval techniques.

[0110] Thus, the above-described system and method for generating a volumetric image advantageously provides more accurate and greater depth information and sufficiently fast computation to generate a volumetric image in real time in a standard laser scanning microscope without the need for a remote optical imaging unit.

[0111] Further modifications may be made without departing from the spirit and scope of the present application.

[0112] References

[0113] [1] Bouchard, Matthew B. et al., “Swept confocally-aligned planar excitation (SCAPE) microscopy for high-speed volumetric imaging of behaving organisms,” Nature photonics, 9.2 (2015): 113–119.

[0114] https: / / doi.org / 10.1038 / nphoton.2014.323 .

[0115] [2] Kumar, Manish et al., “Integrated one-and two-photon scanned oblique plane illumination (SOPi) microscopy for rapid volumetric imaging,” Optics Express, 26.10 (2018): 13027-13041.

[0116] https: / / doi.org / 10.1364 / OE.26.013027 .

[0117] [3] Madaan, Sara et al., “Single-objective selective-volume illumination microscopy enables high-contrast light-field imaging,” Optics Letters, 46.12 (2021): 2860–2863.

[0118] https: / / doi.org / 10.1364 / OL.413849 .

[0119] [4] Yang, Bin et al., “DaXi-high-resolution, large imaging volume and multi-view single-objective light-sheet microscopy”, Nature methods, 19.4 (2022): 461-469.

[0120] https: / / doi.org / 10.1038 / s41592-022-01417-2 。

Claims

1. A microscopy method for volume imaging of a sample located in an object space, comprising: obtaining, using a light field detection device, a set of light field images, each light field image being associated with an illumination slice corresponding to a light sheet projected into the object space from a particular illumination position and at an illumination angle that is non-parallel to an optical axis of the light field detection device, such that each voxel in a set of voxels associated with the object space is illuminated by at least one illumination slice, wherein each voxel is defined by a position in the object space; and for each voxel, determining an intensity of light emitted from an object space associated with the voxel using in part the angular information captured by the light field image, Such that the determined intensity for each voxel in the set of voxels defines a volumetric image of emission light intensity within the object space.

2. The method according to claim 1, further comprising: scanning a light sheet at a plurality of locations to illuminate the object space such that each location is associated with an illumination slice corresponding to the light sheet; and The object space is imaged when illuminated by each illumination slice to generate the set of voxels. 3 . The method according to claim 1 , wherein each voxel is associated with a depth in the object space in a direction transverse to the optical axis.

4. The method according to any one of claims 1 to 3, wherein the light field detection device comprises a microlens array for generating the light field image.

5. The method of claim 4, comprising positioning the microlens array in an image plane.

6. The method of claim 4 or 5, wherein each microlens of the array has an imaging side NA that is smaller than the NA of the microlens array.

7. A method according to any one of claims 1 to 6, wherein the scanning direction is a direction transverse to an angle of the light plane.

8. The method of claim 7, wherein in generating the set of light field images, the illumination position is stepped with a constant step size.

9. The method of any one of claims 1 to 8, wherein the illumination angle is adjustable such that each light field image is associated with a unique combination of illumination position and illumination angle.

10. The method of any one of claims 1 to 9, wherein determining the intensity of light emitted from the object space associated with the voxel comprises summing neighboring intensities within a predefined axial distance of the voxel.

11. A method according to any one of claims 1 to 10, comprising correcting for an apparent axial shift in the apparent position of the voxels.

12. A microscope system for volumetric imaging of an object space of a sample, the system comprising a light field detection device in communication with a computer having a processor and a memory: The light field detection device includes an optical element, which is configured as follows: The light plane is scanned at a plurality of positions, thereby generating a plurality of illumination slices, each illumination slice being associated with a unique position, wherein The light plane is formed by a laser light source at an illumination angle relative to an optical axis for illuminating the object space such that each voxel in a set of voxels associated with the object space is illuminated by at least one illumination slice, wherein each voxel is defined by a position in the object space, and imaging the object space for each illumination slice, thereby generating a corresponding light field image for each illumination slice; and The processor is adapted to execute a plurality of modules stored in the memory, each module being capable of executing a set of instructions, wherein the modules are configured to: for each voxel, determining an intensity of light emitted from the object space associated with the voxel using in part the angular information captured by the light field image, Such that the determined intensity for each voxel in the set of voxels defines a volumetric image of emission light intensity within the object space.

13. The system of claim 12, wherein each voxel is associated with a depth in the object space transverse to the optical axis.

14. The system of claim 12 or 13, wherein the light field detection device comprises a microlens array for generating the light field image.

15. The system of claim 14, wherein the microlens array is located at an image plane.

16. The system of claim 14 or 15, wherein each microlens of the array has an imaging side NA that is smaller than the NA of the microlens array.

17. A system according to any one of claims 12 to 16, wherein: The scan direction is the direction of the angle transverse to the light plane.

18. The system of claim 17, wherein in generating the set of light field images, illumination positions are stepped with a constant step size.

19. The system of any one of claims 12 to 18, wherein the illumination angle is adjustable such that each light field image is associated with a unique combination of illumination position and illumination angle.

20. The system of any one of claims 12 to 19, wherein determining the intensity of light emitted from the object space associated with the voxel comprises summing neighboring intensities within a predefined axial distance of the voxel.

21. The system of any one of claims 12 to 20, wherein the module is further configured to correct for an apparent axial shift in the apparent position of the voxels.

22. A computer-readable storage medium for generating a volumetric image, storing instructions that, when executed by one or more processors of a computer, cause the computer to execute a plurality of modules stored in a memory, each module being capable of executing a set of instructions, and wherein: The modules include: obtaining, using a light field detection device, a set of light field images, each light field image being associated with an illumination slice corresponding to a light sheet projected into the object space from a particular illumination position and at an illumination angle that is non-parallel to an optical axis of the light field detection device, such that each voxel in a set of voxels associated with the object space is illuminated by at least one illumination slice, wherein each voxel is defined by a position in the object space; and for each voxel, determining an intensity of light emitted from an object space associated with the voxel using in part the angular information captured by the light field image, Such that the determined intensity for each voxel in the set of voxels defines a volumetric image of emission light intensity within the object space.

Images