Volumetric Imaging

Abstract

An apparatus (2) for volumetric imaging is provided, the apparatus (2) comprising an illumination assembly (8) arranged to sequentially illuminate a plurality of planes (22) of a sample area (4), each plane (22) extending across a plurality of depths (22a, 22b, 22c, 22d) of the sample area (4) and including a plurality of sections of pixels (12), an image sensor (6) arranged to receive light from the sample area (4) and to sequentially sense each section of the pixels (12), and an image sensor (6) arranged to sequentially sense each section of the pixels (12) and to sequentially sense the light received from each plane (22) of the sample area (4) relative to a different portion of said section of the pixels. The light receiving assembly arranged to direct light to each of the sections (12) of the pixel includes a multi-faceted optical assembly (10) arranged to receive light from a plurality of depths (22a, 22b, 22c, 22d) in the sample region (4) and is configured to simultaneously direct light from each section (12) of each pixel from the plurality of depths (22a, 22b, 22c, 22d) in each plane (22) to a different subsection (12a, 12b, 12c, 12d) for each section (12) of the pixel. The illumination rate is equal to the detection rate, such that each section of the pixel (12) is arranged to detect light from the plurality of depths (22a, 22b, 22c, 22d) in its respective plane (22) when that plane is illuminated by the illumination assembly (8).

Description

[Technical field]

[0001] The present invention relates to an apparatus for volume imaging, for example for volume microscopy of biological samples. [Background technology]

[0002] Conventional microscopy involves two-dimensional imaging of a sample. In transmission microscopy, the sample is typically sectioned thinly, for example with a microtome. However, in many fields, such as biomedical research, it is advantageous to be able to perform volumetric (three-dimensional) imaging of thicker samples. This is typically achieved by imaging several two-dimensional image "slices" of different thin volumes of the sample (e.g., at different depths of the sample). The process of generating 2D images of a thick sample (e.g., different focal planes of the sample) that are substantially free of out-of-focus light is commonly referred to as optical sectioning.

[0003] Traditional approaches to optical sectioning include confocal imaging, which uses raster scanning, where a single point of the sample is illuminated and imaged at a time, using a physical pinhole to remove the out-of-focus light at each point, but can be slow. Another approach is light sheet imaging, where a virtual slice of the sample area is selectively illuminated using a light sheet that extends orthogonally to the imaging axis of the detection objective. However, light sheet imaging also requires complex equipment that can be slow and may not be suitable for many applications. It is possible to record light from several focal planes simultaneously with light field imaging systems, but these do not allow good optical sectioning.

[0004] The paper by Tsang et al., "High-Speed, Multiple-Line Scanning Confocal Microscopy with Axially Distributed Slits," Biomedical Optics Express Vol. 12, pp. 1339-1350 (2021), proposes a method of volumetric imaging that involves sweeping light through a sample to induce fluorescence and then using three reflective slits to spatially separate the fluorescent light from three different depths in the sample. The separated light is directed to three different regions of an image sensor. As the light is swept through the sample, each of these regions progressively captures a respective two-dimensional image of the sample at a specific respective depth. However, using reflective slits to perform optical cutting can introduce aberrations that must be corrected when using bulky correction optics. This can be difficult to set up, and the physical space taken up by the slits and correction optics can limit the number of depths that can be practically imaged simultaneously.

[0005] Therefore, improved approaches are desirable. Summary of the Invention

[0006] According to a first aspect, 1. An apparatus for volumetric imaging, comprising: an illumination assembly arranged to direct light to sequentially illuminate multiple planes of the sample region at an illumination rate, each plane extending across multiple depths of the sample region; a light receiving assembly including a plurality of sections of pixels and arranged to sequentially sense each of the sections of pixels at a detection rate; a light receiving assembly arranged to receive light from the sample area and arranged to direct light received from the planar surface of each of the sample areas to a different one of the sections of the pixel; the light receiving assembly is arranged to receive light from the plurality of depths of the sample area, and includes a multi-faceted optical assembly arranged to simultaneously direct light from each of the plurality of depths in each of the planes to different respective subsections of the section for each of the sections of each of the pixels; An apparatus is provided, wherein the illumination rate is equal to the detection rate, and the sections of each of the pixels are positioned to detect light from the multiple depths in each of the planes when the planes are illuminated by the illumination assembly.

[0007] According to a second aspect, 1. A method of volumetric imaging, comprising: directing the light to sequentially illuminate a plurality of planes of the sample region at an illumination rate, each plane extending across a plurality of depths of the sample region; an image sensor including a plurality of sections of pixels; directing light emanating from the sample area to the image sensor; the image sensor includes a plurality of sections of pixels, and light received from each of the planes of the sample area is directed to a different one of the sections of the pixels, and for each of the sections of the sections of the pixels, light is simultaneously directed to a different one of the subsections of the sections from each of the plurality of depths in each of the planes; A method is provided in which the image sensor detects each of the sections sequentially at a detection rate, a transmission rate equal to the illumination rate, and the sections of each of the pixels detect light from the multiple depths in each of the planes when the planes are illuminated.

[0008] It will thus be appreciated that the present device allows for convenient, fast and high quality capture of volumetric information of a sample area. Light from each depth in each plane is sensed by a separate subsection of the pixel section, allowing three-dimensional information from the sample area to be captured using only one exposure of the entire image sensor (e.g., requiring only one readout process). This may allow for faster imaging times than previous approaches that use multiple exposures of the image sensor (e.g., each exposure corresponds to a different optical section) to build a three-dimensional image. Faster imaging is particularly useful when imaging non-static samples such as cellular organs, individual cells of a cell colony, organoids, or whole tissues where important phenomena may occur on short time scales that require high imaging speeds.

[0009] Furthermore, since the illumination rate is equal to the detection rate, only a limited set of pixels (e.g., a single row or column of pixels) can be arranged to actively detect light when each plane is illuminated, allowing good optical disconnection between the multiple planes. Each section of pixels detects light (e.g., accumulates charge) when the corresponding plane is illuminated. However, the device is preferably configured such that each section does not detect light from any other plane of the multiple planes other than its respective plane. In this way, the output of the image sensor is not affected by light received when other planes of the multiple planes are illuminated for each section. The selective sensitivity of the image sensor can be understood, at least in embodiments, as acting equivalently to a "pinhole."

[0010] Using the selective sensitivity of the image sensor to achieve optical resolution in this manner can reduce constraints on other optical components of the device, such as the illumination assembly and / or the multi-faceted optical assembly. For example, the multi-faceted optical assembly may not be required to provide optical fragmentation between multiple planes (i.e., if the entire sample area is illuminated simultaneously, it may be possible to receive light from the entire sample area simultaneously across the image sensor), which may allow for a smaller and / or less expensive type of multi-faceted optical assembly to be used compared to prior art approaches. Such an embodiment may result in light from near the illuminated plane spilling onto pixels of the image sensor adjacent to the corresponding portion, but this is acceptable since these pixels are sensed at different times. A multi-faceted optical assembly that does not inherently need to provide for optical resolution may be physically smaller than other options and / or may not require correction optics to correct aberrations, allowing for an increased number of depths imaged simultaneously and improving the quality of volumetric imaging. In some embodiments, the multi-faceted optical assembly is positioned to simultaneously direct light from each of at least four depths, at least eight depths, at least twelve depths, or even twenty depths or more, to different respective subsections of the image sensor.

[0011] The sample area may contain a sample. The sample may completely or partially occupy the sample area, or may extend beyond the sample area. The device may include a surface (e.g., of a slide or bath) for holding the sample. The surface may define a boundary of the sample area. In some embodiments, the sample area may contain a biological sample. The sample may include a fluorescent marker.

[0012] The image sensor is preferably an electronic image sensor, such as a CMOS or CCD sensor. It is preferably a two-dimensional (e.g. rectangular) sensor. The image sensor may be arranged to selectively sense each section of pixels through the use of a physical shutter mechanism and / or an electronic shutter circuit. It may include a physical or electronic rolling shutter. For example, the image sensor may include a physical shutter arranged to expose only one section at a time to incident light from the sample area. However, preferably, the image sensor includes an electronic shutter arranged to selectively sense the pixels of each section of the image sensor in successive electronic shutter periods. These periods preferably do not overlap in time. For example, in the case of a CMOS or CCD sensor, the image sensor may be arranged to allow sections of pixels to accumulate charge only during their respective electronic shutter periods (although it is also possible to read out one or all sections in one readout operation).

[0013] The multi-faceted optical assembly may include a multi-faceted prism or a multi-faceted grating. Using a multi-faceted prism, up to eight depths can be reliably imaged simultaneously, but more may be possible. The number of depths that can be imaged simultaneously with a multi-faceted grating may only be limited by the number of pixels in the image sensor.

[0014] The device can be arranged to volumetrically image the sample area repeatedly over time, i.e. to generate image data representing a time series of the volume of the sample area. This may enable the device to capture the motion of the target sample, e.g., dynamic biological processes. The device can be arranged to capture the entire sample area in a single exposure of the image sensor, such that the repetition rate is only limited by the readout speed (frame rate) of the image sensor. In some embodiments, the device is operable to record more than 10 volumes per second, more than 20 volumes per second, more than 50 volumes per second, more than 100 volumes per second, or even up to 1000 or 1500 or more times per second. The device is therefore particularly useful for capturing biological processes that occur in three dimensions and on relatively short time scales.

[0015] The receiving assembly may comprise an objective lens assembly arranged to pass light emanating from the sample area (e.g. reflected or fluorescent light) directly or indirectly to the multi-faceted optical assembly. The objective lens assembly preferably includes at least one objective lens. The illumination assembly may comprise a separate objective lens. However, in some embodiments, the objective lens assembly also forms part of the illumination assembly and is arranged to pass (e.g. focus or direct) light from the illumination assembly to the sample area. Using the same objective lens assembly for illumination and imaging can avoid the need to closely space separate illumination and imaging objective lenses, limiting the physical size of both. Thus, the objective lens assembly may have a higher numerical aperture (NA) than would be possible with separate objective lenses. This may improve both the imaging resolution (scaling linearly with NA) and the photon collection capability (scaling with NA2). Also, using the same objective lens for illumination and imaging allows the device to be used with a wider range of sample mounts (e.g., compared to traditional light sheet microscopy which uses two objective lenses), improving the diversity of microscopy applications in which the device can be used.

[0016] The multiple planes may be parallel planes. They may each be parallel to the imaging axis of the objective. They may be evenly or unevenly spaced across the sample area. The spacing of the planes may be selected based on the desired imaging resolution and / or the expected scale of structures of interest in the sample.

[0017] In one set of embodiments, at least one of the plurality of planes is parallel to an imaging axis of the objective lens assembly.

[0018] In a set of embodiments, at least one of the planes is tilted with respect to the imaging axis of the objective lens assembly. In other words, one or more of the planes may be tilted with respect to the imaging axis of the objective lens. For example, at least one plane may be tilted at an angle of at least 1°, at least 5°, at least 10°, or at least 20° or more. In a set of embodiments, at least one plane is tilted at 30° (or about 30°) with respect to the imaging axis of the objective lens. In some embodiments, at least one plane may be tilted at a larger angle, for example, 45° or more. The angle at which the plane is tilted can be selected based on the maximum allowable thickness of the light (in a direction perpendicular to the illumination plane). The angle can be selected based on the target illumination depth of the sample region (e.g., the angle can be selected so that the light extends to the target depth of the sample region or through the entire sample region). The angle may be selected to be the highest angle with respect to the imaging axis that the illumination light extends to the target depth of the sample region without exceeding a given thickness.

[0019] Each of the multiple planes may be tilted with respect to an imaging axis of the objective lens. In such embodiments, the multiple planes may be parallel (i.e., each plane is tilted at the same angle with respect to the imaging axis).

[0020] The use of one or more tilted planes may allow for improved optical depth sectioning of the sample and improved axial resolution performance (i.e., reducing the minimum dimension of resolvable features in the axial direction). With tilted planes, light from different depths of the sample spreads along a direction perpendicular to the imaging axis (i.e., there is less overlap between light emanating from different depths of the sample). Since each section of the pixel detects light from its respective (tilted) plane, light from different depths in a given axial plane of the sample area is detected separately by the sensor. This improves the axial optical sectioning and improves the achievable axial resolution. When illuminating each of the multiple planes, the illumination assembly may direct light to a volume around the plane, which may be substantially cubic in shape. In embodiments where one or more planes are tilted with respect to the imaging axis, the volume around each tilted plane may be approximately rhombus-shaped. However, each such volume is preferably thin (e.g., compared to the overall thickness of the sample) in a direction perpendicular to the plane.

[0021] A multi-faceted optical assembly for use with an axial plane may be adapted for use with an inclined plane by adjusting individual components of the multi-faceted optical assembly to direct light to a required subsection of the image sensor. Alternatively, the entire multi-faceted optical assembly configured for use with an axial illumination plane may be rotated to address the inclined plane. For example, a multi-faceted optical assembly configured to direct light from multiple depths in the axial plane to a single row or column of the image sensor may be rotated to direct light from multiple depths in the inclined plane to the same single row or column of the image sensor.

[0022] The light sensed by each section (i.e. for each plane) may comprise or consist of a two-dimensional projection in a direction orthogonal to the respective plane of the light generated in the respective volume that includes the plane in the sample area. These volumes preferably do not overlap. The thickness of the volume in the direction orthogonal to the plane (which may depend on the spacing of the illuminated planes) may be selected based on the desired imaging resolution and / or the expected scale of the structures of interest in the sample. The thickness may depend at least in part on other aspects such as the width of the pixels of the image sensor and / or the characteristics of the illumination or light receiving assembly such as the wavelength of the light and / or the numerical aperture and / or the magnification. The illumination assembly may be arranged to fill each of these volumes with light at each instant in time and / or to sweep a beam or sheet (which may be narrower than the volume) across or through the volume. Fluorescence may result in light emanating from a point in the volume for some time even after the illumination of the point has ceased.

[0023] Each subsection of the image sensor can receive light from a respective range of depths around each of the multiple depths, rather than just from a single depth of the multiple depths.

[0024] The illumination assembly (e.g., in cooperation with the objective lens assembly) may be arranged to generate a light sheet that extends parallel (which may include substantially parallel) to an imaging axis of the objective lens assembly (i.e., axially). In such an embodiment, the multiple planes may include substantially axial planes (i.e., planes in which the normal to each plane is orthogonal to the imaging axis).

[0025] The illumination assembly may be arranged (e.g., in cooperation with the objective lens assembly) to generate a light sheet that is tilted (i.e., in an oblique direction) with respect to an imaging axis of the objective lens assembly, in such an embodiment, the multiple planes may include substantially tilted planes (i.e., where a normal to each plane is tilted with respect to the imaging axis).

[0026] The illumination assembly may be arranged to sweep or step the light sheet across the sample region (e.g., to illuminate multiple parallel axial planes or multiple tilted planes). In some embodiments, continuous sweeping may be preferred, as this can enable particularly fast illumination, which can support high frame rates.

[0027] The illumination assembly may comprise a light source (e.g., an LED or laser) or may alternatively receive light from a separate light source. The illumination assembly may include one or more controllable optical components for selectively illuminating different planes of the sample region. For example, the illumination assembly may comprise physically controllable components, such as steerable mirrors, and / or electronically controllable components, such as spatial light modulators.

[0028] The illumination assembly may include one or more lenses arranged to correct the light from the light source. For example, the illumination assembly may include one or more lenses (e.g., one or more cylindrical lenses, scan lenses, and / or tube lenses) arranged to create a light sheet from a light beam (e.g., a Gaussian beam from a laser). The illumination assembly may be arranged to generate a tilted light sheet by directing light from the light source through an off-axis portion of a lens (i.e., away from the imaging axis of the lens, e.g., on one side of the lens).

[0029] A section of pixels may include any discrete set of pixels of an image sensor. However, in some embodiments, it may be advantageous for the section to consist of a respective set of pixels arranged consecutively (e.g., as a single line or rectangle) across the image sensor. This may allow for a reduction in the complexity of the image sensor and / or optical components of the device. In some such embodiments, one or more pairs of consecutively sensed sections are adjacent to one another (i.e., not separated by intervening pixels). This may further reduce complexity and facilitate high speed imaging. In a set of preferred embodiments, one or more of the or each of the sections of pixels includes one or more lines (e.g., rows or columns) of pixels, e.g., adjacent lines or sets of lines are sensed sequentially. In such embodiments, a multi-faceted optical assembly (e.g., in cooperation with an objective lens assembly) is arranged to direct light generated in one or more corresponding planes to one or more lines of pixels (i.e., vectorize the planes). In a particularly preferred set of embodiments, each section includes only a respective single line of pixels (e.g. a respective column), which may all be parallel and may all span the image sensor (e.g. the full height of the sensor). In some embodiments, the image sensor is a line scan image sensor, arranged to sense one line of pixels at a time at a sensing rate.

[0030] In some embodiments where the planes are tilted and parallel, each section of the sensor's pixels can be displaced relative to the section used for axial illumination to match the sensor's unique sensing pattern. Each section of pixels can include one or more sets of lines of pixels, with the beginning of each set offset from the previous set. For example, when using multiple tilted and parallel planes and the image sensor is positioned to sequentially sense lines (e.g., rows) of pixels, the multiple sections of pixels can include offset lines (or sets of lines) with an offset corresponding to the tilt angle. Image data output from the sensor can then be more easily processed into a three-dimensional image of the sample area.

[0031] The device may include a processing system, such as a computer, for receiving image data from the image sensor. It may store and / or process the received image data, for example, performing sub-resolution microscopy of the image data. The image data may be processed to generate a three-dimensional image dataset. It may include a display (e.g., a monitor). The device may be configured to render for display and / or display a three-dimensional image of the sample area, although this may be done by a separate device. The device may be configured to essentially capture two-dimensional images of the planes with different sections of the image sensor, which may then be stacked to build a three-dimensional image dataset of the sample area. However, in some embodiments, for example, in embodiments where light from each plane is sensed by a line of pixels (i.e., the planes are "vectorized" into lines), a transformation step may be required to convert the raw pixel data collected by the image sensor into a three-dimensional image dataset of the sample area. This may include rearranging the data from the pixels into a different layout, but may also or alternatively include distorting (e.g., stretching, compressing, rotating) the pixel data in one or more dimensions.

[0032] Each subsection of pixels may be comprised of any separate set of pixels in the respective section. However, the subsections are preferably comprised of respective contiguous sets of pixels (i.e., arranged in an uninterrupted sequence). In some embodiments, subsections corresponding to adjacent depths of the sample region are adjacent in each section of pixels of the image sensor. For example, in embodiments in which the sections of pixels are respective single or multiple lines of pixels, in a first sensor dimension (e.g., along the X-axis), the subsections can include adjacent portions of each line, such that the subsections are arranged in respective stripes in a second, orthogonal sensor dimension (e.g., along the Y-axis).

[0033] The apparatus may be configured to perform fluorescence volumetric microscopy. The illumination assembly may be configured to excite fluorescence in a sample in the sample region when the planar surface is illuminated, and the multi-faceted optical assembly and image sensor may be configured to direct and detect fluorescent light from the illuminated planar surface.

[0034] Since the illumination rate is equal to the detection rate, light from each of the multiple planes is captured by a different corresponding portion of the pixel (i.e., the image sensor detects portions of the pixel at the same rate that the illumination assembly illuminates the multiple planes). Thus, the time it takes to detect all sections of the image sensor may take the same period of time as it would take to illuminate all of the multiple planes across the sample area.

[0035] In some embodiments, the device can be configured to utilize only a portion of the image sensor when imaging the sample area (i.e., when operating in a "crop sensor mode"). This can be to increase the frame rate and / or optimize the shape or size of the pixels or sections of the sample area. For example, a physical sensor with 4000 lines of pixels can be arranged to use all of them to image the sample area in a first mode, but only use 3000 lines to receive light from 3000 illumination planes that span 75% of the width of the sample area in a second mode.

[0036] The device may use a single image sensor to image the entire sample area. However, in embodiments, the device may include a second image sensor also including multiple sections and arranged to sequentially sense each section of pixels at a detection rate. The multi-faceted optical assembly may additionally be arranged to simultaneously send light from each of a second plurality of depths of the sample area to a respective subsection of each section of pixels of the second image sensor. The multi-faceted optical assembly may be arranged to direct light from different depths to the first and second image sensors. In some embodiments, the multi-faceted optical assembly is configured to (preferably only) direct light from a first set of one or more depths to the first image sensor and (preferably only) direct light from a second set of one or more depths to the second image sensor. The first and second sets of depths may be adjacent sets of depths (e.g., a first set including depths A, B, C, D and a second set including depths E, F, G, H (where A is the deepest and H is the shallowest, or vice versa) or they may be interleaved (e.g., a first set including depths A, C, E, G and a second set including depths B, D, F, H), and by splitting the light from the sample region across two image sensors, the optical cutoff between depths and / or imaging speed and / or imaging resolution may be improved. This approach may be extended, in embodiments, to have a device including three or more image sensors.

[0037] Features of any aspect or embodiment described herein may be applied to any other aspect or embodiment described herein, where appropriate. It should be understood that when referring to different examples, these are not necessarily clear and may overlap.

[0038] One or more non-limiting embodiments will now be described, by way of example only, and with reference to the accompanying drawings. [Brief description of the drawings]

[0039] [Figure 1] FIG. 1 is a schematic diagram of an apparatus for volume microscopy according to one embodiment of the present invention. [Diagram 2] FIG. 2 is another schematic diagram of an apparatus for volume microscopy. [Diagram 3] FIG. 3 is a schematic diagram illustrating how light is transferred by the device to an image sensor. [Figure 4] FIG. 4 is a schematic diagram of an apparatus for volume microscopy according to another embodiment of the present invention. [Diagram 5] FIG. 5 is a diagram illustrating a multi-faceted prism for use in an embodiment of the present invention. [Figure 6] FIG. 6 is a schematic diagram of an apparatus for volume microscopy according to another embodiment of the present invention. [Figure 7] FIG. 7 is a schematic diagram comparing axial and oblique illumination techniques. [Figure 8] FIG. 8 is a schematic diagram of a multi-faceted optical assembly. [Figure 9] FIG. 9 shows examples of optical transfer functions for axial and oblique illumination. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0040] 1 and 2, an apparatus 2 for volumetric microscopy of a sample region 4 occupied by a sample 3 includes an image sensor 6, an illumination and receiving assembly 8, and a multifaceted optical assembly 10 (e.g., a multifaceted prism). The sample 3 (here shown resting on a horizontal surface 5 such as a slide) may be a fluorescently labeled biological sample, although the apparatus 2 may be used to image a wide variety of different objects, potentially at scales greater than microscopic resolution.

[0041] The image sensor 6 in this embodiment is a two-dimensional CMOS image sensor consisting of a plurality of pixels arranged in rows and columns. As will be described in more detail below, the image sensor 6 is arranged to sequentially sense each column of pixels (embodied as a respective "section") at a sensing rate. Sensing here may refer to the period during which the pixels are configured to accumulate charge before the pixels are read out.

[0042] Figure 1 shows a first column 12 of pixels that are exposed and sensed, and Figure 2 shows a second column 14 of pixels that is adjacent to the first column 12 and is also exposed and subsequently sensed. The first and second columns 12, 14 (like all other columns) each comprise four subsections 12a, 12b, 12c, 12d, 14a, 14b, 14c, 14d (the division into subsections relates not to physical attributes of the sensor 6 itself, but instead to how image data from the sensor 6 is processed).

[0043] The illumination and receiving assembly 8 comprises a light source 16 (e.g., a laser), a movable mirror 17, a beam splitter 18 and an objective lens assembly 20. It may optionally include further lenses, mirrors, filters, etc. The illumination and receiving assembly 8 is arranged to illuminate the sample area 4 with a narrow light sheet extending in an axial direction parallel to the imaging axis of the objective lens assembly 20. The position of the light sheet at the sample area 4 is changed by moving the movable mirror 17. The movable mirror 17 is arranged in the Fourier plane of the optical path such that rotating the mirror 17 translates the light sheet laterally (i.e., perpendicular to the sheet) across the sample area 4. In other words, by moving the movable mirror 17, the illumination and receiving assembly 8 is configured to smoothly sweep the axial light sheet across (i.e., through) the sample area 4 and illuminate a continuum of axially extending planes through the sample area. In this continuum, multiple distinct axially extending planes of the sample region can be considered to be sequentially illuminated with an illumination rate equal to the detection rate. In an alternative embodiment, the light sheet can be moved in a series of discrete steps with an illumination rate equal to the detection rate.

[0044] The computer system 21 controls the actuator of the moveable mirror 17. It also receives and processes image data from the image sensor 6 to obtain microscopy data of the sample 3. In an embodiment, it can provide super-resolution imaging of the sample 3 using computational nanoscopy processing.

[0045] FIG. 1 shows the illumination and receiving assembly 8 illuminating a first plane 22 of the sample area 4 with a light sheet, and FIG. 2 shows the illumination and receiving assembly 8 at a later point in time illuminating a second plane 24 of the sample area 4 with the light sheet.

[0046] The objective lens assembly 20 is also positioned to capture light generated at the sample region 4 (e.g., scattered light from the illumination surface through the sample 3 or fluorescent light) and direct it to the multi-faceted optical assembly 10 via the beam splitter 18. Thus, the beam splitter 18, the objective lens assembly 20, and the multi-faceted optical assembly 10 together provide a light receiving assembly (which may optionally include further lenses, mirrors, filters, etc.). The multi-faceted optical assembly 10 separates light from four different depth bands in the sample region 4 and directs them to different respective subsections of pixels of the image sensor 6. For example, FIG. 1 shows how light from the deepest depth 22a of the first plane 22 (i.e., furthest from the objective lens assembly 20) is directed to the first subsection 12a, while light from the shallowest depth 22d of the first plane 22 is directed to the fourth subsection 12d. The lower and upper intermediate depths 22b, 22c are directed towards the second subsection 12b and the third subsection 12c, respectively.

[0047] As described above, in use, the illumination and receiving assembly 8 sweeps the light sheet across the sample area 4, thereby sequentially illuminating multiple planes of the sample. The image sensor 6 sequentially senses each of the rows of pixels with equal detection rate, such that light from each of these planes is sensed by a different respective row of pixels. Any light from the sample area 4 that strikes a part of the sensor 6 outside one row that is actively sensed (i.e. striking a row that is not accumulating charge) is not imaged. This results in a precise vertical slice of the sample area 4. Each row of pixels may receive some additional light from a narrow volume around the illuminated plane due to the width and / or motion of the light sheet (and potentially due to non-zero decay times of any fluorescent markers in the sample 3). The collection optics and pixel width may also affect the segmentation. However, these can be configured to limit the thickness of the volume sensed by each row (i.e. vertical slice) to give a precise vertical section.

[0048] 1 and 2 show the illumination and detection of two planes 22, 24 of the sample area 4. At a first time, the illumination and receiving assembly 8 illuminates the first plane 22 as shown in FIG. 1. Light generated in the first plane 22 as a result of the illumination (e.g., by scattering or by fluorescence) is directed via the objective lens assembly 20 and the multifaceted optical assembly 10 to a first row of pixels 12 of the image sensor. The multifaceted optical assembly 10 directs light from different depths in the first plane 22 to different subsections 12a, 12b, 12c, 12d of the first row 12. Thus, the first row of pixels 12 records an image of the first plane 22 of the sample area 4.

[0049] 2, the illumination and receiving assembly 8 illuminates the second plane 24, and light generated at the second plane 24 (e.g., by scattering or by fluorescence) is directed via the objective lens assembly 20 and the multi-faceted optical assembly 10 to a second row of pixels 14 of the image sensor. The multi-faceted optical assembly 10 directs light from different depths in the second plane 24 to different subsections 14a, 14b, 14c, 14d of the second row 14. Thus, the second row of pixels 14 records an image of the second plane 24 of the sample area 4.

[0050] This process continues as additional planes of the sample area 4 are illuminated and the light therefrom detected by additional rows of pixels of the image sensor 6 until the entire sample area 4 has been imaged. In this manner, the apparatus 2 captures a three-dimensional (volumetric) image (i.e., a 3D data set) of the sample area 4 in a single frame of the image sensor 6. The process of sweeping the light sheet across the sample area 4 and detecting the resulting light can be repeated at high frame rates (e.g., up to 1500 times per second or faster, depending on the maximum electronic shutter rate of the image sensor 6) to record the activity of the sample area 4 in three dimensions and at high speed. If the image sensor 6 is capable of reversing the direction of its rolling shutter, the light sheet may be swept back and forth with image data being collected in both directions, otherwise the light sheet may be swept in the same direction for each frame.

[0051] FIG. 3 is a schematic diagram showing how light emanating from a sample area 4 is translated to the pixels of an image sensor 6.

[0052] For each of four distinct depths 22a-22d in the sample area 4, the apparatus 2 translates light emanating from points at and around that depth into a respective one of four horizontal stripes (subsections) of pixels spanning some or all of the width of the image sensor 6. However, at any one time, only one slice through the sample area 4 is illuminated, so that minimal light reaches the sensor 6 outside the vertical stripes corresponding to the light sheet. In particular, light emanating along each horizontal line, coincident with the illumination plane, is received at a respective pixel along a respective subsection 12a-12d of a single column of pixels.

[0053] The multi-faceted optical assembly 10 joins light from the four depths 22a-22d end-to-end to form the sensor 6 in a row. In the simplified example of FIG. 3, the single pixel wide column 12 is shown as including twelve pixels, three in each of the subsections 12a-12d. In reality, however, there may be hundreds of pixels in each of the four sections of the single column. Each of the twelve pixels in this example receives light emanating from a vicinity of a respective one of twelve points of the sample area 4 that are actively illuminated by the illumination surface. The width W and depth D of this vicinity around each point of the sample area 4 may depend, at least in part, on the collection optics of the device 2. The width W may also depend, at least in part, on the width of the pixels of the image sensor 6, and / or on other aspects of the properties of the combined illumination and receiving assembly 8 and the multi-faceted optical assembly 10, such as the numerical aperture and / or magnification. Here, the depths 22a-22d are shown as continuous (i.e., touching), but in some embodiments there are gaps between them from which light is not sensed (i.e., recorded) by the sensor 6. The length La, Lb, Lc, Ld of the horizontal line through the sample area 4 sampled at each depth 22a-22d can depend on the collection optics and the height and resolution of the image sensor 6. The length L may be the same at all points of the sample area 4 illuminated by the illumination surface (e.g., La=Lb=Lc=Ld), but this is not required. Similarly, the width W may be the same at all points and the depth D may be the same at all points, but this is not required. The width W, depth D, and length L may be the same across the area 4, or may additionally be equal to each other, i.e., W=D=L, such that each pixel samples light from a respective cubic neighborhood around a point of the sample area. However, in other examples, the width W, depth D, and length L are not equal to one another, and a pixel may sample light from a non-cubic neighborhood around each point of the sample area. For example, the depth D may be greater than the width W and / or the length L.

[0054] In some operating modes, only a sub-region of the image sensor 6 is used to capture an image, for example only a central subset of the columns, which potentially allows for higher frame rates to be used in some modes.

[0055] Figure 4 shows another apparatus for a volume microscope 102. The structure of the apparatus 102 shown in Figure 4 is largely the same as that of the apparatus 2 shown in Figures 1 and 2, and comprises an illumination and receiving assembly 108 and a multifaceted optical assembly 110 for imaging a sample area 104 containing a sample (not shown). However, instead of a single image sensor 6, the apparatus 102 comprises a first image sensor 106 and a second image sensor 107.

[0056] The operation of the device 102 is substantially the same as that described above with reference to Figures 1 and 2. However, the multifaceted optical assembly 110 (e.g., the multifaceted prism 400 described below) directs light from first and third depths of the illumination plane 124 of the sample area 104 to different subsections 114a, 114c of the row 114 of the first image sensor 106, and directs light from second and third depths of the sample area 104 to different subsections 115b, 115d of the row 115 of the second image sensor 107. By splitting the light from the illumination plane 124 between the two image sensors 106, 107, the area of ​​each sensor 106, 107 used to sense light from each depth can be increased, improving the imaging resolution.

[0057] FIG. 5 shows a multi-faceted prism 400 used as a multi-faceted optical assembly in an embodiment of the present invention. The prism 400 receives an input light 402 including light from multiple depths A, B, C, D, E, F, G, H of the sample area and generates an output light 404 including the input light 402 separated into different components corresponding to different depths (i.e., different depth ranges). The prism 400 is designed to vector-rise the axial plane into strips with Nyquist optical spacing for projection onto a single exposure line of the image sensor 6. The prism 400 shown in FIG. 5 splits the output light 404 into two different parts so that light from depths A, C, E, and G can be sent to the first image sensor 106 and light from depths B, D, F, and H can be sent to the second image sensor 107. Other prisms can send light to one part for use in a device 2 including only a single image sensor 6.

[0058] In other embodiments, the grating may be used as a multi-faceted optical assembly.

[0059] 6 shows another apparatus 202 for volumetric microscopy of a sample region 4. The apparatus 202 comprises an image sensor 206, a combined illumination and receiving assembly 208, and a multifaceted optical assembly 210 (e.g. a multifaceted prism). As mentioned above, the sample 3 is presented stationary on a horizontal surface 5, such as a slide, and may be a fluorescently labeled biological sample.

[0060] The image sensor 206 is arranged to sequentially sense columns of pixels (embodied as respective "sections") at a sensing rate. Figure 6 shows a column 212 of pixels being exposed and sensed. The column 212 comprises four subsections 212a, 212b, 212c, 212d.

[0061] The illumination and receiving assembly 208 comprises a light source 216 (e.g., a laser), a movable mirror 217, a beam splitter 218 and an objective lens assembly 220. It may optionally include further lenses, mirrors, filters, etc. The illumination and receiving assembly 208 is arranged to illuminate the sample area 4 with a narrow light sheet (tilted light sheet) that extends at an oblique angle with respect to the imaging axis of the objective lens assembly 220. The position of the light sheet on the sample area 4 is changed by moving the movable mirror 217. The movable mirror 217 is arranged in the Fourier plane of the optical path such that rotating the movable mirror 217 translates the light sheet laterally (i.e., perpendicular to the imaging axis) across the sample area 204. In other words, by moving the movable mirror 217, the illumination and receiving assembly 208 is arranged to smoothly sweep the oblique directional light sheet across (i.e., through) the sample area 4, illuminating a continuum of tilted planes through the sample area. In this continuum, multiple distinct inclined surfaces of the sample area can be considered to be sequentially illuminated with an illumination rate equal to the detection rate. In an alternative embodiment, the light sheet can be moved in a series of discrete steps with an illumination rate equal to the detection rate.

[0062] The computer system 221 controls the actuator of the moveable mirror 217. It also receives and processes image data from the image sensor 206 to obtain microscopy data of the sample 3, which in an embodiment can provide super-resolution imaging of the sample 3 using computational nanoscopy processing.

[0063] The objective lens assembly 220 is also positioned to capture light generated at the sample region 4 (e.g., scattered light or fluorescent light from the illumination plane through the sample 3) and direct it to the multi-faceted optical assembly 210 via the beam splitter 218. The multi-faceted optical assembly 210 separates light from four different depth zones of the sample region 4 and directs them to different respective subsections of pixels of the image sensor 6. Because the illumination plane is tilted, the light from the different depths spreads in a direction perpendicular to the imaging axis of the objective lens assembly 220. This facilitates effective separation of light from different depths (i.e., effective axial optical cut) and improves axial resolution performance. FIG. 6 shows how light from the deepest depth 222a of the first plane 222 (i.e., furthest from the objective lens assembly 220) is directed to the first subsection 212a, while light from the shallowest depth 222d of the first plane 222 is directed to the fourth subsection 212d. The lower and upper intermediate depths 222b, 222c are directed into second 212b and third 212c subsections, respectively.

[0064] As the oblique light is swept through the sample, the sensor 206 accumulates image data for the entire volume in one exposure. Because the illumination plane 222 is tilted relative to the imaging axis but still corresponds to a vertical section (column) of the sensor 206, the areas of the sensor that correspond to data from different depths in the sample are slightly shifted (indicated by the dotted lines in FIG. 6).

[0065] Other than the tilted nature of the illumination plane, the operation of apparatus 202 is similar to that of apparatus 2 described above. The illumination and receiving assembly 208 sweeps an angled light sheet across the sample area 4 so as to sequentially illuminate multiple parallel tilted planes of the sample. The image sensor 206 sequentially senses each of the rows of pixels with equal sensitivity such that light from each of these planes is sensed by a different respective row of pixels. In this manner, apparatus 202 captures a three-dimensional image of the sample area 4 in a single frame of the image sensor 206.

[0066] Any light from the sample area 4 that hits a portion of the sensor 206 outside of one actively detected row (i.e., hitting a row that is not accumulating charge) is not imaged. This results in a precise vertical section of the sample area 4. At any given instant, the sensor 206 only detects light from a diamond-shaped area of ​​the sample 3 illuminated by the current tilted illumination plane. As a result, light from different depths in a given axial plane of the sample 3 is detected at different times, improving depth sectioning and axial resolution.

[0067] Each pixel column may receive some additional light from a narrow volume around the illuminated plane due to the width and / or motion of the light sheet (and potentially due to non-zero decay times of any fluorescent markers in sample 3). The collection optics and pixel width may also affect segmentation; however, these can be configured to limit the thickness of the diamond-shaped volume sensed by each column to give an accurate segment.

[0068] Figure 7 compares the operation of an apparatus for volumetric microscopy using on-axis planar illumination (e.g., apparatus 2 described above with reference to Figures 1-5) with that using oblique planar illumination (e.g., apparatus 202 described above with reference to Figure 6). For purposes of understanding, both options are shown together in Figure 7, although in practice only one illumination approach is used at a time.

[0069] For axial plane illumination, a laser 702 generates a first beam 704 that passes through a cylindrical lens 706 and then through the center (on-axis) of a scan lens 708. The first beam 704 is reflected by a moveable mirror 710, passes through two lenses 712, and reflects off a dichroic mirror 714 before entering an objective lens assembly 716. This assembly of lenses and mirrors converts the first beam 704 into an axial light sheet 718 that illuminates the axial plane of a sample region 720. The axial light sheet 718 extends across multiple depths in the sample, four example depths labeled A, B, C, and D.

[0070] For oblique surface illumination, the laser 702 generates a second beam 722 that passes through the cylindrical lens 706 and then through the scan lens 708. However, it passes through the scan lens 708 off-axis (i.e., away from the center of the lens). The second beam 722 is reflected by a movable mirror 710, passes through two further lenses 712, and is reflected off a dichroic mirror 714 before entering the objective lens assembly 716. This assembly of lenses and mirrors converts the second beam 722 into an oblique light sheet 724, which illuminates the oblique surface of the sample area 720. The oblique light sheet 724 also extends across multiple depths of the samples A, B, C, D.

[0071] FIG. 7 includes a detailed inset of a sample area 720 illuminated by an axial light sheet 718 and an inclined light sheet 724 .

[0072] In either case, the illuminated surface of the sample produces light (e.g., by fluorescence) that is captured by the objective lens assembly 716 and directed to a multi-faceted optical assembly 726, which directs light from different depths A, B, C, D of the sample area 720 to different portions of an image sensor 728.

[0073] The movable mirror 710 rotates to sweep either the axial or oblique light sheet 718, 724 through the sample. Each row of the image sensor 728 is sensed at the same rate. FIG. 7 includes a detailed inset of the image sensor 728, with sections of the image sensor 728 corresponding to different depths of the highlighted sample area for both axial and oblique illumination. When an oblique light sheet is used, the sections of the image sensor 728 corresponding to different depths of the sample area are offset. This offset is taken into account when processing the sensed data to generate a three-dimensional image of the sample area 720.

[0074] 8 shows some examples of multi-faceted optical assemblies suitable for use in embodiments of the present invention. A first multi-faceted optical assembly 802 includes a multifocal (MF) grating arranged for use with axial illumination. A second multi-faceted optical assembly 804 also includes an MF grating. However, the second multi-faceted optical assembly 804 is rotated relative to the first multi-faceted optical assembly 802 so that it is arranged for use with oblique illumination.

[0075] The third multi-faceted optical assembly 806 comprises a beam splitter (BS) cascade arranged for use with axial illumination. The fourth multi-faceted optical assembly 808 also comprises a BS cascade. The BS cascades 806, 808 each comprise a number of beam splitter cubes and prism mirrors. Each of the components of the fourth multi-faceted optical assembly 808 are rotated relative to the components of the third multi-faceted optical assembly 806 so that they are arranged for use with oblique illumination.

[0076] The fifth multi-faceted optical assembly 810 includes a multi-focus (MF) prism that can be appropriately rotated for use with axial and oblique illumination.

[0077] 9 shows example optical transfer functions (OTFs) for an apparatus using axial and oblique illumination. The first OTF 902 is for an apparatus using axial illumination. The second OTF 904 is for an apparatus using oblique illumination. Otherwise the apparatus is identical.

[0078] 9 shows how the second OTF 904 (tilted illumination) extends further in the z-direction, showing improved axial resolution. More generally, the second OTF 904 (tilted illumination) extends over a larger area than the first OTF 902, showing that more information along the axial direction may be recoverable from the sample area using tilted illumination.

[0079] It will be understood that although embodiments are shown with features such as sample regions, image sensors, etc. having particular orientations, these may differ in other embodiments, and references herein to "vertical", "horizontal", "width", "height", etc. are adapted accordingly. More generally, the "depth" of a sample region is not limited to any particular orientation.

[0080] While the present invention has been described in detail in connection with only a limited number of embodiments, it should be readily understood that the invention is not limited to such disclosed embodiments. Rather, the present invention can be modified to incorporate any number of variations, alterations, substitutions, or equivalent arrangements not heretofore described, but which are commensurate with the scope of the present invention. Moreover, while various embodiments of the present invention have been described, it should be understood that aspects of the invention may include only some of the described embodiments. Thus, the present invention is not limited by the above description, but only by the appended claims.

Claims

1. 1. An apparatus for volumetric imaging, comprising: an illumination assembly arranged to direct light to sequentially illuminate multiple planes of the sample region at an illumination rate, each plane extending across multiple depths of the sample region; an image sensor including a plurality of pixel sections configured to sequentially detect each pixel section at a predetermined sensing rate; a light receiving assembly including a plurality of sections of pixels and arranged to sequentially sense each of the sections of pixels at a sensing rate; a light receiving assembly arranged to receive light from the sample area and arranged to direct light received from the planar surface of each of the sample areas to a different section of each of the pixels; the light receiving assembly is positioned to receive light from the plurality of depths of the sample region, and includes a multi-faceted optical assembly positioned to simultaneously direct light from each of the plurality of depths in each of the planes to different respective subsections of the section for each of the sections of each of the pixels; the illumination rate is equal to the detection rate, and the sections of each of the pixels are positioned to detect light from the plurality of depths in each of the planes when the planes are illuminated by the illumination assembly.

2. The device described in claim 1, wherein the multi-faceted optical assembly is positioned to simultaneously direct light from each of at least four depths to each of the different subsections of the image sensor.

3. An apparatus as described in claim 1 or 2, wherein the multifaceted optical assembly includes a multifaceted prism or a multifaceted grating.

4. An apparatus as described in claim 1 or 2, wherein the image sensor includes an electronic shutter circuit arranged to selectively detect the pixels of each of the sections of the image sensor during successive electronic shutter periods.

5. An apparatus as described in claim 1 or 2, arranged to repeatedly image the sample area volumetrically to generate image data indicative of a time series of the volume of the sample area.

6. An apparatus as described in claim 1 or 2, wherein the light receiving assembly includes an objective lens assembly positioned to pass light emitted from the sample region through the multifaceted optical assembly.

7. The apparatus of claim 6, wherein the objective lens assembly forms part of the illumination assembly and is positioned to pass light from the illumination assembly to the sample region.

8. The device described in Claim 6, wherein at least one of the multiple planes is inclined with respect to the imaging axis of the objective lens assembly.

9. An apparatus as described in claim 1 or 2, wherein the illumination assembly is arranged to generate a light sheet and sweep or step the light sheet across the sample area to illuminate the multiple planes.

10. An apparatus as described in claim 1 or 2, wherein the multiple planes are parallel planes.

11. An apparatus as described in claim 1 or 2, wherein each of the subsections consists of a respective contiguous set of the pixels.

12. An apparatus as described in claim 1 or 2, wherein the section of each of the pixels includes a line of each of the pixels, and the image sensor is arranged to detect adjacent lines sequentially.

13. An apparatus as described in claim 1 or 2, including a second image sensor including a plurality of sections and arranged to sequentially detect the sections of each of the pixels at the detection rate, and the multifaceted optical assembly arranged to simultaneously direct light from each of a second plurality of depths in the sample region to each of the subsections of the sections of each of the pixels of the second image sensor.

14. An apparatus as described in claim 1 or 2, arranged to perform fluorescence volumetric microscopy of a sample in the sample region.

15. A method of volumetric imaging, comprising: directing the light to sequentially illuminate a plurality of planes of the sample region at an illumination rate, each plane extending across a plurality of depths of the sample region; directing light emanating from the sample area to an image sensor; the image sensor includes a plurality of sections of pixels, and light received from each of the planes of the sample area is directed to a different one of the sections of the pixels, and for each of the sections of the pixels, light is simultaneously directed to a different one of the sections from each of the plurality of depths in each of the planes; The image sensor sequentially detects each of the sections at a detection rate, a transmission rate equal to the illumination rate, and the section of each of the pixels detects light from the plurality of depths in each of the planes when the planes are illuminated.

Images