Confocal imaging device and confocal imaging method
The confocal imaging device and method leverage a multi-pinhole plate and telecentric optical system to address the limitations of previous confocal imaging technologies, achieving improved spatial resolution and brightness for three-dimensional imaging by forming multiple focal points and processing background light, thus enabling effective wide-field observations.
Patent Information
- Application Number
- PCT/JP2024/016941
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-09
- Filing Date
- 2024-05-07
- Publication Date
- 2024-11-14
AI Technical Summary
Current confocal imaging devices face limitations in achieving three-dimensional observation and improving brightness for cell dynamics in tissues and embryos, with restricted spatial resolution in the z-direction and optical axis due to low numerical aperture and background light interference.
A confocal imaging device and method utilizing a multi-pinhole plate to form multiple focal points, combined with a telecentric optical system and low-pass filter processing to enhance spatial resolution and remove background light, allowing for wide-field imaging with improved z-direction resolution and brightness.
The solution enables high-resolution, three-dimensional imaging with enhanced spatial resolution in both the xy and z-planes, effectively addressing the limitations of previous technologies by improving the confocal imaging performance and allowing for wider field of view observations.
Smart Images

Figure JP2024016941_14112024_PF_FP_ABST
Abstract
Description
Confocal imaging device and confocal imaging method
[0001] The present disclosure relates to a confocal imaging apparatus and a confocal imaging method, and more particularly to a confocal imaging apparatus and a confocal imaging method that image a sample using a multi-pinhole plate that forms multiple foci.
[0002] Patent Document 1 discloses a fluorescence imaging device that captures fluorescence from a measurement object using an imaging device such as a CMOS (Complementary Metal-Oxide-Semiconductor) or a CCD (Charge Coupled Device). The fluorescence imaging device in Patent Document 1 uses a telecentric optical system lens with a magnification of 2x.
[0003] Japanese Patent Application Laid-Open No. 2023-3158
[0004] The fluorescence imaging device described in Patent Document 1 can achieve dynamic observation with subcellular level planar (xy plane) spatial resolution within a centimeter observation field of view (FOV). This technology is an extremely powerful tool and can be used in many applied research applications. However, due to its low NA (numerical aperture) of 0.12, there are limitations on the spatial resolution in the optical axis direction (z direction) and the brightness of the imaging system. To expand the possibilities for observing cellular dynamics in tissues and embryos, it is desirable to enable three-dimensional observation and improve brightness.
[0005] The present disclosure has been made in consideration of the above points, and aims to provide a confocal imaging apparatus and a confocal imaging method that are capable of appropriately imaging a sample over a wide field of view.
[0006] The confocal imaging apparatus of this embodiment comprises a light source that generates illumination light to illuminate a sample, a multi-pinhole plate that is positioned conjugate with the sample and has a plurality of pinholes through which the illumination light passes, a drive mechanism that drives the multi-pinhole plate to change the incident position of the illumination light on the sample, an objective lens that focuses the illumination light from the multi-pinhole plate onto the sample and into which signal light from the sample is incident, an imaging lens that is positioned between the objective lens and the multi-pinhole plate and images the signal light from the sample on the multi-pinhole plate, a beam splitter that splits the signal light that has passed through the pinholes from the illumination light heading toward the multi-pinhole plate, a telecentric lens into which the signal light split by the beam splitter is incident, and an imaging device that detects the signal light from the telecentric lens and images the sample.
[0007] In the above-mentioned confocal imaging apparatus, a background light component may be calculated by performing a filtering process using a low-pass filter on image data of the captured image captured by the imaging device, and the background light component may be removed from the image data of the captured image.
[0008] In the above confocal imaging apparatus, image data of the image captured by the imaging device is f 0 (x, y) as image data f 0 By processing (x, y) using a low-pass filter, the surface data L 0 (x, y) is generated, and j is an integer equal to or greater than 1, and image data f is obtained by performing iterative calculations using the following equation (1). j Generate (x, y), The image data f j (x, y) is processed using the low-pass filter to obtain image data L j (x, y) is calculated, and the image data f of the captured image is 0 (x, y) to obtain the image data L j The background light component may be removed by subtracting (x, y).
[0009] In the above confocal imaging apparatus, the low-pass filter may be a one-dimensional infinite impulse response filter, and the background light component may be calculated by sequentially applying the one-dimensional infinite impulse response filter in the x and y directions.
[0010] In the above confocal imaging apparatus, the magnification of the image of the sample formed by the imaging lens may be 2x or less.
[0011] In the above confocal imaging apparatus, the pinhole may have a diameter of 15 μm or less.
[0012] The confocal imaging method according to this embodiment includes the steps of generating illumination light for illuminating a sample, directing the illumination light to a multi-pinhole plate positioned conjugate with the sample, focusing the illumination light that has passed through multiple pinholes in the multi-pinhole plate onto the sample using an objective lens, directing signal light from the sample to the objective lens, and imaging the signal light from the sample onto the multi-pinhole plate using an imaging lens positioned between the objective lens and the multi-pinhole plate, splitting the signal light that has passed through the multiple pinholes from the illumination light heading toward the multi-pinhole plate using a beam splitter, and imaging the signal light split by the beam splitter onto an imaging device using a telecentric lens, driving the multi-pinhole plate to change the incident position of the illumination light relative to the sample, and imaging the sample using the signal light detected by the imaging device when the multi-pinhole plate is driven.
[0013] In the above-described confocal imaging method, a background light component may be calculated by performing a filtering process using a low-pass filter on image data of the captured image captured by the imaging device, and the background light component may be removed from the image data of the captured image.
[0014] In the above confocal imaging method, the image data of the image captured by the imaging device is f 0(x, y) as image data f 0 By processing (x, y) using a low-pass filter, the surface data L 0 (x, y) is generated, and j is an integer equal to or greater than 1, and image data f is obtained by performing iterative calculations using the following equation (1). j Generate (x, y), The image data f j (x, y) is processed using the low-pass filter to obtain image data L j (x, y) is calculated, and the image data f of the captured image is 0 (x, y) to obtain the image data L j The background light component may be removed by subtracting (x, y).
[0015] In the above confocal imaging method, the low-pass filter may be a one-dimensional infinite impulse response filter, and the background light component may be calculated by sequentially applying the one-dimensional infinite impulse response filter in x and y directions.
[0016] In the above confocal imaging method, the magnification of the image of the sample formed by the imaging lens may be 2x or less.
[0017] In the above confocal imaging method, the pinhole may have a diameter of 15 μm or less.
[0018] According to the present disclosure, it is possible to provide a confocal imaging apparatus and a confocal imaging method that can appropriately image a sample over a wide field of view.
[0019] 1 is a schematic diagram showing the basic configuration of an optical system; FIG. 2 is a diagram showing evaluation results of a point spread function (PSF); FIG. 3 is a diagram showing a fluorescent image of cardiomyocytes and its line profile; FIG. 4 is a diagram showing the configuration of a confocal imaging device; FIG. 5 is a diagram showing a pinhole array disk; FIG. 6 is a diagram for comparing images captured with and without a pinhole array disk; FIG. 7 is a diagram showing evaluation results of a point spread function (PSF); FIG. 8 is a diagram showing an image of the cavity chamber structure of a myocardial organoid; FIG. 9 is a diagram showing an image for explaining background removal processing; FIG. 10 is a schematic diagram for explaining an observation method; FIG. 11 is a diagram showing a tomographic image; FIG. 12 is a graph showing imaging results.
[0020] Below, embodiments to which the present invention can be applied are described. The following description is for describing embodiments of the present invention, and the present invention is not limited to the following embodiments. For clarity of explanation, the following description has been omitted and simplified as appropriate. Furthermore, a person skilled in the art would be able to easily modify, add, or convert each element of the following embodiments within the scope of the present invention. Note that elements with the same reference numerals in each drawing indicate similar elements, and descriptions thereof will be omitted as appropriate.
[0021] (Basic Configuration of the Optical System) First, we will explain the basic configuration of the optical system developed by the inventors of this application. The large-field-of-view imaging system (AMATERAS1.0 or AMATERAS-1) previously constructed by the inventors used a 2x telecentric macro lens. It achieved dynamic observation with subcellular-level planar (xy-plane) spatial resolution with a centimeter-level field of view (FOV). Therefore, the inventors of this application developed a giant lens system that increased the NA (numerical aperture) to 0.25 while maintaining 2x magnification. This lens system includes an objective lens and an imaging lens. The lens system has a field of view that covers an image sensor size of 44 mm diagonal and is aberration-corrected in the wavelength range of 400 nm to 700 nm. The objective lens of the lens system is 310 mm long and 144 mm in diameter. The imaging lens of the lens system is 345 mm long and 170 mm in diameter.
[0022] The optical invariant and spatial bandwidth product, which are two key indices for evaluating the ratio of the FOV and spatial resolution of this lens system, are 2.75 and 4.3*10, respectively. 8 Since this system uses a CMOS sensor with a diagonal of 35 mm, the effective values of these indices are 2.19 and 2.8*10, respectively. 8 These values are higher than those of previously reported large-field microscopes, and are much higher than those of commercially available microscopes and microscope lenses for machine vision.
[0023] An epi-fluorescence imaging system constructed using this lens system will be described with reference to Figure 1. Figure 1 is a diagram schematically showing the configuration of an imaging device 100. In the following description, the optical axis direction of the optical system is defined as the z direction, and the xy plane is defined as a plane perpendicular to the z axis. The x direction, y direction, and z direction are perpendicular to one another. The imaging device 100 forms an image of a sample 20 on a camera 32 using a lens system 30 including an objective lens 16 and an imaging lens 31.
[0024] The camera 32 is an imaging device such as a CMOS sensor or CCD sensor, and captures an image of the sample 20. A 120-megapixel camera (VCC-120CXP1M, CIS) or a 250-megapixel camera (VCC-250CXP1M, CIS) can be used as the camera 32. The two cameras are selected depending on the research objective. Both cameras have a diagonal of 35 mm, resulting in a diagonal observation field of view of approximately 17.8 mm. The pixel sizes are 2.2 μm and 1.5 μm, respectively. The sampling intervals are 1.1 μm and 0.75 μm, respectively, at a magnification of 2x. Image data from the camera 32 is input to the processing unit 35 via a CoaXpress frame grabber board (APX-3664G3, Avaldata). The processing unit 35 is a workstation equipped with a processor, memory, display, etc.
[0025] The light source 10 is a high-brightness LED (Light Emitting Diode) that generates illumination light L1 to be irradiated onto the sample 20. For example, the light source 10 is a Thorlabs SOLIS-470C that generates light with a wavelength of 470 nm. The light L1 from the light source 10 becomes excitation light that excites the sample 20. The illumination light L1 from the light source 10 is incident on a homogenizing optical system 11. The homogenizing optical system 11 has a fly's eye lens pair for uniformly illuminating the entire field of view. The homogenizing optical system 11 homogenizes the spatial distribution of the illumination light L1 on the sample 20 in the xy plane.
[0026] Illumination light L1 from the homogenizing optical system 11 is incident on a dichroic mirror 15 via a lens 14. A conjugate plane 13 that is conjugate with the sample 20 is disposed between the homogenizing optical system 11 and the lens 14. The dichroic mirror 15 reflects the illumination light L1 incident from the side upward toward the objective lens 16.
[0027] The dichroic mirror 15 acts as a beam splitter that splits light according to wavelength. The light reflected by the dichroic mirror 15 is incident on the objective lens 16. For example, the NA of the objective lens 16 is 0.25. The objective lens 16 is disposed directly above the dichroic mirror 15.
[0028] The objective lens 16 focuses the illumination light L1 onto the sample 20. The objective lens 16 is positioned directly below the sample 20. The homogenizing optical system 11 provides a uniform spatial distribution of the illumination light L1 across the entire FOV. The sample 20 is, for example, a cell, and is placed on a cell incubator 21. The cell incubator 21 is placed on a sample stage 22. The sample stage 22 is a five-axis drive stage with xyzαβ axes. α is the angle around the x-axis, and β is the angle around the y-axis. The sample stage 22 drives the sample 20, allowing the observation position to be changed. When the illumination light L1 is irradiated onto the sample 20, fluorescence is generated from the sample 20.
[0029] Fluorescence generated in the sample 20 enters the objective lens 16. The fluorescence that enters the objective lens 16 becomes signal light L2 and enters the dichroic mirror 15. The dichroic mirror 15 splits the signal light L2 and the illumination light L1 according to the wavelength. Therefore, the signal light L2 passes through the dichroic mirror 15 and enters the imaging lens 31. The imaging lens 31 serves as an imaging lens that forms an image of the sample 20 on the camera 32. In addition, a filter 33 is disposed between the camera 32 and the imaging lens 31. The filter 33 is a 2-inch dual bandpass fluorescence filter. The filter 33 blocks light of the excitation wavelength.
[0030] The lens system 30, which includes the objective lens 16 and the imaging lens 31, forms an image of the sample 20 at a magnification of 2x. The magnification of the lens system 30 is preferably a low magnification of 4x or less. The objective lens 16 has an NA of 0.25. The imaging lens 31 is an imaging lens that forms a primary image of the sample 20 and has an NA of 0.125. The sensor-side NA (also referred to as the image-side NA) is calculated by dividing the NA of the objective lens 16 (here, 0.25) by the magnification of the lens system 30 (here, 2x). The sensor-side NA here is 0.125. If the NA of the objective lens 16 is 0.25 and the magnification of the lens system 30 is 4x, the sensor-side NA is 0.0625. If the NA of the objective lens 16 is 0.2 and the magnification is 4x, the sensor-side NA is 0.05. Here, the sensor-side NA is preferably 0.05 or more, and more preferably 0.1 or more, which makes it possible to capture an image of a wide field of view while maintaining high spatial resolution.
[0031] For example, if the NA is reduced, it becomes impossible to obtain sufficient spatial resolution. Also, if the magnification is increased, it becomes difficult to obtain a sufficiently wide field of view. Therefore, by setting the value obtained by dividing the NA by the magnification within an appropriate range, it becomes possible to capture an image of a wide field of view with high spatial resolution.
[0032] FIG. 2 is a diagram showing evaluation results of the point spread function (PSF) of the imaging device 100. FIG. 2 shows experimental results of the point spread function obtained using green fluorescent beads with a diameter of 0.2 μm. Measurement result C shown in the upper part of FIG. 2 shows data captured with a 120 million pixel camera. Measurement result D shown in the lower part of FIG. 2 shows data captured with a 250 million pixel camera. Measurement results C and D of FIG. 2 show point spread functions in the xy plane and xz plane on the left side. Measurement results C and D of FIG. 2 show line profiles of the point spread function in the x direction and z direction on the right side.
[0033] Spatial resolution is evaluated by the full width at half maximum of the point spread function calculated by fitting the line profile to a Gaussian function. With a 120-megapixel camera, the spatial resolution is 1.16 μm (xy). Even with a 250-megapixel camera, the spatial resolution is approximately 1.15 μm (xy). The spatial resolution in the xy plane is significantly improved, approximately twice as much as the previous version (AMATERAS 1.0 or AMATERAS-1). However, with the 120-megapixel camera, the sampling interval is coarse (undersampling), so the apparent spatial resolution deteriorates when the bead center is near the boundary between adjacent pixels. Coarse sampling also introduces uncertainty into the estimated position of the bead center coordinates, which varies depending on the relative position of the bead and the pixel. This can be somewhat problematic when quantitative shape evaluation is required or when observing cells, such as E. coli, that are only a few microns in size.
[0034] However, this is not a practical problem when imaging at subcellular spatial resolution. A 250-megapixel camera provides a nearly constant resolution. In actual research settings, a 250-megapixel camera is selected when high-resolution images are required, and a 120-megapixel camera is selected in other cases (when an increase in the number of photons per pixel, a reduction in data size, or a high frame rate is required).
[0035] Similarly, the depth of field (DOF) in the z direction was evaluated using a Gaussian fit of the line profile, and was found to be 15.1 μm for the 120-megapixel camera and 14.0 μm for the 250-megapixel camera. This difference in DOF is thought to be due to the difference in pixel size between the cameras.
[0036] Figure 3 shows a fluorescent image of cardiomyocytes stained with rhodamine phalloidin. Image E in Figure 3 is an image of the entire FOV, and image F is an image of a part of the FOV, that is, an image of a local region indicated by a white frame in image E. Images E and F in Figure 3 are images acquired using a 250 million pixel camera. The size of the FOV is 14.7 x 9.4 mm 2 Image G in Figure 3 shows an image of a local region captured with the previous version (AMATERAS1.0 or AMATERAS-1) for comparison. Graph H in Figure 3 shows the respective line profiles. Image F shows that the filament structure of F-actin stained with rhodamine is spatially resolved with greater precision.
[0037] 1 employs wide-field illumination and detection, making it impossible to selectively acquire images of a specific z-plane. Therefore, in this embodiment, an optical sectioning method is used to achieve three-dimensional imaging capability.
[0038] There are several existing optical sectioning methods that can be applied to fluorescence imaging in the visible wavelength range, including laser scanning confocal microscopy, light sheet microscopy, light field microscopy, and spatiotemporal focusing. Among these options, this embodiment employs confocal imaging. Light sheet microscopy has difficulty uniformly illuminating an area of 1 cm or larger. Furthermore, light field microscopy can achieve three-dimensional resolution in a single shot, but it sacrifices spatial resolution. Spatiotemporal focusing requires the use of a high-power near-infrared ultrashort pulse laser, making it unsuitable for the lenses mentioned above. Therefore, this embodiment employs a confocal method, particularly a multifocal confocal method (multi-confocal method), as a method well-suited to imaging systems for wide-field illumination and detection. This method covers a wide FOV.
[0039] The configuration of a confocal imaging device 100 will be described with reference to FIG. 4. FIG. 4 is a diagram schematically illustrating the overall configuration of the imaging device 100. The imaging device 100 is a confocal imaging device using a multi-pinhole. Specifically, a multi-pinhole plate 40 having a plurality of pinholes is disposed in the optical path of the illumination light L1. This allows the illumination light L1 to form multiple focal points on the sample 20. Note that in the following description, explanations of the same configuration as in FIG. 1 will be omitted as appropriate. For example, the configuration of the lens system 30 is the same as in FIG. 1.
[0040] The light source 10 generates illumination light L1. As described above, the light source 10 is a high-brightness LED. Of course, the light source 10 is not limited to an LED, and may be a laser light source or the like. The illumination light L1 from the light source 10 is incident on the homogenizing optical system 11. The homogenizing optical system 11 has a fly's eye lens pair and a lens, and homogenizes the spatial distribution of the illumination light L1. For example, the homogenizing optical system 11 homogenizes the spatial distribution of the illumination light L1 in a multi-pinhole plate 40, which will be described later. The illumination light L1 from the homogenizing optical system 11 is incident on the mirror 17. The mirror 17 reflects the illumination light L1 incident from the side upward.
[0041] The illumination light L1 reflected by the mirror 17 is incident on the dichroic mirror 15. The dichroic mirror 15 is a beam splitter that splits the illumination light L1 and the fluorescence according to the wavelength. The dichroic mirror 15 has wavelength characteristics that allow the illumination light L1 to pass through.
[0042] The illumination light L1 transmitted through the dichroic mirror 15 is incident on the multi-pinhole plate 40. As shown in FIG. 5, the multi-pinhole plate 40 has multiple pinholes 40a formed therein. The multi-pinhole plate 40 is disposed at a position conjugate with the sample 20. Therefore, the illumination light L1 that has passed through the multiple pinholes 40a forms multiple focal points on the sample 20. In this embodiment, the imaging device 100 uses the multiple pinholes to form multiple focal points on the sample 20.
[0043] By reducing the pinhole diameter, the confocal performance (resolution) improves, but the light transmittance through the multi-pinhole plate 40 decreases. By increasing the pinhole diameter, the light transmittance through the multi-pinhole plate 40 improves, but the confocal performance (resolution) decreases. Here, as shown in FIG. 5 , the diameter of each pinhole 40 a is 6 μm. Of course, the diameter of the pinholes 40 a is not limited to the above value. As a result of using pinholes 40 a with various diameters, it was found that the diameter of the pinholes 40 a is preferably 15 μm or less, and more preferably 12 μm or less.
[0044] The multiple pinholes 40a are arranged in a spiral pattern with intervals between them. The optimal pinhole spacing depends on the brightness and distribution of fluorescent molecules in the sample. Here, as shown in FIG. 5, the multiple pinholes 40a are arranged in an array with a pitch of 24 μm. Of course, the pitch of the pinholes 40a is not limited to the above value. For example, the pitch of the pinholes 40a is preferably 50 μm or less, and more preferably 30 μm or less.
[0045] The multi-pinhole plate 40 is a disk (rotating disk) rotated by a driving mechanism 41. For example, the driving mechanism 41 includes a rotary motor that rotates the multi-pinhole plate 40. The multi-pinhole plate 40 is a circular rotating disk, the center of which serves as the rotation axis of the driving mechanism 41. The rotation axis is parallel to the optical axis and is positioned offset from the optical axis in the x-y plane. The driving mechanism 41 drives the multi-pinhole plate 40 to rotate at a constant rotation speed. This changes the position of the multi-focal points on the sample 20. Therefore, the sample 20 can be scanned with the illumination light L1, and an image of the sample 20 can be captured. The multi-pinhole plate 40 may be mounted on a zαβ three-axis driving stage.
[0046] The illumination light L1 that passes through the multi-pinhole plate 40 is incident on the imaging lens 31. For example, the NA of the imaging lens 31 is 0.125. The illumination light L1 from the imaging lens 31 is incident on the objective lens 16. The objective lens 16 focuses the illumination light L1 onto the sample 20. The sample 20 is, for example, cells held in a cell incubator 21. Furthermore, the cell incubator 21 is mounted on a sample stage 22. The sample stage 22 has a drive mechanism that changes the position of the sample 20. For example, the sample stage 22 is a five-axis drive stage in x, y, z, α, and β.
[0047] When a fluorescent substance in the sample 20 is excited by the illumination light L1, fluorescence is generated. The fluorescence from the sample 20 becomes signal light L2 and enters the objective lens 16. For example, the NA of the objective lens 16 is 0.25. The signal light L2 refracted by the objective lens 16 enters the imaging lens 31. The multi-pinhole plate 40 is disposed on the image plane of the imaging lens 31. Therefore, the imaging lens 31 forms an image of the sample 20 on the multi-pinhole plate 40. The signal light that passes through the pinholes 40a of the multi-pinhole plate 40 enters the dichroic mirror 15. The dichroic mirror 15 is disposed directly below the multi-pinhole plate 40.
[0048] The dichroic mirror 15 splits the optical path of the illumination light L1 and the optical path of the signal light L2 based on the difference in wavelength. The signal light L2 is reflected by the dichroic mirror 15 toward the relay lens 34. The signal light L2 reflected by the dichroic mirror 15 enters the filter 33. The filter 33 is a wavelength filter that transmits light of the fluorescence wavelength and blocks light of the excitation light wavelength. The filter 33 is, for example, a 2-inch bandpass fluorescence filter (#86-992, Edmund Optics).
[0049] The dichroic mirror 15 may be a short-pass dichroic mirror having a cutoff wavelength between the illumination light L1 and the signal light L2. Using a short-pass dichroic mirror can suppress image distortion caused by the signal light. For example, in a configuration in which a dichroic mirror is disposed between the multi-pinhole plate 40 and the relay lens 34, if a long-pass dichroic mirror is used, the image may be distorted when the signal light expanding from the pinhole 40a passes through a glass plate disposed at 45° to the optical axis. Therefore, the dichroic mirror 15 is a short-pass type, and the signal light reflected by the dichroic mirror 15 is detected. This suppresses image distortion.
[0050] The signal light L2 transmitted through the filter 33 enters the relay lens 34. The relay lens 34 is a telecentric lens that forms a confocal image of the sample 20 on the camera 32. The relay lens 34 is, for example, a 1x telecentric macro lens (LSTL10H-F, Myutron). If the relay lens 34 has a 1x magnification and the lens system 30 has a 2x magnification, the total magnification is 2x. By replacing the relay lens 34 with a 2x lens (LSTL20H-F, Myutron), the total magnification can be switched from 2x to 4x. This optical system allows the sample 20 to be imaged at a low magnification of 2x or less, or 4x or less. Furthermore, if the lens system 30 has a 4x magnification, the sample 20 can be imaged at a total magnification of 4x or less, or 8x or less.
[0051] The camera 32 is an imaging device such as a CMOS sensor, and has a plurality of pixels arranged in an array. For example, a 150 million pixel camera or a 250 million pixel camera can be used as the camera 32. The camera 32 outputs detection data detected by each pixel to the processing device 35. Because the imaging device 100 uses a multifocal confocal optical system, pixels at positions corresponding to the multifocal points receive the signal light L2. Each pixel of the camera 32 then outputs detection data indicating the luminance value of the light detected by the signal light L2.
[0052] While the driving mechanism 41 drives the multi-pinhole plate 40, the camera 32 detects the signal light L2. The processing device 35 generates a confocal image based on the detection data from the camera 32. The processing device 35 stores the image data of the captured image of the sample 20 in a memory or the like. In this way, the processing device 35 generates a confocal image of the sample 20 based on the signal light detected by the camera 32 when the multi-pinhole plate 40 is driven. The processing device 35 may control the driving of the driving mechanism 41 and the sample stage 22.
[0053] The imaging device 100 can capture a confocal image of the sample 20 using a multi-pinhole plate 40. The multi-pinhole plate 40 is positioned conjugate to the sample 20. Here, the imaging lens 31 forms a primary image of the sample 20 on the multi-pinhole plate 40. Furthermore, the multi-pinhole plate 40 is positioned conjugate to the light receiving surface of the camera 32. As the multi-pinhole plate 40 rotates, the illumination light L1 scans, changing the position of the multiple foci on the sample 20. The processing device 35 can generate a confocal image (tomographic image) of the sample 20 on the focal plane based on the detection data of each pixel. As the sample stage 22 moves the sample 20 in the z direction, the focal position of the illumination light L1 in the sample 20 changes in the z direction. This allows the imaging device 100 to capture multiple tomographic images (z-stack data).
[0054] The multi-pinhole plate 40 is installed so as to have a conjugate imaging relationship with the sample 20. This allows for the formation of a confocal optical system, improving the resolution in the z direction. The use of the multi-pinhole plate 40 enables imaging in a short time. The imaging device 100 does not include a microlens array for forming multiple foci on the sample 20. In other words, the imaging device 100 forms multiple foci on the sample 20 using only a pinhole array, without using a microlens array. In other words, the imaging device 100 forms multiple foci on the sample 20 without using a microlens array.
[0055] In typical multifocal confocal microscopes, a combination of a microlens array disk and a pinhole array disk is the mainstream. However, this method is difficult to apply to optical systems with low magnification and high NA. This is because the NA of the imaging lens 31 is 0.125, which is much larger than that of a conventional microscope. In other words, to achieve the same NA with a microlens as that of an imaging lens, it is necessary to either enlarge the microlens or shorten the distance between the microlens array disk and the pinhole array disk. However, a dichroic mirror must be placed between the disks, and a larger-than-normal dichroic mirror is required to cover a large field of view, making it impossible to shorten the distance between the disks. One possible way to increase the NA of the microlens array is to increase the size of the microlenses. However, doing so reduces the number of microlenses, i.e., the number of focal points transferred onto the sample, making it impossible to uniformly illuminate a large field of view.
[0056] Therefore, in this embodiment, a pinhole array alone is used to form multiple focal points, without using microlenses. This method is more conventional than the microlens-based method, and although the lack of microlenses results in lower light utilization efficiency, it avoids the above-mentioned problems. Therefore, it is suitable for imaging devices with high NA and low magnification. Confocal systems using microlens array disks are commercially available, but the pinhole array disks used in typical microscopes have diameters of several tens of microns and pinhole spacings of several hundred microns. For example, Yokogawa Electric's CSU-X has pinholes with a diameter of 50 μm and pinhole spacings of 250 μm.
[0057] Considering the magnification (2x) and NA (0.25) of the imaging device 100, a small pinhole size is required. Therefore, the pinhole size is set to 6 μm. A 6 μm pinhole is reduced and projected to 3 μm in the sample. This size is slightly larger than the diameter of the Airy disk at excitation wavelengths (up to 470 nm) such as GFP (Green Fluorescent Protein) and RFP (Red Fluorescent Protein) (up to 570 nm). For example, the Airy disk diameter in the sample 20 with an excitation wavelength of 470 nm is 2.29 μm (= 1.22 * 0.47 / 0.25). Similarly, the Airy disk diameter in the sample with an excitation wavelength of 570 nm is 2.78 μm (= 1.22 * 0.57 / 0.25).
[0058] Because the spread of the excitation light due to diffraction at the pinhole 40a is less than the NA (=0.125) of the imaging lens, the PSF of the excitation light optical system (the outward path from the pinhole 40a to the sample) must take into account the spread due to diffraction at the pinhole 40a. The PSF of the signal light optical system (the return path from the sample to the pinhole) is the convolution integral of the PSF of the point light source and the pinhole. The total PSF of the imaging device 100 is the product of the PSFs of the outward and return paths. Furthermore, because the homogenizing optical system 11 uniformly distributes the excitation light at the multi-pinhole plate 40, the excitation light is converted into multiple plane waves with different angles of incidence on the pinhole 40a.
[0059] The Airy Disk diameter on the multi-pinhole plate 40 at a fluorescence wavelength of 510 nm is 4.98 μm (= 1.22 * 0.51 / 0.125). The Airy Disk diameter on the multi-pinhole plate 40 at a fluorescence wavelength of 600 nm is 5.86 μm (= 1.22 * 0.60 / 0.125). If the pinhole diameter is smaller than this, the confocal performance improves but the transmittance decreases; if the pinhole diameter is larger, the transmittance increases but the confocal performance decreases. Furthermore, the optimal pinhole spacing depends on the brightness and distribution of fluorescent molecules in the sample. In this embodiment, the pinhole spacing is 24 μm. For example, the pinhole spacing is preferably 50 μm or less, and more preferably 30 μm or less.
[0060] The multi-pinhole plate 40 is disposed directly below the imaging lens 31, which is an imaging lens. In other words, the multi-pinhole plate 40 is disposed immediately after the imaging lens 31. This allows the imaging lens 31 and the multi-pinhole plate 40 to be positioned closer to each other. In addition, in the optical system that detects the signal light L2 from the sample 20, the objective lens 16, the imaging lens 31, the multi-pinhole plate 40, and the dichroic mirror 15 are disposed in this order. The signal light L2 branched by the dichroic mirror 15 is incident on the camera 32 via the relay lens 34, which is a telecentric lens. In other words, the relay lens 34 forms an image of the multi-pinhole plate 40 on the camera 32. This configuration allows the sample 20 to be appropriately imaged over a wide field of view.
[0061] Furthermore, the magnification of the relay lens 34 is 1x or 2x. When the diameter of the pinhole 40a is 15 μm or less and the magnification of the relay lens 34 is 1x, the projection size of the pinhole 40a on the camera 32 is 15 μm or less. When the diameter of the pinhole 40a is 15 μm or less and the magnification of the relay lens 34 is 2x, the projection size of the pinhole 40a on the camera 32 is 30 μm or less. In this way, a sufficient amount of light can be detected at each pixel, and the sample 20 can be appropriately imaged with high spatial resolution.
[0062] Figure 6 shows the image capture results obtained by the imaging device 100. A sample was used in which 0.5 μm fluorescent beads were dispersed three-dimensionally in an agarose gel. 2 The images show an area of 292 μm × 202 μm, which corresponds to the area of the multi-pinhole plate 40. The top row shows an image captured without the multi-pinhole plate 40. The middle row shows an image captured with the multi-pinhole plate 40. The bottom row shows the line profiles of these images. By using the multi-pinhole plate 40, background light from outside the focal plane is dramatically suppressed. This shows that optical sectioning is possible with the multi-pinhole plate 40. The imaging device 100 can appropriately capture images over a wide field of view.
[0063] 7 is a diagram showing evaluation results of the point spread function (PSF). The upper part of Fig. 7 shows measurement result D when a relay lens 34 with a magnification of 1x is used, and the lower part shows measurement result E when a relay lens 34 with a magnification of 2x and NA = 0.12 is used. The relay lens 34 with a magnification of 1x has an NA = 0.079. The relay lens 34 with a magnification of 2x has an NA = 0.12.
[0064] The NA of the 1x magnification relay lens is lower than the NA (NA = 0.125) of the imaging lens 31. Therefore, it can be seen that the spatial resolution in the lateral direction is reduced compared to the images obtained without the relay lens 34 (measurement results C and D in Figure 2). On the other hand, the NA of the 2x magnification relay lens (NA = 0.12) is approximately the same as that of the imaging lens, and the reduction in spatial resolution in the lateral direction is relatively small.
[0065] Next, three-dimensional imaging using computational sectioning will be described. In the imaging device 100 shown in FIG. 4, the focal position is scanned in the z direction, and the processing device 35 captures two-dimensional images at each z position. This allows multiple tomographic images, i.e., z-stack data, to be obtained. The imaging device 100 captures multiple tomographic images by changing the focal position in the z direction. The processing device 35 removes background light components by performing image processing on the image data acquired by the camera 32. Specifically, the processing device 35 separates signals from the focal plane and signals from outside the focal plane through computation.
[0066] The image before the background light component is removed is defined as the captured image. The captured image is a superposition of an image of an object within the focal plane and an image of an object outside the focal plane. It is assumed that the imaging data of the captured image has a high spatial frequency for the fluorescent image from within the focal plane and a low spatial frequency for the fluorescent image from outside the focal plane. The processing device 35 estimates the contribution from outside the focal plane as a baseline by iterative low-pass filtering. An image from within the focal plane is obtained by subtracting the baseline image from the captured image. The baseline image becomes two-dimensional data indicating the background light component from outside the focal plane. The processing device 35 performs this calculation for each layer of the z-stack and reconstructs it into a three-dimensional image.
[0067] Figure 8 shows an image in which background light components have been removed by sectioning calculation. Figure 8 shows an image of the cavity structure of a myocardial organoid. This organoid is a 3D organoid that has been extensively studied in developmental biology and regenerative medicine in recent years and is known as a model of human myocardial tissue development. Large-area organoids spanning the entire FOV were prepared and chemically fixed. Troponin was immunostained, and nuclei were stained with Hoechst.
[0068] The maximum height of the organoids was approximately 150 μm. The z-stack data was obtained by repeatedly scanning the 3D region containing the organoids in the z-direction at 4 μm intervals. Images A and B in Figure 8 are images of the organoids across the entire FOV.
[0069] Image A in Figure 8 is the captured image before computational sectioning, and image B is the processed image after computational sectioning. Both images A and B are maximum intensity projection images of the z-stack. Before computational sectioning, strong background light was superimposed on both z-planes, but after computational sectioning, the three-dimensional distribution of nuclei was clearly visualized.
[0070] Image C is an image of the local region indicated by the dashed frame in image A, and image D is an image of the local region indicated by the white frame in image B. Images C and D show tomographic images obtained when the z position is changed at 40 μm intervals.
[0071] The results show that the organoids have a hollow chamber structure, demonstrating the effectiveness of this method for wide-field 3D imaging.
[0072] In principle, the focal plane components on the baseline can be extracted using the above calculations, provided that the signal-to-baseline ratio is sufficiently higher than the bit depth. This method requires that the spatial frequency of the fluorescent image within the focal plane is clearly higher than the spatial frequency of the image outside the focal plane. In actual cell imaging, this method is effective when fluorescent molecules are localized within the cell, such as in the nucleus, or when they have a filament-like structure. However, care must be taken when using a spatially uniform intensity distribution, as this may result in results that emphasize the edges.
[0073] The following describes a specific example of sectioning processing in the processing device 35. The processing device 35 estimates the background light component using a low-pass filter. Specifically, the processing device 35 removes the background light component by repeatedly calculating a filtering process using a low-pass filter as follows:
[0074] Here, the image data of the image captured by the camera 32 is f 0 (x, y). Image data f 0 (x, y) is two-dimensional image data, and a luminance value is set for each x and y coordinate. 0 By applying a low-pass filter to (x, y), the surface data L of a smooth surface is obtained.0 (x, y) is calculated. 0 (x, y) is the image data f 0 (x, y) and specifically indicates the background light component. 0 The low frequency components of (x, y) are extracted as the orientation components.
[0075] The processing unit 35 calculates f 0 (x, y) and L 0 Take the smaller value of (x, y) and 1 (x, y). The processing unit 35 calculates f 1 By applying the same process to (x, y), f 2 (x, y). That is, the processing unit 35 obtains f 1 (x, y) is filtered by a low-pass filter, 1 (x, y). The processing unit 35 generates f 1 (x, y) and L 1 Take the smaller value of (x, y) and 2 Let (x, y) be the coordinates of the coordinate system. The processing unit 35 performs iterative calculations using the following equation (1).
[0076] min(a, b) is a function that returns the smaller value of a and b. j is an integer equal to or greater than 1. The processing unit 35 increments j from 1 by repeating the calculation of equation (1). j-1 (x, y) is the image data f j-1 The image data is obtained by applying a low-pass filter to (x, y). j (x, y) is the image data f 0 It asymptotically approaches the baseline of (x, y). j (x, y) is image data showing the background light component. As shown in the following equation (2), the processing unit 35 performs the iterative calculation of equation (1) using f j (x, y) and L j This is repeated until the standard deviation Var of the difference between (x, y) reaches a preset value ε.
[0077] Then, the processing unit 35 calculates f when the standard deviation converges to a preset value ε. j By processing (x, y) using a low-pass filter, image data L indicating the background light component is obtained. j The processing device 35 calculates (x, y) from the image data L j (x, y) is the baseline data. Image data L j (x, y) is the image data of the captured image f 0 The processing device 35 converts the image data f of the captured image into a baseline of (x, y). 0 (x, y) to the baseline image data L j By subtracting (x, y), a confocal image from which the background light component has been removed is obtained.
[0078] When the image size (number of pixels) is small, the processing device 35 may perform processing using a convolution filter such as a moving average method, or a low-pass filter in Fourier space using FFT (Fast Fourier Transform). Note that when the number of pixels is very large, the processing time becomes long, so the processing device 35 uses an infinite impulse response (IIR) filter to increase speed.
[0079] The processing device 35 sequentially applies one-dimensional IIR filters in two directions (x, y) as a calculation method for a two-dimensional low-pass filter. For example, a one-dimensional infinite impulse response filter may be applied in the x direction and then a one-dimensional impulse response filter in the y direction, or the filters may be applied in the reverse order.
[0080] This method is faster than typical methods that use two-dimensional convolution in real space or filtering on the Fourier plane. This is particularly noticeable when the number of pixels in a single image is very large, as in this case. Although the low-pass filter shape on the Fourier plane is not isotropic, this is not a practical problem. The processing unit 35 can use, for example, a Butterworth filter as the IIR low-pass filter.
[0081] A part or all of the processing in the processing device 35 may be executed by a computer program. The above-mentioned program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of transient computer-readable media. Examples of transient computer-readable media include electric signals, optical signals, and electromagnetic waves. The transient computer-readable medium can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.
[0082] 9 is an image showing the processing result obtained by the computational sectioning. In FIG. 9, the image data f 0 (x, y) indicates the image data captured by the camera 32. Furthermore, in FIG. 9, the image data f 0 To the right of (x, y), the estimated baseline image data L j (x, y) and image data in the focal plane (in focus). 0 (x, y) to image data f j By subtracting (x, y), image data within the focal plane is calculated.
[0083] The processing device 35 separates the image data of the captured image into baseline image data and image data within the focal plane through the above-described processing. Below the image data (raw data) of the captured image, line profiles of the image data of the captured image, the baseline image data, and the image data within the focal plane are shown. In this way, background light components from outside the focal plane can be removed by computational sectioning.
[0084] Demonstration results of 3D imaging obtained by optical sectioning and computed sectioning are shown in Figures 10 to 12. As shown in Figure 10, the sample 20 is a 1.5 mm-thick coronal section of a mouse brain. This section was placed on a transparent glass container 25. The mouse brain was chemically cleared using the tissue clearing reagent CUBIC, and cell nuclei were specifically stained with SYTOX-Green. To cover the entire coronal plane, a 1x relay lens 34 was employed, resulting in an overall magnification of 2x, and the sample 20 was imaged.
[0085] The exposure time for one z-plane was 1 second, and the sample stage 22 scanned the 1.5 mm-thick sample 20 in the z-direction in 4 μm steps. The imaging device 100 then acquired z-stack data for 378 layers. The imaging device 100 acquired images of the sample 20 using both optical and computational sectioning. Figure 11 shows a fluorescence image of one z-plane (z = 700 μm) of the 378 layers. A coronal cross-section approximately 1.2 cm x 0.8 cm wide was clearly observed with single-cell resolution.
[0086] Image C in Figure 12 shows an enlarged view of the xy plane (coronal plane) and xz (transverse plane) and yz (sagittal plane) cross sections of the dashed rectangular area in Figure 11. The xz and yz cross sections clearly show that tomographic images in the z direction have been acquired. Image C in Figure 12 shows processed data after computational sectioning.
[0087] On the other hand, image D in Figure 12 is a 3D representation of the image before computational sectioning (raw data obtained by optical sectioning). While optical sectioning alone allows for surface-selective imaging, strong background light is present. In particular, in areas with high cell density, strong background light is present in the images of the areas before and after the area. By removing this background light through computational processing, we were able to obtain a clear, high-contrast image, as shown in image C in Figure 12.
[0088] In a 1.5 mm thick section, the characteristic cellular distribution of choroid plexus epithelial cells is clearly visualized in three dimensions. Image E in Figure 12 compares three z-plane images in an xy region including the choroid plexus region. Image E in Figure 12 shows three cross-sectional images at z = 700 μm, 720 μm, and 740 μm. The change in structure depending on the position in the z direction is clearly evident.
[0089] Image F in Figure 12 is an isosurface representation of nuclei within a 200 μm × 200 μm × 200 μm cube in the choroid plexus region. Due to the low spatial resolution in the z direction, individual nuclei appear elongated in the z direction, but in three-dimensional space they are spatially separated from surrounding nuclei. The three-dimensional structures of the hippocampal dentate gyrus granule cells and medial habenula, which are known to be related to memory formation and depression, are also clearly visualized (Image C in Figure 12).
[0090] The imaging device 100 uses a confocal optical system, so it can eliminate background light even when imaging with a wide field of view. It can also improve the resolution in the optical axis direction to 15 μm. It can image the sample 20 with a wide field of view with a total magnification of 4x or less.
[0091] The confocal imaging method of this embodiment includes the steps of generating illumination light L1 to illuminate a sample 20, making the illumination light L1 incident on a multi-pinhole plate 40 arranged at a position conjugate with the sample 20, focusing the illumination light L1 that has passed through multiple pinholes in the multi-pinhole plate onto the sample 20 using an objective lens 16, making signal light from the sample 20 incident on the objective lens 16, imaging the signal light from the sample 20 onto the multi-pinhole plate 40 using an imaging lens arranged between the objective lens 16 and the multi-pinhole plate, splitting the signal light that has passed through the multiple pinholes 40a from the illumination light heading toward the multi-pinhole plate using a beam splitter, imaging the signal light split by the beam splitter onto an imaging device using a telecentric lens, driving the multi-pinhole plate to change the incident position of the illumination light on the sample, and imaging the sample using the signal light detected by the imaging device when the multi-pinhole plate is driven.
[0092] The invention made by the inventor has been specifically described above based on an embodiment, but it goes without saying that the present invention is not limited to the above embodiment and can be modified in various ways without departing from the gist of the invention.
[0093] This application claims priority based on Japanese Patent Application No. 2023-077341, filed on May 9, 2023, the disclosure of which is incorporated herein in its entirety.
[0094] REFERENCE SIGNS LIST 100 Imaging device 10 Light source 11 Uniformizing optical system 13 Conjugate surface 14 Lens 15 Dichroic mirror 16 Objective lens 17 Mirror 20 Sample 21 Cell incubator 22 Sample stage 30 Lens system 31 Imaging lens 32 Camera 33 Filter 34 Relay lens 35 Processing device 40 Multi-pinhole plate 41 Driving mechanism L1 Illumination light L2 Signal light
Claims
a multi-pinhole plate arranged at a position conjugate with the sample and having a plurality of pinholes through which the illumination light passes; a drive mechanism for driving the multi-pinhole plate so as to change the incident position of the illumination light on the sample; an objective lens for focusing the illumination light from the multi-pinhole plate on the sample and into which signal light from the sample is incident; an imaging lens arranged between the objective lens and the multi-pinhole plate for imaging the signal light from the sample on the multi-pinhole plate; a beam splitter for splitting the signal light that has passed through the pinholes from the illumination light heading toward the multi-pinhole plate; a telecentric lens into which the signal light split by the beam splitter is incident; and an imaging device for detecting the signal light from the telecentric lens and imaging the sample.
2. A confocal imaging device as described in claim 1, which calculates background light components by performing filtering processing using a low-pass filter on the imaging data of an image captured by the imaging device, and removes the background light components from the image data of the captured image.
3. The imaging data of the image captured by the imaging device is f 0 Image data f 0 By processing (x, y) using a low-pass filter, the surface data L 0 (x, y) is generated, and j is an integer equal to or greater than 1, and image data f is obtained by performing iterative calculations using the following formula (1). j Generate (x, y), The image data f j By processing (x, y) using the low-pass filter, image data L indicating the background light component is obtained. j (x, y) is calculated, and the image data f of the captured image is calculated. 0 From (x, y), the image data L j The confocal imaging apparatus according to claim 2 , wherein the background light component is removed by subtracting (x, y).
4. A confocal imaging device as described in claim 3, wherein the low-pass filter is a one-dimensional infinite impulse response filter, and the background light component is calculated by sequentially applying the one-dimensional infinite impulse response filter in the x and y directions.
5. A confocal imaging device according to claim 1 or 2, wherein the magnification of the image of the sample formed by the imaging lens is 2x or less.
6. A confocal imaging device as described in claim 1 or 2, wherein the pinhole has a diameter of 15 μm or less.
7. A confocal imaging method comprising the steps of: generating illumination light for illuminating a sample; making the illumination light incident on a multi-pinhole plate arranged at a position conjugate with the sample; focusing the illumination light transmitted through a plurality of pinholes in the multi-pinhole plate on the sample by an objective lens; making signal light from the sample incident on the objective lens; imaging the signal light from the sample on the multi-pinhole plate by an imaging lens arranged between the objective lens and the multi-pinhole plate; splitting the signal light that has passed through the plurality of pinholes from the illumination light heading for the multi-pinhole plate by a beam splitter; imaging the signal light split by the beam splitter on an imaging device by a telecentric lens; driving the multi-pinhole plate to change the incident position of the illumination light with respect to the sample; and imaging the sample using the signal light detected by the imaging device when the multi-pinhole plate is driven.
8. A confocal imaging method as described in claim 7, further comprising the steps of: calculating background light components by performing filtering processing using a low-pass filter on image data of an image captured by the imaging device; and removing the background light components from the image data of the captured image.
9. Image data of the image captured by the imaging device is f 0 Image data f 0 By processing (x, y) using a low-pass filter, the surface data L 0(x, y) is generated, and image data f is obtained by performing iterative calculations using the following formula (1) where j is an integer equal to or greater than 1. j Generate (x, y), The image data f j By processing (x, y) using the low-pass filter, image data L indicating the background light component is obtained. j (x, y) is calculated, and the image data f of the captured image is calculated. 0 From (x, y), the image data L j The confocal imaging method according to claim 8 , wherein the background light component is removed by subtracting (x, y).
10. The confocal imaging method according to claim 9, wherein the low-pass filter is a one-dimensional infinite impulse response filter, and the background light component is calculated by sequentially applying the one-dimensional infinite impulse response filter in the x and y directions.
11. A confocal imaging method according to claim 7 or 8, wherein the magnification of the image of the sample formed by the imaging lens is 2x or less.
12. A confocal imaging method according to claim 7 or 8, wherein the pinhole has a diameter of 15 μm or less.
Citation Information
Patent Citations
Complex type observation device
JP2010097768A
Confocal microscope device
JP2010181688A
Systems and methods for three-dimensional imaging
JP2018527607A
Fluorescence imaging apparatus
JP2023003158A
Confocal point microscope and height measuring method using this
WO2002023248A1
