Microscope and methods for SPIM microscopy

DE502014016885D1Active Publication Date: 2024-11-21CARL ZEISS MICROSCOPY GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE502014016885
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2013-12-18
Filing Date
2014-12-17
Publication Date
2024-11-21
Estimated Expiration
2034-12-17

AI Technical Summary

Technical Problem

Current methods for registering three-dimensional image stacks in Plane Illumination Microscopy (SPIM) are computationally intensive and often unreliable, requiring artificial references that can distort images or harm biological samples, and may not always produce accurate results due to the complexity of sample structures.

Method used

A manual registration process is implemented, where image stacks are aligned using maximum intensity projections along multiple axes, allowing for efficient alignment without the need for artificial markers, and enabling the use of various transformation types such as rigid, affine, or elastic transformations, which can be partially automated.

Benefits of technology

This approach allows for accurate and efficient merging of image stacks from different angles into a 3D image without the need for artificial markers, reducing computational burden and sample preparation time, while maintaining high image quality and avoiding biological interference.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to a microscope comprising an imaging lens for imaging a sample onto a detector, as well as means for illuminating the sample with a light sheet in the focal plane of the imaging lens or in a defined plane near this focal plane. The illumination means comprise an illumination source, preferably emitting coherent light.

[0002] A microscope in which the illumination beam path and detection beam path are arranged essentially perpendicular to each other and in which the sample is illuminated with a light sheet in the focal plane of the imaging objective, i.e. perpendicular to its optical axis, is designed for the examination of samples using the selective plane illumination microscopy (SPIM) method. In contrast to confocal laser scanning microscopy (LSM), in which a three-dimensional sample is scanned point by point in individual planes of varying depth and the image information obtained is subsequently combined to create a three-dimensional image of the sample, SPIM technology is based on wide-field microscopy and enables the visual representation of the sample based on optical sections through individual planes of the sample.

[0003] The advantages of SPIM technology include the increased speed with which image information is acquired, the reduced risk of bleaching of biological samples, and an extended depth of penetration of the focus into the sample.

[0004] In principle, SPIM technology involves exciting fluorophores contained in or introduced into the sample with laser light, which is shaped into a so-called light sheet. The light sheet is used to illuminate a selected plane deep within the sample, and imaging optics are used to acquire an image of this sample plane in the form of an optical section. Essentially equivalent to such excitation with a static light sheet is the rapid back-and-forth movement of a thin, rotationally symmetric laser beam in the focal plane of the imaging lens. Effectively, i.e., averaged over the observation period, this also results in the shape of an SPIM light sheet.

[0005] The SPIM technology is described, for example, in Engelbrecht et al., Optics Letters 31, 1477 (2006), in Huisken et al., Science 305, 1007 (2004), in DE 102 57 423 A1 and in WO2004 / 0530558 A1.

[0006] The described "light sheet microscopy" combines optical sectioning with wide-field detection via a spatially resolved camera (CCD camera) by illuminating the entire lateral focal plane (xy plane) of the detection objective with a thin light sheet. The light sheet illumination is perpendicular to the detection axis (z axis).

[0007] The sample is placed in the overlapping area of ​​illumination and detection. Fluorescence signals excited by the illumination light sheet are imaged onto the camera across the entire field of view of the detection lens. The perpendicular illumination with a thin light sheet illuminates only a small portion of the axial extent of the detection optics, thus creating an optical section. To observe another area of ​​the sample, the sample is moved through the light sheet using a sample positioning unit, independently of the optics. By capturing optical sections at various sample positions along the detection axis, it is possible to acquire three-dimensional image stacks. These image stacks can then be reconstructed into a 3D image.

[0008] For example, an image stack contains 200 images.

[0009] In Fig. 1The basic structure of an SPIM microscope for implementing the inventive method is shown. A sample P, which can be located in a sample chamber PK, with the sample or sample chamber being adjustable in X, Y, and Z directions and rotatable about the Y-axis perpendicular to the plane of the drawing, is detected by a detection beam path. This consists of a detection objective O, whose detection axis or optical axis runs in the Z direction and to which a preferably replaceable light filter F, a tube lens TL, and a surface detector CCD are connected in the detection direction.

[0010] Essentially perpendicular to the detection axis Z, here in the X-direction, is an illumination beam path, here consisting of two lasers L1, L2 coupled via a beam splitter BS, which penetrate the sample via an AOTF for wavelength selection and intensity adjustment, a deflection mirror S and a beam expander BE as well as anamorphic optics such as here a cylindrical lens ZL for beam shaping, which generate a planar light distribution.

[0011] A BLjust adjustment unit is shown schematically, which can, for example, adjust or tilt the elements S, BE and ZL in several directions.

[0012] A common control and monitoring unit CU, usually formed by a computer and a display unit (display), is connected to all adjustment devices such as the AOTF, the sample adjustment or sample chamber adjustment (enables adjustment of the sample or sample chamber in X, Y and Z as well as rotation around Y) and the illumination adjustment BLjust in order to be able to carry out the method according to the invention.

[0013] From US 2013 / 0094755 A1 a method for three-dimensional imaging of a sample is known, in which image information from different levels in the depth of the sample is stored in a spatially resolved manner and the three-dimensional image of the sample is then reconstructed from this image information.For this purpose, a reference object is positioned next to or in the sample, the detection direction from which the image information is acquired is changed several times relative to the sample and the reference object, image information from the reference object is acquired from each of the detection directions specified with a change and image information from different levels in the depth of the sample is acquired and, based on the spatial coordinates X, Y, Z of the image information stored by the reference object, transformation operators for translation, rotation and deformation are obtained, with which image information of the same origin from the different detection directions are made to overlap, and these transformation operators are then used as the basis for the reconstruction of the three-dimensional image of the sample from the spatial coordinates X, Y, Z of the image information acquired from the sample.

[0014] US 2010 0177190 A1 discloses a microscopy system that allows image stacks to be acquired by moving the focal plane of an image focusing module. The system allows multiple image stacks to be acquired from different viewing angles, which can then be merged to achieve better resolution in one dimension of the sample. To acquire image stacks from different viewing angles, a sample stage on which the sample is mounted can be rotated at any angle.

[0015] WO 2012 122027 A2 discloses a module for converting a microscope into an SPIM microscope, as well as a converted SPIM microscope. The converted SPIM microscope has an excitation and a detection objective, wherein both the excitation and detection objectives can alternately generate light sheets that are aligned substantially perpendicular to each other and can be detected by the respective other objective. First and second images generated with the converted SPIM microscope can be combined by a processor to generate composite images with improved resolution.

[0016] From "Optical Sectioning Deep Inside Live Embryos by Selective Plane Illumination Microscopy" (Huisken J. et al., Science, Vol. 305, No. 5686, 2004), a method for SPIM microscopy is known in which a rotatably mounted sample can be excited from different angles with a light sheet. Images acquired at different illumination angles are cropped, scaled, rotated to reduce background artifacts, filtered, and correlated. A translation is then applied to the images, determined by an automatically generated correlation peak. The images are subsequently fused to extract high-resolution features from individual images and combine them into a single dataset.

[0017] From US 2011 / 0115895 A1, an SPIM microscope is known which comprises a beam path switch for generating alternating light beams and two illumination units which generate two counter-rotating, coplanar, alternating light sheets from the alternating light beams in order to illuminate the focal plane of the detection unit from opposite directions. Problem and inventive solution

[0018] The sample is placed in the overlapping area of ​​illumination and detection. Fluorescence signals are imaged onto a camera across the entire field of view of the detection objective. The perpendicular illumination with a thin light sheet illuminates only a small axial portion of the detection volume, thus creating an optical section. To observe another region of the sample, the sample is moved through the light sheet using a sample positioning unit, independent of the optics. By capturing optical sections at different sample positions along the detection axis, the generation of three-dimensional image stacks is possible. Such image stacks can be acquired from different angles. An image stack acquired at a specific angle is also referred to as a view.Multiple image stacks captured at a specific angle can be combined into a single 3D image that incorporates the image information from all image stacks. This process is called registration.

[0019] Currently, registration methods are used that automatically combine the individual acquisition angles into a single image using references (e.g., fluorescent "beads" as artificial landmarks) or grayscale intensities. This involves a view, referred to as the reference image, and additional views from different angles that are searched for correspondences to the reference image. Once sufficient correspondences have been found, transformation matrices can be calculated. These are then used to transform each view to the reference image. Finally, all views are merged into a single resulting image.

[0020] Automatically finding correspondences is a computationally intensive process that does not always yield the desired results, as depending on the sample, incorrect or no correspondences may be found. Even the use of fixed references (such as beads), which generally allow for better registration, cannot always be guaranteed, as this requires significantly more time-consuming sample preparation. Furthermore, beads are often undesirable by users, as they can lead to image distortion or have a biological impact on the sample.

[0021] Manual registration, or rather, controlling the process of finding correspondences, is one solution to these problems. Due to the three-dimensional nature of the images and the resulting many degrees of freedom (translation, rotation, scaling, shearing, etc.), a simple and fast procedure is necessary.

[0022] The solution to this problem is an at least partially manual registration procedure that eliminates the disadvantages of the three-dimensionality of the individual acquisition angles. First, the coordinate systems of all image stacks (views) are rotated around the Y-axis by their acquisition angle. The image stacks now appear as if they were acquired from the same acquisition angle.

[0023] Manual alignment and positioning can now be performed by overlaying the image stacks. This positioning takes place in at least two steps and is achieved by maximum intensity projections of the image stacks.

[0024] In the first step, the maximum intensities of the image stacks are projected along the Z-axis. The resulting 2D images of the various image stacks are then positioned onto the 2D image of the reference angle using appropriate transformations (e.g., rigid, affine, or elastic). This transformation is then applied to the corresponding image stacks (thus in 3D).

[0025] In the second step, the maximum intensities of the image stacks are projected along the Y-axis, and the resulting 2D images of the various image stacks (views) are again positioned onto the 2D image of the reference angle using appropriate transformations. The resulting transformation is again applied to the corresponding image stacks.

[0026] The result is a fused image stack that was manually registered from various angles. By positioning in 2D, this can also be done manually with high efficiency and does not require complex automatic

[0027] Registration method. Furthermore, it is not necessary to insert beads into the sample, as positioning can be achieved using sample structures themselves.

[0028] The manual registration process is not limited to the solution described above. In particular, the following modifications are possible: In addition to the previously described projections along the Z and Y axes, projections along the X axis can also be used for alignment. Alignment only requires any sequence of at least two projections, preferably orthogonal to each other. This means that, for example, alignment can alternate between top view, frontal view, and side view. Various transformation types can be used to align the individual projections. These include, for example, rigid, affine, elastic, or locally elastic transformations. A restriction of the degrees of freedom is possible. The described workflow can be partially automated. For example, it is possible to align the projections automatically. Instead of the maximum intensity projection, other projection variants are possible. For example, the subvolume maximum intensity projection or the minimum intensity projection.Images whose views are composed of multiple (spectrally different) channels can also be registered manually. To do this, the same channels must be used to align the various views. Manual multi-view registration can be applied without restriction to images from a time series. For example, it is conceivable that the first time point is registered manually, and the resulting transformation matrices are then automatically applied to all subsequent time points.

[0029] The invention is further explained below with reference to the schematic representations. Fig. 2 shows the basic sequence of the method according to the invention in process steps S1 - S11: S1. Taking a stack of images from at least two directions. S2. Align one image stack to the coordinates of the other. S3. Overlaying the image stacks. S4. Projection from three-dimensional space into a two-dimensional representation. S5. Position correction of the sample features included in the image, recorded from different directions, relative to each other. S6. Determination of a transformation matrix from the position correction and application to the overlaid image stack from S3. S7. Changing the view of the overlaid image stack, which is corrected in X and Y. In Fig. 3 Top view is selected. S8. Projection from three-dimensional space into a two-dimensional representation. S9. Re-alignment of the sample features contained in the image relative to each other. S10. Determination of a transformation matrix from the position correction and application to the overlaid image stack from S7. S11. Changing the view of the image stack from S10 to the original view from S3 (in the example, front view).

[0030] Based on the above-mentioned procedural steps, Fig. 3-5 the process is explained in more detail.

[0031] In Fig. 3 The sample chamber PK is shown (see Fig.1 ), which can rotate around a vertical axis.

[0032] By rotating the sample chamber and moving the light sheet through the sample by adjusting the Z axis of the sample chamber and / or the light sheet, the lens captures image stacks created at different light sheet illumination angles. For example, as shown, the image can be captured using the illumination axes z and z', which are perpendicular to each other.

[0033] Other different recording angles are also possible without restriction, including, for example, three shots with an angle offset of 30 degrees.

[0034] When moving through the light sheet in the Z direction, stacks of recorded images are created which are stored in the image memory (CU in Fig.1 ).

[0035] Schematically shown here is a stack of individual images ST1 and ST2, acquired at angles of 0 and 90 degrees. Different details of the sample are visible in ST1 and ST2, respectively.

[0036] The three individual images are selected here only for the sake of clarity; there could be, for example, 1000 individual images per stack.

[0037] The number of individual images per stack does not have to be identical in order to carry out the method according to the invention.

[0038] In step S2 in Fig. 3 The orientation of the stack ST2 is now adjusted to the orientation of the stack ST1, ie its coordinate system is rotated by 90 degrees along the displayed y` axis.

[0039] In the next step S3, the similarly oriented image stacks ST1 and ST2 are superimposed into a single stack ST1+ST2. The exact order of the individual images is not important; for example, ST2 can be arranged behind ST1.

[0040] From the above-mentioned fundamentally possible different number of individual images of ST1 and ST2, it follows that the Z-distances between the individual images can also be different during recording.

[0041] The image stacks may also have a lateral offset or the individual images in ST1 and ST2 may have different masses in the lateral direction.

[0042] In Fig. 4 In step S4, a projection from the three-dimensional into a two-dimensional representation takes place.

[0043] Here, the image pixel of a single image in the Z direction (axial) with the greatest intensity is determined pixel by pixel from the superimposed images in the Z direction, for example, based on the image pixels of an image from ST1. Instead of a maximum intensity, a specific intensity threshold can be selected, or the minimum intensity in the Z direction can be determined.

[0044] If this is performed for all image pixels, a two-dimensional representation ST1 + ST2 2D is created after step S4, which, as shown schematically, contains image information from both ST1 and (dashed) ST2. This information can be presented to the user in a distinguishable manner.

[0045] In step S5, the user uses input devices (CU in Fig.1) or automatically, or first roughly automatically and then finely by the user, a shift and / or rotation of the individual images ST1 and ST2 in the 2D representation, here in the X / Y plane, thereby correcting the image position of ST1 and ST2 relative to each other. For this purpose, the two image information from ST1 and ST2 are separated from each other in the 2D plane and arranged in a movable / rotatable, generally transformable manner.

[0046] The performed translations / rotations are recorded in CU and converted into a mathematical coordinate transformation for the X / Y coordinates in the two-dimensional representation.

[0047] This can, for example and without restriction, be an affine transformation.

[0048] This is applied to the superimposed image stack ST1+ST2 as it existed before step S4 (step S6), so that a three-dimensional image stack corrected with respect to X / Y is obtained. Fig. 5Starting from this image stack in step S7, a reorientation in ST1+ST2 VO is carried out and a top view in the direction of the Y axis is created. This still contains structural features of ST1 and ST2 and, analogously to step S4, a two-dimensional representation is created from the three-dimensional representation by intensity analysis in step S8, but now in the Y direction. In this subsequent two-dimensional representation, distinguishable and individually movable / rotatable sample features from ST1 and ST2 are located in the X / Z plane. In step S9, again analogous to S5, the automatic and / or manual alignment and overlap with one another takes place by determining a coordinate transformation which, analogous to S6, leads to a three-dimensional representation in S10, which is now aligned both in the X / Z direction and previously in the X / Y direction with regard to the sample details and is suitable for further representation orStorage and later examination is available in CU.

[0049] In a step S11, the return to the original orientation as in step S3 with the axial direction in the Z direction can be carried out in order to better reflect the original recording conditions.

[0050] As mentioned above, there may also be more than two stacks of images, for example three each taken at an angle of less than 30 degrees.

[0051] However, this does not lead to an increase in the process steps described above, but only in the two different orientations as in Fig. 3 shown more than two image stacks superimposed and reduced to two coordinates.

[0052] The tool for the user can be a standard graphical user interface (GUI) on a screen with, for example, several sliders for X, Y and Z translation as well as for rotation.

Claims

1. Method of SPIM microscopy using a microscope, consisting of - an illumination device comprising an illumination light source (L1, L2) and an illumination beam path for illuminating a sample (P) with a light sheet, - a detection device (CCD) for detecting light radiated from the sample (P) using an objective (0), - the light sheet being substantially planar in the focus of the objective (O) or in a defined plane in the vicinity of the focus of the imaging objective (O) and the objective (O) having an optical axis (OA) intersecting the plane of the light sheet at an angle that is different from zero, preferably perpendicularly, - the sample (P) being displaced through the light sheet in the direction of the optical axis (OA) of the objective (O) for the purpose of detecting different sample planes and an illumination of the sample (P) being implemented at at least a first and a second illumination angle and a plurality of sample planes being detected at each illumination angle and being stored as at least first and second image stacks (ST1, ST2) containing sample structures, characterized in that A. there is a manual alignment of the image stacks (ST1, ST2) in relation to one another by virtue of the coordinate systems of all image stacks (ST1, ST2) being aligned into the coordinate system of the first image stack (ST1), B. the at least first and second image stacks (ST1, ST2) are overlaid to form an overlaid image stack (ST1+ST2), C. there is a projection of the overlaid image stack (ST1+ST2) into a two-dimensional representation, D. sample structures detected from different illumination angles and contained in the two-dimensional representation are corrected in terms of their pose in relation to one another, E. a coordinate transformation is ascertained from the coordinates of the pose correction of the sample structures, F. the coordinate transformation is applied to the overlaid image stack (ST1+ST2) for alignment purposes.

2. Method according to Claim 1, wherein the orientation of the overlaid image stack is modified, and steps B-F are applied to the newly oriented overlaid image stack for the purpose of generating a three-dimensionally aligned image stack.

3. Method according to Claim 2, wherein the three-dimensionally aligned image stack is aligned in terms of its orientation with the original illumination direction.

4. Method according to any of the preceding claims, wherein the coordinate transformation is a rigid or affine or elastic or locally elastic transformation.

5. Method according to any of the preceding claims, wherein there is a respective intensity comparison between the images during step C for the individual pixels of the images within the overlaid image stacks.

6. Method according to Claim 5, wherein the pixel with the maximum value or minimum value or a given threshold value or the mean value or the median is used.

7. Method according to any of the preceding claims, wherein the pose correction in accordance with step D is implemented on a computer by image analysis and / or by way of input means for a user.

8. Method according to any of the preceding claims, wherein a coarse alignment on the computer and a fine alignment by way of the input means are provided.

9. Method according to any of the preceding claims, wherein the at least first and second image stacks are detected for the purpose of registering different fluorescent markers in at least two spectral ranges.

10. Method according to Claim 9, wherein a first and a second image stack with the same or a similar spectral range are used in each case for carrying out steps A-F.

11. Method according to any of the preceding claims, wherein, in the case of timeseries recordings, steps A-F and preferably also according to Claim 2 are performed at a first time and the ascertained coordinate transformations are applied to image stacks recorded at further times.

12. Method according to any of the preceding claims, wherein, in the case of timeseries recordings, steps A-F are repeated at a number of times in the timeseries.

13. Method according to any of the preceding claims, wherein, prior to the alignment, the coordinate systems of all image stacks are rotated through their recording angle about a Y-axis such that they are represented as if they were recorded from the same recording angle.

14. Data medium having a computer program configured to carry out the method according to any of the preceding claims.

15. SPIM microscope having - an illumination device comprising an illumination light source (L1, L2) and an illumination beam path for illuminating a sample (P) with a light sheet, - a detection device (CCD) for detecting light radiated from the sample (P) using an objective (O), - the light sheet being substantially planar in the focus of the objective (O) or in a defined plane in the vicinity of the focus of the imaging objective (O) and the objective (O) having an optical axis (OA) intersecting the plane of the light sheet at an angle that is different from zero, preferably perpendicularly, characterized by a monitoring and control unit configured to carry out the method according to any of Claims 1 to 13.

16. Microscope according to Claim 15, having a graphical user interface allowing at least step D to be carried out by a user.