Method for acquiring image and related microscope

Abstract

To provide a method for acquiring images of a specimen.SOLUTION: A method includes the steps of: illuminating a specimen with an illumination beam forming at least one light sheet intersecting the specimen; detecting light emitted from the specimen along two different imaging routes, at multiple positions of the specimen in a z-direction, by using a first imaging object lens and a second imaging object lens having a first imaging focal plane and a second imaging focal plane respectively that are shifted relative to each other in the z-direction and that intersect the specimen; and obtaining corresponding first and second stacks of images, each stack representing multiple planes of the specimen in the z-direction, and each image representing one plane of the specimen in the z-direction.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a method for acquiring a 2D image of a sample, a method for acquiring a 3D image of a sample, a control unit configured to control the microscope system accordingly, and a microscope system comprising said control unit. The present disclosure also relates to a method for generating a 3D image of a sample. [Background technology]

[0002] In a microscope with two imaging objectives and two corresponding imaging paths, the imaging objectives can be positioned so that the imaging focal planes of the two (e.g., opposing) imaging objectives overlap, i.e., are identical. Illumination (e.g., a light sheet) can be configured to illuminate a plane such that images of this same plane can be recorded simultaneously through the two imaging objectives, thus generating two simultaneous (opposing) views of the same plane.

[0003] However, when an object / sample (e.g., cells, hydrogels) is placed between an imaging objective and its focal plane, the focal plane may be translated / shifted depending on the refractive index of the sample (more specifically, depending on the difference between the refractive index of the sample and the refractive index of the medium in the absence of the sample). Thus, when the sample to be imaged is placed, the two focal planes (i.e., the focal planes of each imaging objective) may not overlap, as the focal planes are shifted / offset due to the presence of the sample. The shift in the focal planes may occur depending on the specific samples placed between the two objectives, especially when different samples with different refractive index properties are imaged one after the other in a microscope.

[0004] If the shift in the imaging focal plane is not compensated for in the imaging path, for example by moving the objective lens, tube lens, or camera, the illumination may not simultaneously illuminate the two imaging focal planes due to an offset between the focal planes. Not correcting for the shift in the imaging focal plane may degrade the quality of the recorded views and / or the 3D images reconstructed therefrom.

[0005] Therefore, it is desirable to improve the quality of images that can be obtained with light sheet microscopes. Summary of the Invention [Means for solving the problem]

[0006] The present invention solves the above-mentioned problems and provides a method for acquiring a 2D image of a sample, the method comprising the steps of: illuminating the sample with an illumination beam (extending along an illumination path) that forms at least one light sheet that intersects the sample; detecting light emitted from the sample along two different imaging / detection paths (i.e., the first detection path and the second detection path) at a plurality of positions of the sample in the z direction by a first imaging objective and a second imaging objective, respectively, having first and second imaging focal planes that are shifted relative to each other in the z direction (to image the sample from different viewpoints / directions) and that intersect the sample, to obtain corresponding first and second stacks of images, each stack representing a plurality of planes of the sample in the z direction (corresponding to a plurality of positions of the sample), and each image representing one plane of the sample in the z direction; and for a plurality of positions of the sample in the z direction (for each different position, and optionally for each position), the light sheet is detected according to a first adjustment setting (preferably, the light sheet is incident on a first imaging objective of the first imaging objective). adjusting the light sheet relative to a first imaging focal plane of the first imaging objective (preferably so that the light sheet overlaps with the imaging focal plane or so that the light sheet illuminates at least a portion of the depth of field of the first imaging objective or so that the light sheet is (substantially) coplanar with the first imaging focal plane of the first imaging objective) and adjusting the light sheet relative to a second imaging focal plane of the second imaging objective (preferably so that the light sheet overlaps with a second imaging focal plane of the second imaging objective or so that the light sheet illuminates at least a portion of the depth of field of the second imaging objective or so that the light sheet is (substantially) coplanar with the second imaging focal plane of the second imaging objective). The method includes a preceding adjustment step of adjusting the light sheet relative to a second imaging focal plane of a second imaging objective lens (so that it is (substantially) coplanar with the image focal plane), and for image acquisition, the sample is sequentially illuminated according to the first adjustment setting and the second adjustment setting, and the emitted light is detected via the first imaging objective lens and the second imaging objective lens, respectively, thereby acquiring a first stack and a second stack of images, wherein the first stack and the second stack are characterized by a relative shift of their planes in the z direction.

[0007] In embodiments of the present disclosure, the first and second imaging objectives, each having a first and second imaging focal plane, can be considered to be shifted relative to each other in the z-direction across the sample to image the sample from different viewpoints / orientations, the amount of shift depending on the refractive index mismatch between the sample and the surrounding medium.

[0008] The second adjustment setting may be different from the first adjustment setting, but the first and second adjustment settings may optionally be identical, especially if focal plane shifts due to refractive index mismatches between the sample and the surrounding medium are small or negligible. The adjustment settings may be considered illumination adjustment settings, as they relate to the adjustment of illumination-related parameters (optionally only illumination-related parameters, i.e., excluding adjustment of imaging / detection-related parameters).

[0009] In some embodiments, the detection / imaging may be performed sequentially via the first and second imaging objectives according to the sequential light sheet generation. Such sequential detection / imaging (of 2D images) may be performed while the sample is in a specific position between the two imaging objectives, e.g., no 3D image acquisition is performed during movement of the sample along the direction of the optical axis of the first and / or second imaging objectives.

[0010] In some embodiments, more than two detection / imaging paths and corresponding imaging objectives and / or preferably more than one illumination path and corresponding illumination objectives are possible.

[0011] The adjustment can be considered as an alignment, which allows for subsequent calibration. Adjustment settings can be derived / determined and subsequently applied for each z-position of the sample, each imaging objective, and each illumination objective depending on the specific sample positioned between two opposing imaging objectives, especially if multiple different samples (each with different refractive index and / or surrounding medium properties) are subsequently imaged.

[0012] The shift can be considered as an offset or displacement (misalignment) in the z direction, particularly of the illumination light sheet.

[0013] In embodiments of the present disclosure, the adjustment can refer to the individual adjustment of the first and second imaging focal planes by the light sheet via respective first and second adjustment settings. The adjustments here can provide overlap / identity between the first imaging focal plane and the light sheet according to the first adjustment setting, and overlap / identity between the second imaging focal plane and the light sheet (optionally the same light sheet) according to the second adjustment setting. The adjustment settings can be determined / defined for each position of the sample in the z direction. For example, the light sheet can be repositioned to achieve different adjustment settings, optionally while the imaging path remains the same. In some embodiments of the present disclosure, the adjustment does not include a shift of the imaging focal plane, i.e., a shift during image acquisition, due to, for example, a shift or adjusted movement of the imaging objective relative to the microscope or a shift or adjusted movement of the camera relative to the imaging objective. This allows for efficient yet accurate image acquisition. Therefore, for example, the image quality can be improved in 3D images based on views obtained with shifted imaging focal planes.

[0014] Embodiments of the present disclosure can at least partially compensate for relative shifts between images acquired by two imaging objectives. The shifts may occur during image acquisition, and thus in the acquired 2D images (stacks of images), and this shift can be addressed to reconstruct a 3D image based on the 2D images and / or (only) during reconstruction. Specifically, for example, when multiple planes of a sample are recorded for three-dimensional imaging, if the views obtained through the two imaging objectives shift along the optical axis, the present disclosure can be used to at least partially correct this shift. Therefore, embodiments of the present disclosure can improve image quality.

[0015] Specifically, the (ideally optimal) alignment of illumination can be found for each view individually by adjusting the illumination. Then, two views captured by the respective first and second imaging objectives can be recorded sequentially by illuminating one focal plane at a time. Therefore, the image quality of the two individual views is considered optimal. Then, for example, a series of planes recorded from the two views can be cross-correlated to quantify the shift between the two views. The shift can then be corrected in subsequent processing, for example, to reconstruct a 3D image of the sample using the two views.

[0016] In other words, if the (optimal) alignment of the light sheet may be different for two imaged objects, in some embodiments of the present disclosure, it may be necessary to take into account the alignment difference here in order to obtain the best of the two views.

[0017] The light sheet can sequentially illuminate the first and second imaging focal planes during acquisition, and provides such sequential illumination, thereby eliminating the need for simultaneous illumination of the first and second imaging focal planes, i.e., simultaneous detection via the first and second detection paths.

[0018] The first imaging objective lens and the second imaging objective lens can be arranged to have focal planes that are substantially parallel or coplanar with each other, spaced apart, i.e., with a shift or offset, along the optical axis of each imaging objective lens, for example in the z direction.

[0019] Embodiments of the present disclosure may be utilized, particularly, but not exclusively, when the imaging focal planes of two imaging objectives may be separated (i.e., may not substantially overlap) due to refractive index changes caused by a sample or multiple different samples, and may also be utilized when the imaging focal planes of two imaging objectives do not overlap for other reasons.

[0020] Some embodiments of the present disclosure can be utilized for large samples or for samples in a medium with a high refractive index (eg, Matrigel).

[0021] The planes of stacks of 2D images may be shifted relative to each other, meaning that the physical plane A of the sample shown in the nth 2D image of one stack corresponds to (i.e., is substantially identical to) the mth 2D image of another stack. Thus, physical plane A is shown in image n of the first stack, while it is shown in image m of the second stack. Since the nth and mth images correspond to physical positions of the sample, the physical distance from position m to position n is referred to as the shift / offset in the z-direction.

[0022] Viewed another way, the nth image of the first stack may represent physical plane A of the sample, while the nth image of the second stack may represent physical plane B of the sample. The shift between stacks may indicate the physical distance from plane A to plane B of the sample for the same image number.

[0023] Either way, the shift can be viewed as a shift between the first and second stack of images in the z-direction. The shift can be viewed as an offset between the imaging focal plane and the stack of images.

[0024] The z-direction may be defined by a line connecting the first imaging objective and the second imaging objective, e.g., the optical axis of the imaging objective may extend in the z-direction.

[0025] The present disclosure relates to a method for acquiring a 3D image of a sample, the method including a method for acquiring an image of a sample according to the present disclosure, i.e. as described herein, wherein the 3D image of the sample is obtained based at least in part on a first stack and a second stack of images (to reconstruct the 3D image), and a shift of the planes of the first stack and the second stack relative to each other is at least partially corrected (before and / or during reconstruction) to reconstruct the 3D image of the sample. In embodiments of the present disclosure, efficient and reliable reconstruction can be achieved by compensation / correction of the shift before or during reconstruction.

[0026] Optionally, the illumination beam is switched between the first and second adjustment settings during image acquisition, which allows switching between image acquisition with the first imaging objective and image acquisition with the second imaging objective, particularly at the same sample position, thus enabling efficient image acquisition.

[0027] Optionally, the sample is illuminated sequentially according to the first and second adjustment settings at the same position of the sample in the z-direction, which can be considered as alternating acquisition by switching the illumination beam for the same sample position, which can support efficient image acquisition.

[0028] Optionally, the sample is illuminated at multiple positions of the sample in the z direction (each at a different position, optionally at each position) according to a first adjustment setting, whereby an image is acquired via a first imaging objective, and subsequently, the sample is illuminated at multiple positions of the sample in the z direction according to a second adjustment setting, whereby an image is acquired via a second imaging objective. This can be considered an alternative form of image acquisition disclosed in the previous paragraph. However, these two acquisition procedures need not be mutually exclusive. Efficient image acquisition can be provided by performing subsequent complete z-directional movements of the sample for the subsequent first and second acquisition turns, first via the first imaging objective and then via the second imaging objective, respectively.

[0029] Optionally, the shift between the first and second stack can be identified by image analysis such as image cross-correlation, which allows for a particularly reliable software-based quantification of the shift.

[0030] Further optionally, the image analysis comprises cross-correlating the images of the stacks and / or pixel-by-pixel correlation of at least some of the images of each of the first and second stacks, which represents an efficient implementation.

[0031] Optionally, the adjustment / alignment procedure includes acquiring multiple images of the sample (optionally simultaneously) via both (two different) detection paths while adjusting (and / or positioning) the light sheet (optionally continuously) along the z-direction to generate a first stack of aligned images and a second stack of aligned images, respectively, and determining the image having the highest image sharpness and / or the highest image contrast in each of the acquired first stack of aligned images and the second stack of aligned images. For example, the first and second adjustment settings are determined based on the actual position of the light sheet of the sharpest image in the first and second stacks of aligned images. This can be considered hardware-based quantification of shift and can be an alternative or addition to image analysis-based determination of shift. In this manner, the adjustment / alignment procedure is performed specifically to identify shift between the first and second stacks.

[0032] Optionally, during reconstruction, images from the first and second stacks corresponding to the same plane of the sample are combined, which can result in an improved 3D image.

[0033] Optionally, the first and second imaging paths extend substantially in opposite directions and / or the first and second imaging objectives are spaced apart in the z-direction. The first and second imaging objectives can be arranged opposite each other, e.g., with the same or at least parallel optical axes, which can improve imaging also in terms of the spatial arrangement of entities.

[0034] Optionally, a first illumination path and a second illumination path are selectively defined, in which the illumination beam passes through a first illumination objective and a second illumination objective, respectively, and forms a first light sheet and a second light sheet, respectively, and a corresponding first adjustment setting and a second adjustment setting are provided for each illumination path, and the sample is sequentially illuminated along the first illumination path according to the first adjustment setting and the second adjustment setting for the first illumination path for detection via the first imaging objective and the second imaging objective, respectively, and further along the second illumination path according to the first adjustment setting and the second adjustment setting for the second illumination path for detection via the first imaging objective and the second imaging objective, respectively. Thus, two light sheets can be generated, each light sheet having an individual first adjustment setting and a second adjustment setting (for each imaging objective).

[0035] Another option is to provide a single light source, such as a laser, to generate the two illumination paths. For example, a laser beam can be split into a first illumination path and a second illumination path. This allows for a cost-effective implementation of two light sheets.

[0036] Optionally, the first and second arrangements are provided by an adjustment device common to the first and second lighting paths. In particular, common / shared hardware components can realize the respective adjustments. This can support realization of cost and space savings.

[0037] The present disclosure can be considered to be directed to light sheet microscopy and corresponding light sheet microscopes.

[0038] The present disclosure is also directed to a control unit configured to control a microscope system to perform the methods of the present disclosure.

[0039] The present disclosure is also directed to a microscope system for imaging at least one sample, preferably a plurality of samples, comprising a control unit of the present disclosure.

[0040] The present disclosure is also directed to a method, in particular a computer-implemented method, for generating a 3D image of a sample, which may be a computer-implemented method for generating a 3D image of a sample, comprising: receiving image data including / representing a first stack and a second stack of acquired images, each stack characterized by a relative shift of planes of the sample in the z direction, each stack representing multiple planes of the sample in the z direction (obtained from different viewpoints / or directions), and each image representing one plane of the sample in the z direction; and reconstructing the 3D image of the sample based on the image data, wherein the relative shifts of the planes of the first stack and the second stack are at least partially corrected to reconstruct the 3D image of the sample and / or during the reconstruction. The first and second stacks of images can be generated by illuminating a sample with an illumination beam that forms at least one light sheet that intersects the sample; detecting light emitted from the sample along two different imaging paths at a plurality of positions of the sample in the z direction by a first imaging objective and a second imaging objective, respectively, having respective first and second imaging focal planes that are shifted relative to each other in the z direction and that intersect the sample, to obtain corresponding first and second stacks of images; and detecting the light emitted from the sample along two different imaging paths at a plurality of positions of the sample in the z direction by a first imaging objective and a second imaging objective, respectively, having respective first and second imaging focal planes that are shifted relative to each other in the z direction and that intersect the sample. and a preceding adjustment step of adjusting the light sheet to a first imaging focal plane of a first imaging objective lens according to a first adjustment setting and adjusting the light sheet to a second imaging focal plane of a second imaging objective lens according to a second adjustment setting (for positions (optionally for each position)), wherein the image acquisition further includes sequentially illuminating the sample according to the first adjustment setting and the second adjustment setting and detecting the emitted light via the first imaging objective lens and the second imaging objective lens, respectively, thereby acquiring a first stack and a second stack of images.

[0041] Because at least some of the steps may be performed on a computer system by a computer system, the method may be considered a computer-implemented method.

[0042] The present disclosure is also directed to a data processing apparatus comprising means for performing the methods of the present disclosure, a computer program (product) comprising instructions for causing a computer to perform the methods of the present disclosure when the program is executed by a computer, and a computer-readable medium having stored thereon the computer program (product) of the present disclosure.

[0043] The embodiments of the present invention can be particularly implemented for imaging multiple (different) samples simultaneously or sequentially. More preferably, the embodiments of the present invention can be particularly implemented for imaging multiple (different) samples multiple times / repeatedly sequentially. These (different) samples can be positioned in the sample holder at different spatial positions or different XYZ positions. When the calibration or adjustment step is performed for each sample, the respective shifts of the focal planes of the two imaging objectives for each (different) sample can be different, and therefore the respective adjustment settings can also be different. Preferably, the calibration or adjustment step is performed only once for all samples, and when different samples are repeatedly observed for imaging, for example in a time-lapse experiment, the respective (different) adjustment settings can be applied accordingly. Therefore, different adjustment settings can be applied for each view and sample.

[0044] Detailed embodiments and other advantages and features associated with the present disclosure are described below, but these exemplary embodiments should not be construed as limiting the invention. [Brief explanation of the drawings]

[0045] [Figure 1] FIG. 1 is a diagram illustrating a schematic diagram of a microscope system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating a schematic diagram of an image acquisition step according to an embodiment of the present disclosure. [Figure 3a] FIG. 1 illustrates two stacks of 2D images according to an embodiment of the present disclosure. [Figure 3b] FIG. 1 illustrates an analysis of correlation coefficients according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0046] 1 shows a portion of a microscope system 100 having a first illumination objective 101a and an (optional) second illumination objective 101b, and a first imaging objective 102a and a second imaging objective 102b. An illumination beam is formed along a first illumination path 105a via the first illumination objective 101a and along a second illumination path 105b via the second illumination objective 101b, with each illumination beam forming a light sheet 104. The light sheet formed by the first illumination path 105a and the second illumination path 105b is represented in this example by overlapping light sheets 104. In the following, reference is made to light sheet 104, which can be formed by a single illumination objective (if there is only one illumination path) or by a first illumination objective and a second illumination objective (if there are two illumination paths as shown in FIG. 1). Light emitted or emitted from the sample 103 along the first imaging path 106a and the second imaging path 106b is detected by respective cameras (not shown). The first imaging path 106a and the second imaging path 106b extend substantially in opposite directions. The optical axis of the first imaging objective 102a is substantially coaxial with the optical axis of the second imaging objective 102b. In FIG. 1, the sample 103 is an "ideal" or virtual sample in that there is no refractive index difference (e.g., with respect to the medium surrounding the sample). Therefore, the imaging focal plane 107a of the first imaging objective 102a and the imaging focal plane 107b of the second imaging objective 102b are identical. Specifically, the light sheet 104 formed by the first illumination objective 101a and the second illumination objective 101b is in the same plane as the first imaging focal plane 107a and the second imaging focal plane 107b. The laser light source and other components of the microscope, such as the beam splitter, are not shown in FIG.

[0047] It should be noted that once the "real" sample to be imaged is placed between the two imaging objectives 102a, 102b, the optimum alignment of the light sheet 104 can be found for each view individually by adjusting the illumination accordingly.

[0048] When an object to be imaged is placed between the two imaging objectives 102a, 102b, the imaging focal planes 107a, 107b may no longer overlap but will be shifted relative to each other, and therefore the "ideal" or virtual constellation of Figure 1 no longer applies.

[0049] For the adjustment procedure, the position of the sample 103 can optionally be fixed before the image acquisition procedure, and images are acquired for several adjustments of the light sheet 104. The sharpness of the adjusted images is then analyzed or calculated, and the adjustment corresponding to the sharpest image is set. This is performed sequentially for each view (for each imaging objective 102a, 102b). An example of an adjustment procedure is described in WO 2019 / 016359. The first and second arrangements are provided by an adjustment device (not shown) common to the first illumination path 105a and the second illumination path 105b.

[0050] Thus, for different positions of the sample 103 in the z-direction, by appropriately adjusting the illumination parameters by the adjustment device, the light sheet 104 is adjusted relative to the first imaging focal plane 107a of the first imaging objective 102a according to a first adjustment setting and further adjusted relative to the second imaging focal plane 107b of the second imaging objective 102b according to a second adjustment setting. Preferably, this can be done for each sample introduced between the first and second imaging objectives 102a, 102b. The z-direction can be substantially parallel to the optical axes of the first and second illumination objectives 102a, 102b or to the first and second imaging paths 106a, 106b.

[0051] Instead of performing all of the adjustment steps prior to all of the acquisition steps, it is conceivable to perform several adjustment steps immediately before the corresponding acquisition steps. In some cases, a single adjustment step is sufficient for all subsequent imaging steps, but multiple or re-adjustment steps are not excluded.

[0052] Subsequently, during the acquisition procedure, when the sample 103 is imaged, two views are sequentially recorded for each plane of the sample via the first imaging objective 102a and the second imaging objective 102b, respectively, using the adjustment settings determined in the preceding adjustment procedure. Continuing to refer to FIG. 1 , the sample 103 is moved in the z-direction during image acquisition, while the imaging objectives 102a and 102b remain stationary. At different / multiple (each) sample position, the sample 103 is illuminated in one plane (according to the corresponding adjustment settings), so that different / multiple planes of the sample 103 shifted relative to each other along the z-direction are imaged, i.e., recorded as 2D images.

[0053] Image acquisition involves illuminating the sample 103 with an illumination beam 105a that forms at least one light sheet 104 that intersects the sample. At multiple positions of the sample 103 in the z direction, light emitted from the sample 103 along two different imaging paths 106a, 106b is detected by first and second imaging objectives 102a, 102b, respectively, having respective first and second imaging focal planes 107a, 107b that are shifted relative to each other in the z direction and that intersect the sample 103.

[0054] More specifically, the illumination beam selectively defines a first illumination path 105a and a second illumination path 105b that pass through a first illumination objective 101a and a second illumination objective 101b, respectively, and form respective light sheets 104, and corresponding first and second adjustment settings are provided for each illumination path 105a, 105b, and the sample 103 is sequentially illuminated along the first illumination path 105a according to the first adjustment setting and the second adjustment setting for the first illumination path for detection via the first imaging objective 102a and the second imaging objective 102b, respectively, and further along the second illumination path 105b according to the first adjustment setting and the second adjustment setting for the second illumination path for detection via the first imaging objective 102a and the second imaging objective 102b, respectively.

[0055] 2 shows two views, i.e., two 2D images 208a, 208b, that are recorded sequentially by illuminating one of the first focal plane 107a and the second focal plane 107b at a time. The sample 103 is sequentially illuminated according to the first and second adjustment settings, and the emitted light is detected via the first and second imaging objectives 102a, 102b.

[0056] More specifically, FIG. 2(a) illustrates image acquisition by the first imaging objective 102a, resulting in a 2D image 208a of plane 107a of sample 103, where the plane shown in image 208a corresponds to imaging focal plane 107a. This is indicated by the bolded "active" imaging path 106a. FIG. 2(b) illustrates image acquisition by the second imaging objective 102b, resulting in a 2D image 208b of plane 107b of sample 103, where the plane shown in image 208b corresponds to imaging focal plane 107b. This is indicated by the bolded "active" imaging path 106b. FIGS. 2(a) and 2(b) illustrate the spatial difference, or z-direction distance, between imaging focal plane 107a and imaging focal plane 107b. This difference corresponds to shift 209. Hence, during acquisition, a shift appears in the relative distance of imaging focal plane 107a and imaging focal plane 107b to each other in the z direction.

[0057] The illumination beam is switched between the first and second adjustment settings during image acquisition, so that the sample 103 is illuminated sequentially according to the first and second adjustment settings at the same position of the sample in the z direction.

[0058] The sample 103 is illuminated at a plurality of positions of the sample 103 in the z direction according to a respective plurality of first adjustment settings, and subsequently illuminated at a plurality of positions of the sample 103 in the z direction according to a respective plurality of second adjustment settings. Since a respective adjustment setting per view is used for each image acquisition according to Figures 2(a) and 2(b), the image quality of the two views 208a, 208b is considered to be optimal. The first illumination objective 101a and the second illumination objective 101b are not shown in Figure 2.

[0059] 3(a) shows two stacks of 2D images 310a, 310b, including image 208a and image 208b, respectively. A first stack 310a and a second stack 310b of images are acquired, each stack representing multiple planes of the sample in the z-direction, and each image 208a, 208b representing one plane of the sample in the z-direction. The first stack 310a and the second stack 310b of images are acquired, where the first stack 310a and the second stack 310b are characterized by a relative shift 209 of their planes in the z-direction.

[0060] A shift 209 in the z-direction between the imaging focal planes appears in the acquired 2D images as a corresponding shift 209 in the correspondence between the 2D images of the two stacks. More specifically, comparing stacks 310a and 310b acquired according to FIGS. 2(a) and 2(b), respectively, by comparing the 2D images of the stacks with each other can mean that the sequence of 2D images n=1, 2, 3, ... of the first image stack 310a is compared with the sequence of 2D images n=1, 2, 3, ... of the second image stack 310b. This comparison reveals that image 208a of the first stack 310a and image 208b of the second stack 310b correspond to each other in that they show the same plane of the sample 103. Optionally, image 208a of the first stack 310a and image 208b of the second stack 310b may differ in that they show the same plane of the sample 103 from two different perspectives. Since image 208a is the seventh image in the first stack 310a, n=7, and image 208b is the fourth image in the second stack 310b, n=4, then n-n=3, and the shift corresponds to 3 images. Thus, shift 209 indicates the offset between stacks 310a and 310b, which contain 3 images.

[0061] FIG. 3(b) illustrates one way of performing the above comparison to determine, i.e., quantify, the shift 209. Specifically, a sequence of planes recorded from two views is cross-correlated to quantify the shift 209 between the two views. Specifically, a correlation coefficient between at least some pixel values ​​of the first stack 310a and at least some pixel values ​​of the second stack 310b to which the shift 209 has been applied along the z direction is calculated for multiple values ​​of the shift 209. The maximum value of the correlation coefficient as a function of the applied shift 209 in the z direction indicates the shift 209. In the example illustrated in FIG. 3(a), the shift 209 is three planes.

[0062] Thus, the shift 209 between the first stack 310 a and the second stack 310 b is identified by image analysis, which may include, for example, cross-correlating the images 208 a, 208 b of the stacks 310 a, 310 b, optionally including pixel-by-pixel correlation of at least some of the images of each of the first and second stacks.

[0063] For example, to find the shift 209, the similarity between the first stack 310a and shifted versions of the second stack 310b can be determined. Thus, the similarity is greatest when the shift applied to the second stack 310b corresponds to a physical misalignment, i.e., shift or offset, between the first imaging focal plane 107a and the second imaging focal plane 107b. Illustratively, the similarity can also be calculated as a pixel-by-pixel correlation coefficient between the stacks 310a and 310b.

[0064] Alternatively, the shift 209 between the first stack 310a and the second stack 310b is identified based on an alignment procedure, which includes acquiring a plurality of images of the sample via two different imaging paths 106a, 106b while adjusting the light sheet 104 along the z-direction to acquire a first stack of aligned images and a second stack of aligned images, respectively, and determining an image in each of the acquired first stack of aligned images and second stack of aligned images that has at least one of the highest image sharpness and the highest image contrast.

[0065] This shift 209 is then corrected in subsequent processing (e.g., fusion of the two sequences / stacks of views) to reconstruct a three-dimensional image of the sample 103, for example using the two views 208a, 208b. For each plane of the sample, one can also select the sharpest one between the two views of these planes to reconstruct an optimal 3D image of the sample.

[0066] By comparison, if the imaging focal planes 107a, 107b of the two views overlap (i.e., the shift is zero), the 3D stacks from the two views will be perfectly coincident, i.e., the nth image of the first stack 310a will be in the same plane as the nth image of the second stack 310b.

[0067] In an embodiment of the present invention, once the misalignment has been corrected, i.e., the shift 209 has been quantified and corrected, an optimal 3D image of the sample can be created, for example, by selecting, for each plane of the sample 103, the best of the two views or the best pixel from a pixel-by-pixel analysis of the two views.

[0068] Specifically, the reconstruction can be performed by the computer-implemented method described above. During reconstruction, an image 208a from a first stack 310a and an image 208b from a second stack 310b corresponding to the same plane of the sample are combined. In particular, a computer-implemented method for generating a 3D image of a sample 103 includes receiving image data including a first stack 310a and a second stack 310b of acquired images 208a, 208b, the stacks 310a, 310b being characterized by relative shifts of planes of the sample 103 in the z direction, each stack 310a, 310b representing multiple planes of the sample in the z direction, and each image 208a, 208b representing one plane of the sample in the z direction. Subsequently, a 3D image of the sample 103 is reconstructed based on the image data, at least partially correcting the relative shifts 209 of the planes of the first stack and the second stack 310a, 310b.

[0069] As exemplarily reflected in FIGS. 2, 3(a) and 3(b), an embodiment of the present invention performs at least steps (1) to (3) of the following steps, and optionally performs other steps: (1) for different positions of the sample 103 in the z-direction (for multiple positions, optionally for each position), the light sheet 104 is adjusted relative to a first imaging focal plane 107a of the first imaging objective 102a according to a first adjustment setting, and the light sheet 104 is adjusted relative to a second imaging focal plane 107b of the second imaging objective 102b according to a second adjustment setting; (2) the sample 103 is illuminated with at least one illumination beam 105a, 105b forming at least one light sheet 104 intersecting the sample 103, and light emitted from the sample 103 along two different imaging / detection paths 106a, 106b (i.e., a first detection path and a second detection path) at multiple positions of the sample in the z direction (to image the sample from different viewpoints / directions) is detected by a first imaging objective 102a and a second imaging objective 102b, respectively, having first imaging focal planes 107a and second imaging focal planes 107b that are shifted relative to each other in the z direction and intersect the sample 103, the sample is sequentially illuminated according to the first adjustment setting and the second adjustment setting, and the emitted light is detected via the first imaging objective 102a and the second imaging objective 102b, respectively; (3) corresponding first and second stacks 310a and 310b of images are acquired, each stack representing multiple planes of the sample 103 in the z direction, each image 208a, 208b representing one plane of the sample 103 in the z direction, and the first and second stacks 310a and 310b of images are characterized by a mutually relative shift 209 of the planes in the z direction; (4) A 3D image is reconstructed based at least in part on the first stack 310a and the second stack 310b of images, and a relative shift 209 of the planes of the first stack and the second stack relative to each other is at least partially corrected for reconstructing the 3D image of the sample 103 and / or during the reconstruction.

[0070] Embodiments of the present disclosure can be considered to be characterized by a specific alignment of illumination for each view, sequential recording of views each recorded with a corresponding illumination alignment, and calculation of misregistration between views based on the similarity between the 3D stacks recorded from the two views.

[0071] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0072] While some aspects have been described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or apparatus corresponds to a step or feature of a step, and similarly, aspects described in the context of a step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0073] Some embodiments relate to the microscope system 100 described in connection with FIG. 1 . The microscope may be part of the system 100. FIG. 1 shows a schematic diagram of the system 100 configured to perform the methods described herein. The system 100 includes a microscope and a computer system (not shown). The microscope is configured to capture images and is connected to the computer system. The microscope system 100 may include a control unit configured to perform the data acquisition methods disclosed above. The computer system is configured to perform at least some of the methods described herein. The computer system may be configured to execute machine learning algorithms. The computer system and the microscope may be separate entities or may be integrated within a common housing. The computer system may be part of a central processing system of the microscope and / or part of a subsidiary component of the microscope, such as a sensor, actor, camera, or lighting unit of the microscope.

[0074] A computer system may be a local computing device (e.g., a personal computer, laptop, tablet computer, or mobile phone) with one or more processors and one or more storage devices, or may be a distributed computing system (e.g., a cloud computing system with one or more processors and one or more storage devices distributed at various locations, such as local clients and / or one or more remote server farms and / or data centers). A computer system may include any circuit or combination of circuits. In one embodiment, a computer system may include one or more processors, which may be of any type. As used herein, a processor may contemplate any type of computing circuit, such as, but not limited to, a microprocessor of a microscope or microscope component (e.g., a camera), a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA), or any other type of processor or processing circuit. Other types of circuits that may be included in a computer system may be custom circuits, application specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communications circuits) used in wireless devices such as cell phones, tablet computers, laptop computers, two-way radios, and similar electronic systems. A computer system may also include one or more storage devices, which may include one or more memory elements suitable for a particular application, such as main memory in the form of random access memory (RAM), one or more hard drives and / or one or more drives that handle removable media, such as compact discs (CDs), flash memory cards, digital video discs (DVDs), etc.The computer system may also include a display device, one or more speakers and a keyboard and / or controller which may include a mouse, trackball, touch screen, voice recognition device, or any other device that allows a user of the system to input information to and receive information from the computer system. Some or all of the steps may be performed by (or using) a hardware device such as, for example, a processor, microprocessor, programmable computer, or electronic circuitry.

[0075] In some embodiments, any one or more of the critical steps may be performed by such an apparatus. Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or software. The implementation may be performed by a non-transitory storage medium, such as a digital storage medium, for example, a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory, on which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system to perform the respective methods. Thus, the digital storage medium may be computer-readable.

[0076] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform any of the methods described herein.

[0077] Generally, embodiments of the present invention can be implemented as a computer program product comprising program code which operates to perform any of the methods when the computer program product is run on a computer. The program code may, for example, be stored on a machine-readable carrier. Another embodiment comprises a computer program for performing any of the methods described herein, stored on a machine-readable carrier. In other words, therefore, an embodiment of the present invention is a computer program having a program code for performing any of the methods described herein when the computer program is run on a computer.

[0078] Therefore, another embodiment of the invention is a recording medium (or data carrier or computer readable medium) containing a computer program stored thereon for performing any of the methods described herein when executed by a processor. The data carrier, digital recording medium or recording medium is typically tangible and / or non-transitory. Another embodiment of the invention is an apparatus as described herein, comprising a processor and a recording medium.

[0079] A further embodiment of the present invention is, therefore, a data stream or a sequence of signals representing the computer program for performing any of the methods described herein, the data stream or sequence of signals being for example adapted to be transmitted via a data communication connection, for example the Internet.

[0080] Another embodiment comprises a processing means, for example a computer, or a programmable logic device configured to or adapted to perform any of the methods described herein.

[0081] Another embodiment comprises a computer having the computer program installed thereon for performing any of the methods described herein.

[0082] Another embodiment of the present invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing any of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a storage device, etc. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

[0083] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, the methods may be advantageously performed by any hardware apparatus.

[0084] The detailed description is provided with reference to the embodiments illustrated in the drawings. Obvious modifications and alternatives may occur to those skilled in the art based on the remaining disclosure of the invention, particularly in the summary. These modifications and alternatives are part of the invention to the extent that they are encompassed by the appended claims. [Explanation of symbols]

[0085] 100 Microscope System 101a, 101b Illumination objective lens 102a, 102b Imaging objective lens 103 samples 104 Light Sheet 105a, 105b lighting path 106a, 106b Imaging path 107a, 107b imaging focal plane 208a, 208b 2D images 209 z-shift 310a, 310b Image stack na,nb Image number in the image stack x, y, z spatial directions

Claims

1. 1. A method for acquiring 2D images (208a, 208b) of a sample (103), said method comprising: - illuminating said sample (103) with an illumination beam (105a, 105b) forming at least one light sheet (104) intersecting the sample; detecting light emitted from the sample (103) along two different imaging paths (106a, 106b) by a first imaging objective (102a) and a second imaging objective (102b) having respective first imaging focal planes (107a) and second imaging focal planes (107b) that are shifted relative to each other in the z direction (z) and that intersect the sample (103) at multiple positions of the sample (103) in the -z direction (z), to obtain corresponding first stacks (310a) and second stacks (310b) of images, each stack representing multiple planes of the sample (103) in the z direction (z), and each image (208a, 208b) representing one plane of the sample (103) in the z direction (z); Including, - for a plurality of positions of the sample (103) in the z-direction, adjusting a light sheet (104) relative to the first imaging focal plane (107a) of the first imaging objective (102a) according to a first adjustment setting and adjusting a light sheet relative to the second imaging focal plane (107b) of the second imaging objective (102b) according to a second adjustment setting; including a preceding adjustment step, For image acquisition, the sample (103) is sequentially illuminated according to the first and second adjustment settings, and the emitted light is detected via the first and second imaging objectives (102a) and (102b), respectively, whereby a first stack (310a) and a second stack (310b) of images are acquired, the first stack (310a) and the second stack (310b) being characterized by a mutually relative shift (209) of planes in the z-direction, method.

2. A method for acquiring a 3D image of a sample (103), comprising: The method comprises a method for acquiring an image of a sample according to claim 1, a 3D image of the sample (103) is obtained based at least in part on a first stack (310a) and a second stack (310b) of images, wherein a mutually relative shift (209) of the planes of said first stack (310a) and second stack (310b) is at least partially corrected in order to reconstruct a 3D image of said sample; method.

3. the illumination beams (105a, 105b) are switched between the first and second adjustment settings during image acquisition; 3. The method according to claim 1 or 2.

4. the sample (103) is illuminated sequentially according to the first adjustment setting and the second adjustment setting at the same position of the sample in the z direction; 4. The method according to any one of claims 1 to 3.

5. the sample (103) is illuminated at a plurality of positions of the sample in the z direction according to respective first adjustment settings, and subsequently illuminated at a plurality of positions of the sample in the z direction according to respective second adjustment settings; 5. The method according to any one of claims 1 to 4.

6. a shift (209) between the first stack (310a) and the second stack (310b) is identified by image analysis; 6. The method according to any one of claims 1 to 5.

7. The image analysis includes: - cross-correlating the images (208a, 208b) of said stacks (310a, 310b); - pixel-by-pixel correlation of at least a portion of the images of each of said first and second stacks; at least one of The method of claim 6.

8. The alignment procedure is acquiring a plurality of images of the sample via the two different imaging paths (106 a, 106 b) while adjusting the light sheet (104) along the z-direction to produce a first stack of aligned images and a second stack of aligned images, respectively; determining an image having at least one of the highest image sharpness or the highest image contrast in each of the acquired first stack of aligned images and the second stack of aligned images, in particular to identify a shift (209) between the first stack (310a) and the second stack (310b) based on an alignment procedure; Including, 8. The method according to any one of claims 1 to 7.

9. During reconstruction, the images (208a) of the first stack and the images (208b) of the second stack corresponding to the same plane of the sample are combined.

9. A method according to any one of claims 3 to 8 when dependent on claim 2.

10. The first imaging path (106a) and the second imaging path (106b) extend in substantially opposite directions from each other.

10. The method according to any one of claims 1 to 9.

11. the illumination beam selectively defines a first illumination path (105a) and a second illumination path (105b) that pass through a first illumination objective (101a) and a second illumination objective (101b), respectively, to form respective light sheets (104); a corresponding first adjustment setting and a second adjustment setting are provided for each illumination path (105a, 105b), and the sample (103) is sequentially illuminated along the first illumination path (105a) according to the first adjustment setting and the second adjustment setting for the first illumination path (105a) for detection via the first imaging objective (102a) and the second imaging objective (102b), and further along the second illumination path (105b) according to the first adjustment setting and the second adjustment setting for the second illumination path (105b) for detection via the first imaging objective (102a) and the second imaging objective (102b), respectively; 11. The method according to any one of claims 1 to 10.

12. an adjustment device common to the first illumination path (105a) and the second illumination path (105b) provides the first adjustment setting and the second adjustment setting; The method of claim 11.

13. A control unit configured to control a microscope system (100) to perform the method according to any one of claims 1 to 12.

14. A microscope system (100) for imaging a sample (103), comprising a control unit according to claim 13.

15. A method, in particular a computer-implemented method, for generating a 3D image of a sample (103), said method comprising the steps of: receiving image data comprising a first stack (310a) and a second stack (310b) of acquired images (208a, 208b), each stack (310a, 310b) being characterized by a relative shift of a plane of the sample (103) in the z-direction, each stack (310a, 310b) representing multiple planes of the sample in the z-direction, each image (208a, 208b) representing one plane of the sample in the z-direction; - reconstructing a 3D image of the sample (103) based on the image data, wherein a mutually relative shift of the planes of the first stack (310a) and the second stack (310b) is at least partially corrected in order to reconstruct the 3D image of the sample; Including, said first stack (310a) and second stack (310b) of images - illuminating said sample (103) with an illumination beam (105a, 105b) forming at least one light sheet (104) intersecting said sample; - detecting light emitted from the sample (103) along two different imaging paths (106a, 106b) by a first imaging objective (102a) and a second imaging objective (102b) having respective first imaging focal planes (107a) and second imaging focal planes (107b) that are shifted relative to each other in the z direction and that intersect the sample (103) at a plurality of positions of the sample in the z direction to obtain corresponding first stacks (310a) and second stacks (310b) of images; - for different positions of the sample (103) in the z-direction, adjusting the light sheet (104) relative to a first imaging focal plane (107a) of the first imaging objective (102a) according to a first adjustment setting and adjusting the light sheet (104) relative to a second imaging focal plane (107b) of the second imaging objective (102b) according to a second adjustment setting; a preceding adjustment step; represents the image acquired by Image acquisition is - sequentially illuminating the sample (103) according to the first adjustment setting and the second adjustment setting and detecting the emitted light via the first imaging objective (102a) and the second imaging objective (102b), respectively, thereby acquiring the first stack (310a) and the second stack (310b) of images; method.

Images