Devices and methods for capturing image data

Abstract

This invention relates to an apparatus for capturing image data, comprising a detection beam path and means for separating detected radiation between a first detection path and a second detection path. Detectors are located in each detection path. A microlens array is arranged upstream of at least one detector. The first detector has a first spatial resolution, and the second detector has a second spatial resolution, the first spatial resolution being higher than the second spatial resolution. Alternatively or additionally, the first detector has a first temporal resolution, and the second detector has a second temporal resolution, the first temporal resolution being lower than the second temporal resolution. An evaluation unit is provided for evaluating the image data captured by the first and second detectors, configured such that the image data captured by each detector is computationally combined to form a three-dimensional resolved result image. The invention also relates to a microscope including the apparatus according to the invention, and a method for capturing image data.

Description

Technical Field

[0001] This invention relates to an apparatus and method for capturing image data. Background Technology

[0002] Modern microscopy increasingly emphasizes the rapid capture of three-dimensional sample volumes. An important application of this is measuring neural signals within neural networks, which are distributed over hundreds or thousands of micrometers in the brain. To understand the brain's fundamental capabilities, it is necessary to capture these networks, or at least a large portion of their responses, with high temporal resolution.

[0003] Since this involves not only understanding morphological structures but also understanding functional processes, these methods are also combined under the keywords of functional imaging.

[0004] In the field of microscopy, many different methods are known that can approximate functional imaging. Method-based approaches, such as rapid 2D recording followed by axial scanning, are generally too slow for the aforementioned applications. Methods that rely purely on algorithmic evaluation (computational imaging) are often susceptible to artifacts.

[0005] The disadvantage of confocal scanning methods with low parallelism is that they are relatively slow and operate sequentially. Increased speed is usually accompanied by increased emission power in the sample, where higher emission power can saturate the fluorescent label used and damage the sample. Another point-scanning method is multiphoton microscopy. Parallelism is also low in this case.

[0006] For example, parallelism can be improved using rotating disk microscopy. In this case, a relatively large number of sampling beams are simultaneously guided onto the sample, and then the detection radiation induced under different conditions is captured through a so-called aperture located in the rotating disk. This confocal method allows for, for example, parallel scanning of hundreds of focal volumes.

[0007] Methods and apparatuses using so-called light sheet illumination also exhibit relatively high parallelism. For this purpose, static or dynamic light sheets are generated and introduced into the sample. Because the thickness of the light sheet across its two-dimensional extent (the light sheet thickness) is very small, detection radiation, particularly fluorescence radiation, is induced only in the plane of the current illumination.

[0008] In addition to selectively illuminating the sample area, wide-field illumination can also be used when applying light field microscopy. For microscopy, detection using light field techniques allows for rapid data acquisition with larger volume and good depth resolution. However, the disadvantages include the lack of possibility for optical sectioning and strong background radiation.

[0009] By using a microlens array upstream of the detector, a relatively large volume of capture can be achieved while simultaneously improving resolution. To this end, a microlens array with microlenses of different focal lengths was proposed in the publication of Cong et al. (Cong, L et al., 2017; eLife, 6:e28158). However, a drawback is that each microlens group only utilizes a portion of the aperture. Summary of the Invention

[0010] The present invention is based on the purpose of proposing a possibility that enables imaging of a three-dimensional region of a sample with high temporal resolution and simultaneously high spatial resolution.

[0011] This objective is achieved by a device for capturing image data. The device includes a detection beam path along which the detection radiation of at least one microscope is guided or can be guided. The device can receive the detection radiation of the microscope, for example, at an intermediate image or at another suitable interface. Furthermore, means for separating the detection radiation between a first detection path and a second detection path are present in the detection beam path. A first detector is arranged in the first detection path, and a second detector is arranged in the second detection path, wherein a microlens array is arranged upstream of at least one detector, particularly in the associated first or second detection path. At least one microlens array is arranged in a pupil plane or an image plane. Here, the first detector has a first spatial resolution (image resolution), and the second detector has a second spatial resolution. The first spatial resolution is higher than the second spatial resolution. Additionally or alternatively, the first detector has a first temporal resolution (detection rate), and the second detector has a second temporal resolution, wherein the first temporal resolution is lower than the second temporal resolution.

[0012] The device is characterized by the presence of an evaluation unit, such as a computer, for evaluating the image data captured by the first and second detectors. The evaluation unit is configured such that the evaluations of the image data from the two detectors are computationally combined to form a three-dimensional resolved result image.

[0013] Images, particularly the results of three-dimensional analysis, are understood as datasets that specifically refer to (digital) three-dimensional representations of samples. These datasets can be visualized, for example, as two-dimensional or three-dimensional graphics.

[0014] The core of this invention is to provide image data with different spatial and / or temporal resolutions and combinations thereof to form a resulting image with high temporal and high spatial resolution. Here, each detection element (pixel) of the detector captures and evaluates position-related information, advantageously including the corresponding angular information of the captured beam. The discovery according to the invention is that a suitable combination of two detectors and the computational combination of their captured image data makes it possible to achieve very good three-dimensional resolution, satisfying, for example, requirements regarding spatial resolution for neurobiology, and also enabling, for example, a volume detection rate of 100 full volumes or larger per second.

[0015] The first and second detectors of this device are, for example, two-dimensional detectors with multiple detector elements arranged in a grid. Due to the need for very high sensitivity (low noise), electron multiplication charge-coupled devices (EMCCDs) or super complementary metal-oxide-semiconductor (sCMOS) sensors are suitable as detectors. SPAD detectors, such as SPAD arrays, like CMOS-SPAD array sensors, will also become increasingly important in the future.

[0016] Advantageously, the technical differences between the first and second detectors are chosen such that their respective spatial and temporal resolutions differ to such an extent that, although the two recording modes differ significantly from each other, they can be combined with each other at a reasonable cost. This is effectively used to obtain the respective advantages of the modes and, through their appropriate combination, to obtain and utilize image quality within the scope of the invention.

[0017] For the following statements, unless otherwise explicitly stated, it is assumed that the first detector has a higher spatial resolution but a lower temporal resolution than the second detector. For example, the first spatial resolution may be at least 1.5 times higher than the second spatial resolution, and the first temporal resolution may be at least 2 times lower than the second temporal resolution.

[0018] To support the computational combination of image data captured by the various detectors, the first detector (hereinafter also referred to as the "slowcam") and the second detector (hereinafter also referred to as the "fastcam") are aligned with each other in such a way that the allocation of image data (e.g., individual images or frames) to each other becomes possible. The slowcam and the fastcam are advantageously calibrated relative to each other so that the image data captured separately can be overlapped and / or computationally combined.

[0019] If the first and second detectors are synchronized with each other relative to their respective capture intervals, the goal of combining the captured image data can also be achieved.

[0020] The separation of the detected radiation between the first and second detection paths can be achieved in various ways. The device for separating the detected radiation can be, for example, a beam splitter, particularly a neutral splitter, a dichroic beam splitter, or a switchable mirror.

[0021] Advantageously, the detected radiation is directed into the detection path at a splitting ratio that allows for optimized operation of both detectors. For example, the first detector (slow camera, longer recording duration) with higher spatial resolution can receive a proportionally reduced amount of light compared to a second detector with lower spatial resolution but higher temporal resolution. A neutral density separator, for example, can achieve the corresponding splitting ratio. The detected radiation can also be spectrally separated, with a portion of the spectrum directed into the first detection path and another portion into the second detection path. If a pivotable mirror is used for separation, the detected radiation can be temporarily directed entirely into either the first or second detection path. This is advantageous if the switching of the mirror is controlled temporally so that the slow camera is illuminated by the detected radiation according to its frame rate.

[0022] The detected radiation can be transferred on an intermediate image plane, which serves as the interface between the microscope and the device according to the invention. The detected radiation is transformed from the intermediate image plane to the pupil via a Fourier transform.

[0023] For example, the requirements for examining processes in the brain of a small organism (such as a fruit fly) should be illustrated using an example. In this case, approximately 10,000 neurons need to be observed. The optical resolution should be at least the size of the perinuclear bodies of neurons, approximately 5 μm in size. If the volume of the brain is converted to a cube, then a volume of 400 μm × 400 μm × 400 μm should be observable. Methods suitable for meeting these requirements are particularly known as all-optical principles, light-field microscopy, or integral imaging, and are used in microscopy.

[0024] The literature discloses various methods for imaging three-dimensional volumes using two-dimensional sensors.

[0025] In principle, two variations are distinguished in this respect, both of which share the common feature of having a microlens array (MLA) mounted upstream of the detector. In light field microscopy, which has been used to date, the MLA is typically located in the nominal image plane. Pixels downstream of the individual microlenses then capture information about the angles of light emitted by the sample (object) (e.g., Broxton et al., 2013, Optics Letters, Vol. 21, pp. 25418–39).

[0026] However, if the pupil plane of the device is optically segmented, it provides an advantageous arrangement for microscopy. For this purpose, the microlens array is arranged such that the detector is located in a plane conjugate with respect to the sample plane. In this case, the microlenses of the microlens array are implemented in a plane optically conjugate with respect to the objective pupil. Embodiments of the device according to the invention, described in more detail below, are based on this variation.

[0027] The microlens arrays of the devices according to the invention need not be identical. Different microlens arrays can be selected, for example, to achieve a good fit to the geometry of the detector or its chip, or to optimally balance spatial resolution relative to the detection rate, etc.

[0028] Currently available detectors allow for parallel recording using a large number of pixels across a range of millions of pixels. In the following general description of the invention, a detector from PCO AG (Kelheim, Germany) is highlighted by way of example only.

[0029] For example, the pco.edge 26MP camera allows recording 5120 × 5120 pixels, with each pixel (detector element) having a size of 2.5 μm × 2.5 μm. This results in a sensitive area of ​​12.8 mm × 12.8 mm for the detector. The sensor diagonal is therefore 18.1 mm in length. According to the datasheet (as of July 2020), the camera's image recording rate is 7.1 frames per second (fps) (slow camera), with a given pixel data rate of 267 Mpixels / s.

[0030] Furthermore, for example, the pco.edge 4.2 detector is available, with a sensitive area having a diagonal of 18.8 mm, comparable to the pco.edge 26MP in this respect. However, the pco.edge 4.2 detector has a significantly faster readout speed, thus allowing frame rates up to 100 frames per second (fps) (for fast cameras). The detector has 2048 x 2048 pixels, each pixel measuring 6.5 μm x 6.5 μm. The sensitive area size is 13.3 mm x 13.3 mm.

[0031] Therefore, the two detectors mentioned are identical in size. Crucially, the pco.edge 4.2 detector can operate at frame rates up to 100fps. Consequently, the time difference between two consecutive volumetric records is only about 10ms.

[0032] Depending on the sample geometry, in one configuration of the invention, the recording speed can be further increased by reading only selected portions of the pixels. This can be pre-selected, randomly, or dynamically. For example, a rate of up to 200 volumes per second is possible if only 2048x1024 pixels from a fast camera are read. For example, a rate of up to 400 volumes per second can be achieved, for instance, by further reducing the number of pixels read to 2048x512 pixels. The time difference between two complete volumes captured is only 2.5 ms. These speeds are crucial for capturing the neural network's responses completely, both spatially and temporally. In this regard, for example, the average firing rate of Drosophila neurons is specified as approximately 40 Hz, and the size of the neurons is specified as approximately 2 μm to 5 μm (Weisenburger, S. & Vaziri, A. et al., 2018, Annual Review of Neuroscience, Vol. 41, pp. 431–452).

[0033] In a further configuration of the invention, the speed of image capture can be improved by performing interpolation between pixels and using the interpolation for subsequent calculations. In this process, fewer pixels need to be read.

[0034] If the device according to the invention forms part of a microscope, in one possible embodiment, the latter can be achieved using a scanning light plate for illuminating the sample. This type of illumination is advantageous for various reasons. For example, light from a plane of the sample of no interest can be well distinguished, axial resolution is increased, while low energy input to the sample is achieved, resulting in minimal light loss. Using a second detector in a second detection path, measurements can be taken at, for example, ten locations on the light plate, thereby imaging ten volumes per second at the frame rate of a fast camera of 100 Hz.

[0035] In order to select, for example, the detector to be used in the device according to the invention, the following considerations can be made.

[0036] For lateral resolution in Fourier light field microscopy, the following holds true (Scrofani, G. et al., published in Biomedical Optics Letters, Vol. 9, pp. 335–346, 2018).

[0037]

[0038] In this case, λ is the wavelength of light, and NA MO is the numerical aperture of the microscope objective, ps is the pixel size, and f is the focal length of the tube lens (TL), microscope objective (MO), microlens array (MLA), and lens used for pupil conversion (pup).

[0039] The first expression describes the fluctuating optical resolution under the assumption that the point spread function on the detector is sampled according to the Nyquist criterion. If sampling is no longer performed according to the Nyquist criterion, the second expression is restrictive. In this case, the resolution is limited by the pixel size imaged onto the sample.

[0040] Since better resolution cannot be obtained if the requirements of the Nyquist criterion are significantly exceeded, it is unnecessary to sample with more pixels than necessary. Therefore, it is advantageous to choose the ratio of focal length, number of pixels, and number of microlenses such that the two expressions above are approximately equal.

[0041] The size of the intermediate image (ZBE) is derived according to equation (2), and this intermediate image is used, for example, as the transfer position of the microscope:

[0042]

[0043] For the above arrangement, it is now necessary to consider the case where the ratios in the two detection paths are different. As mentioned earlier, the faster detector (fast camera) has fewer but larger pixels compared to the slower detector (slow camera).

[0044] For example, if the pixel size ps is doubled, the number of pixels #px in equation (2) is halved, then if the focal length f of the microlens array MLA... MLA and the focal length f of the pupil lens pup If the resolution is kept the same, then the intermediate image, and therefore the field of view, remains the same size. However, if the resolution according to equation (1) is set such that the two expressions have the same size, then the resolution in the case of the larger pixel ps is dominated by the second expression. Because the Nyquist criterion is no longer satisfied in this case, the resolution obtained with the larger pixel ps is worse. The resolution will be roughly halved.

[0045] To achieve equal imaging quality in the two detection paths, in another embodiment, the device can be configured to set approximately the same signal-to-noise ratio (SNR) in both detection paths. In the simplified example of the detector described above, the device can be configured such that the slow camera at 7fps has an exposure time of approximately 143ms, while the fast camera integrates for 10ms under different conditions. To set similar SNRs for the two detectors, the measured detection radiation is separated between the two sensors at the same ratio, such that in this example, the fast camera receives 143 samples of detection radiation, while the slow camera measures only 10 samples. Furthermore, the distribution ratio can be modified by considering, for example, lateral sampling, that is, the number of pixels used to capture the PSF (point spread function).

[0046] Without sacrificing quality, the separation ratio can have slight deviations because, if the signal strength is the same, the signal-to-noise ratio is scaled only by the square root of the pixel time. Furthermore, the device can be designed to be flexible enough to set each desired ratio.

[0047] In addition to detecting the distribution of radiation signals as described above, it is also possible that the signal is not of a single wavelength, but rather two different wavelengths distributed across various detectors. This method is particularly advantageous when measuring rapid processes, such as calcium transients. For example, the calcium indicator GCaMP emits light in the green spectral range. Its fluorescence signal can be captured with a second detector having higher second temporal resolution, and the structure of a sample that remains unchanged or changes only slowly can be captured with a detector having lower temporal resolution. For example, structural information can be derived from thin-film dyes that emit light in the red spectral range.

[0048] Therefore, illumination can take the form of monochromatic or multicolor linear or nonlinear illumination. For example, wide-field illumination can be achieved incoherently or with lasers, wherein the illumination is advantageously time-varying so that speckle appearing over the measurement intervals of a fast camera is adequately averaged, thereby reducing or even eliminating speckle.

[0049] Illumination, such as illumination using a light sheet, can be achieved by using an objective lens that serves as both an illumination objective and a detection objective. This embodiment allows for a compact design. In another embodiment, the illumination can be, for example, laterally radiating, and thus independent of the detection objective. Furthermore, it may be advantageous if there is a means for setting time-pulse illumination in the millisecond range in the sense of stroboscopic illumination to stimulate temporal variations in the fluorescence signal.

[0050] In another embodiment of the invention, the MLA may be arranged in the pupil plane or nominal image plane upstream of one of the detectors, while the other detector is a two-dimensional detector (2D detector) known in the prior art, which does not have an MLA arranged upstream of it. This other detector may be present in addition to the first detector and / or the second detector. In this case, the resulting image is generated from the image data of the detector with the upstream MLA and the image data of the 2D detector.

[0051] In another embodiment of the invention, the first detector may also have a higher spatial resolution than the second detector. A so-called aperture, in the form of a pinhole or slit aperture, is present in the first detection path, in the intermediate image plane, and upstream of the first detector. Due to the effect of the pinhole, the defocused portion of the detected radiation is stopped down, resulting in confocal capture of the detected radiation by the first detector. A microlens array is arranged upstream of the second detector. Advantageously, a scanning device is located in the illumination beam path, through which the illumination spot can be guided onto the sample to be imaged. The current alignment of the scanning device allows for a unique allocation of the illumination spot position, for example, in a plane (XY plane) extending orthogonal to the optical axis of the illumination objective. Also captured is the current position (Z position) of the focal point of the detection objective. In such an embodiment, the light source of a laser scanning microscope (LSM) can be used to provide the illumination radiation. Thus, in one possible embodiment of the invention, the LSM is combined with a light field microscope (LFM), for example, both located on a common support, and image data captured by different spatial and / or temporal resolutions and through different functional modes of the microscope are computationally combined to form a three-dimensional resulting image. The illumination radiation of the LSM can be used for both microscopes (LSM and LFM). If necessary, it can be switched back and forth between detection paths. Optionally, the detection radiation can also be proportionally separated between detection paths. The capture of the relevant position of the captured image data, as well as information about the relevant Z-position of the relevant image data, allows the first detector of this embodiment of the device to generate a three-dimensional image dataset. As an alternative to a scanning device, the relative movement between the illumination spot and the sample can be achieved, for example, by controlled adjustment of a motorized sample stage.

[0052] The device according to the invention can be connected to a microscope or be a component of a microscope. A microscope including the device according to the invention can be used in a variety of ways, and particularly allows for three-dimensional imaging of sample volumes with high spatial and temporal resolution. For example, a microscope including the device according to the invention and having a first functional mode can be connected to another microscope having a second functional mode. In this regard, for example, a light field microscope can be technically connected to a laser scanning microscope as described above, such that the light source of the laser scanning microscope, for example, is used in both functional modes.

[0053] A microscope may have a light source located in the path of an illumination beam, an objective lens used as an illumination objective lens, and a device for generating a light sheet, wherein the light sheet is generated in or can be generated in the sample space upstream of the objective lens on the object side.

[0054] In another possible embodiment, the light source can be implemented to provide pulsed illumination. In this regard, pulses with durations in the picosecond or femtosecond range can be generated and provided. This type of illumination can be used for nonlinear fluorescence excitation. Furthermore, a suitable sensor can therefore be used to capture the fluorescence lifetime. For example, a pulsed laser is used as the light source. One detector could be a SPAD array (single-photon avalanche diode array).

[0055] The apparatus for generating a light sheet in the light sheet plane can be, for example, a cylindrical optical unit, a scanning device, or a combination of both. Both embodiments can be designed to generate a light sheet tilted relative to the optical axis of the illumination objective in a light sheet plane that extends at a respective tilt. For this purpose, an illumination beam from a light source shaped by the action of a cylindrical lens or an illumination beam from a light source focused by the scanning device can be guided to an incident position in the objective pupil (hereinafter also referred to as the incident pupil) of the objective lens, said incident position being located outside the optical axis of the objective lens. Such embodiments make it possible to use a common objective lens for illumination and inspection. In an alternative embodiment, an illumination objective and an inspection objective lens can be arranged.

[0056] To adjust the thickness of the resulting sheet laterally relative to the sheet plane, an configurable optical device, such as a zoom optical unit and / or an aperture reduction device, may be present in the illumination beam path. In this case, the thickness can be set manually or automatically, for example, by selecting the measurement task to be performed and adjusting the sheet thickness accordingly. Additionally or optionally, if, for example, the captured image data does not achieve the specified quality parameters, a control loop can influence the sheet thickness. Such quality parameters are, for example, a predetermined signal-to-noise ratio or signal-to-background ratio (SBR).

[0057] The object of the present invention is also achieved by a method for capturing image data, wherein the detection of radiation, particularly at least the detection of radiation by a microscope, is separated between a first detection path and a second detection path.

[0058] In the first detection path, the detected radiation is captured by means of a first detector having a first temporal resolution (first detection rate) and a first spatial resolution. In the second detection path, the detected radiation is captured by means of a second detector having a second temporal resolution (second detection rate) and a second spatial resolution, wherein the first temporal resolution is lower than the second temporal resolution and / or the first spatial resolution is higher than the second spatial resolution.

[0059] According to the present invention, image data captured by two detectors are computationally combined to obtain a three-dimensional analytical result image. In this case, for example, image data captured in the first and second detection paths for each detector can be computationally combined to form a three-dimensional analytical image, and then the three-dimensional analytical images can be combined to form a result image; or image data captured by both detectors can be computationally combined to form a three-dimensional analytical result image.

[0060] In another configuration of the method, image data with higher spatial resolution or image data with higher temporal resolution is used to computationally increase the spatial resolution and / or temporal resolution of image data with lower spatial resolution or lower temporal resolution, respectively.

[0061] The advantage of this configuration of the method is that data with high spatial resolution measured at a lower detection rate can also be used to virtually represent the image with high detection rate but lower spatial data capture (sampling), that is, with at least the same or better resolution at a lower spatial resolution.

[0062] For example, if the same spatial resolution as in an image with higher spatial resolution is achieved, this corresponds to multiple comparisons with the actual sample structure during the image capture sequence (also known as "field comparisons" or "fundamental truth").

[0063] For example, one could assume a fast camera operates at 100Hz, thus capturing image data with lower spatial resolution than a slow camera operating at 1Hz, which provides image data with high spatial resolution. For instance, a "ground truth" comparison could be performed every 100 images (frames).

[0064] In a further configuration of this method, the computational combination of captured image data and / or the combination of image data or images is performed using machine learning applications, particularly using convolutional neural networks (CNNs).

[0065] As described above, a key finding of the present invention is that the way image data is used allows for the advantageous combination of information from two detection paths to achieve rapid image recording and also very high image quality, that is, particularly high spatial resolution. The first possibility is initially the combined presentation of image data. For this purpose, images or image stacks with high spatial resolution are presented in a suitable manner, wherein the image data has high temporal resolution. For example, image data from a slow-motion camera and a fast-motion camera are computationally combined into an "image-to-image mapping" so that, since high-resolution images recorded at various temporal support points are recorded more slowly as the number of pixels in the detector (slow-motion camera) increases, even images recorded at lower spatial resolution but higher temporal resolution possess image quality corresponding to higher spatial resolution.

[0066] Another possibility is the aforementioned Convolutional Neural Network (CNN). In networks with many layers, this is also referred to as a deep central nervous system. These algorithms are used, for example, to virtually increase the resolution of conventionally recorded images. To do this, the network is trained using high-resolution data corresponding to the object type. Afterward, the trained CNN can even compute a virtual high resolution from data with lower resolution (e.g., Wang, H. et al., 2018, *Nature Methods*, Vol. 16, pp. 103–110). For a comparison of various algorithms, see Zhang, H. et al., 2019, *Sensors*, Vol. 19, p. 3234.

[0067] The provided algorithms are trained so that they can be used to improve the resolution of recorded image data. For example, if the intention is to image brain tissue, a sufficient amount of brain tissue image data is used to train the network. The result is an improvement in image quality in both detection paths of a given device.

[0068] Advantageously, different microscope objectives can be used without having to change the setup of the equipment. This is especially evident when the pupil size is roughly the same. Advantageously, objectives with the same M / NA ratio can be used (e.g., 40x / 1.2NA; 20x / 0.6NA, 10x / 0.3NA). Therefore, it is possible to observe volumes of different sizes with slightly different resolutions under different conditions.

[0069] The computational combination of recorded image data can be performed using different magnifications. For this purpose, a smaller volume captured at a higher resolution is embedded within a larger volume captured at a slightly lower resolution.

[0070] To provide illumination, a laser scanning microscope (LSM) can be used, for example, as an add-on module. According to the invention, high-resolution image data captured via LSM mode can be combined, for example, with low-resolution image data from 3D measurements. For this purpose, the LSM can be set to an optional operating mode to allow the generation of light sheets. This can be achieved, for example, by focusing a laser beam into the entrance pupil of the micro-objective.

[0071] Therefore, it is also possible to combine light field technology capture with structured illumination microscopy (SIM).

[0072] In a further configuration of the invention, the captured and / or computationally combined image data can be combined with measurement data from LSM or other imaging microscopy methods (e.g., phase contrast; differential interferometric contrast; structured illumination microscopy, etc.).

[0073] If machine learning is used, image resolution can be improved by building upon the correspondingly trained CNN. The CNN can be continuously adjusted and improved using the image-to-image correspondence between two detection paths.

[0074] Further improvements in image quality can be achieved by incorporating data from, for example, a stack of images recorded in three dimensions into the CNN training process or through other image fusion methods. This data can be obtained and provided, for example, by a simultaneously operating laser scanning microscope. The invention also allows for optical manipulation in both 2D and 3D (e.g., holographically). Furthermore, measurement results can be evaluated in the sense of dSTORM, PALM, etc.

[0075] The advantages of this invention lie in its increasingly better spatial resolution and very good temporal resolution, a result of the highly parallelized voxel capture. Strong background suppression and / or high axial resolution can be achieved when illuminated with a light pane. Furthermore, it is advantageous that this invention can be implemented on existing microscopes, where the numerical aperture of the microscope objective is fully available. The axial resolution obtained after computationally combining this image data is only slightly reduced compared to the depth of field of an objective with full numerical aperture. This invention is advantageously applicable to living samples, for example, due to its low sample load and rapid three-dimensional imaging.

[0076] By means of the device and method according to the invention, a three-dimensional region of a sample can be captured simultaneously with a temporal resolution in the range of milliseconds, while simultaneously possessing sufficient spatial resolution of approximately 2-3 μm. Furthermore, interfering background signals are effectively suppressed, thereby achieving high sensitivity during imaging of the observed region of the sample. Therefore, in 3D samples, the ultimate goal is to simultaneously record as many voxels as possible with the highest possible temporal resolution. The method proposed herein allows for this and provides highly parallelized imaging of the three-dimensional volume.

[0077] This invention particularly allows for the observation of objects on the order of approximately 0.005 mm to approximately 3 mm (e.g., as small as a dendrite). The temporal resolution is approximately 0.01 to 1 second, but can be 0.001 to 100 seconds in certain cases.

[0078] The solution proposed here allows for imaging with high optical resolution, as well as a very high volumetric imaging rate. Attached Figure Description

[0079] The invention will now be explained in more detail with reference to exemplary embodiments and accompanying drawings. In the drawings:

[0080] Figure 1 A schematic diagram of a first exemplary embodiment of the device according to the present invention is shown;

[0081] Figure 2 A schematic diagram of a second exemplary embodiment of the device according to the present invention is shown;

[0082] Figure 3 A schematic diagram of a third exemplary embodiment of the device according to the present invention is shown;

[0083] Figure 4 A schematic diagram of a first exemplary embodiment of a microscope is shown, the microscope having a device according to the invention and an apparatus for generating a light plate;

[0084] Figure 5 A schematic diagram of a second exemplary embodiment of a microscope is shown, the microscope having a device according to the invention and an apparatus for generating a light plate;

[0085] Figure 6 A flowchart illustrating a first configuration of the method according to the invention is shown; and

[0086] Figure 7 A flowchart of a second configuration of the method according to the present invention is shown. Detailed Implementation

[0087] In the general configuration of the device 1 according to the invention, a device 2 for separating detection radiation, having a beam splitter (beam splitter 2) along the beam path, functions to separate the detection radiation between a first detection path 3 having a first detector 4 and a first microlens array 5 arranged upstream thereon, and a second detection path 6 having a second detector 7 and a second microlens array 8 arranged upstream thereon. Microlens arrays 5 and 8 are each arranged in a pupil. If optical lenses 9 are specified in the exemplary embodiment, they may also represent corresponding combinations of optical elements (lens systems).

[0088] The first detector 4 is allowed to have a higher spatial resolution than the second detector 7. The temporal resolution of the first detector 4 (slow camera) is lower than that of the second detector 7 (fast camera). In a further embodiment, the first and second detectors 4 and 7 may also be arranged in another detection path 3 or 6, respectively.

[0089] The detection radiation from microscope 11, focused by the tube lens 9TL, passes through a selectable field stop 14 in the intermediate image plane, reaches optical lens 9, and is separated by beam splitter 2 between the first detection path 3 and the second detection path 6. The pupil plane of microscope 11, particularly microscope objective 18 (see, for example...), Figure 4 Imaging from the pupil plane (back focal plane) of the optical lens array 8 to the plane of the microlens array 8 is achieved through the lens system 9TL, 9. Lens 9TL acts as a tube lens, while downstream lens 9 acts as a Fourier lens, that is, it induces the Fourier transform of the detected radiation.

[0090] Image data captured by detectors 4 and 7 is fed to an evaluation unit 10 in the form of a computer or a field-programmable gate array (FPGA). The latter is configured to evaluate the captured image data by considering position information, angle information, and intensity values, and for example, either to computationally combine the captured portions of angle information from each of the detectors 4 and 7 as image data to form a three-dimensional resolved image under different conditions and then combine them to form a result image, or to combine the captured portions of angle information from both detectors 4 and 7 as image data to form a three-dimensional resolved result image.

[0091] Evaluation unit 10 is optionally connected to display 13 (e.g., a monitor), on which image data and / or result images or stacks of result images can be displayed. Furthermore, evaluation unit 10 is optionally connected to control unit 12, which may in particular be a component of microscope 11. In other possible embodiments, control unit 12 is not a component of microscope 11, but may be connected to microscope 11 in a manner suitable for data transmission (e.g., see...). Figure 2 (Diagram in the diagram).

[0092] The control unit 12 is configured to generate control signals based on the results of the evaluation unit 10. These control signals can be used to control the functions of the microscope 11.

[0093] In another exemplary embodiment of device 1, the microlens arrays 5 and 8 are arranged in the nominal image plane nBE in different cases. Figure 2 Thus, due to the action of the microlenses, the captured detection radiation is guided to the corresponding detector 4 or 7. Therefore, the nominal image plane nBE represents the intermediate image plane. The optical lens 9, which serves as the focusing lens, is arranged upstream of the beam splitter 2. Depending on whether the (point) source (not shown) whose light is to be captured is located in the object plane (focal plane) of the objective lens 18 (e.g., see...), the capture is directed to the corresponding detector 4 or 7. Figure 4 The point source is then imaged (in an idealized manner) onto MLA 5, 8 in the shape of a point. If the point source is located above or below the object plane in the detection direction, it is not imaged precisely into the nominal image plane nBE, but rather into its rear or front (e.g., see [reference needed]). Figure 2 The spatial position of the point light source can be calculated, for example, by means of the evaluation unit 10 configured accordingly, based on the intensity value that has been captured by each pixel and also represents position information, and based on the captured angle information.

[0094] A third exemplary embodiment of device 1 also has microlens arrays 5 and 8, which, in different cases, are located in the nominal image plane nBE. Figure 3 Due to the action of the optical lenses 9 arranged in each detection path 3 and 6, the detection radiation is focused into the nominal image plane nBE.

[0095] Figures 1 to 3 The exemplary embodiment shown can be used, for example, in dual-channel light field microscopy. The spectral composition of the detected radiation can be separated by beam splitter 2.

[0096] In microscope 11 ( Figure 4 In an exemplary embodiment of the device 1 according to the invention, a light source 15 for providing a laser as excitation light, an optical lens 9, a light guiding device 17 or a scanning device 17, a dichroic separator 16, and an objective lens 18 having an entrance pupil EP are present in the excitation beam path, the objective lens serving as an illumination objective lens. The light source 15, especially a light source in the form of a laser light source, may optionally operate in a pulsed manner.

[0097] Optical lens 9 and beam splitter 2 are arranged in the detection beam path (indicated by dashed lines). With the aid of optical lens 9 and beam splitter 2, the detected radiation is guided along the first detection path 3 by means of a first microlens array 5 and a first detector 4, and / or along the second detection path 6 by means of a second microlens array 8 and a second detector 7. Detectors 4 and 7 are connected to evaluation unit 10, which is connected to control unit 12 in a manner suitable for exchanging data. Control unit 12 can generate control commands for controlling scanning device 17 (hereinafter also referred to as scanner 17). In a further embodiment, light source 15 may also be controlled by control unit 12.

[0098] During operation of the microscope 11 including the device 1 according to the invention, a laser emitted by the laser source 15 is focused and transmitted to the scanning device 17. The scanning device 17, controlled by the control unit 12, deflects the laser in a controlled manner in the x-direction (x) and / or y-direction (y). The scanning device 17 can be used to change the incident angle and incident position of the excitation light in the incident pupil EP (objective pupil).

[0099] After passing through the dichroic color splitter 16, the excitation light is guided to an incident position in the entrance pupil EP away from the optical axis oA of the objective lens 18. As a result, a light plate 19 is generated on the object side by the objective lens 18, which is tilted relative to the optical axis oA in a correspondingly tilted light plate plane. If the sample is located in the sample space 20 upstream of the objective lens 18, the light plate 19 can be guided into the sample.

[0100] Optionally, an adjustable optical device 21, such as a zoom optical unit or aperture, may be present in the excitation beam path (= illumination beam path), which functions to make the thickness of the light plate 19 transverse to the light plate plane adjustable (shown only by indication). The adjustable optical device 21 can be controlled by the control unit 12.

[0101] Fluorescence can be excited in the sample and emitted as detection light (detection radiation) due to the action of the light plate 19 formed by the excitation light. The emitted detection light is collected by the objective lens 18, which serves as both an illumination objective and a detection objective. In the dichroic separator 16, the detection light with a wavelength longer than the excitation light is reflected into another segment of the detection beam path and reaches the first microlens array 5 and / or the second microlens array 8 through the beam splitter 2. The microlenses indicated by the indicator can be considered as individual imaging systems. The image points induced by each microlens are captured as image data by the correspondingly positioned detector elements of detectors 4 and 7 and fed to the evaluation unit 10.

[0102] based on Figure 4 ,exist Figure 5Another possible embodiment of the invention is illustrated below. Another dichroic beam splitter 22 is arranged in the illumination beam path between the light source 15 and the scanning device 17. Due to the action of the beam splitter, the detection radiation from the sample space 20 passes through the beam splitter 16 and subsequent optical elements, and is converted into a fixed beam (descanned) by the action of the scanning device 17, and then guided to the final portion of the first detection path 3. In this exemplary embodiment, the beam splitter 16 is also used to separate (beam splitter 2) the captured detection radiation between the first and second detection paths 3, 6, and may be dichroic or separate the detection radiation in a specific ratio. The detection radiation is focused by the optical lens 9 into an intermediate image plane in which a pinhole 23 in the form of a pinhole aperture or slit aperture is provided. Due to the effect of the pinhole 23, portions originating from the defocused region are removed from or at least significantly reduced from the detection radiation beam. For example, a secondary electron multiplier (photomultiplier tube, PMT), an array of multiple PMTs, or a two-dimensional detector (see above) can be used as the first detector 4. The first detector 4 is connected to the evaluation unit 10. The latter is in turn connected to the scanning device 17 to obtain data related to the corresponding current alignment of the scanning device 17. Based on the current alignment, the position in the XY plane can be assigned to the individual image data captured by the first detector 4. Information related to the axial position (position in the z direction, z position) can be determined based on the known position of the current focal plane of the objective lens 18 and optionally taking into account the point spread function (PSF) known to the image capture system. Image data can be captured at different z positions (z stacks). In this way, three-dimensional resolved image data can be captured using the first detector 4. Higher spatial resolution is achieved compared to the second detector 7 because the first detection path 3 is designed as a confocal detection path. If switching is alternately implemented between capture by means of the first detection path 3 (confocal) and the second detection path 6, the configurable optical device 21 can be controlled accordingly to generate an illumination spot or light sheet 19.

[0103] Due to the action of beam splitters / color generators 2 and 16, the second detection path 6 receives the detection radiation imaged onto and captured by the second detector 7. A microlens array 8 is arranged upstream of the second detector 7. In a further embodiment, beam splitters / color generators 2 and 16 can be replaced by, for example, switchable mirrors. The image data captured by the first detector 4 and the second detector 7 are combined by the evaluation unit 10, and a three-dimensional result image is calculated.

[0104] The method according to the invention can be performed in two alternative configurations. In the first alternative ( Figure 6In this process, the detected radiation is directed into the first and / or second detection paths 3, 6, and captured there by detectors 4, 7, which are respectively present according to the principles of light field technology. In each case, a three-dimensional resolved image is calculated from both the image data captured by the slow camera of the first detection path 3 and the image data from the fast camera of the second detection path 6. Subsequently, the three-dimensional resolved images of the two detection paths 3, 6 are combined to form the resulting image.

[0105] In an optional configuration of this method, image data captured by the slow-motion camera and the fast-motion camera are combined to form the final 3D resolved image without pre-generating at least one 3D resolved image for each detection path 3, 6 or each detector 4, 7. Figure 7 ).

[0106] In a further configuration of the method, the image data captured by the first detector 4 and the second detector 7, or the corresponding images calculated therefrom, can be mapped to each other using a CNN (convolutional neural network), and the resulting image of three-dimensional resolution can be calculated.

[0107] Reference marker:

[0108] 1: Equipment

[0109] 2: Devices / beam splitters used for separating and detecting radiation

[0110] 3: First detection path

[0111] 4: First detector (slow-motion camera)

[0112] 5: First microlens array

[0113] 6: Second detection path

[0114] 7: Second detector (fast camera)

[0115] 8: Second microlens array

[0116] 9: Optical Lens

[0117] 10: Evaluation Unit

[0118] 11: Microscope

[0119] 12: Control Unit

[0120] 13: Monitor

[0121] 14: Field Aperture

[0122] 16: Color Separator

[0123] 15: Light source

[0124] 17: Scanning equipment

[0125] 18: Objective lens

[0126] 19: Light film

[0127] 20: Sample Space

[0128] 21: Configurable optical devices

[0129] 22: Beam splitter (used for confocal beam paths)

[0130] 23: Small hole

[0131] EP: Entrance Pupil

[0132] oA: Optical axis

[0133] nBE: Nominal image plane.

Claims

1. A device (1) for capturing image data, comprising: The detection beam path is such that the detection radiation of at least one microscope (11) is guided or can be guided along the detection beam path; as well as Device (2) for separating the detected radiation between the first detection path (3) and the second detection path (6); The first detector (4) in the first detection path (3) and the second detector (7) in the second detection path (6), wherein the microlens array (5, 8) is arranged upstream of at least one of the detectors (4, 7); Wherein, the first detector (4) has a first spatial resolution, the second detector (7) has a second spatial resolution, and the first spatial resolution is higher than the second spatial resolution; and The first detector (4) has a first time resolution, and the second detector (7) has a second time resolution, wherein the first time resolution is lower than the second time resolution. Its features An evaluation unit (10) is provided for evaluating the captured image data of the first detector and the second detector (4, 7). The evaluation unit (10) is configured to enable the evaluation of image data from the two detectors (4, 7) and to generate a three-dimensional resolution result image from the evaluated image data by combining the image data from the first detector and the second detector (4, 7). and i) In the first detection path (3), the aperture (23) exists in the intermediate image plane and in the first detection path (3) and optically upstream of the first detector (4), thereby enabling confocal capture of the detected radiation by means of the first detector (4), and thus achieving a higher spatial resolution by means of the first detector (4) than in the case of the second detector (7); the microlens array (8) is arranged upstream of the second detector (7); or ii) A microlens array (8) is arranged in front of both the first detector (4) and the second detector (7).

2. The device (1) according to claim 1, characterized in that The first spatial resolution is at least 1.5 times higher than the second spatial resolution, and the first temporal resolution is at least 2 times lower than the second temporal resolution.

3. The device (1) according to claim 1 or 2, characterized in that The device (2) for separating the detected radiation is a beam splitter or a switchable mirror.

4. The device (1) according to claim 1 or 2, characterized in that, The device (2) for separating the detected radiation is a dichroic beam splitter.

5. A microscope (11) comprising the device (1) according to any one of the preceding claims.

6. The microscope (11) according to claim 5, characterized in that, A light source (15) and an objective lens (18) used as an illumination objective lens exist in the path of the illumination beam, thereby producing wide field-of-view illumination.

7. The microscope (11) according to claim 6, characterized in that, The light source (15) is a laser light source (15).

8. The microscope (11) according to claim 5, characterized in that, A light source (15), an objective lens (18) used as an illumination objective lens, and a device for generating a light sheet (19) are present in the illumination beam path, wherein the light sheet (19) is generated on the object side upstream of the objective lens (18) in the sample space (20).

9. The microscope (11) according to claim 6 or 7, characterized in that, The light source (15) is implemented to provide pulsed illumination light.

10. The microscope (11) according to claim 9, characterized in that, The light source (15) is implemented to provide pulsed illumination light with a pulse duration in the picosecond or femtosecond range.

11. The microscope (11) according to claim 8, characterized in that, The device for generating the light sheet (19) is a cylindrical optical unit or a scanning device (17), wherein the illumination beam of the light source (15) shaped by the action of the cylindrical lens or the illumination beam of the light source (15) deflected by the scanning device (17) is guided to an incident position in the objective pupil (EP) of the objective lens (18), the incident position being outside the optical axis (oA) of the objective lens (18).

12. The microscope (11) according to claim 8, characterized in that, An configurable optical device (21) is present in the path of the illumination beam, and the function of the optical device (21) is that the thickness of the light sheet (19) transverse to the plane of the light sheet is configurable.

13. A method for capturing image data, wherein Provide a microscope (11) according to claim 5; The detection radiation of at least one microscope (11) is separated between the first detection path (3) and the second detection path (6); In the first detection path (3), the detected radiation is captured by means of the first detector (4). In the second detection path (6), the detected radiation is captured by means of the second detector (7), and The image data captured by the two detectors (4, 7) are combined computationally to form a three-dimensional analytical result image.

14. The method according to claim 13, characterized in that, Image data with higher spatial resolution or image data with higher temporal resolution are used to computationally increase the spatial resolution and / or temporal resolution of image data with lower spatial resolution or lower temporal resolution, respectively.

15. The method according to claim 13 or 14, characterized in that, In each case, only a selected portion of the detector element of the second detector (7) is read.

16. The method according to claim 13 or 14, characterized in that, The detected radiation is guided alternately along the first detection path (3) and the second detection path (6) in time, wherein the switching time point and the duration of the switching are established based on the frame rate of the first detector (4).

17. The method according to claim 13 or 14, characterized in that, The computational combination of the image data used to form the resulting image and / or the combination of the image data or images is performed using machine learning applications.

18. The method according to claim 17, characterized in that, The machine learning application mentioned is the application of convolutional neural networks (CNNs).

Images