Method and device for light sheet microscopy of specimens

Abstract

The invention relates to a method for light sheet microscopy of a sample, wherein the sample is illuminated by a light sheet by means of an illumination objective, and to a light sheet microscope for carrying out the method. Light emitted by the sample is imaged onto a planar detector by means of a detection objective. Here, the optical axis of the detection objective forms an angle different from 0° and 180° with the optical axis of the illumination objective and passes through the light sheet in the light sheet plane. The planar detector captures at least one image. The invention also relates to a corresponding light sheet microscope. According to the invention, by analyzing at least one image with the aid of a neural network, it is determined (i) whether the light sheet plane is at least partially in the focal plane of the detection objective, and / or (ii) in which direction of the focal plane the light sheet plane is located along the optical axis of the detection objective, and / or (iii) how far the light sheet plane is from the focal plane, measured along the optical axis of the detection objective.

Description

Technical Field

[0001] The invention relates to a method for light sheet microscopy of a sample. In this case, the sample is illuminated by means of an illumination objective through a light sheet. The light emitted by the sample is imaged onto a planar detector by means of a detection objective. In this case, the optical axis of the detection objective and the optical axis of the illumination objective enclose an angle different from 0° and 180°. The optical axis passes through the light sheet in the light sheet plane. In this case, the planar detector records at least one image. Background Art

[0002] The illumination objective also has an optical axis which lies in the light sheet plane. The optical axis of the detection objective is preferably perpendicular to the light sheet plane, but this is not mandatory. In general, the optical axis of the detection objective forms an angle with the light sheet plane which is different from 0° and the corresponding 180°. The thicker the light sheet is in its middle part, the greater the angle should be.

[0003] The study of biological samples has become increasingly important in recent years, wherein the sample is illuminated by a light sheet, the plane of which (light sheet plane) intersects the optical axis of the detection (detection direction) at a non-zero angle. Usually, the light sheet plane forms a non-zero, but usually not necessarily right angle, angle with the detection direction (generally relative to the optical axis of the detection objective). This research method is mainly used in fluorescence microscopy and is summarized under the term LSFM (light sheet fluorescence microscopy). An example is the method described in DE 102 57 423 A1 and in WO 2004 / 0535558 A1, which is based on the above-mentioned documents and is called SPIM (selective plane illumination microscopy), with which a three-dimensional image of thicker samples can also be generated in a relatively short time: Based on the optical sectioning, in combination with a relative movement in a direction perpendicular to the sectioning plane, a visual / three-dimensional stretched display of the sample is possible.

[0004] The LSFM method has several advantages over other designed methods, such as confocal laser scanning microscopy or two-photon microscopy. Since detection can be achieved over a wide field of view, a larger sample area can be obtained. Although the resolution is slightly lower than in the case of confocal laser scanning microscopy, thicker samples can be analyzed with the LSFM technique because the penetration depth is greater. In addition, in this method, the exposure of the sample is minimal, which reduces the risk of sample fading, because the sample is only illuminated by a thin light sheet, which has a non-zero angle with the detection direction.

[0005] Instead of a purely static light sheet, a quasi-static light sheet can also be generated by rapidly scanning the sample with a light beam. Light sheet-like illumination is generated in such a way that the light beam undergoes a very rapid relative movement with respect to the sample to be observed and is thus arranged multiple times in succession in time. The integration time of the camera, during which the sample is imaged on its sensor, is selected so that the scan is completed within the integration time.

[0006] One of the main applications of light sheet microscopy is the imaging of medium-sized organisms ranging from a few hundred micrometers to a few millimeters. Typically, the organism is embedded in a gel, such as agar, which in turn is located in a glass capillary. The glass capillary is inserted from above or from below into a sample chamber filled with water, and the sample is pressed out of the capillary as a piece. The sample in the agar is illuminated by the light sheet and the fluorescence is imaged onto a camera using a detection objective (which is preferably, but not necessarily, perpendicular to the light sheet and thus also to the illumination objective of the optical system for generating the light sheet).

[0007] However, the method of light sheet microscopy has certain limitations. On the one hand, the samples that need to be studied are relatively large, coming from developmental biology. On the other hand, due to the size of the sample preparation and the sample chamber, the light sheet is relatively thick and the axial resolution that can be achieved is therefore limited. Third, the sample preparation is complex and not compatible with standard sample preparation and standard experimental holders that are usually used for fluorescence microscopy of single cells.

[0008] In order to partially circumvent the above limitations, a new structure has been realized in recent years, in which the illumination objective and the detection objective are preferably arranged perpendicular to each other and directed at the sample from above at an angle of 45°. The operation method is described in WO 2012 / 110488 A1 and WO 2012 / 122027 A1, for example.

[0009] In order to illuminate the sample, coherent light from a laser is generally used. The wavelength of the light is selected in fluorescence microscopy according to the markers that should be excited to emit fluorescence. In the simplest case, for example, a light beam with an intensity distribution that conforms to the Gaussian function can be formed into a light sheet statically with the aid of a cylindrical objective lens, or to a certain extent statically with the aid of scanning and a camera integration time coordinated therewith. Here, it is advantageous to structure the illumination of the sample, thereby increasing the resolution. Thus, for example, in the article "Propagation-invariant spot arrays" by V. Kettunen et al. (Op. Communications 23 (16), Seite 1247, 1998) the coherent superposition of Bessel beams is introduced. This superposition is achieved by calculating the phase component that can be introduced into the pupil with the aid of an algorithm. When the spectrum of the Bessel beam is imaged into the pupil, the phase component generates a plurality of Bessel beams that are superimposed in the sample. The phase component is similar to a star-shaped grating with phase values ​​from 0 to π.

[0010] In US2013 / 0286181A1, the interference effect between the individual Bessel beams is used in a targeted manner to generate a stretched and structured light sheet. Here, the Bessel beams are arranged close to each other so that the sub-maxima of the individual Bessel beams are superimposed in a deconstructed manner above and below the propagation plane, the light sheet plane. Different interference patterns are obtained depending on the spacing between the individual Bessel beams.

[0011] WO 2014 / 005682 A1 introduces the so-called sinc 3 Generation of a light beam. In the sample, an almost square light sheet with only small secondary maxima can thus be generated.

[0012] Whatever the type of light sheet, the challenge of this type of microscope is to position the light sheet and its light sheet plane and the detection objective relative to each other so that the light sheet plane coincides with the focal plane of the detection objective within a predetermined range of the object field of the detection objective. Here, the optical axis of the detection objective is often perpendicular to the light sheet plane, in which case the light sheet plane and the focal plane of the detection objective must coincide, i.e. be congruent in the same plane. This can be achieved by relative movement of the two objectives to each other. By moving the detection objective along the optical axis of the detection objective, the focal plane of the detection objective can be moved. However, the light sheet is usually also positioned with the aid of an illumination device, of which the illumination objective forms part. Here, the light sheet can be moved both along the optical axis of the detection objective and along the optical axis of the illumination objective.

[0013] The determination of the best focus state is usually done visually. The user usually distinguishes when the best focus state has been reached with the aid of an instantaneous image of the specimen. The visual impression of this illumination in one plane depends strongly on the (usually unknown) specimen and its three-dimensional structure. This process therefore requires a lot of experience and is not always reproducible. For example, a region currently located in the focal plane of the detection objective may have only a few fluorescent structures, while an adjacent region in the same plane or above or below it may have many fluorescent structures. An inexperienced user may now try to move the light sheet to a location where many fluorescent structures are located, either by a lateral shift of the middle part of the light sheet or by a shift along the optical axis of the detection objective. In any case, this leads to an increase in the detected signal and erroneously induces a better focus state. Although the error-proneness in the focus assessment can be reduced by means of, for example, classical criteria for evaluating image quality (e.g. for determining sharpness and / or contrast), which are usually used for the automatic focusing of photographic objectives, it has not been possible to set the best focus state reproducibly for unknown specimens so far. Summary of the invention

[0014] The object of the present invention is therefore to further develop a method of the type described above for light sheet microscopy of a specimen such that the best focus state can be determined reproducibly and thus objectively, independently of the specimen and also independently of the user; optionally, the best focus should be set automatically. Furthermore, a light sheet microscope should be provided which can carry out the method.

[0015] In the simplest case, this object is achieved in the method described initially, in that, by analyzing at least one image, a neural network is used to determine whether the light sheet plane is at least partially in the focal plane of the detection objective. If the light sheet plane and the optical axis of the detection objective are perpendicular to each other, the light sheet plane is in the focal plane in the best focused state. However, the following configuration is also used, in which the light sheet plane and the optical axis of the detection objective form an angle different from 90°, so that the light sheet plane and the focal plane of the detection objective intersect along a straight line, and the light sheet plane is only partially in the focal plane of the detection objective. In principle, all angles between 0° and 180° are possible, which do not contain these two critical angles. However, the advantages of light sheet illumination cannot be achieved around these two critical angles, because the light sheet has a finite thickness, so that the illumination is no longer limited to a small area around the focal plane of the detection objective. In addition to the binary determination of the position of the light sheet plane relative to the focal plane of the detection objective, it is also possible to determine alternatively or in combination in which direction the light sheet plane is located in the focal plane along the optical axis of the detection objective and / or how far the light sheet plane is measured from the focal plane along the optical axis of the detection objective.

[0016] At least one image is passed to a neural network. The neural network then analyses the image with the aid of defined criteria with respect to the position of the light sheet plane relative to the focal plane of the detection objective. These criteria are trained on the neural network with the aid of example data, preferably already by the manufacturer. The neural network can also learn and change the criteria and / or their weights for each detection of the light sheet microscope, with the criteria being improved over time as a number of detections are performed.

[0017] In particular, the type of specimen can also be determined before recording at least one image, and the light sheet can be adapted to the specimen by means of the light sheet parameters. This determination is preferably carried out automatically. For example, the type of specimen depends on its transparency or on the presence of a fluorescent marker. The light sheet parameters (including, among others, the color and structure of the light sheet) are then set, for example, such that they provide an optimal contrast for the identified specimen type during the recording. The light sheet parameters and the specimen type can be provided to the neural network as detection parameters in order to speed up the analysis and increase its accuracy.

[0018] As neural network, preferably a deep neural network (DNN) is used, particularly preferably a convolutional neural network (CNN). This type of neural network is particularly well suited for processing and analyzing image files. In particular, CNN is particularly well suited for applications in which the sample is illuminated by a structured light sheet (e.g. using a fringe pattern): the convolution here can be restricted to one dimension, thereby reducing the dependence on the sample structure, since then only image areas are offset from one another, which have the same phase of the light sheet structuring. Advantageously, the receptive field of the CNN is restricted to the period of the light sheet, in order to further reduce the dependence on the sample. These two measures can be used alternatively or in combination.

[0019] If a single image is recorded, the neural network at least draws a conclusion as to whether the light sheet plane is (at least partially) in the focal plane of the detection objective. The better the neural network is trained, the better and more accurate the conclusions it can draw. In particular, it is then possible to determine with the aid of a single image not only whether the light sheet plane is at least partially in the focal plane of the detection objective, but also, if it is not in the focal plane, in which direction along the optical axis of the detection objective the focal plane is located and how far the focal plane of the detection objective is from the light sheet plane. The distance determination is preferably carried out on the basis of a determination of the focus quality of at least one image.

[0020] However, in a preferred embodiment of the method, a sequence of images is captured, transmitted to a neural network and analyzed by the neural network. Here, the image sequence can be generated in different ways. In a first embodiment, the image sequence can be captured at a constant position of the focus of the detection objective and at different positions of the light sheet along the optical axis of the illumination objective. The program is able to determine the position of the middle part of the light sheet relative to the optical axis of the detection objective and, if necessary, to position it so that the optical axis of the detection objective passes through the middle part of the light sheet. The light sheet is thinnest at this point, viewed perpendicularly to the plane of the light sheet, so that the illuminated sample area along the optical axis of the detection objective also has the smallest extension, which is beneficial for detection by light sheet microscopy. Advantageously, the sequence images are combined into an overall image before being transmitted to the neural network, and the overall image is then transmitted to the neural network.

[0021] However, in the second design, the image sequence can also be captured at different positions of the light sheet or light sheet plane relative to the focal plane of the detection objective along the optical axis of the detection objective (i.e. around the focal plane). To this end, the position of the light sheet or the focal plane of the detection objective can be changed. These positions can be equidistant or variably spaced from each other. The image sequence captured at equidistant positions of the light sheet along the optical axis (also called the Z axis) of the detection objective is also called a Z stack. In the simplest case, this Z stack is transmitted to the neural network for analysis without further processing, wherein each individual image of the sequence is usually transmitted to the network via its own channel. The advantage of the analysis of a single image relative to the only one is that important information from the plane adjacent to the focal plane can be used. Therefore, for example, the neural network can evaluate the structural clarity between the planes, and thus determine with less effort during the analysis of a single image in which direction the light sheet must be moved along the optical axis of the detection objective and how far it must be moved if necessary, so as to be in the focal plane of the light sheet. Here, the processing of the neural network can be performed on the entire Z stack, and then the number of input channels of the neural network corresponds to the number of shots in the Z stack. However, it is also possible to limit the processing to a part of the Z stack according to the three-dimensional folding.

[0022] However, the use of Z-stacks predetermines relatively rigid boundary conditions for the neural network, such as equidistant distances along the optical axis of the detection objective and the number of images. If a network is trained for such a stack, the conditions of the network can no longer be changed, i.e., different trained networks must also be used for Z-stacks with different parameters. However, the use of Z-stacks provides the advantage that public information of all planes is available. In a particularly preferred design, the number of images and the distances between the images or the positions of the images along the optical axis of the detection objective are allowed to be handled more flexibly, and before being transmitted to the neural network, the image sequence (but not the Z-stack) is also combined into an overall image instead. This is done by setting individual images from different focus positions in close proximity and thus merging them into a two-dimensional overall image along an axis or direction in the image coordinate system. In the Cartesian image coordinate system, the following axis is preferably selected along which the number of pixels in the image is small; however, for the neural network, this itself does not have any effect. Then only the overall image is transmitted to the neural network via a single channel. Fully convolutional neural networks (FCNs) are also particularly suitable for analyzing such overall images. FCNs can be trained with a variable number of shots, and FCNs can then also be applied to a variable number of shots. The distance of the recordings along the Z axis does not have to be determined in advance and can also vary. In the analysis of the overall image, it is also advantageous to take information from adjacent planes (in this case adjacent images) into account.

[0023] In the simplest case, in particular when the network is trained with the aid of a single image, the network indicates whether the light sheet plane is at least partially in the focal plane of the detection objective or not in the focal plane of the detection objective. The network can then also analyze in which direction of the focal plane and / or how far from the focal plane the light sheet plane is, in particular when the network has already been trained and / or a sequence of images has already been recorded. Here, the distance is measured along the optical axis of the detection objective, which is particularly important when the light sheet plane forms an angle different from 90° with the focal plane of the detection objective. In the case of a sufficiently trained neural network, a single image is sufficient even in cases where the direction and / or distance can be determined more precisely in an unknown system with the aid of a Z-stack or an overall image.

[0024] In addition to the classification in the categories "in focus" and "out of focus" and in the subcategories "above the focal plane" and "below the focal plane", in a particularly advantageous embodiment, the neural network can also determine the specific distance to the focal plane, which can then be output to indicate to the user how the light sheet must be adjusted, or to position the light sheet correctly in the focal plane in the region of the optical axis of the detection objective by relative displacement of the light sheet plane and the focal plane of the detection objective. If the light sheet is correctly positioned, the actual detection of the specimen is then carried out by recording a single image or a stack of images again, wherein in this case the specimen stage is usually moved since the detection objective is focused on the light sheet.

[0025] In order to determine the distance of the light sheet plane from the focal plane of the detection objective, the focus quality is determined for at least one image in a targeted manner. This can also be normalized in the range between 0 and 1, and then a so-called score corresponding to the focus quality is assigned to the image. A score of 1 here means that the light sheet plane is located in the focal plane; a decrease in the score corresponds to an increase in the distance between the focal plane of the detection objective and the light sheet plane. In particular, when recording a sequence of images, these images can preferably be combined into an overall image by FCN before the analysis. The neural network then determines the focus quality for each image in the overall image and determines a score from the focus quality, which score is usually stored for the overall image as a score vector and contains a number of elements corresponding to the number of images in the overall image. In this case, the focus quality of other images can also be included in the calculation of the individual scores. As a result, a vector with a score is output, with the help of which the position of the light sheet plane can then be determined, wherein the light sheet plane coincides with the focal plane of the detection objective in the area of ​​the object field of the detection objective (for example, the center of the object field). This value can then be used to adjust the light sheet plane accordingly (automatically or manually). The highest score can then be used as the result for adjusting the light sheet plane, but a more precise value can preferably be determined by means of an interpolation method, for example by means of a polynomial regression. The light sheet plane is then preferably automatically moved to the correspondingly determined focus position of the detection objective.

[0026] A repeated search for the position is also possible. In the case of a binary classification, the positioning of the light sheet relative to the focal plane of the detection objective can be repeated, for example by shifting the light sheet plane along the optical axis of the detection objective after the result has been classified into the category "not in focus" and recording a further image and analyzing the further image by means of a neural network. At the latest after recording the second image, the direction of the focal plane of the detection objective relative to the light sheet plane can also be determined by comparison. In this way, the light sheet plane can be repeatedly brought into line with the focal plane.

[0027] In the case of binary classification, another possibility for repeated search is that first a first sequence of images is captured along the optical axis of the detection objective in different positions of the light sheet relative to the focal plane of the detection objective, i.e., a single image is captured at a possibly large fixed distance along the optical axis of the detection objective, and the focus quality is determined for each image by a neural network, for example using the calculated confidence. The neural network then determines the two images of the first sequence with the best focus quality along the optical axis of the detection objective. In the case of equidistant capture, these two images usually have the highest focus quality. The position of the light sheet on the optical axis of the detection objective, in which the two images are captured, defines the distance interval with the best focus quality. Between the positions along the optical axis associated with the two images of the first sequence, i.e., in the distance interval with the best focus quality, another sequence of images is now captured along the optical axis of the detection objective in different positions of the light sheet relative to the focal plane of the detection objective, i.e., a single image is captured again, but only within the interval with the best focus quality. The measured distance is thus reduced and the search is improved. These steps are repeated until the focus position is found. As a termination criterion for the repetition, it can be defined, for example, that the distance between the two images with the best focus quality is below a predetermined value. Alternatively or in combination, the fact that the focus quality of both images exceeds a predetermined value can also be used as a stop criterion. Any of these positions can then be equated to the focal plane of the detection objective within a predetermined tolerance defined by the stop criterion or determined by interpolation from the values ​​of the focus quality at different positions.

[0028] Similarly, it can also be provided when recording a Z stack, which in this case corresponds to the individual image sequences. Here, a Z stack can be recorded along the optical axis of the detection objective with different finely resolved distances of the individual images of the stack. The advantage of this method is that the entire search area does not have to be searched with high resolution in terms of distance.

[0029] Another possibility for reaching the targeted positioning light sheet faster with possibly slightly reduced accuracy compared to a process based on binary classification is to determine the distance of the light sheet plane in the instantaneous position in each image from the focal plane of the detection objective, i.e. the spacing, for at least one image by regression. Here, for example, the spacing of a continuous set of numbers can be determined by a neural network with the aid of regression or the spacing of a discrete set of numbers can be determined with the aid of ordered regression. For this purpose, a Z stack can also be captured, but also individual images can be captured at different spacings between the light sheet plane and the focal plane of the detection objective. Each image is evaluated by the neural network with respect to the determination of the criteria. The neural network determines, for example, a spacing value for at least one image, which gives how far the position of the light sheet at which the image was captured is from the focal plane of the detection objective; in the case of a predetermined sampling rate at a fixed spacing along the optical axis of the detection objective, the neural network can also determine, for example, a step value, which gives how many sampling steps are required under these conditions to reach the focal position. The individual results are then combined into a final decision, for example by averaging (e.g. if the spacing or number of steps from the light sheet to the focal position of the detection objective is estimated at different positions along the optical axis of the detection objective, then the number of steps or the spacing changes can be normalized relative to each other based on the fact that the number of steps or the spacing changes are known to each other). The predictions normalized to each other can then be averaged in order to make a final estimate; this final estimate or decision results in a prediction of the spacing from the light sheet plane to the focal plane of the detection objective. Together with the value of the spacing, a confidence level can also be output, which indicates how reliable the determination of the spacing is.

[0030] In all cases, after the neural network analysis and determination of the position at which the light sheet plane is at least partially located in the focal plane of the detection objective, in a preferred design, the light sheet is automatically positioned so that the light sheet plane coincides with the focal plane of the detection objective in the region of the object field of the detection objective. Preferably, it is noted that there is no collision between the objective and the sample stage or the sample. If the optical axis of the detection objective forms an angle of 90° with the light sheet plane, there is a coincidence in the entire object field. This is not the case when the optical axis of the detection objective is in an inclined position relative to the light sheet plane, i.e. when the light sheet is tilted, so that a coincidence can only be achieved in the region of the object field. As long as at least one image is not divided into sub-regions, the region is preferably located in the middle of the object field. Of course, it is also possible to only output the distance and prompt the user how to adjust the light sheet.

[0031] In particular for such an inclined position, a more flexible type of focusing is achieved if at least one image is divided into sub-areas and for each sub-area it is determined with the aid of a neural network in which direction the light sheet plane lies with respect to the focal plane of the detection objective and / or how far the light sheet plane is from the focal plane of the detection objective (measured parallel to the optical axis of the detection objective). If a sub-area is selected and the distance between the light sheet plane and the focal plane of the detection objective is determined for this sub-area, then only for this sub-area the light sheet plane coincides with the focal plane of the detection objective in the corresponding region of the object field of the detection objective. This enables a flexible adjustment of the light sheet in an inclined position, for example without having to move the sample stage in order to bring a partial region into the center of the object field.

[0032] In order to increase the robustness of the method, the type of sample can be determined before taking at least one image. This provides background information that can be transmitted to the neural network as a parameter. Depending on the type of sample (which also includes the type of experiment to be performed), in particular the parameters of the light sheet can be determined and the neural network can be selected. For example, the type of sample also includes what dye is used to mark the sample. Accordingly, a laser with a wavelength that excites the dye to fluoresce can be selected. The neural network can also be selected according to the type of sample. Purposefully, here, different parameterized versions of the same neural network model (i.e., with the same architecture) trained according to different samples and / or light sheet types are available, from which to choose. Of course, it is also possible to choose between different network architectures. The greater the difference in parameterization with respect to the sample and light sheet type, the more robust and less complex the network is.

[0033] In order to design the determination of the distance between the light sheet plane and the focal plane of the detection objective lens more easily and more efficiently, preferably, before determining whether the light sheet is at least partially in the focal plane of the detection objective lens, at least one image is pre-analyzed, whether the sample is in the image, and if the sample is in the image, whether the sample is completely suitable for the determination. This pre-analysis can be performed automatically with the help of the overall image, for example, with the help of classical methods, but also with the help of a neural network or a neural network provided for pre-analysis, and the image is detected for the area of ​​interest (ROI) that allows the above determination. For this purpose, at least one image is preferably divided into sub-regions that are pre-analyzed individually. The sub-regions can then be divided into suitable and unsuitable regions. The sub-regions have a pixel size of at least a planar detector. Of course, the pre-analysis can also be performed manually, that is, the ROI is selected. Especially when training a neural network, this manual selection will be the preferred path to minimize the error rate and create an automatic model to find such an ROI. The automatic recognition of the ROI can also be performed with the help of conventional detection and image evaluation, and neural networks can also be used if necessary: ​​the image is photographed and the area where the ROI exists or does not exist is detected. The output is then carried out in a list with the plane coordinates and, if necessary, with information on the size of the region within the coordinates. This list, which has the coordinates of such regions with ROIs or positions that are not suitable for determining the position of the light sheet plane relative to the focal plane of the detection objective, can then be transferred to the neural network again, so that the following analysis is restricted to the ROIs, or the determined unsuitable regions are not included in the analysis.

[0034] Advantageously, at least one image is preprocessed before the analysis or pre-analysis by means of image processing algorithms known in the prior art, which include, for example, algorithms for contrast improvement, filtering, projection or normalization.

[0035] The method can be easily integrated into existing light sheet microscopes, which can motorize the movement of the objective lens, the sample stage and the adjustment of the position of the light sheet. Such a light sheet microscope for performing the above method comprises an illumination device for generating a light sheet, the illumination device having an illumination objective lens for illuminating the sample using the light sheet. The light sheet microscope also comprises a detection device having a detection objective lens for imaging the light emitted by the sample onto a planar detector. Here, the optical axis of the detection objective lens and the optical axis of the illumination objective lens form an angle different from 0° and 180°. The angle is preferably 90°. The optical axis of the detection objective lens passes through the light sheet in the plane of the light sheet. Finally, the light sheet microscope has an image evaluation unit connected to the planar detector for evaluating at least one image recorded by the planar detector. Here, a neural network is implemented in the image evaluation unit, which determines during the evaluation whether the light sheet is at least partially located in the focal plane of the detection objective lens. In order to automatically adjust the position of the focal plane of the light sheet and / or the detection objective as a function of the evaluation by the neural network, the light sheet microscope preferably comprises a control unit such that the light sheet plane coincides with the focal plane of the detection objective in the region of the object field of the detection objective or, for an inclined position of the light sheet, in a previously determined sub-region of the object field.

[0036] It goes without saying that the features mentioned above and the features still to be explained below can be used not only in the combination specified but also in other combinations or alone, without departing from the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The present invention is explained in more detail below with reference to the accompanying drawings with the aid of exemplary embodiments, which also disclose features essential to the present invention. These embodiments are for illustration only and should not be construed as restrictive. For example, the description of an embodiment with multiple elements or components should not be construed as implying that all these elements or components are necessary for implementation. On the contrary, other embodiments may include alternative elements and components, fewer elements or components, or additional elements or components. Unless otherwise stated, the elements or components of different exemplary embodiments may be combined with each other. The modifications and variations described for one of the embodiments may also be applicable to other embodiments. To avoid repetition, the same or corresponding elements in different drawings are represented by the same reference numerals and will not be repeated. Wherein:

[0038] Figure 1 Light sheet microscopy is shown;

[0039] Figure 2 A partial view of a light sheet microscope is shown;

[0040] Figure 3 A typical method flow for determining the distance between the light sheet plane and the focal plane of the detection objective is shown;

[0041] Figure 4 showing images at different distances of the light sheet from the focal plane of the detection objective; and

[0042] Figure 5 Shows the results of the neural network's analysis of the image. DETAILED DESCRIPTION

[0043] Figure 1 First, the basic structure of a light sheet microscope is shown, which can be used for light sheet microscopic examination of a sample. Here, the light sheet microscope is shown in an inverted configuration, which can be understood only as an example, that an optical microscope in which the sample can be observed from above or from the side is also a feasible design. The sample 1 is located in a sample chamber 2 and is surrounded by a liquid 3, such as water or a nutrient solution. The sample chamber 2 has side walls and a bottom made of glass with a preset thickness, which is usually equal to the thickness of a common microscope slide, for example 0.17 mm. The sample chamber 2 is supported on a sample stage 4, which can be moved manually or by a motor in three spatial directions. The individual elements of the light sheet microscope are arranged below the sample chamber 2, which has a transparent bottom 5. Between the objective lens of the light sheet microscope and the bottom 5 of the sample chamber 2, there is a so-called virtual relay 6, which has an inner lens and an outer lens. There is also a liquid 3 between the inner lens of the virtual relay 6 and the bottom 5 of the sample chamber. There is a surrounding atmosphere, usually air, between the inner lens and the outer lens of the virtual repeater 6 , and there is also a surrounding atmosphere between the outer lens of the virtual repeater 6 and the objective lens of the light sheet microscope.

[0044] The virtual repeater 6 serves to compensate for aberrations caused by the fact that the optical axes of the illumination objective and the detection objective are not arranged perpendicularly to the bottom 5 of the sample chamber 2. In performing this correction, other correction mechanisms can also be used instead of the virtual repeater 6, such as a front lens or a free-form lens, which is integrated in the objective. In particular, a concentric lens can be used instead of a virtual repeater to adapt to the refractive index of the sample, and aberrations can be corrected by means of a free-form surface inside the objective, by an inclined cover glass channel or by adaptive optics outside the objective. Observation from above requires a different configuration, and then usually an immersion objective is used, which is directly immersed in the liquid 3.

[0045] On the left, a lighting device with an illumination beam path is shown. The light from a laser module 7 is guided via a beam shaping module 8 and a scanning module 9, for which, for example, several lasers of different wavelengths can be accommodated here and a selection can be made between different wavelengths, wherein a plurality of wavelengths can also be selected simultaneously; the beam shaping module 8 and the scanning module 9 can be used, for example, to generate a quasi-static light sheet and / or for angular scanning; the illumination objective 10 images a light sheet into a light sheet plane, which here includes the optical axis of the illumination objective, onto the sample. The sample is thus illuminated by the light sheet via the illumination objective 10. The focus of the illumination objective 10, i.e. the point at which the light sheet has the thinnest extension (also referred to as the middle part), can be adjusted in the direction indicated by the double arrow by means of a positioning unit 11. In the light sheet plane along the optical axis of the illumination objective 10, for example, a piezoelectric drive can be used for adjustment; this movement is indicated by a short double arrow. Perpendicular to the light sheet plane (indicated by a long double arrow), positioning can be performed, for example, by means of a tiltable mirror or a galvanometer scanner arranged in the pupil plane of the illumination objective 10.

[0046] On the right, a detection device with an exemplary detection beam path is shown. The detection device comprises a detection objective 12, which, similar to the illumination objective 10, can be adjusted by means of a drive, here only a piezoelectric drive 13. The optical axis of the detection objective 12 and the light sheet plane in which the optical axis of the illumination objective 10 is located enclose an angle different from 0° and 180°, here a right angle. However, this is not absolutely necessary; the method can also be implemented at a non-zero angle between the light sheet plane and the optical axis of the detection objective 12. The optical axis of the detection objective 12 passes through the light sheet in the light sheet plane. The fluorescence emitted by the sample 1 is guided by the detection objective 12 to the detection module 14. A planar detector is located in the detection module 14, which records the intensity and converts the intensity into a corresponding electrical signal, which then flows into the image processing device. At least the detection module 14 and the positioning unit 11 are connected to the control unit 15. As long as they can be controlled, other elements can also be connected to the control unit 15, but for the sake of clarity, the corresponding connections are not shown here.

[0047] An image evaluation unit is integrated into the control unit 15, via which the image evaluation unit is connected to the flat detector. The image evaluation unit serves to evaluate the image recorded by the flat detector. A neural network is again implemented in the image evaluation unit, which determines during the evaluation whether the light sheet is at least partially in the focal plane of the detection objective 12. A control unit for automatically adjusting the position of the light sheet and / or the focal plane of the detection objective is also integrated into the control unit 15. Based on the evaluation of the neural network, the control unit adjusts the position of the light sheet and / or the focal plane of the detection objective so that the light sheet plane coincides with the focal plane of the detection objective in the region of the object field of the detection objective.

[0048] Figure 2 The illumination and detection are shown in detail, wherein the sample 1 is not shown here for the sake of clarity. The optical axis 16 of the illumination objective 10 and the optical axis 17 of the detection objective 12 are at right angles to each other. The illumination objective 16 radiates a light sheet 18, the plane of which is arranged perpendicularly to the paper plane and contains the optical axis 16 of the illumination objective 10. Due to illumination considerations, the light sheet 18 is not completely flat, but varies in thickness along the optical axis 17 of the detection objective 12. In the ideal state shown here, the thinnest part of the light sheet 18, i.e. the middle part of the light sheet, is located exactly in the focal plane of the detection objective 12. At this time, the optical axis 17 of the detection objective 12 passes through the light sheet plane. The double arrow again indicates the possible positioning of the light sheet 18 relative to the detection objective 12.

[0049] Figure 3 A typical method flow is shown when determining the position of the light sheet plane relative to the focal plane of the detection objective 12 within the scope of a light sheet microscopic examination of a specimen 1. In a first preparatory step 110, the specimen 1 is inserted, i.e., the specimen 1 is placed on a specimen stage. Optionally, background information can be included here to prepare for the recording. For example, the light sheet microscope can already recognize the type of inserted specimen and can set corresponding parameters for the light sheet, so as to optimize the contrast, for example. Of course, the setting of the specimen type and the light sheet parameters can also be entered manually. Typical light sheet parameters first include the type of light sheet, for example, whether the light sheet is structured, and the color of the light sheet selected according to one or more markings. Other parameters also include the length and thickness of the light sheet, which are also selected depending on the specimen. For example, for yeast cells with an extension between 2 μm and 5 μm, very thin and short light sheets are preferred, while for multicellular organisms or organoids, correspondingly longer and thicker light sheets are used.

[0050] Then, in step 120, at least one image of the sample 1 is recorded by means of a planar detector. For this purpose, the sample 1 is illuminated by means of an illumination objective 10 via a light sheet 18. The light emitted by the sample 1 is imaged onto the planar detector by means of a detection objective 12. In this case, a single image can be recorded, but also a sequence of images (each in a constant position of the focus), which is done either at different positions of the light sheet 18 along the optical axis 16 of the illumination objective 10 or at different positions along the optical axis 17 of the detection objective 12.

[0051] In step 130, the images or image sequences are preprocessed. This is done with the aid of image processing algorithms known in the prior art, for example. For example, algorithms for contrast improvement, standardization, filtering and projection can be applied to at least one image. In addition, the images or their data are brought into a form suitable for the analysis described further below; this applies in particular to the image sequences that were taken. For example, these images can be combined into a two-dimensional overall image or a stereoscopic image stack, a so-called Z stack, which is only combined into a Z stack if the image sequence is taken at different positions along the optical axis 17 of the detection objective 12.

[0052] In step 140, at least one image is pre-analyzed as to whether the specimen 1 is present in the image and is suitable for the subsequent analysis. With the aid of this evaluation, a decision is made in step 150 as to whether the analysis can be started. If this is not the case, step 120 is performed again, but before that a new region of the specimen 1 is selected in step 155. This can be done automatically, for example, with the aid of a panoramic recording (using a separate objective to produce a panoramic recording) or by the user.

[0053] Steps 130 and 140 may also be interchanged; in this case, query 150 would occur between steps 130 and 140. However, in this case, it is often helpful to prepare in advance a weak structure suitable for the analysis described below. Furthermore, the preliminary analysis in step 140 may also be performed by a neural network or other neural network, in which case it makes sense to perform step 130 before this.

[0054] If the preprocessing and pre-analysis have been completed successfully, at least one image can be transmitted to a neural network, which is implemented in an image evaluation unit as part of the control unit 15. With the help of the neural network, in step 160, it is determined by analyzing at least one image whether the light sheet plane is at least partially in the focal plane of the detection objective 12. For this, in the simplest case, a single image is sufficient. In particular, for a neural network that has been pretrained by the manufacturer or by a large number of samples that have already been detected, this single image is sufficient to determine in which direction of the focal plane of the detection objective 12 the light sheet is located and / or how far the light sheet is from the focal plane of the detection objective 12 measured along the optical axis 17 of the detection objective 12. In this case, the neural network can be preselected from a plurality of available neural networks in a sample- and / or illumination-specific manner. Convolutional neural networks (CNNs) are particularly suitable for image analysis. CNNs are suitable for the illumination of structured light sheets, wherein only one-dimensional convolutions are performed to reduce the dependence on individual samples.

[0055] In step 170, the result of the analysis is evaluated by the neural network. If the position of the light sheet plane relative to the focal plane of the detection objective 12 is determined with sufficient accuracy or confidence, in step 180 this is either output by informing the user in which direction and by how much the light sheet plane must be adjusted, or the light sheet plane is advantageously made to coincide with the focal plane of the detection objective 12 in the region of the object field of the detection objective 12. In particular in the case of a tilt of the light sheet 18 relative to the focal plane of the detection objective 12, it may be expedient to divide the at least one image into a plurality of sub-regions and then to analyze each sub-region individually by the neural network, i.e. to determine for each sub-region individually in which direction of the focal plane of the detection objective 12 the light sheet plane is located and / or how far the light sheet plane is from the focal plane of the detection objective 12 measured along the optical axis of the detection objective 12. If a particular sub-region is selected, the light sheet plane can be made to coincide with the focal plane of the detection objective 12 for this sub-region. For this purpose, the light sheet 18 can be positioned accordingly in the focal plane of the detection objective 12 by means of a galvanometric scanner in the pupil plane of the illumination objective 10. Alternatively, the focus of the detection objective 12 itself can also be moved, for example by means of a piezoelectric drive 13.

[0056] On the other hand, if the analysis of the neural network shows that it is still not possible to determine in which direction the focal plane of the detection objective 12 is located in the light sheet plane and / or how far the focal plane of the detection objective 12 is from the light sheet plane with the aid of at least one image, adjustments are performed in step 175, and at least one image is captured and analyzed again. When capturing a single image, the direction in which the light sheet plane must be moved can be determined with the aid of the image, and these adjustments can move the position of the light sheet 18 in this direction. In this way, the search area can be repeatedly narrowed, and the correct position of the light sheet plane that coincides with the focal plane of the detection objective 12 can be determined. It is also possible to capture successively at different spacings between the light sheet plane and the focal plane of the detection objective 12, wherein the spacings between adjacent light sheet planes are fixed in each case. These captures are then analyzed individually by the neural network, and for each capture, for example, the focus quality is determined, which is performed, for example, by using the confidence calculated by the neural network. The search is then refined in the spacing interval with the two highest confidences, in that for this interval, the sample 1 is captured again successively at fixed spacings between adjacent light sheet planes, but on a finer spacing grid. In this way, the exact distance of the light sheet plane from the focal plane of detection objective 12 can also be determined repeatedly, and can then be made to coincide with the focal plane in the region of the object field of detection objective 12 .

[0057] Alternatively, the spatial overall image or a Z-stack of images generated as described above can also be transmitted to the neural network for analysis. The Z-stack or the two-dimensional overall image composed of images taken spatially separated from each other along the optical axis 17 of the detection objective 12 is then analyzed as a whole. In particular, when using the overall image, a fully convolutional neural network (FCN) is suitable for analysis, because variable spatial distances of the light sheet plane between shots and a variable number of shots can be processed in the fully convolutional neural network. For example, a larger spacing can be selected along the optical axis 17 in the area of ​​the extreme position of the light sheet plane relative to the optical axis 17 of the detection objective 12, and a smaller spacing can be selected in the area of ​​the inferred or possible position of the focal plane of the detection objective 12.

[0058] The use of an image sequence in particular allows the correct position of the light sheet plane in the focal plane of the detection objective 12 to be determined by means of a polynomial or ordinal regression. In this case, the images do not have to be combined into a Z stack. Figure 4 and Figure 5 To explain. Figure 4 Three shots from a sequence of a total of eleven images are shown, which were recorded at different positions of the light sheet plane and the focal plane of the detection objective 12 relative to one another. In order to adjust the position of the light sheet plane relative to the focal plane of the detection objective 12, the position of the light sheet can be adjusted along the optical axis 17 of the detection objective 12 by means of a galvanometric scanner, or the position of the focal plane of the detection objective 12 can be changed. In this case, the spacing between the light sheet plane and the focal plane, which are equidistant here but can also be easily selected to be variable, varies between the individual shots in each case by a constant amount.

[0059] For each of the three images, a partial view of a specimen with two cells is shown in an enlarged manner. The images have frayed edges at positions 1 and 11, which indicates that there is blur in the individual shots. On the other hand, in the image captured at position 6, the edge contours of the cells are relatively sharp, which indicates a higher definition. In the present example, with the aid of the focus quality, a so-called score is now calculated for each image, which score is higher the smaller the spacing between the light sheet plane and the focal plane of the detection objective 12 in the selected area. This score is output as a score vector, with one element of the vector corresponding to each position. Figure 5In FIG. 1 , for each detected position, the score vector is displayed. If an ordinal regression is applied to the score vector to determine the position of the light sheet plane in the focal plane of the detection objective 12 in the sample area, this position is obtained at position 6 with the highest score. When a polynomial, such as a quadratic regression, is used, a value between position 6 and position 7 is obtained. Depending on the currently set position between the light sheet plane and the focal plane in the selected area, it can be displayed to the user: in which direction the relative position change is to be carried out by adjusting the light sheet position or by adjusting the focal position of the detection objective 12; advantageously, the correct position of the light sheet plane coinciding with the focal plane of the detection objective 12 is automatically set. Finally, the actual light sheet microscopy detection of the sample 1 can be carried out.

[0060] The above method provides the possibility of aligning the light sheet plane and the focal plane of the detection objective 12 in a reproducible manner for a large number of samples, even after sufficient training for unknown samples, so that they are essentially congruent. This reduces the error susceptibility in light sheet microscopy detection and allows better comparison of detection results.

[0061] List of Figures

[0062] 1. Sample

[0063] 2 Sample chamber

[0064] 3 Liquid

[0065] 4. Sample table

[0066] 5. Transparent bottom

[0067] 6 Virtual Repeater

[0068] 7 Laser Module

[0069] 8 Beam Shaping Module

[0070] 9 Scanning module

[0071] 10 Illumination objective

[0072] 11 Positioning unit

[0073] 12. Detection lens

[0074] 13 Piezoelectric Actuator

[0075] 14 Detection Module

[0076] 15 Control Unit

[0077] 16 Optical axis of the illumination objective

[0078] 17 Check the optical axis of the objective lens

[0079] 18 Light Sheet

Claims

1. A method for light sheet microscopy of a sample (1), wherein The sample (1) is illuminated by means of an illumination objective (10) through a light sheet (18), The light emitted by the sample (1) is imaged onto a planar detector by means of a detection objective lens (12), wherein: The optical axis (17) of the detection objective (12) encloses an angle different from 0° and 180° with the optical axis (16) of the illumination objective (10) and passes through the light sheet (18) in the light sheet plane, and The planar detector captures at least one image, characterized in that By analyzing the at least one image, the following aspects are determined using a neural network: i. whether the light sheet plane is at least partially in the focal plane of the detection objective (12), and / or ii along the optical axis (17) of the detection objective lens (12), in which direction of the focal plane the light sheet plane is located, and / or iii. How far the light sheet plane is from the focal plane, measured along the optical axis (17) of the detection objective lens (12).

2. The method according to claim 1, characterized in that: An image sequence is captured at a constant position of the focus of the detection objective (12) and at different positions of the light sheet (18) along the optical axis (16) of the illumination objective (10), or at different positions of the light sheet (18) relative to the focal plane of the detection objective (12) along the optical axis (17) of the detection objective (12), transmitted to the neural network for analysis, and analyzed by the neural network.

3. The method according to claim 2, characterized in that When the image sequence is captured at different positions of the light sheet (18) along the optical axis (16) of the illumination objective (10), the image sequence is combined into an overall image before being analyzed by the neural network; and when the image sequence is captured at different positions along the optical axis (17) of the detection objective (12), the image sequence is combined into a stereoscopic image stack or an overall image before being analyzed by the neural network.

4. The method according to claim 3, wherein: The image sequence is combined into a stereoscopic image stack and the neural network will restrict analysis to only a portion of the images of the stereoscopic image stack.

5. The method according to any one of claims 1 to 4, characterized in that In order to determine the distance of the light sheet plane from the focal plane of the detection objective (12), a focus quality is determined for the at least one image.

6. The method according to any one of claims 1 to 4, wherein capturing a sequence of images at different positions of the light sheet (18) relative to the focal plane of the detection objective lens (12) along the optical axis (17) of the detection objective lens (12), combining the image sequences into an overall image and transmitting the overall image to a neural network for analysis, To determine the distance between the light sheet plane and the focal plane of the detection objective (12), the focus quality is determined for each image in the overall image and a score is calculated by means of the neural network, and The position of the light sheet plane is determined using the fraction, wherein: The light sheet plane coincides with the focal plane of the detection objective (12) in the region of the object field of the detection objective (12).

7. The method according to any one of claims 1 to 4, wherein: Determining whether the light sheet plane is at least partially located in the focal plane of the detection objective (12), characterized in that the position of the light sheet plane is repeatedly determined, the light sheet plane coinciding with the focal plane of the detection objective (12) in the region of the object field of the detection objective (12).

8. The method according to claim 7, wherein a) recording a first sequence of images along the optical axis (17) of the detection objective (12) in different positions of the light sheet (18) relative to the focal plane of the detection objective (12), b) determining the focus quality for each image by means of the neural network, c) determining by means of the neural network those two images in the first sequence which have the best focus quality among the images, d) recording a further sequence of images along the optical axis (17) between the positions corresponding to the two images in the first sequence, in a different position of the light sheet (18) relative to the focal plane of the detection objective (12) along the optical axis (17) of the detection objective (12), and e) Repeating steps b) to d) until the distance between two images with the best focus quality between the images is lower than a predetermined value and / or the focus quality of the two images exceeds a predetermined value.

9. The method according to any one of claims 1 to 4, characterized in that For the at least one image, a distance between the light sheet plane and the focal plane is determined by regression using the neural network.

10. The method according to any one of claims 1 to 4, characterized in that After determining the position of the light sheet plane at least partially in the focal plane of the detection objective, the light sheet plane is made to coincide with the focal plane of the detection objective (12) in the region of the object field of the detection objective (12).

11. The method according to any one of claims 1 to 4, characterized in that The at least one image is divided into a plurality of sub-areas and the neural network is used to determine for each sub-area: in which direction the light sheet plane is located in the focal plane of the detection objective (12), and / or how far the light sheet plane is from the focal plane of the detection objective measured parallel to the optical axis (17) of the detection objective (12), and, when a sub-area is selected and the distance is determined, the light sheet plane is made consistent with the focal plane of the detection objective (12) in the area of ​​the object field of the detection objective (12) for the sub-area.

12. The method according to any one of claims 1 to 4, characterized in that The type of the sample (1) is determined before recording the at least one image, and parameters are determined for the light sheet (18) and a neural network is selected based on the type of the sample (1).

13. The method according to any one of claims 1 to 4, characterized in that A deep neural network (DNN) is used as the neural network.

14. The method according to claim 13, characterized in that A convolutional neural network (CNN) is used as the neural network.

15. The method according to claim 13, wherein: Illumination is performed through a periodically structured light sheet (18) and a convolutional neural network is used which is limited to one or two-dimensional convolutions and / or the receptive field of the convolutional neural network is limited to one period of the light sheet (18).

16. The method according to any one of claims 1 to 4, characterized in that Before the at least one image is analyzed by the neural network, the at least one image is pre-analyzed as to whether the sample (1) is located in the image and, if the sample (1) is located in the image, whether the sample (1) is suitable for the aforementioned determination method.

17. The method according to claim 16, characterized in that The at least one image used for the pre-analysis is divided into a plurality of individually pre-analyzed sub-regions and / or the at least one image is pre-analyzed with the aid of the neural network.

18. The method according to any one of claims 1 to 4, characterized in that Prior to the analysis or pre-analysis, the at least one image is pre-processed by means of an image processing algorithm.

19. A light sheet microscope for implementing the method according to any one of claims 1 to 18, comprising an illumination device for producing a light sheet, the illumination device having an illumination objective (10) for illuminating a sample (1) through the light sheet (18), A detection device having a detection objective lens (12) for imaging the light emitted from the sample (1) onto a planar detector, wherein: The optical axis (17) of the detection objective (12) forms an angle different from 0° and 180° with the optical axis (16) of the illumination objective (10) and passes through the light sheet (18) in the light sheet plane, An image evaluation unit connected to the planar detector for evaluating at least one image recorded by the planar detector, characterized in that A neural network is implemented in the image evaluation unit, which determines the following during the evaluation: i. whether the light sheet (18) is at least partially in the focal plane of the detection objective, and / or ii. along the optical axis (17) of the detection objective lens (12), in which direction of the focal plane the light sheet plane is located, and / or iii. Measure along the optical axis (17) of the detection objective lens (12) to determine how far the light sheet plane is from the focal plane.

20. The light sheet microscope according to claim 19 comprises a control unit for automatically adjusting the position of the light sheet (18) and / or the position of the focal plane of the detection objective (12) based on the evaluation of the neural network, so that the light sheet plane coincides with the focal plane of the detection objective (12) in the region of the object field of the detection objective (12).

Images