Method and microscope for generating an overview image of a sample

Abstract

The present invention proposes a method for generating an overview image of a sample, which is arranged in an observation space of a microscope with the aid of a sample carrier, wherein the sample carrier is illuminated by a first illumination, wherein a preliminary overview image is generated using the first illumination and an overview camera of the microscope, wherein an overview image illumination is selected based on the preliminary overview image, wherein the sample carrier is illuminated by the overview illumination, and wherein the overview image is generated using the overview image illumination and the overview camera.

Description

Technical Field

[0001] The invention relates to a method for generating an overview image of a sample which is arranged in an observation volume of a microscope by means of a sample carrier. Furthermore, the invention relates to a microscope which is configured to carry out the method. Background Art

[0002] Due to the small object field and low depth of field of objectives at high magnification, users of microscopes find it difficult to arrange samples in the observation space so that the features of the sample to be examined are located in the object field of the objective with high magnification.

[0003] Therefore, DE 10 2017 111 718 A1 proposes a method for generating and analyzing an overview contrast image of a sample carrier and / or a sample arranged on a sample carrier. In the process, a sample carrier at least partially arranged in the focus of a detection optical unit is illuminated with transmitted light of a two-dimensional array-type illumination pattern. At least two raw overview images are detected with different illuminations of the sample carrier, and based on the information to be extracted from the overview contrast images, a calculation algorithm is selected for calculating an overview contrast image from the at least two raw overview images. Finally, based on the information to be extracted from the overview contrast image, an image evaluation algorithm is selected for extracting the information. The overview contrast image can be provided to a user for navigation purposes. Summary of the invention

[0004] Different types of sample carriers are used for examining samples with the aid of a microscope. For example, sample carriers used include so-called multiwell plates, chamber slides, culture dishes, slides with cover slips, and the like.

[0005] The object of the present invention is to develop the known method to the effect that an overview image of the sample independent of the sample carrier can be generated very reliably, the overview image allowing the microscope user to navigate to the desired position of the sample, i.e. to position the sample position to be inspected in the field of view and focal plane of an objective with high magnification.

[0006] This object is achieved by the features of the main claim and of the optional independent claim. Advantageous configurations are specified in the dependent claims.

[0007] A method is proposed for generating an overview image of a sample, which is arranged in an observation volume of a microscope by means of a sample carrier, wherein the sample carrier is illuminated by a first illumination, wherein a preliminary overview image is generated using the first illumination and an overview camera of the microscope, wherein an overview image illumination is selected based on the preliminary overview image, wherein the sample carrier is illuminated by the overview illumination, wherein a decision is made based on the preliminary overview image as to how many recordings are required with the overview camera in order to record the entire sample carrier area, and wherein the overview image is generated using the overview illumination and the overview camera.

[0008] The first illumination may be reflected light illumination or transmitted light illumination. A combination of reflected light illumination and transmitted light illumination may also be provided for the first illumination.

[0009] In this case, the overview camera can record a larger object field than that recorded by the microscope objective. It takes less time to generate an overview image using an overview camera than using a microscope objective with a low magnification, in which case a plurality of individual images may optionally have to be combined using stitching techniques in order to be able to image the entire sample. Likewise, the overview camera can have a lens with a greater depth of field than the microscope objective. The overview camera can be arranged on one side of the microscope objective and arranged in an oblique manner on the sample carrier. It is also conceivable to provide a reflector at the location of the objective turret of the microscope, which reflector deflects the light from the sample carrier laterally in the direction of the overview camera. It is also conceivable that the overview camera has a non-telecentric lens.

[0010] The overview image illumination may be reflected light illumination and / or transmitted light illumination. It has been found that many sample carriers can be represented with better contrast when using transmitted light illumination, thereby improving the quality of the overview image of low contrast samples. For example, transmitted light dark field contrast may be used in this process. In the case of some sample carriers, such as multiwell plates, the structure of the sample carrier may cause the light of the transmitted light illumination to be blocked or deflected, so that the raw overview image of the overview image camera cannot always be evaluated with the desired reliability. This can make it more difficult to create an overview image of the sample carrier and the sample. If such a sample carrier is illuminated with reflected light illumination, then the overview image can usually be created very reliably, because the features of the sample carrier can be better identified and can subsequently be honed in on a certain position by the microscope with better reliability. By first generating a preliminary overview image using a first illumination and selecting the overview image illumination based on the preliminary overview image, overview images can be created for different sample carriers with better reliability. For the overview image, a combination of reflected light illumination, which facilitates better identification of the sample carrier, and transmitted light illumination, which allows samples arranged on the sample carrier (e.g. cells arranged in the wells of a multiwell plate) to be represented with better contrast, can then also be used.

[0011] The preliminary overview image may be processed in a trained machine learning based system (ML system), wherein based on the output of the ML system an overview image illumination for further overview recording is selected. For example, the ML system may have been trained using a set of training images, providing for each training image a specification as to whether the overview image may be created with better reliability using reflected light illumination or using transmitted light illumination. It is also conceivable that the ML system is used to determine which contrast is more suitable for the creation of the overview image. For example, brightfield contrast or darkfield contrast may be used for the creation of the overview image. It is also conceivable that for the creation of the overview image a brightfield contrast is combined with a darkfield contrast.

[0012] In one configuration of the method, the output of the trained ML system may include a statement about the sample carrier type of the sample carrier, and the overview lighting may be determined based on the sample carrier type. For example, it is conceivable that the ML system has been trained to determine whether the sample carrier is a multiwell plate or a slide with a cover glass based on the preliminary overview image. In this case, different overview image lighting may be selected depending on the sample carrier type. For example, reflected light lighting may be selected for a multiwell plate.

[0013] In a further configuration, the calibration plate is initially arranged in the observation space of the microscope system. The calibration plate is illuminated by calibration illumination, wherein a calibration overview image is generated using the calibration illumination and the overview camera. In turn, the calibration illumination can be reflected light illumination or transmitted light illumination. Calibration data can be obtained based on the calibration overview image. A preliminary overview image and / or an overview image are subsequently generated with the aid of the calibration data. In particular, the calibration plate can be understood to mean a plate with a predefined calibration structure. For example, the calibration plate may already be provided with a predefined pattern of reference points. The calibration plate can be constructed in such a way that it can be inserted into the stage system of the microscope in a defined manner. In particular, the calibration plate can be configured so that it can be repeatedly inserted into the stage system of the microscope in the same relative position and / or orientation. The predetermined calibration structure can allow correction of optical distortions caused by the lens of the overview camera. Similarly, perspective correction can be facilitated, by means of which a non-orthogonal alignment of the overview camera relative to the stage system of the microscope and therefore also relative to the sample carrier can be compensated. Furthermore, it is conceivable that the calibration overview image is used to describe the relationship between the pixel scaling of the original image recorded by the overview camera and the distance of the sample structure on the sample arranged on the sample carrier located in the observation space of the microscope. Furthermore, the calibration plate can be arranged at different positions in the observation space and the calibration overview image can be generated for the respective position. This can facilitate a further improvement of the above-mentioned corrections and relationships. For example, the calibration plate inserted in the stage system of the microscope can be moved laterally (i.e. in the x-direction and y-direction) and actively (i.e. in the z-direction) by means of a translation unit.

[0014] In particular, the calibration plate may include a first calibration structure. The calibration overview image may include an image representation of the first calibration structure. A first detailed image of the first calibration structure may be generated using a microscope objective of the microscope. Based on the calibration overview image and the first detailed image, calibration data for a sample detail image for locating the sample in the overview image may be obtained.

[0015] Further, the calibration plate may have a second calibration structure. After the first detailed image has been generated, the position of the calibration plate relative to the microscope objective may be changed by a predefined translation vector, and a second detailed image of the second calibration structure may be generated using the microscope objective of the microscope. Calibration data may then be obtained by additionally using the predefined translation vector and the second detailed image.

[0016] By changing the position of the calibration structures relative to the microscope, a first calibration structure can be brought into the object field of the microscope objective at one point, and then a second calibration structure. Thus, a relationship can be established between the subsequently generated overview image and the position on the sample detail image obtained using the microscope objective. In particular, the first calibration structure and the second calibration structure can be run-in several times in order to determine the relationship. By way of example, more than four calibration structures can be considered, which can also be referred to as reference points. In view of the time required to generate the calibration data and the quality of the calibration data, the use of exactly four calibration structures may represent the best results. In particular, the use of four calibration structures may be sufficient to determine the homography.

[0017] The microscope objective used to record the calibration structure can then be used to generate a sample detail image. However, it is also conceivable to use different microscope objectives arranged on the same optical axis. For example, in the case of a microscope objective wheel, the microscope objectives can be arranged on the same optical axis in a repeatable manner. It is therefore conceivable that the calibration structure does not need to be recorded for each individual microscope objective. However, for the purpose of even better positioning the sample detail image in the overview image, a detailed image with the calibration structure recorded can also be provided for each microscope objective.

[0018] In order to facilitate the reliable creation of overview images, calibration data can be determined regularly with the aid of calibration plates. In particular, the calibration data can be determined after transport or refitting of the microscope. After the calibration data or calibration data records have been determined, they can be used to create overview images of a plurality of samples. Expressed in a different way, it can be provided that the calibration data are not determined before each creation of an overview image.

[0019] In a further configuration of the method, for generating the preliminary overview image, at least two preliminary raw overview images can be recorded by the overview camera, wherein the first illumination in the preliminary raw overview images is different. For example, rapidly switchable light sources can be used, such as LEDs, which are used at least in two groups for the purpose of recording the two raw overview images. In this way, reflections of the light sources can be suppressed or reduced via a suitable digital combination by calculation, for example by selecting the minimum of the individual pixel values ​​of the respective raw overview images. It is likewise conceivable to use reflected light illumination for one preliminary raw overview image and transmitted light illumination for a further preliminary raw overview image.

[0020] Likewise, in order to generate an overview image, at least two original overview images are recorded, wherein the overview illumination of the original overview images is different. As described above, for example, it is also possible to use a fast-switching light source and suppress or reduce the reflection of the light source in a suitable digital combination by calculation. Here, the overview illumination for the original overview image can be a plurality of reflected light illuminations (for example in the case of a sample carrier in the form of a multi-well plate) or a plurality of transmitted light illuminations (for example in the case of a sample carrier in the form of a slide with a cover glass). In principle, it is also conceivable to use a combination of transmitted light illumination and reflected light illumination.

[0021] Further, a configuration of the method is proposed, wherein, for generating a preliminary overview image, at least two preliminary raw overview images are recorded by the overview camera, wherein the position of the sample carrier is changed by a known translation vector between the raw overview images, wherein mutually corresponding structures are identified in the raw overview images, and wherein an approximate focus position is determined from the mutually corresponding structures in the raw overview images, the known translation vector and the calibration data. In particular, a distance to the approximate focus position can be determined.

[0022] Further, it is proposed that at least two raw overview images are recorded by the overview camera, wherein the position of the sample carrier is changed by a displacement vector between the raw overview images, wherein the overview image is generated as a mosaic overview image based on the raw overview images and, optionally, based on the displacement vectors. In particular, this may be advantageous if the sample to be imaged in the overview image cannot be fully imaged using a single raw overview image of the overview camera. Combining a plurality of raw overview images to form a mosaic overview image may also be referred to as stitching.

[0023] Further, a microscope is proposed, which comprises an overview camera, at least one microscope objective, a transmitted light unit, a reflected light unit and a control unit, wherein the control unit is configured to perform one of the methods described above.

[0024] Further, a computer program product is disclosed comprising instructions which, when executed by a control unit of a microscope, cause the latter to perform one of the above-mentioned methods.

[0025] Finally, a calibration plate for calibrating a microscope comprising an overview camera and a microscope objective is disclosed, wherein the calibration plate comprises at least one first calibration structure which can be imaged by both the overview camera and the microscope objective, in particular when using calibration illumination.

[0026] The calibration plate can facilitate the automatic generation of calibration data with the help of which deformations and distortions of overview cameras without telecentric lenses can be corrected. This can simplify the generation of navigable overview images.

[0027] The calibration plate may comprise at least one second calibration structure at a predefined distance from the first calibration structure.Providing a plurality of calibration structures at fixed defined spatial positions may further simplify the generation of corresponding calibration data.

[0028] In particular, the calibration structures can be configured such that they can be imaged by the microscope objective in the case of a typical illumination configuration of the microscope and such that their relative position can be determined. Preferably, the relative position of the calibration structures can be determined with an accuracy of better than 20 μm.

[0029] Further, the calibration structures may be configured such that they may be imaged by the overview camera under typical illumination configurations in such a way that their relative position with respect to the image sensor of the overview camera may be determined with sub-pixel accuracy.

[0030] The calibration structures can have a spatial spacing which has a small tolerance, in particular a tolerance of less than 100 μm, and which are defined, stable and known in terms of their relative position, so that the calibration structures can be used to determine distortion and perspective correction parameters for the overview camera. Thus, when different calibration structures of the calibration plate (which can also be referred to as calibration features) are grounded, these calibration structures can be brought into the image field of the microscope objective.

[0031] In particular, the calibration plate can have a checkerboard pattern, wherein the first calibration structure corresponds to a corner of an area of ​​the checkerboard pattern. By way of example, the checkerboard pattern can comprise 20×9 areas. A corner of an area of ​​the checkerboard pattern and / or an edge of an area of ​​the checkerboard pattern can be determined with high precision both in the image generated by the overview camera and in the image generated by the microscope objective, and the checkerboard pattern is therefore particularly suitable for determining calibration data.

[0032] The calibration plate can be arranged to diffusely reflect the light. In this way, reflection effects on the surface of the calibration plate can be avoided so that the calibration structure can be imaged in the worst case with a small number of artifacts.

[0033] The calibration plate can be semi-transparent. This can allow the calibration structure to be imaged even when using transmitted light illumination. This can further simplify the ascertainability of the calibration data.

[0034] Preferably, the calibration structures of the calibration plate are embodied such that they do not degrade even under illumination with high intensity (eg due to lasers).

[0035] The size and properties of the calibration plate may be selected so that the calibration plate may be uniquely and repeatedly inserted into a receptacle of the microscope stage.

[0036] In an exemplary embodiment, the calibration plate has a substantially flat covering surface. In particular, the surface topography of the calibration plate may have a height deviation of less than 100 μm. This may allow that (re)focusing of various calibration structures for calibration features may be simplified or dispensed with when recording the calibration structures using a microscope objective.

[0037] The production process of the calibration plate can ensure that the relative position and orientation of the calibration features with respect to the insertion frame of the microscope stage for various plates has only small tolerances. In particular, relative position tolerances better than ±500 μm and / or angle tolerances better than ±0.5° can facilitate the positioning of the calibration plate in the stage.

[0038] The calibration plate can have a center calibration feature that can mark the center of the stage aperture of the microscope stage so that the center of the stage aperture can be better determined for inspection of the sample.

[0039] At least the first calibration structure of the calibration plate may be asymmetric, so that an incorrect insertion of the calibration plate into the stage of the microscope can be detected automatically.

[0040] Further, the calibration plate may have a through hole. This may allow the application of an immersion liquid on a microscope objective of the microscope without removing the calibration plate from the stage of the microscope. Thus, a reference recording may be made using a microscope objective which is provided for use in combination with an immersion liquid and which may also be referred to as an immersion objective, without the application of a sample.

[0041] In order to be able to determine the scaling factor of a microscope objective and / or to be able to carry out a focus comparison of a plurality of microscope objectives and / or to be able to determine offsets between a plurality of microscope objectives, a calibration structure can be provided in addition to the checkerboard pattern.

[0042] Attention is drawn to the fact that exemplary embodiments of the invention may be described with reference to different implementation categories. Some exemplary embodiments are described with particular reference to methods, whereas other exemplary embodiments may be described in the context of corresponding devices. In any case, a person skilled in the art may recognize and combine possible combinations of features of the method and possible combinations of features with corresponding systems from the above description and the following description, even if these belong to different claim categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Exemplary embodiments are described below with reference to the following drawings, in which:

[0044] Figure 1 A microscope is schematically shown;

[0045] Figure 2 Another microscope is schematically shown;

[0046] Figure 3 A flow chart showing a method for generating an overview image;

[0047] Figure 4 Shown according to Figure 3 Details of the flowchart;

[0048] Figure 5 Shown according to Figure 3 Details of the flowchart;

[0049] Figure 6 Shown according to Figure 3 Details of the flowchart;

[0050] Figure 7 Shown according to Figure 3 Details of the flowchart;

[0051] Figure 8 Shown according to Figure 3 Details of the flowchart;

[0052] Fig. 9 Shown according to Figure 3 Details of the flowchart;

[0053] Fig.10 Shown according to Figure 3 Details of the flowchart;

[0054] Fig.11 Shown according to Figure 3 Details of the flowchart;

[0055] Fig.12 A calibration plate is shown;

[0056] Fig.13 A portion of an original overview image of a calibration plate is shown;

[0057] Fig.14 A detailed image of the calibration plate is shown; and

[0058] Fig.15 The calibration structure in the calibration board is shown. DETAILED DESCRIPTION

[0059] Microscope 1000 is in Figure 1 . The microscope comprises a plurality of microscope objectives 1111, 1112 and 1113 arranged in an objective turret 1110. Further, the microscope 1000 comprises a stage 1120, which is arranged to receive a sample carrier or a calibration plate 1200. The stage 1120 is substantially aligned at right angles to the optical axis of the microscope objectives 1111, 1112, 1113 or the currently used microscope objective. With the aid of a translation unit 1221, the stage 1120 can be changed relative to the distance from the microscope objectives 1111, 1112, 1113 or the currently used objective, i.e. in the z direction. Typically, for this purpose, the objective turret 1110 moves relative to the stage 1120 fixed in the z direction. However, as a matter of principle, it is also conceivable to move the stage 1120 relative to the fixed objective turret. Furthermore, the object stage 1120 can be moved laterally in the x and y direction by means of a translation unit 1221 so that different parts of the sample carrier or the calibration plate 1200 can be brought into the object field of the currently used microscope objective.

[0060] Further, the microscope 1000 comprises a reflected light unit 1320 and a transmitted light unit 1310, each of which is formed by an arrangement of light-emitting diodes. For example, the light-emitting diodes can be arranged in the form of a two-dimensional array of regular planes. It is also conceivable to arrange the light-emitting diodes on a spherical surface. Further, a non-periodic, in particular irregular arrangement of the light-emitting diodes is also possible. With the help of the reflected light unit 1320 and the transmitted light unit 1310, a bright field contrast or a dark field contrast can be set respectively. In an exemplary embodiment, the individual diodes of the reflected light unit 1320 and the transmitted light unit 1310 can also be controlled separately.

[0061] Further, an overview camera 1400 is provided, with the help of which a larger area of ​​the sample carrier or the calibration plate 1200 can be imaged. In particular, the camera can have a non-telecentric lens. In the example shown, the overview camera 1400 is arranged on one side of the objective wheel 1110. Therefore, the overview camera 1400 is aligned with the sample carrier or the calibration plate 1200 in an inclined manner. Lenses with significant optical distortion are usually used in overview cameras, because the overview camera is usually arranged at a small distance from the stage 1120 and is intended to have a large field of view (FOV) at the same time. In addition, due to the tilt of the overview camera 1400 relative to the line of sight of the sample carrier or the calibration plate 1200, perspective distortions may occur.

[0062] The microscope 1000 also comprises a control unit 1500 allowing control of the former.

[0063] The calibration plate 1200 has a calibration structure 1210, by means of which optical distortions and perspective distortions can be corrected, as will be explained below. Furthermore, the calibration plate 1200 with the calibration structure 1210 allows the determination of the relationship between the pixel ratio of the original surround image recorded by the overview camera 1400 and the actual physical distance of the sample structure of the sample located on the sample carrier.

[0064] Figure 2 Another microscope 2000 is shown which is similar to the microscope 1000. In this case, elements 2111, 2112, 2110, 2120, 2121, 2310, 2320 and 2500 correspond to the elements 1111, 1112, 1110, 1120, 1121, 1310, 1320 and 1500 described above.

[0065] with Figure 1 The microscope 1000 shown in Figure 2 In the objective turret 2110 of the microscope 2000 shown in FIG, a mirror 2114 is arranged instead of a microscope objective. The overview camera 2400 of the microscope 2000 is arranged so that a larger area of ​​the sample carrier 2200 can be observed with the aid of the mirror 2114. In this case, the beam path from the sample carrier 2200 to the overview camera 2400 can correspond to a substantially orthogonal alignment of the overview camera 2400 relative to the sample carrier 2200 or the stage 2120.

[0066] and Figure 1 The difference is that it is not the calibration plate but the sample carrier 2200 that is arranged on the stage 2120. In the example shown, the sample carrier 2200 is a multi-well plate having a plurality of wells 2220.

[0067] Figure 3The method schematically described in the above may allow providing a user with a method for using Figure 1 or Figure 2 The microscope shown produces an overview image with the help of which one can orient oneself in the sample. In this case, the necessary method steps can be largely automated and the user can thus obtain an overview image that meets their requirements as easily and quickly as possible and can thus spend more time examining the actual sample structures of interest. In this case, the method is designed such that it works robustly with a very wide range of sample carriers and samples. In particular, the method described can allow the generation of an overview image even for very large sample carriers, such as multiwell plates with dimensions of 128×86 mm, for example.

[0068] Initially, a calibration is performed in step 3100, for example with the aid of a calibration plate; this can be used to reduce distortions and deformations of the raw overview image recorded by the overview camera, facilitate the assignment of the microscope objective relative to the object field of the overview image, and ultimately also facilitate the determination of the physical dimensions of the sample. Step 3100 may be optional. In particular, it is conceivable not to perform step 3100 before the creation of each overview image.

[0069] Further, an overview lighting optimized for the presentation of the overview image is determined in step 3200. The overview lighting can be reflected light lighting or dark field lighting. Here, the overview image lighting can be proposed to the user based on the identified sample carrier type, wherein the user must confirm the lighting, or the determined overview lighting can be automatically adopted. Step 3200 preferably also includes the identification of the sample carrier type.

[0070] If the sample carrier type is identified or an optimized overview image illumination cannot be proposed to the user, the method may be terminated (step 3300).

[0071] Once the sample carrier type has been determined, it can be run-in on the Focus 3400.

[0072] If an approximate focus position is found (step 3500), a decision is made in the next step 3600 as to whether the FOV (field of view) of the overview camera is sufficient to represent the entire sample in the overview image. On the one hand, this can be done manually; on the other hand, however, it is also conceivable that the control unit has recognized that the type of sample carrier used is larger than the FOV of the overview camera.

[0073] In case of sufficient FOV, identification of the sample carrier is performed in the next step, step 3611. Using optimized overview image illumination (e.g., transmitted light illumination with dark field contrast in the case of slides with coverslips, or transmitted light illumination in the case of multiwell plates) can improve the identification of the sample carrier and its structures.

[0074] Once the sample carrier has been identified in step 3612 , a check is performed in step 3613 as to whether the sample should be automatically identified in the overview image.

[0075] If this is the case, the sample will be identified automatically.

[0076] If the FOV of the overview camera established in step 3600 is not sufficient to image the entire sample, a plurality of raw overview images are generated in step 3620 by means of the stage of the microscope which is always slightly displaced to one side. The individual raw overview images can also be referred to as tiles. The individual tiles can then be put together to form a mosaic (stitching). In this case, the mosaic image can be formed based on the known range of motion of the translation unit, so that errors that could occur during the stitching of corresponding structures based on the individual raw overview images can be avoided.

[0077] In step 3621, detection of the sample holder is then performed in step 3621. The sample holder can be used to safely support the sample carrier on the stage of the microscope. Detecting the sample holder helps to avoid a collision between the microscope objective and the sample holder if the distance of the sample carrier to the microscope objective changes or the stage of the microscope shifts to one side.

[0078] If a previous detection of a sample carrier has been determined in step 3622 , a query can be made in a further step 3623 as to whether the sample should also be automatically identified in the overview image.

[0079] If this is the case, an automatic detection of the sample in the overview image is performed in step 3624 .

[0080] exist Figure 4 This is explained in more detail in Figure 3 Step 3100. Initially, in step 4100, the calibration plate is fixed to the stage of the microscope in a defined manner.

[0081] Subsequently, the distance between the overview camera and the objective turret and the microscope stage is adjusted so that the calibration structure of the calibration plate can be imaged in focus with the overview camera. This position can also be referred to as reference focus position 4201.

[0082] In step 4300, a calibration overview image 4301 is created by the overview camera using suitable calibration illumination, such as reflected light illumination. In step 4400, the stage of the microscope may be laterally displaced and in step 4500 a displaced calibration overview image 4501 may be obtained.

[0083] In a further step 4600 calibration data 4601 are obtained from the calibration overview image 4301 and the shifted calibration overview image 4501. This can be facilitated, for example, by triangulation. The calibration data can be used to correct objective lens distortion and / or perspective distortion of the overview camera due to a non-perpendicular alignment of the overview camera relative to the sample stage.

[0084] In step 4700 , a first detailed image 4701 of a calibration structure of a calibration plate may be recorded with the aid of a microscope objective.

[0085] Subsequently, the calibration plate may be laterally displaced in step 4800 in such a way that a second detailed image 4902 of at least one second calibration structure of the calibration plate may be recorded using the microscope objective in step 4900. Typically, further detailed images 4902 of further calibration structures of the calibration plate are recorded in step 4900. In total, for example four calibration structures may be recorded. Subsequently, calibration data 4A01 for positioning the detailed image in the overview image may be obtained in step 4A00.

[0086] exist Figure 5 This is explained in more detail in Figure 3 Step 3400. In a first step 5100, the focus of the overview camera is brought into a reference position. This can be implemented by means of the stage of the microscope being pulled closer to or further away from the overview camera. In step 5200, the stage of the microscope is placed in a center position. Subsequently, in step 5300, a preliminary raw overview image is recorded and, based on the calibration data obtained in step 3100, corrections are made for distortions and deformations, thereby obtaining a preliminary overview image 5301. Similar to step 5400, this preliminary overview image 5301 can be shown to a user of the microscope. This can allow the user to verify the correct functioning of the proposed method in a first step. In addition, in step 5500, the type of sample carrier 5501 located in the observation space of the microscope can be identified based on the preliminary overview image 5301. For this purpose, for example, an ML (machine learning) system can be used, which has been trained using a large number of sample carrier images. To identify or classify sample carrier type 5501, a convolutional neural network (CNN) can be used, for example, as described in more detail in the following articles, Krizhevsky, Sutskever, Hinton: “ImageNet Classification with Deep Convolutional Neural Networks” (NIPS 2012) and He, Zhang, Ren, Sun: “Deep residual learning for image recognition” (CVPR2016).

[0087] Based on the determined sample carrier type 5501, an overview image illumination 5601 intended for creating the overview image may be selected in step 5600. In this case, the overview image illumination 5601 may be selected automatically. Alternatively, it is conceivable that the overview image illumination is proposed to the user of the microscope based on the determined sample carrier type 5501, and that the user has to merely confirm the selection, although they may still select a different type of overview illumination.

[0088] For example, Figure 3 Step 3400 of the method shown may be as follows Figure 6 In a first step 6100 a check is performed as to whether the overview image illumination 5601 differs from the reflected light illumination used for recording the preliminary overview image 5301. If this is not the case, the preliminary overview image 5301 continues to be used as preliminary overview image 6111 (step 6110).

[0089] If the overview image illumination 5601 is different from the reflected light illumination, a new preliminary overview image 6111 is recorded using the overview image illumination 5601 in step 6120. The new preliminary overview image 6111 may in turn be presented to a user of the microscope (6130).

[0090] Segmentation (6200) is performed based on the preliminary overview image and a mask image 6201 is obtained. In particular, the mask image 6201 can visualize the structure of the sample carrier. For example, segmentation can be performed based on a fully convolutional network (FCN), for example, as described in the following articles, Chen, Zhu, Papandreou, Schroff, Adam: "Encoder-decoder with atrous separable convolution for semantic image segmentation" (ECCV2018), and Shelhamer, Long, Darrell: "Fully convolutional networks for semantic segmentation" (PAMI 2016). In particular, segmentation can be performed based on pixel-related classification.

[0091] Subsequently, in step 6300, the stage of the microscope is shifted to one side. Using the overview image illumination 5601, a further preliminary overview raw image is recorded in step 6400, and after correction of distortions and deformations, a shifted preliminary overview image 6401 is obtained. In step 6500, another segmentation may be performed based on the shifted preliminary overview image 6401, so that a shifted mask image 6501 may be obtained.

[0092] In step 6600 , an approximate focus position may be determined based on the mask image 6201 and the shifted mask image 6501 .

[0093] Determining the approximate focus position 6601 may take into account the fact that the distance D1 between the objective turret or overview camera and the calibration plate may be different from the distance D2 between the object turret or overview camera and the sample carrier. In particular, in the case of a sample carrier in the form of a multi-well plate, the focus position should be located at the bottom of the wells rather than the underside of the sample carrier.

[0094] Further details of the method for determining an approximate focus position can be gathered from document DE 10 2018 133 188 A1.

[0095] In step 6700, a corrected preliminary overview image 6701 may be obtained based on the preliminary overview image 6111 and the approximate focus position 6601. Subsequently, the corrected preliminary overview image 6701 may be presented again to the microscope user (step 6800).

[0096] exist Figure 7 This is explained in more detail in Figure 3 Method step 3611. In step 7100, a sample carrier mask image 7101 is generated based on the preliminary overview image 6701. In step 7200, the sample carrier mask image 7101 is used to identify individual areas of the sample carrier. In step 7300, the obtained sample carrier information 7201 can be displayed to the user of the microscope. For example, the individual wells of a multiwell plate can be displayed to the microscope user. Based on step 7400, a sample carrier template 7401 can be generated based on the sample carrier information 7201. For example, the sample carrier template 7401 can represent different wells of a multiwell plate in a simplified manner, and thus individual grinding on these wells is possible. The individual wells of a multiwell plate can be numbered in rows and / or columns.

[0097] Figure 8 In an exemplary manner, Figure 3Details of method step 3614 of . In a first step 8100, the quality of the recording of the sample area is checked based on the overview image 6701. Based on the information about the quality of the recording of the sample area 8101, a decision is made in step 8200 as to whether a segmentation 8300 of the sample area can be performed. Initially, when segmenting the sample area, a mask 8301 is generated based on the overview image 6701. In step 8400, the mask 8301 is used to identify the sample area 8401. In particular, the mask 8301 can be used to define an area that can contain a sample. For example, in the case of a slide, the sample that should be checked is usually arranged only in the area of ​​the cover glass, and in the case of a multi-well plate, only in the wells. In step 8500, the identified sample area 8401 is displayed to the user of the microscope. The latter can thus check with the aid of an automated method whether the correct area has been identified. Based on the sample area 8401, the sample area can be divided into smaller tile areas in step 8600, which can be analyzed separately with the aid of a microscope objective.

[0098] Fig. 9 Shows Figure 3 . Based on the overview image illumination 5601, a plurality of raw overview images are recorded in method step 9100 using a shift of the microscope stage, so that after correction of distortions and deformations, a plurality of tile overview images 9101 are obtained. In step 9200, the various tile overview images 9101 are combined to form an overview image 9201, wherein the combined overview tile image 9201 may also be referred to as a mosaic overview image. Then, in step 9300, the mosaic overview image 9201 is presented to a user of the microscope.

[0099] exist Fig.10 This is described in more detail in Figure 3 In a first step 10100, a plurality of tile mask overview images 10101 are generated based on the tile overview image 9201. In a step 10200, the tile mask overview images are combined to form a mosaic mask image 10201. This can be implemented by means of stitching, for example. In a step 10300, a sample area 10301 is identified based on the mosaic mask overview image 10201. In a step 10400, the sample area is presented to a user of the microscope. In a step 10500, the sample area 10301 is analyzed and a sample carrier template 10501 is created.

[0100] exist Fig.11 This is explained in more detail in Figure 3In a first step 11100, the quality of the tile overview image 9201 is determined. In a step 11200, it is defined whether the method continues based on the determined quality 11101. In a step 11300, a segmentation of the sample area is performed based on the tile overview image and a tile mask overview image 11301 is obtained. In a step 11400, the tile mask overview images 11301 are put together to form a mosaic overview image 11401.

[0101] In step 11500, the mosaic overview image 11401 is used to extract a sample region 11501. The extracted sample region 11501 is then displayed to a user of the microscope in step 11600. In step 11700, the sample region 11501 is divided into tile sample regions 11701, which can be examined in more detail on an individual basis using a microscope objective.

[0102] Fig.12 An exemplary embodiment of a calibration plate 12000 is shown, which can be used in any of the methods described above. The calibration plate 12000 has at least one through hole 12021, which can allow an immersion liquid to be applied to an immersion objective of a microscope. Further, the calibration plate 12000 has a checkerboard pattern 12010 comprising a plurality of calibration structures. The calibration plate 1200 can be translucent, in particular transparent, so that the checkerboard pattern 12010 or a part of the checkerboard pattern 12010 can be recorded by an overview camera and / or a microscope objective when using reflected light illumination and when using transmitted light illumination.

[0103] Fig.13 A portion 13001 of a raw overview image of a first calibration structure of a calibration plate 12000 as recorded by an overview camera of a microscope is shown. Fig.14 A detailed image 14001 of the same calibration structure as recorded using the microscope objective of a microscope is shown. The microscope objective utilized is a 5x microscope objective, which has a significantly smaller image field than the overview camera. Fig.13 and 14 It is shown that in both cases the same calibration structure of the calibration plate can be imaged well and the edges of the area of ​​the checkerboard pattern 12010 can be imaged in focus, so that the generation of calibration data for correcting distortion and / or perspective deformation can be performed in an automated manner.

[0104] As in Fig.15As indicated in FIG. 1 , the calibration plate 12000 may include, in addition to the checkerboard pattern 12010, further calibration structures 12011, 12012. For example, a calibration structure 12011 may be provided which may be used to optically determine the center of a stage of a microscope. Furthermore, the calibration structure 12012 may allow for zooming of the objective. The calibration structure 12012 may also be used to perform focus balancing between multiple microscope objectives and / or determine offsets between multiple microscope objectives.

[0105] In summary, the described method, the proposed microscope and the disclosed calibration plate facilitate the automatic, rapid creation of a navigable overview image. Thus, a user of the microscope can quickly navigate to a sample spot to be examined based on the overview image or starting from a known position of the sample on the sample carrier and can further analyze said spot using a microscope objective with a high magnification. Thus, the user can optimally use their available time and thus they can quickly obtain results.

[0106] Within the scope of the described method, different ML systems have been trained with annotated images of various sample carriers and samples and possibly with annotation data and can be used for identifying sample carrier types, identifying cover slips of slides, for segmenting multiwell plates, determining classification quality and sample area segmentation. In particular, edges and textures in the images can be identified in this process. Target parts can be digitally identified and different targets can be identified.

Claims

1. A method for generating an overview image of a sample, the sample being arranged in an observation space of a microscope by means of a sample carrier, in, illuminating the sample carrier with a first illumination; wherein a preliminary overview image is generated using a first illumination and an overview camera of the microscope; wherein overview image lighting is selected based on the preliminary overview image; wherein the sample carrier is illuminated by the overview image; and Therein, the overview image is generated using the overview image illumination and the overview camera.

2. The method according to claim 1, in, The overview image illumination comprises reflected light illumination and / or transmitted light illumination.

3. The method according to claim 1 or 2, in, processing the preliminary overview image in a trained machine learning based system (ML system); Wherein the overview image lighting is selected based on the output of the ML system.

4. The method according to claim 3, wherein the output of the trained ML system comprises a statement about a sample carrier type of the sample carrier, and in, The overview image illumination is determined based on the sample carrier type.

5. The method according to claim 1 or 2, in, Arranging a calibration plate in the observation space; wherein the calibration plate is illuminated using calibration lighting; wherein a calibration overview image is generated using the calibration illumination and the overview camera; wherein calibration data is obtained by means of the calibration overview image; and Therein, the preliminary overview image and / or the overview image are generated using the calibration data.

6. The method according to claim 5, in, The calibration plate includes a first calibration structure; wherein the calibration overview image comprises an image representation of the first calibration structure; wherein a first detailed image of the first calibration structure is generated using a microscope objective of the microscope; and Therein, based on the calibration overview image and the first detailed image, further calibration data of a sample detail image for locating the sample in the overview image are obtained.

7. The method according to claim 6, in, The calibration plate comprises at least one further calibration structure; wherein, after generating the first detailed image, the position of the calibration plate relative to the microscope objective is changed by at least one further predefined translation vector; wherein at least one second detailed image of the further plurality of calibration structures / a calibration structure is generated using a microscope objective of the microscope; and Therein, the calibration data are obtained by additionally using the further predefined translation vector and evaluating the position of the calibration structure in the at least one second detailed image.

8. The method according to claim 6, in, The sample detail image is generated using the microscope objective or a further microscope objective.

9. The method according to claim 1 or 2, in, In order to generate the preliminary overview image, at least two preliminary raw overview images are recorded by the overview camera, wherein the first illumination is different in the preliminary raw overview images.

10. The method according to claim 1 or 2, in, To generate the overview image, at least two raw overview images are recorded, wherein the overview image illumination of the raw overview images differs.

11. The method according to claim 1 or 2, in, To generate the preliminary overview image, the overview camera records at least two preliminary raw overview images, wherein the position of the sample carrier is changed by a known translation vector between the raw overview images; wherein mutually corresponding structures are identified in the preliminary original overview image; and Therein, an approximate focus position is determined from mutually corresponding structures in the preliminary raw overview image, the known translation vector and calibration data.

12. The method according to claim 1 or 2, in, recording at least two raw overview images by the overview camera, wherein the position of the sample carrier is changed by a displacement vector between the raw overview images; as well as The overview image is based on the original overview image and is generated as a mosaic overview image based on the displacement vector.

13. A microscope, Includes overview camera; including at least one microscope objective; including a transmitted light unit; including a reflective light unit; including a control unit; in, The control unit is arranged to perform a method according to any of the preceding claims.

14. A computer program product comprising instructions which, when executed by a control unit of a microscope, cause the latter to perform the method according to any one of claims 1 to 12.

15. A calibration plate for calibrating a microscope as claimed in claim 13, comprising an overview camera and a microscope objective, in, The calibration plate comprises at least one first calibration structure which can be imaged both by the overview camera and by the microscope objective when using calibration illumination.

16. The calibration plate according to claim 15, in, The calibration plate includes at least a second calibration structure at a predetermined distance from the first calibration structure.

17. The calibration plate according to claim 15 or 16, in, The calibration plate has a checkerboard pattern; and The first calibration structure corresponds to a corner of the area of ​​the checkerboard pattern.

18. The calibration plate according to claim 15 or 16, in, The calibration plate is arranged to diffusely reflect light.

19. The calibration plate according to claim 15 or 16, in, The calibration plate is translucent.

20. The calibration plate according to claim 15 or 16, in, The calibration plate has a height deviation of less than 150 μm.

21. The calibration plate according to claim 15 or 16, in, The first calibration structure is asymmetric.

22. The calibration plate according to claim 15 or 16, in, The calibration plate has at least one through hole.

23. The calibration plate according to claim 20, in, The calibration plate has a height deviation of less than 100 μm.

Images