Method and device for monitoring immersion liquid in a microscope

Abstract

The invention provides a method for monitoring an immersion liquid (6) in a microscope (1) having an objective which images a sample (4) located on a sample carrier (2). In step a) a camera (12) is positioned which has an image field (10) which is aligned such that it detects the sample carrier (2) and a space connected to the sample carrier (2) in the direction to the objective between the sample carrier (2) and the objective, said space serving to accommodate the immersion liquid (6). In step b) the immersion liquid (6) is applied into the space between the sample carrier (2) and the objective. In step c) an image (24) is taken with the immersion liquid (6) in the space between the sample carrier (2) and the objective, and in step d) the position, area and / or contour (34) of the immersion liquid (6) on the sample carrier (2) is determined from the image (24) taken in step c).

Description

Technical Field

[0001] This invention relates to a method for monitoring immersion liquid in a microscope equipped with a microscope objective lens, which images a sample located on a sample carrier. Here, an image of the immersion liquid in the space between the sample carrier and the microscope objective lens is captured. This invention also relates to an apparatus for monitoring immersion liquid in a microscope equipped with a microscope objective lens, which images a sample located on a sample carrier. Here, a camera captures an image of the immersion liquid in the space between the sample carrier and the microscope objective lens. Background Technology

[0002] US6980293B1 discloses an apparatus for automatically introducing an immersion medium into the space between a microscope objective and a sample. The apparatus involves capturing magnified images of the microscope objective and the sample, and automatically introducing the immersion medium into the space between the objective and the sample using a supply device.

[0003] US2007 / 0047093A1 describes a device for automatically applying immersion solution and for automatically cleaning microscope objectives.

[0004] Therefore, a method for automatically monitoring the immersion solution in a microscope should be provided. Summary of the Invention

[0005] The invention is defined in independent claims 1 and 15. Advantageous improvements are given in the dependent claims. Preferred embodiments are applicable to the method and apparatus in the same manner.

[0006] A method is provided for monitoring an infusion in a microscope having a microscope objective that images a sample located on a sample carrier. In step a), a camera having an image field is aligned. Alignment is performed such that the camera detects the sample carrier and the space between the sample carrier and the microscope objective, which is connected to the sample carrier in the direction toward the objective, the space being used to receive the infusion.

[0007] Also provided is an apparatus for monitoring an infusion in a microscope having a microscope objective that images a sample located on a sample carrier. The apparatus has a camera having an image field. The camera is positioned such that the image field detects the sample carrier and a space between the sample carrier and the microscope objective, connected to the sample carrier in a direction toward the microscope objective, the space being used to receive the infusion.

[0008] The specific positioning of the camera is irrelevant, as long as the sample carrier and the immersion liquid, or the space in which they should be located, are contained within the image field and, in particular, clearly visible. The camera can be, for example, mounted on the transmission arm of the microscope or on the microscope stage; other positions for the camera are also possible.

[0009] In step b) of the method, the immersion solution is applied into the space between the sample carrier and the microscope objective. In the apparatus, the immersion solution is applied using an application device.

[0010] In step c) of the method, an image is captured using the immersion liquid in the space between the sample carrier and the microscope objective. In the apparatus, the described image capture is performed using the camera.

[0011] In step d) of the method, the position, area and / or contour of the immersion liquid on the sample carrier are determined from the image taken in step c).

[0012] Since the camera can be placed arbitrarily in principle, as long as the sample carrier and the immersion liquid, or the space set up for them, are clearly visible in the image field, the sample plane is usually distorted in the image taken by the camera. However, the image plays an important role in locating and evaluating the immersion liquid, so it is advantageous to calibrate the camera after step a) of the method, for example, by positioning a pre-known calibration pattern in the plane of the sample carrier and in the image field of the camera and taking a calibration image. However, a reference object in the form of a microscope component, such as an embedded frame or the sample carrier itself, can also be used for calibration.

[0013] Another possible calibration approach is to locate the intersection of the camera objective's optical axis and the sample carrier plane in the image. With the camera centered above or below the sample carrier, this intersection is located at the center of the camera objective. The position of the camera objective in the image is usually known from the camera structure, but can alternatively be determined manually or automatically using a detection algorithm.

[0014] In particular, distortion caused by camera positioning can be identified and eliminated by transformation, and especially preferably, the sample plane in the transformed image is displayed as if the camera had accurately photographed the sample plane from above. To this end, image distortion caused by camera position in the space between the sample carrier and the microscope objective is determined in another step based on the imaged calibration pattern or a photographed (known) reference object.

[0015] To determine the necessary transformations, a pre-known calibration pattern is placed once in the plane of the sample carrier and photographed with a camera. The calibration pattern is designed such that perspective distortion can be identified in the calibration image. For example, the calibration pattern is designed as a grid or checkerboard pattern. Alternatively, as already described above, a reference object is photographed with a camera and used to determine the necessary transformations.

[0016] The necessary transformation, known as homography, can then be determined from the difference between the calibration pattern or reference object and its reproduction in the calibration image, where evaluation is sufficient. Homography is a mapping that maps the camera plane onto the sample carrier plane. Homography allows for the corresponding correction of the image while keeping the camera structure unchanged. Furthermore, distortions caused by camera optics (e.g., barrel curvature) can be calculated from the image using the calibration pattern or reference object and the calibration image. This can also be optionally incorporated into the transformation. Thus, calibration enables the image captured by the camera to be geometrically and perspectiveally corrected through transformation so that all objects in the sample carrier plane, and therefore all objects in the sample carrier plane, and thus the immersion attached to the sample carrier, are displayed largely (within the computationally and resolution-dependent range of transformation accuracy) without distortion.

[0017] The apparatus for monitoring the immersion solution includes a processing unit. A calibration image can be provided to this processing unit, which, for example, displays the space between the sample carrier and the microscope objective in a pre-known calibration pattern in the plane of the sample carrier and in the image field of the camera. The processing unit can also be configured such that it determines the image distortion in the space between the sample carrier and the microscope objective based on the calibration image and thus on the imaged calibration pattern. Similarly, a known reference object can be used as the basis for calibration.

[0018] Following step a) of the method, alternatively or additionally, a background image may also be captured in the space between the sample carrier and the microscope objective. This is preferably performed after any calibration that may be necessary. In the case of the apparatus, the background image is captured using a camera. A background image is an image that describes the appearance of the microscope objective in an image without interference. The background image can be captured, for example, without any immersion liquid in the space between the sample carrier and the microscope objective. Background images can also be generated by capturing many typical images under typical environmental variations, such as different lighting conditions. For a background image captured using immersion liquid, the appearance of the microscope objective without immersion liquid can be estimated from the background image itself.

[0019] Preferably, in step d), the location, area, and / or contour of the immersion on the sample carrier are determined by searching for structures in the image, wherein the structures are located within a certain distance from the center of the microscope objective, cannot be assigned to the microscope objective, and / or have an approximately annular shape.

[0020] In the apparatus, the processing device is configured such that it determines the position, area, and / or contour of the immersion liquid on the sample carrier, wherein the processing device uses an image captured by the camera of the space between the sample carrier and the microscope objective in the presence of the immersion liquid in the space. The processing device may also optionally use a background image of the space between the sample carrier and the microscope objective captured by the camera, an image of the space between the sample carrier and the microscope objective in the presence of the immersion liquid captured by the camera, and / or determined image distortion. The processing device is preferably configured such that it determines the position, area, and / or contour of the immersion liquid on the sample carrier by searching for structures in the image, structures located at a distance from the center of the microscope objective, not assignable to the microscope objective, and / or having an approximately annular shape.

[0021] The position of the microscope objective, and therefore the position of its center, can be determined through optional prior calibration, automatic estimation from one or more overview images, or manual input. Preferably, a maximum search distance is defined. Here, it refers to the radial distance from the center of the microscope objective. This maximum search distance is either automatically determined from the microscopy application based on the selected microscope objective, the size of the sample carrier, or the type of immersion solution, or is determined manually by the user.

[0022] The microscope objective may also generate structures in the image that might resemble the immersion. However, when determining the location, area, and / or outline of the immersion in the image, the structures in the image to be assigned to the microscope objective must not be used. To simplify the distinction between the structures in the image to be assigned to the microscope objective and the structures of the immersion, a previously generated background image can be used, for example. Here, the image taken in step c) is combined with the background image such that, as far as possible, only the structures of the immersion are preserved in the image. In the simplest case, the background image consists of a single reference image in which the immersion is not visible. This combination is achieved by subtracting the background image from the image generated in step c). However, combinations with multiple background images are also possible, which may include multiple environmental variations, such as varying lighting.

[0023] The structure searched in the image preferably has an approximately ring-shaped structure. Deformation of the immersion droplet is permissible here; however, ideally, it should be a circular droplet on the sample carrier, visible as a ring in the image. The location of the approximately ring-shaped structure can be achieved using various methods. One possible approach to localization involves three steps:

[0024] In the first step, the pixel values ​​of the image are converted to grayscale values. Here, dark gray values ​​are assigned to the approximate ring structure of the immersion, rather than light gray values. This grayscale-based scaling can be achieved, for example, by using a previously captured background image.

[0025] In the second step, the image is converted from Cartesian coordinates to polar coordinates. Here, the origin is chosen to be at the center of the microscope objective. The exact location of the center of the microscope objective can be determined, for example, from previous calibration. After the image is converted to polar coordinates, the outline of the immersion around the microscope objective, determined in the first step, appears as a straight line.

[0026] Because the immersion liquid is affected by external factors, such as moving the sample stage, it may not exist as an ideal droplet, and therefore as an approximately ring-shaped structure rather than a precise ring structure. Consequently, the lines in the image converted to polar coordinates are often slightly distorted. This can be taken into account in the implementation and compensated for by distortion transformation.

[0027] In the third step, a path is finally determined across all values ​​of the angular axis of the image converted to polar coordinates. This path allows the sum of all pixel values ​​along it to be minimized, and the path is then converted back to Cartesian coordinates after being determined. This path then corresponds to the contour of the immersion in the image captured in step c). Preferably, to determine the contour, it is assumed that the droplets of immersion in the image have a closed, continuous shape. For this purpose, care can be taken when searching for a path that the variation in the radial coordinates of continuous features along the angular axis does not exceed a predetermined threshold. This involves an optimization problem, which is solved by an algorithm for calculating the shortest path. For example, Dijkstra's algorithm, A* algorithm, or Floyd-Warshall algorithm can be considered.

[0028] Another possible approach to locating the immersion fluid is based on machine learning. For this purpose, a neural network, particularly a deep neural network based on so-called deep learning, and more preferably, at least one convolutional neural network (CNN), is set up as a machine learning model and trained to recognize ring-shaped or near-ring-shaped structures displaying the applied immersion fluid during imaging. The neural network can be trained through supervised learning, unsupervised learning, partially supervised learning, or reinforcement learning.

[0029] In particular, localization can be performed using segmentation, where the region containing the immersion is marked in the captured or image by means of classification or semantic segmentation, where regions with and without immersion are distinguished, and / or regions with and without immersion are distinguished by means of detection.

[0030] The main advantage of using machine learning or machine learning models lies in their robustness, as they can typically compensate for minor changes or disturbances in an overview image without leading to errors. Furthermore, new elements of the sample carrier environment or a complete redesign of the sample carrier environment can be easily added with a new round of training. In contrast, the effort required in classical image analysis to compensate for such disturbances and / or changes is extremely high, as these changes can affect the detection of known features and the surrounding environment.

[0031] In a preferred embodiment, in step e) of the method, the outline and / or area of ​​the immersion on the sample carrier is evaluated, and / or the volume of the immersion is estimated, and / or the remaining duration of the immersion on the sample carrier due to evaporation is determined. Therefore, in the apparatus, after determining the outline and / or area of ​​the immersion in the image, additional steps are performed to evaluate the shape of the immersion droplets on the sample carrier, estimate the droplet volume, or determine the remaining duration due to evaporation.

[0032] Ideally, the immersion droplet completely wets the front lens of the microscope objective and has a circular shape on the sample carrier, which appears as a ring-shaped outline in the image. Due to external influences, such as movement of the sample stage or when the volume of the immersion is too small / too large, it may occur that the front lens is not completely covered, and / or the shape deviates significantly from a circle. Therefore, preferably, the quality of the immersion droplet can be evaluated by the degree to which the front lens of the microscope objective is covered with the immersion. For this purpose, for example, the intersection of the optical axis of the microscope objective and the plane of the sample carrier, known from calibration, is used, i.e., the center of the microscope objective in a top view of the sample. The dimensions of the front lens are taken, for example, from an objective database. Furthermore, a surface area, dependent on the numerical aperture and distance from the sample carrier, is defined around the front lens, and this surface area must be covered with the immersion to ensure optimal image quality. Therefore, it is ensured that the front lens and the surrounding surface area are always covered with the immersion. Thus, appropriate countermeasures can be initiated promptly, i.e., before image quality is compromised, such as warnings to the user, automatic immersion, etc.

[0033] Furthermore, by evaluating suitable features, such as the eccentricity of the ellipse that best approximates the contour, the quality of the droplet can be optionally determined based on its similarity to a circle. For example, the eccentricity of the ellipse in the contour-defined region of the image can be calculated and compared with manually defined data or training data from a machine learning model. Alternatively, the contour of the droplet can be classified from “fully circular” to “excessively deformed” directly from the image captured in step c), without using an evaluation of the droplet’s contour. This can also be based on the area of ​​the droplet. In cases of excessive deformation, however, when the volume of the droplet or the remaining duration of the droplet on the sample carrier is too short, countermeasures can be implemented depending on the microscopy application, such as moving the sample more slowly, tactile feedback, warnings to the user, etc.

[0034] The contact area of ​​the immersion liquid on the sample carrier can also be easily determined based on its contour and / or area. To further estimate the volume of the immersion liquid, calibration measurements are provided for each microscope objective. Here, with the distance between the microscope objective and the sample carrier known, multiple drops of immersion liquid with different known volumes are applied, and the contact area between the immersion liquid and the sample carrier is then automatically determined from the image. This generates a 1:1 mapping between the volume of the immersion drop and its contact area for each microscope objective. Therefore, with the distance between the sample carrier and the microscope objective known, the droplet volume of the immersion liquid is estimated from the contact area. Optionally, it is also estimated how much more volume needs to be added to achieve the desired target volume. Furthermore, by determining the volume of the immersion liquid and understanding the geometry of the microscope objective, it is possible to estimate whether the immersion amount is sufficient for a predetermined shot to ensure the desired image quality at each location. Conversely, with the volume of the immersion liquid known, the distance between the sample carrier and the microscope objective can also be determined from the contact area, which can be used, for example, for collision protection.

[0035] Preferably, a light source is used in the method and in the apparatus for illuminating the image field, the light source being mounted at the camera or present at the microscope. Preferably, the light source is controllable in terms of intensity and wavelength range. Therefore, multiple images with different illuminations can be captured in step c), and the contour, area, and / or location of the immersion liquid can be analyzed from the overall images. Thus, the assessment of immersion can be further optimized.

[0036] Additionally, it is preferable that the camera captures images in the infrared range. To keep dependence on ambient light conditions as low as possible, the camera and light source preferably operate in the infrared range of the electromagnetic spectrum. This allows for parallel monitoring of the immersion solution with a fluorescence microscope. If the camera operates in the visible range of the spectrum, or if there is overlap between the spectral ranges of the camera illumination and the microscope due to the respective microscope application, then image capture is preferably performed temporally separately from the capture of the fluorescence image, or interfering areas in the image are masked.

[0037] It is also advantageous to arrange a polarizing filter on the camera so that images can be captured using the polarizing filter. The device is configured such that the polarizing filter is to be arranged or has been arranged at the camera to capture images using the polarizing filter. Using a polarizing filter can filter out reflections from metal surfaces, such as the metal surface of an objective lens, or reflections from droplets, or can be used to detect droplets, especially when combining images captured using a polarizing filter with images captured without a polarizing filter.

[0038] As described above, if the images of multiple different illuminations are analyzed as a whole in step c), this may also include images from the infrared range and / or images taken using polarizing filters.

[0039] Preferably, in step d) of the method, impurities in the immersion solution are identified and optionally filtered out from the image captured in step c). In the apparatus, the processing device is configured such that it identifies impurities in the immersion solution and optionally filters them out from the image as well.

[0040] The outline, area, and / or location of the immersion liquid are obtained from step d). Therefore, impurities, such as bubbles or dust, in the immersion liquid can be searched in the image after step d). Here, the region of interest in the image is located within the outline and / or surface and has a strong visual deviation from the microscope objective. In this step, a background image (or background model) can also be used again to filter out structures belonging to the microscope objective. Standard algorithms in the fields of image processing and / or machine learning, such as detection, can be used to discover most of the small circular structures within the droplets of the immersion liquid.

[0041] The method and apparatus described above for monitoring the immersion solution enable the automation and control of microscopic procedures such as cleaning microscope objectives and applying immersion solution. These procedures are currently performed manually by the user under constant monitoring and maintenance, according to existing technology. Optional automated application of the immersion solution is implemented by checking the contour, area, and / or position, and is automatically monitored by the method described above. This checks whether the rated volume and rated position of the immersion solution are maintained within predetermined limits. Furthermore, the amount and position of the immersion solution can be automatically monitored during sample stage movement to ensure continuous wetting of the front lens of the microscope objective with the immersion solution and to prevent immersion solution residue from remaining on the sample carrier due to excessive droplet deformation. Attached Figure Description

[0042] The invention will now be described in more detail by way of example with reference to the accompanying drawings. In the drawings:

[0043] Figure 1 A device for monitoring the immersion solution in a microscope is shown;

[0044] Figure 2 A flowchart is shown for a method of monitoring the immersion solution in a microscope;

[0045] Figure 3 A flowchart of the calibration process is shown;

[0046] Figures 4a and 4b show the dependence of the safety distance on the numerical aperture of the objective lens;

[0047] Figures 5a and 5b show the outline of the immersion liquid on the objective lens; and

[0048] Figure 6 The evaporation of the immersion solution on the sample carrier over time is shown. Detailed Implementation

[0049] exist Figure 1 The diagram shows an apparatus for monitoring the immersion liquid 6 in a microscope 1. A microscope with a microscope objective (not shown separately) detects the sample 4 located on a sample carrier 2 along the optical axis OA. The immersion liquid 6 is applied to the sample carrier 2 and across the sample 4. An application device 8 is provided to apply the immersion liquid 6 into the gap between the microscope objective and the sample 4. A camera 12 spanning the image field 10 is pointed at the sample 4 and the immersion liquid 6. Both the application device 8 and the camera 12 are connected to a processing device 16 via wires 14. The processing device is in turn connected to a display device 18 via wires 14.

[0050] The user has virtually complete freedom in positioning the camera 12. Therefore, the camera 12 can, in principle, be placed arbitrarily, as long as the sample carrier 2 with the sample 4 on it and the immersion liquid 6 are clearly visible in the image field 10. Figure 1 The fasteners for camera 12 are not shown; camera 12 can be mounted, for example, at the transmission arm of microscope 1 or at the microscope stage. Because camera 12 can be freely positioned, the plane containing the sample typically exhibits perspective distortion in the image captured by camera 12. The perspective-distorted image is transmitted to processing device 16 via wire 14 or radio after being captured by camera 12.

[0051] In order to detect and evaluate the immersion liquid 6, the perspective-distorted image transmitted from camera 12 to processing device 16 via wire 14 must be transformed and corrected so that the sample plane is displayed in the transformed image as if camera 12 had accurately photographed the sample plane from above. Therefore, according to... Figure 1The apparatus remains unchanged, preferably only sample 4 is removed and a calibration pattern with a known structure is introduced into the plane of sample carrier 2. Here, camera 12 remains in the same position. If the calibration pattern is positioned parallel to the plane of sample carrier 2 (or ideally replaces sample carrier 2), a calibration image is captured by camera 12 and transmitted to processing device 16. Processing device 16 then determines the necessary transformation, so-called homography H, based on the perspective-distorted image and the calibration image, and thereby identifies image distortion in the space between sample carrier 2 and the objective lens. To this end, processing device 16 compares the structure of the calibration pattern with its image according to the calibration image.

[0052] If the distortion is known, camera 12 captures another image, namely background image 22. In a preferred embodiment, background image 22 represents the space between sample carrier 2 and microscope objective without immersion medium 6. Therefore, in background image 22, the appearance of microscope objective is displayed in the image without any interfering effects. Background image 22 can also preferably be generated by capturing many typical images 20 under typical environmental changes, such as different lighting conditions. In a variant, background image 22 is captured using immersion medium 6 introduced into the space between sample carrier 2 and microscope objective. After background image 22 is captured by camera 12, it is transmitted to processing device 16 via wire 14. In the case of capturing multiple images 20 with environmental changes, processing device 16 synthesizes background image 22 from individual images 20. In the case of capturing background image 22 using immersion medium 6 in the space between sample carrier 2 and microscope objective, processing device 16 itself estimates from background image 22 how microscope objective would appear without immersion medium 6.

[0053] The immersion liquid 6 is introduced into the space between the sample carrier 2 and the objective lens using the application device 8. The application device 8 is controlled by the processing device 16 for this purpose.

[0054] If the immersion liquid 6 is located in the space between the sample carrier 2 and the objective lens, the camera 12 uses the immersion liquid 6 in the space between the sample carrier 2 and the objective lens to capture an image and transmit it to the processing device 16. The position of the camera 12 is the same as the position when capturing the background image 22 and the calibration image.

[0055] The processing device 16 corrects the image due to image distortion and then determines the position and contour 34 of the immersion liquid 6 on the sample carrier 2. Here, the processing device 16 searches for structures in the image that are located at a certain distance from the center of the microscope objective, cannot be assigned to the microscope objective, and have an approximately annular shape.

[0056] The processing device 16 is also connected to the imaging device 18 via a wire 14. The imaging device 18 is, for example, a monitor on which the processing device 16 displays the results.

[0057] Preferably, the processing device 16 assesses the profile 34 of the immersion liquid 6 on the sample carrier 2 and / or estimates the volume of the immersion liquid 6 and / or determines the remaining duration of the immersion liquid 6 on the sample carrier 2 due to evaporation.

[0058] In the variant example, according to Figure 1 The device includes a light source that generates an illumination field that illuminates the image field 10. The light source can be, for example, directly mounted to the camera 12; alternatively, a light source already present in the microscope 1 can be used to illuminate the image field 10. When actively illuminating the image field 10 using the light source, both the camera 12 and the light source preferably operate in the infrared range of the electromagnetic spectrum. This allows for parallel monitoring of the immersion solution 6 with a fluorescence microscope.

[0059] In a variant, the processing device 16 identifies impurities in the immersion liquid 6 and filters these impurities out of the image. Here, the processing device 16 searches for regions in the image that are located within the outline 34 of the immersion liquid 6 and have a strong visual deviation from the image. Impurities may be, for example, air bubbles or dust. The processing device 16 is able to combine the identified impurities with the background image 22 and thereby remove these impurities from the image. This can be done using standard algorithms in the fields of image processing and / or machine learning.

[0060] Figure 2 A flowchart is shown for a method of monitoring the immersion liquid 6 in a microscope 1 with a microscope objective that images the sample 4 located on the sample carrier 2.

[0061] In step S1, the camera 12 is aligned such that the image field 10 detects the sample carrier 2 and the space between the sample carrier 2 and the microscope objective lens, which is connected to the sample carrier 2 in the direction of the microscope objective lens and is used to receive the immersion liquid 6.

[0062] In step S2, the camera is calibrated. For this purpose, the calibration pattern is parallel to the plane of the sample carrier 2, ideally replacing the sample carrier 2, and positioned in the image field 10, and the calibration image is captured by the camera 12.

[0063] Subsequently, in step S3, image distortion caused by the position of camera 12 in the space between sample carrier 2 and microscope objective is determined. The homography H that maps the plane of camera 12 onto the plane of sample carrier 2 is also determined or estimated. Thus, images distorted due to the placement of camera 12, captured in a subsequent step of the method, can be corrected based on a calibration image.

[0064] In step S4, a background image 22 of the space between the sample carrier 2 and the objective lens is captured. The background image 22 can be generated, for example, by capturing a single image of the space between the microscope objective lens and the sample carrier 2 without interference, such as the immersion liquid 6. Alternatively, the background image 22 can be generated by capturing multiple images 20 under typical environmental variations, such as different lighting conditions. Then, as shown in FIG. 5, the background image 22 is synthesized by the processing device 16 from the multiple images 20. The appearance of the microscope objective lens without the immersion liquid 6 can also be estimated by the processing device 16 from the background image 22 under the condition that the immersion liquid 6 is present in the space between the objective lens and the sample carrier 2.

[0065] In step S5, the immersion liquid 6 is introduced into the space between the microscope objective and the sample carrier 2 using the application device 8, and then in step S6, the immersion liquid 6 in the space between the microscope objective and the sample carrier 2 is used to capture an image.

[0066] Subsequently, in step S7, the position and contour 34 of the immersion liquid 6 on the sample carrier 2 are determined from the image captured in step S6 and the image distortion determined in step S3. Preferably, the contour 34 of the immersion liquid on the sample carrier is determined by searching for structures in the image, which are located at a certain distance from the center of the microscope objective, cannot be assigned to the microscope objective, and have an approximately annular shape.

[0067] In a variant, after step S7, the profile 34 of the immersion liquid 6 on the sample carrier 2 is evaluated, and / or the volume of the immersion liquid 6 is estimated, and / or the remaining duration of the immersion liquid 6 on the sample carrier 2 due to evaporation is determined.

[0068] To calibrate and determine image distortion, plane E1 of the camera image, plane E2 of the sample carrier 2, and plane E3 of the calibration pattern are preferably used. As described above, image distortion is determined from these three planes based on the calibration process in steps S2 and S3. Figure 3 The image shows an overview of how image distortion is determined.

[0069] Figure 3 The calibration process K related to the determination of image distortion in processing device 16 is illustrated. The image coordinates C1 of plane E1 are related to the calibration coordinates C3 and the sample carrier coordinates C2 via homography H. A translation T occurs between the sample carrier coordinates C2 and the calibration coordinates C3.

[0070] The image coordinates C1 in plane E1 of the camera image thus represent the sample carrier plane E2 as if the camera 12 had accurately captured the image from above. To calculate distortion, the sample carrier coordinates C2 and calibration coordinates C3 are first calculated and scaled in translation T. Coordinates C2 and C3 are then correlated with the image coordinates C1 via homography H. Plane E1 is mapped onto plane E2 using homography H. This allows for image correction. Of course, the camera's position and orientation should not be changed. Furthermore, distortions caused by the camera's optics (e.g., barrel curvature) can also be calculated from the image using calibration K.

[0071] In one variant, the intersection of the optical axis OA of the camera objective and the sample carrier plane E2 is located in the image during calibration K. When the camera 12 is centered above or below the sample carrier 2, this intersection is located at the center of the camera objective. The position of the camera objective in the image is usually known from the camera structure, but can alternatively be determined by a detection algorithm, or manually if necessary.

[0072] Figure 4A A front lens 36 with a diameter D1 is shown. A coverslip 40 is also shown. Immersion liquid 6 is located in the form of a droplet between the coverslip 40 and the front lens 36. This defines the safety area 42 and the angle 44 of the microscope objective lens 38. Figure 4B The front lens 36 of a microscope objective with a relatively high aperture is shown. Its diameter is D2. Immersion liquid 6 is located in the form of a droplet between the coverslip 40 and the front lens 36. This defines a safe area 48 and an angle 50 of the microscope objective 46.

[0073] The quality of the droplets in the immersion liquid 6 can be evaluated by the degree to which the front lens 36 of the microscope objectives 38 and 46 is covered by the immersion liquid 6. Microscope objectives 38 and 46 with different apertures differ in the size of their front lenses 36. The size of the front lens 36 can be taken, for example, from an objective database. Numerical aperture describes the ability of microscope objectives 38 and 46 to focus light. Therefore, a higher aperture is accompanied by a higher angle of focus 44 and 50. Thus, the angle of focus 44 of objective 38 is smaller than the angle of focus 50 of objective 46.

[0074] Therefore, as Figure 4A and Figure 4B As shown, a safety zone 42, 48 is defined around the front lens 36, depending on the distance of the sample carrier 2 from the front lens 36. This safety zone 42, 48 must be covered with the immersion liquid 6 to ensure optimal image quality. This ensures that the front lens 36 and the safety zones 42, 48 are always fully covered with the immersion liquid 6, and that appropriate countermeasures can be activated promptly, such as issuing a warning to the user or automatically introducing the immersion liquid 6 via the application device 8. This ensures that the image quality remains constant throughout the entire duration of the experiment.

[0075] Figure 5A and Figure 5B The resulting images 32 show the outline 34 with the immersion liquid 6.

[0076] The ideal droplet of immersion liquid 6 completely wets the front lens 36 of the microscope objectives 38 and 46 and has a circular shape on the sample carrier 2, which is shown as a ring outline 34 in the image. Figure 5A Due to external influences, such as movement of the sample stage or when the volume of the immersion liquid 6 is too small / too large, it may occur that the front lens 36 does not provide complete coverage and / or its shape deviates significantly from a circle. Figure 5B ).

[0077] In this implementation, the quality of the droplet in the immersion liquid 6 is determined based on its similarity to a circle by calculating suitable features of the region defined by the contour 34 in the resulting image 32, such as the eccentricity of an ellipse that best approximates the contour 34, and comparing this with manually defined data or training data. The contour 34 of the immersion liquid 6 is classified from "fully circular" to "excessively deformed." In cases of excessive deformation, countermeasures are implemented according to the microscope application, such as moving the stage more slowly, providing haptic feedback, or issuing warnings to the user.

[0078] Furthermore, the contact area of ​​the immersion liquid 6 on the sample carrier can be easily determined based on the contour. To also estimate the volume of the immersion liquid, calibration measurements are provided for each microscope objective 38, 46. Here, with the distance between the microscope objectives 38, 46 and the sample carrier 2 known, multiple drops of immersion liquid 6 with different known volumes are applied, and the contact area between the immersion liquid 6 and the sample carrier 2 is automatically determined from the resulting image 32. Thus, a 1:1 mapping between the volume of the immersion drop and its contact area is generated for each microscope objective 38, 46. Therefore, with the distance between the sample carrier 2 and the microscope objectives 38, 46 known, the droplet volume of the immersion liquid 6 is estimated from the contact area. Optionally, how much volume still needs to be applied to achieve the desired target volume is also estimated. Furthermore, by determining the volume of the immersion liquid 6 and understanding the geometry of the microscope objectives 38, 46, it is possible to estimate whether the immersion amount is sufficient for the predetermined imaging to ensure the desired image quality at each location. Conversely, given that the volume of the immersion liquid 6 is known, the distance between the sample carrier 2 and the microscope objectives 38 and 46 can also be determined from the contact area, which can be used, for example, for collision protection.

[0079] exist Figure 6 The image shows the monitoring of droplets of immersion liquid 6 in the resulting images 32a, 32b, and 32c within a specific time period. Immersion liquid 6 is water in this case. Evaporation E is shown. The expected area of ​​the immersion liquid is shown on the y-axis, and the trend of its change over time is shown on the x-axis.

[0080] As the immersion liquid 6 evaporates on the sample carrier 2, the contact area between the sample carrier 2 and the immersion liquid continuously decreases. Regression can be used to predict how the droplets will change over time. The amount of immersion liquid 6 on the sample carrier 2 over time is monitored and predicted by the processing device 16 based on images from the camera 12. This ensures that a timely warning is issued to the user in the event of a sharp decrease in the amount of immersion liquid 6 on the sample carrier 2.

[0081] Explanation of reference numerals in the attached figures:

[0082] 1. Microscope

[0083] 2 Sample carrier

[0084] 4 samples

[0085] 6 Immersion liquid

[0086] 8. Application device

[0087] 10-image field

[0088] 12 cameras

[0089] 14 wires

[0090] 16 processing units

[0091] 18 imaging devices

[0092] 20 typical images

[0093] 22 background images

[0094] 24 images

[0095] 26 difference images

[0096] Difference image in 28 polar coordinates

[0097] Route 30

[0098] 32 Result Images

[0099] 34 outlines

[0100] 36 front lens

[0101] 38 objectives with low aperture

[0102] 40 coverslip

[0103] 42 Safe Zone

[0104] 44 horns

[0105] 46. ​​Objectives with high aperture

[0106] 48 Safe Zone

[0107] 50 corners

[0108] C1 Image Coordinates

[0109] C2 Sample Carrier Coordinates

[0110] C3 Calibration Coordinates

[0111] D1 front lens diameter

[0112] D2 front lens diameter

[0113] E1 camera plane

[0114] E2 Sample Carrier Plane

[0115] E3 calibration plane

[0116] H homography

[0117] K calibration

[0118] OA optical axis

[0119] S1 Step 1

[0120] Step 2 of S2

[0121] Step 3 of S3

[0122] Step 4 of S4

[0123] Step 5 of S5

[0124] Step 5 of S6

[0125] Step 7 of S7

[0126] T translation

Claims

1. A method for monitoring an immersion liquid (6) in a microscope (1) having an objective lens that images a sample (4) located on a sample carrier (2), the method comprising the steps of: Step a) Position the camera (12) having an image field (10) such that it detects the sample carrier (2) and the space between the sample carrier (2) and the objective lens connected to the sample carrier (2) in the direction of the objective lens, the space being used to receive the immersion liquid (6). Step b) Apply the immersion solution (6) into the space between the sample carrier (2) and the objective lens; Step c) Take an image (24) with the immersion liquid (6) in the space between the sample carrier (2) and the objective lens; Step d) Determine the position, area and / or contour (34) of the immersion liquid (6) on the sample carrier (2) from the image (24) taken in step c). In step d), the position, area and / or contour of the immersion liquid (6) on the sample carrier (2) are determined by searching the structure in the image (24), the structure being located at a certain distance from the center of the objective lens, not being assigned to the objective lens and / or having an approximately annular shape.

2. The method according to claim 1, wherein, The camera (12) is calibrated after step a).

3. The method according to claim 2, characterized in that, Calibration is performed by positioning the calibration pattern in the plane of the sample carrier (2) and in the image field (10) of the camera (12) and capturing a calibration image.

4. The method according to any one of the preceding claims, wherein, The determination of image distortion caused by the position of the camera (12) in the space between the sample carrier (2) and the objective lens, and the determination of the position, area and / or contour (34) of the immersion liquid (6) on the sample carrier (2) from the image (24) taken in step c) are performed using the determined image distortion.

5. The method according to any one of claims 1 to 3, wherein, After step a), a background image (22) of the space between the sample carrier (2) and the objective lens is taken, and the position, area and / or contour (34) of the immersion liquid (6) on the sample carrier (2) is determined from the image (24) taken in step c) using the background image (22).

6. The method according to any one of claims 1 to 3, wherein, In step e), the outline and / or area of ​​the immersion liquid (6) on the sample carrier (2) are evaluated, and / or the volume of the immersion liquid (6) is estimated, and / or the remaining duration of the immersion liquid (6) on the sample carrier (2) due to evaporation is determined.

7. The method according to claim 6, characterized in that, In the event that the immersion liquid (6) deforms too much, is too small in volume and / or has too little remaining duration, the sample is moved more slowly, tactile feedback is provided, warnings are given to the user and / or the sample is automatically submerged.

8. The method according to any one of claims 1 to 3, wherein, The image field (10) is illuminated using a light source, which is mounted at the camera (12).

9. The method according to any one of claims 1 to 3, wherein, The image field (10) is illuminated using a light source present at the microscope (1).

10. The method according to claim 9, wherein, The image (24) was captured by the camera (12) in the infrared spectrum.

11. The method according to any one of claims 1 to 3, wherein, The image (24) was captured using a polarizing filter.

12. The method according to any one of claims 1 to 3, wherein, In step c), multiple images under different lighting conditions are captured, and the outline, area, and / or location of the immersion liquid are analyzed from the overall images.

13. The method according to any one of claims 1 to 3, wherein, In step d), impurities in the immersion solution (6) are identified.

14. The method according to claim 13, wherein, The identified impurities are filtered out from the image (24) taken in step c).

15. An apparatus for monitoring an immersion liquid (6) in a microscope (1) having an objective lens, the objective lens imaging a sample (4) located on a sample carrier (2), the apparatus comprising: - A camera (12) having an image field (10) positioned such that the image field (10) is aligned such that it detects the sample carrier (2) and the space between the sample carrier (2) and the objective lens connected to the sample carrier (2) in the direction toward the objective lens, the space being used to receive the immersion liquid (6). - A processing device (16), which is connected to the camera (12) via a wire (14). - The processing device (16) is configured such that it determines the position, area and / or outline (34) of the immersion liquid (6) on the sample carrier (2), wherein the processing device (16) uses an image (24) taken by the camera (12) of the space between the sample carrier (2) and the objective lens in which the immersion liquid (6) is present. The processing device (16) is configured such that it determines the position, area and / or contour of the immersion liquid (6) on the sample carrier (2) by searching for structures in the image (24), the structures being located at a certain distance from the center of the objective lens, not being assigned to the objective lens and / or having an approximately annular shape.

16. The apparatus according to claim 15, wherein, The device has an application device (8) which is connected to the processing device (16) via a wire (14) and applies the immersion liquid (6) into the space between the sample carrier (2) and the objective lens.

17. The apparatus according to claim 15, wherein, The processing device (16) determines, by means of a calibration image provided by the camera (12), the image distortion caused by the position of the camera (12) in the space between the sample carrier (2) and the objective lens, the calibration image being displayed in the plane of the sample carrier (2) and in the space between the sample carrier (2) and the objective lens with a calibration pattern in the image field (10) of the camera (12).

18. The apparatus according to any one of claims 15 to 17, wherein, The processing device (16) is used to determine the position, area and / or outline (34) of the immersion liquid (6) on the sample carrier (2) from a background image (22) of the space between the sample carrier (2) and the objective lens taken by the camera (12).

19. The apparatus according to any one of claims 15 to 17, wherein, The processing device (16) is configured such that it assesses the profile (34) of the immersion liquid (6) on the sample carrier (2), and / or estimates the volume of the immersion liquid (6), and / or determines the remaining duration of the immersion liquid (6) on the sample carrier (2) due to evaporation.

20. The apparatus according to claim 19, wherein, The processing device (16) is configured such that it initiates slower movement of the sample, tactile feedback, warnings to the user, and / or automatic immersion in the event that the immersion liquid (6) deforms too much, is too small in volume, and / or has too little remaining duration.

21. The apparatus according to any one of claims 15 to 17, wherein, A light source is installed at the camera (12) to illuminate the image field (10).

22. The apparatus according to any one of claims 15 to 17, wherein, The image field (10) is illuminated by a light source located at the microscope (1).

23. The apparatus according to claim 21, wherein, The camera (12) captures the image (24) in the infrared spectrum.

24. The apparatus according to any one of claims 15 to 17, wherein, A polarizing filter is arranged at the camera (12) so that the camera (12) uses the polarizing filter to capture the image (24).

25. The apparatus according to any one of claims 15 to 17, wherein, The processing device (16) is configured such that the camera (12) captures multiple images under different lighting conditions in the space between the sample carrier (2) and the objective lens, with the immersion liquid (6) in the space, and analyzes the outline, area and / or position of the immersion liquid from the overall images.

26. The apparatus according to any one of claims 15 to 17, wherein, The processing device (16) is configured to identify impurities in the immersion liquid (6).

27. The apparatus according to claim 26, wherein, The device filters out identified impurities from the image (24).

Images