Microscope system and method for analyzing overview images

Abstract

A microscopy system comprises a microscope (1) having an overview camera (9) for capturing an overview image (11) of at least one sample carrier (7) designed to receive at least one sample fluid (6); a computing device (20) configured to determine at least one sample image region (8) of the at least one sample fluid (6) within the at least one overview image (11). The computing device (20) comprises an evaluation program (30) into which the determined sample image region (8) is inputted and which is configured to determine a fluid state (35) of the associated sample fluid (6) based on the sample image region (8). A method for evaluating an overview image is also described.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a microscopy system for capturing an overview image of a sample carrier and to a method for evaluating an overview image. BACKGROUND

[0002] One application in the field of microscopy is the capturing of a series of images of a dynamic sample, such as a living cell, a microorganism, an organ or an organoid. In order to be able to make meaningful statements about such a sample, the control and setting of the sample environmental conditions is of great importance. In particular, the properties of the sample carrier and the artificially created environment, such as the sample medium, the buffer conditions, the growth factors and the temperature fluctuations, have a significant influence on the sample. It is therefore necessary to control a plurality of parameters, in particular with regard to the sample medium.

[0003] In the presence of small amounts of medium or due to the influence of the surrounding air, evaporation of the medium can lead to damage to the sample, for example, in the case of changes in the pH value, temperature fluctuations or surface oscillations. In order to minimize the influence of evaporation, the sample container can be filled to a higher level, which entails the risk of liquid overflow when the sample stage is moved automatically.

[0004] In order to facilitate automated analysis in particular, microscopy systems have been developed with which a sample carrier can be detected and the sample container positioned in an overview image. This can enable the automatic positioning of the sample stage in order to select a sample container for more detailed analysis. A general microscopy system comprises a microscope with an overview camera for capturing at least one overview image of a sample carrier designed to receive at least one sample fluid; a computing device configured to determine at least one sample image region of the at least one sample fluid within the at least one overview image. In a general method for evaluating an overview image, at least one overview image of a sample carrier designed to receive at least one sample fluid can be received. At least one sample image region of the at least one sample fluid is determined within the at least one overview image. Microscopes for capturing overview images and for positioning sample carriers or sample regions in the overview image by means of segmentation, for example, have been described by the applicant, for example in DE 10 2018 133 188 A1, DE 10 2017 109 698 A1 and DE 10 2017 111 718 A1. SUMMARY

[0005] It can be considered an object of the present application to provide a microscopy system and a method which enable the efficient analysis of one or more samples supported by a sample carrier by evaluation of an overview image.

[0006] This object is achieved by a microscopy system and by a method for evaluating an overview image.

[0007] According to the application, in a microscopy system of the aforementioned type, the computing device comprises an evaluation program into which the determined sample image region is input and which is configured to determine a liquid state of the associated sample fluid on the basis of the sample image region.

[0008] According to the application, in a method of the aforementioned type, the determined sample image region is input into an evaluation program. The evaluation program then determines a liquid state of the associated sample fluid on the basis of the sample image region, respectively.

[0009] The overview image can thus be used not only for sample positioning or documentation purposes, but also for obtaining information about the medium in which the sample is located. For example, a rough estimate of the filling level of all sample containers can be made in order to control the positioning speed of the sample stage accordingly. Defect influences on the liquid in which the sample is located, for example a change in the pH value of the liquid or crystallization of the sample in the liquid, can be detected on the basis of the overview image alone. This makes early intervention possible before a time-consuming in-depth analysis is unnecessarily carried out.

[0010] Alternative embodiments

[0011] Advantageous variants of the microscopy system according to the application and of the method according to the application are the object of the dependent claims and are explained below.

[0012] Evaluation of multiple sample regions

[0013] The sample carrier can comprise a plurality of sample containers for receiving sample fluid. For example, the sample carrier can be a microtiter plate or a chamber slide. The computing device can be configured to determine a plurality of sample image regions representing the sample containers within the same overview image. The evaluation program can be configured to determine a liquid state of the associated sample fluid for each of the sample image regions. It is thus possible to determine the liquid state for a plurality of sample containers or even for all sample containers visible in the overview image, respectively. In an automated process involving a sample carrier with a large number of sample containers, it is particularly advantageous to be able to detect defective cases from the overview image at an early stage.

[0014] In order to make the application more easily understandable, the sample fluid, the sample image region, the liquid state and the sample container are described in the following in parts in the singular. However, although the singular form is chosen for the purpose of illustration, it is understood that the sample fluid can be contained in a plurality of sample containers such that the evaluation program infers the liquid state for a plurality of sample image regions, respectively. In principle, it is also possible to determine a collective liquid state for a plurality of sample image regions, i.e. for a plurality of sample fluids of different sample containers; for example, the collective liquid state can indicate whether all filling levels are within the correct range.

[0015] Evaluation program

[0016] The evaluation procedure can comprise a model learned using training data (hereinafter referred to as evaluation model), or can in principle also be constituted by a classical algorithm without a machine learning model. Features that can be evaluated by a classical algorithm will be described in more detail later.

[0017] The learning evaluation model can in particular be formed by a supervised or unsupervised learning process. In a supervised learning process, the training data can comprise annotated overview images or annotated sample image regions of overview images. The associated liquid states can be predefined in the form of annotations, respectively. Given the high variability of features and imaging conditions captured in the overview images, the learning model is less prone to errors compared to classical image analysis methods. In particular, a deep neural network, such as a convolutional neural network (CNN), can be used, which exhibits a high degree of robustness and universality.

[0018] In the case of unsupervised learning, the evaluation model can be designed as an anomaly detection / novelty detection. The evaluation model can infer an anomaly (e.g. an anomalous liquid state) as a function of the similarity of the input sample image region to the training data. For example, the training data can only describe sample fluids whose state has been classified as correct by a user. Deviations from such image data can be evaluated as (potentially) defective liquid states. For example, the description of a sample image region can differ from the training data due to sample contamination.

[0019] Different liquid states are described below. Annotations of the training data corresponding to these liquid states can be used.

[0020] Liquid states, in particular liquid level, sample color or turbidity

[0021] The liquid states generally relate to features of the liquid in which the sample is located or features of the sample contained in the liquid. In particular, the liquid states can indicate the presence, quantity and / or contamination of the liquid.

[0022] The liquid states can be represented by binary values (e.g. yes / no), ordinal expressions (e.g. little / medium / lot) or specific values (e.g. time until evaporation of the liquid).

[0023] In particular, the detected liquid states can indicate the filling level of the associated sample fluid. If a learning evaluation model is used, the corresponding training data can show sample containers filled to different heights, and the respective filling levels can be indicated in the form of annotations. The evaluation model can be a CNN that maps the input images to the filling level by regression or to an expression of the filling amount such as "little / medium / lot" or "not enough / enough / excess" by classification. Alternatively, the filling level can be expressed as an estimated height, e.g. in mm, as an estimated liquid volume, e.g. in mm3, or as a distance to the upper edge of the associated sample container.

[0024] Determining the respective liquid state for each determined sample image region of a sample container can additionally or alternatively indicate whether the sample container in question is empty or contains a liquid. The computing device can optionally be configured to only position and analyze those sample containers in an automated sample analysis that have been determined to contain a sufficient amount of liquid. Thus, in the case of a screening experiment, the experiment or the presence of a sample is conducted in only those sample containers of a microtiter plate or chamber slide that are analyzed, so that an efficient use of time and data can be achieved. More generally, it can be determined which sample containers to subject to further analysis depending on the determined liquid state. For example, the sample stage can be controlled to only guide those sample containers into the microscope light path whose liquid state indicates no defect.

[0025] Alternatively or additionally, the liquid state can also distinguish the color, color intensity and / or the absorption or turbidity level of the filling. For example, the color intensity generally depends on the filling level, so that the filling level of the relevant sample container can be inferred from the color intensity. In the case of a colorless liquid, the turbidity level can also be captured, which can be used as an indicator of the filling level or for determining which sample containers should be further analyzed. A deviation of the color or turbidity level from an expected color or turbidity range can indicate a contamination or an unintentional modification of the sample fluid.

[0026] It can also be predicted how much time remains until the filling has evaporated or has evaporated to the point that there is no longer a sufficient amount of liquid, as a liquid state to be output.

[0027] The training data of the supervised learning process comprises training images that contain different manifestations of the above-mentioned features (e.g. different turbidity levels) and corresponding annotations (e.g. filling levels corresponding to turbidity levels). An overview image of a sample container can also be captured using a colored liquid (e.g. a red liquid), which is then converted into a grayscale image to generate a training image. The color can be used in the training as a true annotation of the filling level.

[0028] Time series

[0029] Optionally, multiple overview images of the same sample carrier or the same sample container can be captured after a time interval. At least one sample image region of at least one sample fluid is determined in each of these overview images. Tracking can also be used here, so that even if the sample carrier has moved, the same sample container can be found again in different overview images.

[0030] By means of the evaluation program, the filling level or another liquid state can be determined for the same sample region or the same sample container over time. For example, each sample image region of the different overview images can be entered individually into the evaluation program in order to calculate the respective liquid state, which over time form a progression when connected. Alternatively, the sample image regions (or parts thereof) of the various overview images are jointly entered into the evaluation program, from which the evaluation program directly determines the progression over time. In this case, the evaluation model can consist of a CNN with a multi-dimensional input in order to map the sample image regions of the same sample container from the successively captured overview images to the liquid state or the evolution of the liquid state. In addition to CNNs, it is also possible to use, for example, RNNs (recurrent neural networks), in particular LSTMs (long short-term memory). The consideration of several successively captured image portions together can be advantageous for an accurate depiction of the relative color intensities.

[0031] On the basis of the sample image regions of the successively captured overview images, a prediction can also be calculated by means of the evaluation program as to when the filling level will fall below the minimum filling level. This prediction of the type of liquid state described above can additionally or alternatively be calculated. The respective training data can comprise, for example, an overview image captured at the point in time of the minimum filling level or a sample image region thereof, which is defined by evaporation. The evaluation program can be designed to output the prediction directly or to output the respective filling level or the progression of the filling level over time. The prediction of when the filling level will fall below the minimum filling level can then be calculated on the basis of the progression over time. For this purpose, for example, the progression over time can be entered into a learning model (prediction model), which has learned to calculate, using the training data, a prediction of when the filling level will fall below the point in time of the minimum filling level on the basis of the input of the progression over time.

[0032] The time series can also be applied in the described manner to other types of liquid states for which, in addition to the filling level, a respective upper or lower limit can be specified. For example, a time series relating to a pH indicator color change can be generated in order to predict the point in time at which a particular / unacceptable pH value will be reached.

[0033] Overview images at different angles

[0034] The described variant with one overview image can be modified such that multiple overview images of the same sample carrier or the same sample container or the same sample container are captured. While it is preferred to calculate the above-mentioned time series using overview images, wherein the overview camera is oriented in the same position or has the same perspective with respect to the sample carrier, variants will be described below in which two or more overview images are captured from different perspectives with respect to the sample carrier. For this purpose, overview cameras in different positioning can be used, which in particular capture the overview images simultaneously. Alternatively, the sample stage is moved during the image capture or between image capture events, then multiple overview images can be captured by the same overview camera. In the multiple overview images, the sample image regions relating to the same sample fluid or the same sample container are determined separately. The multiple sample image regions from different overview images relating to the same sample fluid are jointly fed to the evaluation program to determine the liquid state. The evaluation program can thus exploit the fact that, for example, the sample image regions differ as a function of the fill level in the different overview images. Thus, in the training of the evaluation model in this case, multiple images from multiple perspectives can jointly constitute the input to the evaluation model.

[0035] Features / criteria for evaluating the liquid state

[0036] The evaluation program can consider features and criteria to infer the described liquid states, which are explained in more detail below. The sample image regions can be evaluated by criteria performed by classic image processing applications. Alternatively, an evaluation model can be trained to consider these features by using training images, which differ in these features and which provide corresponding annotations with the associated liquid states.

[0037] The evaluation program can consider at least one of the following features to determine the liquid state

[0038] - the color intensity of the sample image region;

[0039] - how the image content in the sample image region appears distorted or changed in size or position due to a lens effect caused by the sample fluid surface;

[0040] - whether droplets on the side wall or lid of the associated sample container can be discerned in the sample image region;

[0041] - whether the color or color distribution within the sample image region deviates from the expected sample color or sample color distribution, based on which a contamination of the sample can be inferred.

[0042] To take into account the lens effect of the sample fluid surface, the evaluation program can evaluate the sample image area based on how the visible background passing through the sample fluid appears, in particular how the distortion or degree of change in size of the background appears and / or the size of the sample container visible through the sample fluid. The background can be known; for example, the background can be an LED matrix, wherein the pitch of the LEDs is known. Thus, a background distortion possibly caused by a liquid meniscus can be detected by a smaller or larger LED pitch than expected. According to a further example, while filled wells / sample containers can appear identical to unfilled wells, they can appear smaller. In particular, the edges viewed through the sample fluid can appear smaller. According to yet another example, a cover with a special configuration or texture can be used, for example, a checkerboard pattern. The cover can be arranged on a microtiter plate or some other type of sample carrier. In this case, distortions can be detected relatively easily and quantified precisely.

[0043] Pseudo-ghosts caused by the liquid can also be detected when capturing the overview image. For example, when a checkerboard pattern is activated by an LED matrix, visible squares appear in the sample image area of the sample container filled with liquid.

[0044] The lens effect can be utilized when evaluating a single overview image or directly a plurality of overview images captured in succession. For example, a plurality of overview images can be captured while the sample stage is moved, in particular fast back and forth. At least one sample image area is determined in each of these overview images. The evaluation program takes into account changes in appearance or changes in the lens effect in the sample image area caused by the movement of the sample stage over the plurality of overview images. For example, this movement can result in a displacement and / or deformation of the background visible through the liquid. The liquid state, in particular the liquid level, can be inferred from such changes.

[0045] In view of the movement of the sample stage, the evaluation program can also take into account the movement of the sample or the inertia of the sample in the surrounding liquid. Thus, the plurality of overview images is evaluated to determine the liquid state depending on whether the sample moves relative to the surrounding liquid or not.

[0046] Alternatively or additionally, the evaluation program is designed to detect crystals in the sample image area and use them as an indication of the sample fluid being dry. In this way, a dry sample, a non-dry sample and optionally an empty sample container can be distinguished in terms of the liquid state.

[0047] The evaluation program can also evaluate reflections on the liquid surface in the sample image area to determine the liquid state. In this case, in particular the illumination source and the perspective of the panoramic camera can be aligned from above to the sample carrier.

[0048] Optionally, also different illuminations can be used to capture the plurality of overview images. The evaluation program evaluates the differences between the corresponding sample image regions of the plurality of overview images, i.e. the sample image regions of the same sample container depicted in different overview images. The liquid state can in turn be determined from these differences. In particular, a change of color or polarization can be detected with switching different illumination sources and subsequently. According to another example, an illumination from above can be used to capture the overview images and to segment the sample containers in the overview images. Also an overview image is captured by illuminating from below and again segmenting the sample containers. Now any differences between the segmentation masks can be determined. In case of filled wells / sample containers, when illuminated from below, the actual size of the sample containers can be seen, while when illuminated from above, the sample containers should appear smaller.

[0049] The evaluation program can also infer from the function according to the visible / detected droplets or condensate in the sample image region a filling level or whether there is any sample fluid in the respective sample container. If droplets can be seen, it indicates that there is sample fluid. In case of closed sample containers, it can be inferred from the water droplets or condensate on the sample container lid that the filling level has dropped from the original filling level, e.g. due to evaporation.

[0050] The evaluation program can also be designed to detect the liquid surface and to determine the liquid state accordingly. The liquid surface can be visible, e.g. the size of the ring in the overview image can be evaluated.

[0051] Also a plurality of overview images can be captured consecutively and the evaluation program determines whether the sample has been contaminated by determining a color change in the sample image region as liquid state. Thus it is determined whether there is any change in the color of the sample fluid in the particular sample container on the overview image. The sample fluid can also optionally contain a pH indicator.

[0052] As bacteria and yeast change the pH value of the surrounding medium, the contamination detection is improved. Irrespective of such contamination, the evaluation program can generally also infer a change in the liquid state if the pH indicator color changes. In particular, the pH value can also be estimated as a liquid state. One reason for a change in the pH value can be, for example, too much or too little CO2aeration, which can be caused by incorrect settings of the incubator. In particular, if a change in the pH value of the liquid of a plurality of sample containers is determined, the evaluation program can also directly indicate a possible cause of the pH change, e.g. incorrect aeration.

[0053] Context data

[0054] The evaluation procedure can be designed to additionally consider predefined information (context data) in determining the liquid state, which is related to the sample, the sample carrier, the measurement situation or other device characteristics. As described in more detail below, the context data can be immediately available or selected according to an information function to be determined from a database containing different context data. Both evaluation procedures comprising a learning evaluation model and evaluation procedures running without using a machine learning model can make use of context data.

[0055] The context data can in particular relate to information about the initial filling level (in the form of a binary value or an explicit quantity), which can be specified by the user, for example. The context data can also include details related to the liquid used or the sample type used and / or details related to the sample carrier used. This information can be entered by the user or determined automatically, in particular from the overview image. For example, the overview image can be evaluated on the basis of sample carrier characteristics, such as a descriptive label or the sample container diameter and spacing, in order to infer the sample carrier type from a plurality of predefined sample carrier types. Information about the volume of the sample container can be stored, for example, as context data for a plurality of sample carrier types. In the case of a learning evaluation model, the training data can be used together with the format of the respective sample carrier type, so that the model learns to take the context data of the sample carrier type into account when evaluating the sample image region. If the context data relates to the liquid or the sample type, it is easier to infer evaporation or turbidity behavior, while it is possible to interpret the color or color intensity more precisely.

[0056] Additionally or alternatively, the context data can relate to information about the type of experiment being carried out, which can be entered by the user or determined automatically. This information can allow the evaluation procedure to derive whether the sample image regions of the liquids of different sample containers should appear the same or different in the overview image, for example. The experiment can also indicate whether the sample change over time is intentional or indicates an error.

[0057] Determining the sample image region

[0058] In principle, the sample image region or regions representing the sample fluid or the sample carrier of the sample container can be determined from the at least one overview image in any way.

[0059] In simple cases, for example, the arrangement of the special sample carriers on the sample stage can be predetermined by a receiving frame or stop. If the sample carrier type and the position of the sample stage are known, it can be inferred from this where the sample image region or regions should be located in the overview image. In this case, defining the sample image region does not require an evaluation of the overview image.

[0060] Optionally, the overview image, images derived from the same or at least one further overview image can be evaluated using image analysis methods in order to determine the sample image region. In principle, this can be achieved by classical image processing algorithms without a learning model. However, for robust detection of a plurality of different sample containers or sample types, it is preferred to provide a learning model to find the sample image region of the sample fluid or sample container. Thus, the computing device can comprise a model (localization model) that is learned using training data for determining the sample image region in the overview image. All sample image regions determined by the localization model can be input into the evaluation program. For example, the localization model can perform semantic segmentation or detection by a trained CNN in order to assign an image portion (i.e. sample image region) to each sample container. The CNN is learned using a training data set consisting of images and associated annotations (image coordinates or segmentation masks). The annotations can be provided manually by, for example, an expert.

[0061] Actions taken in view of the determined liquid state

[0062] The determined liquid state can be used to set basic parameters before a planned experiment using the microscope system. The determined liquid state can also be used to control parameters while the experiment is being conducted.

[0063] The determined liquid state can be used, for example, to automatically select sample containers for close monitoring in the next step of a planned sequence.

[0064] If the determined liquid state does not correspond to a predetermined target state or if it is detected that there are sample containers for which the liquid state cannot be determined by the evaluation program, a warning can be output, in particular to a user. In particular, a warning can also be generated when the filling level of certain sample containers cannot be determined, in the case of a determination of contamination, in particular a change in the color of a pH indicator by the pH value, or in the case of a determination of turbidity changes. A warning can also be generated if the filling level is in a critical range (too low or too high) or if there is any change in the filling level.

[0065] If the microscope comprises an upright microscope stand and an immersion objective, the determined liquid level can also be used as part of an overflow protection system. Since the overview image is captured before the immersion of the objective into the sample container, a warning can be issued in time if the filling level exceeds a maximum value and the immersion of the objective could therefore lead to an overflow.

[0066] Instead of or in addition to a warning, parameters can also be adjusted automatically or manually in accordance with instructions output to the user. For example, instructions to modify the incubation conditions can be output.

[0067] Positioning speed of the sample stage

[0068] As an alternative or in addition to the above-described actions, the maximum positioning speed of the sample stage can also be set as a function of the determined fill level. In the case of a plurality of sample containers with different fill levels, the maximum positioning speed can be set as a function of the determined highest fill level. The lower the positioning speed is set, the higher the fill level or the closer the fill level is to the upper edge of the associated sample container. The stage positioning speed can thus be adjusted so that the maximum efficiency of the optical system is achieved under safe conditions.

[0069] The overview image can also be used to determine the size of the cross section of the sample container, whereby the maximum positioning speed can also be set as a function of the determined cross section size. The surface tension can thus be taken into account, in particular in relation to sample containers with a small cross section, so that a high stage positioning speed can also be achieved despite a higher fill level. The size of the cross section of the sample container can be determined, in particular, from the respective sample image region or from a majority of the overview image. The overview image can also be used for the classification of the sample carrier type. For this purpose, for example, the overview image or an image calculated therefrom can be input into a trained classification model which has been trained using training images of different sample carrier types to distinguish images calculated therefrom. The cross section size of the sample container can be stored separately for different sample carrier types, so that the cross section size can be determined by the classification of the sample carrier.

[0070] General features

[0071] It is to be understood that the microscope system is a device comprising at least one computing device and a microscope. In principle, the microscope can be understood as any measuring device with magnifying power, in particular an optical microscope, an X-ray microscope, an electron microscope, a macroscopic microscope or an image capturing device with magnifying power of some other design.

[0072] The computing device can be designed as an integral part of the microscope, arranged separately in the vicinity of the microscope or arranged at any distance from the microscope in a remote location. The computing device can also be designed to communicate with the microscope decentralized by a data link. It can generally be formed by any combination of electronics and software and in particular comprises a computer, a server, a cloud-based computing system or one or more microprocessors or graphics processors. The computing device can also be configured to control the microscope camera, the overview camera, the image capturing, the sample stage driver and / or other microscope components.

[0073] In principle, the sample carrier can be any object for receiving one or more sample liquids, such as a microtiter plate, a chamber slide, a petri dish or a petri dish designed with multiple individual compartments. The sample carrier can comprise one or more sample containers for receiving a respective sample liquid, such as the circular wells / containers of a microtiter plate or the rectangular chambers of a chamber slide. Multiple sample liquids can also be arranged separately from each other, for example in the form of droplets in the same container, for example next to each other on the bottom of a petri dish. It is understood that a sample liquid is a liquid containing a sample to be analyzed. In the context of the present disclosure, the expressions "fluid" and "liquid" can be understood interchangeably. In particular, a sample fluid can be referred to as a sample liquid and a fluidic state can be referred to as a liquid state. The fluid can be any water-based and / or oil-based solution and / or comprise a culture medium. The sample can be any kind of sample and comprise, for example, dissolved particles or floating or deposited objects in the fluid. In particular, the sample can comprise biological cells or cell parts.

[0074] In addition to the sample camera capturing images of the sample region at a higher magnification, an overview camera for capturing overview images can also be provided. Alternatively, one and the same camera can be used which uses different objectives or optical systems to capture the overview images and the sample images at a higher magnification. The overview camera can be mounted on a fixed device frame, such as a microscope stand, or on a movable component, such as a sample stage, a focus drive or an objective revolver. A color camera can be more suitable for more precise image analysis. The overview image can be the raw image captured by the camera or an image processed from one or more raw images. In particular, the captured raw image / overview image can be further processed before it is evaluated in the manner described in the present disclosure. Multiple overview cameras can also be aligned to the sample carrier from different angles. An overview image can thus be calculated based on images from multiple overview cameras or overview images from different overview cameras (or from the same sample image region) are jointly input into the evaluation program. The filling level can potentially be determined more precisely from multiple overview images with different camera orientations. Method variants of the present invention can use pre-captured overview images, for example, the overview images can be received from a memory or, alternatively, the capturing of the overview images can be an integral part of the claimed method variants. The images described in the present disclosure, such as the overview images, can consist of pixels, can be vector graphics or a mixture of both. In particular, the segmentation mask can be a vector graphic or can be converted into a vector graphic.

[0075] The sample image region can be an image region of the overview image or a portion calculated from multiple overview images in which at least a portion of the sample liquid is visible. It can also be divided into multiple sample image regions according to the sample containers, whereby multiple, but not necessarily all, of the determined sample image regions can show sample fluid.

[0076] During the capturing of the overview image, the illumination device can selectively illuminate the sample carrier. The illumination device can be the light source of the microscope which is also used during the sample analysis involving the microscope objective and the microscope camera, e.g. a light source for analysis using incident light or transmitted light. Alternatively, it can be an additional light source which is solely used for capturing the overview image and is directed in a targeted manner, e.g. at an angle to the top side or the bottom side of the sample carrier. The illumination device can be selectively switched to generate different illumination modes in which the evaluation procedure evaluates the sample liquid visibly through this illumination mode.

[0077] The computer program according to the application comprises commands which, when the method variant is executed by a computer, result in the executed one of the method variants.

[0078] The learning model or machine learning model described in the present disclosure respectively denotes a model which is learned using training data by a learning algorithm. The machine learning model may, for example, respectively comprise one or more convolutional neural networks (CNN) which at least receive an input image, in particular an overview image or an image calculated therefrom as input. The training of the machine learning model can be carried out by a supervised learning process in which training overview images are provided with associated annotations / labels. The learning algorithm is used to define model parameters of the machine learning model on the basis of the annotated training overview images. For this purpose, a predefined objective function can be optimized, e.g. a loss function is minimized. The loss function describes the deviation between the predetermined labels and the current output of the machine learning model, which are calculated from the training overview images using the current model parameter values. The model parameter values are modified in order to minimize the loss function, e.g. by gradient descent and backpropagation calculations. In the case of a CNN, the model parameters can in particular comprise the entries of the convolution matrices of the different layers of the CNN. Other deep neural network model architectures can also be used instead of a CNN. In addition to the supervised learning process, an unsupervised training can also be carried out without annotations being provided for the training images. A partially supervised training or a reinforcement learning process can also be carried out.

[0079] When implemented as intended, the features described as additional devices also result in method variants according to the application. Conversely, the microscope system can also be configured to carry out the described method variants. In particular, the computing device can be configured to carry out the described method variants and / or output commands for carrying out the described method steps. The computing device can also comprise the described computer program. While some variants use trained machine learning models, other variants of the application result from the implementation of the respective training steps. BRIEF DESCRIPTION OF DRAWINGS

[0080] Further advantages and features of the present application will become apparent from the following description, given, by way of example only, with reference to the accompanying drawings:

[0081] Figure 1 is a schematic diagram of an example embodiment of a microscopy system of the present invention; Figure 1 Figure 2 is a schematic diagram of an example embodiment of a method of the present invention;

[0082] Figure 3 is a schematic diagram of an example embodiment of a process of a method of the present invention; Figure 2 Figure 4 is a schematic diagram of an example embodiment of a process of a method of the present invention;

[0083] Figure 5 is a schematic diagram of an example embodiment of a process of a method of the present invention; Figure 3 Figure 6 shows a learning process of an evaluation model of the method of Figure 5; and Figure 2 Figure 7 is a flow diagram of another example embodiment of a method of the present invention. DETAILED DESCRIPTION

[0084] Figure 4 The different example embodiments are described below with reference to the accompanying drawings. In general, like elements and elements acting in a similar way are denoted by the same reference signs.

[0085] The different example embodiments are described below with reference to the accompanying drawings. In general, like elements and elements acting in a similar way are denoted by the same reference signs.

[0086] APPENDIX Figure 1

[0087] Figure 1 is a schematic diagram of an example embodiment of a microscopy system of the present invention. The microscopy system 100 comprises a computing device 20 and a microscope 1, which in the example embodiment is an optical microscope, but in principle can be any type of microscope. The microscope 1 comprises a stand 2 through which other microscope components are supported. The other microscope components can comprise, among others, an illumination device 15, an objective changer / rotator 3 on which in the example embodiment an objective 4A is mounted, a sample stage 5 on which a sample carrier 7 can be positioned, and a microscope camera 4B. If the objective 4A has been rotated to be in the microscope light path, the microscope camera 4B receives detection light from one or more samples to capture sample images, the samples being supported by the sample carrier 7. Figure 1 Figure 1 is a schematic diagram of an example embodiment of a microscopy system of the present invention. The microscopy system 100 comprises a computing device 20 and a microscope 1, which in the example embodiment is an optical microscope, but in principle can be any type of microscope. The microscope 1 comprises a stand 2 through which other microscope components are supported. The other microscope components can comprise, among others, an illumination device 15, an objective changer / rotator 3 on which in the example embodiment an objective 4A is mounted, a sample stage 5 on which a sample carrier 7 can be positioned, and a microscope camera 4B. If the objective 4A has been rotated to be in the microscope light path, the microscope camera 4B receives detection light from one or more samples to capture sample images, the samples being supported by the sample carrier 7.

[0088] The microscope 1 further comprises an overview camera 9 for capturing an overview image of the sample carrier 7 or a part of the sample carrier 7. The field of view 9A of the overview camera 9 is larger than the field of view when capturing sample images. In the shown example, the overview camera 9 observes the sample carrier 7 via a mirror 9B. The mirror 9B is arranged on the objective rotator 3 and can be selected instead of the objective 4A. In variants of this embodiment, the mirror or some other deflection element can also be arranged at a different position. Alternatively, the overview camera 9 can also be arranged to observe the sample carrier 7 directly without the mirror 9B. While in the shown example the overview camera 9 observes the top side of the sample carrier 7, the overview camera 9 can instead be directed to the bottom side of the sample carrier 7. In principle, the microscope camera 8 can also be used as an overview camera by selecting a different objective, in particular a macro objective, to capture the overview image via the objective rotator 3. ​

[0089] The computing device 20 uses a computer program 80 according to the invention to process the overview image and selectively control microscope components based on the results of the processing. For example, the computing device 20 can evaluate the overview image to determine the location of the wells in a microtiter plate so that the sample stage 5 can be subsequently controlled in a manner that allows specific wells to be appropriately positioned for further analysis. Example embodiments of the invention can use the overview image to extract further information about the sample, as shown in the appendix. Figure 2 A more detailed description.

[0090] APPENDIX Figure 2

[0091] Appendix Figure 2 The process of an example embodiment of the method of the present invention is illustrated schematically. This method can be performed by... Figure 1 It is executed by a computer program or computing device.

[0092] Appendix Figure 2 The sample carrier 7 is shown in the upper left corner. Here, the sample carrier 7 is a microtiter plate with multiple wells or sample containers 7A-7D. The sample can be arranged in the sample containers 7A-7D respectively and is usually in a fluid. This fluid, or the fluid containing the sample, is called the sample fluid.

[0093] In step S1, the microscope's overview camera captures an overview image 11. In this example, the overview camera observes the top side of the sample carrier 7 at an angle. The illustrated example depicts an inverted microscope setup, where the condenser 10 is positioned above the sample carrier 7, while the objective lens, which is not visible here, is positioned below the sample carrier 7. The circular sample containers 7C and 7D appear elliptical due to the oblique view.

[0094] In step S2, the overview image 11 is input into a computer program 80 with calibration parameters P. Calibration parameters P describe the relationship between positional information from the overview image 11 and positional information related to a microscope reference position. Specifically, calibration parameters P enable the input overview image to be homogenized onto another plane / view. In the illustrated example, computer program 80 uses calibration parameters P to calculate a planar view and outputs the overview image 11 as a planar view in step S3. The circular sample container is accurately displayed as a circle in the planar view, which facilitates subsequent processing steps.

[0095] In step S4, the overview image 11 of the planar view is input into the segmentation model S. In step S5, the segmentation model S calculates the segmentation mask, as shown in Figure 5. Figure 2The combination of the segmentation mask and the overview image 11 is shown as an overlay 21. In the segmentation mask, the wells 7A, 7B are located and distinguished from the background, i.e. from the rest of the image content. For example, the segmentation mask can be a binary mask, in which one pixel value indicates that the corresponding pixel belongs to the well, while the other pixel value indicates that the corresponding pixel belongs to the background.

[0096] By the segmentation of the wells 7A, 7B, image regions, here referred to as sample image regions 8, are labeled. Each sample image region 8 corresponds to one of the wells / sample containers 7A, 7B. In this case, the segmentation model S is a learning model that is learned using annotated training data, i.e. overview images and predefined segmentation masks. In particular, the segmentation model can be designed for a segmentation in which not only the image regions of the sample containers are so labeled, but also a distinction between different sample containers is made. For example, this can be relevant for identifying directly adjacent sample containers that are in contact with a segmentation surface as different sample containers. Alternatively, a classical segmentation algorithm can also be used. In a further variant, the sample image regions 8 can be determined by a detection model that is used to calculate a bounding box around the sample containers 7A, 7B. The bounding box can be rectangular or square and does not necessarily have to coincide completely with the edges of the sample containers 7A, 7B visible in the overview image 11. In general, it is sufficient for each sample image region 8 to depict one part of the sample container or the entire sample container, although it is also possible to depict a region around the sample container. For the purpose of further analysis, each sample image region 8 only shows the image portion from a single well 7A, 7B, although in principle multiple wells can be included in the same sample image region 8 and subsequently evaluated together or separately for individual evaluations.

[0097] Figure 2 A part of the overview image 11 is further schematically shown as a zoomed-in image portion 12. A plurality of sample containers 7A-7D can be discerned, which can be filled with sample fluid 6, respectively. The sample fluid 6 can differ in type or quantity. Different filling levels of the sample fluid 6 in the sample containers 7A-7D result in different color intensities or turbidities. In a purely schematic manner, the strongest color intensity can be found in the sample container 7A with the highest filling level. On the other hand, the sample container 7D appears virtually transparent, as no sample fluid 6 is received therein. For illustrative purposes, the boundaries of the sample image regions 8, as previously determined by segmentation, are indicated as dashed lines. For a clearer illustration, the remaining determined sample image regions 8 are not indicated in the same manner.

[0098] In step S6, the various sample image regions 8 from the magnified image portion 12 are now (individually or jointly) input into an evaluation program 30. The evaluation program in this example is constituted as a learning evaluation model 31, which can in particular comprise a deep neural network, for example a CNN. The evaluation program comprises model parameters, the values of which have been defined using training data. For illustrative purposes, the entries of the convolution matrix of the CNN are shown as model parameters P1-P9. The evaluation model 31 has learned to distinguish different fluid states using the training data and based on the sample image regions 8. In this process, the evaluation model 31 can also take into account, together with the sample image regions 8, associated context data i input in step S7. This will be explained in more detail later. For example, the fluid state can be or comprise an indication of the fill level, which can be estimated from the color intensity or turbidity of the sample fluid 6 visible in the sample image region 8, etc. In step S8, the evaluation model 31 calculates a fluid state 35, for example an indication of the fill level. This can occur, for example, by training the evaluation model 31 to perform a classification which indicates from possible classification results, for example, whether the fill level is too low, whether it is as desired or whether it is too high.

[0099] With the described variant of the application, information about the sample fluid in one or more sample containers 7A-7D can be extracted from the simple overview image 11. The progress of the experiment can thus be monitored and errors detected at an early stage. For example, if one of the sample containers 7A-7D is filled to too high a level - which would result in an overflow occurring in the case of an immersion objective - this can be detected before an automated in-depth analysis of all or several sample containers of a microtiter plate by means of an electric sample stage and an objective is carried out.

[0100] The learning process of the evaluation model 31 is described below with reference to the following figure.

[0101] APPENDIX Figure 3

[0102] Attached Figure 3 The learning process of the evaluation model 31 according to the variant embodiment of the application is shown schematically. Figure 2 The learning process constitutes part of some variants of the method, while other variants of the application use a trained model, so that the steps relating to the training do not constitute part of these variants.

[0103] Training data T comprising a plurality of training images T1-T12 is provided. In the present example, the training images T1-T12 are sample image regions extracted from an overview image. Each sample image region shows one sample container, which can be empty or can contain a sample fluid. The training images T1-T12 differ, inter alia, in the fluid state of the sample fluid depicted, respectively. For each training image T1-T12, the fluid state is made known (ground truth) in the form of a respective annotation A.

[0104] The fluid state can be, for example, a fill level of the sample fluid in the respective sample container, such that the annotation A indicates the respective fill level H1-H12. Alternatively or additionally, the annotation can also indicate, for example, a pH value, which is particularly feasible if a pH indicator is added to the respective sample fluid, which changes the color of the sample fluid depending on the pH value of the sample fluid.

[0105] The evaluation model 31 starts with (e.g. predetermined) starting values of the model parameters, and the training images T1-T12 or a portion (batch) thereof are input to the evaluation model 31 for training. Based on the initial model parameters and the input training images T1-T12, the evaluation model 31 calculates an output 35, which is fed to a loss function L. The loss function L captures the difference of the output 35 calculated using the current model parameter values from the associated annotations A. An optimization function O iteratively minimizes the loss function L, for which the optimization function O iteratively adjusts the values of the model parameter values, for example by gradient descent and backpropagation.

[0106] The training process ends when a stopping criterion is reached, for example a maximum number of iterations. The output 35 calculated by the trained evaluation model 31 now indicates the fluid state. The selected annotations A determine which fluid states the evaluation model 31 can name. The selected training images T1-T12 determine which image content the evaluation model 31 considers or how it considers the image content to calculate the output / fluid state 35.

[0107] The background can be seen, for example, through the transparent or partially transparent sample fluid. In particular, the background can be an arrangement of light sources (switchable). For example, in the training images T1, T2 and T5, a checkered pattern caused by the LED matrix of the microscope illumination device is visible through the sample fluid. Since the training images T1, T2, T5 are captured using the same illumination device, i.e. the spacing between the LEDs remains constant, the differences in the visible checkered pattern are mainly caused by the arrangement of the sample containers and the properties of the sample fluid, in particular its fill level, transparency and / or refractive index. By using training images that depict different fill levels and that each include the visible checkered pattern generated by the illumination device of the microscope, the evaluation model 31 can learn to infer the fill level based on this image content.

[0108] When the background is visible through the sample liquid, lens effects can also be considered, in particular. Thus, the surface of the sample liquid acts as a lens that distorts the background. By using multiple training images that show the same background, but that are traversed by various levels of fill level, the evaluation model 31 can learn to use this difference caused by the lens effect of the sample fluid to determine the fluid state.

[0109] The training images T1-T12 can also show sample fluids with different turbidity and / or color intensity. At higher fill levels, the turbidity or color intensity usually appears higher, as shown in training image T4, as opposed to training image T8, which was captured at a low fill level. Additionally or alternatively, with such training images, the evaluation model 31 can learn to use the respective image content (different turbidity levels and / or color intensity) to determine the fill level or another fluid state specified by the annotation A.

[0110] Furthermore, some of the training images T1-T12 can also show the sample container in which condensation water or droplets have collected on the sample container wall or the transparent lid, as shown purely schematically in training image T6. In principle, droplets or condensate can be detected by the shape of the droplets or condensate and by the fact that the background observed through them appears blurred. By using training images that show droplets or training images with different degrees of droplet formation, together with the respective associated annotations A, the evaluation model 31 can learn to evaluate this image content to determine the fluid state 35.

[0111] Contamination, which is present in the form of deposits at the edge, for example, due to the growth of microorganisms in the sample fluid, can be determined on the basis of the brightness or color gradient within the sample image region. In training image T9, unwanted deposits are discernible at the edge of the sample container. In this case, a plurality of training images with contamination and further training images without contamination can be used together with the respective annotations about the presence or absence of contamination. Thus, the evaluation model learns to detect the contamination based on this image content and to indicate it as a fluid state 35.

[0112] Optionally, context data i can also be provided to some or all of the training images T1-T12. All of the training images T1-T12 are also input into the evaluation model 31 in training, so that the evaluation model 31 learns to take the context data i into account when determining the fluid state 35. The context data i can be, for example, information about the sample or the fluid solution in which the sample is located, for example, a buffer solution. For example, if different solutions have different colors or color intensities, the evaluation model 31 can learn, with the aid of the context data, to distinguish between two cases that involve similar color intensities, one showing a color-intensive fluid at a low fill level and the other showing a fluid with a lower color density at a high fill level.

[0113] The context data i can also indicate the use of a specific pH indicator. Depending on the pH indicator, the color of the color change and the pH limit values are different. If the training images and the associated context data cover one or more different pH indicators, the evaluation model can learn to correctly interpret the color information or pH value information emitted by these pH indicators in the overview image.

[0114] The sample carrier type can be indicated as further context data i. The sample carrier type defines the size of the sample container and thus also the expected fill level. An additional learning classification model can be used to determine the sample carrier type from the overview image, so that the determined sample carrier type can subsequently be used as context data in the evaluation model 35.

[0115] Figure 3 A supervised learning process is illustrated. In other variants, a non-supervised learning process without specified annotations A can be implemented. For example, the evaluation model 31 can be trained for anomaly detection by non-supervised learning. All training images used can correspond to fill levels within an acceptable range (or to acceptable fluid states, free from contamination, from excessive condensation or droplet formation). If the trained evaluation model receives a sample image region that deviates from the distribution of the training images used, an anomaly is detected, which is interpreted as an unacceptable fill level / fluid state.

[0116] The mapping of the image data (image portion of the well) to the desired output (fill level or other fluid state) is calculated by the learning evaluation model 31 using features. The features are learned directly from the training images and do not have to be explicitly modeled for each use case.

[0117] For illustrative purposes, reference has been made to Figure 3 The learning process of the individually processed training images T1-T12 has been described. However, multiple training images of the same sample container captured in succession can also be evaluated. Thereby, a progression over time can be taken into account. For example, the time before the sample fluid evaporates can be estimated as a fluid state, in particular the time at which a predetermined portion of the sample fluid has evaporated, or the time at which the fill level drops below a minimum fill level due to evaporation.

[0118] APPENDIX Figure 4

[0119] Figures Figure 4 A flowchart showing a variant embodiment of the inventive method is shown.

[0120] In step S10, at least one overview image of the sample carrier is captured using an overview camera.

[0121] In step S11, a sample image region, for example an image region of the sample container, is located in the overview image.

[0122] In step S12, the evaluation model is used to determine a fluid state for each sample image region. For example, a fill level for each sample container can be determined.

[0123] In step S13, the operation process of the microscope is controlled in dependence on the detected at least one fluid state. For example, a maximum positioning speed of the sample stage can be set in dependence on the determined fill level or fill levels. For this purpose, a table or model can be stored which determines how to derive the maximum allowed positioning speed or acceleration from the determined fill level. In particular, the maximum allowed positioning speed or acceleration can be defined as a function of the highest fill level among the determined fill levels. In a variant, the evaluation model is directly trained to calculate a control of the microscope operation process as a function of the fluid state. In this case, steps S12 and S13 can take place together. Similarly, steps S11 and S12 can also be performed together by the evaluation model if the evaluation model described before and the previous model for determining the sample image regions are jointly combined into a single model.

[0124] The described example embodiments are purely illustrative and identical variants are possible within the scope of the appended claims. While examples are shown in relation to a microtiter plate, other sample carriers with one or more sample containers can also be used. The sample fluid does not necessarily wet the side walls of the sample containers, but can also be present in the form of droplets at the bottom of the sample containers or sample carrier. Further calculation steps can be added between the described calculation steps. For example, the overview image can be cropped, converted to grayscale or processed in a more complex manner, for example by a trained denoising or a model for calculating a high-resolution image, before or after one of the described steps. The described plan view calculation can be omitted as an optional step or only used for the positioning of the sample image regions, but not for the image data input to the evaluation procedure.

[0125] List of reference signs

[0126] 1 microscope

[0127] 2 holder

[0128] 3 objective revolver

[0129] 4A microscope objective

[0130] 4B microscope camera

[0131] 5 sample stage

[0132] 6 sample fluid

[0133] 7 sample carrier

[0134] 7A-7D Sample containers of sample carrier 7

[0135] 8 Sample image area in overview image

[0136] 9 Overview camera

[0137] 9A Field of view of overview camera

[0138] 9B Mirror

[0139] 10 Condenser

[0140] 11 Overview image

[0141] 12 Image portion of overview image 11

[0142] 15 Illumination device

[0143] 20 Computing device

[0144] 21 Overlay of segmentation mask and overview image

[0145] 30 Evaluation procedure

[0146] 31 Evaluation model

[0147] 35 Fluid state / output of evaluation model 30

[0148] 80 Computer program of the invention

[0149] 100 Microscope system of the invention

[0150] A Annotation

[0151] H1-H12 Annotation indicative of fill level / fluid state

[0152] i Context data

[0153] L Loss function

[0154] O Optimization function

[0155] P Calibration parameter

[0156] P1-P9 Model parameters of evaluation model 31

[0157] T Training data

[0158] T1-T12 Training images

[0159] S Segmentation model / localization model

[0160] S1-S8 Method steps of example embodiments of the invention

[0161] S10-S13 Method steps of example embodiments of the invention

Claims

1. A microscope system, comprising Microscope (1), having an overview camera (9) for capturing at least one overview image (11) of a sample carrier (7) designed to receive at least one sample fluid (6); and The computing device (20) is configured to determine at least one sample image region (8) of at least one sample fluid (6) within at least one overview image (11). Its features are, The computing device (20) includes an evaluation program (30) into which each determined sample image region (8) is input and configured to determine the fluid state (35) of the associated sample fluid (6) based on the sample image region (8). The evaluation procedure (30) includes an evaluation model (31) learned using training data (T). The training data (T) includes labeled training images (T1-T12), which are sample image regions (8) of the overview image (11) or overview image (11) that specify the fluid state (H1-H12) in the form of annotation (A). The determined fluid state (35) indicates the fill level associated with the relevant sample fluid (6); and The training data (T) of the evaluation model (31) shows that the sample containers (7A-7D) are filled to different levels, and the filling levels (H1-H12) are represented by annotations (A).

2. A method for evaluating an overview image, characterized in that, include At least one overview image (11) of the received sample carrier (7), which is designed to receive at least one sample fluid (6); and Determine at least one sample image region (8) of at least one sample fluid (6) within at least one overview image (11); Its features are, Each determined sample image region (8) is input into the evaluation procedure (30), wherein the evaluation procedure (30) determines the fluid state (35) of the relevant sample fluid (6) based on the sample image region (8). The evaluation procedure (30) includes an evaluation model (31) learned using training data (T). The training data (T) includes labeled training images (T1-T12), which are sample image regions (8) of the overview image (11) or overview image (11) that specify the fluid state (H1-H12) in the form of annotation (A). The determined fluid state (35) indicates the fill level associated with the relevant sample fluid (6); and The training data (T) of the evaluation model (31) shows that the sample containers (7A-7D) are filled to different levels, and the filling levels (H1-H12) are represented by annotations (A).

3. The method according to claim 2, characterized in that, The sample carrier (7) includes multiple sample containers (7A-7D) for receiving the corresponding sample fluid (6); Among them, multiple sample image regions (8) depicting sample containers (7A-7D) were identified within the same overview image (11); In this process, the fluid state (35) of the relevant sample fluid (6) is determined for each sample image region (8).

4. The method according to claim 2, characterized in that, in, The evaluation procedure (30) includes an evaluation model (31), which is learned through unsupervised learning using training data (T) and is used for anomaly detection; The evaluation model (31) infers the abnormal fluid state (35) as a function of the similarity between the input sample image region (8) and the training data (T).

5. The method according to claim 2, characterized in that, Multiple overview images (11) are captured after the time interval; Among these overview images (11), at least one sample image region (8) of at least one sample fluid (6) is determined; and The evaluation procedure (30) determines the progress of the fill level or another fluid state (35) over time, or the prediction of when the fill level will fall below the minimum fill level is calculated by the evaluation procedure (30).

6. The method according to claim 2, characterized in that, Multiple overview images (11) are captured by overview cameras (9) at different locations, or multiple overview images (11) are captured as the sample stage moves. Among the multiple overview images (11), sample image regions (8) related to the same sample fluid (6) were identified respectively, and Multiple sample image regions (8) associated with the same sample fluid (6) from different overview images (11) are fed together into the evaluation procedure (30) to determine the fluid state (35).

7. The method according to claim 2, characterized in that, The evaluation procedure (30) used to determine the fluid state (35) considers at least one of the following characteristics: - Color intensity in the sample image region (8); - How the image content in the sample image region (8) is distorted or altered in size or position due to the lensing effect caused by the fluid surface of the sample fluid (6); - Whether droplets are identifiable on the sidewall or lid of the relevant sample container (7A-7D) in the sample image area (8); - Whether the color or color distribution within the sample image region (8) deviates from the expected sample color or sample color distribution, based on which the contamination of the sample can be inferred; - Whether there are crystals in the sample image area (8) is evaluated by the procedure (30) as an indicator of the dryness of the sample fluid (6); - The fluid surface or reflection on the fluid surface in the sample image region (8); - Lens effect on the surface of the sample fluid (6), wherein the evaluation procedure (30) evaluates the sample image area based on the degree of distortion or change in size of the background visible through the sample fluid (6) or the size of the sample container (7A-7D) visible through the sample fluid (6).

8. The method according to claim 2, characterized in that, in, The evaluation procedure (30) infers the presence of any sample fluid (6) in the fill level or the corresponding sample container (7A-7D) based on the droplets or condensates detected in the sample image area (8).

9. The method according to claim 2, characterized in that, The movement of the sample stage occurs in conjunction with the capture of multiple overview images (11), wherein at least one sample image region (8) is determined, and wherein the evaluation procedure (30) considers that the movement of the sample stage causes a change in appearance or lensing effect in the sample image region (8) in order to determine the fluid state (35); or In order to determine the fluid state (35), the evaluation procedure (30) takes into account the motion of the sample stage or the inertia of the sample in the surrounding fluid due to the motion of the sample stage.

10. The method according to claim 2, characterized in that, in, To determine the fluid state (35), multiple overview images (11) are captured under different lighting conditions, and the differences between corresponding sample image regions (8) in the multiple overview images (11) are evaluated by an evaluation procedure (30).

11. The method according to claim 2, characterized in that, Multiple overview images (11) are captured consecutively and the evaluation procedure (30) determines the contamination of the fluid state (35) by color changes in the sample image region (8).

12. The method according to claim 2, characterized in that, The localization model (S) learned using the training data determines at least one sample image region (8) in the overview image (11); All sample image regions (8) determined by the localization model (S) are input into the evaluation procedure (30).

13. The method according to claim 2, characterized in that, in, Based on the determined fluid state (35), determine which sample containers (7A-7D) of the sample carrier (7) are located and subjected to further analysis.

14. The method according to claim 2, characterized in that, The microscope (1) includes an upright microscope stand (2) and an immersion objective (4A), wherein an evaluation procedure (30) determines the filling level of at least one sample container (7A-7D) as a fluid state (35), and wherein if the filling level exceeds a maximum value, a warning is output to prevent potential overflow due to the objective (4A) being immersed in the sample container (7A-7D).

15. The method according to claim 2, characterized in that, The maximum positioning speed of the sample stage (5) is set as a function of the determined fill level, wherein, in the case of multiple sample containers (7A-7D) with different fill levels, the maximum positioning speed is set as a function of the determined highest fill level, and The size of the cross-section of one or more sample containers (7A-7D) is determined by the overview image (11), and the maximum positioning speed is also set as a function of the determined cross-section size.

16. A computer program with commands, characterized in that, When the command is executed by the computer, the method described according to any one of claims 2 to 15 will be performed.

Images