Image analysis for sample on sample carrier with grid arrangement

Abstract

Several examples of the disclosure relate to techniques for evaluating a microscope image (305). The microscope image (305) depicts a sample carrier that has a grid arrangement, which in turn defines several areas with predefined dimensions. Techniques are described by which the grid arrangement can be localized (510, 520) in the microscope image, so that a quantitative image analysis (515, 530) of an arrangement of structures on the sample carrier, for example cells, is possible, taking the areas into account. Exemplary sample carriers are hemocytometers for cell counting, such as Neubauer counting chambers.

Description

TECHNICAL AREA

[0001] Several examples of the disclosure relate to image analysis of a microscope image depicting a scene with an arrangement of structures on a sample carrier. In particular, techniques for the automated performance of a quantitative measurement of a property of the arrangement of the structures are disclosed. BACKGROUND

[0002] In microscopy, so-called counting chambers are used for quantitative measurements. These are sample holders that feature a grating arrangement and enable imaging in transmitted light geometry. The resulting microscope images then depict a scene with an arrangement of structures on the grating of the counting chamber. A commonly used grating arrangement is the Neubauer grating.

[0003] A special type of counting chamber is the hemocytometer, which is primarily used in medical and biological laboratories: hemocytometers allow the counting of cells in a suspension.

[0004] The cells are counted manually while examining the areas defined by the grid through a microscope. This requires considerable accuracy and patience, especially with high cell concentrations. SUMMARY

[0005] There is therefore a need for the automation and simplification of cell counting, or more generally for the automation and simplification of image analysis of a microscope image to quantitatively determine a property of an arrangement of structures that is depicted in the microscope image together with a grid arrangement of a sample carrier.

[0006] This task is solved by the features of the independent patent claims. The features of the dependent patent claims define embodiments.

[0007] Techniques for automating quantitative measurements of an arrangement of structures on a sample carrier are disclosed below. In particular, a property of the arrangement of the structures can be quantitatively determined as a function of their positioning in several areas of the sample carrier. These areas (for example, three-dimensional volumes) are defined by surfaces arranged in a grid. For the sake of clarity, these surfaces will be referred to as counting surfaces, although counting is not the only possible function within the quantitative measurement.

[0008] According to various examples, a microscopic image is obtained using a microscope, which depicts the arrangement of the structures (for example, cells, but also other types of structures such as yeast, sperm, microparticles, etc.) as well as the grid arrangement. The grid arrangement defines one or more counting areas. These have a predefined size and thus enable a quantitative determination of a property of the arrangement of the structures depending on the positioning of the structures within the multiple counting areas.

[0009] The grid arrangement is located in the microscope image. Subsequently, based on the localization of the grid arrangement, an image analysis of the microscope image can be performed to determine a property of the arrangement of the structures: the quantitative dependence of the structures' positioning in the multiple surfaces. In particular, the structures (possibly differentiated by structure type) can be counted for each of the counting surfaces.

[0010] The localization of the grating array for locating the counting surfaces can be achieved using a machine-learned model. In general, using a machine-learned model for grating array localization is helpful because the microscope images used can exhibit different appearances (color, contrast, brightness) depending on the illumination, objective, background, suspension, etc. Image noise can also vary from one microscope image to another. Therefore, it is beneficial to use a machine-learned model that provides robust grating array localization without a strong dependence on the contrast used. In particular, the machine-learned model can provide robust grating array localization compared to conventional, non-machine-learned models that, for example, attempt to fit various candidate grating arrays to the image pixels.Furthermore, the grid arrangements for sample carriers from different manufacturers vary considerably. Different grid arrangements exhibit different numbers and configurations of grid lines. By using a machine-learned model, robust recognition of the counting surfaces can be achieved even for many different grid arrangements. Moreover, it is possible to retrain the machine-learned model, for example, when grid arrangements of a new type become available. The machine-learned model's capabilities can thus be expanded by training it with new training data. Appropriate training prevents a reduction in recognition performance for previously trained grid arrangements.

[0011] As an example, a computer-implemented method comprises the following: A microscope image is obtained, depicting an arrangement of structures on a sample carrier. The sample carrier has a grid arrangement that defines several surfaces with predefined dimensions. The grid arrangement is located in the microscope image using a machine-learned model. Based on the localization of the grid arrangement, an image analysis of the microscope image is performed to quantitatively determine a property of the arrangement of the structures as a function of their positioning within the multiple surfaces.

[0012] The structures can be cells. The grid arrangement can be a hemocytometer grid arrangement. The property can include at least one cell state estimation, cell count, or cell morphology.

[0013] An electronic data processing device is set up to execute such a computer-implemented procedure.

[0014] The features set out above and those described below can be used not only in the corresponding explicitly set out combinations, but also in further combinations or in isolation, without leaving the scope of protection of the present invention. BRIEF DESCRIPTION OF THE FIGURES Fig. Figure 1 is a flowchart of an example procedure. Fig. Figure 2 is a flowchart of an example procedure. Fig. Figure 3 shows a microscope image depicting an arrangement of cells on a sample carrier with a grid arrangement. Fig. 4 shows the microscope image from Fig. 3, where grid lines of the grid arrangement are traced. Fig. Figure 5 illustrates a density map in which grid vertices of the grid arrangement are shown. Fig. 3 are located. Fig. Figure 6 illustrates a density map in which grid points of the grid arrangement are shown. Fig. 3 are located. Fig. Figure 7 schematically illustrates a mask image of the grid arrangement from Fig. 3. Fig. Figure 8 is a flowchart of an example procedure. Fig. Figure 9 shows an example grid arrangement. Fig. Figure 10 schematically illustrates a data processing pipeline according to various examples. Fig. Figure 11 schematically illustrates an electronic data processing device according to various examples. DETAILED DESCRIPTION OF EXAMPLES

[0015] The properties, features and advantages of this invention described above, as well as the manner in which they are achieved, will become clearer and more easily understood in connection with the following description of the exemplary embodiments, which are explained in more detail in conjunction with the drawings.

[0016] The present invention is explained in more detail below with reference to preferred embodiments and the drawings. In the figures, identical reference numerals denote identical or similar elements. The figures are schematic representations of various embodiments of the invention. Elements depicted in the figures are not necessarily shown to scale. Rather, the various elements depicted in the figures are represented in such a way that their function and general purpose are understandable to a person skilled in the art. Connections and couplings between functional units and elements shown in the figures can also be implemented as indirect connections or couplings. A connection or coupling can be implemented as a wired or wireless connection. Functional units can be implemented as hardware, software, or a combination of hardware and software.

[0017] The following describes various techniques for the automated performance of quantitative measurements using a microscope image, where the microscope image depicts an arrangement of structures on a sample carrier. The sample carrier has a grid arrangement that defines several counting surfaces with predefined dimensions. These counting surfaces enable the quantitative measurement. The sample carrier can, for example, have a certain depth, so that the counting surfaces, together with this depth, define three-dimensional volumes.

[0018] In various scenarios described herein, the grid arrangement in the microscope image is localized using a machine-learned model. This machine-learned model could, for example, be a deep neural network.

[0019] The machine-learned model can, for example, provide an image-to-image transformation, where the output image marks specific reference points of the grid arrangement. Alternatively, the machine-learned model could solve a localization task, providing coordinates in list form for the corresponding reference points.

[0020] Using a machine-learned model enables particularly robust localization of the grating arrangement within the microscope image. Specifically, the machine-learned model can be trained to process microscope images with varying contrasts, brightness levels, structure densities, different brightness ratios between structures and the background, image artifacts or aberrations, impurities, and so on. Furthermore, the machine-learned model can be trained to robustly handle different types of sample supports that define different grating arrangements.

[0021] The structures can be, for example, cells. The cells can be suspended on a hemocytometer sample holder. In the various examples described herein, cell state assessment, cell counting, or cell morphology determination can be performed on cells mounted on a hemocytometer grid.

[0022] In principle, the techniques described herein can also be used for other types of structures, such as yeast, sperm, algae, or microplastics. For all such structures, robust localization of the grid arrangement can be helpful in quantitatively determining a property of the structure arrangement as a function of the structure's positioning relative to the counting surfaces. For the sake of simplicity, however, the following discussion primarily focuses on the application of these techniques to the automated quantitative measurement of cells. The described techniques and concepts can, however, also be applied to other types of structures.

[0023] Fig. Figure 1 is a flowchart of an example procedure. The procedure is from Fig. 1 can be executed, at least partially, using computer implementation. This means that a processor can load and execute program code from memory. When the processor executes the program code, this causes the processor to subsequently execute at least some boxes of the procedure. Fig. 1 executes.

[0024] The flowchart of Fig. Section 1 describes the automation of a quantitative measurement for an arrangement of structures. In principle, a wide variety of arrangements of structures can be measured, for example, yeast, sperm, algae, microplastics, etc. However, the following section discusses... Fig. 1. For the sake of simplicity, described for an arrangement of cells.

[0025] A sample is loaded into box 905. The sample is then placed, for example, in the optical path of a light microscope. This can be automated, for instance, by having a robotic arm pick up the sample and position it in the optical path.

[0026] The sample consists of an arrangement of cells (e.g., in suspension) on a sample carrier. The sample carrier has a grid arrangement. In particular, the sample carrier may be a hemocytometer counting chamber.

[0027] The grid arrangement defines several counting areas with a predefined extent.

[0028] In Box 906, the type of sample carrier can optionally be determined first. The sample carrier type can be determined in various ways.

[0029] For example, machine-readable information, such as type information, could be read optically or electrically. The type could be determined based on user input. Alternatively, image analysis could be performed to classify the sample carrier. Such an image analysis could be carried out on an overview image at a very low magnification. Generally, image analysis can be performed based on a calibration image.

[0030] Different types of sample holders have different grid arrangements and correspondingly different counting areas. A common type is the Neubauer counting chamber. Besides the Neubauer counting chamber, there are other counting chambers such as the Burker-Türk counting chamber, Fuchs-Rosenthal counting chamber, Thoma counting chamber, and the Malassez counting chamber. Another example is the Petroff-Hausser counting chamber.

[0031] The classification of the counting chamber manufacturer and / or model can provide important contextual information to determine the exact geometry (grating position, number and / or size of the counting areas, etc.) and any special features for quantitative measurement (e.g., counting). A known grating geometry, in turn, is an important basis for all further grating-based processing steps, such as color calibration, focus determination, position determination, etc. Therefore, in Box 906, a specification for the grating arrangement can be defined based on a sample carrier type.

[0032] If an image-based classification of the sample carrier type is used to determine the required grid arrangement, this can be technically implemented with a machine-learned classification model. Such a classification task typically exhibits a comparatively low level of complexity. In particular, it is possible to train such a machine-learned classification model with a large number of training data points because the corresponding training data do not require localization, etc., but only a class label as the basic truth. In other words, this means that a large amount of suitable training data can be generated relatively quickly, thus enabling robust classification of the sample carrier type using the machine-learned model.

[0033] Box 905 is generally optional; in some variants, the microscope image was already captured at an earlier time, so Box 905 no longer needs to be executed.

[0034] Autofocus can be optionally performed in Box 910. Several examples are based on the understanding that accurately determining the focus position is particularly important when using staining markers to highlight specific structures within cells. For instance, trypan blue can be used to stain cells when distinguishing between living and dead cells for quantitative measurement. If the focus is out of focus, the case where the cell membrane is stained blue can be confused with the case where the dye has penetrated the cell's core. It is precisely this distinction that allows for differentiation between living and dead cells.

[0035] There are several ways to perform autofocus in Box 910. For example, autofocus can be based on the cells themselves. Specifically, the halo, sharpness, and / or cell size can be determined to guide autofocus. Generally, such cell characteristics allow for optimization of image quality, particularly sharpness, and thus improve the accuracy of cell analysis.

[0036] Alternatively or additionally, the microscope could be controlled to perform autofocus based on an image of the grating arrangement in corresponding calibration images. For example, the width of the grating lines could be measured. This measured width could then be compared to a target value for the grating line width, which is known based on the grating arrangement specification from Box 906. Alternatively or additionally, the sharpness of the lines could be measured. Furthermore, a defined offset could be taken into account, since the cell equator is typically located above the grating lines.

[0037] Within the Box 910, a user interface can be controlled to output an indicator that reflects autofocus quality. This allows for the real-time display of a focus score, providing the user with immediate feedback on the focus setting. Furthermore, this can support documentation for quality assurance purposes.

[0038] In the event of insufficient focus quality, the continuation of the various boxes in Fig. 1. This should be prevented in order to avoid so many analyses due to blurry images.

[0039] Box 910 is generally optional; in some variants, the microscope image was already captured at an earlier time, so Box 910 no longer needs to be executed.

[0040] Optional color calibration can be performed in Box 915. Color calibration ensures consistent color reproduction. This is particularly helpful when using staining markers with specific colors for quantitative analysis. An example, as discussed above, would be trypan blue for distinguishing between living and dead cells. Pre-calibration in Box 915 can be performed based on the color of the grid arrangement in a calibration microscope image. For example, semantic segmentation of the grid lines can be carried out using a machine-learned model, followed by a color matching of the grid line color against a predefined color (e.g., based on prior knowledge of the sample carrier in Box 906).

[0041] Color calibration can generally be performed directly on the microscope. It would also be conceivable to perform a preliminary calibration during digital post-processing, meaning that the color space of a captured microscope image can then be digitally transformed.

[0042] Box 915 is generally optional; in some versions, the microscope image was already acquired at an earlier stage, so Box 915 no longer needs to be executed. Even if the microscope image was acquired earlier, color calibration could still be performed through digital post-processing.

[0043] Box 920 will receive a microscope image. Box 920 can include controlling the microscope to capture the corresponding image data. Alternatively, the user can load an existing, previously saved image.

[0044] The user can actively trigger the acquisition of a microscope image, for example by pressing a button. The microscope is then controlled accordingly. The image can also be captured automatically. The capture time can be predefined, for example at regular intervals during an experiment, or adaptive, triggered by a specific event.

[0045] The microscope image can be acquired using transmitted light imaging without fluorescence contrast. Alternatively, the image can be acquired using fluorescence contrast. For example, nuclear staining with DAPI, Hoechst, or other agents can be used.

[0046] The microscope image can exhibit specific or non-specific contrast. Specific contrast highlights particular cell structures in a specific way. For example, a specific fluorescence contrast can be used, or a specific contrast without a fluorescent label. Non-specific contrast would be, for example, phase contrast, brightfield contrast, or autofluorescence. Non-specific contrast could be phase-like contrast. Phase-like contrast can be, for example, phase contrast. Examples include Zernike phase contrast and Normarski phase contrast. Here, special optical elements are used in the light path, such as a phase ring in the objective and an annular diaphragm in the condenser lens. In this way, interference between background and object light can be visualized. By using phase contrast, the image contrast can be increased.This means that the cell structures are particularly well visible. Cells are phase objects that cause no or no significant reduction in the amplitude of light as it passes through the cell sample, so phase contrast is preferred for visualizing the phase shift. However, digital phase contrast can also be used as a phase-like contrast. Here, several images are acquired and then computationally combined into a single phase-contrast image. Therefore, such techniques can be referred to as digital phase contrast. The phase contrast is obtained through digital post-processing of the acquired intensity images. Examples include the Transport of Intensity Equation (TIE) and differential phase contrast (DPC). TIE is described in: Streibl, Norbert. “Phase imaging by the transport equation of intensity.” Optics communications 49.1 (1984): 6-10. DPC is described in: Mehta, Shalin B., and Colin JR Sheppard.“Quantitative phase-gradient imaging at high resolution with asymmetric illumination-based differential phase contrast.” Optics letters 34.13 (2009): 1924-1926. To acquire a TIE dataset, the sample is moved along the optical axis (z-direction), i.e., axially shifted, and a so-called z-stack, consisting of at least two images, is acquired. The data are then processed to obtain a phase-contrast image. This involves solving a diffusion-type partial differential equation. In DPC, the sample is illuminated from at least two different directions (oblique illumination) while remaining at a fixed z-position. Possible sources for oblique illumination include all types of segmented sources; examples are segmented diodes, LED arrays, digital micromirror devices (DMDs), liquid crystal displays (LCDs or SLMs), or variable condenser apertures.The acquired data are then converted into a phase contrast image by solving a deconvolution problem. Using digital phase contrast (as opposed to hardware-based phase contrast) has the advantage that there is no need for the time-consuming process of inserting or removing objects from the light path when acquiring the digital phase contrast. Instead, the illumination can be varied selectively, for example, using a switchable LED array positioned in the illumination pupil plane. This can be done quickly and easily.

[0047] Non-specific contrast types include "label-free" contrasts such as phase contrast, DIC contrast, or TIE contrast, which allow observation of cells without specific labels. Alternatively, stains such as H&E can be used to specifically label certain cell structures. Another option is the use of fluorescent stains, such as DAPI, to fluorescently label specific cell components. Autofluorescence can also be used to localize cells without additional labels.

[0048] In principle, the microscope image could also include several channels (multi-channel microscope image) that have different contrasts (for example, from the set of contrasts discussed above).

[0049] The microscope image can depict the entire counting chamber or only a portion of it. This largely depends on the objective lens used and its magnification. For example, objectives with 4x, 10x, or 20x magnification can be used. A compromise between a large field of view (FoV) and sufficient detail in cell morphology is important. This means that subsequent models are trained at a magnification that allows them to reliably perform the desired differentiation, such as live / dead, while simultaneously maximizing the field of view.

[0050] In Box 935, the microscope image from Box 920 can optionally be rescaled, meaning its size is changed. For example, rescaling can be performed so that the microscope image subsequently depicts the cells or other structures, such as lattice structures of the grid arrangement, at a specific magnification. In other words, the scaling can be done so that the cells in the microscope image have a specific size. This size can be predefined. In particular, the size can correspond to the size of cells in reference microscope images used for training one or more machine-learned models that are subsequently used for image analysis. This reduces the complexity of such machine-learned models, as they only expect cells of a specific image size.The training effort for such machine-learned models can be reduced. The training data does not need to include cells with different image scales, but can be limited to cells with the specified image size.

[0051] Scaling in Box 935 can be done manually, for example. However, it would also be conceivable to use a machine-learned model for scaling in Box 935. For instance, a machine-learned model could be used that performs a picture-to-picture transformation, i.e., outputs the rescaled image. Alternatively, a machine-learned model could be used that outputs a scaling factor, i.e., performs a picture-to-scalar transformation. An exemplary rescaling technique is described in principle in EP 4 053 805 A1. The corresponding techniques are included therein by cross-reference.

[0052] The various techniques described herein can particularly exploit the fact that cells in a suspension are typically round. This prior knowledge about the shape of the cells can be used during rescaling. Furthermore, it is possible to perform rescaling of the microscope image based on the size of the grating arrangement. For example, based on the specification for the grating arrangement in Box 906, prior knowledge about the size of certain structures of the grating arrangement (especially a side length of the counting faces) may be available. It would therefore be possible to rescale the microscope image so that certain grating structures of the grating arrangement subsequently have a specific size.In Box 935, one or more lattice structures can be detected, and then the image size of these lattice structures can be compared with a target size, so that a corresponding rescaling of the microscope image can subsequently be carried out.

[0053] Box 935 can optionally include automatic detection of inappropriate or implausible magnification. For example, it can detect that the size of the cells in the microscope image and the objective magnification do not match (i.e., that the cells are too large or too small relative to the current objective magnification). The objective magnification can be read from the microscope image metadata or obtained as control data from the microscope.

[0054] Box 940 optionally allows for a preliminary analysis of the microscope image from Box 920. This preliminary analysis does not yet involve the actual quantitative measurement; rather, it checks whether certain properties are met in the microscope image, so that the actual quantitative measurement can be carried out later or can achieve a certain level of accuracy.

[0055] It is possible to detect problems in the input data. Examples include impurities or cells at different focus levels. To identify these and other problems, machine learning can be used, particularly classification, detection, or segmentation models trained on known impurities. Alternatively, a so-called "one-class classification" can be used to find objects that are not "known." Anomaly detection can also be performed.

[0056] If problems are detected in the microscope image, various consequences can occur. For example, a warning message can be issued to the user, or a notification can be given indicating incorrect filling of the sample holder or counting chamber. A note can also be created in the metadata. In some cases, it may be necessary to modify the workflow, for example, by excluding certain counting areas from the subsequent quantitative measurement and / or using additional counting areas to improve the statistical accuracy. In general, a subsequent image analysis within the framework of the quantitative measurement can therefore be performed depending on a result of the preliminary analysis from Box 940.

[0057] Box 942 optionally allows for the positioning of the cell arrangement. In other words, the scene—as depicted in the microscope image—is registered in a reference coordinate system. This positioning, or registration of the scene, provides navigation assistance for the user and prevents the duplicate evaluation of the same sample areas by recognizing the absolute location or identifying previously visited sample areas. Optionally, the images can also be stitched together to form a single image based on the determined image positions.

[0058] Position determination can be achieved in various ways. For example, the grid arrangement of the sample holder can be localized. This allows the determination of which parts of the grid arrangement, such as which counting surfaces, are visible in the microscope image. This can be particularly advantageous at high magnifications when only a portion of the grid arrangement is depicted in the microscope image. Technically, this is possible, for example, by recognizing grid points (e.g., using a neural network or filtering) and then comparing them to a "grid database" (grid arrangement specifications, e.g., from Box 906), for example, using an "Iterative Closest Point" algorithm.

[0059] Alternatively, a pattern of the arrangement of already analyzed cells can be used to perform the registration. Technically, this can be done by interpreting the cells as a point cloud. For each image, the current position in the point cloud is continuously estimated and the cloud is extended as needed.

[0060] Another way to determine the position is to use an overview camera that observes the rehearsal room.

[0061] In Box 945, specific areas of the scene can optionally be masked. For example, the user could use a graphical user interface to mask certain areas in the microscope image that should be excluded from subsequent quantitative measurements and / or where a specific image analysis algorithm should be applied. Alternatively or additionally, automated masking can also be performed.

[0062] For example, cell clumps or other areas of the sample that are unsuitable for quantitative measurement due to their structure or that require a specific image analysis algorithm can be identified beforehand and then appropriately masked during image analysis (i.e., excluded from image analysis during quantitative measurement). This masking can, for example, be based on a result from Box 940.

[0063] For example, a model, such as a machine-learned model, could be used to detect and localize cell clumps. Cell clumps are accumulations of cells in which individual cells are no longer isolated in a suspension. Because the distance between adjacent cells within cell clumps is much smaller than outside of them, and because cells within clumps are no longer easily distinguishable from one another visually, different methods or models are typically required to perform a quantitative measurement task, such as cell counting, inside and outside of cell clumps. For example, it would be conceivable to count cells outside of cell clumps and then, based on prior knowledge of cell size and a measured area of ​​the cell clump, estimate how many cells are inside the clump.In other words, this eliminates the need to count cells individually within a cell clump. For example, the area of ​​a cell clump can be determined using semantic segmentation or instance segmentation, and this area can then be compared to the area of ​​individual cells. It is also possible to distinguish between dead and living cells or cell debris. As a concrete example: Consider a cell clump with an area A. K found, whereby this area is approximately five times larger than area A Z an individual cell in the scene. Then, for the purpose of performing a cell count, it can be assumed that there are five cells in the cell cluster.

[0064] In Box 950, the grid arrangement is localized. A machine-learned model is used for this purpose. Localizing the grid arrangement means, in particular, locating the counting areas defined by the grid arrangement. The counting areas could be located in the microscope image.

[0065] Then, in Box 960, the image analysis for the actual quantitative measurement takes place. This can include, for example, the localization of individual cells, possibly with regard to their respective state (e.g., alive or dead). Cells can be counted in the various localized counting areas. Thus, the image analysis in Box 960 takes into account the counting areas localized in Box 950. Optionally, one or more mask areas from Box 945 can also be considered, for example, by selecting the respective image analysis algorithm depending on the mask area (for example, previous examples in connection with Box 945 were described where the counting of cells inside and outside a mask area that marks a cell cluster is performed using different algorithms).

[0066] In Box 960, for example, a density map can be created that marks all cells. Multiple density maps can be created, one for each cell type. For example, a first density map can be created for all living cells and a second density map for all dead cells. Corresponding techniques are described, for example, in DE 10 2021 125 538 A1.

[0067] A density map can therefore exhibit a contrast that is indicative of the density or number of cells at a specific pixel. For example, each targeted cell can be arranged in a "Gaussian bell curve" with a contrast differentiated from the background at the cell center.

[0068] Instead of such an image-to-image transformation, which creates one or more density maps, it would also be conceivable to use a detection model. For example, a list of coordinates could be output in the microscope image, with the entries in the list locating the different cells. Additionally, a class label could be output for each cell, for example, to distinguish between living and dead cells.

[0069] Alternatively, segmentation models could be used. For example, a semantic instance segmentation mask could be created. This would allow, for instance, each pixel of the microscopy image to be assigned to different classes (such as "background," "living cell," and "dead cells"). The individual cells could then be separated, for example, using a non-maximum suppression (NMS) technique.

[0070] Image analysis in Box 960 can be particularly enhanced by the use of suitable dye markers. For example, special dyes that mark dead cells can be used. Trypan blue, visible in bright-field imaging, is one such example. Fluorescence markers can also be used to distinguish living from dead cells. This is especially helpful when analyzing adherent cells.

[0071] In principle, the specific implementation of image analysis in Box 960 is variable and not crucial for the techniques described herein. In particular, various image analysis techniques known from the prior art can be combined with the techniques described herein for localizing the grid arrangement (Box 950). The various techniques described herein depend on the combination of grid arrangement localization for determining the counting areas and the execution of image analysis to enable quantitative measurement, for example, cell counting in a defined suspension area.

[0072] To enable this correlation between counting areas and the properties of the cell arrangement, the image analysis can initially be performed on the entire microscope image. Specifically, the image analysis can be carried out without differentiating between the various counting areas. Subsequently, the results of the image analysis can be broken down to the individual counting areas. This means, for example, that corresponding partial results of the evaluation can be assigned to the different counting areas. However, it would also be possible to perform the image analysis multiple times, each time for parts of the microscope image that correspond to the localized counting areas. This means that several sections of the microscope image corresponding to the different counting areas can be determined beforehand, and then the image analysis is performed separately for each of these sections.

[0073] Based on such techniques, Box 960 can generate statistics on cell arrangement properties (e.g., cell density, cell count, etc.) as a function of cell positioning across multiple counting areas. Such statistics are particularly useful in quantitative measurement to gain an overview of certain macroscopic properties of the arrangement, such as the cell suspension. Furthermore, the results can be used to determine specific information, such as the ratio of live to dead cells or to identify problems with the cell culture and its environmental parameters. Dilution factors can also be suggested to determine the correct amount of medium for transferring cells to other containers.

[0074] In certain situations, specific areas, i.e., parts of counting areas or entire counting areas, can be omitted to calculate the statistics. This can be specified, for example, by corresponding mask areas from Box 945.

[0075] Box 962 can optionally perform post-processing of the output from Box 960. During image analysis in Box 960, contradictory predictions may occur. An example of this is when a cell is classified as both alive and dead. This could indicate problems with the input data and / or the model.

[0076] If such conflicting predictions occur frequently, it may be necessary to discard the corresponding evaluation or issue a warning to the user. In some cases, it may also be necessary to adjust the workflow, for example, by using a different counter for the evaluation.

[0077] Box 965 outputs the results of the image analysis to the user. Specifically, a graphical user interface can be used to display the results of the image analysis. For example, if statistics are generated, these can be displayed via the user interface.

[0078] The results of the image analysis can be presented in various formats to provide the user with a comprehensive overview of the quantitative measurement. For example, the number of live cells can be displayed, as this is the most important indicator of the concentration of reproducible cells. Optionally, the number of dead cells can also be displayed if desired. For instance, a corresponding indicator, indicative of the proportion of live cells, could be overlaid on the microscope image for the different counting areas. A color code could be used to indicate the different counting areas, with green signifying few dead cells and red signifying many dead cells. Furthermore, the ratio of live to dead cells or indications of problems with the cell culture and its environmental parameters can be displayed.Dilution factors can also be suggested to determine the correct amount of medium for transfer to other containers.

[0079] The results from multiple counting surfaces can also be aggregated, for example by counting several surfaces one after the other. This aggregated information can then be output to the user.

[0080] The preceding was discussed in connection with Fig. 1. A technique is described to enable automated quantitative measurement of an array of cells. The techniques described herein are based on the combination of localization of a hemocytometer grid array (Box 950) with image analysis (Box 960). Further details for Box 950 are explained below.

[0081] There are different options for implementing the localization of the grid arrangement in Box 950. An example of a two-stage variant is shown in Fig. 2 shown.

[0082] Fig. Figure 2 is a flowchart of a procedure that represents an exemplary implementation variant for Box 950. Fig. 1 represents.

[0083] In the two-stage version from Fig. First, in box 951, the grid points of the grid arrangement are located. Then, in box 952, the counting surfaces are located.

[0084] In Box 951, a machine-learned model can be used in particular. The machine-learned model is used to locate the grid points.

[0085] The localization of the grid points can be achieved in various ways. For example, a density map can be generated. In other words, the machine-learned model provides a kind of mask for the microscope image, in which the grid points are plotted with high or full contrast. To do this, the machine-learned model can perform an image-to-image transformation that maps the microscope image onto the density map. Alternatively, the machine-learned model could output a list of position coordinates (e.g., the xy position in units of image pixels from the microscope image), where the different list entries mark the different grid points, such as their centers.

[0086] If one or more grid points (for example, grid vertices) are obtained by the machine-learned model, the corresponding positions of the grid points can be further refined locally. The localization can be refined, particularly with super-resolution, that is, with sub-pixel resolution greater than the resolution of the microscope image. This can be achieved, for example, by local fitting (line, Gaussian curve, etc.) to the image data of the microscope image or, for example, by projections of its image axes.

[0087] The grid arrangements of typical hemocytometers generally feature grid points of several types. A first type of grid point defines the corners of the counting surfaces (grid vertices). A second type of grid point does not define any corners of the counting surfaces. Fig. Figure 3, for example, shows a microscope image 305, which depicts two complete counting surfaces and seven truncated counting surfaces. Living and dead cells are also visible, namely the light and dark spots. The two complete counting surfaces each have 16 quadrants. The lower left corner of each of the two counting surfaces is in Fig. 3 marked with an arrow. In Fig. Figure 4 shows microscope image 305 with improved contrast: the grid lines are traced. Fig. Figure 5 shows an exemplary density map 315 for the microscope image 305, which only marks the grid vertices of the counting surfaces. In contrast, in Fig. Figure 6 shows a density map 316 which marks all grid points (regardless of whether these are grid vertices or not).

[0088] It is possible that the counting surfaces in Box 952 are only located after discrimination between grid vertices and other grid points has already taken place. This typically reduces the complexity of the task in Box 952 because the possible arrangements of counting surfaces are reduced by eliminating grid points that do not bound any counting surfaces. However, it would also be conceivable that the counting surfaces in Box 952 are located without prior discrimination between grid vertices and other grid points.

[0089] To locate the counting surfaces in Box 952, discrimination between grid vertices and other grid points is helpful. This can be done in various ways. In one approach, the machine-learned model could locate all grid points of the grid arrangement. Then, by analyzing the neighborhood of a grid point, it could be determined whether it is a grid vertex or not. This means that the machine-learned model would first find all intersections of grid lines in the grid arrangement, regardless of whether they are vertices of counting surfaces or not. Subsequently, some grid points could be discarded. Alternatively, the machine-learned model could locate not all grid points, but only grid points of a specific type.In particular, the machine-learned model can be trained to specifically locate grid vertices.

[0090] Typically, the receptive field for a machine-learned model that selectively localizes grid vertices must be larger than the receptive field of a machine-learned model that localizes all grid points (regardless of whether they are grid vertices). This is because a relatively large neighborhood area must be considered to discriminate between grid vertices and other grid points. Such a large receptive field can be achieved, for example, by a particularly deep neural network with many layers. Alternatively or additionally, the microscope image could be reduced in size before being fed into the deep neural network. On the other hand, the task of localizing grid points is comparatively structurally uncomplicated, so the deep neural network can be relatively lean, meaning it can use fewer channels per layer.

[0091] To locate lattice points, it can also be helpful to use prior knowledge about the type of lattice arrangement. For example, relevant prior knowledge could be determined in Box 906. This is based on the understanding that different types of sample carriers use different types of lattice arrangements. For example, the different lattice arrangements may be simple, double, or (as in Fig. 3 or Fig. (4 for microscope image 305) even shows triple grid lines. Depending on the type of sample holder, closed or open counting surfaces can be used. The number of sub-counting surfaces (that is, in Fig. 3 the 16 quadrants) can also vary. It is evident from the above that the number and relative arrangement of grid points vary considerably. Therefore, the accuracy of the machine-learned model can be improved if this localization is based on prior knowledge of the sample carrier type. For example, different machine-learned models could be available for different types of sample carriers or different grid arrangements. The appropriate machine-learned model could then be selected depending on the sample carrier type. Alternatively, an indicator of the sample carrier type could be passed to a corresponding machine-learned model, but the same machine-learned model could be used to process microscope images depicting different types of sample carriers. For example, it would be conceivable that a mask image 317 of the grid arrangement (compare Fig. 7) is transferred to the machine-learned model, for example as another channel concatenated with the associated microscope image 305.

[0092] Referring again to Fig. 2: Once the grid points or even the grid vertices have been located, it is then necessary to determine the counting areas. This is done in Box 925.

[0093] There are different implementation variants for Box 925. One example is the RANSAC algorithm (random sample consensus). The RANSAC algorithm is a method for robustly estimating parameters in the presence of outliers. In the context of localizing the counting surfaces, the RANSAC algorithm can be used to estimate the parameters of a geometric transformation between the localized grid points and a predefined grid arrangement (in which the counting surfaces are defined). The predefined grid arrangement can be determined, for example, based on prior knowledge about the type of sample carrier from Box 906. The RANSAC algorithm operates iteratively and consists of several steps. In each step, a subset of grid vertices is selected, and the parameters of a geometric transformation between these grid points and the predefined grid arrangement are estimated.The parameters are then used to estimate the positions of the remaining grid points. The RANSAC algorithm is robust against outliers because it only requires a subset of grid points to estimate the parameters of a geometric transformation. This means that the algorithm still works even if some grid points were incorrectly located or if there are outliers in the data. In particular, the RANSAC algorithm can locate the counting surfaces even if no prior discrimination between grid vertices and other grid points has occurred. However, the computational effort may then be comparatively higher because more iterations are required to reach convergence.

[0094] In Box 952, a parameterized model can also be fitted. During iterative fitting, a parameterized model of the lattice arrangement (for example, determined based on the specimen holder type from Box 906, which defines the specified lattice arrangement) can be adapted to the two-dimensional arrangement of the lattice vertices until a minimal deviation between the specified and the observed lattice vertices is seen. The fitting can, for example, perform rotations or scaling, but no further compressions or strains.

[0095] Another implementation variant for Box 952 is in Fig. 8 shown. Fig. Figure 8 is a flowchart of a procedure that represents an exemplary implementation variant for Box 952. Fig. 2 represents.

[0096] Box 970 contains a density map (see density map 315 or density map 316 in Fig. 5 or Fig. 6) evaluated. In this step, all points marking the grid vertices are detected based on a specified predefined value. However, this step can be omitted if a list of coordinates is already available, for example, as output from the machine-learned model.

[0097] Box 975: This checks whether the number of detected grid vertices is within an acceptable range (e.g., at least 4 and at most 256). If not, an error can be raised or handled accordingly. Only if the number of detected grid points is within a specific predefined range is Box 980 then executed.

[0098] Box 980: Here, the neighborhood relationships of the various grid vertices are determined. For example, a distance matrix can be calculated. The distance matrix can, for instance, specify the pairwise Euclidean distances between all detected grid vertices.

[0099] Box 985: Potential squares are identified. This is done by analyzing the neighborhood relationship from Box 980. In this example, it is assumed that counting areas must always be square. In principle, other types of shapes can also be considered. For each combination of four grid points, the pairwise distances are calculated and sorted. The four smallest distances (potential side lengths) are normalized by the average of the first four (smallest) distances. It is checked whether the four smallest distances are similar using a standard deviation threshold. It is checked whether the two largest distances (potential diagonals) are similar using a different standard deviation threshold. The validity of the quadrant is validated by comparing the ratio of the mean diagonal to the mean side length against the expected ratio for a square.If the set of points passes all checks, the points are rearranged to maintain geometric proximity within the respective counting surface. More generally, in Box 985, several candidate counting surfaces can be found, each bounded by a predefined number of localized grid points. In the example above, the square counting surfaces are each bounded by four grid points. The candidate counting surfaces found in this way can then be compared with specifications for shape (square shape in the example above) and / or relative positioning (e.g., non-overlapping counting surfaces) to filter out the actual counting surfaces from the set of candidate counting surfaces.

[0100] Box 990: The identified candidate counting surfaces are sorted. This is done, for example, based on their position in the microscope image. The center of the microscope image is calculated, and the candidate counting surfaces are ordered according to their distance from the image center. By sorting the candidate counting surfaces, certain candidate counting surfaces can be prioritized over others. Box 990 can, for example, correspond to a selection of specific counting surfaces from the set of candidate counting surfaces, so that the quantitative measurement is subsequently performed in these selected counting surfaces. Instead of such a selection (for example, based on the position in the microscope image), it would also be conceivable for the user to perform a manual selection or sorting of the candidate counting surfaces.

[0101] Box 995: Output of the found candidate counting surfaces. The list of quadrants is returned (for example, up to a specific predefined list position from Box 990), where each square counting surface is represented by its four corners.

[0102] In the reconstruction of a square grid surface from the set of localized grid vertices, in the variant of Fig. 8. The following constraints are also taken into account: Counting surfaces consist of four grid vertices and are square. If the magnification and pixel size are known, the known size of a counting chamber (typically 1 mm) can be used to calculate the counting area. 2Certain combinations of vertices can be excluded due to unsuitable distances. If no information on magnification or pixel size is available, the grating / cell size can be estimated from the microscope image and then rescaled in front of the microscope image. This can be done in a further filter step, for example after Box 990.

[0103] Fig. Figure 9 shows an example of a grid arrangement 200 for a Neubauer hemocytometer. Different counting surfaces 221 and 222 are shown. Depending on the application, for example, the large, outer counting surfaces 221 or the small, inner counting surfaces 222 can be used. It is possible that a specific type of counting surface is located based on a user specification. This could be achieved, for example, by corresponding specifications in Box 985 of the procedure in Fig. 8. Corresponding specifications can also be taken into account with a RANSAC algorithm.

[0104] Fig. Figure 10 schematically illustrates a data processing pipeline 599 according to various examples. For example, the data processing pipeline 599 can include at least some boxes of the procedure from Fig. 1. Implement.

[0105] The microscope image 305 is processed in two separate algorithms or models 510 and 515. Model 510 can, for example, be a deep neural network (or another machine-learned model) which generates a density map 315, as already described above in connection with Fig. As discussed in section 5, the density map 315 is provided in the example shown, which only locates grid vertices of the grid arrangement. However, machine-learned models can also be used that locate all grid points of a grid arrangement. Instead of an image-to-image transformation, it would also be conceivable to output a list of position coordinates. Corresponding techniques were discussed previously in connection with Fig. 2: Box 951 or Fig. 1: Box 950 discussed.

[0106] Model 515 provides a further density map 516 that, for example, locates all living cells, all dead cells, or all cells regardless of whether they are living or dead. This is just one example, and various evaluation techniques are conceivable. The specific implementation of model 515 is not crucial for the techniques described herein, and various models known in the prior art can be used. In particular, not only can microscope images depicting a scene with an arrangement of cells be evaluated; microscope images depicting other arrangements, such as microplastics, etc., can also be evaluated. In such cases, different models are typically used.

[0107] The density map 315 is available in the variant of Fig. 10 further processed, namely by means of another model 520. For example, the further model 520 can be produced by the process from Fig. 8 or those related to Box 952 in Fig. The two described techniques must be implemented. The output of model 520 is then a localization of the counting areas in a corresponding map 521 or as position coordinates.

[0108] In the illustrated example of the data processing pipeline 599, the map 521 and the density map 560 are then jointly evaluated in a model 530 to obtain a corresponding final result, for example, statistics on the number of living or dead cells in the two localized counting areas. These statistics can be output to the user, for example, via a graphical user interface. Corresponding techniques have already been described in connection with Box 960 and Box 965. Fig. 1 discussed.

[0109] Fig. Figure 11 schematically illustrates an electronic data processing device 680 according to various examples. The electronic data processing device 680 can be, for example, a PC or a server. The electronic data processing device 680 comprises a processor 681, a memory 682, and a communication interface 683. The processor 681 can load and execute program code from the memory 682.When processor 681 executes the program code, it performs techniques as described herein, such as: receiving a microscope image, for example via the communication interface 683 from a microscope, from an image database, or from local memory 682; applying one or more image evaluation techniques to the microscope image; locating a grating arrangement in a microscope image; and performing image analysis on the microscope image, taking the grating arrangement into account. For example, processor 681 could execute techniques as described above in connection with the flowchart. Fig. 1 and others were described.

[0110] In summary, techniques for processing microscope images have been described. These techniques combine image analysis with the localization of a grid array within a microscope image. Image analysis can, for example, locate cells (such as living and dead cells), while grid array localization determines the positions of counting surfaces. Grid array localization can be performed in various ways. For instance, a machine-learned model can be used to locate grid points. Grid vertices, in particular, can be located. These vertices can then be used to determine the counting surfaces. The techniques can also utilize prior knowledge about the type of sample support defining the grid array to improve localization accuracy.Naturally, the features of the embodiments and aspects of the invention described above can be combined with one another. In particular, the features can be used not only in the combinations described, but also in other combinations or individually, without leaving the scope of the invention. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] EP 4 053 805 A1

[0051] DE 10 2021 125 538 A1

[0066] Cited non-patent literature

[0000] Phase imaging by the transport equation of intensity.“ Optics communications 49.1 (1984): 6-10

[0046] Mehta, Shalin B., and Colin JR Sheppard. „Quantitative phase-gradient imaging at high resolution with asymmetric illumination-based differential phase contrast.“ Optics letters 34.13 (2009): 1924-1926

[0046]

Claims

[1] Computer-implemented method that includes: - Obtaining (920) a microscope image (305) captured by means of a microscope, which depicts an arrangement of structures on a sample carrier, wherein the sample carrier has a grid arrangement (200) that defines several surfaces (221, 222) with predefined extent, - Localizing (950) the grating arrangement (200) in the microscope image (305) using a machine-learned model (510), and - based on locating the lattice arrangement (200), performing (960) an image analysis (515, 530) of the microscope image (305) to quantitatively determine a property of the arrangement of the structures depending on a positioning of the structures in the multiple surfaces (221, 222). [2] Computer-implemented method according to claim 1, where the structures are cells, where the grid arrangement is a hemocytometer grid arrangement, where the property includes at least one of a cell state estimation, a cell count, or a cell morphology. [3] Computer-implemented method according to claim 1 or 2, wherein the localization of the lattice arrangement comprises the localization (951) of lattice points of the lattice arrangement using the machine-learned model. [4] Computer-implemented method according to claim 3, wherein the machine-learned model provides position coordinates or a mask image in which the grid points are marked. [5] Computer-implemented method according to one of claims 3 to 4, wherein the machine-learned model provides a picture-to-picture transformation. [6] Computer-implemented method according to 3 or 5, wherein the machine-learned model is trained to specifically locate grid points of the grid arrangement which define corners of the surfaces. [7] Computer-implemented method according to any one of claims 3 to 6, wherein the grid arrangement comprises grid points of a first type which define corners of the surfaces, wherein the grid arrangement includes grid points of a second type which do not define corners of the surfaces, where the localization of the lattice arrangement includes discriminating between the lattice points of the first type and the lattice points of the second type. [8] Computer-implemented method according to one of claims 3 to 7, wherein a further model (520) based on an analysis (980, 985) of neighborhood relationships between the localized grid points localizes the surfaces. [9] Computer-implemented method according to claim 8, wherein the further model compares candidate surfaces bounded by a predetermined number of localized grid points with specifications about a shape and / or relative positioning of the surfaces in order to select the surfaces from the candidate surfaces. [10] Computer-implemented method according to any one of claims 3 to 9, wherein a further model comprises an iterative fitting of a specification of the grid arrangement to the localized grid points. [11] Computer-implemented method according to one of the preceding claims, wherein the localization of the grid arrangement comprises localization (952) of the surfaces. [12] Computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Performing (915) a color calibration of the microscope image based on a color of the grating arrangement. [13] Computer-implemented method according to any one of the preceding claims, the method further comprising: - Performing (935) a rescaling of the microscope image based on a size of the grating arrangement in the microscope image. [14] Computer-implemented method according to any one of the preceding claims, the method further comprising: - Controlling (910) the microscope to perform autofocusing based on an image of the grating arrangement in further microscope images. [15] Computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Registering (942) the arrangement of the structures in a reference coordinate system based on the arrangement of the structures and / or based on locating the grid arrangement. [16] Computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Determining (906) a specification of the lattice arrangement based on a type of specimen, wherein the localization of the lattice arrangement is based on the specification of the lattice arrangement. [17] Computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Generating statistics for the property of the arrangement of the structures and depending on the positioning of the structures in the multiple surfaces based on quantitative determination. [18] Computer-implemented method according to any one of the preceding claims, the method further comprising: - Performing a preliminary analysis of the arrangement of the structures, whereby the image analysis is carried out depending on a result of the preliminary analysis. [19] Computer-implemented method according to claim 18, wherein the method further comprises: - Masking (945) an area of ​​the sample based on a result of the pre-analysis, wherein the image analysis is performed differently in the masked area than outside the masked area. [20] Computer-implemented method according to claim 2 and according to claim 18 or 19, the preliminary analysis includes the detection of cell clumps, different models are used to perform cell counting inside and outside of cell clusters. [21] Computer-implemented method according to any one of the preceding claims, the method further comprising: - Controlling (965) a graphical user interface to display a result of the image analysis. [22] Electronic data processing device (680) comprising a processor (681) and a memory (682), wherein the processor (681) is configured to load and execute program code from the memory (682), wherein, based on the execution of the program code, the processor (681) is configured to perform the following steps: - Obtaining a microscope image captured by means of a microscope, which depicts an arrangement of structures on a sample carrier, wherein the sample carrier has a grid arrangement that defines several surfaces with predefined extent, - Locating the grating arrangement in the microscope image using a machine-learned model, and - based on locating the lattice arrangement, performing an image analysis of the microscope image to quantitatively determine a property of the arrangement of the structures depending on a positioning of the structures in the multiple surfaces. [23] Electronic data processing device (680) according to claim 22, wherein the processor (681) is configured based on the execution of the program code to execute a method according to any one of claims 1 to 21.

Images