Image analysis of samples on sample carriers with grid array

Abstract

The present disclosure relates to image analysis of a sample on a sample carrier having a grid array. Various examples of the present disclosure relate to techniques for evaluating a microscope image (305). The microscope image (305) shows a sample carrier having a grid array, which in turn defines a plurality of regions having a predetermined extent. Techniques are described that can be used to locate (510, 520) the grid array in the microscope image, thereby enabling quantitative image analysis (515, 530) of the arrangement of structures (e.g., cells) on the sample carrier taking into account the regions. An exemplary sample carrier is a hemocytometer, e.g., a Neubauer counting chamber, used for cell counting.

Description

Technical Field

[0001] The various examples in this disclosure relate to image analysis of microscope images showing a scene of structural arrangement on a sample carrier. In particular, techniques for automatically quantitatively measuring the properties of structural arrangement are disclosed. Background Technology

[0002] In microscopy, a counting chamber is used for quantitative measurements. A counting chamber is a sample carrier with a grid array that allows for transmitted light geometric imaging. The resulting microscopic image reveals the structural arrangement on the grid array within the counting chamber. A commonly used grid array is the Neubauer grid.

[0003] A special type of counting chamber is the blood cell counter, which is mainly used in medical and biological laboratories: the blood cell counter can count cells in a suspension.

[0004] People manually count cells by observing the area defined by a grid under a microscope. This requires considerable precision and patience, especially when the cell concentration is high. Summary of the Invention

[0005] Therefore, there is a need to automate and simplify cell counting, or more broadly, to automate and simplify microscopic image analysis in order to quantitatively determine the characteristics of the structural arrangement displayed in a grid array combined with a sample carrier in a microscopic image.

[0006] This objective is achieved through the features of the independent patent claims. The features of the dependent patent claims define the implementation method.

[0007] The following discloses a technique for the automated quantitative measurement of the arrangement of structures on a sample carrier. Specifically, the characteristics of the structural arrangement can be quantitatively determined based on the location of the structures in multiple regions of the sample carrier. These regions (e.g., three-dimensional volumes) are defined by a grid array. For ease of understanding, these regions are referred to hereinafter as counting regions; although the quantitative measurement is not limited to counting tasks.

[0008] For this purpose, microscopic images are acquired using a microscope, according to various examples, showing the arrangement of structures (e.g., cells, but also other types of structures such as yeast, sperm, microparticles, etc.) and their grid arrays. The grid array defines one or more counting regions. These counting regions have default ranges, so the characteristics of the structural arrangement can be quantitatively determined based on the structure's position within multiple counting regions.

[0009] A grid array is located in the microscope image. Then, image analysis can be performed on the microscope image based on the location of the grid array to quantitatively determine the characteristics of the structural arrangement according to the position of the structures in multiple regions. In particular, structures in each counting region can be counted (and, if necessary, differentiated by structure type).

[0010] Machine learning models can be used to locate grid arrays to determine counting regions. In principle, using machine learning models to locate grid arrays is very useful because the appearance (color, contrast, brightness) of the microscope images used can vary due to factors such as illumination, objectives, background, and suspension. Image noise can also differ between microscope images. Therefore, using machine learning models is very useful, as they can provide robust localization of grid arrays without overly relying on the contrast used. In particular, machine learning models can provide robust localization of grid arrays compared to traditional non-machine learning models (e.g., attempting to fit different candidate grid arrays to image pixels). Furthermore, grid arrays on sample carriers from different manufacturers can appear very different. Different grid arrays have different numbers and arrangements of grid lines. By using machine learning models, robust identification of counting regions can be achieved even with diverse grid arrays. Moreover, machine learning models can be retrained, for example, when new grid arrays appear. This allows the machine learning model to be further trained using new training data to expand its capabilities. Proper training can prevent a decline in recognition performance for previously trained grid arrays.

[0011] For example, a computer-implemented method includes the following steps: acquiring a microscope image captured by a microscope, the image showing the arrangement of structures on a sample carrier. The sample carrier has a grid array that defines multiple regions with predetermined extents. The grid array is located in the microscope image using a machine learning model. Based on the location of the grid array, image analysis is performed on the microscope image to quantitatively determine the characteristics of the structural arrangement according to the position of the structures in the multiple regions.

[0012] The structure can be a cell. The grid array can be a blood cell counter grid array. Properties can include at least one of cell state assessment, cell counting, or cell morphology.

[0013] Electronic data processing equipment is adapted to perform this computer-implemented method.

[0014] The features described above and those described below can be used not only in the combinations explicitly stated therein, but also in other combinations or individually, without departing from the scope of protection of this invention. Attached Figure Description

[0015] Figure 1A flowchart of an exemplary method.

[0016] Figure 2 A flowchart of an exemplary method.

[0017] Figure 3 A microscope image is shown, illustrating the arrangement of cells on a sample carrier with a grid array.

[0018] Figure 4 It shows Figure 3 The image is a microscope image in which the grid lines of the grid array have been redrawn.

[0019] Figure 5 A density map is shown, in which the location is... Figure 3 The grid corners of the medium grid array.

[0020] Figure 6 A density map is shown, in which the location is... Figure 3 Grid points of a medium grid array.

[0021] Figure 7 schematically shown Figure 3 Mask image of a grid array.

[0022] Figure 8 A flowchart of an exemplary method.

[0023] Figure 9 An exemplary grid array is shown.

[0024] Figure 10 A schematic diagram of a data processing pipeline based on different examples is shown.

[0025] Figure 11 Schematic diagrams of electronic data processing devices according to different examples are shown. Detailed Implementation

[0026] The above-described features, characteristics, advantages, and implementation methods of the present invention will become clearer and more explicit when combined with the following embodiments described in more detail with reference to the accompanying drawings.

[0027] The present invention will now be described in more detail with reference to the accompanying drawings and preferred embodiments. The same reference numerals in the drawings denote the same or similar elements. The drawings are schematic diagrams of various embodiments of the present invention. The elements shown in the drawings are not necessarily drawn to scale. Rather, the manner in which the various elements are depicted in the drawings should enable those skilled in the art to understand their function and general purpose. The connections and couplings between the functional units and elements shown in the drawings can also be implemented as indirect connections or couplings. Connections or couplings can be implemented via wired or wireless means. Functional units can be implemented as hardware, software, or a combination of hardware and software.

[0028] The following describes various techniques for automatically performing quantitative measurements based on microscope images showing the arrangement of structures on a sample carrier. The sample carrier has a grid array that defines multiple counting regions with predetermined extents. Quantitative measurements are performed based on these counting regions. The sample carrier may, for example, have a certain depth such that the counting regions, together with the depth extension, define a three-dimensional volume.

[0029] In the various scenarios described in this paper, a grid array is located in a microscope image using a machine learning model. The machine learning model can be, for example, a deep neural network.

[0030] Machine learning models can, for example, provide image-to-image transformations, where the output image is labeled with a grid array of specific reference points. Machine learning models can also perform localization tasks, providing the coordinates of the corresponding reference points in a list format.

[0031] By using machine learning models, particularly robust localization of grid arrays in microscope images can be achieved. In particular, machine learning models can be trained to handle microscope images with varying contrast, brightness, structure density, brightness ratios between structure and background, image artifacts or aberrations, contamination, etc. Furthermore, machine learning models can be trained to robustly handle sample carriers of different types with varying grid arrays.

[0032] The structure can be, for example, a cell. The cell can be located in a suspension on a blood cell counter sample carrier. In particular, in the various examples described herein, cell state assessment, cell counting, or cell morphology determination can also be performed on cells on a blood cell counter grid array.

[0033] In principle, the techniques described in this paper can also be used for other types of structures, such as yeast, sperm, algae, or microplastics. For all such structures, if the grid array is robustly positioned, it helps to quantitatively determine the characteristics of the structural arrangement based on the structure's position relative to the counting region. For simplicity, the following discussion focuses primarily on the application of the techniques described in this paper in the automated quantitative measurement of cells. However, the techniques and concepts described can also be applied to other types of structures.

[0034] Figure 1 A flowchart of an exemplary method. Figure 1 The method described can be implemented at least partially by a computer. This means that the processor can load and execute program code from memory. When the processor executes the program code, it will then execute... Figure 1 At least some of the boxes in the method.

[0035] Figure 1The flowchart illustrates the automated quantitative measurement of structural arrangements. In principle, various structural arrangements can be measured, such as those in yeast, sperm, algae, and microplastics. However, for simplicity, the following description uses cell arrangements as an example. Figure 1 .

[0036] In box 905, load the sample. For example, place the sample in the beam path of an optical microscope. This can be done automatically, for example, by picking up the sample and placing it in the beam path using a robotic arm.

[0037] The sample has cells arranged in a manner on a sample carrier (e.g., in a suspension). The sample carrier has a grid array. In particular, the sample carrier can be a counting chamber of a blood cell counter.

[0038] The grid array defines multiple counting regions with predetermined ranges.

[0039] In box 906, the type of sample carrier can first be optionally determined. The type of sample carrier can be determined in a variety of ways.

[0040] For example, machine-readable information (such as type information) can be read optically or electrically. The type can be determined based on user input. However, image analysis can also be performed to classify sample carriers. This image analysis can be performed on overview images at particularly low magnification. Image analysis can typically be based on calibration images.

[0041] Different types of sample carriers have different grid arrays, and therefore different counting regions. A common type is the Neubauer counting chamber. Besides the Neubauer counting chamber, there are other counting chambers, such as the Burker-Türk counting chamber, the Fuchs-Rosenthal counting chamber, the Thoma counting chamber, and the Malassez counting chamber. Another example is the Petroff-Hausser counting chamber.

[0042] The classification of the counting chamber manufacturer and / or model can provide important background information for obtaining accurate geometry (grid location, number and / or size of counting areas, etc.) and for any special cases in quantitative measurements (e.g., counting). The known grid geometry is also a crucial basis for all other grid-based processing steps (e.g., color calibration, focus determination, location determination, etc.). Therefore, in box 906, the default value of the grid array can be determined based on the type of sample carrier.

[0043] If image-based sample carrier type classification is used to determine the default values ​​for the grid array, this can technically be achieved using a machine learning classification model. Such classification tasks are generally of low complexity. In particular, since the corresponding training data does not require information such as localization, but only class labels as basic facts, a large amount of training data can be used to train this machine learning classification model. In other words, this means that a large amount of corresponding training data can be generated relatively quickly, enabling robust classification of sample carrier types subsequently achieved using a machine learning model.

[0044] Box 905 is optional in principle; in some variants, the microscope images have been acquired at an earlier time, so box 905 is no longer required.

[0045] In box 910, autofocus can optionally be performed. Various examples are based on the understanding that accurately determining the focal point is crucial, especially when using staining markers to color specific structures within cells. For instance, to distinguish between live and dead cells in quantitative measurements, trypan blue can be used to stain cells. If defocusing occurs, the cell membrane being stained blue can be confused with the dye penetrating to the center of the cell. It is precisely this difference that makes distinguishing between live and dead cells possible.

[0046] In principle, there are multiple methods for performing autofocus within box 910. For example, autofocus can be based on the cells themselves. For instance, cell halo, sharpness, and / or size can be determined, and autofocus can be performed based on this. Overall, these cell characteristics can optimize image quality, especially sharpness, thereby improving the accuracy of cell analysis.

[0047] However, in another variant, it is conceivable that, as an alternative or supplement, the microscope could be controlled to autofocus based on imaging of a grid array in a corresponding calibration image. For example, the width of the grid lines could be measured. The measured grid line width could be compared, for example, to a target value for the grid line width, which is known based on the default values ​​of the grid array from box 906. Alternatively or supplementarily, the sharpness of the lines could be measured. Furthermore, since the cell equator is typically located above the grid lines, a defined offset could also be considered, for example.

[0048] Within box 910, the user interface output indicating autofocus quality can be controlled. This allows for real-time display of focus scores, providing immediate feedback to the user regarding focus settings. Furthermore, this method can also support quality assurance documentation.

[0049] When the focus quality is insufficient, it can be prevented Figure 1 Continue execution of each box within the image to avoid extensive analysis based on blurry images.

[0050] Box 910 is optional in principle; in some variants, the microscope images have been acquired at an earlier time, so box 910 is no longer required.

[0051] In box 915, color calibration can optionally be performed. Color calibration ensures consistent color reproduction. This is particularly useful when performing quantitative assessments using staining markers of specific colors. For example, as mentioned above, trypan blue can be used to distinguish between live and dead cells. In box 915, pre-calibration can be performed based on the colors of the grid array in the calibration microscope image. For example, the grid lines can be semantically segmented using a machine learning model, and then the colors of the grid lines can be matched according to default colors (e.g., based on prior knowledge about the sample carrier from box 906).

[0052] Color calibration can generally be performed at the microscope end. Alternatively, it is conceivable to perform pre-calibration during digital post-processing, i.e., digitally converting the color space of the acquired microscope images.

[0053] Box 915 is optional in principle; in some variants, the microscope images have been acquired at an earlier time, so box 915 is no longer necessary. However, even if the microscope images were acquired earlier, color calibration can be performed via digital post-processing.

[0054] In box 920, a microscope image is acquired. Box 920 may include controls for acquiring relevant image data. Alternatively, the user can load existing, pre-stored images.

[0055] Users can actively trigger the acquisition of microscope images, for example, by pressing a button. The microscope is then controlled accordingly. Images can also be acquired automatically. Acquisition times can be preset, such as acquiring images at fixed time intervals during an experiment, or adaptively acquiring images when specific events occur.

[0056] Microscopic images can be acquired through transmission imaging without fluorescence contrast. Alternatively, images can be acquired through fluorescence contrast. For example, nuclear staining can be performed using DAPI, Hoechst, or other reagents.

[0057] Microscopic images can have specific or non-specific contrast. Specific contrast marks specific cellular structures in a specific way. For example, specific fluorescence contrast or non-fluorescent specific contrast can be used. Non-specific contrast can be, for example, phase contrast, bright-field contrast, or autofluorescence. For example, non-specific contrast can be phase-like contrast. Phase-like contrast can be, for example, phase contrast. Examples include Zernike-Phasenkontrast and Normarski-Phasenkontrast. Here, special optical elements are used in the beam path, such as a phase ring in the objective and an annular stop in the condenser. This makes the interference between the background and the object light visible. By using phase contrast, image contrast can be improved. This means that cellular structures are particularly clearly visible. Cells are phase objects; their amplitude does not decrease or does not decrease significantly when light passes through a cellular sample, so phase contrast is preferred for visualizing phase shifts. However, digital phase contrast can also be used as a phase-like contrast. Here, multiple images are acquired and then combined into a single phase-contrast image through calculation. Therefore, this type of technique can be called digital phase contrast. This phase contrast is obtained through digital post-processing of the acquired intensity image. Examples include the Transport of Intensity Equation (TIE) and Differential Phase Contrast (DPC). For an explanation of TIE, see: Streibl, Norbert. "Phase imaging by the transport equation of intensity." Optics Communications 49.1 (1984): 6-10. For an explanation of DPC, see: Mehta, Shalin B., and Colin JR Sheppard. "Quantitative phase-gradient imaging at high resolution with asymmetric illumination-based differential phase contrast." Optics Letters 34.13 (2009): 1924-1926.To acquire a TIE dataset, the sample needs to be translated along the optical axis (z-direction), i.e., axial translation, and a so-called z-axis stack consisting of at least two images needs to be obtained. The data is then processed to obtain a phase-contrast image. This requires solving diffusion-type partial differential equations. In DPC, the sample is illuminated from at least two different directions (oblique illumination) while the sample remains at a fixed z-axis position. Possible sources for oblique illumination include all types of segmented sources; for example, segmented diodes, LED arrays, digital micromirror devices (DMDs), liquid crystal displays (LCDs or SLMs), or variable condenser apertures. The obtained data is then converted into a phase-contrast image by solving a deconvolution problem. The advantage of using digital phase contrast (compared to hardware-based phase contrast) is that no object needs to be inserted or removed in the beam path when acquiring digital phase contrast. Instead, the illumination can be selectively modified, for example, by using a switchable LED array arranged in the illumination pupil plane. This can be done quickly and easily.

[0058] Non-specific contrast types include “label-free” contrast, such as phase contrast, DIC contrast, or TIE contrast, which allow observation of cells without specific labels. Alternatively, dyes such as H&E can be used to specifically label particular cellular structures. Another option is to use fluorescent dyes (such as DAPI) to fluorescently label specific cellular components. Autofluorescence can also be used to locate cells without additional labeling.

[0059] In principle, a microscope image can also contain multiple channels (multi-channel microscope images) with different contrasts (e.g., from those mentioned above).

[0060] Microscopic images can show the entire counting chamber or only a portion of it. This largely depends on the objectives used and their magnification. For example, objectives with magnifications of 4x, 10x, or 20x can be used. Finding a balance between a large field of view (FoV) and sufficient detail in cell morphology is important. This means that subsequent models are trained at magnifications that allow for the largest possible field of view while still robustly making the necessary distinctions, such as distinguishing between live and dead cells.

[0061] In box 935, the microscope image emanating from box 920 can optionally be rescaled, i.e., the size of the microscope image can be changed. For example, it can be rescaled so that the microscope image subsequently displays cells or other structures (e.g., the mesh structure of a grid array) at a specific imaging scale. In other words, this means that scaling can be performed so that the cells in the microscope image have a specific size. This size can be predefined. In particular, this size can correspond to the cell size in a reference microscope image used to train one or more machine learning models (subsequently used for image evaluation). This can reduce the complexity of the corresponding machine learning models, as these models only expect cells with a specific imaging size. The training workload for such machine learning models is reduced. The training data does not necessarily have to contain cells with different imaging scales, but can be restricted to cells with a specific imaging size.

[0062] The scaling in box 935 can be performed manually, for example. However, it is also conceivable to use a machine learning model to perform the scaling in box 935. For example, a machine learning model can be used to perform an image-to-image conversion, i.e., outputting a rescaled image. Alternatively, a machine learning model can be used to output scaling factors, i.e., performing an image-to-scalar conversion. An exemplary rescaling technique is basically described in EP 4 053 805 A1. Corresponding techniques are incorporated herein by cross-reference.

[0063] The various techniques described herein can, in particular, utilize the characteristic that cells in suspension are typically spherical. This prior knowledge about cell shape can be used for rescaling. Furthermore, microscope images can be rescaled based on the dimensions of the grid array. For example, prior knowledge about the dimensions of specific structures within the grid array (especially the side lengths of the counting regions) can be obtained based on the default values ​​of the grid array defined in box 906. Therefore, the microscope image can be rescaled to give specific grid structures of the grid array specific dimensions. Thus, in box 935, one or more grid structures can be identified, and the imaging dimensions of these grid structures can then be compared with the target dimensions so that the microscope image can be rescaled accordingly.

[0064] In box 935, inappropriate or unreasonable magnification can also be optionally automatically identified. For example, it can be identified that the image size of cells in a microscope image does not match the objective magnification (i.e., the cells are too large or too small relative to the current objective magnification). The objective magnification can be read, for example, from the metadata of the microscope image or obtained in the form of microscope control data.

[0065] In box 940, the microscope image from box 920 may optionally be pre-analyzed. This pre-analysis does not involve actual quantitative measurements; rather, it checks whether the microscope image meets specific characteristics in order to perform actual quantitative measurements later or achieve specific precision.

[0066] This can occur when problems are identifiable in the input data, such as contaminants or cells on different focal planes. Machine learning can be used to identify such and other problems, particularly through classification, detection, or segmentation models trained with known contaminants. Alternatively, so-called "one-class classification" can be used to find "unknown" objects. Anomaly detection can be performed.

[0067] If a problem is identifiable in the microscope image, different follow-up actions can occur. For example, a warning may be issued to the user, or an indication of a filling error in the sample carrier or counting chamber may be provided. Annotations can also be created in the metadata. In some cases, it may be necessary to change the workflow, such as excluding certain counting regions from subsequent quantitative measurements, and / or using additional counting regions to enhance statistical data. Therefore, subsequent image analysis can often be performed during the quantitative measurement process based on the pre-analysis results from box 940.

[0068] In box 942, the position of the cell arrangement can optionally be determined. In other words, this means registering the scene shown in the microscope image to a reference coordinate system. This position determination or scene registration can provide navigation assistance to the user and avoid repeated evaluation of the same sample area by identifying the absolute position or identifying the sample area that has been visited. Optionally, the images can also be stitched into a single image based on the determined image position.

[0069] Location determination can be performed in several ways. For example, the grid array of the sample carrier can be located. This allows it to be determined which parts of the grid array (e.g., which counting regions) are visible in the microscope image. This can be particularly advantageous at high magnification when only a portion of the grid array is visible in the microscope image. This is technically feasible, for example, by identifying the grid points (e.g., through a neural network or filtering) and then matching them against a “grid database” (the default for grid arrays, e.g., from box 906), for example using an “Iterative ClosestPoint” algorithm.

[0070] Alternatively, registration can be performed using the already analyzed cell arrangement patterns. Technically, this can be achieved by interpreting the cells as a point cloud. For each image, the current position in the point cloud is continuously estimated and expanded as necessary.

[0071] Another method for determining the location is to use an overview camera that observes the sample space.

[0072] In box 945, specific regions of the scene can be optionally masked. For example, a user can mask specific regions of a microscope image that they want to exclude from subsequent quantitative measurements and / or to which specific image analysis algorithms are to be applied via a graphical user interface. Alternatively or supplementarily, automatic masking can also be performed.

[0073] For example, cell clumps or other regions in a sample that are unsuitable for quantitative measurement or require specific image analysis algorithms can be pre-identified and then appropriately considered in image analysis through masking (e.g., excluding them from image analysis in quantitative measurement). Thus, masking can be performed, for example, based on the results of box 940.

[0074] For example, models (such as machine learning models) can be used to detect and locate cell clumps. Cell clumps are aggregates of cells where individual cells no longer exist in isolated forms in a suspension. Because the distance between adjacent cells within a cell clump is much smaller than outside the clump, or because cells within a clump are not easily distinguishable visually, different methods or models are often needed to perform quantitative measurement tasks inside and outside the cell clump, such as cell counting. For example, one could envision first counting the cells outside the clump, and then estimating the number of cells within the clump based on prior knowledge of cell size and the measured area of ​​the clump. In other words, this eliminates the need to count the cells within the clump individually. For example, the area of ​​a cell clump can be determined through semantic segmentation or instance segmentation, and then the specific area of ​​the clump can be compared with the area of ​​a single cell. Distinguishing between dead cells, live cells, or cell debris can also be achieved. A concrete example is: for instance, detecting a cell clump with an area of... A K The clumps of cells, where the area is approximately the area of ​​a single cell in the scene. A Z Five times that. Therefore, when performing cell counting, we can assume that there are five cells in the cell cluster.

[0075] Within box 950, the grid array is located. A machine learning model is used for this. Locating the grid array specifically means locating the counting region defined by the grid array. This counting region can be located within the microscope image.

[0076] Then, in box 960, image evaluation for actual quantitative measurement is performed. This may include, for example, locating individual cells, and, where necessary, their respective states (e.g., alive or dead). Cells can be counted in different located counting regions. Therefore, in the image evaluation of box 960, the counting region located in box 950 will be taken into account. Optionally, one or more mask regions from box 945 may also be considered, for example, by selecting an appropriate image evaluation algorithm based on the mask region (e.g., the example described above in conjunction with box 945, where different algorithms are used to count cells inside and outside the mask region of the labeled cell cluster).

[0077] In box 960, for example, a density map marking all cells can be created. Multiple density maps can be created, for example, for each cell type. For example, a first density map can be created for all live cells, and a second density map for all dead cells. Corresponding techniques are described, for example, in DE 10 2021 125 538 A1.

[0078] Therefore, density maps can have contrast, which indicates the cell density or number at a specific pixel. For example, each target cell can be arranged like a "Gaussian clock," with a contrast that differs from the background at the cell center.

[0079] In addition to this image-to-image transformation that creates one or more density maps, it is conceivable to use detection models. For example, a list of coordinates in a microscope image could be output, with entries in the list used to locate different cells. Furthermore, category labels could be output separately, for example, to distinguish between live and dead cells.

[0080] In another variant, a segmentation model can be used. For example, a semantic instance segmentation mask can be created. In this way, for example, each pixel of the microscope image can be assigned to a different category (e.g., "background," "live cells," and "dead cells"). Then, the different cells can be separated, for example, using a non-maximum suppression (NMS) technique.

[0081] Image analysis in box 960 can be aided, in particular, by using appropriate dye markers. For example, special dyes can be used to label dead cells. Trypan blue, for instance, is visible in bright-field imaging. Fluorescent markers can also be used to distinguish between live and dead cells. This is especially useful when analyzing adherent cells.

[0082] In principle, the specific implementation of image analysis in box 960 is variable and not critical to the techniques described herein. In particular, various image analysis techniques known in the art can be combined with the techniques described herein for locating the grid array (box 950). The key to the various techniques described herein lies in combining the grid array location for determining the counting region with the execution of image analysis to achieve quantitative measurements, such as cell counting in a defined suspended region.

[0083] To achieve this correlation between counting regions and cell arrangement characteristics, in the first example, image analysis can be performed on the entire microscope image (firstly). This image analysis can be performed, in particular, without distinguishing between different counting regions. The results of the image analysis can then be "decomposed" into different counting regions. This means, for example, that the results of the corresponding portions of the evaluation can be assigned to different counting regions. However, multiple image evaluations can also be performed separately for portions of the microscope image corresponding to the located counting regions. This means that multiple microscope image segments corresponding to different counting regions can be pre-determined, and then image evaluation can be performed on each segment separately.

[0084] Based on these techniques, in box 960, statistical data on cell arrangement characteristics (e.g., cell density, cell number, etc.) can be generated according to the position of cells in multiple counting regions. Such statistical data, especially in quantitative measurements, helps to obtain an overview of specific macroscopic characteristics of the arrangement (e.g., cell suspension). Furthermore, these results can be used to identify specific information, such as the ratio of live to dead cells or indications of problems with cell culture and its environmental parameters. Dilution factors can also be suggested to determine the correct amount of culture medium to use when transferring to other containers.

[0085] In some cases, certain areas, i.e., partial or complete counting areas, can be omitted to calculate statistical data. This can be preset, for example, using the corresponding mask area from box 945.

[0086] In box 962, post-processing of the output of box 960 may be optionally performed. In the image analysis of box 960, contradictory predictions may occur. For example, a cell may be classified as both a live and a dead cell. This may indicate a problem with the input data and / or the model.

[0087] If such contradictory predictions occur frequently, it may be necessary to abandon the corresponding evaluation or issue a warning to the user. In some cases, it may also be necessary to adjust the workflow, such as using a different counting area for evaluation.

[0088] In box 965, the image evaluation results are output to the user. Specifically, the graphical user interface can be controlled to display the image analysis results. For example, if statistics are generated, these statistics can be output through the user interface.

[0089] Image evaluation results can be presented in various formats to provide users with a comprehensive understanding of quantitative measurements. For example, the number of viable cells can be displayed, as this is the most important indicator of reproducible cell concentration. Optionally, the number of dead cells can also be displayed if needed. For instance, corresponding indicators indicating the proportion of viable cells can be overlaid on the microscope image for different counting regions. Color coding can be used to indicate different counting regions, with green indicating fewer dead cells and red indicating more dead cells. Furthermore, prompts can be output regarding the ratio of viable to dead cells or indicating problems with cell culture and its environmental parameters. Dilution factors can also be suggested to determine the correct amount of culture medium for transfer to other containers.

[0090] Results from multiple counting regions can also be summarized, for example, by counting multiple regions sequentially. This summarized information can then be output to the user.

[0091] The above text combined Figure 1 A technique for automated quantitative measurement of cell arrangement is described. The technique described herein is based on the combination of positioning a grid array of blood cell counters (box 950) and image analysis (box 960). Further details of box 950 will be elaborated below.

[0092] Within box 950, there are different implementation schemes for positioning the grid array. Figure 2 An exemplary two-stage variant is shown.

[0093] Figure 2 This is a flowchart of the method, which shows... Figure 1 An exemplary implementation variant of the mid-frame 950.

[0094] exist Figure 2 In the two-stage variant, the grid points of the grid array are first located (box 951). Then, the counting region is located in box 952.

[0095] In box 951, a machine learning model can be used in particular. This machine learning model is used to locate grid points.

[0096] The location of grid points can be done in several ways. For example, a density map can be output. In other words, this means that the machine learning model provides a mask image for the microscope image in which the grid points are drawn with high or full contrast. To do this, the machine learning model can perform an image-to-image transformation, mapping the microscope image onto the density map. Alternatively, it is conceivable that the machine learning model outputs a list of positional coordinates (e.g., xy positions in pixels of the microscope image), where different list entries label different grid points, such as their centers.

[0097] If one or more grid points (especially grid corners) are obtained through a machine learning model, the corresponding positions of these grid points can be further refined locally. Localization can be refined. This can be achieved, in particular, through super-resolution, i.e., using a subpixel resolution greater than the resolution of the microscope image. This can be achieved, for example, by locally fitting (a straight line, a Gaussian curve, etc.) the image data of the microscope image or, for example, its image axis projection.

[0098] A typical blood cell counter's grid array basically has several types of grid points. The first type of grid point defines the corners of the counting region (grid corner points). The second type of grid point does not define the corners of the counting region. For example, Figure 3 Microscopic image 305 is shown, displaying two complete counting regions and seven truncated counting regions. Furthermore, live and dead cells are visible, i.e., bright spots and dark spots. Each of the two complete counting regions has 16 quadrants. Figure 3 In the diagram, the lower left corner of both counting areas is marked with an arrow. Figure 4 The microscope image 305, with enhanced contrast, shows that the grid lines have been redrawn. Figure 5 An exemplary density map 315 of microscope image 305 is shown, which only marks the grid corners of the counted areas. In contrast, Figure 6 Density map 316 is shown, which marks all grid points (whether or not they are grid corner points).

[0099] Only after distinguishing grid corners from other grid points can the counting region be located in box 952. This generally reduces the complexity of the task in box 952 because by excluding grid points that do not define the counting region, the possible arrangements of the counting region are reduced. However, it is also conceivable to locate the counting region in box 952 without first distinguishing grid corners from other grid points.

[0100] Therefore, distinguishing grid corners from other grid points helps in locating the counting region within box 952. This can be achieved in several ways. In one variation, it can be envisioned that a machine learning model locates all grid points in the grid array. Next, it can be determined whether a point is a grid corner by analyzing its neighborhood. This means that the machine learning model first finds all intersections of the grid lines in the grid array, regardless of whether they are corners of the counting region. Some grid points can then be discarded. In another variation, it can also be envisioned that the machine learning model does not locate all grid points, but only specific types of grid points. In particular, the machine learning model can be trained to specifically locate grid corners.

[0101] Typically, the receptive field of a machine learning model that selectively locates grid corners must be larger than that of a machine learning model that locates all grid points (regardless of whether they are grid corners). This is because distinguishing grid corners from other grid points requires considering a relatively large neighborhood. Such a large receptive field can be achieved, for example, through very deep neural networks with multiple layers. Alternatively or supplementarily, the image can be scaled down before being fed into the deep neural network. On the other hand, the task of locating grid points is structurally relatively uncomplex, so deep neural networks can be relatively streamlined, i.e., each layer can use fewer channels.

[0102] To locate grid points, prior knowledge about the type of grid array can also be useful. For example, the relevant prior knowledge can be determined in box 906. This is based on the understanding that different types of sample carriers use different types of grid arrays. For example, different grid arrays have single, double, or even triple grid lines (e.g., Figure 3 or Figure 4 (See microscope image 305). Depending on the type of sample carrier, closed or open counting regions can be used. The number of sub-counting regions (in...) Figure 3 The number of grid points (in the 16 quadrants) can also vary. As seen above, the number and relative arrangement of grid points differ significantly. Therefore, if positioning is based on prior knowledge about the sample carrier type, the accuracy of the machine learning model can be improved. For example, different machine learning models are available for different types of sample carriers or different grid arrays. The appropriate machine learning model can then be selected based on the type of sample carrier. Alternatively, an indicator of the sample carrier type can be passed to the corresponding machine learning model, and the same machine learning model can be used to process microscope images displaying different types of sample carriers. For example, it is conceivable that the mask image 317 of the grid array (see...) Figure 7 ) is passed to the machine learning model, for example as an additional channel connected to the relevant microscope image 305 (konkatenieren).

[0103] Refer again Figure 2 Once the grid points and even the grid corners have been located, the next step is to determine the counting area. This is done in box 925.

[0104] Box 925 has different implementation variations. One example is the RANSAC (random sample consensus) algorithm. RANSAC is a robust parameter estimation method in the presence of outliers. When locating the counting region, RANSAC can be used to estimate the geometric transformation parameters between the located grid points and the default grid array (which defines the counting region). For example, the default grid array can be determined based on prior knowledge about the sample carrier type from Box 906. RANSAC works iteratively, involving multiple steps. In each step, a subset of grid corner points is selected, and the geometric transformation parameters between these grid points and the default grid array are estimated. These parameters are then used to estimate the positions of the remaining grid points. RANSAC is robust to outliers because it only requires a subset of grid points to estimate the geometric transformation parameters. This means that the algorithm remains effective even if some grid points are mislocated or outliers exist in the data. In particular, RANSAC can still locate the counting region even without prior differentiation between grid corner points and other grid points. However, since more iterations are required for convergence, the computational cost may be relatively high.

[0105] In box 952, a parametric model can also be fitted. During the iterative fitting process, the parametric model of the grid array (e.g., determined based on the sample carrier type that defines the default values ​​of the grid array from box 906) can be adjusted to match the two-dimensional arrangement of the grid corner points until the deviation between the default values ​​and the observed grid corner points is minimized. Here, the fitting can perform, for example, rotation or scaling, but cannot perform further compression or stretching.

[0106] Figure 8 Another implementation variant of box 952 is shown. Figure 8 It is a method flowchart, which shows... Figure 2 An exemplary implementation variant of the middle frame 952.

[0107] In box 970, for the density map (see...) Figure 5 or Figure 6 The density map (315 or 316) is used for evaluation. This involves detecting all marked grid corner points based on the specified default size. However, this step can be omitted if a list of coordinates is already available (e.g., as output of a machine learning model).

[0108] Box 975: Checks if the number of detected grid corner points is within an acceptable range (e.g., at least 4, at most 256). If it is not within the acceptable range, an error will be raised or appropriate action will be taken. Box 980 will only be executed if the number of detected grid points is within a specific default range.

[0109] Box 980: Here, the neighborhood relationships of each grid corner point are determined. For example, a distance matrix can be calculated. The distance matrix can, for example, give the pairwise Euclidean distances between all detected grid corner points.

[0110] Box 985: Identifying Potential Quadrants. This is accomplished through neighborhood relation analysis derived from Box 980. In this example, it is assumed that the counting region must always be a square. In principle, other types of shapes can also be considered. For each combination of four grid points, pairwise distances are calculated and sorted. The four minimum distances (potential side lengths) are normalized using the average of the first four (minimum) distances. A standard deviation threshold is used to check if the four minimum distances are similar. Another standard deviation threshold is used to check if the two maximum distances (potential diagonals) are similar. The validity of the quadrant is verified by comparing the ratio of the average diagonal to the average side length with the expected ratio of a square. If the point set passes all checks, the points are rearranged to maintain the geometric neighborhood of the corresponding counting region. More generally, multiple candidate counting regions can be found in Box 985, each region defined by a default number of located grid points. In the example below, each square counting region is defined by four grid points. The candidate counting regions found in this way can then be compared with the shape default (a square in the example above) and / or the relative position default (e.g., non-overlapping counting regions) to filter out the actual counting regions from the candidate counting regions.

[0111] Box 990: Sort the found candidate counting regions. For example, sort them based on their location in the microscope image. The center of the microscope image is calculated, and the candidate counting regions are sorted according to their distance from the image center. By sorting the candidate counting regions, specific candidate counting regions can be prioritized over others. Box 990 can, for example, be equivalent to selecting specific counting regions from the set of candidate counting regions for subsequent quantitative measurements within these selected regions. Besides this selection (e.g., based on location in the microscope image), the user can also manually select or sort the candidate counting regions.

[0112] Box 995: Output the candidate counting regions found. Returns a list of quadrants (e.g., up to a specific default list position from the list in Box 990), where each square counting region is represented by its four corners.

[0113] When reconstructing the square counting region from the set of located grid corner points, in Figure 8 The following constraints should be considered in the variant shown: the counting area consists of four grid corner points and is square. If the magnification and pixel size are known, the counting chamber size (typically 1 mm) can be used as a reference. 2 This excludes specific corner combinations that are not properly positioned. If information about magnification or pixel size is unavailable, the grid / cell size can be estimated from the microscope image and then rescaled in front of the microscope image. This can be done in a further filtering step, for example, after box 990.

[0114] Figure 9 An exemplary grid array 200 for a Newbauer blood cell counter is shown. Different counting regions 221 and 222 are illustrated. For example, depending on the application, a larger outer counting region 221 or a smaller inner counting region 222 may be used. Specific types of counting regions can be positioned based on user default values. For example, this can be achieved through… Figure 8 The default values ​​in block 985 of this method are implemented accordingly. Similarly, default values ​​can also be considered in the RANSAC algorithm.

[0115] Figure 10 The diagram illustrates a data processing pipeline 599 based on different examples. For instance, data processing pipeline 599 can implement... Figure 1 At least some of the boxes in the method.

[0116] Microscopic image 305 is processed by two independent algorithms or models 510 and 515. Model 510 may be, for example, a deep neural network (or other machine learning model), which provides density map 315, as described above. Figure 5 As described above. In the example shown, the provided density map 315 only locates the grid corners of the grid array. However, a machine learning model can also be used to locate all grid points of the grid array. Besides image-to-image transformation, it is also conceivable to output a list of location coordinates. The corresponding techniques have been discussed above. Figure 2 Box 951 or Figure 1 Box 950 was discussed.

[0117] Model 515 provides another density map 516, which, for example, locates all living cells, all dead cells, or all cells (whether living or dead). This is merely an example, and different evaluation techniques can be envisioned. The specific implementation of model 515 is not important to the techniques described herein, and various models known in principle in the prior art can be used. In particular, not only microscopic images showing scenes of cell arrangement can be evaluated; microscopic images showing other arrangements (e.g., arrangements of microplastics, etc.) can also be evaluated. In such cases, other models are typically used.

[0118] exist Figure 10 In the variant, the density map 315 is further processed by another model 520. For example, it can be done through... Figure 8 The methods or combinations in Figure 2 The technique described in the middle frame 952 is used to implement this other model 520. The output of model 520 is the location of the counting region in the corresponding figure 521, or it can be position coordinates.

[0119] In the example of the data processing pipeline 599 shown, Figure 521 and density map 560 are then jointly evaluated in model 530 to obtain corresponding final results, such as statistics on the number of live or dead cells in the two located counting regions. These statistics can be output to a user, for example, via a graphical user interface. The corresponding techniques have been combined... Figure 1 The mid-frame 960 and frame 965 were discussed.

[0120] Figure 11 An electronic data processing device 680 is schematically illustrated according to various examples. The electronic data processing device 680 may be, for example, a PC or a server. The electronic data processing device 680 includes a processor 681, a memory 682, and a communication interface 683. The processor 681 can load and execute program code from the memory 682. When the processor 681 executes the program code, it performs the techniques described herein, such as: acquiring microscope images, such as from a microscope or image database or local memory 682 via the communication interface 683; applying one or more image evaluation techniques to the microscope images; locating a grid array in the microscope images; and performing image analysis on the microscope images taking the grid array into account. For example, the processor 681 may perform the above-described combination... Figure 1 The techniques described by flowcharts and other flowcharts.

[0121] In summary, this paper describes techniques related to microscopic image processing. These techniques combine image evaluation and the localization of grid arrays in microscopic images. Image evaluation can locate, for example, cells (e.g., live and dead cells), while the localization of the grid array determines the location of counting regions. The localization of the grid array can be achieved in various ways. For example, machine learning models can be used to locate grid points. Grid corner points can be specifically located. Then, the grid (corner) points can be used to determine the counting regions. These techniques can also leverage prior knowledge about the type of sample carrier defining the grid array to improve the accuracy of localization. Of course, the features of the foregoing embodiments and aspects of this invention can be combined with each other. In particular, these features can be used not only in the said combinations but also in other combinations or individually without departing from the scope of this invention.

Claims

1. A computer-implemented method comprising the following steps: - Acquire (920) a microscope image (305) captured by a microscope, the microscope image showing the arrangement of structures on a sample carrier having a grid array (200) that defines a plurality of regions (221, 222) with predetermined ranges. - Using a machine learning model (510) to locate (950) the grid array (200) in the microscope image (305), and - Based on the positioning of the grid array (200), the microscope image (305) is subjected to image analysis (515, 530) to quantitatively determine the characteristics of the structure arrangement according to the position of the structure in the plurality of regions (221, 222).

2. The computer-implemented method according to claim 1, The structure described therein is a cell. The grid array mentioned above is a blood cell counter grid array. The aforementioned characteristics include at least one of cell state assessment, cell count, or cell morphology.

3. The computer-implemented method according to claim 1 or 2, The location of the grid array includes locating (951) the grid points of the grid array using the machine learning model.

4. The computer-implemented method according to claim 3, The machine learning model provides location coordinates or a mask image that marks the grid points.

5. The computer-implemented method according to any one of claims 3-4, The machine learning model described therein provides image-to-image conversion.

6. The computer-implemented method according to claim 3 or 5, The machine learning model is trained to specifically locate grid points in the grid array that define the corners of the region.

7. The computer-implemented method according to any one of claims 3-6, in, The grid array contains first-class grid points that define the corners of the region. The grid array includes a second type of grid points with undefined angles. The positioning of the grid array includes distinguishing between the first type of grid points and the second type of grid points.

8. The computer-implemented method according to any one of claims 3-7, Another model (520) locates the region based on the analysis (980, 985) of the neighborhood relationships between the located grid points.

9. The computer-implemented method according to claim 8, The other model therein compares a candidate region defined by a default number of positioned grid points with the default shape and / or default relative position of the region to filter the region from the candidate regions.

10. The computer-implemented method according to any one of claims 3-9, Another model includes an iterative fit between the default values ​​of the grid array and the located grid points.

11. The computer-implemented method according to any one of the preceding claims, The positioning of the grid array includes positioning (952) the area.

12. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - The microscope image is color-calibrated based on the color of the grid array (915).

13. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - The microscope image is rescaled based on the size of the grid array in the microscope image (935).

14. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Control the (910) microscope to autofocus based on the imaging of the grid array in another microscope image.

15. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Register (942) the arrangement of the structure to the reference coordinate system based on the arrangement of the structure and / or the positioning of the grid array.

16. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Determine (906) the default value of the grid array based on the type of the sample carrier, wherein the positioning of the grid array is based on the default value of the grid array.

17. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Based on the quantitative determination, statistical data on the arrangement characteristics of the structure are generated according to the position of the structure in the multiple regions.

18. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - A pre-analysis is performed on the arrangement of the structure, wherein the image analysis is performed based on the results of the pre-analysis.

19. The computer-implemented method of claim 18, wherein the method further comprises: - Based on the results of the pre-analysis, the region of the sample is masked (945). The image analysis within the masked area differs from the image analysis outside the masked area.

20. The computer-implemented method according to claim 2 and claim 18 or 19, The pre-analysis includes identifying cell clumps. Different models were used to count cells inside and outside the cell clusters.

21. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Control (965) graphical user interface to display the results of the image analysis.

22. An electronic data processing apparatus (680) comprising a processor (681) and a memory (682), wherein the processor (681) is adapted to load and execute program code from the memory (682), wherein, based on the execution of the program code, the processor (681) is adapted to perform the following steps: - Acquire microscope images captured by a microscope, the microscope images showing the arrangement of structures on a sample carrier having a grid array that defines multiple regions with predetermined extents. - Use a machine learning model to locate the grid array in the microscope image, and - Based on the positioning of the grid array, image analysis is performed on the microscope image to quantitatively determine the characteristics of the structure arrangement according to the position of the structure in the multiple regions.

23. The electronic data processing apparatus (680) according to claim 22, wherein, based on the execution of the program code, the processor (681) is adapted to perform the method according to any one of claims 1-21.

Images