COMPUTER-IMPLEMENTED TRANSFECTION ANALYSIS

Abstract

Several examples of the disclosure relate to transfection analysis of cells depicted in a microscope image. Various techniques are disclosed for determining a cell-specific transfection rate or a scene-global transfection rate.

Description

TECHNICAL AREA

[0001] Various examples of the disclosure concerning transfection analysis using one or more microscope images. Several examples relate in particular to computer-implemented automation of transfection analysis. BACKGROUND

[0002] One application of microscopy is the study of cells. In particular, microscopic images depicting a scene with cells can be analyzed to determine the degree of cell transfection. To fluorescently label a specific cellular protein, the corresponding gene is linked to the gene sequence of a fluorescent protein (genetic fusion). When the modified gene is taken up into the cell (transfection), the cell expresses the fusion protein consisting of the target protein and the fluorophore. For example, fluorescence imaging can then be used to visualize whether the protein is present inside the cell or not. It is possible to determine for an individual cell whether or not transfection has occurred.

[0003] In reference implementations, the evaluation of corresponding microscope images is done manually and is therefore time-consuming and subjective. SUMMARY

[0004] Therefore, there is a need for improved techniques for transfection analysis. In particular, there is a need for automated techniques that allow the robust and automated determination of the transfection level for cells based on microscope images.

[0005] This task is solved by the features of the independent patent claims. The features of the dependent patent claims define embodiments.

[0006] A computer-implemented method involves obtaining one or more microscope images depicting a scene containing cells. The method also includes performing an initial image analysis based on at least one of the images. This initial analysis yields an instance segmentation mask for the cells. Furthermore, the method includes performing a second image analysis, also based on at least one of the images. This second analysis yields cell-specific outcome data for the scene. The cell-specific outcome data indicates the cell-specific transfection rate for a fluorescent dye-based transfection of the cells.

[0007] A computer-implemented method involves obtaining one or more microscope images. These images depict a scene containing cells. The method includes performing an initial image analysis. This initial analysis is based on at least one of the images. Based on this analysis, a vector field map is generated. This map maps each of several image regions to a corresponding reference image region. These reference image regions are associated with different cells. The method further includes performing a second image analysis. This second analysis, based on at least one of the images and the vector field map, yields cell-specific results for the scene.The cell-specific result data indicate a cell-specific transfection level for a transfection of cells based on a fluorescent dye.

[0008] A computer-implemented method involves obtaining one or more microscope images. These images depict a scene containing cells. The method also includes performing an initial image analysis. This initial analysis is based on at least one of the images. Based on this initial analysis, cell-specific positional information for the cells is obtained. The method also includes performing a second image analysis, again based on at least one of the images. Based on this second analysis, cell-specific outcome data for the scene are obtained. This cell-specific outcome data indicates the cell-specific transfection rate for a fluorescent dye-based transfection of the cells.

[0009] A computer-implemented method involves obtaining one or more microscope images. These images depict a scene containing cells. The method also includes processing these images in a machine-learned model. The machine-learned model provides an image-to-scalar transformation. A scalar output from the machine-learned model is indicative of the scene-global transfection rate for a fluorescent dye-based transfection of the cells.

[0010] A computer-implemented method involves obtaining one or more microscope images depicting a scene containing cells. The method further includes processing these images with at least one machine-learned model to generate a first density map and a second density map. The first density map locates transfected cells, while the second density map locates non-transfected cells. The method also includes comparing the first and second density maps to determine a scene-global transfection rate for fluorescent dye-based cell transfection.

[0011] A computer-implemented method involves obtaining a first microscope image. This first image depicts a scene with cells using a first contrast. The method also involves obtaining a second image, which is registered with the first image. This second image also depicts the scene with cells, but with a second contrast that differs from the first. This second contrast specifically represents a fluorescent dye. The method further includes determining a first confluence mask based on the first image and a second confluence mask based on the second image. Finally, the method includes determining a scene-global transfection rate for fluorescent dye-based cell transfection by comparing the first and second confluence masks.

[0012] The features set out above and those described below can be used not only in the corresponding explicitly set out combinations, but also in further combinations or in isolation, without leaving the scope of protection of the present invention. BRIEF DESCRIPTION OF THE FIGURES Fig. Figure 1 is a flowchart of an exemplary procedure which includes performing an image analysis to determine a degree of transfection. Fig. Figure 2 is a flowchart of an exemplary implementation for performing image evaluation from Fig. 1. Fig. 3 and Fig. Figure 4 shows flowcharts of another exemplary implementation for performing image evaluation. Fig. 1. Fig. Figure 5 is a flowchart of an exemplary implementation for performing image evaluation from Fig. 1. Fig. Figure 6 schematically illustrates a microscope image. Fig. Figure 7 schematically illustrates an instance segmentation mask. Fig. 8 and Fig. Figure 9 schematically illustrates a vector field map. Fig. Figure 10 schematically illustrates position information. Fig. Figure 11 schematically illustrates an electronic data processing device according to various examples. Fig. Figure 12 schematically illustrates data processing according to various examples described herein. DETAILED DESCRIPTION

[0013] The properties, features and advantages of this invention described above, as well as the manner in which they are achieved, will become clearer and more easily understood in connection with the following description of the exemplary embodiments, which are explained in more detail in conjunction with the drawings.

[0014] The present invention is explained in more detail below with reference to preferred embodiments and the drawings. In the figures, identical reference numerals denote identical or similar elements. The figures are schematic representations of various embodiments of the invention. Elements depicted in the figures are not necessarily shown to scale. Rather, the various elements depicted in the figures are represented in such a way that their function and general purpose are understandable to a person skilled in the art. Connections and couplings between functional units and elements shown in the figures can also be implemented as indirect connections or couplings. A connection or coupling can be implemented as a wired or wireless connection. Functional units can be implemented as hardware, software, or a combination of hardware and software.

[0015] The following describes techniques for the image analysis of one or more microscope images. The microscope images are analyzed to provide a transfection analysis, meaning the degree of transfection is determined.

[0016] The degree of transfection can be determined generally for individual cells of a scene or scene-global for the scene.

[0017] When the transfection level is determined for individual cells, it can be a binary transfection level (for example, transfection "yes / no"); however, it is also possible for such a cell-specific transfection level to indicate the probability of protein expression for a particular cell. For example, it is determined whether the cell belonging to the nucleus expresses the dye, does not express it, or whether errors have occurred. Regression to an expression efficiency value (e.g., between 0 and 100) is also possible.

[0018] A global transfection rate can also be determined. Such a global transfection rate can indicate the proportion of transfected cells among all cells in a scene.

[0019] The two options described above are also summarized in Table 1: TABLE 1: Differentiation between local or cell-specific and global transfection levels. Cell-specific transfection rate Indicates for each cell whether that cell is transfected - for example, with a certain probability. Global transfection rate Indicates what proportion of cells in a scene are transfected.

[0020] Transfection analysis uses one or more microscope images. Microscope images can be acquired using different imaging modalities. This means that microscope images with varying contrast levels can be used to determine the degree of transfection using the different techniques described herein.

[0021] In particular, multichannel images can be acquired, comprising several microscope images depicting the scene with cells exhibiting varying contrasts. For example, specific and non-specific contrasts can be combined in a single multichannel image.

[0022] A specific contrast marks certain cell structures in a specific way. For example, a specific fluorescent contrast can be used, or a specific contrast without a fluorescent label. One or more fluorescent contrasts can be used that are particularly helpful in transfection analysis. These are, in particular, fluorescent contrasts that can be used specifically for observing transfected cells. A specific fluorescent contrast is used to visualize the fusion protein; this is subsequently referred to as the transfection fluorescent contrast.

[0023] H&E can be used to visualize the general morphology of cells. DAPI (4',6-diamidine-2-phenylindole) is a fluorescent contrast that labels the cell nucleus. Phalloidin is a fluorescent contrast that labels the actin filament network within a cell.

[0024] A non-specific contrast would be, for example, a phase contrast or a bright-field contrast. For instance, a non-specific contrast could be a phase-like contrast. A phase-like contrast can be, for example, a phase contrast. Examples include Zernike phase contrast and Normarski phase contrast. Here, special optical elements are used in the light path, such as a phase ring in the objective lens and an annular diaphragm in the condenser lens. In this way, interference between background and object light can be visualized. By using a phase contrast, the image contrast can be increased. This means that the cell structures are particularly well visible. Cells are phase objects that cause no or no significant reduction in the amplitude of the light as it passes through the cell sample, so phase contrast is preferred for visualizing the phase shift.However, digital phase contrast can also be used as a phase-like contrast. Here, several images are acquired and then computationally combined into a single phase contrast image. Therefore, such techniques can be referred to as digital phase contrast. The phase contrast is obtained through digital post-processing of the acquired intensity images. Examples include the Transport of Intensity Equation (TIE) and differential phase contrast (DPC). TIE is described in: Streibl, Norbert. “Phase imaging by the transport equation of intensity.” Optics communications 49.1 (1984): 6-10. DPC is described in: Mehta, Shalin B., and Colin JR Sheppard. “Quantitative phase-gradient imaging at high resolution with asymmetric illumination-based differential phase contrast.” Optics letters 34.13 (2009): 1924-1926. To acquire a TIE dataset, the sample is moved along the optical axis (z-direction), i.e.,The sample is axially shifted, and a so-called z-stack, consisting of at least two images, is acquired. The data are then processed to obtain a phase-contrast image. This involves solving a diffusion-type partial differential equation. In DPC, the sample is illuminated from at least two different directions (oblique illumination) while remaining at a fixed z-position. Possible sources for oblique illumination include all types of segmented sources; examples are segmented diodes, LED arrays, digital micromirror devices (DMDs), liquid crystal displays (LCDs or SLMs), or variable condenser apertures. The acquired data are then transformed into a phase-contrast image by solving a deconvolution problem. Combinations of TIE and DPC are also conceivable, as described, for example, in European patent application 24 184 623.7 dated June 26, 2024.The use of digital phase contrast (as opposed to hardware-based phase contrast) has the advantage that no complex process of inserting or removing objects from the light path is necessary when capturing the digital phase contrast. Instead, the illumination can be varied precisely, for example, using a switchable LED array positioned in the illumination pupil plane. This can be done quickly and easily.

[0025] Non-specific contrast types include "label-free" contrasts such as phase contrast, DIC contrast, or TIE contrast, which allow observation of cell structures without specific labels. Alternatively, stains such as H&E can be used to specifically label certain cell structures. Another option is the use of fluorescent stains, such as DAPI, to fluorescently label specific cell components. Autofluorescence can also be used to localize cells without additional labels.

[0026] It is also possible that multiple channels are used to display different fluorescent dyes. In this case, separate predictions can be made for each channel, or a single prediction can be made for all channels.

[0027] In one example, the use of just one channel or a single contrast agent allows for the simultaneous localization of all cells and the determination of the transfection rate. In this case, cell localization can be achieved, for example, via autofluorescence.

[0028] While solutions are described specifically for 2D image data, the techniques described herein can also be used on 3D image data (e.g., from light-sheet microscopy). Fig. Figure 1 shows a flowchart of an example procedure. The procedure is from Fig. 1 can be performed, for example, by an electronic data processing device. The electronic data processing device can include a processor and memory. The processor can load and execute program code from memory. When the processor executes the program code, this causes the processor to complete the procedure. Fig. 1 executes.

[0029] Box 905 receives one or more microscope images. For example, a microscope can be controlled to capture the corresponding images. The microscope images can be received from the microscope. Alternatively, the microscope images can be loaded from memory, such as an image database.

[0030] If multiple microscope images are obtained, they can collectively depict a scene with cells. These multiple microscope images can be part of a single multichannel image. However, it is also conceivable that the multiple microscope images could be acquired sequentially, for example, after individual staining cycles in which specific cells are stained, destained, or otherwise manipulated.

[0031] It is assumed below that at least two microscope images are obtained in Box 905: a first microscope image suitable for the detection of cells (hereinafter referred to as the reference microscope image); and a second microscope image showing the transfection fluorescence contrast.

[0032] The reference microscope image preferably shows all cells with a high contrast ratio against the background. Typically, phase contrast can be used for the reference microscope image, for example, in digital phase contrast. In principle, it would be conceivable to use more than one reference microscope image, with the different reference microscope images then exhibiting different contrasts. For example, a first reference microscope image could be used with brightfield contrast and a second reference microscope image with phase contrast.

[0033] The microscope images from Box 905 are registered together in the optional Box 910. If multiple microscope images are used—for example, as channels of a multi-channel image or acquired entirely separately—the multiple images can be registered. Registration can be image-based or point cloud-based, using localization results. In point cloud-based registration, results from image analysis can be used to create point clouds, which are then used for registration. For example, a first point cloud can be created based on center point detections of the cells in the first channel, while a second point cloud maps local maxima in the fluorescence channel. Registration can then be performed by applying an algorithm such as Iterative Closest Point to determine the relative positioning of the images.

[0034] However, it is also conceivable that the microscope images from Box 905 were already registered beforehand and the corresponding registration parameters already exist. In that case, Box 910 does not need to be executed.

[0035] Microscope images can also be inherently registered with each other, meaning the same pixels represent the same object points in the scene. Such inherent registration of microscope images can occur, for example, when the different microscope images are part of a multichannel image. Typically, the microscope's imaging system is only slightly modified between capturing different microscope images of a multichannel image, for instance, by inserting or removing a color filter from the beam path. Therefore, corresponding microscope images are often already inherently registered with each other (for example, when color aberrations are low).

[0036] The microscope images can optionally be scaled in Box 915. For example, scaling can be performed so that the microscope images subsequently depict the cells at a specific magnification. In other words, the scaling can be done so that the cells in the microscope images have a certain size. This size can be predefined. In particular, the size can correspond to that of cells in reference microscope images, which is used to train one or more machine-learned models that are subsequently used for image analysis. This reduces the complexity of the corresponding machine-learned models, as they only expect cells of a specific magnification. The training effort for such machine-learned models can thus be reduced.The training data does not need to include cells with different image scales, but can be limited to cells with the specified image size.

[0037] Scaling in Box 915 can be done manually, for example. However, it would also be conceivable to use a machine-learned model for scaling in Box 915. For instance, a machine-learned model could be used that performs a picture-to-picture transformation, i.e., outputs the rescaled image. Alternatively, a machine-learned model could be used that outputs a scaling factor, rather than performing a picture-to-scalar transformation. An exemplary rescaling technique is described in principle in EP 4 053 805 A1. The corresponding techniques are included therein by cross-reference.

[0038] Optionally, one or more of the microscope images from Box 905 can be cleaned in Box 920. This allows for the cleaning of microscope images with fluorescence contrast, particularly transfection fluorescence contrast. For example, dirt can be removed. Dye residues, etc., that could potentially cause a fluorescence signal can be removed. Alternatively or additionally, background correction can be performed, for example, if there is an offset on the data.

[0039] In Box 930, image processing takes place of one or more microscope images from Box 905. Fig.Table 1 shows several variants for corresponding image processing. These image processing variants correspond to Boxes 931-936. All of the different image processing variants serve to determine result data that are indicative of the degree of cell transfection. Specifically, the image processing variants according to Boxes 931, 932, and 933 concern the determination of a cell-specific degree of transfection; while the image processing variants according to Boxes 934, 935, and 936 concern the determination of a global degree of transfection (see Table 1).

[0040] Boxes 931-936 employ fundamentally different approaches to determine the degree of transfection. While each yields a global or cell-specific transfection degree, the various image processing methods exhibit different strengths and weaknesses. Accordingly, it is conceivable that, for example, different image processing methods are selected depending on the available contrast of the microscope images from Box 905. For instance, the implementations in the various boxes differ not only in terms of the information content of the resulting data (scene-global transfection degree versus cell-specific transfection degree), but also, alternatively or additionally, in terms of their fundamental algorithmic implementation.While some techniques determine the outline or extent of cells, others are pixel-based or at least based on cell-independent image areas such as superpixels or rectangles of a specific size. Still other techniques perform cell localization but use approximation for the extent of the various cells. The implementation effort for the different techniques can vary significantly, especially when machine-learned models are used for the algorithmic implementation. This is because these machine-learned models are then trained on different training data, and the effort required to create or annotate this training data can vary considerably.Furthermore, the different machine-learned models exhibit varying robustness to variations in the contrasts used, depending on the level of detail in the output of the machine-learned model.

[0041] The different variants of image processing according to Box 931, Box 982, Box 933, Box 934, Box 935 and Box 936 are described one after the other below.

[0042] Box 931 concerns the determination of a cell-specific transfection level based on an instance segmentation mask. A corresponding implementation of Box 931 is described in Fig. 2 shown.

[0043] Fig. 2 is a flowchart which shows a procedure for implementing Box 932 from Fig. 1 illustrated. In the process from Fig.2. A cell-specific transfection degree is determined (see Table 1). Preferably, two microscope images are evaluated. A first microscope image allows for instance segmentation of the cells in Box 1005; while a second microscope image shows the fluorescent dye on which the transfection is based and is used to determine the transfection degree in Box 1010. Thus, the reference microscope image from Box 905 can be used in Box 1005; and the microscope image with transfection fluorescence contrast can be used in Box 1010.

[0044] In Box 1005, an initial image evaluation is performed based on at least one microscope image; an instance segmentation mask is determined in the process.

[0045] The instance segmentation mask typically contains a binary representation of the cells, where each cell is marked as either present (1) or absent (0). This mask can then be used to locate the cells in a microscope image and analyze their properties, such as size, shape, and texture. The instance segmentation mask thus delineates different cells from each other and from the background. Different mask regions of the instance segmentation mask are assigned to different cells. For each pixel of a microscope image, it can therefore be directly determined whether that pixel belongs to a cell and, if so, which cell that pixel belongs to.

[0046] Box 1005 presents several methods for determining the instance segmentation mask. One approach involves using optimization techniques to extract the instance segmentation mask based on cell center locations and confluence masks. Alternatively, Dijkstra algorithms can be employed using the original images and cell center locations as seed points to generate the instance segmentation mask. Additional microscope images from Box 905, such as those with autofluorescence contrast, can be used to aid in distance determination. Derived images, such as probability or density maps for cell wall presence, can also be used. Another method for determining the instance segmentation mask is a graph-cut approach.Another option is to use a machine-learned model, such as multi-stage detectors or single-shot detectors. Transformer-based models can also be used for instance segmentation.

[0047] In Box 1010, a second image analysis is performed. The degree of transfection is determined for each cell in a corresponding microscope image. This means that the success of the transfection is determined for the different cells.

[0048] Box 1010 can, in particular, build upon Box 1005; i.e., the instance segmentation mask can serve as input for image evaluation in Box 1010.

[0049] In particular, the instance segmentation mask can be overlaid with the microscope image using transfection fluorescence contrast, and then an evaluation of the pixel values ​​of this microscope image can be performed in the different mask areas associated with the different cells. The corresponding techniques are explained below.

[0050] In one approach, pixel values ​​in a microscope image depicting the fluorescent dye for transfection are aggregated in the various mask regions of the instance segmentation mask. This integrated intensity can then be compared to a threshold value. This aggregation of fluorescence intensities for a cell can be performed in several ways. One possibility is to sum the intensities, which can also be weighted, for example, according to confidence or distance from the cell center. Alternatively, the intensities can be masked beforehand or converted into an (optionally binary) intensity histogram by comparison with one or more threshold values.

[0051] The thresholds that determine whether a cell is transfected or not can be defined in various ways. One possibility is for the user to interactively define the precise threshold between "transfected" and "non-transfected." This threshold can, for example, be generated on a "calibration microscope image" from Box 905 and then applied to further microscope images. More generally, in Boxes 931, 932, and 933, decision limits for a corresponding classification to determine the degree of transfection can be set based on user input.

[0052] Another method for threshold determination involves using a masked range to transform each cell into a token—that is, a feature vector in a machine-learned feature space—via an embedding model. These feature vectors can then be evaluated. For example, the feature vectors for a set of cells are fed into a transformer model, and the threshold (or more generally, the decision boundary of a corresponding classification) is determined using the so-called CLS token. A transformer model is a machine-learned model specifically designed for processing sequences of data. It comprises multiple layers, each employing a self-attention mechanism to extract the features between different positions in the data sequence. The CLS token is a special element within the architecture of transformer models.The CLS token is typically inserted at the beginning of an input sequence and serves to capture a comprehensive representation of the entire input. After the input has been processed by the transformer model, the output vector at the position of the CLS token represents a condensed representation of all elements of the sequence. This representation is then used to perform the classification into transfected and non-transfected.

[0053] Another possibility is to calculate statistics such as mean, standard deviation, or skewness for the image pixel intensities in the microscope image with transfection fluorescence contrast and in the different mask areas. Histograms, either in one dimension, two dimensions, or radially radiating from the cell nucleus, can also be used to aggregate the fluorescence intensities. A statistical analysis can then be performed, for example, to identify peaks and assign them to "transfected" and "non-transfected" status.

[0054] Another possibility involves the unsupervised determination of the transfection degree: for example, a feature vector in a machine-learned feature space (also referred to as "embedding") is used, which encodes the image sections defined by the instance segmentation masks in the microscope image with transfection fluorescence contrast. Subsequently, the feature vectors of all cells in the feature space are clustered. Unlike the variants described above, a predefined threshold is then not required. In other words, mask regions of the instance segmentation mask in the appropriate microscope image are encoded in a machine-learned feature space to obtain corresponding cell-specific feature vectors for each mask region. Clusters of these feature vectors can then be identified in the machine-learned feature space using a suitable clustering algorithm. The assignment or...The membership of the feature vector in the different clusters then determines whether or not transfection has occurred. Different clusters can therefore correspond to different degrees of transfection. Instead of clustering in the machine-learned feature space, the determined statistics of the fluorescence signal (see above) can also be used directly for clustering.

[0055] In the various variants above, the fluorescence signal in the microscope image is evaluated for each instance (i.e., each cell) using the transfection fluorescence contrast within the corresponding mask area.

[0056] It would also be conceivable for Box 1005 and Box 1010 to be executed by a single machine-learned model. In such a case, it is not strictly necessary for the instance segmentation mask to be used as input for determining the cell-specific transfection level. In other words, it would be conceivable for the instance segmentation mask to be determined in an output branch of the machine-learned model, while in another branch of the machine-learned model the cell-specific transfection level for each cell is determined (and, for example, marked at a cell center). Box 1005 and Box 1010 can therefore be executed in parallel.

[0057] An example would be a Mask Region-based Convolutional Neural Network (Mask R-CNN), a machine-learned model used for object segmentation. A Mask R-CNN model comprises two main components: a backbone network and a header network. The backbone network extracts features from the input image, while the header network uses these features to segment the instances. The model operates in two stages: First, the backbone network is used to create a feature map that encodes the relevant features of the input image. Then, the header network is used to process this feature map and determine the instance segmentation mask. In parallel, the cell-specific transfection level can also be derived from the feature map.

[0058] The preceding was discussed in connection with Fig.Section 2 or Box 931 explains a variant in which an instance segmentation mask allows for the direct assignment of different pixels of a microscope image to different cells. This is achieved by using the mask areas of the instance segmentation mask. Referring again to Fig. 1: A fundamentally different approach is provided in Box 932. There, it is not possible to directly determine which cell a specific pixel corresponds to in the microscope image by comparing its position with mask regions of the instance segmentation mask. Instead, a vector field map is used that maps each of several image regions to a corresponding reference image region. Different reference image regions are then associated with different cells.

[0059] Fig. 3 is a flowchart which shows a procedure for implementing Box 932 from Fig.1 illustrated. In the process from Fig. 3. A cell-specific degree of transfection is determined. Preferably, two microscope images are evaluated. A first microscope image allows the determination of a vector field map in Box 1105; while a second microscope image shows the fluorescent dye on which the transfection is based, i.e., which is to be expressed, and is used to determine the degree of transfection in Box 1110. Thus, the reference microscope image from Box 905 can be used in Box 1105; and the microscope image with transfection fluorescence contrast can be used in Box 1110.

[0060] In Box 1105, an initial image analysis is performed based on at least one microscope image; this involves determining a vector field map. The vector field map maps different image areas—for example, pixels or superpixels—to a corresponding reference image area. Different reference image areas are then associated with different cells. A superpixel is an image area in a microscope image that contains a group of adjacent pixels. This group of pixels is treated as a unit and can be represented by a single vector in the vector field map.

[0061] The vector field map can comprise two or three output channels, depending on whether the image data is 2D or 3D. For example, the vector field map can be represented in Cartesian coordinates as x-channels and y-channels, or in polar coordinates as angle channels and distance channels. The vector field map can be defined for individual pixels of the microscope images. Instead of a vector field map defined for individual pixels, the method can also be applied to larger image areas, i.e., image areas that encompass multiple image pixels. For example, regularly shaped regions, such as squares or rectangles, can be used as image areas. So-called "superpixels" can also be used.

[0062] Optionally, an additional output channel can be used that indicates the confidence level for the respective mapping of image areas to the reference image areas. This makes it possible to ignore uncertain image areas or those not belonging to cells when determining the degree of transfection in Box 1110.

[0063] Mapping pixels from the microscope image to vectors in the vector field map can be accomplished using a machine-learned model. Examples include CNN-based image-to-image networks, such as UNet. Another possibility is the use of transformer-based image-to-image networks, such as ViTMAE. Hybrid networks, such as VQGAN, can also be used.

[0064] One advantage of determining the vector field map using a machine-learned model is that the model does not need to operate based on fluorescence contrasts. For example, a microscope image used as input to determine the vector field map can have a specific phase contrast—such as a digital phase contrast. This allows for particularly robust training of the machine-learned model, which is invariant to variations in fluorescence contrast (for example, due to the use of different dyes for different transfection experiments).

[0065] The second image analysis uses the microscope image with the transfection fluorescence contrast as input, but can also include additional images. For example, one or more further microscope images can be used. Another image that can be used in the image analysis in Box 1110 is a microscope image that labels a specific cell structure for each cell. An example would be a microscope image with a specific DAPI contrast showing the cell nuclei. Instead of cell nuclei, other cell structures can also be labeled—such as the cell center. Preferably, these are cell structures that can be uniquely located for each cell. Based on these cell structures, which are specifically labeled in this additional microscope image, the reference image regions are then determined, onto which the various image regions are mapped by the vector field map.

[0066] As an optional post-processing step, the predicted reference image areas can be combined or consolidated to eliminate any variances in their spatial assignment. For example, all reference image areas within a certain radius can be combined into a single reference image area, and the respective vectors can be corrected accordingly. Other neighborhood relationships can also be considered during such consolidation.

[0067] In Box 1110, a second image analysis is performed. The degree of transfection is determined for each cell in a corresponding microscope image. This means that the transfection success for the different cells is determined. The vector field map determined in Box 1105 is taken into account.

[0068] One way to implement Box 1110 is in Fig.Figure 4 illustrates this process. The process iterates over all image areas distinguished in the vector field map—these can be, for example, pixels, superpixels, or other areas. Box 1205 selects the current image area for a specific iteration (1299). Then, the intensity value of all pixels in the microscope image exhibiting the fluorescence contrast relevant for transfection can be added to a counter value in the corresponding reference image area within the current image area (Box 1215). In an optional variant, however, Box 1215 is only executed if it is previously specified in Box 1210 that this image area should be considered. When determining the cell-specific degree of transfection, it can be helpful to ignore image areas that cannot be assigned to a cell. This can be done in various ways.One possibility is that, in the simplest case, no signals outside of cells are present in the microscope image with the transfection fluorescence contrast. In this case, the value 0 is added if the image region contains pixels outside of cells. However, it is sometimes more accurate to explicitly exclude image regions outside of cells from the summation. One way to do this is to iterate only over image regions that lie within a given confluence mask. The confluence mask can be determined using generally known techniques, for example, by means of a machine-learned model.

[0069] In Fig.Figure 4 shows a variant where the intensity values ​​are added to the counters of the reference image areas iteratively. It would also be possible to achieve this addition of intensity values ​​through matrix multiplication. In this approach, a matrix representing the microscope image can be multiplied by another matrix determined based on the vector field. While such matrix multiplication can be implemented in various ways using an iterative software algorithm, it is also conceivable that the matrix multiplication could be implemented using suitable parallel processing hardware, such as a graphics card – thus enabling hardware acceleration. Hardware multiplication can therefore be hardware-accelerated. This can be particularly helpful for very large images (such as mosaic images composed of a tile scan).

[0070] Once all image areas have been iterated over (see check in Box 1220), Box 1305 can then be executed. In Box 1305, a current reference image area is selected from the set of all reference image areas; 1399 iterations are then performed over all reference image areas. A threshold comparison of the respective counter (the counter value is determined from the 1299 iterations of Box 1215) with a predefined threshold (e.g., from user input) is then performed in Box 1310 to determine whether the respective cell has been transfected or not. In Box 1315, another 1399 iterations are performed for a further reference image area, if one exists.

[0071] The preceding was discussed in connection with Fig. 3 and Fig.Section 4 or Box 932 explains a variant in which a cell-specific transfection rate can be determined using a vector field map. Referring again to Fig. 1: A fundamentally different variant is provided in Box 933. In Box 933, cell-specific position information is determined for the various cells. A corresponding procedure is described in Fig. 5 shown.

[0072] Fig. 5 is a flowchart which shows a procedure for implementing Box 933 from Fig. 1 illustrated. In the process from Fig.5. A cell-specific degree of transfection is determined. Preferably, two microscope images are evaluated. A first microscope image allows the determination of positional information in Box 1405; while a second microscope image shows the fluorescent dye on which the transfection is based, i.e., the fluorescent dye to be expressed, and is used to determine the degree of transfection in Box 1410. Thus, the reference microscope image from Box 905 can be used in Box 1405; and the microscope image with transfection fluorescence contrast can be used in Box 1410.

[0073] In Box 1405, positional information for the cells is determined (localization). Locating each cell can be achieved by determining its center, nucleus, or other characteristic points. This positional information can also be enriched with a kind of "diameter" representing the approximate size of the cell. If prior scaling has taken place (see below), the positional information can be further refined. Fig. 1: Box 915) so it is unnecessary to specify a diameter or other size information for the approximate dimensions of the cells.

[0074] The technical implementation of localization can be achieved in various ways. One example is the use of a model that provides a picture-to-picture mapping, where the output image is a density / probability map representing the presence of cell centers. Another implementation option for the positional information is a list of 2D or 3D coordinates, optionally including the size of the enclosing bounding box. Such lists can be provided by detection models. Examples of such models include RCNN, YOLO, SSD, or transformer-based detection models.

[0075] In Box 1410, the cell-specific transfection rate is determined for each cell located using positional information. The cell-specific transfection rate can be determined in various ways.

[0076] In particular, it is possible that Box 1410 builds upon the result of Box 1405. This means that the positional information from Box 1405 is used to determine the cell-specific transfection rate in Box 1410. Such a technique is described below.

[0077] One possibility is to use a radius-based calculation. Here, the fluorescence signal in the microscope image is aggregated with transfection fluorescence contrast within a radius around the point specified by the position information (e.g., the cell center). Optionally, a confluence mask can be used to crop areas outside the cell. The image can also be distorted before processing so that cells are isotropic, i.e., circular. In general terms, it is therefore possible to determine line-specific image areas based on the cell-specific position information. These can be determined, for example, depending on prior knowledge about the shape and / or size of the cells in the respective microscope image. The assignment of image areas to a confluence mask can also be taken into account. The image evaluation in Box 1410 can then be performed cell-specifically for these image areas.

[0078] After aggregating the fluorescence signal in the image areas of the microscope image with transfection fluorescence contrast, statistics such as mean and variance can be calculated. These statistics can be used to determine whether a cell is transfected, non-transfected, or unusable. A clustering algorithm can be applied. Optionally, the evaluation can also be performed by considering the values ​​of other cells. Corresponding techniques have already been described above in connection with Fig. 2 explained and can also be applied here.

[0079] Another possibility is the use of an instance classifier without segmentation. Here, an image region is extracted from the fluorescence-contrast microscope image around each identified cell center (or other reference point specified by the position information), and then a classification is performed for each extracted image region. The classification result then indicates the degree of transfection. For example, it determines whether the cell belonging to the nucleus expresses the dye, does not express it, or whether errors have occurred. Regression to an expression efficiency value (e.g., between 0 and 100) is also possible. Such image regions for the classification or regression model can have a specific predefined size (for example, as prior knowledge about the size of the cell in the microscope images, as discussed above in connection with Box 915).However, it would also be conceivable to determine such image areas using a detection model. This detection model is applied to patches (i.e., other image areas) that are positioned within the respective microscope image based on cell-specific positional information. Here, instance classifiers are used with the aid of detection models. The detection model either provides the image area, which is then fed into a separate classification model, or the classification is already part of the detection model.

[0080] Techniques have been described above in which a classification or regression value is determined using a machine-learned model, indicating the degree of transfection for each cell. However, it would also be conceivable to define an image area (which, as described above, can be determined using a machine-learned detection model or have a fixed, predefined size) based on the positional information for each cell. The image pixels of the microscope image are then encoded with the transfection fluorescence contrast in a machine-learned feature space to obtain corresponding cell-specific feature vectors for each image area. Clusters of these feature vectors can then be identified in the machine-learned feature space using a suitable clustering algorithm. The assignment or...The membership of the feature vector in the different clusters then determines whether or not transfection is present. Different clusters can therefore correspond to different degrees of transfection.

[0081] The approaches described above are variants in which Box 1410 uses the position information from Box 1405. Besides such a two-stage solution with separate determination of the position information in Box 1405 and subsequent determination of the transfection degree in Box 1410, it is also conceivable that both boxes could be implemented in a single machine-learned model. The input to such a machine-learned model would then include, for example, several microscope images, such as one microscope image with digital phase contrast but without specific labeling of individual cell structures, and another microscope image with the fluorescence contrast relevant for transfection.

[0082] For example, in such a variant, the importance of the different microscope images for the various tasks (determining position information on the one hand and determining the degree of transfection on the other) could be regulated by machine-learned weights within the machine-learned model. This means that there is no one-to-one mapping where the reference microscope image—for example, a phase-contrast image—is used to determine position information, and the image with the transfection fluorescence contrast is used to determine the degree of transfection. Rather, it is possible for both microscope images to be used for both tasks, with the relative importance not necessarily being manually specified but instead being machine-learned.

[0083] Several techniques for determining cell-specific transfection levels have been described above. In particular, the following have been discussed in connection with Fig. 2 or Box 931 of the procedure from Fig. 1 and in connection with Fig. 3 or Box 932 of the procedure from Fig. 1 and in connection with Fig. 5 or Box 933 of the procedure from Fig.One technique is described, each using instance segmentation, a vector field map, or positional information for the different cells. In principle, it would be conceivable to use the transfection level determined for each cell to determine a scene-global transfection level. This means, for example, that based on the cell-specific transfection level, it is possible to determine how many cells in the entire scene express the dye and how many do not, and then calculate a scene-global transfection level from this. Besides such techniques, which determine the cell-specific transfection level, techniques are also conceivable in which the global transfection level is determined directly instead of the cell-specific transfection level (see also Table 1). Referring to Fig.1: Three such techniques are illustrated in connection with Box 934, Box 935 and Box 936.

[0084] Box 934 shows a variant in which one or more microscope images are processed in a machine-learned model that provides an image-to-scalar transformation. The scalar output of the machine-learned model indicates the scene-global transfection rate. This method for determining the global transfection rate thus directly predicts the transfection efficiency for multiple cells. Here, the number of transfected and non-transfected cells is predicted directly as a scalar without explicitly localizing individual cells using the model.

[0085] Scalar prediction can be performed patch-wise or on the entire image. The model can predict the number of transfected and non-transfected cells in the image (section) or even directly the ratio of both, i.e., the transfection efficiency itself, using image-to-scalar regression.

[0086] One advantage of the variant described in Box 934 is that the model can be trained with weakly annotated data. This means that no localization annotations are required, but only a number of cells (e.g., the number of transfected cells). A disadvantage, however, is that the result is more difficult for the user to interpret, since only a scalar is predicted, but it is not apparent how the model arrives at this result.

[0087] Box 935 describes another variant. In Box 935, both transfected and non-transfected cells are counted using one or more machine-learned models. For example, a first machine-learned model can be used to count the transfected cells, and a second machine-learned model can be used to count the non-transfected cells. Alternatively, a single machine-learned model can be used for both tasks.

[0088] An image-to-image transformation can be performed. One or more microscope images can be processed to provide a density map in which the transfected cells are located. Additionally, one or more microscope images can be processed to provide another density map in which non-transfected cells are located.

[0089] For example, a single machine-learned model could be used that provides two output channels corresponding to these two density maps. Based on the corresponding density maps, the number of transfected cells and also the number of non-transfected cells can then be determined.

[0090] It is then possible to compare these two density maps to determine a scene-wide transfection rate. For example, the non-transfected cells can be counted using one density map, and the transfected cells can be identified using the other density map. These two counts can then be compared to determine the scene-wide transfection rate.

[0091] Methodologically, Box 935 is therefore a mixture of the solutions according to Box 933 and Box 934.

[0092] A redundant counting technique can be applied to create each density map. This technique is described in principle in: Paul Cohen, Joseph, et al. “Count-ception: Counting by fully convolutional redundant counting.” Proceedings of the IEEE International conference on computer vision workshops. 2017. A deep neural network (typically a convolutional network) is applied multiple times to overlapping image areas to increase accuracy and reduce error susceptibility. Small image areas or patches are repeatedly and independently analyzed to generate redundant localized count results, which are then aggregated to obtain an overall count. The density maps then display redundant count results. The advantage of this technique is that it enables particularly robust localization of the different cells because many local counts are averaged out.On the other hand, corresponding density maps are comparatively difficult to interpret, as they show redundant counting results.

[0093] Another technique for determining a scene-global degree of transfection is shown in Box 936. This technique uses two microscope images depicting the scene with cells at different contrast levels. Specifically, the first image can have a contrast that is non-transfection-specific, meaning it depicts all cells regardless of whether they are transfected or not. In contrast, the second image can depict the cells with the transfection-specific fluorescence contrast.

[0094] Then, two confluence masks can be determined. Specifically, a first confluence mask can be determined based on the first microscope image, and a second confluence mask can be determined based on the second microscope image. This comparison can be performed, for example, using mask difference or IoU (intersection-over-union) calculations. Subsequently, the first confluence mask can be compared with the second confluence mask to determine the scene-global transfection rate for the fluorescent dye-based transfection.

[0095] This technique, as described in Box 936, is particularly useful if cells express the protein under investigation across their entire cross-sectional area. If this is not the case, the confluence masks may contain "holes" and distort the result.

[0096] Several implementation options for Box 930 have been explained above. The execution of Box 930 yields results relating to the transfection rate of cells. For example, a cell-specific transfection rate can be determined (Box 931, Box 932, and Box 933); alternatively or additionally, a scene-global transfection rate can be determined (Box 934, Box 935, and Box 936; but this is also possible by subsequently aggregating the cell-specific transfection rates for the different cells based on Box 931, Box 932, and Box 933).

[0097] Subsequently, a user interface can be accessed in Box 940. In particular, a graphical user interface can be accessed. Information related to the cell-specific and / or scene-global transfection rate can then be output. For example, the user interface could be accessed to output graphical information determined based on at least one of the microscope images from Box 905 and the result data from Box 930.

[0098] In particular, cell-specific results data can be overlaid with one or more microscope images. For example, if a cell-specific transfection rate is available, each transfected cell could be highlighted in a specific way in a microscope image; alternatively or additionally, each non-transfected cell could be highlighted in a different way in a microscope image.

[0099] Even though instance segmentation according to Box 931 is not used in determining the cell-specific transfection level, it would be conceivable to define an instance segmentation mask for the cells in conjunction with Box 925. Then, for example, the cell-specific transfection level could be displayed in Box 925 along with mask areas that mark different cells. This is a particularly easy-to-interpret representation (see also...). Fig. 12, below).

[0100] One example is representing the cells as colored dots. One dot (or other marker) per cell can be used to visualize the transfection results. The dot's position could be, for example, at the cell's center, centroid, or nucleus.

[0101] Another possibility is to represent the results using colors. For example, green, orange, and red can be used to index the classes "transfected," "overexpressed," and "non-transfected."

[0102] It would also be conceivable to display a sorting of the cells according to the probability of transfection, for example superimposed with the respective microscope image.

[0103] The preceding section described several examples of how to display the cell-specific transfection rate. It is also possible to display the scene-wide transfection rate, either alternatively or additionally, for example, alongside a corresponding microscope image. For instance, the following could be displayed: "79% of all cells successfully transfected".

[0104] Box 945 could optionally allow for corrections to the result data. For example, a user could mark certain cells that are currently labeled as transfected as actually untransfected. The image output in Box 940 would then be adjusted accordingly (dashed arrow).

[0105] Another possibility for correcting the result data 945 involves receiving user input regarding the decision threshold for differentiating between transfected and non-transfected cells. For example, techniques have been described above in which a threshold value, which regulates whether a cell is transfected or non-transfected, is determined in a specific way. It would be conceivable that in Box 945, the user could change this threshold value, for example, using a slider, and then interactively see the effect of such a change on the classification result in the image. For example, it could be shown what effect changing the threshold value has on the classification of cells as transfected or non-transfected (or other classification criteria related to transfection).In other words, continuous human-machine interaction can be provided. This continuous human-machine interaction can include, on the one hand, receiving user input related to setting the decision boundary for classification. On the other hand, it can also include displaying the impact of the user input on the classification result to the user. This has the advantage that the user can interactively "sample" the decision boundary and, for example, identify areas of particularly high sensitivity. This makes it possible to define the decision boundary more precisely, especially in relation to the underlying parameters. In other words, continuous human-machine interaction can be provided.This continuous human-machine interaction can, on the one hand, involve receiving user input regarding the setting of the decision boundary for classification. On the other hand, it can also include providing the user with information about the impact of their input on the classification result. This approach has the advantage that the user can interactively "sample" the decision boundary and, for example, identify areas of particularly high sensitivity. These are areas where even a small change to the decision boundary has a particularly large impact on the classification result. Often, knowing about such areas of high sensitivity is helpful in achieving good classification results.

[0106] Based on such user input, further fundamental truths or training data can be collected. Based on these fundamental truths, one or more machine-learned models used in Box 930 could then be retrained.

[0107] There are various variations of the procedure. Fig.One conceivable approach is to vary the order of the different boxes. For example, Box 920 could be executed before Box 915. Furthermore, multiple iterations of Box 930 could be performed, for instance, for several sets of one or more microscope images, or by processing different areas of microscope images separately. This would allow for individual evaluations of different regions of a sample. For example, a scene-global transfection degree could be determined for different wells of a multiwell plate. Certain parameters related to image evaluation in Box 930 could be synchronized between different instances of Box 330. Examples here include, in particular, decision criteria for distinguishing between different transfection degree values.This would ensure that the same decision criteria are applied to the evaluation of different areas of the sample, thus enabling a consistent evaluation across the various areas of the sample.

[0108] Fig. Figure 6 schematically illustrates a microscope image 811. For example, microscope image 811 may show a fluorescence contrast. Three cells are shown in microscope image 811.

[0109] Fig. Figure 7 illustrates an instance segmentation mask 812 (dashed lines) superimposed with the microscope image 811 (compare Box 931 in Fig.1) The instance segmentation mask 812 defines mask regions bounded by the cell boundaries. For example, the instance segmentation mask could have the same resolution as the microscope image 811 and indicate for each pixel of the microscope image whether that pixel lies inside or outside a cell. The instance segmentation mask could also be defined by other structures, such as splines.

[0110] Fig. Figure 8 schematically illustrates a vector field map 815 (compare Box 932 in Fig.1) The dotted lines represent the grid of the vector field map 815. For each entry of the vector field map 815, a vector is provided that maps the respective image area (for example, pixels or superpixels) to a reference image area; this is shown for the upper left cell by the arrows. This means that all image areas belonging to a common cell are mapped to the same reference image area. Fig. Figure 9 shows another vector field map 816, which is basically equivalent to vector field map 815. In one variant of vector field map 816, image areas that lie outside of cells are mapped to random reference image areas (in Fig. (9, shown in the upper right). In another variant, it would be conceivable that image areas lying outside of cells do not have a mapping to reference image areas.

[0111] From a comparison of Fig. 7 with the Fig. 8 or the Fig. As shown in Figure 9, a vector field does not directly assign image regions to cells. Rather, it specifies for each image region (e.g., each pixel) where the corresponding cell center (or another unique anchor point) of the underlying cell is located. This means that the vectors of two image regions of a cell should point to a common point, but these two image regions are not directly linked to each other in the vector field.

[0112] In Fig. 10 is schematic position information 819 (compare box 933 in Fig. 1) represented by a cell center localization (crosses in Fig. 10).

[0113] Fig.Figure 11 schematically illustrates an electronic data processing device 700 according to various examples. The electronic data processing device 700 comprises a processor 705 and a memory 706. The electronic data processing device also includes a communication interface 707. For example, the processor 705 could process one or more microscope images (compare Figure 11). Fig.1: Box 905) receives data from a microscope via the communication interface 707. The processor 705 could send control data to the microscope to initiate the acquisition of microscope images (for example, with specified imaging parameters or specific imaging modalities and contrasts). In another variant, the processor 705 could load the microscope images from local memory 706. The processor 705 can also load and execute program code from memory 706. When the processor 705 executes the program code, this causes the processor to perform techniques as described in its documentation. For example, the processor could perform techniques such as those related to Fig. Execute as described in section 1.

[0114] Fig.Figure 12 schematically illustrates data processing according to various examples discussed herein. A reference microscope image 611 is illustrated, which in the example shown exhibits phase contrast. For example, a digital phase contrast, generated by oblique illumination, can be used. Furthermore, in Fig. 12 also shows a microscope image 612 with a fluorescence contrast that determines a fluorescent dye in relation to the degree of transfection to be determined (transfection fluorescence contrast).

[0115] A comparison of the reference microscope image 611 with the microscope image 612 shows that only some of the cells visible in the reference microscope image 611 show fluorescence in the microscope image 612; only these cells that appear bright in the microscope image 612 are transfected.

[0116] Furthermore, in Fig.12 also shows an algorithm 615, illustrated here schematically by a deep neural network (in principle, however, as described above, different types of algorithms and also several algorithms in combination or individually can be used).

[0117] In Fig. Figure 12 also shows an output image 620, which represents a superimposition of a cell-specific transfection level with the reference microscope image 611. In the example of the Fig. 12 shows that the output image 620 is created based on an instance segmentation of the different cells: the edges of each mask area of ​​the instance segmentation mask are highlighted in a respective color, depending on whether the respective cell is transfected or not (in Fig.12 different colors are illustrated by different line types; solid lines = non-transfected cells; dotted lines = overexpressed cells; dashed lines = transfected cells).

[0118] In summary, the preceding techniques described enable the automated determination of the degree or rate of transfection using image analysis. This method determines which, and how many, cells in a sample express a specific protein as desired.

[0119] Techniques have been described that generally relate to determining the cell-specific transfection rate of cells imaged under a microscope. Such a cell-specific transfection rate can be determined by using at least one machine-learned model that processes one or more microscope images. This is an alternative to the prior art of manual image evaluation.

[0120] Microscope images are acquired using different contrast methods, such as phase contrast and fluorescence contrast. Phase contrast images typically show all cells equally, regardless of whether they are transfected or not. In contrast, fluorescence contrast images show only cells that have been transfected.

[0121] A model can be used to segment each cell in a microscopic image. This is done using an instance segmentation mask that defines mask regions bounded by the cell edges. Each pixel in the image corresponds to a specific region.

[0122] Alternatively or additionally, a vector field map can be used to determine the degree of transfection. A vector field map assigns each pixel in the image to a reference area or location, such as the center of a cell.

[0123] Another alternative is the use of positional information, which, for example, indicates the center of mass of the cells in a microscope image.

[0124] The determined cell-specific transfection rate can then be displayed along with one or more microscope images. In this display, transfected and non-transfected cells can be highlighted, for example, by different colors.

[0125] In addition to displaying the transfection rate for individual cells, it is also possible to determine a scene-global transfection rate, which indicates the percentage of all transfected cells in the image.

[0126] In summary, the following EXAMPLES were described in particular.

[0127] EXAMPLE 1. Computer-implemented method that includes: - Obtained (905) from one or more microscope images depicting a scene with cells, - Performing (931, 1005) an initial image evaluation based on at least one of the one or more microscope images to obtain an instance segmentation mask for the cells includes, and - Performing (931, 1005) a second image evaluation based on at least one of the one or more microscope images to obtain cell-specific result data for the scene, where the cell-specific result data indicate a cell-specific transfection level for a transfection of the cells based on a fluorescent dye.

[0128] EXAMPLE 2. Computer-implemented method according to EXAMPLE 1, where the second image evaluation is further based on the instance segmentation mask.

[0129] EXAMPLE 3. Computer-implemented method according to EXAMPLE 1 or 2, where the first image evaluation and the second image evaluation are performed in a joint machine-learned model.

[0130] EXAMPLE 4. Computer-implemented method according to EXAMPLE 3, where the jointly machine-learned model is a Mask R-CNN convolutional network.

[0131] EXAMPLE 5. Computer-implemented method according to one of the preceding EXAMPLES, wherein the second image evaluation includes a cell-specific aggregation of pixel values ​​in at least one of the one or more microscope images in each mask area of ​​the instance segmentation mask.

[0132] EXAMPLE 6. Computer-implemented method according to one of the preceding EXAMPLES, wherein the second image evaluation includes encoding mask areas of the instance segmentation mask in at least one of the one or more microscope images in a machine-learned feature space, to obtain corresponding cell-specific feature vectors for each image area.

[0133] EXAMPLE 7. Computer-implemented method according to EXAMPLE 6, where the second image evaluation still includes the recognition of clusters formed by feature vectors in the machine-learned feature space, the procedure further includes: - depending on the assignment of the feature vectors to the clusters, determining the cell-specific degree of transfection.

[0134] EXAMPLE 8. Computer-implemented method according to one of the preceding EXAMPLES, wherein the second image evaluation further includes determining latent feature vectors for several mask areas of the instance segmentation mask and based on at least one of the one or more microscope images, where the second image evaluation still includes processing the latent feature vectors in a transformer network to obtain a classification token, the second image analysis includes a classification to determine the cell-specific degree of transfection, where a decision limit for the classification is set depending on the classification token.

[0135] EXAMPLE 9. Computer-implemented method that includes: - Obtained (905) from one or more microscope images depicting a scene with cells, - Performing (932, 1105) an initial image evaluation based on at least one of the one or more microscope images to obtain a vector field map that maps each of several image regions to a corresponding reference image region, wherein the reference image regions are associated with different cells, and - Performing (932, 1110) a second image evaluation based on at least one of the one or more microscope images and the vector field map, to obtain cell-specific result data for the scene, where the cell-specific result data indicate a cell-specific transfection level for a transfection of the cells based on a fluorescent dye.

[0136] EXAMPLE 10. Computer-implemented method according to EXAMPLE 9, wherein the second image evaluation comprises iterating (1299) over the image areas, wherein in each iteration (1299) one or more pixel values ​​of the at least one of the one or more microscope images in the respective image area are added to a counter (1215) associated with the corresponding reference image area.

[0137] EXAMPLE 11. Computer-implemented method according to EXAMPLE 10, the second image evaluation continues to include comparing (1310) the counter values ​​of the different reference image areas with a threshold value to determine the cell-specific degree of transfection.

[0138] EXAMPLE 12. Computer-implemented method according to one of EXAMPLES 9 to 11, the second image evaluation includes matrix multiplication.

[0139] EXAMPLE 13. Computer-implemented method according to one of EXAMPLES 9 to 12, where each image area comprises several image pixels.

[0140] EXAMPLE 14. Computer-implemented method according to one of EXAMPLES 9 to 13, where the vector field map still provides a confidence for the mapping of the respective image area to the corresponding reference image area.

[0141] EXAMPLE 15. Computer-implemented method according to one of EXAMPLES 9 to 14, where the vector field map is determined using a machine-learned model that performs an image-to-image transformation.

[0142] EXAMPLE 16. Computer-implemented method according to one of EXAMPLES 9 to 15, wherein at least one of the one or more microscope images evaluated in the second image evaluation comprises a first microscope image with a fluorescence contrast specific to the fluorescent dye and optionally a second microscope image, wherein the second microscope image has a contrast that specifically marks cell structures of the cells corresponding to the reference image areas.

[0143] EXAMPLE 17. Computer-implemented method according to one of EXAMPLES 9 to 16, wherein the method further comprises: - Consolidating reference image areas in the vector field map based on neighborhood relationships between the reference image areas.

[0144] EXAMPLE 18. Computer-implemented method that includes: - Obtaining one or more microscope images depicting a scene with cells, - Performing (933, 1405) an initial image evaluation based on at least one of the one or more microscope images to obtain cell-specific positional information for the cells, and - Performing (933, 1410) a second image evaluation based on at least one of the one or more microscope images, to obtain cell-specific result data for the scene, where the cell-specific result data indicate a cell-specific transfection level for a transfection of the cells based on a fluorescent dye.

[0145] EXAMPLE 19. Computer-implemented method according to EXAMPLE 18, the second image analysis uses the position information.

[0146] EXAMPLE 20. Computer-implemented method according to EXAMPLE 18 or 19, wherein the cell-specific position information indicates the two-dimensional or three-dimensional position of one or more characteristic points of the cells in the one or more microscope images.

[0147] EXAMPLE 21. Computer-implemented method according to one of EXAMPLES 18 to 20, where the cell-specific position information includes size information for the cells.

[0148] EXAMPLE 22. Computer-implemented method according to one of EXAMPLES 19 to 21, wherein the method further comprises: - based on cell-specific position information, determining cell-specific image areas, wherein the second image evaluation is cell-specific in the cell-specific image areas of the corresponding at least one of the one or more microscope images.

[0149] EXAMPLE 23. Computer-implemented method according to EXAMPLE 22, where the determination of the cell-specific image areas is carried out depending on prior knowledge about the shape and / or size of the cells in at least one of the one or more microscope images.

[0150] EXAMPLE 24. Computer-implemented method according to EXAMPLE 22 or 23, where the determination of the image areas depends on an assignment of the cell-specific position information to a confluence mask of the scene.

[0151] EXAMPLE 25. Computer-implemented method according to one of EXAMPLES 18 to 24, wherein the second image evaluation includes the application of a machine-learned classification or regression model to cell-specific image areas of at least one of the one or more microscope images, each of which is determined depending on the cell-specific position information.

[0152] EXAMPLE 26. Computer-implemented method according to EXAMPLE 25, wherein the cell-specific image areas are determined by means of a machine-learned detection model, which is applied to further image areas of a predetermined size, which are determined based on the cell-specific position information.

[0153] EXAMPLE 27. Computer-implemented method according to one of EXAMPLES 18 to 26, wherein the second image evaluation comprises encoding image areas of at least one of the one or more microscope images, determined on the basis of cell-specific position information, in a machine-learned feature space, in order to obtain corresponding cell-specific feature vectors, the second image evaluation still includes an evaluation of the feature vectors with a clustering algorithm in the machine-learned feature space.

[0154] EXAMPLE 28. Computer-implemented method according to one of EXAMPLES 18 to 27, where the first image analysis and the second image analysis are performed in a jointly machine-learned model

[0155] EXAMPLE 29. Computer-implemented method according to one of the preceding EXAMPLES, further comprising: - Controlling a user interface to output graphical information based on a superimposition of cell-specific result data with one or more microscope images.

[0156] EXAMPLE 30. Computer-implemented method according to one of the preceding EXAMPLES, the second image analysis includes a classification to determine the cell-specific degree of transfection, where a decision limit for the classification is set depending on user input.

[0157] EXAMPLE 31. Computer-implemented method according to EXAMPLE 30, where continuous human-machine interaction includes receiving user input, where continuous human-machine interaction outputs to the user the influence of user input on a result of the classification.

[0158] EXAMPLE 32. Computer-implemented method according to one of the preceding EXAMPLES, where the one or more microscope images comprise several microscope images registered together.

[0159] EXAMPLE 33. Computer-implemented method according to EXAMPLE 32, wherein the multiple microscope images comprise a first microscope image and a second microscope image, the first image evaluation is based on the first microscope image, the second image analysis is based on the second microscope image, the second microscope image shows a contrast that is specific to the fluorescent dye.

[0160] EXAMPLE 34. Computer-implemented method according to one of the preceding EXAMPLES, further comprising: - Determining a scene-global transfection rate based on the cell-specific transfection rate.

[0161] EXAMPLE 35. Computer-implemented method that includes: - Obtained (905) from one or more microscope images depicting a scene with cells, and - Processing (934) the one or more microscope images in a machine-learned model, wherein the machine-learned model provides an image-to-scalar transformation, wherein a scalar output of the machine-learned model is indicative of a scene-global transfection degree for a fluorescent dye-based transfection of the cells.

[0162] EXAMPLE 36. Computer-implemented method according to EXAMPLE 35, where the scalar output indicates the transfected cells and / or the non-transfected cells and / or the degree of transfection.

[0163] EXAMPLE 37 Computer-implemented method that includes: - Obtained (905) from one or more microscope images depicting a scene with cells, - Processing (935) the one or more microscope images in at least one machine-learned model to provide a first density map and a second density map, wherein the first density map localizes transfected cells and the second density map localizes non-transfected cells, and - Comparing (935) the first density map and the second density map to determine a scene-global transfection level for a fluorescent dye-based transfection of cells.

[0164] EXAMPLE 38. Computer-implemented method according to EXAMPLE 37, wherein the at least one machine-learned model includes at least one machine-learned deep convolution network which processes overlapping image areas of the one or more microscope images to redundantly locate the transfected and non-transfected cells in the first and second density maps.

[0165] EXAMPLE 39. Computer-implemented method that includes: - Obtaining (905) a first microscope image depicting a scene with cells with a first contrast, - Obtaining (905) a second microscope image registered with the first microscope image, which depicts the scene with cells with a second contrast different from the first contrast, wherein the second contrast specifically represents a fluorescent dye, - Determining (936) a first confluence mask based on the first microscope image, - Determining a second confluence mask based on the second microscope image, and - based on a comparison of the first confluence mask with the second confluence mask, determining a scene-global transfection level for a fluorescent dye-based transfection of cells.

[0166] EXAMPLE 40. Electronic data processing device (700) comprising a processor and a memory, wherein the processor is configured to load and execute program code from the memory, wherein, based on the execution of the program code, the processor performs the following steps: - Obtained (905) from one or more microscope images depicting a scene with cells, - Performing (931, 1005) an initial image evaluation based on at least one of the one or more microscope images to obtain an instance segmentation mask for the cells includes, and - Performing (931, 1005) a second image evaluation based on at least one of the one or more microscope images to obtain cell-specific result data for the scene, where the cell-specific result data indicate a cell-specific transfection level for a transfection of the cells based on a fluorescent dye.

[0167] EXAMPLE 41. Electronic data processing device (700) comprising a processor and a memory, wherein the processor is configured to load and execute program code from memory, wherein, based on the execution of the program code, the processor performs the following steps: - Obtained (905) from one or more microscope images depicting a scene with cells, - Performing (932, 1105) an initial image evaluation based on at least one of the one or more microscope images to obtain a vector field map that maps each of several image regions to a corresponding reference image region, wherein the reference image regions are associated with different cells, and - Performing (932, 1110) a second image evaluation based on at least one of the one or more microscope images and the vector field map, to obtain cell-specific result data for the scene, where the cell-specific result data indicate a cell-specific transfection level for a transfection of the cells based on a fluorescent dye.

[0168] EXAMPLE 42. Electronic data processing device (700) comprising a processor and a memory, wherein the processor is configured to load and execute program code from the memory, wherein, based on the execution of the program code, the processor performs the following steps: - Obtaining one or more microscope images depicting a scene with cells, - Performing (933, 1405) an initial image evaluation based on at least one of the one or more microscope images to obtain cell-specific positional information for the cells, and - Performing (933, 1410) a second image evaluation based on at least one of the one or more microscope images, to obtain cell-specific result data for the scene, where the cell-specific result data indicate a cell-specific transfection level for a transfection of the cells based on a fluorescent dye.

[0169] EXAMPLE 43. Electronic data processing device (700) comprising a processor and a memory, wherein the processor is configured to load and execute program code from memory, wherein, based on the execution of the program code, the processor performs the following steps: - Obtained (905) from one or more microscope images depicting a scene with cells, and - Processing (934) the one or more microscope images in a machine-learned model, wherein the machine-learned model provides an image-to-scalar transformation, wherein a scalar output of the machine-learned model is indicative of a scene-global transfection degree for a fluorescent dye-based transfection of the cells.

[0170] EXAMPLE 44. Electronic data processing device (700) comprising a processor and a memory, wherein the processor is configured to load and execute program code from memory, wherein, based on the execution of the program code, the processor performs the following steps: - Obtained (905) from one or more microscope images depicting a scene with cells, - Processing (935) the one or more microscope images in at least one machine-learned model to provide a first density map and a second density map, wherein the first density map localizes transfected cells and the second density map localizes non-transfected cells, and - Comparing (935) the first density map and the second density map to determine a scene-global transfection level for a fluorescent dye-based transfection of cells.

[0171] EXAMPLE 45. Electronic data processing device (700) comprising a processor and a memory, wherein the processor is configured to load and execute program code from memory, wherein, based on the execution of the program code, the processor performs the following steps: - Obtaining (905) a first microscope image depicting a scene with cells with a first contrast, - Obtaining (905) a second microscope image registered with the first microscope image, which depicts the scene with cells with a second contrast different from the first contrast, wherein the second contrast specifically represents a fluorescent dye, - Determining (936) a first confluence mask based on the first microscope image, - Determining a second confluence mask based on the second microscope image, and - based on a comparison of the first confluence mask with the second confluence mask, determining a scene-global transfection level for a fluorescent dye-based transfection of cells.

[0172] EXAMPLE 46. Electronic data processing device according to one of EXAMPLES 40 to 45, wherein the processor executes the procedure according to one of EXAMPLES 1 to 39 based on the execution of the program code.

[0173] Naturally, the features of the embodiments and aspects of the invention described above can be combined with one another. In particular, the features can be used not only in the combinations described, but also in other combinations or individually, without leaving the scope of the invention.

[0174] For example, various techniques related to 2D microscope images have been described above. However, the techniques described here can also be used with 3D microscope images.

[0175] Furthermore, various techniques have been described above that use multiple microscope images with different contrasts to determine results. However, the various techniques described here can also be determined based on a single microscope image, for example, one with an autofluorescence contrast. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] EP 24 184 623.7

[0024] EP 4 053 805 A1

[0037] Cited non-patent literature

[0000] Optics communications 49.1 (1984): 6-10. DPC is described in: Mehta, Shalin B., and Colin JR Sheppard. “Quantitative phase-gradient imaging at high resolution with asymmetric illumination-based differential phase contrast.” Optics letters 34.13 (2009): 1924-1926

[0024] Paul Cohen, Joseph, et al. “Count-ception: Counting by fully convolutional redundant counting.” Proceedings of the IEEE International conference on computer vision workshops. 2017

[0092]

Claims

[1] Computer-implemented method that includes: - Obtained (905) from one or more microscope images depicting a scene with cells, - Performing (931, 1005) an initial image evaluation based on at least one of the one or more microscope images to obtain an instance segmentation mask for the cells includes, and - Performing (931, 1005) a second image evaluation based on at least one of the one or more microscope images to obtain cell-specific result data for the scene, wherein the cell-specific result data indicate a cell-specific transfection grade for a fluorescent dye-based transfection of the cells. [2] Computer-implemented method according to claim 1, wherein the second image evaluation is further based on the instance segmentation mask. [3] Computer-implemented method of claim 1 or 2, wherein the second image evaluation comprises cell-specific aggregation of pixel values ​​in at least one of the one or more microscope images in each mask area of ​​the instance segmentation mask. [4] Computer-implemented method according to one of the preceding claims, wherein the second image evaluation comprises encoding mask regions of the instance segmentation mask in at least one of the one or more microscope images in a machine-learned feature space to obtain corresponding cell-specific feature vectors for each image region. [5] Computer-implemented method according to claim 4, where the second image evaluation still includes the recognition of clusters formed by feature vectors in the machine-learned feature space, the procedure further includes: - depending on the assignment of the feature vectors to the clusters, determining the cell-specific degree of transfection. [6] Computer-implemented method according to any one of the preceding claims, wherein the second image evaluation further includes determining latent feature vectors for several mask areas of the instance segmentation mask and based on at least one of the one or more microscope images, where the second image evaluation still includes processing the latent feature vectors in a transformer network to obtain a classification token, the second image analysis includes a classification to determine the cell-specific degree of transfection, where a decision limit for the classification is set depending on the classification token. [7] Computer-implemented method that includes: - Obtained (905) from one or more microscope images depicting a scene with cells, - Performing (932, 1105) an initial image evaluation based on at least one of the one or more microscope images to obtain a vector field map that maps each of several image regions to a corresponding reference image region, wherein the reference image regions are associated with different cells, and - Performing (932, 1110) a second image evaluation based on at least one of the one or more microscope images and the vector field map, to obtain cell-specific result data for the scene, where the cell-specific result data indicate a cell-specific transfection level for a transfection of the cells based on a fluorescent dye. [8] Computer-implemented method according to claim 7, wherein the second image evaluation comprises iterating (1299) over the image areas, wherein in each iteration (1299) one or more pixel values ​​of the at least one of the one or more microscope images in the respective image area are added to a counter (1215) associated with the corresponding reference image area. [9] Computer-implemented method according to claim 8, wherein the second image evaluation further comprises comparing (1310) the counter values ​​of the different reference image areas with a threshold value to determine the cell-specific degree of transfection. [10] Computer-implemented method according to one of claims 7 to 9, wherein the second image evaluation comprises matrix multiplication. [11] Computer-implemented method according to any one of claims 7 to 10, wherein the vector field map is determined by means of a machine-learned model which performs a picture-to-picture transformation. [12] Computer-implemented method according to one of claims 7 to 11, wherein the at least one of the one or more microscope images evaluated in the second image evaluation comprises a first microscope image with a fluorescence contrast specific for the fluorescent dye and optionally a second microscope image, wherein the second microscope image has a contrast that specifically marks cell structures of the cells corresponding to the reference image areas. [13] Computer-implemented method according to any one of claims 7 to 12, wherein the method further comprises: - Consolidating reference image areas in the vector field map based on neighborhood relationships between the reference image areas. [14] Computer-implemented method that includes: - Obtaining one or more microscope images depicting a scene with cells, - Performing (933, 1405) an initial image evaluation based on at least one of the one or more microscope images to obtain cell-specific positional information for the cells, and - Performing (933, 1410) a second image evaluation based on at least one of the one or more microscope images, to obtain cell-specific result data for the scene, where the cell-specific result data indicate a cell-specific transfection level for a transfection of the cells based on a fluorescent dye. [15] Computer-implemented method according to claim 14, wherein the method further comprises: - based on cell-specific position information, determining cell-specific image areas, wherein the second image evaluation is cell-specific in the cell-specific image areas of the corresponding at least one of the one or more microscope images. [16] Computer-implemented method according to claim 15, where the determination of the cell-specific image areas is carried out depending on prior knowledge about the shape and / or size of the cells in at least one of the one or more microscope images. [17] Computer-implemented method according to claim 15 or 16, where the determination of the image areas depends on an assignment of the cell-specific position information to a confluence mask of the scene. [18] Computer-implemented method according to any one of claims 14 to 17, wherein the second image evaluation includes the application of a machine-learned classification or regression model to cell-specific image areas of at least one of the one or more microscope images, each of which is determined depending on the cell-specific position information. [19] Computer-implemented method according to claim 18, wherein the cell-specific image areas are determined by means of a machine-learned detection model, which is applied to further image areas of a predetermined size, which are determined based on the cell-specific position information. [20] Computer-implemented method according to any one of claims 14 to 19, wherein the second image evaluation comprises encoding image areas of at least one of the one or more microscope images, determined on the basis of cell-specific position information, in a machine-learned feature space, in order to obtain corresponding cell-specific feature vectors, the second image evaluation still includes an evaluation of the feature vectors with a clustering algorithm in the machine-learned feature space. [21] Computer-implemented method according to any one of the preceding claims, the second image analysis includes a classification to determine the cell-specific degree of transfection, where a decision limit for the classification is set depending on user input, where continuous human-machine interaction includes receiving user input, where continuous human-machine interaction outputs to the user the influence of user input on a result of the classification. [22] Computer-implemented method according to any one of the preceding claims, where the one or more microscope images comprise several microscope images registered together, wherein the multiple microscope images comprise a first microscope image and a second microscope image, the first image evaluation is based on the first microscope image, wherein the second image evaluation is based on the second microscope image, wherein the second microscope image has a contrast that is specific to the fluorescent dye. [23] Computer-implemented method that includes: - Obtained (905) from one or more microscope images depicting a scene with cells, and - Processing (934) the one or more microscope images in a machine-learned model, wherein the machine-learned model provides an image-to-scalar transformation, wherein a scalar output of the machine-learned model is indicative of a scene-global transfection degree for a fluorescent dye-based transfection of the cells. [24] Computer-implemented method that includes: - Obtained (905) from one or more microscope images depicting a scene with cells, - Processing (935) the one or more microscope images in at least one machine-learned model to provide a first density map and a second density map, wherein the first density map localizes transfected cells and the second density map localizes non-transfected cells, and - Comparing (935) the first density map and the second density map to determine a scene-global transfection level for a fluorescent dye-based transfection of cells. [25] Computer-implemented method that includes: - Obtaining (905) a first microscope image depicting a scene with cells with a first contrast, - Obtaining (905) a second microscope image registered with the first microscope image, which depicts the scene with cells with a second contrast different from the first contrast, wherein the second contrast specifically represents a fluorescent dye, - Determining (936) a first confluence mask based on the first microscope image, - Determining a second confluence mask based on the second microscope image, and - based on a comparison of the first confluence mask with the second confluence mask, determining a scene-global transfection level for a fluorescent dye-based transfection of cells.

Images