Computer-implemented transfection analysis with vector field maps

Abstract

Various examples of the present disclosure relate to transfection analysis of cells displayed in a microscope image. Techniques are disclosed for determining cell-specific transfection levels or scene-global transfection levels using vector field maps.

Description

Technical Field

[0001] The examples in this publication relate to transfection analysis based on one or more microscope images. In particular, the examples relate to computer-implemented automation of transfection analysis, which utilizes vector field diagrams. Background Technology

[0002] One application of microscopy is cell research. In particular, it allows for the evaluation of microscopic images showing scenes with cells to determine the degree of cell transfection. To fluorescently label specific cellular proteins, the corresponding gene is fused to the gene sequence of the fluorescent protein (gene fusion). If the modified gene is absorbed by the cell (transfection), the cell will express a fusion protein composed of the target protein and a fluorophore. For example, fluorescence imaging can show the presence of this protein within the cell. This allows determination of whether transfection has occurred in a single cell.

[0003] In the reference implementation, the evaluation of the relevant microscope images is performed manually, which is both time-consuming and subjective.

[0004] Automated techniques for transfection analysis are known, such as those proposed in CN116609311 A or CN118782156 A. These techniques are sometimes not precise enough or robust enough. For example, identifying image regions with fluorescence signals is not always straightforward, as the fluorescence signal may also contain interfering signal components. Depending on the specific imaging modality, the appropriate threshold filter used may also be inaccurate. Summary of the Invention

[0005] Therefore, there is a need to improve transfection analysis techniques. In particular, there is a need for automated techniques that can robustly and automatically determine the degree of cell transfection based on microscope images.

[0006] This objective is achieved through the features of the independent patent claims. The features of the dependent patent claims define the implementation method.

[0007] A computer-implemented method includes acquiring one or more microscope images. The one or more microscope images show a scene containing cells. The computer-implemented method includes performing a first image evaluation. The first image evaluation is based on at least one of the one or more microscope images. Based on the first image evaluation, a vector field map is obtained. The vector field map maps multiple image regions to corresponding reference image regions. Here, these reference image regions are associated with different cells. The method also includes performing a second image evaluation. The second image evaluation, based on at least one of the one or more microscope images and the vector field map, is used to obtain cell-specific result data for the scene. The cell-specific result data indicates the degree of cell-specific transfection based on fluorescent dye transfection.

[0008] The present invention also discloses an electronic data processing apparatus. This electronic data processing apparatus is adapted to perform the computer-implemented method described above.

[0009] Furthermore, the present invention also discloses a system comprising such electronic data processing equipment and a microscope adapted for acquiring microscope images.

[0010] The features described above and those described below can be used not only in the combinations explicitly stated therein, but also in other combinations or individually, without departing from the scope of protection of this invention. Attached Figure Description

[0011] Figure 1 The flowchart shows an example method that uses a vector field graph to perform transfection analysis.

[0012] Figure 2 Details of the first and second image evaluations are shown to determine the vector field map, and the degree of transfection is then determined based on the vector field map.

[0013] Figure 3 , Figure 4 , Figure 5 and Figure 6 An exemplary vector field diagram is shown.

[0014] Figure 7 and Figure 8 The edge conditions of the vector field graph are shown with different examples.

[0015] Figure 9 An instance segmentation graph is schematically shown, which can be used to determine the vector field graph, for example, during training or inference.

[0016] Figure 10 The flowchart shows an example method that determines the degree of transfection based on a vector field graph.

[0017] Figure 11 and Figure 12 A technique for determining a threshold used to determine whether a particular cell has been transfected is shown.

[0018] Figure 13 Electronic data processing devices according to various examples are illustrated schematically.

[0019] Figure 14 The data processing flow for transfection analysis using vector field diagrams is illustrated schematically according to various examples.

[0020] Figure 15 An exemplary phase-contrast microscope image is shown.

[0021] Figure 16 The diagram shows the cell center point map, and the cell center point map is compared with... Figure 15 Superimposed phase-contrast microscope images.

[0022] Figure 17 The vector field diagram is schematically shown. Figure 15 Superimposed phase-contrast microscope images.

[0023] Figure 18 The superposition of vector field maps and fluorescence microscope images is illustrated schematically according to various examples.

[0024] Figure 19 The illustration schematically shows how, according to various examples, vector field maps are used to aggregate fluorescence signals indicating pixel values ​​in a fluorescence microscope image to the cell center point.

[0025] Figure 20 The cell-specific transfection levels are shown according to various examples.

[0026] Figure 21 A flowchart of an exemplary method.

[0027] Figure 22 A flowchart of an exemplary method. Detailed Implementation

[0028] The above-described features, characteristics, advantages, and implementation methods of the present invention will become clearer and more explicit when combined with the following embodiments described in more detail with reference to the accompanying drawings.

[0029] The present invention will now be described in more detail with reference to the accompanying drawings and preferred embodiments. The same reference numerals in the drawings denote the same or similar elements. The drawings are schematic diagrams of various embodiments of the present invention. The elements shown in the drawings are not necessarily drawn to scale. Rather, the manner in which the various elements are depicted in the drawings should enable those skilled in the art to understand their function and general purpose. The connections and couplings between the functional units and elements shown in the drawings can also be implemented as indirect connections or couplings. Connections or couplings can be implemented via wired or wireless means. Functional units can be implemented as hardware, software, or a combination of hardware and software.

[0030] The following describes techniques for evaluating one or more microscopic images. Microscopic images are evaluated for transfection analysis. This means determining the degree of transfection. The degree of transfection can be determined generally for a single cell in a scene or globally for the entire scene. It is conceivable that the local degree of transfection for each cell in the scene is first determined; then the entire scene can be aggregated accordingly to determine the global degree of transfection based on the local degree of transfection.

[0031] If the degree of transfection is determined for a single cell, it can be a binary degree of transfection (e.g., a "yes / no" transfection classification); however, this cell-specific degree of transfection can also represent the probability that a particular cell expresses a protein. For example, it can be determined whether a cell corresponding to the nucleus expresses the dye or whether an error has occurred. It can also be regressed to an expression efficiency value (e.g., between 0 and 100).

[0032] It can also determine the global transfection level. This global transfection level can represent the proportion of transfected cells among all cells in the scene.

[0033] Transfection analysis uses one or more microscope images. In principle, microscope images can be acquired using different imaging modes. This means that, among the various techniques described herein, microscope images with different contrasts can be used to determine the degree of transfection. In particular, multi-channel images containing multiple microscope images displaying a scene with cells at different contrasts can be acquired. For example, specific and non-specific contrasts can be combined into a single multi-channel image.

[0034] Specific contrast is used to label specific cellular structures in a specific way. For example, specific fluorescence contrast or non-fluorescently labeled specific contrast can be used. One or more fluorescence contrasts that are particularly useful in transfection analysis can be used. In particular, those fluorescence contrasts that can be specifically used to observe transfected cells are used. Specific fluorescence contrasts are used to make fusion proteins visible; these are referred to below as transfection fluorescence contrast.

[0035] Hematoxylin and eosin (H&E) staining can be used to observe the overall morphology of cells. DAPI (4',6-diamidinyl-2-phenylindole) is a fluorescent contrast agent used to label cell nuclei. Phalloidin is a fluorescent contrast agent used to label intracellular actin filament networks.

[0036] Non-specific contrast can be, for example, phase contrast or bright-field contrast. For instance, non-specific contrast can be phase-based contrast. Phase-based contrast can be, for example, phase contrast. Examples include Zernike-Phasenkontrast and Normarski-Phasenkontrast. Special optical elements are used in the beam path here, such as a phase ring in the objective and an annular stop in the condenser. This makes the interference between the background and object light visible. By using phase contrast, image contrast can be improved. This means that cellular structures are particularly clearly visible. Cells are phase objects; their amplitude does not decrease or does not decrease significantly when light passes through a cell sample, therefore phase contrast is preferred for visualizing phase shifts. However, digital phase contrast can also be used as a phase-based contrast. Here, multiple images are acquired and then combined into a single phase-contrast image through computation. Therefore, this type of technique can be called digital phase contrast. This phase contrast is obtained by digitally post-processing the acquired intensity images. Examples include the Transport of Intensity Equation (TIE) and Differential Phase Contrast (DPC). For a description of TIE, see: Streibl, Norbert. "Phase imaging by the transport equation of intensity." Optics Communications 49.1 (1984): 6-10. For a description of DPC, see: Mehta, Shalin B., and Colin JR Sheppard. "Quantitative phase-gradient imaging at high resolution with asymmetric illumination-based differential phase contrast." Optics Letters 34.13 (2009): 1924-1926. To obtain the TIE dataset, samples need to be translated along the optical axis (z-direction), i.e., axial translation, and a so-called z-axis stack consisting of at least two images needs to be obtained. The data is then processed to obtain a phase contrast image. This requires solving diffusion-type partial differential equations.In DPC, the sample is illuminated from at least two different directions (oblique illumination) while the sample remains at a fixed z-axis position. Possible light sources for oblique illumination include all types of segmented light sources; for example, segmented diodes, LED arrays, digital micromirror devices (DMDs), liquid crystal displays (LCDs or SLMs), or variable condenser apertures. The obtained data is then converted into a phase contrast image by solving a deconvolution problem. A combination of TIE and DPC is also conceivable, as described, for example, in European Patent Application 24,184,623.7, dated June 26, 2024. The advantage of using digital phase contrast (compared to hardware-based phase contrast) is that no object needs to be inserted or removed in the beam path when acquiring digital phase contrast. Instead, the illumination can be selectively changed, for example, by a switchable LED array arranged in the illumination pupil plane. This can be done quickly and easily.

[0037] Non-specific contrast types include “label-free” contrast, such as phase contrast, DIC contrast, or TIE contrast, which allow observation of cellular structures without specific labeling. Alternatively, dyes such as H&E can be used to specifically label particular cellular structures. Another option is to use fluorescent dyes (such as DAPI) to fluorescently label specific cellular components. Autofluorescence can also be used to locate cells without additional labeling.

[0038] Multiple channels can also be used to display different fluorescent dyes. In this case, each channel can be predicted individually or all channels can be predicted as a whole.

[0039] In one example, even using only a single channel or a single contrast ratio, all cells can be located simultaneously and the degree of transfection determined. In this case, cell localization can be achieved, for example, by autofluorescence.

[0040] While the solutions described herein are specifically for two-dimensional image data, these techniques can also be applied to three-dimensional image data (e.g., from light-sheet microscopy). For this purpose, for example, two-dimensional slice images of one or more surfaces (e.g., planes, such as stacked planes) can be extracted from three-dimensional image data and then processed using the two-dimensional techniques described herein. For example, a first image evaluation and a second image evaluation can be applied to each two-dimensional slice image. This yields cell-specific result data for each two-dimensional slice image. Subsequently, these cell-specific result data obtained for different two-dimensional slice images can be combined to obtain three-dimensional cell-specific result data in a reference coordinate system associated with the imaging volume of the three-dimensional image data. When combining cell-specific result data, integration can be performed, for example, if cell-specific result data for the same cells are determined separately for different two-dimensional slice images. This occurs when the corresponding surfaces are close to or even overlap each other within the imaging volume. In another example, vector field maps determined for different slice images can also be combined with each other. This yields a three-dimensional vector field map associated with the imaging volume.

[0041] Figure 1 A flowchart of an exemplary method is shown. Figure 1 The methods described herein can be executed, for example, by an electronic data processing device. This electronic data processing device may include a processor unit and memory. The processor can load and execute program code from memory. When the processor unit executes the program code, it triggers the processor to execute... Figure 1 The method in the middle.

[0042] Optional boxes are shown in dashed lines.

[0043] In box 905, one or more microscope images are acquired. For this purpose, the microscope can be controlled to acquire the corresponding microscope images, for example. Microscope images can be received from the microscope. Microscope images can also be loaded from memory, such as from an image database.

[0044] If multiple microscope images are acquired, they can collectively display a scene containing cells. Multiple microscope images can show a single cell sample. Multiple microscope images can be part of a common multi-channel image. However, it is also conceivable to acquire multiple microscope images sequentially, for example, after each staining cycle in which specific cells were stained, destained, or otherwise manipulated.

[0045] The following assumptions are made that at least two microscope images are obtained in box 905: a first microscope image suitable for cell detection (hereinafter referred to as the reference microscope image); and a second microscope image with transfection fluorescence contrast (hereinafter referred to as the fluorescence microscope image).

[0046] The reference microscope image preferably displays all cells and provides high contrast against the background. Typically, the reference microscope image can use phase contrast (phase-contrast microscopy), such as digital phase contrast. In principle, it is conceivable to use multiple reference microscope images, where different reference microscope images have different contrasts. For example, the first reference microscope image can use bright-field contrast, while the second reference microscope image can use phase contrast.

[0047] In optional box 910, the microscope images from box 905 are registered to each other. If multiple microscope images are used (e.g., as channels in a multi-channel image or completely independently acquired images), registration of multiple microscope images can be performed. Registration can be based, for example, on images or on point clouds, through localization results. In point cloud-based registration, a point cloud can be created using image evaluation results and then used for registration. For example, a first point cloud can be created based on the detection results of cell center points in a first channel, while a second point cloud shows local maxima in the fluorescence channel. Registration can then be performed using algorithms such as Iterative Closest Point to determine the relative positions between the images.

[0048] However, it is also conceivable that the microscope image from box 905 has already been registered and the corresponding registration parameters already exist. In this case, there is no need to execute box 910.

[0049] There can also be inherent registration relationships between microscope images, meaning that the same pixels represent the same object points in the scene. This inherent registration relationship occurs, especially when different microscope images constitute a multi-channel image. Typically, between different microscope images acquired in a multi-channel image, the microscope's imaging system only changes slightly, such as inserting or removing filters in the beam path. Therefore, the corresponding microscope images are usually inherently registered with each other (e.g., when chromatic aberration is small).

[0050] Optionally, the microscope image can be scaled in box 915. For example, it can be scaled so that the microscope image subsequently displays cells at a specific imaging scale. In other words, this means that it can be scaled so that the cells in the microscope image have a specific size (structural unit size). This size can be predefined. In particular, this size can correspond to the cell size in a reference microscope image used to train one or more machine learning models (subsequently used for image evaluation). This can reduce the complexity of the corresponding machine learning models, as these models only expect cells with a specific imaging size. The training workload for such machine learning models is reduced. The training data does not have to include cells with different imaging scales, but can be limited to cells with a specific imaging size.

[0051] The scaling in box 915 can be performed manually, for example. However, it is also conceivable to use a machine learning model to perform the scaling in box 915. For example, a machine learning model can be used to perform an image-to-image conversion, i.e., outputting a rescaled image. Alternatively, a machine learning model can be used to output scaling factors, i.e., performing an image-to-scalar conversion. An exemplary rescaling technique is basically described in EP 4 053 805 A1. Corresponding techniques are incorporated herein by cross-reference.

[0052] Optionally, one or more microscope images (possibly rescaled) from box 905 can be preprocessed in box 920, such as cleaned or filtered. In particular, microscope images with fluorescence contrast, especially transfection fluorescence contrast, can be cleaned. For example, dirt can be removed. Dye residues that may produce fluorescence signals can be removed. Alternatively or supplementarily, background correction can also be performed, for example, when data offset exists. Smoothing operations can be performed in box 920, for example. Noise reduction can be applied to one or more microscope images.

[0053] Optionally, interfering signal components of the fluorescence signal in the fluorescence microscopy image can also be reduced or eliminated in box 920. This can be achieved, for example, based on a cell mask map (Zellmaskenkarte). The cell mask map can be determined, for example, based on a phase-contrast microscopy image. Based on the cell mask map, interfering signal components in the extracellular regions can be determined. This is based on the understanding that interfering signal components cannot be robustly determined within cellular regions because, in addition to the interfering signal components, the region may also contain foreground signal components. Based on the determination of interfering signal components in the extracellular regions (these image regions are determined based on the cell mask map), interfering signal components in the intracellular image regions can also be estimated. In this way, an interfering signal map can be determined, and this interfering signal map can be used to reduce or eliminate interfering signal components of the fluorescence signal. Such interfering signal components can typically have different components, such as background signal components (usually having only small spatial frequency components); but it is also conceivable that such interfering signal components have components with higher spatial frequencies, for example, due to dye accumulation, etc.

[0054] For example, in some variations, it is conceivable to swap the order of box 915 with the box 920 below it. Preprocessing can be performed on the image that has not yet been rescaled.

[0055] In box 930, one or more microscopic images from box 905 are subjected to image evaluation or image processing. For example, the degree of cell-specific transfection can be determined.

[0056] In box 930, a vector field map is used to map multiple image regions to corresponding reference image regions. Different reference image regions are then associated with different cells. Details of one possible implementation of box 930 will be provided below. Figure 2 To elaborate.

[0057] Subsequently, the user interface can be controlled in box 940. In particular, the graphical user interface can be controlled. Information related to the degree of cell-specific transfection and / or the degree of scene-wide transfection can then be output. For example, the user interface can be controlled to output graphical information determined based on at least one microscope image from the microscope images from box 905 and the result data from box 930.

[0058] In particular, cell-specific results data can be overlaid with one or more microscopic images for output. For example, when the degree of cell-specific transfection is known, each transfected cell can be highlighted in a specific way in the microscopic image; alternatively or supplementarily, each untransfected cell can be highlighted in another way in the microscopic image.

[0059] It can be envisioned that a cell instance segmentation mask is determined using box 940. Then, for example, the degree of cell-specific transfection can be output in box 940 along with mask regions labeling different cells. This is a particularly easy-to-interpret display method. For example, a vector field map can be converted into an instance segmentation map. This instance segmentation map can then be displayed to the user. The conversion from a vector field map to an instance segmentation map can use, for example, a watershed algorithm or a Dijkstra algorithm (where a “peak” is defined by the vector length). For example, each vector can be assigned to a specific cell center point, and the pixels of the microscope image can be colored according to this assignment. For example, each cell center point can have a specific color. If necessary, the resulting mask can be post-processed. For example, it can be smoothed using morphological operations. Holes can be filled.

[0060] One example is displaying cells as colored dots. Each cell can be visualized using a dot (or other marker) to indicate the transfection result. The dot's location could be, for example, at the cell's center, centroid, or nucleus. In other words, a microscopic image (e.g., a fluorescence microscope image) can be overlaid with a cell-specific graphic indicator that shows the degree of cell-specific transfection.

[0061] Another approach is to use colors to display the results. For example, green, orange, and red can be used to represent the categories of "transfected," "overexpressed," and "not transfected."

[0062] Alternatively, it could be envisioned that cells be categorized and displayed according to their transfection probability, for example, by overlaying them with corresponding microscope images.

[0063] The above description provides examples of how cell-specific transfection levels can be displayed. Alternatively, or supplementarily, the overall transfection level can be output, for example, displayed next to the corresponding microscope image. For instance, it could display: "79% of cells successfully transfected."

[0064] Another approach is to determine the pixel value distribution of the image pixels displaying transfected cells. It's also possible to generate the pixel value distribution of the image pixels displaying untransfected cells. The corresponding distribution can then be output to the user.

[0065] Optionally, the resulting data can be corrected in box 945. For example, a user can mark certain cells that were marked as transfected as actually untransfected. The image output in box 940 (indicated by the dashed arrow) can then be adjusted.

[0066] Another method 945 for refining the resulting data is to receive user input relating to a decision boundary used to distinguish between transfected and untransfected cells. For example, some techniques have been described above where a threshold for determining the presence of transfected or untransfected cells is found in a specific way. It is conceivable that, in box 945, a user (e.g., via a slider) can change this threshold and then interactively view the effect of this change on the classification result in the image. For example, the effect of changing the threshold on classifying cells as transfected or untransfected (or other or further classification criteria related to transfection) can be displayed. In other words, continuous human-computer interaction can be provided. This continuous human-computer interaction can, on the one hand, include receiving user input relating to the setting of the classification decision boundary. On the other hand, this continuous human-computer interaction can include outputting to the user the effect of the user input on the classification result. The advantage is that the user can interactively “sample” the decision boundary and can, for example, identify regions of particularly high sensitivity. This allows for a better determination of the decision boundary, especially when considering underlying parameters. In other words, continuous human-computer interaction can be provided. This continuous human-computer interaction can, on the one hand, include receiving user input relating to the setting of the classification decision boundary. On the other hand, this continuous human-computer interaction can also include showing the user the impact of their input on the classification results. The advantage of this approach is that users can interactively "sample" the decision boundary and, for example, identify regions with particularly high sensitivity. In these regions, small changes in the decision boundary can have a particularly large impact on the classification results. Understanding these highly sensitive regions often helps to achieve good results in classification.

[0067] Based on this type of user input, more ground truth or training data can be collected. Then, based on these ground truths, one or more machine learning models used in box 930 can be retrained (which will be discussed later). Figure 22 To elaborate further.

[0068] Figure 1 The method described herein can be conceived in several variations. For example, the order of the different boxes can be changed. For instance, it is conceivable to execute box 920 before box 915. Furthermore, it is conceivable to perform multiple iterations on box 930, for example, for multiple sets of one or more microscope images or for processing different regions of a microscope image separately. In this way, separate evaluation results can be obtained for different regions of the sample. For example, the scene-wide transfection level of different wells in a multi-well plate can be determined. Specific parameters related to the image evaluation in box 930 can be synchronized between different instances of box 330. The examples here specifically involve decision criteria used to distinguish different transfection level values. This ensures that the same decision criteria are used for different regions of the sample, thereby achieving consistent evaluation of different regions of the sample.

[0069] The following describes the details of the image evaluation used to determine the degree of transfection in box 930.

[0070] Figure 2 The flowchart illustrates the method for determining the vector field graph and using it to determine the degree of transfection. Figure 2 Implementation shown Figure 1 The method for the 930-inch mid-frame.

[0071] exist Figure 2 The method shown, for example, determines the degree of cell-specific transfection. Preferably, two microscopic images are evaluated: a reference microscopic image and a fluorescence microscopic image. The first microscopic image enables the determination of the vector field map in box 1105; while the second microscopic image displays the fluorescent dye on which the transfection is based, i.e., the dye to be expressed, and is used to determine the degree of transfection in box 1110. Therefore, the reference microscopic image from box 905 can be used in box 1105; while a fluorescence microscopic image with transfection fluorescence contrast can be used in box 1110. This will be explained in detail below.

[0072] The method begins at box 1101. First, optionally, image evaluation for determining the vector field map in box 1105 is configured in box 1101 and / or image evaluation for determining the degree of transfection is configured in box 1110. Alternatively or supplementarily, possible image preprocessing may also be configured (see [link to relevant documentation]). Figure 1 (See box 920 in the configuration file; this preprocessing can also be performed after box 1101). When configuring box 1110, you can, for example, set the method of using a vector field graph to determine the degree of transfection.

[0073] The corresponding settings can be configured based on the type of one or more microscope images. For example, the contrast techniques of the reference microscope image and / or the fluorescence microscope image can be used. The fluorescent dye used can be considered, as can which cellular structures are stained by the fluorescent dye. This allows, for example, identification of whether the cell membrane or other cellular structures are stained. It also allows verification of whether the contrast is specific or non-specific. For example, if the cell membrane is not stained, a model that provides a specific tolerance range or "safe distance" at each cell edge can be selected for determining the vector field map in box 1105. This technique is based on the understanding that when staining cellular structures outside the cell membrane, accurate determination of the degree of transfection does not require capturing precise fluorescence signals within the cell wall region; therefore, setting a certain tolerance in this region when creating the vector field map is desirable.

[0074] In box 1105, a first image evaluation is performed based on at least one microscope image; wherein a vector field map is determined. The vector field map maps different image regions (e.g., pixels or superpixels) to corresponding reference image regions. The different reference image regions are then associated with different cells. A superpixel is an image region in a microscope image that contains a group of adjacent pixels. This group of pixels is treated as a unit and can be represented by a single vector in the vector field map.

[0075] Vector field maps can contain two or three output channels, depending on whether the image data is two-dimensional or three-dimensional. For example, a vector field map can be represented in Cartesian coordinates as x and y channels, or in polar coordinates as angle and distance channels. Vector field maps can be defined for a single pixel of a microscope image. Besides vector field maps defined for single pixels, this method can also be applied to larger image regions, i.e., image regions containing multiple image pixels. For example, regions with regular shapes (such as squares or rectangles) can be used as image regions. So-called "superpixels" can also be used.

[0076] The following shows some variations of vector field diagrams. Figure 3 An exemplary vector field graph 815 is shown. Dotted lines represent the grid of the vector field graph 815. For each entry in the vector field graph 815, a vector is provided that maps the corresponding image region (e.g., a pixel or superpixel) to a reference image region; this is indicated by an arrow for the top-left cell. This means that all image regions belonging to the same cell are mapped to the same reference image region. Figure 4 Another vector field diagram 816 is shown, which is substantially consistent with vector field diagram 815. In a variant of vector field diagram 816, image regions located outside the cells are mapped to random reference image regions (such as...). Figure 4 (As shown in the upper right corner). In another variant, it is conceivable that image regions located outside the cell are not mapped to reference image regions. Figure 5 Another vector field diagram 817 is shown. This vector field diagram 817 is also substantially consistent with vector field diagrams 815 or 816. However, vector chains exist in the variant vector field diagram 817. This means that the end point of one vector is simultaneously the starting point of another vector. In other words, such a vector chain can map an image region to a reference image region, which in turn is mapped to another reference image region through another vector. Figure 6 Another vector field diagram, 818, derived from actual calculations, is shown. From Figure 3 , Figure 4 and Figure 5 And especially from Figure 6 As can be seen, the vector field does not directly assign image regions to cells. Instead, it indicates the location of the cell center point (or other unique anchor point) of the corresponding related cell for each image region (i.e., each pixel). This means that the vectors of two image regions of a cell should point to the same point, but these two image regions are not directly related to each other in the vector field.

[0077] Refer again Figure 2 Box 1105. There are several distinct methods for determining the vector field graph. Some algorithmic implementations will be described below. For example, one could... Figure 2In box 1105.1, a machine learning model is used to determine the vector field map. The mapping from pixels of a microscope image to vectors in the vector field map can be accomplished using a machine learning model. One example is an image-to-image network based on a convolutional neural network (CNN), such as UNet. Another approach is to use a transformer-based image-to-image network, such as ViTMAE. Hybrid networks, such as VQGAN, can also be used for this purpose. One advantage of determining the vector field map using a machine learning model is that the corresponding machine learning model does not need to be run based on fluorescence contrast. For example, the microscope image used as input to the machine learning model to determine the vector field map can have a specific phase contrast (e.g., digital phase contrast). This allows the machine learning model to be trained particularly robustly to remain invariant to changes in fluorescence contrast (e.g., due to the use of different dyes in different transfection experiments). When the machine learning model is used to determine the vector field map in box 1105, the output of the machine learning model may sometimes not satisfy certain edge conditions or properties. Therefore, in various variations, post-processing of the output of the machine learning model (box 1105.2) after determining the vector field map using the machine learning model (box 1105.1) can be desirable. In box 1105.2, as an optional post-processing step (e.g., after applying a machine learning model to determine the vector field map), predicted reference image regions can be merged or integrated to eliminate discrepancies in location correspondence. Consistency checks can be performed. For example, all reference image regions located within a specific radius can be merged into a single reference image region, and the corresponding vectors can be corrected accordingly. Other neighborhood relationships can also be considered when performing such integration. Another technique for post-processing in box 1105.2 includes vector chain elimination. For example, vector chains can be replaced with corresponding end-to-end vectors.

[0078] In box 1105.2, one or more edge conditions can typically be considered. For example, edge conditions can include preset values ​​for the spatial distance between reference image regions corresponding to adjacent vectors of the vector field map. For example, a value gradient can be defined for the entire microscope image. For instance, for an image 1000 pixels wide, this value gradient can be set to values ​​from -500 to +500. Then, for each pixel or each image region mapped to a reference image region by the vector field map, an offset relative to the gradient value associated with the corresponding reference image region can be predicted. Only one offset is allowed per cell. Figure 7 This situation is illustrated. Figure 7 In the diagram, the cross represents the length of different vectors 2610, which are mapped to image regions at different x positions within the fluorescence microscope image. Figure 7 Only one sub-region along the x-axis is shown, which is located within a single cell. Figure 7The lower bound 2611 and upper bound 2612 are shown, obtained as linear gradients with offsets. These lower and upper bounds define the tolerance range for length variations or differences between adjacent vectors, thus defining the spatial distance between the reference image regions corresponding to adjacent vectors in the vector field map. For example, the three vectors (marked with dotted circles) determined by the machine learning model in box 1105.1 exceed the tolerance range, thus violating the required edge conditions. These vectors can be corrected in box 1105.2, for example, by moving them to their respective nearest boundaries 2611, 2612, or by deleting them and replacing them with interpolations based on adjacent vectors.

[0079] Combination Figure 8 Another edge condition is shown, which can be used for consistency checks of the vector field graph in box 1105.2. Figure 8 In the diagram, the solid circle represents the prediction of the cell center point. The figure also uses dashed lines to mark a tolerance range 2621 around a specific distance from the cell center point, which defines the edge conditions for the vector endpoint. Several vectors from the vector field map are also shown. Five of these vectors point to reference image regions sufficiently close to the cell center point, specifically within circle 2621. Only one vector ends outside circle 2621, which can be corrected during integration. Therefore, in general, the edge conditions considered when determining the vector field map can include preset values ​​for the distance between the reference image region pointed to by the vector and the cell center point plotted on the cell center point map. For vectors mapped to reference image regions outside tolerance range 2621 (marked with small dotted circles), various measures can be taken: for example, the vector can be deleted, meaning the value of the vector field map at the corresponding image region constituting the origin of the vector is zero. Alternatively, it is conceivable to correct the reference image region, i.e., the endpoint of the vector, for example, based on interpolation of adjacent vectors. In the current specific case, the vector can be "stretched". More generally, edge conditions can selectively prohibit or allow vectors in a vector field map based on which subregion of the cell the corresponding vector begins and / or terminates. Apply this concept to... Figure 8 This leads to the following result: vectors terminating in a reference image region more than a certain preset distance from the cell center are prohibited. Alternatively or supplementarily, it is conceivable, for example, to prohibit vectors originating in cell edge regions (e.g., near the cell membrane). As described above, such edge conditions can be selectively activated or deactivated based on the fluorescence contrast used in the fluorescence microscope image. In other words, edge conditions can be selected based on the contrast type of the fluorescence microscope image used or the reference microscope image. In another variation, for vectors originating in cell edge regions, certain edge conditions (e.g., combined with...) can be excluded. Figure 8 (Discussion on edge conditions). Cell edge regions can be determined based on cell mask maps.

[0080] In the preceding text, combined with Figure 2 Boxes 1105.1 and 1105.2 and Figure 7 and Figure 8 The paper describes the following technique: first, a machine learning model is used to determine the vector field graph (box 1105.1), followed by post-processing to enforce edge conditions (box 1105.2). In other variations, the vector field graph can also be determined directly based on the corresponding edge conditions. This is in... Figure 2 As shown in box 1105.3. In box 1105.3, for example, an instance segmentation map of the cells in the cell sample is determined by a corresponding machine learning model. This is in Figure 9 The instance segmentation is shown in more detail in Figure 812. Within a cell, the length of the vector is determined based on the corresponding value gradient 2605. This value gradient can be predicted as an offset relative to the global value gradient of the image (not shown), or it can be predefined individually for each cell. More generally, as Figure 7 As shown, edge conditions can be obtained through this value gradient 2605. These edge conditions include preset values ​​for the length differences of adjacent vectors in the vector field map and are used to determine the vector field map. The length differences of adjacent vectors can only differ by one pixel within a cell.

[0081] Figure 9 In determining the vector field diagram, this edge condition is inherently considered, and the preceding combination... Figure 7 and Figure 8 The described techniques retrospectively consider edge conditions after the vector field graph is determined, in order to correct the vector field graph, for example, when edge conditions are not met. These methods are used to achieve... Figure 2 The distinctly different approach to middle frame 1105 has been described above in conjunction with frames 1105.1 and 1105.3.

[0082] Further reference Figure 2 It is conceivable that multiple vector field maps can be determined in box 1105. For example, if multiple reference microscope images are available, multiple vector field maps can be determined separately based on the different reference microscope images. This technique is based on the understanding that if different reference microscope images with different contrasts exist, there are alternative methods for determining the vector field maps, thereby allowing differences to be tested during the determination process.

[0083] Optionally, a confidence map can be determined in box 1006. This confidence map can provide confidence values ​​for vectors from one or more vector field maps derived from box 1105. The degree of transfection can then be determined in box 1110, taking into account the confidence map. For example, the uncertainty of the degree of transfection can be determined. There are several distinct options for determining such a confidence map. For example, multiple vector field maps can be determined based on reference microscope images with different contrasts. For example, a first vector field map can be determined based on a phase-contrast microscope image and a second vector field map based on a bright-field microscope image. Multiple reference microscope images with different phase contrasts (e.g., optical phase contrast and digital phase contrast) can also be used. This allows for different estimates of the vector field maps, which can then be compared; based on this comparison, the confidence level can be determined according to the deviation between the two vector field maps. In another variation, the statistical properties of the vectors in the vector field maps can be considered. For example, the length distribution and / or orientation distribution of the vectors in the vector field maps can be considered. Based on this distribution, the confidence level can be determined. For example, the existence of particularly large dispersion in the lengths of the vectors can be checked. The width of the peaks in the distribution can also be considered. These techniques are based on the understanding that different cells in a typical cell sample are similar in size. This is reflected in a specific distribution of vector lengths. Based on this prior knowledge of the nominal form of the vector length distribution, lower confidence can be derived when corresponding deviations occur. This can also be applied to individual vectors, for example, those located at the edge of a peak, which can be assigned a lower confidence. Another technique is based on the use of cell centroid maps. For example, the distance between certain vectors in the vector field map and the cell centroids plotted in the cell centroid map can be examined. If the distance is large, a lower confidence can also be derived. Alternatively or supplementarily, cell mask maps can also be considered when determining the confidence map. Such cell mask maps can be used to indicate the degree to which different image regions in a microscope image are occupied by cells. The confidence can then be determined, for example, based on the dispersion of vector lengths or the deviation of vector lengths from the gradient of linear values ​​within the cells. In another example, it can be envisioned that the confidence map is output directly from the corresponding machine learning model as an additional channel. In the corresponding machine learning training, the corresponding confidence predictions can be learned.

[0084] In box 1110, a second image analysis is performed. This determines the degree of transfection for each cell in the corresponding microscopic image. This means determining the transfection success rate for different cells. Here, the vector field map determined in box 1105 is taken into consideration. Figure 10 One possible implementation of box 1110 is shown.

[0085] Figure 10 This is a flowchart of an exemplary method. Figure 10 The methods in [the document] are used for evaluation (e.g., by [method / method]). Figure 2 The method described in the text uses a pre-defined vector field map to determine the degree of cell transfection in a cell sample. Therefore, Figure 10 Showing Figure 2 The implementation method of the middle frame 1110.

[0086] First, in box 1201, the counter value can be normalized according to the associated image region. The more image regions mapped to the reference image region with the corresponding cell, the larger the corresponding normalization denominator or normalization factor may be. For example, a vector field can be applied to the reference image containing only "1" pixel values. This yields a counter value corresponding to the number of image regions mapped to the corresponding reference image region via the vector field. Then, the counter value of the reference image region can be divided element-wise by the reference counter. Thus, each row contains a counter value corresponding to the average of the aggregated pixel values ​​in that row—independent of the number of vectors pointing to the corresponding reference image region, and therefore also independent of the corresponding cell size.

[0087] After this optional normalization in box 1201 (normalization can also be performed later, for example before box 1310, or omitted entirely), all image regions (e.g., pixels, superpixels, or other regions) distinguished in the vector field map are iterated (iteration 1299): box 1205 selects the current image region for a specific iteration 1299. The intensity values ​​of all pixels in each current image region of the microscope image with transfection-related fluorescence contrast can then be added to the counter values ​​in the corresponding reference image region (box 1215). However, in an optional variant, box 1215 is performed only if the image region was previously determined to be considered in box 1210. It is helpful to ignore image regions that cannot be assigned to cells when determining the degree of cell-specific transfection. This can be achieved in several ways. One approach is, in the simplest case, where there is no extracellular signal in the microscope image with transfection fluorescence contrast. In this case, if the image region contains extracellular pixels, a value of 0 is added. However, sometimes it is more accurate to explicitly exclude extracellular image regions from the summation. One approach is to iterate only over image regions located within a given confluence mask. The confluence mask can be determined using generally known techniques, such as machine learning models. Unlike instance segmentation masks, this confluence mask does not require distinguishing between different cell instances. This means that the confluence mask does not segment instances but simply specifies the cell occupancy of different image regions. Compared to instance segmentation masks (e.g., combined with the above...), this approach... Figure 9 Compared to the instance segmentation mask described in 812, this convergence mask can generally be determined particularly robustly.

[0088] Then, in box 1220, it is checked whether there are any other image regions that need to be considered in the subsequent iteration 1299 of box 1205. The iteration will continue until all image regions have been processed.

[0089] Figure 10 One variation is shown where intensity values ​​are iteratively added to a counter in a reference image region. The addition of intensity values ​​can also be achieved through matrix multiplication. Here, the matrix representing the microscope image is multiplied by another matrix determined based on a vector field. This matrix multiplication can be implemented in various variations using iterative software algorithms. However, it is also conceivable to implement matrix multiplication using suitable parallel processor hardware (e.g., a graphics card), thus achieving hardware acceleration. Hardware multiplication can be accelerated in hardware. This is particularly useful for very large images (e.g., mosaic images composed of tile scans).

[0090] In various examples, iteration 1299 can be performed multiple times on the image regions selected one by one in box 1205. This outer loop is formed by iteration 1298 and is shown in conjunction with box 1230. In box 1230, it is checked whether the inner loop of iteration 1299 on the image regions needs to be performed multiple times. If so, box 1205 is performed again on the first image region; subsequently, in the re-executed multiple iterations 1299, all image regions are selected again. This technique of performing iteration 1299 multiple times on all image regions has the following advantages: for vector formation vector chains (see... Figure 5In a vector field map, pixel values ​​can be gradually moved along a vector chain. This is based on the understanding that vectors far from the cell center are generally inaccurate, merely shifting the fluorescence signal closer to the cell center. If further iterations 1298 are performed, these pixel values ​​can be moved from counters far from the cell center to counters closer to the cell center. However, some techniques described above, particularly in conjunction with box 1105.2, can pre-remove the vector chain from the vector field map through appropriate post-processing. This is achieved by replacing two or more vectors that together constitute the vector chain with a single vector from the starting point of the first vector in the vector chain to the last vector in the vector chain. In this case, multiple outer loops based on iteration 1298 are generally unnecessary. However, if multiple iterations 1298 are performed, one or more termination conditions can be considered. For example, one or more termination conditions may include a specific number of iterations 1298. Alternatively or supplementarily, such termination conditions may also consider the spatial distribution of pixel values ​​in a reference image region. For example, this spatial distribution can be compared with a specific preset value. In other words, this means, for example, checking whether the dispersion of the counter values ​​in the reference image region is spatially small enough, or whether the counter values ​​are sufficiently localized to achieve non-zero values. Alternatively or supplementarily, the change in the distribution of pixel values ​​in the reference image region between iterations 1298 and 1298 can also be considered. If the distribution of counter values ​​in the reference image region no longer changes or changes only slightly between iterations 1298 and 1298, a steady state is reached: the counter values ​​have moved to the end, even on the longest vector chain of the vector field graph.

[0091] Then, the degree of transfection is determined in box 1310. In particular, the degree of cell-specific transfection can be determined for each cell. For this purpose, for example, counter values ​​for different reference image regions can be compared to a threshold. If the counter value exceeds the threshold, transfection can be considered to have occurred. If the counter value does not exceed the threshold, the corresponding cell can be considered untransfected, i.e., not expressing the fluorescent dye. Such a threshold can be obtained in various ways. For example, the threshold can be obtained from user input. It is also conceivable to consider the distribution of counter values ​​for different reference image regions (see...). Figure 11The diagram shows the distribution 2451 of counter values ​​2450 for a reference image region and the corresponding threshold 2452, which separates the corresponding frequency peaks. This means that the frequency of a specific counter value for a set of reference image regions can be considered. The threshold can then be determined based on this distribution. Therefore, the threshold can be adaptively adjusted. Another example of an implementation of box 1310 involves determining sub-regions in one or more microscope images, where these sub-regions correspond to different cells. These sub-regions can be determined, for example, based on a vector field map. For example, sub-regions can be formed such that all vectors or vector chains pointing to the same point label the image region assigned to the same cell. However, it is also conceivable to perform individual instance segmentation. When different image regions correspond to different cells, the distribution of pixel values ​​of the image pixels for each cell can be determined. In other words, this means that each cell can obtain the distribution of the corresponding values ​​of its image pixels. These distributions can then be compared, for example, based on a distance metric, to assign different cells to transfected or untransfected cells. For example, cluster analysis can be performed in the space of the pixel value distribution of the image pixels for each cell (see...). Figure 12 The figure shows the distribution of pixel values ​​2460 belonging to different clusters using dotted and dashed lines. Before comparing distributions, the counter values ​​or distributions can be normalized, for example, based on the number of reference image regions, the percentage of image pixels, or cell size. When comparing distributions, the distinctions can be formed as (linear) trajectories across the feature space between clusters of the distribution (see [reference]). Figure 12(Boundary line 2462). The trajectory moves on a hyperplane that represents the interface between transfected and untransfected cells. These points can also be converted to a continuum, for example, through principal component analysis. This yields a one-dimensional feature space. In another implementation example of box 1310, latent feature vectors of pixel values ​​for each cell's image pixels are determined, and these latent feature vectors are then compared with each other. In other words, this means embedding all cell center points. The corresponding feature vectors can then be compared, for example, clusters can be found in the corresponding feature space. A straight line or a manifold of the corresponding dimension can be fitted in the feature space. Another implementation variant of box 1310 involves determining an image patch (Bildpatch) for each cell. Again, cells can be located, for example, based on a vector field map, as described above. These image patches can then be processed using a machine learning network to determine the cell-specific transfection level for each cell. For example, it can be envisioned that a series of image patches (e.g., all with the same resolution) are passed to the machine learning network. The transfection level can then be determined, for example, through a cross-attention mechanism across different channels, based on comparisons between the corresponding latent features. By incorporating cross-attention mechanisms, such machine learning networks can also consider the similarities or differences in appearance between transfected and untransfected cells. This allows for particularly robust differentiation between transfected and unsuccessfully transfected cells based on their appearance in fluorescence microscopy images.

[0092] Figure 13 An electronic data processing device 700 according to various examples is schematically illustrated. The electronic data processing device 700 includes a processor unit (CPU) 705 and a memory (MEM) 706. The electronic data processing device also includes a communication interface (IF) 707. For example, the processor unit 705 can receive one or more microscope images from a microscope via the communication interface 707 (see...). Figure 1 (See box 905). Processor unit 705 can send control data to the microscope to capture microscope images (e.g., using specified imaging parameters or a specific imaging mode and contrast). In another variant, processor unit 705 can load microscope images from local memory 706. Processor 705 can also load and execute program code from memory 706. When processor unit 705 executes program code, it triggers the processor to perform the techniques described herein. For example, the processor unit can perform a combination of... Figure 1 , Figure 2 and Figure 10 The technology described above.

[0093] Figure 14 The data processing flow is illustrated schematically according to different examples. For example, according to Figure 14The data processing flow can be executed by the processor unit 705 based on the program code in the memory 706.

[0094] exist Figure 14 In the process, phase-contrast microscopy image 2905 and fluorescence microscopy image 2906 were obtained. Both images show a scene with one or more cells. Figure 15 Phase-contrast microscopy image 2905 is also shown. See also: [link to reference]. Figure 14 The phase-contrast microscope image 2905 is processed in a machine learning model 2910. The machine learning model 2910 includes one encoding branch 2911 and two decoding branches 2912 and 2913. The machine learning model 2910 outputs a cell mask map 2915, typically a confluence map. The machine learning model 2910 also outputs a cell centroid map 2920. The cell mask map 2915 and / or the cell centroid map 2920 can be further used in various ways. For example, it is conceivable that, as described above, [the image is processed in combination with other methods]. Figure 2 As described in box 1106, these graphs are used to determine confidence maps of the vector field. Alternatively or supplementarily, these graphs can also be used directly to determine the vector field. For example, these graphs can be used to examine the confidence of a machine learning model's predictions of the vector field and to make local corrections where necessary. For example, a cell mask map can be determined, and then the degree of cell-specific transfection can be determined based on that cell mask map. Although Figure 14 The illustration shows a scenario where machine learning models 2910 are jointly trained and have multiple decoding branches 2912 and 2913 to determine cell mask map 2915 and cell centroid map 2920. However, in different examples, it is also conceivable to use different machine learning models to determine cell mask map 2915 and cell centroid map 2920. Furthermore, it is conceivable that the phase contrast microscope image 2905 and fluorescence microscope image 2906 are rescaled before the machine learning model 2910 processes them. Figure 14 Not shown in the text; but it has been combined with the above text. Figure 1 (This is discussed in box 910). Other image preprocessing can also be performed, such as those described above. Figure 1 As described in box 920; this is in Figure 14 It is not shown in the middle either. Figure 16 The cell center point diagram 2920 shows the phase contrast microscopy image 2905; Figure 16 Also shown is a superimposed image 2921 of cell center point diagram 2920 and phase contrast microscopy image 2905.

[0095] Refer again Figure 14 Although in Figure 14Although not shown in the diagram, it is conceivable that cell masking image 2915 could be used to eliminate or reduce interfering signal components of the fluorescence signal in fluorescence microscopy image 2906. The corresponding techniques are described above in conjunction with box 920.

[0096] Based on the cell centroid map 2920 and the phase-contrast microscopy image 2905, the vector field map 2955 can be determined in the machine learning model 2950. The corresponding techniques have been incorporated above. Figure 2 Boxes 1105, 1105.1, and 1105.2 in the text describe this. For example, a machine learning model can be used to determine the vector field graph first, followed by a consistency check. Such techniques have been combined above. Figure 7 and Figure 8 A description was provided. Besides, as... Figure 14 In addition to using a machine learning model to determine the vector field graph 2955, it is also conceivable to use an instance segmentation graph to determine the vector field graph (the corresponding techniques have been combined above). Figure 9 The linear value gradient (see boxes 2605 and 1105.3) is illustrated in these examples. In some cases, machine learning models can be used at inference time to determine the vector field graph, such as... Figure 14 As shown; in order to generate training data for this machine learning model, instance segmentation maps are used to generate weakly supervised or unsupervised ground truth values ​​for vector field maps based on phase contrast microscopy images. Figure 17 The superposition of phase-contrast microscopy image 2905 with the determined vector field map 2955 is shown; it can be seen that outside the cell, the vector field is forced to zero (e.g., based on cell mask map 2915, which indicates confluence without segmenting cell instances, i.e., based on confluence map); inside the cell, the vectors point to a common reference image region located at or near the center point of the corresponding cell. Figure 18 The superposition of vector field image 2955 and fluorescence microscope image 2906 is shown.

[0097] Refer again Figure 14 The fluorescence microscope image 2906 (possibly after the aforementioned background correction and / or rescaling) is then processed in algorithm 2960 along with vector field map 2955. Algorithm 2960 corresponds to box 1110; here the degree of transfection for each cell is determined. This algorithm, for example, can be based on… Figure 10 This means aggregating the fluorescence signal values ​​in the corresponding cells (corresponding to the pixel values ​​in the fluorescence microscope image 2906), for example, by applying a vector field to the fluorescence microscope image 2906. For example, an image can be obtained that aggregates the sum of all pixel values ​​belonging to the cell in the corresponding counter to the center point of the cell. Figure 19 Image 2961 is shown. The corresponding counter values ​​can also be normalized, as described above. Figure 10 As described in box 1201. Subsequently, cells can be binary-assigned to either the "transfected cells" or "untransfected cells" class. For this purpose, counter values ​​(normalized as described above, if necessary) can be compared to corresponding thresholds. Techniques for determining such thresholds have been incorporated above. Figure 11 and Figure 12 It has been described. Figure 20 The corresponding cell classifications are shown. Figure 20 Image 2961 is shown (see image 2961) Figure 19 ) and cell center point diagram 2920 (see Figure 16 The superposition of 2969; in addition, Figure 20 The document also illustrates the allocation of cells into "transfected" and "untransfected" classes based on corresponding thresholds. The overall transfection level can also be determined based on this cell-specific allocation or the degree of cell-specific transfection. For example, statistical data can be generated indicating the proportion of cells in each class. The corresponding transfection analysis results can then be output to the user (e.g., ...). Figure 20 (The cell-level visualization results shown) and / or overall statistics. The relevant techniques have been incorporated above. Figure 1 Box 940 is described.

[0098] Figure 21 This is a flowchart of an exemplary method. Figure 21 The methods described involve computer-implemented transfection analysis. For example, Figure 21 The method in [the document] can achieve this. Figure 16 The data processing flow in the process. Figure 21 The methods in [the document] can also be implemented, for example. Figure 1 The method in the middle.

[0099] In box 1310, a phase-contrast microscopy image is obtained. This phase-contrast microscopy image shows a scene with cells. In box 1315, a fluorescence microscopy image is obtained. This fluorescence microscopy image has fluorescence contrast and also shows a scene with cells. Therefore, boxes 1310 and 1315 can, for example, correspond to box 905. An exemplary phase-contrast microscopy image and a fluorescence microscopy image have been combined. Figure 15 (Phase-contrast microscopy image 2905) and Figure 18 (Fluorescence microscope image 2960) is described. The two microscope images from boxes 1305 and 1310 can be mutually registered, either implicitly or explicitly. If they are not registered, the microscope images can still be mutually registered (not shown; see [link]). Figure 1 (Box 910 in the middle).

[0100] Subsequently, in box 1320, the scaling of the phase-contrast microscopy image 1110 or the fluorescence microscopy image from box 1315 can be checked. Based on the check results, box 1325 can optionally be executed; in box 1325, the microscopy images from boxes 1305 and 1310 are rescaled to display cells according to the structural unit size. This structural unit size can be the cell size in the training image data used to train one or more machine learning models (e.g., for determining vector field maps and / or cell mask maps and / or cell centroid maps). If the scaling is already appropriate, box 1325 can be skipped.

[0101] Image evaluation is then performed in box 1330, taking into account the phase-contrast microscopy image from box 1310. This yields a vector field map. Aspects related to determining the vector field map have been discussed above. Figure 14 In particular, it was described in conjunction with machine learning model 2950. Aspects related to alternative algorithms have also been incorporated. Figure 9 The sum gradient 2605 is described. Therefore, box 1230 corresponds to Figure 2 Box 1105 of the method.

[0102] In box 1335, a second image analysis is performed based on the fluorescence microscopy image from box 1315 and the vector field map from box 1330. This yields scene-specific, cell-specific results indicating the degree of transfection for each cell. Related aspects, especially... Figure 18 , Figure 19 , Figure 20 and Figure 10 This is explained in the text. For example, Figure 10 This describes how to perform multiple iterations (1299) on an image region, such that the counter values ​​associated with a reference image region aggregate the pixel values ​​of each image region. These counter values ​​can then be compared to one or more thresholds. This allows determination of whether a specific cell has been transfected (see [link to documentation]). Figure 20 ).

[0103] Figure 22 This is a flowchart of an exemplary method. Figure 22 Involves the inference phase in box 1810 (where one or more machine learning models are combined with, for example) Figure 1 and Figure 21 The described transfection analysis or Figure 14 The separation of the data processing flow (as described in the diagram) from the training phase in Box 1805. The training phase in Box 1805 involves the training of this type of machine learning model.

[0104] For example, such machine learning models can be used to determine vector field graphs (see...). Figure 14(Machine learning model 2950). Phase-contrast microscopy images can be used as input. Alternatively, other information can be used to determine the vector field map, such as a cell centroid map (for determining the reference image region) and / or a cell mask map, especially, for example, a confluence map (for forcing extracellular vector lengths to zero). A corresponding machine learning model can also be used to determine such cell mask maps and / or cell centroid maps, which can be trained in box 1805 ( Figure 14 Machine learning model 2910).

[0105] The training is performed in box 1805 based on training data. This training data includes corresponding input-output pairs. For example, instance segmentation masks can be specifically used during training, which are determined, for example, by a convolutional neural network based on masked regions (Mask-RCNN). These instance segmentation masks can then be converted into vector field maps (as described in conjunction with box 1105.3—although the description here is for inference). In particular, during training in box 1805, the ground truth of the vector field map can be determined based on these instance segmentation masks by determining the vectors of the vector field map based on linear gradients defined within cell instances, such that these vectors are consistent within each cell and all point to the cell center point. This provides the vector field map as the ground truth for a machine learning model that generates vector field maps based on phase-contrast microscopy images, used to train the model in box 1805. Since no manual annotation is required, the training of the image-to-vector field map model in box 1805 can also be referred to as weakly supervised or unsupervised training. In this context, for example, cell mask maps can also be automatically generated using the Dijkstra algorithm, which is applied to phase-contrast microscopy images. In the inference phase of box 1810, instance segmentation maps are no longer needed.

[0106] In some variations, it is conceivable to use feedback from boxes 1810 to 1805. This is in Figure 22 The dashed arrow indicates this. For example, based on user interactions related to displaying transfection analysis results (the corresponding techniques have been incorporated above). Figure 1 (As described in boxes 940 and 945), more fundamental truths can be obtained and used to retrain one or more machine learning models.

[0107] In summary, the above describes a technique that enables the automated determination of transfection degree or transfection rate through image analysis. That is, it can determine which cells in a sample, and how many cells, are expressing a specific protein as expected.

[0108] The described technique typically involves determining the degree of cell-specific transfection of cells imaged under a microscope. This degree of cell-specific transfection can be determined using at least one machine learning model that processes one or more microscope images. This is an alternative to the manual evaluation of images in existing techniques.

[0109] Microscopic images are acquired at different contrast levels, such as phase contrast and fluorescence contrast. Phase-contrast images typically show all cells consistently, regardless of whether they have been transfected. In contrast, fluorescence-contrast images show only transfected cells.

[0110] Vector field maps can be used to determine the degree of transfection. A vector field map assigns each pixel in an image to a reference region or reference location (e.g., the center of a cell).

[0111] The determined degree of cell-specific transfection can then be displayed along with one or more microscopic images. In this display, transfected and untransfected cells can be highlighted, for example, by using different colors.

[0112] In addition to displaying the transfection level of individual cells, it can also determine the scene-wide transfection level, which indicates the percentage of all transfected cells in the image.

[0113] Of course, the features of the foregoing embodiments and aspects of the present invention can be combined with each other. In particular, these features can be used not only in the said combinations, but also in other combinations or individually, without departing from the scope of the present invention.

[0114] For example, the above describes various techniques related to two-dimensional microscope images. However, the techniques described in this article can also be used for three-dimensional microscope images.

[0115] Furthermore, various techniques for determining the resulting data using multiple microscope images with different contrasts have been described above. However, the various techniques described herein can also be based, for example, on a single microscope image (e.g., with autofluorescence contrast).

[0116] Furthermore, some techniques described above involve separate implementations of the first image analysis for determining the vector field map and the second image analysis for determining the transfection level, e.g., through separate algorithms. However, it is also conceivable to use a common machine learning model, trained end-to-end, to simultaneously determine the vector field map and the transfection level. It is also conceivable to incorporate one or more preprocessing steps into such an end-to-end machine learning model, e.g., rescaling the input image to a structural standard size. Such a machine learning model can also determine a cell centroid map, where, for example, different decoding branches can be mapped from the same feature space to determine both the vector field map and the cell centroid map. Multiple decoding branches can also be used to determine the vector field map on one hand and (as an alternative or supplement to) the cell centroid map on the other. This cell mask map can be formed, for example, as a convergence mask or foreground mask, or it can separate cell instances from each other, i.e., form an instance segmentation map. Using such a cell mask map, the vectors of the vector field map can be corrected (e.g., combined with the above). Figure 7 and Figure 8 (as described above). Alternatively or additionally, this cell mask map can be used to determine the confidence level of a vector field map.

[0117] Furthermore, the techniques for determining two-dimensional vector field maps for two-dimensional microscopic images have been described above. Such two-dimensional microscopic images can be slice images extracted along a surface (e.g., a plane) from a three-dimensional volumetric imaging dataset. Various volumetric imaging modalities are known to provide datasets of microscopic images of a volume. Examples include laser scanning microscopy, light-sheet microscopy, two-photon microscopy, wide-field microscopy, light-field microscopy, and rotating disk microscopy. The same techniques described herein can be applied to each slice image; subsequently, it is conceivable to combine the two-dimensional vector field maps into a three-dimensional vector field map. It is also conceivable to determine cell-specific outcome data for each slice image; then, these two-dimensional cell-specific outcome data are combined to obtain three-dimensional cell-specific outcome data within the imaging volume. When combining two-dimensional vector field maps, it is considered that the endpoints of vectors in different vector field maps (associated with different slice images) may nominally represent the same cells, but terminate at different three-dimensional coordinates in the reference coordinate system of the imaging volume due to the different associated surfaces. This fact can be taken into account during the integration process, for example, by utilizing prior knowledge related to the arrangement of different surfaces or prior knowledge related to the cell extent.

Claims

1. A computer-implemented method, comprising: - Acquire one or more microscope images showing a scene with cells. - Based on at least one of the one or more microscope images, a first image evaluation (1105) is performed to obtain vector field maps (815, 816, 817, 818), which map multiple image regions to corresponding reference image regions, wherein these reference image regions are associated with different cells, and - Based on at least one of the one or more microscope images and the vector field map (815, 816, 817, 818), a second image evaluation (1110) is performed to obtain cell-specific result data for the scene. The cell-specific result data indicates the degree of cell-specific transfection based on fluorescent dyes.

2. The computer-implemented method according to claim 1, in, The second image evaluation includes iterating over the image region (1299). In each iteration (1299), one or more pixel values ​​of at least one of the microscope images in the corresponding image region are added to a counter associated with the corresponding reference image region.

3. The computer-implemented method according to claim 2, in, At least some vectors of the vector field graph form a vector chain. In this process, the image region is iterated (1299) multiple times (1298), causing the pixel values ​​to gradually move along the vector chain.

4. The computer-implemented method according to claim 3, in, The image region is iterated multiple times until the termination condition (1230) is met.

5. The computer-implemented method according to claim 4, in, When the pixel value distribution on the reference image region meets the corresponding preset value, the termination condition (1230) is satisfied.

6. The computer-implemented method according to claim 4 or 5, in, When the change in pixel value distribution in the reference image region is less than a preset value between iterations, the termination condition (1230) is satisfied.

7. The computer-implemented method according to any one of claims 2-6, in, The second image evaluation also includes comparing the counter values ​​of different reference image regions with a threshold (2452) (1310) to determine the degree of cell-specific transfection.

8. The computer-implemented method according to claim 7, in, The second image evaluation also includes determining (2451) the distribution (2451) of the counter values ​​(2450) of the counters for different reference image regions. The threshold (2452) is determined based on the distribution.

9. The computer-implemented method according to any one of claims 2-8, in, The second image evaluation also includes, for example, determining, based on the vector field map, the sub-region of the cell assigned to at least one of the one or more microscope images. The second image evaluation further includes determining the pixel value distribution of each cell's image pixels. The second image evaluation further includes comparing the pixel value distribution to determine the degree of cell-specific transfection.

10. The computer-implemented method according to any one of claims 2-9, in, The second image evaluation (1110) further includes normalizing the counter value of the counter according to the corresponding relevant image region (1201).

11. The computer-implemented method according to any one of claims 2-10, in, The second image evaluation also includes, for example, determining, based on the vector field map, the sub-region of the cell assigned to at least one of the one or more microscope images. The second image evaluation further includes determining the latent feature vector of the pixel values ​​of each cell's image pixels. The second image evaluation further includes comparing the latent feature vectors to determine the degree of cell-specific transfection.

12. The computer-implemented method according to any one of claims 2-10, in, The second image evaluation also includes, for example, determining, based on the vector field map, the sub-region of the cell assigned to at least one of the one or more microscope images. The second image evaluation also includes determining image patches for each cell. A machine learning network is used to determine the degree of cell-specific transfection based on the image patch.

13. The computer-implemented method according to any one of the preceding claims, in, The second image evaluation includes matrix multiplication.

14. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Configure at least one of the first image evaluation, the second image evaluation, or image preprocessing based on the contrast type of at least one of the microscope images.

15. The computer-implemented method according to any one of the preceding claims, in, The vector field graph is determined by a machine learning model (1105.1) that evaluates the first image, and the machine learning model performs image-to-image conversion.

16. A computer-implemented method for training a machine learning model for evaluating a first image according to claim 15, wherein the method comprises: - Determine instance segmentation maps for multiple training images separately. - Based on the instance segmentation map, a vector field map is determined such that the vector of each cell points to the same corresponding point of the cell, and - Use the vector field graph as the base ground truth for training the machine learning model in an unsupervised or weakly supervised manner.

17. The computer-implemented method according to any one of the preceding claims, in, The execution of the first image evaluation includes: determining (1105.3) an instance segmentation map (812) of the corresponding microscope image, segmenting different cells; and determining a vector of a vector field map of each instance of the instance segmentation map based on a linear value gradient (2605) defined between the edges of each cell.

18. The computer-implemented method according to any one of the preceding claims, in, The first image evaluation determines the vector field map while taking into account at least one edge condition.

19. The computer-implemented method according to claim 18, in, The at least one edge condition includes a preset value for the spatial distance between the reference image regions corresponding to adjacent vectors of the vector field map.

20. The computer-implemented method according to claim 18 or 19, in, The at least one edge condition includes a preset value for the distance (2621) between the reference image region and the cell center point drawn on the cell center point map.

21. The computer-implemented method according to any one of claims 18-20, in, The at least one edge condition includes a preset value for the length difference between adjacent vectors of the vector field graph.

22. The computer-implemented method according to any one of claims 18-21, The at least one edge condition selectively disables or enables vectors in the vector field map based on which sub-region of the cell the corresponding vector begins and / or ends in.

23. The computer-implemented method according to any one of claims 18-22, in, In the cell edge region, the first image evaluation does not consider at least one edge condition. The cell edge region is selectively determined based on a cell mask map.

24. The computer-implemented method according to any one of claims 18-23, wherein the method further comprises: - Select the at least one edge condition based on the contrast type of at least one of the one or more microscope images.

25. The computer-implemented method according to any one of claims 18-24, wherein the method further comprises: - If at least one edge condition is not met: replace the corresponding vector in the vector field graph based on one or more neighboring vectors of the vector field graph.

26. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Based on the neighborhood relationships between the reference image regions, integrate the reference image regions in the vector field map.

27. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Determine (1106) a confidence map, the confidence map indicating the confidence values ​​of the vectors in the vector field map, The second image evaluation is also based on the confidence map.

28. The computer-implemented method according to claim 27, in, The confidence map is determined based on a comparison between the vector field map and another vector field map, which is determined based on another microscope image among the one or more microscope images.

29. The computer-implemented method according to claim 28, in, The at least one microscope image associated with the vector field map and the at least one other microscope image associated with the other vector field map have different contrasts.

30. The computer-implemented method according to any one of claims 27-29, in, The confidence map is determined based on the distribution of the length and / or direction of the vectors in the vector field map, and / or based on the distance from the reference image region to the cell center point plotted on the cell center point map, and / or based on the cell mask map.

31. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Determine the cell mask image, The second image evaluation is also based on the cell mask image.

32. The computer-implemented method according to any one of the preceding claims, in, At least one of the one or more microscope images evaluated in the second image evaluation includes: a first microscope image with fluorescence contrast specific to the fluorescent dye, and an optional second microscope image, wherein the second microscope image has contrast of cellular structures specifically marked to correspond to the cell region of the reference image.

33. The computer-implemented method according to any one of the preceding claims, in, The first image evaluation and the second image evaluation are performed by a common machine learning model.

34. The computer-implemented method according to any one of the preceding claims, in, The one or more microscope images include multiple slice images, each slice image being associated with a different surface of the three-dimensional volume of the scene containing cells. The second image evaluation combines the vector field map with at least one other vector field map, wherein the vector field map and each of the at least one other vector field map are associated with a different slice plane.

35. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Convert the vector field graph into an instance segmentation graph, and - Display the instance segmentation diagram described in (940).

36. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Before performing the first image evaluation and the second image evaluation, perform at least one of the following operations on the one or more microscope images: (915, 920) smoothing operation, denoising operation, or rescaling operation.

37. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Based on the degree of cell-specific transfection, determine the first pixel value distribution of the image pixels assigned to the transfected cells, and - Based on the cell-specific transfection level, determine the second pixel value distribution of the image pixels assigned to untransfected cells, and - Display at least one of the first distribution or the second distribution.

38. The computer-implemented method according to any one of the preceding claims, wherein the method further comprises: - Display (940) at least one of the microscope images of the one or more microscope images superimposed with a cell-specific graphic indicator of the degree of cell-specific transfection.

39. A computer-implemented method, comprising: - Acquire microscopic images with phase contrast, showing cellular scenes. - Acquire microscope images with fluorescence contrast, showing cellular scenes. - Inspect the scaling ratio of the microscope image with phase contrast and the microscope image with fluorescence contrast, and selectively change the scaling ratio based on the inspection results. Based on the phase-contrast microscope image, image evaluation is performed to obtain a vector field map that maps multiple image regions to corresponding reference image regions, wherein these reference image regions are associated with different cells. - Based on the fluorescence-contrast microscopic image and the vector field map, a second image evaluation is performed to obtain cell-specific result data for the scene, wherein the cell-specific result data indicates the degree of cell-specific transfection based on fluorescent dye transfection. The second image evaluation includes iterating over the image region. In each iteration, one or more pixel values ​​from at least one of the one or more microscope images in the corresponding image region are added to a counter associated with the corresponding reference image region. The second image evaluation further includes comparing the counter values ​​of counters in different reference image regions with a threshold to determine the degree of cell-specific transfection.

40. An electronic data processing apparatus comprising a processor unit (705) adapted to perform the method according to any one of the preceding claims.

Images