Transfection imaging of cell wall annotations

Abstract

Aspects of the present disclosure relate to techniques for identifying cellular structures in microscopic images. In particular, techniques are described for identifying cell walls (i.e., cell edges) in microscope images, such as phase contrast images. For this purpose, a machine learning algorithm can be used, for example, using an artificial neural network. Techniques are described how annotations can be created as the basis facts for training of machine learning algorithms, for example, based on fluorescence channels transfecting image data. Further operations can then be performed based on the corresponding localized cell walls, such as performing cell instance annotations that divide cell instances, for further training of machine learning algorithms.

Description

Technical Field

[0001] Various examples of the present invention relate to a technique related to the recognition of cell structures in a microscope image, and particularly relate to a technique of machine learning and a technique of training a machine learning algorithm. Background Art

[0002] In biology and medicine, microscopy is used to study biological samples. Microscope image analysis is often used to identify and classify individual cells, for example to gain information about the cell's structure, function, and potential abnormalities. The concept of instance segmentation is often used for this task.

[0003] Instance segmentation is the name for the process by which each cell in a microscope image is individually identified and separated from other cells. This means determining not only the presence and location of a cell within an image, but also its exact shape and size. Instance segmentation makes it possible to examine cell properties and cell behavior at an individual level.

[0004] The challenge of instance segmentation lies in identifying the boundaries of each cell, especially in complex images or densely populated images where cells often overlap or have very similar features. In addition, cells vary in shape, size, and structure, making instance segmentation even more difficult.

[0005] To overcome these challenges, various techniques and algorithms are used, including machine learning algorithms.

[0006] Ronneberger et al. developed a neural network called U-Net that is specifically designed for segmenting biomedical images. See Ronneberger, O., Fischer, P., and Brox, T. (October 2015) U-net: Convolutional networks for biomedical image segmentation, In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 234–241), Springer, Cham. U-Net uses a special architecture that allows the network to exploit both local and global context information to make accurate segmentation decisions. U-Net has also been used for instance segmentation of cells, see, e.g., Falk, Thorsten et al. U-Net: deep learning for cell counting, detection, and morphometry, Nature methods 16.1 (2019): 67–70.

[0007] To train a machine learning algorithm, such as Ronneberger et al.'s U-Net, annotations are used as ground truth ("ground truth annotations" or simply annotations). These annotations serve as a comparison standard or "source of truth" for the training and / or validation of machine learning algorithms (ML algorithms). These annotations provide the correct answers that the ML algorithm should learn to reproduce.

[0008] However, one challenge lies in the creation of annotations. Since each individual cell in the microscope image must be precisely delineated, the manual creation of annotations requires considerable effort. This is a time-consuming and error-prone task that requires expertise and is therefore often costly. Furthermore, differences in interpretation between different annotators can lead to inconsistencies in the annotations, which in turn affect the quality of the trained model. Summary of the invention

[0009] Therefore, manual creation of annotations is a major barrier to scaling and efficiency of instance segmentation of cells in microscopy images. Solutions that can automate or at least simplify this process are needed to increase efficiency while ensuring accuracy and reliability of the results.

[0010] More generally, techniques are needed to train machine learning algorithms that recognize cellular features or cellular structures. Annotations that serve as ground truth for such training need to be obtained.

[0011] This object is achieved by the features of the independent patent claims. The dependent claims describe embodiments.

[0012] A computer-implemented method is disclosed. An ML algorithm is trained based on cell wall annotations as ground truth and non-fluorescent channels as input.

[0013] There are several variants for creating cell wall annotations. In some variants, cell wall annotations can be created manually. For example, cell wall annotations can be manually generated from microscope image data in a phase contrast channel. Alternatively or additionally, cell wall annotations can be generated from transfection image data in a non-fluorescent channel. For this purpose, automatic image processing of the fluorescent channel can be implemented to locate the boundary area between the fluorescent area and the non-fluorescent area in this way.

[0014] The ML algorithm trained based on the cell wall annotation can then be further used in various ways. For example, the output of the ML algorithm can be evaluated in order to perform further evaluation of the corresponding microscope image based on the cell wall identified by the ML algorithm. For example, instance segmentation of cells can be performed based on the identified cell wall. Alternatively or additionally, visible cells can be counted and / or the degree of confluence can be determined.

[0015] Another way to use an ML algorithm trained on cell wall annotations is to generate further annotations as ground truth for training further ML algorithms. For example, cells can be segmented based on the cell wall annotations output by an ML algorithm, thereby creating cell instance annotations. These cell instance annotations are used as training further ML algorithms that solve the corresponding segmentation task.

[0016] Such techniques described above are based on the following discovery: obtaining cell wall annotations is often easier than, for example, obtaining cell instance annotations. In some examples, cell wall annotations can be easily created manually. Annotating a single cell wall takes less time than noting down an entire cell. In a further example, it was found that transfection imaging is particularly suitable for automatically determining cell wall annotations on corresponding image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A data processing device according to various examples is schematically illustrated.

[0018] Figure 2 is a flow chart of an exemplary method.

[0019] Figure 3 is a flow chart of an exemplary method.

[0020] Figure 4 An example of data processing of transfection image data is shown. DETAILED DESCRIPTION

[0021] Techniques for identifying and locating structures of cells in microscope images are described below. For example, cell walls or other cell structures can be identified and located. It is also conceivable to identify organelles. Appropriate segmentation of these parts of the cell can be provided. Other localization techniques (e.g., point localization or bounding boxes) are also conceivable.

[0022] The cell wall is sometimes called the cell membrane. The cell wall surrounds the cell and regulates the flow of substances in and out. The cell wall is mainly composed of a double layer of phospholipids and proteins. The cell wall is different from other cellular components such as the cytoplasm and the nucleus.

[0023] In the following, particular reference is made to variants in which the cell wall is identified and localized. However, the corresponding technology can also be used in conjunction with the identification and localization of other structures in the cell.

[0024] A technique for identifying and locating the cell wall of a cell using an ML algorithm is used. In particular, a deep neural network can be used, for example a convolutional neural network can be used. A transformer network can also be used.

[0025] Techniques are described that allow automatic provision of ground truth annotations to train ML algorithms to solve the corresponding task of locating cell walls.

[0026] The output of the trained ML algorithm is then the location of the cell wall. For example, the cell wall can be marked with the aid of a spline. A binary mask image can be output that determines the location of the cell wall.

[0027] This output can be used in several ways. Further evaluations can be performed in downstream image processing steps based on the identified cell wall. For example, instance segmentation of cells can be implemented based on the identified cell wall (or other identified structure). The degree of confluence can also be determined. Alternatively or additionally, the number of cells or the cell density can be determined. Then, such data can then be used as ground truth annotations for the training of further ML algorithms, which directly (i.e., without first locating the cell wall) solve the corresponding task (e.g., determination of the degree of confluence of cells, cell instance segmentation or cells). This means that, in such an example, the training of the ML algorithm for the location of the cell wall is merely a means for the purpose of creating further annotations for the training of further ML algorithms, the tasks of which are related to or based on the location of the cell wall, but nevertheless have different concerns. Such a technique is based on the discovery that the training of the ML algorithm for the location of the cell wall can be performed relatively simply and reliably, and in particular the training can also be performed automatically. Then, based on such a first step, the more complex problem of creating annotations for more complex tasks can be solved in a simplified manner.

[0028] To create annotations that serve as ground truth for training an ML algorithm that localizes cell walls (referred to as the Cell Wall ML algorithm), transfection imaging was used in some examples.

[0029] Transfection imaging is a technique known in the field of molecular biology. Transfection imaging is used to study the role of specific genes in living cells. The study is achieved by introducing DNA (or RNA) into cells encoding the gene of interest, often combined with a reporter gene encoding a fluorescent protein. As a result, cells that have successfully absorbed the DNA and integrated the DNA into their genome (successful transfection) express the fluorescent protein and emit cold light under the microscope, while cells that failed to be transfected do not express or emit cold light. Microscopy used in such studies usually uses two channels. The first is a phase contrast channel that does not use fluorescence. Phase contrast microscopy is a technique that uses the refraction of light to observe details in a sample. This technique allows living cells to be seen in culture without staining or fixation, which is often necessary in fluorescence microscopy. Therefore, the phase contrast channel provides a "natural" view of the cell and can be used to determine the morphology and position of the cell. In the technology described in this article, the phase contrast channel is used as the input of the machine learning algorithm. The second channel is the fluorescence channel. In this channel, fluorescence generated by the expression of the fluorescent protein is detected. Fluorescence microscopy uses the ability of specific molecules to absorb light and radiate light again to generate images. In the context of transfection imaging, the fluorescence channel shows which cells have successfully taken up the transfected DNA, because only these cells will express the fluorescent protein and emit luminescence.

[0030] In the framework of the techniques described herein, the fluorescence channel is used to create annotations for training a cell wall ML algorithm. Various examples are based on the finding that it is irrelevant for successfully training a cell wall ML algorithm that only some cells express fluorescent proteins and appear fluorescent in the fluorescence channel. Nevertheless, a cell wall ML algorithm can be trained in such a way that cell walls can be reliably identified. Details of this aspect are described in detail below using the accompanying drawings.

[0031] Figure 1A data processing device 90 according to various examples is shown. The data processing device 90 includes a processor 91 and a memory 92 and a communication interface 93. The processor 91 is capable of receiving a microscope image or general image data via the communication interface 93. For example, the processor 91 is capable of receiving transfection image data. The processor 91 is capable of receiving image data from a database via the communication interface 93. The processor 91 is capable of receiving image data from a microscope, for example, based on a control command previously sent to the microscope and triggering the microscope to acquire corresponding image data. The processor 91 is capable of loading image data from the memory 92 alternatively or additionally. The processor 91 is capable of loading program code from the memory 92 and executing the program code. When the processor 91 executes such program code, the effect is that the processor implements the technology as described herein. In particular, the processor 91 is capable of implementing technology related to the training of one or more ML algorithms. Alternatively or additionally, the processor is also capable of identifying cell structures based on the execution of the ML algorithm as part of reasoning, or solving tasks such as cell instance segmentation.

[0032] Figure 2 is a flow chart of an exemplary method. Figure 2 The method can be performed by a data processing device, for example, by Figure 1 The data processing device 90 is used to execute. For example, Figure 2 The method can be executed by a processor, for example, the processor 91 , based on a program code from a memory, for example, the memory 92 .

[0033] Figure 2 The method involves techniques related to the localization of cell walls in microscopy images using a cell wall ML algorithm.

[0034] Figure 2 The method comprises a training phase 3098 and an inference phase 3099 of the cell wall ML algorithm. As a general rule, in the various examples described herein, it is conceivable to perform either the training phase 3098 or the inference phase 3099, but not both.

[0035] First, aspects related to the training phase 3098 of the cell wall ML algorithm are described.

[0036] In block 3005, transfection image data is received. These data include a fluorescent channel and a non-fluorescent channel. The non-fluorescent channel is typically a phase contrast channel. The transfection image data forms an accumulation of cells. Not all cells express the dye. This means that not all cells in the fluorescent channel are fluorescent. Only a portion of all cells fluoresce in the fluorescent channel.

[0037] In block 3010, automatic image processing of the fluorescence channel is then performed. This locates the border regions between fluorescent and non-fluorescent regions. Such border regions are placed where the cell walls are - but not every cell wall has a border region: there are cells in the fluorescence channel that contain the fluorescent dye (successful transfection), and cells that do not (failed transfection). Nevertheless, based on this image processing, one or more types of annotations can be created in block 3015 that are valuable for subsequent training of the cell wall ML algorithm. In particular, these annotations can be automatically determined from the transfection image data.

[0038] The following text explains what types of annotations can be determined in box 3015.

[0039] The first type is a cell wall annotation along the localized border region between the fluorescent and non-fluorescent regions. Thus, the cell wall annotation marks the cell wall—that is, the edge of the cell, the region where the cell edge is located.

[0040] The second type is non-cell wall annotations. These non-cell wall annotations mark image areas for which it can be assumed with sufficiently high certainty that no cell wall is localized in this area. In particular, non-cell wall annotations can be created in image areas adjacent to localized border areas. This is based on the finding that in localized border areas, it can be assumed with high probability that a cell wall is localized there; conversely, it can be assumed with high probability that no cell wall is localized adjacent to and parallel to such a localized cell wall.

[0041] The third type is unknown annotations. These annotations mark image areas where no statement can be made about the location of the cell wall. Unknown annotations can be created for image areas with the minimum distance to all located border areas. In image areas where no fluorescent signal is detected, due to partial transfection, no statement about the presence of the cell wall or the absence of the cell wall can be made. Unknown annotations can contain all areas of the fluorescence channel for which no cell wall annotation or non-cell wall annotation is provided, for example. Unknown annotations can still be created based on an estimate of the presence of cells in the corresponding image area, which estimate is generated based on a non-fluorescent channel (e.g., phase contrast). For example, with the aid of grayscale variance analysis or other simple image processing and / or the use of a convergence mask, a statement about the probability that a cell is located in the corresponding image area can be made. Then, when creating unknown annotations, this probability can be further taken into account.

[0042] In summary, several types of annotations have been described above that can be used as the basis for training a cell wall ML algorithm. Training is then performed in box 3020. Training of a U-Net or another neural network is an iterative process that typically consists of multiple stages. The first step is forward propagation, in which the network processes an input image (or a batch of images) (in this case, a fluorescence channel) and generates an output, such as a localized cell wall. In this stage, the weights and distortions of the network that were initially randomly initialized are used to calculate the output. The generated output is then compared to the annotations (see box 3015) to calculate the error or difference between the network prediction and the annotations. The error is typically referred to as a loss function and is a measure of how well the network performs the task. Once the loss is calculated, back propagation begins. During this stage, the weights and distortions of the network are adapted to minimize the loss. Minimizing the loss is typically accomplished through a process of gradient descent, in which the derivative of the loss for each weight and each distortion is calculated. The weights and distortions are then updated in the opposite direction of the gradient to reduce the loss. This process is repeated over multiple epochs (passes through the entire training dataset) until the network reaches acceptable performance or the loss no longer decreases significantly. In this way, the neural network learns to effectively localize cell walls in microscopy images based on the annotations, which serve as guides.

[0043] During training, image regions with cell wall annotations and non-cell wall annotations are typically incorporated into the training; image regions with unknown annotations (for regions outside the boundary between fluorescent signal and non-fluorescent signal) are ignored. For the ML algorithm, it is possible to "ignore" regions annotated with unknown annotations (ignore labels) by not including the predictions of the ML algorithm in the calculation of the loss. Instead of hard ignoring labels, it is also possible to implement a soft weighting of each pixel, for example depending on how close the pixel is to another annotation, for example how close the corresponding pixel is to the boundary between fluorescent signal and non-fluorescent signal.

[0044] It is conceivable that, in particular, large image regions are provided with unknown annotations. In order to still enable accurate training, a so-called consistency loss function can be calculated; in particular (but not necessarily exclusively) in image regions provided with unknown annotations. See, for example, German patent application 102022114888 of June 14, 2022. The consistency loss function provides a loss signal based only on the input of the consistency transformation. Such a consistency loss function can be combined with a conventional loss function, for example in a weighted combination. However, it is also conceivable to determine a consistency loss function for the entire input image. In another variant, similar to the consistency loss function, a loss signal can be generated that minimizes the entropy of the output in image regions with unknown annotations, thereby maximizing the certainty of the prediction. For example, see Wang Dequan et al., "Tent: Fully test-time adaptation by entropy minimization", arXiv preprint arXiv:2006.10726 (2020).

[0045] The training of the cell wall ML algorithm can be performed iteratively in the example, which is called bootstrapping. In the first iteration of training, the cell wall ML algorithm provides the location of the cell wall, especially within the cell aggregate. This information can then be used in the next iteration to expand the location of the edge area of ​​the cell even within the cell aggregate. Therefore, the training program is optionally performed several times, and the respective results enrich the basic facts related to the previous iteration. In the first iteration, only the annotations determined based on the transfection image data as described above are available. The ML algorithm trained in this way then also provides cell edge predictions, which are not yet available in the training data of the first iteration (e.g., cell edges within cell aggregates). The predictions of the ML algorithm can now be used to generate extended annotations that are more extensive (but not perfect) than the original annotations. Then, this will train the ML algorithm again, for example, until convergence is determined.

[0046] After training, the trained cell wall ML algorithm can be applied to the microscope image, such as the fluorescence channel, in block 3025. The cell wall ML algorithm can then correctly distinguish between "cell wall" and "non-cell wall" for all pixels of the microscope image - even if only a portion of the transfected cell culture was used for training. It has been found that the cell wall ML algorithm trained in this manner reliably identifies all cell walls in the microscope image; even if the cell walls were only partially provided with corresponding cell wall annotations during training.

[0047] In a subsequent evaluation step, in block 3030, which is generally optional, an instance segmentation of individual cells in the corresponding image data can then be performed, for example, based on the localized cell walls from block 3025. This determination of instance segmentation based on the localized cell walls can be achieved with the aid of various image processing techniques. For example, conventional image processing techniques such as watershed, flood filling, graph segmentation, etc. can be used. Here, image regions are separated based on (binarized) cell walls. Since the relevant regions thus identified can represent individual instances or background regions, post-processing is necessary. In this process, regions that do not meet the criteria for cell instances are discarded. Different criteria are conceivable here, such as the size, shape or arrangement of the regions; and / or the statistics / properties of the phase contrast and / or fluorescence contrast in the regions; and / or the output of existing algorithms for estimating cell centers and / or cell confluences, see, for example, European patent application 21200420.4 of October 1, 2021. For each determined region, a check is performed whether the region is assigned (exactly one) cell center and / or whether the region coincides with the determined confluence.

[0048] In a further variant, an extension of the segmentation method from German patent application 102021125575.9 of October 1, 2021 can be used in block 3030. The method described therein uses a cell confluence estimate together with a determined cell center to perform a Voronoi partitioning and thus a separation of cell regions. Based on a determined cell wall, the method can be extended to a "weighted" partitioning, i.e. the Voronoi partitions do not grow evenly in all directions until they coincide, but are locally slowed down / accelerated by a determined probability of the cell wall. Thus, when a partition encounters a cell wall, the partition spreads more slowly. Thus, for irregular shapes, the separation of regions can also be determined more accurately.

[0049] In a further variant, an ML algorithm that maps a cell wall image (as output of a cell wall ML algorithm) to cell instances can be used in block 3030. The ML algorithm can be used as a model for semantic segmentation (input: cell wall image; output: segmentation mask) or as an instance segmentation model (input: cell wall image; output: single coordinates and associated instance segmentation mask). The advantage is that the training data for the ML algorithm can be simply generated by simulation, since no textual attributes of the cells are used as input, but only morphological attributes (shape, size, extent, arrangement of the cell wall). Such input data with associated output masks can be easily generated in large quantities by a parameterizable simulation model without manual annotation and without recording on the device.

[0050] and Figure 2In connection with this, a variant is described above in which a machine learning algorithm is trained to locate cell walls based on non-fluorescent channels (block 3020) as part of the training in block 3098. In the context of inference in block 3099, the cell walls can then be located, and further evaluation 3030 can then be performed to obtain, for example, cell instance segmentation. Figure 2 In this variant of , no ML algorithm is used to directly perform cell instance segmentation based on non-fluorescent channels. Instead, a two-step process occurs in which (i) cell walls are located and (ii) further evaluation is performed, such as for cell instance segmentation. Figure 3 Another variation is shown in which a corresponding end-to-end ML algorithm is trained in block 3098 to predict cell instance segmentations as part of the training.

[0051] Figure 3 is a flow chart of an exemplary method. For example, Figure 3 The method can be performed by a data processing device, such as by Figure 1 The data processing device 90 executes. Figure 3 The method can be implemented, for example, by a processor, such as the processor 91 , based on program codes from a memory, such as the memory 92 .

[0052] In block 3105, transfection image data is received. Block 3105 corresponds to Figure 2 Box 3005.

[0053] In block 3110, image processing is performed on the transfection image data from block 3105. Block 3110 corresponds to Figure 2 Box 3010.

[0054] Then, one or more types of annotations are created in block 3115. Block 3115 corresponds to Figure 2 Box 3015.

[0055] In block 3120, a cell wall ML algorithm is then trained based on the annotations from block 3115. Block 3120 corresponds to block 3020.

[0056] The cell wall ML algorithm trained in block 3120 is then used in block 3125 to locate the cell wall based on the non-fluorescent channel of the transfection image data from block 3105 or based on the non-fluorescent channel of further image data (for which no fluorescent channel need be available at all).

[0057] In block 3130, an evaluation is then performed based on the found cell walls. In particular, instance segmentation of individual cells is performed in the non-fluorescent channel.

[0058] This means that box 3125 corresponds to box 3025, and box 3130 corresponds to box 3030.

[0059] The instance segmentation in block 3130 is then used in block 3135 to train another ML algorithm for instance segmentation (a cell instance segmentation ML algorithm). The instance segmentation from block 3130 is used as the ground truth for this training, and the non-fluorescent channel based on which the cell wall is localized is used as input in block 3125. Then, in the context of inference, block 3099, the cell instance segmentation ML algorithm can be used to directly segment the cell wall of the non-fluorescent channel, block 3140. Thus, the cell instance segmentation ML algorithm can be applied, for example, directly to conventional phase contrast microscopy images and directly output cell instance segmentations.

[0060] For example, a cell instance segmentation ML algorithm can be a U-net. The main components of a U-net are the encoder side and the decoder side. The encoder side consists of a continuous sequence of convolutional layer blocks followed by downsampling operations such as max pooling to reduce the spatial dimension. These layers are used to obtain background information and extract features of the input image. The decoder side consists of a continuous sequence of convolutional layer blocks followed by upsampling operations such as bilinear upsampling or transposed convolution to restore the spatial dimension. The decoder side aims to amplify and refine features to generate detailed segmentation maps. To connect information from the encoder side to the corresponding layers on the decoder side, U-Net uses skip connections. This connection bridges the spatial distance between layers and allows the decoder side to obtain more detailed information from the encoder side for more accurate segmentation results. In addition, each convolutional layer block of U-Net uses a convolution operation and an activation function, such as the ReLU function, to map nonlinearity and improve the model's ability to extract features.

[0061] A further conceivable architecture for a cell instance segmentation ML algorithm is the so-called Mask R-CNN: This is an extension of the Faster R-CNN algorithm, which was developed specifically for instance segmentation. Mask R-CNN adds an additional branching layer to Faster R-CNN, which creates a binary mask for each region of interest (ROI). This way, Mask R-CNN is able to determine not only the location and class of objects in the image, but also the exact shape of the objects by means of segmentation. See Fujita, Seiya, and Xianhua Han, "Cell detection and segmentation in microscopy images with improved mask R-CNN," Proceedings of the Asian Conference on Computer Vision, 2020.

[0062] A further architecture for cell instance segmentation ML algorithms is StarDist. See Fazeli, Elnaz et al., Automated cell tracking using StarDistand TrackMate, bioRxiv (2020): 2020-09. StarDist can be considered an extension of U-Net and is based on the idea of ​​representing the shape and position of each instance by a set of star coordinates (hence the name "StarDist"). The StarDist architecture consists of two main parts: one part predicts the probability of the cell center (similar to U-Net) and the other part predicts the star coordinates of each cell. The star coordinates represent the shape and position of the cell with respect to its center. By combining these two parts, StarDist is able to segment each cell in a microscopy image individually and determine the exact shape and position of the cell. The main advantage of the StarDist architecture is its ability to effectively segment overlapping cells and touching cells, which is a common challenge in microscopy image analysis. In addition, StarDist is able to handle cells of different sizes and shapes, making it a versatile solution for instance segmentation.

[0063] With the help of Figure 3 This approach enables the creation of a large number of cell instance annotations, e.g. on any desired phase contrast image. Large amounts of data are also necessary for accurate training, since the corresponding cell instance segmentation ML algorithms are usually very data-intensive and often require manual annotation. However, using Figure 3 The method described in can automatically create the corresponding annotations and greatly speed up the training process.

[0064] Although a variant has been described above in which a cell instance segmentation ML algorithm is trained in block 3115 to solve the instance segmentation task, other types of ML algorithms can also be trained. For example, annotations to an ML algorithm for estimating cell centers or cell confluence can be created in block 3130. Each mask itself provides a confluence from a phase contrast image, and the cell center can be determined from the center of the mask. Here, as mentioned above in connection with the cell instance segmentation task, manual annotation work is also reduced.

[0065] Figure 4 The processing pipeline 100 is shown in Figure 2 Aspects of the method and Figure 3 aspects of the method.

[0066] Figure 4 A data processing pipeline 100 is shown according to various examples. Figure 4It is shown how the cell wall ML algorithm 190 can be trained to identify cell walls.

[0067] Figure 4 At the top an overlay 101 of the fluorescent channel 106 and the non-fluorescent channel 105 (here: phase contrast channel) of the transfection image data is shown (compare box 3005 and box 3105). Figure 4 A corresponding zoom-in 102 is also shown, where an accumulation of cells can be seen, not all of which express the dye, being imaged in the transfection image data. Figure 4 In the figure, short dashed lines are used to highlight the transfected cell aggregates; contrasting differences are characterized by different colors that are not easily seen in the grayscale rendering. In addition, dot-dashed lines are used to highlight the cell boundaries within the transfected cell aggregates. Finally, non-transfected cells are marked with arrows. )

[0068] In addition, an exemplary embodiment of block 3010 and block 3110 is shown, i.e., automatic image processing of the fluorescence channel 106, in order to find the boundary area between the fluorescent area and the non-fluorescent area. For example, a low-pass filtering of the fluorescence channel 106 can be initially implemented. This type of low-pass filtering is usually used to reduce high-frequency noise in the image data, thereby improving the quality of the input image. Binarization can then be implemented so that a corresponding mask image 111 is obtained. The mask image 111 can also be called a binary image. There, a fixed threshold or an automatically determined threshold is used, for example, by means of the Otsu method.

[0069] Then, in the mask image 111, contours are identified, for example with the aid of the Suzuki algorithm, see Suzuki, S. (1985) "Topological structural analysis of digitized binary images by border following", Computer Vision, Graphics and Image Processing, 30(1), 32–46. The corresponding cell wall image 112 is Figure 4 is shown in .

[0070] The corresponding contours can then be enlarged or "washed out". This gives a mask for the class "cell wall" (cell wall annotation). By further enlarging or further washing out these contour edges, a mask for the class "no cell wall" (non-cell wall annotation) is obtained. This Figure 4(where the cell wall annotation - the darker area between two bright areas, is marked with a short dashed line, and the non-cell wall annotation is marked with an adjacent bright line; see also 184, 185). As a result, two masks are obtained, one mask encoding pixels containing cell walls; the other mask encoding pixels not containing cell walls. All uncoded pixels can be assigned to unknown annotations.

[0071] Techniques have been described above how cell wall annotations and non-cell wall annotations can be created based on the transition from one pixel having a fluorescent signal to another pixel not having a fluorescent signal.

[0072] It should be noted that there are other possible implementation variants of the image processing algorithm for cell wall recognition. These variants can include both conventional algorithms and ML algorithms, in particular deep neural networks. In the latter case, a network designed to recognize the interface between fluorescent and non-fluorescent signals has an easier task and can therefore be trained more easily and with less data than a network designed to directly recognize cell boundaries in phase contrast images.

[0073] Variants or extensions that exploit the spatial structure of the fluorescence channel are also conceivable. For example, even if cells in a cell aggregate all express a fluorescent dye, this allows the identification of cell edges within a cell aggregate. Such an analysis of the fluorescence channel structure can be performed, for example, with the aid of a corresponding ML algorithm.

[0074] Figure 4 Also shown is an overlay 105-1 of a cell wall annotation 184 (highlighted by a short dashed line) with the phase contrast channel 105. Additionally, an overlay 105-2 of a non-cell wall annotation 185 (solid line) with the phase contrast channel 105 is shown. These annotations (e.g., along with the unknown annotations) are then used as ground truth for the training of the machine learning algorithm 190, which identifies the cell wall as output 238.

[0075] The output 238 of the ML algorithm 190, i.e., the estimated cell wall, implicitly contains information about individual cell instances. Optionally, the estimated cell wall can then be explicitly converted into individual cell fragments by post-processing. This has been described above in conjunction with blocks 3030 and 3130. The output 238 can be used to create annotations for training further machine learning algorithms - for example, for direct instance segmentation of cells - for comparison Figure 3 ; or output 238 can be in this way (see Figure 2 ) are evaluated in the context of inference on segmented cells.

[0076] In summary, a technique has been described above that allows the creation of annotations for the training of one or more ML algorithms based on transfection image data. The transfection image data shows a variety of cells, only some of which express the dye. The annotations can then be used to train an ML algorithm to locate all cell walls. Based on this, cell instance segmentation can be performed or further ML algorithms can be trained to directly solve the corresponding task.

[0077] It goes without saying that the features of the embodiments and aspects of the present invention described above can be combined with each other. In particular, these features can be used not only in the described combinations, but also in other combinations or alone without departing from the scope of the present invention.

[0078] Various examples have been described above in conjunction with automatic image processing of the fluorescent channel from transfection image data, so as to obtain the boundary area between the fluorescent area and the non-fluorescent area. Based on this, the cell wall annotation is then created along the located boundary area. In some variants, it is also conceivable to manually create the cell wall annotation based on general image data, for example, based on the phase contrast channel. Even in such an example, it is possible to then implement the training of the cell wall ML algorithm based on these manually created cell wall annotations as described in detail above. Even in this variant of the manual creation of the cell wall annotation, the training work can be significantly reduced compared to the reference implementation in which the cell instances are manually annotated. The manual annotation of any continuous cell is easier and faster than the complete annotation of all individual cells in the image.

[0079] Furthermore, techniques have been described above in which cells fluoresce in a fluorescence channel. It is also possible that only the cell wall fluoresces in a fluorescence channel, i.e. the interior of the cell does not fluoresce. The corresponding algorithms described herein can be directly adapted to provide cell wall annotations based on such information.

Claims

1. A computer-implemented method, wherein: The method comprises: - receiving (3005, 3105) transfection image data, the transfection image data comprising a fluorescent channel (101) and a non-fluorescent channel (105), wherein the transfection image data represents an accumulation of cells, not all of which express a dye, - performing (3010, 3110) automatic image processing of the fluorescence channel (105) to locate the boundary region (112) between the fluorescent area and the non-fluorescent area, - creating (3015, 3115) cell wall annotations (184) along said located border regions, and - Training (3020, 3120) a machine learning algorithm (190) based on the cell wall annotation as ground truth and the non-fluorescent channel (105) as input.

2. The computer-implemented method of claim 1 , wherein: The method further comprises: - creating (3015, 3115) a non-cell wall annotation (185) in an image region adjacent to said located border region (12), Therein, the training (3020, 3120) is also based on the non-cell wall annotation as further ground truth.

3. The computer-implemented method of claim 1 , wherein: The method further comprises: - creating (3015, 3115) an unknown annotation in the image region having the minimum distance to all located said boundary regions (112), Wherein, the training (3020, 3120) is also based on the unknown annotation as further ground truth.

4. The computer-implemented method of claim 3, wherein: The unknown annotation is also created based on an estimate of the presence of cells in the corresponding image region from the non-fluorescent channel (105).

5. The computer-implemented method of claim 3, wherein: During training of the machine learning algorithm (190), predictions of the machine learning algorithm in regions with the unknown annotations are not included in the loss function.

6. The computer-implemented method of claim 3, wherein: During training of the machine learning algorithm (190), a weighting of the predictions of the machine learning algorithm in regions having the unknown annotation is included in a loss function, wherein the weighting is determined based on the distance to at least one other annotation.

7. The computer-implemented method of claim 1 , wherein: The method further comprises: - Applying (3025, 3125) the machine learning algorithm (190) to locate the cell wall in a non-fluorescent channel of the inference image data.

8. The computer-implemented method of claim 7, wherein: The method further comprises: - Based on the localized cell walls in the output of the machine learning algorithm (190): performing (3030) instance localization of individual cells in the inference image data.

9. The computer-implemented method of claim 8, wherein: The instance localization is performed to identify continuous regions, and each continuous region is then divided into a cell instance or a cell background.

10. The computer-implemented method of claim 1, wherein: The method further comprises: - applying (3125) the machine learning algorithm to locate cell walls in the non-fluorescent channel of the transfection image data or other image data, - based on the cell wall found: performing (3130) instance localization of the single cell in the non-fluorescent channel, and -Training (3135) a further machine learning algorithm based on the instance localization as ground truth and the non-fluorescent channel as input.

11. The computer-implemented method of claim 1 , further comprising: - evaluating the spatial structure of the fluorescent area in said fluorescent channel, Wherein, the cell wall annotation is also created according to the spatial structure.

12. The computer-implemented method of claim 1, wherein: The training of the machine learning algorithm (190) uses a consistency loss function or a loss function, which minimizes the entropy value of the output of the machine learning algorithm (190) in at least some image regions.

13. A data processing device, the data processing device being configured to: - receiving (3005, 3105) transfection image data, the transfection image data comprising a fluorescent channel (101) and a non-fluorescent channel (105), wherein: The transfection image data shows an accumulation of cells, not all of which express the dye. - performing (3010, 3110) automatic image processing of the fluorescence channel (105) to locate the boundary region (112) between the fluorescent area and the non-fluorescent area, - creating (3015, 3115) cell wall annotations (184) along said located border regions, and - Training (3020, 3120) a machine learning algorithm (190) based on the cell wall annotation as ground truth and the non-fluorescent channel (105) as input.

Images