Methods and systems for providing training datasets for training machine learning segmentation algorithms

Abstract

Methods and systems for providing training datasets for training machine learning segmentation algorithms are disclosed. The subject matter of the present invention is a method and corresponding system for providing training datasets for training segmentation algorithms for segmenting whole slide images in digital pathology, as well as the use of the training data and corresponding ML segmentation algorithms. In particular, a first segmentation of a whole slide image is refined based on automatically generated annotations, which have a higher level of detail than the first segmentation. The second segmentation result can be used as ground truth for training ML segmentation algorithms on the basis of the whole slide image.

Description

TECHNICAL FIELD

[0001] Various examples of the present disclosure relate to determining ground truth when training machine learning algorithms in the context of digital pathology. More specifically, various examples relate to determining segmentation of whole slide images depicting multiple types of tissue and using the segmentation to train segmentation algorithms. BACKGROUND

[0002] In histopathology, a tissue sample of a patient is examined to study the manifestation of a disease, e.g. cancer.

[0003] Conventionally, a medical practitioner can examine a tissue sample using a microscope, e.g. using chemical staining for staining. In more detail, a thin tissue slide is prepared by cutting a tissue sample and staining and focusing on the cut part. Then, an image is taken depicting the tissue slide. The result is often referred to as a whole slide image. Digitalization of such images enables digital analysis and processing of the tissue sample. This is referred to as digital pathology.

[0004] Recently, algorithmic analysis of digitized tissue samples has become possible. Reference techniques for digital pathology use convolutional neural networks (CNNs) as an example of machine learning (ML) algorithms. Such techniques are described in Karen Simonyan and Andrew Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv: 1409.1556, 2014. Such techniques are further described in Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Deep residual learning for image recognition, CoRR, abs / 1512.03385, 2015. Such techniques are further described in Mingxing Tan and Quoc V Le, EfficientNet: Rethinking model scaling for convolutional neural networks, arXiv preprint arXiv: 1905.11946, 2019. Such techniques are further described in Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollar, Designing network design spaces, In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (IEEE / CVF Conference on Computer Vision and Pattern Recognition), pp. 10428-10436, 2020.

[0005] In general, digital pathology entails processing large images, such as whole slide tissue images (WSIs). Whole slide images depict microscopically magnified tissue slides at a macroscopic scale (e.g., centimeters); thus, whole slide images can include a large number of pixels. Whole slide images typically have a size of tens of thousands of pixels in each dimension. Tissue slides from biopsied or resected tumor tissue are prepared on glass slides, which can be scanned by a digital microscope at high resolution (gigapixels) and high throughput. Due to their size and number, whole slide images are particularly suitable for data-hungry ML algorithms. They can be processed by ML algorithms as a whole or based on small image tiles cut out from the large image.

[0006] It has been found that training ML algorithms to process whole slide images can be time consuming and / or error prone. Large amounts of image data often require large amounts of expensive annotations for supervised learning. Annotations, including ground truth labeling for training processing, are expensive and time consuming, and pathology requires specific solutions using training and validation data for a large number of specific tissue or cancer types, organs, resections versus biopsies, and possible preparation parameters for physical tissue samples (fixation, mold, slicing, etc.). For example, for the standard tumor identification task used as an intermediate representation in many ML-based pathology systems, pathologists need to provide local, ideal, pixel-wise ground truth labeling for the training processing of the ML algorithms. Additionally, obtaining high resolution ground truth labeling can be challenging because there are many fine structures and regions that are difficult to classify, even for experts.

[0007] There are several ways to address the high cost and lack of annotation. First, not all training data samples have the same amount of information. Frequently annotated and / or easily detected samples can not contribute the same amount to better performance as annotating rare and / or difficult to detect samples of training data. A good set of samples is diverse in the various types to be detected by the ML algorithm and covers the entire distribution of possible samples. Active learning is an interactive process in which the added value of labeling a certain sample is explicitly modeled, presenting the annotator with carefully selected valuable samples. Visual similarity can be assessed in an unsupervised way to maximize the diversity of samples. Uncertainty can be used as an indicator of the need for more of a certain data. Most active learning methods use the latter or a combination of both criteria to maximize the training performance while minimizing the amount of labeled data. An example of this process that also applies to whole slide images can be found in Yang, Lin, et al. "Suggestive annotation: A deep active learning framework for biomedical image segmentation." International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, Cham, 2017. An example that applies a one-shot strategy and thus relies mainly on the first criterion can be found in Zheng, Hao, et al. "Biomedical image segmentation via representative annotation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.

[0008] Therefore, there is a need for advanced techniques to provide training datasets for ML algorithms to process whole slide images. In particular, there is a need to train ML algorithms that segment whole slide images to detect different tissue types. SUMMARY

[0009] It is an object of the present invention to provide methods and systems that allow for an improved way of providing training datasets for training ML segmentation algorithms for segmenting whole slide images according to a plurality of different tissue types. Furthermore, it is an object of the present invention to provide methods and systems that enable training ML segmentation algorithms on this basis and segmenting whole slide images using such algorithms.

[0010] This object is solved by the method for providing a training dataset, the method for training an ML segmentation algorithm, the method for segmenting a whole slide image, the corresponding system, the corresponding computer program product and the computer readable storage medium according to the present application. Alternative and / or preferred embodiments are objects of the technical solution of the present application.

[0011] In the following, the technical solution according to the present application is described with respect to the claimed apparatus as well as with respect to the claimed method. Features, advantages or alternative embodiments described herein can equally be assigned to the other claimed objects and vice versa. In other words, the inventive method can be improved by features described or claimed with respect to the apparatus. In this case, for example, functional features of the method are embodied by target units or elements of the apparatus.

[0012] According to one aspect, a computer-implemented method for providing a training dataset for training a segmentation algorithm is provided, wherein the segmentation algorithm is to be trained for segmenting a whole slide image depicting tissue of a plurality of different tissue types according to the plurality of different tissue types. The method comprises a plurality of different steps. A first step is directed to obtaining a whole slide image depicting a tissue section of (a patient). A further step is directed to obtaining the whole slide image and a first segmentation of the plurality of different tissue types, the first segmentation having a first level of detail. A further step is directed to obtaining an annotation of the tissue section, the annotation having a greater level of detail than the first level of detail. A further step is directed to generating a second segmentation of the whole slide image and the plurality of different tissue types based on the first segmentation and the annotation, the second segmentation having a greater level of detail than the first level of detail. A further step is directed to providing the training dataset comprising the whole slide image and the second segmentation.

[0013] In particular, the segmentation algorithm can be a machine learning (ML) segmentation algorithm. Generally, ML algorithms mimic cognitive functions of human beings associated with other human minds. In particular, by means of training based on training data, ML algorithms are able to adapt to new environments and to detect and infer patterns.

[0014] Generally, parameters of the ML algorithm can be adjusted by means of training. In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning and / or active learning can be used. Further, representation learning (alternative term: feature learning) can be used. In particular, the parameters of the ML algorithm can be iteratively adjusted by several training steps.

[0015] In particular, the training function can comprise a neural network, a support vector machine, a decision tree, and / or a Bayesian network, and / or the ML algorithm can be based on k-means clustering, Q-learning, a genetic algorithm, and / or association rules. In particular, the neural network can be a deep neural network, a convolutional neural network, or a convolutional deep neural network. Further, the neural network can be a generative adversarial network, a deep generative adversarial network, and / or a generative adversarial network.

[0016] As a general rule, a neural network comprises a plurality of layers. The input of the first layer is the input image (in this case the whole slide image). Each layer can apply one or more mathematical operations to the input values, such as a convolution, a non-linear activation, a pooling operation, to name a few examples. The input of a layer can be formed by the output of the preceding layer (feed-forward). Feedback or skip connections skipping layers are possible.

[0017] The neural network for digital pathology, i.e. as comprised in the ML segmentation algorithm, can infer at least one semantic histopathological feature. The at least one semantic histopathological feature can describe whether the tissue sample is a manifestation of a disease. Depending on the various types, it can be possible to segment the tissue in the whole slide image. It can be possible to detect and specifically locate healthy and / or unhealthy tissue. In particular, it can be possible to employ a convolutional neural network as the ML segmentation algorithm. For example, it can be possible to use Resnet-18, see Ayyachamy, Swarnambiga, et al. “Medical image retrieval using Resnet-18.” Medical Imaging 2019: Imaging Informatics for Healthcare, Research, and Applications, vol. 10954, International Society for Optics and Photonics, 2019. In particular, it can be possible to use a VGG-16 or VGG-19 CNN, see Mateen, Muhammad, et al. “Fundus image classification using VGG-19 architecture with PCA and SVD.” Symmetry 11.1 (2019): 1; or Kaur, Taranjit, and Tapan Kumar Gandhi. “Automated brain image classification based on VGG-16 and transfer learning.” 2019 International Conference on Information Technology (ICIT). IEEE, 2019.

[0018] The whole slide image to be processed by the ML segmentation algorithm can be a two-dimensional digital image having a plurality of pixels. The whole slide image can have a size of at least 4.000 x 4.000 pixels, or at least 10.000 x 10.000 pixels, or at least 1E6 x 1E6 pixels. The whole slide image can image a tissue section or slide of a patient.

[0019] The preparation of the tissue section from the tissue sample can comprise preparing a portion from the tissue sample, e.g. with a punch tool, wherein the portion is cut into a micron-thick section, i.e. the tissue section. Another word for the portion is a block or punch biopsy. Under microscopic observation, the tissue section can show the fine tissue structure of the tissue sample and in particular the cellular structure or cells contained in the tissue sample. When observed on a larger length scale, the whole slide image can show an overview of the tissue structure and tissue density.

[0020] The preparation of the tissue section can further comprise staining the tissue section with a histopathological stain. In this case, the staining can serve to highlight different structures in the tissue section, such as e.g. cell walls or cell nuclei, or to test for medical indications, such as e.g. cell proliferation levels. In such a case, different histopathological stains serve different purposes.

[0021] For creating the whole slide image, the stained tissue section is digitized or scanned. To this end, the tissue section is scanned with a suitable digitizing station such as e.g. a whole slide scanner, which preferably scans the entire tissue section mounted on an object carrier and converts it into a pixel image. In order to preserve the color effects from the histopathological stain, the pixel image is preferably a color pixel image. Because in the evaluation both the overall impression of the tissue and the finely resolved cellular structure are important, the individual images contained in the histopathological image data usually have a very high pixel resolution. The data size of the individual images can usually amount to several gigabytes.

[0022] In general, the whole slide image can depict a plurality of tissue types relevant to the diagnostic process in digital pathology. The ML segmentation algorithm shall recognize a plurality of these tissue types and provide the segmentation result as a segmentation or segmentation mask. In this respect, "plurality" can mean at least two tissue types. In particular, the ML segmentation algorithm can segment the whole slide image to distinguish between one type of "cancerous" tissue and another type of "non-cancerous" tissue. Other possible examples exist. For example, it would be possible to segment the whole slide image to determine tissue infiltrated by immune cells as a respective tissue type. Other tissue types can relate to blood vessels, connective tissue, necrotic tissue, etc. present in the whole slide image. It would also be possible to segment regions in the whole slide image that do not show tissue at all, i.e. a "no tissue" tissue type. The specific types and number of tissue types to be distinguished by the ML segmentation algorithm depend on the training of the ML segmentation algorithm and also on the histopathological stain used for the whole slide image.

[0023] The first segmentation can identify a plurality of regions in the whole slide image. In particular, the first segmentation can comprise a segmentation mask delineating regions in the whole slide image, each region corresponding to a respective one of the plurality of tissue types (the ML segmentation algorithm should identify / segment in the deployment phase). A plurality of regions can be defined in the first segmentation. Different regions can be associated with different tissue types.

[0024] The first segmentation can thus comprise a plurality of labels. Each label can be associated with a respective region identified by the first segmentation. Each label can indicate one of the plurality of tissue types. In particular, each label can indicate the majority type of tissue in the region. There can also be minority types of tissue in the respective region, at least to some extent.

[0025] The first segmentation can thus comprise a plurality of segments / regions of the whole slide image associated with different ones of the plurality of tissue types.

[0026] According to some examples, the first segmentation comprises one or more contours or segmentation masks of cancerous or tumorous regions, each cancerous or tumorous region comprising a plurality of cancerous or tumorous cells.

[0027] The term“detail level” as used herein can relate to a spatial resolution defining the boundaries of a segmented or annotated region. For example, at a lower detail level, the edge points of a polygonal line delineating a region associated with a given label can be spaced at a greater distance compared to a higher detail level. Alternatively or additionally, the“detail level” can relate to a size threshold below which structures are ignored. For example, a region associated with the label“non-cancerous tissue” can comprise small cancerous structures below a size threshold. Alternatively or additionally, the“detail level” can relate to a maximum count of minority types of structures / tissue portions in a region; for example, consider a region associated with the label“non-cancerous tissue”, such a region can comprise up to a certain quota of instances of cancerous tissue.

[0028] The first (or initial) level of detail can be related to determining a certain accuracy of the first segmentation. For example, the first level of detail can specify a spatial resolution of boundaries of regions associated with a common tissue type (of a plurality of tissue types) or a corresponding label. For example, these boundaries can not be specified with a resolution with respect to a pixel level. For example, the first level of detail can allow for a certain quota and / or a certain small size of deviating types (i.e. minority types within the scope of regions associated with a majority type) of tissue portions or segments (i.e. tissue blocks). According to some examples, the first level of detail can correspond to a resolution of the first segmentation with respect to a supra-cellular level, which means that individual cells of a tissue slice are not resolved (first resolution) in the first segmentation. According to some examples, the first segmentation comprises contours of tumor regions of the first level of detail (i.e. rather “coarse” contours compared to the annotation or the second segmentation).

[0029] Generally, the first level of detail of the first segmentation can be relatively low. Thus, in all cases, the first segmentation can not be suitable to be used as a basis for setting parameters of the ML segmentation algorithm such that the ML segmentation algorithm provides a sufficient level of detail in inference.

[0030] The annotation can identify a plurality of (further) regions in the whole slide image. In particular, the annotation can comprise a segmentation mask delineating the (further) regions in the whole slide image. The plurality of (further) regions can be defined in the annotation. Different (further) regions can be associated with different tissue types.

[0031] Thus, the annotation can comprise a plurality of segments / regions of the whole slide image. Thus, the plurality of segments / regions comprised in the annotation can be different from the segments / regions comprised in the first segmentation. Thus, the annotation can be considered as a complementary (i.e. different) segmentation to the first segmentation.

[0032] According to some examples, the annotation comprises a contour or segmentation mask of one or more cancer cells or tumor cells.

[0033] According to some examples, the annotation does not involve manual annotation and / or automatic image processing based on image data related to the tissue slice.

[0034] The annotation has a greater level of detail compared to the first segmentation. This can be related to a higher accuracy in determining the annotation compared to the first segmentation. For example, the level of detail of the annotation can specify a greater spatial resolution of the region boundaries compared to the first segmentation. For example, these boundaries can be specified with a resolution on a pixel level. For example, the greater level of detail can allow for smaller quotas and / or smaller types of deviating tissue portions or segments (i.e. tissue patches) compared to the first segmentation. This can mean that the size threshold and / or the quota for a minority type instance can be smaller than for the first level of detail. According to some examples, the level of detail of the annotation can correspond to a resolution on a cell level, which means that the annotation resolves individual cells of the tissue section. According to some examples, the greater level of detail can mean that the annotation and the second segmentation comprise contours of the tumor regions of the greater level of detail (i.e. contours that are comparatively “fine” compared to the first segmentation).

[0035] The second segmentation can identify a plurality of regions in the whole slide image. In particular, the second segmentation can comprise a segmentation mask delineating regions in the whole slide image, each region corresponding to a respective one of the plurality of tissue types (the ML segmentation algorithm should identify / segment in the deployment phase).

[0036] Thus, the second segmentation can comprise a plurality of labels. Each label can be associated with a respective region identified by the second segmentation. Each label can indicate one of the plurality of tissue types.

[0037] Like the first segmentation, the second segmentation can comprise a plurality of segments / regions of the whole slide image associated with different ones of the plurality of tissue types.

[0038] However, the second segmentation has a greater level of detail than the first level of detail of the first segmentation. Thus, the plurality of segments / regions comprised in the second segmentation can be different from the segments / regions comprised in the first segmentation, although the second segmentation can in principle comprise the same labels and distinguish between the same tissue types as the first segmentation.

[0039] The second segmentation can be considered as a refined version of the first segmentation, which has been refined using the information contained in the annotation. Thus, the second segmentation has a level of detail that is at least higher than the first level of detail. According to some examples, the level of detail of the second segmentation is at least equal to the level of detail of the annotation.

[0040] Thus, the second segmentation can serve as a ground truth for setting the parameters of the ML segmentation algorithm. Thus, the whole slide image and the second segmentation can constitute training data for training the ML segmentation algorithm.

[0041] According to various examples, the segmentation of the whole slide image can be refined for the purpose of training the ML segmentation algorithm, i.e. by obtaining a first segmentation (initial segmentation) having a first level of detail, and based on this initial segmentation, determining a second segmentation (refined segmentation) of the whole slide image having a second level of detail higher than the first level of detail.

[0042] By such a technique, the workload of the annotation process for determining the initial segmentation can be reduced. In other words, the ground truth for performing the training of the ML segmentation algorithm can be determined relatively quickly.

[0043] The setting of the parameters of the ML segmentation algorithm can benefit from this higher level of detail of the second segmentation. Thus, the ML segmentation algorithm can be trained to deliver segmentation results at the level of detail of the second segmentation.

[0044] As compared to active learning, this process does not involve detailed expert input, as the annotations can be provided automatically. The coarse tumor region outlines as provided by the first segmentation can be produced within a few minutes per whole slide image, as they neither need to be complete, nor pixel-precise or precise at the cellular level.

[0045] In addition, the annotations are not used for inference time, as the training data does not include this information, and the training is only done based on the fine segmentation. Thus, the final method can be applied directly to whole slide images.

[0046] According to some examples, in the step of obtaining annotations, the annotations are obtained independently of the first segmentation.

[0047] This is beneficial, as cross-talk between the generation of the first segmentation and the subsequent refinement is avoided, excluding artifacts in the training phase. Furthermore, this can ensure that no annotations are needed when the ML segmentation algorithm is deployed in the field.

[0048] According to one aspect, the annotations comprise a segmentation of at least one supplementary tissue type, which is different from the plurality of tissue types processed by the first segmentation.

[0049] The supplementary tissue type can be a tissue type which the ML segmentation algorithm should not recognize / segment in the deployment phase. Thus, the annotations can comprise at least one supplementary label which is not included in the labels of the first segmentation. The at least one supplementary label can be associated with one or more regions identified by the annotations. The supplementary label can indicate a supplementary tissue type which is not included in the plurality of tissue types.

[0050] For example, while the plurality of tissue types labeled in the first segmentation can comprise cancer tissue regions and non-cancer tissue regions, the supplementary tissue type can comprise cancer cells. In particular, the annotations can identify adenocarcinoma as a tissue type.

[0051] Hence, the annotations can be seen as a supplemental segmentation providing additional information not included in the first segmentation. By this, the first segmentation can be effectively refined based on the annotations.

[0052] According to one aspect, obtaining the annotations comprises obtaining a supplemental whole slide image different from the whole slide image, the supplemental whole slide image depicting the tissue section or a proximal tissue section of the tissue section, and obtaining the annotations by processing the supplemental whole slide image, optionally by applying a threshold to the supplemental whole slide image.

[0053] The supplemental whole slide image can essentially be in the form of a whole slide image as described above. It differs from the whole slide image in that it comprises different image data, although depicting essentially the same tissue region as the whole slide image. Hence, the supplemental whole slide image can provide supplemental information about the tissue section that can be used to obtain the annotations. For example, the supplemental whole slide image can have undergone different image (pre-)processing steps (e.g. different contrast and brightness adjustments) compared to the whole slide image. In particular, the supplemental whole slide image can be a whole slide image that is specifically generated for providing annotations that are (mostly) not generated in clinical routine when deploying the ML segmentation algorithm. In other words, the supplemental whole slide image is not “seen” by the ML segmentation algorithm, neither in the training phase nor at deployment (just like the annotations are not directly input into the ML segmentation algorithm).

[0054] Providing the supplemental whole slide image can allow for a relatively easy processing to obtain the annotations. According to some examples, the annotations can simply be obtained by applying an image threshold to the supplemental whole slide image (i.e. filtering for predetermined pixel values in the supplemental whole slide image). By this, for example, cancer cells such as adenocarcinoma cells can be obtained in a pixel-precise manner.

[0055] As an alternative, the supplemental whole slide image can be input in a dedicated ML segmentation algorithm (different from the ML segmentation algorithm for which the training data is to be provided) that has been trained to predict annotations based on the supplemental whole slide image. In principle, the dedicated ML segmentation algorithm can be in the same form as the ML segmentation algorithm.

[0056] According to one aspect, the whole slide image depicts the tissue section stained with a first histopathological stain, and the supplemental whole slide image depicts the tissue section or a proximal tissue section stained with a second histopathological stain different from the first histopathological stain.

[0057] By supplementing the whole slide image based on different histopathology stains, supplementary image information can be provided that can be used for annotation. Since each histopathology stain highlights specific structures in the tissue section, a second histopathology stain can be used to particularly highlight structures that contribute to increasing the level of detail of the first segmentation. For example, the second histopathology stain can be configured to highlight cellular structures, which can allow for a cell-wise identification of tumor regions. In particular, the second histopathology stain can be a stain that is less common in clinical routine compared to the first histopathology stain.

[0058] According to one aspect, the first histopathology stain is an H&E stain, and / or the second histopathology stain is an immunohistochemistry stain, in particular comprising a keratin targeting biomarker.

[0059] In this regard, H&E stands for hematoxylin and eosin. Hematoxylin stains cell nuclei, and eosin stains extracellular matrix and cytoplasm. H&E is the most widely used stain in digital pathology, which also makes the ML segmentation algorithm widely applicable.

[0060] Immunohistochemistry staining, or IHC staining for short, involves a process of selectively recognizing antigens (proteins) in cells of a tissue portion by exploiting the principle of specific binding of antibodies to antigens in biological tissue. By this, structures can be highlighted that cannot be reached with other stains such as H&E. By this, additional readings can be provided to further detail the first segmentation.

[0061] In particular, the IHC stain can comprise a biomarker (e.g. in the form of an antibody) configured to target keratin (cytoskeletal). Keratin forms part of the cytoskeleton and defines the mechanical properties of a cell. Thus, the abundance of keratin constitutes a good tumor marker, as keratin expression levels are often altered in tumor cells. Thus, using a keratin IHC stain can enable the identification of adenocarcinoma (neoplastic formation of epithelial tissue). Specifically, the IHC stain can comprise keratin biomarkers for different keratin forms, such as CK-5, CK-8, CK-14, CK-18 (wherein “CK” stands for “cytoskeletal keratin”). The IHC stain can comprise different or additional biomarkers per se, such as p63 and AMACR biomarkers.

[0062] According to one aspect, obtaining the supplementary whole slide image comprises removing the first histopathology stain from the tissue section, staining the tissue section with a second histopathology stain to generate a re-stained tissue section, and imaging the re-stained tissue section to generate the supplementary whole slide image.

[0063] In other words, first the tissue section is washed and then re-stained using a second histopathological stain. By this, for the supplementary whole slide image, a structure substantially identical to the structure in the whole slide image can be imaged. This not only enables to acquire supplementary information, but also ensures that the supplementary whole slide image is easily comparable to the whole slide image.

[0064] According to one aspect, obtaining the supplementary whole slide image comprises providing an image processing function configured to simulate image data depicting a tissue section stained with a second histopathological stain based on image data of the tissue section stained with the first histopathological stain, and generating the supplementary whole slide image by applying the image processing function on the whole slide image.

[0065] The image processing function can be a machine-learned function that has been trained according to the above-described task of simulating a supplementary whole slide image. In particular, the image processing function can be an image-to-image neural network, or more specifically a convolutional image-to-image neural network. According to some examples, the image processing function can be a generative adversarial network (GAN). According to some examples, the image processing function can be trained based on whole slide images and “real” supplementary whole slide images that can be provided as described herein.

[0066] Providing the supplementary whole slide image by the image processing function allows to automatically generate the supplementary whole slide image without the need for manual processing steps. Moreover, since the tissue section is not physically altered compared to the washing step, where washing can destroy some areas of the tissue, the result is inherently well comparable to the whole slide image.

[0067] According to one aspect, obtaining the supplementary whole slide image comprises obtaining a proximal tissue section from the same tissue sample as the tissue section, the proximal tissue section being preferably a proximal section to the tissue section and / or a consecutive section to the tissue section, staining the proximal tissue section with the second histopathological stain, and imaging the stained proximal tissue section to generate the supplementary whole slide image.

[0068] The proximal or even consecutive section is typically spaced from the tissue section on the order of several pm. Thus, the proximal or consecutive section can show a portion of the tissue that is similar to the tissue section to a degree that allows for a one-to-one comparison. The advantage of providing the supplementary whole slide image based on the proximal or consecutive section is that the supplementary whole slide image can be prepared in parallel to the whole slide image. Moreover, destruction of areas of the tissue due to washing can be avoided.

[0069] According to an aspect, the step of obtaining the annotation comprises extracting one or more features from the supplementary whole slide image at a greater level of detail than the first level of detail, the features optionally being related to one or more cytoskeletal features of the cells depicted in the supplementary whole slide image, wherein the annotation is determined based on a segmentation of the one or more features.

[0070] According to some examples, the one or more features can be related to one or more features of cancer cells and / or adenocarcinoma. In particular, the one or more features can be related to cytoskeletal keratins highlighted by the second histopathology stain. According to an aspect, the feature extraction can be performed by a dedicated ML segmentation algorithm that has been trained to predict the annotation based on the supplementary whole slide image.

[0071] By automatically extracting features at a greater level of detail than the first level of detail, annotations with an inherently improved resolution can be obtained.

[0072] According to an aspect, the step of generating the second segmentation comprises aligning and / or registering the first segmentation with the annotation.

[0073] According to some examples, aligning the first segmentation with the annotation can be performed manually by a user or automatically by one or more image processing functions by spatially matching the whole slide image with the supplementary whole slide image.

[0074] By aligning the first segmentation with the annotation, the quality of the second segmentation, and thus the quality of the training data, can be improved.

[0075] According to an aspect, the step of aligning and / or registering comprises providing an image registration between the whole slide image and the supplementary whole slide image and aligning the first segmentation with the annotation based on the image registration.

[0076] According to some examples, providing the at least one image registration can generally comprise registering a target image (e.g. the whole slide image) with a reference image (e.g. the supplementary whole slide image). According to some examples, this can comprise obtaining a transformation function between the target image and the reference image, the transformation function determining a relationship between coordinate systems of the target image data and the reference image data such that each physiological location in the target image is mapped to the same physiological location in the reference image and vice versa. The transformation can thus comprise a plurality of individual displacement vectors associated with pixels of the target image and the reference image, respectively.

[0077] According to some examples, the registration can comprise a rigid registration. A rigid registration can comprise a registration in which the coordinates of the pixels in one image undergo a rotation and a translation to register the image with another image. According to some examples, the registration can comprise an affine registration. An affine registration can comprise a registration in which the coordinates of the data points in one image undergo a rotation, a translation, a scaling and / or a shear to register the image with another image. Thus, a rigid registration can be considered a specific type of affine registration. According to some examples, the registration can comprise a non-rigid registration. A non-rigid registration can provide a different displacement for each pixel of the images to be registered, and can for example use a non-linear transformation in which the coordinates of the pixels in one image undergo a flexible deformation to register the image with another image. According to some examples, a vector field, for example a warp field or other field or function, can be used to define the non-linear transformation, thereby defining a separate displacement for each pixel / voxel in the image. For more details on image registration, reference is made to US 2011 / 0081066 and US 2012 / 0235679. Rigid image registration is very effective in the expectation that there is no deformation. Non-rigid image registration has significantly greater flexibility compared to rigid image registration, but can be more complex to process, since it can manage local distortions between two image sets.

[0078] According to some examples, the step of generating the second segmentation comprises transforming the annotations into the coordinate system of the first registration (or vice versa) using image registration, and generating the second registration (or vice versa) by combining the first registration with the transformed annotations.

[0079] An advantage of using image registration techniques in generating the second registration is that the first and second abnormal images can be transformed into a common coordinate system. By this, it can be ensured that the structures of the segmentations have the same scale and the same position in the image data. In turn, it can be easier to compare the segmentation masks, and artifacts in the computation of the second segmentation are avoided.

[0080] According to one aspect, the step of generating the second segmentation comprises segmenting the first segmentation (mathematically) with the annotations, and / or correcting (or refining) the first segmentation by applying the annotations to the first segmentation.

[0081] According to some examples, the step of segmenting and / or correcting (or refining) can be based on image registration. Further, the step of segmenting and / or correcting (or refining) can be based on annotations transformed into the coordinate system of the first segmentation according to image registration.

[0082] According to an aspect, the step of obtaining the first segmentation comprises providing an initial segmentation algorithm different from the segmentation algorithm to be trained with the training dataset, the initial segmentation algorithm being configured to segment the whole slide image depicting a plurality of types of tissue of the plurality of different tissue types at a first level of detail, and applying the initial segmentation algorithm to the whole slide image to generate the first segmentation.

[0083] In other words, a "coarse" segmentation algorithm operating at the first level of detail is used to generate the first segmentation. Although the general architecture of such a segmentation algorithm can in principle be of the same type as the final ML segmentation algorithm, the "coarse" segmentation algorithm can be relatively easily trained as it does not require detailed annotations. At the same time, such a coarse segmentation algorithm can be completely independent from the final ML segmentation algorithm as the training data and process are conceived to provide an independent ML segmentation algorithm.

[0084] According to an aspect, a computer-implemented method for providing a trained segmentation algorithm for segmenting a whole slide image depicting a plurality of types of tissue of a plurality of different tissue types according to the plurality of different tissue types in digital pathology is provided. The method comprises a plurality of steps. A first step is aimed at providing a training dataset according to any one of the aspects described herein. A further step is aimed at providing an ML segmentation algorithm for segmenting a whole slide image depicting a plurality of types of tissue of a plurality of different tissue types according to the plurality of different tissue types. A further step is aimed at training the ML segmentation algorithm based on the training dataset to obtain the trained segmentation algorithm. A further step is aimed at providing the trained ML segmentation algorithm.

[0085] The ML segmentation algorithm can be completely untrained or pre-trained to accept further training. The training data comprises the whole slide image and the corresponding second segmentation as ground truth. The training can comprise inputting the whole slide image into the ML segmentation algorithm to obtain a training segmentation, comparing the training segmentation with the second segmentation, and adjusting the ML segmentation algorithm based on the comparison. In particular, this can comprise minimizing a loss function based on the second segmentation.

[0086] According to an aspect, a computer-implemented method for segmenting a whole slide image depicting a plurality of types of tissue of a plurality of different tissue types according to the plurality of different tissue types is provided. The method comprises a plurality of steps. A first step is aimed at providing a trained segmentation algorithm according to any one of the aspects described herein. A further step is aimed at providing the whole slide image. A further step is aimed at applying the trained segmentation algorithm to the whole slide image to segment the whole slide image according to the plurality of different tissue types.

[0087] According to an aspect, there is provided a computer program product comprising program elements which, when loaded into the memory of a computing unit, direct the computing unit of the system to perform the steps according to one or more of the above method aspects.

[0088] According to another aspect, there is provided a computer readable medium having stored thereon program elements which, when executed by the computing unit of a system, are readable and executable by the computing unit to perform the steps according to one or more of the above method aspects.

[0089] The advantages of the present application by means of a computer program product and / or a computer readable medium are that an already existing providing system can easily be adapted by a software update to work as proposed by the present application.

[0090] The computer program product can be, for example, a computer program or such including another element next to the computer program. The other element can be hardware (e.g. a memory device on which the computer program is stored, a hardware key for using the computer program, etc.) and / or software (e.g. a document or software key for using the computer program). The computer program product can also include development material, a runtime system and / or a database or library. The computer program product can be distributed over several computer instances.

[0091] According to an aspect, the present application comprises use of a trained segmentation algorithm provided according to any one of the aspects described herein for segmenting a whole slide image depicting a plurality of types of tissue according to a plurality of different tissue types.

[0092] According to an aspect, there is provided a system for providing a training data set for training a segmentation algorithm for segmenting a whole slide image depicting a plurality of types of tissue according to a plurality of different tissue types. The system comprises an interface unit and a computing unit. The interface unit is configured to receive a whole slide image depicting a tissue section of (a patient). The computing unit is configured to obtain (compute) a first segmentation of the whole slide image and the plurality of different tissue types, the first segmentation having a first level of detail. The computing unit is further configured to obtain (compute) an annotation of the tissue section, the annotation having a greater level of detail than the first level of detail. The computing unit is further configured to generate (compute) a second segmentation of the whole slide image and the plurality of different tissue types based on the first segmentation and the annotation, the second segmentation having a greater level of detail than the first level of detail. The computing unit is further configured to provide (via the interface unit) the training data set comprising the whole slide image and the second segmentation.

[0093] According to an aspect, there is provided a system for providing a trained segmentation algorithm for segmenting a whole slide image depicting tissue of a plurality of tissue types according to a plurality of different tissue types. The system comprises an interface unit and a computing unit. The interface unit is configured to receive a training dataset provided according to any one of the aspects described herein and a ML segmentation algorithm for segmenting a whole slide image depicting tissue of a plurality of tissue types according to a plurality of different tissue types. The computing unit is instructed to train the ML segmentation algorithm based on the training dataset to obtain a trained segmentation algorithm. The computing unit is instructed to provide the trained segmentation algorithm (via the interface unit).

[0094] According to an aspect, there is provided a system for segmenting a whole slide image depicting tissue of a plurality of types according to a plurality of different tissue types. The system comprises an interface unit and a computing unit. The interface unit is configured to receive a trained segmentation algorithm according to any one of the aspects described herein and a whole slide image. The computing unit is configured to apply the trained segmentation algorithm to the whole slide image to segment the whole slide image according to a plurality of different tissue types.

[0095] The computing unit can be implemented as a data processing system or as part of a data processing system. Such a data processing system can for example comprise a cloud computing system, a computer network, a computer, a tablet, a smartphone and / or the like. The computing unit can comprise hardware and / or software. The hardware can comprise for example one or more processors, one or more memories and combinations thereof. The one or more memories can store instructions for performing the method steps according to the present application. The hardware can be configured by software and / or operated by software. Generally, all units, sub-units or modules can at least temporarily exchange data with each other, for example via a network connection or respective interfaces. Thus, the individual units can be placed apart from each other.

[0096] The interface unit can comprise an interface for data exchange with a local server or a central network server via an internet connection for receiving the reference image data or the subsequent image data. The interface unit can also be adapted to interface with one or more users of the system, for example by displaying the results of the processing by the computing unit to the user (for example in a graphical user interface), or by allowing the user to adjust parameters for image processing or visualization, for annotating or aligning image data, and / or for selecting whole slide images for processing.

[0097] According to other aspects, the present application also relates to a digital pathology image analysis system comprising at least one of the above systems and a digital pathology image system (or digital pathology information system) configured to acquire, store and / or forward whole slide images. Thereby, the interface unit is configured to receive the whole slide images from the digital pathology image system.

[0098] According to some examples, the digital pathology image system comprises one or more archive stations for storing whole slide images, which can be implemented as cloud storage or local storage or distributed storage. Further, the digital pathology image system can comprise one or more imaging modalities, such as a slide scanning device or the like.

[0099] According to other aspects, the system is adapted to implement the inventive method in its respective aspects for providing training data, for providing an ML segmentation algorithm, or for segmenting a whole slide image. The advantages described in connection with the method aspects can also be achieved by the respective configured system components. BRIEF DESCRIPTION OF DRAWINGS

[0100] The above-mentioned features, characteristics and advantages of the invention and the manner in which they are achieved will become more apparent and the application itself will be better understood from the following description of embodiments, with reference to the accompanying drawings. The following description is not intended to limit the application to the embodiments contained herein. Identical parts, portions or steps can be marked with the same reference signs in different figures. Generally, the figures are not drawn to scale. In the following:

[0101] Figure 1 Embodiments of a system for providing training data for training an ML segmentation algorithm, and / or an ML segmentation algorithm, and / or segmenting a whole slide image by applying an ML segmentation algorithm are schematically depicted;

[0102] Figure 2 A method for providing a segmentation of a whole slide image using an ML segmentation algorithm is schematically depicted;

[0103] Figure 3 A method for providing a segmentation of a whole slide image and / or training data for training an ML segmentation algorithm is schematically depicted;

[0104] Figure 4 A whole slide image according to an embodiment is schematically depicted;

[0105] Figure 5 A segmentation of a whole slide image according to an embodiment is schematically depicted;

[0106] Figure 6 A whole slide image according to another embodiment is schematically depicted;

[0107] Figure 7 A segmentation of a whole slide image according to another embodiment is schematically depicted;

[0108] Figure 8 Method steps for providing a whole slide image according to an embodiment are schematically depicted;

[0109] Figure 9 Method steps for providing a segmentation of a whole slide image according to embodiments are schematically depicted;

[0110] Figure 10 Method steps for providing a segmentation of a whole slide image according to embodiments are schematically depicted;

[0111] Figure 11 Method steps for providing a segmentation of a whole slide image according to embodiments are schematically depicted;

[0112] Figure 12 Method steps for providing a segmentation of a whole slide image according to embodiments are schematically depicted;

[0113] Figure 13 Method steps for providing a segmentation of a whole slide image according to embodiments are schematically depicted;

[0114] Figure 14 Method steps for training a ML segmentation algorithm for segmenting a whole slide image according to a plurality of tissue types according to embodiments are schematically depicted; and

[0115] Figure 15 Method steps for training a ML segmentation algorithm for segmenting a whole slide image according to a plurality of tissue types according to embodiments are schematically depicted. DETAILED DESCRIPTION

[0116] Figure 1 A system 91 according to various examples is schematically depicted. The system 91 comprises a processor 92 (computing unit) coupled to a memory 93. The processor 92 can load program code from the memory 93. The processor 92 can execute the program code. The processor 92 can further communicate with other devices and / or databases via an interface 94 (interface unit). For example, the processor 92 can receive, via the interface 94, a whole slide image WSI1, WSI2 for subsequent processing of the whole slide image WSI1, WSI2. The processor 92 can output, via the interface 94, semantic information associated with the whole slide image WSI1, WSI2 and obtained from said processing of the whole slide image WSI1, WSI2. The semantic information can for example comprise segmentation results of a previously trained ML segmentation algorithm. Based on such segmentation results, a tumor type tissue can be detected in the whole slide image WSI1, WSI2.

[0117] Further, the processor 92 can receive, via the interface 94, a supplemental whole slide image WSI2 for subsequent processing of the supplemental whole slide image WSI2. The processor can be configured to output an annotation A associated with the supplemental whole slide image WSI2.

[0118] In loading and executing program code, the processor 92 can perform techniques as described herein, e.g. training of ML algorithms; performing annotation processing for determining annotations of whole slide images WSI1, WSI2; determining ground truth for training of ML algorithms; refining annotations; setting parameters of ML algorithms, e.g. using backpropagation or another training optimization that minimizes loss values of a loss function, etc.

[0119] Figure 2 is a flowchart of a method according to various examples. For example, the method according to Figure 2 may be performed by a processor in loading program code from a memory. For example, the method according to Figure 2 may be performed by the processor 92 of the system 91 in loading program code from the memory 93 and executing the program code.

[0120] At step D10, a ML segmentation algorithm for digital pathology is provided. This can include training the ML segmentation algorithm as described herein. In particular, the training can be based on training data generated as described herein.

[0121] An input image depicting a tissue sample can be processed using the ML segmentation algorithm. The ML segmentation algorithm can receive the input image and process the input image. An example ML algorithm is a neural network algorithm (NN).

[0122] As a general rule, a NN comprises multiple layers. The input of the first layer is the input image. Each layer can apply one or more mathematical operations to the input values, e.g. convolution, non-linear activation, pooling operations, to name a few examples. The input of a layer can be formed by the output of the preceding layer (feed-forward). Feedback or skip connections skipping layers are possible.

[0123] The NN for digital pathology can infer at least one semantic histopathological feature. The at least one semantic histopathological feature can describe whether the tissue sample is a manifestation of a disease. According to various types, it can be possible to segment the tissue in the input image. Healthy and / or unhealthy tissue can be detected and specifically located. In view of tumor growth, it can be possible to rate the organ fitness of an organ dissection. Tumors can be graded according to a predefined scale, e.g. to determine the severity. Example semantic histopathological features that can be inferred can be selected from the group consisting of: Gleason score, cancer grade / cancer stage estimation, clinical pathway prediction, sub-tumor classification, metastasis assessment, microsatellite instability (MSI) or stability. For example, the cancer grade can be according to a predefined grading system / scale. Examples would include the Bloom-Richardson score or the TNM classification of malignant tumors. Classification systems can be used for brain tumors, breast tumors, prostate cancer (Gleason score) and other kinds of tumors.

[0124] The input image received by the ML segmentation algorithm can have a size of at least 4.000 x 4.000 pixels, or at least 10.000 x 10.000 pixels, or at least 1E6 x 1E6 pixels. The input image can be a whole slide image WSI1, WSI2. The input image can be acquired using an optical microscope. The tissue section from which the input image is acquired can be stained using chemical staining. Illumination can be used to generate the corresponding contrast.

[0125] Various examples disclosed herein are described in the practical example of a ML segmentation algorithm segmenting a whole slide image (ML segmentation algorithm). This means that image regions of the whole slide image comprising a specific type of tissue are delimited by a respective segmentation mask defining a segment. A plurality of segments can be determined. Different segments can be associated with different types of tissue.

[0126] Different implementations of such types of tissue to be detected by the ML algorithm are conceivable. For example, the ML algorithm can segment the whole slide image WSI1 to distinguish between one type of “cancer” tissue and another type of non-cancer “tissue”. Other examples are possible. For example, it would be possible to segment the whole slide image to determine tissue infiltrated by immune cells as a respective type. It would be possible to segment parts of the whole slide image where no tissue is fully displayed, i.e. “no tissue” segments. The specific types of tissue to be distinguished by the ML algorithm depend on the training of the ML segmentation algorithm.

[0127] In particular, the ML segmentation algorithm can be implemented as a classification algorithm configured to classify image regions (e.g. patches) of the WSI according to the type as indicated by the image data. In particular, the segmentation algorithm can be configured to classify cells depicted in the whole slide image WSI1, WSI2 according to different types as indicated by the WSI image data of these cells. Thus, according to some examples, the level of detail the segmentation algorithm shall implement is at least the cell level. In other words, the segmentation provided by the segmentation algorithm distinguishes between cells depicted in the whole slide image WSI1, WSI2 according to different types. According to some examples, the segmentation thus provided segments individual cells.

[0128] A CNN can be employed as segmentation algorithm. For example, Resnet-18 can be used, see Ayyachamy, Swarnambiga, et al. "Medical image retrieval using Resnet-18." Medical Imaging 2019: Imaging Informatics for Healthcare, Research, and Applications, vol. 10954, International Society for Optics and Photonics, 2019. VGG-16 or VGG-19 CNN can be used, see Mateen, Muhammad, et al. "Fundus image classification using VGG-19 architecture with PCA and SVD." Symmetry 11.1 (2019): 1; or Kaur, Taranjit, and Tapan Kumar Gandhi. "Automated brain image classification based on VGG-16 and transfer learning." 2019 International Conference on Information Technology (ICIT). IEEE, 2019.

[0129] Generally, in such an architecture of a CNN, the fully connected layer is the output layer, preceded by a pooling layer. The pooling layer is preceded by a plurality of convolutional layers. There can be skip connections.

[0130] At step D20, a whole slide image WSI1 is obtained (see Figure 4 ). For example, the whole slide image WSI1 can be loaded from a picture archiving system or another database. For example, the whole slide image WSI1 can be acquired using a microscope during a laboratory process. The whole slide image WSI1 can be acquired using various imaging modalities, e.g. fluorescence of a stained tissue sample or tissue section, etc. In particular, the obtained whole slide image WSI1 depicts a tissue section stained with H&E staining.

[0131] At step D30, the trained ML segmentation algorithm is used for an inference task as part of a digital pathology workflow. This means that the whole slide image WSI1 can be input to the ML segmentation algorithm, resulting in segmentation results. Based on these segmentation results, it is possible to detect tumor type tissue in the whole slide image WSI1.

[0132] Figure 3 A method for providing a segmentation SEG2 of a WSI is depicted. The segmentation SEG2 is to be used as a reference or ground truth for training the ML segmentation algorithm. Thus, Figure 3 The method of Figures 4 to 7 corresponding WSIs are shown in Figures 8 to 13 additional optional sub-steps according to further embodiments are shown in Figure 3 The method of Figure 3 The method of

[0133] At step S10, a whole slide image WSI1 is obtained. The whole slide image WSI1 depicts a tissue section of a tissue sample from a patient. The whole slide image WSI1 depicts the tissue section stained with a first histopathology stain. In particular, the first histopathology stain can be a hematoxylin and eosin stain (H&E stain). H&E is a combination of two histological stains: hematoxylin and eosin. Thus, hematoxylin stains cell nuclei (in a purplish blue color), and eosin stains extracellular matrix and cytoplasm (in a pink color), with other structures appearing in different shades, hues, and combinations of these colors (see Figure 4 In this regard, H&E is the most widely used histopathology stain in digital pathology and is often the first or standard stain tailored for histopathological analysis. The whole slide image WSI1 can be obtained essentially as described in connection with step D20.

[0134] At step S20, a first or initial segmentation SEG1 of at least a portion of the whole slide image WSI1 is obtained. As Figure 5 depicted, the initial segmentation SEG1 can comprise a segmentation of the whole slide image WSI1 into a plurality of tiles. In particular, the initial segmentation SEG1 can comprise a segmentation of the whole slide image WSI1 into a plurality of tiles, each tile comprising a respective tile image. The initial segmentation SEG1 can be obtained by a tiling algorithm. The tiling algorithm can be a tiling algorithm as described in connection with step D20. Figure 4one or more labels 121-124 of different types of tissue of the whole slide image WSI1. The initial segmentation can be according to a plurality of tissue types, i.e. a label can be defined for each type. The labels can relate to regions delineated by polylines. According to some examples, the initial segmentation SEG1 covers only a portion of the entire whole slide image WSI1. Further, the initial segmentation SEG1 can be a binary segmentation segmenting the whole slide image WSI1 according to two different tissue types or features.

[0135] The initial segmentation SEG1 has a first level of detail. The first level of detail can relate to a certain precision with which the initial segmentation SEG1 is determined. For example, the initial level of detail can specify a spatial resolution of boundaries of regions associated with the common labels. For example, these boundaries can not be specified at a pixel level resolution. The first level of detail can resolve the labels of the initial segmentation SEG1 at a certain spatial resolution which can be coarser than a pixel-precise resolution.

[0136] In particular, the initial segmentation SEG1 can identify individual regions (cf. the regions 121-124) in the whole slide image WSI1. Figure 5 In particular, the initial segmentation SEG1 can identify individual regions (cf. the regions 121-124) in the whole slide image WSI1.

[0137] According to some examples, the initial segmentation SEG1 can be obtained from an annotation process involving user interaction. According to other examples, the initial segmentation SEG1 can be obtained by inputting the whole slide image WSI1 into an “initial” ML segmentation algorithm as described above. This initial segmentation algorithm can have been trained to process WSI (in particular H&E stained WSI) to provide a segmentation at the first level of detail. Such a segmentation algorithm can thus be denoted as initial ML segmentation algorithm.

[0138] At step S30, an annotation A is obtained. According to some examples, the annotation A is at least partly complementary to the initial segmentation SEG1. This can mean that the annotation A is not based on the plurality of types on which the initial segmentation SEG1 is based. Instead, the annotation A can be an annotation according to a type different from the plurality of types. Thus, the annotation A can be based on a second set of labels 125 different from the first set of labels of the initial segmentation SEG1. The annotation A can be directed to a portion of the whole slide image WSI1 at least partly overlapping with the initial segmentation SEG1. Moreover, the annotation A can be a binary segmentation segmenting the whole slide image according to two different tissue types or features.

[0139] The annotation A can have a second level of detail greater than the first level of detail. In particular, the second level of detail can resolve the labels of the annotation at a greater resolution than the initial segmentation SEG1, in particular, the resolution of the second level of detail can be pixel-precise. Moreover, the second level of detail greater than the first level of detail can mean that the quota of specific small-sized tissue portions or fragments (i.e. tissue patches) of deviating types (i.e. minority types within the range of majority types associated with the plurality of types) is smaller compared to the initial segmentation SEG1.

[0140] There are a plurality of ways in which such an annotation A can be provided. In particular, such an annotation A can be provided by processing a complementary whole slide image WSI2 of the whole slide image WSI1, the complementary whole slide image WSI2 comprising complementary image information compared to the whole slide image WSI1. In particular, the complementary whole slide image WSI2 can be a whole slide image of the same tissue section as the whole slide image WSI1 or at least a tissue section related to the whole slide image WSI1. However, the tissue section depicted in the complementary whole slide image WSI2 is stained with a second or complementary histopathological stain different from the first histopathological stain. In particular, a histopathological stain can be used which is less common in clinical practice than the first histopathological stain, but which can allow a more precise identification of structures and thus a more precise assignment of labels in the annotation A. Since the second histopathological stain will highlight at least partly different features in the tissue section compared to the first stain, the labels covered by the annotation A and the tissue types can also be different compared to the initial segmentation SEG1. Figure 8 Corresponding steps are shown in

[0141] In particular, Figure 8 A plurality of optional sub-steps of step S30 are schematically shown. The order of the steps does not necessarily correspond to the numbering of the steps, but can also vary between different embodiments of the present application. Moreover, individual steps can be omitted and individual steps or a series of steps can be repeated.

[0142] At a first optional sub-step S31, a complementary whole slide image WSI2 is obtained. The complementary whole slide image WSI2 shows the same tissue section as the whole slide image WSI1 or at least a tissue section related to the whole slide image WSI1. However, the tissue section depicted in the complementary whole slide image WSI2 is stained with a second or complementary histopathological stain different from the first histopathological stain. In particular, a histopathological stain can be used which is less common in clinical practice than the first histopathological stain, but which can allow a more precise identification of structures and thus a more precise assignment of labels in the annotation A. Since the second histopathological stain will highlight at least partly different features in the tissue section compared to the first stain, the labels covered by the annotation A and the tissue types can also be different compared to the initial segmentation SEG1.

[0143] The second histopathological staining can be an immunohistochemical staining. In particular, the second histopathological staining can be configured to highlight keratins in the tissue section. Generally, keratin isoforms build intermediate filaments that constitute the cytoskeleton of basal epithelial cells. A corresponding immunohistochemical staining that specifically targets such keratin structures provides a very clear optical readout of the cellular structures in the whole slide image. Thus, such a second histopathological staining can allow for an accurate labeling of cells and / or whole cells and / or cellular structures within a cell type at high resolution. Since the variability of the image pattern is low and the labeling for the second annotation A can in principle be low key (not distinguishing between tissue types but only detecting cellular structures), the annotation A can be provided rather easily at a better resolution than the initial segmentation SEG1.

[0144] More importantly, the use of an immunohistochemical staining can also allow for a complementary disease readout since the expression of keratins is typically suppressed in cancer cells compared to benign structures. For example, in the complementary whole slide image WSI2 shown in Fig. 1 1 1, the label 125 would correspond to benign areas. Figure 6 and Figure 7 In the complementary whole slide image WSI2 shown in Fig. 1 1 1, the label 125 would correspond to benign areas.

[0145] There are different options for the second histopathological staining. For example, the second histopathological staining can comprise biomarkers targeting p63, CK-5, CK-8, CK-18, CK-14, AMACR and combinations thereof, wherein p63 relates to so-called “tumor proteins”, CK-… relates to different forms of cellular keratins and AMACR stands for the human enzyme alpha-methylacyl-CoA racemase. In particular, a mix of p63, cellular keratin and AMACR biomarkers can be used for specifically labeling adenocarcinomas, e.g. the so-called PIN-4 mix, targeting p63, CK-5, CK-14 and AMACR.

[0146] At a second optional sub-step S32, the annotation A is obtained based on the complementary whole slide image WSI2. According to some examples, the annotation A can be obtained by applying a threshold to the complementary whole slide image WSI2, which can be configured to filter image data corresponding to structures targeted by the second histopathological staining and / or areas in the complementary whole slide image WSI2 that are not stained with the second histopathological staining depicted in the complementary whole slide image WSI2. In particular, a thresholding process can be applied to individual pixels or color values of pixels contained in the complementary whole slide image WSI2. In this regard, the thresholding process can imply a distinction between different color or brightness values. For example, all pixels having a value above a certain threshold value can be identified as belonging to a certain type of structure and attributed to a corresponding label according to the annotation A.

[0147] According to an alternative example, the annotation A can be obtained by a further ML segmentation algorithm configured to annotate the complementary whole slide image WSI2 according to the label set 125 of the annotation A. As such a ML segmentation algorithm is particularly suited to identify types highlighted with the second histopathology stain, it can be referred to as a ML IHC segmentation algorithm. The ML IHC segmentation algorithm can essentially be in the form of a general ML segmentation algorithm as described above.

[0148] According to a further example, obtaining the annotation A can involve extracting features from the complementary whole slide image WSI2 at a second level of detail. As explained above, the features can relate to one or more cellular features depicted in the complementary whole slide image WSI2 and highlighted by the second histopathology stain. The annotation A can then be based on the extracted features. In particular, the feature extraction can be performed by the ML IHC segmentation algorithm.

[0149] Generally, the annotation A can provide at the second level of detail a region explicitly highlighted by the second histopathology stain. Thus, the second level of detail can be pixel-precise. Moreover, the region can be a keratin-positive cell wall. Furthermore, the region can be related to adenocarcinoma.

[0150] At step S40, a second segmentation SEG2 is obtained based on the first segmentation SEG1 and the annotation A. In particular, the annotation A can be applied to or merged with the first segmentation SEG1 to generate the second segmentation SEG2. As the annotation A has a higher level of detail than the first segmentation SEG1, the first segmentation SEG1 can be improved or refined, at least such that the resulting second segmentation SEG2 has a greater level of detail than the first segmentation SEG1. In an optimal implementation, the level of detail of the second segmentation SEG2 can be at least equal to the level of detail of the annotation A.

[0151] According to some examples, generating the second segmentation SEG2 based on the first segmentation SEG1 and the annotation A can comprise applying the annotation A on the first segmentation SEG1 and adjusting, i.e. removing or adding, portions of the first segmentation SEG1 that are not reflected in the annotation A. If the annotation A comprises a pixel-precise region, pixels of the labeled regions of the first segmentation SEG1 that do not correspond to the annotation A can be added or removed.

[0152] To improve the accuracy of the second segmentation SEG2, the annotation A and the first segmentation SEG1 can be aligned or registered with each other when generating the second segmentation SEG2. This can be performed in an optional step S41. In particular, this can involve providing a registration between the first segmentation SEG1 and the annotation A at a further optional sub-step S41-A10 and aligning the first segmentation SEG1 and the annotation A based on the registration in a further optional sub-step S41-A20.

[0153] Typically, providing registration in steps S41-A10 means identifying corresponding data points in the first segment SEG1 and annotation A. Having identified such corresponding data points, the local offsets between these points can be calculated, providing an indication of the local offsets in the coordinate system between the first segment SEG1 and annotation A. Doing so for multiple corresponding data points sufficiently distributed in the first segment SEG1 and annotation A already provides a good indication of the displacement and deformation between the corresponding image data. Various registration techniques can be used to properly aggregate these individual contributions into a coherent two-dimensional or three-dimensional transformation function or deformation field. These techniques can include rigid registration, affine registration, non-rigid registration, non-affine registration, and any combination thereof.

[0154] Based on some examples, registration can be provided based on the full-slide image WSI1 and the supplementary full-slide image WSI2, i.e., the full-slide image WSI1 and the supplementary full-slide image WSI2 can be registered with each other. Such image-based registration can produce better results than registration based on segmentation and annotation masks because the matching of image data may be less ambiguous due to more context for each matched data point. Therefore, the transformation function used to transform the coordinate system of the supplementary full-slide image WSI2 to the coordinate system of the full-slide image WSI1 (or vice versa) is also applicable to transforming annotation A to the coordinate system of the first segment SEG1 (or vice versa).

[0155] Aligning the first segment SEG1 and annotation A at step S41-A20 may involve transforming annotation A into the coordinate system of the first segment SEG1 (or vice versa) using the transformation function determined at step S41-A10. The second segment SEG2 can then be determined by applying the transformed annotation A to the first segment SEG1.

[0156] Using the second segmentation SEG2, a baseline truth with better quality than the first segmentation SEG1 can be provided at step S50 for training the ML segmentation algorithm. Therefore, the corresponding training dataset includes the full-slide image WSI1 and the second segmentation SEG2. Specifically, the second segmentation SEG is obtained via a roundabout process of providing and evaluating a supplementary full-slide image WSI2 for each full-slide image WSI1 (this will not be the case during the deployment of the ML segmentation algorithm).

[0157] Several options exist for providing supplemental whole-slide images (WSI2), which will be referred to... Figures 11 to 13 Let me explain three of them in more detail.

[0158] exist Figure 11In the illustrated example, the tissue section depicted in the whole slide image WSI1 is re-used to obtain the complementary whole slide image WSI2. Specifically, at step S31-A10, the first histopathology stain is removed from the tissue section. This can be performed by washing or rinsing the tissue section with a suitable solvent. At the next step S31-A20, the washed tissue section can be re-stained with the second histopathology stain in the usual manner. Subsequently, at step S31-A30, the re-stained tissue section can be imaged, e.g. with a slide scanning device, substantially as described above.

[0159] In Figure 12 In the illustrated example, the appearance of the complementary whole slide image WSI2 is "simulated" based on the whole slide image WSI1. To this end, a machine-learned image processing function can be provided at step S31-B10, which has been trained to simulate or predict image data depicting a tissue section stained with a second histopathology stain based on image data of a tissue section stained with a first histopathology stain. In other words, the ML image processing function has been trained to convert the "style" of a tissue section obtained with a first histopathology stain to the style that a tissue section would likely have if stained with a second histopathology stain. At step S31-B20, the ML image processing function is applied to the whole slide image WSI1 to generate the corresponding complementary whole slide image WSI2.

[0160] According to some examples, the ML image processing function can be a deep convolutional neural network. In one example, a fully convolutional image-to-image neural network can be used, in which the whole slide image WSI1 is input into the network and the output is the complementary whole slide image WSI2. The ML image processing function can be trained based on corresponding pairs of whole slide images WSI1 and "real" complementary whole slide images WSI2, which can be provided, e.g. in the same manner as described in connection with the above-described method. Figure 11 or Figure 13 in the same manner as described.

[0161] According to some examples, the ML image processing network can be a generative adversarial network, a deep generative adversarial network and / or a generative adversarial network. A generative adversarial network or function comprises a generator part or function and a classifier or discriminator part or function. According to some examples, the generator creates a supplementary whole slide image WSI2 image from a whole slide image WSI1 and the discriminator distinguishes between the synthetically created supplementary whole slide image WSI2 and a real supplementary whole slide image WSI2. The training of the generator and / or the discriminator is based on a minimization of a cost function in each case, in particular. The cost function can be minimized, in particular by backpropagation. If the generator and the discriminator are given by a network, in particular by an artificial neural network, the generative adversarial (or simply "GA") algorithm is also referred to as a GA network (also "GAN", which is an acronym for "generative adversarial networks"). These are known in particular from the publication by Ian J. Goodfellow, "Generative Adversarial Networks", arxiv 1406.2661 (2014).

[0162] According to Figure 13 In the illustrated implementation, a supplementary whole slide image WSI2 is provided based on a supplementary tissue section (step S31-C10). The supplementary tissue section can be an adjacent or neighboring tissue section, and in particular a consecutive tissue section, with respect to the tissue section depicted in the whole slide image WSI1. At step S31-C20, the supplementary tissue section is stained with a second histopathological stain. Subsequently, at step S31-C30, the restained tissue section can be imaged, e.g. with a slide scanning device, substantially as described above.

[0163] Figure 14 A method for providing a trained ML segmentation algorithm according to an embodiment is illustrated. The method comprises several steps. The order of the steps does not necessarily correspond to the numbering of the steps, but can also vary between different embodiments of the present application. Furthermore, individual steps or a series of steps can be repeated. The steps are performed by the system 91, in particular by the processor 92 of the system 91, in particular when loading program code from the memory 93 and executing the program code. Figure 3 and Figures 8 to 11 The steps denoted by the same reference signs as the steps described in connection with Figure 3 and Figures 8 to 11 The steps described in connection with the method according to Figure 14 and its substeps can be performed by the processor when loading program code from the memory. For example, the method according to Figure 14 and its substeps can be performed by the processor 92 of the system 91 when loading program code from the memory 93 and executing the program code.

[0164] After providing the second segmentation SEG2 at step S50 and the (not easily trainable) ML segmentation algorithm at step S60, the second segmentation can be used in step S70 to (further) train the ML segmentation function. In particular, the second segmentation SEG2 can be used as a ground truth for judging the performance of the ML segmentation algorithm. At step S80, the thus optimized ML segmentation algorithm is provided for deployment.

[0165] The ML segmentation algorithm according to some examples can differ from the initial ML segmentation algorithm with which the first segmentation SEG1 can have been obtained. This has the advantage that the ML segmentation algorithm can be optimized for a higher level of detail and that cross-talk between different segmentations and their generation is avoided.

[0166] In Figure 15 , a method for providing a ML segmentation algorithm is schematically illustrated. The order of the steps does not necessarily correspond to the numbering of the steps, but can also vary between different embodiments of the present application. Furthermore, individual steps or a series of steps can be repeated. For example, the method according to Figure 15 and sub-steps thereof can be executed by a processor when loading a program code from a memory. For example, the method according to Figure 15 may be executed by a processor 92 of a system 91 when loading a program code from a memory 93 and executing the program code.

[0167] The first step T10 aims at providing a plurality of training data sets. The training data sets comprise a whole slide image WSI1 and a corresponding reference segmentation SEG2, respectively. The reference segmentation SEG2 is of the type of the second segmentation SEG2 and can be obtained substantially as described in connection with Figures 3 to 11 .

[0168] Next, at step T20, the whole slide image WSI1 of one training data set is provided to the (not easily trainable) ML segmentation algorithm.

[0169] In step T30, based on the whole slide image WSI1, the ML segmentation algorithm will determine a training segmentation of the whole slide image WSI1 according to the learned task. In particular, the training segmentation of the whole slide image WSI1 can be a segmentation according to a plurality of tissue types depicted in the whole slide image WSI1.

[0170] In a subsequent step T40, based on a comparison of the reference segmentation and the training segmentation, the performance (i.e. the quality of the segmentation) of the ML segmentation algorithm is evaluated. One way to achieve this is to compare the reference segmentation with the training segmentation (e.g. on a pixel-by-pixel basis with respect to the whole slide image WSI1).

[0171] At step T50, this comparison is used as a loss function to adjust the weights of the ML segmentation algorithm.

[0172] At step T60, the steps of obtaining a training segmentation (step T30) and comparing the result with the reference segmentation (step T40) are repeated with pairs of the whole slide image WSI1 and the reference segmentation SEG2 until the ML segmentation algorithm is able to generate an acceptable result (i.e. until a local minimum of the loss function is reached). Once all pairs have been used, the pairs are then shuffled randomly for the next round.

[0173] The various embodiments or aspects and features thereof can be combined or exchanged with one another, wherever possible, without limiting or expanding the scope of the invention. Where applicable, the advantages described with respect to one embodiment of the invention are also advantageous for other embodiments of the invention.

Claims

1. A computer-implemented method for providing a training dataset for training a segmentation algorithm for segmenting a whole slide image (WSI1) depicting tissue of a plurality of tissue types according to the plurality of different tissue types, the method comprising: obtaining (S10) a whole slide image (WSI1) depicting a tissue section, obtaining (S20) a first segmentation (SEG1) of the whole slide image (WSI1) and the plurality of different tissue types, the first segmentation (SEG1) having a first level of detail, obtaining (S30) an annotation (A) of the tissue section by obtaining (S31) a complementary whole slide image (WSI2) different from the whole slide image (WSI1), processing the complementary whole slide image (WSI2) to extract one or more features from the complementary whole slide image (WSI2) at a level of detail greater than the first level of detail, and segmenting the one or more features to obtain the annotation (A), the annotation (A) having a level of detail greater than the first level of detail, the complementary whole slide image (WSI2) depicting the tissue section or a proximal tissue section of the tissue section, and the one or more features relating to one or more cytoskeleton features of cells depicted in the complementary whole slide image (WSI2), generating (S40) a second segmentation (SEG2) of the whole slide image (WSI1) and the plurality of different tissue types based on the first segmentation (SEG1) and the annotation (A), the second segmentation (SEG2) having a level of detail greater than the first level of detail, and providing (S50) a training dataset comprising the whole slide image (WSI1) and the second segmentation (SEG2).

2. The method of claim 1, wherein the whole slide image (WSI1) depicts the tissue section stained with a first histopathology stain, and the complementary whole slide image (WSI2) depicts the tissue section or the proximal tissue section stained with a second histopathology stain different from the first histopathology stain.

3. The method of claim 2, wherein the first histopathology stain is H&E staining, and / or the second histopathology stain is an immunohistochemistry stain.

4. The method of claim 3, wherein, the second histopathology stain is an immunohistochemistry stain comprising a keratin-targeting biomarker.

5. The method of any one of claims 2 to 4, wherein obtaining (S31) the complementary whole slide image (WSI2) comprises: removing (S31-A10) the first histopathology stain from the tissue section, staining (S31-A20) the tissue section with the second histopathology stain to generate a restained tissue section, and imaging (S31-A30) the restained tissue section to generate the complementary whole slide image (WSI2).

6. The method of any one of claims 2 to 4, wherein, obtaining the complementary whole slide image (WSI2) comprises: - providing (S31-B10) an image processing function configured to simulate image data depicting a tissue section stained with the second histopathology stain based on image data of a tissue section stained with the first histopathology stain, and - generating (S31-B20) the complementary whole slide image (WSI2) by applying the image processing function to the whole slide image (WSI1).

7. The method of any one of claims 2 to 4, wherein, obtaining the complementary whole slide image (WSI2) comprises: - obtaining (S31-C10) a proximal tissue section from the same tissue sample as the tissue section, the proximal tissue section being preferably a proximal section to the tissue section and / or a consecutive section to the tissue section, - staining (S31-C20) the proximal tissue section with the second histopathology stain, and - imaging (S31-C30) the stained proximal tissue section to generate the complementary whole slide image (WSI2).

8. The method of any one of claims 1 to 4, wherein, the step of generating (S40) the second segmentation comprises aligning and / or registering (S41) the first segmentation (SEG1) with the annotation (A).

9. The method of claim 8, wherein, the step of aligning and / or registering (S41) comprises: - obtaining (S41-A10) an image registration between the whole slide image (WSI1) and the complementary whole slide image (WSI2), and - aligning (S41-A20) the first segmentation (SEG1) with the annotation (A) based on the image registration.

10. The method of any one of claims 1 to 4, wherein, the step of generating (S40) the second segmentation (SEG2) comprises: - segmenting the first segmentation (SEG1) with the annotation (A), and / or - correcting the first segmentation (SEG1) by applying the annotation (A) to the first segmentation (SEG1).

11. The method of any one of claims 1 to 4, wherein, the step of obtaining (S20) the first segmentation (SEG1) comprises: - providing an initial segmentation algorithm different from the segmentation algorithm to be trained with the training data set, the initial segmentation algorithm being configured to segment a whole slide image (WSI1) depicting a plurality of types of tissue from the plurality of different tissue types at the first level of detail, and - applying the initial segmentation algorithm to the whole slide image (WSI1) to generate the first segmentation (SEG1).

12. A computer-implemented method for providing a trained segmentation algorithm for segmenting a whole slide image (WSI) depicting a plurality of types of tissue according to a plurality of different tissue types in digital pathology, the method comprising: - providing (S50) a training data set according to any one of claims 1 to 11, - providing (S60) a segmentation algorithm for segmenting a whole slide image (WSI) depicting tissue of a plurality of tissue types according to a plurality of different tissue types, - training (S70) the segmentation algorithm based on the training data set to obtain a trained segmentation algorithm, and - providing (S80) the trained segmentation algorithm.

13. A computer-implemented method for segmenting a whole slide image (WSI1) depicting tissue of a plurality of tissue types according to a plurality of different tissue types, the method comprising: - providing (D10) a trained segmentation algorithm according to claim 12, - providing (D20) the whole slide image (WSI1), and - applying (D30) the trained segmentation algorithm to the whole slide image (WSI1) to segment the whole slide image according to the plurality of different tissue types.

14. A computer readable medium having stored thereon program elements, which, when executed by a computing unit (30) of a system (1), are readable and executable by the computing unit (30) to perform the steps of the method according to any one of claims 1 to 13.

Images