Marker system for marking medical images

Abstract

The invention relates to the labeling of medical images using modular models. The modules on which the models can be based comprise: a) a shared first module (10) for different label (40, 41, 42) types, configured to take a medical image as input and to provide a medical image representation (13) as output; b) label type specific first modules (11, 12), specific to a label type and configured to take the representation provided by the shared first module as input and to provide a label type specific representation (14, 15) as output; and c) second modules (20, 21, 22), wherein each second module is specific to a respective label type and configured to take the medical image representation as input and to provide a label of the respective label type as output. This modular structure allows for efficient multi-task training, thus allowing for more accurate labeling.

Description

Technical Field

[0001] This invention relates to labeling systems, methods, and computer programs for labeling medical images. It also relates to models and modules of models to be used by the labeling systems, methods, and computer programs. Furthermore, it relates to training systems, methods, and computer programs for training modules of models. Background Technology

[0002] Deep learning is a tool for medical image analysis because it can learn and extract useful visual representations. Methods for implementing deep learning in medical image analysis involve pre-training models on extensive natural image datasets, such as ImageNet-1K as described in J. Deng et al.'s paper "ImageNet: A largescale hierarchical image database," IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255 (2009), and then fine-tuning them or directly applying the pre-trained features to specific medical target tasks, as described in HEKim et al.'s paper "Transfer learning for medical image classification: a literature review," BMC Medical Imaging, 22(1):69 (2022), https: / / doi.org / 10.1186 / s12880-022-00793-7. Fine-tuning utilizes the weights of the pre-trained model as an initial base, enabling accelerated training and enhanced performance even with limited data. Alternatively, pre-trained models can remain fixed, and their features can be directly used for downstream medical tasks, as disclosed in M. Raghu et al.'s paper "Transfusion: Understanding Transfer Learning with Applications to Medical Imaging", CoRR. 2019; abs / 1902.07208, https: / / arxiv.org / abs / 1902.07208 and X. Wang et al.'s paper "Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases", CoRR. 2017; abs / 1705.02315, https: / / arxiv.org / abs / 1705.02315.

[0003] However, effective deep learning typically requires large amounts of annotated training data, which is often scarce in medical imaging due to the scarcity of rare diseases or the high cost of annotation, reducing the effectiveness of pre-training, as disclosed in the article "A survey on deep learning in medical image analysis" by G. Litjens et al., Medical ImageAnalysis, December 2017; 42: 60–88, https: / / doi.org / 10.1016 / j.media.2017.07.005. While many small and medium-sized public datasets exist in the medical field, no single pre-trained dataset compares to ImageNet, posing a significant challenge to the development of effective medical image analysis models.

[0004] Several approaches have been proposed to address the problem of data scarcity when pre-training deep networks for medical image analysis. One approach is to use self-supervised learning, which learns visual representations from unlabeled data by solving excuse tasks. However, a significant performance gap remains between self-supervised pre-training and labeled-supervised pre-training, as also outlined in the article “Generalized radiograph representation learning via cross-supervision between images and free-text radiology reports” by HY Zhou et al., Nature Machine Intelligence, 4(1): 32–40 (2022).

[0005] Another approach is to use domain-specific supervised pre-training. For example, as explained in the paper by Zhou et al. mentioned above, a general representation of chest X-rays can be trained using a large text-tagged chest X-ray dataset. Evaluations were conducted on four unseen datasets, and it was found that the chest X-ray encoder described in this paper outperformed ImageNet pre-training by 10% when applied to other chest X-ray analysis tasks. However, supervised pre-training is only applicable to domains where large amounts of training data are available, such as ray maps.

[0006] In the paper “RadimageNet: an open radiologic deep learning research dataset for effective transfer learning” by X.Mei et al., Radiology: Artificial Intelligence, 2022; 4(5): e210315, it was proposed to combine multiple medical classification datasets into a larger dataset and use it to pre-train a deep network for radiology tasks. In this paper, a target task is demonstrated where this approach achieves better results than ImageNet pre-training. However, this approach has some limitations: it only uses classification labels and may not capture all relevant information in medical images, and it requires the network to predict all categories in the combined dataset, even if they are irrelevant or meaningless for a given case.

[0007] The third approach uses hand-designed features, which are designed to be as general as possible to different modalities, organs, and diseases. For example, in the article “Radiological tumour classification across imaging modality and histology” by J. Wu et al., Nature Machine Intelligence, 3(9), pp. 787–798 (2021), a set of morphological and spatial heterogeneity features was proposed that could be applied to lung, breast, and brain tumors from both computed tomography (CT) and magnetic resonance (MR) imaging. However, hand-designed features cannot capture the complex and subtle patterns that deep learning can learn from data. Furthermore, they require more time and expertise to develop and validate compared to the automatic features learned by deep neural networks, as presented in the article by G. Litjens et al. cited above.

[0008] Multi-label task learning (MTL) offers a potential solution to data scarcity in medical image analysis by enabling the simultaneous training of a single model that generalizes well across multiple tasks, as disclosed in the paper "ASurvey on Multi-Labeling task Learning" by Y. Zhang et al., arXiv, from: https: / / arxiv.org / abs / 1707.08114. This method leverages the diversity of small and medium-sized datasets in medical imaging, effectively utilizing different label types to optimize pre-training even in data-sparse domains. MTL has been applied to medical image analysis in various ways, such as training on multiple small and medium-sized datasets from different tasks, particularly limited to classification as disclosed in “Multi-labeling task pre-training of deep neural networks for digital pathology” by R. Mormont et al., IEEE Journal of Biomedical and Health Informatics, 25(2): 412–421 (2020), or segmentation as disclosed in “TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation” by J. Chen et al., arXiv, https: / / arxiv.org / abs / 2102.04306, or one model as disclosed in “One Model is All You Need: Multi-Labeling task Learning Enables Simultaneous Histology Image Segmentation” by S. Graham et al. The use of multiple label types for a single image is disclosed in arXiv, https: / / arxiv.org / abs / 2203.00077. For example, in a paper cited by Mormont et al., MTL was used to pre-train a general representation of histology by aggregating 22 datasets and transforming segmentation and detection tasks into classification tasks. Summary of the Invention

[0009] One object of the present invention is to provide an improved labeling system for labeling medical images, which specifically addresses one or more of the problems mentioned above. Another object of the present invention is to provide a training system for training the labeling system, and to provide a model and modules of the model to be used by the labeling system. Furthermore, an object of the present invention is to provide corresponding labeling and training methods and computer programs.

[0010] In a first aspect of the invention, a labeling system for labeling medical images is provided, wherein the labeling system comprises: - A model module providing unit, adapted to provide a first module configured to output a representation of any one of different types of medical images and a second module for different label types, wherein the respective second module is specific to the respective label type and configured to take the representation of the medical image as input and a label of the respective label type as output, wherein the first module includes: a) a shared first module for different label types, configured to take a medical image of any one of different types of medical images as input and a representation of the medical image as output; and b) a label type-specific first module, wherein the label type-specific first module is specific to the label type and configured to take the representation provided by the shared first module as input and a label type-specific representation as output. - A medical image providing unit, suitable for providing medical images of one of different types of medical images. - A tag type providing unit, suitable for providing one of different tag types. - A model generation unit, adapted to generate a model based on at least a first module and a second module containing provided tags specific to the provided type, the model taking a medical image of any type of medical image as input and a tag of the provided tag type as output, and - A labeling unit suitable for generating labels for the provided medical images using the generated model.

[0011] Since the model used to generate the labels for the provided medical images is based on a shared first module and a second module specific to the type of labels provided, where the shared first module is configured to take any type of medical image as input and output a representation of the medical image, and the second module is configured to take the representation of the medical image as input and output a label of the corresponding label type, the same representation of the provided medical images can be used as the basis for all types of labels. Therefore, each time labels for medical images are generated, it is not necessary to compute several representations. Instead, only a single representation always needs to be computed, where otherwise only the second module specific to the corresponding label type might be needed to generate the label. Computational resources are saved by avoiding the computation of representations that are not actually used in generating labels of a given type. Furthermore, due to the “decoupling” between the shared first and second modules, the model can be easily adapted to other types of labels, as only the corresponding additional second module needs to be provided. In principle, a non-modular model architecture can also be adapted to provide different types of labels. However, training such an architecture would require all types of training labels available for each training image dataset. Therefore, it would be impossible to train using dispersed small and medium-sized datasets that can be used for supervised basic pre-training.

[0012] For some types of tags, the representation provided as output by the shared first module may be unsuitable, or may require an unnecessary complex structure of a second module specific to the corresponding tag type. Therefore, as part of the first module, a tag type-specific module is also provided, wherein the tag type-specific module is specific to the tag type and is configured to take the representation provided by the shared first module as input and take the tag type-specific representation as output.

[0013] Therefore, the model generation unit can be adapted to select whether to generate a model based on a tag type-specific first module, depending on the type of tag provided. Since the second module that generates the model is specific to the type of tag provided, and thus indicates the type of tag provided, the selection of whether to generate a model based on a tag type-specific module can also be considered to depend on the second module that generates the model. For example, a rule can be defined for each second module, wherein the rule defined for the second module indicates whether the corresponding second module is suitable for directly receiving the representation provided as output by the shared first module. In this case, and where the model module providing unit provides more than one tag type-specific first module—which is preferred—the rule can also indicate the tag type-specific module to be used additionally for generating the model, i.e., the tag type-specific first module that is “suitable” for the corresponding second module used. Therefore, the tag type-specific first module can also be considered an “adapter” between the shared first module and the second module.

[0014] As indicated above, without any tag type-specific first module, the resulting model will have limited applicability to different tag types, or the second module must be given a more complex tag type-specific structure. For example, more tag type-specific parameters might be needed overall, which would be inefficient and could even lead to situations where, during model training, only the parameters of the second module are effectively learned, without affecting the shared parameters of the first module. In this way, the model's ability to handle any one type of tag will hardly benefit from training on a model handling any other tag type.

[0015] It should be noted that while it is preferable to provide only a single shared first module for generating the model, multiple additional modules may be provided for model generation. In particular, as will be discussed in detail below, in addition to the single shared first module, the model may also be generated based on one of two provided tag type-specific first modules and further based on one of a plurality of provided second modules, wherein the number of provided second modules can be substantially arbitrary.

[0016] Preferably, the different types of medical images include at least two types from the following list: digital histological images, computed tomography (CT) images, and X-ray images. CT images are, for example, magnetic resonance (MR) images, computed tomography (CT) images, positron emission tomography (PET) images, or single-photon emission computed tomography (SPECT) images. X-rays are preferably two-dimensional X-ray projection images. Therefore, the labeling system can be able to process very different types of medical images, which allows for the use of a wider variety of training medical images to train the labeling system. This larger number of training medical images can lead to improved quality of the trained labeling system, and thus improved quality of the final labeling of the medical images.

[0017] However, it is certainly possible to train the system using only a single type of medical image. Furthermore, the aforementioned types of medical images are certainly not exhaustive and particularly allow for finer-grained classification. Thus, for example, the labeling system could be trained only for labeling digital histology images, but preferably for labeling multiple subtypes, such as H&E-stained, immunohistochemical (IHC)-stained, fluorescent (FL), and / or other digital histology images. Moreover, the training of the labeling system can take into account not only images acquired with different medical imaging modalities, but also medical images that differ from each other in terms of where they were acquired (i.e., in which laboratory). It has been found that even for the same imaging modality, images from different laboratories tend to have large differences, making a model trained solely on data from one laboratory ineffective for data from different laboratories. For example, in the case of digital histology images, the differences may specifically involve staining or tissue preparation.

[0018] In a preferred embodiment, the different types of labeling include at least two types from the following list: dense prediction, particularly segmentation; classification; and object detection. Since the labeling system can provide different types of labels, training medical images already labeled with different types of labels can be used to train the system. Therefore, training, especially sharing the first module, does not need to be limited to a single type of label, thus allowing the use of more training data to train the labeling system. This results in an improved labeling system, and therefore ultimately leads to further improved quality of labeling of medical images performed by the labeling system.

[0019] Label types should be understood as possible subtypes of labels and / or groupings of labeling tasks. A labeling task, or simply “task” below, should refer to the application of a corresponding label type or, in particular, a subtype of label in a specific medical use case. For example, a task might be defined as the segmentation or detection of a specific type of lesion associated with a specific disease in medical images acquired in a specific laboratory using an imaging modality, or the classification of a specific type of such an image according to a corresponding disease state. Although tasks could theoretically be grouped differently, in order to define separate modules for the model used for labeling, it has been chosen to group them according to their corresponding label types, thus grouping tasks of different label subtypes into the same group, regardless of any other differences. It has been found that differences in label types can be most reasonably considered in a modular model structure compared to other types of differences between tasks, thus allowing for particularly efficient multi-task training.

[0020] Preferably, the tag type-specific first module is also not specific to a specific task, that is, not specific to a specific task that shares the corresponding tag type. Therefore, the tag type-specific first module can be regarded as shared between different tasks with the same tag type, and the second module can be regarded as a task-specific module.

[0021] Furthermore, the tag type-specific first modules are preferably not even specific to a particular tag subtype, that is, they are preferably not specific to a particular tag subtype of the same tag type. Therefore, the tag type-specific first modules can also be considered as shared between different tag subtypes of the same tag type.

[0022] Furthermore, regarding the specificity of a module to a given label type or subtype, this specificity may only refer to the pre-training phase, i.e., the pre-training of the model. In principle, the first pre-trained module can then be used as the basis for tasks associated with the label type or subtype, i.e., combined with a corresponding newly defined second module, with respect to the fact that the first module was not used in the pre-training. For example, it has been found that the representation determined as output by the first module can be used to compress medical images by determining a corresponding representation for each of multiple regions of an image (which may also be referred to as an image "patch"), wherein the compressed medical image (i.e., multiple representations determined for the corresponding regions of the image) can be used to predict patient prognosis. In particular, it has been found that representations provided as output by a shared first module in this way allow for accurate patient prognosis prediction.

[0023] The classification of medical images (i.e., the classification type labeling of medical images) is understood in this paper as assigning one or more labels to the entire image, where each of the one or more labels corresponds to one or more categories associated with the entire image. For example, in (single-label) binary classification, a single label is assigned to the image, indicating which of two possible categories the image belongs to. In (single-label) multi-class classification, a single label is still assigned to the entire image, but the image can fall into three or more possible categories. Another type of classification is multi-label classification, where more than one label is assigned to the image, each label indicating one of two or more possible categories of the image. While in binary and multi-class classification, the possible categories that can be indicated by a single label are mutually exclusive, i.e., making it logically possible for an image to be in only one of the categories, multi-label classification allows more than one category to be assigned to a single image via more than one label. While each individual label in multi-label classification still indicates only one of two or more mutually exclusive categories for the corresponding image, the possible categories that can be indicated by different labels are not mutually exclusive.

[0024] Therefore, for example, the same medical image can have a single or one binary classification label. For instance, when multiple diseases can coexist and whether the patient is male or female is relevant, multiple binary classifications can be used, i.e., binary multi-label type labeling. For example, a chest X-ray can be labeled "pneumonia," "female," and "enlarged heart." For single-label, multi-class classification, three different labeling tasks and corresponding second modules must be defined, which is computationally inefficient in this common scenario because some unnecessary recalculations depending on the image representation of the corresponding task will be necessary.

[0025] Dense prediction type labeling of medical images is preferably understood herein as assigning a label to each element of the image, wherein the element of the image may be, in particular, its pixel, and the label may include one or more numerical values. The label may correspond to one or more categories associated with the corresponding image element, but its value may, in principle, be, a new value for the corresponding image element. Dense prediction type labeling includes a label according to which several image elements, in particular pixels, are assigned the same label. For example, pixels may be grouped and labels may be assigned to pixel groups.

[0026] Object detection type labeling for medical images is preferably understood herein as including dense prediction type labeling for each of a predefined number of scales for the image. Therefore, it can also be referred to as including multi-scale dense type labeling. For this type of labeling, each pixel element or group thereof is assigned a representation at each scale, and the corresponding label is derived based on these representations.

[0027] However, object detection type labeling for medical images can also be understood as including dense prediction type labeling and subsequent post-processing of the labeling. Therefore, instead of distinguishing between detection type labeling and dense prediction type labeling based on multiple scales of labeling based on presence, object detection type labeling can also be defined as differing from dense prediction type labeling in terms of post-processing. Post-processing preferably refers to processing the dense prediction type labeling output into countable objects. The final object detection type labeling output preferably does not include output for each image element.

[0028] In the example, the provided second module is a label type-specific second module configured to: a) take the representation provided by the label type-specific first module as input; and b) output the label type. Specifically, the label type-specific first module is specific to dense prediction type and / or object detection type labels, and is therefore dense prediction (i.e., dense prediction-specific first module) and / or object detection (i.e., object detection-specific first module), wherein the provided second module is a label type-specific second module configured to: a) take the representation provided by the label type-specific first module as input; and b) output the label type, the second module being configured to output dense prediction type and / or object detection type labels, and is therefore a dense prediction second module and / or object detection second module. Thus, for a certain type of label, a certain module sequence is used, which is formed by a shared first module, a label type-specific first module for a certain type of label, and a second module also specific to a certain type of label. It has been found that this structure of modules further improves the labeling of medical images for a specific type of label, particularly for dense prediction type and object detection type labels.

[0029] Preferably, the dense prediction-specific first module includes a shared pixel-dense network and / or the object detection-specific first module includes a shared feature pyramid network. It has been found that using these networks can further improve both dense prediction and object detection labeling. Using these networks in label type-specific modules also makes it relatively easy to define compatible second modules for new label subtypes, which can lead to more robust training. Furthermore, these networks allow for the combination of different image sizes, which can further improve labeling quality.

[0030] In the example, the provided second module is a tag-type-specific second module, configured to: a) take the representation provided by the shared first module as input; and b) take the tag-type tags as output. Specifically, the provided second module is a tag-type-specific second module, configured to: a) take the representation provided by the shared first module as input; and b) take the tag-type tags as output, and is configured to output category-type tags and is therefore a category second module.

[0031] Many diagnostic problems require the classification of medical images, including, for example, tumor grading and binary disease prediction. Furthermore, it has been recognized that the image representation required for accurate classification using a relatively low-complexity second module is of such type that it can also be used to perform accurate segmentation and / or object detection in medical images, even with only additional intermediate modules, whereas conversely, image classification using the image representation required for segmentation and / or object detection would be inefficient. Therefore, the above example is preferred, where the labeling type-specific first module does not need to be used for classification but can be used for dense prediction and / or object detection.

[0032] Since labels are always assigned to the entire image in image classification, the image representation used to make category decisions does not need to consider the spatial dimension of the image. Therefore, the classification second module can be configured to take a one-dimensional representation of the medical image as input, while the shared first module can be configured to take the one-dimensional representation as output.

[0033] Preferably, the shared first module uses pooling to output a representation of the medical image, based on which the classification second module can output a label for the classification type. That is, the shared first module may include pooling layers for computing one or more combinations (e.g., one or more weighted combinations) of values ​​from corresponding previous layers. For example, the pooling (i.e., the combination of values ​​from corresponding previous layers) could be global average pooling. During training, this can lead to more robust features compared to other pooling types (e.g., global max pooling), for example, because gradients are computed and applied to all parts of the features, not just the most dominant ones. In other words, more stable training can be achieved due to the improved gradient flow.

[0034] Preferably, the second classification module uses a fully connected layer to output a classification type label. It has been found that, for a specific type of label, particularly for classification type labels, a fully connected layer can lead to a more general representation and allows for the combination of different numbers of categories for the classification task, i.e., classification type labels with different numbers of possible categories for each label.

[0035] The above-described example of patient prognosis prediction can be viewed as a specific instance of a classification task, since classification type labels can be used to indicate patient prognosis. Therefore, the shared first module can be configured to take any type of medical image from different categories as input and output a representation of each of multiple regions of the medical image, wherein the classification second module can be configured to: a) take the multiple representations provided by the shared first module as input; and b) output labels indicating the predicted patient prognosis. The classification second module then becomes the patient prognosis prediction second module. Furthermore, the multiple representations of the medical image can correspond to a compressed version of the medical image; "compression" refers to data size. Predicting patient prognosis based on a compressed version of the medical image can be computationally more efficient.

[0036] In the example, the corresponding second module comprises a single linear layer or a single convolutional layer for computing the corresponding labels. It has been found that, for a specific second module, this architecture enables efficient pre-training by using less resource-intensive computation in downstream medical applications. Furthermore, it has been found that limiting the size of the second module ensures that learning primarily occurs in the first module, resulting in a more generalizable first module.

[0037] Preferably, one of the first modules (especially the shared first module) encodes the image representation at multiple scales to make them applicable to both dense and non-dense types of labels. Good results have been obtained when using a Transformer architecture for the shared first module, particularly the Swin Transformer architecture disclosed in the paper “SwinTransformer: Hierarchical Vision Transformer using Shifted Windows” by Z. Liu et al., CoRR 2021, abs / 2103.14030, https: / / arxiv.org / abs / 2103.14030, which is incorporated herein by reference. However, at least acceptable (if not equally good) results can be expected from a shared first module with convolutional structures. Furthermore, label-type-specific first modules can also have convolutional structures.

[0038] Preferably, the labeling system further includes a preprocessing unit adapted to preprocess the medical images before using them as input to the generated model, wherein the preprocessing includes at least one of the following steps: a) transforming the medical image to a predefined spatial dimension; b) normalizing the image values ​​of the medical image; and c) transforming the medical image into a predefined number of image elements. This allows the labeling system to handle very different types of medical images, all of which contribute to the training of the labeling system due to the shared first module. Therefore, even if only a small amount of training data exists for a single type of medical image, the total amount of training data can be relatively high, as training data associated with different types of medical images can be used to train the labeling system. This allows for improved training of the labeling system, and thus ultimately, improved labeling of medical images based on a better-trained labeling system.

[0039] For example, if the provided medical image is not a two-dimensional image, it can be transformed into a two-dimensional image. Normalization can include normalizing image values ​​to a range from 0 to 1. A predefined number of image elements can refer to a predefined number of rows and columns in the arrangement of the image elements. For example, a predefined number of rows and columns could be 512 × 512. The predefined number of rows and columns can be the same or different.

[0040] Preprocessing of medical images can also be used to generate several inputs for a model from the medical images, where the inputs can be considered as related to each other. For example, a three-dimensional medical image can be transformed into several two-dimensional medical images corresponding to slices traversing the three-dimensional image. Similarly, the original two-dimensional medical image can be used to form several smaller two-dimensional images corresponding to regions of the original two-dimensional image. In both cases, i.e., when slicing a three-dimensional image and when segmenting a two-dimensional image into regions, several image portions of the original medical image are formed, where the image portions can be considered as contextualizing each other in a sense that each of them can help interpret the meaning of the others. To label medical videos comprising several frames that vary over time, preprocessing can also include extracting frames from the video, where frames can correspond to two-dimensional or three-dimensional medical images and can be considered as image portions of the medical video, which forms medical images that vary over time.

[0041] Therefore, in a particular embodiment, preprocessing includes forming several image portions of a medical image, wherein the model generation unit is adapted to generate a model such that the model takes the several image portions as input, wherein each of the several image portions is provided as input to a shared first module, such that the shared first module generates several representations as output, each of the several representations being a representation of a corresponding image portion, wherein the provided tag type as the output of the model is determined based on several representations generated as outputs of the shared first module and at least a second module, the second module being specific to the provided tag type. Thus, the model generation unit can be adapted to generate a model taking several inputs determined based on a medical image as input, wherein each of the several inputs is provided as input to the shared first module, thereby causing the shared first module to generate several outputs, wherein the inputs for a tag type-specific first module or a corresponding second module are based on the several outputs of the shared first module. Using this type of model, 3D volumetric, gigapixel images and videos can be directly processed, wherein the several outputs of the shared first module can be considered contextualized with each other. The inputs provided to the shared first module can be generated by the preprocessing indicated above, and thus can correspond to different parts of a medical image, such as two-dimensional slices of a three-dimensional image provided as original input, blocks of a two-dimensional image provided as original input, or frames of a video provided as original input. For each of the image parts, the shared first module can provide separate outputs, i.e., representations of the corresponding image parts, thereby deriving several intermediate outputs derived from the same original input, which can be viewed as contextualizing the intermediate outputs relative to each other. The intermediate outputs or image part representations can then be used to form a single input to a label type-specific first module or a corresponding second module. More generally, the intermediate outputs can be used to form one or more inputs to a label type-specific first module or a corresponding second module. Thus, as further indicated below, instead of forming a single representation that can be viewed as a representation of the entire medical image from several representations of several image parts, several representations can also be mapped to several additional representations before being provided as input to a label type-specific first module or a corresponding second module. The several additional representations can be viewed as contextualized versions of the representations output by the shared first module.

[0042] Several inputs can be continuously provided to a shared first module, generating corresponding outputs (i.e., corresponding representations) of the shared first module one after another. The outputs (i.e., several representations) of the shared first module can be temporarily stored until, for each of the inputs, the shared first module has generated a corresponding output (representation), and the outputs (representations) can be further processed together in the model. Alternatively, several inputs can be provided to the shared first module at once, allowing the corresponding outputs (representations) to be determined simultaneously. Therefore, the shared first module can also be configured to take a medical image as input based on several inputs corresponding to portions of the medical image, and to take several outputs corresponding to the representations of the portions of the medical image received as input as outputs.

[0043] In fact, the inputs to the shared first module do not need to originate from the same medical images. Instead, in other embodiments, the labeling system remains a labeling system for labeling combinations of input elements, but the input elements include: a) several medical images or b) one or more medical images and non-image data associated with one or more medical images. During the training of the shared first module, only the auxiliary first module may be used; that is, the auxiliary first module may not be trained. However, both the shared first module and the auxiliary first module may also be trained in combined training.

[0044] Preferably, the following relationship is trained, according to which several representations of the model input (i.e., several image portion representations determined by the shared first module, or a) several medical images from the shared first module, or b) several representations from one or more medical images from the shared first module, and for example several representations determined by non-image data from the auxiliary first module) are formed as inputs for a label type-specific first module or a corresponding second module. For example, the inputs for the label type-specific first module or the corresponding second module can be formed as a weighted combination of several representations, such as a weighted average, wherein the weights of the combination can be generated by training. On the other hand, the model module providing unit can be adapted to provide an additional first module, wherein the additional first module is configured to take several representations of the model input as input and take one or more additional representations (i.e., several inputs provided as inputs to the model or one or more additional representations of input elements) as outputs. Then, the corresponding second module and the optional label type-specific first module can be configured to take at least one of the one or more additional representations as inputs, and the model generation unit can be adapted to further generate a model based on the additional first module. The additional first module can be referred to as the contextual first module, and one or more additional representations can be considered as contextualized representations or as combined representations. Particularly preferred is that the additional first module is configured to output more than one contextualized representation. Using the additional first module, context in model input data with several labels can be considered, such as the context between different regions in a medical image or between different frames in a medical video. It has also been observed that more meaningful representations can be learned from training as well as from non-contextualized tasks.

[0045] The additional first module is preferably attention-based. Additionally or alternatively, the additional first module may include several convolutional layers. A particular attention-based additional first module may have an architecture such as the known BERT encoder, where an attention mask is used to train the additional first model, preferably to guide its attention to context, i.e., to the representations it receives as input. Attention mechanisms are described, for example, in the paper “Attention is All You Need,” Advances in Neural Information Processing Systems (2017), https: / / arxiv.org / abs / 1706.03762, which is incorporated herein by reference in its entirety. In a preferred embodiment, the additional first module is trained based on a random attention mask. The random attention mask can randomly combine the representations received as input by the additional first module. For example, if the additional first module receives several images or partial image representations as input and is configured to provide additional representations for each of them, each of the additional representations may depend on half or 50 percent of the other representations received as input.

[0046] In another aspect of the invention, a training system for a module of a model for training a labeling system is proposed, wherein the training system comprises: - A model module providing unit, adapted to provide a first module configured to output a representation of any one of different types of medical images and a label type-specific second module for different label types, wherein the corresponding second module is specific to the corresponding label type and configured to take the representation of the medical image as input and a label of the corresponding label type as output, wherein the module can be modified by modifying the module parameters, wherein the first module includes: a) a shared first module for different label types, configured to take any one of different types of medical images as input and a representation of the medical image as output; and b) a label type-specific first module, wherein the label type-specific first module is specific to the label type and configured to take the representation provided by the shared first module as input and a label type-specific representation as output. - Training dataset providing unit, suitable for providing training dataset, wherein the corresponding training dataset includes medical training images of one type of medical training image and training labels of one type of different labeling. - A model generation unit, adapted to generate label type-specific models for different types of training labels, wherein a model for the corresponding training label type is generated based on at least a shared first module and a second module provided for the corresponding training label type, the model taking a medical training image of any one of different types of medical training images as input and labels of the corresponding training label type as output. - A loss function providing unit, suitable for providing a label type-specific loss function, wherein the corresponding loss function depends on the deviation between the training labels and the labels generated by using the generated label type-specific model for the corresponding label type, and - Training unit, suitable for training modules via the following: a) Divide the training dataset into batches, and then divide the batches into batch groups. b) For each batch in one of the groups, perform the following steps: i) Generate labels for the medical training images of the corresponding batch using the generated model, wherein the corresponding medical training images are labeled with labels of the corresponding training label type using the corresponding model generated for the corresponding training label type; ii) Calculate the deviation between the training labels of the training dataset of the corresponding batch and the corresponding generated labels generated for the medical training images of the training dataset of the corresponding batch; iii) Calculate the label type-specific gradient of the label type-specific loss function based on the calculated deviation. c) Update the model modules by: i) combining gradients computed for the same type of label, such that a corresponding combined gradient is determined for the corresponding label type; and ii) modifying the module parameters based on the combined gradient. d) Repeat steps b) and c) for all batch groups.

[0047] Because the model uses a shared first module, different types of medical training images and different types of training labels can both contribute to the training of the shared first module, allowing more training data to be used for training. This leads to improved training and, consequently, an improved labeling system for determining the labels for real medical images. Furthermore, by combining gradients and then modifying the module parameters based on the combined gradients as described above, a single update step can be based on different types of labels, where different types of labels can even be used in any order.

[0048] Because the update step considers different types, and preferably also label subtypes, and thus different modules, wherein model parameters for different label types, and preferably label subtypes, are updated simultaneously—that is, at least the parameters of the first module are shared, and preferably also the parameters of another first module—the update step can also be called a shared update step. Through this shared update step, it can be ensured that all tasks within this step are jointly optimized, thereby producing the optimal computational step for all types of labels and all the training data involved.

[0049] Preferably, in step a) above, the training dataset is divided into batches of training data, such that each batch corresponds to a corresponding label type, and more preferably to a label subtype, i.e., the training labels of any given batch have the same type, and preferably the same subtype. Then, in step b) above, for each batch of one of the groups, i.e., the batch group formed in step a) above, the following steps are preferentially performed: i) generating labels for the medical training images of the corresponding batch, wherein the corresponding medical training images are labeled by labels of the corresponding training label type corresponding to the corresponding batch using a corresponding model generated for the corresponding training label type corresponding to the corresponding batch; ii) calculating the deviation between the training labels of the training dataset of the corresponding batch and the corresponding generated labels generated for the medical training images of the training dataset of the corresponding batch; iii) calculating the label type-specific gradient of the loss function for the corresponding label type based on the calculated deviation. Furthermore, in step c) above, the modules of the model are then preferentially updated by: i) combining the gradients calculated for the batch, such that a corresponding combined gradient is determined for the corresponding label type; and ii) modifying the module parameters based on the combined gradient.

[0050] Preferably, the different types of labels include classification labels, wherein the loss function providing unit is adapted to provide a classification loss function to be used for classification labels as a loss function specific to the different types of labels, wherein the classification loss function includes a cross-entropy function, particularly a classification or binary cross-entropy function, which depends on the deviation between the training labels and the labels generated for the classification labels using the corresponding generated model.

[0051] In the example, different types of labels include categorical labels, where the loss function providing unit is adapted to provide a classification loss function to be used for categorical labels as a type-specific loss function for different types of labels, wherein the classification loss function includes normalization over multiple possible categories. This allows for joint training of the second module, which, without normalization, would have very different loss magnitudes. This would result in a task, i.e., label subtypes, where the largest loss magnitude dominates all other loss magnitudes.

[0052] Preferably, the different types of labels include densely predicted labels, wherein the loss function providing unit is adapted to provide a dense prediction loss function to be used for densely predicted labels as a loss function of a label type-specific loss function for the different types, wherein the dense prediction loss function includes a DICE loss function and / or a focal loss function. Specifically, the dense prediction loss function includes an equally weighted combination of the DICE loss function and the focal loss function. This allows for addressing challenges associated with class imbalance, such as large background regions.

[0053] The training system can be a training system for a module used to train a model for a labeling system using an additional first module as defined above, wherein the additional first module is referred to as a context first module. In this case, the model module providing unit is preferably adapted to further provide the additional first module, and the model generation unit is preferably adapted to further generate a label type-specific model based on the additional first module.

[0054] Furthermore, if the labeling system is a labeling system for labeling combinations of several input elements, wherein the several input elements include: a) several medical images or b) one or more medical images and non-image data associated with one or more medical images, then preferably, the corresponding training dataset includes: A) a combination of several input elements, said several input elements including: a) several medical images or b) one or more medical images and non-image data associated with one or more medical images; and B) training labels of one of different labeling types.

[0055] While supervised training has yielded good results, it has been found that more meaningful representations of medical images can be obtained through unsupervised training, particularly in addition to supervised training. For unsupervised training, the training data may consist only of medical training images, i.e., without additional training labels for the training images. The additional labels are preferably replaced by the medical training images themselves or by images or portions thereof obtained based on the medical training images. In other words, the training labels may correspond to the training images or portions thereof. Therefore, in this case, the type of label is preferably dense prediction, particularly image generation, where image generation should be understood herein to include image translation. In implementations, an additional label type-specific first module and / or an additional dense prediction second module, specific to the dense prediction type, are used for unsupervised training. For example, an autoencoder can be computed using the additional label type-specific first module and the additional dense prediction second module. According to another example, the masked image loss can be computed as disclosed in the paper “SimMIM: a Simple Framework for Masked Image Modeling” (2021), 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9643–9653, https: / / arxiv.org / abs / 2111.09886, which is incorporated herein by reference in its entirety. Additionally or alternatively, unsupervised training can be performed as contrastive learning. Furthermore, it should be noted that unsupervised training is not limited to training data consisting only of medical training images. In particular, non-image data can also be used as training data.

[0056] In another aspect of the invention, a set of model modules is proposed for generating label-type-specific models for different types of labels, wherein the model for training the corresponding label type takes a medical image of any one of different predefined types of medical images as input and a label of the corresponding label type as output, and is generated based on at least a shared first module and a second module of the set of model modules, the second module being specific to the corresponding label type, wherein the set of model modules includes a first module and a second module for different label types, wherein the corresponding second module is specific to the corresponding label type and is configured to take a representation of the medical image as input and a label of the corresponding label type as output, wherein the first module includes: a) a shared first module for different label types, configured to take a medical image of any one of different types of medical images as input and a representation of the medical image as output; and b) a label-type-specific first module, wherein the label-type-specific first module is specific to the label type and is configured to take a representation provided by the shared first module as input and a label-type-specific representation as output.

[0057] In another aspect of the invention, a model is proposed that takes a medical image of any one of different predefined types as input and a tag of a tag type of different predefined types as output, wherein the model includes at least a shared first module of the module group of claim 16 and a second module of the model module group of claim 16, the second module being specific to the tag type.

[0058] In another aspect of the invention, a labeling method for labeling medical images is provided, wherein the labeling method includes: - A model module providing unit provides a first module configured to output a representation of any one of different types of medical images and a second module for different label types, wherein each second module is specific to a corresponding label type and configured to take the representation of the medical image as input and a label of the corresponding label type as output. The first module includes: a) a shared first module for different label types, configured to take any one of the different types of medical images as input and a representation of the medical image as output; and b) a label type-specific first module, wherein the label type-specific first module is specific to the label type and configured to take the representation provided by the shared first module as input and a label type-specific representation as output. - A medical image of one of different types is provided by the medical image providing unit. - A tag type is provided by the tag type providing unit, which provides one of the different tag types. - A model is generated by the model generation unit based on at least a shared first module and a provided second module. This model takes a medical image of any type from different medical image types as input and outputs a provided tag type as output. The second module is specific to the provided tag type. - The provided medical images are labeled by the labeling unit using the generated model.

[0059] In another aspect of the invention, a training method is provided for training modules of a model of a labeling system as defined in any one of claims 1 to 10, wherein the training method comprises: - A model module providing a unit provides a first module configured to output a representation of a medical image of one of different types of medical images, and a label type-specific second module for different label types, wherein the corresponding second module is specific to the corresponding label type and configured to take the representation of the medical image as input and the label of the corresponding label type as output, wherein the module can be modified by modifying the module parameters, wherein the first module includes: a) a shared first module for different label types, configured to take any one of the different types of medical images as input and the representation of the medical image as output; and b) a label type-specific first module, wherein the label type-specific first module is specific to the label type and configured to take the representation provided by the shared first module as input and the label type-specific representation as output. - The training dataset is provided by the training dataset provider, wherein the corresponding training dataset includes medical training images of one type of medical training image and training labels of one type of label. - A model generation unit generates a label-type-specific model for different types of training labels, wherein a model for the corresponding training label type is generated based on at least a shared first module and a second module provided for the corresponding training label type. This model takes a medical training image of any one of different types of medical training images as input and labels of the corresponding training label type as output. - A label-type-specific loss function is provided by the loss function providing unit, wherein the corresponding loss function depends on the deviation between the training labels and the labels generated by using a label-type-specific model generated for the corresponding label type, and - The module is trained by the training unit using the following methods: a) Divide the training dataset into batches, and then divide the batches into batch groups. b) For each batch in one of the groups, perform the following steps: i) Generate labels for the medical training images of the corresponding batch using the generated model, wherein the corresponding medical training images are labeled with labels of the corresponding training label type using the corresponding model generated for the corresponding training label type; ii) Calculate the deviation between the training labels of the training dataset of the corresponding batch and the corresponding generated labels generated for the medical training images of the training dataset of the corresponding batch; iii) Calculate the label type-specific gradient of the label type-specific loss function based on the calculated deviation. c) Update the model modules by: i) combining gradients computed for the same type of label, such that a corresponding combined gradient is determined for the corresponding label type; and ii) modifying the module parameters based on the combined gradient. d) Repeat steps b) and c) for another batch group in the same batch group.

[0060] In another aspect of the invention, a labeling computer program for labeling medical images is provided, wherein the labeling computer program includes program code means that, when the labeling computer program is run on a computing system, the program code means are used to cause the computing system to perform the steps of the labeling method as defined in claim 14.

[0061] In another aspect of the invention, a training computer program is provided for training a module of a model of a labeling system as defined in any one of claims 1 to 10, wherein the training computer program includes program code means that, when the training computer program is run on a computing system, the program code means are used to cause the computing system to perform the steps of the training method defined above.

[0062] In another aspect, the present invention relates to the use of a medical image representation which has been determined as the output of a first module of a medical image—particularly after the first module has been trained as part of the training method defined above—for a labeling task that was not previously considered, i.e., in the training that leads to the trained first module and thus in the representation generated by the first module.

[0063] For example, the image representation determined by the shared first module can be used for image compression, and preferably also for patient prognostic prediction based on the compressed image. In particular, as further indicated above, multiple representations can be provided for an image, each representing a region of the image, wherein one or more representations can be used as input to a second module for defining and / or training to provide labels indicative of patient prognosis. These second modules, which may be referred to as patient prognostic prediction second modules, can be redefined and / or retrained after pre-training, i.e., training leads to training of the first module, and thus results in representations provided by the trained first module.

[0064] However, in order to predict patient prognosis based on one or more representations, any other known machine learning architecture can also be used in principle, as long as the machine learning architecture is configured to receive one or more representations as input and provide an output indicating the patient's prognosis.

[0065] Therefore, the representation provided by the first module as the corresponding output can not only be provided internally for further processing within the model, but also as a separate output of the model itself.

[0066] It should be understood that the use of the labeling system, training system, model module group, model, labeling method, training method, labeling computer program, training computer program and the above-defined uses have similar and / or identical preferred embodiments, especially as defined in the dependent claims.

[0067] It should be understood that the preferred embodiments of the present invention may also be any combination of the dependent claims or the above embodiments and the corresponding independent claims.

[0068] These and other aspects of the invention will be apparent from the embodiments described below and will be set forth with reference to those embodiments. Attached Figure Description

[0069] In the attached diagram below: Figure 1 An embodiment of a labeling system for labeling medical images is illustrated schematically and exemplary. Figure 2 The schematic and exemplary illustration shows the modules of a model to be used for labeling medical images. Figure 3 The various aspects of sharing the first module are illustrated schematically and exemplary. Figure 4 The various aspects of the classification module are illustrated schematically and exemplary. Figure 5 The aspects of the dense prediction first module and the dense prediction second module are illustrated schematically and exemplary. Figure 6 The first and second object detection modules are illustrated schematically and exemplary. Figure 7 The preprocessing of different types of medical images is illustrated schematically and exemplary. Figure 8 An implementation of a training system for a module used to train a model of a labeling system is illustrated schematically and exemplary. Figure 9The training dataset for the modules to be used to train the model and the test dataset for testing the trained modules are illustrated schematically and exemplary. Figure 10 The various aspects of module training are illustrated schematically and exemplary. Figure 11 A flowchart illustrating an exemplary implementation of a labeling method for labeling medical images is provided. Figure 12 A flowchart illustrating an exemplary implementation of a training method for a module used to train a model of a labeling system is provided. Figure 13A and Figure 13B The changes in memory state during some steps of the training method are illustrated schematically and exemplary. Detailed Implementation

[0070] Figure 1 An embodiment of a labeling system 1 for labeling medical images is schematically illustrated. The labeling system 1 includes a model module providing unit 2, which is adapted to provide first modules 10, 10', 11, 12 and second modules 20, 21, 22, which are in... Figure 2 The diagram is illustrated schematically and exemplarily.

[0071] First modules 10, 11, and 12 are configured to output representations 13, 14, and 15 of any of predefined different types of medical images. The first module includes a shared first module 10 for predefined different types of labels, wherein the shared first module 10 is configured to take a medical image as input and output a representation 13 of the medical image. The first module also includes label-type-specific first modules 11 and 12, wherein the label-type-specific first modules 11 and 12 are specific to a corresponding label type and are configured to take the representation 13 provided by the shared first module 10 as input and output label-type-specific representations 14 and 15.

[0072] The second module includes modules specific to a corresponding label type. In this embodiment, the three sets of second modules 20, 21, and 22 are specific to three predefined label types: classification, dense prediction, and object detection. Each of the individual second modules in the set is further specific to a corresponding subtype of the label type, or more specifically for a particular task. The second modules are configured to take a corresponding representation of a medical image provided by the first module as input and a corresponding label of the corresponding label type as output. In this embodiment, each of the second modules 20 is configured to take a representation 13 provided by a shared first encoder, i.e., module 10, as input; each of the second modules 21 is configured to take a representation 14 provided by a label type-specific first module 11 as input; and each of the second modules 22 is configured to take a representation 15 provided by a label type-specific first module 12 as input.

[0073] In this embodiment, the label-type specific first module 11 is specific to labels of the dense prediction type, and the label-type specific first module 12 is specific to labels of the object detection type. Therefore, the label-type specific first module 11 can also be considered as the dense prediction first module, and the label-type specific first module 12 can be considered as the object detection first module. Accordingly, in this embodiment, each second module 21 is configured to output labels of the dense prediction type, and it can be considered as the dense prediction second module, and each second module 22 is configured to output labels of the object detection type, and therefore can also be considered as the object detection second module. Meanwhile, the second module 20 is configured to output labels of the classification type as labels, and therefore can be considered as the classification second module 20.

[0074] The additional first module 10' is configured to take as input several representations 13 provided as output by the shared first module 10, and to output another representation, which will be referred to as combined representation 13', and may also be referred to as contextualized representation. In response to receiving several inputs consecutively, several representations 13 can be provided as output by the shared first module 10, wherein the inputs may correspond to image portions of a medical image, several medical images, and / or non-image data associated with one or more medical images. Therefore, combined representation 13' can be understood as a representation of several inputs from the shared first module 10. Instead of representation 13, the label type-specific first modules 11, 12, and second module 20 can also be configured to take combined representation 13' as input. In fact, combined representation 13' can be of the same type as representation 13, such as the same dimensions. Although as Figure 2As shown, representation 13' can appear as a single representation from a plurality of representations 13, but in other embodiments, an additional first module 10' can be configured to map a plurality of representations 13 to a plurality of representations 13'.

[0075] Although classification, object detection, and dense prediction are illustrated as outputs of the second module, the second module can also be configured to output other types of tags.

[0076] Preferably, a label is defined as any type of information or annotation that indicates or describes a given medical image. The label may be derived from the medical image, for example, or it may be associated with, for example, an attribute of the medical image. A label for a medical image is preferably understood as a label of the image content of the medical image. Therefore, the label is preferably based on the pixel or voxel values ​​of the medical image. Thus, for example, a label for a medical image may be a label based on one or more physical features visible in the image.

[0077] Each classification second module is preferably configured to classify the corresponding medical image, wherein the classification may refer to assigning a corresponding single label to the corresponding medical image.

[0078] The dense prediction second module is preferably configured to assign a label to each image element of the medical image. The dense prediction second module is preferably configured to segment the medical image or generate a new image based on the different image elements of the medical image to which the labels are assigned.

[0079] Image elements are preferably image pixels. Preferably, the dense prediction second module assigns only a single label to the corresponding single image element. For example, if an object in a medical image should be segmented, a first label can be assigned to the image element belonging to that object, and a second label can be assigned to other image elements, or no label can be assigned.

[0080] The object detection second module is preferably configured to assign a number of tags to the same image element, wherein one or more tags preferably indicate the object category of the image element, and one or more other tags preferably indicate the position of the bounding box around the object of the corresponding object category.

[0081] Preferably, the first modules 10, 10', 11, and 12 are neural networks, and therefore preferably, representations 13, 13', 14, and 15 are neural representations. More preferably, the dense prediction first module 11 includes a shared pixel-dense network, and the object detection first module 12 includes a shared feature pyramid network. While pixel-dense networks and shared feature pyramid networks are generally known, the term "shared" should be used herein to refer to the fact that the same instances of such generally known networks can be used for different tasks, thus being shared across label subtypes.

[0082] On the other hand, for example, the second modules 20, 21, and 22 preferably do not include deep neural networks, but only include a single fully connected or convolutional layer.

[0083] Several modules 10, 11, 12, 20, 21, and 22 will be referred to later. Figures 3 to 5 To describe in more detail.

[0084] like Figure 3 As illustrated schematically and exemplary, the shared first module 10 is preferably configured to take a medical image as input and, in addition to representation 13, output several feature maps 16. Feature maps are understood herein to include multi-valued maps, which can also be considered as having multiple layers, where each layer can itself be considered a single-valued feature map. Therefore, Figure 3 The feature maps 16 are displayed as blocks. While representation 13 may correspond to a fixed-size embedding, for example, that might be used in classification by the second classification module 20, the various feature maps 16 may each have different sizes, i.e., each feature map corresponds to a different scale. Preferably, the shared first module 10 is configured to use global average pooling to transform the feature maps with the smallest scale (i.e., in particular all their layers) into representations 13 corresponding to fixed-size embeddings. The various feature maps 16 output by the shared first module 10, or at least some of them, are preferably provided as input to the type-specific first modules 11, 12 for labeling.

[0085] In such Figure 3 In the illustrated embodiment, the shared first module 10 can be constructed as a convolutional neural network or a Swing transform, but other implementations are also possible. Regardless of the implementation, it is preferred that the shared first module 10 is configured to output feature maps 16 at four consecutive coarse-grained scales, wherein the number of feature maps 16 output at each scale (i.e., the number of layers of the corresponding multi-valued feature maps) increases with coarsening. The structure of the feature maps 16 can be, for example, H / n × W / n × N. H / 2n × W / 2n × 2N H / 4n × W / 4n × 4N H / 8n × W / 8n × 8N, where H refers to the first (e.g., height) dimension of the input medical image, W refers to the second (e.g., width) dimension of the input medical image, n can be equal to 4, and N can be equal to 96, for example, and where the corresponding first and second numbers refer to the size of the feature map, and the corresponding third number refers to the number of feature maps determined by the shared module 10 at the corresponding scale. Then, representation 13 can have a dimension of 1 × 1 × 8N, where each of its values ​​is associated via a respective global average pooling with a corresponding layer in 8N layers having the coarsest-grained feature map patch determined by the shared first module 10 in dimension H / 8n × W / 8n.

[0086] Figure 4 The classification second module 20 is illustrated schematically and exemplary in more detail. This classification second module 20 receives representation 13 from a shared first module 10, or representation 13' from another first module, each of which can be considered a neural image representation 13, 13'. Figure 4 As can be seen, the second classification module 20 uses dropout layers, ReLU activation function, and linear layers ( Figure 4 (in box 25) and the maximum layer (especially the softmax layer) Figure 4 Box 26 in the middle provides the category decision C, which is in Figure 4 The figure is indicated by reference numeral 40 in the attached drawing.

[0087] Therefore, the classification module 20 can be configured to use fully connected layers to output labels for the classification type. In fact, as will be explained below, all second modules 20, 21, and 22 may include a single linear layer or a single convolutional layer for computing the corresponding labels.

[0088] Figure 5 The dense prediction first module 11 and the dense prediction second module 21 are illustrated schematically and exemplary in more detail.

[0089] Figure 5As illustrated in the preferred embodiment described above, the dense prediction first module 11 includes a shared pixel-dense network. This shared pixel-dense network can be considered as outputting a representation of each pixel or each (small, e.g., 2 × 2) group of pixels. As input, it requires feature maps of different scales, which in this embodiment are feature maps 16 output by the shared first module 10 as described above (not to be confused with the output of the shared feature pyramid network of the object detection first module 11, which will be described in more detail below). For example, it is useful for a second module with pixel-dense tags for type segmentation, image generation, image transformation, and autoencoders. Along with any shared first module 10, the dense prediction first module 11, including the shared pixel-dense network, can particularly form an encoder-decoder structure of the known U-Net type. This is described in more detail in the article “U-Net: Convolutional Networks for Bio-medical Image Segmentation” by O. Ronneberger et al., Computer Science, Computer Vision and Pattern Recognition, arXiv:1505.04597 [cs .CV], which is incorporated herein by reference.

[0090] In the illustrated embodiment, representation 14 corresponds to the output of the dense prediction first module 11 having dimensions H / 2 × W / 2 × 32. This output representation 14 is then received by the dense prediction second module 21.

[0091] In this example, the dense prediction second module 21 provides the label pixel category decision, thereby segmenting the medical image. Thus, each pixel is assigned to a certain category, thereby segmenting the medical image. As shown, the dense prediction second module 21 preferentially uses 1 × 1 convolutional layers 28 and maximum layers (specifically, softmax layers 29) to provide the final category decision C, which in turn... Figure 4 The figure is indicated by reference numeral 41. Since a category decision is made for each pixel, the output of the dense prediction second module 21 has dimensions H × W × C. Instead of a 1 × 1 convolutional layer, any convolutional layer with C output channels can be used.

[0092] The dense prediction second model 21 is configured to output a vector of values ​​for each pixel, i.e., output a corresponding value for each class, where the corresponding pixel is assigned to the class with the highest value. Figure 5In this diagram, variable C defines the number of different categories, variable H defines the height of the corresponding medical image, and variable W defines the width of the corresponding medical image. Therefore, H × W defines the number of image elements (i.e., pixels) of the corresponding medical image. Since the corresponding pixel is assigned to the category with the highest value, or in other words, since the category or label with the highest value has been determined and assigned to the corresponding pixel, a single label or category is assigned to each pixel.

[0093] Figure 6 The object detection first module 12 and the object detection second module 22 are illustrated schematically and exemplary in more detail.

[0094] Figure 6 As illustrated in the preferred embodiment described above, the object detection first module 12 includes a shared feature pyramid network. Similar to the dense prediction first module 11, the object detection first module 12 also requires feature maps of different scales as input; in this embodiment, these feature maps are the feature maps 16 output by the shared first module 10. However, unlike the shared pixel-dense network of the dense prediction first module 11, the shared feature pyramid network corresponding to the dense prediction first module 11 outputs multi-scale pixel-dense features as output, rather than a single output, to adapt to processing objects of different scales.

[0095] Preferably, the object detection first module 12 is adapted to use weight sharing when using representation levels. Therefore, objects of different scales are treated equally. To provide only a simple, intuitive example of different scales, different scales can refer to near and far objects of the same actual size that appear to be different sizes due to their different distances (a near elephant, a far elephant). These objects of the same size at different distances are treated equally. For more details on the shared feature pyramid network, refer to the article "Feature Pyramid Networks for Object Detection" by T.-Y. Lin et al., Computer Science, Computer Vision and Pattern Recognition, arXiv:1612.03144v2 [cs.CV], which is incorporated herein by reference.

[0096] exist Figure 6In the exemplary embodiment shown, three outputs are initially generated by the object detection first module 12. It is noteworthy that these initial three outputs, generated by the object detection first module 12, are not based on all feature maps 16 received from the shared first module 10, but rather on a subset 16', i.e., a subset 16' of the feature maps 16. The subset 16' includes the three coarsest-grained feature maps 16, but is not the first feature map 16, i.e., not based on the feature map initially generated from the input medical image in the shared first module 10. The three initial outputs of the object detection first module 12 have dimensions H / 8 × H / 8 × 128, H / 16 × H / 16 × 128, and H / 32 × H / 32 × 128, respectively. The outputs with dimensions H / 8 × H / 8 × 128 and H / 16 × H / 16 × 128 are generated based on the smallest coarse-grained level of the shared feature pyramid corresponding to the object detection first module 12, and the output with dimensions H / 32 × H / 32 × 128 is generated from the previous level, i.e., the level of the shared feature pyramid before the smallest coarse-grained level. To generate representation 15, the initial output with dimensions H / 32 × H / 32 × 128 is further processed into an intermediate output with dimensions H / 64 × H / 64 × 128, which is then further processed into another intermediate output with dimensions H / 128 × H / 128 × 128. The initial output with dimensions H / 8 × H / 8 × 128 and H / 16 × H / 16 × 128 and the intermediate output with dimensions H / 128 × H / 128 × 128 are then combined into the final output corresponding to representation 15.

[0097] Based on the representation 15 received as input, the object detection second module 22 provides bounding boxes 42 around detected objects within the medical image as output. To this end, the object detection second module 22 includes two 3 × 3 convolutional layers 61, 62, one configured to generate an output of size H × W × 4, and the other configured to generate an output of size H × W × C. Post-processing 65 is performed on these two outputs to arrive at the bounding boxes 42 around the corresponding detected objects in the respective medical images. For example, the post-processing can be loaded from known packages such as the MMDetection package from the OpenMMLab project.

[0098] The dimensions, particularly the depth, of the corresponding outputs of the two convolutional layers 61 and 62 in the object detection second module 22 are selected such that, for each pixel in the corresponding medical image, and for each scale considered by the feature pyramid network, the distance to the edge of the corresponding surrounding box is determined in all four directions, in addition to the category. Therefore, convolutional layer 61 with an output of size H × W × C can be considered as the predicted box category, and convolutional layer 62 with an output of size H × W × 4 can be considered as the predicted box edge.

[0099] In the example, object detection module 22 is a fully convolutional one-stage object detection module (FCOS), as described, for example, in the paper “FCOS: Fully Convolutional One-Stage Object Detection” by Z. Tian et al., Computer Science, Computer Vision and Pattern Recognition, arXiv: arXiv:1904.01355 [cs.CV], which is incorporated herein by reference.

[0100] In summary, the output of the feature pyramid network corresponding to the first object detection module 12 is the input to the object detection task. For example, the Swin transformer (encoder) corresponding to the shared first module 10 outputs multi-scale feature maps, and the feature pyramid network acquires these multi-scale feature maps and transforms them into multi-scale feature maps with the same tensor depth. Then, the second object detection module 22 convolves its output at each scale of the feature pyramid network, producing an output for each scale, which is then post-processed using, for example, the MMDetection package from the OpenMMLab project.

[0101] Refer again Figure 1 The labeling system 1 further includes a medical image providing unit 3, which is adapted to provide a medical image of one of the different types of medical images to be labeled. The medical image providing unit 3 may be a storage device storing one or more medical images, and may retrieve a medical image from the storage device to provide the medical image. The medical image providing unit may also be a receiving unit for receiving medical images, wherein the medical image providing unit can be configured to provide the received medical image. The medical image providing unit may also be an image generating unit configured to generate a medical image. The medical image may be, for example, a digital histological image, an MR image, a CT image, a PET image, a SPECT image, or an X-ray image, especially a two-dimensional projection X-ray image.

[0102] The tagging system 1 also includes a tag type providing unit 4, which is adapted to provide a tag type of one of predefined different tag types. For example, the tagging system 1 may include an input unit 8, such as a keyboard, computer mouse, touchpad, etc., wherein the tag type providing unit 4 may be adapted to provide a user interface on the display 9 of the tagging system 1, which allows the user to indicate the desired tag type, wherein the tag type providing unit 4 can then provide the indicated tag type. Desired tag types may also be provided and stored in advance, wherein the tag type providing unit 4 may be adapted to retrieve previously stored tag types and provide the retrieved previously stored tag types.

[0103] Furthermore, the labeling system 1 may include a model generation unit 5, which is adapted to generate a model based on at least a shared first module 10 and a second module providing labels specific to the provided type. This model takes a medical image of any of a predefined different type of medical image as input and outputs a label of the provided label type. In this embodiment, if the medical image should be labeled using type-classified labels, the model is generated by combining the shared first module 10 and the classification second module 20. If the provided type of label is dense prediction, the shared first module 10, the dense prediction first module 11, and the dense prediction second module 21 are combined to generate the model. If the provided type of label is object detection, the shared first module 10, the object detection first module 12, and the object detection second module 22 are combined to generate the model.

[0104] The labeling system 1 also includes a labeling unit 7, which is adapted to generate labels for the provided medical images using the generated model.

[0105] The labeling system 1 also includes a preprocessing unit 6 adapted to preprocess the medical image before it is used as input to the generated model. The preprocessing includes at least one of the following: a) transforming the medical image to a predefined spatial dimension, b) normalizing the image values ​​of the medical image, and c) transforming the medical image into a predefined number of image elements. This is in Figure 7 As exemplarily shown in the figure.

[0106] Figure 7 Exemplary examples illustrate MR image 31, X-ray image 32, and digital histology image 30 as three different types of images. MR image 31 is a three-dimensional image with grayscale values ​​for its image elements, X-ray image 32 is a two-dimensional image with grayscale values ​​for its image elements, and digital histology image is a two-dimensional image with color values ​​for its image elements. Preferably, the two-dimensional digital histology image is a gigapixel image.

[0107] exist Figure 7In step 34, as shown, 3D enhancement can be applied to the 3D MR image, and a 2D slice image can be generated using the random orientation of the 2D planes within the 3D MR image. The 2D slice image can be normalized to image values ​​ranging from 0 to 1, and it can also be transformed into a predetermined number of image elements. Specifically, the image elements ultimately used as input to the model are arranged in rows and columns, where the number of rows and columns is predefined, and the 2D MR slice can be transformed to have the predefined number of rows and columns. For example, if desired, the number of image elements can be reduced by averaging or increased by interpolation. The resulting transformed image 37 can be input into the generated model.

[0108] Specifically, to apply 3D augmentation, 3D MR images can be loaded into cache 59 one after another, either as a whole 3D image or as all slices of the corresponding 3D image. If images are loaded into cache 59 piece by piece, performing 3D augmentation becomes more difficult (e.g., with higher requirements for memory and / or computation) or even impossible, which would make model training less diverse, resulting in lower-quality labels for the trained model.

[0109] Before inputting the resulting transformed image 37 into the generated model, it can be modified by applying domain-specific two-dimensional enhancements (i.e., two-dimensional enhancements specifically for CT or X-ray images). Such enhancements can optionally be used to increase robustness during model training. Two-dimensional enhancements preferably include Gaussian blurring, shearing, and / or skewing of medical images. Other two-dimensional enhancements can also be used, such as known enhancements for computed tomography or X-ray images, or known enhancements for digital histology or microscopic images.

[0110] The X-ray image 32 is already two-dimensional, so it is only necessary to normalize the image values ​​to a range from 0 to 1 and transform the number of image elements so that the number of rows and columns are the same as the predetermined number of rows and columns. Typically, this would refer to shrinking to a common size, which... Figure 7 This is executed in step 35. (As per...) Figure 7 As indicated by reference numeral 38 in the accompanying drawings, random enhancement is subsequently provided to the resulting image of the adaptation model 40. Although not limited thereto, random enhancement preferably includes random redirection.

[0111] exist Figure 7In step 33, the two-dimensional digital histology images are normalized so that their image values ​​lie within a fixed range from 0 to 1 (this fixed range may differ in other embodiments), and their size is reduced so that the number of rows and columns in the transformed two-dimensional digital histology image is the same as the number of predefined columns and rows. The resulting image is then copied multiple times, with each copy being provided with random enhancements (such as in...). Figure 7 The image (indicated by reference numeral 36 in the attached figure) is input into model 40. Similarly, as with MR images, to perform these processing and / or enhancement steps, the entire original image (i.e., the entire digital histology image in this case) can be loaded one after another into cache 59. Normalization, particularly size reduction, is then performed in cache 59, which can also be considered as the formation of image patches. Furthermore, once each image (i.e., patch) is extracted from cache 59 for training, subsequent enhancement is preferably performed on each image (i.e., patch), i.e., subsequent patch enhancement. If enhancement has already been applied when filling cache 59, a second extraction of the same patch (it is generally meaningful not to always delete the patch after extraction from the cache, especially for slowly loaded images) will mean that it is being trained twice with the same enhancement.

[0112] Using the images obtained from steps 36, 37, and 38, a sequence 39 of training images can be generated, which can be used to train the model. In other embodiments, i.e., by Figure 7 In embodiments not shown, training images used to train the model may not be preprocessed, or preprocessing may at least not include image segmentation. Specifically, when using the additional first module 10', gigapixel images such as digital histological images 30 do not need to be cropped into tiles, and three-dimensional images such as MR images 31 do not need to be sliced ​​to form individual training images labeled by separate training tags. Instead, due to the use of the additional first module 10', a combined representation 13' can be obtained for all tiles or slices of the corresponding images 30, 31, based on which the model can obtain labels, thus images 30, 31 can be directly used as training images. Furthermore, according to other variations, not only training images can be provided, but also non-image data associated with the training images, such as clinical values ​​or diagnoses in textual form.

[0113] Figure 8 An embodiment of a training system 50 for training modules of a model is illustrated schematically and exemplaryly. The training system 50 includes a model module providing unit 2 and a training dataset providing unit 51 adapted to provide a training dataset, wherein the corresponding training dataset includes medical training images of one type of medical training images and training labels of one type of labeling. The medical training images are preferably as described above, for example... Figure 7 The described preprocessing is performed. Therefore, in Figure 7 The training labels are also shown. For example, in Figure 7 In step 34, several organs are segmented, where segmentation 58 can be considered a marker of dense prediction type. Figure 7 In the diagram, segment 58 is the training label.

[0114] Figure 9 Training datasets for different imaging types and different label types are illustrated schematically and exemplary. In particular, Figure 9 The diagram shows digital histological images from training dataset 60, which includes Kather100k (containing labeled colon histological images) and HubMAP (containing labeled digital histological images of the lungs, spleen, etc.). Therefore, training datasets used to train modules, and especially the same generative model, can include digital histological images from different parts of the body and with different types of labels.

[0115] Figure 9 The diagram also schematically and exemplaryly illustrates a training dataset 61 comprising MR images 31, wherein the training dataset 61 may include labeled MR images from different parts of the body (e.g., from the prostate, breast, lungs, etc.). The training dataset may also include different types of labels, such as tumor segmentation and prostate cancer grading. In this embodiment, the training dataset includes labeled images from a database called "Brats2020". Therefore, the module, and especially the same generative model, can be trained based on MR images depicting different parts of the body and including different types of labels.

[0116] Figure 9 A training dataset 62 with two-dimensional x-ray images 32 is also illustrated schematically and exemplary, wherein the training dataset 62 may include labeled x-ray images from databases named "CheXpert" and "VinBigData". The training dataset 62 may include images from different parts of the body and include different types of labels.

[0117] Therefore, a shared first module 10 is trained using medical training images of all different types and based on all different types of training labels. Furthermore, both label-specific first and second modules can be trained using training images of all types with corresponding training label types. This allows for training modules with a large amount of training data, leading to improved training and thus ultimately improved labeling accuracy when using the trained model.

[0118] The training system 50 also includes a model generation unit 5, which, when used by the training system 50, is adapted to generate a label type-specific model for different types of training labels. The model is generated based on at least a shared first module 10 and a provided second module specific to the corresponding training label type. The model takes a medical training image of any one of different types of medical training images as input and labels of the corresponding training label type as output.

[0119] In addition, the training system 50 includes a loss function providing unit 52, which is adapted to provide a label type-specific loss function, wherein the corresponding loss function depends on the deviation between the training labels and the labels generated by the label type-specific model of the corresponding training label type.

[0120] The training system 50 also includes a training unit 53 adapted to train the module using a training dataset. Specifically, the training unit 53 divides the training dataset into batches and further divides these batches into batch groups. Then, for each batch in one of the groups, the training unit 53 performs the following steps: i) generating labels for the corresponding batch of medical training images using the generated model, wherein the corresponding medical training images are labeled by labels of the corresponding training label type using a corresponding model generated for the corresponding training label type; ii) calculating the deviation between the training labels of the corresponding batch of training dataset and the corresponding generated labels for the medical training images of the corresponding batch of training dataset; and iii) calculating a label type-specific gradient for a label type-specific loss function based on the calculated deviation.

[0121] After these steps have been performed for each batch of one of the groups, training unit 53 updates the model modules by: i) combining gradients computed for the same type of label, such that a corresponding combined gradient is determined for the corresponding label type; and ii) modifying the model parameters based on the combined gradients. These steps, performed for one group, are repeated for all batch groups. The combination of gradients is preferably the sum of gradients.

[0122] Specifically, training unit 53 is configured to form batches of training data such that each batch originates from only one of training datasets 60, 61, and 62. Thus, each batch is associated with a unique task (although there are typically many batches from each training dataset 60, 61, and 62, and therefore many batches for each task). Furthermore, training unit 53 may then specifically perform the following steps for each batch of one of the groups: i) generating labels for the corresponding batch of medical training images, wherein the corresponding medical training images are labeled by labels of the corresponding training label type corresponding to the corresponding batch, using a corresponding model generated for the corresponding training label type corresponding to the corresponding batch; ii) calculating the deviation between the training labels of the training dataset for the corresponding batch and the corresponding generated labels for the medical training images of the training dataset for the corresponding batch; and iii) calculating a label type-specific gradient for a loss function specific to the corresponding label type based on the calculated deviation. Thereafter, and before moving to the next corresponding set of batches, training unit 53 will preferably update the modules of the model by: i) combining the gradients calculated for the batches such that a corresponding combined gradient is determined for the corresponding label type; and ii) modifying the module parameters based on the combined gradient.

[0123] Figure 10 The training of Model 19 is illustrated schematically and exemplary. Element 80 indicates gradient-based... Modification of the parameters of the shared first module 10, the gradient It is a combination of all batch training data from a given batch group, and therefore a combination across all tasks T1…Tn in that group.

[0124] In this embodiment, the loss function providing unit 52 is adapted to provide a classification loss function to be used for classifying the labels 40. Preferably, the classification loss function includes a cross-entropy function, particularly a categorical or binary cross-entropy function, which depends on the deviation between the training labels and the labels generated for the classification labels using the correspondingly generated model.

[0125] The classification loss function also includes a normalization of the number of possible categories, which, for example, can be performed using the logarithm of the number of possible categories.

[0126] The loss function also includes a dense prediction loss function for dense prediction of labels and an object detection loss function for object detection of labels. In this embodiment, the dense prediction loss function includes a dice loss function and / or a focal loss function, particularly an equally weighted combination of the dice loss function and the focal loss function, especially an equally weighted sum. The dice loss function is described in the article “Morphometric analysis of whitematter lesions in MR images: method and validation” by AP Zijdenbos et al., IEEE Transactions on Medical Imaging, Vol. 13, No. 4, pp. 716–724 (1994), which is incorporated herein by reference. The focal loss function is described in the article “Focal Loss for Dense Object Detection” by T.-Y. Lin, Computer Science, Computer Vision and Pattern Recognition, arXiv: arXiv:1708.02002 [cs .CV], which is incorporated herein by reference.

[0127] Figure 11 A flowchart illustrating an exemplary implementation of a labeling method for labeling medical images is provided.

[0128] In step 101, a first module is provided, configured to output a representation of any one of different types of medical images, and a second module is provided for different types of labeling performed by the model module providing unit 2. As explained above, the corresponding second module is specific to the corresponding labeling type and is configured to take a representation of the medical image as input and a label of the corresponding labeling type as output. The first module includes: a) a shared first module for different labeling types, configured to take a medical image of any one of different types of medical images as input and a representation of the medical image as output, and b) a labeling type-specific first module, wherein the corresponding labeling type-specific first module is specific to the corresponding labeling type and is configured to take a representation provided by the shared first module as input and a labeling type-specific representation as output.

[0129] Still in step 101, the medical image providing unit 3 provides a medical image of one of different types of medical images, and the marker type providing unit 4 provides a marker type of one of different marker types.

[0130] In step 102, a model is generated based on at least a shared first module and a second module containing provided tags specific to the provided type. This model takes a medical image of any of different types as input and outputs tags of the provided tag type. Step 102 is performed by the model generation unit 5. In step 103, the tagging unit 7 tags the provided medical image using the generated model. In step 104, the tagged medical image can be output to the user using the display unit 9.

[0131] Figure 12 A flowchart illustrating an exemplary implementation of a training method for a module used to train a model of a labeling system is shown.

[0132] In step 201, the model module providing unit 2 provides a first module and a second module to be trained, wherein these modules include a shared first module, and preferably also include a label type-specific first module and a label type-specific second module. Specifically, these modules include a shared first module 10, a dense prediction first module 11, an object detection first module 12, a classification second module 20, a dense prediction second module 21, and an object detection second module 22.

[0133] Furthermore, in step 201, the training dataset providing unit 51 provides a training dataset, wherein the corresponding training dataset includes a medical training image of one type of medical training image and a training label of one type of label. Also in step 201, the model generation unit 5 generates a label type-specific model for the different types of training labels. As explained above, a model for the corresponding training label type is generated based on at least a shared first module and a provided second module specific to the corresponding training label type, which takes a medical training image of any one of the different types of medical training images as input and labels of the corresponding training label type as output.

[0134] Furthermore, in step 201, the loss function providing unit 52 provides a label type-specific loss function, wherein the corresponding loss function depends on the deviation between the training labels and the labels generated by using the corresponding label type-specific model that has been generated for the corresponding label type.

[0135] In step 202, training unit 53 divides the provided training dataset into batches, and further divides the batches into batch groups. In step 203, training unit 53 selects one of the batch groups, and in step 204, training unit 53 selects a batch from the selected batch group.

[0136] In step 205, for each of the selected batch of medical training images, a label is generated using the generated model, wherein the corresponding medical training image is labeled with a label of the corresponding training label type using a corresponding model generated for the corresponding training label type.

[0137] In step 206, the training unit 53 calculates the deviation between the training labels of the selected batch of training datasets and the corresponding generated labels.

[0138] Furthermore, in step 207, training unit 53 computes a label-type-specific gradient for the label-type-specific loss function based on the calculated bias. For example, the gradient can be determined based on the bias using backpropagation techniques known from and implemented in the function tensor.backward from the PyTorch 2.0 package.

[0139] It should be noted again that each batch can be associated with a corresponding label type or label subtype, so that only a single model needs to be generated in step 205 and only a single loss function needs to be considered in step 207.

[0140] In step 208, training unit 53 determines whether steps 205 to 207 have been performed for all batches of the group. If not, the method proceeds to step 204. Otherwise, the method proceeds to step 209.

[0141] In step 209, training unit 53 determines whether steps 203 to 208 have been performed for all groups. If not, the method continues to step 203, where the next group in the batch is selected. Otherwise, the method continues to step 210.

[0142] In step 210, training unit 53 determines whether the termination criterion is met, and if so, the method terminates in step 211. Otherwise, the method continues to step 203, where steps 204 through 209 are executed again for all batch groups. If it is determined in step 210 that the termination criterion is not met, the method may also continue to step 202, where, in this case, batch group formation is performed again, but this time resulting in the formation of other batch groups. For example, the termination criterion could be that the deviation between the training labels and the labels generated during the training process is less than a predefined threshold, or that the training dataset has been used to train the module a predefined number of times.

[0143] Figure 13A The changes in the memory state of the computer performing the above training method during steps 205 to 208 are illustrated schematically and exemplary, with the elapsed time from top to bottom indicated by the arrows on the left. Steps 205 to 208 can be viewed as forming a single optimization or update step for the trained model. At the beginning of each such optimization or update step, i.e., before processing the first batch of images in the corresponding batch group, the initial model parameters M of all the first and second modules 10, 11, 12, 20, 21, 22 are stored in memory. It should be noted that all M are stored, even though only some of them will always be needed each time thereafter, depending on the corresponding model generated for the corresponding batch being processed.

[0144] During the execution of step 205 for the first batch in the group, in addition to the markers mentioned above, for each image in the batch, additional activations A1, i.e. node outputs, are determined along the entire model generated for the first batch and stored in memory.

[0145] Then, in steps 206 and 207, A1 is used to calculate the gradient, which consists of a first part G1 of one or more first modules of the model generated for the first batch and a second part g1 of the second module for the first batch, both of which are stored in memory. Once the gradients G1 and g1 from the first batch are known, the basis for calculating these gradients, A1, can be removed from memory. Therefore, at the end of the processing of the first batch of images, only the (unmodified) module parameters M and the gradients, along with their two parts G1 and g1, are stored in memory.

[0146] By repeating the above processing steps on the second batch of images in the corresponding group, a second activation A2 is determined, and a second gradient is obtained based on this. This second gradient is now divided into a first part G2 for one or more first modules in the second batch and a second part g2 for the second module in the second batch. Now, it is not necessary to store the entire first part G2 of the gradient in memory. Instead, since it is at least partially applied to the first module already used in the first batch, and therefore the first part of the already stored gradient G1 is also applied to that first module, G2 can be added to G1 at least to some extent; that is, G1 can be replaced at least to some extent by the sum of G1 + G2. This is referred to herein as gradient accumulation. On the other hand, assuming the second batch means training a different task than the first batch, the second module used for the second batch will also be different from the second module used for the first batch, such that the second gradient part g2 cannot be added to its previous corresponding part g1. In the illustrated implementation, g2 is therefore stored separately in memory. Furthermore, similar to after processing the first batch, activation A2 is removed from memory, and it is no longer needed. It should be noted that, since the second module has relatively fewer parameters compared to the first module, the increase in memory space occupied by the additional storage of g2 will generally be negligible compared to the memory space saved by: a) gradient accumulation; b) removal of used activations from memory; and c) the fact that not all modules or even only the required modules are kept in memory and / or loaded into memory in order to process the second batch.

[0147] By performing similar processing on all n image batches in the group, specifically by always discarding the corresponding activations after computing the corresponding gradients, and by storing only the accumulated portion of the gradients of the first module in memory, rather than each gradient portion from the first module of the corresponding image batch, except for the gradients used for the corresponding second module, the memory requirement remains substantially constant throughout the processing of a set of image batches. Based on the model parameters M (which remain unchanged during steps 205 to 208, and therefore remain unchanged throughout the processing of a given set of training image batches), and based on at least the partially accumulated portion G1 + G2 + … + Gn of the gradients of the first module and the portions g1, g2, …, gn of the gradients of the second module, the model parameters can eventually (i.e., after the last batch in the group has been processed) be updated to M'. Steps 205 to 208 are then repeated for the next set of training image batches using the updated model parameters M'. Thus, once the updated model parameters M' have been stored, the previous model parameters M can also be discarded from memory, meaning that the memory requirement can remain substantially constant throughout the training.

[0148] Although Figure 13AThis illustrates an implementation that sequentially processes different batches of training images (while different images in any given batch can still be processed in parallel), but in other implementations, these batches can be processed in parallel, such as... Figure 13B As illustrated and illustratively. When using batch parallel processing to execute the training method, two or more of steps 205 through 208 can be executed in separate but essentially similar processing threads. Then, the activations A1, A2… and the corresponding gradient pairs (G1, g1), (G2, g2),… are stored in parallel by the respective separate threads, but the model parameters M still only need to be stored once. Figure 13B The left-to-right arrangement of the elements stored in the table indicates which thread stores the corresponding element. As shown, if there are more batches to be processed than the number of threads scheduled for parallel processing (n > t, where t is the number of threads), then the processing is performed sequentially by one or more threads that process more than one batch, such as... Figure 13A As shown. Therefore, in the illustrated embodiment, in such a thread, in Figure 13B The algorithm includes at least the first two threads, since it is assumed that n ≥ t + 2. The gradient parts of the first module (G1, G(t+1)), (G2, G(t+2)), ... are accumulated separately. Once all image batches from a group have been processed, the partially accumulated gradients (G1, G(t+1)), (G2, G(t+2)), ... from different threads are combined in the final accumulation, and the updated model parameters M' are determined based on this and based on the gradient parts g1, g2, ... of the second module collected by different threads, as shown in... Figure 13A In the sequence of cases.

[0149] The modules, and therefore the models generated based on them, can handle heterogeneous data sources such as histology, MR imaging (MRI), and X-ray imaging, where the provided training dataset includes data from these heterogeneous data sources. Furthermore, different types of medical image labels (e.g., classification, dense prediction, particularly segmentation and object detection) can be processed and integrated into a multi-label training process, which can also be considered a multi-task training process if different types of labels are considered to refer to different types of tasks. Generating label-type-specific models by combining modules can be considered a modular self-assembly pipeline employing a shared first module or a combination of a shared first module and one of the label-type-specific first modules. The generation of the first module can be considered a representation of a neural representation, as it is generated by a neural network defined by a shared first encoder optionally combined with one of the label-type-specific first modules. The first modules can also be considered encoders, as they generate neural representations, and the generation of such neural representations can be considered encoding. Correspondingly, the second modules can be considered decoders. In fact, the shared first module can also be considered solely as an encoder, and the label-type-specific first module, alone or in combination with the corresponding second module, can be considered as a decoder.

[0150] Label type classification can refer to the classification of brain tumors in MR images, the classification of breast tumors in digital histology images, and so on. Compared with known models used to perform labeling tasks (such as classifying medical images), the corresponding generated model can perform the corresponding labeling task with excellent sample efficiency. In particular, it can achieve the same classification accuracy with less training data compared with known labeling models.

[0151] The second object detection module can be adapted to output the region within the corresponding medical image where the object to be detected is located. For example, this region can be a two-dimensional bounding box determined by the second object detection module, such that the bounding box includes the object to be detected.

[0152] Since a label-type-specific model is generated by combining corresponding first and second modules, a corresponding channel or "computation graph" can be provided for each label. This graph can be dynamically constructed and stored only during the label-type-specific forward computations during training. To combine the individual label-type-specific computation graphs during training, the prediction errors of the individual second modules are preferentially accumulated. Therefore, only one computation graph needs to be stored in memory preferentially. To update the parameters of the first and second modules, the corresponding weights are preferably adjusted using known backpropagation methods, such as those implemented in the pytorch 2.0 package using the function tensor.backward. For gradient accumulation, the weights are combined (i.e., accumulated) before updating the module parameters as described above. The combination of gradients allows for the establishment of training schemes where a single step of updating the module parameters can consider different types of labels even in any order.

[0153] Because models for generating and training different types of labels use the same shared first module, the parameters of the shared first module, especially the corresponding weights and biases of the neural network forming the shared first module, do not need to be copied and stored separately for each type of label. This allows training of models for generating different types of labels to be performed with reduced storage requirements. Therefore, training units can be adapted to store module parameters, especially the corresponding weights and biases of the neural network forming the module, only once, i.e., they are not repeated for each type of label.

[0154] The neural network of the module includes activation functions, where training units can be adapted to store values ​​computed using the activation function, i.e., the node's output, also known as the activation, in memory only for one type of label at a time. After the node's output (i.e., the activation) has been used to compute the gradient of the corresponding label type to be used to update the module parameters, i.e., after the backpropagation of the corresponding label type has been performed, the activation of the corresponding label type is discarded, such that not all activations for all label types are stored simultaneously. Therefore, in a preferred embodiment, the only increase in memory requirement due to multiple types of labels is due to the gradients of different types of labels, such as those computed using the activation function. Figure 13A and Figure 13B As shown. Alternatively, memory requirements can be further reduced by discarding gradient portions required to update the parameters of the second module. According to this option, after a batch of training data has been processed and the corresponding gradients for the respective label type (especially subtype) have been computed, the gradient portions associated with the corresponding second module that has been used to process the batch of training data are used to update the parameters of that module, wherein the gradient portions are then discarded from memory.

[0155] Furthermore, preferably, the modules are configured such that the number of module parameters adapted during training of the shared first module is greater than the number of module parameters updated during training of other modules that together form a model of the corresponding label type. In other words, preferably, the shared portion of the corresponding model represents the majority of the total model size. These aspects of modules, models, and training enable multi-label learning across many types of labels, even on hardware with limited computing power.

[0156] The concept of computing several gradients for different types of labels and for different training datasets of the same type of labels, and combining these gradients before the next update step (i.e., before the next step after modifying the module parameters), allows for enhanced training versatility by considering multiple instances of different types of labels and even the same type of labels in each update step. Multiple instances (i.e., multiple tasks of label subtypes trained in combination at each update step) correspond to their respective second-order modules. A task or label subtype can refer to a specific, label-related problem, such as, for example, tumor classification and pneumonia classification. Both share the same label type and receive their inputs from first-order modules, but their second-order modules differ, allowing them to solve only one specific problem (e.g., tumor classification or pneumonia classification).

[0157] The training unit is preferably adapted to prevent the size of the training dataset for different types of labels from affecting the contribution of the corresponding label type to the module training. Preferably, each type of label contributes equally to the module training. To achieve this, the training unit is preferably configured to provide one training batch for each label subtype for each update step, i.e., the batches are grouped such that each group contains exactly one batch for each label subtype.

[0158] Furthermore, the training unit is preferably configured to restart image loading for that subtype once all images for that subtype have been used; that is, to then provide additional batches of images for that subtype based on the images already used for training. In this way, information about the length of the image dataset is not required beforehand, allowing each epoch to have a different length depending on the data augmentation. An epoch is understood herein to refer to a complete training cycle on the training dataset. In the case of a training dataset based on more complex medical images (e.g., 3D or gigapixel images), its length (i.e., the number of 2D images used for training) can vary depending on the corresponding augmentation applied. For example, if an entire 2D image is ultimately used for training, the length can change due to the original non-cubic 3D image being rotated and subsequently split into 2D images in a direction independent of rotation, or depending on the scaling factor applied to the original 2D image.

[0159] Since the step of modifying module parameters, i.e. the update step, takes priority on the sum of gradients, and since this sum is commutative, the order of the label types used for the update does not affect the result, thus further contributing to the described versatility of training.

[0160] Loss functions for different types of labels can vary significantly in magnitude, often leading to a situation where the loss for one type of label dominates the entire training process. To counteract this, the loss function providing unit can be adapted to normalize the corresponding loss function for each type of label, such that different loss functions for different types of labels provide loss values ​​within the same range. This same range could be, for example, between 0 and 1, but in principle, it could also be approximately 1 or even approximately 1000. In an implementation, the loss function providing unit is configured to normalize the loss function such that, for each loss function, the expected value provided by the corresponding loss function for random input and reinitialized weights is the same, for example, 1.

[0161] The loss function providing unit is preferably configured to provide a normalized loss function for each label subtype, wherein a corresponding normalization is selected individually for each label subtype by selecting a corresponding factor, and the loss function for the corresponding label subtype is multiplied by the corresponding factor. For label subtypes of classification type, the loss function providing unit is configured to select a factor based on the number of categories considered in the classification. For label subtypes of dense prediction type, it has been found that normalization may not be necessary, i.e., the loss function providing unit can be configured to not normalize the loss function for dense prediction type labels. For label subtypes of object detection type, the loss function providing unit can be configured to provide: a) a classification loss function to be used as the loss function for those parts of the label indicating the box classification, and b) an additional loss function to be used as the loss function for those parts of the label indicating the position and size of the corresponding box, and normalize: a) the classification loss function and b) the additional loss function with a constant factor using the number of box categories considered.

[0162] In this implementation, the training unit is adapted to use the AdamW optimizer for the training process. This particular optimizer is disclosed in I. Loshchilov's paper "Decoupled Weight Decay Regularization," arXiv: 1711.05101, which is incorporated herein by reference. In this implementation, the learning rate is 0.001 and the weight decay is 0.05 during training. Furthermore, in this implementation, the training settings disclosed in Z. Liu et al.'s paper "Swin transformer: Hierarchical vision transformer using shifted windows," Proceedings of the IEEE / CVF International Conference on Computer Vision, pp. 10012–10022 (2021) are used, which is also incorporated herein by reference. However, training can also be performed using other training settings.

[0163] The training unit can be adapted to perform training loops that iterate through batches in a given order. After each iteration step, gradients are computed by evaluating a loss function specific to the corresponding label type relative to the corresponding batch. Gradients are accumulated, specifically summed, until the optimization step, i.e., the step that modifies the module parameters, is scheduled by the training unit. During the optimization step, the module parameters are updated once, using the accumulated gradients, relative to all types of labels since the last update step. After the accumulated gradients have been used in the optimization step, the previously stored gradients and the accumulated gradients are removed from the memory of the training unit. Note again that while each training update can include all types of labels, and in particular their subtypes, each batch preferably contains only images of the same label type. Furthermore, all images contained in a batch are preferentially passed to the same second module.

[0164] A module comprises at least one shared module (i.e., at least a shared first module) and a label-type-specific module. Modules can be assembled for their respective label types. This provides a flexible architecture that can adapt to the needs of a given label type. For a given label type, one or more shared modules can be used in their forward pass, i.e., they can be combined to generate a label-type-specific model. In other words, each type of label can independently generate its computational graph, even in each iteration during the training process. This flexibility liberates the training scheme from a rigid, predefined architecture.

[0165] Preferably, the original normalized layer in the shared modules, and if more than one shared module exists, the original normalized layer in all shared modules is recursively replaced with a layer normalization that does not utilize the batch range parameter. Layer normalization is disclosed in the paper "Layer Normalization" by JL Ba et al., arXiv: 1607.06450 (2016), which is incorporated herein by reference. Since one or more shared modules of the model can be based on a known architecture, different corresponding normalizations can be used within them.

[0166] Normalization is a method to accelerate training and prevent gradient explosion or reduction during error backpropagation. A more detailed description can be found in the paper "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" by S. Ioffe et al., CoRR 2015, abs / 1502.03167, https: / / arxiv.org / pdf / 1502.03167.pdf, which is incorporated herein by reference. This paper also describes parameters that are adjusted with each forward pass; these parameters can be referred to as batch-range parameters. When looping through batches of different tasks in different domains, these parameters adapt to each domain in the loop, thus hindering generalization. Therefore, layer normalization is applied.

[0167] Normalization layers can improve the overall performance of a model (i.e., achievable label accuracy) and generalization (i.e., the ability to maintain label accuracy across tasks) as well as, in particular, training speed.

[0168] Different types of medical images that can be used to train the module may include, for example, two-dimensional medical images, particularly two-dimensional X-ray images and / or two-dimensional gigapixel images, as well as three-dimensional images, particularly tomographic three-dimensional images.

[0169] The preprocessing unit can be configured to also apply known 3D enhancements to the 3D tomographic image, followed by slicing, and apply a set of known domain-specific (i.e., specific to tomography, X-ray, or histological imaging) 2D enhancements. Specifically, the orientation of the imaging volume can be enhanced if the maximum edge length is greater than twice the minimum edge length. The latter is considered a good indicator of whether the corresponding image (i.e., its volume) is cubical. Therefore, if the imaging volume is cubical, its orientation is essentially enhanced. An example of domain-specific 2D enhancement would be the inversion of X-ray images. In X-rays, randomly inverting the image makes sense because in DICOM, there is a "show inverted grayscale" property that some users might use while others do not. Therefore, it makes sense to use enhancements that prevent the network from changing with variations.

[0170] The preprocessing unit can also be configured to enhance intensity-based 2D images, such as grayscale 2D images, using standard enhancements that utilize the albumin library disclosed in the paper “Albumentations: fast and flexible image augmentations” by A. Buslaev et al., arXiv 1809.06839 (2018), which is incorporated herein by reference. For example, standard enhancements include CLAHE, Sharpen, Emboss, RandomBrightnessContrast, RandomGamma, Gaussian Noise, HueSaturationValue, and ShiftScaleRotate, as disclosed in the paper by A. Buslaev et al.

[0171] The preprocessing unit can also be adapted to perform enhancements using domain-specific knowledge for each different domain. That is, the preprocessing unit can be configured to perform preprocessing, particularly enhancement, depending on the type of image, especially its domain.

[0172] The preprocessing unit can also be configured to preprocess the two-dimensional X-ray image by performing image inversion. Therefore, in one embodiment, the correspondingly processed X-ray image is input into the generated model. In another embodiment, image inversion is performed only on a certain percentage of the two-dimensional X-ray image, where this percentage could be, for example, 30%. However, many other percentages can also be used.

[0173] The preprocessing unit can also be configured to flip digital histological images and two-dimensional images extracted from three-dimensional tomographic images. Mirroring can also be applied to two-dimensional images extracted from three-dimensional tomographic images. Furthermore, preprocessing of digital histological images can also include channel washing.

[0174] Regarding the order of image enhancement steps, it is preferable that source enhancement—that is, if applicable, such as 3D enhancement—precede any 2D enhancement schemes. Within 2D enhancement schemes, the order may be less relevant.

[0175] The classification labels allow each training medical image to be associated with a single training label from a set of categories, where the number of categories is indicated by c below. A latent representation (i.e., representation 13 or 13') computed by a shared first module 10 or another first module 10' is used by the second module 20 to perform the classification labeling. The classification second module 20 preferably includes fully connected layers to perform the classification.

[0176] The loss function used for classification labels is preferably a classification cross-entropy function, wherein preferably, the classification cross-entropy function is normalized with respect to the number of classes to prevent bias in classification labels for classification types with a large number of classes. For normalization, the logarithm of the number of classes for the corresponding classification type can be used. Therefore, in the example, the loss function for classification labels can be based on the following equation: (1) in, It is a classification loss function. y1 is the classification cross-entropy function, y2 is the training label, and y3 is the label obtained during training using the module to be trained.

[0177] In the case of multi-label classification, the loss function providing unit can be adapted to consider inputs, each with multiple binary classification targets y2. Preferably, in this case, the classification loss function includes, and preferably is normalized, a binary cross-entropy loss function. In this case, the loss is preferably normalized to 1. Preferably, this is achieved by adding a constant factor. This is used to perform normalization to 1. Therefore, the classification loss function in this case can be described by the following equation: (2) in, It is a binary cross-entropy loss function. Variables y1 and y2 each have two entries, which can be regarded as a two-dimensional vector or an array with two elements. The classification target y2 can be referred to as one of the baseline truth labels for multi-label binary classification, and can take the value [1.0, 0.0] or [0.0, 1.0], while y1 refers to the corresponding training label output by the model, and can therefore be regarded as a prediction and can take any two true values, i.e., for example, [0.3, 0.9]. Equation (2) can be used for each of the multiple labels in multi-label classification, where y2 refers to the corresponding different baseline truth, and y1 refers to the respective model prediction.

[0178] Between single-label and multi-label types, the cross-entropy loss function is preferred for multi-class classification, and the binary cross-entropy function is preferred for binary classification.

[0179] For reference Figure 5 and Figure 6 As described, skip connections can be provided between the feature map of the shared first module 10 and the upsampling layer of any one of the label type-specific first modules 11, 12, wherein these skip connections can be implemented by concatenating the corresponding feature map with the upsampling layer.

[0180] The first module 11, the second module 21, and the shared first module 10 are used to perform dense prediction labeling, specifically semantic segmentation labeling. Semantic segmentation labeling assigns a category label to each pixel of a two-dimensional image, where the output for each pixel can be a number ranging from 1 to c, where c is the number of possible categories.

[0181] Although the first modules 11 and 12 have been described above as labels specific to dense prediction and object detection types, respectively, it has been found that the representations 14 and 15 provided as outputs by the first modules 11 and 12, respectively, and the representations 13 and 13' provided as outputs by the shared first module 10 and the additional first module 10', can actually be used more broadly. In particular, representations 13, 13', 14, and 15 have been found to be useful for image compression. For example, medical images compressed based on representations 13, 13', 14, and 15 can be more easily correlated with patient prognostic data such as survival time, treatment data, or diagnosis. Therefore, compressed versions of images can be used to predict future medical indicators, such as the likelihood of successful treatment or general medical outcomes for a given patient. Compression is particularly relevant for histological images, which consist of a large number of pixels and a large three-dimensional magnetic resonance (MR) volume.

[0182] Therefore, the present invention also relates to the use of representations 13, 13', 14, and 15, provided respectively by the first modules 10, 10', 11, and 12, for predicting patient prognosis. Prediction of patient prognosis based on a given medical image can be understood as an additional type of label, or a subtype of classification type label.

[0183] In this implementation, the provided medical image is labeled with a marker indicating patient prognosis by: a) dividing the image into several image blocks; b) determining a representation 13, 14, 15 for each of the blocks or a single representation 13' for a combination of blocks using one or more corresponding first modules 10, 10', 11, 12; and c) providing the representations 13, 13', 14, 15 determined individually or in combination for the several blocks as input to a second module for patient prognosis prediction, wherein if the second module for patient prognosis prediction receives the representations 13, 13', 14, 15 of the patient's medical image as input, then the second module for patient prognosis prediction has been trained to provide a marker indicating patient prognosis as output.

[0184] It has been found that patient prognosis can be predicted with particularly high accuracy if representations 13, 13' provided as output by the shared first module 10 or another first module 10' are used as representations of tiles in the corresponding medical image. Therefore, preferably, patient prognosis is predicted by providing tiles of the medical image as input to the shared first module 10, such that the shared first module 10 provides a corresponding representation 13 for each tile as output, wherein these representations are optionally provided as input to another first module 13' to obtain a combined representation 13', wherein one or more representations 13, 13' are provided as input to a second module for patient prognosis prediction.

[0185] Similar to modules 20, 21, and 22, the patient prognosis prediction module may include a single linear layer or a single convolutional layer for computing corresponding labels for the respective patient prognosis. To train the patient prognosis prediction module, a training dataset comprising medical training images and associated training labels indicating the corresponding patient prognosis may be used, wherein the patient prognosis indicated by the training labels is preferably real, i.e., the prognosis of patients in actual clinical records.

[0186] However, as further indicated above, in order to predict patient prognosis based on one or more representations provided as output by one of the first modules 10, 10', 11, 12 for a given medical image, any other known machine learning architecture may also be used in principle, provided that the machine learning architecture is configured to receive one or more representations as input and provide an output indicating patient prognosis.

[0187] Based on a study of the accompanying drawings, the disclosure, and the appended claims, other variations of the disclosed embodiments can be understood and implemented by those skilled in the art in practicing the claimed invention.

[0188] In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality.

[0189] A single unit or device can fulfill the functions of several items listed in the claims. The mere fact that certain measures are listed in mutually different dependent claims does not indicate that a combination of these measures cannot be used advantageously.

[0190] Steps such as providing medical images, providing label types, providing modules, generating models, generating labels, and modifying module parameters, which are performed by one or more units or devices, can be performed by any other number of units or devices. These steps, and especially the labeling and training methods, can be implemented as program code devices and / or dedicated hardware for computer programs.

[0191] Computer programs can be stored / distributed on suitable media (such as optical storage media or solid-state media), supplied with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0192] Any reference numerals in the claims should not be construed as limiting the scope.

[0193] This invention relates to the use of modular models for labeling medical images. Based on these modules, a model can be generated comprising: a) a shared first module for different label types, configured to take a medical image as input and a representation of the medical image as output; b) a label type-specific first module, specific to the label type and configured to take a representation provided by the shared first module as input and a label type-specific representation as output; and c) second modules, wherein each second module is specific to a corresponding label type and configured to take a medical image representation as input and a label of the corresponding label type as output. This modular structure allows for efficient multi-task training, thus enabling more accurate labeling.

Claims

1. A labeling system for labeling medical images, wherein, The marking system (1) includes: - A model module providing unit (2) is adapted to provide a first module (10, 11, 12) configured to output representations (13, 14, 15) of any of different types of medical images (30, 31, 32) and a second module (20, 21, 22) for different label (40, 41, 42) types, wherein the corresponding second module (20, 21, 22) is specific to the corresponding label (40, 41, 42) type and is configured to take the representations (13, 14, 15) of the medical images (30, 31, 32) as input and the labels (40, 41, 42) of the corresponding label (40, 41, 42) type as output, wherein the first module (10, 11, 12) is adapted to provide a first module (10, 11, 12) configured to output representations (13, 14, 15) of the corresponding label (40, 41, 42) type, wherein the first module (10, 11, 12) is adapted to provide a first module (10, 11, 12) configured to output representations (13, 14, 15) of the medical images (30, 31, 32) and a second module (20, 21, 22) for different label (40, 41, 42) types ... second module (20, 21, (0, 11, 12) includes: a) a shared first module (10) for different label (40, 41, 42) types, the shared first module (10) being configured to take a medical image (30, 31, 32) of any one of the different types of medical images as input and a representation (13) of the medical image (30, 31, 32) as output; and b) a label type-specific first module (11, 12), wherein the label type-specific first module (11, 12) is specific to the label (41, 42) type and is configured to take the representation (13) provided by the shared first module (10) as input and a label type-specific representation (14, 15) as output. - Medical image providing unit (3), adapted to provide a medical image (30, 31, 32) of one of the different types of medical images. - A tag type providing unit (4) is adapted to provide a tag (40, 41, 42) type of one of the different tag types. - A model generation unit (5) is adapted to generate a model based on at least the shared first module (10) and a second module (20, 21, 22) provided with a type specific to the provided tags (40, 41, 42), the model taking a medical image (30, 31, 32) of any of the different types of medical images as input and a tag of the provided tag type (40, 41, 42) as output, and - Labeling unit (7), adapted to generate labels (40, 41, 42) of the provided medical image (30, 31, 32) by using the generated model.

2. The marking system according to claim 1, wherein, The different types of medical images (30, 31, 32) include at least two types from the following list: digital histology images, tomographic images, X-ray images, and ultrasound images.

3. The marking system according to any one of claims 1 and 2, wherein, The different label (40, 41, 42) types include at least two types from the following list: a) dense prediction, particularly segmentation or image generation; b) classification; c) object detection.

4. The marking system according to any one of the preceding claims, wherein, The second module (21, 22) of the provided second module, which is specific to the tag (41, 42) type, is configured to: a) take the representation (14, 15) provided by the tag type-specific first module (11, 12) of the tag (41, 42) type as input; and b) take the tag (41, 42) of the tag type as output.

5. The marking system according to claim 4, wherein, The first module (11, 12) specific to the tag type is specific to the tag (41, 42) of the dense prediction type and / or object detection type, wherein the second module (21, 22) of the provided second module is specific to the tag (41, 42) type and is configured to: a) take the representation (14, 15) provided by the first module (11, 12) as input; and b) take the tag (41, 42) of the tag type as output, the second module (21, 22) being configured to output the tag (41, 42) of the dense prediction type and / or the object detection type and is therefore a dense prediction second module (21) and / or an object detection second module (22).

6. The marking system according to any one of the preceding claims, wherein, The second module in the provided second module, which is specific to the tag (40) type, is configured to: a) take the representation (13) provided by the shared first module (10) as input; and b) take the tag (40) of the tag type as output.

7. The marking system according to claim 6, wherein, The second module (20) of the provided second module is specific to the tag (40) type and is configured to: a) take the representation (13) provided by the shared first module (10) as input; and b) take the tag (40) of the tag type as output, the second module (20) being configured to output the tag (40) of the classification type and thus being a classification second module (20).

8. The marking system according to claim 7, wherein, The shared first module (10) is configured to take any one of the different types of medical images (30, 31, 32) as input and output a representation (13) of each of a plurality of regions of the medical images (30, 31, 32), wherein the classification second module (20) is configured to: a) take the plurality of representations (13) provided by the shared first module (10) as input; and b) take a marker (40) indicating the predicted patient prognosis as output.

9. The marking system according to any one of the preceding claims, wherein, The corresponding second modules (20, 21, 22) include a single linear layer or a single convolutional layer for computing the corresponding labels (40, 41, 42).

10. The marking system according to any one of the preceding claims, wherein, The labeling system further includes a preprocessing unit (6) adapted to preprocess the medical images (30, 31, 32) before using them as input to the generated model, wherein the preprocessing includes at least one of the following steps: a) transforming the medical images (30, 31, 32) to a predefined spatial dimension; b) normalizing the image values ​​of the medical images (30, 31, 32); and c) transforming the medical images (30, 31, 32) into a predefined number of image elements.

11. The marking system according to claim 10, wherein, The preprocessing includes forming a plurality of image portions of the medical image (30, 31, 32), wherein the model generation unit (5) is adapted to generate the model such that the model takes the plurality of image portions as input, wherein each of the plurality of image portions is provided as input to the shared first module (10) such that the shared first module (10) generates a plurality of representations (13) as output, each of the plurality of representations (13) being a representation of the corresponding image portion, wherein the provided tag (40, 41, 42) type of the output of the model is determined based on the plurality of representations generated as output by the shared first module (10) and at least the second module (20, 21, 22), the second module (20, 21, 22) being specific to the provided tag (40, 41, 42) type.

12. The marking system according to any one of the preceding claims, wherein, The labeling system is a labeling system for labeling combinations of several input elements, the several input elements including: a) several medical images (30, 31, 32) or b) one or more medical images (30, 31, 32) and non-image data associated with the one or more medical images (30, 31, 32), wherein the model generation unit (5) is adapted to generate the model such that the model takes the several input elements as input, wherein at least the second module (2) is based on several representations and at least the type of the provided label (40, 41, 42). 0, 21, 22) to determine the provided tags (40, 41, 42) as the output of the model, the plurality of representations being generated by the plurality of input elements, wherein the plurality of representations of the plurality of medical images (30, 31, 32) are generated by providing the plurality of medical images (30, 31, 32) as inputs to the shared first module (10), such that the shared first module (10) generates one or more outputs, each output being a representation or corresponding medical image (30, 31, 32).

13. The marking system according to claim 11 or 12, wherein, The model module providing unit (2) is adapted to provide an additional first module (10') configured to take the plurality of representations (13) as input and take one or more additional representations (13') as output, wherein a corresponding second module (20, 21, 22) and an optional tag type-specific first module (11, 12) are configured to take the one or more additional representations (13') as input, wherein the model generation unit (5) is adapted to further generate the model based on the additional first module (10').

14. A training system for a module for training a model of a labeling system as described in any of the preceding claims, wherein, The training system (50) includes: - A model module providing unit (2) is adapted to provide a first module (10, 11, 12) configured to output representations (13, 14, 15) of any of different types of medical images (30, 31, 32) and a second module (20, 21, 22) specific to the label type (40, 41, 42), wherein the corresponding second module (20, 21, 22) is specific to the corresponding label type (40, 41, 42) and configured to take the representations (13, 14, 15) of the medical images (30, 31, 32) as input and take the labels (40, 41, 42) of the corresponding label type as output, wherein the module can be modified by the module's module The parameters are modified, wherein the first module (10, 11, 12) includes: a) a shared first module (10) for the different label (40, 41, 42) types, configured to take a medical image (30, 31, 32) of any one of the different types of medical images as input and a representation (13) of the medical image (30, 31, 32) as output; and b) a label type specific first module (11, 12), wherein the label type specific first module (11, 12) is specific to the label (41, 42) type and is configured to take the representation (13) provided by the shared first module (10) as input and a label type specific representation (14, 15) as output. - Training dataset providing unit (51), adapted to provide training datasets (60, 61, 62), wherein the corresponding training datasets (60, 61, 62) include medical training images of one type of medical training images and training labels of one type of different labels (40, 41, 42). - Model generation unit (5), adapted to generate label type-specific models for the different training label (40, 41, 42) types, wherein a model for the corresponding training label (40, 41, 42) type is generated based on at least the shared first module (10) and a second module (20, 21, 22) provided for the corresponding training label (40, 41, 42) type, the model taking a medical training image of any one of the different types of medical training images as input and labels of the corresponding training label (40, 41, 42) type as output. - A loss function providing unit (52) is adapted to provide a label type-specific loss function, wherein the corresponding loss function depends on the deviation between the training labels and the labels generated by using the generated label type-specific model for the corresponding label type, and - Training unit (53), adapted to train the module by: a) Divide the training dataset (60, 61, 62) into batches, and further divide the batches into batch groups. b) For each batch of one of the groups, perform the following steps: i) generate labels for the medical training images of the corresponding batch using the generated model, wherein the corresponding medical training images are labeled with labels of the corresponding training label (40, 41, 42) type using the corresponding model generated for the corresponding training label (40, 41, 42) type; ii) calculate the deviation between the training labels of the training dataset of the corresponding batch and the corresponding generated labels generated for the medical training images of the training dataset of the corresponding batch (60, 61, 62); iii) calculate the label type-specific gradient of the label type-specific loss function based on the calculated deviation. c) Update the module of the model by: i) combining gradients computed for labels (40, 41, 42) of the same type, such that a corresponding combined gradient is determined for the respective label (40, 41, 42) type; and ii) modifying the module parameters based on the combined gradient. d) Repeat steps b) and c) for all batch groups.

15. The training system according to claim 14, wherein, The different label types include classification labels (40) and / or object detection type labels, wherein the loss function providing unit (52) is adapted to provide a classification loss function to be used for the classification labels (40) and / or the object detection type labels as a loss function specific to the different label types, wherein the classification loss function includes normalization of a plurality of possible categories for classification.

16. A set of model modules is to be used to generate tag-type-specific models for different tag types (40, 41, 42), wherein, The model for the corresponding training label (40, 41, 42) type takes a medical image (30, 31, 32) of any of the different predefined types of medical images as input and the label of the corresponding label (40, 41, 42) type as output, and is generated based on at least a shared first module (10) and a second module (20, 21, 22) of the set of model modules, the second module (20, 21, 22) being specific to the corresponding label (40, 41, 42) type, wherein the set of model modules includes a first module (10, 11, 12) and a second module (20, 21, 22) for the different label (40, 41, 42) types, wherein the corresponding second module (20, 21, 22) is specific to the corresponding label (40, 41, 42) type and is configured to take the medical image (30, 31, 32) of any of the different predefined types of medical images as input and the label of the corresponding label (40, 41, 42) type as output, and is generated based on at least a shared first module (10) and a second module (20, 21, 22) of the set of model modules, the second module (20, 21, 22) being specific to the corresponding label (40, 41, 42) type and configured to take the medical image (30, 31, 32) of any of the different predefined types of medical images as input and the label of the corresponding label (40, 41, 42) type as output, the second module (20, 21, 22) being specific to the corresponding label (40, 41, 42) type and is configured to take the medical image (30, 31, 32) of any of the different predefined types of medical images as input and the label of the corresponding label (40, 41, 42) type as output, the The first module takes the representations (13, 14, 15) of the medical images (30, 31, 32) as input and outputs the corresponding tags (40, 41, 42) as tags, wherein the first module includes: a) a shared first module (10) for the different tags (40, 41, 42) types, configured to take any one of the different types of medical images (30, 31, 32) as input and output the representation (13) of the medical images (30, 31, 32); and b) a tag type specific first module (11, 12), wherein the tag type specific first module (11, 12) is specific to the tag (41, 42) type and is configured to take the representation (13) provided by the shared first module (10) as input and output the tag type specific representation (14, 15).

17. A labeling method for labeling medical images, wherein, The marking method includes: - The model module providing unit (2) provides a first module (10, 11, 12) configured to output representations (13, 14, 15) of any of the different types of medical images (30, 31, 32) and a second module (20, 21, 22) for different label (40, 41, 42) types, wherein the corresponding second module (20, 21, 22) is specific to the corresponding label (40, 41, 42) type and is configured to take the representations (13, 14, 15) of the medical images (30, 31, 32) as input and the labels of the corresponding label (40, 41, 42) type as output, wherein the... The first module includes: a) a shared first module (10) for the different label (40, 41, 42) types, configured to take a medical image (30, 31, 32) of any one of the different types of medical images as input and a representation (13) of the medical image (30, 31, 32) as output; and b) a label type specific first module (11, 12), wherein the label type specific first module (11, 12) is specific to the label (41, 42) type and is configured to take the representation (13) provided by the shared first module as input and a label type specific representation (14, 15) as output. - A medical image (30, 31, 32) of one of the different types of medical images is provided by the medical image providing unit (3). - The tag type is provided by the tag type providing unit (4) for one of the different tag (40, 41, 42) types. - The model generation unit (5) generates a model based on at least the shared first module (10) and the provided second module (20, 21, 22), which takes any one of the different types of medical images (30, 31, 32) as input and takes the provided labels (40, 41, 42) as output, and - The provided medical images (30, 31, 32) are labeled by the labeling unit (7) using the generated model.

18. A labeling computer program for labeling medical images, the labeling computer program including program code means, when the labeling computer program is run on a computing system, the program code means being configured to cause the computing system to perform the steps of the labeling method as claimed in claim 17.

Images