Method for generating images for training and / or testing an image segmentation system

Abstract

A computer-implemented method for generating training images and training label maps for training and / or testing an image segmentation system wherein the method comprises the steps of: obtaining a first image and a first label map corresponding to the first image, and obtaining a second image and a second label map corresponding to the second image; determining a third image, where the third image is determined by providing the second image as input to a machine learning system, where the machine learning system is configured for determining an image in the style of the first image based on the provided image; determining a training image by replacing pixels from the first image with pixels from a third image, wherein the pixels from the third image are determined based on classes from the second tag map; determining a training tag map by replacing class tags from the first tag map with class tags indicative of abnormal classes, wherein class tags in the first tag map that are replaced for corresponding pixels in its first image are replaced.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for generating images for training and / or testing an image segmentation system, a computer-implemented method for training an image segmentation system, a computer-implemented method for detecting anomalies in an image, a training system, a control system, a computer program and a machine-readable storage device. Prior art

[0002] Tian et al., "Pixel-wise energy-biased abstention learning for anomaly segmentation on complex urban driving scenes", https: / / arxiv.org / pdf / 2111.12264.pdf, 2022 discloses an anomaly segmentation method called pixel-wise energy-biased abstention learning (PEBAL), which explores pixel-wise abstention learning (AL) using a model that learns adaptive pixel-level anomaly classes and an energy-based model (EBM) that learns the distribution of inlier pixels. PEBAL is based on the joint training of EBM and AL, where EBM is trained to output high energy for abnormal pixels (from outlier exposure), and AL is trained so that these high-energy pixels receive adaptive low penalties for being included in the anomaly class.

[0003] Nayal et al., “RbA: Segmenting Unknown Regions Rejected by All”, https: / / arxiv.org / pdf / 2211.14293.pdf, 2022 discloses an outlier scoring function called RbA by defining an event as an outlier as one rejected by all known classes. Experiments show that mask classification improves the performance of existing outlier detection methods, and the proposed RbA achieves the best results.

[0004] Li et al., “Intra-&Extra-Source Exemplar-Based Style Synthesis for Improved Domain Generalization”, https: / / arxiv.org / pdf / 2307.00648.pdf, 2023 discloses a sample-based style synthesis pipeline to improve domain generalization in semantic segmentation, in particular, a masked noise encoder combined with StyleGAN2. Background Art

[0006] Perceiving the environment and detecting its contents is an important task in different technological applications (e.g., robotics). A typical goal is to detect objects, plants, or other items of interest using sensors such as cameras, for example, without colliding with elements of the environment.

[0007] Especially for safety-critical applications such as autonomous driving (AD), the ability to detect unknown objects (out-of-distribution objects, OoD objects) is crucial. This can also be understood as identifying anomalous objects. More specifically, detecting pixels of unknown or anomalous objects in semantic segmentation scenarios is a core perception task for mobile robots (such as, for example, autonomous vehicles).

[0008] The semantic segmentation task is typically solved by neural networks. Therefore, it is desirable to enable a neural network configured for semantic segmentation to also detect unknown or anomalous pixels, so that in an overall system using a neural network for modeling an environment, unknown or anomalous objects can be accurately identified and separated from the drivable space.

[0009] A core difficulty in OoD detection is the large number of possible OoD objects compared to the relatively small number of object classes on which the neural network is trained. Furthermore, semantic segmentation typically processes context-centric input images, i.e., images in which OoD objects are co-located with many other known objects and environmental elements. This poses an even greater challenge than OoD detection on object-centric data (e.g., image classification benchmarks). Another challenge that comes with context-centric images is class imbalance. For example, road pixels in AD scenarios often constitute a large portion of the training data and bias the neural network towards predicting roads. Due to the imbalance in the training data, neural networks often exhibit a tendency to predict the majority class of OoD objects found on roads, which is a serious error that carries the highest risk level.

[0010] A particularly successful technique for detecting unknown or anomalous objects in semantic segmentation tasks (also known as anomaly segmentation) is a form of outlier exposure. That is, while potential OoD scenarios cannot be accessed at inference time, there are many readily available datasets that are different from the training data that can serve as OoD samples during training. For example, the MS COCO dataset is a popular OoD proxy dataset for training semantic segmentation. For example, it is possible to cut and then paste specific objects included by the MSCOCO dataset into a road scenario to create anomaly examples such as AD scenarios.

[0011] OoD data constructed in this way can be used during training a neural network from scratch or during fine-tuning of a pre-trained model. The inventors found that using OoD data for training from scratch may degrade the in-distribution generalization performance of the neural network (i.e., its semantic segmentation performance on known and / or non-anomalous pixels) and require architectural changes in the neural network. In contrast, the authors found that when OoD data is used only during fine-tuning of a pre-trained neural network, the neural network is typically constrained to small weight changes and is therefore less prone to overfitting.

[0012] Based on these findings, the present invention advantageously proposes a new method for generating OoD data by adapting the style of the OoD data copied and pasted onto an existing image. Style transfer results in training data that is more suitable for training to detect unknown or anomalous objects, because a neural network trained with the data cannot rely on the style of the pasted content (i.e., it looks "out of place" with respect to the rest of the image) to determine whether a pixel is anomalous. The present invention further proposes an advantageous method for training a neural network based on data as generated according to the method for generating OoD data. Summary of the invention

[0013] In a first aspect, the present invention relates to a computer-implemented method for generating training images and training label maps for training and / or testing an image segmentation system, wherein the method comprises the steps of:

[0014] Obtain a first image and a first label map corresponding to the first image, and obtain a second image and a second label map corresponding to the second image;

[0015] determining a third image, wherein the third image is determined by providing the second image as input to a machine learning system, wherein the machine learning system is configured to determine an image in the style of the first image based on the provided image;

[0016] determining a training image by replacing pixels from the first image with pixels from a third image, wherein the pixels from the third image are determined based on the classes from the second label map;

[0017] Determining a training label map by replacing class labels from the first label map with class labels indicative of anomaly classes, wherein class labels in the first label map for which corresponding pixels in the first image are replaced are replaced.

[0018] The method can be understood as enhancing a first image with pixels from a second image, where the second image is first brought into a style similar or identical to the first image. Advantageously, this style alignment between the first image and the second image allows the pixels of the second image to be realistically embedded into the first image. In other words, the pixels from the third image (i.e., the second image after style alignment) look as if they may actually belong to the first image.

[0019] Training an image segmentation system using such generated images increases the diversity of the training data and leads to improved performance of the image segmentation system. Even more advantageously, the copied pixels are labeled as depicting anomalies in a training label map (either by anomaly class or by additional indication in the label map), which training label map can then be used to train an image segmentation system for determining whether a particular pixel in a given image is anomaly.

[0020] Because the pixels from the third image are in the style of the pixels from the first image, the image segmentation system (60) cannot rely on the style of the replaced pixels as an indicator of anomalous pixels, and therefore advantageously must learn to reason about whether pixels in the image are anomalous based on the semantic content of the image (e.g., a non-flying object located in the sky is likely to be an anomaly).

[0021] Label Figure 1 It can generally be understood as an annotation of an image, where the annotation assigns a class label to a subset of pixels of the image corresponding to the label map, preferably all pixels. The correspondence can typically be achieved with the aid of a label map having the same width and height as the image, where each pixel can be assigned a location in the image and can additionally be assigned a class label in the label map (e.g., an integer representing the index of the class), where the class label itself can be assigned a location in the label map. The location can be characterized by an index along the width and height dimensions of the image and the label map, respectively. Therefore, the label map can be given by an integer matrix indicating the class labels.

[0022] The label map corresponding to an image can be understood as a label map including the classification of each pixel of the image.

[0023] If the replaced class label in the first label map is further indicated as resulting from a replacement, this indication can be achieved by, for example, assigning a special class label that indicates both the corresponding class from the second label map and the "anomaly" (e.g., by adding a predefined offset to an integer representing an additional non-anomalous class). Alternatively, an additional binary map can be provided next to the generated training label map, indicating for each pixel of the generated training image whether the pixel represents an anomaly, i.e., whether the pixel has been copied from a third image. The binary map can be provided in the form of a matrix of binary or integer elements.

[0024] Anomalies can be understood as image parts (depicted by image pixels) that constitute rare, uncommon or unreasonable objects. An anomalous pixel can be understood as a pixel belonging to an anomalous object.

[0025] The machine learning system may in particular be understood as being configured such that the semantic content of its input does not change in its output, but the style changes. For example, the machine learning system may be given by a neural network comprising a masked noise encoder and a StyleGAN2, wherein the neural network is further configured (i.e., trained) to change the style of an image provided as input.

[0026] The term “determining an image in the style of a first image based on a provided image” may in particular be understood in relation to a machine learning system being configured to accept an image as input, wherein the machine learning system then transforms the provided input image into an output image, wherein the output image has a different style than the input image, but has similar or identical semantic content.

[0027] An image can be understood as being obtained by synthesis (e.g., rendering from a virtual scene in a computer) or by recording from a sensor, in particular an optical sensor. In this case, an image can be understood as the output of a camera, a lidar, a radar, an ultrasonic sensor or a thermal sensor.

[0028] Acquiring an image may be understood as any way of acquiring an image, for example by means of recording the image using a corresponding sensor or retrieving the image from a database of images.

[0029] The first images can in particular be obtained randomly from a data set of first images. This can be understood as the first images having a similar visual style, for example because they are recorded by the same sensor and / or the same surrounding conditions of the sensor.

[0030] Determining the training image by replacing pixels from the first image with pixels from the third image may be understood as providing an image resulting from replacing pixels from the first image with corresponding pixels of the third image as the training image.

[0031] The pixels in the first image that are replaced by the pixels from the third image may be randomly selected. For example, a random location may be selected in the first image, and the pixels from the third image may be superimposed into the location such that the center of the pixels from the third image is located at the random location. In this case, the pixels to be replaced in the first image may be understood to be implicitly determined. Alternatively, the pixels to be replaced in the first image may be selected based on the classes of the first label map. For example, if the pixels from the third image to be pasted onto the first image constitute pixels of an object, the pixels in the first image may be selected such that they represent pixels that represent a portion of the first label map that may reasonably include the object, for example, free space of the object such as a road, a sidewalk, etc.

[0032] Alternatively, it is also possible to select a location in the first image that characterizes the first label map and that may reasonably include a part of the object, and the pixels from the third image may be copied into the first image such that the center of the pixels from the third image is located at this location.

[0033] In all examples, the center position is selected only as an example. Other positions, such as corners or other desired positions, may also be used.

[0034] Preferably, the pixels to be replaced are determined randomly.The expression is also understood to include methods in which a single position in the first image is randomly chosen and a pixel from the third image is copied into the first image, as described above.

[0035] Preferably, replacing the class label from the first label map is achieved by replacing the class label from the first label map with the class label from the second label map, wherein the class label from the second label map whose corresponding pixel has been used as a replacement in the first image is used for replacement, and wherein the replaced class label in the first label map is further indicated as being obtained from the replacement.

[0036] Alternatively, it is also possible to assign the designated “anomaly” class to the elements of the training label map for which the corresponding pixels in the training images have been determined by copying from the third image.

[0037] Preferably, before determining the training images, the first image is transformed in an enhancement operation.

[0038] For example, the first image may be cropped or resized before being subjected to the replacement of pixels. The enhancement may be any known enhancement operation. The result of the enhancement operation may then be used as the first image for a subsequent operation, for example for determining a training image.

[0039] Advantageously, the authors found that the image segmentation system performed even better when trained with an enhanced first image.

[0040] Preferably, the machine learning system determines its output based on providing its input to the masked noise encoder of the machine learning system and providing the output of the masked noise encoder to StyleGAN2.

[0041] The masked noise encoder and StyleGAN2 in particular can be understood as subnetworks of a combined neural network. The architecture of the masked noise encoder can be the architecture of the pSp encoder. The masked noise encoder can map the provided input image to the expanded latent space of the pre-trained StyleGAN2 generator The pre-trained StyleGAN2 generator can be particularly pre-trained on images including complex scenario data such as urban driving scenes or other similar images.

[0042] The inventors found that, advantageously, the combination of the masked noise encoder and StyleGAN2 results in better style transfer of the style of the first image, and therefore in better first training images, which in turn results in improved image segmentation system performance when the training images are used for training, or an improved assessment of the generalization ability of the image segmentation system when the training images are used during testing of the image segmentation system.

[0043] Alternatively, other style transfer machine learning systems may also be used on top of this approach, for example, machine learning systems based on diffusion models, such as diffusion models configured to interpolate between images.

[0044] Preferably, the first image consists of a first dataset of first images, and the machine learning system is trained to transfer the input image to the style of the first image.

[0045] This may in particular be understood as the method comprising an additional step of training the machine learning system as part of the method for generating the image. Alternatively, this may be understood as obtaining the machine learning system after training on the first image, for example by downloading it from a resource on the Internet.

[0046] Some or all of the preferred embodiments of the method for training as described above may be combined to form further embodiments.

[0047] In another aspect, the present invention relates to a computer-implemented method for training an image segmentation system, wherein the method comprises the steps of:

[0048] Generating training images and corresponding training label maps according to the method of any of the preceding claims;

[0049] Use the generated training images and training label maps to perform supervised training on the image segmentation system.

[0050] The image segmentation system may be configured to perform semantic segmentation and / or instance segmentation of the provided input image. This may in particular be achieved by an image segmentation system comprising or consisting of a neural network configured for image segmentation and / or instance segmentation. Semantic segmentation and / or instance segmentation may in particular be understood as a specific form of image classification.

[0051] The method for training may in particular comprise training the neural network in a supervised manner.The image segmentation system may further comprise a pre-processing step applied to an input image provided to the image segmentation system and / or a post-processing step applied to an output of the image segmentation system.

[0052] The method for training can be run "from scratch", i.e., using an untrained image segmentation system, or as a fine-tuning method, where the image segmentation system is trained on other data before being trained using one or more generated images.

[0053] The method for training can preferably also be run with the generated batch of training images, i.e., the method for generating training images can be run multiple times with different images and / or different replacement pixels in order to determine the batch of training images. Preferably, the generated images and images without any replacement pixels (e.g., the first image) are combined into batches.

[0054] Preferably, supervised training further comprises the following steps:

[0055] providing the training image as input to an image segmentation system, wherein the image segmentation system determines values ​​representing logits for pixels of different classes in the training image;

[0056] Determining a loss value based on a loss function, wherein the loss function includes a first term characterizing a logarithm of a sigmoid of a logit determined for pixels in the training image, wherein the logarithm of the sigmoid is determined for pixels labeled as an anomaly class or indicated as an anomaly in the training label map.

[0057] Supervised training of image segmentation systems by minimizing the loss value.

[0058] The term "minimizing the loss value" can preferably be understood as performing steps such that the loss value is smaller if the same input is provided to the image segmentation system. This can be achieved in particular by means of a gradient descent method, in particular a stochastic gradient descent method. Alternatively, other training methods are also possible, for example, methods based on evolutionary algorithms.

[0059] Advantageously, the inventors have found that using the logit of pixels indicated as "abnormal" in this way can be understood as training the image segmentation system as a one-vs-the-rest (OvR) classifier for each pixel, that is, the system can indicate for each pixel and each "non-abnormal" class how likely the pixel is to belong to that class. If each "non-abnormal" class of the pixel is unlikely (e.g., the logit characterization probability is equal to or lower than 50%), the pixel can be considered abnormal. The inventors have found that this advantageously increases the ability of the image segmentation system to correctly indicate whether a pixel is abnormal.

[0060] Even more advantageously, the authors found that the proposed loss function can be applied only during fine-tuning. This can be understood in particular as the method for training does not require any other ground truth data, and the image segmentation system is fine-tuned only based on the loss function, preferably including only the first term.

[0061] Alternatively, the loss function may also include a second term that characterizes the standard loss for semantic segmentation of image pixels that are not indicated as abnormal. For example, a softmax function may be applied to the logit of non-abnormal pixels, and the second loss term may be characterized or given by a cross entropy loss with respect to the training label map that may be determined for non-abnormal pixels, preferably for all non-abnormal pixels.

[0062] Preferably, the first term characterizes the logarithm of the sigmoid of the maximum logit of a predefined quantity.

[0063] Advantageously, the inventors have found that limiting the amount of logits for which the logarithm of the sigmoid for which the logit is determined further improves the performance of the image segmentation system, ie its ability to correctly indicate whether a pixel is anomalous or not.

[0064] The number of logits for which the logarithm of the sigmoid is determined can be considered a hyperparameter of the method. It can be tuned according to any suitable hyperparameter tuning method. The authors found that small values ​​performed particularly well, especially using three, five, or seven largest logits per pixel. However, other quantities may also be suitable.

[0065] Preferably, the first loss term is characterized by the following formula:

[0066]

[0067] where i, j are the height and width coordinates of the pixel, ood is the pixel coordinate of the pixel indicated as an anomaly, and N ood is the number of pixels in ood, K is the number of logits for which the logarithm of the sigmoid is determined, and S i,j is a set of K logits, and is the kth logit in the set of logits for the pixel at location i, j.

[0068] The first loss term may preferably be multiplied by a factor constituting a further hyperparameter of the method used for training. The loss function from which the loss value is determined may thus be characterized by the following formula

[0069]

[0070] in characterizes the second loss term and λ ood Characterization factor.

[0071] The loss function can be extended with further terms, for example, to regularize the training of an image segmentation system.

[0072] Some or all of the preferred embodiments of the method for training may be combined to form another embodiment of the method for training.

[0073] In another aspect, the invention relates to a computer-implemented method for determining an output signal, wherein the output signal characterizes the classification of pixels of an input image as anomalous or non-anomalous, wherein the method comprises the steps of:

[0074] providing an input image as input to an image segmentation system, wherein the image segmentation system has been trained using a method according to any of the preceding claims;

[0075] The pixel logit is determined by the image segmentation system.

[0076] • If the difference between the determined maximum logit and the determined minimum logit is less than or equal to a predefined threshold, providing the classification of the pixel as an anomaly in the output signal, and otherwise providing the classification of the pixel as a non-anomalous in the output signal.

[0077] The term "wherein the image segmentation system has been trained using a method" can be understood as steps of a method for training being performed as part of a method for determining an output signal. Alternatively, it can also be understood as obtaining an image segmentation system that has been trained as described above, for example from a resource on the Internet.

[0078] Preferably, the image segmentation system determines for a plurality of pixels, preferably for all pixels of the input image, whether the corresponding pixel is abnormal. Even more preferably, the pixels determined to be abnormal can be grouped, for example by means of a connected component algorithm, so as to determine the area in the image that is considered abnormal. For example, the grouped area can also be represented (put out) in the output signal by a bounding box of pixels of a single connected component enclosing the abnormal pixel.

[0079] In order to determine whether the classification of a pixel is abnormal, an abnormality score may be determined for the pixel, wherein the abnormality score is preferably represented by the following formula:

[0080]

[0081] In other words, the logit can be determined for all classes that the pixel may be classified into, and then the difference between the maximum and minimum logit can be used as the anomaly score. Alternatively, it is also possible to multiply the result of the difference by -1 so that a larger anomaly score corresponds to a higher likelihood that the pixel is abnormal. The anomaly score can also be determined for each pixel individually.

[0082] An output signal comprising a classification may in particular be understood as an output signal comprising or consisting of a value characterizing the classification, such as an anomaly score or a binary value indicating whether the anomaly score is less than or equal to a predefined threshold (or equal to or greater than a predefined threshold if the difference is multiplied by -1).

[0083] Some or all of the preferred embodiments of the method for determining an output signal may be combined to form another embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Embodiments of the present invention will be discussed in more detail with reference to the following figures. The figures show:

[0085] Figure 1 is a method for generating training images and training label maps;

[0086] Figure 2 is a training system configured to perform a method for training an image segmentation system;

[0087] Figure 3 are control systems, including machine learning systems that control actuators in their environment;

[0088] Figure 4 is a control system for controlling an at least partially autonomous robot;

[0089] Figure 5 It is the control system that controls the manufacturing machines;

[0090] Figure 6 It is a control system that controls the medical analysis system. DETAILED DESCRIPTION

[0091] Figure 1 A method for generating training images and training label maps is schematically shown.

[0092] The method comprises obtaining a first image (i1) and a first label map (l1) corresponding to the first image (i1), and obtaining a second image (i2) and a second label map (l2) corresponding to the second image (i2). The first image (i1) and / or the second image (i2) may be obtained from a corresponding image database, or may be recorded by a sensor, in particular an optical sensor such as a camera. Recording may also be part of the method. Preferably, the first image (i1) and the second image (i2) have different styles, for example, the color distribution of the pixels is significantly different. In this embodiment, the first image (i1) depicts a road scene, for example an urban traffic scene, and the second image (i2) includes a bear, which is typically not seen in an urban traffic scene. The corresponding label maps (l1, l2) may be obtained by labeling the corresponding images (i1, i2) by humans or by an automatic algorithm, i.e., may be automatically labeled.

[0093] The method further includes determining a third image (i3), wherein the third image is determined by providing the second image (i2) as input to a machine learning system (70), wherein the machine learning system (70) is configured to determine an image in the style of the first image (i1) based on the provided image.

[0094] The term “determining an image in the style of a first image based on a provided image” may in particular be understood in relation to a machine learning system being configured to accept an image as input, wherein the machine learning system then transforms the provided input image into an output image, wherein the output image has a different style than the input image, but has similar or identical semantic content.

[0095] In particular, the machine learning system takes as input a second image (i2, in this embodiment an image of a bear) and provides a third image (i3, an image of a bear having a color distribution similar to the urban traffic scene in the first image).

[0096] The machine learning system may in particular be configured to consist of a neural network comprising a masked noise encoder and a StyleGAN2 for determining the third image (i3).

[0097] Based on the second label map (l2), objects from the second image (i2) can then be cut from the third image (i3) in a cutting operation (c). Preferably, the cutting operation selects objects from the second label map (l2) based on connected class labels of the object class (i.e., connected components of multiple specific class labels from the same object class).

[0098] The first image (i1) and the first label map (l1) may be subjected to optional enhancement operations (80), such as cropping, resizing, rotating or distorting, etc. The result may then be used again as the first image (i1) and the first label map (l1).

[0099] Whether or not the augmentation operation is performed, the method further comprises determining a training image (x) by replacing pixels from the first image (i1) with pixels from the third image (i3). i ). In other words, the pixels obtained from the cutting operation can then be copied (ie, overlaid) on the pixels from the first image (i1) in order to determine the training image (x i ).

[0100] The pixels in the first image (i1) to be replaced by the pixels from the third image (i3) may be randomly selected. For example, a random location may be selected in the first image (i1), and the pixels from the third image may be superimposed on the location so that the center of the pixel from the third image (i3) is located at the random location. In this case, the pixels to be replaced in the first image (i1) may be understood to be implicitly determined. Alternatively, the pixels to be replaced in the first image (i1) may be selected based on the classes of the first label map (l1). For example, the pixels from the third image (i3) to be pasted onto the first image (i1) constitute the pixels of an object, and the pixels in the first image (i1) may be selected so that they represent pixels representing a portion of the first label map (l1) that may reasonably include an object, for example, the free space of the object such as a road, a sidewalk, etc.

[0101] The method further comprises determining a training label map (t1) by replacing the class labels from the first label map (l1) with class labels indicating abnormal classes. i ), where the class labels of the first label map (l1) for the corresponding pixels in the first image (i1) are replaced. In other words, for the pixels in the training image (x i ) and the pixels obtained by copying the training label map (t i ) is determined to include the class labels of the anomaly class.

[0102] Figure 2 An embodiment of a training system (140) for training an image segmentation system (60) with the aid of a training data set (T) is shown. The training data set (T) comprises a plurality of training images (x i ) and training label map (t i ), where each training image (x i ) and training label map (t i ) has passed Figure 1 The training image (xi ) and training labels (l i ) For example, by using a different first image (i1) and / or a different second image (i2) to show different content. In a method for generating and / or using different positions, an object cut out from a third image (i3) is copied into the first image (i1).

[0103] For training, the training data unit (150) accesses a computer-implemented database (St2) which provides a training data set (T). The training data unit (150) preferably randomly determines at least one training image (x i ) and corresponding to the training image (x i )’s training label map (t i ), and the training image (x i ) is transmitted to the image segmentation system (60). The image segmentation system (60) is based on the training image (x i ) determines the output signal (y i ), where the output signal (y i ) represents the training image (x i ), preferably including logits for different classes for each pixel.

[0104] Training label map (t i ) and the determined output signal (y i ) is transmitted to the modification unit (180).

[0105] Based on the training label map (t i ) and the determined output signal (y i ), the modification unit (180) then determines new parameters (Φ′) of the image segmentation system (60), for example new parameters of a neural network included in the image segmentation system (60). For this purpose, the modification unit (180) compares the training label map (t i ) and the determined output signal (y i ), wherein the loss function includes a first term representing the logarithm of the sigmoid of the logit determined for the pixels in the training image, wherein the logarithm of the sigmoid is determined for the pixels marked as anomaly classes or indicated as anomalies in the training label map. The loss function determines a loss value as an output. Preferably, the first loss term is represented by the following formula:

[0106]

[0107] where i, j are the height and width coordinates of the pixel, ood is the pixel coordinate of the pixel indicated as an anomaly, and N oodis the number of pixels in ood, K is the number of logits for which the logarithm of the sigmoid is determined, and S i,j is a set of K logits, and is the kth logit in the set of logits for the pixel at location i, j. Preferably, the first term can be understood as the only term in the loss function, that is, the loss function is the first term. Alternatively, Can be combined with further terms to form a loss function. For example, the loss function can include a second term that characterizes a standard loss for semantic segmentation of image pixels that are not indicated as abnormal. For example, a softmax function can be applied to the logit of non-anomalous pixels, and the second loss term can be characterized or given by a cross entropy loss with respect to the training label map that can be determined for non-anomalous pixels, preferably for all non-anomalous pixels.

[0108] The first loss term may preferably be multiplied by a factor constituting a further hyperparameter of the method used for training. Thus, the loss function according to which the loss value is determined may be characterized by the following formula

[0109]

[0110] in Characterize the second loss term and and λ ood Characterization factor.

[0111] The loss function can be extended with further terms, for example, to regularize the training of an image segmentation system.

[0112] The modification unit (180) determines new parameters (Φ') based on the loss value. In the given embodiment, this is done using a gradient descent method, preferably stochastic gradient descent, Adam or Adam W. In further embodiments, the training can also be based on evolutionary algorithms or second order methods for training neural networks.

[0113] In other preferred embodiments, the described training is iteratively repeated for a predefined number of iteration steps, or iteratively repeated until the loss value falls below a predefined threshold. Alternatively or additionally, it is also possible that the training is terminated when the average first loss value with respect to the test or validation data set is below the predefined threshold. In at least one of the iterations, the new parameters (Φ′) determined in the previous iteration are used as parameters (Φ) of the image segmentation system (60).

[0114] Furthermore, the training system (140) may include at least one processor (145) and at least one machine-readable storage medium (146) containing instructions which, when executed by the processor (145), cause the training system (140) to perform a training method according to one of the aspects of the present invention.

[0115] In a further embodiment (not shown), the training image (x i ) and training label map (t i ) can be generated in each iteration of training instead of being obtained from the database. In other words, the training images (x i ) and training label map (t i ) can be generated on the fly.

[0116] Figure 3 An embodiment of a control system (40) for controlling an actuator (10) in its environment (20) based on the output of an image segmentation system (60) is shown. The actuator (10) and its environment (20) will be collectively referred to as the actuator system. At preferably evenly spaced points in time, a sensor (30) senses a condition of the actuator system. The sensor (30) may include several sensors. Preferably, the sensor (30) is an optical sensor that captures an image of the environment (20). An output signal (S) of the sensor (30) encoding the sensed condition (or, where the sensor (30) includes multiple sensors, an output signal (S) of each of the sensors) is transmitted to the control system (40).

[0117] Thus, the control system (40) receives a stream of sensor signals (S). It then calculates a series of control signals (A) depending on the stream of sensor signals (S), and the series of control signals (A) are then transmitted to the actuator (10).

[0118] The control system (40) receives a stream of sensor signals (S) from the sensor (30) in an optional receiving unit (50). The receiving unit (50) transforms the sensor signals (S) into an input image (x). Alternatively, in the absence of a receiving unit (50), each sensor signal (S) can be taken directly as an input image (x). The input image (x) can be given, for example, as an excerpt from the sensor signal. Alternatively, the sensor signal (S) can be processed to generate the input image (x). In other words, the input signal (x) is provided as a function of the sensor signal (S).

[0119] The input image (x) is then passed to the image segmentation system (60).

[0120] The image segmentation system (60) is parameterized by parameters (Φ) which are stored in and provided by a parameter memory (St1).

[0121] The image segmentation system (60) determines an output signal (y) from an input image (x). The output signal (y) is a classification of pixels of the input image (x), preferably a corresponding classification for each pixel, wherein the classification characterizes whether the pixel is abnormal. The output signal (y) is transmitted to an optional conversion unit (80), which converts the output signal (y) an abnormality score and determines a control signal (A) based on the classification. Alternatively, the output signal (y) can be directly used as the control signal (A).

[0122] The classification is preferably determined by the image segmentation system (60) by first determining an anomaly score for each pixel to be classified. The anomaly score may preferably be determined according to the following formula

[0123]

[0124] in is the kth logit λ determined for the pixel of the input image (x) at location i, j in the input image (x).

[0125] The anomaly score can then be compared to a predefined threshold, and if the anomaly score is equal to or lower than the predefined threshold, the pixel can be classified as an anomaly. Alternatively, the anomaly score can be multiplied by -1, and if the result of the multiplication is equal to or higher than the predefined threshold, the pixel can be classified as an anomaly. The threshold can be understood as a hyperparameter of the method used for training.

[0126] If the pixel is not classified as an anomaly, it may preferably be assigned a class indicating a normal pixel. This may be achieved, for example, by assigning the pixel to the class corresponding to the highest logit determined by the image segmentation system (60).

[0127] The control signal (A) is then transmitted to the actuator (10) for controlling the actuator (10) accordingly.

[0128] The actuator (10) receives the control signal (A), is controlled accordingly, and performs an action corresponding to the control signal (A). The actuator (10) may include control logic that transforms the control signal (A) into a further control signal, which is then used to control the actuator (10).

[0129] In a further embodiment, the control system (40) may include a sensor (30). In still further embodiments, the control system (40) may alternatively or additionally include an actuator (10).

[0130] In yet further embodiments, it is contemplated that the control system (40) controls the display (10a) instead of or in addition to the actuator (10).

[0131] Still further, the control system (40) may include at least one processor (45) and at least one machine-readable storage medium (46) storing instructions thereon which, if executed, cause the control system (40) to perform a method according to aspects of the present invention.

[0132] Figure 4 An embodiment is shown in which a control system (40) is used to control an at least partially autonomous robot, such as an at least partially autonomous vehicle (100).

[0133] The sensors (30) may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more lidar sensors. Some or all of these sensors are preferably, but not necessarily, integrated into the at least partially autonomous robot (100).

[0134] The image segmentation system (60) can be configured to perform semantic segmentation or instance segmentation of the vicinity of the at least partially autonomous robot (100) based on the input image (x). In the event that an abnormal region (i.e., a pixel or connected component of pixels classified as abnormal) is detected in the input image (x), a control signal (A) can be determined based on this information, for example to avoid the path of the at least partially autonomous robot (100) passing through the abnormal region, or to avoid the path of the at least partially autonomous robot (100) being closer to the abnormal region than a predefined threshold.

[0135] The actuator (10), which is preferably integrated in the vehicle (100), may be provided by the brakes, propulsion system, engine, transmission or steering of the vehicle (100).

[0136] Alternatively or additionally, the control signal (A) may also be used to control the display (10a), for example to display abnormal pixels detected by the image segmentation system (60). It is also possible that the control signal (A) may control the display (10a) so that it generates a warning signal if the vehicle (100) approaches an area comprising abnormal pixels. The warning signal may be a warning sound and / or a tactile signal, such as vibration of the vehicle's steering wheel.

[0137] In a further embodiment, the at least partially autonomous robot may be provided by another mobile robot (not shown), which may move, for example, by flying, swimming, diving or walking. The mobile robot may in particular be an at least partially autonomous lawnmower, or an at least partially autonomous cleaning robot. In all of the above embodiments, the control signal (A) may be determined such that a propulsion unit and / or a steering and / or a braking of the mobile robot is controlled such that the mobile robot may avoid a collision with the identified object.

[0138] Figure 5 An embodiment is shown in which a control system (40) is used to control a manufacturing machine (11) of a manufacturing system (200), such as a punch, a cutter, a gun drill or a fixture, for example as part of a production line. The manufacturing machine may include a transport device (13), such as a conveyor belt or an assembly line, which moves the manufactured products (12). The control system (40) controls an actuator (10), which in turn controls the manufacturing machine (11).

[0139] The sensor (30) may be given by an optical sensor that captures, for example, properties of the manufactured product (12).

[0140] The image segmentation system (60) can determine a portion of the input image (x) of the device that includes the manufactured product (12). The actuator (10) can then be controlled depending on the positioning of the manufactured product (12) for subsequent manufacturing steps of the manufactured product (12). For example, the actuator (10) can be controlled to cut the manufactured product at a specific location on the manufactured product itself. If the output signal (y) determines that there are abnormal pixels on the transport device (13) and / or the manufactured product (12), the operation of the machine can be stopped and / or a human operator can be called to evaluate the problem. Alternatively, if an abnormality is detected on a pixel belonging to the manufactured product, the actuator (10) can be controlled to remove the manufactured product from the transport device.

[0141] Figure 6 An embodiment of a medical imaging system (500) controlled by a control system (40) is shown. The imaging system may be, for example, an MRI device, an x-ray imaging device or an ultrasound imaging device. The sensor (30) may be, for example, an imaging sensor that captures at least one image of a patient, for example showing different types of body tissue of the patient.

[0142] The image segmentation system (60) may be configured to detect different types of tissue in the sensed image, for example by classifying the tissue displayed in the image as malignant or benign tissue. This may be accomplished by semantic segmentation of the input image (x) performed by the image segmentation system (60). The control signal (A) may then be determined to cause the display (10a) to display different tissues, for example, by displaying the input image (x) and coloring different areas of the same tissue type with the same color. The control signal (A) may further be selected to color abnormal areas with different colors in order to direct the doctor's attention to the abnormal areas and examine them in detail.

[0143] In further embodiments (not shown), the imaging system (500) may be used for non-medical purposes, for example, to determine material properties of a workpiece. In these embodiments, the image segmentation system (60) may be configured to receive an input image (x) of at least a portion of the workpiece and perform semantic segmentation of the input image (x) to thereby classify the material properties of the workpiece. A control signal (A) may then be determined to cause the display (10a) to display the input image (x) and information about the detected material properties.

[0144] The term "computer" may be understood to cover any device for processing predefined computing rules. These computing rules may be in the form of software, hardware or a combination of software and hardware.

[0145] In general, a plurality may be understood to be indexed, i.e. each element in the plurality is assigned a unique index, preferably by assigning consecutive integers to the elements contained in the plurality. Preferably, if the plurality comprises N elements, where N is the number of elements in the plurality, the elements are assigned integers from 1 to N. It may also be understood that the elements of the plurality may be accessed by their index.

Claims

1. For generating a training image (x) for training and / or testing an image segmentation system (60) i ) and training label map (t i ), wherein the method comprises the following steps: Obtain a first image (i1) and a first label map (l1) corresponding to the first image (i1), and obtain a second image (i2) and a second label map (l2) corresponding to the second image (i2); determining a third image (i3), wherein the third image is determined by providing the second image (i2) as input to a machine learning system (70), wherein the machine learning system (70) is configured to determine an image in the style of the first image (i1) based on the provided image; The training image (x) is determined by replacing pixels from the first image (i1) with pixels from the third image (i3) i ), wherein pixels from the third image (i3) are determined based on classes from the second label map (l2); Determine the training label map (t1) by replacing the class labels from the first label map (l1) with the class labels indicating the abnormal class i ), where the class labels in the first label map (l1) are replaced for the corresponding pixels in the first image (i1).

2. A method according to claim 1, wherein replacing the class label from the first label map is achieved by replacing the class label from the first label map with the class label from the second label map, wherein the class label from the second label map whose corresponding pixel has been used as a replacement in the first image is used for replacement, and wherein the replaced class label in the first label map is further indicated as being obtained from the replacement.

3. A method according to any of the preceding claims, wherein before determining the training images, the first image is transformed in an enhancement operation.

4. A method according to any of the preceding claims, wherein the machine learning system determines its output based on providing its input to a masked noise encoder of the machine learning system and providing the output of the masked noise encoder to StyleGAN2.

5. A method according to any one of the preceding claims, wherein the first image is constituted by a first dataset of first images, and the machine learning system is trained to transfer the input image to the style of the first image.

6. A computer-implemented method for training an image segmentation system (60), wherein the method comprises the steps of: A method according to any one of the preceding claims, generating a training image (x i ) and the corresponding training label map (t i ); Using the generated training images (x i ) and training label map (t i ) performs supervised training on the image segmentation system (60).

7. The method according to claim 6, wherein the supervised training further comprises the steps of: ·The training image (x i ) is provided as input to an image segmentation system (60), wherein the image segmentation system (60) is trained on a training image (x i )’s different classes of pixels determine the value representing the logit; Determine a loss value based on a loss function, wherein the loss function includes a representation of the training image (x i ) is the first term of the logarithm of the sigmoid of the logit determined by the pixels in the training label map (t i ) is marked as anomaly class or indicated as anomaly to determine the logarithm of the sigmoid. • Performing supervised training on the image segmentation system (60) by minimizing the loss value.

8. The method of claim 7, wherein the first term characterizes the logarithm of the sigmoid of the maximum logit of a predefined quantity.

9. The method according to claim 7 or 8, wherein the first loss term is characterized by the following formula: where i, j are the height and width coordinates of the pixel, ood is the pixel coordinate of the pixel indicated as an anomaly, and N ood is the number of pixels in ood, K is the number of logits for which the logarithm of the sigmoid is determined, and S i,j is a set of K logits, and is the kth logit in the set of logits for the pixel at location i, j.

10. A computer-implemented method for determining an output signal (y), wherein the output signal (y) characterizes the classification of a pixel of an input image (x) as anomalous or non-anomalous, wherein the method comprises the steps of: providing an input image (x) as input to an image segmentation system (60), wherein the image segmentation system (60) has been trained using a method according to any of the preceding claims; Determine the logit of the pixel by the image segmentation system (60); • If the difference between the determined maximum logit and the determined minimum logit is less than or equal to a predefined threshold, providing the classification of the pixel as anomaly in the output signal (y), and otherwise providing the classification of the pixel as non-anomalous in the output signal (y).

11. A training system (140) configured to carry out the training method according to any one of claims 6 to 9.

12. A control system (40) configured to implement the method according to claim 10, wherein the control system (40) determines a control signal (A) based on the output signal (y), wherein the control signal (A) is configured to control an actuator (10) and / or a display (10a).

13. Computer program configured to cause a computer to carry out the method according to any one of claims 1 to 10 and all its steps, if said computer program is executed by a processor (45, 145).

14. A machine-readable storage medium (46, 146) having stored thereon a computer program according to claim 13.

Images