Apparatus and method for training image segmentation system

Abstract

A computer-implemented method for training an image segmentation system (60), the method comprising the steps of: obtaining a training image (xi) and a training label map (ti), where the training label map (ti) indicates pixels of the training image (xi) belonging to an anomaly category; providing the training image (xi) as input to an image segmentation system (60), wherein the image segmentation system (60) determines values of logits characterizing different classes of pixels of the training image (xi); determining a loss value on the basis of a loss function, where the loss function comprises a first term characterizing the logarithm of sigmoid of the logit determined for the pixels in the training image (xi), where the logarithm of sigmoid is determined for the pixels marked as belonging to the anomaly category in the training label map (ti); training an image segmentation system (60) by minimizing the loss value.

Description

Technical Field

[0001] The invention relates to a computer-implemented method for training an image segmentation system, a computer-implemented method for detecting anomalies in an image, a training system, a control system, a computer program and a machine-readable storage device. Prior art

[0002] Tian et al., "Pixel-wise energy-biased abstention learning for anomaly segmentation on complex urban driving scenes", https: / / arxiv.org / pdf / 2111.12264.pdf, 2022 discloses an anomaly segmentation method called pixel-wise energy-biased abstention learning (PEBAL), which explores pixel-wise abstention learning (AL) with a model that learns adaptive pixel-level anomaly categories, and an energy-based model (EBM) that learns the distribution of inlier pixels. PEBAL is based on the joint training of EBM and AL, where EBM is trained to output high energy for abnormal pixels (from outlier exposure), and AL is trained so that these high-energy pixels receive adaptive low penalties for being included in the anomaly category.

[0003] Nayal et al., “RbA: Segmenting Unknown Regions Rejected by All”, https: / / arxiv.org / pdf / 2211.14293.pdf, 2022 discloses an outlier scoring function called RbA by defining an event as an outlier as one rejected by all known classes. Experiments show that mask classification improves the performance of existing outlier detection methods, and the best results are achieved using the proposed RbA.

[0004] Li et al., “Intra-&Extra-Source Exemplar-Based Style Synthesis for Improved Domain Generalization”, https: / / arxiv.org / pdf / 2307.00648.pdf, 2023 discloses an exemplar-based style synthesis pipeline to improve domain generalization in semantic segmentation, specifically a masked noise encoder combined with StyleGAN2. Background Art

[0005] Perceiving an environment and detecting its contents are important tasks in different technical applications, such as robotics. A typical goal is to detect objects, plants or other items of interest using sensors such as cameras, for example in order not to collide with elements of the environment.

[0006] Especially for safety-critical applications such as autonomous driving (AD), the ability to detect unknown objects (out-of-distribution objects, OoD objects) is crucial. This can also be understood as identifying anomalous objects. More specifically, detecting pixels of unknown or anomalous objects in a semantically segmented scene is a core perception task for mobile robots (such as, for example, autonomous vehicles).

[0007] The semantic segmentation task is usually solved by a neural network. Therefore, it is desirable to enable a neural network configured for semantic segmentation to also detect unknown or anomalous pixels so that, across the entire system, using the neural network to model the environment can accurately identify unknown or anomalous objects and separate these objects from the drivable space.

[0008] The core difficulty of OoD detection is the large number of possible OoD objects compared to the relatively small number of object categories on which the neural network is trained. In addition, semantic segmentation typically processes scene-centric input images, i.e., images in which OoD objects are co-located with many other known objects and elements of the environment. This poses an even greater challenge than OoD detection performed on object-centric data (e.g., image classification benchmarks). Another challenge posed by scene-centric images is class imbalance. For example, road pixels in AD scenes typically constitute the majority of the training data and bias the neural network towards predicting roads. Due to the imbalance in the training data, neural networks typically exhibit a tendency to predict the majority class for OoD objects found on roads, which is a serious error with the highest risk level.

[0009] A particularly successful technique for detecting unknown or anomalous objects in semantic segmentation tasks (also known as anomaly segmentation) is a form of outlier exposure. That is, while there is no access to potential OoD scenarios at inference time, many ready-to-us datasets that are different from the training data can serve as OoD examples during training. For example, the MS COCO dataset is a popular OoD proxy dataset for training semantic segmentation. For example, it is possible to cut a specific object composed of the MS COCO dataset and paste it into a road scene to create anomaly examples for AD scenarios, for example.

[0010] OoD data constructed in this way can be used during training a neural network from scratch, or can be used during fine-tuning of a pre-trained model. The inventors found that using OoD data for training from scratch can degrade the in-distribution generalization performance of the neural network (i.e., its semantic segmentation performance on known and / or non-anomalous pixels) and require architectural changes in the neural network. In contrast, the authors found that when OoD data is used only during fine-tuning of a pre-trained neural network, the neural network is typically under the constraint of small weight changes and is therefore less prone to overfitting.

[0011] Based on these findings, the present invention advantageously proposes a new method for generating OoD data by taking a copy of the OoD data in one style and pasting it on an existing image. Style transfer results in training data that is more suitable for training to detect unknown or anomalous objects, because a neural network trained using such data cannot rely on the style of the pasted content (i.e., it appears "off" relative to the rest of the image) to determine whether a pixel is anomalous. The present invention further proposes an advantageous method for training a neural network based on data generated by the method for generating OoD data. Summary of the invention

[0012] In a first aspect, the present invention relates to a computer-implemented method for training an image segmentation system, the method comprising the following steps: obtaining a training image and a training label map, wherein the training label map indicates pixels of the training image that belong to an abnormal class; providing the training image as input to an image segmentation system, wherein the image segmentation system determines values ​​of logits of different classes representing pixels of the training image; Determining a loss value based on a loss function, wherein the loss function includes a first term that represents a logarithm of a sigmoid of a logit determined for pixels in the training image, wherein the logarithm of the sigmoid is determined for pixels labeled as belonging to an anomaly class in the training label map.

[0013] Train the image segmentation system by minimizing the loss value.

[0014] The image segmentation system may be configured to perform semantic segmentation and / or instance segmentation on the provided input image. This may be achieved in particular by an image segmentation system comprising or consisting of a neural network configured for image segmentation and / or instance segmentation. Semantic segmentation and / or instance segmentation may be understood in particular as a specific form of image classification.

[0015] The method for training may in particular comprise training the neural network in a supervised manner.The image segmentation system may further comprise a pre-processing step applied to an input image provided to the image segmentation system and / or a post-processing step applied to an output of the image segmentation system.

[0016] The method used for training can be run "from scratch", i.e., using an untrained image segmentation system, or can be run as a fine-tuning method, in which the image segmentation system is trained on other data before training it using one or more generated images.

[0017] The method for training can also preferably be run with a batch of generated training images, i.e., the method for generating training images can be run multiple times with different images and / or different replacement pixels to determine a batch of training images. Preferably, the generated images and images without any replacement pixels (e.g., the first image) are combined in a batch.

[0018] An image can be understood as being obtained by synthesis (e.g., from a virtual scene rendered in a computer) or by recording from a sensor, in particular an optical sensor. In this case, an image can be understood as the output of a camera, a lidar, a radar, an ultrasonic sensor or a thermal sensor.

[0019] Acquiring an image may be understood as any way of acquiring an image, for example by means of recording the image with a corresponding sensor or retrieving the image from an image database.

[0020] The label map can generally be understood as an annotation of an image, where the annotation assigns a class label to a subset of pixels of the image to which the label map corresponds, preferably all pixels. The correspondence can generally be achieved with the aid of a label map having the same width and height as the image, where each pixel can be assigned a position in the image and can additionally be assigned a class label in the label map (e.g., an integer representing the index of the class), where the class label itself can be assigned a position in the label map. The position can be represented by an index along the width and height dimensions of the image and the label map, respectively. Therefore, the label map can be given by a matrix of integers indicating the class labels.

[0021] A label map corresponding to an image may be understood as a label map including a classification for each pixel of the image.

[0022] The term "minimizing the loss value" may preferably be understood as performing steps so that the loss value is smaller when the same input is provided to the image segmentation system. This can be achieved in particular by means of a gradient descent method, in particular a stochastic gradient descent method. Alternatively, other training methods are also possible, for example, methods based on evolutionary algorithms.

[0023] Advantageously, the inventors have found that using the logit of pixels indicated as "abnormal" can be understood as training the image segmentation system as a one vs. the rest (OvR) classifier for each pixel, that is, the system can indicate for each pixel or for each "non-abnormal" category how likely the pixel belongs to that category. If each "non-abnormal" category for a pixel is unlikely (for example, the logit characterization is equal to or less than 50% probability), the pixel can be considered abnormal. The inventors have found that this advantageously increases the ability of the image segmentation system to correctly indicate whether a pixel is abnormal.

[0024] Even more advantageously, the authors found that the proposed loss function can be applied only during fine-tuning of the image segmentation system. This can be understood in particular as the method for training does not require any other ground truth data and only fine-tunes the image segmentation system based on the loss function (preferably including only the first term).

[0025] Alternatively, the loss function may also include a second term that characterizes a standard loss for semantic segmentation of pixels of the image that are not indicated as abnormal. For example, a softmax function may be applied to the logit of non-abnormal pixels, and the second loss term may be characterized or given by a cross entropy relative to the training label map, which may be determined for non-abnormal pixels, preferably for all non-abnormal pixels.

[0026] Preferably, the first term characterizes the logarithm of the sigmoid of the maximum logit of the predetermined quantity.

[0027] Advantageously, the inventors have found that limiting the amount of logits for which the logarithm of the sigmoid of the logits is determined further increases the performance of the image segmentation system, ie its ability to correctly indicate whether a pixel is anomalous or not.

[0028] The amount of logits for which the logarithm of the sigmoid is determined can be considered a hyperparameter of the method. It can be tuned according to any suitable hyperparameter tuning method. The authors found that small values ​​performed particularly well, especially when using three, five, or seven maximum logits per pixel. However, other amounts may also be suitable.

[0029] Preferably, the first loss term is characterized by the following formula: where i,j are the height and width coordinates of the pixel, ood is the pixel coordinate of the pixel indicated as an anomaly, and N oodis the number of pixels in ood, K is the number of logits for which the logarithm of the sigmoid is determined, and S i,j is a set of K logits, and is the kth logit in the logit set for the pixel at position i,j.

[0030] The first loss term may preferably be multiplied by a factor that constitutes a further hyperparameter of the method used for training. Thus, the loss function from which the loss value is determined may be characterized by the following formula: in characterizes the second loss term, and λ ood Characterize this factor.

[0031] The loss function can be extended with further terms, for example to regularize the training of image segmentation systems.

[0032] Preferably, the training image (x i ) and training label mapping (t i ) is generated by generating training images (x i ) and training label mapping (t i ) to obtain, where the training image (x i ) and training label mapping (t i ) comprises the following steps: Obtaining a first image (i1) and a first label map (l1) corresponding to the first image (i1) and obtaining a second image (i2) and a second label map (l2) corresponding to the second image (i2); determining a third image (i3), wherein the third image is determined by providing the second image (i2) as input to a machine learning system (70), wherein the machine learning system (70) is configured to determine an image in the style of the first image (i1) based on the provided image; The training image (x) is determined by replacing pixels from the first image (i1) with pixels from the third image (i3) i ), wherein pixels from the third image (i3) are determined based on categories from the second label map (l2); Determine a training label map (t1) by replacing the class labels from the first label map (l1) with class labels indicating the abnormal class i ), where the class label in the first label map (l1) is replaced, and for the class label, the corresponding pixel in the first image (i1) is replaced.

[0033] This can be understood as enhancing the first image with pixels from the second image, where the second image is first brought into a similar or identical style as the first image. Advantageously, this style alignment between the first and second images allows the pixels of the second image to be realistically embedded in the first image. In other words, the pixels from the third image (i.e., the second image after style alignment) appear as if they actually belong to the first image.

[0034] Utilizing such generated images to train an image segmentation system increases the diversity of the training data and results in increased performance of the image segmentation system. Even more advantageously, the replicated pixels are labeled in a training label map as depicting anomalies (either by anomaly class, or by additional indication in the label map), which training label map can then be used to train an image segmentation system for determining whether a particular pixel in a given image is anomaly.

[0035] Because the pixels from the third image are in the style of the pixels from the first image, the image segmentation system cannot rely on the style of the replaced pixels as an indicator of anomalous pixels, and therefore, advantageously has to learn to reason about whether pixels in the image are anomalous based on the semantic content of the image (e.g., a non-flying object located in the sky may be an anomaly).

[0036] If the replaced class label in the first label map is further indicated as being caused by a replacement, this indication can be achieved by, for example, assigning a special class label that indicates both the corresponding class from the second label map and "anomaly" (e.g., by adding a predefined offset to an integer representing an additional non-anomalous class). Alternatively, an additional binary map can be provided together with the generated training label map, which indicates for each pixel of the generated training image whether the pixel represents an anomaly, i.e., whether the pixel has been copied from the third image. The binary map can be provided in the form of a matrix of binary or integer elements.

[0037] Anomalies may be understood as those parts of an image (depicted by pixels of the image) that constitute rare, unusual or unreasonable objects. Abnormal pixels may be understood as pixels belonging to abnormal objects.

[0038] The machine learning system may be particularly understood as being configured such that the semantic content of its input is not changed in its output, but the style is changed. For example, the machine learning system may be given by a neural network comprising a masked noise encoder and StyleGAN2, wherein the neural network is further configured (i.e., trained) for changing the style of an image provided as input.

[0039] The term “determining an image in the style of the first image based on the provided image” may in particular be understood as follows: the machine learning system is configured to accept an image as input, wherein the machine learning system then converts the provided input image into an output image, wherein the output image has a different style than the input image, but has similar or identical semantic content.

[0040] The first images can in particular be obtained randomly from a data set of first images. This can be understood as the first images having a similar visual style due to being recorded, for example, by the same sensor and / or the same environmental conditions of the sensor.

[0041] Determining the training image by replacing pixels from the first image with pixels from the third image may be understood as providing an image obtained by replacing corresponding pixels from the first image with pixels from the third image as the training image.

[0042] The pixels in the first image that are replaced by the pixels from the third image can be selected randomly. For example, a random position can be selected in the first image, and the pixels from the third image can be superimposed on this position so that the center of the pixels from the third image is located at the random position. In this case, the pixels to be replaced in the first image can be understood as being implicitly determined. Alternatively, the pixels to be replaced in the first image can be selected based on the categories of the first label mapping. For example, if the pixels from the third image to be pasted on the first image constitute the pixels of an object, the pixels in the first image can be selected so that they represent pixels that represent a portion of the first label mapping that can reasonably include an object (e.g., free space, such as a road, sidewalk, etc.).

[0043] Alternatively, it is also possible to select a position in the first image which characterizes a part of the first label map which may reasonably include an object, and pixels from the third image may be copied into the first image such that the center of the pixels from the third image is located at this position.

[0044] In all examples, the center position is selected for exemplary purposes only. Other positions, such as corners or other desired positions, may also be used.

[0045] Preferably, the pixels to be replaced are determined randomly.This expression is also understood to include methods in which a single position in the first image is randomly selected and a pixel from the third image is copied into the first image, as described above.

[0046] Preferably, replacing a category label from a first label map is achieved by replacing the category label from the first label map with a category label from a second label map, wherein the category label from the second label map is used for replacement, for which the corresponding pixel has been used as a replacement in the first image, wherein the replaced category label in the first label map is further indicated as being caused by the replacement.

[0047] Alternatively, it is also possible to assign a specified “anomaly” class to the element of the training label map for which the corresponding pixel in the training image has been determined by copying from the third image.

[0048] Preferably, the first image is transformed in an enhancement operation prior to determining the training images.

[0049] For example, the first image may be cropped or resized before it undergoes pixel replacement. The enhancement may be any known enhancement operation. The result of the enhancement operation may then be used as the first image for subsequent operations, such as for determining a training image.

[0050] Advantageously, the authors found that the image segmentation system performed even better when trained with the augmented first image.

[0051] Preferably, the machine learning system determines its output based on providing its input to a masked noise encoder of the machine learning system and providing the output of the masked noise encoder to StyleGAN2.

[0052] The masked noise encoder and StyleGAN2 can be specifically understood as subnetworks of a combined neural network. The architecture of the masked noise encoder can be the architecture of the pSp encoder. The masked noise encoder can map the provided input image to the expanded latent space of the pre-trained StyleGAN2 generator. The pre-trained StyleGAN2 generator can be pre-trained specifically on images comprising complex scene data, such as urban driving scenes, or other similar images.

[0053] The inventors found that, advantageously, the combination of the masked noise encoder and StyleGAN2 results in better style transfer of the style of the first image, and therefore in better first training images, which in turn results in increased performance of the image segmentation system when the training images are used for training, or an improved assessment of the generalization ability of the image segmentation system when the training images are used during testing of the image segmentation system.

[0054] Alternatively, other style transfer machine learning systems may be used on top of this approach, for example, machine learning systems based on diffusion models (such as a diffusion model configured to interpolate between images).

[0055] Preferably, the first image consists of a first dataset of first images, and the machine learning system is trained to transfer the input image to the style of the first image.

[0056] This may be understood in particular as the method comprising the additional step of training a machine learning system as part of the method for generating an image. Alternatively, this may be understood as obtaining a machine learning system after training on a first image, for example by downloading the machine learning system from a resource on the Internet.

[0057] Some or all of the preferred embodiments of the method for training described above may be combined to form further embodiments.

[0058] In another aspect, the invention relates to a computer-implemented method for determining an output signal, wherein the output signal characterizes a classification of whether a pixel of an input image is anomalous or non-anomalous, wherein the method comprises the following steps: providing an input image as input to an image segmentation system, wherein the image segmentation system has been trained using a method according to any one of the preceding claims; The image segmentation system determines the logit of the pixel; If the difference between the largest determined logit and the smallest determined logit is less than or equal to a predefined threshold, then providing a classification of the pixel as abnormal in the output signal, otherwise providing a classification of the pixel as non-anomalous in the output signal.

[0059] The term "wherein the image segmentation system has been trained using the method" can be understood as the steps of the method for training being performed as part of the method for determining the output signal. Alternatively, it can also be understood as obtaining an image segmentation system that has been trained as described above (e.g. from a resource on the Internet).

[0060] Preferably, the image segmentation system determines whether a corresponding pixel is abnormal for a plurality of pixels, preferably all pixels, of the input image. Even more preferably, pixels determined to be abnormal can be grouped, for example by means of a connected components algorithm, so as to determine regions in the image that are considered abnormal. The grouped regions can also be output in the output signal, for example by a bounding box of pixels of a single connected component that surrounds the abnormal pixel.

[0061] In order to determine whether the pixel is abnormal or non-abnormal, an abnormality score may be determined for the pixel, wherein the abnormality score is preferably represented by the following formula:

[0062] In other words, the logit can be determined for all categories that the pixel may be classified into, and then the difference between the maximum logit and the minimum logit can be used as the anomaly score. Alternatively, it is also possible to multiply the result of the difference by -1 so that a larger anomaly score corresponds to a higher probability that the pixel is abnormal. The anomaly score can also be determined for each pixel individually.

[0063] The output signal comprising a classification may in particular be understood as the output signal comprising or consisting of a value characterizing the classification, such as an anomaly score, or a binary value indicating whether the anomaly score is less than or equal to a predefined threshold (if the difference is multiplied by -1, it is equal to or greater than a predefined threshold).

[0064] Some or all preferred embodiments of the method for determining an output signal may be combined to form another embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Embodiments of the present invention will be discussed in more detail with reference to the following drawings. The drawings show: Figure 1 : Methods for generating training images and training label mappings; Figure 2 : A training system configured to perform a method for training an image segmentation system; Figure 3 : A control system including a machine learning system that controls actuators in its environment; Figure 4 : A control system for controlling a robot that is at least partially autonomous; Figure 5 : Control systems that control manufacturing machines; Figure 6 : Control systems for controlling medical analysis systems. DETAILED DESCRIPTION

[0066] Figure 1 A method for generating training images and training label maps is schematically shown.

[0067] The method comprises obtaining a first image (i1) and a first label map (l1) corresponding to the first image (i1) and obtaining a second image (i2) and a second label map (l2) corresponding to the second image (i2). The first image (i1) and / or the second image (i2) may be obtained from a corresponding image database, or may be recorded by a sensor, in particular an optical sensor (such as a camera). Recording may also be part of the method. Preferably, the first image (i1) and the second image (i2) have different styles, for example, the color distribution of the pixels is significantly different. In this embodiment, the first image (i1) depicts a road scene, such as an urban traffic scene, and the second image (i2) includes a bear, which is usually not seen in an urban traffic scene. The corresponding label maps (l1, l2) may be obtained by humans marking the corresponding images (i1, i2) or by marking the corresponding images (i1, i2) by an automatic algorithm (i.e., may be automatically marked).

[0068] The method further includes determining a third image (i3), wherein the third image is determined by providing the second image (i2) as input to a machine learning system (70), wherein the machine learning system (70) is configured to determine an image in the style of the first image (i1) based on the provided image.

[0069] The term “determining an image in the style of the first image based on the provided image” may in particular be understood as follows: the machine learning system is configured to accept an image as input, wherein the machine learning system then converts the provided input image into an output image, wherein the output image has a different style than the input image, but has similar or identical semantic content.

[0070] In particular, the machine learning system takes as input the second image (i2, an image of a bear in this embodiment) and provides a third image (i3, an image of a bear having a color distribution similar to that of the urban traffic scene in the first image).

[0071] The machine learning system may in particular be configured to consist of a neural network including a masked noise encoder and StyleGAN2 for determining the third image (i3).

[0072] Based on the second label map (l2), objects from the second image (i2) can then be cut from the third image (i3) in a cutting operation (c). Preferably, the cutting operation selects objects from the second label map (l2) based on connected class labels of the object class (i.e., connected components of multiple specific class labels from the same object class).

[0073] The first image (i1) and the first label map (l1) may undergo an optional augmentation operation (80), such as cropping, resizing, rotating, warping, etc. The result may be used again as the first image (i1) and the first label map (l1).

[0074] Whether or not the augmentation operation is performed, the method further comprises determining a training image (x) by replacing pixels from the first image (i1) with pixels from a third image (i3). i ). In other words, the pixels obtained from the cutting operation can then be copied (ie, superimposed) onto the pixels from the first image (i1) in order to determine the training image (x i ).

[0075] The pixels in the first image (i1) that are replaced by the pixels from the third image (i3) can be randomly selected. For example, a random position can be selected in the first image (i1), and the pixels from the third image can be superimposed on this position so that the center of the pixels from the third image (i3) is located at the random position. In this case, the pixels to be replaced in the first image (i1) can be understood as being implicitly determined. Alternatively, the pixels to be replaced in the first image (i1) can be selected based on the categories of the first label mapping (l1). For example, the pixels from the third image (i3) to be pasted onto the first image (i1) constitute the pixels of an object, and the pixels in the first image (i1) can be selected so that they represent pixels that characterize a portion of the first label mapping (l1) that can reasonably include an object (e.g., free space, such as a road, sidewalk, etc.).

[0076] The method further comprises determining a training label map (t1) by replacing the class labels from the first label map (l1) with class labels indicating the abnormal class. i ), where the class label in the first label map (l1) is replaced, and for this class label, the corresponding pixel in the first image (i1) is replaced. In other words, for the training image (x i ) by copying the pixels in the training label map (t i ) is determined to include the category label of the abnormal category.

[0077] Figure 2 An embodiment of a training system (140) for training an image segmentation system (60) with the aid of a training data set (T) is shown. The training data set (T) comprises a plurality of training images (x i ) and training label mapping (l i ), where each training image (x i ) and training label mapping (l i ) has passed Figure 1The training image (x i ) and training labels (l i ) shows different content, for example, by using a different first image (i1) and / or a different second image (i2) and / or using a different position in the method for generating, copying a cut object from a third image (i3) into the first image (i1).

[0078] For training, the training data unit (150) accesses a computer-implemented database (St2), which provides a training data set (T). The training data unit (150) preferably randomly determines at least one training image (x i ) and the training images (x i ) corresponding to the training label mapping (t i ), and the training image (x i ) is transmitted to the image segmentation system (60). The image segmentation system (60) is based on the training image (x i ) to determine the output signal (y i ), where the output signal (y i ) represents the training image (x i ), preferably including logits for different classes for each pixel.

[0079] Training label mapping (t i ) and the determined output signal (y i ) is transmitted to the modification unit (180).

[0080] Based on the training label mapping (t i ) and the determined output signal (y i ), the modification unit (180) then determines new parameters (Φ′) of the image segmentation system (60), for example, new parameters of a neural network included in the image segmentation system (60). For this purpose, the modification unit (180) transforms the training label mapping (t i ) and the determined output signal (y i ), wherein the loss function includes a first term, the first term representing the logarithm of the sigmoid of the logit determined for the pixels in the training image, wherein the logarithm of the sigmoid is determined for the pixels marked as abnormal classes or indicated as abnormal in the training label map. The loss function determines the loss value as output. Preferably, the first loss term is represented by the following formula: where i,j are the height and width coordinates of the pixel, ood is the pixel coordinate of the pixel indicated as an anomaly, and N oodis the number of pixels in ood, K is the number of logits for which the logarithm of the sigmoid is determined, and S i,j is a set of K logits, and is the kth logit in the logit set for the pixel at position i,j.

[0081] Preferably, the first can be understood as the only term in the loss function, i.e., the loss function is the first term. Alternatively, Can be combined with additional terms to form a loss function. For example, the loss function can include a second term that characterizes a standard loss for semantic segmentation of pixels of the image that are not indicated as abnormal. For example, a softmax function can be applied to the logit of non-anomalous pixels, and the second loss term can be characterized or given by the cross entropy relative to the training label map, which can be determined for non-anomalous pixels, preferably for all non-anomalous pixels.

[0082] The first loss term may preferably be multiplied by a factor constituting a further hyperparameter of the method used for training. Thus, the loss function from which the loss value is determined may be characterized by the following formula: in characterizes the second loss term, and λ ood Characterize this factor.

[0083] The loss function can be extended with further terms, for example to regularize the training of image segmentation systems.

[0084] The modification unit (180) determines new parameters (Φ') based on the loss value. In a given embodiment, this is done by using a gradient descent method, preferably stochastic gradient descent, Adam or AdamW. In further embodiments, the training can also be based on an evolutionary algorithm, or a second order method for training neural networks.

[0085] In other preferred embodiments, the described training is iteratively repeated for a predetermined number of iteration steps, or iteratively repeated until the loss value falls below a predefined threshold. Alternatively or additionally, it is also possible that the training is terminated when the average first loss value with respect to the test or validation data set falls below a predefined threshold. In at least one iteration, the parameters (Φ′) determined in the previous iteration are used as parameters (Φ) of the image segmentation system (60).

[0086] Furthermore, the training system (140) may include at least one processor (145) and at least one machine-readable storage medium (146) containing instructions which, when executed by the processor (145), cause the training system (140) to perform a training method according to one of the aspects of the present invention.

[0087] In a further embodiment (not shown), the training image (x i ) and training label mapping (l i ) may not be obtained from the database, but may be generated in each iteration of training. In other words, the training images (x i ) and training label mapping (l i ) can be generated on the fly.

[0088] Figure 3 An embodiment of a control system (40) for controlling an actuator (10) in its environment (20) based on the output of an image segmentation system (60) is shown. The actuator (10) and its environment (20) will be collectively referred to as the actuator system. At preferably evenly spaced time points, a sensor (30) senses the state of the actuator system. The sensor (30) may include several sensors. Preferably, the sensor (30) is an optical sensor that takes an image of the environment (20). An output signal (S) of the sensor (30) (or, in the case where the sensor (30) includes multiple sensors, an output signal (S) for each sensor) encoding the sensed state is transmitted to the control system (40).

[0089] Thus, the control system (40) receives a stream of sensor signals (S). It then calculates a series of control signals (A) from the stream of sensor signals (S), which are then transmitted to the actuator (10).

[0090] The control system (40) receives a stream of sensor signals (S) from the sensor (30) in an optional receiving unit (50). The receiving unit (50) transforms the sensor signals (S) into an input image (x). Alternatively, in the absence of a receiving unit (50), each sensor signal (S) can be used directly as an input image (x). For example, the input image (x) can be given as an excerpt from the sensor signal (S). Alternatively, the sensor signal (S) can be processed to generate the input image (x). In other words, the input signal (x) is provided based on the sensor signal (S).

[0091] The input image (x) is then passed to the image segmentation system (60).

[0092] The image segmentation system (60) is parameterized by parameters (Φ) which are stored in a parameter storage device (St1) and provided thereby.

[0093] The image segmentation system (60) determines an output signal (y) from an input image (x). The output signal (y) is a classification of pixels of the input image (x), preferably a corresponding classification for each pixel, wherein the classification characterizes whether the pixel is abnormal. The output signal (y) is transmitted to an optional conversion unit (80), which converts the output signal (y) abnormality score and determines a control signal (A) based on the classification. Alternatively, the output signal (y) can be directly used as the control signal (A).

[0094] The classification is preferably determined by the image segmentation system (60) by first determining an anomaly score for each pixel to be classified. The anomaly score may preferably be determined according to the following formula: in is the kth logit λ determined for the pixel at position i,j in the input image (x).

[0095] The anomaly score can then be compared to a predefined threshold, and if the anomaly score is equal to or below the predefined threshold, the pixel can be classified as an anomaly. Alternatively, the anomaly score can be multiplied by -1, and if the result of the multiplication is equal to or above the predefined threshold, the pixel can be classified as an anomaly. The threshold can be understood as a hyperparameter of the method used for training.

[0096] If the pixel is not classified as an anomaly, it may preferably be assigned a class indicating a normal pixel. This may be achieved, for example, by assigning to the pixel the class corresponding to the highest logit determined by the image segmentation system (60).

[0097] The control signal (A) is then transmitted to the actuator (10) for controlling the actuator (10) accordingly.

[0098] The actuator (10) receives the control signal (A), is controlled accordingly, and performs an action corresponding to the control signal (A). The actuator (10) may include control logic that transforms the control signal (A) into a further control signal that is then used to control the actuator (10).

[0099] In further embodiments, the control system (40) may include a sensor (30). In still further embodiments, the control system (40) may alternatively or additionally include an actuator (10).

[0100] In still further embodiments, it is contemplated that the control system (40) controls the display (10a) instead of or in addition to the actuator (10).

[0101] Furthermore, the control system (40) may include at least one processor (45) and at least one machine-readable storage medium (46) on which instructions are stored that, when executed, cause the control system (40) to perform a method according to aspects of the present invention.

[0102] Figure 4 An embodiment is shown in which a control system (40) is used to control an at least partially autonomous robot, such as an at least partially autonomous vehicle (100).

[0103] The sensors (30) may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more LiDAR sensors. Some or all of these sensors are preferably, but not necessarily, integrated into the at least partially autonomous robot (100).

[0104] The image segmentation system (60) may be configured to perform semantic segmentation or instance segmentation of the vicinity of the at least partially autonomous robot (100) based on the input image (x). In the event that an abnormal region (i.e., a pixel or connected component of pixels classified as abnormal) is detected in the input image (x), a control signal (A) may be determined based on this information, for example to avoid that the path of the at least partially autonomous robot (100) is directed through the abnormal region or to avoid that the path of the at least partially autonomous robot (100) is closer to the abnormal region than a predefined threshold.

[0105] The actuator (10), which is preferably integrated in the vehicle (100), may be provided by the brakes, propulsion system, engine, transmission or steering of the vehicle (100).

[0106] Alternatively or additionally, the control signal (A) may also be used to control the display (10a), for example, for displaying abnormal pixels detected by the image segmentation system (60). It is also possible that the control signal (A) may control the display (10a) so that it generates a warning signal when the vehicle (100) approaches an area including abnormal pixels. The warning signal may be a warning sound and / or a tactile signal, such as vibration of the steering wheel of the vehicle.

[0107] In a further embodiment, the at least partially autonomous robot may be provided by another mobile robot (not shown), which may move, for example, by flying, swimming, diving or walking. The mobile robot may be, among other things, an at least partially autonomous lawn mower or an at least partially autonomous cleaning robot. In all the above embodiments, a control signal (A) may be determined such that a propulsion unit and / or a steering device and / or a brake of the mobile robot are controlled so that the mobile robot may avoid a collision with the identified object.

[0108] Figure 5 An embodiment is shown in which a control system (40) is used to control a manufacturing machine (11), such as a punch cutter, a cutter, a gun drill or a gripper, of a manufacturing system (200), for example as part of a production line. The manufacturing machine may include a transport device (13), such as a conveyor belt or an assembly line, for moving the manufactured products (12). The control system (40) controls an actuator (10), which in turn controls the manufacturing machine (11).

[0109] The sensor (30) may be provided by an optical sensor that captures, for example, properties of the manufactured product (12).

[0110] The image segmentation system (60) can determine the portion of the input image (x) that includes the manufactured product (12) device. The actuator (10) can then be controlled based on the location of the manufactured product (12) for subsequent manufacturing steps of the manufactured product (12). For example, the actuator (10) can be controlled to cut the manufactured product at a specific location on the manufactured product itself. If the output signal (y) determines that there is an anomaly on the transport equipment (13) and / or the manufactured product (12), the operation of the machine can be stopped and / or a human operator can be called to evaluate the problem. Alternatively, if an anomaly is detected on a pixel belonging to the manufactured product, the actuator (10) can then be controlled to remove the manufactured product from the transport equipment.

[0111] Figure 6 An embodiment of a medical imaging system (500) controlled by a control system (40) is shown. For example, the imaging system may be an MRI device, an x-ray imaging device, or an ultrasound imaging device. For example, the sensor (30) may be an imaging sensor that acquires at least one image of a patient, for example showing different types of body tissue of the patient.

[0112] The image segmentation system (60) may be configured to detect different types of tissue in the sensed image, for example by classifying the tissue displayed in the image as malignant or benign tissue. This may be accomplished by semantic segmentation of the input image (x) by the image segmentation system (60). A control signal (A) may then be determined to cause the display (10a) to display different tissues, for example by displaying the input image (x) and coloring different areas of the same tissue type with the same color. The control signal (A) may further be selected to color abnormal areas with different colors to guide a physician to pay attention to the abnormal areas and examine them in detail.

[0113] In further embodiments (not shown), the imaging system (500) may be used for non-medical purposes, such as for determining material properties of a workpiece. In these embodiments, the image segmentation system (60) may be configured to receive an input image (x) of at least a portion of the workpiece and perform semantic segmentation on the input image (x) to classify the material properties of the workpiece. A control signal (A) may then be determined to cause the display (10a) to display the input image (x) and information about the detected material properties.

[0114] The term "computer" may be understood to cover any device for processing predefined computing rules. These computing rules may be in the form of software, hardware, or a mixture of software and hardware.

[0115] In general, a plurality may be understood to be indexed, that is, each element in the plurality is assigned a unique index, preferably by assigning consecutive integers to the elements contained in the plurality. Preferably, if the plurality comprises N elements, where N is the number of elements in the plurality, integers from 1 to N are assigned to these elements. It may also be understood that the elements in the plurality may be accessed by their indexes.

Claims

1. A computer-implemented method for training an image segmentation system (60), the method comprising the following steps: Get the training image (x i ) and training label mapping (t i ), where the training label mapping (t i ) indicates the training image (x i ) of pixels belonging to the abnormal category; Provide training images (x i ) as input to an image segmentation system (60), wherein the image segmentation system (60) determines a representation of the training image (x i )’s logit values ​​for different categories of pixels; Determine a loss value based on a loss function, wherein the loss function includes a first term, the first term representing the loss value for the training image (x i ), where the logarithm of the sigmoid of the logit is the logarithm of the pixels in the training label map (t i ) is determined by the pixels marked as belonging to the abnormal category; • Training the image segmentation system (60) by minimizing the loss value.

2. The method of claim 1, wherein the first term represents the logarithm of the sigmoid of the maximum logit of the predetermined quantity.

3. The method according to claim 1 or 2, wherein the first loss term is characterized by the following formula: where i,j are the height and width coordinates of the pixel, ood is the pixel coordinate of the pixel indicated as an anomaly, and N ood is the number of pixels in ood, K is the number of logits for which the logarithm of the sigmoid is determined, and S i,j is a set of K logits, and is the kth logit in the logit set for the pixel at position i,j.

4. The method according to any one of claims 1 to 3, wherein the training image (x i ) and training label mapping (t i ) is generated by generating training images (x i ) and training label mapping (t i ) to obtain, where the training image (x i ) and training label mapping (t i ) comprises the following steps: Obtaining a first image (i1) and a first label map (l1) corresponding to the first image (i1) and obtaining a second image (i2) and a second label map (l2) corresponding to the second image (i2); determining a third image (i3), wherein the third image is determined by providing the second image (i2) as input to a machine learning system (70), wherein the machine learning system (70) is configured to determine an image in the style of the first image (i1) based on the provided image; The training image (x) is determined by replacing pixels from the first image (i1) with pixels from the third image (i3) i ), wherein pixels from the third image (i3) are determined based on categories from the second label map (l2); Determine a training label map (t1) by replacing the class labels from the first label map (l1) with class labels indicating the abnormal class i ), wherein the category labels in the first label map (l1) are replaced, and for the category labels, the corresponding labels in the first image (i1) are replaced.

5. A method according to claim 4, wherein replacing a category label from a first label map is achieved by replacing the category label from the first label map with a category label from a second label map, wherein the category label from the second label map is used for replacement, for which the corresponding pixel has been used as a replacement in the first image, and wherein the replaced category label in the first label map is further indicated as being caused by the replacement.

6. A method according to claim 4 or 5, wherein the first image is transformed in an enhancement operation prior to determining the training images.

7. A method according to any one of claims 4 to 6, wherein the machine learning system determines its output based on providing its input to a masked noise encoder of the machine learning system and providing the output of the masked noise encoder to StyleGAN2.

8. A method according to any one of claims 4 to 7, wherein the first image consists of a first dataset of first images, and the machine learning system is trained to transfer the input image to the style of the first image.

9. A computer-implemented method for determining an output signal (y), wherein the output signal (y) characterizes a classification of a pixel of an input image (x) as being anomalous or non-anomalous, wherein the method comprises the following steps: providing an input image (x) as input to an image segmentation system (60), wherein the image segmentation system (60) has been trained using a method according to any one of the preceding claims; Determining the logit of the pixel by an image segmentation system (60); If the difference between the largest determined logit and the smallest determined logit is less than or equal to a predefined threshold, providing in the output signal (y) a classification of the pixel as abnormal, otherwise providing in the output signal (y) a classification of the pixel as non-abnormal.

10. A training system (140) configured to perform the training method according to any one of claims 1 to 8.

11. A control system (40) configured to perform the method according to claim 9, wherein the control system (40) determines a control signal (A) based on the output signal (y), wherein the control signal (A) is configured to control an actuator (10) and / or a display (10a).

12. A computer program configured to cause a computer to perform the method according to any one of claims 1 to 9 and all its steps if the computer program is executed by a processor (45, 145).

13. A machine-readable storage medium (46, 146) having stored thereon a computer program according to claim 12.

Images