Device and method for training image segmentation system

Abstract

To provide a computer-implemented method for training an image segmentation system (60).SOLUTION: A method includes the steps of: obtaining a training image (xi) and a training label map (ti), the training label map (ti) indicating that pixels of the training image (xi) belongs to an anomaly class; providing the training image (xi) as input to an image segmentation system (60), the image segmentation system (60) determining values characterizing logits for different classes for pixels of the training image (xi); determining a loss value based on a loss function, the loss function comprising a first term that characterizes a logarithm of a sigmoid of logits determined for pixels in the training image (xi), the logarithm of the sigmoid being determined for pixels labeled as belonging to the anomaly class in the training label map (ti); and training the image segmentation system (60) by minimizing the loss value.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a computer-implemented method for training an image segmentation system, a computer-implemented method for detecting anomalies in an image, a training system, a control system, a computer program, and a machine-readable storage device. [Background technology]

[0002] Tian et al., “Pixel-wise energy-biased abstention learning for anomaly segmentation on complex urban driving scenes”, https: / / arxiv.org / pdf / 2111.12264.pdf, 2022, discloses an anomaly segmentation method called pixel-wise energy-biased abstention learning (PEBAL), which explores pixel-wise anomaly suppression learning (AL) using a model that learns adaptive pixel-level anomaly classes and an energy-based model (EBM) that learns the anomaly pixel distribution. PEBAL is based on the joint training of EBM and AL, where the EBM is trained to output high energy for anomaly pixels (from outlier exposure) and the AL is trained to adaptively low penalty for these high energy pixels being included in the anomaly class.

[0003] Nayal et al., “RbA: Segmenting Unknown Regions Rejected by All”, https: / / arxiv.org / pdf / 2211.14293.pdf, 2022, discloses an outlier scoring function called RbA by defining an outlier event as one rejected by all known classes. Experiments show that masked classification improves the performance of existing outlier detection methods, with the best results being achieved with the proposed RbA.

[0004] Li et al., “Intra-&Extra-Source Exemplar-Based Style Synthesis for Improved Domain Generalization”, https: / / arxiv.org / pdf / 2307.00648.pdf, 2023, discloses an example-based style synthesis pipeline for improving domain generalization in semantic segmentation, in particular a masked noise encoder combined with StyleGAN2.

[0005] Perceiving the environment and detecting the contents of the environment are important tasks in various technological applications, e.g., robotics. A typical goal is, for example, to use sensors such as cameras to detect objects, vegetation, or other relevant items in order to avoid colliding with elements of the environment.

[0006] Especially in safety-critical applications such as autonomous driving (AD), the ability to detect unknown objects (Out-of-Distribution, OoD objects) is important. This can also be understood as identifying anomalous objects. More specifically, detecting pixels of unknown or anomalous objects in semantic segmentation scenarios is a core perception task for mobile robots, e.g., autonomous vehicles.

[0007] Semantic segmentation tasks are typically solved by neural networks, and it is therefore desirable to enable neural networks configured for semantic segmentation to also detect unknown or anomalous pixels so that systems that use neural networks to model the environment can accurately identify unknown or anomalous objects and separate them from the drivable space.

[0008] A central problem of OoD detection is the vast amount of possible OoD objects compared to the relatively small number of object categories on which neural networks are trained. Moreover, semantic segmentation typically deals with scene-centric input images, i.e., images in which OoD objects are co-located with many other known objects and elements of the environment. This poses an even greater challenge compared to OoD detection on object-centric data (e.g., image classification benchmarks). Another challenge with scene-centric images is class imbalance. For example, road pixels in AD scenes often constitute a large portion of the training data, biasing neural networks towards predicting roads. Due to the imbalance of the training data, neural networks often show a tendency to predict a majority class for OoD objects found on the road, which is a significant error with the highest level of risk.

[0009] A particularly successful technique for detecting unknown or anomalous objects in semantic segmentation tasks (also known as anomaly segmentation) is a form of outlier exposure. That is, while potential OoD scenarios are not accessible at inference time, there are many readily available datasets different from the training data that can serve as OoD examples during training. For example, the MS COCO dataset is a popular OoD proxy dataset for training semantic segmentation. For example, certain objects included in the MS COCO dataset can be cut out and then pasted into a road scene to create anomalous examples of, for example, AD scenes.

[0010] The OoD data constructed in this way can be used during training of a neural network from scratch or during fine-tuning of a pre-trained model. The inventors found that using OoD data for training from scratch can degrade the within-distribution generalization performance of the neural network (i.e., its semantic segmentation performance for known and / or non-anomalous pixels) and may require architecture modification of the neural network. In contrast, the authors found that when using OoD data only during fine-tuning of a pre-trained neural network, the neural network is usually under the constraint of small weight changes and is therefore less susceptible to overfitting. [Prior art documents] [Non-patent literature]

[0011] [Non-Patent Document 1] Tian et al., ““Pixel-wise energy-biased abstention learning for anomaly segmentation on complex urban driving scenes”, https: / / arxiv.org / pdf / 2111.12264.pdf, 2022.” [Non-Patent Document 2] Nayal et al., ““RbA:Segmenting Unknown Regions Rejected by All”, https: / / arxiv.org / pdf / 2211.14293.pdf, 2022.” [Non-Patent Document 3] Li et al., ““Intra-&Extra-Source Exemplar-Based Style Synthesis for Improved Domain Generalization”, https: / / arxiv.org / pdf / 2307.00648.pdf, 2023.” Summary of the Invention [Problem to be solved by the invention]

[0012] Based on these findings, the present invention advantageously proposes a new method for generating OoD data by adapting and pasting the style of an OoD data copy onto an existing image. Style transfer results in training data that is more suitable for training to detect unknown or anomalous objects, since a neural network trained on this data cannot rely on the style of the pasted content (i.e., which appears "off" relative to the rest of the image) to determine whether a pixel is anomalous or not. The present invention further proposes an advantageous method for training a neural network based on data generated from the method for generating OoD data. [Means for solving the problem]

[0013] In a first aspect, the present invention relates to a computer implemented method for training an image segmentation system, the method comprising: obtaining training images and a training label map, the training label map indicating pixels of the training images as belonging to an abnormal class; providing training images as input to an image segmentation system that determines values ​​characterizing the logits of different classes of pixels of the training images; determining a loss value based on a loss function, the loss function including a first term characterizing a logarithm of a sigmoid of logits determined for pixels in the training images, the logarithm of the sigmoid being determined for pixels labeled as belonging to an anomalous class in the training label map; training an image segmentation system by minimizing a loss value; Includes.

[0014] The image segmentation system may be configured to perform semantic and / or instance segmentation of the provided input image. This may be achieved in particular by an image segmentation system including or consisting of a neural network configured for image segmentation and / or instance segmentation. Semantic and / or instance segmentation may in particular be understood as a particular form of image classification.

[0015] The method for training may in particular include training the neural network in a supervised manner. The image segmentation system may also include pre-processing steps applied to input images provided to the image segmentation system and / or post-processing steps applied to the output of the image segmentation system.

[0016] The method for training may be performed "from scratch", i.e., with an untrained image segmentation system, or as a fine-tuning method in which the image segmentation system is trained on other data using one or more generated images before training on other data.

[0017] The method for training may preferably be performed on a batch of generated training images, i.e. the method for generating training images may be performed multiple times with different images and / or different replacement pixels to determine a batch of training images. Preferably, the generated images and an image without replacement pixels, e.g. the first image, are combined in a batch.

[0018] An image can be understood to be obtained by synthesis (e.g. rendering from a virtual scene in a computer) or by being recorded from a sensor, in particular an optical sensor, in which case an image can be understood as the output of a camera, a lidar, a radar, an ultrasonic sensor or a thermal sensor.

[0019] Acquiring an image may be understood as any method of acquiring an image, for example by recording the image with a respective sensor or by retrieving it from a database of images.

[0020] A label map can generally be understood as an annotation of an image, where the annotation assigns a class label to a subset of pixels of the image to which the label map corresponds, preferably all pixels. The correspondence can be achieved by a label map that typically has a width and height comparable to the image, where each pixel can be assigned a location in the image and can further be assigned a class label (e.g., an integer characterizing an index of the class) in the label map, which itself can be assigned a location in the label map. The location can be characterized by an index along the width and height dimensions of the image and the label map, respectively. Thus, the label map may be given by a matrix of integers indicating the class labels.

[0021] A label map corresponding to an image can be understood as a label map that contains the classification for each pixel in the image.

[0022] The term "minimizing the loss value" can preferably be understood as taking steps such that the loss value is smaller when the same input is provided to the image segmentation system. This can be achieved in particular by gradient descent, in particular stochastic gradient descent. Alternatively, other training methods are possible, for example methods based on evolutionary algorithms.

[0023] Advantageously, the inventors have found that using the logits of pixels designated as "anomalous" in this way can be understood as training the image segmentation system as a one-pair residual (OvR) classifier for each pixel, i.e., the system can indicate for each pixel and each "non-anomalous" class the likelihood that the pixel belongs to the class. If the likelihood of each "non-anomalous" class for a pixel is low (e.g., the logit characterizes a probability of 50% or less), then the pixel can be considered anomalous. The inventors have found that this advantageously enhances the ability of the image segmentation system to accurately indicate whether a pixel is anomalous or not.

[0024] Even more advantageously, the authors found that the proposed loss function can be applied only during fine-tuning of the image segmentation system, which does not require any other ground truth data, but only fine-tunes the image segmentation system based on the loss function, and can be particularly understood as a training method that preferably includes only the first term.

[0025] Alternatively, the loss function may also include a second term that characterizes a standard loss for semantic segmentation of pixels of the image that are not marked as anomalous. For example, a softmax function may be applied to the logits of the non-anomalous pixels, and the second loss term may be characterized or given by a cross-entropy loss over the training label map, determined for the non-anomalous pixels, preferably all non-anomalous pixels.

[0026] Preferably, the first term characterizes the logarithm of the sigmoid of the maximum logit of a given quantity.

[0027] Advantageously, the inventors have found that by limiting the amount of logits for which the logarithm of the sigmoid of the logits is determined, the performance of the image segmentation system, i.e., its ability to accurately indicate whether a pixel is anomalous or not, is further improved.

[0028] The amount of logits on which the logarithm of the sigmoid is determined can be considered a hyperparameter of the method. It can be adjusted according to any suitable hyperparameter adjustment method. The authors found that small values ​​work particularly well, in particular using a maximum of 3, 5 or 7 logarithms per pixel. However, other amounts may also be suitable.

[0029] Preferably, the first loss term is characterized by the following equation:

number

number

[0030] The first loss term may preferably be multiplied by a coefficient constituting a further hyperparameter of the method for training. The loss function for which the loss value is determined can thus be characterized by the following formula: L=L semseg +λ ood L ood , Here, L semseg characterizes the second loss term, λ ood characterizes the factor.

[0031] The loss function may be extended, for example, by an additional term to regularize the training of the image segmentation system.

[0032] Preferably, the training images (x i ) and the training label map (t i ) is the training image (xi ) and the training label map (t i ), and the training images (x i ) and the training label map (t i ) obtaining a first image (i1) and a first label map (l1) corresponding to the first image (i1), and obtaining a second image (i2) and a second label map (l2) corresponding to the second image (i2); determining a third image (i3), the third image being determined by providing the second image (i2) as an input to a machine learning system (70), the machine learning system (70) being configured to determine an image in the style of the first image (i1) based on the provided images; · Transform the training image (x) by replacing pixels from the first image (i1) with pixels from the third image (i3). i ), where pixels from a third image (i3) are determined based on a class from a second label map (l2); The training label map (t i ) where corresponding pixels in the first image (i1) are replaced to replace class labels in the first label map (l1); Includes.

[0033] This can be understood as augmenting a first image with pixels from a second image, where the second image is first made to have a similar or identical style to the first image. Advantageously, this style alignment between the first and second images allows the pixels of the second image to be realistically embedded in the first image. In other words, pixels from a third image (i.e. the second image after style alignment) look as if they could actually belong to the first image.

[0034] Training an image segmentation system with such generated images increases the diversity of the training data, leading to improved performance of the image segmentation system. Even more advantageously, the copied pixels are marked in a training label map (either via an anomaly class or via an additional indication in the label map) as indicative of anomalies, and the training label map can then be used to train the image segmentation system to determine whether a particular pixel in a given image is anomalous or not.

[0035] Because the pixels from the third image are in the style of the pixels from the first image, the image segmentation system cannot rely on the style of the replaced pixels as an indicator of anomalous pixels, and therefore must advantageously learn to determine whether a pixel in an image is anomalous based on the semantic content of the image (e.g., a non-flying object located in the sky is likely to be anomalous).

[0036] If the replaced class label in the first label map further indicates as a result of replacement, the indication can be achieved, for example, by assigning a special class label that simultaneously indicates the corresponding class from the second label map and "anomaly" (e.g., by adding a predetermined offset to the integer that otherwise represents the non-anomalous class). Alternatively, an additional binary map may be provided alongside the generated training label map, indicating for each pixel of the generated training images whether the pixel characterizes an anomaly or not, i.e. whether the pixel was copied from a third image or not. The binary map may be provided in the form of a matrix of binary or integer elements.

[0037] Anomalies can be understood as parts of an image (depicted via the pixels of the image) that constitute rare, unusual or irrational objects. An anomalous pixel can be understood as a pixel that belongs to an anomalous object.

[0038] The machine learning system may be understood to be particularly configured such that the semantic content of its input is not altered in its output, but the style is altered. For example, the machine learning system may be provided by a neural network including a masked noise encoder and StyleGAN2, the neural network being further configured (i.e., trained) to alter the style of images provided as input.

[0039] The term "determining an image in the style of a first image based on a provided image" may be particularly understood as a machine learning system configured to accept an image as input, and convert the provided input image into an output image, where the output image has a different style from the input image but has similar or identical semantic content.

[0040] The first images may in particular be randomly acquired from the dataset of first images, which may be understood as first images having a similar visual style, for example because they have been recorded by the same sensor and / or similar environmental conditions of the sensor.

[0041] Determining a training image by replacing pixels from a first image with pixels from a third image can be understood as providing an image resulting from replacing pixels from the first image with respective pixels of the third image as training images.

[0042] The pixels in the first image to be replaced by pixels from the third image may be selected randomly. For example, a random location may be selected in the first image and the pixels from the third image may be overlaid on the location such that the center of the pixel from the third image is located at the random location. In this case, the replaced pixels in the first image may be understood to be implicitly determined. Alternatively, the replaced pixels in the first image may be selected based on a class of the first label map. For example, if the pixels from the third image to be overlaid on the first image constitute object pixels, the pixels in the first image may be selected to represent pixels characterizing portions in the first label map that may reasonably include objects, such as open spaces such as roads, sidewalks, etc.

[0043] Alternatively, a location in the first image that characterizes a portion of the first label map that could reasonably contain an object may be selected, and a pixel from the third image may be copied into the first image such that the center of the pixel from the third image is located at that location.

[0044] In all examples, the central locations are chosen merely as examples: other locations can also be used, for example corners or any other desired location.

[0045] Preferably, the replaced pixel is determined randomly, this expression is also understood to include a method in which a single location in the first image is randomly selected and a pixel from the third image is copied into the first image as described above.

[0046] Preferably, permuting the class label from the first label map is achieved by replacing the class label from the first label map with a class label from the second label map, the class label from the second label map being used for the substitution whose corresponding pixel was used as the substitution in the first image, and the permuted class label in the first label map further indicating as a result of the substitution.

[0047] Alternatively, a designated "anomalous" class can be assigned to elements of a training label map whose corresponding pixels in the training images were determined by copying from a third image.

[0048] Preferably, the first image is transformed with an augmentation operation before determining the training images.

[0049] The first image may, for example, be cropped or resized before undergoing pixel replacement. The augmentation may be any known augmentation operation. The result of the augmentation operation may then be used as the first image for a subsequent operation, for example, to determine training images.

[0050] Advantageously, the authors found that the image segmentation system performed better when trained with the augmented first images.

[0051] Preferably, the machine learning system determines its output based on providing its input to a masked noise encoder of the machine learning system and providing the output of the masked noise encoder to StyleGAN2.

[0052] The masked noise encoder and StyleGAN2 can be specifically understood as sub-networks of a combined neural network. The architecture of the masked noise encoder may be the architecture of a pSp encoder. The masked noise encoder converts the provided input image into the augmented latent space W of the pre-trained StyleGAN2 generator. + The pre-trained StyleGAN2 generator can be specifically pre-trained on images containing complex scene data, such as urban driving scenarios or other similar images.

[0053] The inventors have found that advantageously, a combination of a masked noise encoder and StyleGAN2 results in better style transfer of the style of the first image, and therefore better first training images, which results in improved performance of the image segmentation system when using the training images for training, or improved assessment of the generalization ability of the image segmentation system when using the training images during testing of the image segmentation system.

[0054] Alternatively, other style transfer machine learning systems can also be used in the present method, for example machine learning systems based on diffusion models, such as diffusion models configured to interpolate between images.

[0055] Preferably, the first image is constituted by a first dataset of first images, and the machine learning system is trained to transfer the input image to the style of the first image.

[0056] This may be understood as a method that includes, in particular, an additional step of training a machine learning system as part of the method for generating an image, or as obtaining the machine learning system after training on a first image, for example by downloading it from a resource on the Internet.

[0057] Some or all of the preferred embodiments of the training method as described above may be combined to form further embodiments.

[0058] In another aspect, the invention relates to a computer-implemented method for determining an output signal, the output signal characterizing a classification of pixels of an input image as anomalous or not, the method comprising: providing an input image as input to an image segmentation system, the image segmentation system having been trained with a method according to any one of the preceding claims; determining logits of pixels by an image segmentation system; providing a classification of the pixel as anomalous in the output signal if the difference between the maximum determined logit and the minimum determined logit is less than or equal to a predefined threshold, and providing a classification of the pixel as non-anomalous in the output signal otherwise; Includes.

[0059] The term "the image segmentation system is trained in a certain manner" may be understood as a method step for training that is performed as part of the method for determining the output signal, or as obtaining an image segmentation system trained as described above, for example from a resource on the Internet.

[0060] Preferably, the image segmentation system determines for a number of pixels, preferably all pixels of the input image, whether each pixel is anomalous or not. Even more preferably, the pixels that have been determined to be anomalous may be grouped, for example by a connected components algorithm, to determine regions in the image that are considered to be anomalous. The grouped regions may also be output in the output signal, for example by a bounding box surrounding the pixels of a single connected component of the anomalous pixel.

[0061] To determine the classification of a pixel as anomalous or not, an anomaly score may be determined for the pixel, where the anomaly score is preferably characterized by the following formula:

number

[0062] In other words, the logits can be determined for all classes into which the pixel is likely to be classified, and the difference between the maximum and minimum logits can be used as the anomaly score. Alternatively, the result of the difference can be multiplied by -1, so that the larger the anomaly score, the more likely the pixel is to be anomalous. The anomaly score can also be determined for each pixel individually.

[0063] An output signal comprising the classification may in particular be understood as an output signal comprising or consisting of a value characterising the classification, e.g. an anomaly score or a binary value indicating whether the anomaly score is below a predefined threshold (or above a predefined threshold if the difference is multiplied by -1).

[0064] Some or all of the preferred embodiments of the method for determining an output signal can be combined with other embodiments.

[0065] Embodiments of the invention will now be described in more detail with reference to the following drawings. [Brief description of the drawings]

[0066] [Figure 1] FIG. 1 illustrates a method for generating training images and a training label map. [Diagram 2] FIG. 1 illustrates a training system configured to perform a method for training an image segmentation system. [Diagram 3] FIG. 1 illustrates a control system that includes a machine learning system that controls actuators in the environment. [Figure 4] FIG. 1 illustrates a control system for controlling an at least partially autonomous robot. [Diagram 5] FIG. 1 illustrates a control system for controlling a manufacturing machine. [Figure 6] FIG. 1 illustrates a control system for controlling a medical analysis system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0067] FIG. 1 shows a schematic of a method for generating training images and a training label map.

[0068] The method comprises the steps of obtaining a first image (i1) and a first label map (l1) corresponding to the first image (i1) and obtaining a second image (i2) and a second label map (l2) corresponding to the second image (i2). The first image (i1) and / or the second image (i2) may be obtained from a respective database of images or may be recorded by a sensor, in particular an optical sensor such as a camera. Recording may also be part of the method. Preferably, the first image (i1) and the second image (i2) have different styles, e.g. the color distribution of the pixels is significantly different. In this embodiment, the first image (i1) depicts a road scene, e.g. an urban traffic scene, and the second image (i2) includes a bear, which is not typically found in urban traffic scenes. The respective label maps (l1, l2) may be obtained by labeling the respective images (i1, i2) by a human or an automated algorithm, i.e. they may be auto-labeled.

[0069] The method further comprises a step of determining a third image (i3), the third image being determined by providing the second image (i2) as an input to a machine learning system (70), the machine learning system (70) being configured to determine an image in the style of the first image (i1) based on the provided images.

[0070] The term "determining an image in the style of a first image based on a provided image" may be particularly understood as a machine learning system configured to accept an image as input, and convert the provided input image into an output image, where the output image has a different style from the input image but has similar or identical semantic content.

[0071] In particular, the machine learning system takes as input a second image (i2, an image of a bear in the embodiment) and provides a third image (i3, an image of a bear) having a similar color distribution as the urban traffic scene in the first image.

[0072] The machine learning system may be configured to consist of a neural network including, in particular, a masked noise encoder and StyleGAN2 for determining a third image (i3).

[0073] Then, based on the second label map (l2), the objects from the second image (i2) can be cut from the third image (i3) in a cutting operation (c). Preferably, the cutting operation selects objects from the second label map (l2) based on connected class labels of the object classes (i.e. connected components of multiple specific class labels from the same object class).

[0074] The first image (i1) and the first label map (l1) may be subjected to optional augmentation operations (80), such as, for example, cropping, resizing, rotating, warping, etc. The results may then be used again as the first image (i1) and the first label map (l1).

[0075] Regardless of whether an augmentation operation is performed, the method further comprises: i In other words, next, the training images (x i ), the pixels obtained from the cutting operation can be copied (i.e., overlaid) over pixels from the first image (i1).

[0076] The pixels in the first image (i1) to be replaced by pixels from the third image (i3) may be selected randomly. For example, a random location may be selected in the first image (i1) and a pixel from the third image (i3) may be superimposed on the location such that the center of the pixel from the third image (i3) is located at the random location. In this case, the pixel to be replaced in the first image (i1) may be understood to be determined implicitly. Alternatively, the pixel to be replaced in the first image (i1) may be selected based on the class of the first label map (l1). For example, the pixels from the third image (i3) to be pasted into the first image (i1) may constitute object pixels, and the pixels of the first image (i1) may be selected such that they represent pixels characterizing parts of the first label map (l1) that may reasonably contain objects, such as open spaces such as roads, sidewalks, etc.

[0077] The method creates a training label map (t i ), where the corresponding pixels in the first image (i1) are replaced to replace the class labels in the first label map (l1). In other words, for a training image (x i ), the training label map (t i ) is determined to contain the class labels of the abnormal classes.

[0078] 2 illustrates one embodiment of a training system (140) for training an image segmentation system (60) with a training data set (T). The training data set (T) includes a number of training images (x i ) and the training label map (l i ), and for each training image (x i ) and the training label map (l i ) are obtained by the method shown in Figure 1. i ) and training labels (l i) shows different content, for example by using a different first image (i1) and / or a different second image (i2) in a method for generating and / or using different positions where a cut-out object from a third image (i3) is copied into the first image (i1).

[0079] For training, the training data unit (150) accesses a computer-implemented database (St2), which provides a training data set (T). The training data unit (150) selects, preferably randomly, at least one training image (x i ) and training images (x i ) corresponding to the training label map (t i ) and determine the training image (x i ) to an image segmentation system (60). The image segmentation system (60) receives the training images (x i ) based on the output signal (y i ) and output signal (y i ) is the training image (x i ) and preferably includes the logits of the different classes for each pixel.

[0080] The training label map (t i ) and the determined output signal (y i ) is sent to the modification unit (180).

[0081] Then, the training label map (t i ) and the determined output signal (y i ), the modification unit (180) determines new parameters (Φ') for the image segmentation system (60), e.g., new parameters of a neural network included in the image segmentation system (60). To this end, the modification unit (180) uses a loss function to modify the training label map (t i ) and the determined output signal (y i), and the loss function includes a first term characterizing the logarithm of a sigmoid of logits determined for pixels in the training images, the logarithm of the sigmoid being determined for pixels labeled as an anomaly class or indicated as anomaly in the training label map. The loss function determines a loss value as an output. Preferably, the first loss term is characterized by the following equation:

number

number

[0082] Preferably, the first term L ood can be understood as the only term in the loss function, i.e., the loss function is the first term. Alternatively, L ood may be combined with further terms to form a loss function. For example, the loss function may include a second term that characterizes a standard loss for semantic segmentation of pixels of an image that are not marked as anomalous. For example, a softmax function may be applied to the logits of the non-anomalous pixels, and the second loss term may be characterized or given by a cross-entropy loss over the training label map, determined for the non-anomalous pixels, preferably all non-anomalous pixels.

[0083] The first loss term may preferably be multiplied by a coefficient constituting a further hyperparameter of the method for training. The loss function for which the loss value is determined can thus be characterized by the following formula: L=Lsemseg +λ ood L ood , Here, L semseg characterizes the second loss term, λ ood characterizes the factor.

[0084] The loss function may be extended, for example, by an additional term to regularize the training of the image segmentation system.

[0085] The refinement unit (180) determines new parameters (Φ') based on the loss value. In the given embodiment, this is done using a gradient descent method, preferably stochastic gradient descent, Adam, or AdamW. In further embodiments, the training may also be based on evolutionary algorithms or second-order methods for training neural networks.

[0086] In another preferred embodiment, the described training is repeated iteratively for a predefined number of iteration steps or until the loss value falls below a predefined threshold. Alternatively or additionally, training can be terminated when the average first loss value for the test or validation data set falls below a predefined threshold. In at least one of the iterations, new parameters (Φ') determined in the previous iteration are used as parameters (Φ) of the image segmentation system (60).

[0087] Additionally, the training system (140) may include at least one processor (145) and at least one machine-readable storage medium (146) containing instructions that, when executed by the processor (145), cause the training system (140) to perform a training method according to one of the aspects of the present invention.

[0088] In a further embodiment (not shown), the training images (x i ) and the training label map (l i) may not be obtained from a database, but may be generated at each iteration of training. In other words, the training images (x i ) and the training label map (l1) can be generated on the fly.

[0089] 3 shows an embodiment of a control system 40 that controls an actuator 10 in its environment 20 based on the output of an image segmentation system 60. The actuator 10 and its environment 20 are collectively referred to as an actuator system. A sensor 30 senses a state of the actuator system, preferably at equally spaced time points. The sensor 30 may include several sensors. Preferably, the sensor 30 is an optical sensor that takes images of the environment 20. An output signal (S) of the sensor 30 (or the output signal (S) of each of the sensors, if the sensor 30 includes multiple sensors), encoding the sensed state, is transmitted to the control system 40.

[0090] Thereby, the control system (40) receives a stream of sensor signals (S) and then calculates a set of control signals (A) in response to the stream of sensor signals (S), which are then transmitted to the actuators (10).

[0091] The control system (40) receives a stream of sensor signals (S) of the sensor (30) at an optional receiving unit (50). The receiving unit (50) converts the sensor signals (S) into an input image (x). Alternatively, in the absence of the receiving unit (50), each sensor signal (S) may be directly obtained as an input image (x). The input image (x) may, for example, be provided as an excerpt from the sensor signals (S). Alternatively, the sensor signals (S) may be processed to obtain the input image (x). In other words, the input signal (x) is provided according to the sensor signals (S).

[0092] The input image (x) is then passed to an image segmentation system (60).

[0093] The image segmentation system (60) is parameterized by parameters (Φ) provided and stored in a parameter store (St1).

[0094] The image segmentation system (60) determines an output signal (y) from an input image (x). The output signal (y) is a classification of pixels of the input image (x), preferably a respective classification for each pixel, characterizing whether the pixel is anomalous or not. The output signal (y) is sent to an optional transformation unit (80) that transforms the output signal (y) anomaly score and determines a control signal (A) based on the classification. Alternatively, the output signal (y) may be used directly as the control signal (A).

[0095] The classification is preferably determined by the image segmentation system 60 by first determining an anomaly score for each pixel to be classified. The anomaly score may preferably be determined according to the following formula:

number

number

[0096] The anomaly score can then be compared to a predefined threshold, and if the anomaly score is less than or equal to the predefined threshold, the pixel can be classified as anomalous. Alternatively, the anomaly score can be multiplied by -1, and if the result of the multiplication is greater than or equal to the predefined threshold, the pixel can be classified as anomalous. The threshold can be understood as a hyperparameter of the method for training.

[0097] If a pixel is not classified as abnormal, it is preferably assigned a class indicative of a normal pixel, which may be accomplished, for example, by assigning to the pixel the class corresponding to the highest logit as determined by the image segmentation system 60.

[0098] The control signal (A) is then sent to the actuator (10) to control the actuator (10) accordingly.

[0099] Actuator (10) receives control signal (A) and is controlled accordingly to perform an action corresponding to control signal (A). Actuator (10) may include control logic that converts control signal (A) into further control signals used to control actuator (10).

[0100] In further embodiments, the control system 40 may include the sensor 30. In yet other embodiments, the control system 40 may alternatively or additionally include the actuator 10.

[0101] In a further embodiment, the control system (40) may be envisaged to control the display (10a) instead of or in addition to the actuator (10).

[0102] Additionally, the control system (40) may include at least one processor (45) and at least one machine-readable storage medium (46) having stored thereon instructions that, when executed, cause the control system (40) to perform a method according to one aspect of the present invention.

[0103] FIG. 4 illustrates an embodiment in which the control system (40) is used to control an at least partially autonomous robot, such as an at least partially autonomous vehicle (100).

[0104] The sensors (30) may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more LiDAR sensors, some or all of which are preferably, but not necessarily, at least partially integrated into the autonomous robot (100).

[0105] The image segmentation system (60) may be configured to perform a semantic or instance segmentation of a neighborhood of the autonomous robot (100) based at least in part on the input image (x). If an anomalous region (i.e., a pixel or a connected component of pixels classified as anomalous) is detected in the input image (x), the control signal (A) may be determined according to this information, for example, to at least in part avoid paths that take the autonomous robot (100) through the anomalous region or to at least in part avoid paths that take the autonomous robot (100) closer to the anomalous region than a predefined threshold.

[0106] The actuator (10), which is preferably integrated into the vehicle (100), may be provided by the vehicle's (100) brakes, propulsion system, engine, drive train, or steering.

[0107] Alternatively or additionally, the control signal (A) may also be used to control the display (10a), for example to display anomalous pixels detected by the image segmentation system (60). The control signal (A) may also control the display (10a) to generate a warning signal when the vehicle (100) is close to an area containing anomalous pixels. The warning signal may be an audible warning and / or a haptic signal, for example a vibration on the steering wheel of the vehicle.

[0108] In further embodiments, the at least partially autonomous robot may be provided by another mobile robot (not shown), which can move, for example, by flying, swimming, diving or by stepping. The mobile robot may in particular be an at least partially autonomous lawnmower or an at least partially autonomous cleaning robot. In all of the above embodiments, the control signal (A) may be determined such that the propulsion unit and / or steering and / or braking of the mobile robot are controlled so that the mobile robot can avoid a collision with the identified object.

[0109] 5 illustrates an embodiment in which a control system (40) is used to control a manufacturing machine (11) of a manufacturing system (200), such as a punch cutter, cutter, gun drill, or gripper, for example as part of a production line. The manufacturing machine may include a conveying device (13), such as a conveyor belt or an assembly line, that moves a manufactured product (12). The control system (40) controls actuators (10), which in turn control the manufacturing machine (11).

[0110] The sensor (30) may be provided, for example, by an optical sensor that captures a characteristic of the manufactured product (12).

[0111] The image segmentation system 60 can determine the portion of the input image x that includes the manufactured product 12 device. The actuator 10 can then be controlled according to the position of the manufactured product 12 for a subsequent manufacturing step of the manufactured product 12. For example, the actuator 10 may be controlled to cut the manufactured product at a specific location on the manufactured product itself. If the output signal y identifies the presence of anomalous pixels on the conveyor 13 and / or the manufactured product 12, machine operation can be stopped and / or a human operator can be involved to evaluate the problem. Alternatively, the actuator 10 can be controlled to remove the manufactured product from the conveyor if an anomaly is detected in a pixel belonging to the manufactured product.

[0112] 6 shows an embodiment of a medical imaging system 500 controlled by the control system 40. The imaging system may be, for example, an MRI device, an X-ray imaging device, or an ultrasound imaging device. The sensor 30 may be, for example, an imaging sensor that acquires at least one image of the patient, for example, displaying different types of body tissue of the patient.

[0113] The image segmentation system (60) can be configured to detect different types of tissues in the sensed image, for example, by classifying tissues displayed in the image as either malignant or benign tissue. This can be done by semantic segmentation of the input image (x) by the image segmentation system (60). The control signal (A) can then be determined to cause the display (10a) to display the different tissues, for example, by displaying the input image (x) and coloring different regions of similar tissue type with the same color. The control signal (A) can further be selected to color abnormal regions with different colors to draw the physician's attention to the abnormal regions and inspect them in detail.

[0114] In further embodiments (not shown), the imaging system (500) may be used for non-medical purposes, for example to determine material properties of a workpiece. In these embodiments, the image segmentation system (60) may be configured to receive an input image (x) of at least a portion of a workpiece and perform a semantic segmentation of the input image (x) to thereby classify material properties of the workpiece. A control signal (A) may then be determined to cause the display (10a) to display information regarding the input image (x) and the detected material properties.

[0115] The term "computer" may be understood to encompass any device for processing predefined computational rules. These computational rules may be in the form of software, hardware, or a mixture of software and hardware.

[0116] In general, the plurality may be understood to be indexed, i.e., each element of the plurality is assigned a unique index, preferably by assigning consecutive integers to the elements contained in the plurality. Preferably, if the plurality includes N elements, N is the number of elements in the plurality, and the elements are assigned integers from 1 to N. It may also be understood that the elements of the plurality may be accessed by their index.

Claims

1. 1. A computer-implemented method for training an image segmentation system (60), the method comprising: Training image (x ｉ ) and the training label map (t ｉ ), obtaining the training label map (t ｉ ) is the training image (x ｉ ) as belonging to the abnormal class; The training image (x ｉ ) as an input to the image segmentation system (60), which segment the training images (x ｉ determining values ​​characterizing the logits of different classes of pixels of determining a loss value based on a loss function, the loss function being a function of the training images (x ｉ ), the logarithm of the sigmoid being the logarithm of the logits determined for the pixels in the training label map (t ｉ ) for pixels labeled as belonging to the anomalous class; training the image segmentation system (60) by minimizing the loss value; A method comprising:

2. The method of claim 1 , wherein the first term characterizes the logarithm of a sigmoid of a maximum logit of a given quantity.

3. The first loss term is characterized by the following equation: [0010] where i, j are the height and width coordinates of a pixel, ood is the pixel coordinate of a pixel marked as an anomaly, and N ｏｏｄ is the amount of pixels in ood, K is the amount of logits to determine the logarithm of the sigmoid, and S ｉ、ｊ is a set of K logits, [0025] The method of claim 1 or 2, wherein k is the kth logit in the set of logits for the pixel at location i,j.

4. The training image (x ｉ ) and the training label map (t ｉ ) is the training image (x ｉ ) and the training label map (t ｉ ), and the training images (x ｉ ) and the training label map (t ｉ ) is The first image (i １ ) and the first image (i １ ) corresponding to the first label map (l １ ) and a second image (i ２ ) and the second image (i ２ ) corresponding to the second label map (l ２ ) and The third image (i ３ ), wherein the third image is used to determine the second image (i ２ ), and the machine learning system (70) determines the first image (i １ ) determining an image in the style of The first image (i １ ) to the third image (i ３ ) by replacing the training images (x ｉ ), wherein said third image (i ３ ) is added to the second label map (l ２ ) based on a class from The first label map (l １ ) with a class label indicating an abnormal class, ｉ ), wherein said first image (i １ ) is replaced with a corresponding pixel in the first label map (l １ ) the class labels in The method of any one of claims 1 to 3, comprising:

5. 5. The method of claim 4, wherein permuting a class label from the first label map is accomplished by replacing a class label from the first label map with a class label from the second label map, the class label from the second label map being used in a substitution where a corresponding pixel was used as a substitution in the first image, and the permuted class label in the first label map is further denoted as a result of the substitution.

6. The method according to claim 4 or 5, wherein the first image is transformed with an augmentation operation before determining the training images.

7. 7. The method of claim 4, wherein the machine learning system determines its output based on providing its input to a masked noise encoder of the machine learning system and providing an output of the masked noise encoder to a StyleGAN2.

8. 8. The method of claim 4, wherein the first images are constituted by a first dataset of first images, and the machine learning system is trained to transfer an input image to a style of the first images.

9. 1. A computer-implemented method for determining an output signal (y), the output signal (y) characterizing a classification of pixels of an input image (x) as anomalous or not, the method comprising: providing said input image (x) as an input to an image segmentation system (60), said image segmentation system (60) having been trained with a method according to any one of claims 1 to 8; determining, by said image segmentation system (60), the logits of said pixels; providing a classification of the pixel as anomalous in the output signal (y) if the difference between the maximum determined logit and the minimum determined logit is less than or equal to a predefined threshold, and providing a classification of the pixel as non-anomalous in the output signal (y) otherwise; A method comprising:

10. A training system (140) configured to carry out the training method of any one of claims 1 to 8.

11. A control system (40) configured to implement the method according to claim 9, comprising: The control system (40) determines a control signal (A) based on the output signal (y); The control system (40), wherein the control signal (A) is configured to control an actuator (10) and / or a display (10a).

12. A computer program configured to cause a computer to carry out the method according to any one of claims 1 to 9, when said computer program is executed by a processor (45, 145), with all of the steps of said method.

13. A machine-readable storage medium (46, 146) having stored thereon a computer program according to claim 12.

Images