Generative adversarial network for processing and generating image and label map

Abstract

To relate to a generative adversarial network.SOLUTION: A generative adversarial network comprises: a generator configured for generating an image and a corresponding label map; and a discriminator configured for determining a classification of a provided image and a provided label map. The classification characterizes whether the provided image and the provided label map have been generated by the generator or not. Determining the classification comprises the steps of: determining a first feature map of the provided image; masking the first feature map according to the provided label map for determining the masked feature map; globally pooling the masked feature map for determining a feature representation of the provided image masked by the provided label map; and determining a classification of the image based on the feature representation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a generative adversarial network for processing and generating images, a method for training a generative adversarial network, a method for training a machine learning system, a method for classifying images by a generative adversarial network, a training system, a computer program and a machine-readable storage medium. [Background technology]

[0002] Prior art Zhang et al., "DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort" (April 20, 2021, https: / / arxiv.org / abs / 2104.06490v2), discloses a generative adversarial network for generating image and label maps.

[0003] Background technology Machine learning systems, such as neural networks, are used in almost every technical field. However, the performance of such systems depends critically on the data on which they are trained. Satisfactory generalization capabilities of machine learning systems can typically only be achieved when they are trained using large and diverse datasets.

[0004] Collecting such datasets is a challenging task, especially for computer vision tasks such as semantic segmentation or object detection, since labeling the required data requires a lot of expert labeling time. It is therefore desirable to devise a method that allows for the automatic generation of labels for semantic segmentation or object detection, and preferably also allows for the generation of images corresponding to the generated labels. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Zhang et al., “DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort” (April 20, 2021, https: / / arxiv.org / abs / 2104.06490v2) Summary of the Invention [Problem to be solved by the invention]

[0006] An advantage of the generative adversarial network proposed in this invention is that it enables accurate generation of images and their corresponding semantic segmentation or object detection labels. As an additional feature, the generative adversarial network allows accurate classification of whether a provided image and its corresponding provided label are from the distribution used to train the generative adversarial network. [Means for solving the problem]

[0007] Disclosure of the Invention In a first aspect, the present invention relates to a generative adversarial network. a generator configured to generate images and corresponding label maps; a classifier configured to determine an output characterizing a classification of the provided image and the provided label map; Including, The classification characterizes whether the provided image and the provided label map were generated by the generator, and determining the classification is determining a first feature map of the provided image; masking the first feature map according to the provided label map to determine a masked feature map; Globally pooling the masked feature maps to determine a feature representation of the provided image masked by the provided label map; determining a classification of the image based on the feature representation; Includes:

[0008] The generative adversarial network (GAN) proposed in this invention can be understood as a special form of GAN known in the prior art. It is a model from the field of machine learning, and this model includes two components: a generator and a discriminator. Compared to known GANs, the generator is configured not only to generate images, but also to generate a label map corresponding to each image.

[0009] A label map can be understood as characterizing the labels of an image to which the label map corresponds, with the term "corresponding" being understood as a one-to-one relationship between the label map and the image it characterizes. In other words, a label map can be understood as characterizing meta-information about an image. A label map can be understood, in particular, as characterizing the class membership of pixels of the image to which the label map corresponds. The label map can be provided, for example, in the form of a matrix with width and height equal to the image, where elements of the matrix characterize classes of pixels of the image. In this case, elements are characterized by their position along the width and height of the matrix, characterizing the class of pixels at the same position in the image. Alternatively, the label map can be provided by a tensor, preferably a three-dimensional tensor with a specific height, width, and depth. The height and width are preferably equal to the image, while the tensor preferably includes as many elements along its depth dimension as there are possible classes for pixels of the image. In this case, the tensor can encode the classes of pixels, in particular, by one-hot encoding. Here, a one-hot encoding is a slice of a tensor at a particular position along the width and height of the tensor, i.e., a one-hot encoding is a vector.

[0010] However, the label map may also be provided as a more abstract definition of class membership. For example, the label map may be provided by a bounding box or multiple bounding boxes, each bounding box characterizing a corresponding rectangular region of the image. Each bounding box may further characterize information about the class of the object depicted in the image at the location of each bounding box, including, for example, a class label or other information associated with the class. However, the label map need not be limited to a bounding box or multiple bounding boxes. The label map may generally be any type of polygon or multiple polygons of any type, preferably (but not necessarily) convex polygons. The polygons can be understood as indicating regions of the image that belong to a particular class. For example, objects in an image can be annotated using convex hulls (i.e., convex polygons).

[0011] When a label map characterizes one polygon or multiple polygons (including bounding boxes), pixels in the image corresponding to labels that are not included in any of the polygons characterized by the label map can be understood as a special class, typically referred to as the "background class."

[0012] If the label map characterizes a polygon or polygons, the label map may be converted into a label map characterizing a semantic segmentation by assigning each pixel in the image corresponding to this label map to the class of the polygon in which it is contained, or to the background class if it is not contained in any polygon.

[0013] The GAN further includes a classifier configured to determine an output characterizing a classification of the provided image and the provided label map. In other words, the classifier is configured to classify the provided image and the provided label map. As is common in GANs, the classifier attempts to classify whether its input (i.e., the provided image and the provided label map) was generated by the generator. In general GAN ​​terminology, the class referring to the image and the label map generated by the generator may be referred to as a "fake class," while the other class may be referred to as a "true class" or "real class." The classifier is configured to determine a first feature map based on the image. The first feature map may, for example, be the image itself. However, preferably, the first feature map may be the output of at least one convolutional layer, preferably multiple convolutional layers. For example, the classifier may include a block of convolutional layers connected as a feedforward neural network. This block may be understood as a sub-neural network of the classifier. The sub-neural network takes the provided image as input and determines a feature map from it.

[0014] A feature map may be understood as characterizing information about an image. The feature map may preferably be provided in the form of a three-dimensional tensor. The tensor includes height, width, and depth axes. The tensor may be understood as including vectors (along the depth axis) located at specific spatial positions along the height and width axes of the tensor. In this sense, a first feature map may be provided by a three-dimensional tensor having a width and height equivalent to the provided image. A vector at a specific spatial position of the tensor can then be understood as a feature vector characterizing a pixel of the image, where the image has the same spatial position in the image as that having this feature vector in the tensor (this is indicated, for example, by coordinates along the width and height axes of the tensor and image, respectively).

[0015] In the following, when referring to feature maps and their tensor features, it is also understood that when an image is used as a feature map, the feature map can still be understood as a tensor with as many elements along the depth dimension as there are channels in the image.

[0016] The classifier is configured to mask the first feature map according to the provided label map. In this sense, masking can be understood as setting certain elements of the first feature map to a predetermined value, preferably 0. In particular, the first feature map may be masked by setting all elements of the first feature map that do not correspond to a class characterized by the provided label map to a predetermined value. This may technically be achieved by first determining a matrix of the same height and width as the provided image, where an element of the matrix is ​​set to 1 if a pixel having the same spatial position in the image as the element of the matrix belongs to the class. Otherwise, the element may be set to 0. In this way, a matrix consisting only of 0s and 1s is constructed. The first feature map and the matrix may then be multiplied element-wise to determine the masked feature map. This element-wise multiplication may in particular be copied along the depth dimension of the first feature map (also known as a matrix broadcast over the first feature map).

[0017] The masked feature map is then processed by a global pooling operation. The masked feature map can be understood as a three-dimensional tensor. It can also be understood as multiple matrices stacked along the depth dimension of the tensor. The global pooling operation can be understood as extracting a feature representation by extracting one value from each matrix of multiple matrices. In other words, global pooling can be understood as a pooling operation in which the received field is the entire input. The pooling operation can be particularly realized as a pooling layer of a neural network. The global pooling operation may preferably be a global average pooling operation. Other possible global pooling operations include global max pooling.

[0018] An advantage of the proposed GAN is that the specific design of the generator and classifier allows the GAN to generate images that look like other images from the training dataset on which it was trained, and also to generate highly accurate label maps, i.e., annotations for the generated images. As an additional benefit, this design of the classifier allows it to accurately predict whether a provided image, in combination with a provided label map, looks like data from the training dataset. This can be particularly advantageous for detecting, for example, whether a provided image, combined with a label map, characterizes an image identified from a sensor or whether this image characterizes a tampered image. The inventors have found that the high accuracy of the classifier's predictions is due to its design, i.e., its ability to mask elements according to the provided label map, thereby guiding the classification process to view the object suggested by the provided label map. In this way, classification is based on more information than just the image.

[0019] It should be noted that the proposed GAN can be understood as a trained GAN, specifically, that is, the generator and discriminator of the GAN are trained to achieve their desired objectives.

[0020] Preferably, the generator and the classifier are realized as neural networks. Since GANs are configured to generate and / or classify images, the generator and the classifier may preferably be realized as convolutional neural networks.

[0021] The inventors have found that the accuracy of the generated images and generated label maps as well as the classification accuracy of the classifier can be improved if the generator and / or classifier are implemented as neural networks, in particular deep neural networks.

[0022] The generation of an image and corresponding label map by the generator may preferably be achieved by providing at least one random number, preferably a plurality of random numbers, e.g., in the form of a vector, matrix or tensor, as input to the generator, and determining the image and label map as output of the generator given this input.

[0023] The provided label map characterizes a semantic segmentation of the provided image, and a masked feature map can be determined for the classes characterized by the semantic segmentation. Preferably, a masked feature map is determined for each class characterized by the semantic segmentation.

[0024] This embodiment can be understood as determining at least one, but preferably multiple, masked feature maps, i.e., one masked feature map for each class present in the provided label map. The classification of the classifier can then be based on whether each of the masked feature maps characterizes the provided image and the provided label map as originating from the generator. By determining a masked feature map for each class, accuracy is further improved because the provided label map should be accurate for all classes, not just one or a subset of classes. During training of the GAN, this guides the generator to generate more accurate label maps.

[0025] It is also possible that the provided label map characterizes regions of the provided image and a masked feature map is determined for classes characterized by these regions. Preferably, a masked feature map is determined for each class characterized by these regions.

[0026] The provided label map can, for example, characterize polygonal regions of an image that belong to a particular class, a typical example of which is a bounding box, where the bounding box carries additional information about the class of the object enclosed by the bounding box in the image.

[0027] The advantage of this embodiment is similar to the embodiment that uses semantic segmentation as the provided label map: the classifier can base its classification on whether each of the masked feature maps characterizes the provided image and the provided label map as coming from the generator.

[0028] In general, it is possible, but not necessary, for the generator to generate a label map of the same type that the classifier is configured to process. For example, the generator may generate a label map in the form of a semantic segmentation, and the classifier may be configured to process a label map provided in the form of a semantic segmentation (similarly in the case of a label map characterizing a region). However, it is also possible for the generator to generate a label map of a different type than the label map for which the classifier is configured. In this case, the generated label map may be processed before being used as input to the classifier. For example, if the generator generates a label map characterizing a region, the label map characterizing the semantic segmentation may be obtained by setting each element of the semantic segmentation to a class characterized by a region in the generated label map. Similarly, the generated label map characterizing the semantic segmentation may be converted to a label map characterizing a region by determining the convex hull of the connected elements of the semantic segmentation.

[0029] In other words, both the provided label maps characterizing regions and the provided label maps characterizing semantic segmentations can be understood as characterizing class membership of pixels of the provided image.

[0030] The masked feature map may be determined specifically for classes characterized by semantic segmentation, or may be characterized by regions by setting all pixels of the first feature map that do not belong to a class to 0.

[0031] A pixel of the first feature map may be understood as a vector of a tensor characterizing the first feature map. The vector is arranged along the depth dimension of the tensor and has a spatial position relative to the width and height of the tensor. The expression "pixel of a tensor" may therefore be understood as an analogy to "pixel of an image." Masking of the first feature map may then be achieved by setting a particular pixel of the tensor to a predetermined value. Setting a pixel to a predetermined value may be understood as setting an element of a vector to a predetermined value, in particular setting all elements of a vector to a predetermined value. Advantageously, if the predetermined value is 0, subsequent operations will implicitly consider each pixel as masked. Therefore, consideration is given to how the classifier is implemented on a computer.

[0032] In other embodiments, the classifier may be further configured to determine the classification based on a second feature map, where the second feature map is determined by applying a 1×1 convolution to the first feature map.

[0033] These embodiments can be understood as allowing the classifier to determine its classification based on the layout of the provided image. Preferably, the second feature map is the result of applying a single 1×1 convolution to the first feature map. The application of the 1×1 convolution may be preferably achieved by a convolutional layer including a filter (preferably only a single filter) with a 1×1 kernel size. By applying the 1×1 convolution, the information of the first feature map is compressed. The inventors have found that this compression also allows the classifier to derive information about the layout of the provided image in addition to content information derived from masking the first feature map. In other words, the classifier can determine that a particular masked feature map may essentially characterize a meaningful object, but its position in the provided image may be significantly incorrect (e.g., a floating car) with respect to the images provided in the training dataset used to train the GAN. The layouts may be classified separately by the classifier, i.e., the classifier may output two classifications, one classification characterizing whether the masked feature map characterizes content belonging to a fake class or a real class, and the other classification characterizing whether the layout characterizes content from a fake class or a real class.

[0034] The inventors have found that by using 1x1 convolutions to determine classification based on the layout of the provided image, the classification accuracy of the classifier is further improved.

[0035] In another aspect, the present invention relates to a computer-implemented method for training a generative adversarial network according to an embodiment of the first aspect, wherein the training comprises: generating a first image and a corresponding first label map from a generator of a generative adversarial network; determining, by a classifier of the generative adversarial network, a first output characterizing a classification of the first image and the first label map; training a classifier to classify the first image and the first label map into a first class that characterizes the image and the label map generated by the generator based on the first output; training the generator based on the first output to generate images classified into a second class and corresponding label maps that characterize images and label maps not generated by the generator; determining, by a classifier, a second output characterizing a classification of a provided second image and a provided second label map, the second image and the second label map not being provided by a generator; training a classifier to classify the second image and the second label map into a second class based on the second output; Includes:

[0036] The second image and second label map may be understood to arise specifically from a training dataset for the GAN. Training a GAN may be understood as teaching the GAN to learn about the probability distributions of the images and corresponding label maps in the training dataset, thereby enabling the GAN to generate images and label maps that "look" like data from the training dataset.

[0037] The training method can be understood as a common zero-sum game approach for training a generative adversarial network. However, due to the advantageous design of a GAN (i.e., its unique generator and its unique discriminator), this training method determines a GAN that is more accurate in generating images and corresponding label maps, as well as in classifying provided images and provided label maps. Because training is performed as a zero-sum game, improving the classification accuracy of the discriminator directly improves the accuracy of the images and label maps generated by the generator. This is due to the fact that to maintain balance in the zero-sum game, the generator must be able to determine more accurate outputs for the training dataset.

[0038] Training of the GAN may preferably be accomplished by a (preferably stochastic) gradient descent algorithm using the negative log-likelihood of the classifier's classification as a loss function. If the classifier determines multiple classifications, for example, one classification per masked feature map or a classification for a second feature map, the negative log-likelihood values ​​determined for each classification may be averaged to determine the loss value of the gradient descent algorithm.

[0039] Advantageously, the inventors have found that the unique design of GANs allows them to be trained even with very limited amounts of training data: in fact, GANs can be trained with a single image without experiencing common problems with such small training datasets, such as overfitting or mode collapse.

[0040] In another aspect, the present invention relates to a computer-implemented method for training or testing a machine learning system, the method comprising: determining images and corresponding label maps from a generator of a generative adversarial network according to a GAN embodiment of the present invention; training a machine learning system to determine an output that characterizes the label map when provided with an image as input, or testing the machine learning system as to the extent to which its output characterizes the label map when provided with an image as input; Includes:

[0041] The method for training may be understood as a supervised training method that uses generated (sometimes referred to as synthetic) images to train a machine learning system, which may be, for example, a neural network trained for image analysis.

[0042] The method for training may preferably include training the machine learning system using the image and label maps from the training dataset and the image and label maps generated by the generator of the GAN, preferably the GAN is previously trained with the training dataset to generate the image and label maps.

[0043] This training method improves the accuracy of the machine learning system because it allows for an automated approach to obtain more image and label maps from the GAN.

[0044] When used for testing, this method may be understood as generating a test dataset for the machine learning system being tested. The degree to which the output of the machine learning system characterizes the label map may be understood as a metric for evaluating the performance of the machine learning system. For example, if the machine learning system is configured for semantic segmentation, mean intersection over union may be used as the metric. If the machine learning system is configured for object detection, mean average precision may be used as the metric.

[0045] In another aspect, the present invention relates to a computer-implemented method for classifying images and corresponding label maps, wherein the images and corresponding label maps are classified by a generative adversarial network classifier according to an embodiment of the first aspect of the present invention.

[0046] The method for classifying an image may be understood as evaluating whether the image and corresponding label map are part of a real class, i.e., whether they are likely to be from a probability distribution characterized by a training dataset of the GAN. The method for classification may, for example, be used to detect adversarial samples. For example, a machine learning system may be configured to determine a label map for the image. The image and the determined label map may then be forwarded to a classifier. If the classifier determines that the image and the determined label map belong to a real class, the label map may be further processed, for example, by an autonomous device. Otherwise, the determined label map may be rejected or ignored in further processing.

[0047] Embodiments of the present invention will now be described in more detail with reference to the following drawings. [Brief explanation of the drawings]

[0048] [Figure 1] FIG. 1 illustrates a generative adversarial network. [Figure 2] FIG. 1 illustrates a method for training a generative adversarial network. [Figure 3] FIG. 1 illustrates a control system including a generative adversarial network. [Figure 4] FIG. 1 illustrates a control system for controlling an at least partially autonomous robot. DETAILED DESCRIPTION OF THE INVENTION

[0049] Description of the embodiment 1 illustrates a generative adversarial network (GAN). The GAN includes a generator (71) configured to determine an image (711) and a corresponding label map (712) as output based on randomly drawn values ​​(R) used as input. The randomly drawn values ​​(R) may be part of a plurality of randomly drawn values ​​used as input to the generator (71), e.g., in the form of a vector, matrix, or tensor. Determining an output from the generator (71) may be referred to as generating an output. The generator (71) may preferably be implemented as a neural network.

[0050] The GAN further includes a classifier (72), which accepts a provided image (711) and a provided label map (712) and determines a classification (y1, y2, y3) of the provided image and the provided label map. n ,y l ) The classifier (72) is preferably implemented by a neural network.

[0051] For this purpose, the classifier (71) may include an optional first unit (721), which is configured to determine a first feature map (F1) based on the provided image (711). The first unit (721) may be, in particular, a neural network, in particular a convolutional neural network. If the first unit (721) is a neural network, the first unit (721) may process the provided image (711) by transferring the provided image (711) to its layer. The output thus determined may then be used as the first feature map (F1). Alternatively, the provided image (711) may be directly used as the first feature map (F1).

[0052] Next, the first feature map (F1) is masked according to the provided label map (712). In this embodiment, the provided label map (712) characterizes the semantic segmentation. In particular, the provided label map (712) can characterize a tensor of one-hot encodings of classes of pixels. The one-hot encodings may in particular be pixels of the tensor, i.e., arranged along the depth dimension of the tensor and having spatial positions along the width and height of the tensor. For masking, the tensor can be sliced ​​along the depth dimension to extract various matrices. Each matrix consists of 0s and 1s. Each of these matrices corresponds to a class characterized by the provided label map. The various matrices may be understood as various masks, where there is a mask for each class. Each mask is then used in the masking operation (726) of the classifier (72). The result of this masking operation (726) is preferably a masked feature map (M1, M2) for each mask, i.e., a masked feature map (M1, M2) for each class.

[0053] If the provided label map (712) characterizes a matrix of class indices, the provided label map may be converted to a one-hot encoding tensor before the masking operation. Similarly, if the provided label map (712) characterizes a region, e.g., a polygonal region such as a bounding box, the provided label map (712) may be converted to one-hot encoding before the masking operation. To this end, each pixel in the provided image (711) may be assigned a class according to the region in which the pixel is located. If a pixel is not located in any region characterized by the provided label map (712), the pixel may be assigned to a "background" class. In this manner, a semantic segmentation is determined from the regions characterized by the provided label map (712). The semantic segmentation may then be used for masking as described above.

[0054] The determined masked feature maps (M1, M2) are then processed by a global pooling operation. The global pooling operation may preferably be a global average pooling operation. In another embodiment, the global pooling operation may be a global max pooling operation. The result of the global pooling operation may be understood as a feature representation that characterizes the provided input image (711) masked according to the class of the provided label map (712). Preferably, the classifier (72) determines a feature representation for each of the masked feature maps (M1, M2).

[0055] The feature representations are then processed by a second unit (725) of the classifier (72). The second unit (725) may in particular be a neural network, in particular a multi-layer perceptron, i.e. a fully connected neural network. For each feature representation, the second unit (725) generates a classification (y1, y2, yn ), each classification characterizing whether each feature representation characterizes a real class or a fake class. The second unit (725) may, in particular, perform multi-class classification, where the fake class is one of multiple classes that can be predicted by the second unit (725). In this embodiment, since the provided image (711) and the provided label map (712) originate from the generator (71), it is desirable for the second unit (725) to predict a fake class for each feature representation. In other embodiments, the provided image (711) and the provided label map (712) may originate, for example, from a training dataset used to train the generative adversarial network (70). In this case, it is desirable for the second unit (725) to predict that the feature representation is included in a real class. When the second unit (725) is configured to perform multi-class classification, it is desirable for the second unit (725) to predict a class that characterizes a class used for masking in the process of determining the feature representation. Then, the classification (y1, y2, y n ) is provided as the output (y) of the classifier (72).

[0056] In another embodiment, the classifier (72) may include an operational unit for evaluating whether the layout of the provided image indicates that the image was provided by the generator (71). Preferably, the classifier (72) includes a convolutional layer including a single filter with a kernel size of 1×1. The convolutional layer processes the first feature map (F1) and thereby determines a second feature map (F2). The second feature map (F2) may then be used as input to a third unit (723), which is preferably a neural network, in particular a convolutional neural network. The third unit (723) receives the second feature map (F2) as input and characterizes the second feature map (F2), thereby generating a classification (y) characterizing the layout of the provided image (711). l ) is determined. l ) may be specifically a binary classification characterizing either real or fake classes. l ) may then be provided at the output (y) of the classifier (72).

[0057] FIG. 2 illustrates a schematic of a method (100) for training a generative adversarial network (70).

[0058] In a first step (101), a generator (71) is provided as input with a vector of randomly drawn values ​​(R) and determines an output that characterizes a first image (711) and a first label map (712).

[0059] In the second step (102), the classifier (72) generates one classification (y1, y2, y n ,y l ), possibly multiple classifications (y1, y2, y n ,y l ) to determine the output (y) that characterizes the

[0060] Then, in a third step (103), a classifier (72) is trained to classify the first image (711) and the first label map (712) into fake classes. This is advantageously achieved by a gradient descent algorithm. Each classification (y1, y2, y n ,y l ) may be provided to each loss function using the fake class as the desired class for each loss function. For layout-related classification, binary cross-entropy loss may be used as the loss function, while for other classifications characterized by output, multinomial cross-entropy loss may be used as the loss function. Each loss function determines a loss value. The loss values ​​may then be aggregated into a single loss value using a weighted sum. This single loss value can then be used as the loss value for the gradient descent algorithm. Based on this loss value, gradients for parameters of the first unit (721) and / or the second unit (725) and / or the convolutional layer (722) and / or the third unit (723) may then be determined, for example, by automatic differentiation. These parameters may then be updated according to the gradients.

[0061] In a fourth step (104), the parameters of the generator (71) are updated based on this one loss value. To this end, the gradient of this one loss value with respect to the parameters of the generator (71) is determined. This may also be achieved by automatic differentiation. Then, to train the generator (71), the parameters of the generator (71) may be updated according to the positive direction of the gradient, i.e., by gradient ascent.

[0062] In a fifth step (105), the classifier (72) is provided with a second image from the training dataset and a corresponding second label, and the classifier then determines an output for the second image and the second label map.

[0063] In a sixth step (106), the classifier (72) is trained to classify the second image and the second label map into real classes. This is preferably achieved by a gradient descent algorithm. Each classification characterized by the output for the second image and the second label map may be provided to a respective loss function using the class of the mask used to determine a feature representation for each desired class of feature representation. For the layout classification, a binary cross-entropy loss may be used as the loss function, and for the other classifications characterized by the output, a multinomial cross-entropy loss may be used as the loss function. Each loss function determines a loss value. The loss values ​​may then be aggregated into a single loss value using a weighted sum. In this case, the single loss value may be used as the loss value for the gradient descent algorithm. Based on the loss values, gradients of parameters for the first unit (721) and / or the second unit (725) and / or the convolutional layer (722) and / or the third unit (723) may then be determined, for example, by automatic differentiation. The parameters can then be updated according to the gradient.

[0064] Steps 101 through 106 may then be repeated a predetermined number of iterations. Alternatively, training may be terminated when a predetermined performance metric, such as bits per dimension, falls below a predetermined threshold.

[0065] 3 shows an embodiment of an actuator 10 in its environment 20. The actuator 10 interacts with a control system 40. The actuator 10 and its environment 20 together are referred to as the actuator system. A sensor 30 senses the state of the actuator system, preferably at equally spaced time points. The sensor 30 may include multiple sensors. The sensor 30 is an optical sensor that images the environment 20. An output signal S of the sensor 30 (or the output signal S of each sensor, if the sensor 30 comprises multiple sensors) encoding the sensed state is transmitted to the control system 40.

[0066] Thereby, the control system 40 receives the stream of sensor signals S. The control system then calculates a series of control signals A in response to the stream of sensor signals S, which are then transmitted to the actuator 10.

[0067] The control system 40 receives a stream of sensor signals (S) from the sensors 30 at an optional receiving unit 50. The receiving unit 50 converts the sensor signals (S) into input signals (x). Optionally, in the absence of a receiving unit 50, each sensor signal (S) may be directly incorporated as an input signal (x). The input signal (x) may be provided, for example, as an excerpt of the sensor signals (S). Optionally, the sensor signals (S) may be processed to produce the input signals (x). In other words, the input signals (x) are provided in accordance with the sensor signals (S).

[0068] The input signal (x) is then sent to a classifier (60) configured for semantic segmentation or object detection.

[0069] The classifier (60) is parameterized by provided parameters (Φ) stored in a parameter store (St1).

[0070] The classifier (60) determines an output signal (o) from the input signal (x), which characterizes the semantic segmentation or object detection of the input signal (x). The output signal (o) is transmitted to a transformation unit (80), which transforms the output signal (o) into a control signal (A).

[0071] A control signal (A) is then transmitted to the actuator (10) to control the actuator (10) accordingly.

[0072] The input signal (x) and the output signal (o) are also provided to the generative adversarial network (70). The generative adversarial network (70) evaluates whether the input signal (x) and the output signal (o) characterize “real data,” i.e., the data used to train the classifier (60). For this purpose, the generative adversarial network (70) is trained with the same data as the classifier (60). In other words, the generative adversarial network (70) knows what the input signal (x) and the output signal (o) should look like. If the input signal (x) and the output signal (o) are classified as “fake data,” this indicates that the data obtained from the sensor (30) may be critical, e.g., data not intended for training the classifier (60) and therefore good classification results cannot be expected for this data, or may be intentionally malicious data, such as adversarial samples, and / or the classification determined by the classifier (60) is inaccurate or false. The generative adversarial network classifies an input signal (x) and an output signal (o) by providing them to its classifier (72), and if the output signal (o) is not a semantic segmentation map in one-hot encoding, it preprocesses the output signal (o). The output (y) of the classifier (72) is then provided as the output of the generative adversarial network (70) and is also forwarded to the transformation unit (80).

[0073] The actuator 10 receives the control signal A from the conversion unit 80 and is controlled accordingly to perform an action corresponding to the control signal A. The actuator 10 may include control logic that converts the control signal A into another control signal that is then used to control the actuator 10.

[0074] In other embodiments, the control system 40 may include the sensor 30. In still other embodiments, the control system 40 may alternatively or additionally include the actuator 10.

[0075] In yet another embodiment, it is envisioned that the control system 40 controls the display 10a instead of or in addition to the actuator 10.

[0076] Additionally, the control system (40) may include at least one processor (45) and at least one machine-readable storage medium (46) having stored thereon instructions that, when executed, cause the control system (40) to perform methods according to aspects of the present invention.

[0077] FIG. 4 illustrates an embodiment in which the control system (40) is used to control an at least partially autonomous robot, such as an at least partially autonomous vehicle (200).

[0078] The sensors 30 may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more LiDAR sensors, some or all of which are preferably, but not necessarily, mounted on the vehicle 200. Thus, the input signal x may be understood as an input image, and the classifier 60 may be understood as an image classifier.

[0079] The image classifier (60) may be configured to detect an object in the vicinity of the at least partially autonomous robot based on an input image (x). The output signal (y) may include information characterizing where the object is located in the vicinity of the at least partially autonomous robot. A control signal (A) may then be determined according to this information, for example, to avoid a collision with the detected object.

[0080] Preferably, the actuators 10 on the vehicle 200 may be provided by the brakes, propulsion system, engine, drivetrain, or steering of the vehicle 200. The control signal A may be determined to control the actuators 10 to cause the vehicle 200 to avoid a collision with the detected object. The detected object may be classified according to what the image classifier 60 determines to be most likely, e.g., a pedestrian or a tree, and the control signal A may be determined in response to that classification.

[0081] The classification (y1, y2, y n ,y l) characterizes a fake class, the autonomous vehicle 200 may be controlled accordingly. This may mean handing over control to the driver or operator of the vehicle 200, assuming a safe condition by, for example, stopping in an emergency lane, slowing down the vehicle 200, or sending the input signals (x) and / or output signals (o) to a designated location, such as a control center, for analysis of the input signals (x) and / or output signals (o). Suitable control of the vehicle 200, as exemplified above, involves detecting at least a predetermined amount of classifications (y1, y2, y3) contained in the output (y) of the generative adversarial network 70. n ,y l ) may also be triggered if it characterizes a fake class.

[0082] Alternatively or additionally, the control signal (A) may be used to control the display (10a), for example to display the objects detected by the image classifier (60). It is also envisioned that the control signal (A) may control the display (10a) to generate a warning signal if the vehicle (200) is close to colliding with at least one of the detected objects. The warning signal may be an audible warning and / or a haptic signal, for example, a vibration in the vehicle's steering wheel.

[0083] Also, the classifications (y1, y2, y3) included in the output (y) of the generative adversarial network (70) may be corrected, for example, by an appropriate warning message stating that the detected environment may be unreliable. n ,y l ) characterizes a fake class, the driver or operator may be notified by a display.

[0084] In other embodiments, the at least partially autonomous robot may be provided by another mobile robot (not shown), which may move by, for example, flying, swimming, diving or walking. The mobile robot may in particular be an at least partially autonomous lawn mower or an at least partially autonomous cleaning robot. In all of the above embodiments, the control signal (A) may be determined such that the propulsion unit and / or steering and / or braking of the mobile robot are controlled so that the mobile robot can avoid a collision with the identified object.

[0085] In another embodiment, the at least partially autonomous robot may be provided by a gardening robot (not shown) that uses sensors 30, preferably optical sensors, to determine the condition of plants in the environment 20. The actuators 10 may control nozzles for spraying liquid and / or cutting devices, such as blades. Depending on the identified species and / or the identified condition of the plants, a control signal A may be determined such that the actuators 10 spray the appropriate amount of liquid on the plants and / or cut the plants.

[0086] In yet another embodiment, the at least partially autonomous robot may be provided by a home appliance (not shown), such as a washing machine, stove, oven, microwave, or dishwasher. A sensor (30), such as an optical sensor, may detect the state of an object being processed by the home appliance. For example, if the home appliance is a washing machine, the sensor (30) may detect the state of laundry in the washing machine. A control signal (A) may then be determined depending on the detected material of the laundry.

[0087] The term "computer" may be understood to encompass any device for processing predetermined computational rules, which may be in the form of software, hardware, or a mixture of software and hardware.

[0088] In general, the plurality of elements may be understood to be indexed, preferably by assigning consecutive integers to elements within the plurality of elements, where each element within the plurality of elements is assigned a unique index. Preferably, if the plurality of elements includes N elements, where N is the number of elements in the plurality of elements, the elements are assigned integers from 1 to N. It may also be understood that the plurality of elements are accessible by their indexes.

Claims

1. - A generator (71) configured to generate an image (711) and a corresponding label map (712); ・ Classification of the provided image and the provided label map (y １ , y ２ , y ｎ ) to determine an output (y) characterizing the identifier (72) configured to A adversarial generation network (70) comprising: The classification (y １ , y ２ , y ｎ ) characterizes whether the provided image and the provided label map were generated by the generator, and determining the classification (y １ , y ２ , y ｎ， y ｌ ) is ・Determining a first feature map (F １ ) of the provided image; · To determine the masked feature map (M １ , M ２ ), masking the first feature map according to the provided label map (726); ・To determine the feature representation of the provided image masked by the provided label map, the masked feature map (M １ , M ２ ) is globally pooled (724). - Classifying the image (y １ , y ２ , y ｎ ) based on the feature expression, and comprising the adversarial generation network (70).

2. The provided label map characterizes the semantic segmentation of the provided image, and the masked feature map (M １ , M ２ ) is determined for the classes characterized by the semantic segmentation, the adversarial generation network (70) according to claim 1.

3. The adversarial generation network (70) according to claim 1, wherein the provided label map characterizes regions of the provided image, and the masked feature map is determined for a class characterized by the regions.

4. The identifier is further configured to determine the classification based on a second feature map (F ２ ), and the second feature map (F ２ ) is determined by applying a 1×1 convolution (722) to the first feature map (F １ ). The adversarial generation network (70) according to claim 1.

5. The adversarial generation network (70) according to claim 1, wherein the provided label map characterizes pixel class membership of the provided image.

6. The masked feature map (M １ , M ２ ) is determined for the classes characterized by the semantic segmentation or, by setting to zero the pixels of the first feature map (F １ ) that do not belong to the classes, an adversarial generation network (70) according to claim 5, characterized by a region.

7. The adversarial generation network (70) according to claim 1, wherein the generator (71) is configured to generate the image and the corresponding label map based on a randomly drawn value (R).

8. The adversarial generation network (70) according to claim 1, wherein the generator (71) and / or the discriminator (72) characterize a convolutional neural network.

9. A computer-implemented method (100) for training the adversarial generation network according to claim 1, the training comprising: - generating, from the generator (71) of the adversarial generation network (70), a first image (711) and a corresponding first label map (712) (step 101); ・A step (102) of determining a first output (y) characterizing the classification (y １ , y ２ , y ｎ , y ｌ ) of the first image (711) and the first label map (712) by the discriminator (72) of the adversarial generation network (70); - training the discriminator (72) to classify the first image (711) and the first label map into a first class characterizing the image and label map generated by the generator based on the first output (y) (step 103); - training the generator (71) to generate an image and a corresponding label map classified into a second class characterizing an image and label map not generated by the generator (71) based on the first output (step 104); - determining, by the discriminator (72), a second output characterizing the classification of a provided second image and a provided second label map, wherein the second image and the second label map are not provided by the generator (71) (step 105). - A step (106) of training the discriminator (72) to classify the second image and the second label map into the second class based on the second output; including A computer-implemented method (100) for training an adversarial generation network. **Claim 10** A computer-implemented method for training or testing a machine learning system, - A step of determining an image and a corresponding label map from the generator (71) of the adversarial generation network (70) according to Claim 1; - A step of training the machine learning system to determine an output characterizing the label map when the image is provided as an input, or a step of testing the machine learning system regarding to what extent the output of the machine learning system characterizes the label map when the image is provided as an input; including a computer-implemented method. **Claim 11** A computer-implemented method for classifying an image and a corresponding label map, wherein the image and the corresponding label map are classified by the discriminator (72) of the adversarial generation network (70) according to Claim 1. **Claim 12** The method according to Claim 11, wherein an actuator (10) and / or a display (10a) are controlled based on the classification of the adversarial generation network (70). **Claim 13** A training system (140) configured to implement the training method according to Claim 9 or 10. **Claim 14** A computer program configured to cause a computer to perform all steps of the method according to any one of Claims 9 to 12 when executed by a processor (45). **Claim 15** A machine-readable storage medium (46) storing the computer program according to Claim 14.

Images