Apparatus and method for evaluating saliency map determiner

Abstract

Apparatuses and methods for evaluating a saliency map determiner are provided. According to various embodiments, a method of evaluating a saliency map determiner is described, the method comprising: adding a predefined pattern to a plurality of training dataset units to train identification of a data class, wherein each training dataset unit comprises a representation of the data class to be identified; training a neural network with the plurality of training dataset units comprising the predefined pattern; determining, by the saliency map determiner, a saliency map for the data class; and evaluating the saliency map determiner based on whether the determined saliency map comprises a context of the data class introduced by the addition of the predefined pattern.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a device and a method for evaluating a saliency map determiner. BACKGROUND

[0002] Neural networks are becoming more and more widely used for classifying images into a predefined number of classes. Understanding how a neural network arrives at its results can greatly help in determining how confident the classification is, but well-known difficulties.

[0003] The publication "Deep inside convolutional networks: Visualising image classification models and saliency maps" by Karen Simonyan, Andrea Vedaldi and Andrew Zisserman in the 2013 International Conference on Learning Representations (ICLR) describes interpreting the network decision process by generating low-level visual explanations. Such and similar approaches mainly focus on the task of image classification and can be divided into two categories: gradient-based methods and perturbation-based methods.

[0004] Gradient-based methods compute saliency maps visualizing the sensitivity of each image pixel to the prediction of a specific class, which is obtained by backpropagating the gradient of the prediction with respect to the image and estimating how moving along the gradient affects the class output. Gradient-based methods mainly rely on heuristics for backpropagation and can provide explanations that are not faithful to the model or the data. Perturbation-based methods evaluate the change in class prediction with respect to a perturbed image, e.g. a specific region of the image is replaced with mean image values or removed by applying a blur or Gaussian noise.

[0005] Since not all approaches for saliency determination perform equally well, e.g. gradient-based methods are typically susceptible to noise, it is desirable to allow for an approach to evaluate saliency determination methods. SUMMARY

[0006] The method and device having the features of the invention allow for an evaluation of saliency determination approaches, i.e. a verification whether the saliency determination indeed identifies saliency (e.g. the context considered by a neural network for a classification task).

[0007] Further examples are described in the following.

[0008] A method of evaluating a saliency map determiner performed by one or more processors can include adding a predefined pattern to a plurality of training data set units to train identification of a data class, wherein each training data set unit includes a representation of the data class to be identified; training a neural network with the plurality of training data set units including the predefined pattern, the saliency map determiner determining a saliency map for the data class, and evaluating the saliency map determiner based on whether the determined saliency map includes a context of the data class introduced by the addition of the predefined pattern. The method referred to in this paragraph provides a first example.

[0009] The adding of the predefined pattern to the plurality of training data set units can include adding the predefined pattern to training data that is interrelated to the data class. The feature referred to in this paragraph in combination with the first example provides a second example.

[0010] The method can include evaluating the saliency map determiner based on a comparison of a context of the data class of the neural network trained on the training data without adding the predefined pattern that is interrelated to the data class and a context of the data class of the neural network trained on the training data with the addition of the predefined pattern that is interrelated to the data class. The feature referred to in this paragraph in combination with any one of the first example to the second example provides a third example.

[0011] The method can include evaluating the saliency map determiner based on an overlap of a region of a sensor data set unit representing an instance of the data class and a region of the saliency map for the instance of the data class of the sensor data set unit. The feature referred to in this paragraph in combination with any one of the first example to the third example provides a fourth example.

[0012] The method can include evaluating the saliency map determiner based on a context specified by the saliency map outside of a requested region. The feature referred to in this paragraph in combination with any one of the first example to the fourth example provides a fifth example.

[0013] The evaluating of the saliency map determiner based on whether the determined saliency map includes the context of the data class introduced by the addition of the predefined pattern can include determining whether the saliency map includes a representation of the predefined pattern and evaluating the saliency map determiner based on whether the determined saliency map includes the representation of the predefined pattern. The feature referred to in this paragraph in combination with any one of the first example to the fifth example provides a sixth example.

[0014] The plurality of training data set units can be part of a training data set including a plurality of further training data set units, and the neural network is trained with the training data set. The feature referred to in this paragraph in combination with any one of the first example to the sixth example provides a seventh example.

[0015] It is possible that at least some of the further training data set units do not contain a data class. The feature mentioned in this paragraph in combination with the seventh example provides an eighth example.

[0016] At least some of the further training data set units can contain a further data class. The feature mentioned in this paragraph in combination with any one of the seventh to eighth examples provides a ninth example.

[0017] The method can comprise, after training the neural network, applying the neural network on a sensor data set unit comprising instances of the data class and the pattern, and determining a saliency map based on an output of the neural network. The feature mentioned in this paragraph in combination with any one of the first to ninth examples provides a tenth example. The feature mentioned in this paragraph in combination with any one of the first to ninth examples provides a tenth example.

[0018] The method can comprise checking whether the neural network has learned the interrelation between the pattern and the data class, and evaluating the saliency map determiner if the neural network has learned the interrelation between the pattern and the data class. The feature mentioned in this paragraph in combination with any one of the first to tenth examples provides an eleventh example.

[0019] Each sensor training data set unit can comprise one or more sensor data values for each of the multiple data points. The feature mentioned in this paragraph in combination with the features of any one of the first to eleventh examples provides a twelfth example.

[0020] The predefined pattern can be a predefined combination of sensor data values. The feature mentioned in this paragraph in combination with the twelfth example provides a thirteenth example.

[0021] Each of the multiple training data set units can comprise an instance of the data class, and adding the predefined pattern to the training data set unit can comprise changing the sensor data values of the training data set unit to a predefined combination of sensor values in addition to the sensor data values of the instance of the data class. The feature mentioned in this paragraph in combination with the thirteenth example provides a fourteenth example.

[0022] The data class can be an object. The feature mentioned in this paragraph in combination with any one of the first to fourteenth examples provides a fifteenth example.

[0023] The method can comprise training the neural network to identify the data class. The feature mentioned in this paragraph in combination with any one of the first to fifteenth examples provides a sixteenth example.

[0024] The method can comprise training a neural network for semantic segmentation or object classification. The features mentioned in this paragraph in combination with any of the first to sixteenth examples provide a seventeenth example.

[0025] The training data set unit can be an image. The features mentioned in this paragraph in combination with any of the first to seventeenth examples provide an eighteenth example.

[0026] The pattern can comprise a texture. The features mentioned in this paragraph in combination with any of the first to eighteenth examples provide a nineteenth example.

[0027] An apparatus can be configured to perform the method of any of the first to nineteenth examples. The features mentioned in this paragraph provide a twentieth example.

[0028] A computer program can have program instructions configured to, when executed by one or more processors, cause the one or more processors to perform the method according to one or more of the first to nineteenth examples.

[0029] The computer program can be stored in a machine-readable storage medium. BRIEF DESCRIPTION OF DRAWINGS

[0030] In the drawings, like reference characters generally refer to the same parts throughout the different views of the Figure One drawings. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the application. In the following description, various aspects of the application are described with reference to the following drawings, in which:

[0031] Figure 1 An example of object detection in an autonomous driving scenario is shown.

[0032] Figure 2 An example of a neural network that can be used to classify images into a predefined number of classes is shown.

[0033] Figure 3 A saliency determination is illustrated.

[0034] Figure 4 An arrangement for evaluating a saliency determination is shown.

[0035] Figure 5 An example of generated input data based on the MNIST data set is illustrated.

[0036] Figure 6A An example is given of a neural network picking up a bias introduced in the training data.

[0037] Figure 6B An evaluation result of a saliency determiner is shown.

[0038] Figure 7 A flowchart illustrating a method for evaluating a saliency map determiner according to an embodiment is shown. DETAILED DESCRIPTION

[0039] The following detailed description references the drawings, wherein the details of one or more embodiments of the present disclosure are illustrated by way of specific examples. Other embodiments can be utilized, and structural, logical, and electrical changes can be made without departing from the scope of the present disclosure. The various aspects of the present disclosure are not necessarily mutually exclusive, as some aspects of the present disclosure can be combined with one or more other aspects of the present disclosure to form new aspects.

[0040] In the following, various examples will be described in more detail.

[0041] Figure 1 An example 100 of object detection in an autonomous driving scenario is shown.

[0042] In Figure 1 In the example, a vehicle 101, e.g. a car, a truck or a motorcycle, is provided with a vehicle controller 102.

[0043] The vehicle controller 102 comprises data processing components, e.g. a processor, e.g. a CPU (Central Processing Unit), 103 and a memory 104 for storing control software according to which the vehicle controller 102 operates and data on which the processor 103 operates.

[0044] For example, the stored control software comprises instructions which, when executed by the processor 103, cause the processor to implement a neural network 107.

[0045] The data stored in the memory 104 can comprise image data from one or more image sources 105, e.g. acquired by one or more cameras. The images can comprise a collection of data representing one or more objects or patterns. The one or more image sources 105 can for example output a grayscale or color picture of the vehicle’s environment. The one or more image sources 105 can respond to visible light or non-visible light, such as for example infrared or ultraviolet light, ultrasound or radar waves or other electromagnetic or sonic signals.

[0046] The vehicle controller 102 can determine the presence of objects, e.g. stationary objects such as traffic signs or road markings and / or moving objects such as pedestrians, animals and other vehicles, based on the image data.

[0047] The vehicle 101 can then be controlled by the vehicle controller 102 in dependence on the results of the object determinations. For example, the vehicle controller 102 can control an actuator 106 to control the speed of the vehicle, e.g. to actuate the brakes of the vehicle.

[0048] The control can be performed on the basis of the object classification performed by the neural network 107.

[0049] Figure 2 An example of a neural network 200 is shown, which can be used for classifying images into a predefined number of classes.

[0050] In this example, the neural network 200 comprises one input layer 201, two layers 202a and 202b, and one output layer 203.

[0051] It should be noted that the neural network 200 is a simplified example of an actual deep neural network (e.g. a deep feedforward neural network) for classification purposes, which can comprise many more processing nodes and layers.

[0052] The input data corresponds to the input layer 201 and can generally be seen as a multidimensional array of values, e.g. an input image can be seen as a 2-dimensional array of values corresponding to the pixel values of the image.

[0053] The inputs from the input layer 201 are then connected to processing nodes 204. A typical node 204 multiplies each input with a weight and adds the weighted values together. Additionally, the node 204 can add a bias to the sum.

[0054] The nodes 204 are typically each followed by a non-linear activation function 205, e.g. a rectified linear unit ReLU (f(x) = max(x, 0)) or a sigmoid function The resulting values are then output to the next layer.

[0055] The layers 202a and 202b can be hidden layers, e.g. fully connected layers, as shown in Figure 2 where each node of one layer is connected to each node of the other layer.

[0056] The layers can also be (or be supplemented by) non-fully connected layers, e.g. convolutional layers or pooling layers (typically followed by one or more hidden layers) in the case of a convolutional neural network CNN.

[0057] In a convolutional layer, the input is modified by convolutional filters. These filters operate on a subset of the input data and can help to extract features of the input data, e.g. specific shapes or patterns. The filters implemented by a convolutional layer cause several corresponding nodes 204 of the convolutional layer to receive input only from a part of the previous layer.

[0058] A pooling layer can be seen as a form of non-linear down-sampling, which reduces the dimensionality of the data by combining the outputs of several nodes into a single node in the next layer, e.g. by taking the maximum value of the outputs.

[0059] In a neural network designed for classification, such as the neural network 200, the output layer 203 receives values from at least one preceding layer, e.g. from the hidden layer 202b. These values can then be transformed into probabilities by the output layer, e.g. by applying a softmax function where v i (i = 1,..., K) are the values received by the output layer) or a sigmoid function, while transforming them into probabilities. The highest probability value contained in the output vector corresponds to the class prediction.

[0060] In the following, the class prediction can also be referred to as prediction, predicted class label or predicted classification label.

[0061] The output vector of the output layer 203 is thus a probability vector, which for each predefined class indicates the probability that the image corresponds to the predefined class, e.g. that the image shows the predefined object. For example, assuming that there are 10 predefined classes for the input images of numbers (0, 1,..., 9), the output vector is a vector consisting of 10 elements, where each element corresponds to the probability of a number. The class prediction will be the number corresponding to the highest probability in the output vector. The output layer 203 can output the entire vector consisting of probability values, or only the class prediction.

[0062] In order to be able to classify images, the neural network 200 is first trained accordingly. In the case of car driving, this can be done based on a collection of traffic scenes such as cityscapes.

[0063] It should be noted in the above example that one image is classified, e.g. the image is classified to show a pedestrian. However, in e.g. a car driving scenario, an image (e.g. taken by the camera 105) typically comprises multiple objects. Therefore, in such applications, dense prediction (or semantic segmentation) can be used, which can be seen as classifying each pixel of an image. For example, certain pixels can be classified to show a pedestrian, while other pixels are classified to show another vehicle. Such dense prediction can similarly be performed using a neural network for image classification as explained above, with the difference that the output comprises a class prediction for each pixel of the image, rather than a class prediction for each image. The output of an image x can thus be another image indicating the class prediction for each pixel (e.g. encoded by color, e.g. pedestrian green, vehicle red, background grey, etc.), i.e. f(x) itself is an image with multiple channels corresponding to possible class predictions. For semantic segmentation, a CNN can be used.

[0064] In many real-world scenarios, the existence of an object, its location and appearance are highly interrelated with the context information around that object, such as the existence of other nearby objects or more global scene semantics. For example, in the case of a city street scene, a cyclist is more likely to co-occur with a bicycle, and a car is more likely to appear on a road under the sky and buildings. These semantic inter-relationalities inherently exist in real-world data. Data-driven models such as deep neural networks 200 tend to exploit these statistical biases in order to improve their prediction performance. The effective and safe exploitation of deep learning models for real-world applications, e.g. autonomous driving, requires a good understanding of these contextual biases inherent in the data and the extent to which the learned model incorporates them into its decision process. Otherwise, there is a risk of misclassification of an object, e.g. because it appears in an unusual location.

[0065] Interpretability is a core element of both model verification (did we build the model correctly?) and validation (did we build the right model?). Network models can typically be easily fooled to provide false, high-confidence predictions, and thus their true generalization performance has often been questioned. As with any other software component intended to be deployed in the field, it is desirable to quantitatively establish the extent to which a model can be verified and validated from this perspective, i.e. the extent to which the model has learned the desired input-output relationship.

[0066] Compared to the amount of available data, models working with limited (and thus inherently biased) datasets can clearly be over-parameterized. This often creates a wide gap between what the model has actually learned and what the implementer believes the model has learned.

[0067] As an effort to explain deep networks for classification networks, saliency maps, i.e. heat maps over input images, can be constructed that highlight which parts of an image are relevant or useful for classifying the image. Besides classification models, semantic segmentation is another use case that is worth looking at in this respect. Semantic segmentation models in particular tend to learn the aforementioned inherent biases in their training datasets. For example, when learning to recognize a common street scene (e.g. as in a cityscape dataset), there is a clear bias that the network will learn, e.g. that cars are always driving on a road. However, other biases can be effects of missing or over-represented data and can represent relationships that can harm the model's generalization performance. It is thus desirable to help explain these model predictions or segmentations.

[0068] Saliency methods can explain the predictions of a trained model (e.g. a trained neural network) by highlighting parts in the input that can be assumed to have a high relevance for the model's prediction, i.e. by identifying the image pixels that contribute most to the network's prediction.

[0069] In the following, as an example, a scheme for saliency determination, i.e. a saliency method, is described, which can be seen as an extension of the saliency method for image classification towards (pixel-level) dense prediction tasks, and which allows to generate spatially coherent explanations (and spatially distinguish between prediction explanations) by exploiting spatial information in dense predictions.

[0070] In the following example, the scheme is referred to as grid saliency, which is a perturbation-based saliency method that is based on a formulation as an optimization problem that identifies the minimal unperturbed region of an image that is required to preserve the network prediction inside the region of the target object. Since grid saliency allows to distinguish between an object and its associated context region in the saliency map, it allows to produce context explanations for a semantic segmentation network, discovering which context information impacts the most on the class prediction inside the region of the target object.

[0071] Let f : I → O denote a prediction function, e.g. implemented by a deep neural network 200, that maps a grid input space I = RH I × W I × C I (e.g. an input image of an input image space) to a grid output space O = RH O × W O × C O (e.g. an output image of an output image space), where W and H are the respective width and height of the input (image) and output (image), and C I and C O is the number of input channels (e.g. 3 or 1 for images) and the number of output prediction channels (e.g. the number of classes for semantic segmentation).

[0072] In the following, for ease of explanation, only images are considered as input and each-pixel dense predictions of a network are considered as output, i.e. the input is an image, x e I, and the output is the each-pixel dense prediction f(x). Moreover, for simplicity, it is assumed that the input and output space dimensions are the same.

[0073] According to an embodiment, the grid saliency determination for an input image x can be seen as based on finding a minimal saliency (map) M e [0, 1]H I × W I that has to be preserved in the image x, in order to keep a requested mask region R e {0, 1}H O × W O for a class (e.g. a channel) c e {1,..., C Othe significance of M is based on the pixel values of M. For example, if the II-norm of the pixel values of M is smaller than the II-norm of the pixel values of M', then M is smaller than M'.

[0074] According to various embodiments, the grid significance is based on a perturbation significance scheme. This means that the salient image region that is primarily responsible for the classifier decision (in the request region) is determined by replacing parts of the image with non-informative pixel values (i.e. perturbing the image) and evaluating the corresponding class prediction change.

[0075] Let p denote a perturbation function that removes information outside the significance M from the image x (where outside means pixels for which M is not 1). For example, such a perturbation function can be an interpolation between x and a e I, where a can be a constant color image, a Gaussian blur or random noise. In this case, where denotes the Hadamard product.

[0076] It should be noted that in practice, the significance M can be defined on a lower resolution than the input image to avoid adversarial artifacts and can be upsampled to the input image resolution later. Furthermore, the pixel values of the perturbed image p(x, M) can be clipped to maintain the range of the original image pixel value space.

[0077] Based on the above notation, the task of finding a saliency map M for class c prediction can be formulated as the following optimization

[0078] where ||. ||1 denotes the II-norm and f c (x) is the network prediction for class c, i.e. the pixel value of the output image for class (channel) c.

[0079] The first term on the right-hand side of equation (1) can be seen as a mask loss that minimizes the salient image region such that the original image is perturbed as much as possible. The second term acts as a keep loss that ensures that the network prediction f c (p(x, M)) for class c on the perturbed image p(x, M) at least reaches the confidence of the network prediction f c (x) on the original, unperturbed input image. Thus, the second loss term can be seen as a penalty for not satisfying the constraint f c (p(x, M)) > f c (x). Hence, the max(·, 0) is used in equation (1). The parameter λ controls the sparsity of M, i.e. how strongly the size of the saliency is penalized. An exemplary value for λ is 0.05, but other values can possibly generate smaller or larger saliencies.

[0080] It is further possible to separate spatially the interpretation given to the network prediction in the requested region of interest R in the saliency map M from the interpretation given to other predictions by restricting the loss of holding in equation (1) to the request mask R according to

[0081]

[0082] In the following, the result of equation (2) is referred to as grid saliency map.

[0083] The grid saliency formula from equation (2) can be adapted to specifically provide a contextual interpretation for the requested region of interest R. For semantic segmentation, a contextual interpretation is of particular interest, as the context typically usually serves as one of the main cues for semantic segmentation networks.

[0084] Thus, according to various embodiments, there is a focus on the contextual interpretation of semantic labeling predictions, and let R be a region covering an object of interest in the input image x. To optimize the salient part of the object context, the object request mask R is integrated into the perturbation function. For the request mask R, the perturbed image p(x, R) e I contains only the object information inside R and (e.g., in case of a constant colored image a) removes all contextual information outside R.

[0085] To optimize (i.e., the determination of saliency), this new perturbed image p(x, R) is used instead of the maximally perturbed image p(x, M = 0) = a, and the contextual perturbation function is In other words, when the input image is perturbed, the image information inside R is relevant (thus, the saliency does not "need" to include the request region R).

[0086] The contextual saliency map for class c and the request object R is given by

[0087]

[0088] This can be seen as an optimization of the saliency map to select the minimum context that is necessary to produce at least the original prediction for class c inside the request mask R.

[0089] It should be noted that the contextual saliency map can be seen as a special case of the grid saliency map, as the perturbation takes a special form.

[0090] Figure 3 Fig. 1 illustrates a saliency determination.

[0091] In 301, a neural network is trained for dense prediction, which is denoted by the function f.

[0092] In 302, given a network trained for f (dense prediction task), an input image x, and the network f for this input image and class c (e.g. output image channel c) c (x) of the prediction (map), the target regions R in the prediction map for which a visual explanation is required are selected.

[0093] In 303, these components are used to define an optimization problem according to equation (3), which is solved to obtain the saliency map as a post-hoc explanation of the prediction in the target regions R.

[0094] The optimization problem can be solved (i.e. the saliency map can be optimized) using various optimization techniques such as stochastic gradient descent (SGD) with momentum or Adam. For example, for SGD, a momentum of 0.5 and a learning rate of 0.2 can be used for 100 steps, and for each pixel, the saliency map can be initialized with 0.5. Another example is the optimization of a coarse 16 by 32 pixel mask using SGD with a learning rate of 1 for 80 steps.

[0095] In 304, the resulting saliency map is used for various purposes, e.g. as a visual explanation for the network prediction, and for debugging errors in the trained network and determination of classification reliability.

[0096] The grid saliency can for example be used to enhance a trained model (e.g. any dense prediction network) by detecting biases picked up by the trained network via the obtained grid saliency map. The obtained grid saliency map can be used for anomaly detection by checking inconsistencies across saliency explanations for the same semantic object class and by identifying the cause of anomalies via the obtained saliency map. More generally, it can be used to explain any unexpected model behavior, e.g. in case of erroneous predictions for corner cases, the grid saliency can be used to understand which part of the image (any data sample) is the “outlier” (anomaly) and causes the failure.

[0097] The grid saliency can in particular be used in an online diagnostic tool, e.g. for a controller such as the vehicle controller 102.

[0098] It should be noted that while in the above example the grid saliency is determined via perturbation, it can also be determined based on other schemes (such as in a gradient-based manner as explained in the following).

[0099] Let denote the gradient of the prediction fcwith respect to the input image x e I for class c. For VG (vanilla gradient), IG (integral gradient) and SG (smoothed gradient) saliency, the saliency map is computed as

[0100]

[0101] where n is the number of approximation steps for IG or the number of samples for SG, and denotes Gaussian noise with standard deviation σ.

[0102] Similar to the perturbation-based approach, the explanation given by the saliency M for the network prediction in the request region R is spatially separated from other predictions. For a given input x and binary request mask R, the normalized network prediction score for class c in the request region R is denoted by

[0103]

[0104] Similar to G(x, c), for grid saliency, define which directly yields the grid saliency grid (x, R, c) by replacing G(x, c) in equation (4) with For gradient-based contextual saliency, similar to the perturbation-based approach, only salient pixels outside the object region (i.e. the region of interest) are considered, i.e.

[0105] It should be noted that gradient-based saliency maps tend to be noisy. Therefore, to circumvent this, a spatial mean filter with a (W1 / W S ) x (H1 / H S ) kernel and stride can be used on top of the saliency map, where W S x H S is the resolution of the perturbation-based saliency map.

[0106] Nevertheless, the perturbation-based saliency determination can perform better, i.e. determine the context with higher accuracy, compared to the gradient-based approach. However, this is not immediately apparent from the saliency approach used.

[0107] In view of this, according to various embodiments, a scheme for evaluating (or benchmarking) a saliency determiner is provided, e.g. a quantitative quality measure for a saliency determination method. Such a measure can in particular be used to show that the results from the above-described grid saliency method are meaningful and non-random.

[0108] According to various embodiments, therefore, a scheme is provided for determining the reliability and quality of a contextual saliency determination (or detection) method such as the above grid saliency determination. Such a scheme allows to quantitatively show to what extent the saliency determination method identifies contextual biases. In principle, such a scheme can be used to evaluate (or benchmark) any method that aims at detecting contextual biases in data.

[0109] Figure 4 An arrangement 400 for evaluating a saliency determination is shown.

[0110] The arrangement comprises a contextual bias detector 401, which can be seen as an input to the evaluation method implemented by the arrangement 400. The contextual bias detector 401 is denoted D and can for example implement the grid saliency method explained above. For example, but not limiting, the detector is configured to determine the saliency of a neural network that is trained to identify and localize the presence of a specific object structure at pixel level in an image. For example, in case of a road scene from a cityscape dataset, this can be a car, a pedestrian, a cyclist, etc.

[0111] According to various embodiments, a family (set) of data generation processes P is provided that are used to create a given data distribution in an appropriate input space (e.g. images, audio, video, etc.). The data generation process can for example be modeled by any data generation method such as using a generative adversarial network, a manual process or other alternatives.

[0112] For simplicity, the following explanation focuses on the image domain as the main input modality, although as pointed out above other alternatives are possible as well. This means that in the following, a neural network is provided with an input image, performs e.g. a classification or semantic segmentation of the input image, and a contextual bias detector D determines the context of the decision of the neural network (e.g. a saliency map or a grid saliency map) and uses an evaluation method to evaluate the performance of the contextual bias detector. As a specific example, a classification of a number as present in the MNIST dataset is given for simplicity.

[0113] Let P unbiased be an element in P, i.e. a process that generates an input dataset for a neural network (in this example input images with numbers) that is unbiased in the following sense. P unbiased is configured to generate a data distribution that is constrained on:

[0114] 1. There shall only be one object as defined by the input data modality present in a given sample (e.g. the input data modality dictates that the object is a number) (i.e. one number per input image). The object is considered "foreground", everything else in the sample (i.e. input image) is considered "background".

[0115] 2. The background is evenly divided into parts. Process P unbiased is configured to generate its input dataset such that each individual background part is "in-painted" with a different texture pattern drawn at random from a set of texture patterns.

[0116] 3. Optionally, process P unbiased is configured to generate its input dataset such that the foreground is also in-painted with a randomly selected texture.

[0117] 4. Optionally, configure process P unbiased such that the statistics of the generated samples (in the input dataset) are balanced by ensuring that all texture patterns appear equally as background / foreground elements, or equally as groups of other texture patterns, etc.

[0118] The input dataset generated by P unbiased is referred to as the P unbiased dataset. In the P unbiased dataset, all foreground and background texture patterns appear with (essentially) equal probability for all objects (digits).

[0119] Let P biased be another element in P, P is a process configured to generate an input dataset (in this example, input images with digits) for a neural network, which process is biased (compared to the dataset generated by P unbiased .

[0120] P biased is configured very much like P unbiased , except that it is configured to generate an input dataset (which is referred to as the P biased dataset) in which certain texture patterns appear more frequently than others in the background, given by the probability as a function of the objects in the sample, i.e. a particular texture is coupled to a particular object (in this example, a digit). In the extreme case (when the co-occurrence probability is 100%), a background texture appears in a sample (image) if and only if a particular object instance appears in that sample.

[0121] Figure 5 An example of generated input data based on the MNIST dataset is illustrated.

[0122] The MNIST dataset can be used as a basis for P biased and P unbiased . In this example, the foreground corresponds by definition to the digits present in the images, and the background corresponds to everything else. The background is evenly split into two parts, each in-painted with a random texture. Here, the foreground digits are also in-painted. Pbiased and P unbiased The difference is that the probability that a given object (number) will appear in the same image with a particular texture. It should be noted that texture is an example here and other image features (e.g. certain color ranges, etc.) can be used. In general, objects can be correlated (in biased datasets) with certain patterns, which can be textures of images, but also other structures (or combinations of data point values) of other kinds of input data.

[0123] The arrangement 400 comprises an implementation 402 of P biased configured to generate a biased input dataset, e.g. from a reference dataset 404, such as the MNIST dataset. unbiased The arrangement 400 further comprises an implementation 403 of P unbiased configured to generate an unbiased input dataset from the reference dataset 404.

[0124] It should be noted that MNIST is only an example for ease of explanation and other datasets can also be used, in particular datasets with traffic scenes, such as urban landscapes for applications in the context of car driving.

[0125] The neural networks are trained using the biased dataset and independently using the unbiased dataset to perform classification or semantic segmentation of input images, resulting in two trained neural networks 405, 406.

[0126] The trained neural networks 405, 406 are applied to the unbiased dataset to perform classification or semantic segmentation of input images of the unbiased dataset. The saliency (or context) detector 401 detects the saliency of the results provided by the neural networks 405, 406.

[0127] The evaluator 407 receives the saliency determined by the detector 401 and evaluates the performance of the detector 401 in terms of its saliency determination capability.

[0128] The evaluator 407 can use a metric as a basis for evaluating the detector performance. In the following, three examples of metrics are given, which can be used to measure the D’s ability to recognize context bias. All metrics can be computed as a mean over a set of objects (generally a data class) O.

[0129] 1. A first measure is the mean IoU (intersection over union) difference between D applied to a neural network trained on the P biased dataset and evaluated on the P unbiased dataset and D applied to a neural network trained and evaluated on both the P unbiased and P biasedObject is explicitly associated with texture, so the background texture T associated with object O should also constitute positive evidence for the classification of another instance of object O, even in settings where T and O were not associated (here: P unbiased ).

[0130] On the other hand, D for a neural network trained on P unbiased and evaluated on P unbiased should not determine any saliency (outside the numbers themselves).

[0131] Hence, for example, the IoU is measured between the original object mask (where does O appear in the image?) and the saliency returned by D for a neural network trained on P biased and evaluated on P unbiased (where here it is assumed that the saliency detector D is set to include the object itself in the saliency). A lower IoU value corresponds to a stronger context bias detection, as this means that D detects a large context region outside the object (number) itself.

[0132] The first measure can in particular be used to assess to what extent a neural network is able to pick up on context biases present in the training data.

[0133] 2. The second measure is the mean CBD (Context Bias Detection measure), which can be seen as a generalization of the first (IoU) measure above by including a request mask of arbitrary size. As explained above in the context of grid saliency, the request mask is a region of interest in the image that is chosen for optimization. For the first measure, it can be considered equivalent to the entire object (number). For the second measure, it is generalized to e.g. different object subparts or components.

[0134] Let C-I-R, i.e. let C be the part of the input image x that is not the request mask R (which e.g. is the ground truth mask of the object in the image x). Let c be the class assigned to the object at the request mask R (i.e. to the pixels assigned to R). Using the definition of M context from equation (3) above (where here the asterisk and argument are omitted for simplicity), the CBD measure is formally defined as

[0135]

[0136] Intuitively, the percentage of background pixels marked as salient by D is computed.

[0137] The second measure can in particular be used to benchmark how well different saliency methods (e.g. perturbation-based or gradient-based) can detect context biases of a semantic segmentation neural network.

[0138] 3. The third measure is the mean CBL metric (Contextual Bias Locating metric), which can be seen as a more specific metric that includes in the metric of equation (7) a constraint on where the saliency determined by D lies. This means that only salient regions are taken into account that correspond to textures that are already positively associated with the current object in the input image.

[0139] Let C bias be a mask of the input image x that marks where the texture T for the object O appears, where T, O have been associated via a data generation process P biased The CBL metric is then defined as follows:

[0140]

[0141] That is, the metric achieves its maximum value when D provides saliency that is completely focused on the background, and specifically on the background part that corresponds to the texture T that is associated with the current object in the input image x. R is a subset of O, but for simplicity can be considered to span the entire object O. Intuitively, this quantifies how specifically attention is shifted in the background from O to T. A higher percentage indicates that the input method D is highly capable of identifying the contextual bias present in P biased .

[0142] The third measure can be used in particular to benchmark the ability of different saliency methods to locate the contextual bias.

[0143] Each of the three metrics can be averaged (e.g. over multiple input images and / or object classes, in general over multiple input datasets and data classes) to provide a mean metric. This is denoted in the following with a small "m", i.e. mIOU, mCBD and mCBL.

[0144] P biased and P unbiased can be chosen to represent a simple dataset in which the contextual bias is clearly identifiable such as in the example based on MNIST, which also forms the basis for the following examples. Figure 4

[0145] For the following examples, the process P biased is configured to generate two variants of a biased dataset - a weak bias version DS no-bias and a strong bias version DS w-bias - from an unbiased dataset DS s-bias of digits. For the dataset with strong induced bias DS s-bias , a particular texture appears if and only if a certain digit class is present. For the weak induced bias DS​w-bias , the specific texture always occurs together with the biased digit, but also consistently with other digits. For example, P biased From a pool of 25 textures, 5 textures are randomly selected to induce a contextual bias for one of the 10 digits. For all 50 texture / digit combinations, it generates weakly and strongly biased dataset variables with train / test splits and trains the neural network for both (independently).

[0146] With respect to the bias in the trained network, the evaluator 407 can first investigate whether the weak and strong contextual bias induced in the training data has been picked up by the network trained on DS w-bias and the network trained on DS s-bias . For this purpose, their performance can be evaluated in terms of mIoU on the unbiased dataset DS no-bias .

[0147] Figure 6A The per-digit mIoU for biased digits with respect to weak bias (top heatmaps 601) and strong bias (bottom heatmaps 602) is visualized. The first row (labeled N) of heatmaps 601, 602 shows the performance of the network trained on DS no-bias . Compared to the first row, a clear drop in performance for the biased digits (diagonal elements) can be seen. As expected, the stronger the bias, the higher the drop in performance. In addition, the mIoU for unbiased digits (non-diagonal elements) is also affected by the introduced bias. For example, the introduced bias for digit nine leads to a reduced performance for digit four (see second row of heatmaps 601, 602). This effect mainly occurs for similar-looking digits and can be assumed to be caused by the fact that on the unbiased dataset, the biased textures also occur for unbiased digits, leading to a confusion of similar-looking digits by the network. From the observed drop in mIoU for biased digits, the evaluator 407 can conclude that the network 405 has picked up the introduced bias. However, in a real-world application, it can not be possible to collect completely unbiased data. Therefore, the evaluator 407 can evaluate the ability of the detector 401 to detect contextual bias using only biased data (e.g. grid saliency).

[0148] Figure 6B Heatmaps 603 to 610 are shown, which illustrate the performance of saliency detection with VG, SG, IG, and grid saliency, respectively, and weak bias (top heatmaps 603 to 606) and strong bias (bottom heatmaps 607 to 610), which is determined by using the CBD metric.

[0149] The heatmaps give the per-digit mIoU with respect to the different digit-biased DS s / w-biasnetwork trained on data of different digit classes (x-axis) of the unbiased test set.

[0150] The only exception is the first row (labeled N), where the unbiased results of the network trained on DS no-bias It can be seen that grid saliency shows a large amount of evidence of contextual bias for the digits with introduced bias (both strong and weak) (diagonal elements). Even for weak bias, grid saliency still clearly distinguishes between biased and unbiased digits (diagonal elements vs. non-diagonal elements).

[0151] The contextual saliency extension of gradient-based methods, i.e., VG, SG, and IG, do not reliably detect contextual bias, while IG achieves comparable performance to grid saliency. However, in contrast to perturbation saliency, IG also has high mCBD values for unbiased digits, complicating its use in practice, as one would need to tune a reliable detection threshold, which is particularly challenging for weak bias data.

[0152] In addition to detection performance, evaluator 407 can also evaluate how well saliency detector 401 can localize contextual bias using the mCBL measure. The evaluation shows that VG, IG, and SG have low localization performance comparable to random guessing, while grid saliency can accurately localize contextual bias on both strong bias and weak bias data.

[0153] In summary, according to various embodiments, there is provided a method as illustrated in Figure 7

[0154] Figure 7 A flowchart 700 is shown, illustrating a method for evaluating a saliency map determiner according to an embodiment.

[0155] In 701, a predefined pattern is added to a plurality of training dataset units for identification of a data class, wherein each training dataset unit comprises a representation of the data class to be identified.

[0156] In 702, a neural network is trained with the plurality of training dataset units comprising the predefined pattern.

[0157] In 703, the saliency map determiner determines a saliency map for the data class.

[0158] In 704, the saliency map determiner is evaluated based on whether the determined saliency map comprises a context of the data class introduced by the addition of the predefined pattern.

[0159] ​According to various embodiments, in other words, the training data is modified such that certain data classes are correlated with certain patterns in the training data. Thus, a certain bias is introduced in the training data. Since it can be assumed that the neural network takes this bias into account when identifying the data classes, the saliency map determiner should at least partly include this pattern in the saliency map.

[0160] For example, for a neural network trained to identify data features (e.g. objects in an image), an original training data set comprising images is provided, which images comprise objects (or generally image features, such as regions of a certain depth or motion), and for at least one object, a region is included whose content is not correlated with the object (e.g. in case of constant content). The region not correlated with the object can be understood as the distribution of the object in the training data set being (substantially) independent of the content of the region. An example of such a region is the background in a MNIST image: the content of the background (black) is not correlated with the object (the digit).

[0161] For an image having such an object and such a not correlated region, the image is modified such that the region comprises content that is correlated with the region.

[0162] This means that for one or more images of at least one object, the content of the not correlated region is changed such that the content of the not correlated region (in the original training set) becomes correlated with the appearance of the object in the image in the modified training set. In the MNIST example, as mentioned above, a certain texture can for example be introduced into the background of a certain digit such that the content of the image region (in this example the background) becomes correlated with the object (the digit).

[0163] The result of the modification of the training data set with respect to the training data units (e.g. images) and the possibly multiple data classes (e.g. object images) results in a modified training data set (referred to as biased training data set in the above example).

[0164] The neural network is now trained based on the modified training data set and the neural network behavior is analyzed by means of the neural network analysis tool (i.e. the saliency determiner) indicating the context of the network decision. The network analysis tool can then be evaluated based on whether it identifies the context in the correlated region (e.g. indicates the correlated region as salient).

[0165] In other words, a bias is introduced into the training data and it is evaluated whether the saliency determiner is able to detect that the neural network uses the introduced bias.

[0166] While in the above example, the neural network is used for object classification or semantic segmentation with images as input data, the same applies to other neural networks and other types of input data. Figure 7The approach of the solution can be used for neural networks receiving sensor signals received from any sensor, i.e. operating on any kind of input sensor data such as video, radar, lidar, ultrasound and motion. The mode can thus be a sensor data type mode, i.e. a certain value combination of values of a sensor data type, e.g. texture, motion values (or motion in a certain video region), lidar values in a certain range, etc.

[0167] It should be specifically noted that the neural network input data (and the corresponding training data, in particular the training data units) are not limited to images, but can also be applied to any image-like data (e.g. data structured in one or more two-dimensional or even higher-dimensional arrays), such as sound spectrograms, radar spectra, ultrasound images, etc. Furthermore, raw 1D (e.g. audio) or 3D data (video, or RGBD (red green blue depth) data) can also be used as input.

[0168] The generated saliency determination result (e.g. the grid saliency map) can be used as a basis for computing a control signal for controlling a physical system (like e.g. a computer-controlled machine, like a robot, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant or an access control system), or a system for conveying information (like a surveillance system or a medical (imaging) system) in order to interpret and understand the decision process of the neural network used in the above physical system. It does so by generating a low-level visual explanation (e.g. the grid saliency map). In particular, the result image allows to identify the cause of an anomaly by analyzing the explanation given by the result image. The controller can use the determined saliency as a basis for forming a decision based on the neural network output.

[0169] The method of evaluating a saliency map determiner provided according to various embodiments can be executed by one or more processors. The term “processor” can be understood as any kind of entity that allows processing data or signals. For example, data or signals can be handled in accordance with at least one (i.e. one or more than one) specific function executed by the processor. The processor can comprise or be formed by an analog circuit, a digital circuit, a complex signal circuit, a logic circuit, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field-programmable gate array (FPGA) integrated circuit, or any combination thereof. Any other means of implementing a respective function (which will be described in more detail below) can also be understood as a processor or logic circuit. It will be understood that one or more method steps described in detail herein can be executed (e.g. implemented) by the processor by one or more specific functions executed by the processor.

[0170] While particular embodiments have been illustrated and described herein, ordinary skilled in the art will appreciate that various alternatives and / or equivalents can be substituted for the specific embodiments shown and described without departing from the scope of the present application. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that the application be limited only by the claims and the equivalents thereof.

Claims

1. A method of evaluating a saliency map determiner performed by one or more processors, the method comprising: • adding a predefined pattern to a plurality of training images to train the recognition of an object, wherein each training image comprises a representation of the object to be recognized, wherein adding a predefined pattern to the plurality of training images comprises adding a predefined pattern to the training data associated with the object, wherein the predefined pattern is a predefined combination of image data values; • training a neural network with the plurality of training images comprising the predefined pattern; • determining, by the saliency map determiner, a saliency map for the object by applying the neural network to an image comprising an instance of the object and the pattern and determining the saliency map based on an output of the neural network; and • evaluating a bias detection of the saliency map determiner with respect to the neural network, wherein the bias detection with respect to the neural network comprises determining, in case of a misclassification or a semantic segmentation of the neural network, an image portion contributing to the misclassification or the semantic segmentation by determining whether the determined saliency map comprises a context of the object introduced by adding the predefined pattern, wherein the context is an area around a plurality of image data values in the image representing the object.

2. The method according to claim 1, comprising evaluating the bias detection of the saliency map determiner based on a comparison of a context of the object of the neural network trained on the training data without adding the predefined pattern interrelated to the object and a context of the object of the neural network trained on the training data with adding the predefined pattern interrelated to the object.

3. The method according to claim 1 or 2, comprising evaluating the bias detection of the saliency map determiner based on an overlap of an area of the image representing the instance of the object and an area of the saliency map for the instance of the object of the image.

4. The method according to claim 1 or 2, comprising evaluating the bias detection of the saliency map determiner based on a context specified by the saliency map outside a requested area.

5. The method according to claim 1 or 2, wherein evaluating the bias detection of the saliency map determiner based on whether the determined saliency map comprises a context of the object introduced by adding the predefined pattern comprises • determining whether the saliency map comprises a representation of the predefined pattern; and • evaluating the bias detection of the saliency map determiner based on whether the determined saliency map comprises the representation of the predefined pattern. The plurality of training images is part of a training data set, the training data set comprising a plurality of further training images, and the neural network is trained with the training data set.

6. The method of claim 1 or 2, wherein, 7. The method according to claim 6, wherein at least some of the further training images do not contain the object.

8. The method according to claim 6, wherein at least some of the further training images contain a further object. ​ 9. The method of claim 1 or 2, comprising checking whether the neural network has learned the association between the pattern and the object, and if the neural network has learned the association between the pattern and the object, evaluating the bias detection of the saliency map determiner.

10. The method of claim 1 or 2, wherein each of the plurality of training images comprises an object instance, and adding a predefined pattern to the training image comprises changing the data values of the training image to a predefined combination of sensor values in addition to the data values of the object instance.

11. The method of claim 1 or 2, comprising training the neural network to identify the object.

12. The method of claim 1 or 2, wherein the pattern comprises a texture.

13. A device configured to perform the method of any one of claims 1 to 12.

Images