Apparatus and method for operating a neural network

Abstract

Devices and methods for operating a neural network are provided. According to various embodiments, a method for operating a neural network is described, including: determining, using a neural network, neural network output data for neural network input sensor data; selecting a portion of the output data points to form a region of interest; and for each of at least some output data points outside the region of interest, determining a contribution value representing a contribution of one or more input data points associated with the output data point to the output data point value assigned to the output data point in the region of interest.

Description

Technical Field

[0001] This disclosure relates to devices and methods for operating neural networks. Background Technology

[0002] Neural networks are being used more and more widely to classify images into a predefined number of classes. Understanding how neural networks achieve their results can greatly help in determining how reliable the classification is, but it is notoriously difficult.

[0003] The publication "Deep inside convolutional networks: Visualizing image classification models and saliency maps" by Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman, presented at the 2013 International Conference on Learning Representations (ICLR), describes interpreting network decision-making processes by generating low-level visual interpretations. Such and similar approaches primarily focus on image classification tasks and can be divided into two categories: gradient-based methods and perturbation-based methods.

[0004] Gradient-based methods compute a saliency map visualizing the sensitivity of each image pixel to a specific class prediction. This saliency map is obtained by backpropagating the gradient of the prediction relative to the image and estimating how moving along the gradient affects the class output. Gradient-based methods rely heavily on heuristics used for backpropagation and can provide interpretations that are not faithful to the model or the data. Perturbation-based methods evaluate changes in class prediction relative to a perturbed image; for example, for a perturbed image, specific regions of the image are replaced using the mean image value or removed by applying blur or Gaussian noise.

[0005] Efficient schemes for saliency determination are desirable, and these schemes are not limited to the interpretation of image classification networks, but also applicable to other neural networks such as dense prediction networks. Summary of the Invention

[0006] The methods and apparatus having the features of independent claims 1 (first example) and 18 (twenty-third example) allow for spatially coherent interpretations of neural networks, such as dense prediction (or semantic segmentation) networks. For semantic segmentation neural networks, spatial and semantic interrelationships can thus be discovered in the training data picked up by the neural network, and, for example, vehicle control for automated driving can be taken into account in further processing.

[0007] Another example is described below.

[0008] A method for operating a neural network, executed by one or more processors, may include: determining neural network output data using sensor data as input to the neural network, wherein the neural network input data includes multiple input data points, each input data point being assigned one or more input data point values, and wherein the neural network output data includes multiple output data points, each output data point being assigned one or more output data point values, and wherein each output data point is associated with one or more input data points; selecting a portion of the output data points from the multiple output data points to form a region of interest, wherein the region of interest includes a plurality of output data points; and for each of at least some output data points outside the region of interest, determining a contribution value representing the contribution of one or more input data points associated with the output data points of the neural network, the neural network determining the output data point values ​​assigned to the output data points in the region of interest. The method mentioned in this paragraph provides a first example.

[0009] By mapping input data point coordinates to output data point coordinates, each output data point can be associated with one or more input data points. The features mentioned in this paragraph, combined with the first example, provide a second example.

[0010] Input data points can be constructed as an input array, and output data points can be constructed as an output array. Each output data point is associated with one or more input data points through a mapping from positions in the input array to positions in the output array. The features mentioned in this paragraph, combined with any of the first to second examples, provide a third example.

[0011] Input data points can be constructed as an input image, and output data points can be constructed as an output image. Each output data point can be associated with one or more input data points through a mapping from pixel positions in the input array to pixel positions in the output array. The features mentioned in this paragraph, combined with any of the first through third examples, provide a fourth example.

[0012] The method may include presenting the user with output data points having contribution values ​​and output data points for the region of interest. The method may further include comparing the relative positions of the output data points with contribution values ​​to the output data points for the region of interest. Both procedures have the advantage that the output of the neural network can be validated based on the contribution values.

[0013] The method may further include determining the contribution value of a data point based on a measure of the influence of a perturbation of the input data point value associated with the output data point on one or more output data point values. The features mentioned in this paragraph, combined with any of the first through fourth examples, provide a fifth example.

[0014] The contribution value can be determined based on a trade-off between the total metric of the contribution value and the retention loss, which occurs when determining the output data point value assigned to the region of interest and when information in the input data values ​​is ignored based on the contribution value. A sixth example is provided by combining the features mentioned in this paragraph with any of the first through fifth examples.

[0015] The portion of the output data points selected to form the region of interest can be a true subset of multiple output data points. The features mentioned in this paragraph, combined with any of the first through sixth examples, provide the seventh example.

[0016] The output data point value for each output data point can specify the data class of the input data point values ​​for one or more input data points associated with the output data point. The features mentioned in this paragraph, combined with any of the first through seventh examples, provide an eighth example.

[0017] The contribution value of an output data point can represent the contribution of one or more input data points associated with the output data point to the neural network's decision to set the output data point value to specify the data class. The features mentioned in this paragraph, combined with the eighth example, provide the ninth example.

[0018] The neural network input sensor data can include one or more images. The tenth example is provided by combining the features mentioned in this paragraph with any of the first through ninth examples.

[0019] The output data of the neural network may include the resulting image. The eleventh example is provided by combining the features mentioned in this paragraph with any of the first through tenth examples.

[0020] The region of interest can be an image region in the resulting image. The features mentioned in this paragraph, combined with the eleventh example, provide the twelfth example.

[0021] Neural networks can be trained for image segmentation, where the resulting image represents semantic segmentation. The thirteenth example is provided by combining the features mentioned in this paragraph with any of the eleventh or twelfth examples.

[0022] The region of interest can correspond to one or more segments of semantic segmentation. The features mentioned in this paragraph, combined with the thirteenth example, provide the fourteenth example.

[0023] The resulting image can be a depth image or a motion image. The features mentioned in this paragraph, combined with any of the eleventh or twelfth examples, provide the fifteenth example.

[0024] The method may include generating a saliency map representing contribution values. The features mentioned in this paragraph, combined with any of the first through fifteenth examples, provide a sixteenth example.

[0025] The contribution value can be a pixel value from the saliency map. The features mentioned in this paragraph, combined with the sixteenth example, provide the seventeenth example.

[0026] Each input data point can be associated with exactly one output data point. The features mentioned in this paragraph, combined with any of the first through seventeenth examples, provide the eighteenth example.

[0027] A portion of the output data points can be selected such that the output data point values ​​fall within a predetermined range. The nineteenth example is provided by combining the features mentioned in this paragraph with any of the first through eighteenth examples.

[0028] The method may further include controlling the training of the neural network based on contribution values. The features mentioned in this paragraph, combined with any of the first through nineteenth examples, provide a twentieth example.

[0029] The method may further include controlling the actuator based on the contribution value. The features mentioned in this paragraph, combined with any of the first through twentieth examples, provide a twenty-first example.

[0030] The method may further include evaluating the performance of the neural network based on contribution values. The features mentioned in this paragraph, combined with any of the first through twenty-first examples, provide a twenty-second example.

[0031] A device can be configured to execute the methods of any of the first through twenty-two examples. The features mentioned in this paragraph provide a twenty-third example.

[0032] A vehicle may include: at least one image sensor configured to provide digital image data; and a driver assistance system including a neural network operating according to any of the first to twenty-second examples, wherein the neural network is configured to classify the digital image data, and wherein the driver assistance system is configured to control the vehicle based on the classified digital image data and contribution values. The features mentioned in this paragraph provide a twenty-fourth example.

[0033] A computer program may have program instructions configured to, when executed by one or more processors, cause the one or more processors to perform one or more methods according to one or more of the first to twenty-second examples.

[0034] The computer program can be stored in a machine-readable storage medium. Attached Figure Description

[0035] In the accompanying drawings, the same reference characters are used throughout different views. Figure 1 Generally, the same parts are referred to. The accompanying drawings are not necessarily to scale, but generally focus on illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings, in which:

[0036] Figure 1 An example of object detection in an autonomous driving scenario is shown.

[0037] Figure 2 An example of a neural network that can be used to classify images into a predefined number of classes is shown.

[0038] Figure 3 The illustration shows the determination of salience according to an embodiment.

[0039] Figure 4 and 5 An example of salience determined according to an embodiment is shown.

[0040] Figure 6 A flowchart illustrating a method for operating a neural network according to an embodiment is shown. Detailed Implementation

[0041] The following detailed description is taken with reference to the accompanying drawings, which illustrate by way of illustration specific details and aspects in which the present disclosure may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made, without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

[0042] The various examples will be described in more detail below.

[0043] Figure 1 Example 100 of object detection in an autonomous driving scenario is shown.

[0044] exist Figure 1 In the example, vehicle 101 (e.g., car, truck, or motorcycle) is provided with vehicle controller 102.

[0045] The vehicle controller 102 includes data processing components, such as a processor (e.g., a CPU (central processing unit)) 103 and a memory 104 for storing control software of the vehicle controller 102 and data operated by the processor 103 according to its operation.

[0046] For example, the storage control software includes instructions that cause the processor to implement the neural network 107 when executed by the processor 103.

[0047] The data stored in memory 104 may include image data from one or more image sources 105 (e.g., acquired by one or more cameras). The image may include a set of data representing one or more objects or patterns. One or more image sources 105 may, for example, output grayscale or color images of the vehicle environment. One or more image sources 105 may be responsive to visible or invisible light, such as, for example, infrared or ultraviolet light, ultrasonic or radar waves, or other electromagnetic or acoustic signals.

[0048] The vehicle controller 102 can determine the presence of objects based on image data, such as stationary objects like traffic signs or road markings and / or moving objects like pedestrians, animals and other vehicles.

[0049] The vehicle controller 102 can then control the vehicle 101 based on the result determined by the object. For example, the vehicle controller 102 can control the actuator 106 to control the speed of the vehicle, such as to actuate the vehicle's brakes.

[0050] Control can be performed based on object classification performed by neural network 107.

[0051] Figure 2 An example of a neural network 200 is shown, which can be used to classify an image into a predefined number of classes.

[0052] In this example, the neural network 200 includes an input layer 201, two layers 202a and 202b, and an output layer 203.

[0053] It should be noted that Neural Network 200 is a simplified example of a real deep neural network (such as a deep feedforward neural network) used for classification purposes, which can include many more processing nodes and layers.

[0054] The input data corresponds to input layer 201 and can generally be viewed as a multidimensional value array. For example, the input image can be viewed as a 2D value array corresponding to the pixel values ​​of the image.

[0055] The input from input layer 201 is then connected to processing node 204. A typical node 204 multiplies each input by a weight and sums the weighted values. Additionally, node 204 can add a bias to the sum.

[0056] Nodes 204 are typically followed by nonlinear activation functions 205, such as rectified linear units (ReLU). ) or sigmoid function ( The resulting value is then output to the next layer.

[0057] Layers 202a and 202b can be hidden layers, such as fully connected layers, as shown in... Figure 2 As shown, each node in one layer is connected to each node in another layer.

[0058] Layers can also be partially connected layers (or supplemented by partially connected layers), such as convolutional layers or pooling layers (typically followed by one or more hidden layers) in the case of convolutional neural networks (CNNs).

[0059] In convolutional layers, the input is modified by convolutional filters. These filters operate on a subset of the input data and can help extract features of the input data, such as specific shapes or patterns. The filters implemented by the convolutional layers cause several corresponding nodes 204 of the convolutional layer to receive input only from a portion of the previous layers.

[0060] A pooling layer can be viewed as a form of non-linear downsampling that reduces the dimensionality of data by combining the outputs of several nodes into a single node in the next layer (e.g., by taking the maximum value of the output).

[0061] In a neural network designed for classification (such as neural network 200), the output layer 203 receives values ​​from at least one previous layer, for example, from hidden layer 202b. These values ​​can then be processed by the output layer, for example, by applying a softmax function to it. ,in v i (i = 1, ..., K) are the values ​​received by the output layer) or a sigmoid function, which transforms them into probabilities. The highest probability value contained in the output vector corresponds to the class prediction.

[0062] In the following text, class prediction may also be referred to as prediction, predicted class labeling, or predicted classification labeling.

[0063] Therefore, the output vector of output layer 203 is a probability vector that, for each predefined class, indicates the probability that the image corresponds to a predefined class—for example, the image shows a predefined object. For instance, assuming there are 10 predefined classes (0, 1, ..., 9) for an input image of a digit, the output vector is a vector of 10 elements, where each element corresponds to the probability of the digit. The class prediction will be the digit with the highest probability in the output vector. Output layer 203 can output the entire vector of probability values, or it can output only the class prediction.

[0064] In order to classify images, a neural network 200 is first trained accordingly. In the case of car driving, this can be done based on a collection of traffic scenes such as urban landscapes.

[0065] In the examples above, it should be noted that one image is classified, for example, an image is classified as showing a pedestrian. However, in a driving scene, such as one captured by camera 105, an image typically includes multiple objects. Therefore, in such applications, dense prediction (or semantic segmentation) can be used, which can be viewed as classifying each pixel of the image. For example, some pixels might be classified as showing a pedestrian, while other pixels might be classified as showing another vehicle. Such dense prediction can be performed similarly using a neural network for image classification as explained above, the difference being that the output includes a class prediction for each pixel of the image, rather than a class prediction for each image. x The output can therefore be another image indicating the class prediction for each pixel (e.g., encoded by color, such as green for pedestrians, red for vehicles, gray for the background, etc.), i.e. It is an image that has multiple channels corresponding to possible class predictions. For semantic segmentation, CNNs can be used.

[0066] In many real-world scenarios, the presence, location, and appearance of an object are highly correlated with the contextual information surrounding it, such as the presence of other nearby objects or more global scene semantics. For example, in a city street scene, cyclists are more likely to co-occur with bicycles, and cars are more likely to appear in the sky and on roads below buildings. These semantic interrelationships are inherent in real-world data. Data-driven models, such as Deep Neural Networks, tend to exploit these statistical biases to improve their predictive performance. Effective and safe utilization of deep learning models for real-world applications (e.g., autonomous driving) requires a good understanding of these inherent contextual biases in the data and the extent to which the learned model incorporates them into its decision-making process. Otherwise, there is a risk of objects being misclassified, for example, because they appear in unusual locations.

[0067] The saliency method can explain the predictions of a trained model (e.g., a trained neural network) by highlighting the parts of the input that can be assumed to be highly relevant to the model’s predictions, i.e., by identifying the image pixels that contribute the most to the network’s predictions.

[0068] According to various embodiments, a scheme for saliency determination, namely a saliency method, is provided, which can be seen as an extension of saliency methods for image classification toward (pixel-level) dense prediction tasks, and which allows for the generation of spatially coherent interpretations (and spatial distinctions between prediction interpretations) by utilizing spatial information in dense predictions.

[0069] In the following example, the scheme is referred to as grid saliency, a perturbation-based saliency method. This method is based on a formula that serves as an optimization problem identifying the minimum undisturbed region of the image needed to preserve the network's predictions within the target object region. Because grid saliency allows for differentiation between objects and their associated context regions in the saliency map, it enables the generation of contextual interpretations for semantic segmentation networks, thereby discovering which contextual information has the greatest impact on class predictions within the target object region.

[0070] let Example: A prediction function implemented by a deep neural network 200, which takes a grid as input space. (For example, an input image in an input image space) is mapped to a grid output space. (For example, the output image in the output image space), where W and H These are the corresponding width and height of the input (image) and output (image), and and It is the number of input channels (e.g., 3 or 1 for an image) and the number of output prediction channels (e.g., the number of classes for semantic segmentation).

[0071] In the following text, for ease of explanation, we will only consider the image as input and the network's per-pixel dense predictions as output; that is, the input is the image. And the output is a dense prediction per pixel. Furthermore, for simplicity, we assume that the input and output spaces have the same dimensions.

[0072] According to an embodiment, for the input image x The determination of grid saliency can be viewed as being based on finding the grid that must be preserved in the image. x Minimal significance in (Figure) This is so that it can be targeted to classes (e.g., channels). Keep the request mask area Network prediction in The significance level is based onM The pixel value. For example, if M pixel values norm less than M 'pixel value Norm, then M Less than M '.

[0073] According to various embodiments, grid saliency is based on a perturbation saliency scheme. This means that the salient image regions (in the requested region) that are primarily responsible for the classifier's decision are determined by replacing portions of the image with non-informative pixel values ​​(i.e., perturbing the image) and evaluating the corresponding class prediction changes.

[0074] let p Marking from image x Remove salience M The disturbance function of external information (where external means with respect to it) M (Pixels that are not 1). For example, such a perturbation function could be... x and Interpolation between, where a This could be a constant color image, a Gaussian blur, or random noise. In this case, ,in" "Indicates the Hadama product."

[0075] It should be noted that, in practice, significance M It can be defined at a lower resolution than the input image to avoid adversarial artifacts, and can be upsampled to the input image resolution later. Furthermore, the perturbed image... The pixel values ​​can be clipped to maintain the range of the original image pixel value space.

[0076] Based on the above symbols, find the class c Predicted significance plot M The task can be formulated as follows optimization

[0077] (1)

[0078] in Marking norm, and Is class c Network prediction, i.e., for class (channel) c The output image's pixel values.

[0079] The first term on the right-hand side of equation (1) can be viewed as a masking loss that minimizes the salient image region, thereby perturbing the original image as much as possible. The second term acts as a preservation loss, which ensures that the perturbed image... Upper category c Network prediction At least achieve network prediction on the original, undisturbed input image The confidence level. Therefore, the second loss term can be regarded as the confidence level of the non-satisfied constraint. The penalty, therefore used in equation (1) .parameter control M The sparsity of the control, i.e. how strongly the magnitude of saliency is penalized. An example value is 0.05, but other values ​​may produce smaller or larger significance.

[0080] Furthermore, it is possible to limit the retention loss to the request mask in equation (1) according to the following formula. R In the saliency graph M The region of interest given in the text is the region of interest for the request. R The explanations for network predictions are spatially separated from the explanations for other predictions.

[0081] (2)

[0082] In the following text, the result of equation (2) is referred to as the grid saliency plot.

[0083] The grid saliency formula from equation (2) can be adapted to specifically provide a region of interest for the requested region. R Contextual interpretation is of particular interest for semantic segmentation because context typically serves as one of the main cues used in semantic segmentation networks.

[0084] Therefore, according to various embodiments, there is concern about the contextual interpretation of semantic annotation prediction, and it is assumed that... R It is to cover the input image x The region containing the object of interest. To optimize salient parts of the object context, an object request mask is used. R It is integrated into the perturbation function. For the request mask... R Disturbed image Only contains R Internal object information, and (for example, in the case of image a with a constant color) remove R All external contextual information.

[0085] To optimize (i.e., to determine significance), this new perturbed image is used. Instead of the image with the greatest disturbance And the context perturbation function is In other words, when the input image is disturbed, R The image information within is relevant (therefore, salience does not "need" include the requested area). R ).

[0086] kind c and the request object R The contextual saliency map is given by the following formula.

[0087] (3)

[0088] This can be viewed as an optimization of the saliency graph to select at least the request mask. R Internal targeting class c The minimum context required for the original prediction.

[0089] It should be noted that contextual saliency graphs can be viewed as a special case of grid saliency graphs because the perturbations take a special form.

[0090] Figure 3 The illustration shows the determination of salience according to an embodiment.

[0091] In 301, the neural network is trained for dense prediction, which is determined by a function. f Marking.

[0092] In 302, given the training used for f (Dense prediction task) network, input image x And for the input image and class c (e.g., output image channels) c ) network The prediction (figure) is used to select the target area in the prediction figure that requires visual interpretation. R .

[0093] In 303, these components are used to define an optimization problem according to equation (3), and solve the optimization problem to obtain a saliency map as the target region. R The post-hoc interpretation of the prediction.

[0094] The optimization problem can be solved using various optimization techniques (i.e., the saliency map can be optimized), such as stochastic gradient descent (SGD) with momentum or Adam. For example, with SGD, a momentum of 0.5 and a learning rate of 0.2 can be used for 100 steps, and the saliency map can be initialized with 0.5 for each pixel. Another example is the optimization of a coarse 16x32 pixel mask using SGD, with a learning rate of 1 for 80 steps.

[0095] In 304, the resulting saliency map is used for various purposes, such as as a visual interpretation of network predictions, and for debugging errors in trained networks and determining classification reliability.

[0096] For example, controller 102 can determine the reliability of the classification based on the saliency map, and can further process the class prediction based on the saliency map. For example, depending on the determined context, the class prediction can be accepted or discarded for further processing (e.g., vehicle control). For example, if the context of the classification is not meaningful (e.g., when the determined context does not include an image region that can be expected to be relevant), controller 102 can discard the classification, as explained below.

[0097] Figure 4 and 5 An example of salience determined according to an embodiment is shown.

[0098] exist Figure 4 In the image, the first image 401 shows the semantic segmentation of a first traffic scene determined by a neural network. The second image 402 shows the context of the class "rider" (determined by grid saliency), the third image 403 shows the context of the class "bicycle", and the fourth image 404 shows the context of the class "car".

[0099] exist Figure 5 In the image, the first image 501 shows the semantic segmentation of the second traffic scene determined by the neural network. The second image 502 shows the context of the "pedestrian" class, the third image 503 shows the context of the "leg" class, and the fourth image 504 shows the context of the "car" class.

[0100] It can be seen that grid saliency can be used to interpret correct predictions from context: in Figure 4 In the second image 402, the neural network views the motorcycle to correctly predict the rider class. However, the contextual interpretation of grid saliency used for semantic segmentation can also explain incorrect predictions: in Figure 5 In the example, the rider's upper body was incorrectly predicted as a pedestrian. For example, in... Figure 5 As can be seen in the second image 502, for this prediction, the rider's leg was correctly classified ( Figure 5 In contrast to the third image (503), the bicycle is not prominent.

[0101] Here is one of the situations in which controller 102 can discard a classification: Since bicycles are not significant for the rider's classification, but are significant for the rider's legs, the controller may suspect that bicycles have been incorrectly ignored for the rider as a pedestrian classification, and may ignore that classification.

[0102] Grid saliency can be used, for example, to enhance a trained model (e.g., any dense prediction network) by detecting biases picked up by the trained network via the obtained grid saliency map. The obtained grid saliency map can be used for anomaly detection by examining inconsistencies in saliency interpretations across the same semantic object class and identifying the cause of the anomaly via the obtained saliency map. More generally, it can be used to explain any unexpected model behavior; for example, in the case of incorrect predictions for a corner case, grid saliency can be used to understand which part of an image (any data sample) is the "outlier" (anomaly) and causes the failure.

[0103] Mesh saliency can be particularly useful in online diagnostic tools, such as in controllers like vehicle controller 102.

[0104] It should be noted that although the saliency of the grid was determined by perturbation in the example above, it can also be determined based on other schemes, such as gradient-based methods as explained below.

[0105] let Labels relative to the input image The gradient of the predicted fc for class c. For the significance of VG (ordinary gradient), IG (integral gradient), and SG (smooth gradient), the significance maps are calculated as follows:

[0106] (4)

[0107] Where n is the approximate number of steps for IG or the number of samples for SG, and Indicates that it has a standard deviation Gaussian noise.

[0108] Similar to perturbation-based schemes, saliency M For the requested area R The interpretation given by the network prediction is spatially separate from other predictions. For a given input x and a binary request mask... R Request area R China targets class c The normalized network prediction score is labeled as

[0109] .

[0110] Similar to For grid saliency, define It utilizes Replace the equation (4) And directly generate grid saliency For gradient-based contextual saliency, similar to perturbation-based schemes, only salient pixels outside the object region (i.e., region of interest) are considered.

[0111] (5)

[0112] It should be noted that gradient-based saliency maps tend to be noisy. Therefore, to circumvent this, a gradient-based saliency map can be used on top of the saliency map. The spatial mean filter of kernel and stride, where It is the resolution of the saliency map based on the perturbation.

[0113] In summary, according to various embodiments, the following are provided: Figure 6 The method is illustrated in the figure.

[0114] Figure 6 A flowchart 600 illustrating a method for operating a neural network according to an embodiment is shown.

[0115] In 601, neural network input sensor data is used to determine neural network output data, wherein the neural network input sensor data includes multiple input data points, each input data point being assigned one or more input data point values, and wherein the neural network output data includes multiple output data points, each output data point being assigned one or more output data point values, and wherein each output data point is associated with one or more input data points.

[0116] In 602, a portion of the output data points is selected from multiple output data points to form a region of interest, wherein the region of interest includes multiple output data points.

[0117] In step 603, for each of at least some output data points outside the region of interest, a contribution value representing the contribution of one or more input data points associated with the output data points of the neural network is determined, the neural network determining the output data point value assigned to the output data points in the region of interest.

[0118] According to various embodiments, in other words, the salience of input data values ​​is determined for a region of interest in the output. For example, the preservation loss used as the basis for determining contribution values ​​(which can be viewed as a salience determination result, such as forming a (grid) salience map) is limited to the region of interest.

[0119] For example, determining the contextual interpretation of semantic segmentation based on contribution values. In the example above, contribution values ​​could be, for example,... The value of , for example, a value between 0 and 1, represents the contribution of a pixel (or generally one or more input data point values ​​corresponding to an output data point) to the neural network's decision about one or more data point values ​​of another data point (e.g., classification). Here, a contribution value of 0 indicates the minimum contribution, and a contribution value of 1 indicates the maximum contribution.

[0120] Figure 6 The methods described herein can be executed by one or more processors. The term "processor" can be understood as any type of entity that allows the processing of data or signals. For example, data or signals can be disposed of according to at least one (i.e., one or more) specific functions performed by a processor. A processor can include analog circuits, digital circuits, composite signal circuits, logic circuits, microprocessors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable gate arrays (FPGAs), integrated circuits, or any combination thereof. Any other means of implementing the corresponding functions (described in more detail below) can also be understood as a processor or logic circuit. It will be understood that one or more of the method steps described in detail herein can be executed (e.g., implemented) by a processor through one or more specific functions performed by the processor.

[0121] Although in the example above, the neural network is a dense prediction network for vehicles with camera images as input data, Figure 6 The scheme can be used to receive sensor signals from any sensor, that is, to operate on any kind of input sensor data (such as video, radar, lidar, ultrasound and motion).

[0122] It should be noted that the input data is not limited to images, but can also be applied to any image-like data (e.g., data constructed in the form of one or more two-dimensional or higher-dimensional arrays), such as sound spectrograms, radar spectrograms, ultrasound images, etc. In addition, raw 1D (e.g., audio) or 3D data (video, or RGBD (red, green, blue, and depth) data) can also be used as input.

[0123] The generated saliency determination results (e.g., grid saliency maps) can be used as the basis for calculating control signals used to control physical systems (such as computer-controlled machines, robots, vehicles, household appliances, power tools, manufacturing machines, personal assistants, or access control systems) or systems used to transmit information (such as monitoring systems or medical (imaging) systems) in order to interpret and understand the decision-making processes of dense predictive networks used in the aforementioned physical systems. It does this by generating low-level visual interpretations (e.g., grid saliency maps). In particular, the resulting images allow for the identification of the causes of anomalies by analyzing the interpretations given by the resulting images.

[0124] According to various embodiments, the input data is in the form of an image (or in an image-like data structure). Therefore, according to various embodiments, a method for analyzing a neural network, executed by one or more processors, is provided, comprising: determining a result image for an input image by means of a neural network, wherein the result image comprises multiple pixels, each pixel having one or more pixel values; selecting a region of interest (ROI) of the result image; and determining a region of the result image outside the ROI, depending on the region of the result image, the neural network having determined the pixel values ​​of pixels within the ROI.

[0125] Although specific embodiments have been illustrated and described herein, those skilled in the art will appreciate that various alternatives and / or equivalent implementations may be made in place of the specific embodiments shown and described without departing from the scope of the invention. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that the invention be limited only by the claims and their equivalents.

Claims

1. A method performed by one or more processors for operating a neural network, the method comprising: • determining, using a neural network, neural network output data for neural network input sensor data, wherein the neural network input data sensor data comprises a multiplicity of input data points, each input data point being assigned one or more input data point values, and wherein the neural network output data comprises a multiplicity of output data points, each output data point being assigned one or more output data point values, and wherein each output data point is associated with one or more input data points, wherein the input data points are structured as an input image and the output data points are structured as an output image, and wherein the one or more input data point values specify pixel values and the output data point values of each output data point specify a classification of the one or more input data points associated with the output data point; • selecting a portion of the output data points from the multiplicity of output data points to form a region of interest, wherein the region of interest comprises a plurality of output data points; and • for each of at least some output data points outside the region of interest, determining a contribution value representing a contribution of the one or more input data points associated with the output data point to a determination by the neural network of the output data point value assigned to the output data point in the region of interest.

2. The method of claim 1, wherein the output data points with the contribution values and the output data points of the region of interest are presented to a user.

3. The method of claim 1, wherein the output data points with the contribution values are compared to the relative positions of the output data points of the region of interest.

4. The method of any one of claims 1 to 3, wherein each output data point is associated with one or more input data points by mapping a pixel position in the input image to a pixel position in the output image.

5. The method of any one of claims 1 to 3, comprising determining the contribution value of a data point based on a measure of the impact that perturbations of the input data point values of the input data points associated with the output data point have on the one or more output data point values of the output data point.

6. The method of any one of claims 1 to 3, wherein, The contribution value is determined based on a trade-off between a total measure of the contribution value and a preservation loss that occurs when determining the output data point value assigned to the output data point in the region of interest, and when disregarding information in the input data values based on the contribution value.

7. The method of any one of claims 1 to 3, wherein the portion of the output data points selected to form the region of interest is a proper subset of the multiplicity of output data points of the output data.

8. The method of claim 1, wherein the neural network is trained for image segmentation, wherein the output image represents a semantic segmentation, and wherein the region of interest corresponds to one or more segments of the semantic segmentation.

9. The method of claim 1, wherein the output image is a depth image or a motion image.

10. The method of any one of claims 1 to 3, comprising generating a saliency map representing the contribution values, and wherein the contribution values are pixel values of the saliency map.

11. An apparatus comprising means for carrying out the method of any one of claims 1 to 10.

12. A computer program product comprising instructions which, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 10.

13. A computer-readable storage device having stored thereon the computer program product of claim 12.

14. A vehicle comprising: • at least one image sensor configured to provide digital image data; and • a driver assistance system comprising a neural network operating in accordance with any one of claims 1 to 10, wherein the neural network is configured to classify the digital image data, and wherein the driver assistance system is configured to control the vehicle based on the classified digital image data and the contribution value. ​

Images