Apparatus and methods for generating counterfactual data samples for neural networks

Abstract

This disclosure relates to methods and apparatus for generating counterfactual data samples for a neural network. According to various embodiments, a method for generating counterfactual data samples for a neural network based on input sensor data samples is described, comprising: using the neural network to determine a class prediction for the input sensor data samples; determining, in addition to the class prediction, an estimate of the uncertainty of the class prediction; generating candidate counterfactual data samples for which the neural network determines class predictions different from those of the input sensor data samples; determining a loss function, wherein the loss function includes an estimate of the uncertainty of the class predictions made by the neural network for the candidate counterfactual data samples; modifying the candidate counterfactual data samples based on the determined loss function to obtain counterfactual data samples; and outputting the counterfactual data samples.

Description

Technical Field

[0001] This disclosure relates to methods and apparatus for generating counterfactual data samples for neural networks. Background Technology

[0002] Deep learning models using neural networks are being used more and more widely; however, before they are deployed in the field, especially when they are applied to high-risk tasks such as autonomous driving or medical diagnosis, it is crucial to understand how these models arrive at their results (predictions).

[0003] To understand a model, it's crucial to be able to quantitatively determine how well it has learned the desired input-output relationships. However, deep learning models and techniques typically lack the metrics and practices to measure this performance, often resulting in models that are over-parameterized compared to the amount of data available. This is especially true for models used in classification tasks, where a large number of model parameters allows the decision boundaries between object classes to become increasingly complex and non-linear. This often leads to a significant gap between what the model actually learns and what the model implementer believes it has learned.

[0004] In Alex Kendall and Yarin Gal's paper "What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision" The method described in the paper provides an estimate of the prediction accuracy of the neural network and thus helps to assess what the model / neural network has truly learned.

[0005] The paper "Interpretable Explanations of Black Boxes by Meaningful Perturbation" by Ruth C. Fong and Andrea Vedaldi describes a method to increase the interpretability of deep learning models. Such and similar methods primarily focus on image classification tasks and generate saliency maps that show which parts / pixels of an image are most responsible for the neural network's classification.

[0006] However, a significant problem in searching for meaningful explanations is typically the existence of adversarial effects, which consist of small changes to the input data sample leading to large changes in the classification score. These small changes, such as just a few pixels in the input image, may be irrelevant to the part representing the semantic object and therefore may not help in interpreting the classification score.

[0007] Given the above, it is desirable to obtain meaningful interpretations of the results (predictions) of deep learning models, particularly interpretations that describe the model decision boundaries near a given input sample.

[0008] Furthermore, what is desirable is an indication of the confidence level of the reliability of the model results (predictions). Summary of the Invention

[0009] The method and apparatus of the independent claims allow for the generation of counterfactual data samples, thereby providing meaningful interpretations for the predictions of neural networks. This allows for the measurement of the reliability of the neural network's predictions based on the differences between the counterfactual data samples and the corresponding input sensor data samples. For the counterfactual generation apparatus, the interpretation and measurement of the reliability of the category predictions obtained from the neural network can be further used for the control of the device, such as vehicle control for autonomous driving.

[0010] The following describes further examples:

[0011] A method for generating counterfactual data samples for a neural network based on input sensor data samples may include: using the neural network to determine a class prediction for the input sensor data samples; determining, in addition to the class prediction, an estimate of the uncertainty of the class prediction; generating candidate counterfactual data samples for which the neural network determines class predictions different from those for the input sensor data samples; determining a loss function, wherein the loss function includes an estimate of the uncertainty of the class prediction of the candidate counterfactual data samples by the neural network; modifying the candidate counterfactual data samples based on the determined loss function to obtain counterfactual data samples; and outputting the counterfactual data samples.

[0012] The method mentioned in this paragraph provides a first example.

[0013] Determining an estimate of the uncertainty in class prediction and adding that estimate to the loss function has the following effect: the generated counterfactual data samples are less likely to be unwanted and / or (for finding an explanation for the classification) useless data samples, such as, for example, adversarial data samples. This means that the generated counterfactual data samples will be more likely to be counterfactual data samples that the network has high confidence in for its classification.

[0014] This enables the use of the generated counterfactual data samples to find meaningful (human-understandable and / or seemingly reasonable) explanations for class predictions, and / or to identify the root causes of classification, and / or to show users of the method which significant and meaningful changes are necessary for the neural network to alter the classification score of a given input sensor data sample.

[0015] This method may include iteratively generating a sequence of candidate counterfactual data samples, wherein, in each iteration, based on a determined loss function, the current candidate counterfactual is modified into a subsequent candidate counterfactual, or the current candidate counterfactual is accepted as a counterfactual data sample. The features mentioned in this paragraph, combined with the first example, provide a second example.

[0016] (For example, by using multiple iterations) Iteratively determining counterfactual data samples allows for the discovery of high-quality counterfactual data samples.

[0017] The method may include modifying the current candidate counterfactual to a subsequent candidate counterfactual if the determined loss function is above a predetermined threshold, and accepting the current candidate counterfactual if the determined loss function is below the predetermined threshold. A third example is provided by combining the features mentioned in this paragraph with the first or second example.

[0018] Even if the true minimum (optimal value) of the loss function has not yet been found, the iterative determination of the loss function is stopped as soon as a good approximation of it is found. This ensures that the generation of counterfactual data samples will stop after a reasonable period of time.

[0019] It should be noted that the goal of iteratively determining the loss function is usually to minimize the loss function; however, in some cases, the goal may be to maximize the loss function.

[0020] When the goal of iteratively determining the loss function is to maximize the loss function, the method may include modifying the current candidate counterfactual to a subsequent candidate counterfactual if the determined loss function is below a predetermined threshold, and accepting the current candidate counterfactual if the determined loss function is above the predetermined threshold.

[0021] The method may include: a loss function comprising (at least) a first term containing the output of the neural network for a sample of input sensor data and a second term containing an estimate of the uncertainty in the category prediction. A fourth example is provided by combining the features mentioned in this paragraph with any of the first through third examples.

[0022] This method may include: a loss function that further includes a term representing the difference between the input sensor data samples and the counterfactual data samples. The features mentioned in this paragraph, combined with any of the first through fourth examples, provide a fifth example.

[0023] Knowing the difference between the input sensor data samples and the counterfactual data samples allows for the finding of the “minimum” counterfactual, i.e., the counterfactual that differs from the input sensor data samples as little as possible. This allows the counterfactual data samples to approach the decision boundary, and thus allows for precise testing of the decision boundary and / or improvement of the decision boundary when the generated counterfactual data samples are used for further training.

[0024] The method may include: a loss function further comprising a term indicating the target category of the counterfactual data sample generated by the neural network. The features mentioned in this paragraph, combined with any of the first through fifth examples, provide a sixth example.

[0025] Indicating the target category of a counterfactual data sample allows the neural network to be forced to classify the sample using the desired / specific target category. This helps test how the neural network distinguishes a particular category. In particular, the target category of the counterfactual data sample can be close to (for humans) the category of the input sensor data sample; for example, the category of the input sensor data sample could be bicycle, while the target category of the counterfactual data sample could be motorcycle.

[0026] This method can include modifying candidate counterfactual data samples using gradient descent based on a loss function. The seventh example is provided by combining the features mentioned in this paragraph with any of the first through sixth examples.

[0027] The method may include: generating candidate counterfactual data samples by applying a mask to input sensor data samples, and wherein modifying the candidate counterfactual data samples includes modifying the mask. The features mentioned in this paragraph, combined with any of the first through seventh examples, provide an eighth example.

[0028] The advantage of using masks as the basis for counterfactual generation processing is that the output is a saliency map that highlights portions of the input sensor data samples (e.g., pixels of the input image) that bear the greatest responsibility for changes in classification (scores). Masks applied to the input sensor data samples can be specifically chosen in such a way that the perturbations that cause changes in classification have certain characteristics, such as being small, speckled, reducing image artifacts, etc.

[0029] This method may include: generating candidate counterfactual data samples using a generative adversarial network (GAN), wherein modifying the candidate counterfactual data samples includes modifying the input to the GAN. The features mentioned in this paragraph, combined with any of the first through eighth examples, provide a ninth example.

[0030] This method may include: the neural network input sensor data samples being image data samples. The tenth example is provided by combining the features mentioned in this paragraph with any of the first through ninth examples.

[0031] Applying this method to images can, for example, allow for object classification in autonomous driving or visual inspection systems. In particular, it can allow for finding meaningful interpretations for object classification used in autonomous driving and visual inspection systems.

[0032] The method may include: the neural network being a Bayesian neural network (BNN), wherein the estimation of uncertainty is derived from the prediction uncertainty arising from the weight probability distribution of the BNN. The features mentioned in this paragraph, combined with any of the first through tenth examples, provide an eleventh example.

[0033] Using BNNs as neural networks to classify input sensor data samples enables the capture of at least a portion of the uncertainty contained in the category predictions, since these are no longer point estimates but distributions. Furthermore, using BNNs makes it possible to use uncertainty measures based on this induced prediction uncertainty.

[0034] The method may include: determining a difference between an input sensor data sample and a counterfactual data sample; storing the determined difference between the input sensor data sample and the counterfactual data sample in a storage device; and controlling the device based on at least the stored difference. A twelfth example is provided by combining the features mentioned in this paragraph with any of the first through eleventh examples.

[0035] Based on the differences between stored input sensor data samples and counterfactual data samples, the device's controller can determine what modifications need to be made to the neural network to change the classification of the input sensor data samples with high confidence. For example, in manufacturing processes / systems, such as... Figure 1 In the manufacturing process / system illustrated in the diagram, which employs a (visual) inspection system that relies on learned model predictions, the control parameters of the manufacturing process can be adjusted after considering the reasons why a sample (part) is classified as "bad." The stored discrepancies further enable the detection of unexpected or critical anomalies. Upon detecting such anomalies, the controller can, for example, stop the manufacturing process, place the process in a safe mode, or prompt a human operator to take over control.

[0036] A method for determining the reliability of a category prediction determined using a neural network for two or more input sensor data samples may include: generating a counterfactual data sample for each input sensor data sample according to any one of the first through twelfth examples; determining the difference between the input sensor data sample and the corresponding counterfactual data sample for each input sample; calculating at least one statistic of the determined difference between the input sensor data sample and the counterfactual data sample; comparing the at least one statistic against at least one predefined criterion; and determining the reliability of the category prediction made by the neural network based on the result of the comparison. A thirteenth example is provided by combining the features mentioned in this paragraph with any one of the first through twelfth examples.

[0037] The method in Example 13 provides a safe way to determine how much “energy” (or “work”) a model needs to classify a given input differently. By calculating statistics (such as mean and variance) about the “energy” required to modify several input sensor data samples into counterfactual data samples and comparing them to a previously defined threshold, it is possible to identify potentially mislabeled / misclassified input sensor data samples, or where the network is uncertain about classification.

[0038] For example, if the amount of "energy" required to modify an input sensor data sample is significantly lower than the expected (average) amount of "energy" required to modify the input sensor data sample into a counterfactual data sample, the generated counterfactual data sample may be adversarial. In such cases, the generated counterfactual data sample may be discarded and / or the input sensor data sample may be labeled as (potentially) misclassified. Furthermore, depending on the amount of "energy" required to misclassify the input sensor data sample, the system can automatically label potentially misclassified input sensor data samples.

[0039] Therefore, this method can be particularly used for verification and validation processing of systems that use neural networks to classify input sensor data samples.

[0040] A method for training a neural network may include: providing training sensor data samples of a training dataset; training the neural network using the training dataset; generating one or more counterfacts based on any of the first through eleventh examples; adding the generated one or more counterfacts to the training dataset to obtain an enhanced training dataset; and training the neural network and / or another neural network using the enhanced training dataset. The fourteenth example is provided by combining the features mentioned in this paragraph with any of the first through thirteenth examples.

[0041] The method in the fourteenth example enables the refinement of existing models / datasets to obtain better-trained and more robust neural networks, or the training of new models / neural networks from scratch. In particular, when the decision boundary of the original model is highly nonlinear near the input sensor data samples (indicating samples that may be misclassified), using newly generated counterfactual data samples as additional data samples in the enhanced training dataset will help smooth the decision boundary.

[0042] Methods for determining the reliability of category predictions may include: the input sensor data samples and the counterfactual data samples being data arrays comprising individual data elements, and the difference between the input sensor data samples and the corresponding counterfactual data samples being the sum of the differences between the data elements of the input sensor data samples and the elements of the corresponding counterfactual data samples. The features mentioned in this paragraph, combined with the fourteenth example, provide the fifteenth example.

[0043] Methods for determining the reliability of category predictions may include: the at least one predefined criterion being a predefined threshold. The features mentioned in this paragraph, combined with examples fourteen or fifteen, provide a sixteenth example.

[0044] Methods for determining the reliability of category predictions may include outputting an alarm signal when at least one predefined criterion is not met. The features mentioned in this paragraph, combined with any of the examples fourteen through sixteen, provide a seventeenth example.

[0045] The counterfact generation device can be configured to execute the methods of any of the first through seventeenth examples. The device mentioned in this paragraph provides an eighteenth example.

[0046] The vehicle may include: at least one sensor providing samples of input sensor data and a driver assistance system configured to generate counterfactual data samples based on any of the first through seventeenth examples, wherein the driver assistance system is configured to control the vehicle based on the difference between the at least one input sensor data sample and the generated counterfactual data sample. The features mentioned in this paragraph provide a nineteenth example.

[0047] The vehicle may include: actuators for controlling the vehicle, including controlling the vehicle. The features mentioned in this paragraph, combined with the nineteenth example, provide a twentieth example.

[0048] The vehicle may include: input sensor data samples being images, and controlling the vehicle based on the difference between the input sensor data samples and counterfactual data samples, including determining whether the difference semantically corresponds to a category prediction for the image. The features mentioned in this paragraph, combined with the nineteenth or twentieth example, provide the twenty-first example.

[0049] A computer program may have program instructions configured to, when executed by one or more processors, cause the one or more processors to perform one or more methods according to the first example to the seventeenth example.

[0050] Computer programs can be stored in machine-readable storage media.

[0051] The methods described in one or more of the first to seventeenth examples can be computer-implemented methods. Attached Figure Description

[0052] In the accompanying drawings, the same reference numerals are used throughout different views and generally refer to the same parts. The drawings are not necessarily drawn to scale; instead, the emphasis is usually placed on illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings, wherein:

[0053] Figure 1 An exemplary manufacturing system for detecting defective parts is shown;

[0054] Figure 2 An example of object classification in an autonomous driving environment is shown;

[0055] Figure 3 An example of a neural network is shown;

[0056] Figure 4 The counterfact generation arrangement is shown;

[0057] Figure 5 A flowchart illustrating an exemplary method for determining the reliability of results output by a neural network is shown.

[0058] Figure 6 A flowchart illustrating an exemplary method for generating counterfactual data samples using a neural network is shown. Detailed Implementation

[0059] The following detailed description relates to the accompanying drawings, which illustrate by way of illustration specific details and aspects in which the invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made, without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

[0060] The various examples will be described in more detail below.

[0061] Figure 1 A manufacturing system 100 is shown, illustrating an example for detecting defective parts.

[0062] exist Figure 1 In the example, component 101 is located on assembly line 102.

[0063] The controller 103 includes data processing components, such as a processor (e.g., a CPU (Central Processing Unit)) 104 and a memory 105, which stores control software that the controller 103 operates according to and data that the processor 104 operates on.

[0064] In this example, the stored control software includes instructions that, when executed by processor 104, cause processor 104 to implement inspection system 106, which includes a counterfact generation system and contains neural network 107 (or may contain multiple neural networks 107).

[0065] Input data samples can be formed from data arrays, where each data array includes multiple individual data elements.

[0066] Counterfactual data samples are generated data samples that have a different classification from the corresponding input data sample (when classified by neural network 107) and / or a classification score that is significantly different from the corresponding input data sample (e.g., the classification score drops below a predetermined threshold).

[0067] The data stored in memory 105 may include, for example, image data from one or more image sources 108 (e.g., a camera). An image may include a collection of data representing one or more objects or patterns. One or more image sources 108 may, for example, output one or more grayscale or color images of each of components 101. One or more image sources 108 may respond to visible light or non-visible light such as infrared or ultraviolet light, ultrasonic or radar waves, or other electromagnetic or acoustic signals.

[0068] The image data is classified by neural network 107. The counterfactual generation system included in the inspection system 106 provides an estimate of how reliable the classification performed by neural network 107 is.

[0069] It should be noted that classifying an image can be considered equivalent to classifying the objects shown in the image. If the original image shows multiple objects or patterns, segmentation (possibly by another neural network) can be performed such that each segment shows one object or pattern, and the segments are used as input to the image classification neural network.

[0070] The controller 103 can determine a defect in part 101 based on image data from one or more image sources 108. For example, if a part meets all quality criteria, the neural network 107 classifies the part as "good"; otherwise, if the part does not meet at least one quality criterion, the part is classified as "bad".

[0071] The inspection system 106 can further provide an explanation for the classification of component 101 as "bad". By design, the explanation can be constrained to a set of discrete possibilities, such as physical defects in a specific location of the component or changes in control conditions such as lighting during manufacturing.

[0072] If the controller 103 has determined that component 101 is defective, it can send a feedback signal 109 to the error handling module 110. The feedback signal 109 contains an explanation of why component 101 was determined to be defective, i.e., information representing the cause (such as the characteristics of the component).

[0073] The error handling module 110 can then use the feedback signal 109 to adapt the manufacturing system / process accordingly. For example, it may indicate that a part has a specific physical defect at a particular location. In this case, the error handling module 110 can modify the operating parameters of the manufacturing process, such as the applied pressure, heat, welding time, etc., to reduce the risk of such failures. This adaptation of the operating parameters of the manufacturing process in response to a defective part can be viewed as similar to a reinforcement learning method.

[0074] If the interpretation generated by the inspection system 106 is, for example, unexpected or critical according to predefined (and / or user-defined) criteria, the error handling module 110 can control the system (manufacturing process) to operate in a safe mode after receiving the feedback signal 109.

[0075] Furthermore, this interpretation can be used to determine how "bad" images must be modified so that neural network 107 classifies them as "good." This information can then be used to generate new data samples that can be incorporated into future training datasets to refine the deep learning model used (or train another model). Counterfactual data samples generated by the counterfactual generation system included in inspection system 106 can also be added to future training datasets to refine the deep learning model used.

[0076] It should be noted that the "good" / "bad" classification used to provide explanations is just one specific example of how a counterfactual generation system might be helpful. Other examples include semantic segmentation of complex scenarios, standard object classification, scene recognition, and so on.

[0077] In addition to Figure 1 Beyond the illustrated system, machine learning systems can be trained to learn the optimal operating parameter settings for manufacturing systems / processes by learning the correspondence between interpretations and operational parameter settings.

[0078] and Figure 1 The system illustrated in the figure can be used in other technological fields, such as access control systems, computer-controlled machines, such as robots, home appliances, power tools, or personal assistants.

[0079] Figure 2 Example 200 for object detection in autonomous driving scenarios is shown.

[0080] exist Figure 2 In the example, vehicle 201 (e.g., car, truck, or motorcycle) is provided with vehicle controller 202.

[0081] The vehicle controller 202 includes data processing components, such as a processor (e.g., a CPU (central processing unit)) 203 and a memory 204, which stores data operated by the vehicle controller 202 according to its control software and the processor 203.

[0082] For example, the stored control software includes instructions that, when executed by processor 203, cause the processor to implement counterfactual generation system 205 and neural network 206 (or possibly multiple neural networks 206).

[0083] The data stored in memory 204 may include input sensor data from one or more sensors 207. For example, the one or more sensors 207 may be one or more cameras acquiring images. The images may include a set of data representing one or more objects or patterns. The one or more sensors (cameras) 207 may, for example, output grayscale or color images of the vehicle environment. The one or more sensors 207 may respond to visible light or non-visible light such as infrared or ultraviolet light, ultrasonic or radar waves, or other electromagnetic or acoustic signals. For example, sensor 207 may output radar sensor data that measures the distance to objects in front of and / or behind vehicle 201.

[0084] The neural network 206 can determine the presence of an object based on input sensor data (e.g., image data), such as stationary objects (e.g., traffic signs or road markings) and / or moving objects (e.g., pedestrians, animals, and other vehicles).

[0085] The counterfact generation system 205 can provide counterfact data samples for the input image, and in particular for each of the identified objects. The counterfact data samples are generated data samples that have a different classification (when classified by the neural network 206) and / or a significantly different classification score (e.g., the classification score drops below a predetermined threshold) than the corresponding input data samples.

[0086] The generated counterfactual data samples can be data arrays, where each data array comprises multiple individual data elements. In particular, the counterfactual data samples can be generated images that have a small, preferably minimal, difference from the corresponding input data samples, while having a different classification.

[0087] The difference between the counterfactual data sample and the associated input data sample can be expressed (according to the saliency representation method) as a saliency map. A saliency map can interpret the neural network's prediction (classification) by highlighting the parts of the input data sample that are likely highly correlated with the prediction, i.e., by identifying the image pixels that contribute the most to the neural network's prediction.

[0088] Vehicle 201 can be controlled by vehicle controller 202 based on the presence of the object and the determination of the generated counterfactual (short for counterfactual data sample) and / or the corresponding salience map. For example, vehicle controller 202 can control actuator 208 to control the speed of the vehicle, such as to actuate the vehicle's brakes, or can prompt a human driver to take over control of vehicle 201.

[0089] If neural network 206 detects an object in an input image received from one or more sensors 207, but the corresponding saliency map highlights parts of the image that are semantically unrelated to the object, the object determination by neural network 206 is deemed unreliable. For example, if the neural network determines that an object near vehicle 201 is a bus, but the corresponding saliency map highlights parts of the image used for object classification that are unrelated to the bus (e.g., traffic signs, another vehicle, or the sky), then the object identified as a bus is considered not to have been reliably classified.

[0090] In such a situation, where the classification of an object is determined to be unreliable, the controller 202 may request new images from one or more image sensors 207, or may activate emergency procedures, such as stopping the vehicle 201 by actuating the vehicle's brakes, or may prompt the human driver to take over control of the vehicle.

[0091] like Figure 1 and 2 As mentioned in the example of the control system illustrated, control is performed based on object classification executed by a neural network (or possibly multiple neural networks).

[0092] Figure 3 An example of a neural network 300 is shown, which can be used to classify input sensor data into a predefined number of categories.

[0093] In this example, the neural network 300 includes an input layer 301, two hidden layers 302a and 302b, and an output layer 303.

[0094] It should be noted that Neural Network 300 is a simplified example of a real deep neural network (such as a deep feedforward neural network) used for classification purposes, which can include many more processing nodes and layers.

[0095] The input data corresponds to input layer 301 and can generally be viewed as a multidimensional array of values. For example, the input image can be viewed as a two-dimensional array of individual values ​​corresponding to the pixel values ​​of the image.

[0096] The input from input layer 301 is then fed into processing node 304. Typically, node 304 multiplies each input by a weight and sums the weighted values. Additionally, node 304 can add biases to the sum. The weights can be provided as a distribution with a learned mean and variance, such as a Gaussian distribution. In such a case, the neural network is called a Bayesian neural network, or BNN.

[0097] Node 304 is typically followed by a non-linear activation function 305, such as a rectified linear unit or ReLU. ) or sigmoid function ( The resulting values ​​are typically input into the next layer.

[0098] like Figure 3 As shown, hidden layers 302a and 302b can be fully connected layers, where each node in one layer is connected to each node in the other layer.

[0099] Hidden layers can also be non-fully connected layers (or supplemented by non-fully connected layers), such as convolutional or pooling layers, recurrent layers, or self-attention layers in the case of convolutional neural networks.

[0100] In a neural network designed for classification (such as neural network 300), the output layer 303 receives values ​​from at least one of the preceding hidden layers (e.g., from hidden layer 302b). The output layer can then, for example, apply a softmax function to these values. , where v i The values ​​(i = 1,…,K, are the values ​​received by the output layer) are converted into probabilities using either a sigmoid function. The highest probability value contained in the output vector corresponds to the class prediction.

[0101] In the following text, category prediction may also be referred to as prediction, predicting category labels, or predicting classification labels.

[0102] Therefore, the output vector of output layer 303 is a probability vector that, for each of the predefined categories, indicates the probability that an input sensor data sample corresponds to a predefined category (e.g., one that represents a predefined object). For example, assuming an input image of a digit has 10 predefined categories (0, 1, ..., 9), the output vector is a vector of 10 elements, where each element corresponds to the probability of a digit. The category prediction will be the digit corresponding to the highest probability in the output vector. Output layer 303 can output the entire vector consisting of probability values, or it can output only the category prediction.

[0103] It should be noted that in the case of BNN, the output prediction of the neural network is a distribution, not a single (floating-point) number.

[0104] In order for the neural network 300 to classify input sensor data, especially image data, the neural network 300 is first trained based on the input training (image) data.

[0105] A core problem when using deep learning models, including deep neural networks, for classification tasks is the difficulty in explaining how the neural network achieves its classification output. This is especially true when a large number of model parameters allow the decision boundaries between object classes to become increasingly complex and nonlinear.

[0106] Predictions for input samples mapped near the decision boundary are less reliable than those further away. Assuming the input samples follow a distribution near a severely nonlinear boundary, the amount of "energy" (i.e., the amount of "work" or total difference) required to change the prediction of an input sample is less than in the case where the decision boundary is smooth and doesn't quite fit the sample distribution precisely. The amount of "energy" or "work" required to change the prediction of an input sample can be represented by, for example, the total number of (pixel) changes / edits to the input sample required to move the sample across the decision boundary (i.e., change its classification). Lower "energy" indicates a smaller change required in the input sample for the neural network to change its classification.

[0107] In such Figure 1 In the manufacturing system illustrated in the diagram, it is important to have a high degree of confidence and trust in classifying parts as “good” / “bad”, and therefore, a deep learning model may be needed to provide the rationale for its classification in a form that is understandable and / or reasonable to human operators.

[0108] These interpretations are typically in the form of visual interpretations, such as saliency maps, which highlight which parts of the image are most important for the neural network to obtain its classification predictions. Ideally, the pixels most responsible for classification predictions should be associated with the image parts representing semantic objects or portions—that is, with the image parts that a human operator would also see when determining what objects are in the image.

[0109] Counterfacts are data samples with classifications different from the input samples they are based on. Generating counterfacts can be used as a way to provide better explanations for neural network / model predictions. Counterfacts can be used to explore the causal relationship between the input and output of a given model. More specifically, counterfacts can be designed to investigate causal relationships in model behavior in the form of "if X does not happen, then Y will not happen," or alternatively, "if Y happens, then it implies that X has happened." This is done by analyzing the "energy" necessary to generate the counterfacts corresponding to the input data samples. For example, in... Figure 1 In the manufacturing system illustrated in the diagram, X can be interpreted as "the part is defective," while Y can be interpreted as "the part is classified as bad."

[0110] In practice, one problem with using counterfactual to increase the interpretability of neural network predictions is the existence of adversarial effects, which are small changes to the input data samples that lead to large changes in classification (scores).

[0111] In terms of boundary delineation, these adversarial effects can be interpreted as shortcuts taken by counterfactual generation processes. It may be useless for a human operator to observe that changing a few random pixels on a saliency map causes the neural network to change its classification. For the model's predictions to be understood and for the model to be validated and confirmed, it is important that the interpretation is meaningful; that is, the pixel most responsible for the change in classification (score) is related to the image portion representing the semantic object or part—that is, the image portion that a human operator might also see when classifying objects. However, the more complex the model, the less likely the interpretation will be meaningful and / or understandable to a human operator, and the more likely adversarial effects are the cause of changes in classification (score).

[0112] Therefore, it is desirable to generate counterfactuals that avoid adversarial effects, and thus provide a meaningful interpretation of the model decision boundary near a given input sample.

[0113] According to one embodiment, a counterfactual generation method and system are provided that allow for the reduction / mitigation of adversarial effects by employing an uncertainty regularization term on the counterfactual generator.

[0114] According to a further embodiment, a method for training a neural network is provided, which allows counterfactual data samples generated by the provided counterfactual generation method and system to be added to the training dataset. This allows the neural network to be more robust and smooth the (highly) nonlinear decision boundary near the input sensor data samples.

[0115] The following section uses image data to explain counterfactual generation processing. It should be noted that the proposed method and system are not limited to use with images as input data, but can also be used with other data types such as video, sound, radar, etc. In general, they can be used with any type of data for which an effective data generation process can be defined.

[0116] Figure 4 A counterfact generation arrangement 400 according to one embodiment is shown.

[0117] The counterfact generation arrangement 400 includes a (deep learning) classification model 401, hereinafter referred to as a neural network, for example, corresponding to neural network 300, and input data samples 402, hereinafter referred to as input images for model 401. The input data samples 402 are provided, for example, by one or more sensors, hereinafter referred to as image sensors.

[0118] The input image 402 is classified by the neural network 401, which outputs a classification O for each input image.

[0119] The counterfact generation device 400 performs counterfact (data) generation processing 403, that is, generates data samples with a classification O' that is different from the corresponding input image 402.

[0120] The explanation of the local behavior of the classification model 401 for a given input image 402 is derived by subjecting the input image to a counterfactual generation process 403 and evaluating the causal effect of O.

[0121] The exemplary counterfact generation process 403 is described below. Note that many other types of data (counterfact) generation processes, such as those modeled with the help of generative adversarial networks (GANs), can be used to construct counterfacts.

[0122] make Annotation, for example, is a prediction function implemented by a neural network 401, which maps the input space X to the output space Y.

[0123] In this example, the counterfactual generation process 403 is a mask during its operation. Refined processing. Mask m The size is the same as the input image, and the output is an element in the range [0, 1]. In each iteration of the counterfactual generation process 403, the mask is... m Applied to one of the input images 402 (for which a counterfactual will be generated), resulting in a perturbed image, the scalar value... With each pixel Related.

[0124] For masks mand input image pixel values ​​of the perturbed image u The following is given:

[0125] (1)

[0126] in It is a function representing the disturbance. For example, It can be independent and identically distributed (iid) Gaussian noise samples for each pixel, fixed color, salt and pepper noise, color desaturation, local affine transformation, etc.

[0127] For a given pixel u If the mask value at a pixel is equal to 1, the original image is displayed at that pixel. If the mask value is zero, the pixel value of a pre-selected perturbation target is displayed. The perturbation target is selected a priori and represents the "rules" by which the image can be modified. For example, if a completely black image is selected as the perturbation target, then the process P will attempt to add black pixels to the original input image I. Mask m Interpolation is performed between the two extreme values ​​of 0 and 1.

[0128] The counterfact generation process 403 uses simple, unrefined masks—such as random masks, default masks, or random inputs when using GANs to generate counterfacts—to generate first candidate counterfacts 404.

[0129] The goal of Counterfact Generation System 412 is to find the factors that cause the classification score. The minimum mask that decreases significantly, where c It is the category of the input data sample, and Refers to the category in the output layer c Input x The output classification score.

[0130] Ideally, the mask should have further characteristics, such as being speckled, which would be helpful for images in dense scene recognition. Ideally, the mask should also not rely too heavily on local image artifacts.

[0131] Taking all these factors into account, find a suitable mask. m This can be formalized, for example, into the following optimization problem:

[0132] (2)

[0133] In the current context, it should be noted that optimization is understood as optimization based on a certain time period or a certain number of iterations. Therefore, the above optimization problem can only be approximated, for example, by stochastic gradient descent.

[0134] In equation (2) above, Encourage disabling most of the mask, i.e., only deleting the input image. A small subset of the input image. The second term of Equation (2) forces the mask to change its classification score relative to the input, and the third term of Equation (2) has the effect of making the mask more speckled by regularizing it using the total variation norm. Finally, using the fourth term of Equation (2), the mask is calculated as the average of the jittered version of the original input image, thus avoiding its over-reliance on local image artifacts.

[0135] However, for data generation processing 403 using optimization problems such as those described by equation (2), unwanted masks / counterfactuals, especially adversarial masks, remain a problem despite the first and third terms of equation (2). That is, the masks returned by data generation processing 403 often represent adversarial masks that do not correspond to semantic objects that are understandable to human operators.

[0136] Therefore, in this example, to avoid unwanted / adversarial masks, the total loss function is solved. The minimization problem of 411 is used to determine the mask. m The minimization problem can be formulated, for example, as follows:

[0137] (3)

[0138] Therefore, in each iteration, m* should be smaller than in the previous iteration.

[0139] It should be noted that minimization is understood as minimization based on a certain time period or a certain number of iterations. Therefore, the above minimization problem can only be solved approximately.

[0140] In system 406, a total loss function 411 is calculated for the current candidate counterfact 404, and the result is fed back (illustrated by arrow 413) to the counterfact generation process 403, which accordingly adjusts the mask. m The calculated total loss function 411 is adapted, and then the adapted mask is applied to the input image 402, thereby modifying the current candidate counterfactual 404.

[0141] At point 405, the final iteration produces the final counterfactual data sample output.

[0142] The total loss function 411 in the k-th iteration includes the following: Item 407, marked Item 409 and optional by Further terms 410 are labeled, such as, for example, further loss regularization terms. Therefore, the total loss function to be minimized in the k-th iteration is... 411 is given by the following equation:

[0143] (4).

[0144] Depend on Item 407 is a measure of the (sub)loss during counterfactual generation and can be formulated, for example, as follows:

[0145] (5)

[0146] Where I is the input image 402, It is the modification of the candidate counterfact at iteration k. This is the output of data generation and processing step 403 in the k-th iteration. It is the input-to-output mapping of model 401 (e.g., the output of the softmax layer), and the function This involves choosing a loss metric, such as the p-norm or cross-entropy.

[0147] Therefore, item Specifically, it can be viewed as a term that includes the output of a neural network, such as the output of a softmax layer, i.e., the softmax value.

[0148] function It is optional and is a function that returns a counterfactual output of input I, therefore the counterfactual target category 408 can be a... The input is 407. For example, if the category "car" is encoded as [0, 1] in the output and the category "bicycle" is encoded as [1, 0], and the optimization aims to create bicycle counterfactuals starting from images of cars (car samples), then Therefore, item 407 occurs when using a function The counterfactual loss is defined when the target counterfactual category is explicitly specified. Alternatively, the function can be omitted when a specific counterfactual category is irrelevant. Then, the counterfactual for the car is not generated by forcing the model to explicitly misidentify it as a bicycle, but rather for any other category contained in the output Y.

[0149] To avoid adversarial masks, or at least reduce / mitigate their effects, a measure (estimate) of the uncertainty in the neural network's predictions is introduced as part of the total loss function 411. Additional item 409 (Uncertainty Loss).

[0150] Neural networks traditionally use point estimation, which gives the result of a given input. x and weight configuration W In the case of category yDistinguishing probability The probability of an input data sample belonging to a given category is expressed as a function of all other possible categories. In other words, the output of a neural network represents the probability that the input data sample belongs to the corresponding category. However, these point estimates are not suitable for interpretation as confidence or deterministic values, and cannot distinguish whether the uncertainty in category predictions is fundamentally due to characteristics of the data itself (e.g., sensor noise) or characteristics of the model (e.g., the model was not trained on a specific type of data that is part of the test set). In particular, the fact that the classification score output by the softmax layer of a neural network is not suitable for providing a measure of the confidence of the neural network in its predictions can be observed, for example, when the neural network outputs a classification score for an adversary, which typically has a high classification score despite being misclassified by the network.

[0151] Therefore, in order to obtain a measurement of the uncertainty of neural networks in their predictions, uncertainty estimation methods use confidence estimation to extend point prediction, that is, allow the model to measure how confident it is in the prediction while making the prediction, and further provide a measurement of where the uncertainty comes from, whether it is from the model or the data.

[0152] For this purpose, BNN can be used as a neural network 401, i.e., where each weight is modeled as a neural network with a learned distribution of mean and variance (typically a Gaussian distribution), which is used to provide a measurement of uncertainty.

[0153] In BNNs, the variance of the weight distribution describes the probability of observing a given weight, and thus measures the uncertainty as the distance from the learned mean. Output prediction is also modeled in this way. Finding the weights requires calculating the posterior probability. Here, X is the training point, Y is the corresponding prediction, and W is the weight we want to find. Calculating this posterior probability is typically tricky because it requires a number of terms. The calculation of the posterior probability marginalizes (averages) across all possible weight configurations, which is a tricky task in large neural networks with many weights. However, there are several methods to find a (numerical) approximation of this posterior probability.

[0154] However, it should be noted that uncertainty estimation does not require the use of BNN; any network that allows uncertainty estimation analysis can be used.

[0155] For example, in the case of a 401 neural network using point estimation, the Monte Carlo dropout method can be used. In the Monte Carlo dropout method, a certain percentage of nodes are randomly dropped out. Therefore, the prediction is no longer deterministic, but depends on which nodes are randomly selected to remain. Thus, given the same input image, the model can predict different values ​​each time; that is, Monte Carlo dropout can generate different predictions and interpret them as samples from a probability distribution, which is sometimes called a Bayesian interpretation.

[0156] The first of the two types of uncertainty mentioned above is often referred to as random uncertainty, which is caused by noise inherent in the sensor input data samples (such as sensor noise or motion noise). Even collecting more data cannot reduce random uncertainty.

[0157] The second type of uncertainty is often referred to as cognitive uncertainty. It takes into account the uncertainty in the model parameters, i.e., the quality of the model, and represents what the model cannot predict based on its training data. Cognitive uncertainty can be reduced by utilizing more training data; that is, given infinite training data, cognitive uncertainty will be zero.

[0158] It should be noted that using a BNN as a classifier allows for the output to be a predicted distribution, rather than a point estimate. When marginalizing on the (approximate) weighted posterior distribution, the uncertainty in the weights of the BNN introduces uncertainty in the prediction. Therefore, using a BNN as a classifier makes it possible to capture the prediction uncertainty of the neural network.

[0159] Random uncertainty is further divided into homoscedastic uncertainty and heteroscedastic uncertainty. In homoscedastic uncertainty, the uncertainty remains constant for different inputs. In heteroscedastic uncertainty, the uncertainty depends on the input to the model. Some inputs may have more noise output than others. For example, different areas in the scene have different uncertainties due to occlusion and glare.

[0160] Random uncertainty can also be considered in uncertainty measurements. This can be achieved, for example, by placing a distribution on the model's output—e.g., by adding Gaussian random noise to the output—whereby a homoscedastic model assumes a constant noise for each input point (pixel), while a heteroscedastic model assumes a varying noise depending on the region / input point (pixel). In non-Bayesian neural networks, the noise parameter is typically fixed, as part of the model weight decay, and is ignored, but it can be made dependent on the data and learned as a function of the data.

[0161] Therefore, uncertainty term 409 can model (reflect) cognitive uncertainty contained in the input image, accidental uncertainty contained in the input image, or a combination of both.

[0162] Depend on The labeled loss term 409 incorporates the uncertainty measure for the current (intermediate) candidate counterfactual data sample generated via the counterfactual generation process 403. A lower value for uncertainty term 409 indicates a higher confidence level of neural network 401 in its classification. This means that in the minimization process of the total loss function 411, data samples for which model 401 has a higher confidence level in its classification will be favored.

[0163] Therefore, this method discards potentially unwanted or useless counterfacts, especially those obtained using adversarial masks, and thus enables meaningful interpretations of how neural networks classify images.

[0164] Uncertainty term 409 can be determined, for example, based on the random deactivation Monte Carlo method mentioned above, where the overall prediction for a single input image 402 can be interpreted as a sample of a probability distribution. Uncertainty term 409 reflects whether the distribution has the expected form, such as being expected to be Gaussian, or whether it has an unexpected form, such as being expected to be Gaussian but having a larger tail than expected.

[0165] In the case of BNN, uncertainty term 409 can be determined based on the predicted distribution and reflects unexpected forms of the predicted distribution.

[0166] Furthermore, more sophisticated methods can be used to determine the uncertainty term 409, such as those described above in Alex Kendall and Yarin Gal's paper "What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision". The method described in "".

[0167] It should be noted that different uncertainty estimates Different requirements will be imposed on the original model 401 regarding training, architecture, etc.

[0168] Depend on The labeled total loss function 411 may include further (loss) terms 410, such as, for example, further loss regularization terms.

[0169] At the end of the minimization (optimization) process, at point 405, the final counterfact is output by the counterfact generation system 412.

[0170] The counterfact generation system 412 therefore includes a loop that utilizes counterfact generation processing 403 and iteratively optimizes candidate counterfacts 404 using system 406.

[0171] The difference between the final counterfactual output at 405 and the original input image 402 can be visualized as a saliency map, thus highlighting the differences at each spatial location necessary for the model to produce different classifications in which model 401 has a certain (high) confidence.

[0172] Depend on Figure 4 The illustrated counterfactual generation arrangement further enables a reliable measurement of how much "energy" is required for the model / neural network to classify different inputs. This is a useful metric for model validation and verification purposes. It can also be used to control the machine and / or processing.

[0173] Figure 5 Flowchart 500 is shown, illustrating an exemplary method for determining the reliability of a result output by a neural network by measuring how much “energy” is required to classify a given input image by changing the model.

[0174] In step 501, several samples, such as a batch of samples, are used as input to the neural network.

[0175] In step 502, run for each input sample Figure 4 The counterfact generation system illustrated in the figure.

[0176] In step 503, for each input sample, the total "energy" required to generate the (final) counterfactual is recorded. For example, the measurement of the total "energy" required could be the sum of all active elements in the (final) optimization mask.

[0177] In step 504, statistics of the results from step 503 are calculated. For example, the mean and variance of the total “energy” are calculated over all (or a subset) of the inputs.

[0178] In step 505, the statistics (such as mean and variance) calculated in step 504 are compared against certain criteria, such as predefined thresholds for the mean and variance of the desired total "energy", and if the criteria are not met, such as if the predetermined thresholds are not met, a feedback signal is provided, which may take the form of returning a single or several input samples.

[0179] In conclusion, such as Figure 6 As illustrated in the figure, according to various embodiments, a method is provided for generating counterfactual data samples for a neural network based on input data samples.

[0180] In step 601, a neural network is used to determine the category prediction for the input sensor data sample.

[0181] In step 602, a neural network is generated to predict candidate counterfactual data samples for categories that differ from the input sensor data samples.

[0182] In step 603, a loss function is determined, wherein the loss function includes a term representing the uncertainty of the category predictions made by the neural network for the candidate counterfactual data samples.

[0183] In step 604, the candidate counterfactual data samples are modified based on the determined loss function to obtain counterfactual data samples.

[0184] In step 605, a counterfactual data sample is output.

[0185] According to various embodiments, in other words, optimization processing based on a loss function generates and modifies candidate counterfactual data samples, wherein the loss function includes a term that considers the predictive uncertainty of the neural network; that is, it considers the random and / or cognitive uncertainty contained in the category prediction of the neural network. The result of modifying the candidate counterfactual data samples is a counterfactual data sample, which is then output.

[0186] It should be noted that "loss function-based optimization" means that in each iteration, the weights and / or other parameters are adjusted to reduce (optimize) the loss function. The optimization process may only last for a certain period of time or a certain number of iterations, meaning that the optimization problem may only be solved approximately.

[0187] Figure 6 The methods can be executed by one or more processors. The term "processor" can be understood as any type of entity that is allowed to process data or signals. For example, data or signals can be processed according to at least one (i.e., one or more) specific functions performed by a processor. A processor can include analog circuits, digital circuits, composite signal circuits, logic circuits, microprocessors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable gate arrays (FPGAs), integrated circuits, or any combination thereof, or formed therefrom. Any other means of implementing the corresponding functions, which will be described in more detail below, can also be understood as processors or logic circuits. It should be understood that one or more of the method steps described in detail herein can be executed (e.g., implemented) by a processor through one or more specific functions performed by that processor.

[0188] Figure 6The method can be used for neural networks that receive sensor signals from any sensor, i.e., to operate on any kind of input sensor data such as video, radar, LiDAR, ultrasound, and motion.

[0189] It should be noted that the input data is not limited to images, but can also be applied to any image-like data (e.g., data constructed in the form of one or more two-dimensional or even higher-dimensional arrays), such as sound spectrograms, radar spectra, ultrasound images, etc. In addition, raw 1D (e.g., audio) or 3D data (video or RGBD (red, green, and blue depth) data) can also be used as input.

[0190] Although specific embodiments have been illustrated and described herein, those skilled in the art will appreciate that various alternatives and / or equivalent implementations may be made in place of the specific embodiments illustrated and described without departing from the scope of the invention. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, the invention is intended to be limited only by the claims and their equivalents.

Claims

1. A computer-implemented method (600) for generating counterfactual image data samples (405) for an image data classification neural network (300, 401) based on input image data samples (402), the computer-implemented method (600) comprising: A neural network is used to determine (601) the category prediction for the input image data sample; Generate (602) a neural network for predicting candidate counterfactual image data samples for categories different from the input image data samples (404), wherein generating candidate counterfactual image data samples includes applying a mask to the input image data samples, and wherein modifying candidate counterfactual image data samples includes modifying the mask; Determine an estimate of the uncertainty of the category prediction for the candidate counterfactual image data sample (409), wherein the estimate of the uncertainty is derived from the prediction uncertainty induced by the Monte Carlo random deactivation method, or wherein the image data classification neural network is a Bayesian neural network (BNN), and the estimate of the uncertainty is derived from the prediction uncertainty induced by the weight probability distribution of the BNN. Determine (603) a loss function (411), wherein the loss function includes an estimate (409) of the uncertainty (409) of the category prediction made by the neural network for the candidate counterfactual image data sample, wherein the loss function includes a first term (407) containing the output of the neural network for the input image data sample, and a second term (409) containing an estimate of the uncertainty of the category prediction for the candidate counterfactual image data sample, wherein the loss function further includes a term representing the difference between the input image data sample and the counterfactual image data sample, wherein the loss function further includes a term (408) indicating the target category to be generated by the neural network for the counterfactual image data sample. Based on the determined loss function, modify the (604) candidate counterfactual image data samples to obtain counterfactual image data samples; and Output (605) counterfactual image data samples.

2. The computer-implemented method according to claim 1, comprising: A sequence of candidate counterfactual image data samples (403) is generated iteratively, wherein in each iteration, the current candidate counterfactual is modified to the subsequent candidate counterfactual based on the determined loss function, or the current candidate counterfactual is accepted as a counterfactual image data sample.

3. The computer-implemented method according to claim 2, comprising: If the determined loss function is higher than a predetermined threshold, the current candidate counterfactual is modified into a subsequent candidate counterfactual; if the determined loss function is lower than the predetermined threshold, the current candidate counterfactual is accepted.

4. The computer-implemented method according to any one of claims 1 to 3, wherein the neural network is a Bayesian neural network (BNN), wherein, The estimation of the uncertainty is derived from the prediction uncertainty caused by the weight probability distribution of the BNN.

5. The computer-implemented method according to any one of claims 1 to 3, comprising: Determine the differences between the image data samples and the counterfactual image data samples; The differences between the determined image data samples and counterfactual data samples are stored in memory; and Control the device based on at least the stored differences.

6. A computer-implemented method (500) for determining the reliability of a category prediction determined for two or more input image data samples using an image data classification neural network, the computer-implemented method (500) comprising: Generate (502) counterfactual image data samples for each input image data sample using the method of any one of claims 1 to 5; For each input sample, determine the difference between the input image data sample and the corresponding counterfactual image data sample; Calculate (504) at least one statistical information of the determined difference between the input image data sample and the counterfactual image data sample; The at least one statistical information is compared against at least one predefined criterion (505); and Based on the comparison results, the reliability of the category predictions made by the neural network is determined.

7. A computer-implemented method for training a neural network for classifying image data, comprising: Provide training image data samples for the training dataset; Train the neural network using the training dataset; Generate one or more counterfactual image data samples according to any one of claims 1 to 5; The generated one or more counterfactual image data samples are added to the training dataset to obtain an enhanced training dataset; and The enhanced training dataset is used to train neural networks and / or additional neural networks.

8. A counterfactual generation apparatus configured to perform the method of any one of claims 1 to 7.

Images