Determine a perturbation mask for a classification model

Abstract

Determine a perturbation mask for a classification model. A system (100) for determining a mask for an input instance for a classification model is disclosed, the mask indicating perturbations that interfere with the classification of the input instance by the classification model. The classification model determines the classification of a certain type of input instance. A generative model generates synthetic instances of the type based on a latent space representation. Given an input instance, its classification according to the classification model, and a latent space representation that causes the generative model to approximate the input instance, the mask is determined. The mask indicates a perturbation to the latent space representation for the input instance and is determined based on a classification score of the classification model for the perturbed input instance. The perturbed instance is determined using the mask by: masking the latent space representation with the mask; and generating a perturbed input instance using the generative model based on the masked latent space representation.

Description

Technical Field

[0001] The present invention relates to a system for determining a mask for classification by a classification model, for example to obtain an understanding of whether the classification model has learned an input / output relationship. The present invention also relates to a corresponding computer-implemented method. The present invention furthermore relates to a computer-readable medium comprising instructions for performing the method. Background Art

[0002] Classification models are increasingly being used in environments such as automated control systems. For example, the control systems of autonomous vehicles can use image classification to detect objects in their vicinity, such as traffic signs and obstacles, and use that detection to control the vehicle. However, there are also non-autonomous vehicles today that have driver assistance systems that use image classification, for example to brake if the vehicle is in danger of colliding with an object. Image classification is also used in various other critical contexts such as medical image classification. In addition to images, there are other types of sensor data that can be used as input to a classification model, for example, audio data, video data, radar data, LiDAR data, ultrasonic data, or various individual sensor readings or their history.

[0003] Based on the classification, control signals can be generated, for example to control robots, vehicles, household appliances, power tools, manufacturing machines, personal assistants, access control systems, and so on. However, classification also occurs in various other contexts such as medical image analysis. Various types of classification models are known, such as neural networks, deep neural networks, support vector machines, and so on. Classification models are sometimes also referred to as discriminative models.

[0004] To ensure the robustness and reliability of a classification model, it is important to establish to what extent the classification model has learned the desired input-output relationship. This is especially the case for more complex models, such as neural networks, which can be regarded as black boxes that provide a classification but alone provide little insight into how that classification was obtained. For example, such a model can be trained on a restricted, and thus inherently biased, data set, resulting in a model that is overparameterized compared to the amount of available data. As a result, such a model may seem to work well in testing, but may provide poor results in deployment. For example, an image classifier for a vehicle trained on data for good weather conditions may not generalize well to poor weather conditions.

[0005] In addition, it is known that various prior art machine learning models, such as neural networks, can be deceived into providing false, high-confidence predictions with potentially dangerous results. For example, it may be possible to determine modifications to an image that are difficult to see with the naked eye, but nonetheless cause an image classifier to change its mind from making a high-confidence classification into one class, such as a colliding object, to making a high-confidence classification into another class, such as no danger. And in such cases, the true generalization performance of the classification model may be called into question.

[0006] To obtain an understanding of whether a classification model has learned the input / output relationship, it is known to use perturbation masks. For example, in "Interpretable explanations of black boxes by meaningful perturbation" by R. Fong and A. Vedaldi (available at https: / / arxiv.org / abs / 1704.03296 and incorporated herein by reference), a framework is presented that, given an image, determines a mask of the same size as the image and highlights the parts of the image that are most responsible for the classifier's decision. SUMMARY

[0007] Various embodiments are directed to determining masks that indicate perturbations that interfere with the classification of input instances by a classification model. It can be advantageous to identify perturbations that are critical in the sense that they interfere with the classification, such as changing the image classification from "traffic sign" to "billboard", or reducing the certainty of classifying as "traffic sign". It can also be beneficial to identify perturbations that are minimal in the sense that they are designed to best preserve the input instance.

[0008] In addition, it can be advantageous to determine intuitive perturbations, such as those that contain semantic meaning, rather than highlighting artifacts of the classification model that lead to a particular classification. For example, given a classifier trained to distinguish between images of cats and dogs, the classifier can be useful if the mask can highlight how changing the appearance or presence of a dog's muzzle, tail, or nose reduces the model's output score for the dog class and / or when replacing each with a cat counterpart increases the score for the cat class. Note that determining masks such that they correspond to meaningful changes preferably excludes adversarially generated masks, e.g., small, localized changes that are designed to be difficult to recognize but still cause the classification model to change its classification.

[0009] In other words, it can be beneficial for a mask to provide meaningful counterfactuals for a given classification model, or in other words, to provide controlled and semantic perturbations that change the model's view on the input instance. Such perturbations can be put to various uses as shown in the following various examples. The perturbations and / or the perturbed input instances are useful outputs in themselves and for themselves, e.g., for presenting to the model developer as a way to debug the classification model, or presenting to the end user to determine whether the model is looking at the right aspects of the input instance. The perturbations as described herein can maintain their interpretability and allow the end user to better understand the decision boundary of the model, e.g., the sensitivity of the classification model to changes in its input, which is important when performing model checking / verification. The perturbations and / or the input instances can also be used, for example, in a control system using image classification. For example, the perturbations and / or the input instances can be used to quantify the model sensitivity in order to obtain verification metrics, for performing additional classifications, etc.

[0010] According to a first aspect of the present invention, a system for determining a mask is proposed. According to a further aspect of the present invention, a method for determining a mask is proposed. According to a further aspect of the present invention, a computer-readable medium is provided.

[0011] Various embodiments relate to determining a mask indicating a perturbation that interferes with the classification of an input instance, e.g., an input image, by a classification model. The perturbed input instance can thus be considered counterfactual for the classification model. Interfering with the classification can include causing the classification model to classify the input instance differently, or at least causing a significant drop in the classification score. For example, a traffic sign can be perturbed into an advertising billboard. Interfering with the classification can also include moving the input instance to the decision boundary of the classification model, or at least close to the decision boundary of the classification model, e.g., a region of the input space where the classification model cannot be trusted to make a correct classification. For example, a traffic sign in an image can become a billboard that still has the characteristics of a traffic sign, e.g., a red border and a white background. The mask indicating the perturbation is referred to throughout this specification as a perturbation mask. By indicating the perturbation that interferes with the classification, the mask can provide an explanation of why the classification model classifies the input instance in a certain way.

[0012] Interestingly, the inventors realized that instead of determining a mask that directly masks the input instance, instead, a mask can be determined that indicates a perturbation to the latent space representation of the input instance for a generative model.

[0013] The generative model can be configured to generate synthetic input instances for the classification model. In other words, the generative model can be configured to generate synthetic instances of the input type of the classification model based on a latent space representation. Various generative models, and ways of training them such that they can generate representative synthetic instances, are known in the art. For example, the generative model can include the decoder part of a variational autoencoder (VAE). Such a VAE typically includes an encoder part and a decoder part. The encoder part can generate a latent space representation based on an instance of a given type. The decoder part can generate a synthetic instance of said type based on the latent space representation, e.g., a feature vector of the same length. The VAE can be trained to maximize the likelihood of the decoder part, so as to reconstruct the input instance based on its latent space representation. However, various other types of generative models can also be used, such as generative adversarial networks (GANs), flow models, or autoregressive models.

[0014] Generally, the generative model can be trained such that the generative model generates input instances representing the input instances of the classification model based on latent representations from the latent space. For example, the input instances can generally be approximated by the latent space representation. The generative model is separate from the classification model, although both are typically trained on the same or a similar dataset. For example, the generative model can be trained to optimally generate representative input instances of the training dataset. The generative model can then be used together with the classification model to diagnose its behavior.

[0015] The latent space representations come from the latent space and typically have a smaller dimension than the input instances they generate, e.g., at most 1% or at most 0.1% of the number of features of the input instances. Thus, the latent space representations approximating the input instances can effectively be regarded as compressed representations of the input instances. By training the generative model to optimally reproduce the training dataset, semantic meanings can be ascribed to the features of the latent feature representations. For example, in a generative model for generating images of traffic situations, specific features can indicate the presence of a traffic light and the color it shows, weather conditions, etc.

[0016] Thus, the mask can indicate a perturbation to the latent space representation of the input instance. The latent space representation can be a latent space representation such that when used as an input to a generative model, it causes the generative model to generate a synthetic instance that, for example, approximates the input instance in terms of norm in visual appearance and / or distance. The mask can interfere with the classification of the input instance in the sense that if the latent space representation is modified as indicated by the mask and the result is input to the generative model, this results in a synthetic instance for which the classification is interfered with. The synthetic instance can be regarded as a perturbed input instance. For example, the synthetic instance can be classified differently from the original input instance, it can be classified the same but with less certainty, it can be close to the decision boundary of the classification model, and so on.

[0017] The mask can indicate the perturbation in various ways. For example, the mask can indicate perturbations to individual features of the latent space representation that are to be applied, such as separate perturbations to be applied to each feature. The mask can indicate, for example, addition or subtraction for numerical features; bit flips, setting, or resetting for binary features, and so on. For example, the mask can be a heatmap that indicates the bits of the latent space representation to be set or reset. The mask can also indicate various other types of modifications, such as applying a non-linear function such as a threshold or sigmoid to one or more features simultaneously, and so on. It is not necessary for the mask to be explicitly represented in memory. For example, determining the mask can include determining the masked latent space representation based on the latent space representation of the input instance, where the mask is implicitly defined as the difference between the original and the masked latent space representations. The mask can also include successive perturbations. For example, the mask can be determined as a sequence of constituent masks for gradually updating the latent space representation in an iterative process.

[0018] As the inventors have realized, since the latent space representation is essentially a compressed representation of the input instance, a mask of the latent space representation can also allow for a relatively concise representation of the perturbation to the input instance. Typically, perturbing a particular feature component of the latent space representation results in changes in multiple feature components of the corresponding input instance. Moreover, such changes are typically related to each other and can represent higher-level features that affect classification by the classification model. In many cases, a particular feature component of the latent space representation can have a human-interpretable meaning, and thus, in this sense, the mask for perturbing them can provide an explanation of which features of the input instance are used by the classification model in its classification.

[0019] Accordingly, the various measures involved can include learning how the latent space representation can be modified, e.g., how it can grow or shrink and in which directions, so that the classification model under diagnosis behaves in a desired manner. Thus, a structured approach can be provided for finding minimal, intuitive, and / or critical perturbations of the input instance that turn the input instance into a meaningful counterfactual for the classification model. By determining a mask for interfering with classification, the mask can be obtained in a controlled manner, which provides semantic perturbations that change the model's view of the input instance. Such masks can be valuable, e.g., because they retain their interpretability and allow the end user to better understand the decision boundary of the model. For example, understanding the perturbations can help understand how sensitive the model is to changes in its input, which can be desirable in model checking / verification. These sensitivities can be quantified and developed into standardized verification metrics.

[0020] Since the mask can be determined from the latent space representation rather than directly from the input instance, the susceptibility to adversarial type modifications of the input instance can also be reduced. As discussed, it is known for various classification models that an adversary can determine small, e.g., imperceptible, modifications to the input instance that result in a different classification. Such small modifications are not typically useful as masks for explaining how the classification model arrives at the classification because they typically represent artifacts of the classification model rather than meaningful changes. Determining the mask of the input instance directly can result in a mask that represents such adversarial changes. Various ways to mitigate this effect based on specific characteristics, such as high-frequency or low-intensity mask activations, may not adequately exclude adversarial changes and / or exclude meaningful masks.

[0021] However, by instead determining a mask that modifies the latent space representation, a mask can be determined that indicates more meaningful perturbations, such as perturbations that carry semantic meaning and / or result in larger changes in the input instance that affect the classification through the classification model. In fact, prior knowledge of the input semantics can be incorporated. Since the masks carry more meaning, they are less likely to represent adversarial type artifacts, e.g., more meaningful ones.

[0022] The determined mask can also be beneficially used as a metric that indicates the amount of change required to interfere with the transformation of the input instance into a classification. In other words, the determined mask can indicate the amount of "energy" required to interfere with the classification and thus indicate the confidence in the classification of the classification model. In this sense, the mask can provide an explanation of the classification that is not only interpretable but also quantifiable, e.g., based on the behavior of the model.

[0023] In particular, a mask can be determined based on the classification scores of a classification model for perturbed input instances. Various classification models provide classification scores that indicate the probability that an input instance belongs to a given class. Classification scores are sometimes also referred to as prediction scores. A classification score can be a probability score, but can also more generally indicate a probability, e.g., the higher the classification score, the higher the probability. For example, in the case of a neural network, such a classification score can include the output of the softmax output layer of the neural network. In the case of a multi-class classifier, separate classification scores can be provided for the respective classes.

[0024] By using a mask to mask a latent space representation and using a generative model to generate a perturbed input instance from the masked latent space representation, a mask can be used to determine a perturbed instance. In other words, starting from a fixed starting latent space representation, a type of in-latent-space interpolation can be performed, where a masked latent space representation is determined, which results in a classification score that indicates that the classification of the input instance is perturbed, e.g., changed or made more uncertain, and can thus reveal some interesting properties of the original classification model. For example, determining a mask based on the classification score can yield instances for which the regions of the input space that the model is performing for are not well-behaved, e.g., have noisy classification boundaries, and / or instances of adversarial classes generated using a minimal amount of change to the input instance.

[0025] In various embodiments, determining a mask can include optimizing the mask based on the classification score, e.g., selecting a mask that minimizes a loss that includes the classification score. Such optimization can be performed using known optimization methods, e.g., iterative methods such as stochastic gradient descent. It will be appreciated that the optimization can be heuristic and / or can reach a local optimum. The optimization need not be iterative, e.g., it can be performed as follows: by brute-force search of the latent space, by selecting a mask with the minimum classification score from random samples of the latent space, by using a closed-form expression to determine the mask, etc. As also discussed, optimizing the mask can include optimizing the masked latent space representation, which is implicitly defined by the masked latent space representation. For example, the previously mentioned optimization techniques can be equivalently applied to obtain the masked latent space representation.

[0026] Instead of or in addition to the mask or the masked latent space representation, it can also be beneficial to provide the determined perturbed input instance for the mask as an output. For example, the perturbed input instance, especially when presented alongside the original input instance, can show features that are relevant to the classification of the original input instance. Additionally, the classification model may be less certain in classifying the input instance, indicating a type of instance that the classification model has difficulty with, such that adding training instances of this kind to the training dataset for the classification model can improve its performance.

[0027] Optionally, the processor subsystem is configured to determine a mask by at least minimizing the difference between a classification score and a classification threshold. For example, a mask can be determined by minimizing a loss function that includes the difference or at least has an increase in the difference. The loss function can include other terms. For example, here and elsewhere, "at least minimize" should be interpreted to mean: optimize to prefer a mask with a lower difference, all other things being equal.

[0028] For example, the classification threshold can be the classification threshold for the class in which the original input instance is classified. A classification score above the threshold can result in classification into that class. Minimizing the difference can result in a mask that is used to move the input instance closer to the decision boundary of the classification model and / or move the perturbed input instance closer to the decision boundary, highlighting aspects of the latent space representation that the classification model uses to arrive at its classification of the input instance.

[0029] Optionally, the processor subsystem is configured to determine a mask by at least maximizing the gradient of the classification score with respect to the mask. For example, similar to the difference between classification and threshold, the loss function can include the gradient in such a way that higher gradients are preferred, all other things being equal. For example, the norm of the gradient can be maximized. Using the gradient can focus the optimization on irregular parts of the latent feature space, such as parts close to the decision boundary and / or parts where the classification model exhibits non-uniform or unpredictable behavior. Such parts can indicate the types of training instances that the model finds particularly difficult to classify, and / or can indicate modifications to the input instance that make the classification model particularly uncertain. Using the gradient can be particularly useful for multi-class classification, in which case changes in the perturbation score can be more informative than a particular classification score for a particular class.

[0030] Optionally, the processor subsystem is configured to determine a mask by at least minimizing a discriminator score that quantifies the dissimilarity between a perturbed input instance and a training data set used to train a generative model. For example, as above, the loss function can include the discriminator score such that lower dissimilarities are preferred, all other things being equal. Including the discriminator score can encourage optimization to rule out solutions that do not look natural, thus providing more meaningful results. For example, if the generative part of a generative adversarial network (GAN) is used, the discriminator score can be given by the discriminator part of the GAN that is used to train the generative model. In the case of a variational autoencoder or various other types of generative models, a suitable optimization objective function can be used to determine the discriminator score, e.g., the reconstruction loss of training the generative model to produce instances that represent the training data set.

[0031] Optionally, the processor subsystem is configured to determine a mask by at least minimizing a norm of the mask. For example, as described above, the loss function may include the norm such that, ceteris paribus, smaller norms are preferred. Minimizing the norm may encourage relatively small masks, e.g., resulting in relatively small perturbations of the input instances. It may be desirable to determine a mask that uses the least amount of energy to have the greatest possible impact on the classification, and thus, based on the norm, unnecessary activations of the mask may be penalized. The norm is preferably the l1 norm due to its sparsity-inducing properties, but this is not required.

[0032] Optionally, the generative model includes the generative part of a generative adversarial network (GAN). A GAN typically includes a generative part and a discriminative part. The generative part may be trained to generate instances based on a latent feature representation, and the discriminative part may be trained to distinguish instances generated by the generative part from the training instances. For example, the two parts may be trained repeatedly and alternately. This may result in the generative part being able to generate a wide range of input instances similar to the training instances, including instances that approximate the training instances themselves and their variants. Thus, the generative part may be suitable as a generative model for the techniques described herein.

[0033] Optionally, the generative model includes the decoder part of an autoencoder, e.g., a variational autoencoder (VAE). An autoencoder typically includes an encoder and a decoder. The encoder may be trained to generate a latent space representation based on an instance, and the decoder may be trained to generate an instance based on the latent space representation. Both may be trained, e.g., in a joint optimization, for their ability to reproduce the original instance. Autoencoders such as VAEs are particularly suitable because the encoder part can be used to determine synthetic instances that approximate the input instances, as described.

[0034] Optionally, the classification model can be an image classifier. The processor subsystem can be configured to obtain an input image as an input instance and determine a perturbed input image as a perturbed input instance. The interpretability of classification is particularly significant for image classification because the image classification task is typically relatively complex, and as a result, relatively complex classification models, such as convolutional neural networks and the like, can be used. Additionally, for images, being able to visualize the perturbed instance images provides a particularly attractive and intuitive way for users to interpret the determined masks. Note that various kinds of sensor inputs can actually be represented as images and processed as images, as disclosed, for example, for sound data in Chris Donahue et al.'s "Adversarial Audio Synthesis" (available at https: / / arxiv.org / abs / 1802.04208 and incorporated herein by reference).

[0035] Optionally, the determined mask and / or the perturbed input instance generated by the generative model based on the determined mask are output to the user in a perceptually perceivable manner, such as being shown on a display. For example, the perturbed input instance can be presented beside the original input instance for comparison. The user can be an end user of the classification model, in which case the mask and / or the perturbed input instance can be used as a justification for the decision of the classification model. For example, in medical image classification, the perturbed input instance can highlight the part of the image of the organ used for making the decision, and / or the mask can highlight semantic features such as volume, color, shape, etc., based on which the decision is made. Similarly, the output can inform the user, such as the model designer or user, of what data close to the decision boundary can be collected to improve the model.

[0036] Optionally, the processor subsystem is configured to determine a plurality of masks for the latent space representation of the input instance and determine a stability value based on the plurality of masks, the stability value indicating the stability of the classification of the input instance by the classification model. Providing the stability value together with the classification is particularly relevant in control systems. For example, when obtaining a classification with a low stability value, the system can request user confirmation, store the classification and the results associated with it, activate the safety subsystem, etc. The stability value can indicate the amount of energy generally required to modify the classification of the input instance, where the energy measures the amount of information contained in the mask, which is, for example, the amount of information calculated by the norm on the mask. More energy can indicate higher stability because more changes may be required to perturb the classification. The norm can be the same norm that is also minimized when determining the mask.

[0037] Determining multiple masks—such as corresponding to moving to the decision boundary of a classification model in multiple directions—can result in a more accurate assessment of stability. For example, the decision boundary can be close in one direction but far in another direction. For example, a processor subsystem can be configured to determine a mask among multiple masks by: randomly generating an initial value for the mask and iteratively updating the mask based on the initial value, thereby enabling optimization to move to the decision boundary in various directions.

[0038] Optionally, the processor subsystem is further configured to obtain the classification of perturbed input instances. The processor subsystem can retrain the classification model using at least the perturbed input instances and the classification. As discussed, the mask can interfere with the classification of the input instance. For example, the perturbed instance can be on or close to the decision boundary of the classification model. A perturbed instance located near the decision boundary can indicate that insufficient training instances similar to the perturbed instance were used to train the classification model. Thus, using the perturbed instance and its classification to retrain the model can improve its accuracy. Optionally, multiple perturbed instances and corresponding classifications are determined based on the mask for further improving the classification near the decision boundary. For example, random noise can be added to the masked latent space representation, and a noise-perturbed instance can be generated according to the corresponding noise-masked latent space representation.

[0039] Optionally, an input interface is used to further access an additional classification model. The additional classification model can be configured to determine the classification of the mask for the latent space representation, such as being classified into two or more predefined classes. The processor subsystem can further be configured to determine the classification through the additional classification model of the determined mask. The inventors have realized that since the latent space representation essentially provides a compressed representation of the input instance, the mask can be regarded as providing information related to how the classification model determines its classification in a compressed form. In an embodiment, the classification model is used to detect anomalies in a control system, for example, and the additional classification model is used to classify the anomalies into multiple types. By providing information in a more compressed and / or semantically structured manner, a particularly accurate additional classifier can be obtained. Generally, the additional classifier can classify the mask into multiple root causes for the classification by the original classification model.

[0040] Those skilled in the art will appreciate that the above-mentioned embodiments, implementations, and / or optional aspects of the present invention can be combined in any way considered useful, two or more at a time.

[0041] Those skilled in the art can implement modifications and variations to any computer-implemented method and / or any computer-readable medium based on this description, corresponding to the modifications and variations of the corresponding system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] These and other aspects of the invention will be made apparent from the embodiments and the drawings and will be further elucidated by reference to the embodiments and the drawings, which are described as examples in the following description, in which:

[0043] Figure 1 A system for determining a perturbation mask is shown;

[0044] Figure 2 A detailed example of how a perturbation mask can be determined is shown;

[0045] Figure 3 A method for determining a perturbation mask is shown;

[0046] Figure 4 A computer-readable medium including data is shown.

[0047] It should be noted that the figures are purely illustrative and not drawn to scale. In the figures, elements corresponding to elements already described may have the same reference numerals. DETAILED DESCRIPTION

[0048] Figure 1 A system 100 for determining a perturbation mask is shown. The perturbation mask can indicate a perturbation for an input instance of a classification model, the perturbation interfering with the classification of the input instance by the classification model. The system 100 can include an input interface 120 and a processor subsystem 140 that can communicate internally via a data communication 124.

[0049] The processor subsystem 140 can be configured to access a classification model 040 and a generative model 060 during operation of the system 100 and using the input interface 120. The classification model 040 can be configured to determine the classification of a certain type of input instance. The generative model 060 can be configured to generate a synthetic instance of that type based on a latent space representation. For example, as Figure 1 shown, the input interface 120 can provide access 122 to an external data storage device 020, which can include the models 040, 060. Alternatively, the models 040, 060 can be accessed from an internal data storage device that is part of the system 100. Alternatively, the models 040, 060 can be received from another entity via a network. Generally, the input interface 120 can take various forms, such as a network interface for a local area network or a wide area network, such as the Internet, a storage interface for an internal or external data storage device, and so on. The data storage device 020 can take any known and suitable form.

[0050] The processor subsystem 140 may additionally be configured to obtain, during operation of the system 100, an input instance for the classification model 040, a classification of the input instance according to the classification model 040, and a latent space representation of the input instance. For example, the processor subsystem 140 may obtain the classification by classifying the input instance using the classification model 040. For example, the processor subsystem 140 may determine the latent space representation based on the input instance, e.g., by using an encoder portion of a generative adversarial network. When used as an input to the generative model 060, the latent space representation may cause the generative model to generate a synthetic instance that approximates the input instance, e.g., is close to the input instance in norm and / or in visual appearance. The processor subsystem 140 may additionally be configured to determine, during operation of the system 100, a mask that indicates perturbations that interfere with the classification of the input instance by the classification model 040. The mask may indicate perturbations to the latent space representation of the input instance. The mask may be determined based on the classification score of the classification model 040 for the perturbed input instance. By masking the latent space representation with the mask and using the generative model 060 to generate a perturbed input instance from the masked latent space representation, the mask may be used to determine the perturbed instance.

[0051] As an optional component, the system 100 may include an image input interface 150, or any other type of input interface, for obtaining sensor data from a sensor such as a camera 170. The sensor data may be included in the input instance. For example, the camera may be configured to capture image data 172 and the processor subsystem 140 may be configured to obtain an input instance 152 from the image data 172 obtained via the input interface 150.

[0052] As an optional component, the system 100 may include a display output interface 160 or any other type of output interface for outputting the determined mask and / or the perturbed input instance to a presentation device such as a display 180. For example, the display output interface 160 may generate display data 162 for the display 180 that causes the display 180 to present the determined mask and / or the perturbed instance 142 in a perceptually perceivable manner, e.g., as a screen visualization 182.

[0053] Reference will be made Figure 2 to further clarify various details and aspects of the operation of the system 100, including its optional aspects.

[0054] Typically, system 100 may be embodied as or in a single device or apparatus such as a workstation, e.g., a laptop or desktop-based workstation, or a server. The device or apparatus may include one or more microprocessors executing appropriate software. For example, the processor subsystem may be embodied by a single central processing unit (CPU), and also by a system or combination of such a CPU and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory such as volatile memory like RAM, or non-volatile memory such as flash memory. Alternatively, functional units of the system, such as data interfaces and the processor subsystem, may be implemented in a device or apparatus in the form of programmable logic, such as a field programmable gate array (FPGA). Generally, each functional unit of the system may be implemented in circuit form. Note that system 100 may also be implemented in a distributed manner, e.g., involving different devices or apparatuses such as distributed servers in the form of cloud computing, for example.

[0055] Figure 2 An example is shown of how a mask can be determined for an input instance, for example, by Figure 1 system 100, which is detailed but non-limiting.

[0056] Shown in the figure is classification model CM, 240. The classification model CM may be configured to determine the classification of an input instance. The classification model CM may classify data into one or more predefined classes, e.g., two (binary classifier) or more than two (multi-class classifier) classes. For simplicity, the following description will focus on image data as the input to an image classification model in a binary classification scenario.

[0057] Various known types of classification models may be used, but for illustrative purposes, a neural network is used in this example. Generally, the classification model may allow determination of a classification score, e.g., a probability score, which indicates the probability that the input instance belongs to a given class. For example, neural network CM may provide a softmax score. Shown in the figure is classification operation CI, 242, which uses the classification model to determine the classification score of a perturbed input instance, as discussed in more detail later. The classification model CM may be trained based on a labeled training data set in a training operation (not shown).

[0058] Also shown in the figure is an input instance II, 210, for the classification model CM. The input instance representing the sensor data can be represented as a feature vector, but this is not necessary. For example, graph-structured data can also be used. The input instance II can be obtained in various ways, such as from or based on data from a sensor interface, from a user, from different subsystems, and so on. The figure illustrates determining a mask M, 222, which indicates a perturbation that interferes with the classification of the input instance II by the classification model CM. In various embodiments, the classification of the input instance II according to the classification model CM (not shown) is obtained, for example, by using the classification model CM to classify the input instance II, by receiving the classification as an input, and so on.

[0059] Also shown in the figure is a generative model GM, 260. The generative model GM can be configured to generate synthetic instances of the same type as those input to the classification model CM according to a latent space representation. In other words, the generative model GM outputs instances suitable as inputs for the classification model CM. The latent space representation is typically smaller than the instances generated therefrom. As a specific example, the input instance II can be, for example, an image with a size of 256x256, for which the latent space representation can include 100 features.

[0060] A generative model can generally be decomposed into an encoder part and a decoder part, where the decoder part can generate synthetic instances according to the latent space representation, and the encoder part can generate the latent space representation according to the synthetic instances. The generative model can be trained in a training operation (not shown), for example, on the same dataset on which the classification model CM is trained, or on a similar dataset. When using a dataset to train the generative model, the dataset is typically unlabeled. Typically, the generative model is trained to minimize a loss function that quantifies the loss of reproducing the instances of the training dataset by the generative model.

[0061] The generative model GM can be a variational autoencoder (VAE). In the case of a VAE, the latent space can be explicitly modeled by choosing its latent distribution, for example, as a normal distribution with a learned mean and standard deviation. The VAE can be parameterized by the weights and biases, mean, and / or standard deviation of the encoder and decoder parts.

[0062] However, other types of generative models, such as generative adversarial networks (GANs) and the like are also possible. In the case of a GAN, the generator part can be trained together with a discriminator network. The generator part can be trained to generate synthetic instances, and the discriminator network can be trained to determine a discriminator score that quantifies the dissimilarity between the generated instances and the instances from the training dataset. Training can continue until a Nash equilibrium is reached such that the generator part can produce samples that a highly discriminative discriminator can no longer distinguish from the actual samples from the training dataset. In this way, the generator can learn to model the input data distribution.

[0063] Also shown is the operation Enc, 262, where a latent space representation LSR, 220 of an input instance II is obtained by determining it according to the input instance II. The latent space representation LSR, when used as an input to a generative model GM, causes the generative model GM to generate a synthetic instance that approximates the input instance II, e.g., is close to the input instance II in norm and / or visual appearance. In this example, Enc can obtain the latent space representation LSR by applying the encoder part of a variational autoencoder GM. For other types of generative models, it is also known in the art to determine a latent space representation that gives an approximation of the input instance. For example, determining the latent space representation can be formulated as an optimization problem that can be solved using known techniques. This is disclosed, e.g., for image GANs in Zachary C. Lipton and Subarna Tripathi's "Precise Recovery of Latent Vectors from Generative Adversarial Networks" (available at https: / / arxiv.org / abs / 1702.04782 and incorporated herein by reference).

[0064] Given the trained generative model GM, shown in this figure is an optimization process that, for example, learns a mask M for perturbing an input instance II to interfere with the classification of the input instance by a classification model CM in a search for the decision boundary of the classification model. The decision boundary can be "critical" in the sense that non-smooth and / or overfitted decision boundaries can be found where the predictions of the model are non-uniform and / or unpredictable and thus less trustworthy.

[0065] The mask M can indicate perturbations to the latent space representation LSR. The mask M can have the same shape and size as the latent space representation LSR and values in the range [-max, max], where max is a user-specified parameter and represents the size of the search space.

[0066] The mask M can be determined based on the classification score CS for the perturbed input instance PII, 212, which is perturbed as indicated by the mask M.

[0067] To determine the perturbed input instance PII, in operation Ma, 224, the latent space representation LSR can be masked using the mask M. For example, operation Ma can apply the perturbation indicated by the mask M to the latent space representation LSR. Operation Ma can result in a masked latent space representation MLS, 226. Labeling the latent space representation LSR by the vector z and the mask M by the vector m, the result can be labeled as φ(z, m). A broad group of possible functions can be used. For example, in this example, operation Ma can perform element-wise addition z + m.

[0068] In operation Dec, 264, a generative model G, GM can be used to generate the perturbed input instance PII according to the masked latent space representation MLS, which for example results in G(φ(z, m)). For example, the decoder part of a variational autoencoder, or the generative part of a GAN, etc. can be applied.

[0069] In operation CI, 242, a classification model CM can be applied to the perturbed input instance PII to obtain a classification score CS, 244, which indicates the probability that the perturbed input instance PII belongs to the class c of the input instance II. For example, the class c can be obtained by applying the classification model CM. For example, the classification score CS can be the softmax output f of the classification model CM used for diagnosing the class c c which results in f c (G(φ(z, m))).

[0070] In this example, in the optimization operation Opt, 250, the mask M is determined based on the classification score CS, and the optimization operation Opt, 250 minimizes a loss function based on the classification score In the example shown here, the optimization is an iterative optimization. For example, the mask M can be initialized randomly, and then gradually updated based on the classification score CS using, for example, stochastic gradient descent or the like. However, this is not necessary. For example, a brute-force search or a closed-form solution for the optimization can also be used.

[0071] As a specific example, the following loss function that combines various terms can be used, where each of the various terms contributes to a mask m, which provides a perturbation that interferes with the classification of the input instance II:

[0072]

[0073] It will be understood that variations are possible, e.g., a subset of the above terms can be used, additional terms can be included, various terms can be controlled by hyperparameters, etc.

[0074] As illustrated by the first term of the above loss function the mask can be optimized to minimize the difference between the classification score f c (G(φ(z, m))) and the classification threshold f cr For example, for a neural network CM, the classification threshold f cr can represent the target softmax output of the classification model, which is used to diagnose the class c of the input instance II. For example, in binary classification, f cr can be set to the classification threshold previously determined based on the ROC curve, or can be set in another way to represent a softmax boundary that divides samples of one class from samples of another class. Including the difference term in the optimization can help ensure that the mask M is learned, which affects the latent space representation LSR of the input instance for the perturbed instance PII that reaches close to the decision boundary of the model.

[0075] As illustrated by the second term of the above loss function the mask can be further optimized to minimize the discriminator score, e.g., D(G(φ(z,m))), which quantifies the dissimilarity between the perturbed input instance φ(z,m) and the training dataset used to train the generative model GM. For example, if the generative model GM is a GAN, the discriminator part of the GAN can be used to obtain the discriminator score, which is trained, for example, to return 1 if the input is a generated example and 0 if the input is real. For other types of generative models, this term can be omitted or replaced by another appropriate discriminator term. Including the discriminator score can help ensure that the perturbed input instance PII maintains the semantic quality of the input instance II. As a result, solutions that do not look natural can be encouraged to be excluded.

[0076] As illustrated by the third term of the above loss function the mask can be further optimized to maximize the gradient of the classification score G(φ(z,m)) - e.g., the softmax output with respect to the mask -, e.g., Including gradients can effectively place constraints on how the decision boundary of the classification model CM reacts to the class of the input instance II. By including gradients, masks can be favored to obtain perturbed input instances II for which the model behaves non-uniformly and / or unpredictably. To control this effect, for example, the hyperparameter λ1 can be applied to the classification scores. In terms of the history of the learning optimization process, including gradients can favor solutions with noisy and / or high-frequency softmax curves.

[0077] As illustrated by the fourth term of the loss function above the mask can be further optimized to minimize the norm of the mask, e.g., ||m||1. Minimizing the norm can constrain the amount of modification made to the latent space representation LSR. It may be desirable to find a mask with relatively small modifications, e.g., a mask with the minimum amount of energy for the maximum possible impact. In an example, the l1 norm is chosen for minimizing the number of modifications, e.g., favoring sparse masks, but other norms are possible. As shown, the hyperparameter λ2 can be used to control the effect of including the norm.

[0078] Through the various measures described above, the perturbed input instance PII and / or the mask M indicating its perturbation can be determined. Both can be beneficially used in various ways. In some embodiments, the determined mask M and / or the perturbed input instance PII can be output to the user in a perceptually perceivable manner, thereby providing the user with insights into the way the classification model CM has classified the input instance II. For example, the mask M and / or the perturbed instance PII can be used by the model developer for model validation and / or debugging, e.g., to obtain insights into the stability of the model predictions and to obtain an understanding of the general behavior of the model, or by the user to understand how a particular classification is made.

[0079] In some embodiments, multiple masks M are determined for the latent space representation LSR of the input instance. Based on the multiple masks M, a stability value (not shown) can be determined, which indicates the stability of the classification of the input instance II by the classification model CM. The stability value can be based on the energy value of the mask, e.g., the norm, as described below. In practice, this can to some extent allow the automation of the model validation discussed above.

[0080] More specifically, the mask M among the multiple masks can be determined as follows: randomly generate an initial value for the mask and iteratively update the mask based on the initial value using, for example, the operations Ma, Dec, Cl, and / or Opt discussed above. In effect, this provides a Monte Carlo-like assessment of maintaining the mask energy after an optimization process over several iterations of the process detailed above. The initial mask can be randomly generated using a known expected energy, for example, by sampling from a normal distribution with a fixed standard deviation . In this way, the optimization Opt can be encouraged to explore different regions of the solution space. The parameters of the noise model, such as σ, can also be at least partially based on previous iterations of the mask generation process. For example, if slow convergence for the mask is determined, a larger initial value can be used.

[0081] When the optimization for determining the mask M is successfully completed, the energy of the mask M can be stored. The energy can be determined based on the l1 norm, l2 norm, entropy, etc. The associated perturbed input instance PII can also be stored.

[0082] Based on the determined mask and the corresponding energy, one or more stability values can be determined. Generally, a higher energy value indicates a higher stability of the classification of the input instance II. For example, a high mask energy can correspond to significant changes made to the input instance. Thus, a model for which only such high-energy changes cause it to decide to change can be considered more stable than a model for which a very small energy can cause a difference in the classification by the model. Therefore, the stability value can be based on, for example, the average energy, the confidence interval of the energy, etc. For example, an alert can be issued for the stability value, which indicates low stability of the classification, for example, below a certain threshold.

[0083] In addition to the determined stability value, the generated perturbed input instance PII can also be beneficially output, for example, for visual verification as a representative variant of the data in the dataset, and / or for augmenting the dataset or otherwise improving the classification model CM, as also discussed below.

[0084] In various embodiments, a classification of a perturbed input instance PII can be obtained, and the classification model CM can be retrained in a retraining operation (not shown) using at least the perturbed input instance PII and the obtained classification. For example, the perturbed input instances can be added to the original training dataset, after which the training process for training the classification model CM can be repeated, or the classification model CM can be updated based on one or more perturbed input instances PII and their classifications. Since the perturbed input images PII can be determined in such a way that the classification model CM finds it difficult to classify them, using them as training data can lead to a particularly efficient and / or cost-effective refinement of the model. The generative model GM can also be retrained based on the perturbed input instances.

[0085] In various embodiments, an additional classification model can be used in an additional classification operation (not shown) to determine the classification of the determined mask M, e.g., to find the root cause for the classification by the classification model CM. Since the determined mask M can essentially provide an explanation of the classification by the model CM in a compressed form, such an additional classification can be particularly effective.

[0086] For example, in various embodiments, the classification of the determined mask M can be used as a control signal for a control system. For example, the classification model CM can be trained to detect defects in a manufactured product as part of a quality inspection component in a manufacturing process. For example, the classification model CM can classify an input instance, e.g., an image, into "OK (good)" and "not OK (bad)" classes. The additional classification model can then use the mask M to determine the root cause of the "not OK" classification, e.g., various anomalies, faults, etc. For example, the model can distinguish between physical defects in a sample, changes in control conditions such as illumination during the manufacturing process, etc. Based on the classification, e.g., classified as an explanation / anomaly, a control signal can be determined. For example, based on the classification of the mask M, the operating parameters of the manufacturing process can be adjusted and / or the system can be brought into a safe mode. Thus, insights into the classification of the input instance II as embodied in the mask M can be used to improve the quality of the control signal provided by the control system.

[0087] Figure 3 A block diagram of a computer-implemented method 300 for determining a mask for an input instance to a classification model is shown, the mask indicating a perturbation that interferes with the classification of the input instance by the classification model. Method 300 can correspond to Figure 1 the operation of system 100. However, this is not a limitation as method 300 can also be performed using another system, apparatus, or device.

[0088] The method may include accessing 310 a classification model and a generative model in an operation entitled "ACCESSING CLASSIFICATION MODEL AND GENERATIVE MODEL". The classification model may be configured to determine a classification of an input instance. The input instance may have a type. The generative model may be configured to generate a synthetic instance of the type based on a latent space representation. The method may further include obtaining 320 an input instance for the classification model, a classification of the input instance according to the classification model, and a latent space representation of the input instance in an operation entitled "OBTAINING INPUT INSTANCE, CLASSIFICATION, LATENT SPACE REPRESENTATION". The latent space representation, when used as an input to the generative model, may cause the generative model to generate a synthetic instance approximating the input instance. The method may also include determining 330 a mask in an operation entitled "DETERMINING MASK", the mask indicating a perturbation that interferes with the classification of the input instance by the classification model. The mask may indicate a perturbation to the latent space representation of the input instance. The mask may be determined based on a classification score of the classification model for a perturbed input instance. As part of operation 330, the mask may be used to determine a perturbed instance by masking 332 the latent space representation with the mask in an operation entitled "MASKING LATENT SPACE REPRESENTATION". As part of operation 330, to determine the perturbed instance, a perturbed input instance may be generated 334 using the generative model based on the masked latent space representation in an operation entitled "GENERATING PERTURBED INPUT INSTANCE".

[0089] It will be appreciated that, generally, Figure 3 the operations of method 300 may be performed in any suitable order, such as sequentially, simultaneously, or a combination thereof, which in applicable cases is subject to a particular order required, for example, by input / output relationships.

[0090] (One or more) methods may be implemented on a computer as a computer-implemented method, as dedicated hardware, or as a combination of both. As also in Figure 4As illustrated, instructions for a computer, such as executable code, can be stored on a computer-readable medium 400, for example, in the form of a series of 410 machine-readable physical markings and / or as a series of elements having different electrical (e.g., magnetic) or optical properties or values. The executable code can be stored in a transient or non-transient manner. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, and the like. Figure 4 An optical disc 400 is shown.

[0091] Examples, embodiments, or alternative features—whether or not indicated as non-limiting—should not be construed as limiting the invention as claimed.

[0092] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The use of the verb “comprise” and its conjugations does not exclude the presence of elements or steps other than those stated in the claim. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. Expressions such as “at least one of” when preceding a list of elements or groups mean selecting all elements or any subset of elements from the list or group. For example, the expression “at least one of A, B, and C” should be understood to include only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The invention can be implemented by means of hardware including several different elements, and by means of a suitably programmed computer. In a device claim enumerating several components, several of these components can be embodied by the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A system (100) for determining a mask for an input instance to a classification model, the mask indicating perturbations that interfere with the classification of the input instance by the classification model, the interference causing the classification to change or become less certain, the system comprising: - An input interface (120) for accessing a classification model (040) and a generative model (060), wherein the classification model is configured to determine the classification of an input instance, the input instance having a type, and wherein the generative model is configured to generate synthetic instances of the type according to a latent space representation. - A processor subsystem (140) configured to: - Obtain an input instance for the classification model, the classification of the input instance according to the classification model, and a latent space representation for the input instance, wherein the latent space representation, when used as an input to the generative model, causes the generative model to generate synthetic instances approximating the input instance; - Determine a mask that indicates perturbations that interfere with the classification of the input instance by the classification model, wherein the mask indicates perturbations to the latent space representation for the input instance, and wherein the mask is determined based on a classification score of the classification model for the perturbed input instance, the classification score indicating the probability that the input instance belongs to a given class, the perturbed input instance being determined using the mask by: - Masking the latent space representation with the mask; And - Using the generative model to generate a perturbed input instance according to the masked latent space representation, wherein the classification model is an image classifier, and wherein the processor subsystem (140) is configured to obtain an input image as the input instance and determine a perturbed input image as the perturbed input instance.

2. The system (100) according to claim 1, wherein the processor subsystem (140) is configured to determine the mask by minimizing the difference between the classification score and a classification threshold.

3. The system (100) according to claim 1 or 2, wherein the processor subsystem (140) is configured to determine the mask by maximizing the gradient of the classification score of the mask.

4. The system (100) according to claim 1 or 2, wherein the processor subsystem (140) is configured to determine the mask by minimizing a discriminator score that quantifies the dissimilarity between the perturbed input instance and a training data set used to train the generative model.

5. The system (100) according to claim 1 or 2, wherein the processor subsystem (140) is configured to determine the mask by minimizing the norm of the mask.

6. The system (100) according to claim 1 or 2, wherein the generative model comprises a generative part of a generative adversarial network or a decoder part of an autoencoder.

7. The system (100) according to claim 1 or 2, further comprising an output interface (160) for outputting the determined mask and / or the perturbed input instance to a user in a perceptually perceivable manner.

8. The system (100) according to claim 1 or 2, wherein the processor subsystem (140) is configured to determine a plurality of masks for a latent space representation of an input instance and determine a stability value based on the plurality of masks, the stability value indicating the stability of the classification of the input instance by the classification model.

9. The system (100) according to claim 8, wherein the processor subsystem (140) is configured to determine a mask among the plurality of masks by: randomly generating an initial value for the mask and iteratively updating the mask based on the initial value.

10. The system (100) according to claim 8, wherein the processor subsystem (140) is configured to determine the stability value by: calculating a norm of the masks among the plurality of masks and calculating the stability value based on the norm.

11. The system (100) according to claim 1 or 2, wherein the processor subsystem (140) is further configured to obtain a classification of a perturbed input instance and retrain the classification model using the perturbed input instance and the classification.

12. The system (100) according to claim 1 or 2, wherein the input interface (120) is further for accessing an additional classification model, wherein the additional classification model is configured to determine a classification of a mask for a latent space representation, and wherein the processor subsystem (140) is further configured to determine a classification by the additional classification model of the determined mask.

13. A computer-implemented method (300) for determining a mask for an input instance for a classification model, the mask indicating a perturbation that interferes with the classification of the input instance by the classification model, the interference causing the classification to change or become less certain, the method comprising: - accessing (310) a classification model and a generative model, wherein the classification model is configured to determine a classification of an input instance, the input instance having a type, and wherein the generative model is configured to generate a synthetic instance of the type according to a latent space representation, - obtaining (320) an input instance for the classification model, a classification of the input instance according to the classification model, and a latent space representation of the input instance, wherein the latent space representation, when used as an input to the generative model, causes the generative model to generate a synthetic instance approximating the input instance; - determining (330) a mask that indicates a perturbation that interferes with the classification of the input instance by the classification model, wherein the mask indicates a perturbation to the latent space representation of the input instance, wherein the mask is determined based on a classification score of the classification model for the perturbed input instance, the classification score indicating the probability that the input instance belongs to a given class, the perturbed input instance being determined using the mask by: - masking (332) the latent space representation with the mask; and - using the generative model to generate (334) a perturbed input instance according to the masked latent space representation. Wherein the classification model is an image classifier, and wherein the input instance is an input image, and the perturbed input instance is a perturbed input image.

14. A computer-readable medium (400) comprising transient or non-transient data representing instructions arranged to cause a processor system to perform the computer-implemented method according to claim 13.

Images