determining an explanation for the classification

Abstract

The invention relates to a computer-implemented method for explaining a classification of one or more classifier inputs by a trained classifier. A generative model is used, the generative model generating inputs for the trained classifier. The generative model comprises a plurality of filters. Generator inputs corresponding to the one or more classifier inputs are obtained, wherein the generator inputs cause the generative model to approximately generate the corresponding classifier inputs. Filter suppression factors are determined for the plurality of filters of the generative model. The filter suppression factor of a filter indicates a degree of suppression of a filter output for the filter. The filter suppression factors are determined based on adapting an influence of the classifier inputs on the classification of the trained classifier according to the filter suppression factors. The classification explanation is based on the filter suppression factors.

Description

Technical Field

[0001] This invention relates to a computer-implemented method for determining a classification interpretation, and to a corresponding computer-implemented method. The invention further relates to a computer-readable medium comprising instructions for performing the above-described method. Background Technology

[0002] Machine learning-based (“trained”) classifiers (also known as classification models) are widely used in many real-world applications, such as optical quality inspection in manufacturing; hazard detection in autonomous driving; robotics; building control, and more. For example, machine-learnable classifiers, such as neural network classifiers, can be trained to classify the state of a physical system (such as an autonomous vehicle or robot) or its environment (e.g., objects being manufactured in a factory, roads on which vehicles travel, robot workspaces, etc.) based on sensor data acquired by one or more sensors. This state can be classified into two or more categories, such as “OK” or “not OK”; “safe” or “unsafe”; “cat,” “dog,” or “tree,” etc. After the state has been classified, the physical system can be monitored or controlled, for example, by adjusting control parameters of a manufacturing system based on the classification; by controlling one or more actuators of a robot; and so on.

[0003] Typically, as is known in machine learning, machine-learnable classifiers, such as neural network classifiers, can be trained on training data to provide classification. By training a classifier, it can provide this classification without explicitly programming it with instructions on how to do so. For example, to classify images, a classifier can be trained on a training dataset that includes images whose classifications are known. For instance, the classifier can be trained to use multi-class identifiers, which allow it to classify images as, for example, "cat," "dog," or "tree." Another example is that a trained classifier can be trained as a binary classifier, such as classifying an input into only one of two complementary categories, "normal" or "abnormal." However, a trained classifier can also be trained on datasets with unknown categories, for example, through clustering / automatic classification.

[0004] As machine learning classifiers are deployed in real-world scenarios, determining why a classifier categorizes one or more inputs into a particular class becomes increasingly important. Interpretability is a core element of both model verification (have we built the model correctly?) and validation (have we built the correct model?). It is a well-known finding that large, existing-technology network models can easily be fooled into providing false, high-confidence predictions; therefore, their true generalization performance is questionable. As with any other software component intended for field deployment, it is crucial that the model can be quantitatively verified and validated, for example, to establish the extent to which the model has learned the expected input-to-output relationships. Current state-of-the-art techniques in machine and deep learning lack the metrics and practices to measure this effect, often working with limited and therefore inherently biased datasets, resulting in over-parameterized models compared to the available data. Consequently, there is often a significant gap between what the model has actually learned and what the implementer believes the model has learned.

[0005] Over the past few years, the need for explanation and model classification has been recognized, and several methods have been proposed and used at the academic level, achieving varying degrees of success. A series of works attempts to visualize what a neural network classifier has learned by visualizing the activation patterns of each pixel. The goal is to find out how individual pixels influence the classification of an input image.

[0006] An example of this approach is provided in the paper "Interpretable explanations of black boxes by meaningful perturbation" by R. Fong and A. Vedaldi (incorporated hereby by reference). In that work, an optimization is proposed where the objective is to find a minimal deletion mask such that perturbing the image indicated by this mask leads to a significant drop in its classification. Therefore, a causal relationship is sought between changes made to the input and corresponding changes achieved in the resulting classification. For the end user, this change can confirm whether the model is sensitive to anticipated elements, such as whether a pedestrian detector is sensitive to pedestrians, or to sky or weather conditions. Summary of the Invention

[0007] Improving the techniques used to interpret classifications would be desirable. While various interpretability techniques exist in the literature, inventors have observed that these existing techniques are often inconsistent with each other. Furthermore, existing techniques typically fail to present results intuitively for laypeople to understand and grasp, thus requiring extensive explanations from experts in the field. For example, in many cases, pixel-level explanations have been developed that lead to changes in the classifier's output, but these do not correspond to meaningful concepts present in the image. Because such pixel-level explanations do not actually represent meaningful changes to the image, they are not only difficult for humans to understand or understand, but also difficult to extract useful information from them in an automated manner, such as to further classify the explanations based on root causes.

[0008] Furthermore, the inventors recognized that the existing methods of perturbing inputs to obtain changes in classification are not applicable to the diverse datasets encountered in practice. For example, the inventors envisioned using interpretability techniques for optical quality inspection. They found that, in various manufacturing processes, images of manufactured products possess statistical information significantly different from the types of datasets to which existing interpretability techniques are typically applied; therefore, such techniques are unsuitable for this type of data. For instance, as the inventors recognized, adding black squares as perturbations to a black-and-white image of a manufactured product might cause the perturbation to actually add information that the classifier might be sensitive to. Therefore, in this case, it is unclear whether the change in the output classification truly stems from information being retained or removed, or more precisely, whether it truly stems from the addition of such a new element as new information that the model is attempting to interpret. Thus, in this situation, the interpretation may not provide a meaningful explanation for the information present in the image.

[0009] To address these and other problems, according to a first aspect of the invention, a computer-implemented method for determining classification interpretations for a trained classifier, as defined in claim 1, is described. According to another aspect of the invention, a corresponding system as defined in claim 14 is described. According to another aspect of the invention, a computer-readable medium as defined in claim 15 is described.

[0010] Various features are involved in determining a classification interpretation for a trained classifier. This interpretation can be for one or more classifier inputs that are all classified into the same class by the classifier. The interpretation can indicate factors that contribute to the classifier classifying the input into that class; in other words, it influences the classifier's adaptation when applied to classify the input into that class. For example, this adaptation might lead to a change in the classification of that class, or at least a decrease in the classification score for that class.

[0011] Using pixel-level adaptation to an input image to explain its classification is known in itself, as seen in the earlier documented paper "Interpretable explanations of black boxes by meaningful perturbation". For example, as is known, if blackening a specific input pixel in the classifier's input image leads to a significant drop in the classification score, this can be considered an explanation in the sense of indicating aspects of the classifier input that make a significant contribution to the original classification score.

[0012] However, what is of interest is that, instead of using this pixel-level adaptation, the inventors envision using generative models to adapt categorical inputs. A generative model is a model trained on a training dataset to generate synthetic instances that resemble those instances in the training dataset; for example, the generator part of a Generative Adversarial Network (GAN) or the decoder part of a Variational Autoencoder (VAE). For instance, when trained on a dataset of real human facial images, a generative model can learn to generate realistic but synthetic facial images. To generate such instances, a generative model can use multiple filters. For example, a generative model can be a convolutional network (also known as a convolutional neural network) or other types of neural networks, where filters are used to generate filter outputs at corresponding layers of the network.

[0013] As has been demonstrated, this type of generative model using filters has a property of interest: suppression of the filter outputs of specific filters in the generative model often leads to meaningful changes in the generated synthetic instances. Therefore, the corresponding filters of the generative model can be associated with the corresponding aspects of the generated instances. For example, in image generation models, a corresponding filter or filter bank can be associated with the generation of a corresponding object (or other type of feature) that is part of the image to be generated. In a face image generator, one set of filters might be responsible for generating wrinkles, while another set might be responsible for generating a pair of glasses. If some or all of the filters responsible for generating a particular object are suppressed, for example, if their filter outputs are modulated during generation, this could result in the corresponding object being absent or at least less prominent in the resulting instance. However, the resulting instance usually still appears realistic. For example, suppression of the glasses would not leave empty spaces in the image but would result in, for example, the generation of synthetic skin. Essentially, filter-based generative models can therefore compensate for the information loss due to the suppression of filters by incorporating other information.

[0014] The inventors envision using this property to generate meaningful fits to classifier inputs and interpreting the impact of these meaningful fits on the classification performed by the trained classifier. For this purpose, a generative model can be used, whose output can be fed into a trained classifier. Given an input instance of a trained classifier for which its interpretation is to be determined, a generator input can be determined (or otherwise obtained) that causes the generative model to approximately generate the classifier input. By manipulating the way the generative model thereby (approximately) generates the classifier input, meaningful fits to the classifier input can be obtained (e.g., those preserved on the natural image manifold).

[0015] Therefore, many fittings can be made that are meaningful in terms of the information contained in the classifier input (e.g., information indicating the presence of a particular object or other type of feature in the generated instance, such as whether a person is wearing glasses). However, not all of these meaningful fittings are relevant to the trained classifier (e.g., a trained classifier can classify a facial image based on whether a person has sunburn, but wearing glasses would be irrelevant for sunburn). The inventors envision trying various such meaningful fittings and observing which of them influences the classification made by the trained classifier, thereby obtaining an interpretation of the classification based on these meaningful fittings.

[0016] Specifically, the adaptation of the generative model's generation process can be captured based on the filter suppression factors of the corresponding filters in the generative model. The filter suppression factor indicates the degree of suppression applied to the filter output. The effect of the filter suppression factors can be determined by adapting the classifier input according to these filter suppression factors (meaning adjusting the filter output of the generative model as it generates the classifier input), and then applying the trained classifier to the resulting adapted classifier input. For example, filter suppression factors can be applied individually to observe their respective effects, or more preferably, the effect of adjusting several filters simultaneously on classification can be determined.

[0017] Therefore, the classification of input instances by a trained classifier can be explained based on these filter suppression factors. For example, if adjusting the filter output through the corresponding filter suppression factor causes the adapted classifier input to be classified into a different class than the original classifier input, then the adapted classifier can provide an explanation for the classification in the form of a counterfactual for the original classifier input. This counterfactual provides important insights into which features of the original classifier input led the trained classifier to its classification.

[0018] Filter suppression factors can themselves be considered as representations of interpretation. Specifically, given multiple classifier inputs classified into the same class by a trained classifier, a set of filter suppression factors can be determined that indicates which aspects of the classifier input typically affect the classification performed by the trained classifier. This is possible because it is of interest that filter suppression factors generally have similar effects for different classifier inputs. For example, a particular generator filter can be associated with the generation of trees, so suppressing the filter can remove trees from different images, even if these trees appear in different locations within the corresponding images. Furthermore, due to the use of the generative model, these trees can be compensated for in the generated instances based on the context of the specific classifier input, for example, by inserting grass or flowers in appropriate places. This contrasts, for example, with determining the pixel mask of the input image to be applied to an image classifier—the corresponding images will typically contain objects in different locations, so a common pixel mask cannot provide a common interpretation for multiple images regarding the presence or absence of such objects.

[0019] Preferably, the set of filters for which the filter suppression factor is determined represents a broad range of objects or aspects that may contribute to the classification decisions of the trained classifier, rather than just a specific contributing object or aspect. For example, the filter suppression factor can be determined for all filters of the generative model or for all filters of one or more layers of the generative model. The training of the generative model tends to partition its latent space in such a way that different concepts are orthogonally represented across different filters, thus allowing different aspects to be suppressed independently of each other. It is also preferred to use a generative model trained on a dataset substantially the same as the trained classifier (e.g., the same dataset as the training dataset of the trained classifier or a randomly sampled subset). In this way, the instances generated by the generative model are more representative of the typical inputs of the trained classifier, and therefore represent a more meaningful fit.

[0020] In general, using the technique proposed in this paper, interpretations of a trained classifier can be obtained based on a semantically meaningful adaptation to its input instances. This effectively models perturbations in a more sophisticated way, making changes to the classifier input more likely to result in another natural classifier input. Consider an image classifier as an example: this classifier relies on the presence or absence of glasses on a human face to make classification decisions. For example, using pixel-level methods, removing glasses from only one eye would suffice for a decrease in classification score. However, since generative models are typically trained to either generate glasses or not, suppressing filters associated with glasses often leads to removing glasses from both eyes, resulting in a more natural and therefore easier-to-understand interpretation.

[0021] As a result of adapting the classifier input in a more natural way, the probability of obtaining adversarial interpretations (e.g., almost invisible pixel-level changes that still lead to alterations in the classification performed by the trained classifier) ​​is also reduced. Furthermore, the problem of pixel-level perturbations potentially introducing information into the classifier input instead of removing it is addressed. Moreover, providing interpretations based on filter suppression factors allows for the efficient representation of interpretations, for example, based on a set of factors rather than as an image. This is advantageous for the further automated processing of these interpretations, not only for applying further machine learning models to them, but also for interpreting these interpretations by humans.

[0022] It also makes interpretations for different input instances more comparable; for example, suppressing glasses at different locations in an input image can be similarly represented based on the filter suppression factor. This allows, for instance, the compilation of statistics or other means to obtain insights into how often a trained classifier uses specific aspects of the classifier input to make decisions.

[0023] Interestingly, trained classifiers can essentially be considered black boxes. Therefore, these techniques are applicable to many trained classifiers. A trained classifier does not need to provide additional functionality beyond classification; for example, in many cases, a trained classifier is not a segmentation model, etc. Therefore, an interpretation can be determined for a trained classifier where no interpretation is yet available for its output.

[0024] Alternatively, the trained classifier can be an image classifier. For image classification, various generative models are known, which work particularly well when combined with the presented techniques (e.g., convolutional neural networks, and more specifically, fully convolutional networks). It has been observed in various practical settings that adapting image generation within such networks using suppression filters results in seemingly realistic fits, and various relevant semantically interpretable aspects of the generated images can be tuned using suppression filters.

[0025] Optionally, the input instance may include images of products manufactured during the manufacturing process. A trained classifier can categorize such images into, for example, "normal" and "abnormal" (NOK) categories. In this case, it is important not only to know that the manufactured product is problematic, but also to know why the problem exists. For example, information about individual products or aggregated over a batch of products can be reported to the supervisor of the manufacturing process so that appropriate action can be taken. As an additional optional step, the image interpretation determined based on filter suppression factors can be categorized into a predefined set of possible anomalies. The detected anomalies can then be reported to the supervisor, for example, to help take appropriate action, or automatically used to determine control signals for controlling the manufacturing process, such as by adapting one or more control parameters of the manufacturing process (e.g., speed / temperature / pressure), or by operating the manufacturing process in a safe mode.

[0026] More generally, the input instances for the trained classifier can be various types of sensor signals, including video data, radar data, LiDAR data, ultrasonic data, motion data, and thermal imaging camera data. Specifically, as is known per se, these various sensor signals can be represented in 2-D or ND spatial grids and therefore can be processed by various image processing techniques or generalizations thereof (e.g., image generation models and / or image classifiers). Thus, by using appropriate generative models and trained classifiers, a wide variety of sensor signal types can be supported.

[0027] Optionally, the filter suppression factor can be determined by optimizing its effect on the classification of the classifier input. This optimization can include minimizing the difference between the target classifier output and the affected classifier output of the trained classifier for one or more classifier inputs affected by the filter suppression factor. For example, the target classifier output may indicate a class different from the class to which the class(s) input(s) are classified, or it may indicate a decision boundary for that class. Therefore, this difference can be used to encourage changes in the classification of the classifier input, or at least a decrease in the classification score. The optimization may also include a regularization term, such as a regularization term to minimize the overall degree of suppression indicated by the filter suppression factor. The regularization term is preferably sparsity-inducing, for example, using the L0 norm or its approximation to limit the number of suppressed filters. This encourages limiting the amount of change made to the classifier input and, consequently, obtaining an interpretation that is as relevant as possible to the original classifier input.

[0028] Optionally, this optimization can be further configured to minimize the degree to which the output of the generative model appears synthetic. This degree can be determined by a machine-learnable model called a discriminative model. For example, the discriminative model can be the discriminator part of a generative adversarial model (GAN), such as a GAN with a generator as its generator part. However, this is not required; the discriminative model can also be trained separately from the generative model. Including this degree of realism further encourages the optimization to determine a seemingly realistic fit to the trained input and, accordingly, further improves the quality of the determined explanation.

[0029] Alternatively, when minimizing the overall degree of suppression indicated by the filter suppression factor, more unique filters may be penalized less than less unique filters. The degree to which a filter is unique can be indicated by the uniqueness score of the corresponding filter. Therefore, including more unique filters in the classification explanation can be encouraged, leading to reduced redundancy in the output explanation. This improves the quality of the explanation, for example, by reducing the risk of adversarial solutions, since individual filter suppression more directly corresponds to a specific and individual fit to the classifier input.

[0030] Alternatively, the output classification interpretation can be determined by assessing the difference between the classifier input and the corresponding adapted classifier input based on one or more metrics. For example, metrics could include pixel-wise differences, differences in color distribution, and / or entropy. These metrics, or combinations thereof, can be characteristics of the specific way the trained classifier makes decisions. Therefore, they can, for example, be used as features of further machine learning models to be applied to the interpretation. This allows for a particularly compressed representation of the classification decision, thereby simplifying further automated processing.

[0031] Alternatively, the filter output of the generative model's filter can be adjusted based on a filter suppression factor by multiplying the elements of the filter output by that suppression factor. For example, the filter suppression factor can be a value between 0 and 1, where 0 indicates the filter will be fully suppressed, and 1 indicates the filter is not suppressed at all. This is a flexible but effective (and optimizable) solution, but alternatives exist. For example, the filter suppression factor can be a Boolean value indicating no suppression or full suppression; or it can be applied using a type of computation other than multiplication, such as based on any function that increments both the activation value to which the suppression factor will be applied and the suppression factor itself.

[0032] Optionally, a classifier input can be obtained, and based on this, a generator input corresponding to the classifier input can be determined. This allows for the determination of an interpretation for a given classifier input. Given a classifier input, it is generally impossible to determine a generator input that precisely reproduces the classifier input, for example, because the number of possible classifier inputs is typically much larger than the number of possible generator inputs. Therefore, a generator input that approximates the classifier input can be determined, for example, based on optimization involving the difference between the classifier inputs and their approximations. For example, this optimization can be performed to determine the generator input based on the classifier input (which is known in itself for generative adversarial networks), or the model can be trained to determine the generator input based on the classifier input (e.g., the encoder of an autoencoder). In any case, such optimization is typically heuristic and / or reaches local optima. Therefore, the generator input corresponding to the classifier input will generally not be the generator input that provides an absolutely best approximation over all possible generator inputs.

[0033] Optionally, the class of the trained classifier can be obtained, and based on this, one or more generator inputs can be generated that cause the generative model to generate classifier inputs according to that class. This allows for the determination of explanations for the more global behavior of the trained classifier, such as why the trained classifier typically classifies classifier inputs into a given class. For example, classifier inputs from that class can be obtained through rejection sampling: by generating classifier inputs and checking whether they belong to that class. Thus, a representative set of generator inputs can be obtained. It is also possible to check whether the generated classifier inputs satisfy specific properties in addition to belonging to that class, or whether the generator inputs or classifier inputs are hand-picked. This allows for the study of how the trained classifier derives its classification decisions for a given set of classifier inputs that share specific characteristics.

[0034] Optionally, the classification explanation can be output to the user in a sensorily perceptible manner, such as being displayed on a screen. For example, the classification explanation may include an adaptation of one or more classifier inputs(s) based on a determined filter suppression factor, the classification of which will be interpreted. This adapted classifier input, whose classification is affected (e.g., it may belong to a different class or may be located at a decision boundary), can be referred to as a counterfactual. For example, the original classifier input can be output along with one or more counterfactuals; counterfactuals can be used to emphasize parts of the presented classifier input; or vice versa. Other types of explanations determined based on filter suppression factors (e.g., classification based on the filter suppression factor) can also be output in any suitable manner. Thus, actionable information about the state of the trained classifier and / or available data can be delivered. For example, hidden biases that cannot be detected, for example, using conventional performance evaluation methods such as sensitivity or specificity measurements, may be revealed.

[0035] The classification explanations provided in this paper can be particularly useful to end users, such as those supervising the manufacturing process who want to understand problems with the manufactured products, or clinicians who want to understand why a medical image classification model arrives at a given classification. In this context, the classification explanation can be viewed as an explanation of why the classifier input actually belongs to that given class. In other words, it can be assumed that the classifier correctly classified the input into that class, and the user may want to use information about why the classifier (correctly) classified the input into that class to improve their own understanding of the output. For example, this would correspond to a fully-fledged, trained classifier deployed in the field. This feedback to the end user is greatly improved by providing a more semantically meaningful explanation, such as classifier input that has been adapted in a meaningful and / or realistic way.

[0036] However, classification interpretations can be valuable to users when the trained classifier itself is still under development (perhaps after field deployment). In this case, the user can be a model developer who can use the interpretations to further improve the model. For example, if the interpretation indicates a part of the classifier input that seems irrelevant to a particular class (e.g., an image region), this can indicate a problem in the training context, such as hidden bias in the training dataset, a lack of training examples of a given type, or a problem with the model architecture of the trained classifier. In this case, the classification of the trained classifier is not necessarily assumed to be correct, and the interpretation can be used to understand why the classifier incorrectly classified the input into that class, allowing for the correction of such misclassification.

[0037] Optionally, the adapted classifier input can be output to a user, who can then provide the desired classification for the adapted classifier input. This desired classification can then be associated with the adapted classifier input for use when retraining the trained classifier using the adapted classifier input and the desired classification (e.g., as training and / or test data). Retraining can be performed by the same or different systems. For example, the adapted classifier input can be obtained by performing optimization that encourages classification near the decision boundary of the trained classifier. Thus, valuable additional training data can be obtained, which can improve the trained classifier. By using a generative model, the adapted classifier input becomes more realistic, and therefore, more high-quality training data can be obtained. Furthermore, by changing the generator input of existing training instances, or by randomly generating generator inputs, it is possible to freely generate additional training data in a counterfactual form for any given number of classifier or generator inputs.

[0038] For example, the generated counterfactuals can, for instance, visually represent the decision boundary of a given input model. If this decision boundary represents an undesirable bias (e.g., a given counterfactual should not be classified differently from the original image), then the counterfactuals can be used to retrain and / or fine-tune the existing model accordingly.

[0039] When optimizing a process that balances the difference between the target classifier output and the adapted classifier output (one aspect) with the overall suppression level (another aspect), these two aspects can be weighted in the optimization process by appropriate hyperparameters. These hyperparameters can be learned automatically, which may involve repeating the process of determining the filter suppression factor several times for the corresponding (e.g., random) generator input. The performance of the techniques presented in this paper can be tuned by appropriately selecting the hyperparameters. For example, hyperparameters can be selected such that the difference in classifier outputs is sufficiently weighted to ensure that the adapted classifier input actually represents the counterfactual of the different classes, while on the other hand, hyperparameters can be selected such that the overall suppression level is sufficiently weighted to ensure that there is no significant change from the original input. For example, grid search can be used to learn the hyperparameters.

[0040] The various trained models used in this paper (e.g., trained classifiers and generative models) can be parameterized by their respective parameter sets. For example, generative models and / or trained classifiers can be neural networks. Neural networks are also known as artificial neural networks. Examples include deep neural networks and convolutional neural networks. In this case, the parameter set can include the weights of the nodes in the neural network. For example, the number of layers in the model can be at least 5 or at least 10, and the number of nodes and / or weights can be at least 1000 or at least 10000. Depending on the specific application, various known architectures of neural networks and other types of machine-learnable models can be used. From a training efficiency perspective, it is beneficial to use models that follow gradient-based optimization, for example, models that are continuous and / or differentiable in their parameter set.

[0041] Those skilled in the art will appreciate that two or more of the above embodiments, implementations and / or optional aspects of the present invention can be combined in any manner deemed useful.

[0042] Those skilled in the art can make modifications and variations to any system and / or any computer-readable medium based on this description, corresponding to the modifications and variations described in the description of the methods implemented on the respective computers. Attached Figure Description

[0043] These and other aspects of the invention will become apparent and further illustrated by reference to the embodiments described by way of example in the following description and with reference to the accompanying drawings, in which: Figure 1 A system for determining classification interpretations for a trained classifier is shown; Figure 2a An example of a generative model is shown, in which the generative model is the decoder part of an autoencoder; Figure 2b An example of a generative model is shown, in this case, the generative model is the generator part of a generative adversarial network (GAN); Figure 3 A generative model comprising multiple filters is shown, along with an example of its manipulation using filter suppression factors; Figure 4a An example is shown of how to determine one or more filter suppression factors used to interpret the classification; Figure 4b An example of determining the generator input is shown; Figure 4c An example of determining the generator input is shown; Figure 5 An example is shown where a classification interpretation is determined and used to control the manufacturing process; Figure 6 A method for determining the computer implementation of the interpretation is shown; Figure 7 A computer-readable medium containing data is shown.

[0044] It should be noted that these figures are purely schematic and not drawn to scale. In the figures, elements corresponding to those already described may have the same reference numerals. Detailed Implementation

[0045] Figure 1 A system 100 for determining classification interpretations for a trained classifier is illustrated. The classification interpretation can be for one or more classifier inputs classified into the same class by the trained classifier. System 100 may include a data interface 120 and a processor subsystem 140, which can communicate internally via data communication 121. Data interface 120 can be used to access model data 030 defining the trained classifier. Data interface 120 can also be used to access model data 040 defining a generative model. The generative model can be configured to generate classifier inputs for the trained classifier from generator inputs. The generative model may include multiple filters. The filters of the generative model can be configured to generate filter outputs at internal layers of the generative model.

[0046] The processor subsystem 140 can be configured to access model data 030, 040 during operation of system 100 and by using data interface 120. For example, Figure 1 As shown, data interface 120 can provide access 122 to external data storage device 021, which may include the model data 030, 040. Alternatively, the model data 030, 040 can be accessed from an internal data storage device that is part of system 100. Alternatively, the model data 030, 040 can be received from another entity via a network. Typically, data interface 120 can take various forms, such as a network interface to a local area network or wide area network (e.g., the Internet), a storage interface to an internal or external data storage device, etc. Data storage device 021 can take any known and suitable form.

[0047] The processor subsystem 140 can be configured to obtain generator inputs corresponding to one or more classifier inputs during operation of the system 100 and by using the data interface 120. The generator inputs enable the generative model to approximately generate the corresponding classifier inputs.

[0048] The processor subsystem 140 can be further configured to determine filter suppression factors of multiple filters for the generative model during operation of the system 100 and by using the data interface 120. The filter suppression factor indicates the degree of suppression for the filter output. The filter suppression factor can be determined based on adapting the classifier input to the classification of the trained classifier according to the filter suppression factor.

[0049] To determine this effect, processor subsystem 140 can adapt the classifier input based on one or more filter suppression factors by applying the generative model to the generator input corresponding to the classifier input, while simultaneously adjusting the filter output of the generative model's filters according to one or more filter suppression factors. To further determine the effect, processor subsystem 140 can apply the trained classifier to the adapted classifier input to obtain a classifier output affected by the one or more filter suppression factors. To further determine the effect, processor subsystem 140 can further determine the classification interpretation based on the filter suppression factors and output the classification interpretation.

[0050] The processor subsystem 140 can also be configured to train a classifier and / or a generative model, for example, based on a public dataset. The processor subsystem 140 can also be configured to learn hyperparameters used in the optimization for determining the filter suppression factor. For example, the determination of the filter suppression factor can be performed multiple times to learn such hyperparameters, for example, in a grid search.

[0051] As an optional component, system 100 may include an image input interface 160 or any other type of input interface for acquiring sensor data 124 from a sensor (such as camera 071). Processor subsystem 140 may be configured to determine one or more classifier inputs leading to a trained classifier based on the acquired sensor data 124. For example, the classifier inputs may be used to determine a corresponding generator input, based on which a filter suppression factor may be determined to interpret how the trained classifier classifies these classifier inputs. For example, camera 071 may be configured to capture image data 124, and processor subsystem 140 may be configured to determine classifier inputs based on the image data 124 acquired from input interface 160 via data communication 123. The input interface may be configured for various types of sensor signals, such as video signals, radar / LiDAR signals, ultrasonic signals, etc.

[0052] As an optional component, system 100 may include a display output interface 180 or any other type of output interface for outputting the determined classification interpretation to a presentation device (e.g., display 190). For example, display output interface 180 may generate display data 182 for display 190, which causes display 190 to present the interpretation in a sensorily perceptible manner, such as as input to classifier(s) and a corresponding adapted on-screen visualization 192 based on a determined filter suppression factor. Optionally, system 100 may include a user input interface (not shown), such as a keyboard or mouse. Using the user input interface, the system can obtain the desired classification from adapted classifier input output to a user. The adapted classifier input may be stored along with its desired classification for retraining the trained classifier. Processor subsystem 140 may perform this retraining itself, but may also provide the adapted classifier input and desired classification to another party performing the retraining.

[0053] As an optional component, system 100 may include a control interface (not shown) for providing control signals to control external processes (e.g., a monitored manufacturing process). For example, the control interface may be a dedicated control interface, such as Foundation Fieldbus, PROFIBUS, HART, Modbus, etc.; or a general-purpose communication interface arranged for digital communication with the control system, such as using USB, IEEE 1394, or a similar interface. The control interface can communicate via a computer network, such as a wireless personal area network, the Internet, an intranet, a LAN, WLAN, etc. For example, the control interface may include connectors, such as wireless connectors suitable for computer networks, Ethernet connectors, Wi-Fi, 4G or 4G antennas, ZigBee chips, etc. The control interface may also be an internal communication interface, such as a bus, API, storage interface, etc. Specifically, the control interface may be an actuator interface for providing actuator data to actuators that enable the actuators to operate within the environment of system 100. For example, processor subsystem 140 may be configured to determine the actuator data at least in part based on a determined classification interpretation.

[0054] Reference Figure 2a-5 This will further elaborate on the various details and aspects of the operation of System 100, including its optional aspects.

[0055] Typically, System 100 may be embodied as or in a single device or apparatus, such as a workstation, e.g., a laptop- or desktop-based workstation, or a server. This device or apparatus may include one or more microprocessors executing appropriate software. For example, the processor subsystem may be embodied by a single central processing unit (CPU), but may also be embodied by a combination or system of such a CPU and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory, such as volatile memory (e.g., RAM) or non-volatile memory (e.g., flash memory). Alternatively, the functional units of the system (e.g., data interfaces and processor subsystems) may be implemented in the device or apparatus as programmable logic, e.g., as field-programmable gate arrays (FPGAs) and / or graphics processing units (GPUs). Typically, each functional unit of the system may be implemented as a circuit. It should be noted that System 100 may also be implemented in a distributed manner, e.g., involving different devices or apparatuses, such as a distributed server employing cloud computing.

[0056] Figure 2a A detailed, but non-limiting, example of a generative model is shown, in which the generative model is the decoder part of the autoencoder. Various types of autoencoders, including regularized autoencoders and / or variational autoencoders, can be used in this paper. Variational autoencoders are preferred due to their improved generalization. Many autoencoder architectures suitable for various types of input data, such as images, are known and can be used.

[0057] The figure illustrates an autoencoder, comprising an encoder section Enc 250 and a decoder section Dec 240. Given an input instance II 210, the encoder section Enc of the autoencoder can be configured to determine a representation GI 220 of the input instance II. Based on this representation, the decoder section Dec can be configured to determine a synthesized output GO 270.

[0058] An autoencoder can be trained on a training dataset of input instances to output a synthetic output SO that is as similar as possible to the corresponding input instance II, for example in optimization that minimizes the difference between the training input instance II and the corresponding synthetic output SO (possibly in combination with other terms, such as regularization). Thus, the autoencoder can learn to determine the representation GI that best allows for such reconstruction. In the context of an autoencoder, the output of the encoder part Enc can be referred to as "code," "latent representation," or "encoding."

[0059] Therefore, the decoder part Dec of the autoencoder can be used as a generative model that can generate the output GO from the generator input GI (e.g., the input to a trained classifier). Interestingly, the encoder part Enc provides an inverse transformation from the classifier input to the generator input; for example, the autoencoder provides an efficient and accurate method to determine, given an instance, the generator input that causes the generative model Dec to approximately generate that instance.

[0060] Figure 2b A detailed but non-limiting example of a generative model is shown, in which the generative model is the generator part of a generative adversarial network (GAN).

[0061] The GAN shown in the figure includes a generator part Gen 240' and a discriminator part Disc 260. The generator part Gen can be configured to generate a synthesized generator output GO 270' from the generator input GI 220'. The discriminator part Disc can be configured to provide a discriminator output that distinguishes between the synthesized instance SII and the training instance TII 210'. 280.

[0062] The generator (Gen) can be effectively trained to generate instances GO that the discriminator (Disc) cannot distinguish from the real training instances TII, while the discriminator (Disc) is trained to correctly distinguish between real instances TII and synthetic instances SII. For example, the generator (Gen) can be trained together with the discriminator (Disc) until a Nash equilibrium is reached, in which case the generator (Gen) can generate efficient instances that the discriminator (Disc) can no longer distinguish from the actual sample TII. In this way, the generator can learn to model the data distribution of the input TII.

[0063] Therefore, the generative part Gen of a generative adversarial network can be used as a generative model that can generate an output GO from the generator input GI (e.g., the input to a trained classifier). Interestingly, the discriminative part Disc itself is also useful, i.e., as a model used to determine the extent to which the output of the generative model Gen appears synthetic.

[0064] As an illustrative example, a known training objective for training a GAN is: In this example, D It is the discriminator (Disc). G It is the generator Gen. x It is the input sample TII, and z From the hidden distributionp z The latent variable GI is extracted from (e.g., a random normal distribution or a similar distribution). However, many variations of the above training objective are possible.

[0065] Figure 3 A generative model comprising multiple filters is shown, along with detailed but non-limiting examples of its manipulation using filter suppression factors.

[0066] The diagram illustrates a generative model configured to generate an output GO 370 from a generator input GI 320 (e.g., an input instance leading to a trained classifier). For example, this generative model could be the decoder portion of an autoencoder, as shown in the diagram. Figure 2a The subject of discussion; or the generator part of generative adversarial networks, such as, for example, regarding Figure 2b The subject of discussion.

[0067] Typically, the generator input GI is a feature vector, for example, comprising at most or at least 100, or at most or at least 1000 entries. The generated output GO can typically take various shapes, but in many cases it is generated as ND space data (e.g., 2-D or 3-D). Image data is an example of this type of data, but various other types of sensor data can also be represented in this way and therefore can be synthetically generated using generative models that output ND space data.

[0068] Typically, the input GI to the generative model is much smaller than the generated output GO in terms of the number of elements. For example, the number of elements (e.g., features) in the generator input GI can be at most 10% or even 1% of the number of entries in its output. Therefore, the generator input GI can effectively provide a compressed representation of its output. It has been observed in various practical scenarios that semantic meaning can be associated with specific entries or combinations of entries in the generator input GI, for example, for generative adversarial networks and for autoencoders; for example, one set of entries in the generator input GI might be responsible for generating ears in a facial image, while another set might be responsible for generating glasses. Therefore, by adjusting the generator input GI, variations of the generated output GO can be obtained.

[0069] The generative model shown in the figure uses a set of filters to generate its output GO. These filters are typically arranged across multiple layers of the generative model. In particular, the generative model can be evaluated by calculating the activation values ​​of subsequent layers of the generative model (starting from the generator input GI and eventually reaching the output GO) based on the activation volumes of previous layers. For example, the number of inner layers (and therefore all layers other than the input and output layers) can be at most or at least 5, at most or at least 10, or at most or at least 20.

[0070] At least some of these layers include one or more filters. The figure illustrates layer Lk341 of the generative model, which includes multiple filters (and in this case, is actually composed of multiple filters). The figure shows filters Fk,1 341-1 up to Fk,I341-2. The number of filters at a layer can be, for example, at most or at least 128, or at most or at least 1024.

[0071] The filter can be a parameterized function that calculates a portion of the output activation volume AVk372 of layer Lk based on the activation volume AVk-1 371 of the previous layer. Therefore, as shown in the figure, the activation volume at layer Lk of the generated model can include slices determined by the corresponding filters, each slice having the same spatial dimensions, for example, in 2-D or 3-D. For example, the figure shows corresponding slices 373, 374, and 375 of the activation volume AVk of layer Lk; slice 375 is, for example, determined as the filter output of filter Fk,1; slice 373 is determined as the filter output of filter Fk,I; and so on. Thus, a layer can be decomposed into multiple filters Fk,j, which can be evaluated individually to determine their respective filter outputs, and therefore, their activity can be suppressed individually.

[0072] Typically, the filter Fk,j is a convolutional filter, for example, a filter whose output is computed by applying a convolution operator to the activation volume Avk-1 at a previous layer. A model using such a convolution operator is called a convolutional neural network. The convolutional filter can be parameterized by the convolution coefficients, which are trained accordingly when training the generative model. However, it is also possible to perform separate transformations on other filters at layers of the generative model.

[0073] Various model architectures for filter-based generative models are known and can be used on their own. Some models are so-called fully convolutional networks, where each layer of the network includes multiple convolutional filters. An example of such a generative model, the so-called "DCGAN," is given in A. Radford et al.'s "Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks" (incorporated by reference). Another example of a fully convolutional network is the U-NET architecture from O. Ronneberger et al.'s "U-Net: Convolutional Networks for Biomedical Image Segmentation" (incorporated by reference).

[0074] However, it is not necessary to have multiple filters at every layer of the generative model. For example, the model can have one or more fully connected layers, followed by multiple layers, each containing multiple filters. In many generative models, the number of filters at subsequent layers is typically reduced or at least not increased. For example, one layer can have half the number of filters as the previous layer, such as 512 filters, followed by 256 filters, then 128 filters, and so on. Therefore, the total number of filters in a generative model can be, for example, at most or at least 500, at most or at least 2000, or at most or at least 2000. The size of subsequent activation volumes is typically chosen to increase or at least not decrease. For example, the width and height of the activation volume at one layer can be twice that of the previous layer.

[0075] As shown in the figure, the process of generating the output GO based on a given generator input GI can be manipulated by adjusting one or more filters of the generative model according to the filter suppression factor. Therefore, if the generator input GI corresponds to the input of a trained classifier, this adjustment can obtain an adapted input for the trained classifier.

[0076] Specifically, the figure shows the vector FSV 380 of the filter suppression factors for the corresponding filters of the generative model. Specifically, filter suppression factors 383, 384, and 385 corresponding to the corresponding filters of layer Lk are shown; for example, filter suppression factor 383 corresponds to filter Fk,I, while filter suppression factor 385 corresponds to filter Fk,1. In general, the vector FSV can provide filter suppression factors for all filters of the generative model or for a subset thereof (e.g., for all filters at a given subset of layers, or for any subset of filters). The appropriate selection of filters for determining the classification interpretation is discussed elsewhere.

[0077] Determining the output of a generative model adapted to a set of filter suppression factors (FSVs) can include adjusting the filter output of the corresponding filter according to the respective filter suppression factor (FSV). The filter suppression factor can indicate the degree of suppression for the corresponding filter output in various ways. For example, the filter suppression factor can be a value between 0 and 1, where 0 indicates complete suppression of the filter output, and 1 indicates complete preservation of the filter output (e.g., no suppression). Therefore, the filter can be adjusted according to the filter suppression factor by multiplying each feature of the filter output by that filter suppression factor. Similarly, the filter suppression factor can be a Boolean value, where True indicates suppression, and False indicates no suppression. In this case, if the filter suppression factor is set to False, the filter output can be adjusted by preserving the filter output; otherwise, the filter output is reset (e.g., reset to zero or another intermediate value). Many variations will be apparent.

[0078] Therefore, an adapted output of the generative model can be obtained by applying the generative model. This can include determining the output volume (e.g., AVk) of the generative model at a layer (e.g., Lk). Specifically, filters can be applied to the output volume AVk-1 of the previous layer. The determined output volume AVk can include the filter outputs of one or more filters for which filter suppression factors are to be applied. In this case, the filter outputs of the filters included in the output volume can be adjusted according to the corresponding filter suppression factors, for example, by multiplication as described above. For example, as illustrated in the figure, the filter output 373 of filter Fk,I can be adjusted according to filter suppression factor 383; the filter output 374 can be adjusted according to filter suppression factor 384; and the filter output 375 can be adjusted according to filter suppression factor 385. Of course, the calculation of the filter output of fully suppressed filters can be skipped. Thus, an adapted output volume AVk can be obtained, which can be used to evaluate subsequent layers of the generative model (potentially involving additional suppression). Thus, the filter suppression factor FSV can be applied at the corresponding layer of the generative model to ultimately obtain the generative model output GO.

[0079] Figure 4a A detailed, but non-limiting example of how to determine one or more filter suppression factors used to interpret the classification is shown.

[0080] In this example, the interpretation of the classification of one or more classifier inputs CI 410 of the trained classifier TC 430 is determined. The classification is interpreted by adapting the influence of the classifier inputs CI on the classification according to a set of filter suppression factors FSV 480. To this end, firstly, a generator input GI 420 can be determined such that the generative model Dec 440 approximately generates the classifier inputs CI. Having determined the generator inputs GI, the filter suppression factors FSV can then be used to manipulate the process by which classifier inputs are generated from the determined generator inputs GI, where these manipulations result in an adapted classifier input ACI 470. If these manipulations affect the classification performed by the trained classifier TC (e.g., the adapted classifier input ACI is classified into a different class), then the filter suppression factors FSV corresponding to these manipulations indicate the filters associated with the classification performed by the classifier TC, and can be accordingly considered to explain the classification.

[0081] This figure illustrates the classifier input CIs whose classification will be interpreted. While a single classifier input CI is shown in this figure, as discussed throughout the text, the interpretation of multiple classifier input CIs, all classified into the same class, can also be determined. For example, the classifier input CI can be an image, such as one with a size of [missing information]. M × N ×l (grayscale) or M × N ×3 (color, e.g., RGB). The classifier input CI can also represent various other types of sensor data, such as images, feature vectors, etc.

[0082] The figure also illustrates a trained classifier TC. Given a classifier input CI, the trained classifier TC can be configured to classify the classifier input CI into one of a set of predefined classes. For example, the classifier can be a binary classifier, which classifies the classifier input CI into one of two classes; or a multi-class classifier, which classifies the classifier input CI into one of three or more classes. For example, an image of a manufactured product can be classified as “normal” or “abnormal,” or traffic signs detected around an autonomous vehicle can be classified as a specific type of traffic sign, such as a speed limit sign, a pedestrian crossing sign, etc. Many types of classifiers are known from the literature, and generally, any classifier suitable for the type of classifier input CI can be used.

[0083] The output of the trained classifier TC is referred to as the classifier output in this paper. For example, as shown in the figure, given a classifier input CI, the trained classifier TC can determine the original classifier output OCO 460. Although the classifier output determined by the trained classifier TC can simply represent the class to which the input is classified, the classifier output typically contains additional information. In particular, the classifier output can include a classification score for the corresponding class. The classification score for a class can indicate the similarity between the classifier input CI and that class, or in other words, the probability that the classifier input CI belongs to that class. Therefore, the classifier input CI can be classified into the class with the highest classification score. In the case of binary classification, the classifier output can also include a classification score for only one of the two classes, with the classification score for the other class implicitly following.

[0084] The diagram also shows the encoder part Enc 450 and the decoder part Dec 440 of the autoencoder. The encoder part Enc is configured to determine the generator input GI 420 from the classifier input CI, while the decoder part Dec is a generative model configured to generate classifier inputs for the trained classifier TC from the generator input GI. For example, the encoder part Enc and the decoder part Dec can be configured as follows: Figure 2a As discussed in [the document]. As explained here, the encoder part Enc and the decoder part Dec are typically trained such that when the encoder part Enc is applied to the classifier input CI to obtain the generator input GI, the generator input GI causes the decoder part Dec to generate the classifier input CI as closely as possible. This generation is often approximate because the encoding performed by the encoder part Enc usually loses some information contained in the classifier input CI.

[0085] Therefore, in this specific example, the classifier input CI can be obtained as input. Based on this classifier input CI, the generator input GI can be determined by the encoder part Enc, which causes the decoder part Dec to approximately generate the classifier input. Instead of using the decoder part Dec of an autoencoder, other types of generative models can also be used. Again, in this case, given the classifier input CI, the generator input GI that approximately generates the classifier input can be determined. Figure 4b An example of this situation is given in the text. In other cases, the generator input GI can be obtained as input, and the classifier input CI can be determined from the generator input GI by applying a generative model. Figure 4c An example of this situation is given in the document.

[0086] The classification of one or more classifier inputs (CI) can be explained by a set of filter suppression factors from multiple filters of the generative model Dec, in which case the set of filter suppression factors is represented by a vector of filter suppression factors (FSV). (See also: ...) Figure 3 As explained, the generation process of the generative model Dec can be manipulated by adjusting the filter outputs of the filters in the generative model Dec according to one or more filter suppression factors. Therefore, by manipulating the generation process of the generative model Dec when it is applied to the generator input GI, an adapted ACI to the classifier input CI can be obtained. As shown in the figure, by applying the trained classifier TC to this adapted classifier input ACI, an adapted classifier output ACO 462 influenced by these filter suppression factors can be obtained.

[0087] Generally, there are various ways to determine the filter suppression factor FSV based on its impact on the classification of the trained classifier TC. For example, it is possible to suppress individual filters, such as by adapting the classifier input CI to individual corresponding filter suppression factors, and selecting the filter suppression factor that has the greatest individual impact on classification, such as the filter suppression factor that causes the largest decrease in the classification score for the class of the classifier input CI. For example, a fixed number of filters with the greatest impact, or filters whose impact exceeds a threshold, can be selected.

[0088] However, this figure illustrates a preferred approach to determining the vector of the filter suppression factor FSV based on optimization Opt 490 relative to the target classifier output TCO 461. This optimization can be configured to minimize the difference between the target classifier output and the affected classifier output ACO of the trained classifier for one or more classifier inputs affected by the filter suppression factor FSV. Simultaneously, the optimization Opt can minimize the overall degree of suppression indicated by the filter suppression factor FSV. Therefore, this optimization encourages approximating the target classification output TCO with minimal suppression. In practice, this optimization provides a trade-off where additional suppression is enabled in the vector of the filter suppression factor FSV only if the contribution of additional suppression to reaching the target classifier outweighs the increase in the degree of suppression. Thus, the most applicable and relevant interpretation for the classifier inputs can be provided.

[0089] Various methods are possible for selecting the target classifier output TCO and measuring its difference from the adapted classifier output ACO. For example, the target classifier output TCO can represent the classification score for a particular class (e.g., a value between 0 and 1, or a Boolean value). The difference from the adapted classifier output can be measured as the difference in the classification score for that particular class.

[0090] For example, the class could be the original class to which the trained classifier TC classifies the classifier input CI, as indicated by the original classifier output OCO. The classification score could be a low score for that class. In this case, the optimization could encourage the removal of aspects from the classifier input CI that cause the trained classifier TC to classify the classifier input CI into the original class. Thus, an adapted classifier input ACI representing the counterfactual can be determined from which the minimum set of information required for the classification is removed.

[0091] The classification score can also be set to or approximate the decision boundary of the trained classifier for the original class. This allows for the generation of an adapted classifier input ACI such that the uncertainty of the trained classifier TC increases for that input ACI. Such an adapted classifier instance can be used to retrain the trained classifier; for example, the desired classification of the adapted classifier input ACI can be obtained, which can then be used as the label for this retraining. However, it should be noted that other adapted classifier inputs (e.g., for which the trained classifier provides an incorrect classification) can also be used for retraining.

[0092] The specific class for which the classification score is used can also be another class besides the original class. A high classification score for another class can encourage optimization of Opt to determine the adapted classifier input ACI for that other class. Therefore, an adapted classifier input ACI representing a counterfactual can be determined, for which minimal removal of information leads to classification in that specific class.

[0093] Many variations are possible. The target classification output TCO can also include multiple classification scores for the corresponding class, which are compared, for example, with the classification score of the adapted classifier output OCO on an L1 or L2 distance basis. For example, the classification score of the original class can be low, while the classification scores of one or more specific other classes can be high to encourage the generation of counterfactuals from these classes. The target classification score can be determined automatically, for example, based on the original classifier output OCO, or obtained as, for example, input from a user. The difference can also be based on the class rather than the classification score.

[0094] As a concrete example of the optimization Opt to be performed, let P This represents a trained classifier that takes instance CI (e.g., ...) as input. M × N × d The input image (of a certain size) is mapped to a 1D output vector. y The 1D output vector y This represents the classification score for the corresponding class; in other words, it indicates whether a class of elements exists in the input image.y k The probability of.

[0095] As a first concrete example, the following loss function can be minimized across multiple generator inputs. z Optimize the above Opt: (1) The loss function is the integral variable. z Defined above. A variable of this kind. z Corresponding to generative models G The possible generator inputs GI of Dec. Therefore, interpretations can be provided for multiple generator inputs being integrated, and correspondingly, a global interpretation on these multiple generator inputs can be determined. These multiple generator inputs GI can be obtained once and kept static throughout the optimization, but it is also possible to dynamically sample generator inputs belonging to a given class as the optimization progresses, such as regarding... Figure 4c The subject of discussion. Through multiple z The loss is defined above, and the filter suppression factor FSV indicates that the filter can reflect the changes that generally need to be made, rather than for any particular input.

[0096] In formula (1), p t The target classification output TCO is represented as a classification score for a specific class. Therefore, p t It can be a scalar value between 0 and 1, representing a certain probabilistic goal we want to achieve. For example, the target classification output can be set to 1 to encourage the generation of adapted classification instances from a certain class, or set to 0 to prevent such generation.

[0097] Therefore, the items shown here Here is an example of the following: This item measures the target classification output TCO and probability output. The distance between ACOs, and the probability output. ACO is a vector of the filter in Dec that already has the filter suppression value FSV. x The output generated under the specified modified constraints, given a Dec. As input, it is returned by the trained classifier TC. As explained elsewhere, instead of... Various other distance measurement items are also possible.

[0098] The second term in Equation (1) is optional, but it illustrates how to further configure the optimization of Opt to minimize the degree to which the output ACI of the generative model appears synthetic. The function in this example... DIt is configured to provide high values ​​for inputs that appear to be synthetic and low values ​​for inputs that appear to be non-synthetic. For example, if G If it is the generative part of a generative adversarial network, then... D This could be a discriminative part. However, it's also possible to use a separately trained model configured to determine the degree to which the output of the generative model appears synthetic. Including the degree of syntheticity is useful because it encourages the optimization Opt to prefer more realistic samples over those that might appear more artificial.

[0099] As the third term, formula (3) includes regularization. L ( x This regularization L ( x This minimizes the overall suppression level indicated by the filter suppression factor FSV. Several options are available for minimizing the overall suppression level.

[0100] For example, regularization L ( x The norm () can be a sparse induced norm of the filter suppression factor (FSV), such as the L0 norm or L1 norm. Of course, depending on how the FSV represents suppression, these values ​​can be transformed before calculating the sparse induced norm, such that a filter suppression factor that does not indicate suppression is represented by 0, and a filter suppression factor that indicates suppression is represented by a non-zero value, such as 1 for complete suppression. By optimizing the sparse vector of the FSV, a more specific classification interpretation is determined. Moreover, having a relatively small number of filter suppression factors that indicate suppression makes it easier, for example, to represent this interpretation to a user (e.g., in text form) or in further automated processing.

[0101] Regularization is based on the uniqueness score indicating the uniqueness of the corresponding filter, compared to the suppression of less unique filters. L ( x It's possible to less strongly penalize the suppression of more unique filters. This also helps to provide a more focused interpretation. For example, it's preferable to provide an interpretation based on a filter that is more uniquely responsible for generating the specific object in the generated instance ACI (e.g., glasses in an image of a certain situation) rather than one filter from a set of filters performing a similar task, because it makes it easier to compare interpretations for different classifier input CIs. Another advantage is that more unique filters tend to reduce the likelihood of adversarial solutions, since the suppression of more unique filters more directly corresponds to manipulation of the specific object in the generated adapted classifier input ACI.

[0102] For example, the uniqueness score of a filter can be obtained by applying an auxiliary network (not shown) to the activations of the corresponding filters for the generator input GI. This network can return a quantitative measure of the number of nearby neighbors for each filter. Based on this quantitative measure, high-cost filters with many nearby neighbors are assigned high-cost values, while low-cost filters are assigned low-cost values. Instead of such an auxiliary network, statistical information such as differential entropy can also be used to provide the uniqueness score, for example.

[0103] As another concrete example, a loss function is now given for use in optimizing Opt, which determines the filter suppression factor FSV for a single classifier input CI and the corresponding generator input GI, and accordingly examines the local behavior of the classifier trained on that particular input sample. Compared to Equation (1) above, this loss function uses an approximate generator input... T CI's single generator input z * GI: (2) In optimizing Opt, the relative importance of the difference between the target classifier output TCO and the adapted classifier output ACO, as well as the overall degree of suppression, can be controlled by weighting the corresponding terms using hyperparameters. Therefore, the total energy distribution in this loss function can be balanced, for example, to allow the optimizer Opt (such as SGD or ADAM) to converge or converge faster. For example, these hyperparameters can be included in the objective functions (1) and (2) above. Hyperparameters can be determined automatically, for example, based on whether optimizing Opt successfully produces an output such that model TC classifies the output differently from the original input image CI. For example, hyperparameters can be determined in a grid search, for example, by starting at a certain value and increasing the weight of the term penalizing the distance between the output classification ACO and the target classification TCO if optimizing Opt does not produce a counterfactual.

[0104] As mentioned earlier, in general, the filter suppression factor FSV can be determined for all filters of the generative model Dec, or for any subset of the filters. A broad selection of filters is generally preferred to allow for interpretations based on a wide range of objects or other aspects that may or may not be present in the instance. For example, all filters (or at least 80%) of some or all layers of the generative model can be used; for example, one, two, or at most or at least five layers. For example, using earlier layers (e.g., in the first half of the network) can encourage more low-level interpretations based on relatively small aspects present in the instance, while using later layers (e.g., in the second half of the network) can encourage more high-level interpretations. Another good option is to select at least 25%, or even at least 50%, of all filters in the generative model. For efficiency reasons, it is possible to randomly select and use filters, in which case, for example, using at least 10% of the filters may also be sufficient.

[0105] Another good option is to select a subset of filters based on their uniqueness. As mentioned above, it is possible to obtain uniqueness scores for the corresponding filters. The optimization Opt can then determine the filter suppression factor only for the most unique filters, such as a fixed number or percentage of the most unique filters, for example, up to 25% or even up to 10% of the most unique filters. This not only provides a more focused and easily comparable interpretation but also makes the optimization more efficient by limiting the search space.

[0106] Typically, any known optimization scheme can be used to perform the optimization (Opt), such as gradient-based optimizations like stochastic gradient descent (SGD) or ADAM (see Kingma and Ba's "Adam: A Method for Stochastic Optimization," incorporated herein by reference). Such optimization methods are known to be heuristic and / or reach local optima. Training can be performed on an instance-by-instance basis or in batches, for example, up to or at least 64 instances, or up to or at least 256 instances. Typically, the optimization is initialized starting with a filter suppression factor indicating no suppression (e.g., a value of 1.0). Various known stopping criteria can be used for this optimization, such as the maximum number of iterations, a threshold for the relative change in the objective function between adjacent iterations, etc.

[0107] Then, the classification interpretation for the classifier input CI can be determined based on the filter suppression factor FSV. Several possibilities exist for this. One possibility is to output the filter suppression factor FSV itself as the interpretation. Another option is, for example, to output the adapted classifier input ACI along with the corresponding original classifier input CI. For example, the classifier input ACI could be used to highlight different parts of the adapted classifier input in the classifier input CI, or vice versa.

[0108] Another possibility is to perform one or more comparisons between the classifier input CI and the corresponding adapted classifier input ACI to compute a corresponding measure of difference. This measure of difference can serve as a compact representation of the classification interpretation, and therefore as a compact representation of the classification decision itself, which can be used for further automated processing, such as as a feature of a machine learning model. For example, this measure of difference can be based on pixel-wise difference, color distribution difference, entropy difference, or various other summary statistics. For example, based on pixel-wise difference... It is possible to determine the measure of difference. and / or Another example is calculation. ,in This is an operator that returns a color histogram of a given input image. It can also determine and aggregate these statistics (e.g., calculate the mean and / or standard deviation) across multiple classifier input CIs.

[0109] Figure 4b A detailed, but non-limiting, example of determining the generator input is shown. This example can be found in... Figure 4a It is used as an alternative to obtain classifier input and / or generator input.

[0110] In this example, a classifier input CI 410 is obtained as input, and based on this classifier input CI, a generator input GI 420 that approximately generates the classifier input CI can be determined. In this example, this is achieved by performing an optimization IOpt 495. This optimization is configured to determine the generator input GI by minimizing the difference between the classifier input CI'410 generated by the generator input GI and the classifier input CI given as input. For example, this optimization may include minimizing a loss function that incorporates this difference. Thus, a classifier input CI' that best approximates the given classifier input CI can be found. The classifier input CI' is typically determined by the generative model Gen without applying a feature suppression vector. This optimization may include additional terms, such as regularization terms.

[0111] For example, this example applies when the generative model Gen is the generative part of a generative adversarial network. For instance, optimizing IOpt can be implemented as disclosed in A. Creswell et al., “Inverting The Generator Of A Generative Adversarial Network (II)” (incorporated hereby by reference). However, optimizing IOpt can also be applied to other types of generative models. Typically, for... Figure 4a Various options for optimizing Opt also apply to optimizing IOpt, such as using stochastic gradient descent or ADAM.

[0112] Figure 4c A detailed, but non-limiting, example of determining the generator input is shown. This example can be found in... Figure 4a It is used as an alternative to obtain classifier input and / or generator input.

[0113] In this example, class CLS 463 is taken as input, and one or more generator inputs GI 420 belonging to class CLS and corresponding classifier inputs CI 410 are generated. More specifically, a sampler Sam 496 is used to generate generator inputs GI, which cause the model to generate classifier inputs CI from class CLS. This can be done using rejection sampling, for example, the sampler Sam (e.g., randomly) generates generator instances GI; the generative model Gen generates the corresponding classifier inputs CI; the trained classifier TC 430 classifies the classifier inputs CI into a certain class; and the sampler Sam checks whether the trained classifier has already classified the classifier inputs CI into class CLS. For example, this can be repeated a given number of times, or until the sampler Sam has generated a given number of generator instances GI that cause the generative model to generate classifier inputs from class CLS. The expected number of classifier inputs to be generated can be, for example, one or more, such as at most or at least 10 or at most or at least 50.

[0114] While the sampler Sam typically generates only instances from class CLS, additional selection criteria can be applied. For example, the sampler can require that classification scores for a given class CLS exceed a given threshold, and / or that classification scores for one or more other classes do not exceed a given threshold. These additional selection criteria can also be based on further machine learning models, such as a classifier (e.g., sampler Sam) that also satisfy another property indicated by that further machine learning model (e.g., bad weather conditions, the presence of another object) from class CLS (e.g., a picture of a traffic sign). This allows for the study of classification behavior for a given class under boundary conditions.

[0115] The sampler Sam can also generate generator input GI and classifier input CI from class CLS as needed. For example, also regarding... Figure 4a As described, optimization Opt can be performed with respect to multiple classifier and generator inputs; the sampler Sam can sample a new batch of inputs as needed (e.g., in each iteration). By obtaining new samples, biases caused by specific initial selections of generator / classifier inputs can be avoided.

[0116] Figure 5 A detailed, but non-limiting, example is shown of determining a filter suppression factor and using the determined filter suppression factor to control the manufacturing process.

[0117] The diagram shows manufacturing process MP 500. During manufacturing, manufacturing operation Man 540 occurs. Manufacturing operation can be an assembly operation. Manufacturing process MP is based on control parameter CP 520.

[0118] As is known in the field of manufacturing engineering, manufacturing processes are typically highly parameterized processes, defining strict rules governing how their physical systems operate to produce individual samples. For example, in an assembly line, the control parameters CP of a manufacturing process may include the amounts of applied pressure, heat, torque, force, welding time, etc. Therefore, it is valuable to gain insights into how variations in these parameters can affect, for example, the production of non-standard (NOK) samples. For instance, applying too much pressure may cause a sample to break. Such insights can lead to more efficient manufacturing processes. Various aspects of the present invention can be applied to obtain interpretations, for example, to identify anomalies from images OQI510 of products produced in the manufacturing process MP, produced by an optical quality inspection system. Optionally, these insights can be fed back to the manufacturing process MP itself, thereby shutting down automated loops by incorporating, for example, knowledge of how non-standard samples are produced.

[0119] Therefore, the optical quality inspection image (OQI) can be fed into a trained classifier TC 530 to determine whether the resulting process belongs to the "normal" class of successfully manufactured products or the "abnormal" (NOK) class of unsuccessfully manufactured products. For example, based on the output of the trained classifier TC, the manufacturing process MP can be configured to further process the product or discard it.

[0120] Furthermore, if the product is not successfully manufactured, the classification interpretation CE 580 can be determined in interpretation operation Exp 590 as described herein. For example, the classification interpretation CE can be based on a filter suppression factor determined as described herein, such as the filter suppression factor and / or a metric derived from it.

[0121] The classification interpretation CE can be fed into another classifier FC 560 to classify the CE into a predefined set of possible anomalies. The classifier can be trained automatically, but may also include the application of one or more predefined criteria, such as including adapted classifier inputs that differ from the original optical quality inspection image OQI in a specific portion of the image. Classifying the CE can also involve applying a pre-trained model or performing clustering, for example, into a set of interpretations / anomalies generated by a predefined set of validation data samples.

[0122] In Manufacturing Control Operation MC 565, the classification of anomalies within this predefined set can be signaled back to control the Manufacturing Process (MP). For example, this signaling notification can be performed using known automation protocols. The manufacturing process can take various types of actions, including but not limited to marking products for manual inspection, manipulating the sorting system, operating the manufacturing process in a safe mode, or changing one or more control parameters (CP) of the Manufacturing Process (MP) to reflect the latest interpretation.

[0123] For example, adjustments can be performed according to a set of predetermined rules. For instance, if the difference between an optical quality inspection image II and a determined, adapted classifier instance indicates a small, localized defect known to be a possible side effect of a specific setting (e.g., high temperature, pressure, etc.) in the manufacturing process at a known stage of the processing line, the control parameter CP can be automatically adjusted to reduce the probability of that defect recurring in the future. A machine learning system can also be trained to learn optimal operating parameter settings using the correspondence between interpretations and operating parameter settings on multiple samples. For example, a reinforcement learning-based system can be used, which takes the classification interpretation CE and / or further classification output FCO as input and determines the action to be performed based on that input. A reinforcement-based system can also replace this additional classifier.

[0124] By using a set of predefined anomalies, the interpretation can be limited to a set of discrete possibilities, such as physical defects in a specific location on the sample, changes in control conditions such as lighting during the manufacturing process, etc. For example, such physical defects can be characterized by an interpretation that substantially highlights a specific area of ​​the image, while changes in lighting conditions may lead to an interpretation that substantially highlights the entire image.

[0125] Figure 6 A block diagram of a computer-implemented method 600 for determining a classification interpretation for a trained classifier is shown. The classification interpretation can be based on one or more classifier inputs classified into the same class by the trained classifier. Method 600 can correspond to... Figure 1 The operation of system 100. However, this is not a limitation, as another system, apparatus, or device may also be used to perform method 600.

[0126] Method 600 may include, in an operation entitled “Accessing Classifier, Generator,” accessing, 610, model data defining a trained classifier and model data defining a generative model. The generative model may be configured to generate classifier inputs for the trained classifier from generator inputs. The generative model may include multiple filters. The filters of the generative model may be configured to generate filter outputs at internal layers of the generative model.

[0127] Method 600 may include, in an operation entitled “Obtaining Generator Inputs”, obtaining 620 generator inputs corresponding to one or more classifier inputs. The generator inputs enable the generative model to approximately generate the corresponding classifier inputs.

[0128] Method 600 may include, in an operation entitled “Determining Filter Suppression Factors”, determining filter suppression factors for multiple filters of a generative model 630. The filter suppression factors of a filter can indicate the degree of suppression for the filter output of the filter. The filter suppression factors can be determined based on adapting the classifier input to the classification of a trained classifier according to the filter suppression factors.

[0129] Determining the 630 filter suppression factor can include: in the operation entitled "Adapting the classifier input according to the filter suppression factor", the generator model is applied to the generator input corresponding to the classifier input, and the filter output of the generator model's filter is adjusted according to one or more filter suppression factors, thereby adapting the 632 classifier input according to one or more filter suppression factors.

[0130] Determining the 630 filter suppression factor may further include: in the operation entitled “Applying the classifier to the adapted classifier input”, applying the trained classifier 634 to the adapted classifier input to obtain a classifier output affected by one or more filter suppression factors.

[0131] Method 600 may include, in an operation entitled "Determining Explanation", determining a 640-classification explanation based on a filter suppression factor. Method 600 may further include, in an operation entitled "Outputting Explanation", outputting a 650-classification explanation.

[0132] It should be understood that, generally speaking, Figure 6 The operation of method 600 can be performed in any suitable order (e.g., sequentially, simultaneously, or a combination thereof), which, where applicable, is subject to a particular order necessary, for example, by input / output relationships.

[0133] This method(s) can be implemented on a computer as a computer-implemented method, dedicated hardware, or a combination of both. For example... Figure 7 As illustrated, instructions for a computer (e.g., executable code) can be stored on a computer-readable medium 700. These instructions may be stored, for example, as a machine-readable physical tag sequence 710 and / or as a sequence of elements with different electrical, such as magnetic or optical, properties or values. The executable code can be stored in a temporary or non-temporary manner. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 7 An optical disc 700 is shown. Optionally, the computer-readable medium 700 may include temporary or non-temporary data 710, which represents decoder model data for predicting object feature vectors of multiple interacting physical objects as described herein; and / or encoder model data for classifying pairwise interactions between multiple physical objects into a set of multiple interaction types as described herein.

[0134] Examples, embodiments, or optional features, whether or not indicated as non-limiting, should not be construed as limiting the claimed invention.

[0135] It should be noted that the above embodiments are illustrative rather than limiting of the invention, and those skilled in the art will be able to devise many alternative embodiments without departing from the scope of the appended claims. Any reference numerals in parentheses in the claims should not be construed as limiting the claims. The use of the verb "comprising" and its variations does not exclude the presence of elements or stages other than those recited in the claims. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Expressions such as "at least one" preceding a list or group of elements indicate the selection of all elements or any subset of elements from that list or group. For example, "at least one of A, B, and C" should be understood to include only A, only B, only C, both A and B, both A and C, both B and C, or all A, B, and C. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In device claims enumerating several components, several of these components may be embodied by the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that combinations of these measures cannot be advantageously used.

Claims

1. A computer-implemented method (600) for determining a classification interpretation for a trained classifier, the classification interpretation being based on one or more classifier inputs classified into the same class by the trained classifier, the method comprising: - Access (610) defines the model data of the trained classifier and the model data of the generative model, which is configured to generate classifier input for the trained classifier from the generator input. The generative model includes multiple filters, which are configured to generate filter outputs at the inner layers of the generative model. - Obtain (620) generator inputs corresponding to one or more classifier inputs, which enable the generative model to approximately generate the corresponding classifier inputs; - Determine (630) filter suppression factors for multiple filters of the generative model, the filter suppression factors indicating the degree of suppression for the filter output of the filter, the filter suppression factors being determined based on the effect of the classifier input on the classification of the trained classifier according to the filter suppression factors, the determination including: - By applying the generative model to the generator input corresponding to the classifier input, and adjusting the filter output of the filter of the generative model according to one or more filter suppression factors, the (632) classifier input is adapted according to one or more filter suppression factors; - Apply the trained classifier (634) to the adapted classifier input to obtain a classifier output affected by one or more filter suppression factors; - Based on the filter suppression factor, determine the (640) classification explanation and output the (650) classification explanation. - The method (600) further includes determining a filter suppression factor by performing an optimization, the optimization being configured to: - For one or more classifier inputs affected by filter suppression factors, minimize the difference between the target classifier output and the affected classifier output of the trained classifier; - Minimize the overall suppression level indicated by the filter suppression factor. - The trained classifier is an image classifier, and the classifier input includes images of products produced during the manufacturing process.

2. The method (600) of claim 1 further includes classifying the classification interpretation into a predefined set of possible anomalies.

3. The method (600) according to claim 2, further comprising controlling the manufacturing process based on classification interpretation.

4. The method (600) of claim 1, further comprising accessing a discriminative model configured to determine the degree to which the output of the generative model appears to be synthetic, the optimization further configured to minimize the degree for one or more adapted classifier inputs.

5. The method (600) according to claim 1 or 4, further comprising obtaining a uniqueness score indicating the uniqueness of the respective filter, wherein minimizing the overall degree of suppression less strongly penalizes the suppression of the more unique filter compared to the suppression of the less unique filter.

6. The method (600) of claim 1, wherein determining the classification interpretation includes determining the difference between the classifier input and the corresponding adapted classifier input.

7. The method (600) according to claim 1, wherein the filter output of the filter of the generating model is adjusted by multiplying the elements of the filter output with the filter suppression factor according to the filter suppression factor.

8. The method (600) according to claim 1, comprising obtaining a classifier input and determining a generator input corresponding to the classifier input.

9. The method (600) of claim 1, comprising obtaining a class of a trained classifier, and generating one or more generator inputs, the one or more generator inputs causing a generative model to generate classifier inputs from the class.

10. The method (600) of claim 1, comprising outputting a classification explanation to the user in a sensory-perceptible manner.

11. The method (600) of claim 10, wherein at least the adapted classifier is input and output to the user, the method further comprising: The desired classification is obtained from the user with the adapted classifier input, and is used to retrain the trained classifier using the adapted classifier input and the desired classification.

12. A system (100) for determining a classification interpretation for a trained classifier, the classification interpretation being based on one or more classifier inputs classified into the same class by the trained classifier, the system comprising: - Data interface (120) for accessing model data (030) defining a trained classifier and model data (040) defining a generative model, the generative model being configured to generate classifier input for the trained classifier from the generator input, the generative model including multiple filters, the filters of the generative model being configured to generate filter outputs at the inner layers of the generative model; - Processor subsystem (140), which is configured to: - Obtain generator inputs corresponding to one or more classifier inputs, such that the generative model approximately generates the corresponding classifier inputs; - Determine the filter suppression factors for multiple filters of the generative model. The filter suppression factor indicates the degree of suppression for the filter output. The filter suppression factor is determined based on the effect of the classifier input on the classification of the trained classifier, and the determination includes: - By applying the generative model to the generator input corresponding to the classifier input, and adjusting the filter output of the generative model's filter according to one or more filter suppression factors, the classifier input is adapted according to one or more filter suppression factors. - Apply the trained classifier to the adapted classifier input to obtain a classifier output that is affected by one or more filter suppression factors; - The classification interpretation is determined based on the filter suppression factor, and the classification interpretation is output. - The processor subsystem (140) is further configured to determine the filter suppression factor by performing optimization, the optimization being configured to: - For one or more classifier inputs affected by filter suppression factors, minimize the difference between the target classifier output and the affected classifier output of the trained classifier; - Minimize the overall suppression level indicated by the filter suppression factor. - The trained classifier is an image classifier, and the classifier input includes images of products produced during the manufacturing process.

13. A computer-readable medium (700) comprising temporary or non-temporary data (710) representing instructions that, when executed by a processor system, cause the processor system to perform the method according to any one of claims 1 to 11.

Images