Improved method and system for generating concept-based interpretations for target model

Abstract

Improved methods and systems are provided for generating concept-based interpretations for a target model. Some embodiments relate to calculating metrics that quantify effects of concepts defined by a conceptual model on classification of a target model. This may include calculating an output of a similarity function applied to an output of the target model and an output of the conceptual model for sensor data in the evaluation set.

Description

Technical Field

[0001] The presently disclosed subject matter relates to a method for generating a concept-based explanation for a target model, a system for generating a concept-based explanation for a target model, and a computer storage medium. Background Art

[0002] In the field of machine learning (ML), there is an increasing emphasis on understanding the rationale behind the predictions made by ML systems, especially since they play an important role in all aspects of our lives. A significant challenge is the opaque or black-box nature of many ML models, which makes their decision-making process invisible to the end user. To address this opacity, the field of explainable AI (XAI) has emerged, dedicated to creating methods to elucidate the mechanisms and decisions of these models.

[0003] Post hoc explanations can provide valuable insights into how a model works without compromising its accuracy. These explanations provide additional information such as feature attributes that assess each feature’s contribution to the prediction. In addition to feature attribution explanations, there are example-based methods such as counterfactual explanations and influential examples. Counterfactual explanations highlight what changes must be made to the inputs to change the model’s output. Influential examples reveal which examples are most important to the model’s decision-making process. Overall, these post hoc explanations can help provide a better understanding of the inner workings of a model.

[0004] One prominent type of explanation is conceptual explanation, which assesses the importance of a human-understandable concept on the prediction. Conceptual explanation is a form of post hoc explanation that focuses on determining the impact of concepts on the model’s predictions. These concepts are usually human-understandable properties that can be represented as vectors or scalars. Different studies have proposed methods to represent concepts in different ways and return the impact of concepts on neural network models. For example, when predicting the presence of “zebra” in an image, the model may be influenced by recognizable concepts such as “stripes”.

[0005] Several methods have been developed to capture the importance of concepts relevant to prediction. However, several challenges remain. Existing methods are often model-specific and require white-box access. This can be a limitation, for example, in cases where the model is proprietary and not freely accessible for inspection. Furthermore, ambiguity remains regarding the interpretation, evaluation criteria, and uniqueness of the various methods developed for concept explanation. Summary of the invention

[0006] Improved methods with quantified conceptual impact would be advantageous.

[0007] Most existing concept explanation methods are designed for specific models. In addressing this limitation, embodiments are provided that achieve model-agnostic metrics. We introduce an approach to concept explanation that follows three natural axioms: separability, recursion, and similarity. We show experimentally that the approach is effective in identifying sufficient and necessary concepts and is applicable to a wide variety of model types. The approach is also significantly more efficient in execution than traditional methods. While dealing with the associated challenges, we develop an axiomatic approach to measuring the importance of concepts with respect to predictions. This is initiated by setting certain axioms and subsequently formulating a method that is consistent with these axioms. Starting from these fundamental principles ensures that the approach remains robust and transparent.

[0008] Some embodiments are directed to computing a metric that quantifies the influence of concepts defined by a conceptual model on the classification of a target model. This may include computing the output of a similarity function applied to the output of the target model and the output of the conceptual model for sensor data in an evaluation set.

[0009] Embodiments of the method may be implemented on a computer as a computer-implemented method, or in dedicated hardware, or in a combination of the two. The executable code of embodiments of the method may be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Preferably, the computer program product includes non-transitory program code stored on a computer-readable medium for performing embodiments of the method when the program product is executed on a computer.

[0010] In an embodiment, the computer program comprises computer program code adapted to perform all or part of the steps of an embodiment of the method when the computer program is run on a computer.Preferably, the computer program is embodied on a computer readable medium.

[0011] Another aspect of the presently disclosed subject matter is a method of making a computer program available for download. This aspect is used when a computer program is uploaded, and when a computer program is available for download. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Further details, aspects and embodiments will be described by way of example only with reference to the accompanying drawings. The elements in the drawings are illustrated for simplicity and clarity and are not necessarily drawn to scale. In the drawings, elements corresponding to elements already described may have the same reference numerals. In the drawings,

[0013] Figure 1 schematically illustrates an example of an embodiment of a system for computing a metric quantifying the impact of a concept,

[0014] Figure 2a and2b schematically illustrates an example of an embodiment of a system for computing a metric for quantifying the impact of a concept,

[0015] Figure 3a Schematically showing examples of metrics for quantifying concept impact,

[0016] Figure 3b Schematically showing examples of metrics for quantifying concept impact,

[0017] Figure 4 schematically illustrates an example of an embodiment of a method for calculating a metric for quantifying the impact of a concept,

[0018] Figure 5a schematically shows a computer-readable medium having a writable portion comprising a computer program according to an embodiment,

[0019] Figure 5b A representation of a processor system according to an embodiment is schematically shown.

[0020] Reference numerals list

[0021] The following list of references and abbreviations corresponds to Figure 1-2b , and are provided to facilitate the explanation of the drawings and should not be construed to limit the claims.

[0022] 110 A system for computing metrics that quantify the impact of concepts

[0023] 111 processor system

[0024] 112 Storage Devices

[0025] 113 Communication Interface

[0026] Evaluation set of more than 210 sensor data

[0027] 211 Sensor data

[0028] 220 Target Model

[0029] 230 Conceptual Model

[0030] 221 Target Model Output

[0031] 231 Conceptual Model Output

[0032] More than 240 similarity functions

[0033] 241 Similarity function

[0034] 251-253 Similarity function output

[0035] 260 Sum function

[0036] 271 Sum output

[0037] 1000, 1001 Computer readable medium

[0038] 1010 Writable part

[0039] 1020 Computer Programs

[0040] 1110 Integrated Circuit

[0041] 1120 Processing Unit

[0042] 1122 Memory

[0043] 1124 ASIC

[0044] 1126 Communication Components

[0045] 1130 Interconnect

[0046] 1140 Processor System DETAILED DESCRIPTION

[0047] While the presently disclosed subject matter is susceptible of embodiments in many different forms, one or more specific embodiments are shown in the drawings and will be described in detail herein, it being understood that this disclosure is considered an example of the principles of the presently disclosed subject matter and is not intended to be limited to the specific embodiments shown and described.

[0048] Hereinafter, for ease of understanding, the elements of the embodiments are described in operation. However, it will be apparent that the various elements are arranged to perform the functions described as being performed by them.

[0049] Furthermore, the presently disclosed subject matter is not limited only to the embodiments, but also comprises all other combinations of features described herein or recited in mutually different dependent claims.

[0050] Various embodiments are presented that provide numerous advantages.

[0051] The metric calculated according to the embodiment is based on three axioms. The third axiom emphasizes the importance of the similarity between concepts and predictions in achieving a high value for the metric. By taking these axioms into account, a more accurate metric can be developed in assessing the impact of concepts on predictions. In addition, at least for some choices of similarity, a unique metric that satisfies all axioms is obtained.

[0052] Metrics according to embodiments may be model agnostic; no knowledge of the inner workings of the model is required. This makes embodiments different from traditional methods that require access to the gradients or hidden layers of the model. Lack of access to the inner workings of the model may prove particularly beneficial when the model is confidential and cannot be disclosed.

[0053] Conceptual explanations extend the notion of feature attributes, which primarily attribute importance to predictive features. However, within this subset of explanations, different attribution methods produce different results.

[0054] To validate the merits of the approach, experiments have been conducted to evaluate its performance in different scenarios. These demonstrate how the metric can provide conceptual explanations for various model classes.

[0055] Figure 1 An example of an embodiment of a system 110 for computing a metric quantifying the impact of a concept is schematically shown. The system 110 may include a processor system 111 , a storage device 112 and a communication interface 113 .

[0056] The system 110 may be provided with a target model and a concept model. The target model is the model that needs to be measured. The concept model defines the concepts. The system 110 is configured to calculate a value indicating the impact of the concept on the classification of the model. This can be used for many purposes.

[0057] In various embodiments of the communication interface 113, the communication interface can be selected from various alternatives. For example, the interface can be a network interface to a local area network or a wide area network (such as the Internet), a storage interface to an internal or external data storage device, an application program interface (API), etc.

[0058] The storage device 112 may be, for example, an electronic storage device, a magnetic storage device, etc. The storage device may include a local storage device, such as a local hard drive or an electronic memory. The storage device 112 may include non-local storage, such as cloud storage. In the latter case, the storage device 112 may include a storage interface to the non-local storage. The storage device may include multiple discrete sub-storage devices, which together constitute the storage device 112.

[0059] The storage device 112 may be a non-transitory storage device. For example, the storage device 112 may store data in the presence of power, such as a volatile memory device, such as a random access memory (RAM). For example, the storage device 112 may store data in the presence of power and in the absence of power, such as a non-volatile memory device, such as flash memory. The storage device 112 may include a volatile writable portion, such as RAM, and a non-volatile writable portion, such as flash memory. The storage device may include a non-volatile non-writable portion, such as ROM, such as a portion storing software.

[0060] For example, the storage device may include parameters defining a target model, a conceptual model, and an evaluation set of multiple sensor data that may be used to compute a metric.

[0061] The system 110 can communicate internally with other systems, other devices, external storage devices, input devices, output devices, and / or one or more sensors via a computer network. The computer network can be the Internet, an intranet, a LAN, a WLAN, a WAN, etc. The computer network can be the Internet. The system 110 includes a connection interface that is arranged to communicate within the system 100 or outside the system 100 as needed. For example, the connection interface may include a connector, such as a wired connector (e.g., an Ethernet connector, an optical connector, etc.), or a wireless connector (e.g., an antenna, such as a Wi-Fi, 4G or 5G antenna). The system 110 can be implemented on a single device, such as a server.

[0062] The communication interface 113 may be used to send or receive digital data, such as target and / or conceptual models, inputs (such as similarity function selections), and outputs, in particular the metrics.

[0063] The system 110 may have a user interface, which may include well-known elements, such as one or more buttons, a keyboard, a display, a touch screen, etc. The user interface may be arranged to accommodate user interaction to initiate calculation of a metric.

[0064] The execution of system 110 can be implemented in a processor system. System 110 may include functional units that implement various aspects of the embodiment. The functional units may be part of the processor system. For example, the functional units shown here may be implemented in whole or in part with computer instructions stored in the storage device of the device and executable by the processor system.

[0065] The processor system may include one or more processor circuits, such as a microprocessor, a CPU, a GPU, etc. The system 110 may include multiple processors. The processor circuit may be implemented in a distributed manner, such as as multiple sub-processor circuits. For example, the system 110 may use cloud computing.

[0066] Typically, system 110 includes one or more microprocessors that execute appropriate software stored in the device; for example, the software may have been downloaded and / or stored in corresponding memory, such as volatile memory such as RAM or non-volatile memory such as flash memory.

[0067] Instead of using software to implement the functionality, the system 110 may be implemented in whole or in part in programmable logic, for example as a field programmable gate array (FPGA). The device may be implemented in whole or in part as a so-called application specific integrated circuit (ASIC), for example an integrated circuit (IC) customized for its specific use. For example, the circuit may be implemented in CMOS, for example using a hardware description language such as Verilog, VHDL, etc. Specifically, the system 110 may include circuits for cryptographic processing and / or arithmetic processing, for example.

[0068] In a hybrid embodiment, the functional unit is implemented partly in hardware, e.g. as a coprocessor, e.g. an AI coprocessor, e.g. for neural network evaluation, and partly in software stored and executed on the device.

[0069] Figure 2a and 2b An example of an embodiment of a system for computing a metric quantifying the impact of a concept is schematically illustrated. Figure 2a shows the first part of the calculation, and Figure 2b The second part is shown. Figure 2a and Figure 2b The system can be Figure 1 The system 110 is shown as being implemented. Figure 2a and Figure 2b The system can be implemented on a computer.

[0070] The system computes metrics that quantify the influence of concepts defined by the conceptual model on the classification of the target model. Such metrics serve as concept-based explanations of the target model. Quantifying the influence of concepts enhances model interpretation by providing tangible metrics that reveal how specific human-interpretable concepts influence model decisions. This not only increases transparency by highlighting concepts that are important for prediction, but also helps identify and potentially correct biases or errors rooted in these concepts.

[0071] The system is provided with two models: a target model, sometimes abbreviated as h, and a concept model, sometimes abbreviated as c. Both models are configured to receive at least sensor data (x) as input and produce a classification as output; a target classification h(x) and a concept classification c(x), respectively. In an embodiment, multiple concept models are used to evaluate target models for multiple concepts.

[0072] The output of the target and conceptual models can take a variety of forms. For binary classification, a classifier typically outputs one of two categories, for example, determining whether an email is "spam" or "not spam." In multi-class or multi-label classification scenarios, a classifier can determine one of several different categories—such as labeling an image of fruit as "apple," "banana," or "cherry"—or can assign multiple labels to a single instance, such as labeling a movie as both "comedy" and "romance." In this case, the output of the model can be a vector indicating the relevance of each category. For example, if the elements in the vector can be interpreted as probability scores, the vector can sum to 1. However, this is not required, for example, if multiple categories or labels can be applied to the same input sensor data, the sum of the vector may not be 1.

[0073] Instead of vectors, the output of the goal and / or conceptual model may be a scalar value, e.g., ranging from -1 to 1 or from 0 to 1, or it may be a discrete value, such as -1 or 1, or 0 or 1. The ability to handle discrete output values ​​is particularly useful when evaluating proprietary goal models that discretize their output values, e.g., spam or not spam, without providing access to internal continuous scalar values ​​that may be available.

[0074] If the output of the target model or conceptual model is a vector, the calculation of the measure of quantifying the impact can be performed at the level of the vector, but can also be performed at the level of selected elements of the vector. For example, the target model can classify fruits and can produce vectors with elements corresponding to specific fruits. The conceptual model can classify colors and can produce vectors with elements corresponding to specific colors. This calculation can be performed at the vector level, thereby showing the impact of color on the classification of fruits, or it can be performed at the element level, such as showing the impact of yellow on the classification of bananas.

[0075] The target model may be opaque, or even a black-box model. An opaque model refers to a machine learning model whose internal workings or decision-making processes are difficult to explain or understand, usually due to their complexity. For example, neural networks, especially deep networks, are considered opaque models. In a black-box model, the internal processes are hidden from the user, making it impossible for the user to directly understand the decision-making rationale of the model. Interestingly, embodiments can be applied to both opaque and black-box target models.

[0076] Concept models can also be opaque and / or black-box. This is an advantage because it allows the use of off-the-shelf, pre-trained concept classifiers. The target model and / or concept model can be obtained in a trained state, but the target model and / or concept model can also be trained by the system. For example, one or more concept models can be specifically trained to evaluate target models for various concepts.

[0077] For example, the target model and / or the conceptual model may include any of a neural network, a support vector machine, a linear regression, a logistic regression, a random forest, etc. For example, both the target model and the conceptual model may include a neural network. For example, both the target model and the conceptual model may include a neural network image classifier.

[0078] The target model and the conceptual model are configured to receive the same type of sensor data as input. The sensor data is obtained from sensors, such as microphones, cameras, radars, etc.

[0079] Typically, sensor data is at least one-dimensional, consisting of multiple values.

[0080] For example, the sensor data may be one-dimensional (1D), eg, where the sensor data includes one or more of: audio clips, time series data from an instrument like a temperature or pressure monitor, or waveform data from an electromagnetic sensor.

[0081] For example, the sensor data can be two-dimensional (2D), e.g., where the sensor data includes one or more of: 2D spatial data from a tool such as a geographic mapping instrument, images captured by a camera operating in the visible or infrared spectrum, or a thermal image from a thermal camera.

[0082] For example, the sensor data may be three-dimensional (3D), eg, where the sensor data includes one or more of: 3D spatial data from an advanced scanner, a volumetric image from medical tomography, or a three-dimensional representation of a physical object.

[0083] For example, the sensor data may be higher dimensional than three dimensions, for example, where the sensor data includes one or more spatiotemporal data that capture spatial information over time, such as a stereoscopic movie clip showing a dynamic process.

[0084] Specifically, a 3D MRI or CT scan that evolves over time represents 4D data. For example, in cardiology, 4D MRI can be used to capture the volume (3D) changes of the heart over the cardiac cycle (time). Similarly, in oncology, periodic 3D scans can be used to track the growth or regression of tumors over time, thus generating 4D data. Another non-medical example is tracking a moving 3D object over time, such as a drone.

[0085] In particular, sensor data can include image data. For example, sensor data can be images obtained from a camera operating in the visible light or infrared spectrum, etc. Sensor data may be monocular, providing a single viewpoint. Or it can be multi-view: for example, a stereo image presents two perspectives of the same object, enhancing depth perception. More advanced setups may provide multi-view images, capturing different viewpoints, not just two. In addition, there are range-based forms of sensor data, such as range images or point clouds acquired from LIDAR devices, depth images from time-of-flight cameras, or even radar-based data that measures the distance and speed of objects. In some complex applications, sensor data may fuse multiple modalities, for example, combining traditional color images with depth images, or juxtaposing infrared images with point clouds obtained by LIDAR, to provide more comprehensive insights.

[0086] Sensor data may include sensor measurements from different sources, such as images and temperature measurements.

[0087] The metric that quantifies the impact of a concept is calculated using an evaluation set 210 of multiple sensor data, sometimes denoted X. The evaluation set includes the same type of sensor data as the target and the sensor data received by the concept model. For example, the metric can be expressed as a sum over the evaluation set.

[0088] In a typical embodiment, the evaluation set can be taken to be the same as the training set of multiple sensor data on which the target model is trained. However, it is possible to reduce the evaluation set to reduce the computational cost of the metric. For example, the evaluation set can be a strict subset of the training set, e.g., having 50% less sensor data than the training set. For example, in the case of an image classifier, the evaluation set may have less than 50% of the images in the training set.

[0089] One way to select the evaluation set from the training is to select it randomly from the training set. This way an unbiased subset is obtained, especially for larger evaluation sets. Another approach is controlled selection, aiming to obtain a distribution of the target classification in the evaluation set that is consistent with the distribution present in the training set. For example, stratified sampling can be used. In a stratified approach, the training set is divided into strata based on some criteria, in particular by the target classification of the target model. The evaluation set is then selected to represent all these strata. For example, a random selection can be made for each stratum, for example, proportional to the size of the stratum.

[0090] For example, if the classification output is a scalar value, layers can be defined by the bands of that scalar output to capture all data points x for which, for specific bounds a and b, a ≤ h(x) < b. If the output of the target model is a vector, layers can be established by setting bounds on two or more elements within that vector to ensure comprehensive sampling considering multiple dimensions of the data.

[0091] As will be discussed in more detail, the metrics affected by the quantization concept preferably satisfy certain conditions. For example, the separability requirement is evaluated separately for each example, meaning that the influence of one example is not enhanced or diminished by the specific values of different examples. Another preferred condition is recursiveness, which is further discussed below. It turns out that even when these and other conditions are met, there is still room for choice in how to compute the metrics. This remaining degree of freedom in defining the metrics can be represented by a similarity function, sometimes abbreviated as s.

[0092] Using a similarity function is advantageous because, on the one hand, it concentrates the remaining degree of freedom in defining the metrics in one place: the definition of the similarity function. Additionally, changing the similarity function allows for obtaining different views of the concept influence. In fact, in embodiments, the metrics are computed multiple times for the same concept model but using different similarity functions.

[0093] The similarity function receives as input the output of the target model of the sensor data and the output of the concept model of the sensor data. For example, given sensor data x, such as an image, then the similarity function s(h(x), c(x)) can be computed. Note that this applies to models with both scalar and vector outputs. Note: By selecting a specific element of a vector, a vector output can always be reduced to a scalar output. For example, the similarity function for two vectors can be the inner product, such as the dot product (possibly scaled). For example, the similarity function for two vectors can be the cosine similarity.

[0094] Although not strictly necessary, it is convenient to use a continuous similarity function. It is also convenient to use a bounded similarity function, for example, with bounds from 0 to 1, or from -1 to +1, etc. Another convenient condition is monotonicity. For example, the similarity function can be monotonic in any independent variable. For example, assuming both the target model and the concept model have scalar outputs, then the similarity function s(a, b) is monotonic in a in the following case: for any fixed b, if a1 ≤ a2, then we have s(a1, b) ≤ s(a2, b) or s(a1, b) ≥ s(a2, b). Similarly, the similarity function is preferably monotonic in b.

[0095] In an embodiment, the similarity function can have discontinuities, but is zero in cases where the similarity function is discontinuous.

[0096] The system provides a selection for a similarity function from a plurality of similarity functions 240. Figure 2a In the example of FIG. 2 , a selection function 241 is selected from a plurality of functions 240. By making different selections from the plurality of functions, different types of metrics can be calculated. The plurality of functions can be predetermined in the system, for example, some useful similarity functions are defined here. The plurality of functions can also or alternatively be user configurable.

[0097] To facilitate the selection of a similarity function, the system can provide various methods. On the one hand, a representation of multiple similarity functions can be presented on a display, allowing the user to visually select the function they prefer. In this context, the user can make a selection directly from the display. Alternatively, the system can execute a script in which the desired similarity function is specified, automating the selection process based on predefined criteria or configurations. This solution may be preferred for automated testing of target models for multiple concepts. In addition, the user can also select the similarity function through a computer interface. For example, the computer interface can be an API.

[0098] Various choices of similarity functions are possible. For example, suppose the target model output includes a value x indicating the target classification t , and the conceptual model output includes a value x indicating the concept classification c . For example, these can be scalars indicating how likely the classification of the corresponding model is applicable. Shown here are possible choices for similarity functions.

[0099] The similarity function can include the target model output value x t and the conceptual model output value (x c ) multiplied by (x t x c ). For example, we might have s(x t ,x c )=x t x c .

[0100] If the target model value is below the threshold a, the similarity function can be zero, and the similarity function is monotonic with respect to the concept model value. For example, we can have s(x t ,x c )=x c max(0,sign(x t -a)). If the concept model value is below the threshold, the similarity function can be zero, and the similarity function is monotonic with respect to the target model value. For example, we can have s(x t ,x c )=x t max(0,sign(xc -a)). In these examples, where the similarity function needs to consider model values ​​above a threshold, for x t and / or x c , sign(xa) can be replaced by sign(ax).

[0101] The threshold a can be selected relative to the distribution of the target model on the evaluation set. Preferably, a large amount of sensor data in the evaluation model has a target output exceeding a. For example, assuming the output is in the range of 0-1, a can be selected to be 0.8.

[0102] To calculate the metric, it is calculated for each sensor data in the evaluation set 210. Figure 2a , such a calculation is shown for sensor data 211. Both the target model 220 and the conceptual model 230 are applied to the sensor data 211. In this way, a target model output 221 and a conceptual model output 231 are obtained. Both outputs are then provided to a similarity function 241. For example, if the sensor data 211 is represented as x, then this can calculate s(h(x), c(x)).

[0103] Figure 2b shows how the calculation can proceed. Figure 2a The similarity function output, i.e., value, is obtained for each sensor data in the evaluation set 210. Figure 2b Similarity function outputs 251-253 are shown. Typically, there will be more than three similarity function outputs, depending on the size of the evaluation set 210. For example, as many as the size of the evaluation set 210. The number of similarity function outputs can range from at least 100, at least 1000 to at least 100,000, or even at least 1,000,000.

[0104] These similarity function outputs are summed in summation function 260 to obtain summation output 271. For example, summation function 260 can add the similarity function outputs and then possibly divide by the size of the evaluation set, for example, summation function 260 can calculate an average similarity function output. The metric can be derived from the summation function. For example, the metric can be equal to the sum or average. The sum of the metric can first be scaled, etc.

[0105] In an embodiment, a probability function is provided for the evaluation set. The probability function assigns values ​​from 0 to 1 to each element in the evaluation set, so that the sum is 1. The probability function can represent the weight of each element of the evaluation set, such as importance. For example, the probability function can indicate the frequency with which the elements of the evaluation set appear, for example, in practice or in a training set. If n is the size of the evaluation set, a constant function 1 / n can be used, possibly adjusted for the selected elements, for example, to correct for abnormal elements. Using the probabilities, the sum is a weighted sum weighted by the probability function. In this case, a weighted average can be obtained.

[0106] If we want to target the conceptual model c1,…,c k The target model output h(x) can be calculated once and cached. This reduces s(h(x),c i (x)), where 1≤i≤k.

[0107] Once the metric is calculated, it can be presented to a user, for example on a display. The metric can be used for a variety of purposes, some of which are discussed below.

[0108] As noted, if desired, the target model and / or the conceptual model can be trained on the system. Alternatively, for any of them, a pre-trained model can be obtained from an external source. Training can be performed on training data including a plurality of sensor data and a training set of corresponding classifications.

[0109] In an embodiment, the metric can be used to select a training method for a target model. In practice, there are various methods to train machine learning models. For example, one can choose between stochastic gradient descent (SGD) or Adams. For example, various choices can be made for meta-parameters that affect neural network training, such as learning rate, batch size, number of epochs, etc. Different choices of training methods will result in different models that behave differently, even if the differences are small and imperceptible. Calculating concept-based influence values ​​for differently trained models is one way to select models.

[0110] For example, a first target model and a second target model are trained according to at least two different training methods on the training data. Typically, the two target models are similar to the desired target model, but are smaller and / or trained on less training data. For example, if the desired target model is a neural network, the first and second target models can be smaller neural networks. The first and second target models can have the same architecture, but this is not required. For example, the first and second target models can be trained with SGD versus Adams, or with different learning rates, etc. For example, the first and second target models can have different architectures.

[0111] According to an embodiment, once training is complete, a first metric may be calculated for a first target model and a second metric may be calculated for a second target model. Typically, the same metric will be calculated, for example, using the same conceptual model and similarity function. In an embodiment, multiple metrics may be calculated, for example, using conceptual models and / or similarity functions. Of the first and second target models, one may be selected that has the most desired metric. The training method used for the selected target model may then be applied to a larger target model. For example, as explained below, a model may appear to have good performance, but may appear to base its decisions on irrelevant aspects of the image. Such a target model may have poor generalizability. This can be detected by looking at the metric.

[0112] In addition to training, concept-based influence measures can be applied in various situations.

[0113] For example, a metric may be used to analyze a target model for defects, such as for debugging it. As further explained below, it may be predicted whether the metric is high or low, depending on the situation. If a technical analyst of the target model finds that the metric has a value that is different from the expected value, this may indicate a defect in the model. For example, a system may be provided with multiple concept models, and multiple metrics of the target model may be calculated corresponding to the multiple concept models. As indicated above, outliers in the multiple metrics may indicate a defect in the target model. For example, outliers may be determined in the multiple metrics and reported to an analyst. For example, one or more concept models, such as the concept model corresponding to the highest value or the minimum value, may be selected depending on the multiple metrics and reported to the user.

[0114] Many machine learning models, such as neural networks, are notoriously difficult to debug, and having the ability to analyze model concepts provides an important new tool for neural network engineers.

[0115] The desired multiple metrics can be obtained from a variety of sources. For example, the desired multiple metrics can be obtained by defining which concepts should or should not be relevant to the target model, and the desired concept values ​​can be similarly defined. The desired multiple metrics can also be obtained by calculating the desired multiple metrics for a successful model (e.g., a much larger or better trained model), and the resulting metrics can be used as the desired metrics for another model that, for example, uses less or different training or a different architecture, etc.

[0116] With a list of desired multiple metrics and measured multiple metrics, the analyst can do a variety of things. For example, he / she can adjust the target model or the training data to enhance the alignment between the calculated multiple metrics and the desired multiple metrics. In particular, the measured multiple metrics can be used to identify deviations in the target model from the multiple metrics. For example, if the conceptual model is trained to classify images as male or female, and the target model is trained to detect diseases in medical images, such as lung cancer in x-ray images, then the male / female classification may preferably have a low impact on the classification. This is relevant because historically smokers are above average for males, so the above-average portion of the training material may actually correspond to males. However, determining a high probability of lung cancer based solely on gender may be undesirable.

[0117] For example, once such biases, or more generally undesirable conceptual influences, are discovered, they can be addressed by changing the model or training. For example, in the lung cancer example above, examples of female lung cancer patients or male non-lung cancer patients could be added, or could be given increased weight, such as through a probability function.

[0118] As another example of using metrics to quantify the impact of concepts on a target model, consider a target model that includes an image classifier for an autonomous vehicle. The image classifier can be configured to classify objects in a traffic scene. For example, the image classifier can identify objects, including other vehicles, such as cars, trucks, motorcycles, and bicycles. Traffic signals can also or alternatively be identified, such as one or more of the following: signs (such as stop signs), traffic lights, yield signs, road markings (such as lane dividers), zebra crossings, and arrows indicating directions. In addition, the image classifier can classify static objects, such as one or more of the following: lamp posts, bridges, billboards, buildings, etc.

[0119] Another application of image classifiers is to classify traffic conditions as safe for autonomous driving. For example, image classifiers can detect one or more of the following: insufficient lighting, unsuitable road conditions, unexpected scenes, etc.

[0120] In both cases, control signals for controlling the autonomous vehicle may be derived from the target model and used to control the autonomous vehicle, such as a car.

[0121] Concept models can be trained for concepts that are relevant to safety. For example, a concept model can be trained to classify lighting, or to classify road conditions. On the other hand, concept models can also be trained for concepts that are not relevant to safety, such as classifying billboard ads, vehicle brands, etc. Such concepts should have a low impact on the predictions of the target model.

[0122] Concept-based impact metrics can be used in a variety of scenarios both in and outside of training. They can be used as a tool to analyze the target model to identify potential flaws or anomalies. If a metric deviates from the expected value, it indicates a potential problem in the model. The system can be equipped with multiple concept models and metrics for the target model can be calculated relative to these concept models. Anomalies, such as outliers in these measurements, can flag flaws. In some cases, metrics from well-trained models can be used as a benchmark for other models. These metrics help analysts adjust the target model, training parameters, or training data, for example, to reduce bias or misalignment. For example, gender bias in medical image classification can be detrimental and should have minimal impact on the model's results. In the context of autonomous vehicles, safety-related concepts should have high impact, while unimportant concepts should have low impact. Selecting concepts can be left to technicians and / or determined empirically.

[0123] Embodiments include selecting a similarity function from a plurality of similarity functions. Another useful method is to predetermine the similarity function. For example, the similarity function can be predetermined based on a similarity function described herein or in the claims.

[0124] An example of such a method may include calculating a metric that quantifies the impact of concepts defined by a conceptual model on a classification of a target model, the method comprising: providing a target model, the target model being configured to receive at least sensor data as input and to produce a target classification as output; providing a conceptual model, the conceptual model being configured to receive at least sensor data as input and to produce a concept classification as output; providing an evaluation set of multiple sensor data; calculating the metric, the calculation comprising calculating, for each sensor data in the evaluation set, an output of the target model for the sensor data, an output of the conceptual model for the sensor data, and an output of a selected similarity function applied to the output of the target model and the output of the conceptual model, the similarity function being configured to receive as input an output of the target model for the sensor data and an output of the conceptual model for the sensor data; summing the outputs of the similarity functions and deriving the metric from the summation.

[0125] Several further optional improvements, details and embodiments are shown below. In addition, mathematical notations are introduced that help describe some embodiments.

[0126] We use to represent the set of input examples. The target model to be explained is represented by h: It indicates that it Takes an input and outputs +1 or -1 to indicate whether the prediction is true or false, respectively. The concept we are interested in measuring is represented by c: (or in the discrete case by c: ) indicates that it will input from is mapped to a value between -1 and +1, where c(x)=1 indicates that the concept holds, and c(x)=-1 indicates that it does not hold. The probability distribution of the example is given by p: It will input from Mapping to probability, ∑ x p(x) = 1. For targets and / or conceptual models that produce vectors, these formulas can be applied to one element of the output vector. Alternatively, the same approach can be applied, but using a similarity function that is configured to receive a vector rather than a scalar. Other ranges besides [-1, +1] can be used; furthermore, the specific examples given here can be mapped to another range if desired.

[0127] The goal is to identify a metric, denoted M, that quantifies the impact of concept c on the predictions of an opaque model h; for example, consider h as a predictor of the class "zebra" and c as the concept "stripes". This allows, for example, to gain a deeper understanding of the model h by leveraging human-understandable concepts. In an embodiment, the metric M is a function of the following parameters: [(i)] the model h, the concept c, and optionally a prior probability p(·) over the examples.

[0128] Axiom 1: Divisibility

[0129] When measuring the impact of a concept, the separability axiom requires that each example be evaluated separately, which means that the impact of one example is not enhanced or weakened by the specific values ​​of different examples. In other words, the impact of an example is considered in isolation and is not affected by other examples. From M, one might expect it to be analytic, i.e. infinitely differentiable everywhere, which directly implies that M is additive: If M satisfies the separability axiom and is analytic, then M is additive, i.e. it can be written as:

[0130]

[0131] This is for some local influence function m:{-1,+1}×[-1,1]×[0,1].

[0132] Axiom 2: Recursion

[0133] The second axiom delves deeper into the local metric m and how it relates the probability example p(x) to the properties of the examples h(x) and c(x). This axiom is specifically aimed at studying how m changes with respect to p(x).

[0134] The recursiveness axiom requires that when the probability p(x) of an example is split between two new examples with the same values ​​of h(x) and c(x), the value of M remains unchanged. In other words, the axiom requires that replacing example x with two examples with probabilities p1 and p2 such that p1+p2=p(x) and the same values ​​of h(x) and c(x) (keeping all other examples unchanged) does not change the value of M.

[0135] From the last two axioms, we deduce that M is additive and linear in function p: If M satisfies the separability and recursion axioms, M is analytic, and p is in the range of rational numbers, then M can be written as

[0136]

[0137] This is true for some function s:{-1,+1}×[-1,+1].

[0138] This axiom guarantees the linearity of the probabilities. If no explicit probability function is used, p(x) can be replaced by a constant 1 / n, where n is size.

[0139] Axiom 3: Similarity

[0140] In the third axiom, the term s(h(x),c(x)) is treated as focusing on a single example with some probability; for s, the exact probability is irrelevant. It aims to determine the contribution of h(x) and c(x) to the overall value of M.

[0141] The similarity axiom asserts that for each example x, M will be high when the values ​​of c(x) and h(x) are close, and will be low when their values ​​are far. To satisfy this axiom, any similarity function can be used. When M is viewed as a function of only c(x) and h(x), a natural choice is to take s(v(x), c(x)) = c(x) h(x). The resulting metric is as follows.

[0142] Symmetric metric: For predictor h: Concept c: and probability p on examples, the symmetric measure of concept influence is defined as:

[0143] Mathematically, it can be shown that if all the axioms hold, then M must be a symmetric metric. A symmetric metric is a unique metric that satisfies the axioms of separability, recursion, and similarity to s(h(x),c(x)) = c(x) h(x), and is analytic.

[0144] There are various other similarity functions that can be considered. For example, one might want to focus on a particular class, such as class +1. In this case, examples with different classes will have no effect on the metric, and for related examples, we define s(h(x),c(x))=c(x)1 h(x)＝1 After normalization to obtain the expected value, the resulting metric M is as follows.

[0145] Class-conditional metrics: For predictor h: Concept c: and the probability p on the example, the class-conditional measure of concept influence is defined as

[0146] This metric quantifies how often concept c appears in a class, for example, if c always appears when h(x) = 1, then the value of M will be high. In other words, the class conditional metric assesses the level of necessity of c for the prediction of h.

[0147] Another possible similarity function s to consider is one that focuses on a particular concept and ignores all examples for which c does not satisfy a threshold θ. In particular, we define s(h(x),c(x))=h(x)1 c(x)≥θ The resulting metric M is as follows.

[0148] Concept conditional metrics: For predictor h: Concept c: With a threshold θ∈[-1,+1] and probability p over examples, the concept-conditional measure of concept influence is defined as

[0149] When the concept holds (i.e., exceeds a threshold), the metric quantifies the probability of the class. For example, if the class is 1 whenever c is sufficiently present (i.e., c(x) ≥ θ), then M will be high. In other words, the concept conditional metric assesses whether c is sufficient to predict h.

[0150] Various other similarity functions are possible. In some embodiments, it is preferred to use a bounded similarity function, e.g., having the same range as the output of the model, e.g., in this case the range is [-1, +1], and / or being monotonic when all but one input is held fixed. Other examples of similarity functions s(x, y) include 1-|xy| or The similarity function is preferably continuous, but this is not required; for example, the similarity function may be discontinuous when conditional on a particular concept being present or being above a threshold.

[0151] Note that some traditional concept metrics, such as TCAV, see B. Kim et al., "Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV)" (incorporated herein by reference), require training the model for each target and concept model combination considered. Embodiments are much faster than this approach. And embodiments are at least as accurate as TCAV.

[0152] experiment

[0153] Experiments are conducted to demonstrate the effectiveness and applicability of the embodiments; some of these embodiments are described below. We show that our metric captures the relationship between predictors and concepts. The proposed method is then applied to a wide variety of models, regardless of their inner workings.

[0154] As a proof of concept, this section verifies the meaning of the metric through two scenarios. In the first scenario, consider a classifier h that tries to predict the fine-grained category of the cat family, for example, whether a given image x is a Persian cat (labeled as 1) or not (labeled as 0). Note that the range used for classification in this section is [0, 1] rather than [-1, 1]. Then the condition that h(x) = 1 should expect the concept "the animal in the image belongs to the cat family" to have a very high score. That is, In other words, for any necessary concept, it is expected that for any reasonable classifier, there will be a large value of On the other hand, if one wants to understand the impact of the concept “the animal in the image is a wolf”, the aforementioned score should be low, since this concept is largely irrelevant to the prediction of Persian cat.

[0155] In the second scenario, imagine that h simply predicts whether x is a feline (label 1) or not (label 0), and then conditional on the concept that the animal in the image is a Persian cat, then the expected That is, for any sufficient concept, for any reasonable classifier there is a large value

[0156] According to experience, Figure 3a and 3bAs can be shown in , a ResNet-18 [7] pre-trained on ImageNet [3] exhibits these expected behaviors given the necessary and sufficient concepts. Specifically, among the 1000 categories in ImageNet, consider the categories under the "cat" level, which include category ids ranging from 281 to 293. When predicting coarser categories, for example, when trying to predict whether an image contains a cat, rather than identifying fine-grained types of cats, for simplicity, we set h(x) = 1 whenever the ResNet outputs any index in {281, ..., 293}.

[0157] Figure 3a An example of a metric for quantifying the impact of a concept is shown schematically. Where h(x) predicts a fine-grained cat class, e.g., Persian cat or cheetah. At 302, the value of the concept c(x) is shown as the cat concept. At 301, the value of the concept c(x) is shown as the wolf concept. That is, c(x) is defined by the image classifiers for cats and wolves, respectively. Note that the values ​​at 302 are all positive and close to 1, while the values ​​at 301 are all negative and close to -1. This indicates the accuracy of the metric. The target model h(x) classifies the following animals:

[0158] 311 Cheetah

[0159] 312 Tiger

[0160] 313 Lion

[0161] 314 Jaguar

[0162] 315 Snow Leopard

[0163] 316 Leopard

[0164] 317 Lynx

[0165] 318 Cougar

[0166] 319 Egyptian cat

[0167] 320 Siamese cat

[0168] 321 Persian cat

[0169] 322 Tiger Cat

[0170] 323 Tabby Cat

[0171] Figure 3b An example of a metric for quantifying the impact of a concept is shown schematically. Where h(x)=1 means that the image is a wolf or a feline; at 301, the value when h(x)=1 predicts a wolf is shown, and at 302, the value when h(x)=1 predicts a feline is shown. The concept model c(x) classifies the fine-grained feline concept; in particular, labels 311-323 represent the same category except for c(x).

[0172] For example, in row 311, the concept model classifies cheetahs, while the target model classifies wolves (at 301) or felines (at 302). Note again that at 302, all values ​​are positive and most are close to 1, while at 301, all values ​​are negative and close to -1.

[0173] Further experiments are explored to demonstrate the applicability of the newly designed metric. The experiments are conducted on the a-Pascal dataset, see “Describing objects by their attributes” by A. Farhadi et al. This dataset contains 6340 training images and 6355 test images, unevenly distributed over 20 categories. Each image is associated with a {0,1} 64 , with concepts such as “wings” and “shiny”. The real concepts are labeled by human experts, and it is later shown that current image description generators (captioners) can also achieve reliable performance on labeled concepts. Unless otherwise stated, we will use human-labeled concepts in our experiments.

[0174] An advantage of the embodiments is that they can work with any model. This flexibility enables one to apply the metrics to both logistic regression (LR) and random forest (RF) models, making it different from traditional methods that only apply to neural networks. In the experiments, these metrics were used to compare LR and RF models. The LR classifier was used Regularizer training. The RF classifier has 100 trees and the maximum depth of each tree is 15. Both models achieve 87% accuracy. The metrics and Although LR and RF achieve almost the same accuracy, these metrics provide valuable insights into how the models differ. Specifically, when predicting the cat class, RF places more weight on the concepts of "tail", "trunk", and "legs", while LR places more weight on "eyes". When predicting the chair class, RF is more sensitive to the concept of "leather" than LR, which is inconsistent with the true data distribution.

[0175] The examples also provide a convenient way to study the behavior of different optimizers. While the traditional view is that minima found by SGD generalize better, and local minima found by ADAM are faster, we are able to show which concepts in the image are picked up by each optimizer, which can lead to explanations for differences in optimizer behavior and generalization capabilities.

[0176] We use a ResNet-18 pre-trained on ImageNet and fine-tune with the following optimizers (unspecified hyperparameters are set to PyTorch defaults): [(a)] SGD, learning rate 1e-3 for the last linear layer, 1e-9 for other layers, momentum 0.9, weight decay 5e-4. AdamW, learning rate 1e-3 for the last linear layer, 1e-9 for other layers. Both models are trained for 100 epochs, and the SGD model achieves a test accuracy of 58.46%, and the AdamW model achieves 60.26%. Metrics The classifications for various concepts and objects were computed. Although the models had similar accuracy, various differences between the two models were found. For example, when h(x) predicted the "motorcycle" category, it was found that AdamW over-predicted the motorcycle category, gave too much weight to several irrelevant concepts (such as "legs"), and thus gave too little weight to relevant concepts (such as "engine"). This is contrary to both SGD and the ground truth. Therefore, training with AdamW can be improved by increasing the weights of some training examples or adding specific training examples to reduce the importance of legs for identifying motorcycles.

[0177] Recent developments in visual-language models like CLIP have achieved zero-shot classification performance through cue engineering. See “Learning transferable visual models from natural language supervision” by A. Radford et al. These models have an image encoder f(x): And the text encoder g(t): These encoders are contrastively trained to align matching f(x) and g(t), where text t is the true caption of image x. During inference time, given a set z of categories, for each z∈z, we can construct a category hint “{t z} photos", and for an input image x, we predict In this experiment, we use the CLIP model with the ViT-B16 visual hub on the a-Pascal dataset and achieve a zero-shot test accuracy of 78.2% and an F1 score of 0.6796. Various metrics can be computed to study the model's predictions. For example, the metric E[c(x)|h(x)=1] can be computed, where h(x)=1 predicts the class "bottle". We identify 10 class cues that incorrectly overweight 13 irrelevant concepts. For example, when conditioned on the original CLIP cues to predict "bottle", too many predicted images contain irrelevant concepts like "head", "ears", and "torso".

[0178] To combat the false emphasis, the original category cues for the 10 categories can be edited by subtracting irrelevant concepts. In particular, consider a concept w that is irrelevant to category z, and use g(t z )-λ·g(w) as the new hint embedding for classifying category z, where λ<1 is a scalar that prevents us from over-subtraction. Heuristically, we choose λ=0.1 based on trial and error using the training data, however, λ can certainly be learned in a few-shot fashion. See the Appendix for details. The edited hint improves the F1 score to 0.7042. In addition, the edited hint shows better alignment.

[0179] Figure 4 An example of an embodiment of a method 400 for calculating a metric for quantifying the impact of a concept is schematically shown. The method 400 is a computer-implemented method (400) and can be used to generate a concept-based explanation for a target model by calculating a metric for quantifying the impact of concepts defined by the concept model on the classification of the target model. The method 400 comprises:

[0180] - providing (410) a target model (h), the target model being configured to receive at least sensor data (x) as input and to produce a target classification (h(x)) as output,

[0181] - providing (420) a concept model (c), the concept model being configured to receive at least sensor data (x) as input and to produce a concept classification (c(x)) as output,

[0182] - providing (430) an evaluation set (X) of a plurality of sensor data,

[0183] - selecting (440) a similarity function(s) from a plurality of similarity functions, wherein each similarity function of the plurality of similarity functions is configured to receive as input an output of the target model for the sensor data and an output of the conceptual model for the sensor data,

[0184] - calculating (450) the metric, the calculating comprising:

[0185] - for each sensor data in the evaluation set, calculating (460) the output of the target model for the sensor data, the output of the conceptual model for said sensor data, and the output of the selected similarity function (s(h(x), c(x))) applied to the output of the target model and the output of the conceptual model,

[0186] - summing (470) the outputs of the similarity functions and deriving the metric from the sum.

[0187] The target model and / or the conceptual model can be defined by a plurality of parameters, for example, at least 10, 100, 10000 or 100000, etc. The target model and / or the conceptual model can include a neural network. For example, the neural network can have at least 2, 5, 10, 15, 20 or 40 hidden layers, or more hidden layers, etc. The number of neurons in the neural network can be, for example, at least 10, 100, 1000, 10000, 100000, 100000 or more, etc.

[0188] It will be apparent to those skilled in the art that many different ways of performing the method are possible. For example, the order of steps may be performed in the order shown, but the order of steps may vary, or some steps may be performed in parallel. In addition, other method steps may be inserted between steps. The inserted steps may represent improvements to the method described herein, or may be unrelated to the method. For example, some steps may be performed at least partially in parallel. In addition, a given step may not be fully completed before the next step begins.

[0189] Embodiments of the method may be performed using software that includes instructions for causing a processor system to perform method 400. The software may include only those steps taken by a specific sub-entity of the system. The software may be stored in a suitable storage medium, such as a hard disk, a floppy disk, a memory, an optical disk, etc. The software may be sent as a signal along a wired or wireless or using a data network (e.g., the Internet). The software may be used for downloading and / or remote use on a server. Embodiments of the method may be performed using a bitstream that is arranged to configure a programmable logic, such as a field programmable gate array (FPGA), to perform the method.

[0190] It will be understood that the subject matter disclosed herein also extends to computer programs, particularly computer programs on or in a carrier, suitable for putting the subject matter disclosed herein into practice. The program can be in the form of source code, object code, code intermediate source and object code (e.g., partially compiled form), or any other form suitable for implementing a method embodiment. An embodiment relating to a computer program product includes computer executable instructions corresponding to each processing step of at least one method described. These instructions can be subdivided into subroutines and / or stored in one or more files that can be statically or dynamically linked. Another embodiment relating to a computer program product includes computer executable instructions corresponding to each device, unit and / or part of at least one system and / or product described herein.

[0191] Figure 5a A computer readable medium 1000 having a writable portion 1010 is shown, as well as a computer readable medium 1001 also having a writable portion. The computer readable medium 1000 is shown in the form of an optically readable medium. The computer readable medium 1001 is shown in the form of an electronic memory, in this case a memory card. The computer readable media 1000 and 1001 can store data 1020, wherein the data can indicate instructions, which, when executed by a processor system, cause the processor system to perform an embodiment of a method according to an embodiment. The computer program 1020 can be embodied on the computer readable medium 1000 as a physical mark or by magnetization of the computer readable medium 1000. However, any other suitable embodiment is also conceivable. In addition, it will be understood that although the computer readable medium 1000 is shown here as an optical disk, the computer readable medium 1000 can be any suitable computer readable medium, such as a hard disk, a solid state memory, a flash memory, etc., and can be non-recordable or recordable. The computer program 1020 includes instructions for causing the processor system to perform the method for generating concept-based interpretations.

[0192] Figure 5b A schematic representation of a processor system 1140 is shown according to an embodiment of a system or apparatus for generating concept-based interpretations. The processor system includes one or more integrated circuits 1110. Figure 5bThe architecture of one or more integrated circuits 1110 is schematically shown. The circuit 1110 includes a processing unit 1120, such as a CPU, for running a computer program component to perform a method according to an embodiment and / or implement its module or unit. The circuit 1110 includes a memory 1122 for storing programming code, data, etc. A portion of the memory 1122 may be read-only. The circuit 1110 may include a communication element 1126, such as an antenna, a connector, or both, and the like. The circuit 1110 may include a dedicated integrated circuit 1124 for performing some or all of the processing defined in the method. The processor 1120, the memory 1122, the dedicated IC 1124, and the communication element 1126 may be interconnected by an interconnect 1130 (such as a bus). The processor system 1110 may be arranged to communicate in contact and / or contactless using an antenna and / or a connector, respectively.

[0193] For example, in an embodiment, the processor system 1140 (e.g., an interpretation generating device) may include a processor circuit and a memory circuit, and the processor is arranged to execute software stored in the memory circuit. For example, the processor circuit may be an Intel Core i7 processor, an ARM Cortex-R8, etc. The memory circuit may be a ROM circuit or a non-volatile memory, such as a flash memory. The memory circuit may be a volatile memory, such as an SRAM memory. In the latter case, the device may include a non-volatile software interface, such as a hard drive, a network interface, etc., arranged to provide software.

[0194] Furthermore, the memories 1122 may all be considered “non-transitory machine-readable media.” As used herein, the term “non-transitory” will be understood to exclude transitory signals, but rather include all forms of storage, including both volatile and non-volatile memory.

[0195] Although system 1140 is shown as including one of each described component, in various embodiments, various components may be repeated. For example, the processor may include multiple microprocessors that are configured to independently perform the methods described herein, or are configured to perform the steps or subroutines of the methods described herein, so that multiple processors collaborate to implement the functions described herein. In addition, in the case where system 1140 is implemented in a cloud computing system, various hardware components may belong to separate physical systems. For example, the processor may include a first processor in a first server and a second processor in a second server.

[0196] It should be noted that the above-mentioned embodiments illustrate rather than limit the presently disclosed subject matter, and that those skilled in the art will be able to design many alternative embodiments.

[0197] In the claims, any reference numerals placed between brackets shall not be interpreted as limiting the claims. The use of the verb "comprise" and its variations does not exclude the presence of elements or steps other than those described in the claims. The article "one" or "an" in front of an element does not exclude the presence of multiple such elements. Expressions such as "at least one of..." when in front of a list of elements indicate that all elements or any subset of elements are selected from the list. For example, the expression "at least one of A, B, and C" should be understood to include only A, only B, only C, both A and B, both A and C, both B and C, or all of A, B, and C. The subject matter currently disclosed can be implemented by hardware including several different elements and by a properly programmed computer. In a device claim that lists several parts, several of these parts can be embodied by the same hardware. The mere fact that certain measures are recorded in mutually different dependent claims does not indicate that the combination of these measures cannot be used advantageously.

[0198] In the claims, references in parentheses refer to reference signs in the drawings of exemplary embodiments or formulas of embodiments, thereby increasing the intelligibility of the claims. These references should not be construed as limiting the claims.

Claims

1. A computer-implemented method (400) for generating concept-based explanations for a target model, comprising computing a metric quantifying the influence of concepts defined by a concept model on a classification of the target model, the method comprising: - providing (410) a target model (h), the target model being configured to receive at least sensor data (x) as input and to produce a target classification (h(x)) as output, - providing (420) a concept model (c), the concept model being configured to receive at least sensor data (x) as input and to produce a concept classification (c(x)) as output, - providing (430) an evaluation set (X) of a plurality of sensor data, - selecting (440) a similarity function(s) from a plurality of similarity functions, wherein each similarity function of the plurality of similarity functions is configured to receive as input an output of the target model for the sensor data and an output of the conceptual model for the sensor data, - calculating (450) the metric, the calculating comprising: - for each sensor data in the evaluation set, calculating (460) the output of the target model for the sensor data, the output of the conceptual model for said sensor data, and the output of the selected similarity function (s(h(x), c(x))) applied to the output of the target model and the output of the conceptual model, - summing (470) the outputs of the similarity functions and deriving the metric from the sum.

2. The method of claim 1, wherein the target model is an opaque and / or black box.

3. A method according to any one of the preceding claims, comprising: - A probability function is provided for the evaluation set, where the sum is a weighted sum weighted by the probability function.

4. A method according to any one of the preceding claims, wherein the target model output comprises a value (x t ), the conceptual model output includes values ​​indicating the concept classification (x c ), and where -The similarity function includes the target model value (x t ) and the conceptual model value (x c ) multiplied by (x t x c ), and / or - if the target model value is below a threshold, the similarity function is zero, and the similarity function is monotonic with respect to the concept model value, and / or - If the concept model value is below the threshold, the similarity function is zero, and the similarity function is monotonic with respect to the target model value.

5. A method according to any one of the preceding claims, wherein the sensor data received by the object model and the conceptual model is dimensional, the dimensionality being: - one-dimensional (1D), e.g., where the sensor data includes one or more of audio clips and time series data; or - two-dimensional (2D), for example, where the sensor data comprises one or more of 2D spatial data, images; or - three-dimensional (3D), e.g., where the sensor data includes one or more of 3D spatial data and a volumetric image; - Higher dimensional than three dimensions, for example where the sensor data comprises one or more of spatiotemporal data, stereoscopic movie clips.

6. A method according to any one of the preceding claims, comprising: - training a target model on training data, the training data comprising a plurality of sensor data and a training set of corresponding classifications, wherein the evaluation set is a subset of the training set and / or wherein the training set is equal to the evaluation set, and / or -Training a concept model on concept training data, where the concept training data includes a plurality of sensor data and a concept training set corresponding to concept classification.

7. The method of any one of the preceding claims, comprising training the target model on training data, the training data comprising a plurality of sensor data and a training set of corresponding classifications, wherein the evaluation set is a subset of the training set, and wherein the evaluation set is selected based on one of the following criteria: - randomly selected from the training set, - controlled selection to obtain a distribution of target classes produced by the target model on the evaluation set that is consistent with the distribution of target classes when applied to the training set, - Stratified sampling method, where the training set is divided into strata according to at least the target classification of the target model, and the evaluation set is selected to represent all strata in the training set.

8. A training method for training a target model, - provide at least two different training methods for the target model, - training the first and second target models on training data according to at least two different training methods, the training data comprising a plurality of sensor data and a training set of corresponding classifications, the target model and the first and second target models being configured to receive at least the sensor data as input and to generate a target classification (h(x)) as output, - a method according to any of the preceding claims, determining a first metric for a first target model and a second metric for a second target model, - selecting a training method from at least two different training methods depending on the first and second metrics, - Train the target model using the selected training method.

9. A method according to any one of the preceding claims, comprising: - Determining defects in the target model by comparing the metric to a threshold.

10. A method according to any one of the preceding claims, comprising: - providing a plurality of conceptual models, and calculating a plurality of metrics for a target model corresponding to the plurality of conceptual models, wherein the method further comprises one or more of the following operations: - select one or more conceptual models from a plurality of conceptual models depending on a plurality of measures and report the selected conceptual models, and - adjusting the target model to enhance the alignment between the computed plurality of metrics and the desired plurality of metrics, - adjusting the training data to enhance the alignment between the computed multiple metrics and the desired multiple metrics, and retraining or improving the target model, - Identify biases in target models from multiple metrics.

11. A method according to any one of the preceding claims, wherein the target model comprises an image classifier for the autonomous vehicle, the image classifier being configured to classify objects in a traffic scene and / or to classify a traffic situation as safe for autonomous driving, the method comprising: -Evaluate the safety of the target model by comparing the metric against the target model with a threshold.

12. The method of any one of the preceding claims, wherein selecting a similarity function comprises: - presenting a representation of a plurality of similarity functions on a display and receiving a selection from a user, and / or - execute a script in which a similarity function is selected, and / or - Receiving, at the computer interface, a selection of a similarity function.

13. A method according to any one of the preceding claims, wherein -Multiple similarity functions are monotonic in the output of the target model and monotonic in the output of the conceptual model.

14. A system comprising: one or more processors; and one or more storage devices storing instructions which, when executed by one or more processors, cause the one or more processors to perform the operations of any one of the preceding claims.

15. A transitory or non-transitory computer storage medium encoded with instructions which, when executed by one or more computers, cause the one or more computers to perform the method according to any one of claims 1-13.

Images