reliability calibrated explanation of predictions of ai-based operational models
By calibrating industrial AI models with perturbation datasets, calibrated models are generated to improve the reliability of interpretation results. This solves the problem of insufficient transparency and credibility of black-box models in industrial environments and ensures the accuracy of interpretation results when faced with perturbation inputs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SIEMENS AG
- Filing Date
- 2024-12-16
- Publication Date
- 2026-07-21
AI Technical Summary
Existing AI-based black-box models struggle to provide transparent and reliable explanations in industrial environments, leading to a lack of trust in the model's capabilities among data analysts and operators. Furthermore, existing explanation methods are unreliable in the face of perturbation inputs.
By receiving instances of input data, a calibration function is applied to generate a calibrated model. The running model is then calibrated using a perturbation dataset to ensure it is suitable for the perturbation dataset in the interpretation function, thereby determining the importance of the input data features.
It improves the reliability of perturbation-based interpretation functions, ensuring that the interpretation results can still accurately reflect the actual reliability of the model when faced with perturbation inputs, and enhances the transparency and credibility of AI model outputs in industrial environments.
Smart Images
Figure CN122439162A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to a computer-implemented method for obtaining a reliability-calibrated interpretation of predictions for AI-based operating models used in industrial environments, a computer-implemented method for generating a calibration model of an AI-based operating model, corresponding interpretation and calibration apparatus, and corresponding computer program products. Background Technology
[0002] Ensuring the credibility of machine learning and AI-based solutions is one of the core requirements for applying them in industrial environments as operational (operative) AI models for purposes such as quality monitoring in manufacturing processes, control of machines like robots in the shop floor, anomaly detection and predictive maintenance of machines, or object detection and control of autonomous vehicles. However, modern machine learning methods like deep neural networks are opaque black-box models, known to be difficult to interpret. This lack of transparency regarding the reasoning behind the provided outputs makes it difficult to rigorously assess the true reliability of the models.
[0003] Without the ability to explain it, data analysts, domain experts, and potential operators often struggle to understand the logic of underlying AI systems, frequently leading to distrust in the true capabilities of ML or AI-based solutions. Furthermore, emerging laws and regulations are likely to require AI-based solutions to be transparent to end-users in specific applications, a requirement not currently adequately guaranteed by modern machine learning models. Due to the significance of the problem, various approaches have emerged to improve the interpretability of black-box models—often referred to as interpretable AI (XAI). However, as demonstrated by numerous related academic studies, their true effectiveness in providing practical insights into black-box models is frequently questionable.
[0004] Interpretable AI methods often ensure the ability to interpret the output of a black-box model through perturbation-based model evaluation, thereby deriving the importance of individual input features. However, since black-box models typically do not see any such perturbation input data during training, it is likely that they will not behave appropriately when faced with perturbation input data during the computation of interpretation. Therefore, the reliability of the retrieved interpretation can be severely affected, and numerous academic studies have demonstrated the relevant limitations.
[0005] Uncertainty calibration techniques are used to ensure that the model's output probabilities reflect the likelihood of actual correctness. This means that if a model predicts a defective part with an 80% probability during production, then if the part is indeed defective in exactly 80% of such predictions, the model is uncertainty-calibrated. Various ex-post calibration methods have been proposed that fine-tune running the model based on validation datasets to produce well-calibrated predictions. However, this does not guarantee that the model will work stably and robustly in the face of particularly perturbative input data, as is the case when using perturbation-based interpretability techniques to understand the model's predictions. Summary of the Invention
[0006] Therefore, one object of the present invention is to improve the quality of interpretability methods for providing interpretation of results from AI-based models operating at an industrial level.
[0007] This objective is achieved through the features of the independent claim. The features of the dependent claim define the embodiments.
[0008] The first aspect relates to a computer-implemented method for obtaining a reliability-calibrated interpretation of predictions for an AI-based operational model used in an industrial environment, comprising the steps of: - Receive instances of input data collected in the industrial environment to be inspected and output the predicted operational model based on the instances of input data. - A calibration function is applied to the running model to obtain a calibrated model, wherein the calibration function makes the running model suitable for processing a perturbation dataset, the perturbation dataset being constructed to resemble the explained dataset. - The calibrated model and the input data instance are fed into an interpretation function, which determines the importance of each feature of the input data for the prediction based on the interpretation dataset. - Output the importance of each feature of the input data instance as a calibrated interpretation result, thereby providing an indication of the ground truth for predictions in the industrial environment.
[0009] The main advantage of this invention is that by calibrating the operating model for the perturbations used by the interpreted function, the reliability of the calibrated interpretation results determined by the existing perturbation-based interpretation function is improved.
[0010] In an embodiment of the method, an explanatory dataset is generated by applying at least one first perturbation function to the input data instance.
[0011] Therefore, the explanatory dataset consists of multiple perturbation versions of the input data instances of interest. If the calibrated model to be interpreted outputs similar predictions when processing undisturbed input data instances and when processing the explanatory dataset where the first feature is perturbed, then the importance of the first feature is determined to be low. Generating such an explanatory dataset is simple, requires minimal processing power, and yields a wide variety of different datasets. Furthermore, such a perturbation-based explanatory function provides reliable information regarding feature importance. Overall, this approach provides reliable results with low processing effort.
[0012] In an embodiment of the method, the perturbation dataset is generated by applying a second perturbation function to the input data instance, wherein the first perturbation function and the second perturbation function include similar perturbation patterns.
[0013] This ensures that the running model is suitable for input data that has been constructed to resemble the explanatory dataset.
[0014] In an embodiment of the method, the perturbation dataset includes several modified verification datasets, wherein each of the modified verification datasets is generated by applying a different second perturbation function to the verification dataset.
[0015] This helps generate a sufficient amount of data in the perturbation dataset.
[0016] In an embodiment of the method, each perturbation function in the second perturbation function is parameterized by a different perturbation separation value.
[0017] This allows for flexible settings of the extent of modification to the modified validation dataset.
[0018] In an embodiment of the method, the calibration function is any calibration method used for post-hoc uncertainty calibration.
[0019] In an embodiment of the method, the calibration method is one of temperature scaling, vector scaling, spline calibration, Dirichlet calibration, order-preserving regression, and matrix scaling.
[0020] Post-calibration methods determine calibrated confidence scores that represent the true probability of a prediction made by a deep neural network. A classification model is calibrated if the predicted confidence scores match the probability of being correct across all confidence levels. Therefore, using known post-calibration methods to adapt running AI models requires only a small amount of additional "coding" work and results in a tight fit of the running model to perturbed datasets.
[0021] In an embodiment of the method, the interpretation function for the calibrated interpretation result is determined to be a perturbation-based interpretation function, preferably one of the following: Shapley value, locally interpretable model-independent interpretation (LIME), integral gradient, feature ablation, or occlusion.
[0022] Perturbation-based explanatory functions are well-known and provide valuable explanations for features that contribute most to the predictions of the examined running AI model.
[0023] The second aspect relates to a computer-implemented method for generating a calibrated model of an AI-based operational model for use in industrial environments, comprising: - Receive the validation dataset, at least one second perturbation function, and the running model. - A perturbation dataset is generated by applying at least one second perturbation function to the validation dataset, wherein the first and second perturbation functions include similar perturbation patterns, and wherein an explanation dataset is generated by applying at least one first perturbation function to the input data instances. - Applying a calibration function to the running model yields a calibrated model, wherein the calibration function makes the running model suitable for processing perturbation datasets, the perturbation datasets being constructed to resemble the explanatory datasets used in the explanatory function, and - Output the calibrated model for inputting the calibrated model (fc) and the input data instance (x) into the interpretation function (e), which determines the importance of each feature of the input data instance (x) to the prediction (y) based on the interpretation dataset (xe), thereby providing an indication of the basic facts of the prediction in the industrial environment.
[0024] The method for generating calibrated models, and in particular applying calibration functions to running models, prepares running models for this type of perturbation so that they behave well during the interpretation phase.
[0025] The third aspect relates to an interpretation apparatus for obtaining a reliability-calibrated interpretation of predictions for an AI-based operating model used in an industrial environment, comprising: - An input module configured to receive instances of input data collected in the industrial environment to be inspected and to output a predictive operational model based on said instances of input data. - An interpretable reliability calibrator module is configured to apply a calibration function to the running model to obtain a calibrated model, wherein the calibration function adapts the running model to handle perturbation datasets constructed to resemble the interpretable dataset. - An explanation module, configured to input the calibrated model and the input data instance into an explanation function, the explanation function determining the importance of each feature of the input data for the prediction based on the explanation dataset, and - An output module configured to output the importance of each feature of the input data instance as a calibrated interpretation result, thereby providing an indication of the basic facts of the prediction in the industrial environment.
[0026] The fourth aspect relates to a calibration apparatus for generating a calibration model of an AI-based operating model for use in industrial environments, comprising: - Input module, which is configured to receive a validation dataset, at least one second perturbation function, and a running model. - A perturbation module configured to generate a perturbation dataset by applying at least one second perturbation function to a validation dataset, wherein the first and second perturbation functions include similar perturbation patterns, and wherein an explanatory dataset is generated by applying at least one first perturbation function to an input data instance. - A calibration module configured to apply a calibration function to the running model to obtain a calibrated model, wherein the calibration function makes the running model suitable for processing a perturbation dataset constructed to resemble the explanatory dataset used in the explanatory function, and - An output module configured to output the calibrated model for inputting the calibrated model (fc) and the input data instance (x) into the interpretation function (e), which determines the importance of each feature of the input data instance (x) to the prediction (y) based on the interpretation dataset (xe), thereby providing an indication of the basic facts of the prediction in the industrial environment.
[0027] The fifth aspect relates to a computer program product that can be directly loaded into the internal memory of a digital computer, comprising a software code portion for obtaining a reliability-calibrated interpretation of predictions for an AI-based operating model applied in an industrial environment when the product is run on the digital computer.
[0028] The sixth aspect relates to a computer program product that can be directly loaded into the internal memory of a digital computer, comprising a software code portion for generating a calibrated model of an AI-based operating model for use in an industrial environment when the product is run on the digital computer. Attached Figure Description
[0029] The invention will be explained in more detail with reference to the accompanying drawings. Similar objects will be labeled with the same reference numerals.
[0030] Figure 1 The illustration schematically depicts an industrial environment where machines are monitored by an AI-based operating model, and predictions are interpreted by an interpreting device.
[0031] Figure 2 An embodiment of the main steps of a method for obtaining a reliability-calibrated interpretation is illustrated schematically.
[0032] Figure 3 An embodiment of the method of the present invention for generating a calibrated model of an AI-based operating model is illustrated schematically.
[0033] Figure 4 The illustration shows an example of a calibration apparatus configured to perform a method for generating a calibrated model of an AI-based operating model.
[0034] Figure 5 An example of an interpretation apparatus configured to perform a method for obtaining a reliability-calibrated interpretation is illustrated. Detailed Implementation
[0035] Note that in the following detailed description of the embodiments, the accompanying drawings are merely schematic, and the illustrated elements are not necessarily shown to scale. Rather, the drawings are intended to illustrate the cooperation of functions and components. It should be understood here that any connection or coupling of functional units, modules, components, or other physical or functional elements may also be achieved through direct or indirect connection of coupling elements (e.g., via one or more intermediate elements). Connections or couplings of entities or components may be achieved, for example, through cable-based connections, wireless connections, and / or a combination of cable-based and wireless connections. Functional modules may be implemented by dedicated hardware (e.g., a processor), firmware, or software, and / or a combination of dedicated hardware and firmware and software. It should also be noted that functional steps of the associated methods can be performed for each functional module described for the apparatus, and vice versa.
[0036] Explainable AI (XAI) (which often overlaps with interpretable AI or explainable machine learning) refers either to the possibility of retaining intelligent oversight of the AI system for human intervention, or to the methods used to achieve this. The main focus is typically on making the reasoning behind the AI's decisions or predictions more understandable and transparent. XAI counters the "black box" tendency of machine learning, where even the AI's designers cannot explain why the AI arrives at a particular decision.
[0037] Ensuring the credibility of machine learning and other AI-based solutions is one of the core requirements for applying AI-based solutions in industrial environments as running AI models, which are used for, for example, quality monitoring of manufacturing processes, control of machines such as robots in the shop floor, anomaly detection and predictive maintenance of machines, or object detection and control for autonomous vehicles.
[0038] Figure 1 An exemplary illustration shows a machine 1 in an industrial environment controlled or monitored by an AI-based operational model f. Data is collected at machine 1, for example, via sensors, and input data instance x from these sensors is fed into the AI-based operational model f implemented in an auxiliary device 2. The AI-based operational model f processes the input data instance x and outputs a prediction y, such as a classification result. The input data instance x and the operational model f are then fed into an interpretation device 3, which provides an interpretation result E as output, indicating the importance of different features of the input data instances that contribute to the prediction y.
[0039] The interpretation result E not only provides confidence in the predictions and the AI-based operating model itself, but also offers an indication of the underlying facts of the predictions. For example, if the operating model classifies machine 1's operating patterns as normal or abnormal, the interpretation result provides an indication of which features in the input data contribute the most. Determining the cause of detected abnormal patterns can be aided or derived from the most important features.
[0040] The method described in this paper aims to improve the fidelity and reliability of interpretations determined by the interpretation function applied to the predicted output of the AI-based running model by recalibrating the AI-based running model according to the specific needs of the interpretation function utilized. The interpretability of black-box models, such as AI-based running models f, is typically ensured through perturbation-based model evaluation to derive the importance of individual input features. However, since AI-based running models typically do not see any such perturbation input data during training, it is likely that the AI-based running model will not behave appropriately when faced with such perturbation input data during the computation of interpretation. Therefore, the reliability of the obtained interpretation can be severely affected and reduced.
[0041] Figure 2 The diagram illustrates the method for obtaining information such as... Figure 1 The main steps of the method for interpreting the prediction y of the AI-based operating model, as shown, are described.
[0042] In the first step S1, input data instance x collected in the industrial environment to be inspected and a running model f that outputs a prediction y based on input data instance x are received. The running model (e.g., a machine learning model) The model f has been trained to predict y for input data instances x in a set of input data X. In the next step S2, a calibration function c is applied to the running model f to obtain a calibrated model fc. The calibration function c makes the running model f suitable for processing the perturbation dataset xp.
[0043] The perturbation dataset xp is constructed in the same manner as the explanatory dataset xe, which is applied to the explanatory function e. The explanatory dataset xe consists of multiple perturbation versions of a single input data instance x of interest. The calibration function c outputs the calibrated model fc.
[0044] The calibrated model fc and the input data instance x are fed into the interpretation function e, which determines the importance of each feature of the input data instance x for the prediction y based on the interpretation dataset xe (see step S3). Finally, the importance of each feature of the input data instance x is output as the calibrated interpretation result Ec (see S4).
[0045] Compared to simply applying interpretation techniques, the calibrated interpretation result Ec interprets the predictions in a more reliable way. Here, the interpretation function e is applied to the original running model f without proper reliability calibration.
[0046] exist Figure 3 The method for generating a calibrated model fc of an AI-based running model f is described in the text and further detailed below.
[0047] In the first step C1, the validation dataset Xv, at least one second perturbation function Pv, and the running model f are obtained.
[0048] The perturbation dataset xp is generated by applying at least one second perturbation function Pv to the validation dataset Xv. The at least one second perturbation function Pv consists of different perturbation functions: , These perturbation functions are separated by perturbation values with values between 0 and 1. To parameterize.
[0049] At least one second perturbation function Pv includes a perturbation pattern similar to at least one first perturbation function Pe, which is applied to the interpretation function e to generate the interpretation dataset xe, see C2. Preferably, the second perturbation function Pv contains exactly the same type of perturbation as the type of perturbation subsequently used by the interpretation function e to derive the interpretation. This is crucial because the goal is to 'prepare' the model for this type of perturbation so that it behaves well during the interpretation phase. As described above, the interpretation dataset xe is generated by applying at least one first perturbation function Pe to the input data instance x.
[0050] The calibration function c is applied to the running model f, resulting in the calibrated model fc. The calibration function c makes the running model f applicable to processing the perturbation dataset xp, which is constructed to resemble the explanatory dataset xe used in the explanatory function e, see step C3.
[0051] Finally, output the calibrated model fc, see step C4.
[0052] The calibration function fc is also used for any calibration technique employed in post-hoc uncertainty calibration. Uncertainty calibration is known to be applicable to deep neural networks, a possible AI-based model applied as a running model f in industrial environments. Due to the high predictive power of deep neural networks, they are increasingly being used as part of decision-making systems in real-world applications. However, such systems require not only high accuracy but also reliable and calibrated uncertainty estimates.
[0053] If the predicted confidence level matches the probability of being correct across all confidence levels, then a classifier, such as that implemented by a deep neural network, is calibrated. This is particularly important in safety-critical applications in healthcare where average-case performance is insufficient, and in dynamic environments in industry, where practitioners need reliable predictive uncertainty access throughout the model's lifecycle. This means that confidence scores (or predictive uncertainty) should be well-calibrated not only for in-domain predictions but also for gradual domain drift—that is, as the distribution of input samples gradually changes from in-domain to true out-of-distribution.
[0054] A popular approach for well-calibrated predictions is post-calibration, where a validation set extracted from the generated distribution of the training data is used to rescale the output returned by the trained neural network, ensuring that the in-domain predictions are well-calibrated. More details on post-calibration can be found in the paper "Post-hoc Uncertainty Calibration for Domain Drift Scenarios" by Christian Tomani et al., published at https: / / arxiv.org / abs / 2012.1098.
[0055] Compared with existing uncertainty calibration methods, using post-calibration in interpretability reliability calibration methods to improve the reliability of interpretation results is a completely different and novel calibration objective.
[0056] exist Figure 4 An embodiment of the calibration device is depicted in the figure and described in more detail below.
[0057] The calibration device 10 includes an input module 11, a disturbance module 12, a calibration module 13, and an output module 14. Modules 11-13 are communicatively linked and implemented by at least one processor.
[0058] Input module 11 is configured to receive a validation dataset Xv, at least one second perturbation function Pv, and a running model f. The validation dataset Xv includes validation data instances xv and predictions yv assigned to the validation data instances xv. The validation dataset Xv includes, for example, a subset of training data used to train the running model f. In another embodiment, the validation dataset includes any set of data points sampled on similar or identical machines deployed in the same industrial environment and associated labels. Input module 11 passes the validation dataset Xv and at least one second perturbation function Pv to perturbation module 12.
[0059] Perturbation module 12 is configured to create a perturbation dataset xp. The perturbation dataset xp is represented as: , Here, u is a uniform distribution with the same dimension as the predicted y. Therefore, the perturbation dataset xp comprises several modified validation datasets. Each of the modified validation datasets is generated by applying different second perturbation functions Pv to the validation datasets. As described above, each perturbation function in the second perturbation function Pv is parameterized by different perturbation separation values. It is important to note that the second perturbation function Pv and the first perturbation function Pe are the same or at least similar perturbation functions.
[0060] The modified validation dataset xp is passed to calibration module 13. Calibration module 13 is configured to receive the modified validation dataset xp and the running model f. A calibration function c is implemented in calibration module 13, which, together with the modified validation dataset xp, is applied to the running model f to obtain a calibrated model fc. Initially, calibration function c itself is trained to satisfy the condition that calibration function c, when processed on the running model f at the modified validation dataset xp, outputs the label yval assigned to the modified validation dataset xp. That is, the parameters of calibration c are thus optimized to satisfy... For each .
[0061] It is important to note that only the parameters of the calibration function c are optimized, not the parameters of the operation function f, to ensure that, for example, the same operating model f is still interpreted through the interpretation function e.
[0062] The calibration function c implemented in calibration module 13 can be any calibration technique also used for post-uncertainty calibration, such as temperature scaling, vector scaling, spline calibration, Dirichlet calibration, ordinal-preserving regression, and matrix scaling. What is special about calibration function c is that it runs on a specifically modified validation dataset, i.e., a perturbed dataset xp.
[0063] When the running model f is input to the calibration module 13, the calibration function c is applied to the running model f to obtain the calibrated model fc. The calibration module is configured to transmit the calibrated model fc to the output module 14, which outputs it.
[0064] Calibration device 10 can be used as a module within interpretation device 20 for post-processing of operating model f (see...). Figure 5 ).
[0065] The interpretation device 20 provides a reliability-calibrated model interpretation system. The interpretation device 20 is configured to perform the aforementioned method for obtaining a reliability-calibrated interpretation of predictions for an AI-based operating model used in an industrial environment.
[0066] The interpretation device includes an input module 21, an interpretation reliability calibrator module 22, an interpretation module 23, and an output module 24. These modules are implemented by at least one processor and are communicatively connected.
[0067] Input module 21 is configured to receive input data instance x collected in an industrial environment, run model f based on input data instance x and optionally based on a set of first perturbation functions Pe, and output a prediction y. In one embodiment, the first perturbation function Pe has been included in interpretation module 13, for example as part of interpretation function e.
[0068] The interpretability calibrator module 22 is configured to take a running model f as input and output a calibrated model fc. The interpretability calibrator module 22 includes a calibration function c that uses the interpretability dataset xe to prepare the running model f to be analyzed by the interpretability module 23. The interpretability calibrator module 22 can be implemented by the calibration device 10.
[0069] Interpretation module 23 is configured to receive a calibrated model fc from interpretation reliability calibrator module 22 and an input data instance from input module 21. Interpretation module 23 is configured to input the calibrated model fc and the input data instance x into an interpretation function e, which determines a calibrated interpretation result Ec based on the interpretation dataset xe. This calibrated interpretation result Ec represents the importance of each feature x1, ..., x3 of the input data instance x for the prediction y. Interpretation module 23 includes the interpretation function e that determines the calibrated interpretation result Ec. This interpretation function e is a perturbation-based interpretation function, preferably one of Shapley value, Locally Interpretable Model-Independent Interpretation (LIME), integral gradient, feature ablation, or occlusion.
[0070] Output module 24 is configured to output the calibrated interpretation result Ec. In one embodiment, output module 24 includes a graphical user interface, which allows for... Figure 5 The diagram depicts the calibrated interpretation result Ec, which shows the importance of each feature x1, ..., x3.
[0071] In other words, instead of simply applying interpretability techniques to the original running model f, the running model f is enhanced with the interpretability reliability calibrator module 22 to obtain a calibrated model fc, which is suitable for processing perturbed datasets similar to the interpretable datasets applied in the interpretability module.
[0072] The calibrated model fc, along with the input data instance x, is passed to the interpretation module 23. The interpretation module 23 is designed to identify, based on the input data instance x, how important each input feature of x, symbolized by x1, x2, and x3, is to the prediction output by the calibrated model fc.
[0073] The calibrated interpretation result Ec is typically calculated by evaluating the calibrated model fc using a perturbation version xe based on the input data instance x, which is obtained using a perturbation in the first perturbation function Pe. The output calibrated interpretation result Ec is more reliable than simply applying the interpretation technique to the original running model f without proper reliability calibration.
[0074] For an instance of input data as an image, the interpretation dataset includes several versions of the image, where different regions of the image are either "greyed out" or modified elsewhere.
[0075] The main advantage of the proposed method is that it improves the reliability of existing perturbation-based interpretation techniques by calibrating the running model against the perturbations used in the interpretation module. This ensures that the interpretation module does not misbehave during feature importance calculations—which would otherwise lead to clearly biased interpretations that could confuse data scientists and domain experts, and potentially result in erroneous conclusions, such as regarding the root cause.
[0076] It should be understood that the above description of the examples is for illustrative purposes, and the illustrated components allow for various modifications. For example, the illustrated concepts can be applied to different technical systems, and in particular, to different subtypes of the respective technical systems with minimal adaptation.
Claims
1. A computer-implemented method for obtaining a reliability-calibrated interpretation of predictions for an AI-based operational model (f) used in an industrial environment, comprising the following steps: - Receive (S1) an input data instance (x) collected in the industrial environment to be inspected and an operating model (f) that outputs a prediction (y) based on the input data instance (x). - Apply the calibration function (c) (S2) to the running model (f) to obtain the calibrated model (fc), wherein, The calibration function (c) makes the running model (f) suitable for processing perturbation datasets (xp) constructed similarly to the interpreted dataset (xe). - The calibrated model (fc) and the input data instance (x) are input (S3) into the interpretation function (e), which determines the importance of each feature of the input data instance (x) based on the interpretation dataset (xe), and - Output (S4) the importance of each feature of the input data instance (x) as a calibrated interpretation result (Ec), thereby providing an indication of the basic facts of the prediction in the industrial environment.
2. The computer-implemented method according to claim 1, wherein, The explanatory dataset (xe) is generated by applying at least one first perturbation function (Pe) to the input data instance (x).
3. The computer-implemented method according to claim 2, wherein, The perturbation dataset (xp) is generated by applying a second perturbation function (Pv) to the input data instance (x), wherein the first perturbation function (Pe) and the second perturbation function (Pv) include similar perturbation patterns.
4. The computer-implemented method according to claim 3, wherein, The perturbation dataset (xp) includes several modified verification datasets, wherein each modified verification dataset is generated by applying a different second perturbation function (Pv) to the verification dataset (xv).
5. The computer-implemented method according to any one of claims 3 or 4, wherein, Each perturbation function in the second perturbation function (Pv) is parameterized by different perturbation separation values.
6. The computer-implemented method according to any one of the preceding claims, wherein, The calibration function (c) is any calibration method used for post-hoc uncertainty calibration.
7. The computer-implemented method according to claim 6, wherein, The calibration method (c) is one of temperature scaling, vector scaling, spline calibration, Dirichlet calibration, ordinal-preserving regression, and matrix scaling.
8. The computer-implemented method according to any one of the preceding claims, wherein, The interpretation function (e) is any perturbation-based interpretation function, preferably one of Shapley value, locally interpretable model-independent interpretation (LIME), integral gradient, feature ablation, or occlusion.
9. A computer-implemented method for generating a calibrated model (fc) of an AI-based operational model for use in an industrial environment, comprising the following steps: - Receive (C1) a validation dataset (xv), at least one second perturbation function (Pv), and a running model (f). - A (C2) perturbation dataset (xp) is generated by applying at least one second perturbation function (Pv) to the validation dataset (xv), where, The first perturbation function (Pe) and the second perturbation function (Pv) include similar perturbation patterns, and wherein an explanatory dataset (xe) is generated by applying at least one first perturbation function (Pe) to an input data instance (x). - Apply the calibration function (c) (C3) to the running model (f) to obtain a calibrated model (fc), wherein the calibration function (c) makes the running model (f) suitable for processing a perturbation dataset (xp), which is constructed to resemble the explanatory dataset (xe) used in the explanatory function (e), and - Output (C4) the calibrated model (fc) for inputting the calibrated model (fc) and the input data instance (x) into the interpretation function (e), which determines the importance of each feature of the input data instance (x) to the prediction (y) based on the interpretation dataset (xe), thereby providing an indication of the basic facts of the prediction in the industrial environment.
10. An interpretation apparatus (20) for obtaining a reliability-calibrated interpretation of predictions for an AI-based operating model (f) used in an industrial environment, comprising: - Input module (21), which is configured to receive an input data instance (x) collected in the industrial environment to be inspected and an operating model (f) that outputs a prediction (y) based on the input data instance (x). - An interpreter reliability calibrator module (22) is configured to apply a calibration function (c) to the running model (f) to obtain a calibrated model (fc), wherein the calibration function (c) makes the running model (f) suitable for processing a perturbation dataset (xp) similar to the interpreter dataset (xe) being constructed. - An interpretation module (23) is configured to input the calibrated model (fc) and the input data instance (x) into an interpretation function (e), which determines, based on the interpretation dataset (xe), the importance of each feature of the input data instance (x) to the prediction (y), and - Output module (24), which is configured to output the importance of each feature of the input data instance (x) as a calibrated interpretation result (Ec), thereby providing an indication of the basic facts of the prediction in the industrial environment.
11. A calibration apparatus (10) for generating a calibration model (fc) of an AI-based operating model (f) for use in an industrial environment, comprising: - Input module (11), which is configured to receive a validation dataset (xv), at least one second perturbation function (Pv) and a running model (f). - A perturbation module (12) is configured to generate a perturbation dataset (xp) by applying at least one second perturbation function (Pv) to the validation dataset (xv), wherein the first perturbation function (Pe) and the second perturbation function (Pv) include similar perturbation patterns, and wherein an explanatory dataset (xe) is generated by applying at least one first perturbation function to an input data instance (x). - A calibration module (13) is configured to apply a calibration function (c) to the running model (f) to obtain a calibrated model (fc), wherein the calibration function (c) makes the running model (f) suitable for processing a perturbation dataset (xp), the perturbation dataset (xp) being constructed similarly to the explanatory dataset (xe) used in the explanatory function (e), and - Output module (14), configured to output the calibrated model (fc) for inputting the calibrated model (fc) and the input data instance (x) into the interpretation function (e), which determines the importance of each feature of the input data instance (x) to the prediction (y) based on the interpretation dataset (xe), thereby providing an indication of the basic facts of the prediction in the industrial environment.
12. A computer program product capable of being directly loaded into the internal memory of a digital computer, comprising instructions that, when the product is run on the digital computer, cause the digital computer to perform the steps of claims 1 to 8.
13. A computer program product capable of being directly loaded into the internal memory of a digital computer, comprising instructions that, when the product is run on the digital computer, cause the digital computer to perform the steps of claim 9.