An iterative self-explanatory artificial intelligence system for trustworthy decision making

Abstract

To provide an iterative self-explaining artificial intelligence system for trustworthy decision making.SOLUTION: A method for generating a self-explaining decision in an artificial intelligence (AI) system is provided, comprising receiving or defining a graph for a task in the AI system, the graph including a plurality of nodes connected by edges. Message passing is performed among the nodes of the graph, and a discrete attention mechanism is implemented during the message passing, thereby transforming features of each node into a discrete representation designed to vary depending on which neighboring node a message is passed to. The self-explaining decision is generated for one of the nodes based on the message passing.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS Priority is claimed to U.S. Provisional Patent Application No. 63 / 384,235, filed November 18, 2022, the entire disclosure of which is incorporated herein by reference.

[0002] The present invention relates to artificial intelligence (AI) and machine learning, and in particular to explainable AI and methods, systems, and computer-readable media for iteratively learning human-understandable explanations along with AI-driven predictions and decisions. [Background technology]

[0003] According to the European Union (EU) AI Act and General Data Protection Regulation (GDPR), transparency is a key requirement for the use of AI technologies, especially in high-risk systems such as digital healthcare and critical urban infrastructure. Based on these EU regulations, it is no longer possible to deploy so-called "black box" AI systems in the EU unless they provide explanations along with their predictions. Although there are explainable AI methods developed to interpret how an AI-based system reaches a decision, the majority of existing explainable AI (XAI) methods focus on post-hoc explaining (i.e. explaining a decision made by another AI system). See, for example, Hao Yuan et al., "On Explainability of Graph Neural Networks via Subgraph Explorations," Proceedings of the 38th International Conference on Machine Learning, arXiv:2102.05152v2 (May 31, 2021), hereafter "Yuan," and Phillip Pope et al., "Explainability Methods for Graph Convolutional Neural Networks," Proceedings of Conference on Computer Vision and Pattern Recognition, DOI:10.1109 / CVPR.2019.01103 (June 2019), hereafter "Pope," each of which is incorporated herein by reference. These techniques can help legacy AI systems comply with regulations and increase end-user engagement. However, there are significant limitations in such XAI methods that remain to be addressed.First, because post-mortem explanations are learned by one AI from the behavior of another, the fidelity of learned explanations has become an important concern for society (see, e.g., Chih-Kuan Yeh et al., “On the (In)fidelity and Sensitivity of Explanations,” Advances in Neural Information Processing Systems, arXiv:1901.09392v4 (November 3, 2019), incorporated herein by reference). Moreover, most existing explainable AI methods aim to learn algorithmic explanations, e.g., sparse subgraphs that maximize a pre-determined score (see Yuan) or the gradient of a score function with respect to the observations (see Pope). Such explanations are often difficult to understand for humans, especially for users without AI knowledge. Such implausible explanations not only do not increase users' trust in the AI ​​system, but on the contrary, they may actually increase users' doubt and hesitation (see, e.g., Himabindu Lakkaraju et al., "'How do I fool you?': Manipulating user trust via misleading black box explanations," In Proceedings of the AAAI / ACM Conference on AI, Ethics, and Society, arXiv:1911.06473v1 (November 15, 2020), incorporated herein by reference).

[0004] Zaixi Zhang et al., "ProtGNN: Towards Self-Explaining Graph Neural Networks," Association for the Advancement of Artificial Intelligence, arXiv:2112.00911v1 (December 2, 2021), incorporated herein by reference, describes a method for graph data that addresses the graph classification problem. In this method, explanations learned are based on subgraphs, and these are still algorithm-driven (minimizing a pre-defined score), and there is no mechanism for providing feedback on the learned explanations. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] "On Explainability of Graph Neural Networks via Subgraph Explorations", Proceedings of the 38th International Conference on Machine Learning, arXiv:2102.05152v2 (May 31, 2021) [Non-Patent Document 2] Phillip Pope et al., “Explainability Methods for Graph Convolutional Neural Networks,” Proceedings of Conference on Computer Vision and Pattern Recognition, DOI:10.1109 / CVPR.2019.01103 (June 2019) [Non-Patent Document 3] Chih-Kuan Yeh et al., “On the (In)fidelity and Sensitivity of Explanations,” Advances in Neural Information Processing Systems, arXiv:1901.09392v4 (November 3, 2019) [Non-Patent Document 4] Himabindu Lakkaraju et al., "'How do I fool you?': Manipulating user trust via misleading black box explanations," In Proceedings of the AAAI / ACM Conference on AI, Ethics, and Society, arXiv:1911.06473v1 (November 15, 2020) [Non-Patent Document 5] Zaixi Zhang et al., “ProtGNN: Towards Self-Explaining Graph Neural Networks,” Association for the Advancement of Artificial Intelligence, arXiv:2112.00911v1 (December 2, 2021) Summary of the Invention [Means for solving the problem]

[0006] In one embodiment, the invention provides a method for generating self-explanatory decisions in an AI system, comprising the steps of receiving or defining a graph for a task in the AI ​​system, the graph including a plurality of nodes connected by edges. Message passing is performed between the nodes of the graph, and a discrete attention mechanism is implemented during the message passing, whereby features of each node are transformed into a discrete representation, which changes depending on which neighboring node a message is passed to. A self-explanatory decision is generated for one of the nodes based on the message passing.

[0007] The subject matter of the present disclosure will be described in more detail below based on exemplary figures. All features described and / or illustrated herein may be used alone or combined in different combinations. The features and advantages of various embodiments will become more apparent by reading the following detailed description with reference to the attached figures. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 illustrates a self-explanatory decision engine and its workflow according to an embodiment of the present invention. [Diagram 2] FIG. 2 illustrates a self-explanatory decision learner of a self-explanation based decision engine according to an embodiment of the present invention. [Diagram 3] FIG. 2 illustrates tracking of a predicted path according to an embodiment of the present invention. [Figure 4] 4 is a flowchart illustrating method steps for iterative self-explanation based node classification according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Embodiments of the present invention provide methods and self-explanatory, graph-based AI systems that learn human-understandable explanations along with AI-driven predictions and decisions to enable transparent decision-making. Embodiments of the present invention technically improve upon state-of-the-art AI systems by (1) being self-explanatory, (2) enabling user-centricity, (3) providing comprehensive explanations based on instances, attributes, and / or graph structure, and (4) enabling iterative explanations.

[0010] The embodiments of the present invention may also be practically applied to provide further improvements in various technologies that utilize explainable AI systems, especially in high-risk applications. One exemplary high-risk application that may be improved by the embodiments of the present invention is an AI-driven patient diagnostic support system that facilitates doctors to predict which medicines will be effective for a particular patient, given the patient's symptoms, test, and gene sequence data, and disease information. In this system, if the AI-driven system only provides an end user, such as a doctor, with a list of candidate medicines without an explanation of why it predicted the medicine for the patient, the doctor cannot trust the result and prescribe the recommended medicine to the patient. Another example of a high-risk application that may be improved by the embodiments of the present invention is a predictive maintenance management system for water / gas / electricity supply networks. If the AI-driven system only predicts which pipes are dangerous and need to be repaired without an explanation of why it made such a prediction, the operator will not trust the result and take action.

[0011] The present invention addresses these challenges by providing a self-explaining graph-based AI method and system that generates human-understandable explanations along with AI-driven predictions and decisions. Inspired by case-based learning in medical education, the present invention learns to generate case-based explanations. For example, in a patient diagnosis support system, the explanations learned could be which patients (i.e., nodes in the patient graph) are similar to a test patient (also a node in the patient graph) in terms of certain features, and where on the patient graph the patient is connected to the test patient. Based on the human-understandable explanations, the end user can verify the AI-driven decisions and decide with higher confidence to allow or not allow the system to execute the decision.

[0012] In an embodiment, the present invention provides a method for semi-supervised situations. For example, in one practical application, test patients (e.g., unknown drugs) and training patients (e.g., known drugs) are given as a batch. Thus, model parameters are learned by considering both training data and test data. This is an advantage of semi-supervised learning, since it considers the data characteristics of the test data. When a small number of new test patients arrive, it is possible to directly use the already learned model parameters for message passing and calculating important features. However, when the newly available test patients are large or show different characteristics from the previous ones, the model may need to be trained again.

[0013] In a first aspect, the present invention provides a method for generating self-explanatory decisions in an AI system. The method includes receiving or defining a graph for a task in the AI ​​system, the graph including a plurality of nodes connected by edges. Message passing is performed between the nodes of the graph, and a discrete attention mechanism is implemented during the message passing, whereby features of each node are transformed into a discrete representation, which changes depending on which neighboring node a message is passed to. A self-explanatory decision is generated for one of the nodes based on the message passing.

[0014] In a second aspect, the invention provides a method according to the first aspect, wherein the discrete attention mechanism comprises using an importance vector that is different for each pair of nodes and indicates which features of the pair of nodes are important, the message passing comprises passing the important features determined by the importance vector, and the self-explanatory decision comprises an explanation of which of the nodes are similar with respect to each of the features to the node about which the self-explanatory decision is made.

[0015] In a third aspect, the present invention provides a method according to the first or second aspect, wherein parameters of the model are learned during a training procedure using as input observed features of a portion of the nodes to provide a learned model that is used for message passing and generating self-explanatory decisions.

[0016] In a fourth aspect, the present invention provides a method according to any of the first to third aspects, further comprising performing iterative explanation generation and validation, wherein feedback is received on the explanations and the feedback is used to improve the learned model.

[0017] In a fifth aspect, the present invention provides a method according to any of the first to fourth aspects, wherein performing the iterative explanation generation and validation is based on feedback comprising at least one of the following: correctness of identified important features of one or more nodes, correctness of identified similar neighbors of the node, and a request to follow the predicted path.

[0018] In a sixth aspect, the present invention provides a method according to any of the first to fifth aspects, wherein a refined self-explaining decision is determined by performing further message passing using the refined learned model.

[0019] In a seventh aspect, the present invention provides a method according to any of the first to sixth aspects, wherein an importance vector for a node for which a self-explanatory decision is made is refined based on feedback to modify at least one of the nodes indicated as similar by the explanation and / or to modify at least one important feature for at least one of the nodes indicated as similar by the explanation.

[0020] In an eighth aspect, the invention provides a method according to any of the first to seventh aspects, wherein the message passing comprises passing from each neighbouring node in the graph of the node for which the self-explanatory decision is to be made one of the features determined by an importance vector that is different for each of the neighbouring nodes and indicates in each case that the different features of the neighbouring nodes are important to the node for which the self-explanatory decision is to be made.

[0021] In a ninth aspect, the present invention provides a method according to any of the first to eighth aspects, further comprising calculating a similarity between each of the neighbouring nodes and the node to which the self-explanatory decision is made, the self-explanatory decision being based only on significant features passed by neighbouring nodes that are determined to be similar.

[0022] In a tenth aspect, the present invention provides a method according to any of the first to ninth aspects, wherein the discrete attention mechanism is a multi-layer discrete attention mechanism based on a Gumbel approximator or a sparsemax, and each layer of the multi-layer attention mechanism calculates the proximity of node features using hidden vectors calculated from the previous layer.

[0023] In an eleventh aspect, the present invention provides a method according to any of the first to tenth aspects, wherein the hidden vector for a node for which a self-explanatory decision is made comprises features determined by message passing in a previous layer from neighbouring nodes determined to be similar, and comprises different importance features determined by a different importance vector for each of the neighbouring nodes.

[0024] In a twelfth aspect, the invention provides a method according to any of the first to eleventh aspects, wherein the AI ​​system is programmed for use in automated medicine, for drug development and patient diagnosis assistance, and the task is to predict a patient outcome or determine an effective drug, and the decision is the predicted patient outcome or drug, and the human understandable explanation indicates one or more other similar patients as determined by the discrete attention mechanism having the features used to make the decision.

[0025] In a thirteenth aspect, the present invention provides a method according to the first to twelfth aspects, wherein the AI ​​system is used in automated decision making in a smart city for maintenance of a utility grid, the task being to identify control or maintenance targets, the decision being an automated identification of the control or maintenance targets or an associated automated control or maintenance action, and the human understandable explanation indicating one or more other parts of the utility grid determined by the discrete attention mechanism having the features used to make the decision.

[0026] In a fourteenth aspect, the present invention provides a system for generating self-explanatory decisions in an AI system, the system comprising one or more hardware processors configured, alone or in combination, to effect the execution of the steps of the method according to any of the first to thirteenth aspects.

[0027] In a fifteenth aspect, the present invention provides a tangible, non-transitory computer-readable medium having instructions which, when executed by one or more processors, effect the performance of a method for generating self-explanatory decisions in an AI system according to any of the first to fourteenth aspects.

[0028] FIG. 1 illustrates a self-explaining decision engine 100 for an automated AI system and its workflow. Input includes a graph structure 102 (nodes and edges) and features of the nodes. The self-explaining decision engine 100 receives feedback from a device 110 that is used as input to an iterative updater 112. The self-explaining decision learner 104 and the iterative updater 112 provide a learned model 108 that can be used to output an improved decision 106 of the self-explaining decision engine 100 for a task in the AI ​​system, such as for node classification. This improved decision 106 can be, for example, a prediction or an automated action for the task, accompanied by a human-understandable explanation.

[0029] In the training procedure or training phase, the self-explaining decision learner 104 and the iterative updater 112 learn the parameters of the learned model 108. After training, the parameters of the learned model 108 are fixed. Thus, after training, the learned model 108 can be applied to a new machine learning task (e.g., to predict a drug for a patient) without the need to perform further training. In this case, the message passing of a particular patient was optimized during training.

[0030] In the test procedure or application phase, the learned model 108 is called by the self-explanatory decision learner 104 and the iterative updater 112 to generate decisions and explanations, also called performing inference. A single model can be used for a number of different machine learning tasks, characterized by the parameters of a message passing procedure. The message passing procedure is completely determined by the parameters in the exemplary formulas provided herein. The parameters are learned in a training period. The learned parameters can then be used to calculate how to pass messages for each test patient and ultimately obtain a decision. Then, after the decision is provided to a user, for example a physician, an iterative procedure using the iterative updater 112 can be performed to provide an improved decision 106. The physician then sees which patients are similar to the test patient in terms of which features, so that the physician can determine whether the explanation is valid and provide feedback thereon. Based on the input, the model defined by the parameters of the exemplary formulas provided herein is updated by the iterative updater 112 and saved as an updated learned model 108. Finally, the decisions and explanations may be recalculated using the updated learned model 108 to provide improved decisions 106.

[0031] As shown in FIG. 2, the self-explanatory decision learner 104 includes two components: an explanation learner 204 and a decision estimator 206. The explanation learner 204 is programmed to learn and output human-understandable explanations. Specifically, the form of explanation is which node (e.g., x, y) is similar to node v in terms of which important features, and where on the graph nodes x, y, etc. are connected to node v. The important features or sets of important features between pair (x, y) and those between pair (y, v) may be different. The decision estimator 206 is programmed to generate and output decisions for the AI ​​system based on the learned explanations that explain the similar neighboring nodes. Thus, according to this exemplary embodiment, the decision estimator 206 outputs decisions that are node characteristic predictions, accompanied by human-understandable explanations from the explanation learner 204.

[0032] The explanation learner 204 and decision estimator 206 use message passing to learn which important features of neighboring nodes are important for different decisions. Referring to an embodiment in which the nodes of the graph represent patients, the explanation learner 204 can calculate which patients are similar to a patient in terms of which features, and then ignore the features that are not important, and dissimilar patients are also ignored. Only messages of the important features of similar patients are passed to the test patient to finally infer the decision.

[0033] The explanation learner 204 and the decision estimator 206 are trained simultaneously in a self-explanatory graph-based AI method according to one embodiment of the present invention, which is based on multi-layer discrete attention. i and its neighbours j It is first learned to select the important features of feature x i ∈R D Node v with i Assuming that, the neighborhood v j ∈N(v i ) for each neighbour v j is the feature x j Since a soft mixture of features may confuse a user, especially if the user is not AI savvy, a discrete-like attention mechanism may be used to identify important features for pairs of nodes. The discrete-like attention mechanism may be based on, for example, Gumbel softmax and sparsemax. i and v j For each feature d in a pair, node v i and v j The variable ω, which specifies whether the feature d is important for those nodes in quantifying the similarity of i,j,d is introduced. The importance of features is node-pair specific. In other words, each pair of nodes is associated with a separate importance vector ω i,j ∈{0,1} D We have the important vector ω i,jis a binary vector {0,1} D where D is the number of all features. For each dimension, ω i,j,d = 1, otherwise 0. For different pairs of patients, the vector ω i,j,d For example, for one pair of patients, the age feature is important, but for another pair, the blood pressure feature is important. The importance vector ω i,j can be calculated by the following formula, which takes the features of two patients as input. An example of hard attention (D-head 2-dimensional Gumbel attention) is

number

[0034] Node.v i and v j Given the important features of a pair of, we can project messages between pairs of nodes, e.g. m j→(i,j) =x j 〇ω i,j ·W (0) ;m i→(i,j) =x i 〇ω i,j ·W (0) where the symbol 〇 denotes the element-wise product, and W (0) is the parameter matrix to be learned during the training procedure. These messages are i and v j represents the information projected onto the importance vector ω i,j Because is node-pair specific, important features are learned for each pair of nodes to construct a message, which biases the message to ignore potentially noisy information, and more importantly, this biased message passing process is more human understandable.

[0035] Message passing can also be described by the following pseudocode / listing of steps: For each computing layer of the proposed neural network: For each node i: For each neighbor j of node i: 1. Importance vector ω i,j Calculate. 2. Omit unimportant features of node j. ωi,j,d If d is 0, it means that feature d is not important for i and j and can be omitted when passing messages between i and j. 3. Project the remaining features of node j into a vector space, which is the message for passing. 4. Perform steps 2 and 3 for node i. 5. For example, using the following formula, z i,j Calculate z i,j = 1, then the two nodes are considered as similar nodes and the message may be passed from j to i, otherwise these two nodes are not similar and the procedure moves to the next neighbor of node i. Collect all messages from i’s similar neighbors and update i’s hidden vector (see equation below). The hidden vector is i’s “feature” for the next layer of computation.

[0036] It then learns to identify similar neighbors based on the messages. i,j If is 1, the two nodes are similar and z i,j If is 0, then they are not similar. To avoid introducing extra confusion to the user, another layer of hard (discrete) attention is utilized to find a limited number of most similar neighbors (soft similarity) from all the neighbors (e.g., all the neighbors of a node) rather than using a soft mixture of all the neighbors. Any hard attention can be used here, e.g., Gumbel approximator or sparsemax. For example (N i -head 2-dimensional gumbel attention)

number

[0037] The detected significant neighbors are node v i Then, it passes those messages to v for the next layer. i For example, if the hidden layer of

number

[0038] That is, in layer l, node v i Hidden vector of

number

number

number

[0039] For these layers (l>1) of the neural network, there are different options to determine the important hidden features for each pair of nodes i and j. All hidden features can be used, which may lead to a more stable learning procedure, but implies that the intermediate steps are not human understandable. As an alternative, more discrete human understandable features can be used, which ensures that this step remains human understandable, but may lead to a more difficult learning procedure. One further possibility is to use the former option during training and the latter during inference. The best choice depends on the application and can be made on a case-by-case basis. In the last layer, for each node v i A softmax function is used to predict the probability of the property of . The softmax function is a known mathematical function available in commercially available mathematics packages. The softmax function converts a vector z of K real numbers into a probability distribution. Given training data that includes labels for some of the nodes, a neural network according to an embodiment of the present invention can be trained using, for example, a gradient descent algorithm with cross-entropy error. The training procedure optimizes the parameters of the model defined by the exemplary formulas herein such that the predicted labels approach the observed labels. The predicted labels are calculated using a message passing procedure defined by the exemplary formulas herein.

[0040] Thus, in a computational layer l of a neural network, for node i, a message is passed from each of its similar neighbors to produce the hidden vector for node i.

number

number

[0041] In an embodiment where the nodes represent patients, in the first layer, l=1, the input features are patient observations that are meaningful to the user (e.g., age, sex, lab results, genetic biomarkers, etc.). i,j are computed to determine which features are important. A user (e.g., a doctor) can then decide whether the learned important features are meaningful and provide feedback about them. However, when moving to the next layer, i.e., l>1, the input features of these layers are not hidden features.

number

[0042] Referring again to FIG. 1, the iterative updater 112 is programmed to iteratively revise the learned decisions and explanations based on feedback received from the device 110, in particular the user device by which the user provides feedback on the learned decisions and explanations. During the test procedure or application phase, for the test nodes, the method infers predictions and explanations (which nodes are similar to the test node with respect to which features), and the predictions and explanations are presented to the user. The user can then give feedback, for example, by correcting the explanation using expert knowledge. Based on the feedback, the model (i.e., the learned parameters) is updated. The self-explanatory decision engine 100 defines three types of feedback: Type I: correctness of identified important features, Type II: correctness of identified similar neighbors, and Type III: request to follow the predicted path.

[0043] For a test node, the method calculates the importance vector ω i,j which is the binary vector {0,1} Dwhere D is the number of features, 1 means that the feature is important for the patient pair i and j, 0 means that it is not. The determined important features (e.g., age, gender, and some test results) are presented to the doctor for the patient pair, so that the doctor can provide feedback in the form of a correction (e.g., the doctor may think that gender is not an important feature, but that another test result, given in the data but not selected by the method as an important feature for the patient pair, is actually important). In this case, the value (0 / 1) is then changed to provide the user feedback ω̂.

[0044] When Type I feedback is received, node (v i ,v j The important features of the pair are

number

number

[0045] For a test node, the method calculates the similarity z i,j This value is calculated for patient v i is its neighbour v j If the patient is calculated as similar to the patient, it is 1, otherwise it is 0. The selected similar patients are presented to the physician to provide Type II feedback. The modified value is

number

[0046] Similar to the handling of type I feedback, when type II feedback is collected, node v i Similar neighborhoods of

number

number

[0047] Parameter ρ F and ρ Z are the coefficients. After incrementally learning the new loss, the decisions and explanations are updated for further feedback and validation.

[0048] When a type III feedback is received, the self-explanatory decision engine 100 tracks the predicted path based on the learned model 108. Figure 3 shows the tracking procedure. Specifically, the decision is the predicted label of node v. The identified explanation is the similar neighboring nodes a and b, shown as the left panel 300 of Figure 3. If the feedback is to track how node a is predicted, the self-explanatory decision engine 100 asks the self-explanatory decision learner 104 to generate a predicted label and explanation for the suspected neighboring node a, as shown in the center panel 310 of Figure 3. To further track the prediction, for example the second-order neighbor node e, the prediction of e is tracked, as shown in the right panel 320 of Figure 3. The iteratively tracked predicted path shows how the decision is made by the self-explanatory decision engine, leading to a comprehensive understanding and validation of the generated decision of the AI ​​system. The boxes next to the nodes in Figure 3 represent the features of the node, and the important features are indicated by the dark boxes. The dashed circles in Figure 3 represent the n-th order neighbors of a test node v in the graph. Nodes a and b are direct neighbors, i.e. there is an edge connecting node v to a or b in the graph. Nodes c, d, and e are second order neighbors of node v. The nodes may represent, for example, patients in one practical application of the method.

[0049] Thus, Type III feedback inspires the application of the method to infer decisions for similar nodes. For example, when a doctor asks about patient v, the method infers the decision and explanation (which patients are similar in terms of which features) for patient v, shown as the left panel 300. The doctor may suspect that patient a is similar, so he asks about that patient, and the method infers the decision and explanation for patient a, shown as the middle panel 310. This allows the doctor to track the message passing process to find questionable or erroneous parts of the AI-driven decision. The doctor can also provide Type I or II feedback for similar patients.

[0050] Certain embodiments of the present application may be practically applied in the field of automated medicine, for example, to AI systems programmed for drug development and patient diagnostic assistance (e.g., patient outcome prediction). For example, for complex diseases, such as cancer, a drug may be effective for some patients but not for others, even for the same disease. Thus, it is recognized that it is important to identify susceptible patients when developing drugs, such as cancer vaccines and targeted therapy drugs. In addition, it is also known that it is important to find existing drugs that are effective for such patients. Prescribing drugs that do not help delays the treatment of patients. By design, the self-explanatory decision engine according to the embodiments of the present invention helps doctors iteratively identify effective drugs for patients more accurately and efficiently, while avoiding delays and other problems caused by inaccurate or poorly understood decisions. Then, the verified decision is forwarded to the smart hospital system to administer the drug. Such improved AI systems may also be used for simple diseases.

[0051] In this embodiment, the data source for input is a patient graph established using information extracted from an electronic health record (EHR) system, including patient, symptoms, examination, and gene sequence data and disease information. The label is whether the drug is effective for the patient. A self-explaining decision engine applying a method according to an embodiment of the present invention predicts the drug for the patient and provides an explanation of the prediction. The decision is iteratively improved through learning from the received feedback. Thus, the output of the self-explaining decision engine in this embodiment is a verified decision about the effective drug for the patient. As a preferably automated result, the final predicted effective drug is sent to the doctor or automatically provided or administered.

[0052] In another embodiment, the present invention can be practically applied in the field of automated utilities / smart cities, for example, to AI systems programmed for predictive diagnosis and maintenance of water / gas / electricity grids. For example, here, a self-explaining decision engine can be applied to improve AI-driven predictions and decisions for diagnosing problems in maintaining and improving city infrastructure such as water, gas, and electricity grids. Such AI systems for smart cities are programmed to realize automated or semi-automated control and maintenance. For each grid, there are many potential problems that can be monitored. For example, for a water grid, if a pipe is broken, this can cause a leak and affect water quality, ultimately endangering the life, health, and property of citizens. To monitor this without an AI system, a technician would need to check the pipes regularly, which is an expensive and time-consuming job. To make this more efficient, smart city platforms can use AI-driven systems for predictive maintenance so that they can more efficiently identify which pipes of the entire grid need to be maintained / repaired. Embodiments of the present invention further improve these AI systems by realizing self-explaining decisions that are more reliable and improve iteratively using feedback. In a water supply network, the data source is a graph of pipes. In the graph, the nodes are pipes, since it is desired in this AI task to predict the state of the pipes. If two pipes are connected, there is an edge between them. The features are the characteristics of the pipes, such as age, location, surrounding environment, and average water pressure collected by a sensor. The labels are binary, 1 if the pipe needs maintenance and 0 otherwise. A limited number of pipe labels can be provided by the technician. A self-explaining decision engine applying a method according to an embodiment of the present invention predicts which pipes are dangerous and explains the reasons for the prediction. The technician's knowledge and experience can be learned by the self-explaining decision engine from feedback received iteratively.Thus, the output in this application is a verified decision as to which pipes should be maintained. As a result, a final decision can be sent to a technician, for example as an automated message identifying the pipes that need to be replaced, to coordinate maintenance efforts. Embodiments of the present invention can be applied to gas and electricity grids, as well as water grids.

[0053] Embodiments of the present invention enable the following improvements over existing AI systems: During message aggregation, which here refers to the process by which information is collected from other nodes, a node interacts with various neighbors. To make this process more transparent and human understandable, some embodiments of the present invention use more discrete node features (importance vector ω i,j ), which varies depending on which neighbors a node is currently passing messages to. These node features are derived iteratively from user feedback. - Joint learning of discrete node weights and node feature weights is integrated into the learning goal of neural networks, e.g., graph neural networks (GNNs). This involves iterative guidance of the discrete attention that each node pays to each of its neighbors after observing feedback. Thus, the method achieves joint learning of similar nodes (e.g., patients) and their important features, while also inferring an explanation of which nodes are similar to which other nodes with respect to which features. In contrast to most existing explainable AI systems, which focus on post-hoc explanations and can therefore only learn to approximate why another AI system reached a certain outcome, the method according to an embodiment of the invention provides a self-explanatory decision engine that can explain itself, and outcomes are inferred from explanations, which increases trust and leads to improved accuracy (fidelity) of explanations. - Explanations according to embodiments of the present invention are human-centric, and self-explanatory decision learners learn explanations that are understandable to humans. In contrast, existing explainable AI systems focus on algorithmic explanations, such as minimizing predefined mathematical functions, which often introduces unnecessary confusion and thus further reduces users' trust in the AI ​​system. - An iterative updater is provided as an iterative method for improving explanations based on feedback from the device to further increase confidence and result in even more accurate explanations and decisions.

[0054] FIG. 4 shows a flow chart illustrating a method 400 for node classification according to an embodiment of the present invention. The method 400 is self-explainable and is updated based on feedback and validation received from the device. The method 400 comprises a first step 402 of defining a graph for the task at hand. The graph defined in the first step 402 may include one or more graph features, such as one or more definitions of nodes, one or more node edges, and one or more selections of node attributes. In a second step 404, the method 400 includes a step of implementing a multi-layer discrete attention mechanism during message passing to bias the process towards important features of each pair of selected neighboring nodes. Message passing involves using the nodes of the graph, and specifically using node information locally present at the node and / or incident edge, and further involves sending information only to adjacent nodes. The discrete attention mechanism implemented during message passing may be used to limit the number of most similar neighbors from all candidates of a node and / or the number of node features used, thereby discretizing the relationship between the node and its neighbors based on messages. As a result, users may be less susceptible to confusion from being exposed to too many relationships that are not adequately discretized based on strength of similarity.

[0055] In the multi-layer discrete attention mechanism in the second step 404, the features of each node are transformed into a more discrete representation, which changes depending on which neighboring node the message is passed to. This allows the message to focus on the important features of the pair of nodes. Furthermore, in the second step 404, each attention that a node pays to another node is discrete.

[0056] In a third step 406, the method 400 includes training parameters of the method 400 (e.g., parameters of an exemplary formula herein). During training, the parameters are learned by an exemplary formula herein to match the labels provided for some of the nodes as training data. After the parameters of the model used by the method are learned, the message passing is fully defined and can be used to infer a decision for any test node (e.g., test patient). In a fourth step 408, the method 400 includes performing inference such that each classification is self-explanatory. For example, during inference, nodes similar to the test node are determined in terms of which features are important for each pair, and message passing of the important features results in a prediction for the test node. In a fifth step 410, the method 400 includes performing inference such that each classification is self-explanatory. In a sixth step 412, the method 400 includes performing iterative explanation generation and validation, where feedback is received on the learned explanations, and the feedback is used to improve the learning of the explanations. For example, physician feedback correcting which patients the physician believes are similar and / or which features of the test patients are deemed important to the decision is used as Type I or II feedback to update the model. The updated model can then be used to recalculate explanations and message passing can be performed again to provide updated decisions for the test patients. Note that steps 402-410 need not be performed in the order listed, e.g., method parameters may be trained prior to message passing.

[0057] The embodiments of the present invention can be applied to improve any existing AI system where data and tasks can be modeled as inputs to a graph and a GNN. There are many technical applications where data and tasks can be represented as graphs and computed using GNNs. However, existing AI systems for such tasks are often black boxes. In contrast, to gain and increase end-user trust, the embodiments of the present invention improve the transparency of AI-driven decisions by explaining to end users in a human-understandable way how the AI ​​system reached a certain prediction. Our system is designed to meet these requirements. Examples of AI systems where data and tasks can be modeled as inputs to a graph and a GNN can be found in automated healthcare (e.g., drug development), automated utilities or smart cities (e.g., automated control of smart cities), automated public services (e.g., reemployment assistance), and security (e.g., insurance fraud detection, where explanations are organized to explain why a case is fraudulent).

[0058] While enabling several improvements in these various technical areas of the present application, embodiments of the present invention also provide improvements to explainable AI systems in general. In particular, embodiments of the present invention improve the capabilities of these explainable AI systems by providing self-explanatory as well as comprehensive, more human-understandable explanations, and by iterative learning of improved explanations and decisions. These advanced features bridge the gap between end-user demands and graph AI methods, increasing user trust in AI systems.

[0059] Human understandable explanations generated according to embodiments of the present invention are based on instances, attributes, and graph structures, and increase understandability by iteratively learning from feedback on explanations. This enables transparent decision-making and mimics the process humans go through when learning to make decisions. By providing more human understandable explanations and iteratively revising these explanations and decisions using feedback, embodiments of the present invention not only provide more accurate and understandable explanations, but also greatly improve users' trust in AI systems.

[0060] While the subject matter of the present disclosure has been illustrated and described in detail in the drawings and the foregoing description, such illustration and description should be considered as illustrative or exemplary, and not restrictive. Since the present invention is defined by the claims, any statements made herein characterizing the invention should be considered as illustrative or exemplary, and not restrictive. It will be understood that changes and modifications may be made by those skilled in the art within the scope of the appended claims, which may include any combination of features from the various embodiments described above.

[0061] The terms used in the claims should be considered to have the broadest reasonable interpretation consistent with the foregoing description. For example, the use of the articles "a" or "the" in introducing an element should not be construed as excluding a plurality of elements. Similarly, statements of "or" should be construed as inclusive, such that statements of "A or B" do not exclude "A and B," unless it is clear from the context or from the preceding description that only one of A and B is intended. Furthermore, statements of "at least one of A, B, and C" should be construed as one or more of the group of elements consisting of A, B, and C, and should not be construed as requiring at least one of each of the listed elements A, B, and C, regardless of whether A, B, and C are related as a category or otherwise. Moreover, statements of "A, B, and / or C" or "at least one of A, B, or C" should be construed as including any single entity from the listed elements, e.g., A, any subset from the listed elements, e.g., A and B, or the entire list of elements A, B, and C. [Explanation of symbols]

[0062] 102 Node Features and Graph Structure 104 Self-Explaining Decision Learner 106 Improved Decision 108 trained models 110 Feedback from Devices 112 Iterative updater 204 Explanation Learner 206 Decision Estimator

Claims

1. 1. A method for generating self-explanatory decisions in an artificial intelligence (AI) system, comprising: receiving or defining a graph for a task in the AI ​​system, the graph including a number of nodes connected by edges; performing message passing between the nodes of the graph, where a discrete attention mechanism is implemented during the message passing, whereby features of each node are converted into a discrete representation, and the discrete representation changes depending on which neighboring node a message is passed to; generating the self-explanatory decision for one of the nodes based on the message passing.

2. 2. The method of claim 1, wherein the discrete attention mechanism comprises using an importance vector for each pair of nodes indicating which features of the pair of distinct nodes are important, the message passing comprises passing the important features determined by the importance vector, and the self-explanatory decision comprises an explanation of which of the nodes are similar with respect to each of the features to the node about which the self-explanatory decision is made.

3. The method of claim 2, wherein model parameters are learned during a training procedure that uses observed features of a portion of the nodes as input to provide a learned model that is used for the message passing and for generating the self-explanatory decisions.

4. 4. The method of claim 3, further comprising performing iterative explanation generation and validation, wherein feedback is received on the explanations and used to improve the learned model.

5. 5. The method of claim 4, wherein the performing of iterative explanation generation and validation is based on feedback including at least one of: correctness of identified important features of one or more nodes, correctness of identified similar neighbors of the node, and a request to follow a predicted path.

6. The method of claim 4 , wherein improved self-explanatory decisions are determined by performing further message passing using the improved learned model.

7. 5. The method of claim 4, wherein the importance vector for the node for which the self-explanatory decision is made is refined based on the feedback to modify at least one of the nodes indicated as similar by the explanation and / or to modify at least one of the important features for at least one of the nodes indicated as similar by the explanation.

8. 2. The method of claim 1 , wherein the message passing comprises passing one of the features from each neighboring node in the graph of the node for which the self-explanatory decision is being made, determined by an importance vector that is different for each of the neighboring nodes and indicates in each case that the different features of the neighboring nodes are important to the node for which the self-explanatory decision is being made.

9. 9. The method of claim 8, further comprising the step of calculating a similarity between each of the neighboring nodes and the node for which the self-explanatory decision is made, the self-explanatory decision being based only on the significant features passed by the neighboring nodes that are determined to be similar.

10. The method of claim 1, wherein the discrete attention mechanism is a multi-layer discrete attention mechanism based on a Gumbel approximator or a sparsemax, and each layer of the multi-layer discrete attention mechanism calculates the closeness of the features of the nodes using hidden vectors calculated from the previous layer.

11. 2. The method of claim 1 , wherein the hidden vector for the node for which the self-explanatory decision is made comprises features determined by the message passing in a previous layer from neighboring nodes determined to be similar, and comprises different important features determined by a different importance vector for each of the neighboring nodes.

12. 13. The method of claim 1, wherein the AI ​​system is used in automated medicine and programmed for drug development and patient diagnosis assistance, the task is to predict a patient outcome or determine an effective drug, the decision is a predicted patient outcome or drug, and a human understandable explanation indicates one or more other similar patients determined by the discrete attention mechanism having features used to make the decision.

13. 2. The method of claim 1, wherein the AI ​​system is used in automated decision-making in a smart city for maintenance of a utility grid, the task being to identify control or maintenance targets, the decision being an automated identification of the control or maintenance targets or an associated automated control or maintenance action, and a human understandable explanation indicating one or more other portions of the utility grid determined by the discrete attention mechanism having features used to make the decision.

14. 1. A system for generating self-explanatory decisions in an artificial intelligence (AI) system, the system comprising: receiving or defining a graph for a task in the AI ​​system, the graph including a number of nodes connected by edges; performing message passing between the nodes of the graph, where a discrete attention mechanism is implemented during the message passing, whereby features of each node are converted into a discrete representation, and the discrete representation changes depending on which neighboring node a message is passed to; generating the self-explanatory decision for one of the nodes based on the message passing; 16. A system comprising: one or more hardware processors configured to effect execution of

15. 1. A tangible, non-transitory computer-readable medium having instructions that, when executed by one or more processors, perform a method for generating self-explanatory decisions in an artificial intelligence (AI) system, the method comprising: receiving or defining a graph for a task in the AI ​​system, the graph including a number of nodes connected by edges; performing message passing between the nodes of the graph, where a discrete attention mechanism is implemented during the message passing, whereby features of each node are converted into a discrete representation, and the discrete representation changes depending on which neighboring node a message is passed to; generating the self-explanatory decision for one of the nodes based on the message passing; 1. A tangible, non-transitory computer-readable medium comprising:

Images