Knowledge-based reasoning method and system with improved transparency properties

Abstract

This disclosure provides a computer-implemented method for improving the transparency of knowledge-based (KB) reasoning systems. According to one embodiment, the method includes the steps of: training a black-box AI system (110) to predict the probability that an input triple is true—original prediction probability—using triples in a factual knowledge base (122) as training data, where each triple identifies a subject entity, an object entity, and a predicate between the subject and object entities; The method includes using a trained model (124) of the black-box AI system (110) to determine one or more predicted paths that support the prediction of the input triples and / or one or more predicted paths that contradict the prediction of the input triples, and providing as output information derived therefrom as explanations for the determined one or more predicted paths or predictions along with an original prediction probability that the input triples are true.

Description

[Technical Field]

[0001] The present invention relates to a method and system for improving the transparency of knowledge-based (KB) reasoning systems. [Background technology]

[0002] According to the EU Artificial Intelligence (AI) Act and the General Data Protection Regulation (GDPR), transparency is a mandatory requirement for the use of AI technologies, especially in high-risk areas. Therefore, existing AI systems in these applications need solutions to improve their transparency, i.e., to generate correct and human-understandable explanations for their AI-driven predictions; otherwise, these AI systems will not be able to be deployed in the EU in the near future due to EU regulations. Similar regulations with comparable results exist in many other countries around the world.

[0003] Knowledge-based reasoning (KBI) is an important family of AI techniques that has been successfully applied to many domains with complex entities and relationships, such as digital health, a high-risk field due to its direct impact on human life and health. A typical example of a high-risk application is drug discovery. In the context of a drug discovery project, pharmaceutical companies must discover the side effects of their new drug along with existing drugs when patients take them simultaneously. Additionally, to optimize revenue or treat some emerging disease, such as COVID-19, pharmaceutical companies may explore possible new uses for their developed drug beyond its original scope of application. That is, the development may aim to learn whether drug / drug "a" can be used to treat disease "d": <drug a, treatment, ?disease d>.

[0004] KBI systems can give predictions, but what is even more important for pharmaceutical companies is knowing the reasons for the predictions, which makes it easier to (1) verify whether the predictions make sense based on existing knowledge, and (2) provide advice and prioritize physics experiments, which are often expensive and time-consuming.

[0005] Existing techniques learn large sets of single-neighbor facts (for reference, see, e.g., Carolin Lawrence, Timo Sztyler, and Mathias Niepert: "Explaining Neural Matrix Factorization with Gradient Rollback," in: Proceedings of AAAI, 2021, https: / / arxiv.org / abs / 2010.05516) or complex subgraphs (for reference, see, e.g., Hao Yuan et al., "On Explainability of Graph Neural Networks via Subgraph Explorations," in: Proceedings of the 38th International Conference on Machine Learning, 2021, https: / / arxiv.org / pdf / 2102.05152.pdf) as explanations, which are often difficult for humans to understand, especially users without an AI background. Not only do such implausible explanations fail to improve user trust in AI systems, on the contrary, they amplify users' doubts and hesitation, as acknowledged, for example, in Himabindu Lakkaraju and Osbert Bastani: "'How do I fool you?': Manipulating user trust via misleading black box explanations," in Proceedings of the AAAI / ACM Conference on AI, Ethics, and Society, 2020, https: / / arxiv.org / pdf / 1911.06473.pdf. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] WO 2021 / 244766 A1 [Non-patent literature]

[0007] [Non-Patent Document 1] Carolin Lawrence, Timo Sztyler and Mathias Niepert: "Explaining Neural Matrix Factorization with Gradient Rollback", in: Proceedings of AAAI, 2021, https: / / arxiv.org / abs / 2010.05516 [Non-patent document 2] Hao Yuan et al. “On Explainability of Graph Neural Networks via Subgraph Explorations”, in: Proceedings of the 38th International Conference on Machine Learning, 2021, https: / / arxiv.org / pdf / 2102.05152.pdf [Non-patent document 3] Himabindu Lakkaraju and Osbert Bastani: “‘How do I fool you?’: Manipulating user trust via misleading black box explanations,” in Proceedings of the AAAI / ACM Conference on AI, Ethics, and Society, 2020, https: / / arxiv.org / pdf / 1911.06473.pdf [Non-patent document 4] A. Garcia-Duran and M. Niepert: “KBLRN:End-to-End Learning of Knowledge Base Representations with Latent, Relational, and Numerical Features”, arXiv:1709.04676 Summary of the Invention [Problem to be solved by the invention]

[0008] In view of the above, there may be a desire to provide improved concepts for increasing the transparency of knowledge-based (KB) reasoning systems. [Means for solving the problem]

[0009] This need is addressed by the subject matter of the independent claims.

[0010] In accordance with one embodiment of the present disclosure, the aforementioned needs are addressed by a computer-implemented method for improving the transparency of knowledge-based (KB) reasoning systems, the method including the steps of: training a black-box AI system to predict the probability that an input triple is true—original prediction probability—using triples of a factual knowledge base as training data, where each triple identifies a subject entity, an object entity, and a predicate between the subject entity and the object entity; using the trained model of the black-box AI system to determine one or more prediction paths that support the prediction of the input triple and / or one or more prediction paths that contradict the prediction of the input triple; and providing as output information derived therefrom as explanations for the determined prediction path or paths or predictions along with the original prediction probability that the input triple is true.

[0011] The method may be performed by an apparatus that may be a computer system or part of a computer system. Accordingly, one aspect of the concepts proposed herein relates to an apparatus comprising one or more processors and one or more storage devices. The apparatus is configured to perform the above method. Another aspect relates to a corresponding computer program comprising instructions that, when executed by a computer, cause the computer to perform the above method. Another aspect relates to a corresponding non-transitory computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the above method.

[0012] According to embodiments of the present disclosure, the aforementioned challenges are addressed by providing a method and system for extracting (1) supportive predicted pathways, (2) contradictory predicted pathways, and (3) questionable unknowns that can be used to construct strongly predicted pathways to support or contradict the prediction. For example, for a prediction that drug "a" targets a new use for disease "d," the system disclosed herein may be configured to extract a set of predicted pathways from a drug knowledge base as descriptions, such as <drug a, target, protein p>, <protein p, interaction, protein q>, <protein q, pathway, disease d>.

[0013] Compared with existing XAI approaches, the method and system disclosed herein exhibit significant characteristics. In particular, the predicted pathways learned by the system according to the concepts of the present disclosure are consistent with the human reasoning process, thereby effectively resolving the limitations of existing XAI methods. Furthermore, the detected suspicious unknowns identify the most important unobserved features or relationships of entities to be examined for drug repurposing and drug-drug side effect detection.

[0014] In one embodiment, the present disclosure provides a system for improving the transparency of a knowledge-based reasoning system, where input preparation for the system may include an ML-based knowledge-based reasoning system and include facts to improve transparency. Given a specific prediction of the knowledge-based reasoning system, the system may be configured to detect supportive paths for the prediction with importance scores (by running a supportive prediction path generator), contradictory paths for the prediction with importance scores (by running a contradictory prediction path generator), and questionable facts for the prediction (by running a questionable unknown fact identifier). The questionable unknown fact identifier may be configured to filter out important unknown facts that can build strong prediction paths for the prediction. Thus, as outputs, the system may provide supportive prediction paths, contradictory prediction paths, and questionable unknown facts.

[0015] Therefore, embodiments of the method disclosed herein generate human-understandable explanations based on examples, attributes, and graph structure, and enhance understandability by learning predictive paths. This mimics the process of how humans learn to make decisions and explain them. Most existing XAI systems focus on single triple and subgraph explanations, which have been documented as difficult to understand. In contrast, the proposed method provides predictive paths to explain predictions, is human-centered, and resembles the human reasoning procedure for unknowns, making it easy for users to understand. According to embodiments, the method provides an effective way for end users to filter out suspicious unknowns, so that they know what to do to further verify the accuracy of the predictions.

[0016] According to one embodiment, it may be provided that paths between subject and object entities of input triples are extracted from a knowledge base, and that for each path a new predicted probability is calculated for the input triple, the calculation being based on the assumption that the respective path does not exist in the knowledge base. Each path extracted from the knowledge base may be assigned a importance score based on the respective difference between the original predicted probability and the new predicted probability.

[0017] According to one embodiment, it may be provided that the new predicted probabilities are smaller than the original predicted probabilities and that some paths with the highest importance scores are selected to be output as predicted paths that support the prediction of the input triple. Alternatively or additionally, it may be provided that the new predicted probabilities are larger than the original predicted probabilities and that some paths with the highest importance scores are selected to be output as predicted paths that contradict the prediction of the input triple.

[0018] According to one embodiment, it may be provided that the length of the paths extracted from the knowledge base is configurable by a user of the KB inference system.

[0019] For example, for an extracted path of length L=1, the extracted path consists of one training triple from the knowledge base. In this case, a new predicted probability for an input triple may be calculated as follows: First, the embedding vectors of each of the subject and object entities of the input triple from the trained model are updated with the scaled gradients of each of the subject and object entities of the one training triple corresponding to each path. Then, the trained black-box AI system with its updated embedding vectors is used as input to infer a new predicted probability for the input triple.

[0020] In the case of an extracted path of length L=2, the extracted path consists of a first training triple from the knowledge base linked with the subject entity of the input triple and a second training triple from the knowledge base linked with the object entity of the input triple, the first and second training triples being linked via the same object entity. In this case, a new predicted probability for the input triple may be calculated as follows: first, the embedding vector of the subject entity of the input triple from the trained model is updated with the scaled gradient of the subject entity of the first training triple, and the embedding vector of the object entity of the input triple from the trained model is updated with the scaled gradient of the object entity of the second training triple. The trained black-box AI system with its updated embedding vectors is then used as input to infer a new predicted probability for the input triple.

[0021] In the case of an extracted path of length L=3, the extracted path consists of a first training triple from the knowledge base linked with the subject entity of the input triple, a third training triple from the knowledge base linked with the object entity of the input triple, and a second training triple from the knowledge base linking the first and third training triples. In this case, the influence of the second training triple on the embedding vectors of the subject and object entities of the input triple may be taken into account by updating the embedding vector of the subject entity of the second training triple from the trained model with the scaled gradient of the subject entity of the second training triple.

[0022] For extracted paths of length L > 3, i.e., L = n and n > 3, the new predicted probabilities can be calculated equivalently. In this case, the extracted path consists of the first training triple in the knowledge base linked with the subject entity of the input triple, the nth training triple in the knowledge base linked with the object entity of the input triple, and some (i.e., n-2) intermediate training triples in the knowledge base linking the first and nth training triples. The influence of the intermediate training triples on the embedding vectors of the subject and object entities of the input triple may be taken into account by updating the embedding vector of the subject entity of the intermediate training triples from the trained model with the scaled gradient of the subject entity of the intermediate training triples.

[0023] According to one embodiment, sensitive entities of an input triple may be identified, and it may be specified that the sensitive entities include the subject and object entities of the input triple as well as n-th order neighbors of the subject and object entities in the knowledge base, where n is a predefined or configurable parameter. The probability of unknown triples among all identified sensitive entities may be predicted by using a black-box AI system.

[0024] According to one embodiment, it may be provided that unknown triples between all identified sensitive entities are added to the knowledge base as training triples. A graph search method may be used to identify paths between the subject and object entities of the input triples. From the identified paths, those paths containing unknown triples between sensitive entities may be selected and scored. Based on the scoring, some top-scoring paths may be removed, and unknown triples in the removed paths may be selected as suspect unknowns to be examined to confirm the prediction.

[0025] According to one embodiment, the factual knowledge base may be a knowledge base of drugs, diseases, proteins and related information. The triples in the factual knowledge base may be used as training data to train a black-box AI system to predict the probability that a given drug targets a given disease.

[0026] According to one embodiment, the factual knowledge base may be a knowledge base of drugs, diseases, proteins, and related information. The triples in the factual knowledge base may be used as training data to train a black-box AI system to predict the probability that a given first drug will interact with a given second drug.

[0027] According to one embodiment, the factual knowledge base may be a city knowledge base, including information about neighborhoods, schools, bus / metro lines, stores, entertainment, and possibly further related information. The factual knowledge base triples may be used as training data to train a black-box AI system to predict the probability that a given activity will meet a given goal.

[0028] There are several ways of designing and further developing the teaching of the present invention in an advantageous manner. For this, reference is made on the one hand to the dependent claims and on the other hand to the following description of preferred embodiments of the present invention by way of example, which are illustrated by the figures. In connection with the description of preferred embodiments of the present invention with the aid of the figures, preferred embodiments and further developments of the teaching as a whole will be explained. [Brief explanation of the drawings]

[0029] [Figure 1] FIG. 1 is a schematic overview of a transparency booster engine configured to output predicted paths and suspect unknown facts to determine the accuracy of predictions of a black-box AI system, in accordance with one embodiment of the present invention. [Figure 2] FIG. 1 illustrates a knowledge graph and predicted paths extracted therefrom in accordance with an embodiment of the present invention. [Figure 3] FIG. 1 illustrates a process for extracting predicted pathways from a drug knowledge graph according to one embodiment of the present invention. [Figure 4] FIG. 2 illustrates the operation of a suspect fact identifier in accordance with one embodiment of the present invention. [Figure 5] FIG. 1 illustrates an example of a suspect unknown from which a strong predictive path to prediction can be constructed according to one embodiment of the present invention. [Figure 6] 1 shows a schematic diagram of an example of an apparatus for carrying out at least one of the methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0030] Embodiments of the present disclosure provide a solution to improve the transparency of knowledge-based reasoning systems through automatically learning (1) supportive prediction pathways, (2) contradictory prediction pathways, and (3) questionable unknowns that can construct strong prediction pathways to support or contradict predictions. Therefore, methods and systems according to embodiments of the present disclosure can provide human-understandable explanations for why a knowledge-based reasoning system makes a certain prediction. The present invention has various application areas, such as drug repurposing or drug-drug side effect detection in digital health.

[0031] 1 schematically illustrates a system 100 for improving the transparency of a black-box AI system 110 for knowledge-based reasoning according to one embodiment of the present disclosure. For easier understanding of the concepts proposed herein, the operation mode of the system 100 will be introduced based on the example of a drug discovery run, aiming at drug retargeting, i.e., finding new uses for existing drugs beyond their original scope of application. However, the chosen application scenario should be understood to be merely illustrative, and it will be recognized by those skilled in the art that the proposed concepts can be applied to many different contexts and applications.

[0032] Referring to FIG. 1 , assume there is a drug a. A black-box AI system 110 predicts <drug a, treatment, disease d> with probability P. Embodiments of the present disclosure provide methods and systems for extracting predicted pathways with significant scores that support or contradict the prediction of the black-box AI system 110. Furthermore, embodiments of the present disclosure provide—additionally or alternatively—methods and systems for identifying suspicious unknowns by knowing which strongly predicted pathways can be constructed to support or contradict the prediction.

[0033] As illustrated in FIG. 1, a system 100 for improving transparency of an existing black-box AI system 110 includes an input preparation component 120 and a transparency booster engine 130, which is the actual component configured to improve transparency and provides detected predicted paths and / or filtered suspect facts as output 140, as will be described in detail below.

[0034] Input preparation According to one embodiment, the system 100 includes the following inputs:

[0035] Knowledge Base: The knowledge base 122 contains a (large) collection of triples, which serve as training data. Each triple is of the form <subject, predicate, object>, e.g. <protein q, pathway, disease d>, where the subject and object are entities that can be visualized as nodes in a knowledge graph, and the predicate specifies the relationship between the subject and the object, which can be seen as the label of the edge between the two nodes.

[0036] Black-box AI system: The black-box AI system 100 is a knowledge-based reasoning (KBI) system, e.g., the KBlrn framework, as described in A. Garcia-Duran and M. Niepert: "KBLRN: End-to-End Learning of Knowledge Base Representations with Latent, Relational, and Numerical Features," arXiv:1709.04676, which is incorporated herein by reference. The black-box AI system 100 provides predictions without an explanation of how the predictions are achieved. There is a desire to improve this transparency, i.e., to generate an explanation for each prediction, based on AI rules.

[0037] Training refers to training the black-box AI system 100 with the triples in the knowledge base 122 as training data to obtain a trained AI model 124 that can be used for prediction. During the training procedure, a scaled gradient for each training triple may be recorded. The scaled gradient is the multiplication of the learning rate and the gradient. The gradient is calculated as the partial derivative of the loss of the training triple with respect to the embedding vector of each participating entity. For example, to efficiently obtain the gradient, an existing method called gradient rollback may be utilized, as described in WO 2021 / 244766 A1, which is incorporated herein by reference. The gradient specifies the extent to which the training triple contributes to the embedding vector of the entity. The larger the gradient, the more important the training triple is to the entity. The recorded scaled gradient may be stored in a database, and is shown in FIG. 1 as factual gradient 126.

[0038] - Prediction: After the black-box AI system 110 has been trained, the trained model 124 can be used to make a prediction request 128 and to obtain a prediction in response. A prediction is expressed as the probability that an unknown triple is true. For example, a user of the system may want to know whether drug a can be used to treat disease d. Thus, the predicted triple <drug a, treatment, disease d> may be provided as an input triple to the AI ​​system 110, which then provides the probability P that the triple is true.

[0039] In this embodiment, the data flow as illustrated in Figure 1 is as follows: The knowledge base 122, which is formatted as a collection of triples, is input to the black-box AI system 110 to train the model, i.e., to obtain the trained AI model 124. During training, scaled gradients 126, represented as vectors, are saved for each triple. The trained model 124, the knowledge base 122, and the gradients 126 of each triple in the knowledge base 122 are input to the proposed transparency booster engine 130. During prediction, i.e., when a user requests a predicted triple, for example, formatted as <subject, predicate, object>, the black-box AI system 110 uses the trained model 124 to calculate the probability that the triple is true, and the proposed XAI system 100 provides an explanation for the prediction according to the input as output 140.

[0040] Supportive and contradictory predicted path generators 132, 134 As illustrated in the embodiment of FIG. 1, the transparency booster engine 130 comprises a supportive predicted path generator 132 and a contradictory predicted path generator 134, the operation of which is described in further detail below.

[0041] Black box AI system 110 * is the subject, and o * is the object, and r * For each triple t predicted with * = * ,r * ,o * For >, the generators 132, 134 may be configured to extract predicted paths with importance scores. The supportive generator 132 outputs predicted paths that support the prediction, i.e., triple t if the respective path does not exist in the training data. * ​The predicted probability of a path will decrease. Conversely, the contradiction generator 134 outputs predicted paths that contradict the prediction, i.e., without the respective path, the predicted probability would increase. The importance score of a path is proportional to the probability change. The larger the change in predicted probability after removing a path, the more important the path is to the respective prediction.

[0042] In particular, the predicted triple t * = * ,r * ,o * >, first s * and * It may be specified that paths between t and t are extracted from the knowledge base 122. For this purpose, any off-the-shelf graph search method can be used, such as depth-first search (DFS) and breadth-first search (BFS). The paths can be of length L=1, 2, 3... and may be a pre-implemented parameter or may be selected by the user of the system according to specific requirements. Then, for each existing path to be extracted from the knowledge base 122, it is determined that the predicted triple t * In other words, a new predicted probability may be calculated for the predicted triple t * For each path extracted from the knowledge base 122 with respect to * The predicted probability for

[0043] A path of length L=1 (i.e., s * is the subject / object, and o * is the object / subject, r is the predicate, r ≠ r * For a training triple t, the prediction generators 132, 134 calculate p(t * =T|t=F), where T and F are contractions for true and false, respectively. Thus, the above formula can be used to compute a training triple t (where the predicate r≠r * ) is false (corresponding to the absence of the respective path), * represents the probability that is true.​

[0044] According to one embodiment of the present disclosure, the above probabilities are calculated by multiplying the s of the training triple t by * and o * For this purpose, first, the s of the training triple t * and o * The scaled gradient of s may be looked up from the input data, and then s from the trained AI model 124 * and o * The embedding vector (in this specification

[0045]

number

[0046] ) may be updated as follows:

[0047]

number

[0048] In the formula δ(s * ,t) and δ(o * , t) are the training triples t, respectively * and o * The updated embedding vectors are then used as inputs to the black-box KBI system 110 to generate the predicted triple t * A new probability of f(Δ p ) where Δ p represents the difference between the new and original probabilities, where f(·) can be any monotonically increasing function.

[0049] Consider a path of length L=2. The path consists of two training triples, namely, the subject s and* Links t1 and object o * and t2, where the two triples t1 and t2 are linked by an arbitrary entity e. In this case, the prediction generators 132, 134 calculate p(t * =T|t1=F,t2=F), i.e., the predicted triple t under the condition that the training triples t1 and t2 are false (corresponding to the absence of the respective paths). * is true.

[0050] According to one embodiment of the present disclosure, the above probabilities are calculated by multiplying the s * Scaled gradients and training triple t2 * For this purpose, first, the s of the training triple t1 may be calculated based on the scaled gradient of * Scaled gradients and training triple t2 * The scaled gradient of s may be obtained from the input data. * and o * The embedding vector of follows the above equation, i.e.

[0051]

number

[0052] may be updated by calculating δ(s * ,t1) is the s of training triple t1 * represents the scaled gradient of δ(o * ,t2) is the training triple t2 * represents the scaled gradient of

[0053] Again, s * and o * The updated embedding vectors of t are used as inputs to the black-box KBI system 110 to generate the predicted triple t * If the new probability is less than the original probability, then the respective path is a supporting predicted path; otherwise, it is a contradictory predicted path.

[0054] Consider a path of length L=3. The path consists of three training triples, namely, the subject s, * Linked with t1, object o * The problem here is how to quantify the influence of t2. * and object o * Since there is no direct link to , it is not possible to compute a new embedding vector as in the case of short paths (L≦2) as described above. However, the training triple t2 is * and o * As the gradient of changes the point at which it is calculated, * and o * , which affects the embedding of . To calculate the impact of the entire path, embodiments of the present disclosure propose a probabilistic approach, which can be easily extended to calculate the impact of longer paths where L>3. In particular, according to one embodiment, the prediction generators 132, 134 use a probability p(t * =T, t1=F, t2=F, t3=F), i.e., the predicted triple t under the condition that the training triples t1, t2, and t3 are false (corresponding to the absence of the respective paths). * is true.

[0055] Probability can be decomposed based on the chain rule in probability theory: p(t * =T,t1=F,t2=F,t3=F) ≒p(t * =T|t1=F,t3=F)p(t1=F|t2=F)p(t3=F|t2=F)p(t2=F)

[0056] The first term p(t * =T|t1=F, t3=F) can be calculated as above (i.e., as in the case of a path of length L=2).

[0057] The second term p(t1=F|t2=F) may be calculated as follows: first, the scaled gradient of entity e1 of training triple t2 is determined, then the embedding vector of e1 is

[0058]

number

[0059] will be updated as.

[0060] The updated embedding is then used as input to the black-box KBI system 110 to calculate the probability of triple t1, which is p(t1=F|t2=F).

[0061] The third term, p(t3=F|t2=F), can be calculated in a similar manner to the second term.

[0062] The final term p(t2=F) can be calculated directly by the black-box KBI system 110.

[0063] For paths of length L>3, the predicted triple t * The new probability of may be calculated equivalently as above for a path of length L=3.

[0064] Suspicious unknown fact discriminator 136 This component may be implemented as part of the Transparency Boost Engine 130 and aims to screen unknown triples by knowing which strongly predictive paths can be constructed to support or contradict a given prediction.

[0065] According to one embodiment of the present disclosure, illustrated in FIG. 4, the identifier 136 may be configured to first identify sensitive entities of the predicted triple (as shown in FIG. 4 at step S400). The sensitive entities include the subject and object entities of the predicted triple, i.e., s * and o * In addition to the above, s in the training knowledge base 122 *and o * The parameter n may be a pre-implemented parameter or may be configurable to be specified by a user of the system according to specific requirements. In one embodiment, it may be provided that the user may also add additional entities as sensitive entities or remove some entities from the set.

[0066] Given a specified set of sensitive entities, in the next step (shown in step S402), the black-box KBI system 110 may be used to predict the probability of unknown triples among all the sensitive entities. Then (shown in step S404), these triples involving the sensitive entities are added to the training triples, and a graph search method is applied (step S406) to find the s * and o * , i.e., find a path between the subject and object entities of the predicted triples.

[0067] According to the illustrated embodiment, s * and * Given the set of paths found between and, it may be specified to select those paths from the set containing unknown triples between sensitive entities (as shown in step S408). In the next step, shown in step S410, these selected paths may be scored according to the methods described above for different path lengths in connection with the operation of the predictive graph generators 132, 134. A filter may then be applied which may be configured to filter out the K top-scoring paths or paths with a score above a given score threshold. Both the parameter K as well as the score threshold may be pre-implemented parameters or may be configurable to be specified by the user of the system according to specific requirements.

[0068] According to one embodiment, as shown in step S412, the identifier 136 may be configured to select unknown triples in the pruned paths and provide the selected unknown triples as suspected unknowns in output 140. These unknown triples can then be examined in detail to confirm the prediction.

[0069] 5 illustrates an example of suspected unknown fact identification in a drug knowledge graph. In the illustrated example, the possible prediction is prediction <drug a, treatment, disease d>. In the illustrated example, it is assumed that step S400 described above in connection with FIG. 4 revealed the entities "protein p2," "protein p3," and "protein q2" as sensitive entities of the predicted triple in question, including but not limited to these.

[0070] Furthermore, it is assumed that step S406 described above has revealed several paths between the subject of the prediction (i.e., "drug a") and the object of the prediction (i.e., "disease d"), including the path exemplarily illustrated in FIG. 5. The selection made in step S408 will include the path illustrated in FIG. 5, as it includes an unknown triple between sensitive entities, i.e., the triple <protein p3, interaction ?, protein q2>. Consequently, this path will participate in the scoring performed in step S410 and be eliminated if it receives a high score. In this case, in step S412, the unknown triple <protein p3, interaction ?, protein q2> with BS is selected as a suspected unknown and can be separately examined to confirm the prediction. If it is found that "protein p3" actually interacts with "protein q2," this will support the prediction that "drug a" actually treats "disease d."

[0071] FIG. 6 shows a schematic diagram of an example of an apparatus 600 for performing at least one of the methods described herein. The apparatus 600 includes one or more processors 620 and one or more storage devices 630. Optionally, the apparatus further includes one or more interfaces 610. For example, the apparatus 600 may be a computer system. The one or more processors 620 are coupled to the one or more interfaces 610 and the one or more storage devices 630. For example, the one or more processors 620 may be configured to provide the functionality of the apparatus 600, e.g., in conjunction with the one or more interfaces 610 (for exchanging information with one or more entities outside the apparatus 600) and / or the one or more storage devices 630 (for storing information, such as machine-readable instructions, knowledge graphs, machine learning models, or link prediction machine learning models). In various examples, the one or more processors 620 may be configured to execute machine-readable instructions of a computer program. In other words, the functionality of the apparatus 600 may be provided by the one or more processors executing the machine-readable instructions of the computer program.

[0072] For example, apparatus 600 may be used (e.g., configured via corresponding machine-readable instructions) to perform at least one of the methods described herein, such as the method illustrated in connection with Figures 2 and 3 or the method of Figure 4.

[0073] For example, interface 610 may include or correspond to a network interface and / or a device interface configured to be communicatively coupled to one or more other devices, such as processor 620 or sensors in the area of ​​interest. For example, one or more interfaces 610 may include a transmitter, a receiver, or a combination thereof (e.g., a transceiver) and may enable wired communication, wireless communication, or a combination thereof. Interface 610 may provide information to a user interacting with said interface 610.

[0074] For example, the one or more processors 620 may include or correspond to a digital signal processor (DSP), a graphical processing unit (GPU), and / or a central processing unit (CPU). The one or more processors 620 may be coupled to a storage device 630. The storage device 630 may include instructions (e.g., executable instructions), such as computer-readable instructions or processor-readable instructions. The instructions may include one or more instructions executable by a computer, such as by each of the one or more processors 620.

[0075] For example, the one or more storage devices 630 may include or correspond to non-volatile storage devices, such as magnetic disks, optical disks, or flash memory devices. The storage devices 630 may include both removable and non-removable memory devices. The storage devices 630 may be configured to store an operating system, an operating system image, applications, and program data. The storage devices 630 may store the supportive predicted path generator 132, the contradictory predicted path generator 134, the questionable fact identifier 136, the knowledge base 122, the trained model 124, and the fact gradient database 126. In certain embodiments, the storage devices 630 include tangible computer-readable media.

[0076] Some application scenarios of the methods and systems disclosed herein will be described below. Those skilled in the art will recognize that the described application scenarios are merely illustrative, and many other application scenarios in various different technical fields can be realized as well.

[0077] A possible use case for the concepts disclosed herein is in the field of drug repurposing. In drug discovery, drug repurposing is an important task for generating additional revenue for pharmaceutical companies and treating emerging diseases. Drug repurposing is the process of identifying new uses for approved drugs. For example, some pharmaceutical drugs were retargeted to COVID-19 in the early stages of the pandemic. To avoid expensive physical testing, AI models are used to computationally screen new targets. Biomedical researchers may want to know why AI models make specific predictions, and importantly, they may want an XAI system to help them select useful clues for why specific drugs can target specific diseases.

[0078] In the context of this use case, it may be specified that the data source is a knowledge base of drugs, diseases, proteins, and other related information. Typical predicates between the entities in each knowledge graph would include "target," "interaction," "pathway," etc. Using such knowledge base triples as training data, a black-box AI system can be trained to predict the probability that a given drug targets a given disease.

[0079] According to one embodiment of the present disclosure, a trained model of a black-box AI system may be used to determine one or more predicted pathways that support the prediction of the input triples and / or that contradict the prediction of the input triples. In other words, the system may learn predicted pathways for ML-driven predictions. For example, for a prediction that "drug a" targets "disease d," the system may provide a set of predicted pathways along with their respective importance scores, such as <drug a, target, protein p>, <protein p, interaction, protein q>, <protein q, pathway, disease d>, etc.

[0080] Accordingly, an embodiment of the present disclosure is a computer-implemented method for improving transparency of a knowledge-based (KB) reasoning system, comprising: training a black-box AI system to predict the probability that a specific drug targets a specific disease - the raw predicted probability - using the knowledge-based triples of drug, disease, protein and possibly further related information as training data; using a trained model of a black-box AI system to determine one or more predicted pathways that support the prediction that the drug will target the disease and / or one or more predicted pathways that contradict the prediction that the drug will target the disease; and providing as output the determined predicted path(s) together with the original predicted probabilities or information derived therefrom as explanations for the predictions.

[0081] The predicted pathway may be provided as an explanatory output of why the drug is predicted to target the disease. Additionally, suspected unknown facts may be generated according to the concepts described herein and provided as an output to be tested for substantiating and / or verifying the prediction. Thus, the above method may further include a step of using the output of one or more predicted pathways or information derived therefrom and / or the output of suspected unknown facts to perform testing for substantiating and / or verifying the prediction.

[0082] The detected predicted pathways identify why the black-box AI system predicts that a drug can treat a new disease. The identified suspicious unknowns provide biomedical researchers with more contextual information on what to check and investigate for new uses of the drug if they are unsure about the prediction. This can facilitate screening and physical examination.

[0083] Another potential use case for the concepts disclosed herein is in the field of adverse drug reaction detection. During drug discovery and use, a complex but important medical question is: What happens if a patient takes two compounded drugs at the same time? Crucially, pharmaceutical companies and physicians need to know whether two drugs interact with each other, potentially causing side effects. Testing the effects of drug combinations in a physical environment is impractical because there are many combinations and variables, making validation extremely expensive and time-consuming. A more efficient process is to screen with an AI model, which makes novel predictions about what will happen when two drugs are taken simultaneously. Biomedical researchers can then focus on physical testing to investigate promising combinations. However, AI models are often black boxes that simply provide results. Biomedical researchers may want to know why the AI ​​model makes the predictions it does, so that their confidence can be increased; more importantly, the explanation provides clues about how the two drugs interact with each other.

[0084] In the context of this use case—as in the previous one—it may be specified that the data source is a knowledge base of drugs, diseases, proteins, and other related information. Typical predicates between the entities in each knowledge graph would include "target," "interaction," "pathway," etc. Using such knowledge base triples as training data, a black-box AI system can be trained to predict the probability that two given drugs will interact with each other to cause an adverse reaction.

[0085] According to one embodiment of the present disclosure, a trained model of a black-box AI system may be used to determine one or more predicted pathways that support the prediction of an input triple and / or one or more predicted pathways that contradict the prediction of the input triple. In other words, the system may learn to extract predicted pathways of how a drug interacts with another drug. For example, for a prediction that "drug a" interacts with "drug b," the system may provide a set of predicted pathways along with their respective importance scores, such as <drug a, target, protein p>, <protein p, interaction, protein q>, <protein q, targeted by, drug b>, etc.

[0086] Accordingly, an embodiment of the present disclosure is a computer-implemented method for improving transparency of a knowledge-based (KB) reasoning system, comprising: training a black-box AI system to predict the probability that a specific first drug interacts with a specific second drug - the raw predicted probability - using the knowledge-based triples of drug, disease, protein and possibly further related information as training data; using a trained model of a black-box AI system to determine one or more predicted pathways that support a prediction that the first drug interacts with the second drug and / or one or more predicted pathways that contradict a prediction that the first drug interacts with the second drug; and providing as output the determined predicted path(s) together with the original predicted probabilities or information derived therefrom as explanations for the predictions.

[0087] The predicted pathway may be provided as an explanatory output of why or how a drug interacts with another drug. Additionally, suspected unknown facts, generated according to the concepts described herein, may be provided as an output to be examined to substantiate and / or verify the prediction. Thus, the above method may further include a step of conducting an examination to substantiate and / or verify the prediction using the output of one or more predicted pathways or information derived therefrom and / or the output of suspected unknown facts.

[0088] The detected predicted pathways identify why the black-box AI system predicts that two drugs will interact with each other. The identified suspicious unknowns provide biomedical researchers with more contextual information on what to check and investigate for new drug uses if they are unsure about the prediction. This can facilitate screening and physical testing.

[0089] Yet another possible use case of the concepts disclosed herein is digital government systems, particularly activity checking for policy optimization in digital government systems. In digital government systems, end users, i.e., city policymakers, often need to optimize city activities to meet specific goals, such as making the city livable. Identifying optimal activities and explaining the reasons for the activities is important for digital government systems so that policymakers know what to do and why. According to embodiments of the present disclosure, the explanation not only explains how the AI-driven system reaches a result, but also provides important features of the selected activity and advice on the rationality of the activity. Importantly, AI-driven results must be explained by EC regulations; otherwise, the system cannot be deployed and used in the EU.

[0090] In the context of this use case, a data source may be specified to be a city knowledge base, containing, for example, information about neighborhoods, schools, bus / subway routes, shops, and entertainment. Typical predicates between each knowledge graph entity would include "connects," "targets," "interacts," "attracts," etc. Using such knowledge base triples as training data, a black-box AI system can be trained to predict the probability that specific actions taken by city policymakers will meet or fulfill specific goals.

[0091] According to one embodiment of the present disclosure, a trained model of a black-box AI system may be used to determine one or more predicted paths that support the prediction of an input triple and / or one or more predicted paths that conflict with the prediction of an input triple. In other words, the system may learn predicted paths for ML-driven prediction activities. For example, for a prediction that "activity a" will satisfy "goal t," the system may provide a set of predicted paths along with their respective importance scores, such as <activity a, improve, bus line b>, <bus line b, connect, subway line s>, <subway line s, support, goal t>, etc.

[0092] Accordingly, an embodiment of the present disclosure is a computer-implemented method for improving transparency of a knowledge-based (KB) reasoning system, comprising: training a black-box AI system to predict the probability that a specific activity will meet a specific goal - the raw prediction probability - using as training data triples of a city knowledge base, e.g., including neighborhood, school, bus / metro line, shop, entertainment information, and possibly further related information; using a trained model of a black-box AI system to determine one or more predicted paths that support a prediction that the activity will meet the goal and / or one or more predicted paths that contradict a prediction that the activity will meet the goal; and providing as output the determined predicted path(s) together with the original predicted probabilities or information derived therefrom as explanations for the predictions.

[0093] The predicted path may be provided as an explanatory output of why the activity is predicted to meet the goal. Additionally, suspect unknown information about the city, generated in accordance with the concepts described herein by knowing which strongly predicted paths can be constructed to support or contradict the predicted activity, may be provided as an output to be examined to substantiate and / or verify the prediction. Thus, the above method may further include a step of using the output of one or more predicted paths or information derived therefrom and / or the output of suspect unknown facts to perform an examination to substantiate and / or verify the prediction.

[0094] The detected predicted path identifies why the black-box AI system predicts a new activity, for example by identifying why a certain area of ​​the city does not have more people (e.g., there are not enough schools, so the new activity is to build a new school). If the prediction is certain according to the explanation, the activity is added to the city's to-do list. Otherwise, the identified suspicious unknown information provides the end user with more contextual information on what to confirm and investigate for the new activity.

[0095] It should be noted that the above example of <activity a, meet?, goal t> prediction can be in many different contexts in various technology fields, such as in the context of predictive maintenance applications.

[0096] In summary, the present disclosure provides a method for determining the accuracy of a new prediction by extracting positive, i.e., supporting, and negative, i.e., contradictory, prediction paths from a knowledge base via gradient-based probabilistic inference. If a large number of contradictory prediction paths with significance scores above a configurable score threshold (e.g., above a configurable threshold) are detected, the new prediction is determined to be false. In addition, the present disclosure provides a method for identifying suspicious unknowns for determining the accuracy of a new prediction by constructing positive and contradictory prediction paths with large significance scores. This is an enabler for sorting promising factors from a large number of candidates for expensive and time-consuming testing.

[0097] Many modifications and other embodiments of the inventions set forth herein will come to mind to one skilled in the art to which these inventions pertain having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. It is to be understood, therefore, that the invention is not limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation. [Explanation of symbols]

[0098] 100 systems 110 Black Box AI System 120 Input Preparation Components 122 Knowledge Base 124 pre-trained AI models 126 Gradient of Facts 128 Forecast Requests 130 Transparency Booster Engine 132 Supportive Predictive Path Generator 134 Inconsistent Prediction Path Generator 136 Suspicious Unknown Fact Identifier 140 output 600 equipment 610 Interface 620 processor 630 Storage Devices

Claims

1. 1. A computer-implemented method for improving transparency of a knowledge-based (KB) reasoning system, comprising: training a black-box AI system (110) to predict the probability that an input triple is true—a raw prediction probability—using triples in a factual knowledge base (122) as training data, each triple identifying a subject entity, an object entity, and a predicate between the subject entity and the object entity; using a trained model (124) of the black-box AI system (110) to determine one or more predicted paths that support the prediction of the input triples and / or one or more predicted paths that conflict with the prediction of the input triples; providing as output the original prediction probability that the input triple is true together with the determined prediction path(s) or information derived therefrom as an explanation for the prediction; A method comprising:

2. extracting from the knowledge base (122) a path between the subject entity and the object entity of the input triple; calculating, for each path, a new predicted probability for the input triple, said calculation being based on the assumption that each path does not exist in said knowledge base (122); The method of claim 1 further comprising:

3. assigning an importance score to each path extracted from the knowledge base (122) based on the respective difference between the original predicted probability and the new predicted probability; The method of claim 2 further comprising:

4. selecting several paths with the highest importance scores, where the new predicted probabilities are smaller than the original predicted probabilities, to be output as predicted paths that support the prediction of the input triple; and / or selecting several paths with the highest importance scores, where the new predicted probabilities are greater than the original predicted probabilities, to be output as predicted paths that conflict with the prediction of the input triple; The method of claim 3, further comprising:

5. For an extracted path of length L=1, the extracted path consists of one training triple from the knowledge base (122), and the new predicted probability for the input triple is: updating the embedding vectors of the subject and object entities of each of the input triples from the trained model (124) with the scaled gradients of the subject and object entities of each of the training triples corresponding to each of the paths; using the trained black-box AI system (110) with the updated embedding vector as input to infer the new predicted probabilities for the input triples.

5. The method according to claim 2, wherein the calculated value is:

6. In the case of an extracted path of length L=2, the extracted path consists of a first training triple in the knowledge base (122) linked with the subject entity of the input triple and a second training triple in the knowledge base (122) linked with the object entity of the input triple, the first training triple and the second training triple being linked via the same object entity, and the new predicted probability for the input triple is: updating the embedding vector of the subject entity of the input triple from the trained model (124) with the scaled gradient of the subject entity of the first training triple, and updating the embedding vector of the object entity of the input triple from the trained model (124) with the scaled gradient of the object entity of the second training triple; using the trained black-box AI system (110) with the updated embedding vector as input to infer the new predicted probabilities for the input triples.

5. The method according to claim 2, wherein the calculated value is:

7. In the case of an extracted path of length L=3, the extracted path consists of a first training triple in the knowledge base (122) linked with the subject entity of the input triple, a third training triple in the knowledge base (122) linked with the object entity of the input triple, and a second training triple in the knowledge base (122) linking the first training triple and the third training triple, taking into account the influence of the second training triples on the embedding vectors of the subject and object entities of the input triples by updating the embedding vectors of the subject entities of the second training triples from the trained model (124) with the scaled gradients of the subject entities of the second training triples.

5. The method of claim 2, further comprising:

8. identifying sensitive entities of an input triple, the sensitive entities including the subject and object entities of the input triple as well as n-th order neighbors of the subject and object entities in the knowledge base (122), where n is a predefined or configurable parameter; predicting the probability of unknown triples between all identified sensitive entities by using the black-box AI system (110); 8. The method of claim 1, further comprising:

9. adding the unknown triples between all identified sensitive entities to the knowledge base (122) as training triples; using a graph search method to identify a path between the subject entity and the object entity of the input triple; selecting from said identified paths those paths that include said unknown triples between sensitive entities; The method of claim 8, further comprising:

10. scoring the selected paths and eliminating some of the top scoring paths; selecting the unknown triples in the pruned paths as suspect unknowns to be checked for confirmation of the prediction; 10. The method of claim 9, further comprising:

11. 11. The method of claim 1, wherein the knowledge base of facts is a knowledge base of drugs, diseases, proteins and related information, and triples in the knowledge base of facts are used as training data to train the black-box AI system to predict the probability that a given drug targets a given disease.

12. 11. The method of claim 1, wherein the knowledge base of facts is a knowledge base of drugs, diseases, proteins and related information, and triples in the knowledge base of facts are used as training data to train the black-box AI system to predict the probability that a given first drug interacts with a given second drug.

13. 11. The method of claim 1, wherein the knowledge base of facts (122) is a city knowledge base containing information about neighborhoods, schools, bus / metro lines, shops, entertainment, and possibly further related information, and wherein triples in the knowledge base of facts (122) are used as training data to train the black-box AI system (110) to predict the probability that a given activity will meet a given goal.

14. 14. An apparatus comprising one or more processors and one or more storage devices, the apparatus being configured to perform the method of any one of claims 1 to 13.

15. A computer program comprising instructions that, when said computer program is executed by a computer, cause said computer to carry out the method of any one of claims 1 to 13.

Images