Dynamic reasoning method and system based on language model and knowledge graph

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to a dynamic reasoning method and system based on a language model and a knowledge graph. The method comprises the following steps: step 1, encoding an input QA context text by using a language model encoder; reasoning the knowledge graph by using a graph encoder; 2, iteration is carried out in a hierarchical aggregation module, a joint reasoning module and a dynamic pruning module, so that multi-modal fusion is carried out on the features of the QA context text and the knowledge graph; and step 3, calculating a final output answer. The dynamic reasoning method and the dynamic reasoning system have better answer accuracy, have the advantages of convenience and wide application range, can provide new thoughts and choices for the multi-modal fusion technology in the field of natural languages, and have good application prospects.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a dynamic reasoning method and system based on language model and knowledge graph. Background Art

[0002] Question answering system is an important task in natural language processing. It not only needs to obtain relevant knowledge in the context of question and answer (QA), but also needs to use knowledge from related fields to reason about entities and the relationships between entities. Pre-trained language models (LMs) such as BERT and GPTS implicitly encode a lot of knowledge through unstructured text. As the basis of QA systems, language models can capture the situational constraints of context. When targeting downstream QA tasks in specific fields, language models show strong learning ability, so fine-tuning large language models has become the mainstream model for QA tasks, but these models lack the representation of relationships between concepts, which is crucial for complex reasoning. Large knowledge graphs (KGs), such as UMLS, Freebase, and ConceptNet, represent structured knowledge through triplets consisting of entities and relationships, providing explainable reasoning. According to existing work, knowledge graphs play an important role in structured reasoning and QA tasks. However, the knowledge covered by existing knowledge graphs is limited. Therefore, it is very important to study the technology of using knowledge graphs as external knowledge to enhance the knowledge representation of language models.

[0003] The fusion technology of knowledge graph and language model is widely used in the fields of entity linking, KG completion and recommendation system. Although the multimodal fusion technology based on deep learning has achieved good results in question-answering tasks, there are still some problems in fields with high accuracy requirements (such as medicine), such as factual errors. Therefore, the algorithm still needs to be further improved.

[0004] Previous methods proposed to improve reasoning by using large language models with extensive knowledge coverage and structured knowledge graphs (Feng Y, Chen X, Lin BY, et al. Scalable multi-hop rela-tional reasoning for knowledge-aware question answering [J]. arXiv preprint arXiv: 2005.00646, 2020; Lin BY, Chen X, Chen J, et al. Kagnet: Knowledge-aware graph networks for commonsense reasoning [J]. arXiv preprint arXiv: 1909.02151, 2019.). Given the QA context and KG, the existing LM+KG reasoning methods have the following main problems: First, researchers retrieve triples related to the question through string matching or semantic matching to construct a knowledge subgraph. In order to obtain enough relevant knowledge, the subgraph will inevitably contain many noisy nodes. Then, the designed GNN module is used to constrain and reason on the knowledge subgraph. However, these methods only model KG in isolation, encode the two modalities separately, and either there is no interaction between QA context text and knowledge graph, or there is only limited static interaction, and the model fuses two different representations at output for prediction, or uses one to enhance the input of the other. Therefore, existing methods may limit the potential of the model to model and learn deep interactions between text and KG.

[0005] In summary, multimodal fusion is one of the indispensable technologies in QA tasks. How to solve the interaction between the two modal information to further improve the accuracy of the answer is still an important research topic in this field. Summary of the invention

[0006] In response to the problems of the prior art, the present invention provides a dynamic reasoning method and system based on a language model and a knowledge graph.

[0007] A dynamic reasoning method based on a language model and a knowledge graph includes the following steps:

[0008] Step 1: Use the language model encoder to encode the input QA context text to obtain the encoded QA context representation; use the graph encoder to infer the knowledge graph to obtain the representation of the node entity;

[0009] Step 2, iterate and execute the following steps at least once:

[0010] Step 2.1: Input the node entity representation and QA context representation into the hierarchical aggregation module to capture the corresponding hierarchical features in the QA context and entity, and then integrate them through the multi-head attention mechanism to obtain the enhanced QA context representation;

[0011] Step 2.2: Input the node entity representation and the enhanced QA context representation into the joint reasoning module, and use the bidirectional attention mechanism to connect the two encoders to capture the fine-grained interaction between different modal information to obtain the updated node entity representation and the updated QA context representation.

[0012] Step 2.3: Input the updated node entity into the dynamic pruning module. Under the guidance of the LM-to-KG attention value calculated in step 2.2, delete the nodes that are not related to the QA context text to obtain the representation of the node entity after dynamic pruning.

[0013] Step 3: Based on the node entity representation and QA context representation finally obtained in step 2, the final output answer is calculated.

[0014] Preferably, in step 1, the language model encoder is selected from BERT, RoBERTa-large, AristoRoBERTa, BioBERT or SapBERTBase.

[0015] Preferably, in step 1, the process of using the graph encoder to reason about the knowledge graph includes:

[0016] Step 1.1, use BERT’s average pooling to obtain the initial representation of the node;

[0017] Step 1.2, after the graph encoder GNN layer, the representation of the node is updated by iterative information transmission of neighboring nodes on the graph:

[0018]

[0019] in, or is a set of entity nodes in the knowledge graph;

[0020] Specific Calculated according to the following formula:

[0021]

[0022] Among them, N i is the set of neighbor nodes of node i, m ji is the information transmitted from node j to node i, α ji is the weight of information transmission between control nodes, f n It is a 2-layer neural network function;

[0023] m ji The calculation of is as follows:

[0024] r ji =f r (e ji ,u i ,u j )

[0025]

[0026] Among them, f r is a 2-layer multilayer perceptron, f m is a linear transformation, e ji Indicates the type of edge (i, j), u i Indicates the type of node i, u j Indicates the type of node j;

[0027] α ji The calculation of is as follows:

[0028]

[0029] α ji =softmax(γ ji )

[0030] Among them, f j and f i They are all linear transformations.

[0031] Preferably, step 2.1 specifically includes:

[0032] Step 2.1.1, use a two-layer perceptron network to extract the hierarchical features of the entity nodes in the QA context text and knowledge graph:

[0033]

[0034] in, or is the representation of the node entity, or It is a QA context representation;

[0035] Step 2.1.2, use the multi-head attention mechanism to update the representation of the QA text and obtain the enhanced QA context representation:

[0036]

[0037] in, or is the QA context representation, d is Dimension.

[0038] Preferably, step 2.2 specifically includes:

[0039] Step 2.2.1, input node entity representation and enhanced QA context representation Construct a new matrix:

[0040]

[0041] Among them, f s is a linear transformation, is element-wise multiplication, [;] is vector concatenation across rows;

[0042] Step 2.2.2, row by row Normalize and get the KG-to-LM attention map:

[0043]

[0044] Step 2.2.3, column by column Normalize and get the LM-to-KG attention mapping and attention representation:

[0045]

[0046] in, represents the matrix Kronecker product, It is the QA text representation modified according to the knowledge graph representation. It is a knowledge graph representation modified according to the features of the QA text;

[0047] Step 2.2.4, concatenate the initial feature representation and the information representation after interactive fusion, and update the features of the two modalities in the following way:

[0048]

[0049]

[0050] Among them, f q , f v is a linear transformation.

[0051] Preferably, step 2.3 specifically includes:

[0052] Define a node retention rate K to determine the number of nodes to be retained; N is the number of nodes in the knowledge graph when inputting the model.

[0053] Select according to the attention value of LM-to-KG The highest nodes constitute the knowledge graph subgraph predicted by the model, that is, the representation of the node entity after dynamic pruning:

[0054] Preferably, step 3 specifically includes:

[0055] Step 3.1, calculate the credibility score of each candidate answer:

[0056] p=(a|q,G)=MLP(Q;V)

[0057] Where a is the candidate answer, q is the question text, G is the retrieved knowledge graph subgraph, Q is the QA context representation finally obtained in step 2, and V is the node entity representation finally obtained in step 2;

[0058] In step 3.2, use the softmax function to normalize all questions to get the final probability, and take the option with the highest probability value as the final output answer.

[0059] The present invention also provides a dynamic reasoning system based on a language model and a knowledge graph, comprising:

[0060] Input module, used to input QA context text;

[0061] Storage module, used to store knowledge graphs;

[0062] The question-answering module is used to obtain the final output answer according to the above-mentioned dynamic reasoning method based on the language model and knowledge graph.

[0063] Preferably, the question-answering module includes: a language model encoder, a graph encoder and at least one layer of cross-modal interaction fusion module, and the cross-modal interaction fusion module includes a hierarchical aggregation module, a joint reasoning module and a dynamic pruning module.

[0064] The present invention also provides a computer-readable storage medium, on which is stored: a computer program for implementing the above-mentioned dynamic reasoning method based on a language model and a knowledge graph, or for implementing the above-mentioned dynamic reasoning system based on a language model and a knowledge graph.

[0065] The present invention improves the use of KG and the hierarchical interaction between LM and KG in the question answering system, thereby providing an improved algorithm and system. After adopting the technical solution of the present invention, the following beneficial effects can be achieved:

[0066] 1. In the method and system of the present invention, the interaction between LM and KG is more comprehensive and in-depth, which can effectively improve the accuracy of the answer. In particular, the present invention has a significant effect on improving the accuracy of the answer on the medical data set.

[0067] 2. The present invention can be used directly and is very convenient. It allows users to integrate the modules of the invention into the existing fusion model in the pre-trained language model and the knowledge graph fusion model, thereby improving the performance of the model. Moreover, this modular design enhances the versatility of the invention and can be applied to any tasks and application fields that require KG as background knowledge to enhance LM.

[0068] 3. The modular design of the present invention facilitates the subsequent upgrade and improvement of the algorithm. According to different task requirements, the performance of the entire model can be quickly updated by modifying or replacing specific modules.

[0069] In short, the invention can improve the accuracy of answers, and has the advantages of convenience and a wide range of applications. It can provide new ideas and options for multimodal fusion technology in the field of natural language and has good application prospects.

[0070] Obviously, according to the above contents of the present invention, in accordance with common technical knowledge and customary means in the art, without departing from the above basic technical ideas of the present invention, other various forms of modification, replacement or change may be made.

[0071] The above contents of the present invention are further described in detail below through specific implementation methods in the form of embodiments. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following examples. All technologies realized based on the above contents of the present invention belong to the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 This is a diagram of the model architecture in the method of Example 1. DETAILED DESCRIPTION

[0073] It should be noted that the algorithms of data collection, transmission, storage and processing steps not specifically described in the embodiments, as well as the hardware structure, circuit connection, etc. not specifically described can all be implemented through the contents disclosed in the prior art.

[0074] Example 1 Dynamic reasoning method based on language model and knowledge graph

[0075] The algorithm structure diagram used in this embodiment is as follows Figure 1 The model takes a question-answer (question-answer) and a retrieved knowledge graph subgraph as input and outputs a probability value to measure the similarity between each option and the answer.

[0076] Specifically, the Multiple Choice Question Answering task in the present invention is simply expressed as Y = {Q, A}, where Q represents a set of questions and A represents a set of candidate answers, i.e., the options of the multiple choice questions. For a given QA context text, there are multiple candidate answers {a1, a2, ..., a n}, n is the number of candidate answers. Linking the entities involved in the question and options to the external knowledge graph yields an initial subgraph G = (V, R), where V is the set of entity nodes related to the QA text, R is the set of relationship classes formed by the entities in V, and E = V × R × V defines a set of edges connecting the above nodes. The goal of the question-answering task is to calculate the similarity score between the question and the candidate answers, and select the option with the highest similarity score as the correct answer.

[0077] The model structure diagram of this embodiment is as follows Figure 1 As shown, the following steps are included:

[0078] Step 1: Use the language model encoder to encode the input QA context text to obtain the encoded QA context representation; use the graph encoder to infer the knowledge graph to obtain the representation of the node entity.

[0079] The language model encoder may adopt any one of the prior art, for example, RoBERTa-large, AristoRoBERTa or SapBERTBase.

[0080] For a given QA context (containing questions and candidate answers), and obtain their representation using a pre-trained language model encoder.

[0081] Q=LM-layer({w1,w2,...,w M})

[0082] in is the last hidden layer vector of each partition unit in the problem, and Q represents the encoded QA context representation.

[0083] For the entity node set in the knowledge graph Each entity node v i ∈V can be seen as a tag sequence. The process of using the graph encoder to reason about the knowledge graph includes:

[0084] Step 1.1, use BERT’s average pooling to obtain the initial representation of the node;

[0085] Step 1.2, after the graph encoder GNN layer, the representation of the node is updated by iterative information transmission of neighboring nodes on the graph:

[0086]

[0087] in, or is a set of entity nodes in the knowledge graph;

[0088] The GNN network of this embodiment is an improved graph attention network. Specifically, Calculated according to the following formula:

[0089]

[0090] Among them, N i is the set of neighbor nodes of node i, m ji is the information transmitted from node j to node i, α ji is the weight of information transmission between control nodes, f n It is a 2-layer neural network function;

[0091] m ji The calculation of is as follows:

[0092] r ji =f r (e ji ,u i ,u j )

[0093]

[0094] Among them, f r is a 2-layer multilayer perceptron, f m is a linear transformation, e ji Indicates the type of edge (i, j), u i Indicates the type of node i, u j Indicates the type of node j;

[0095] α ji The calculation of is as follows:

[0096]

[0097] α ji =softmax(γ ji )

[0098] Among them, f j and f i They are all linear transformations.

[0099] The above formula is used to obtain the embedded set of nodes At this time, the node information has integrated the information of neighboring nodes in the graph, and the node representation is richer.

[0100] Step 2, iteratively execute steps 2.1 to 2.3 at least once (in this embodiment, the maximum number of times is set to 30 times, and the preferred number of iterations on the three data sets is {5, 6, 3} respectively).

[0101] QA context representation after encoder encoding and the representation of node entities New information has been integrated, but there is a gap between the two different sources of information, which makes it difficult to use it directly for reasoning. The existing model uses the same KG representation to interact with multiple layers of QA context text, which ignores the gaps between and within layers. In order to promote the fusion of the two information, perform step 2.1:

[0102] Step 2.1: Input the node entity representation and QA context representation into the hierarchical aggregation module to capture the corresponding hierarchical features in the QA context and entity, and then integrate them through the multi-head attention mechanism to obtain the enhanced QA context representation;

[0103] Step 2.1 specifically includes:

[0104] Step 2.1.1, use a two-layer perceptron network to extract the hierarchical features of the entity nodes in the QA context text and knowledge graph:

[0105]

[0106] in, or is the representation of the node entity, or It is a QA context representation;

[0107] Step 2.1.2, use the multi-head attention mechanism to update the representation of the QA text and obtain the enhanced QA context representation:

[0108]

[0109] in, or is the QA context representation, d is Through step 2.1, the model uses dynamic inter-layer and intra-layer interactions to integrate the relationships between nodes and edges in the knowledge graph into the QA context representation, so that external knowledge information can enhance the QA context representation.

[0110] To further fuse the QA context representation and the node entity representation to obtain their updated representations, perform step 2.2:

[0111] Step 2.2: Input the node entity representation and the enhanced QA context representation into the joint reasoning module, and use the bidirectional attention mechanism to connect the two encoders to capture the fine-grained interaction between different modal information to obtain the updated node entity representation and the updated QA context representation.

[0112] Step 2.2 specifically includes:

[0113] Step 2.2.1, input node entity representation and enhanced QA context representation Construct a new matrix:

[0114]

[0115] Among them, f s is a linear transformation, is element-wise multiplication, [;] is vector concatenation across rows;

[0116] Step 2.2.2, row by row Normalize and get the KG-to-LM attention map:

[0117]

[0118] Step 2.2.3, column by column Normalize and get the LM-to-KG attention mapping and attention representation:

[0119]

[0120] in, represents the matrix Kronecker product, It is the QA text representation modified according to the knowledge graph representation. It is a knowledge graph representation modified according to the features of the QA text;

[0121] Step 2.2.4, concatenate the initial feature representation and the information representation after interactive fusion, and update the features of the two modalities in the following way:

[0122]

[0123] Among them, f q , f v is a linear transformation.

[0124] The attention values ​​calculated in the joint reasoning module reflect the importance of different nodes in the knowledge graph to the QA context text. Therefore, step 2.3 is further performed:

[0125] Step 2.3: Input the updated node entity into the dynamic pruning module. Under the guidance of the LM-to-KG attention value calculated in step 2.2, delete the nodes that are not related to the QA context text to obtain the representation of the node entity after dynamic pruning.

[0126] Step 2.3 specifically includes:

[0127] Define a node retention rate K to determine the number of nodes to be retained; N is the number of nodes in the knowledge graph when inputting the model.

[0128] Select according to the attention value of LM-to-KG The highest nodes constitute the knowledge graph subgraph predicted by the model, that is, the representation of the node entity after dynamic pruning:

[0129] Step 3: Based on the node entity representation and QA context representation finally obtained in step 2, the final output answer is calculated.

[0130] Step 3 specifically includes:

[0131] Step 3.1, calculate the credibility score of each candidate answer:

[0132] p=(a|q,G)=MLP(Q;V)

[0133] Where a is the candidate answer, q is the question text, G is the retrieved knowledge graph subgraph, Q is the QA context representation finally obtained in step 2, and V is the node entity representation finally obtained in step 2;

[0134] In step 3.2, use the softmax function to normalize all questions to get the final probability, and take the option with the highest probability value as the final output answer.

[0135] Example 2 Dynamic Reasoning System Based on Language Model and Knowledge Graph

[0136] The system of this embodiment includes:

[0137] Input module, used to input QA context text;

[0138] Storage module, used to store knowledge graphs;

[0139] The question-answering module is used to obtain the final output answer according to the dynamic reasoning method based on the language model and knowledge graph described in Example 1.

[0140] Among them, the question-answering module includes: a language model encoder, a graph encoder and a cross-modal interaction fusion module that is repeatedly stacked N times (according to the method of Example 1, the number of repeated stackings in this embodiment on three data sets is {5, 6, 3} respectively), and the cross-modal interaction fusion module includes a hierarchical aggregation module, a joint reasoning module and a dynamic pruning module.

[0141] The technical solution of the present invention is further illustrated by experiments below.

[0142] Experimental Example 1 Comparison of the answer accuracy of the method of the present invention and the existing method

[0143] 1. Experimental Methods

[0144] The text corpus can provide complementary knowledge in addition to the knowledge graph. The present invention focuses on improving the use of KG and the hierarchical interaction between LM and KG. Therefore, models related to LM and KG are used as comparison models. This experimental example verifies the effectiveness of the present invention on the common sense dataset CommonsenseQA internal split dataset, OpenBookQA test dataset, and biomedical dataset MedQA-USMLE, and compares the performance of the present method with different benchmark models.

[0145] The description of each experimental group is as follows:

[0146] The DLM-KG model corresponding to the method and system of the present invention in this experimental example is based on an existing language model encoder as a benchmark model and is executed according to the method of Example 1 and the system of Example 2.

[0147] The baseline model or additional model algorithm of the control group is implemented according to the literature. Model and its literature include

[0148] 1. For CommonsenseQA

[0149] ROBERTa-large (w / o KG): Liu Y, Ott M, Goyal N, et al. Roberta: A robustly optimized bert pretraining approach [J]. ArXivpreprint, 2019, abs / 1907.11692.

[0150] +RGCN:Schlichtkrull M,Kipf T.N,Bloem P,et al.Modeling relational datawith graph convolutionalnetworks[C].The semantic web:15th internationalconference,ESWC 2018,Heraklion,Crete,Greece,June 3–7,2018,proceedings15,2018,593-607.

[0151] +GconAttn:Wang X,Kapanipathi P,Musa R,et al.Improving naturallanguage inference using externalknowledge in the science questions domain[C].Proceedings of the AAAI conference on artificialintelligence,2019,7208-7215.

[0152] +KagNet:Lin B.Y,Chen X,Chen J,et al.KagNet:Knowledge-aware graphnetworks for commonsensereasoning[C].Proceedings of the 2019Conference onEmpirical Methods in Natural LanguageProcessing and the 9th InternationalJoint Conference on Natural Language Processing(EMNLP-IJCNLP),Hong Kong,China,2019,2829-2839.

[0153] +RN:Santoro A,Raposo D,Barrett D.G,et al.A simple neural networkmodule for relational reasoning[C].Advances in Neural Information ProcessingSystems,2017,4967-4976.

[0154] +MHGRN: Feng Y, Chen X, Lin B.Y, et al. Scalable multi-hop relational reasoning for knowledge-aware question answering[C]. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing(EMNLP), Online, 2020, 1295-1309.

[0155] +QA-GNN: Yasunaga M, Ren H, Bosselut A, et al. QA-GNN: Reasoning with language models and knowledge graphs for question answering[C]. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, 2021, 535-546.

[0156] +JointLK: Sun Y, Shi Q, Qi L, et al. JointLK: Joint reasoning with language models and knowledge graphs for commonsense question answering[C]. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, 2022, 5049-5060.

[0157] 2. For OpenBookQA

[0158] Fine-tunedLMs(w / o KG):Clark P,Etzioni O,Khot T,et al.From‘f’to‘a’onthe ny regents science exams:An overviewof the aristo project[J].Ai Magazine,2020,41(4):39-53.

[0159] +RGCN:Schlichtkrull M,Kipf T.N,Bloem P,et al.Modeling relational datawith graph convolutional networks[C].The semantic web:15th internationalconference,ESWC 2018,Heraklion,Crete,Greece,June 3–7,2018,proceedings15,2018,593-607.

[0160] +GconAttn:Wang X,Kapanipathi P,Musa R,et al.Improving naturallanguage inference using external knowledge in the science questions domain[C].Proceedings of the AAAI conference on artificial intelligence,2019,7208-7215.

[0161] +RN:Santoro A,Raposo D,Barrett D.G,et al.A simple neural networkmodule for relational reasoning[C].Advances in Neural Information ProcessingSystems,2017,4967-4976.

[0162] +MHGRN: Feng Y, Chen X, Lin B.Y, et al. Scalable multi-hop relational reasoning for knowledge-aware question answering[C]. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, 2020, 1295-1309.

[0163] +QA-GNN: Yasunaga M, Ren H, Bosselut A, et al. QA-GNN: Reasoning with language models and knowledge graphs for question answering[C]. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, 2021, 535-546.

[0164] +JointLK: Sun Y, Shi Q, Qi L, et al. JointLK: Joint reasoning with language models and knowledge graphs for commonsense question answering[C]. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, 2022, 5049-5060.

[0165] 3. For MedQA-USMLE

[0166] PMl(Clark et al,2016):Clark P,Etzioni O,Khot T,et al.Combiningretrieval,statistics,and inference to answer elementary science questions[C].Proceedings of the AAAI Conference on Artificial Intelligence,2016,2580-2586.

[0167] MAX OUT(Mihaylovet al,2018):Mihaylov T,Frank A.Knowledgeable reader:Enhancing cloze-style reading comprehension withexternal commonsenseknowledge[C].Proceedings of the 56th Annual Meeting of the Association forComputational Linguistics(Volume 1:Long Papers),Melbourne,Australia,2018,821-832.

[0168] IR-ES(Chen et al,2017):Danqi Chen,Adam Fisch,Jason Weston,andAntoineBordes.2017.Reading wikipedia to answer open-domain questions.InProceedings of the 55th Annual Meeting of the Association forComputationalLinguistics,pages 1870–1879,Vancouver,Canada.Association forComputational Linguistics.

[0169] IR-CUSTOM(Chen et al,2017):Danqi Chen,Adam Fisch,Jason Weston,andAntoineBordes.2017.Reading wikipedia to answer open-domain questions.InProceedings of the 55th Annual Meeting of the Association forComputationalLinguistics,pages 1870–1879,Vancouver,Canada.Association forComputational Linguistics.

[0170] BERT-Base(Devlin t al,2019):Jacob Devlin,Ming-Wei Chang,Kenton Lee,andKristina Toutanova.2019.BERT:Pre-training ofdeep bidirectionaltransformers for language understanding.In Proceedings of the 2019 Conferenceofthe North American Chapter of the Association forComputational Linguistics:Human Language Technologies,Volume 1(Long and Short Papers),pages4171–4186,Minneapolis,Minnesota.Association forComputational Linguistics.

[0171] clinicalBERT-Base(Alsentzer et al,2019):Emily Alsentzer,John Murphy,William Boag,WeiHung Weng,Di Jindi,Tristan Naumann,andMatthewMcDermott.2019.Publicly available clinical BERT embeddings.In Proceedings ofthe 2 ndClinical Natural Language Processing Workshop,pages 72–78,Minneapolis,Minnesota,USA.Association for Computational Linguistics.

[0172] BioBERT-Base(Lee et al2020):Jinhyuk Lee,Wonjin Yoon,Sungdong Kim,DonghyeonKim,Sunkyu Kim,Chan Ho So,and Jaewoo Kang.2020.Biobert:a pre-trainedbiomedical languagerepresentation model for biomedical textmining.Bioinformatics,36:1234–1240.

[0173] BioRoBERTa Base(Gururangan et al2020):Suchin Gururangan,Ana Marasovi′c,SwabhaSwayamdipta,Kyle Lo,Iz Beltagy,Doug Downey,and Noah A.Smith.2020.Don’t stop pretraining:Adapt language models to domains and tasks.InProceedingsof the 58th Annual Meeting of theAssociation forComputational Linguistics,pages8342–8360,Online.Association forComputationalLinguistics.

[0174] RoBERT-Large(liu t al,2019):Liu Y,Ott M,Goyal N,et al.Roberta:Arobustly optimized bert pretraining approach[J].ArXiv preprint,2019,abs / 1907.11692.

[0175] BioBERTa-large(Lee et al,2020):Jinhyuk Lee,Wonjin Yoon,Sungdong Kim,Donghyeon Kim,Sunkyu Kim,Chan Ho So,and Jaewoo Kang.2020.Biobert:a pre-trained biomedical language representation model for biomedical textmining.Bioinformatics,36:1234–1240.

[0176] SapBERT Base(Liu et al,2021):Liu F,Shareghi E,Meng Z,et al.Self-alignment pretraining for biomedical entity representations[C].Proceedings ofthe 2021 Conference of the North American Chapter of theAssociationforComputational Linguistics:Human Language Technologies,Online,2021,4228-4238.

[0177] +QAGNN(Yasunaga et al,2021):Yasunaga M,Ren H,Bosselut A,et al.QA-GNN:Reasoning with language models and knowledge graphs for question answering[C].Proceedings of the 2021Conference of the North American Chapter of theAssociation for Computational Linguistics:Human Language Technologies,Online,2021,535-546.

[0178] +GreaselM(Zhang et al,2020):Zhou J,Cui G,Hu S,et al.Graph neuralnetworks:A review of methods and applications[J].AIopen,2020,1:57-81.

[0179] +JointLK(Sun et al,2022):Sun Y,Shi Q,Qi L,et al.JointLK:Jointreasoning with language models and knowledge graphs for commonsensequestionanswering[C].Proceedings of the 2022Conference of the NorthAmericanChapter of the Association for Computational Linguistics:Human LanguageTechnologies,Seattle,United States,2022,5049-5060.

[0180] 2. Experimental Results

[0181] As shown in Table 1, on the CommonsenseQA dataset, RoBERTa-large, which is suitable for common sense reasoning, is selected as the benchmark model. RoBERTa (w / o KG) means that only the language model is used to perform question-answering tasks without the help of external knowledge. Compared with the fine-tuned LM and the existing LM+KG model, the accuracy of the DLM-KG model of the present invention is improved by about 5.8% over RoBERTa. The model effect is very close to the experimental results of the best LM+KG model JointLK in the past, and is competitive.

[0182] Table 1 Performance comparison on CommonsenseQA in-house split

[0183]

[0184] On the OpenBookQA dataset, two language models in BERT, RoBERTa-large and AristoRoBERTa, were selected and tested in combination with different benchmark models. The experimental results (see Table 2) show that the accuracy of the model DLM-KG is improved by about 8.2% compared with the fine-tuned AristoRoBERTa. These improvements prove that the hierarchical interaction and joint reasoning of LM and KG proposed in the present invention can better integrate the features of QA context text and knowledge graph, ensuring that the model can reason along the correct path.

[0185] Table 2 Performance comparison on the OpenBookQA dataset

[0186]

[0187] Table 3 shows the accuracy comparison of different models on the biomedical dataset MedQA-USMLE. The language models BERT, SapBERT, RoBERTa and BioBERT are used as benchmark models. The SapBERT model uses a self-supervised learning strategy and can automatically align with the representation space of biomedical entities, which is beneficial for capturing fine-grained semantic relationships in this field. Compared with other benchmark models, SapBERT has the highest accuracy. Therefore, SapBERT is used as the language encoder in the DLM-KG model. For the LM+KG model, the model DLM-KG uses a similar model framework as the predecessors, that is, LM is used as the encoder of the QA context text and GNN is used as the encoder of KG. Compared with the fine-tuning model JointLK, the accuracy of DLM-KG is significantly improved, which shows that the model framework of this article performs better in language reasoning methods.

[0188] Table 3 Performance comparison on the medical dataset MedQA-USMLE

[0189]

[0190]

[0191] It can be seen that the performance of the DLM-KG model provided by the present invention in the common sense dataset is comparable to the best fine-tuning model of the same type in the prior art, the JointLK. In the more professional biomedical dataset, the DLM-KG model of the present invention performs better than the fine-tuning model JointLK.

[0192] It can be seen from the above embodiments and experimental examples that the present invention provides a new dynamic reasoning method and system based on language model and knowledge graph, which has better answer accuracy, and at the same time has the advantages of convenience and wide application range. It can provide new ideas and choices for multimodal fusion technology in the field of natural language, and has good application prospects.

Claims

1. A dynamic reasoning method based on language model and knowledge graph, characterized in that: The steps include: Step 1: Use the language model encoder to encode the input QA context text to obtain the encoded QA context representation; use the graph encoder to infer the knowledge graph to obtain the representation of the node entity; Step 2, iterate and execute the following steps at least once: Step 2.1: Input the node entity representation and QA context representation into the hierarchical aggregation module to capture the corresponding hierarchical features in the QA context and entity, and then integrate them through the multi-head attention mechanism to obtain the enhanced QA context representation; Step 2.2: Input the node entity representation and the enhanced QA context representation into the joint reasoning module, and use the bidirectional attention mechanism to connect the two encoders to capture the fine-grained interaction between different modal information to obtain the updated node entity representation and the updated QA context representation. Step 2.3: Input the updated node entity into the dynamic pruning module. Under the guidance of the LM-to-KG attention value calculated in step 2.2, delete the nodes that are not related to the QA context text to obtain the representation of the node entity after dynamic pruning. Step 3: Based on the node entity representation and QA context representation finally obtained in step 2, the final output answer is calculated.

2. The dynamic reasoning method based on language model and knowledge graph according to claim 1, characterized in that: In step 1, the language model encoder is selected from BERT, RoBERTa-large, AristoRoBERTa, BioBERT or SapBERTBase.

3. The dynamic reasoning method based on language model and knowledge graph according to claim 1, characterized in that: In step 1, the process of using the graph encoder to reason about the knowledge graph includes: Step 1.1, use BERT’s average pooling to obtain the initial representation of the node; Step 1.2, after the graph encoder GNN layer, the representation of the node is updated by iterative information transmission of neighboring nodes on the graph: in, or is a set of entity nodes in the knowledge graph; Specific Calculated according to the following formula: Among them, N i is the set of neighbor nodes of node i, m ji is the information transmitted from node j to node i, α ji is the weight of information transmission between control nodes, f n It is a 2-layer neural network function; m ji The calculation of is as follows: r ji =f r (e ji ,u i ,u j ) Among them, f r is a 2-layer multilayer perceptron, f m is a linear transformation, e ji Indicates the type of edge (i, j), u i Indicates the type of node i, u j Indicates the type of node j; α ji The calculation of is as follows: a ji =softmax(γ ji ) Among them, f j and f i They are all linear transformations.

4. The dynamic reasoning method based on language model and knowledge graph according to claim 1, characterized in that: Step 2.1 specifically includes: Step 2.1.1, use a two-layer perceptron network to extract the hierarchical features of the entity nodes in the QA context text and knowledge graph: in, or is the representation of the node entity, or It is a QA context representation; Step 2.1.2, use the multi-head attention mechanism to update the representation of the QA text and obtain the enhanced QA context representation: in, or is the QA context representation, d is Dimension.

5. The dynamic reasoning method based on language model and knowledge graph according to claim 1, characterized in that: Step 2.2 specifically includes: Step 2.2.1, input node entity representation and enhanced QA context representation Construct a new matrix: Among them, f s is a linear transformation, is element-wise multiplication, [;] is vector concatenation across rows; Step 2.2.2, row by row Normalize and get the KG-to-LM attention map: Step 2.2.3, column by column Normalize and get the LM-to-KG attention mapping and attention representation: in, represents the matrix Kronecker product, It is the QA text representation modified according to the knowledge graph representation. It is a knowledge graph representation modified according to the features of the QA text; Step 2.2.4, concatenate the initial feature representation and the information representation after interactive fusion, and update the features of the two modalities in the following way: Among them, f q , f v is a linear transformation.

6. The dynamic reasoning method based on language model and knowledge graph according to claim 1, characterized in that: Step 2.3 specifically includes: Define a node retention rate K to determine the number of nodes to be retained; N is the number of nodes in the knowledge graph when inputting the model. Select according to the attention value of LM-to-KG The highest nodes constitute the knowledge graph subgraph predicted by the model, that is, the representation of the node entity after dynamic pruning:

7. The dynamic reasoning method based on language model and knowledge graph according to claim 1 is characterized in that: Step 3 specifically includes: Step 3.1, calculate the credibility score of each candidate answer: p=(a|q,G)=MLP(Q;V) Where a is the candidate answer, q is the question text, G is the retrieved knowledge graph subgraph, Q is the QA context representation finally obtained in step 2, and V is the node entity representation finally obtained in step 2; In step 3.2, use the softmax function to normalize all questions to get the final probability, and take the option with the highest probability value as the final output answer.

8. A dynamic reasoning system based on language model and knowledge graph, characterized in that: include: Input module, used to input QA context text; Storage module, used to store knowledge graphs; A question-answering module is used to obtain the final output answer according to the dynamic reasoning method based on the language model and knowledge graph as described in any one of claims 1-7.

9. The dynamic reasoning system based on language model and knowledge graph according to claim 8, characterized in that: The question-answering module includes: a language model encoder, a graph encoder and at least one layer of cross-modal interaction fusion module, and the cross-modal interaction fusion module includes a hierarchical aggregation module, a joint reasoning module and a dynamic pruning module.

10. A computer-readable storage medium, characterized in that: Stored thereon are: a computer program for implementing the dynamic reasoning method based on a language model and a knowledge graph as described in any one of claims 1 to 8, or a computer program for implementing the dynamic reasoning system based on a language model and a knowledge graph as described in claim 9.

Images