Adaptive Multi-hop Retrieval Knowledge Graph Construction Method for Large Language Model Question Answering

Abstract

The present invention discloses an adaptive multi-hop retrieval knowledge graph construction method for large language model question answering, including: extracting triples of the input question and corpus; adaptive multi-hop retrieval matching to retrieve and identify specific triples related to the question in the corpus; constructing a knowledge graph using the triples of the question and corpus in the adaptive multi-hop retrieval matching process; answer predictor model: node embedding, message passing, and selective filtering, where node embedding is used to encode the question and knowledge graph; message passing is performed based on the encoded question and knowledge graph using a graph convolutional network; selective filtering is used to filter redundant information in the knowledge graph; the present invention helps to mitigate the problem of hallucinations generated by large language models, helps to understand complex reasoning processes, and thus improves the question answering and logical reasoning performance of large language models.

Description

Technical Field

[0001] The present invention relates to the field of semantic retrieval and matching, and particularly to an adaptive multi-hop retrieval knowledge graph construction method for large language model question answering. Background Art

[0002] With the continuous progress of artificial intelligence (AI) and natural language processing (NLP) technologies, how to efficiently extract knowledge from massive unstructured data and generate accurate and natural answers has become the focus of research. However, when dealing with complex questions, existing answer generation systems are often limited by problems such as information retrieval errors, lack of semantic understanding, and insufficient generation fluency. These technical bottlenecks not only degrade the user experience but also significantly limit the widespread deployment of intelligent question answering systems in practical applications. To address this issue, the Knowledge Graph (KG), as a structured knowledge organization method, provides important semantic and reasoning support for large language models.

[0003] The knowledge graph expresses entities and their relationships in the form of nodes and edges, and can play a core role in deep semantic understanding and relationship reasoning. By accurately modeling entities, attributes, and relationships, the knowledge graph can help large models better identify multi-level entity relationships when facing complex queries, and perform semantic expansion and context supplementation. Compared with pure text retrieval, the structured information of the knowledge graph can reduce generation errors caused by ambiguous relationships or scarce information, thereby improving the accuracy of answer generation.

[0004] Knowledge graph construction mainly involves the retrieval of triples. Most existing methods are based on the retrieval of a fixed number of hops between entities, retrieving corresponding triples from existing corpora to construct the knowledge graph. The basic idea is as follows: extract entities and relationships from the existing question information; associate and match the extracted entities and relationships with the entity relationships in the corpus; repeat the matching process for the associated entities until the answer entity is found. Since the entity matching process has a high time complexity, most people choose to match entities with a fixed number of hops to their related triples to form the knowledge graph. Among them, entity extraction and relationship extraction mainly rely on open-source tools.

[0005] Li et al. proposed the G-NAG framework in the literature "Knowledge based natural answer generation via masked-graph transformer". In the knowledge graph construction process of this framework, a multi-hop retrieval mechanism (mainly within two hops) is first used to identify and expand the entities and their relationships mentioned in the question, and then the entity disambiguation phase is used to ensure the correctness of the entities in the graph. Subsequently, the graph simplification step refines the graph structure and removes unnecessary redundant nodes. On this basis, the Masked-Graph Transformer encodes the graph using the communication path and the masking mechanism to capture the complex semantic relationships between vertices, thereby providing rich context information for generating accurate and fluent natural language answers. Saxena et al. formalized the graph structure by defining a set of entities and relationships in the literature "Improving Multi-hop Question Answering over Knowledge Graphs using Knowledge Base Embeddings", and enhanced the graph integrity using the link prediction task. Knowledge Graph Embedding (KGE) methods, such as ComplEx, are applied to generate high-dimensional vector representations of entities and relationships, enabling the model to predict the missing links in the knowledge graph by learning the interaction patterns of entities and relationships, thereby providing rich background knowledge for the multi-hop question answering system.

[0006] Disadvantages of the prior art:

[0007] The deficiencies of existing methods in the knowledge graph construction process are as follows: (1) Existing methods only retrieve a limited or fixed number of hops of relevant question entities in the knowledge base. When these entities contain multi-hop relationships, they cannot perform adaptive multi-hop retrieval based on the existing information. This may lead to the loss of necessary information related to answer generation; (2) The recognition capabilities of existing widely used open information extraction tools, such as Minie, StanfordCoreNLP, etc., for information extraction are insufficient for triple extraction of complex sentences; (3) Most existing models use Graph Transformer as the encoder, which mainly uses the attention mechanism. This mechanism is easily affected by noise or irrelevant information in the input sequence, and thus cannot filter out redundant information with low semantic information in the question and answer. This may cause the model to focus on incorrect elements, especially when dealing with complex or ambiguous data. Summary of the Invention

[0008] The objective of the present invention is: aiming at the above-mentioned problems, the present invention provides an adaptive multi-hop retrieval knowledge graph construction method for large language model question answering, which accurately extracts triples related to the user's given question based on the paragraph information in the corpus, and realizes the construction of the knowledge graph through the triples, which can give the large model structured and explicit factual knowledge, help reduce the problem of hallucinations in the large language model, and help understand complex reasoning processes, thereby improving the question answering and logical reasoning performance of the large language model.

[0009] The technical solution adopted by the present invention is as follows:

[0010] An adaptive multi-hop retrieval knowledge graph construction method for large language model question answering, including:

[0011] Data extraction, extracting the input question and the triples of the corpus;

[0012] Adaptive multi-hop retrieval matching, retrieving and identifying specific triples related to the question in the corpus;

[0013] Knowledge graph construction, using the question and the corpus triples in the adaptive multi-hop retrieval matching process to construct the knowledge graph;

[0014] Answer predictor model, including: node embedding, message passing and selective filtering. Node embedding is used to encode the question and the knowledge graph; message passing is performed based on the encoded question and the knowledge graph using a graph convolutional network; selective filtering is used to filter redundant information in the knowledge graph.

[0015] Furthermore, denoising the extracted triples, as shown in the following formula:

[0016]

[0017] In the formula, T q , T d respectively represent the extracted question entity triples and corpus entity triples, and g and d represent the question and the corpus.

[0018] Furthermore, removing redundant information from the extracted triples, the redundant information processing includes:

[0019] Assume that the entities and relationships in triple T1 are included in triple T2, that is, T1 ∈ T2, then use triple T1 to replace triple T2;

[0020] If two triples are in parallel, then concatenate the two triples.

[0021] Furthermore, the adaptive multi-hop retrieval matching is specifically to retrieve from the corpus the question entity E q and the answer entity E aRelevant data, i.e., by using the triples T extracted from the knowledge base d to quantify the matching information between the corpus and the question q.

[0022] Furthermore, using the corpus triples structuredly extracted by the pre-trained language model PLM, the sequence model BART is used as an encoder to encode the question entity E q and the corpus triples T d , and they are tokenized as follows:

[0023] Question entity E q , the question entity E q is marked as the token vector v q ={w1, w2,..., w n} and then the classification token "[CLS]" and the separation token "[SEP]" are added to mark the beginning of the sequence and to separate sentences or paragraphs, respectively;

[0024] Corpus triples T d , each triple t i ∈T d is flattened into a sequence s, and then s is tokenized as v q ∈R s in a similar encoding to the question triple T {m×n} ;

[0025] After encoding, the corpus triples T d with the highest similarity to the question encoding v q are selected as the basis for the next iteration; the correlation score and the similarity score are defined as follows:

[0026] R(v q , T d ) = max(S m ) (2)

[0027] S m ={s1, s2,..., s m}(3)

[0028] s i =cos(v q , t i ), t i ∈T d (4)

[0029] In the formula, R(v q , T d ) represents the correlation score, s i represents the similarity score, and t i is the entity encoding in the triple.

[0030] Furthermore, the construction of the knowledge graph specifically includes:

[0031] Utilize the problem entity E q and the corpus triples T d to construct the knowledge graph G; for each hop i, select the entity as a node and divide the construction of the edge into an initial edge and a dynamic edge;

[0032] Initial edge, for the first-hop retrieval, connect the problem entity E q with the problem entity E used as the initial-edge retrieval d ;

[0033] Dynamic edge, for the next-hop retrieval i (i≠1), use the problem entity of the previous hop and the most similar knowledge-base entity retrieved to update the query, and connect the retrieved knowledge-base entity with as the dynamic edge.

[0034] Furthermore, the node embedding in the answer predictor model includes:

[0035] Before using the graph convolutional network GCN to perform message passing to update the node encoding in the neighborhood, it is necessary to initialize the node encoding using an encoder, specifically:

[0036] Question encoding, for the original question q, add the specific flags "[CLS]" and "[SEP]" to the question q, and use the sentence-level model BART to generate the encoding

[0037] Knowledge graph encoding, for each entity in the knowledge graph, use the character-level encoder to encode the entity as

[0038] Furthermore, the message passing includes:

[0039] Utilize GCN to accelerate information aggregation, specifically as follows:

[0040] V * = W r (AVW g ) + V (5)

[0041] In the formula, V is the adjacency matrix, A is the entity node matrix, and W r and W g are learnable model network parameters.

[0042] Furthermore, the selective filtering is specifically performed through the Graph based Mamba module for selection and filtering.

[0043] In summary, due to the adoption of the above technical solutions, the beneficial effects of the present invention are as follows:

[0044] The adaptive multi-hop retrieval knowledge graph construction method for large language model question answering of the present invention solves the challenges in natural answer generation and improves the accuracy and fluency of answers by proposing a knowledge-enhanced interpretable Graph Mambe framework. Compared with the existing retrieval methods with fixed hop counts, this framework not only improves the prediction accuracy but also reduces the computational complexity. In addition, by using a pre-trained large language model for answer optimization, the filtering ability of the model for redundant information is enhanced, overcoming the problem that existing models are easily interfered by noise or irrelevant information. By adopting an adaptive multi-hop retrieval algorithm and based on the Graph Mamba module, the construction process of the knowledge graph is optimized, effectively filtering redundant information. This framework overcomes the problem that existing models are easily interfered by noise or irrelevant information when dealing with complex or ambiguous data. By selectively memorizing or forgetting certain tokens in the input sequence, the attention of the model to key information is improved, thereby enhancing the robustness of the model when dealing with complex or ambiguous data and improving the quality of natural answer generation. In addition, the interpretability of the model answers is also enhanced, making the reasoning process of the model more transparent and helping to understand the decision-making logic of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a framework diagram of the adaptive multi-hop retrieval knowledge graph construction in the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] The present invention will be described in detail below with reference to the accompanying drawings.

[0047] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] This embodiment provides a dynamic crack detection and pixel-by-pixel object segmentation method based on parameter redefinition, as Figure 1 shown, which specifically includes:

[0049] Triple extraction

[0050] For the input questions and corpus, it is first necessary to extract their respective triples for subsequent construction of the knowledge graph. Due to the complexity of semantics in the corpus, there are noises and redundant items in the extracted triples T d in it.

[0051] Noise processing: The extracted triples lack a specific representation of the source corpus and are irrelevant to the problem entity because they exist in multiple triples and have no specific semantic association with the current problem entity; these triples may not be conducive to subsequent corpus retrieval. Therefore, select the triples that contain as many problem entities as possible to delete the noisy triples. Use the relevance score to quantitatively describe this process.

[0052]

[0053] Redundancy processing: The extracted triple T d has a redundancy problem, which significantly affects the efficiency of corpus retrieval. There are mainly two types of redundant information. First, the entities and relationships in triple T1 are completely contained in T2, which means T1 ∈ T2, and triple T1 will be used to replace triple T2; second, parallel: two triples T1 = <s1, p1, o1> and T2 = <s2, p2, o2> have a high similarity in terms of structure and semantic information, and we replace them with their concatenated triple T3 = <s3, p3, o3>, which means s3 = s1 U s2, and p3 and o3 are calculated in a similar way.

[0054] Adaptive multi-hop retrieval matching

[0055] To answer the multi-hop retrieval question q, it is necessary to retrieve knowledge related to the problem entity E q and the answer entity E a from the corpus. These entities contain clues that help answer the question; by using the triples T d extracted from the corpus to quantify the matching information between the knowledge base and the question q; therefore, the goal of retrieval is to identify the specific triples related to the question in the knowledge base. We use a pre-trained language model PLM to retrieve structured triples from the extracted corpus; in this way, the sequence model BART is adopted as the encoder to encode E q and T d , and tokenize them in different ways.

[0056] The problem entity E q , label the problem entity E q as the token vector v q = {w1, w2,..., w n}, and then add the classification token "[CLS]" and the separation token "[SEP]" to label the beginning of the sequence and split the sentence or paragraph respectively;

[0057] The corpus triple T d , flatten each triple t i ∈ T d into a sequence s, and then in the same way as the question triple Tq A similar encoding marks s as v s ∈R {m×n} ;

[0058] After encoding, the corpus triple T is selected according to the relevance score d in the one with the problem encoding v q The entity with the highest similarity as the basis for the next iteration; the relevance score and similarity score are defined as follows:

[0059] R(v q , T d ) = max(S m ) (2)

[0060] S m = {s1, s2,..., s m}} (3)

[0061] s i = cos(v q , t i ), t i ∈T d (4)

[0062] In the formula, R(v q , T d ) represents the relevance score, s i represents the similarity score, and t i is the entity encoding in the triple.

[0063] Knowledge graph construction

[0064] This embodiment uses the problem entity E in the multi-hop retrieval process q and the knowledge base triple T d to construct the knowledge graph G. For each hop i, the entity is selected as the node and the construction of the edge is divided into two types: the initial edge and the dynamic edge;

[0065] Initial edge. For the first-hop retrieval, this embodiment connects the problem entity E q with the entity E d retrieved as the initial edge;

[0066] Dynamic edge. For the next-hop retrieval i (i≠1), this embodiment uses the problem entity of the previous hop and the most similar knowledge base entity retrieved to update the query. Therefore, this embodiment selects to connect the knowledge base entity retrieved this time with as the dynamic edge.

[0067] Answer predictor model

[0068] The model divides this process into three parts: node embedding, message passing, and Mamba-based selective filtering.

[0069] Node embedding: Before updating the node encoding in the neighborhood using message passing with the graph convolutional network (GCN), it is necessary to initialize the node encoding using an encoder:

[0070] Question encoding. For the original question q, in this embodiment, specific flags "[CLS]" and "[SEP]" are added to q in the common way, and the sentence-level model BART is used to generate the encoding.

[0071] Knowledge graph encoding. For each entity in the knowledge graph, in this embodiment, a word-level encoder is used to encode the entity into

[0072] Message passing: The core of GCN lies in the information transmission process, enabling the network to understand the relationships between entity nodes. In this stage, information spreads between adjacent nodes using the graph structure. This embodiment uses GCN to accelerate information aggregation.

[0073] V * =W r (AVW g )+V (5)

[0074] Mamba-based selective filtering. In this embodiment, the Graph based Mamba (GBM) module is proposed; GBM utilizes the selectivity and long-range dependence capabilities of Mamba to achieve graph sparsification, effectively filtering redundant information in the graph. The core of GBM is the selective structured state space model, which is a sequence model developed from the SSM and is updated by selectively hiding state information; the SSM is mainly divided into a state equation and an output equation; the state equation updates the hidden state of the input sequence through a first-order ordinary differential equation, while the output equation uses these hidden states to map the input sequence to the output sequence, as shown in the following formula:

[0075]

[0076] In the formula, and represent the state transition matrix, input control matrix, output matrix, and direct transfer matrix respectively. Usually in deep learning, Dx(t) = 0 is regarded as an easy-to-compute skip connection and can be simplified to y = Ch(t), where h(t) is the state at time t and x(t) is the input at time t.

[0077] Equation (6) is applicable to continuous data. However, in practical scenarios, discrete data after sampling is usually processed. Therefore, the structured state space sequence model proposes to discretize the SSM and constructs a state transition matrix A based on HIPPO. The discretization equation is as follows:

[0078]

[0079] In the formula, and are the discretized state transition matrix and input control matrix, h t-1 is the state matrix at time t - 1, and x t is the input at time t.

[0080] The simulation experiment of this embodiment is carried out as follows:

[0081] Simulation conditions: Experiments are carried out using the deep learning framework pytorch on a central processing unit of Intel(R) Xeon(R) Platinum 8369C CPU, NVIDIA A10 Tensor Core GPU, and Ubuntu operating system. The public datasets HotpotQA and WikiHop are used for data collection.

[0082] Comparison method: GenQA is an end-to-end model based on the encoder-decoder architecture. It uses bidirectional LSTM to encode and decode questions to generate answers;

[0083] COREQA combines the copy and retrieval mechanisms, enabling it to directly extract information from the question text or knowledge base and integrate it into the generated answers;

[0084] G-NAG is a graph2seq model using the encoder-decoder structure. It designs Masked-GraphTransformer to encode the knowledge graph and then uses a sequence attention decoding model to generate natural answers;

[0085] QRGAT encodes the reasoning process as an inference graph and uses a relation-aware graph attention network to capture the dependencies between different relation paths.

[0086] Simulation process: Calculate the accuracy of question answering according to the method constructed in this embodiment and compare it with the accuracy of the comparison methods mentioned above. The results are shown in Tables 1 and 2.

[0087] Table 1 Answer prediction performance on the HotpotQA dataset

[0088] model Acc BLEU-4 Meteor GenQA 0.649 0.362 0.371 COREQA 0.656 0.393 0.412 G-NAG 0.784 0.415 0.433 QRGAT 0.853 0.422 0.446 this method 0.874 0.454 0.481

[0089] Table 2 Answer Prediction Performance on the WikiHop Dataset

[0090]

[0091]

[0092] As can be seen from Table 1 and Table 2, the knowledge-enhanced interpretable Graph Mambe framework adopted in this embodiment can fully exploit the semantic information of the questions and the corpus, making the model answers have higher accuracy and fluency, verifying the advancement of the present invention.

[0093] Specific embodiments are applied in this article to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention. It should be noted that for those of ordinary skill in the art in this technical field, without departing from the principle of the present invention, several improvements and modifications can still be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims

1. An adaptive multi-hop retrieval knowledge graph construction method for large language model question answering, characterized in that: include: Data extraction, extracting triples of input questions and corpus; Adaptive multi-hop retrieval matching, retrieving the entity E from the corpus q and answer entity E a Related data, that is, by using the triples T extracted from the knowledge base d To quantify the matching information between the corpus and question q, specifically including: Use the pre-trained language model PLM to structure the extracted corpus triples and use the sequence model BART as the encoder to encode the question entity E q and corpus triples T d , and tokenize it as follows: Problem Entity E q , the problem entity E q Labeled as label vector v q ={w1,w2,…,w n }, then add the classification tag "[CLS]" and separation tag "[SEP]" to mark the beginning of the sequence and separate sentences or paragraphs respectively; Corpus triples T d , each triplet t i ∈T d Flattened into a sequence s, then replaced with the problem triple T q Similar encoding marks s as v s ∈R {m×n} ; After encoding, corpus triples T are selected based on the relevance score d Chinese and question coding v q The entity with the greatest similarity serves as the basis for the next iteration; the relevance score and similarity score are defined as follows: R(v q ,T d )=max(S m ) (2) S m ={s1,s2,...,s m } (3) s i =cos(v q ,t i ),t i ∈T d (4) In the formula, R(v q ,T d ) represents the correlation score, s i represents the similarity score, t i is the entity encoding in the triple; Knowledge graph construction, using the question and corpus triples in the adaptive multi-hop retrieval matching process to build a knowledge graph; The answer predictor model includes: node embedding, message passing and selective filtering, wherein node embedding is used to encode questions and knowledge graphs; message passing is performed based on a graph convolutional network based on the encoded questions and knowledge graphs; and selective filtering is used to filter redundant information of the knowledge graph.

2. The method for constructing an adaptive multi-hop retrieval knowledge graph for large language model question answering according to claim 1, characterized in that: The extracted triples are denoised as follows: Where, T q ,T d They represent the extracted question entity triples and corpus entity triples respectively, q and d represent the question and corpus.

3. The method for constructing an adaptive multi-hop retrieval knowledge graph for large language model question answering according to claim 1, characterized in that: The extracted triples are subjected to redundant information removal, wherein the redundant information processing comprises: Assume that the entities and relations in triple T1 are contained in triple T2, that is, T1∈T2, then replace triple T2 with those in triple T1; If two triplets are in parallel, the two triplets are connected in series.

4. The method for constructing an adaptive multi-hop retrieval knowledge graph for large language model question answering according to claim 1, characterized in that: The knowledge graph construction specifically includes: Using the problem entity E q and corpus triples T d To construct the knowledge graph G; for each hop i, select entity as nodes and divide the construction of edges into initial edges and dynamic edges; Initial edge, for the first hop retrieval, the problem entity E q and the question entity E retrieved as the initial edge d connect; Dynamic edge, for the next hop retrieval i (i≠1), use the problem entity of the previous hop and the most similar knowledge base entity retrieved To update the query, the retrieved knowledge base entities and Connections act as dynamic edges.

5. The method for constructing an adaptive multi-hop retrieval knowledge graph for large language model question answering according to claim 1, characterized in that: The node embedding in the answer predictor model includes: Before using the graph convolutional network GCN for message passing to update the node encoding in the neighborhood, the node encoding needs to be initialized using the encoder, specifically: Question encoding: For the original question q, add specific tags "[CLS]" and "[SEP]" to the question q, and use the sentence-level model BART to generate the encoding Knowledge graph encoding, for each entity in the knowledge graph, a word-level encoder is used to encode the entity as 6. The method for constructing an adaptive multi-hop retrieval knowledge graph for large language model question answering according to claim 5, characterized in that: The message transmission includes: GCN is used to accelerate information aggregation, as follows: V * =W r (AVW g )+V (5) Where V is the adjacency matrix, A is the entity node matrix, and W r and W g are learnable model network parameters.

7. The method for constructing an adaptive multi-hop retrieval knowledge graph for large language model question answering according to claim 1, characterized in that: The selective filtering is specifically performed by selective filtering based on the Graphbased Mamba module.

Images