LLM thinking chain-driven knowledge graph question and answer method and system

Abstract

The invention relates to the technical field of natural language processing and knowledge graph crossing, in particular to an LLM thinking chain driven knowledge graph question and answer method and system, and the method comprises the steps: obtaining a knowledge graph question and answer data set and knowledge subgraphs; constructing a knowledge graph question and answer network model driven by the LLM thinking chain; applying the trained LLM thinking chain driven knowledge graph question and answer network model to a knowledge graph question and answer task; according to the method, a more stable, robust and accurate knowledge graph question and answer result can be obtained.

Description

Technical Field

[0001] This invention belongs to the field of interdisciplinary technology of natural language processing and knowledge graph, specifically involving an LLM-driven knowledge graph question answering method and system. Background Technology

[0002] The core objective of knowledge graph question answering is to retrieve relevant information from a knowledge graph storing structured knowledge and infer answers that conform to semantic constraints based on natural language questions input by users. However, with the continuous expansion of knowledge graphs, the increasing complexity of entity relationships, and the evolution of user questions into more complex forms such as multi-hop and semantic ambiguity, traditional KGQA methods face numerous challenges.

[0003] Existing knowledge graph question answering methods typically rely on two main technical paths: One type of approach is based on semantic parsing, which transforms natural language questions into logical forms and performs graph queries. The other type is based on deep learning, which uses neural networks to model the semantic alignment between the question and the knowledge graph. However, the former often requires a large number of manually defined rules and labeled logical forms, making it difficult to adapt to open-domain problems. While the latter can automatically learn semantic representations, it suffers from significant technical limitations in practical applications. First, language models may output entities or relationships that do not exist in the knowledge graph during the generation process, leading to distortion of the reasoning basis; Secondly, the lack of effective constraints on the relational path search space leads to low retrieval efficiency. Third, the structured information mining of knowledge graphs is insufficient, failing to fully utilize the topological relationships between entities and affecting the accuracy of reasoning. Summary of the Invention

[0004] To address the problems of distorted and unstable reasoning, low retrieval efficiency, and insufficient reasoning accuracy in existing knowledge graph question answering methods, this invention provides an LLM-driven knowledge graph question answering method and system. This method and system are conducive to obtaining more stable, robust, and accurate knowledge graph question answering results.

[0005] The technical solution of the present invention is as follows: A knowledge graph question-answering method driven by LLM mind chain includes the following steps: Step 1: Obtain the knowledge graph question-answering dataset and knowledge subgraph; Step 2: Construct an LLM-driven knowledge graph question-answering network model; Step 3: Apply the trained LLM-driven knowledge graph question answering network model to the knowledge graph question answering task.

[0006] Furthermore, the knowledge graph question-answering dataset contains the original natural language question and its corresponding question entity and answer entity, wherein the knowledge subgraph provides a knowledge graph subgraph structure related to the original question, including entity set, relation set and multiple triples; The entities are derived from the dataset, and each entity contains the following fields: id, question, answer entity, question entity, and knowledge subgraph.

[0007] Furthermore, the knowledge graph question-answering network model includes a semantically guided subgraph retrieval module, a subgraph structure encoding module, and a structure-semantic fusion module; The multiple modules work together to perform subgraph retrieval tasks, subgraph structure encoding tasks, and answer reasoning tasks. The semantically guided subgraph retrieval module utilizes a thought chain-driven problem decomposition mechanism to decompose the original natural language problem into one or more sub-problems, guide the direction of subgraph retrieval, and filter the candidate relation list most relevant to the semantics of the current sub-problem to retrieve high-quality reasoning subgraphs related to the problem. The subgraph structure encoding module performs structural modeling on high-quality reasoning subgraphs and aggregates them to form a subgraph structure representation. The structure-semantic fusion module uses the subgraph structure representation as a soft cue, integrates the textual representation of subgraph triples and the semantic representation of the question, and achieves the fusion of semantics and structure, thereby guiding the generation of the final answer; The constructed LLM-driven knowledge graph question answering network model was trained using a knowledge graph question answering dataset.

[0008] Furthermore, the implementation method of the LLM-driven knowledge graph question-answering network model is as follows: Step 2.1: Input the original natural language question, question entities, and prompt words into the semantic-guided subgraph retrieval module. The semantic-guided subgraph retrieval module utilizes a thought-chain-driven question decomposition mechanism. First, through prompt word engineering, based on the designed task instructions, few-sample examples, and output format constraints, it guides the LLM to gradually decompose the original natural language question, generating a sequence of sub-questions and guiding the direction of subgraph retrieval. For each sub-question, it performs two-level semantic matching: "sub-question-relation" and "relation-entity". First, it calculates the semantic similarity between the sub-question and the knowledge graph relationship using a pre-trained language model, and selects the Top-k candidate relations to eliminate the illusion that occurs when generating large language models. Then, based on the candidate relation list and combined with the knowledge graph, it samples entities connected to the candidate relations, calculates the semantic similarity between the relations and the knowledge graph entities using a language model, and selects the Top-k candidate entities, thereby forming a high-quality reasoning subgraph. Step 2.2: Input the high-quality reasoning subgraph into the subgraph structure encoding module; the subgraph structure encoding module first initializes the embedding representation of each node in the subgraph, and the graph encoder aggregates the neighbors of each node through the attention mechanism to obtain the aggregated representation of the node; then it performs mean pooling aggregation on all nodes in the subgraph to construct the structure vector representation of the subgraph, thereby realizing the subgraph structure modeling and fully capturing the semantic associations and graph structure dependencies between nodes; Step 2.3: Input the question text, subgraph structure representation, textual description of the subgraph, and prompt words into the structure-semantic fusion module; using the fusion strategy of soft and hard prompts, the subgraph structure representation serves as a soft prompt to guide the LLM to understand the graph structure, while the subgraph is textualized into language prompts to provide explicit reasoning information, thereby realizing joint structure-semantic modeling and enhancing the structural compressibility and semantic interpretability of the model; Step 2.4: Input the fused semantic and structural representations into the LLM, and generate the final answer using an autoregressive generation method; Step 2.5: During training, the negative log-likelihood loss function is used to calculate the loss value of the generated answer. The parameters of the graph neural network encoder and LLM are optimized through backpropagation. The LLM can use the low-rank adaptive LoRA technique to efficiently fine-tune the parameters to reduce training costs.

[0009] Furthermore, the specific method for autoregressive generation is as follows: LLM generates the answer sequence sequentially from left to right. ; At every moment Knowledge graph question answering network models are all based on previously generated sequences. By combining the fused hint vector, the next token can be predicted. The specific generation probability can be formalized as: ; in, This means that LLM predicts the next token based on the input and the previous generated token. The probability distribution.

[0010] Furthermore, the negative log-likelihood loss function The formula is:

[0011] Among them, the summation symbol This represents the sequence of answers from the first position to the last position. The losses for all time steps (token positions) are summed up.

[0012] Finally, by iteratively minimizing the above loss function, the knowledge graph question answering network model can gradually learn to generate natural language answers that conform to the constraints of the knowledge graph under the condition of integrating graph structure and semantic information.

[0013] Furthermore, the specific implementation method of the semantic-guided subgraph retrieval module is as follows: The semantic-guided subgraph retrieval module accepts raw natural language questions. The set of subject entities involved in the problem And artificially designed prompts As input, where For natural language questions raised by users, Let m represent the set of entities related to the gold theme, and m represent the number of question entities. The goal of the semantically guided subgraph retrieval module is to generate a high-quality reasoning subgraph that is highly semantically relevant to the question and capable of answering it, based on semantic guidance. This subgraph contains candidate relations and entities related to the problem; The semantic-guided subgraph retrieval module employs a thought-chain-driven problem decomposition mechanism. First, it uses prompt word engineering to semantically decompose the input question and design task instructions. It includes a task description. Small sample examples and output format constraints Task instructions It can be represented as: ; in The roles and generation requirements of LLM are defined, by... Inputting into an LLM yields a sequence of subproblems. It can be represented as: ; in, Indicates the first The issue of height. This represents the number of hops in the original problem. This represents the recursive problem decomposition function based on CoT hints, for the generated first... Size problem Its content consists of the answer generated in the previous step and the first step. Size problem Launch, when hour, Represent the original question; Secondly, for each subproblem generated The model first starts from the current entity set Let's start with knowledge graphs. Retrieve all one-hop relationships connected to these entities, construct their topological neighborhoods, and form a candidate relationship set: ; Among them, when hour, For the initial set of gold-themed entities ; The semantic-guided subgraph retrieval module employs a lightweight vector-based semantic similarity metric, using a sentence vector encoder as the semantic encoder. The encoder will convert the sub-problems into their respective sub-problems. and each candidate relationship Encode it as a dense vector representation, and use cosine similarity. Calculate their semantic similarity: ; in This indicates the calculation of the L2 norm of a vector. This represents the vector dot product operation.

[0014] Finally, the models are selected based on similarity scores. The best-matching relations form the candidate relation set for the current step. :

[0015] in Representing subproblems and all candidate relationships similarity, This indicates selecting the top-ranked relation from all candidate relation similarity scores. The relationship between the highest scores; For each candidate relation In the knowledge graph, sample all connected target entities, construct their structural neighborhoods, and form a candidate entity set. : ; The semantic-guided subgraph retrieval module introduces a second-level semantic filtering mechanism at the entity level, implementing dual constraints of structural reachability and semantic relevance for each sampled candidate entity. sub-problems As a semantic benchmark, calculate the semantic similarity between the entity text description and the sub-problems: ; Finally, based on semantic similarity scores, the selected products are... The most relevant entity: ; in Representing subproblems and all candidate entities semantic similarity, This indicates selecting the top-ranked entities from all candidate entity similarity scores. The entity with the highest score; The process achieves precise expansion through dual constraints, ensuring that entities originate from real connections in the knowledge graph, and improving reasoning accuracy through semantic filtering. After the above process, at each step A candidate relation set is obtained. and entity set Together, they constitute the extended edge set and node set of the subgraph:

[0016] in Indicates the first The updated knowledge subgraph is derived from the subgraph of the previous step. It is formed by merging with the newly extended set of triples, symbol The union operation represents the set operation. Describes the newly expanded set of triples. Indicates the first The set of entities in the step; Iterate continuously until the final step of the reasoning chain is reached. Thus, the complete reasoning subgraph is obtained: ; The final output is a high-quality inference subgraph. .

[0017] Furthermore, the specific implementation method of the subgraph structure encoding module is as follows: First, the subgraph structure encoding module receives high-quality reasoning subgraphs output by the semantically guided subgraph retrieval module. As input, the subgraph consists of a set of candidate nodes and their connections. Nodes contain entities from the knowledge graph, and edges represent semantic relationships between entities. For the set of entity nodes in the subgraph, This is the set of relation edges in the subgraph; Input inference subgraph ,Include Each node and Each edge can be initially represented by its attribute description or adjacency structure, using a sentence vector encoder. Let the initial embedding representation of the node be denoted as ,in Embed dimensions for nodes; ; At the same time, the semantic description of the label of each edge is also embedded and represented as denoted as , .

[0018] After obtaining the initial representation, the graph encoder performs neighbor aggregation on each node through a multi-layer attention mechanism. For any node in the subgraph... First, calculate its relationship with its neighboring nodes. The attention weights in the current layer depend not only on the node's own representation. to the neighbors It also incorporates the relationship between the two. To capture the dual features of semantic and structural dependencies, attention weights It can be represented as: ; in, Represents a node For its neighboring nodes The normalized attention weights at the nth attention head, Indicates the first Transpose of the scoring vector of each attention head The weight matrix representing the edge attributes. For activation function, For normalized exponential functions, This represents the weight matrix used for linear transformation node features in the nth attention head. Representing nodes respectively and In the The feature vector representation of a layer; Based on this weight, each node's... Layer representation Aggregate neighbors through the following multi-headed attention: ; in , express The set of neighboring nodes, express Size spliced ​​together Indicates the first Weight matrix of the size, Represents a node In the The feature vector representation of the layer, after Layer iterative updates, nodes The final representation is: ; It integrates the semantic and structural information of multi-hop neighbors, thus enabling cross-entity context modeling capabilities; After node aggregation is complete, the module represents the entire subgraph as a global vector. This is used for downstream reasoning and question answering. Therefore, pooling aggregation is performed on the representations of all nodes in the subgraph, which can be represented as: ; in This indicates mean pooling. Represents a node The vector representation, updated through multiple iterations, reflects the overall semantics of the subgraph. The final subgraph structure encoding vector It captures key topological patterns, inference path features, and structural relationships between entities in the subgraph, which are important inputs for subsequent fusion inference.

[0019] Furthermore, the specific implementation method of the structure-semantic fusion module is as follows: After obtaining the structured encoding of the subgraph, a hierarchical hint injection mechanism is added, which integrates soft and hard hints to present graph structure information in different forms and at different levels. Injecting into LLM; First, the structure of the subgraph is encoded through a multi-layer nonlinear projection network. Mapped to a continuous vector of the same dimension as the LLM word embeddings. This ensures that the graph structure information can be effectively aligned with the LLM space:

[0020] Among them, soft prompts As an additional virtual token, it is injected into the beginning of the input sequence to serve as a continuous learnable prefix vector; at the same time, the triples in the subgraph are linearized into natural language descriptions as hard cues, providing explicit semantics for entities and relations. Constructing the final input representation sequence At that time, the model follows structural embedding and textual description. Question text Separator order organization information: ; in, This indicates a soft prompt, which is a structural embedding of subgraph information; Hard hints are embedded textual descriptions of subgraph triples; Embedding of the question text; These represent the start boundary symbol and the end boundary symbol, respectively.

[0021] A knowledge graph question-answering system driven by LLM (Low-Level Mechanism) thinking chain is provided. The system includes a data acquisition module, a model building module, and a question-answering reasoning module. The functions and connections of each module are as follows: The data acquisition module is used to acquire knowledge graph question-and-answer datasets and knowledge subgraphs; The model building module is connected to the data acquisition module and is used to build an LLM-driven knowledge graph question answering network model. The question-answering reasoning module is connected to the model building module and is used to call the trained model to perform knowledge graph question-answering tasks.

[0022] Compared with the prior art, the present invention has the following beneficial effects: This invention introduces a semantic guidance mechanism to decompose complex natural language problems into multiple sub-problems, effectively reducing the blindness in subgraph retrieval and the illusionary risk in relation generation during LLM (Local Language Modeling). Simultaneously, it combines graph structure modeling capabilities, utilizing subgraph retrieval and graph structure encoding to capture structural dependencies between entities and relations. Based on this, a structure-semantic fusion module enables joint structure-semantic reasoning, thereby enhancing the interpretability and robustness of the reasoning chain. This method not only improves the accuracy of answer generation but also significantly enhances the model's generalization ability in complex multi-hop question-answering scenarios, better addressing challenges such as the large scale of knowledge graphs, complex entity relationships, difficulties in semantic understanding of natural language problems, and the need for multi-hop reasoning. Attached Figure Description

[0023] Figure 1 This is an architecture diagram of the knowledge graph question-answering network model driven by the LLM mind chain in an embodiment of the present invention. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0025] This embodiment provides an LLM-driven knowledge graph question answering method, including the following steps: Step 1: Obtain the knowledge graph question-answering dataset and knowledge subgraph. The knowledge graph question-answering dataset contains the original natural language question and its corresponding question entity and answer entity. The knowledge subgraph provides the knowledge graph subgraph structure related to the original question, including entity set, relation set and multiple triples.

[0026] Step 2: Construct an LLM-driven knowledge graph question-answering network model. This model comprises multiple components: a semantically guided subgraph retrieval module, a subgraph structure encoding module, and a structure-semantic fusion module. These components collaboratively execute subgraph retrieval, subgraph structure encoding, and answer reasoning tasks. The semantically guided subgraph retrieval module utilizes a chain-driven question decomposition mechanism to break down the original natural language question into one or more sub-questions, guiding the direction of subgraph retrieval and filtering a list of candidate relations most semantically relevant to the current sub-question, thus retrieving high-quality reasoning subgraphs related to the question. The subgraph structure encoding module performs structural modeling on the high-quality reasoning subgraphs, aggregating them to form a subgraph structure representation. The structure-semantic fusion module uses the subgraph structure representation as a soft cue, fusing the textualization of subgraph triples with the question's semantic representation to achieve semantic and structural fusion, thereby guiding the generation of the final answer. The constructed LLM-driven knowledge graph question-answering network model is trained using a knowledge graph question-answering dataset. Step 3: Apply the trained LLM-driven knowledge graph question answering network model to the knowledge graph question answering task.

[0027] In this embodiment, a knowledge graph question-answering dataset is obtained from public datasets including WebQSP and CWQ. The knowledge subgraph provides a knowledge graph subgraph structure related to the original question, including a set of entities, a set of relations, and multiple triples.

[0028] In this embodiment, the LLM large language model specifically adopts the open-source Llama series LLM, the semantic guided subgraph retrieval module uses Llama3-70B-Instruct; and the answer reasoning stage uses Llama2-7B-Chat or Llama3-8B-Instruct.

[0029] Figure 1 This is the architecture diagram of the LLM-driven knowledge graph question-answering network model in this embodiment. For example... Figure 1 As shown, the specific implementation method of the knowledge graph question answering network model is as follows.

[0030] A) The original natural language question, question entities, and prompt words are input into the semantic-guided subgraph retrieval module. The semantic-guided subgraph retrieval module utilizes a thought-chain-driven question decomposition mechanism. First, through prompt word engineering, based on the designed task instructions, few-sample examples, and output format constraints, it guides the LLM to gradually decompose the original natural language question, generating a sequence of sub-questions and guiding the direction of subgraph retrieval. For each sub-question, a two-level semantic matching process of "sub-question-relation" and "relation-entity" is performed. First, the semantic similarity between the sub-question and the knowledge graph relationship is calculated using a pre-trained language model, selecting the Top-k candidate relations to eliminate the illusions that occur during large language model generation. Then, based on the candidate relation list and combined with the knowledge graph, entities connected to the candidate relations are sampled. The semantic similarity between the relations and the knowledge graph entities is calculated using the language model, selecting the Top-k candidate entities, thus forming a high-quality inference subgraph. The specific implementation method of the semantic-guided subgraph retrieval module is as follows: The semantic-guided subgraph retrieval module accepts raw natural language questions. The set of subject entities involved in the problem And artificially designed prompts As input, where For natural language questions raised by users, Let m represent the set of entities related to the gold theme, where m represents the number of question entities. The goal of this module is to generate a high-quality reasoning subgraph that is highly semantically relevant to the question and capable of answering it, based on semantic guidance. This subgraph contains candidate relations and entities related to the problem, minimizing the "illusion" that may occur during LLM generation and providing sufficient structured evidence for subsequent subgraph structure modeling and LLM inference.

[0031] This module employs a thought-chain-driven problem decomposition mechanism. First, it utilizes prompt word engineering to semantically decompose the input question. Then, it designs task instructions. It includes a task description. Small sample examples and output format constraints The instruction can be represented as: ; in The roles and generation requirements of LLM are defined, such as "You are an assistant responsible for decomposing complex problems into subproblems" and "Please think step by step and decompose the given problem into multiple subproblems"; It contains several question-answer pairs to constrain the generation pattern and provide generation examples for LLM. Each example contains several decomposed examples of "real question - sub-question - relation - sufficiency judgment". These examples can effectively constrain the model's generation pattern and, to a certain extent, enable it to capture the mapping pattern between question semantics and knowledge graph relations. This allows the model to not only understand "how to decompose the question" but also "when to stop decomposing", thereby explicitly modeling the deep control mechanism of multi-hop reasoning. Specifying the output format involves using explicit tags such as "Step1", "Step2", and "Output" to regulate the hierarchy and structural consistency of the generated content, ensuring clear and controllable semantic boundaries between different reasoning steps. This is achieved by... Inputting into an LLM yields a sequence of subproblems. It can be represented as: ; in, Indicates the first The issue of height. The number of hops in the original problem is determined by the "information sufficiency judgment" of LLM during the iteration process. This represents a recursive problem decomposition function based on CoT hints, which takes the original problem as an example. With prompt template As input, the algorithm iterates through multiple rounds of "sub-problem generation → information sufficiency judgment" and outputs a semantically progressive sequence of sub-problems. For the generated first Size problem Its content consists of the answer generated in the previous step and the first step. Size problem Launch, when hour, This refers to the original question.

[0032] Secondly, for each subproblem generated The model first starts from the current entity set Starting from this point, retrieve all one-hop relationships connected to these entities in the knowledge graph, construct their topological neighborhood, and form a candidate relationship set: ;

[0033] Among them, when hour, For the initial set of gold-themed entities .

[0034] To balance retrieval efficiency and semantic alignment accuracy, this module employs a lightweight vector-based semantic similarity metric. Specifically, it uses the Sentence Vector Graph Encoder (SBERT) as the semantic encoder. The encoder will convert the sub-problems into their respective sub-problems. and each candidate relationship Encode it as a dense vector representation, and use cosine similarity. Calculate their semantic similarity: ;

[0035] in This indicates the calculation of the L2 norm of a vector. This represents the vector dot product operation.

[0036] Finally, the models are selected based on similarity scores. The best-matching relations form the candidate relation set for the current step. :

[0037] in Representing subproblems and all candidate relationships similarity, This indicates selecting the top-ranked relation from all candidate relation similarity scores. The relationship between the highest scores.

[0038] This "structure-first, semantic-second screening" mechanism avoids the illusion of semantically similar but structurally nonexistent relationships through the dual constraints of "structural reachability + semantic relevance," thus overcoming the limitations of traditional subgraph retrieval.

[0039] The candidate relation set indicates the potential direction of reasoning, while the answer to the question needs to be located at a specific entity node. Therefore, the candidate relation set... Building upon this foundation, the model needs to be further expanded to include entities. For each candidate relation... In the knowledge graph, sample all connected target entities, construct their structural neighborhoods, and form a candidate entity set: ;

[0040] However, while relation retrieval provides a clear direction for reasoning, a knowledge graph may connect a large number of neighboring entities under the same relation, many of which may be irrelevant to the current problem's intent. Entity expansion based solely on structural connections may introduce a large number of semantically irrelevant entities, potentially resulting in a large inference subgraph containing numerous unrelated nodes, thus reducing the accuracy of the reasoning. Therefore, this module introduces a second-level semantic filtering mechanism at the entity level, implementing a dual constraint of "structural reachability + semantic relevance." Specifically, for each sampled candidate entity... Similarly, using sub-problems As a semantic benchmark, calculate the semantic similarity between the entity text description and the sub-problems: ;

[0041] Finally, based on semantic similarity scores, the selected products are... The most relevant entity:

[0042] in Representing subproblems and all candidate entities semantic similarity, This indicates selecting the top-ranked entities from all candidate entity similarity scores. The entity with the highest score.

[0043] This process achieves precise expansion through dual constraints, ensuring that entities originate from real connections in the knowledge graph, and improving reasoning accuracy through semantic filtering.

[0044] After the above process, at each step A candidate relation set is obtained. and entity set Together, they constitute the extended edge set and node set of the subgraph:

[0045] in Indicates the first The updated knowledge subgraph is derived from the subgraph of the previous step. It is formed by merging with the newly extended set of triples, symbol The union operation represents the set operation. Describes the newly expanded set of triples. Indicates the first A set of entities in the step.

[0046] Iterate continuously until the final step of the reasoning chain is reached. Thus, the complete reasoning subgraph is obtained:

[0047] The module's final output is a high-quality inference subgraph. .

[0048] B) Input the high-quality inference subgraph into the subgraph structure encoding module; the subgraph structure encoding module first initializes the embedding representation of each node in the subgraph, and the graph encoder aggregates the neighbors of each node through an attention mechanism to obtain the aggregated representation of the node; then, it performs mean pooling aggregation on all nodes in the subgraph to construct the structure vector representation of the subgraph, thereby realizing subgraph structure modeling and fully capturing the semantic associations and graph structure dependencies between nodes. The specific implementation method of the subgraph structure encoding module is as follows: First, the subgraph structure encoding module receives high-quality reasoning subgraphs output by the semantically guided subgraph retrieval module. As input, the subgraph consists of a set of candidate nodes and their connections. Nodes contain entities from the knowledge graph, and edges represent semantic relationships between entities. For the set of entity nodes in the subgraph, Let be the set of relation edges in the subgraph. Assume the input inference subgraph is... Include Each node and Each edge can be initially represented by its attribute description or adjacency structure. A sentence vector encoder is used. Let the initial embedding representation of the node be denoted as ,in Embed dimensions for nodes.

[0049] ; At the same time, the semantic description of the label of each edge is also embedded and represented as denoted as , .

[0050] After obtaining the initial representation, the graph encoder performs neighbor aggregation for each node through a multi-layer attention mechanism. Specifically, for any node in the subgraph... First, calculate its relationship with its neighboring nodes. The attention weights in the current layer. These attention weights depend not only on the node's own representation. to the neighbors It also incorporates the relationship between the two. This is to capture both semantic and structural dependencies. Attention weights. It can be represented as: ; in Represents a node For its neighboring nodes In the Normalized attention weights on each attention head. Indicates the first The scoring vector of each attention head This indicates the transpose operation. The weight matrix represents the edge attributes (relation embedding). For activation function, For normalized exponential functions, This represents the weight matrix used for linear transformation node features in the nth attention head. Representing nodes respectively and In the The feature vector representation of the layer. Based on this weight, the first layer of each node... Layer representation aggregates neighbors through the following multi-head attention: ; in , express The set of neighboring nodes, express Size spliced ​​together Indicates the first The weight matrix of the head, Represents a node In the The feature vector representation of the layer. After... Layer iteration updates, the node is finally represented as: ; This integrates the semantic and structural information of multi-hop neighbors, thus enabling cross-entity context modeling capabilities.

[0051] After node aggregation is complete, the module needs to represent the entire subgraph as a global vector. This is used for downstream reasoning and question answering. To this end, pooling aggregation is performed on the representations of all nodes in the subgraph, which can be represented as: ; in This indicates mean pooling. Represents a node The vector representation, updated through multiple iterations, reflects the overall semantics of the subgraph.

[0052] The final subgraph structure encoding vector It captures key topological patterns, inference path features, and structural relationships between entities in the subgraph, which are important inputs for subsequent fusion inference.

[0053] C) Input the question text, subgraph structure representation, textual description of the subgraph, and cue words into the structure-semantic fusion module; utilize a soft-hard cue fusion strategy, the subgraph structure representation serves as a soft cue to guide the LLM in understanding the graph structure, while the subgraph is textualized into language cue to provide explicit reasoning information, achieving joint structure-semantic modeling and enhancing the model's structural compressibility and semantic interpretability. The specific implementation method of the structure-semantic fusion module is as follows: After obtaining the structured encoding of the subgraph, a hierarchical cue injection mechanism was designed, which integrates soft cues (continuous vectors) and hard cues (discrete text) to deliver graph structure information in different forms and hierarchically. Injecting LLM. First, the structure of the subgraph is encoded through a multi-layer nonlinear projection network. Mapped to a continuous vector of the same dimension as the LLM word embeddings. This ensures that the graph structure information can be effectively aligned with the LLM space: ; Among them, soft prompts As an additional virtual token injected into the beginning of the input sequence, it acts as a continuously learnable prefix vector. This design allows graph structure information to be encoded before the LLM processes the text sequence, providing structured contextual guidance for subsequent semantic understanding. Simultaneously, the triples in the subgraph are linearized into natural language descriptions as hard cues, providing explicit semantics for entities and relations.

[0054] Constructing the final input representation sequence At that time, the model follows the "structural embedding" Textual description Question text The sequential organization information of the "separator markers": ; in, This indicates a soft prompt, which is a structural embedding of subgraph information; Hard hints are embedded textual descriptions of subgraph triples; Embedding of the question text; These represent the start and end boundary symbols, respectively. This design ensures the organic integration of structural information, background knowledge, and problem semantics.

[0055] D) Input the fused semantic and structural representations into the LLM, and generate the final answer using an autoregressive generation method. The specific method of autoregressive generation is as follows: The answer generation process uses an autoregressive approach, meaning that LLM generates the answer sequence sequentially from left to right. At every moment The models will all be based on the sequences that have already been generated. By combining the fused hint vector, the next token can be predicted. The specific generation probability can be formalized as: ; E) During training, the negative log-likelihood loss function is used to calculate the loss value of the generated answer. The parameters of the graph neural network encoder and LLM are optimized through backpropagation. The LLM can use the low-rank adaptation (LoRA) technique to efficiently fine-tune the parameters to reduce training costs.

[0056] During the training process for answer reasoning generation, a causal language modeling objective is used to optimize the trainable parameters. To achieve efficient training with limited computational resources, low-rank adaptation (LoRA) is employed for efficient parameter fine-tuning of the LLM. During training, only the tokens for the answer portion are supervised, and the objective function is the standard negative log-likelihood. ; Finally, by iteratively minimizing the above loss function, the model can gradually learn to generate natural language answers that conform to the constraints of the knowledge graph under the condition of integrating graph structure and semantic information.

[0057] Table 1 shows a case requiring two-hop reasoning and its corresponding correct answer. LLM first decomposes the original question into a logically progressive sequence of sub-questions ["What country has Northern District?", "What type of government is used in Israel?"]; then it retrieves key triples (Nord-OuestDepartment, first_level_division_of, Haiti), (Haiti, languages_spoken, Haitian Creole) and (Haiti, countries_spoken_in, French) from the knowledge graph, forming a complete chain of reasoning evidence; finally, LLM generates the correct answer "French, Haitian Creole" based on this during the reasoning stage.

[0058] Original question The people from the country that containsNord-Ouest Departmentspeakwhat languages ​​today? Standard Answer French, Haitian Creole LLM decomposes the original problem into a sequence of subproblems. ["What country has Northern District?","What type of government is used in Israel?"] Subgraph retrieval is performed in the knowledge graph based on the sub-problem sequence, and the resulting reasoning subgraph contains these triples. (Nord-Ouest Department, first_level_division_of, Haiti),(Nord-OuestDepartment, administrative_division, m.0q3f16x),(Nord-OuestDepartment, partially_contains, Hispaniola),(Nord-Ouest Department,contains, Port-de-Paix),(Nord-Ouest Department, contains, Môle-Saint-Nicolas),(Haiti, languages_spoken, Haitian Creole),(Haiti,countries_spoken_in, French),(Peguero Jean Philippe, nationality,Haiti),(Gabard Fénélon, nationality, Haiti),(Jonathas Granville,nationality, Haiti),(Haiti, basin_countries, Artibonite River),(Haiti, national_anthem, m.0d9ggc4]) Predicted answer French, Haitian Creole Table 1 This embodiment also provides an LLM-driven knowledge graph question answering system, which includes a data acquisition module, a model building module, and a question answering reasoning module. The functions and connections of each module are as follows: The data acquisition module is used to acquire knowledge graph question-and-answer datasets and knowledge subgraphs; The model building module is connected to the data acquisition module and is used to build an LLM-driven knowledge graph question answering network model. The question-answering reasoning module is connected to the model building module and is used to call the trained model to perform knowledge graph question-answering tasks.

[0059] In this embodiment, the validation sets of the WebQSP and CWQ datasets are used for comparative validation. Table 1 shows the comparison results between the proposed method and other knowledge graph question answering methods on the WebQSP and CWQ datasets. As can be seen from Table 2, the method of this invention has higher answer accuracy compared to other knowledge graph question answering methods, specifically Hits@1 is the best.

[0060] Table 2

[0061] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A knowledge graph question-answering method driven by LLM thinking chain, characterized in that, Includes the following steps: Step 1: Obtain the knowledge graph question-answering dataset and knowledge subgraph; Step 2: Construct an LLM-driven knowledge graph question-answering network model; Step 3: Apply the trained LLM-driven knowledge graph question answering network model to the knowledge graph question answering task.

2. The knowledge graph question-answering method driven by LLM mind chain according to claim 1, characterized in that, The knowledge graph question-answering dataset contains the original natural language question and its corresponding question entity and answer entity. The knowledge subgraph provides the knowledge graph subgraph structure related to the original question, including entity set, relation set and multiple triples. The entities are derived from the dataset, and each entity contains the following fields: id, question, answer entity, question entity, and knowledge subgraph.

3. The knowledge graph question-answering method driven by LLM mind chain according to claim 1, characterized in that, The knowledge graph question-answering network model includes a semantically guided subgraph retrieval module, a subgraph structure encoding module, and a structure-semantic fusion module. The multiple modules work together to perform subgraph retrieval tasks, subgraph structure encoding tasks, and answer reasoning tasks. The semantically guided subgraph retrieval module utilizes a thought chain-driven problem decomposition mechanism to decompose the original natural language problem into one or more sub-problems, guide the direction of subgraph retrieval, and filter the candidate relation list most relevant to the semantics of the current sub-problem to retrieve high-quality reasoning subgraphs related to the problem. The subgraph structure encoding module performs structural modeling on high-quality reasoning subgraphs and aggregates them to form a subgraph structure representation. The structure-semantic fusion module uses the subgraph structure representation as a soft cue, integrates the textual representation of subgraph triples and the semantic representation of the question, and achieves the fusion of semantics and structure, thereby guiding the generation of the final answer; The constructed LLM-driven knowledge graph question answering network model was trained using a knowledge graph question answering dataset.

4. The knowledge graph question-answering method driven by LLM mind chain according to claim 3, characterized in that, The implementation method of the LLM-driven knowledge graph question-answering network model is as follows: Step 2.1: Input the original natural language question, question entity, and prompt words into the semantic guidance subgraph retrieval module; The semantically guided subgraph retrieval module utilizes a thought-chain-driven problem decomposition mechanism. First, through prompt word engineering, based on the designed task instructions, few-sample examples, and output format constraints, it guides the LLM to gradually decompose the original natural language problem, generating a sequence of sub-problems and guiding the direction of subgraph retrieval. For each sub-problem, it performs two-level semantic matching: "sub-problem-relation" and "relation-entity". First, it calculates the semantic similarity between the sub-problem and the knowledge graph relationship using a pre-trained language model, selecting the Top-k candidate relations to eliminate the illusions that occur when generating large language models. Then, based on the candidate relation list and combined with the knowledge graph, it samples entities connected to the candidate relations, calculates the semantic similarity between the relations and the knowledge graph entities using a language model, and selects the Top-k candidate entities, thus forming a high-quality reasoning subgraph. Step 2.2: Input the high-quality reasoning subgraph into the subgraph structure encoding module; the subgraph structure encoding module first initializes the embedding representation of each node in the subgraph, and the graph encoder aggregates the neighbors of each node through the attention mechanism to obtain the aggregated representation of the node; then it performs mean pooling aggregation on all nodes in the subgraph to construct the structure vector representation of the subgraph, thereby realizing the subgraph structure modeling and fully capturing the semantic associations and graph structure dependencies between nodes; Step 2.3: Input the question text, subgraph structure representation, subgraph textual description, and prompt words into the structure-semantic fusion module; By employing a fusion strategy of soft and hard cues, the subgraph structure representation serves as a soft cue to guide the LLM in understanding the graph structure, while the subgraph is textualized into a language cue to provide explicit reasoning information, thereby achieving joint structure-semantic modeling and enhancing the structural compressibility and semantic interpretability of the model. Step 2.4: Input the fused semantic and structural representations into the LLM, and generate the final answer using an autoregressive generation method; Step 2.5: During training, the negative log-likelihood loss function is used to calculate the loss value of the generated answer. The parameters of the graph neural network encoder and LLM are optimized through backpropagation. The LLM can use the low-rank adaptive LoRA technique to efficiently fine-tune the parameters to reduce training costs.

5. The knowledge graph question-answering method driven by LLM mind chain according to claim 4, characterized in that, The specific method for autoregressive generation is as follows: LLM generates the answer sequence sequentially from left to right. At every moment Knowledge graph question answering network models are all based on previously generated sequences. By combining the fused hint vector, the next token can be predicted. The specific generation probability can be formalized as: ； in, This means that LLM predicts the next token based on the input and the previous generated token. The probability distribution.

6. The knowledge graph question-answering method driven by LLM mind chain according to claim 4, characterized in that, The negative log-likelihood loss function The formula is: ； Among them, the summation symbol This represents the sequence of answers from the first position to the last position. The losses of all time steps (token positions) are summed up; Finally, by iteratively minimizing the above loss function, the knowledge graph question answering network model can gradually learn to generate natural language answers that conform to the constraints of the knowledge graph under the condition of integrating graph structure and semantic information.

7. The knowledge graph question-answering method driven by LLM mind chain according to claim 3, characterized in that, The specific implementation method of the semantic-guided subgraph retrieval module is as follows: The semantic-guided subgraph retrieval module accepts raw natural language questions. The set of subject entities involved in the problem And artificially designed prompts As input, where For natural language questions raised by users, Let m represent the set of entities related to the gold theme, and m represent the number of question entities. The goal of the semantically guided subgraph retrieval module is to generate a high-quality reasoning subgraph that is highly semantically relevant to the question and capable of answering it, based on semantic guidance. This subgraph contains candidate relations and entities related to the problem; The semantic-guided subgraph retrieval module employs a thought-chain-driven problem decomposition mechanism. First, it uses prompt word engineering to semantically decompose the input question and design task instructions. It includes a task description. Small sample examples and output format constraints Task instructions It can be represented as: ； in The roles and generation requirements of LLM are defined, by... Inputting into an LLM yields a sequence of subproblems. It can be represented as: ； in, Indicates the first The issue of height. This represents the number of hops in the original problem. This represents the recursive problem decomposition function based on CoT hints, for the generated first... Size problem Its content consists of the answer generated in the previous step and the first step. Size problem Launch, when hour, Represent the original question; Secondly, for each subproblem generated The model first starts from the current entity set Let's start with knowledge graphs. Retrieve all one-hop relationships connected to these entities, construct their topological neighborhoods, and form a candidate relationship set: ； Among them, when hour, For the initial set of gold-themed entities ; The semantic-guided subgraph retrieval module employs a lightweight vector-based semantic similarity metric, using a sentence vector encoder as the semantic encoder. The encoder will convert the sub-problems into their respective sub-problems. and each candidate relationship Encode it as a dense vector representation, and use cosine similarity. Calculate their semantic similarity: ； in This indicates the calculation of the L2 norm of a vector. This represents the vector dot product operation; Finally, the models are selected based on similarity scores. The best-matching relations form the candidate relation set for the current step. : ； in Representing subproblems and all candidate relationships similarity, This indicates selecting the top-ranked relation from all candidate relation similarity scores. The relationship between the highest scores; For each candidate relation In the knowledge graph, sample all connected target entities, construct their structural neighborhoods, and form a candidate entity set. : ； The semantic-guided subgraph retrieval module introduces a second-level semantic filtering mechanism at the entity level, implementing dual constraints of structural reachability and semantic relevance for each sampled candidate entity. sub-problems As a semantic benchmark, calculate the semantic similarity between the entity text description and the sub-problems: ； Finally, based on semantic similarity scores, the selected products are... The most relevant entity: ； in Representing subproblems and all candidate entities semantic similarity, This indicates selecting the top-ranked entities from all candidate entity similarity scores. The entity with the highest score; The process achieves precise expansion through dual constraints, ensuring that entities originate from real connections in the knowledge graph, and improving reasoning accuracy through semantic filtering. After the above process, at each step A candidate relation set is obtained. and entity set Together, they constitute the extended edge set and node set of the subgraph: ； in Indicates the first The updated knowledge subgraph is derived from the subgraph of the previous step. It is formed by merging with the newly extended set of triples, symbol The union operation represents the set operation. Describes the newly expanded set of triples. Indicates the first The set of entities in the step; Iterate continuously until the final step of the reasoning chain is reached. Thus, the complete reasoning subgraph is obtained: ； The final output is a high-quality inference subgraph. .

8. The knowledge graph question-answering method driven by LLM mind chain according to claim 5, characterized in that, The specific implementation method of the subgraph structure encoding module is as follows: First, the subgraph structure encoding module receives high-quality reasoning subgraphs output by the semantically guided subgraph retrieval module. As input, the subgraph consists of a set of candidate nodes and their connections. Nodes contain entities from the knowledge graph, and edges represent semantic relationships between entities. For the set of entity nodes in the subgraph, This is the set of relation edges in the subgraph; Input inference subgraph ,Include Each node and Each edge can be initially represented by its attribute description or adjacency structure, using a sentence vector encoder. Let the initial embedding representation of the node be denoted as ,in Embed dimensions for nodes; ； At the same time, the semantic description of the label of each edge is also embedded and represented as denoted as , ; After obtaining the initial representation, the graph encoder performs neighbor aggregation on each node through a multi-layer attention mechanism. For any node in the subgraph... First, calculate its relationship with its neighboring nodes. The attention weights in the current layer depend not only on the node's own representation. to the neighbors It also incorporates the relationship between the two. To capture the dual features of semantic and structural dependencies, attention weights It can be represented as: ； in, Represents a node For its neighboring nodes In the Normalized attention weights on each attention head. Indicates the first Transpose of the scoring vector of each attention head The weight matrix representing the edge attributes. For activation function, For normalized exponential functions, This represents the weight matrix used for linear transformation node features in the nth attention head. Representing nodes respectively and In the The feature vector representation of a layer; Based on this weight, each node's... Layer representation Aggregate neighbors through the following multi-headed attention: ； in , express The set of neighboring nodes, express Size spliced ​​together Indicates the first Weight matrix of the size, Represents a node In the The feature vector representation of the layer, after Layer iterative updates, nodes The final representation is: ； It integrates the semantic and structural information of multi-hop neighbors, thus enabling cross-entity context modeling capabilities; After node aggregation is complete, the module represents the entire subgraph as a global vector. This is used for downstream reasoning and question answering. Therefore, pooling aggregation is performed on the representations of all nodes in the subgraph, which can be represented as: ； in This indicates mean pooling. Represents a node The vector representation, updated through multiple iterations, reflects the overall semantics of the subgraph. The final subgraph structure encoding vector It captures key topological patterns, inference path features, and structural relationships between entities in the subgraph, which are important inputs for subsequent fusion inference.

9. The knowledge graph question-answering method driven by LLM mind chain according to claim 3, characterized in that, The specific implementation method of the structure-semantic fusion module is as follows: After obtaining the structured encoding of the subgraph, a hierarchical hint injection mechanism is added, which integrates soft and hard hints to present graph structure information in different forms and at different levels. Injecting into LLM; First, the structure of the subgraph is encoded through a multi-layer nonlinear projection network. Mapped to a continuous vector of the same dimension as the LLM word embeddings. This ensures that the graph structure information can be effectively aligned with the LLM space: ； Among them, soft prompts As an additional virtual token, it is injected into the beginning of the input sequence to serve as a continuous learnable prefix vector; at the same time, the triples in the subgraph are linearized into natural language descriptions as hard cues, providing explicit semantics for entities and relations. Constructing the final input representation sequence At that time, the model follows structural embedding and textual description. Question text Separator order organization information: ； in, This indicates a soft prompt, which is a structural embedding of subgraph information; Hard hints are embedded textual descriptions of subgraph triples; Embedding of the question text; These represent the start boundary symbol and the end boundary symbol, respectively.

10. A knowledge graph question-answering system driven by LLM thinking chain, characterized in that, The system includes a data acquisition module, a model building module, and a question-answering reasoning module. The functions and connections of each module are as follows: The data acquisition module is used to acquire knowledge graph question-and-answer datasets and knowledge subgraphs; The model building module is connected to the data acquisition module and is used to build an LLM-driven knowledge graph question answering network model. The question-answering reasoning module is connected to the model building module and is used to call the trained model to perform knowledge graph question-answering tasks.

Images