Method and system for constructing knowledge graph question answering model based on reflection and experience enhanced knowledge

Abstract

The application provides a knowledge graph question answering model construction method and system, which utilizes a teacher model to construct an initial training data set, and trains a strategy model to be trained on the basis of the initial training data set to obtain an initial strategy model, and then takes the initial strategy model and the training data set as a current strategy model and a training data set respectively, and iterates the following steps multiple times: using the current strategy model to answer each question in the current training data set; evaluating whether the answers are correct; for incorrect answers, guiding the current strategy model to reflect and answer again and evaluating whether the answers are correct; adding correct answers, reasoning tracks and questions to the current training data set; and incrementally fine-tuning the current strategy model based on the updated current training data set. The knowledge graph question answering model constructed in this way can support more accurate multi-round interactive question answering.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, specifically to a method and system for constructing knowledge graph question-answering models. Background Technology

[0002] A knowledge graph is a semantic network that describes entities, concepts, and their relationships in the real world in a structured form. Its basic building block is the triple, formally represented as a head entity, a relation, and a tail entity. Multiple triples are interconnected to form a graph-like knowledge graph structure. In this structure, there are multiple entities similar to nodes, and each entity is connected by a predicate (i.e., the relation in the triple). Knowledge graph question answering technology aims to understand natural language questions posed by users and provide accurate answers based on structured knowledge graphs through reasoning or retrieval. It is one of the key technologies in the fields of artificial intelligence and natural language processing.

[0003] Existing knowledge graph question answering methods can be mainly categorized into four types: (1) Subgraph retrieval-based methods (retrieve related entities and their relationships from one entity) adopt a two-stage process of "retrieval-reasoning". This process lacks a reflection mechanism, and errors between retrieval and reasoning will accumulate; (2) Semantic parsing-based methods convert questions into formal queries such as SPARQL and execute them on the knowledge graph. They have high accuracy, but rely on a large amount of manual annotation of "question-query" and are highly bound to the knowledge graph pattern, making cross-graph migration difficult; (3) Large language model enhancement methods use large language models as subgraph retrieval machines or semantic parsers to improve the above two methods. Essentially, they are still pipeline modeling and cannot actively interact with the knowledge graph in multiple steps; (4) Agent-based reasoning methods treat large language models as agents, making them follow the "thinking-action-observation" cycle paradigm in the knowledge graph environment, calling query tools step by step and dynamically adjusting the reasoning direction according to environmental feedback, unifying question understanding, knowledge query and answer reasoning into the interaction process.

[0004] Compared to the previous three methods, agent-based reasoning methods can proactively explore the knowledge graph using a knowledge graph question-answering model, dynamically adjust subsequent actions based on intermediate execution results, and integrate retrieval, parsing, and reasoning into a unified interactive process. This results in greater flexibility and scalability when facing complex problems such as multi-step reasoning and multi-constraint aggregation. In fact, agent-based reasoning methods employ a multi-turn interactive knowledge graph question-answering model (which can also be referred to as a strategy model in this paper), which follows a "think-action-observation" cyclical paradigm in the knowledge graph environment. Thinking refers to the strategy model's reasoning and analysis based on currently acquired information, planning the next action to be taken, reflecting the strategy model's reasoning and decision-making process; action refers to the strategy model calling specific query tools to query and perceive the knowledge graph based on the thinking results; observation refers to the strategy model receiving feedback results returned after the actions are executed in the knowledge graph environment and using them as input for the next round of thinking. These three stages are repeated until the strategy model determines that it has obtained sufficient information and outputs the final answer. Summary of the Invention

[0005] However, during their research on knowledge graph question answering technology based on agent-based reasoning, the inventors discovered that while this method, which supports multi-turn interactive knowledge graph question answering, can handle complex problems requiring multi-step reasoning, it still has shortcomings in the construction of training data for policy models and the reuse of experience for difficult problems during reasoning. Specifically, when training the policy model, a process-supervised decomposition approach is used to construct training data. This relies on pre-annotated logical forms (such as formal query statements like SPARQL), that is, a correctly annotated logical form is decomposed into step-by-step operations according to its query structure, thereby obtaining the operations that the policy model should execute, and using this as a reference answer during training. However, this training data constructed based on process-supervised decomposition is "idealized," containing only the interactive tool call sequences decomposed from the annotated logical forms, and not including abnormal feedback and failed attempts that the policy model may encounter in real interactions, such as empty results, timeouts, and syntax errors. This results in the trained policy model lacking the ability to handle abnormal situations and recover from failures. At the same time, this construction method, which relies on high-quality logical form annotations, has high annotation costs in low-resource scenarios. Another approach to constructing training data is based on generating training data from a large, closed-source model. This method uses a powerful, closed-source model as the teacher model, actually performing question-and-answer tasks within a knowledge graph environment. The multi-turn interaction records generated during this process are collected and used as training data to train a smaller student model (the policy model to be trained). This approach is costly, and the difference in capabilities between the teacher model and the policy model to be trained means that the reasoning trajectories generated by the teacher cannot specifically optimize the weaknesses of the policy model itself, making it difficult to steadily improve the policy model's reflection and error recovery capabilities. Furthermore, existing knowledge graph question-answering models generally struggle with long-tail problems, which refer to a distribution of questions where a few questions (head questions) appear very frequently, while most questions (tail questions) appear very infrequently. The few head questions occupy most of the attention and resources, while the tail questions, although numerous, lack sufficient samples due to their low frequency. Therefore, underfitting to tail questions often occurs during training. Consequently, existing knowledge graph question-answering models have weak generalization ability for low-frequency, high-difficulty query patterns.

[0006] While conducting research on knowledge graph question-answering technology, the inventors discovered that the key to enabling open-source, low-parameter strategy models to possess stable multi-turn interactive reasoning capabilities lies not in acquiring more idealized training data, but in establishing a training mechanism that allows the model to continuously improve from its own real-world trial and error. Specifically, regarding the shortcomings of existing technologies, the inventors believe that: First, to address the problem of "idealized" training data leading to a lack of anomaly handling capabilities, multi-turn interactive reasoning trajectories containing real interactive tool calls, anomaly feedback, and failed attempts can be constructed as training data, rather than simply decomposing logical forms to obtain anomaly-free idealized reasoning trajectories. This allows the strategy model to handle anomalies such as empty results, timeouts, and syntax errors from the cold start stage. Second, to address the problem of high costs and inability to optimize weak areas of student models due to reliance on closed-source teacher models, it is possible to explore using the strategy model's own reflective capabilities to identify the causes of erroneous reasoning trajectories generated during its autonomous exploration and generate targeted corrective reasoning trajectories without introducing external strong teacher models. This transforms failed samples into training data targeting the model's own weaknesses, thereby steadily improving its reflective and error recovery capabilities. Third, regarding the lack of a mechanism for accumulating and reusing lessons learned from failures in long-tail problems, for samples that cannot be corrected through self-reflection, we can try to abstract and precipitate their lessons learned into structured external knowledge that can be retrieved and reused semantically during reasoning. This can compensate for the shortcomings of the model's inherent capabilities and improve its generalization ability for low-frequency, high-difficulty query patterns without increasing the model's parameters.

[0007] This application aims to provide a new solution for multi-turn interactive knowledge graph question answering, in order to address the problems of existing multi-turn interactive knowledge graph question answering methods, such as the lack of real trial and error processes in training data, insufficient model reflection and correction capabilities, and poor generalization to tail questions.

[0008] The above objective is achieved through the following technical solution:

[0009] According to a first aspect of the embodiments of this application, a method for constructing a knowledge graph question-answering model is provided, comprising the following steps: S1: Constructing an initial training dataset using a teacher model, and training a policy model to be trained on the basis of the initial training dataset to obtain an initial policy model, wherein each sample in the training dataset includes at least a question, the correct answer to the question, and an inference trajectory for obtaining the predicted answer to the question; S2: Using the initial policy model as the current policy model and the initial training dataset as the current training dataset, performing the following iterative training steps: S21 Using the current policy model in a knowledge graph environment to generate a predicted answer for each question in the current training dataset and an inference trajectory for obtaining the predicted answer to the question; S22 S23. Evaluate the degree of deviation between the predicted answer and the correct answer for each question; S24. For predicted answers with a deviation greater than a set threshold, use the corresponding reasoning trajectory and question, along with the degree of deviation, as input to guide the current policy model to generate a corrected predicted answer and its corresponding reasoning trajectory, and evaluate the degree of deviation between the corrected predicted answer and the correct answer; S25. Add predicted answers with a deviation less than the set threshold, along with their corresponding reasoning trajectory and question, as new samples to the current training dataset; S26. Incrementally fine-tune the current policy model based on the updated current training dataset; Repeat steps S21-S25 until the preset number of iterations is met; S3: Use the current policy model as the constructed knowledge graph question answering model.

[0010] In some embodiments, step S1, which involves constructing an initial training dataset using a teacher model, further includes: constructing a hybrid question set consisting of multiple questions, their correct answers, and reference queries; providing an interactive tool for interacting with the knowledge graph; using the teacher model in the knowledge graph environment to generate a predicted answer and at least three inference paths for obtaining the predicted answer for each question in the hybrid question set via the interactive tool, wherein the inference path must satisfy the following conditions: its corresponding predicted answer hits the correct answer and the predicted answer appears in the observation history of the inference path; for each question, adding the question, the correct answer to the question, the inference path for obtaining the predicted answer to the question, and the reference query as samples to the training dataset; and training the policy model to be trained based on the initial training dataset in step S1 includes: performing supervised fine-tuning of the policy model to be trained based on the initial training dataset.

[0011] In some embodiments, assessing the degree of deviation between the predicted answer and the correct answer for each question includes calculating a reward score for the predicted answer and the correct answer for each question using the F1 reward function, where a higher reward score indicates a smaller degree of deviation.

[0012] In some embodiments, the training dataset also includes a reference query statement for each question; and in step S23, for predicted answers with a deviation greater than a set threshold, the reference query statement of the predicted answer, along with the question, the inference trajectory corresponding to the predicted answer, and the deviation, are used as input to guide the current policy model to generate a corrected predicted answer and its corresponding inference trajectory, and the current policy model is restricted from directly using the reference query statement.

[0013] In some embodiments, the training dataset also includes a reference query statement for each question, and after step S23, the following steps are further included: for predicted answers with a deviation greater than a set threshold, the predicted answer, its corresponding reasoning trajectory and question, the correct answer, and the corresponding reference query statement are used as input to guide the current policy model to generate experience entries corresponding to the question. The experience entries include at least an error pattern field and an experience field. The error pattern field is a general description of common errors in this type of question, and the experience field includes the reasoning trajectory for obtaining the correct answer using the reference query statement for the question and the reason for each step in the reasoning trajectory; the generated experience entries and their corresponding questions are added to the experience base.

[0014] In some embodiments, the iterative training step S2 further includes: for a question whose predicted answer deviation is greater than a set threshold and for which a corresponding experience entry has been generated, calculating the similarity between the question and each question in the experience base, selecting the experience entry corresponding to the question whose similarity exceeds a set similarity threshold; and providing the selected experience entry to the current policy model to guide the current policy model to regenerate multiple inference trajectories for the corresponding question.

[0015] Evaluate the degree of deviation between the predicted answer and the correct answer to the question for each reasoning trajectory; add the predicted answer with the smallest deviation and its corresponding reasoning trajectory and question as new samples to the current training dataset.

[0016] In some embodiments, step S21 further includes: calculating the similarity between each question in the current training dataset and each question in the experience base, selecting the experience entry corresponding to the question whose similarity exceeds a set similarity threshold, and providing the selected experience entry to the current policy model to guide the current policy model to generate a predicted answer for each question in the current training dataset and obtain the reasoning trajectory of the predicted answer for that question in the knowledge graph environment.

[0017] According to a second aspect of the embodiments of this application, a knowledge graph question answering method is provided, which utilizes a knowledge graph question answering model constructed according to the method described in the first aspect of the embodiments of this application to generate answers to input questions.

[0018] According to a third aspect of the embodiments of this application, a computer program product is provided, including computer instructions that, when executed by a processor, implement the steps of the method described according to the first and / or second aspects of the embodiments of this application.

[0019] According to a fourth aspect of the embodiments of this application, a knowledge graph question-answering model construction system is provided, including a memory and a processor; the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described according to the first aspect of the embodiments of this application.

[0020] Compared with the prior art, the solution according to the embodiments of this application has at least the following technical effects:

[0021] First, the embodiments of this application preserve the authentic trial-and-error process. Compared to maskless training, it avoids the interference of environmental observation terms on gradient calculation, enabling the model to possess basic multi-round "think-action-observation" interaction capabilities from the cold start stage. Second, the embodiments of this application also propose a self-reflective inference trajectory correction strategy, utilizing the model's own capabilities to correct failed samples throughout the process, without the need to introduce external strong teachers. Failed samples are transformed into training data targeting the current model's weaknesses, achieving iterative optimization of the training data. Furthermore, the embodiments of this application transform lessons learned from failures in long-tail problems that remain unresolved into external experience, compensating for the model's inherent capabilities without increasing model parameters; simultaneously, experience-guided inference data construction eliminates the inconsistency between training and inference distributions. Attached Figure Description

[0022] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0023] Figure 1 A flowchart illustrating a method for constructing a knowledge graph question-answering model according to an embodiment of this application is shown. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description, in conjunction with the accompanying drawings and specific embodiments, further illustrates this application. It should be understood that the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0025] As mentioned above, existing multi-turn interactive knowledge graph question answering methods suffer from problems such as a lack of real-world trial-and-error processes in training data, insufficient model reflection and correction capabilities, and poor generalization to tail-end questions. This application proposes a knowledge graph question answering solution based on reflection and experience enhancement. It employs a multi-stage training and multi-turn iteration framework to train the strategy model, enabling the model to continuously improve its multi-turn interactive planning capabilities by utilizing successful and failed reasoning trajectories generated during its own exploration, without relying on external strong teacher supervision or process annotation. This solution revolves around an iterative closed loop of "exploration-reflection-experience consolidation-strategy update," categorizing failed reasoning trajectories generated by the strategy model itself into two types: those that can be reflected and corrected, and those that cannot. For failed reasoning trajectories that can be reflected and corrected, the model's own capabilities are used to correct the failed samples, eliminating the need for external strong teachers. These failed samples are transformed into training data targeting the current model's weaknesses, achieving iterative optimization of the training data and thus improving the strategy model's capabilities from the training side. For failed reasoning trajectories that cannot be reflected and corrected, experience consolidation is used to transform them into external memory, compensating for the model's capabilities from the reasoning side. In other words, lessons learned from failures that cannot be resolved through reflection are transformed into external memories, compensating for the model's intrinsic capabilities without increasing model parameters. Simultaneously, experience-guided inference data construction eliminates the inconsistency between training and inference distributions. Thus, during the training of the policy model, the model and experience base are continuously iterated and updated through these two different processing paths. Simple errors that can be corrected through reflection are incorporated into model parameters, while difficult errors that are hard to correct through reflection are preserved as external experience, enabling the policy model to achieve steady performance improvements on samples of varying difficulty.

[0026] Figure 1 A flowchart of a knowledge graph question-answering model construction method according to an embodiment of this application is provided. Figure 1 As shown, the method mainly includes: S1: Constructing an initial training dataset using a teacher model, and training the policy model to be trained on the basis of the initial training dataset to obtain an initial policy model; S2: Using the initial policy model and the training dataset as the current policy model and the training dataset respectively, iteratively executing the following steps: using the current policy model to answer each question in the current training dataset; evaluating whether these answers are correct; for incorrect answers, guiding the current policy model to answer again and evaluating whether they are correct; adding the correct answers and their reasoning trajectories and questions to the current training dataset; incrementally fine-tuning the current policy model based on the updated current training dataset; S3: Using the current policy model as the constructed knowledge graph question answering model.

[0027] Specifically, in S1, firstly, a hybrid question set consisting of multiple question types is constructed based on various question datasets that are widely used in the knowledge graph question answering field. The hybrid question set can include one-hop questions, two-hop questions, multi-constraint combination questions, topic entity questions, etc. The constructed hybrid question set includes not only the question but also the correct answer to the question, as well as the corresponding reference query statement (e.g., a formalized query statement that correctly answers the question). The WebQSP, CWQ, and GrailQA datasets are all widely used question datasets. One-hop and two-hop questions can be sampled in a balanced manner from the WebQSP dataset (a one-hop question is answered with only one inference / retrieval, and the same applies to two-hop questions). Multi-constraint combination questions (questions with at least two constraints) can be upsampled from the CWQ dataset. Topic entity questions (questions without a specified subject) can be upsampled from the GrailQA dataset. The hybrid question set constructed in this way can cover different types of questions with varying inference depths and question patterns, thus avoiding the situation where the subsequently trained policy model can only solve a single type of question.

[0028] Next, based on the constructed mixed problem set, an initial training dataset is built using a teacher model. This training dataset is subsequently used to train the policy model. Typically, a high-performance, large model is chosen as the teacher model. For example, Gemini 2.5 Flash or DeepSeek-V3 can be used as the teacher model. This application does not impose any restrictions on the choice of teacher model.

[0029] To enable the teacher model to answer each question in a hybrid question set within a knowledge graph environment, embodiments of this application also provide interactive tools for interacting with the knowledge graph, allowing the model to retrieve information from the knowledge graph. In one embodiment, the following three basic interactive tools are provided: (1) a type retrieval tool, which returns candidate types sorted by semantic similarity given a type name, used to initialize queries with type constraints when topic entities are missing, or to introduce type restrictions when there are too many intermediate results; (2) a subgraph pattern exploration tool, which returns one- or two-hop candidate subgraph patterns related to the query variables given a partial query sketch, which can be rearranged according to the expected predicate semantics via parameters, used to discover scalable relational paths and complete predicates; and (3) a query execution tool, which executes complete or partial query statements and returns results, used to verify intermediate hypotheses and obtain the final answer. In some embodiments, one or more other interactive tools may be added as needed to provide more means for the model to interact with the knowledge graph.

[0030] Then, using the teacher model in a knowledge graph environment, interactive tools generate predicted answers for each question in the mixed question set. These predicted answers are compared with the correct answer to the question. If they match or are similar, the question, the correct answer, and the reasoning trajectory used to obtain the predicted answer are added as a training sample to the initial training dataset. The reasoning trajectory refers to the complete record of the model's problem-solving process, including the model's thought process, actions, observations returned by the environment, and the predicted answer output by the model, where the predicted answer consists of one or more tokens. In one embodiment, a subset of questions from the mixed question set can be selected to construct the initial training dataset, while the remaining questions are used in subsequent processing stages. In some embodiments, the training sample may include not only the question, the correct answer, and the reasoning trajectory used to obtain the predicted answer to the question, but also a reference query statement for obtaining the correct answer to the question.

[0031] In another embodiment, a prompt template containing a description of the problem to be solved, the provided interactive tools, and high-quality manually written examples is first constructed to instruct the teacher model to use the provided interactive tools to generate a predicted answer for each problem and output multiple candidate inference trajectories to obtain the predicted answer, for example, up to three. Then, the multiple candidate inference trajectories generated by the teacher model are filtered by a result-based strategy, retaining inference trajectories that simultaneously meet the following conditions: (1) Correct answer condition: the predicted answer hits at least one of the correct answers; (2) Evidence anchoring condition: the predicted answer appears in the observation history of the inference trajectory, that is, the predicted answer is indeed found by the model from the knowledge graph during the interaction, rather than guessed or fabricated by the model based on prior knowledge. The scheme of this embodiment can avoid too many similar repetitive inference trajectories and a single inference method filling the dataset, and can allow the subsequent policy models to be trained to learn multiple legitimate problem-solving ideas of the teacher model under the same problem, thereby improving the policy flexibility of the measurement model. In the process of constructing the training dataset, the teacher model is used as training data through multiple rounds of interaction trajectories involving calls to real knowledge graph interaction tools, abnormal feedback, and failed attempts, rather than simply decomposing the logical form in reverse to obtain an ideal trajectory without abnormalities. This allows the subsequent training strategy model to have the ability to handle abnormal feedback such as empty results, timeouts, and syntax errors from the very beginning.

[0032] Continue to refer to Figure 1 In step S1, the policy model to be trained is trained on the initial training dataset to obtain the initial policy model. The policy model to be trained can be an open-source large language model that has been pre-trained on a large corpus and has general language understanding and generation capabilities, such as Meta's LLaMA large language model, Alibaba's Qwen series of open-source models, etc.

[0033] Using each sample in the training dataset as input, the policy model is trained in a supervised manner so that it gradually learns to solve knowledge graph question-answering problems in accordance with the teacher model's "think-action-observation" approach.

[0034] In some embodiments, a loss masking mechanism can be used to supervise the fine-tuning of the parameters of the policy model to be trained, wherein the loss function is defined as:

[0035] (1)

[0036] Where θ represents the set of all parameters in the policy model, L SFT (θ) is the loss function, D SFT Given the current training dataset, Q and τ * These are the questions in the training dataset and their corresponding inference trajectories, τ. * agent For the inference trajectory τ * The set of lexical units generated by the policy model (excluding observed lexical units returned by the environment, i.e., lexical units observed by the policy model from the external problem rather than derived by itself), y t p is a single lexical in the lexical set. θ (y t | y <t , Q) represents the policy model where y is the next term in the objective in problem Q. t The conditional probability of the assignment. The loss function is the negative log-likelihood of the target word. That is, in incremental fine-tuning, the policy model to be trained is the next target word y. t The higher the conditional probability of the assignment, the lower the loss function. Therefore, training is guided by the goal of achieving a lower loss function, ultimately training the policy model to be trained into the initial policy model π. θ0 This mechanism only considers the lexical units generated by the policy model, thus shielding the influence of observed lexical units returned by the environment on the gradient. This allows the policy model to focus on learning inference and interactive tool invocation capabilities, resulting in the initial policy model π. θ0 .

[0037] In this embodiment, step S1 preserves the real trial-and-error process compared to the traditional process-supervised decomposition method, and avoids the interference of environmental observation terms on gradient calculation compared to the lossless mask training method, so that the policy model to be trained has basic multi-round "think-action-observation" interaction capabilities from the cold start stage.

[0038] It should be understood that the above embodiments are preferred embodiments of this application. In some embodiments, the methods in the prior art described above can also be used to construct an initial training dataset through a teacher model, and the policy model to be trained can be trained based on the initial training dataset to obtain the current policy model.

[0039] In step S2, the initial policy model π is used. θ0 Using the initial training dataset as the current policy model, the following iterative training steps are performed:

[0040] Step S21, utilize the current policy model π θi In a knowledge graph environment, for each question Q in the current training dataset i The reasoning trajectory τ for generating the predicted answer and obtaining the predicted answer to the question. i This strategy model also uses the interactive tools provided above to obtain relevant information from the knowledge graph.

[0041] Step S22: Evaluate the degree of deviation between the current policy model's predicted answer for each question and the correct answer for that question. For example, an F1-based reward function can be used. For each inference trajectory τ i Calculate the corresponding reward score R(τ) i This score is used to measure the degree of bias. This reward score, also known as the F1 score, is used to evaluate the Q-squared problem. i Predicted answer With the correct answer Both sets are consistent. F1 is the harmonic mean of precision and recall, calculated based on the predicted and correct answers, used to comprehensively and balancedly evaluate the overall performance of the model. Precision is the proportion of words in the predicted answer that actually belong to the correct answer. Recall is the proportion of words in the correct answer that were successfully predicted by the predicted answer. The F1 score ranges from 0 to 1; a higher score indicates a closer match between the predicted and correct answers, and a smaller deviation. An F1 score of 1 indicates a perfect match. In some other embodiments, other methods may be used to assess the deviation between the predicted and correct answers.

[0042] Step S23: For predicted answers with a deviation greater than a set threshold, use the corresponding reasoning trajectory and question, along with the deviation level, as input to guide the current policy model π. θiThe system generates revised predicted answers and their corresponding reasoning trajectories, and evaluates the degree of deviation between these revised predictions and the correct answer. For example, if the reward score of a reasoning trajectory is less than a pre-set reward score threshold ρ, the deviation between the predicted answer and the correct answer for that trajectory is considered to be greater than the set threshold. Then, these reasoning trajectories with deviations greater than the set threshold, along with the questions and the degree of deviation, are used as input to guide the current policy model π. θi Further, a revised predicted answer and its reasoning trajectory are generated. In some preferred embodiments, the input provided to the current policy model may also include a reference query statement corresponding to the question. The purpose of introducing the reference query statement is to provide structural hints for the correct reasoning path, enabling the current policy model to... θi It can identify key errors in the original inference trajectory, such as omitted constraints or incorrectly selected relations (i.e., guiding the model to self-reflect). In this case, the prompt template provided to the policy model should explicitly prohibit the current policy model π. θi The answer is obtained directly using the reference query statement; the current strategy model is π. θi The entities and predicates in the problem still need to be re-validated independently through interactive tools.

[0043] After obtaining further revised inference trajectories, it is necessary to further evaluate the degree of deviation between the revised predicted answer and the correct answer. For revised predicted answers and their inference trajectories with reduced deviation, they are retained as predicted answers and inference trajectories; otherwise, they are discarded. In some embodiments where the reward score of the predicted answer is calculated based on the F1 reward function, the reward functions of the aforementioned inference trajectories with deviations greater than a threshold and the revised inference trajectories for their corresponding questions are compared. If the reward function of the revised inference trajectories after reflection is significantly improved compared to the original inference trajectories, the revised inference trajectories are retained; otherwise, they are filtered out. For the remaining revised inference trajectories, if the reward score of the revised inference trajectories is less than 0.5, the revised inference trajectories are filtered out. For the remaining revised inference trajectories, evidence anchoring filtering is performed, i.e., the current policy model π in the inference trajectory is required to be included. θi The set of predicates used for actions is included in the set of predicates that appear in the observation history returned by the interactive tool, in order to prevent the current policy model π from being used. θi The predicates in the reference query statement are directly reused, and the corrected predicted answer and its inference trajectory after the above filtering are retained as the predicted answer and inference trajectory. The multi-stage filtering in the above embodiments aims to retain the corrected inference trajectory for the question that has a low degree of bias, shows improvement after correction, and meets the requirements. Therefore, it is understandable that, due to different requirements for the strictness of the dataset, in other embodiments, only one or more of the above-mentioned multiple filtering methods may be used, or even more other filtering methods may be used.

[0044] Step S24 involves adding the predicted answers with a deviation less than a set threshold, along with their corresponding inference trajectories and questions, as new samples to the current training dataset. These predicted answers and their corresponding inference trajectories with a deviation less than the set threshold include both the portions of the predicted answers and inference trajectories generated in S21 with a deviation less than the threshold, and the portions of the corrected predicted answers and inference trajectories from subsequent steps with a deviation less than the threshold. This process introduces ideal samples into the training dataset in each iteration, enabling the training of a more performant policy model in subsequent steps.

[0045] Step S25, adjust the current policy model π based on the updated current training dataset. θi The new policy model π is obtained by performing incremental fine-tuning as described in S1 above. θi+1 The new strategy model π θi+1 As the current strategy model.

[0046] Repeat steps S21-S25 until the preset number of iterations is met. For example, if the number of iterations is k, the current policy model π is obtained. θk .

[0047] In some preferred embodiments, when constructing the training dataset using the teacher model in step S1, not all questions in the mixed question set are selected. Therefore, before each iteration, some new questions that have not yet been selected can be chosen from the mixed question set for the policy model to answer, thereby introducing more diverse samples into the training dataset. In other words, this preferred scheme is equivalent to first dividing the mixed question set into several batches, and then in each iteration, training the current policy model on samples from only one batch, and in the next iteration, training the current policy model on samples from another batch, and so on.

[0048] In step S3, the current policy model obtained from the iterative training step in step S2 is used as the constructed knowledge graph question answering model.

[0049] In the process of building the knowledge graph question answering model described above, a reasoning trajectory correction strategy based on self-reflection was implemented. During the iteration process, the strategy model itself is used to correct the failed samples without the need to introduce external strong teachers. The failed samples are transformed into training data for the weak links of the current model, thereby realizing iterative optimization of the training data.

[0050] In the embodiments provided in this application, some problems exist after step S23, where the deviation between the corrected predicted answer and the correct answer is still greater than the set threshold. These problems are referred to as difficult problems. These problems are typically tail problems within the long-tail problems mentioned earlier. These difficult problems suffer from a lack of sufficient training samples, resulting in a large deviation between the predicted answer and the correct answer.

[0051] To address the issue of significant discrepancies between the predicted and correct answers of current policy models for difficult problems, in another embodiment of this application, the iterative step in step S2, following step S23, may further include the following steps: For predicted answers with a deviation exceeding a set threshold, the predicted answer, its corresponding reasoning trajectory and question, the correct answer, and the corresponding reference query statement are used as input to guide the current policy model in generating an experience entry corresponding to the problem. This experience entry includes at least an error pattern field and an experience field. The error pattern field is a general description of common errors for this type of problem, and the experience field includes the reasoning trajectory for obtaining the correct answer using the reference query statement for the problem, as well as the reason for each step in the reasoning trajectory. Furthermore, the generated experience entry and its corresponding question are added to an experience base.

[0052] For example, for a problem where the F1 score is less than a set threshold ρ (e.g., ρ=0.5) and the reward function of both the inference trajectory and the corrected inference trajectory does not reach 1.0, the problem, the incorrect inference trajectory, the predicted answer, the standard answer, and the reference query are used as input to guide the strategy model to output experience items m in a preset structured format (e.g., JSON format). i An experience entry includes one or more fields. An experience entry includes (1) an error pattern field, which includes a general description of common errors in this type of problem; and (2) an experience field, which includes a generalized error pattern and correct strategic thinking derived from a reference query statement, which can be transferred to other problems for reasoning guidance. In some embodiments, experience entry m i It may also include other fields describing the problem. The experience entries m generated by the strategy model i and its question Q i Constructing an experience base Q i It corresponds to experience item m i The above embodiments define the difficult problems as follows: In the above embodiments, problems where the reward function is less than a set threshold ρ (e.g., ρ=0.5) and the reward functions of both the inference trajectory and the corrected inference trajectory do not reach 1.0 are considered difficult problems that require the generation of experience entries, and experience entries are generated to establish an experience base. However, it is understood that in other embodiments, alternatively, problems of different ranges can be selected according to other different conditions and considered as difficult problems.

[0053] In embodiments that incorporate an experience base, when the policy model processes a question to be answered, it can calculate the similarity between the question and each difficult question in the experience base, and select one or more experience entries from the difficult questions that are closest to the question to inject into the policy model as prior knowledge. In some embodiments, the cosine similarity between the new question and each difficult question in the experience base can be calculated using a vector embedding model ϕ (e.g., all-MiniLM-L6-v2).

[0054] (2)

[0055] Wherein, ϕ(Q) i ) and ϕ(Q new These are the questions to be answered, Q and Q respectively. new With difficult question Q i The semantic vectors are used. Experience entries from one or more (e.g., three) of the questions, ranked from highest to lowest similarity, are selected and injected into the policy model prompt as prior knowledge. Preferably, a similarity threshold (e.g., cosine similarity greater than 0.6) can be set to filter the experience entries to be injected, avoiding the injection of irrelevant experience entries.

[0056] In the embodiment that introduces an experience base, in step S21, for each question in the current training dataset, the similarity between the question and each question in the experience base is first calculated. Then, the experience entries corresponding to the questions whose similarity exceeds the set similarity threshold are selected, and the selected experience entries are provided to the current policy model to guide the current policy model to generate predicted answers for each question in the current training dataset and obtain the reasoning trajectory of the predicted answers for the question in the knowledge graph environment.

[0057] In this embodiment, through the iterative training step S2, not only are the training dataset and policy model continuously updated, but the experience base is also continuously updated. The final experience base can also be used in the process of knowledge question answering using the final knowledge graph question answering model output in step S3. For example, when the knowledge graph question answering model generates an answer for an input question, it calculates the similarity between the input question and each question in the experience base, selects the experience entries corresponding to questions with similarity exceeding a set similarity threshold, and provides the selected experience entries to the knowledge graph question answering model to guide the model in generating a predicted answer for the input question in the knowledge graph environment and obtaining the reasoning trajectory of the predicted answer for the question.

[0058] In some preferred embodiments, the iterative step S2 further includes the following steps after the above steps: for difficult problems whose predicted answer deviation is greater than a set threshold and for which corresponding experience entries have already been generated, select the experience entries corresponding to the difficult problem; provide the selected experience entries to the current policy model to guide the current policy model to regenerate multiple predicted answers for the corresponding problem and obtain multiple inference trajectories for the predicted answers of multiple problems, and retain the inference trajectories corresponding to the predicted answers whose deviation is less than the set threshold. In an embodiment that calculates cosine similarity using a vector embedding model, for problems whose predicted answer deviation is greater than a set threshold and for which corresponding experience entries have already been generated, calculate the cosine similarity between the problem and each problem in the experience base, select the experience entries corresponding to problems whose cosine similarity exceeds the set similarity threshold of 0.6; provide the selected experience entries to the current policy model to guide the current policy model to regenerate multiple inference trajectories using the three interactive tools provided above, and retain the inference trajectories of the predicted answers that completely hit the correct answer.

[0059] In this process, when the current strategy model regenerates inference trajectories for these questions under the guidance of the experience base, the bias of the predicted answers generated in this step is usually less than the set threshold, or even completely hits the correct answer, because the experience base contains experience entries that can be referenced for these questions. Therefore, the predicted answers with smaller biases and their corresponding inference trajectories for these difficult questions that lack samples and are prone to repeated errors are also added to the current data training set in step S4 to provide ideal samples for correct answers to difficult questions.

[0060] By employing the methods described above, difficult problems are constructed into an experience base. When encountering similar problems, the strategy model, in conjunction with the experience base, can refer to the experience of historical difficult problems to compensate for the model's inherent capabilities without increasing model parameters. This experience guides the model to address the inconsistency between training and inference encountered in these problems. Furthermore, the correct answers to difficult problems guided by the experience base and their inference trajectories can be added to the training dataset to achieve better reflective effects.

[0061] This application also provides a knowledge graph question answering method, which uses the knowledge graph question answering model constructed by the method described above to generate answers to input questions.

[0062] This application also provides a computer program product, which includes computer instructions that, when executed by a processor, implement the knowledge graph question answering model construction method and / or knowledge graph question answering method described above.

[0063] This application also provides a system for constructing a knowledge graph, including a memory and a processor; the memory stores a computer program, and the processor executes the computer program to implement the steps of the knowledge graph question-answering model construction method described above.

[0064] This application revolves around an iterative closed loop of "exploration-reflection-experience consolidation-strategy update," categorizing inference trajectories with excessively large biases generated by the model into two types: those that can be reflected upon and those that cannot. The former improves the model's capabilities from the training side by transforming reflection into training data, while the latter compensates for the model's capabilities from the inference side by transforming experience consolidation into external memory. These two paths form a natural division of labor in terms of sample difficulty, jointly covering performance bottlenecks at different difficulty levels. This allows open-source, small-parameter models to continuously improve their multi-turn interactive planning capabilities through correct and incorrect inference trajectories during their own exploration, without relying on strong teacher supervision and process annotation. Experimental results on three widely used knowledge graph question-answering datasets—WebQSP, CWQ, and GrailQA—show that the proposed method achieves performance comparable to or even better than prompting methods based on closed-source large models like GPT-4 under low-resource conditions, validating the effectiveness of this method in complex multi-step interactive knowledge graph question-answering tasks.

[0065] References to "some embodiments," "one embodiment," or "embodiment," etc., in this specification refer to a particular feature, structure, or property described in connection with an embodiment that is included in at least one embodiment. Therefore, the appearance of the phrases "in various embodiments," "in some embodiments," "in one embodiment," or "in an embodiment," etc., throughout this specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or property can be combined in any suitable manner in one or more embodiments. Therefore, a particular feature, structure, or property shown or described in connection with one embodiment can be combined, in whole or in part, with features, structures, or properties of one or more other embodiments without limitation, provided that such combination is not illogical or inoperable.

[0066] The various embodiments in this specification focus on the differences from other embodiments or implementation methods. Similar or identical parts between the embodiments of this application can be referred to mutually. The implementation principles and technical effects of the inventive concept can be mutually referenced, and will not be repeated here. Where there is no conflict, the various embodiments or implementation methods in this application can be combined with each other.

[0067] The embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used in the embodiments of this application is chosen to best explain the principles, practical applications, or technological improvements in the market, or to enable other those skilled in the art to understand the embodiments disclosed in this application.

Claims

1. A method for constructing a knowledge graph question-answering model, comprising the following steps: S1: Construct an initial training dataset using the teacher model, and train the policy model to be trained on the basis of the initial training dataset to obtain the initial policy model. Each sample in the training dataset includes at least a question, the correct answer to the question, and the reasoning trajectory for obtaining the predicted answer to the question. S2: Using the initial policy model as the current policy model and the initial training dataset as the current training dataset, perform the following iterative training steps: S21 uses the current policy model to generate predicted answers for each question in the current training dataset and obtains the reasoning trajectory for the predicted answers to that question in the knowledge graph environment; S22 assesses the degree of deviation between the predicted answer and the correct answer for each question; S23 For predicted answers with a deviation greater than a set threshold, the corresponding reasoning trajectory and question, along with the deviation level, are used as inputs to guide the current strategy model to generate a corrected predicted answer and its corresponding reasoning trajectory, and to evaluate the deviation level between the corrected predicted answer and the correct answer. S24 adds the predicted answers with a deviation less than a set threshold, along with their corresponding reasoning trajectories and questions, as new samples to the current training dataset. S25 performs incremental fine-tuning of the current policy model based on the updated current training dataset; Repeat steps S21-S25 until the preset number of iterations is met; S3: At least the current strategy model should be used as the knowledge graph question answering model to be constructed.

2. The method according to claim 1, wherein, Step S1, which involves constructing the initial training dataset using the teacher model, further includes: Construct a mixed question set consisting of multiple questions, their correct answers, and reference query statements; Provide interactive tools for interacting with knowledge graphs; Using a teacher model in a knowledge graph environment, the interactive tool generates a predicted answer and up to three inference paths for each question in a mixed question set. The inference path must satisfy the following conditions: its corresponding predicted answer is the correct answer and the predicted answer appears in the observation history of the inference path. For each question, the question, the correct answer to the question, the reasoning trajectory for obtaining the predicted answer to the question, and the reference query statement are added as samples to the training dataset; And the training of the policy model to be trained based on the initial training dataset in step S1 includes: Supervised fine-tuning of the policy model to be trained based on the initial training dataset.

3. The method of claim 1, wherein assessing the degree of deviation between the predicted answer to each question and the correct answer to the question comprises: The F1 reward function is used to calculate the reward score for the predicted answer and the correct answer for each question. The higher the reward score, the smaller the deviation.

4. The method according to claim 1, wherein the training dataset further includes a reference query statement for each question; and in step S23, for a predicted answer with a deviation greater than a set threshold, the reference query statement of the predicted answer, together with the inference trajectory and deviation corresponding to the question and the predicted answer, are used as input to guide the current policy model to generate a corrected predicted answer and its corresponding inference trajectory, and the current policy model is restricted from directly using the reference query statement.

5. The method according to claim 1, wherein the training dataset further includes a reference query statement for each question, and the method further includes the following steps after step S23: For predicted answers with a deviation greater than a set threshold, the predicted answer, its corresponding reasoning trajectory and question, the correct answer, and the corresponding reference query statement are used as input to guide the current strategy model to generate experience entries corresponding to the question. The experience entries include at least an error pattern field and an experience field. The error pattern field is a general description of common errors in this type of question. The experience field includes the reasoning trajectory for obtaining the correct answer using the reference query statement of the question and the reason for each step in the reasoning trajectory. Add the generated experience entries and their corresponding questions to the experience library; And in step S3, the current strategy model and experience base are used as the constructed knowledge graph question answering model.

6. The method according to claim 5, further comprising the iterative training step S2: For questions whose predicted answer deviation is greater than a set threshold and for which corresponding experience entries have already been generated, select the experience entry corresponding to that question. The selected experience items are provided to the current strategy model to guide the current strategy model to regenerate multiple inference trajectories for the corresponding problem; Evaluate the degree of deviation between the predicted answer for each reasoning trajectory and the correct answer to the question; The predicted answer with the smallest deviation, along with its corresponding reasoning trajectory and question, is added as a new sample to the current training dataset.

7. The method according to claim 5, wherein step S21 further comprises: For each question in the current training dataset, calculate the similarity with each question in the experience base, and select the experience entries corresponding to questions whose similarity exceeds the set similarity threshold; The selected experience items are provided to the current policy model to guide it in generating predicted answers for each question in the current training dataset and obtaining the reasoning trajectory for the predicted answers to that question in the knowledge graph environment.

8. A knowledge graph question answering method, which uses a knowledge graph question answering model constructed according to any one of claims 1-7 to generate answers to input questions.

9. A computer program product comprising computer instructions that, when executed by a processor, implement the steps of the knowledge graph question-answering model construction method according to any one of claims 1-7.

10. A knowledge graph question-answering model construction system, comprising a memory and a processor; The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the knowledge graph question-answering model construction method according to any one of claims 1-7.

Images