Knowledge graph reasoning method and system combining reinforcement learning and attention mechanism

Abstract

The invention discloses a knowledge graph inference method and system combining reinforcement learning and an attention mechanism, and the method comprises the steps: inputting a knowledge graph into a knowledge graph inference model, building the environment of the knowledge graph, representing an entity in the knowledge graph as a state, taking a relation-entity pair as an action, constructing a Markov decision process, and obtaining the knowledge graph inference model. The agent performs an action in the environment to find a target entity; through mutual cooperation of a path history coding module, a neighbor entity information extraction module, a strategy decision-making module and a reward shaping module, an intelligent agent is guided to perform multi-hop path reasoning on entities and relationships thereof in a knowledge graph; and completing a knowledge graph reasoning task based on a reasoning result of multi-hop path reasoning. The method can effectively improve the accuracy and efficiency of the knowledge graph reasoning task, solves the problems of path selection difficulty, information aggregation insufficiency and reward sparseness in a traditional reasoning method, and has good practicability and popularization value.

Description

Technical Field

[0001] The present invention belongs to the field of knowledge graph technology, and specifically relates to a knowledge graph reasoning method and system that combines reinforcement learning and attention mechanism. Background Art

[0002] As a crucial component of artificial intelligence, knowledge graphs represent entities and their semantic relationships in a structured form, and are widely used in scenarios such as intelligent question-answering, recommendation systems, and semantic search. In natural language question-answering tasks, knowledge graphs support knowledge-based reasoning and answers to natural language questions by providing clear entity relationships and logical paths, effectively improving the accuracy and interpretability of the system.

[0003] In recent years, with the rapid construction and application of large-scale knowledge graphs, the number of nodes, edges, and relationship types has continued to grow, and the complexity of reasoning has increased dramatically. In this context, multi-hop question answering has become a research focus. This task requires the system to reason along multiple relationship paths in the graph, exploring deep connections between entities to answer complex user questions. However, current knowledge graph question answering systems still face many challenges in terms of the rationality of path selection, the accuracy of entity representation, and the efficiency of the reasoning process.

[0004] Traditional methods mostly adopt rule-based path matching or graph-based search strategies, and often rely on manually designed features, rule-based path templates and other means. They not only have poor flexibility and poor transferability, but also easily lead to inaccurate reasoning paths when faced with redundant entities and ambiguous relationships in the graph, ultimately affecting the performance of the question-answering system.

[0005] With the development of deep learning, attention mechanisms and reinforcement learning have demonstrated powerful capabilities in sequence modeling and decision optimization. Previous studies have shown that applying attention mechanisms to feature extraction of neighboring entities can effectively identify entity information that contributes to the current reasoning state; while reinforcement learning frameworks can dynamically optimize paths in complex environments, improving the efficiency and accuracy of overall reasoning strategies. DeepPath is the first model to apply reinforcement learning to entity path search, learning paths between entities in link prediction tasks through supervised learning. MINERVA introduces reinforcement learning to search for answer entities and jointly selects entity-relationship pairs through a single policy network under the supervision of the query relationship at each step. Although the above methods have made some progress in the direction of multi-hop reasoning, when dealing with actual business scenarios with complex relationships and diverse semantics, especially in the medical field where accuracy and practicality are higher, the existing methods still have problems such as limited path selection, large reasoning errors, and excessive entity interference. There is an urgent need to design a more robust and scenario-adaptive question-answering reasoning method. Summary of the Invention

[0006] In response to the above-mentioned deficiencies in the prior art, the knowledge graph reasoning method and system combining reinforcement learning and attention mechanism provided by the present invention solve the problems of insufficient utilization of path information, weak entity representation ability and sparse reward feedback in the existing knowledge graph reasoning methods.

[0007] In order to achieve the above-mentioned purpose of the invention, the technical solution adopted by the present invention is: a knowledge graph reasoning method combining reinforcement learning and attention mechanism, including the following steps: S1. Input the knowledge graph into the knowledge graph reasoning model, which includes a path history encoding module, a neighbor entity information extraction module, a strategy decision module, and a reward shaping module; S2. Build a knowledge graph environment, represent entities in the knowledge graph as states, and use relationship-entity pairs as actions. Construct a Markov decision process, and the agent performs actions in the environment to find the target entity. S3. The path history encoding module encodes the path history sequence currently visited by the agent and generates a path history encoding vector. The neighbor entity information extraction module extracts the neighbor information of the current entity and generates a neighbor entity aggregation vector. The strategy decision module outputs the action selection probability distribution based on the current state representation, the path history encoding vector and the neighbor entity aggregation vector. The reward shaping module calculates the final reward by combining the real-time reward, efficiency reward and terminal reward. The agent selects the action with the highest reward value based on the final reward, guiding the agent to perform multi-hop path reasoning on the entities and their relationships in the knowledge graph. S4. Based on the reasoning results of multi-hop path reasoning, complete the knowledge graph reasoning task.

[0008] Furthermore: In S3, the method for generating the path history encoding vector is specifically as follows: A1. Add position codes to entities and relations at each time step and encode them into real-valued vectors at each time step. A2. Integrate all real-valued vectors at the current time step to form a path sequence, input the path sequence into the multi-layer Transformer encoder, extract the path history representation of the current time step from the hidden state sequence encoded by the multi-layer Transformer encoder, and use it as the path history encoding vector.

[0009] Further: In A1, the current time step t A real-valued vector The specific expression is: Where, For the current entity, is the relationship of the current entity, Encode for position; In A2, extract the time step t Path history representation The specific expression is: Where, is the encoded hidden state sequence.

[0010] Furthermore, in S3, the method for generating the neighbor entity aggregation vector is as follows: B1. Extract all neighbor entities directly connected to the current entity and their associated relationships from the knowledge graph to form a set; B2. Project the current entity and path history encoding vector into the embedding space of neighboring entities through learnable parameters, calculate their semantic relevance, and obtain the similarity score of neighboring entities; B3. Use the Softmax function to normalize the similarity score to generate the weight coefficient of the neighbor entity. Calculate the local feature vector based on the weight coefficient and the neighbor entity, and use it as the neighbor entity aggregation vector.

[0011] Further: In B1, the collection The specific expression is: Where, For the i Neighbor entities, is the relationship between neighbor entities, For knowledge graph; In B2, i Similarity scores of neighbor entities The specific expression is: Where, is a matrix, T is the transpose symbol; B3, no. i The expression of the weight coefficient of the neighbor entity is specifically: Where, For the k Similarity scores of neighbor entities; Calculate local eigenvectors The specific expression is: .

[0012] Further: In S3, the output action selection probability distribution The specific expression is: Where, is a set of actions, For action, For status, is the activation function, To query the global information of triples, and is the bias term, and is a learnable parameter, is the Softmax operator.

[0013] Further: In S3, the reward shaping module calculates the final reward The specific expression is: Where, It is the terminal reward, which is used to give the overall result reward when the reasoning task is completed, to ensure that the intelligent body can optimize the reasoning end. is the reward after weighted fusion, and its specific expression is: Where, is an efficiency reward, which is used to encourage the agent to choose a fast and low-complexity reasoning path. It is a real-time reward used to feedback the effect of the current action. is a tunable hyperparameter.

[0014] Further: Efficiency Rewards The specific expression is: Where, For the path, For path The total mass, is the maximum path length limit, For path length, To obtain the maximum value; Where, For the The confidence of the entity or relation, Is the path The total number of entities or relations in Where, is the embedding representation of the current query, For the l entity, is the Euclidean norm of the vector; Real-time rewards The specific expression is: Where, Embed for current issue Relation embeddings corresponding to the actions taken The similarity between is the trade-off factor, In state Unreasoned information evaluation under In state Unreasoned information evaluation under; Where, is the cosine similarity formula; Where, Embedded as the initial question, is the dot product; Terminal Rewards The specific expression is: Where, The correct answer entity.

[0015] A knowledge graph reasoning system that combines reinforcement learning and attention mechanism. The system includes: The path history encoding module uses the Transformer network to encode the path history sequence currently visited by the agent and generate a path history encoding vector; The neighbor entity information extraction module uses the attention mechanism to extract neighbor entity features, models the semantic relevance of the relationship-entity pairs in the current entity neighbor set, calculates the attention weights, and generates a neighbor entity aggregation vector. The policy decision module builds a policy network in the form of a feedforward neural network, outputs the action selection probability distribution based on the current state representation, the path history encoding vector, and the neighbor entity aggregation vector, and guides the intelligent agent to explore the optimal path in the knowledge graph; The reward shaping module combines real-time rewards, efficiency rewards, and terminal rewards to calculate the final reward, guiding the agent to make better corresponding actions.

[0016] The beneficial effects of the present invention are: (1) This invention provides a knowledge graph reasoning method and system that combines reinforcement learning with an attention mechanism. The knowledge graph is modeled as a Markov decision process (MDP). Through the cooperation of a path history encoding module, a neighbor entity information extraction module, a policy decision module, and a reward shaping module, an intelligent agent is guided to perform multi-hop path reasoning. The path selection difficulty problem is alleviated by the policy optimization mechanism in the reinforcement learning framework, enabling the intelligent agent to effectively explore high-quality paths in large-scale graphs. The problem of insufficient information aggregation is solved by introducing an attention mechanism to dynamically aggregate semantically related neighbor information, thereby improving the expressive power of state representation. The problem of sparse rewards is solved by designing a multi-dimensional reward function (including real-time rewards, efficiency rewards, and terminal rewards) to shape rewards and enhance the guidance of the intermediate reasoning stage.

[0017] (2) The present invention designs a path history encoding module. Compared with traditional recurrent neural networks (such as LSTM), Transformer has stronger long-term dependency modeling capabilities, enabling the model to fully utilize the semantic information in the historical path during the reasoning process.

[0018] (3) The present invention designs a neighbor entity information extraction module, adopts a neighbor entity feature extraction method based on the attention mechanism, models the semantic relevance of the relationship-entity pairs in the current entity neighbor set, and calculates the attention weight to achieve weighted aggregation of neighbor information, thereby enhancing the representation ability of the current state.

[0019] (4) The present invention designs a strategy decision module that comprehensively considers the current state representation, path history encoding vector and neighbor entity aggregation vector, scores all optional actions, outputs the action selection probability distribution, and guides the intelligent agent to explore the optimal path in the knowledge graph.

[0020] (5) This paper designs a reward shaping module to overcome the sparse reward problem in reinforcement learning. It introduces a multi-level final reward function consisting of real-time rewards, efficiency rewards, and terminal rewards. During the inference process, rewards are not only given based on the final target matching result, but also adjusted for intermediate steps based on factors such as the number of path hops and entity semantic consistency to improve training efficiency and model stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of the knowledge graph reasoning method that combines reinforcement learning and attention mechanism in the present invention.

[0022] Figure 2 Schematic diagram of the reinforcement learning framework of the present invention.

[0023] Figure 3 Schematic diagram of the strategy decision module of the present invention.

[0024] Figure 4 Schematic diagram of the knowledge graph reasoning system that combines reinforcement learning and attention mechanism of the present invention. DETAILED DESCRIPTION

[0025] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0026] like Figures 1 and 2 As shown, in one embodiment of the present invention, a knowledge graph reasoning method combining reinforcement learning and attention mechanism includes the following steps: S1. Input the knowledge graph into the knowledge graph reasoning model, which includes a path history encoding module, a neighbor entity information extraction module, a strategy decision module, and a reward shaping module; S2. Build a knowledge graph environment, represent entities in the knowledge graph as states, and use relationship-entity pairs as actions. Construct a Markov decision process, and the agent performs actions in the environment to find the target entity. S3. The path history encoding module encodes the path history sequence currently visited by the agent and generates a path history encoding vector. The neighbor entity information extraction module extracts the neighbor information of the current entity and generates a neighbor entity aggregation vector. The strategy decision module outputs the action selection probability distribution based on the current state representation, the path history encoding vector and the neighbor entity aggregation vector. The reward shaping module calculates the final reward by combining the real-time reward, efficiency reward and terminal reward. The agent selects the action with the highest reward value based on the final reward, guiding the agent to perform multi-hop path reasoning on the entities and their relationships in the knowledge graph. S4. Based on the reasoning results of multi-hop path reasoning, complete the knowledge graph reasoning task.

[0027] Among them, knowledge graph reasoning tasks include knowledge graph question answering and triple completion tasks.

[0028] like Figure 2As shown, in this embodiment, the reasoning process of the model of the present invention is carried out within a reinforcement learning framework. The agent continuously explores and learns through interaction with the environment. Specifically, each time the agent selects an action (such as selecting a relationship or entity), the environment returns a reward signal reflecting the quality of that action. The agent optimizes its reasoning strategy by accumulating rewards, resulting in more accurate and efficient reasoning results. In addition to considering the correctness of reasoning, the reward signal design also incorporates efficiency rewards and terminal rewards to improve the efficiency of the reasoning process and mitigate the problem of reward sparsity.

[0029] In S3, the path history encoding module uses a Transformer network structure to process historical paths. Specifically, given the current agent's historical path sequence, the path history encoding module converts these historical paths into a path history encoding vector with a high-dimensional context representation through the Transformer encoder. The path history encoding vector not only contains information about the path itself, but also captures the long-term dependencies between paths. In this way, the model can better understand the impact of historical paths on current reasoning decisions. The specific method for generating the path history encoding vector is: A1. Add position codes to entities and relations at each time step and encode them into real-valued vectors at each time step. A2. Integrate all real-valued vectors at the current time step to form a path sequence , the path sequence is input into the multi-layer Transformer encoder, and the path history representation of the current time step is extracted from the hidden state sequence encoded by the multi-layer Transformer encoder, which is used as the path history encoding vector.

[0030] In this embodiment, each entity and relation in the knowledge graph is mapped into a continuous vector space through embedding so that the model can perform subsequent neural network processing. Specifically, the dimension of the vector space is set to , then each entity and relationships At time step Each moment is coded as a dimensional real-valued vector, i.e. , R is a matrix.

[0031] In A1, the current time step t A real-valued vector The specific expression is: Where, For the current entity, is the relationship of the current entity, Encode for position; In this embodiment, to more effectively capture the temporal structure of the reasoning path—that is, to record the order in which the model accesses entities and relationships during reasoning—we employ a path history encoding approach and introduce a positional encoding mechanism. Each time step in the path contains both an entity and a relationship, so the information from both must be fused to form a holistic representation of the current step.

[0032] In A2, extract the time step t Path history representation The specific expression is: Where, is the encoded hidden state sequence.

[0033] In S3, the neighbor entity information extraction module is responsible for extracting the neighbor information of the current entity based on the attention mechanism. Given a specific entity, the model extracts neighbor entities related to the entity from the knowledge graph, and performs weighted aggregation based on the semantic similarity of these neighbor entities to generate a neighbor entity aggregation vector. Specifically, the neighbor entity information extraction module uses the self-attention mechanism to calculate the relationship weight between the current entity and its neighbor entities, and aggregates the neighbor entity information with higher weights. In this way, the model can dynamically select the neighbor information that is most relevant to the current entity semantics, providing valuable background information for reasoning decisions. The specific method for generating the neighbor entity aggregation vector is: B1. Extract all neighbor entities directly connected to the current entity and their associated relationships from the knowledge graph to form a set; B2. Use learnable parameters to transform the current entity and path history encoding vector Project it into the embedding space of neighbor entities, calculate its semantic relevance, and obtain the similarity score of neighbor entities; B3. Use the Softmax function to normalize the similarity score to generate the weight coefficient of the neighbor entity. Calculate the local feature vector based on the weight coefficient and the neighbor entity, and use it as the neighbor entity aggregation vector.

[0034] In B1, the collection The specific expression is: Where, For the i Neighbor entities, is the relationship between neighbor entities, For knowledge graph; In B2, i Similarity scores of neighbor entities The specific expression is: Where, is a matrix, T is a transposed symbol; in this embodiment, the present invention aligns the matrix The path history information With neighbor entities Semantic alignment of the two methods can dynamically adapt to different task requirements.

[0035] B3, no. i The expression of the weight coefficient of the neighbor entity is specifically: Where, For the k Similarity scores of neighbor entities; Calculate local eigenvectors The specific expression is: .

[0036] In S3, the strategy decision module is responsible for determining the reasoning action that the agent should take based on the path history encoding vector and the neighbor entity aggregation vector. Figure 3 As shown in Figure 2, the policy decision module contains a feedforward neural network. The specific workflow of the policy decision module is as follows: Path history encoding vector , neighbor entity aggregation vector and global information of query triples , the policy network outputs a probability distribution over the set of all possible actions , output action selection probability distribution The specific expression is: Where, is a set of actions, For action, For status, is the activation function, To query the global information of triples, and is the bias term, and is a learnable parameter, is the Softmax operator.

[0037] According to the calculated probability distribution, the policy network calculates all possible actions. Sampling is performed, and the agent selects the most likely correct action as the next step to perform reasoning and decision-making.

[0038] In S3, the reward shaping module combines real-time rewards, efficiency rewards, and terminal rewards to ensure that the agent receives sufficient feedback signals during the reasoning process to promote rapid convergence. Real-time rewards: Provide feedback on the effectiveness of the current action, such as the accuracy of entity matching. Efficiency rewards: Encourage the agent to choose fast and low-complexity reasoning paths, reducing computing resource consumption. Terminal rewards: Provide rewards for the overall results when the reasoning task is completed, ensuring that the agent optimizes the final reasoning result.

[0039] The reward shaping module calculates the final reward The specific expression is: Where, For terminal rewards, is the reward after weighted fusion; In this embodiment, the final reward Indicates that the agent is in state The final reward feedback at the terminal reward It is used to measure whether the entire reasoning task is successful, such as whether the agent finally reaches the correct target entity or whether it successfully infers the expected path. , the agent completes the correct reasoning task and obtains the full final reward; on the contrary, when , the agent fails to complete the task successfully, then the final reward will be the integrated reward To make decisions, so as to ensure that the agent can obtain continuous and effective feedback during the exploration process and promote reasonable reasoning path search.

[0040] Where, Reward for efficiency, For real-time rewards, is a tunable hyperparameter.

[0041] In this embodiment, the efficiency reward Immediate feedback that measures the agent's decisions in the current state, such as whether it has chosen a more favorable path or reduced ineffective exploration. Characterize the local feedback obtained by the agent at each step of the reasoning process to promote the agent to make reasonable action choices; It is an adjustable hyperparameter that controls the weight of the two reward signals and usually takes a value between [0,1]. When the value is large, the model tends to focus on efficiency rewards and encourages the agent to choose a shorter reasoning path; When the value is smaller, more attention is paid to real-time rewards, which enhances the ability of gradual optimization during the reasoning process.

[0042] Efficiency Rewards The specific expression is: Where, A path, defined as a series of relationships , For path The total mass, is the maximum path length limit, For path length, To obtain the maximum value; Where, For the The confidence of the entity or relation, Is the path The total number of entities or relations in Where, is the embedding representation of the current query, For the l entity, is the Euclidean norm of the vector; Real-time rewards The specific expression is: Where, Embed for current issue Relation embeddings corresponding to the actions taken The similarity between is the trade-off factor, In state Unreasoned information evaluation under In state Unreasoned information evaluation under; Where, is the cosine similarity formula; Where, Embedded as the initial question, is the dot product; Terminal Rewards The specific expression is: Where, The correct answer entity.

[0043] In this embodiment, the principle process of design efficiency reward is as follows: In path reasoning tasks, it's easy to observe that shorter paths often provide more reliable reasoning evidence than longer paths. Shorter relationship chains can reduce the computational effort required for each reasoning step, thereby improving reasoning efficiency. In particular, shorter paths shorten the agent's interaction time with the environment, reducing training and reasoning overhead. To incentivize the model to choose shorter reasoning paths, the efficiency reward is defined as follows: in, Represents a path, defined as a series of relationships The design goal of the efficiency reward is to reward shorter paths, making the reasoning process more efficient while limiting the length of the agent's interaction with the environment. In the subsequent practice, considering the shortcomings of the efficiency reward. In order to avoid the model being overly inclined to choose extremely short paths, a maximum path length limit is introduced here. , the new expression is as follows.

[0044] In this way, the length of the path will have a reasonable upper limit to prevent over-excitation of extremely short paths, which may cause misjudgment. At the same time, we hope that the path is not only short but also has high reasoning reliability. Here, a weighting mechanism is introduced to weight the confidence of each entity and relationship in the path. Specifically, for the path Each entity in and relationships , assigning it a confidence score , this score can be calculated based on the relevance, confidence or other metrics of entities and relationships. The total quality of the path It is given by the weighted sum of the confidences of all entities and relations in the path. The formula is: in, Indicates the The confidence of an entity or relationship, Is the path The total number of entities or relations in the network. After introducing the path quality factor, the efficiency reward can be further adjusted to take path quality into account. A weighted term for path quality is added to the efficiency reward to ensure that the length and quality of the path jointly affect the inference efficiency. The new efficiency reward can be expressed as: The relevant confidence is calculated by the similarity between the representation in the knowledge graph (such as the embedding vector) and the query semantic information. Among them, is the embedding representation of the current query , then the entity Confidence The expression is as above.

[0045] The principle process of designing real-time rewards is as follows: The design goal of real-time rewards is to enhance supervision in the intermediate reasoning stage. It ensures that the agent makes reasonable decisions based on the current state of the reasoning task by evaluating the agent's actions at each stage. The first is to evaluate the relevance of the information taken by the current action to the semantic information being reasoned in the problem. The more relevant, the better, that is, the direction of reasoning must meet the needs of the problem. Secondly, the reduction of non-reasoning information is also important. After the agent takes an action, the reasoning progress should move forward, remove the reasoned part from the problem, and reduce the remaining "non-reasoning information" as much as possible, that is, irrelevant or redundant information. Initial question embedding Contains all the information needed for subsequent reasoning. With each step of reasoning, consistent reasoning information should be removed and the question embedding should be updated. To reflect the new reasoning progress. Regarding the measurement of unreasoned information, the agent needs to evaluate the remaining "unreasoned information" in the current state at each step of reasoning. This amount of information is obtained by comparing the current problem embedding and the initial question embedding The relevant formula is as follows: in, It means in the state The unreasoned information evaluation under Embedded with the initial question similarity. yes and The dot product between them represents the similarity between them. The lower the similarity, the more unreasoned information there is, which means that there is more unprocessed information left in the current reasoning task. Corresponding to the exponential function, it is used to ensure that the similarity measure can reflect the amount of unreasoned information after normalization. The value indicates that the similarity in the current state is low and there is more unreasoned information in the problem. This design considers two parts: the fitness of the current action and the reduction of unreasoned information. The formula is as follows: in, Indicates the current question embedding Relation embeddings corresponding to the actions taken The similarity between them. It measures the adaptability of the current action to the problem and reflects whether the selected relationship matches the semantic information of the current problem. The expression is as follows: in, is the cosine similarity between question embedding and relation embedding, Ensure that the similarity will not be negative to avoid unreasonable calculation results. Indicates the difference between the unreasoned information of the current jump and the unreasoned information of the previous jump. If the reduction of unreasoned information after the current jump is large, it means that the current action is effective and helps to advance the reasoning process. It indicates that the reasoning effect is significant and irrelevant information in the question is greatly reduced. Is a trade-off factor that controls the balance between the two parts of the reward. , you can decide whether to focus more on the fitness of the current action or on the reduction of unreasoned information.

[0046] The principle process of designing terminal rewards is as follows: The terminal reward is a reward mechanism based on the final goal, used to evaluate whether the agent has achieved the predetermined goal. The role of the terminal reward in the task is to map the final result of the task into a numerical reward, allowing the agent to identify which behaviors lead to success during the learning process and encourage the agent to develop towards these behaviors. The relevant formula is as follows: in, During the inference process, the current state Entities inferred from . is the correct answer entity, such as the target entity.

[0047] like Figure 4 As shown in the figure, the knowledge graph reasoning system that combines reinforcement learning and attention mechanism includes: The path history encoding module uses the Transformer network to encode the path history sequence currently visited by the agent and generate a path history encoding vector; The neighbor entity information extraction module uses the attention mechanism to extract neighbor entity features, models the semantic relevance of the relationship-entity pairs in the current entity neighbor set, calculates the attention weights, and generates a neighbor entity aggregation vector. The policy decision module builds a policy network in the form of a feedforward neural network, outputs the action selection probability distribution based on the current state representation, the path history encoding vector, and the neighbor entity aggregation vector, and guides the intelligent agent to explore the optimal path in the knowledge graph; The reward shaping module combines real-time rewards, efficiency rewards, and terminal rewards to calculate the final reward, guiding the agent to make better corresponding actions.

[0048] In this embodiment, the present invention utilizes the collaborative work of the aforementioned modules to effectively integrate path history and graph structure information during reasoning, resulting in strong contextual modeling and decision-making capabilities. For example, in a medical question-answering system, when faced with ambiguous user questions such as "Why do I feel anxious when I sleep at night?", this method can automatically infer possible diseases (such as thyroid dysfunction or autonomic nervous system disorders) from the medical knowledge graph through multi-hop reasoning. It then combines symptoms, medications, and treatment recommendations to recommend appropriate preliminary diagnoses and medication recommendations. By incorporating an attention mechanism, this system focuses on important entities semantically relevant to symptoms. Through a reward-shaping mechanism, the model prefers reasoning paths with short, logically clear paths, effectively alleviating the incomplete information coverage and inefficient path exploration inherent in traditional rule-based systems. Experimental results demonstrate that this method significantly improves reasoning accuracy and path rationality across a variety of knowledge graph question-answering and triple completion tasks, demonstrating its broad applicability and application value.

[0049] In the description of the present invention, it should be understood that the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying the relative importance or the number of technical features implicitly specified. Therefore, the features defined by "first", "second", and "third" may explicitly or implicitly include one or more of such features.

Claims

1. A knowledge graph reasoning method combining reinforcement learning and attention mechanism, characterized by: The following steps are involved: S1. Input the knowledge graph into the knowledge graph reasoning model, which includes a path history encoding module, a neighbor entity information extraction module, a strategy decision module, and a reward shaping module; S2. Build a knowledge graph environment, represent entities in the knowledge graph as states, and use relationship-entity pairs as actions. Construct a Markov decision process, and the agent performs actions in the environment to find the target entity. S3. The path history encoding module encodes the path history sequence currently visited by the agent and generates a path history encoding vector. The neighbor entity information extraction module extracts the neighbor information of the current entity and generates a neighbor entity aggregation vector. The strategy decision module outputs the action selection probability distribution based on the current state representation, the path history encoding vector and the neighbor entity aggregation vector. The reward shaping module calculates the final reward by combining the real-time reward, efficiency reward and terminal reward. The agent selects the action with the highest reward value based on the final reward, guiding the agent to perform multi-hop path reasoning on the entities and their relationships in the knowledge graph. S4. Based on the reasoning results of multi-hop path reasoning, complete the knowledge graph reasoning task.

2. The knowledge graph reasoning method combining reinforcement learning and attention mechanism according to claim 1 is characterized in that: In S3, the method for generating the path history encoding vector is as follows: A1. Add position codes to entities and relations at each time step and encode them into real-valued vectors at each time step. A2. Integrate all real-valued vectors at the current time step to form a path sequence, input the path sequence into the multi-layer Transformer encoder, extract the path history representation of the current time step from the hidden state sequence encoded by the multi-layer Transformer encoder, and use it as the path history encoding vector.

3. The knowledge graph reasoning method combining reinforcement learning and attention mechanism according to claim 2 is characterized in that: In A1, the current time step t A real-valued vector The specific expression is: Where, For the current entity, is the relationship of the current entity, Encode for position; In A2, extract the time step t Path history representation The specific expression is: Where, is the encoded hidden state sequence.

4. The knowledge graph reasoning method combining reinforcement learning and attention mechanism according to claim 3 is characterized in that: In S3, the method for generating the neighbor entity aggregation vector is as follows: B1. Extract all neighbor entities directly connected to the current entity and their associated relationships from the knowledge graph to form a set; B2. Project the current entity and path history encoding vector into the embedding space of neighboring entities through learnable parameters, calculate their semantic relevance, and obtain the similarity score of neighboring entities; B3. Use the Softmax function to normalize the similarity score to generate the weight coefficient of the neighbor entity. Calculate the local feature vector based on the weight coefficient and the neighbor entity, and use it as the neighbor entity aggregation vector.

5. The knowledge graph reasoning method combining reinforcement learning and attention mechanism according to claim 4 is characterized in that: In B1, the collection The specific expression is: Where, For the i Neighbor entities, is the relationship between neighbor entities, For knowledge graph; In B2, i Similarity scores of neighbor entities The specific expression is: Where, is a matrix, T is the transpose symbol; B3, no. i The expression of the weight coefficient of the neighbor entity is specifically: Where, For the k Similarity scores of neighbor entities; Calculate local eigenvectors The specific expression is: 。 6. The knowledge graph reasoning method combining reinforcement learning and attention mechanism according to claim 5 is characterized in that: In S3, the output action selection probability distribution The specific expression is: Where, is a set of actions, For action, For status, is the activation function, To query the global information of triples, and is the bias term, and is the learnable parameter matrix, is the Softmax operator.

7. The knowledge graph reasoning method combining reinforcement learning and attention mechanism according to claim 6 is characterized in that: In S3, the reward shaping module calculates the final reward The specific expression is: Where, It is the terminal reward, which is used to give the overall result reward when the reasoning task is completed, to ensure that the intelligent body can optimize the reasoning end. is the reward after weighted fusion, and its specific expression is: Where, is an efficiency reward, which is used to encourage the agent to choose a fast and low-complexity reasoning path. It is a real-time reward used to feedback the effect of the current action. is a tunable hyperparameter.

8. The knowledge graph reasoning method combining reinforcement learning and attention mechanism according to claim 7 is characterized in that: Efficiency Rewards The specific expression is: Where, For the path, For path The total mass, is the maximum path length limit, For path length, To obtain the maximum value; Where, For the The confidence of the entity or relation, Is the path The total number of entities or relations in Where, is the embedding representation of the current query, For the l entity, is the Euclidean norm of the vector; Real-time rewards The specific expression is: Where, Embed for current issue Relation embeddings corresponding to the actions taken The similarity between is the trade-off factor, In state Unreasoned information evaluation under In state Unreasoned information evaluation under; Where, is the cosine similarity formula; Where, Embedded as the initial question, is the dot product; Terminal Rewards The specific expression is: Where, The correct answer entity.

9. A knowledge graph reasoning system combining reinforcement learning and attention mechanism, applied to the knowledge graph reasoning method combining reinforcement learning and attention mechanism as described in any one of claims 1 to 8, characterized in that: The system includes: The path history encoding module uses the Transformer network to encode the path history sequence currently visited by the agent and generate a path history encoding vector; The neighbor entity information extraction module uses the attention mechanism to extract neighbor entity features, models the semantic relevance of the relationship-entity pairs in the current entity neighbor set, calculates the attention weights, and generates a neighbor entity aggregation vector. The policy decision module builds a policy network in the form of a feedforward neural network, outputs the action selection probability distribution based on the current state representation, the path history encoding vector, and the neighbor entity aggregation vector, and guides the intelligent agent to explore the optimal path in the knowledge graph; The reward shaping module combines real-time rewards, efficiency rewards, and terminal rewards to calculate the final reward, guiding the agent to make better corresponding actions.

Images