Knowledge graph multi-hop reasoning method based on transformer deep reinforcement learning

Abstract

The application discloses a knowledge graph multi-hop reasoning method based on a Transformer deep reinforcement learning, and comprises the following steps: S1, a large number of RDF triples are accessed, a relationship is completed through relationship completion, and a knowledge graph with complete relationships is created; S2, representation learning is respectively performed on a topological structure and a triple semantic of the knowledge graph, representation tensors of the topological structure and the triple semantic are comprehensively combined in a linear combination manner, and a representation layer of the knowledge graph is mapped to a unified knowledge space; and S3, a GPT-2 model based on a Transformer decoding layer stack is used to model a reasoning layer of the knowledge graph, and a bundle search is used to generate a next time step action A T+1 The application avoids defects such as uncontrollable design of behavior strategy constraints and distribution of confidence by using artificial experience in traditional reinforcement learning.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of knowledge graph multi-hop reasoning, and particularly relates to a knowledge graph multi-hop reasoning method based on Transformer deep reinforcement learning. BACKGROUND

[0002] A knowledge graph describes concepts, entities and their relationships in the objective world in a structured form, and provides the ability to better organize, manage and understand the massive information on the Internet. The sparse knowledge density limits the function of the knowledge graph. Multi-hop reasoning, as an important means of knowledge completion, accepts entity and relationship query conditions, calculates the target entity that meets the relationship mapping of the starting entity, and constructs appropriate relationships between multiple entities that are originally unrelated. The dense relationships connected thereby can provide more rich knowledge information and improve the generalization ability of the downstream task model of the knowledge graph.

[0003] Multi-step reasoning is a sequential decision problem, and therefore reinforcement learning can be used to learn the process. The agent of policy-based reinforcement learning learns the reasoning path through interaction between the knowledge graph environment. DeepPath is the first multi-step reasoning model based on reinforcement learning, but it can only infer the intermediate relationship chain given two entities. MINERVA improves this and introduces LSTM into the policy function to encode the sequence history information. M-Walk introduces Monte Carlo tree search into the reinforcement learning model to search for the reasoning path, effectively reducing the path sparsity problem. The history information of the same is encoded by RNN. Multi-Hop points out that the reward function of the reinforcement learning process needs to be defined artificially. To solve this problem, the method proposes a learnable reward function, which can adaptively adjust the exploration of the path. At the same time, it also uses a technology similar to Dropout in policy selection.

[0004] In order to solve the problems of conflict resolution difficulty, poor generalization ability, weak result interpretability and low credibility in knowledge graph multi-hop reasoning, the knowledge graph multi-hop reasoning can be modeled as a partially observable Markov decision process with limited horizon. When solving the approximate optimal policy, a small error in the value function may be amplified constantly when bootstrapping the value function, which may easily lead to overestimation. The updating method based on Bellman's optimal equation propagates relatively slowly and does not work well in the sparse reward scenario. SUMMARY

[0005] In view of the above problems in the prior art, the knowledge graph multi-hop reasoning method based on Transformer deep reinforcement learning provided by the application solves the problems of poor generalization ability, uninterpretability and difficulty in application to large-scale knowledge graphs in the prior art knowledge graph multi-hop reasoning method.

[0006] In order to achieve the above-mentioned purposes, the technical scheme adopted by the present application is as follows: a knowledge graph multi-hop reasoning method based on a Transformer deep reinforcement learning, comprising the following steps:

[0007] S1, access a large number of RDF triples, create a complete relationship knowledge graph through relationship completion, reduce the sparse distribution of knowledge graph nodes and edges, and improve the knowledge density of the knowledge graph used for reasoning;

[0008] S2, respectively, the topological structure and the triple semantics of the knowledge graph are learned, and the representation tensors of the topological structure and the triple semantics are combined in a linear combination manner, and the representation layer of the knowledge graph is mapped to a unified knowledge space;

[0009] S3, using a GPT-2 model based on a stack of Transformer decoding layers to model the reasoning layer of the knowledge graph, using the historical trajectory H t =(T0,T1,…,T t ) of the previous time step, and using bundle search to generate the next time step action A T+1 in an autoregressive manner.

[0010] Further, the specific steps for creating a complete relationship knowledge graph in step S1 are as follows:

[0011] S11, for the existing knowledge graph, access the RDF triples to the JanusGraph graph database, which facilitates subsequent graph relationship completion, query and reading operations;

[0012] wherein the existing knowledge graph is modeled as Φ=(E,Γ,Λ), denotes the set of all entities in the knowledge graph, is the set of all relationships in the knowledge graph, is the set of all fact triples, in the fact triple (ε μ ,γ,ε v )∈Λ, ε μ is the head entity of the fact triple, ε v is the tail entity of the fact triple, and γ is the relationship from the head entity ε μ to the tail entity ε v ;

[0013] S12, on the basis of the existing relationships of the knowledge graph, actively complete the missing relationships in the equal mutual relationship, and mirror inverse relationships that are beneficial to perfect the calculation path, such as (ε μ ,γ,ε v )∈Λ, the mirror inverse relationship triple of (ε μ ,γ -1 ,εv )∈Λ,γ -1 This is the mirror image of γ;

[0014] S13. Add a self-loop relation to each vertex, and add a loop relation to the entity of each vertex with itself, i.e. (ε μ ,γ μμ ,ε μ )∈Λ, which helps to increase the search space of the reasoning agent, expand the search path of the reasoning agent, and help the reasoning agent's exploration ability; γ μμ For the self-loop relationship of knowledge graph node μ;

[0015] S14. Write the newly added relations back to the JanusGraph graph database to obtain a complete knowledge graph.

[0016] Further: The specific steps for mapping the representation layer in step S2 are as follows:

[0017] S21. Extract the pure topology without text content and the semantic triples containing only text content from the complete graph network generated by the graph layer, and import all the topology graph network and semantic triples at once.

[0018] S22. Feed the graph network topology into the graph convolutional network (GCN) for training, and learn the graph network nodes ε. i Topological representation and edge γ j Topological representation

[0019] S23. Feed the fact triples of the graph network into the ConvE network for training to learn the graph network node ε. i semantic representation and edge γ j semantic representation

[0020] S24. Linearly connect the topological representation tensor and the semantic representation tensor to obtain node ε. i Knowledge representation and edge γ j Knowledge representation

[0021] Furthermore, the specific steps for modeling the inference layer in step S3 are as follows:

[0022] S31. Model a multi-hop reasoning reinforcement learning environment based on knowledge graphs, including state, action, immediate reward, subsequent cumulative reward and trajectory;

[0023] S32. Use GPT for sequence modeling training and select the trajectory that maximizes the benefit.

[0024] S33, search for hidden actions in the trajectory based on the cluster search, and form a complete action sequence;

[0025] S34, backtrack the trajectory of the action sequence to generate an explainable reasoning path, and present explanation items supporting the conclusion.

[0026] Further, the state in step S31 wherein ε t is the entity node at time t, ε μσ and γ σ are the starting entity node and relationship edge at time t=0, ε νσ is the final answer node, S is the state space, the initial state S0=(ε μσ ,ε μσ ,γ σ ,ε νσ ), and the terminal state S T =(ε νσ ,ε μσ ,γ σ ,ε νσ );

[0027] The action A t is defined as:

[0028] wherein γ t is the relationship edge at time t, and ε is the knowledge graph node;

[0029] The immediate reward R t is R(S t ), which is defined as:

[0030] R(S t ) = I(ε t = ε νσ ) - I(ε t ≠ ε νσ )

[0031] In the above formula, I(·) is a binary reward function;

[0032] The subsequent cumulative reward G t is defined as:

[0033]

[0034] In the above formula, T is the end time step of the agent reasoning, K is the start time step of the agent reasoning, and α is the penalty coefficient of the immediate reward;

[0035] The trajectory is defined as:

[0036]

[0037] In the above formula, τ is a trajectory.

[0038] Further, the sequence modeling training in step S32 is:

[0039] During training, real trajectories are used, the trajectory field is directly and parallelly input, all predicted nodes are output through the autoregressive model and the bundle search, each node is a multi-dimensional vector, the probability distribution is obtained through softmax normalization processing, the cross entropy of the real label corresponding thereto is calculated, and the log likelihood is obtained.

[0040] The target L(τ) of maximizing the log likelihood is:

[0041]

[0042] For the conditional probability P θ (S t |S ＜t ,τ ＜t ), the meaning is the probability of the current state S t occurring under the condition that the state before the current time t and the trajectory τ before the current time are given, and logP θ (S t |S ＜t ,τ ＜t ) is the log conditional probability of the state; for the conditional probability P θ (A t |A ＜t ,S t ,τ ＜t ), the meaning is the probability of the current state A t occurring under the condition that the state at the current time t, the action taken before the time t and the trajectory τ before the current time are given, and logP θ (A t |A ＜t ,S t ,τ ＜t ) is the log conditional probability of the action; for the conditional probability P θ (G t |A t ,S t ,τ ＜t ), the meaning is the probability of the cumulative return G t occurring under the condition that the state at the current time t, the action taken before the time t and the trajectory τ before the current time are given, and logP θ (G t |A t ,S t ,τ ＜tLet P be the log-conditional probability of the cumulative return; and P be the conditional probability of the immediate return. θ (R t |A t ,S t ,τ ＜t The meaning is that, given the state at the current time t, the action taken in that state, and the trajectory τ before the current time, the immediate benefit R occurs. t The probability, logP θ (R t |A t ,S t ,τ ＜t Let P be the log-conditional probability of the payoff, θ be the orbital space parameter, and P be the log-conditional probability of the payoff. θ (S t |S i ,τ ＜t Let P be the conditional probability of the state. θ (A t |A j ,S t ,τ ＜t Let P be the conditional probability of an immediate action. θ (G t |A k ,S t ,τ ＜t Let P be the conditional probability of cumulative reward. θ (R t |A t ,S t ,τ ＜t ) represents the conditional probability of immediate benefit;

[0043] Use log probabilities for all intermediate processes (S) t A t ,R t G t Importance sampling is performed, and the importance sampling ratio is calculated using the following method: given the initial state S t And the set of all possible actions A after time step t t:T-1 The subsequent state-action alternation trajectory A t ,S t+1 A t+1 ,...,S T The probability of this occurring under policy π is:

[0044]

[0045] Where P is the state transition probability function, and A k S represents the action taken by the reasoning entity at time step t. kFor the state of the knowledge graph at time step t, the importance sampling ratio ρ under the target policy π and the action policy b trajectory is defined as:

[0046]

[0047] In the above formula, ρ t:T-1 is the importance sampling ratio of the agent in the time period from time step t to T-1.

[0048] Further, the step S33 is specifically: in the time step t, the conditional probability and The maximum of the first two actions and The bundle search algorithm takes these two actions as the action input of the next time step, assuming The conditional probability of the branch at the next time step tends to be infinitely small, and only As the action input of the next time step, by the time step T, the bundle search will only select the action that makes the conditional probability The maximum, finally forming a complete action sequence

[0049] Further, the step S34 is specifically: the search width B of the bundle search, in the first time step, the first word of the B candidate output sequences with the maximum conditional probability is selected, and in each subsequent time step, based on the output sequence of the last time step, the B candidate output sequences with the maximum conditional probability in all combinations are selected as the candidate output sequences of the time step, and the B candidates are always maintained, and finally the optimal one is selected from the B candidates, and the process of the bundle search can be described as:

[0050]

[0051] In the above formula, α is an adjustable parameter, B α is used to punish long sequences, y t is the candidate at time step t, y B is the candidate at time step B, and d is the end-of-sequence symbol.

[0052] The application has the beneficial effects that the application proposes a TDRL-MHR (Transformer based Deep Reinforcement Learning for Multi-Hop Reasoning) algorithm, which is a framework for transforming a knowledge graph multi-hop reasoning task into a conditional sequence generation task based on a Transformer deep reinforcement learning. A Transformer encoder maps a knowledge graph to a high-dimensional knowledge space, and the state, action and return modeled accordingly are sent to a Transformer decoder GPT (General Presentation Transformer) to predict the next candidate action in an autoregressive manner. A bundle search is used to select the action with the maximum expected return conditional probability. The algorithm avoids the uncontrollable defects such as the use of artificial experience to assign confidence and the design of behavior strategy constraints in traditional reinforcement learning. Experiments show that the Transformer modeling method can simulate a wide range of behavior distributions and improve the generalization and migration ability of multi-hop reasoning on knowledge graphs of different magnitudes. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 is a knowledge graph multi-hop reasoning processing flowchart based on the Transformer deep reinforcement learning of the application;

[0054] Figure 2 is an architecture principle block diagram of Figure 1 ;

[0055] Figure 3 is a flowchart of a complete graph construction module of the application;

[0056] Figure 4 is a flowchart of a complete graph topology semantic comprehensive representation module of the application;

[0057] Figure 5 is a flowchart of a trajectory sequence multi-hop reasoning module based on the application. DETAILED DESCRIPTION

[0058] The specific embodiments of the application are described below to facilitate the understanding of the application by those skilled in the art, but it should be clear that the application is not limited to the scope of the specific embodiments. For those skilled in the art, it is obvious that all kinds of changes within the spirit and scope of the application defined and determined by the appended claims are obvious, and all kinds of applications utilizing the concept of the application are included in the protection.

[0059] As shown in Figure 1 , a knowledge graph multi-hop reasoning method based on Transformer deep reinforcement learning comprises the following steps:

[0060] S1, access a large number of RDF triples, complete the relationship, create a complete knowledge graph, reduce the sparse distribution of knowledge graph nodes and edges, and improve the knowledge density of the knowledge graph used for reasoning;

[0061] S2, respectively, the topological structure and the triple semantic of the knowledge graph are learned, and the representation tensors of the topological structure and the triple semantic are comprehensively combined in a linear combination manner, so that the representation layer of the knowledge graph is mapped to a unified knowledge space;

[0062] S3, a GPT-2 model based on a stack of Transformer decoding layers is used to model the reasoning layer of the knowledge graph, the historical trajectory H t =(T0,T1,...,T t ) of the previous time step is used, and the next time step action A T+1 is generated in an autoregressive manner by using bundle search.

[0063] Figure 2 It is the schematic diagram of the architecture principle of the multi-hop reasoning of the application. With a large number of triples and a query problem as input, and with a problem solution and a reasoning path as output, the application adopts a hierarchical structure, which is divided into a graph layer, a representation layer and a reasoning layer. The graph layer converts a large number of RDF triples into a standardized graph network, which is used as an operation object for subsequent representation and reasoning. The representation layer maps the graph layer to a knowledge space through a topological semantic integrated network. The reasoning layer first generates knowledge space component modeling of each reinforcement learning element according to the definition of standard reinforcement learning, combines the characteristics of the knowledge graph, converts the knowledge space environment into a reinforcement learning environment, and takes the knowledge space state in the reinforcement learning dimension, the search action of the reasoning intelligent agent on the knowledge graph, the reward for the reasoning intelligent agent behavior based on the yield shaping technology and the subsequent cumulative yield of the reinforcement learning elements as input, and sends them into the GPT-2 network architecture. The probability path is calculated, the calculation performance and the path exploration coverage are comprehensively considered, the bundle search is used to derive the motion path with the highest probability of the reasoning intelligent agent, and the motion path is combined into a complete full-process reasoning path through backtracking of each hop until the number of hops within the specified range is exhausted.

[0064] Figure 3 It is the completion and construction process of the graph layer of the application. The knowledge graph modeling is Φ=(E,Γ,Λ), representing the set of all entities in the knowledge graph, is the set of all relationships in the knowledge graph, is the set of all fact triples, in the fact triple (ε μ ,γ,ε v )∈Λ, ε μ is the head entity of the fact triple, ε v is the tail entity of the fact triple, and γ is the relationship from the head entity εμ Mapped to tail entity ε v The relationship. (S11) The content, quality, and storage method of the triples are not within the scope of this invention. In order to obtain better data ACID operation performance, this invention first imports a large number of existing RDF triples into the JanusGraph graph database to facilitate subsequent graph relation completion, query, and reading operations; (S12) Based on the existing relations, this invention actively completes the missing relations in equal and mutually exclusive relations, as well as the mirror inverse relations that are not logically self-consistent but are beneficial to improving the computation path, such as (ε μ ,γ,ε v The mirror inverse relation triple of )∈Λ is (ε μ ,γ -1 ,ε v (S13) This invention adds a self-loop relation to each vertex, adding a loop relation to itself to the entity of each vertex, i.e. (ε) μ ,γ μμ ,ε μ (S14) The newly added relation is written back to the JanusGraph graph database. Using the Graphexp visualization tool, users can easily view the overall content and structural details of the graph, and also view the complete reasoning path after the reasoning is completed.

[0065] Figure 4 This is the learning mapping process of the representation layer in this invention. (S21) This invention extracts the pure topological structure without text content and the semantic triples containing only text content from the complete graph network generated by the graph layer, and imports all the topological graph network and semantic triples at once; (S22) The topological structure of the graph network is fed into the graph convolutional network GCN for training to learn the graph network node ε. i Topological representation and edge γ j Topological representation (S23) Feed the fact triples of the graph network into the ConvE network for training to learn the graph network node ε. i semantic representation and edge γ j semantic representation (S24) Linearly connect the topological representation tensor and the semantic representation tensor to obtain node ε. i Knowledge representation and edge γ j Knowledge representation

[0066] Figure 5is the modeling reasoning process of the reasoning layer of the present application. The reasoning agent is the subject of learning and implementing decisions, and the knowledge graph is the environment of the activity of the reasoning agent. The reasoning agent decides the behavior path of the next moment according to the state and the benefit of the knowledge graph environment, and presents a new knowledge graph subgraph state to the reasoning agent. The cumulative benefit generated by the knowledge graph environment is the goal that needs to be maximized in the continuous path selection exploration of the reasoning agent.

[0067] 1) State

[0068] The state space S contains all the effective combinations of the knowledge graph node information, and the state of the reasoning agent not only contains the entity node ε t at the moment t, but also includes the starting entity node ε μσ at the moment t = 0, the relationship edge γ σ , and the final answer node ε νσ . Therefore, the state at time step t can be defined as:

[0069]

[0070] The initial state S0 = (ε μσ , ε μσ , γ σ , ε νσ ), and the terminal state S T = (ε νσ , ε μσ , γ σ , ε νσ ). The state update function is the core part of solving the observability problem of the solution department. Before seeing the state, no action can be taken or any prediction can be made.

[0071] 2) Action

[0072] The action space A is the combination of the effective edge information of the knowledge graph environment of the reasoning agent in the state . The action A t under the state S t is defined as:

[0073]

[0074] That is, the next step entity node selected at the moment t is not in the list of historical nodes traversed. The reasoning agent starts from the starting node ε μσ of the query condition σ, predicts the most likely path through the strategy network, and continuously explores until the target answer entity node ε νσ is reached.

[0075] 3) Timely reward

[0076] In the traditional knowledge graph reinforcement learning path reasoning, a binary reward function is generally used:

[0077] R(S t )=Ι(ε t =ε νσ )

[0078] That is, only when the current entity node is the answer node, the reward +1 can be obtained, otherwise the reward obtained at other time steps is 0. After the reasoning agent continuously executes the strategy and strategy evaluation, only a little feedback can be obtained, even if the reinforcement learning action strategy based on sequence generation, the reward signal can be long-term propagated and the decay is small, but the delay and sparsity of the reward signal still cause the reinforcement learning to slow down and even not to converge, and the effective reasoning strategy cannot be learned. By converting the "maximum reward" idea to the "minimum loss" idea, the timely reward function can be set as:

[0079] R(S t )=Ι(ε t =ε νσ )-Ι(ε t ≠ε νσ )

[0080] 4) Subsequent cumulative reward

[0081] In order to enable the Transformer to learn meaningful patterns, and should be able to conditionally generate actions at test time. It is very important to establish a reward model, and TDRL-MHR hopes that the reasoning agent can generate actions according to the expected return in the future instead of the past reward, that is, maximize the probability expectation value of the cumulative sum of the scalar signal (reward) received by the reasoning agent. The reasoning agent tries to select an action so that the total sum of the rewards it receives in the future weighted by the discount factor is maximized. The subsequent cumulative expected return is defined as:

[0082]

[0083] 5) Trajectory

[0084] The core of the algorithm shown in the application is to send the trajectory data as a structured sequence into the Transformer modeling, and to jointly model the state and action to generate the tendency of action allocation. The trajectory τ is composed of state, action, reward and subsequent cumulative return:

[0085]

[0086] Benefiting from the scalability of the results of large-scale unsupervised learning, enough elements can be accommodated in the trajectory tensor.

[0087] (S32) TDRL-MHR uses GPT for sequence modeling training

[0088] The attention in GPT is causal, i.e. the current field will only do attention calculation with its previous field. When training, the real trajectory is used, the trajectory field can be directly and parallelly input, and all the predicted nodes are output through the autoregressive model and beam search, each node is a multi-dimensional vector, and the probability distribution is obtained through softmax normalization processing, and then the cross entropy of the real label corresponding to it is calculated to obtain the log likelihood. θ is the parameter of the trajectory space, and the goal of maximizing the log likelihood in model training is:

[0089]

[0090] The objective function is the log likelihood of the predicted state, action, return and reward. For the conditional probability P θ (S t |S ＜t ,τ ＜t ), the meaning is the probability of the current state S t occurring under the condition of the state before the current time t and the trajectory τ before the current time, and log P θ (S t |S ＜t ,τ ＜t ) is the log conditional probability of the state; for the conditional probability P θ (A t |A ＜t ,S t ,τ ＜t ), the meaning is the probability of the current state A t occurring under the condition of the state at the current time t, the action taken before time t under the state, and the trajectory τ before the current time, and log P θ (A t |A ＜t ,S t ,τ ＜t ) is the log conditional probability of the action; for the conditional probability P θ (G t |A t ,S t ,τ ＜t ), the meaning is the probability of the cumulative return G t occurring under the condition of the state at the current time t, the action taken before time t under the state, and the trajectory τ before the current time, and log P θ (G t |A t ,S t ,τ＜t ) is the log conditional probability of the cumulative return; P θ (R t | A t , S t , τ ＜t ) is the probability of the immediate reward R t given the state at the current time t, the action taken in that state, and the trajectory τ before the current time, log P θ (R t | A t , S t , τ ＜t ) is the log conditional probability of the reward.

[0091] TDRL-MHR uses Monte Carlo methods with a large number of random components to produce value estimates, because Monte Carlo algorithms do not require prior knowledge of the environment dynamics when sampling state, action, and reward sequences from the environment, nor do they require generating probability distributions for all possible transitions as in dynamic programming, nor do they require updating current estimates with estimates of successor states, i.e., they do not require bootstrapping. The TDRL-MHR algorithm uses a trajectory autoregressive network and a beam search strategy to maximize the reward. Compared to traditional Monte Carlo algorithms that solve reinforcement learning problems by averaging the returns of samples, Monte Carlo value estimates can lead to reduced sample complexity and convergence of the inferred trajectory to suboptimal behavior. By using predicted reward signals instead of field-predicted log probabilities, this problem is circumvented. Compared to greedy search, the beam search method optimizes the probability of sequences under data distribution, but there is still a risk of short-sighted behavior when calculating the reward maximization. To solve this problem, the TDRL-MHR algorithm adds the subsequent cumulative return G t to each training trajectory update and increases the penalty term coefficient.

[0092] All learning control methods face a dilemma: the actions they want to learn can make the subsequent agent behavior optimal, but in order to search for all actions, they need to take non-optimal actions. The compromise method is not to learn the action value of the optimal target policy, but to learn an action policy action value that is close to the optimal and still allows exploration. Using Monte Carlo value estimates instead of relying on Bellman optimal updates, learning the value function of the action policy is simpler than learning the value function of the target policy.

[0093] TDRL-MHR uses log probabilities for all intermediate processes (S t , A t , R t , G tImportance sampling is a general method to estimate the expectation of a distribution given samples from other distributions. Importance sampling ratio is defined as: t t:T-1 t t+1 t+1 T

[0094]

[0095] where P is the state transition probability function. The importance sampling ratio ρ of the target policy π and the behavior policy b is defined as:

[0096]

[0097] The importance sampling ratio only depends on the sampled sequence data of the two policies, and is independent of the dynamic property of the Markov Decision Process, i.e., the state transition probability.

[0098] TDRL-MHR adopts a sequential modeling approach, which can be described as a model-based planning algorithm: sample a candidate action sequence, evaluate its effect using the predictive model, and select the one that maximizes the return. By jointly modeling and sampling actions, states, returns, and rewards using the same process, the model can be prevented from being queried with out-of-distribution actions. Treating the action sequence as an unconstrained optimization variable that does not depend on the state makes it easier to fully utilize the model, as the maximum return problem under the learned model is very similar to the problem of finding adversarial examples for a classifier.

[0099] For the generation task, the model gives a conditional probability based on the history of the generated results at each time step. To generate a complete sentence, the outputs of the model at multiple time steps need to be fused, and the final sequence probability should be maximized. By taking the output with the maximum conditional probability at each time step and using the result from the beginning to the current step as input to obtain the output at the next time step, the process is repeated until the model gives a signal indicating the end of generation. Although this approach can compress the exponential solution space to linear length, it discards most of the possible solutions, and the current-focused strategy cannot guarantee that the final sequence probability is optimal. The beam search module keeps a certain number of outputs with the highest scores at each time step to obtain an approximately optimal solution by trading off performance for time.

[0100] For convenience of description, assume that the search width of the beam search is 2. At time step t, the output with the maximum conditional probability is selected as​​​​​​ and The first two actions with the largest and The beam search algorithm will let these two actions be the action input of the next time step. Assume The conditional probability of branching at the next time step tends to be infinitely small, and for the convenience of drawing, only The case of being the action input of the next time step is considered, and the same is true for other time steps. At the final time step T, the beam search will only select the action that makes the conditional probability The largest, and finally form a complete action sequence

[0101] (S34) Backtracking behavior trajectory generation to generate an interpretable reasoning path

[0102] The TDRL-MHR algorithm establishes the relationship between the four of state, action, return and cumulative return. The TDRL-MHR algorithm implicitly fits a Q function through return constraints. A more vivid description is that in a three-dimensional coordinate system, the X-axis, Y-axis and Z-axis are state, action and return respectively. By fitting the Q function with existing data, when reasoning, the TDRL-MHR algorithm can find the most likely behavior on this function. The learned Q function has certain generalization ability and exploration ability.

[0103] The causal connection attention mask of GPT ensures that the prediction only depends on the previous tokens in the sequence. In the context of multi-hop reasoning in the knowledge graph, it means that future choices are not allowed to affect past decisions. It can be directly used as a method to achieve the goal by adjusting the expected final state. Adjust the sequence on a final goal state, keep the lower diagonal attention mask unchanged, and do not need to modify the standard attention implementation. The transformer will continuously update the reward probability according to the event, forming the state reward correlation discussed by Raposo et al., and achieving accurate value prediction.

[0104] The beam search is used to select the optimal action from the candidate trajectory of each time step, and a complete action sequence is formed after reasoning. The search width B of the beam search, in the first time step, selects the B candidate items with the largest conditional probability to output the first word of the sequence. In each subsequent time step, based on the output sequence of the previous time step, select the B candidate items with the largest conditional probability from all combinations as the candidate output sequence of this time step, always keep B candidates, and finally select the optimal one from the B candidates. The process of beam search can be described by the following formula:

[0105]

[0106] where a is an adjustable parameter, B α to penalize long sequences, y t is the candidate at time step t, d is the end-of-sequence symbol, and the logarithm is used because it converts multiplication into addition.

Claims

1. A multi-hop reasoning method for knowledge graphs based on Transformer deep reinforcement learning, characterized in that, Includes the following steps: S1. Integrate a large number of RDF triples and create a complete knowledge graph through relation completion; S2. Representation learning is performed on the topological structure and triple semantics of the knowledge graph respectively, and the representation tensors of the topological structure and triple semantics are combined in a linear combination to map the representation layer of the knowledge graph to a unified knowledge space. S3. Use the GPT-2 model based on stacked Transformer decoding layers to model the inference layer of the knowledge graph. Utilize bundle search to autoregressively generate the next time step action that maximizes the reward. , Count the time steps; The specific steps for creating a relation-complete knowledge graph in step S1 are as follows: S11. For existing knowledge graphs, RDF triples are connected to the JanusGraph graph database to facilitate subsequent graph relationship completion, query and reading operations; Among them, existing knowledge graph modeling is , Represents the set of all entities in a knowledge graph. It is the collection of all relations in a knowledge graph. It is the set of all fact triples, in fact triples middle, It is the head entity of the fact triple. It is the tail entity of the fact triple. It is a fact triple from the beginning entity Mapped to tail entity Relationship; S12. Based on the existing relationships in the knowledge graph, proactively supplement the missing relationships in equal and mutually exclusive relationships, as well as the mirror and inverse relationships that are conducive to improving the computation path. The mirror inverse relation triple is , for The mirror image of the inverse relation; S13. Add a self-loop relation to each vertex, that is, add a loop relation between each vertex and itself. This is beneficial for increasing the search space of the reasoning agent, expanding the search path of the reasoning agent, and helping the reasoning agent's exploration capabilities; For knowledge graph nodes Self-cyclic relationship; S14. Write the newly added relations back to the JanusGraph graph database to obtain a complete knowledge graph.

2. The knowledge graph multi-hop reasoning method based on Transformer deep reinforcement learning according to claim 1, characterized in that, The specific steps for mapping the representation layer in step S2 are as follows: S21. Extract the pure topology without text content and the semantic triples containing only text content from the complete graph network generated by the graph layer, and import all the topology graph network and semantic triples at once. S22. Feed the graph network topology into the Graph Convolutional Network (GCN) for training to learn the graph network nodes. Topological representation and edge Topological representation ; S23. Feed the fact triples of the graph network into the ConvE network for training to learn the graph network nodes. semantic representation and edge semantic representation ; S24. Linearly connect the topological representation tensor and the semantic representation tensor to obtain the node. Knowledge representation and edge Knowledge representation .

3. The knowledge graph multi-hop reasoning method based on Transformer deep reinforcement learning according to claim 2, characterized in that, The specific steps for modeling the inference layer in step S3 are as follows: S31. Model a multi-hop reasoning reinforcement learning environment based on knowledge graphs, including state, action, immediate reward, subsequent cumulative reward and trajectory; S32. Use GPT for sequence modeling training and select the trajectory that maximizes the benefit. S33. Search for hidden actions in the trajectory based on cluster search to form a complete action sequence; S34. Tracing back the action sequence generates an interpretable reasoning path, presenting explanatory terms that support the conclusion.

4. The knowledge graph multi-hop reasoning method based on Transformer deep reinforcement learning according to claim 3, characterized in that, The state in step S31 ,in Let be the entity node at time t. and Let be the starting entity node and relation edge at time t=0. Let S be the final answer node, and S be the state space, with the initial state being... Termination status ; The action Defined as: ,in For a moment The relationship it occupies For knowledge graph nodes; The timely benefits That is Defined as: In the above formula, It is a bivariate payoff function; The subsequent cumulative income Defined as: In the above formula, This is the time step at which the agent's inference ends. This is the time step for the agent to begin inference. The penalty coefficient for immediate benefits; Define the trajectory as: In the above formula, For trajectory.

5. The knowledge graph multi-hop reasoning method based on Transformer deep reinforcement learning according to claim 4, characterized in that, The sequence modeling training in step S32 is as follows: During training, real trajectories are used, and the trajectory field is directly input in parallel. After passing through an autoregressive model and beam search, all predicted nodes are output. Each node is a multi-dimensional vector. After softmax normalization, the probability distribution is obtained. Then, the cross-entropy of the corresponding real label is calculated to obtain the log-likelihood. The objective of maximizing the log-likelihood yes: Conditional probability of a state The meaning is that at a given current moment Previous state and trajectory before the current moment Under the given conditions, the current state occurs. The probability, For the log-conditional probability of the state; for the conditional probability of the action. The meaning is that at a given current moment The state at any given time in this state Previous actions and trajectories prior to the current moment Under the given conditions, take the current action. The probability, Let be the log-conditional probability of the action; and the conditional probability of the cumulative reward. The meaning is that at a given current moment The state at any given time in this state Previous actions and trajectories prior to the current moment Under certain conditions, cumulative returns occur. The probability, The log-conditional probability of cumulative returns; the conditional probability of immediate returns. The meaning is that at a given current moment The state, the actions taken in that state, and the trajectory before the current moment. Under certain conditions, timely benefits occur The probability, Let be the log-conditional probability of the payoff. To solve for the spatial parameters of the orbit, Let be the conditional probability of the state. The conditional probability of a timely action. The conditional probability of cumulative return; Use logarithmic probability for all intermediate processes Importance sampling is performed, and the importance sampling ratio is calculated using the following method: given the initial state... and time step The set of all possible actions after Subsequent state-action alternation trajectory In strategy The probability of the following occurring is: in, It is the state transition probability function. For the reasoning body at time step The actions taken at that time For knowledge graphs at time steps The state of time; Target Strategy and action strategies Importance sampling ratio under trajectory Defined as: In the above formula, For the agent in time step arrive Importance sampling ratio within a time period.

6. The knowledge graph multi-hop reasoning method based on Transformer deep reinforcement learning according to claim 5, characterized in that, Specifically, step S33 involves: at time step... In the middle, choose conditional probability and The two largest movements and The cluster search algorithm uses these two actions as inputs for the next time step, assuming... The conditional probability of the branch in the next time step tends to infinitesimal; only considering... As the input for the next time step action, until the final time step At that time, beam search will only select items that make the conditional probability... The largest movements ultimately constitute a complete sequence of movements. .

7. The knowledge graph multi-hop reasoning method based on Transformer deep reinforcement learning according to claim 6, characterized in that, Specifically, step S34 involves: the search width of the beam search. In the first time step, select the one with the highest current conditional probability. The first word of each candidate output sequence is used. At each subsequent time step, based on the output sequence of the previous time step, the combination with the highest conditional probability is selected. Each candidate sequence is selected as the output sequence for that time step and is always maintained. There were 10 candidates, and finally... The process of selecting the optimal candidate from 10 candidates, the beam search process can be described as follows: In the above formula, It is an adjustable parameter. Used to punish excessively long sequences. It is a candidate for time step t. It is a candidate for time step B, and d is the sequence terminator.

Images