Knowledge graph reasoning method based on logic rules and reinforcement learning

Abstract

The present application relates to a knowledge graph reasoning method based on logical rules and reinforcement learning, aiming at the problems of weak path correlation and difficulty in determining exploration reward existing in the multi-hop reasoning method based on reinforcement learning, a method of optimizing reinforcement learning path reward by using logical rules is proposed, which solves the defect that the exploration reward of reinforcement learning is difficult to determine in the knowledge graph reasoning process. In the setting of reinforcement learning reward, the present application adopts the combination of hit reward, embedding reward and rule reward, so that the agent fully utilizes logical rules as a guide in the process of exploration and learning, effectively improving the learning efficiency and accuracy of the model. For the setting of single-step reward, a decay factor is introduced to distinguish the importance of different paths. Through the analysis of experimental results, the improved method proposed by the present application can effectively improve the accuracy of the knowledge graph reasoning task.

Description

Technical Field

[0001] The technical solution of this invention relates to a method for knowledge graph reasoning, specifically a knowledge graph reasoning method based on logical rules and reinforcement learning. Background Technology

[0002] Knowledge graphs employ graph-based data structures to represent facts of the real world in a structured way. Leveraging the advantages of graph representation, knowledge graphs play a crucial role in natural language processing tasks such as question answering and semantic search. Since Google launched its first knowledge graph in 2012, numerous knowledge graphs have emerged, including the world knowledge base Freebase, the multilingual knowledge base DBpedia, the English knowledge base WordNet, the comprehensive knowledge bases YAGO and NELL, and Wikidata based on Wikipedia. However, despite their large size, these knowledge graphs still struggle to represent all objective knowledge, especially when faced with ever-increasing amounts of new knowledge, making it difficult to provide robust support for downstream applications. To address this issue, knowledge graph reasoning starts from existing entity relationships in the knowledge base, using one or more existing pieces of knowledge to establish new connections between entities through computational reasoning, thereby enriching the knowledge network and providing a solution for knowledge graph completion tasks.

[0003] Currently, knowledge graph reasoning models can be divided into two categories: single-hop reasoning models and multi-hop reasoning models. Single-hop reasoning models, also known as triple-based reasoning models, aim to represent entities and relations in the knowledge graph using low-dimensional embeddings in a vector space. Typically, a scoring function is defined to measure the correctness of the triples. Single-hop reasoning models can be divided into three categories: distance models, tensor decomposition models, and semantic matching models. Distance models, also known as translation models, typically include algorithms such as TransE, TransR, and RotatE. They represent each relation in the knowledge graph as a translation transformation from the head entity vector to the tail entity vector. By minimizing the translation transformation error, the model maps entities and relations to a low-dimensional space. Tensor decomposition models improve and adjust basic tensor decomposition algorithms and apply them. RescAL is a representative example. It decomposes relation tensors into multiple matrices using specific techniques and uses these matrices to construct a low-dimensional embedding representation of the knowledge graph. Semantic matching models use the similarity information of entities and relations to design objective functions and discover the latent semantics of different entities and relations in the low-dimensional vector space. DistMult matches binary and trigram semantics in the knowledge graph. QuatE, based on the complex vector representation, proposes a vector representation of a hypercomplex space and maps entities and relations to this space. For knowledge graph reasoning, single-hop reasoning models can effectively uncover the implicit semantic information between entities and relations, and have better performance than other methods, but poor interpretability.

[0004] The goal of multi-hop inference models is to find multiple triples related to a question in a knowledge graph and model multi-hop long paths to find the correct answer. Specifically, given a question triple, the model predicts the tail entity, and continuously finds suitable next-hop triples based on the neighborhood information of the head entity until the inference ends. At the end of the inference, the model provides an interpretable inference path. The DeepPath model uses a reinforcement learning framework for relation path inference and has demonstrated excellent performance in relation path inference. MINERVA uses the REINFORCE algorithm to complete multi-hop inference tasks. MultiHop, based on the MINERVA model, adds action discarding and reward shaping methods to reduce the misleading effect of false paths on the model. RLH introduces a hierarchical reinforcement learning framework to simulate human thinking, decomposing macro-operations into simple sub-tasks, and then learning the latent semantics of each relation. For long-tail data in knowledge graphs, MetaKGR further introduces meta-learning into multi-hop inference, enabling the model to quickly learn a better parameter when facing few-sample tasks. For sparse knowledge graphs, DacKGR proposes dynamic prediction and dynamic completion strategies, introducing additional actions during the inference process to increase the number of paths. To adapt to the dynamic growth of knowledge graphs, CPL combines multi-hop reasoning with fact extraction to find paths and extract facts from text. Compared to single-hop reasoning models, multi-hop reasoning models have slightly lower accuracy but better interpretability and can effectively provide reasoning paths.

[0005] However, multi-hop reasoning methods based on reinforcement learning still have two problems: (1) weak path correlation. Because reinforcement learning uses random sampling in path selection, the correlation between paths is weakened, resulting in poorer interpretability of paths compared to logical rules. (2) difficulty in determining exploration rewards. Reinforcement learning depends to some extent on reward feedback. The model needs to optimize the distribution of action probabilities based on the amount of reward. However, due to the complexity of the knowledge graph environment, the model often finds it difficult to provide appropriate rewards. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a knowledge graph reasoning method based on logical rules and reinforcement learning. This method is a way to optimize the reward of reinforcement learning path using logical rules. It overcomes the shortcomings of existing reinforcement learning methods, which are difficult to determine the exploration reward in the knowledge graph reasoning process, resulting in low exploration efficiency of the agent. It improves the accuracy of reinforcement learning reasoning and enhances the interpretability of the path.

[0007] The technical solution adopted by this invention to solve this technical problem is: a knowledge graph reasoning method based on logical rules and reinforcement learning, which is a method to optimize the reward of reinforcement learning path using logical rules. The specific steps are as follows:

[0008] The first step is reinforcement learning state modeling:

[0009] (1.1) Knowledge graph triple embedding representation:

[0010] The original triplet (s,r,o) data is input into the embedding method for representation learning, and is transformed into a vector representation using the scoring function formula (1).

[0011]

[0012] Where e s e represents the embedding representation of the head entity s in the original triple. o r represents the embedding representation of the tail entity o in the original triple. r It is a relational parameter that depends on r. and They are e s and r r Reconstructed vectors in a two-dimensional space, stacked together. * denotes a convolution operation, and ω represents the filter of the convolutional layer. The method utilizes the reconstructed vectors... and As input to the convolutional layer, ω serves as the convolution kernel to compute the feature tensor. The feature tensor is then reconstructed into a vector using the transformation function vec, and a linear transformation is performed using matrix W. Finally, the embedding of the tail entity is matched using the inner product. o .

[0013] Next, the fractions are normalized, p = σ(ψ) r (e s ,e o ), where σ represents the sigmoid function. The embedding representation method utilizes the loss function of formula (2). Optimize parameters.

[0014]

[0015] Where t represents the label vector, which is 1 if there is a relationship between entities and 0 if there is no relationship.

[0016] (1.2) Representation of historical path information:

[0017] The path length to be explored is defined as T steps. The action (r, e) at step t is vectorized into a based on the relation vector and entity vector of step (1.1) above. t =[r;e], using the Long Short Term Memory Network (LSTM) of formula (3) to store historical path information.

[0018] h t =LSTM(h t-1a t-1 (3)

[0019] (1.3) Components of reinforcement learning states:

[0020] The entity and relation embedding representations in step (1.1) and the historical information in step (1.2) are combined, and the state of step t in reinforcement learning is constructed using formula (4).

[0021] s t =[r q ;e t h t (4)

[0022] Where, r q The embedded representation of the relation to be queried, e t h represents the embedding representation of the entity reached at step t. t This represents the path history information explored at step t.

[0023] This completes the construction of the reinforcement learning state, yielding the state s at step t. t .

[0024] The second step involves inputting the state into a reinforcement learning policy network. This network guides the agent to explore the action space, obtaining the probability distribution of the next action. Random sampling is then used to select the next action, thus initiating the next path exploration.

[0025] The state s obtained in step t above is used as the first step. t The input is fed into the reinforcement learning policy network, and the probability distribution of the action to be taken at step t is obtained using formula (5).

[0026] π θ (a t |s t )=σ(A t (W1ReLU(W2s t ))) (5)

[0027] Where, π θ (a t |s t ) indicates in s t In this state, the action space The probability distribution of all actions in σ. σ represents the softmax function, W1 and W2 represent two linear neural networks, and ReLU represents the activation function. The action space is used to represent the probability distribution of all actions in σ. t In a given state, the next action to be taken is selected; the action space is represented by a vector. From action space The state transition is composed of a stack of all actions within the action space, where d represents the dimension of the entity and relation vectors. Then, a random sampling method is used to select an action from the action space to initiate the state transition, thus proceeding to the next step.

[0028] The third step is to optimize reinforcement learning rewards using logical rules:

[0029] After exploring the second step (T steps) above, the reinforcement learning reward is optimized using the embedding representation score and the confidence score of the logical rule. First, it is determined whether the obtained result is the expected target result. If it is consistent with the expected result, the reward is 1. Otherwise, the reward is optimized based on the reasoning process and the result. The process is as follows: First, the search result is scored using the embedding representation according to formula (1) to obtain the embedding reward. Then, the reasoning path is compared with the rules obtained by the rule miner, and the confidence score of the rules obtained by the rule miner is used as the reward to obtain the rule reward. Finally, the embedding reward and the rule reward are weighted and combined to obtain the final reward.

[0030] Step 4: Optimize the policy network using rewards:

[0031] The reward obtained in the third step above is decayed, so the single-step reward obtained in step t can be expressed as r. t =γ t R(s T |e s Then, the gradient descent strategy network parameters are optimized using formula (6).

[0032]

[0033] Where θ represents the parameters of the policy network, and J(θ) represents the expected reward the model can obtain when the network parameters are θ. β represents the learning rate. v(st) represents the state value function, which is the expected reward obtained from the current state. The state value function formula is v(st). t =W3ReLU(W4s) t ), where W3 and W4 represent two linear neural networks, and ReLU represents the activation function.

[0034] The fifth step involves using a bundle search method combined with a policy network to select actions, thereby enabling knowledge graph reasoning.

[0035] The knowledge graph triplet data input in the first step is used as follows: 70% is used as the training set, and 30% as the test set. After processing the training set through steps one through four, a trained policy network is obtained. Then, the test set data is processed through the first step to obtain reinforcement learning states, which are input into the policy network for prediction. For each output of the policy network, a beam search method is used to select the action with the highest probability until the pre-defined number of steps is reached. This achieves reasoning from the knowledge graph.

[0036] The above knowledge graph reasoning method based on logical rules and reinforcement learning optimizes reinforcement learning rewards by integrating embedded rewards and rule rewards on the basis of the original hit rewards, adding logical rules as guidance, and using formula (7) to calculate specific path rewards.

[0037] R eward =R hit +(1-R hit )(δR emb +λR rule (7)

[0038] Among them, R hit This indicates a hit reward; if the exploration results match expectations, R... hit If the value is 1, then it is 0; otherwise, it is 0. R emb Indicates embedded reward, R rule Let δ and λ represent the rule reward, and let δ and λ represent the weights corresponding to the embedding reward and the rule reward, respectively. To ensure that the reward is less than 1 in the case of a miss, we have δ + λ = 1.

[0039] The beneficial effects of this invention are as follows: Compared with the prior art, the outstanding substantive features and significant progress of this invention are as follows:

[0040] (1) The method of the present invention adds rule rewards when determining the overall reward of the path, which takes advantage of the strong interpretability of logical rules to guide reinforcement learning to prioritize the exploration of paths with strong interpretability, thereby improving the correlation between paths.

[0041] (2) In determining the single-step reward, the method of the present invention uses a decay factor to distinguish the importance of asynchronous paths and introduces a state value function to give different adaptive rewards for each step in combination with the current state, which overcomes the problem of difficulty in determining the reward of reinforcement learning and improves the accuracy of knowledge graph reasoning. Attached Figure Description

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] Figure 1 This is a schematic diagram of the structure of the method of the present invention.

[0044] Figure 2 This is a schematic diagram illustrating the embedded structure of the method of the present invention. Detailed Implementation

[0045] Figure 1 The illustrated embodiment shows that the process of the method of the present invention is as follows: knowledge graph triple data embedding representation → embedding entities and relations into historical information to obtain reinforcement learning state → using policy network to obtain probability distribution of different actions → obtaining the next action required through random sampling → calculating the final path reward based on the final inference result and the rules mined by the rule miner → determining the single-step reward through decay processing and optimizing the policy network using the reward → using the policy network to implement knowledge graph inference in the training set using the bundle search method.

[0046] Figure 2 The illustrated embodiment shows that the training process for the knowledge graph triplet embedding representation in the method of the present invention is as follows: Entities and relations are initialized into vectorized representations and the results are stacked. Compared to one-dimensional convolution, two-dimensional convolution can extract more features. Therefore, the vectorized entity and relation embeddings are reconstructed and concatenated to transform these vectors into two-dimensional vectors. Then, a convolution operation is performed to obtain a feature map. The obtained feature map is subjected to a fully connected linear transformation to obtain a hidden layer. This hidden layer is then multiplied by the entity matrix composed of all entity vectors and normalized using a sigmoid function to obtain the matching degree of each entity. Candidate targets and results are selected and scored, and then the entity and relation embedding representations are optimized using a loss function. The goal of the loss function is to minimize the difference between the predicted result and the expected result vector, making the result obtained by the embedding method closer to the expected result.

[0047] Example 1

[0048] The knowledge graph reasoning method based on logical rules and reinforcement learning in this embodiment is a method that optimizes the reward of reinforcement learning paths using logical rules. The specific steps are as follows:

[0049] The first step is reinforcement learning state modeling:

[0050] (1.1) Knowledge graph triple embedding representation:

[0051] The original triplet (s, r, o) data is input into the embedding method for representation learning, and is transformed into a vector representation using the scoring function formula (1).

[0052]

[0053] Where es represents the embedding representation e of the head entity s in the original triple. o r represents the embedding representation of the tail entity o in the original triple. rIt is a relational parameter that depends on r. and They are e s and r r The reconstructed vectors in a stacked two-dimensional space are used in this invention, with the entity and relation embedding vectors having a dimension of 200. * represents a convolution operation, and ω represents the filter of the convolutional layer. The method utilizes the reconstructed vectors... and As the input to the convolutional layer, ω serves as the convolution kernel. This invention uses a 3x3 kernel to calculate the feature tensor. The feature tensor is then reconstructed into a vector using the transformation function vec. A linear transformation is performed using matrix W, which employs a 10368x200 linear transformation. The embedding of the tail entity is matched using the inner product. o .

[0054] Next, the fractions are normalized, p = σ(ψ) r (e s e o ), where σ represents the sigmoid function. The embedding representation method utilizes the loss function of formula (2). Optimize parameters.

[0055]

[0056] Where t represents the label vector, which is 1 if there is a relationship between entities and 0 if there is no relationship.

[0057] (1.2) Representation of historical path information:

[0058] The path length for exploration is defined as T steps, and the exploration path length used in this invention is 3. The action (r, e) at step t is vectorized into a based on the relation vector and entity vector of step (1.1) above. t =[r;e], using the Long Short Term Memory Network (LSTM) of formula (3) to store historical path information, the present invention uses 3 LSTM layers and 200 dimensions of historical information.

[0059] h t =LSTM(h t-1 a t-1 (3)

[0060] (1.3) Components of reinforcement learning states:

[0061] The entity and relation embedding representations in step (1.1) and the historical information in step (1.2) are combined, and the state of step t in reinforcement learning is constructed using formula (4).

[0062] st =[r q ;e t h t (4)

[0063] Where, r q The embedded representation of the relation to be queried, e t h represents the embedding representation of the entity reached at step t. t This represents the path history information explored at step t.

[0064] This completes the construction of the reinforcement learning state, yielding the state s at step t. t .

[0065] The second step involves inputting the state into a reinforcement learning policy network. This network guides the agent to explore the action space, obtaining the probability distribution of the next action. Random sampling is then used to select the next action, thus initiating the next path exploration.

[0066] The state s obtained in step t above is used as the first step. t The input is fed into the reinforcement learning policy network, and the probability distribution of the action to be taken at step t is obtained using formula (5).

[0067]

[0068] Where, π θ (a t |s t ) indicates in s t In this state, the action space The probability distribution of all actions is given. σ represents the softmax function, and W1 and W2 represent two linear neural networks; the linear networks used in this invention are 600*200 and 200*200, respectively. ReLU represents the activation function. The action space is used to define the probability distribution of all actions in s. t In a given state, the next action to be taken is selected; the action space is represented by a vector. From action space The state transition is composed of a stack of all actions within the action space, where d represents the dimension of the entity and relation vectors. Then, a random sampling method is used to select an action from the action space to initiate the state transition, thus proceeding to the next step.

[0069] The third step is to optimize reinforcement learning rewards using logical rules:

[0070] After exploring step T of the second step above, in this invention, T is set to 3. The reinforcement learning reward is optimized using the embedding representation score and the confidence of the logical rule. The rule miner in this method adopts the AnyBURL method. First, it is determined whether the obtained result is the expected target result. If it is consistent with the expected result, the reward is 1. Otherwise, the reward is optimized according to the reasoning process and the result. The process is as follows: First, the search result is scored according to the embedding representation using formula (1) to obtain the embedding reward. Then, the reasoning path is compared with the rule obtained by the rule miner. The confidence of the rule obtained by the rule miner is used as the reward to obtain the rule reward. Finally, the embedding reward and the rule reward are weighted and combined to obtain the final reward. The final reward formula is obtained using formula (6).

[0071] R eward =R hit +(1-R hit )(δR emb +λR rule (6)

[0072] Among them, R hit This indicates a hit reward; if the exploration results match expectations, R... hit If yes, otherwise 0. R emb Indicates embedded reward, R rule Let δ and λ represent the rule reward, and let δ and λ represent the weights corresponding to the embedding reward and the rule reward, respectively. To ensure that the reward is less than 1 in the case of a miss, we have δ + λ = 1. In this invention, δ = 0.9 and λ = 0.1.

[0073] Step 4: Optimize the policy network using rewards:

[0074] The reward obtained in the third step above is decayed, so the single-step reward obtained in step t can be expressed as r. t =γ t R(s T |e s In this invention, γ is taken as 0.99. Then, the gradient descent strategy network parameters are optimized using formula (7).

[0075]

[0076] Where θ represents the parameters of the policy network, and J(θ) represents the expected reward the model can obtain when the network parameters are θ. β represents the learning rate. v(s) t ) represents the state value function, which is the expected reward obtained from the current state. The formula for the state value function is v(s). t =W3ReLU(W4s) tIn this invention, W3 and W4 represent two linear neural networks, which are 600*200 and 200*200 respectively, and ReLU represents the activation function.

[0077] The fifth step involves using a bundle search method combined with a policy network to select actions, thereby enabling knowledge graph reasoning.

[0078] The knowledge graph triplet data input in the first step is used as follows: 70% is used as the training set, and 30% is used as the test set. After processing the training set through steps one through four, a trained policy network is obtained. Then, the test set data is processed through the first step to obtain the reinforcement learning state, which is then input into the policy network for prediction. For each output result of the policy network, a beam search method is used to select the action with the highest probability. In this invention, the beam width is set to 100, until the number of steps specified before training is reached. In this invention, the number of steps is set to 3. This enables reasoning based on the knowledge graph.

[0079] This embodiment conducted experiments on two datasets: the UMLS dataset and the NELL-995 dataset. The UMLS dataset is a medical dataset containing 135 entities, 46 relations, and 5316 facts. The mean degree of entities in this dataset is 38.6, and the median degree is 28. The NELL-995 dataset is a subset of the NELL database, mainly consisting of triples automatically extracted from the network. The NELL-995 dataset contains 75492 entities, 200 relations, and 154213 facts. The mean degree of entities in this dataset is 4, and the median degree is 1. The method of this invention is written in Python v3.6.0, using PyTorch version 1.5.1 as the deep learning framework. The CPU is an Intel(R) Core(TM) i7-6600U, the GPU is an NVIDIA GeForce RTX 2080Ti, the server memory is 128GB, and the operating system is Linux. The mean reciprocal rank in this embodiment is 0.955 and 0.754, respectively.

[0080] To verify the advantages of the method of the present invention in knowledge graph reasoning, this embodiment selects five widely used knowledge graph reasoning methods and compares them with the knowledge graph reasoning method of the present invention based on logical rules and reinforcement learning. The five knowledge graph reasoning methods are: embedding-based methods: DistMult, RotatE; logical rule-based methods: AnyBURL; and reinforcement learning-based methods: MINERVA, MultiHop.

[0081] The results of comparing the above five knowledge graph reasoning methods with the knowledge graph reasoning method based on logic rules and reinforcement learning of this invention are shown in Table 1.

[0082] Table 1. Comparison of knowledge graph reasoning results of different methods on two databases

[0083]

[0084] The evaluation metrics in the table are Mean Reciprocal Rank (MRR), Hits@1, Hits@3, and Hits@10.

[0085] MRR represents the average of the last-ranked entities in the candidate entity ranking list, and is calculated using the following formula:

[0086]

[0087] Where m represents the number of triples in the test set, and rank i This represents the ranking of the tail entity in the inference among all candidate entities for the i-th triple.

[0088] Hits@1, Hits@3, and Hits@10 represent the proportions of the tail entity in the inference that are ranked 1st, 3rd, and 10th in the candidate entity list, respectively. The calculation formula is as follows:

[0089]

[0090] Where, k∈{1,3,10}, This indicates the indicator function, rank, which is used when the condition is met. i When k ≤ k, the function value is 1; otherwise, the function value is 0.

[0091] Table 1 compares the experimental results of our proposed method with those of commonly used knowledge graph reasoning methods. The results show that our proposed method outperforms other knowledge graph reasoning methods on both the UMLS and NELL-995 datasets.

[0092] In the above embodiments, the embedding representation method, rule miner, reinforcement learning method, beam search method, etc., are all well known in the art.

[0093] It should be noted that this invention is not limited to the specific embodiments described above. Any other embodiments obtained by those skilled in the art under the guidance of this invention without departing from its principles are considered to be within the protection scope of this invention.

Claims

1. A knowledge graph reasoning method based on logical rules and reinforcement learning, characterized by: Includes the following steps: The first step is reinforcement learning state modeling: (1.1) Knowledge graph triple embedding representation: The original triplet The data is input into an embedding method for representation learning, and is transformed into a vector representation using the scoring function formula (1). in Represents the head entity in the original triple. Embedded representation, Represents the tail entity in the original triplet Embedded representation, It depends on A relational parameter, and They are and Reconstructed vectors in a stacked two-dimensional space. This represents the convolution operation. The filter represents the convolutional layer, and the method utilizes the reconstructed vector. and As input to the convolutional layer, As the convolution kernel, the feature tensor is computed, and then the feature tensor is transformed using a transformation function. Reconstructing vectors using matrices Perform a linear transformation and match the embedding of the tail entity using the inner product. , Next, the scores are normalized. ,in express The function, the embedding representation method utilizes the loss function of formula (2). Optimize parameters, in, This represents a label vector; a value of 1 indicates a relationship between entities, and a value of 0 indicates a relationship between entities. (1.2) Representation of historical path information: The length of the explored path is defined as T steps, and the first step is... Step movement Based on the relation vector and entity vector from step (1.1) above, the vectorized representation is as follows: Historical path information is stored using a Long Short Term Memory Network (LSTM) based on formula (3). (1.3) Components of reinforcement learning states: The entity and relation embedding representations from step (1.1) and the historical information from step (1.2) are combined, and the reinforcement learning step is completed using formula (4). The construction of the state of the step. in, The embedded representation of the relation to be queried. Indicates the first The step reaches the embedded representation of the entity. Indicates the first Historical information on the path explored step by step This completes the construction of the reinforcement learning state, obtaining the state at the [number]th [year]. Step state , The second step involves inputting the state into a reinforcement learning policy network. This network guides the agent to explore the action space, obtaining the probability distribution of the next action. Random sampling is then used to select the next action, thus initiating the next path exploration. The first step above obtained in the first... Step state The input is fed into the reinforcement learning policy network, and the result is obtained using formula (5). When taking a step, a probability distribution of the action needs to be used. in, Indicates in In this state, the action space The probability distribution of all actions in the given... express function, and This represents two linear neural networks. This represents the activation function, and the action space is used for... In a given state, the next action to be taken is selected; the action space is represented by a vector. From action space All the actions within are stacked together, among which, The dimension of the entity and relation vectors is represented. Then, a random sampling method is used to select an action in the action space to initiate a state transition, thus proceeding to the next transition. The third step is to optimize reinforcement learning rewards using logical rules: After exploring the second step T times, the reinforcement learning reward is optimized using the embedding representation score and the confidence of the logical rule. First, it is determined whether the obtained result is the expected target result. If it is consistent with the expected result, the reward is 1. Otherwise, the reward is optimized according to the reasoning process and the result. The process is as follows: First, the search result is scored according to the embedding representation using formula (1) to obtain the embedding reward. Then, the reasoning path is compared with the rules obtained by the rule miner. The confidence of the rules obtained by the rule miner is used as the reward to obtain the rule reward. Finally, the embedding reward and the rule reward are weighted and combined to obtain the final reward. Step 4: Optimize the policy network using rewards: The reward obtained in the third step above is subject to decay, and then... The single-step reward obtained in a step can be represented as , We set the value to 0.99, and then use formula (6) to optimize the network parameters of the gradient descent strategy. in, The parameters represent the policy network. This indicates that the network parameters are At that time, the expected reward that the model can obtain is Indicates the learning rate. Let represent the state value function, which is the expected reward that can be obtained starting from the current state. The formula for the state value function is: ,in, and This represents two linear neural networks. This represents the activation function. The fifth step involves using a bundle search method combined with a policy network to select actions, thereby enabling knowledge graph reasoning. Take 70% of the knowledge graph triplet data input in the first step above as the training set and 30% as the test set. After processing the training set through the first to fourth steps above, a trained policy network is obtained. Then, the test set data is processed through the first step above to obtain the reinforcement learning state, which is then input into the policy network for prediction. For each output result of the policy network, the beam search method is used to select the action with the highest probability until the number of steps specified before training is reached, thereby realizing the reasoning of the knowledge graph.

2. The knowledge graph reasoning method based on logical rules and reinforcement learning according to claim 1, characterized in that: The optimization of reinforcement learning rewards using logical rules is based on the original hit reward, which integrates embedded rewards and rule rewards, and adds logical rules as guidance. The specific path reward is calculated using formula (7). in, This indicates a hit reward if the exploration results match expectations. It is 1 if it is 1, otherwise it is 0. Indicates embedded rewards, Indicates a rule-based reward. and This indicates the weights corresponding to the embedding reward and the rule reward. To ensure that the reward is less than 1 in the event of a miss, there is... .

Images