A knowledge graph multi-hop reasoning method based on reinforcement state modeling

Abstract

The application discloses a kind of knowledge graph multi-hop reasoning methods based on reinforcement state modeling, comprising the following steps: using a set of embedding model, knowledge graph data is pre-trained, and the embedding representation of entity and relation is obtained, and using test set to evaluate the prediction performance of each embedding model, obtain a set of MRR index;According to MRR index, calculate model weight, and sort each embedding model, obtain embedding model with weight;According to embedding model with weight, calculate the corresponding global target embedding representation for each query;According to global target embedding representation, construct reinforcement learning state model;The reinforcement learning state model is trained, and the inference completion work of knowledge graph is carried out using the trained reinforcement learning state model, to complete the knowledge graph multi-hop reasoning based on reinforcement state modeling.The application solves the problem that the global information representation of reinforcement agent is insufficient, the inference on sparse graph lacks basis, thereby limiting the model link prediction performance.

Description

Technical Field

[0001] This invention belongs to the field of knowledge graph completion technology, specifically relating to a multi-hop reasoning method for knowledge graphs based on reinforced state modeling. Background Technology

[0002] In recent years, Knowledge Graphs (KGs), as a form of structured representation of human knowledge, have been widely used in the field of artificial intelligence, providing a solid foundation for diverse downstream tasks such as question answering, recommendation, and decision-making. However, with the continuous expansion of KG research, its inherent defects, such as incompleteness and insufficiency, have become increasingly apparent, becoming key factors limiting the practical application performance of KGs. Considering that many knowledge graphs are constructed manually or semi-automatically, even large-scale knowledge graphs still exhibit sparse entity relation mining. For small and medium-sized or domain-specific knowledge graphs with richer topic types and broader application scenarios, the sparsity problem is even more prominent. Therefore, industry practitioners urgently need to find a method to solve the sparsity of graphs. To address this need, Knowledge Graph Completion (KGC) technology, also known as link prediction technology, has emerged. Knowledge Graph Completion (KGC) aims to predict missing entities and relations in a knowledge graph, explore unknown facts, and thus improve and enrich the information content of the knowledge graph as much as possible. This is a key means to improve the quality of knowledge graphs.

[0003] Initially, researchers employed embedding-based methods to complete knowledge graphs by mapping entities and relations to the same feature space. However, these methods lacked interpretability and failed to model complex relational paths. Later, path-based methods, by encoding path information into the graph structure, overcame this limitation to some extent. However, their performance remained difficult to improve due to the randomness of pathfinding and path explosion. Recently, multi-hop reasoning models based on reinforcement learning have gradually emerged in the field. This reasoning method aligns with human reasoning processes and intuition, predicting missing triples through finite-step reasoning on the knowledge graph (KG), and exhibits strong interpretability.

[0004] Although current mainstream models exhibit subtle differences in their specific state modeling methods, they generally follow the aforementioned framework, which involves concatenating three embeddings: the agent's current entity, pathfinding history, and global query information. While this state modeling approach can represent the agent's current state, it fails to effectively guide inference based on global information. This results in the agent lacking a basis for reasoning on a sparse graph, thus limiting the model's link prediction performance. Summary of the Invention

[0005] This invention provides a knowledge graph multi-hop reasoning method based on reinforced state modeling, which solves the problem that insufficient global information representation of reinforced agents leads to a lack of basis for reasoning on sparse graphs, thus limiting the model's link prediction performance.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is: a knowledge graph multi-hop reasoning method based on reinforced state modeling, comprising the following steps:

[0007] S1. Use a set of embedding models to pre-train the knowledge graph data to obtain the embedding representations of entities and relations, and use the test set to evaluate the prediction performance of each embedding model to obtain a set of MRR indicators, MRRs.

[0008] S2. Based on the MRR index, calculate the model weights of each embedding model, and reorder the embedding models in descending order of model weights to obtain weighted embedding models.

[0009] S3. Based on the weighted embedding model, calculate the corresponding global target embedding representation for each query;

[0010] S4. Construct a reinforcement learning state model based on the global target embedding representation;

[0011] S5. Train the reinforcement learning state model using the knowledge graph, and use the trained reinforcement learning state model to complete the reasoning and inference work of the knowledge graph, thus completing the multi-hop reasoning of the knowledge graph based on reinforcement state modeling.

[0012] The beneficial effects of this invention are as follows: By pre-training the knowledge graph data using an embedding model, a global target embedding representation is calculated without increasing excessive additional computational costs. This representation is then concatenated into the reinforcement learning model to guide the agent in pathfinding decisions. Therefore, the agent can refer to the pre-calculated global target and combine it with the experience learned by the original reinforcement learning model to make more accurate multi-step inference decisions. This solves the problem of insufficient global information representation for reinforcement learning agents, leading to a lack of basis for inference on sparse graphs and thus limiting the model's link prediction performance.

[0013] The formula for calculating the model weights of each embedded model in step S2 is as follows:

[0014]

[0015] Where, α i Represents the weights of the i-th embedded model, MRR i Let represent the MRR metric of the i-th embedding model, and k' represent the total number of MRR elements in the set.

[0016] The beneficial effect of the above further scheme is that the model weights of the embedded models demonstrate the contribution of each individual embedded model to the subsequent global objective calculation. For prediction models with good prediction performance, their predicted answer entities are more accurate, and constructing the global objective using them as a reference will be more accurate.

[0017] Furthermore, step S3 specifically comprises the following steps:

[0018] S31. Input the query q into a set of weighted embedding models, and predict the query q through each weighted embedding model to obtain a set of global targets Targets;

[0019] S32. Multiply each confidence score in the set of global targets Targets by the weight corresponding to each weighted embedding model, accumulate the confidence scores of the predicted entities with the same answer, and arrange all results in descending order of confidence score. Take the first L answers to form a global target candidate list T0 for query q.

[0020] S33. Use a confidence filter to filter out the poor-quality answers in the global target candidate list T0 to obtain the global target list T1;

[0021] S34. Based on the global target list T1, calculate the global target embedding representation e by weighted average. gt .

[0022] The beneficial effects of the above-mentioned further scheme are as follows: By using a weighted embedding model, the expected target embedding can be computed for each query without increasing computational cost, thus supplementing the global information of the learning state model. Furthermore, to prevent the limited performance of a single embedding model from misleading the agent in some cases, an integrated combination mode is adopted, with multiple embedding models generating the representation together. Finally, to prevent poorly designed answers with excessively low confidence from introducing noise during the generation of the global target representation, a confidence filter is introduced to remove entities that deviate too much from the best answer.

[0023] Further, in step S34, the global target embedding representation e is calculated. gt The formula is:

[0024]

[0025] Among them, e gt This represents the global target embedding representation, where N represents the number of targets in the global target list, and e i p represents the embedding representation of the answer entity in the i-th embedding model. i This represents the confidence level of the entity output by the i-th embedding model.

[0026] Furthermore, step S4 specifically involves the following steps:

[0027] S41. Based on the global target embedding representation e gt Define the state of the agent when it reaches the t-th time step;

[0028] S42. Define the action space based on the knowledge graph and perform agency based on the action space;

[0029] S43. Select the Long Short-Term Memory (LSTM) network and the fully connected network as the memory network and the policy network, respectively.

[0030] S44. At time step t, the agent inputs the state into the policy network and executes a certain action;

[0031] S45. After the agent performs an action, move the agent to the entity node connected to that action;

[0032] S46. Define the reward value for the agent to complete the construction of the learning state model.

[0033] The beneficial effects of the above-mentioned further scheme are: constructing a reinforcement learning state model to represent the agent's state at each time step, and the state at each time step contains extended global information, which helps to make pathfinding decisions after inputting into the policy network.

[0034] Furthermore, in step S41, the expression defining the state of the agent at the t-th time step is:

[0035] s t =(e t ,h t ,(e s ,r q ,e gt ))

[0036] Among them, s t Let e ​​represent the state of the reinforcement learning model at time step t. t h represents the agent's position on the knowledge graph at time step t. t e represents the historical decision information obtained by the agent through the Long Short-Term Neural Network at time step t. s Represents the head entity in the query, r q Indicates the relation in the query, e gt This represents the global target embedding representation.

[0037] Furthermore, the expression for defining the action space in step S42 is:

[0038] A t ={(r n ,en )|(e t ,r n ,e n )∈G}

[0039] Among them, A t Let r represent the action space of the model at time step t. n Indicates the next-hop neighbor relationship of the current entity, e n Indicates the next-hop neighbor entity of the current entity, e t G represents the entity location of the agent at time step t, and G represents the knowledge graph.

[0040] Furthermore, the expression for defining the agent's reward value in step S46 is:

[0041] If (e) s ,r q ,e gt )∈G,R b (s t ) = 1

[0042] Otherwise, R s (s t )=f(e s ,e t )

[0043] R(s t ) = R b (s t )+(1-R b (s t ))R s (s t )

[0044] Among them, s t and e t R represents the final state of the learned state model and the final predicted answer entity. b Represents a binary reward, R s e represents the similarity reward. s Represents the head entity in the query, r q Indicates the relation in the query, e gt Let f() represent the global target embedding representation, f() represent the calculation of cosine similarity, R represent the total reward calculated based on the binary reward and similarity reward, G represent the knowledge graph, and R b (s t ) indicates that the agent's final state is s t The binary reward given at that time, R s (s t ) indicates that the agent's final state is s t The similarity reward given at the time, R(s) t) indicates that the agent's final state is s t The total reward given at that time.

[0045] Furthermore, step S5 specifically involves the following steps:

[0046] S51. Based on the action space, search for entity nodes in the knowledge graph G, and select the neighbor of the entity node to execute an action;

[0047] S52. Based on the actions performed by the neighbors, calculate the state of the agent at each time step, and input the state of each time step into the policy network to obtain the agent's action in the next hop and the entity node connected to the next hop action;

[0048] S53. Input the entity node of the next hop into the long short-term neural network to obtain the historical information of the agent's decision-making.

[0049] S54. Input the historical information of the agent's decision-making into the policy network, calculate the agent's reward value, and evaluate the agent's decision-making at each time step based on the agent's reward value.

[0050] S55. Based on the decisions made by the agent at each time step, train the reinforcement learning state model using beam search, and determine whether the number of iterations has reached the preset threshold. If so, complete the training of the reinforcement learning state model and proceed to step S56; otherwise, return to step S51.

[0051] S56. Perform reasoning completion work on the knowledge graph based on the completed reinforcement learning state model, and complete multi-hop reasoning of the knowledge graph based on reinforcement state modeling. Attached Figure Description

[0052] Figure 1 This is a flowchart of the knowledge graph multi-hop reasoning method based on reinforced state modeling of the present invention.

[0053] Figure 2 This is a flowchart illustrating the calculation of the global target embedding representation in this invention.

[0054] Figure 3 This is a structural diagram of the reinforcement learning state model of the present invention.

[0055] Figure 4 This is the link prediction result of this invention on the NELL-995 dataset.

[0056] Figure 5 This is the link prediction result of this invention on the WM18RR dataset. Detailed Implementation

[0057] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

[0058] Example 1

[0059] like Figure 1 As shown, this invention provides a knowledge graph multi-hop reasoning method based on reinforced state modeling, comprising the following steps:

[0060] S1. Use a set of embedding models to pre-train the knowledge graph data to obtain the embedding representations of entities and relations, and use the test set to evaluate the prediction performance of each embedding model to obtain a set of MRR indicators, MRRs.

[0061] S2. Based on the MRR index, calculate the model weights of each embedding model, and reorder the embedding models in descending order of model weights to obtain weighted embedding models.

[0062] S3. Based on the weighted embedding model, calculate the corresponding global target embedding representation for each query;

[0063] S4. Construct a reinforcement learning state model based on the global target embedding representation;

[0064] S5. Train the reinforcement learning state model using the knowledge graph, and use the trained reinforcement learning state model to complete the reasoning and inference work of the knowledge graph, thus completing the multi-hop reasoning of the knowledge graph based on reinforcement state modeling.

[0065] In this embodiment 1, the embedding model in step S1 is Embs = {Emb1,Emb2,...,Emb} k}, where Emb k Let represent the k-th embedding model. Also, in step S1, MRRs = {MRR1, MRR2, ..., MRR...} k},in MRR k Let denot 'k' be the MRR metric of the k-th embedding model, Q be the set of all query triples, |Q| be the number of all query triples (i.e., the number of queries in the test set), and rank be the MRR metric. i The rank represents the confidence ranking of the actual correct answer to the query within the list of predictions from the embedded model. |Q| This indicates the confidence ranking of the last correct answer in the triplet within the prediction results list.

[0066] In Example 1, three classic embedding models were selected: the bilinear model DistMult, the complex model ComplEx, and the convolutional embedding model ConvE, as the pre-trained model set. First, an input set is required: knowledge graph data, specifically including the knowledge graph G and the training set D. train and test set D test The two datasets are distinct and each contains several triples, in the form of (e s ,r q ,e t In addition, missing triples of tail entities (e) will be removed. s ,r q A query is defined as a set of k embedded models, where {Emb1, Emb2, ..., Emb...}. The goal of knowledge graph reasoning is to predict the correct tail entity based on the query. i ,...,Emb k The number and type of embedded models are arbitrary, where the answer length hyperparameter L = 10 and the confidence hyperparameter ρ = 0.5.

[0067] After correctly inputting the initial data, the training set D will be used first. train Each model in the Embedded Models group (Embs) is trained individually until convergence, and then the test set D is used. test Evaluate the predictive performance of each model in the Embedded Models group (Embs). After validation, a set of MRR metrics will be obtained: MRRs = {MRR1, MRR2, ..., MRR...} k}

[0068] The formula for calculating the model weights of each embedded model in step S2 is as follows:

[0069]

[0070] Where, α i Represents the weights of the i-th embedded model, MRR i Let represent the MRR metric of the i-th embedding model, and k' represent the total number of MRR elements in the set.

[0071] In Example 1, after obtaining the weight of each embedding model, the models in Embs are logically reordered in descending order of their weights, i.e., Emb1 represents the best-performing embedding model.

[0072] The specific steps of step S3 are as follows:

[0073] S31. Input the query q into a set of weighted embedding models, and predict the query q through each weighted embedding model to obtain a set of global targets Targets;

[0074] S32. Multiply each confidence score in the set of global targets Targets by the weight corresponding to each weighted embedding model, accumulate the confidence scores of the predicted entities with the same answer, and arrange all results in descending order of confidence score. Take the first L answers to form a global target candidate list T0 for query q.

[0075] S33. Use a confidence filter to filter out the poor-quality answers in the global target candidate list T0 to obtain the global target list T1;

[0076] S34. Based on the global target list T1, calculate the global target embedding representation e by weighted average. gt .

[0077] like Figure 2 As shown, in Example 1, when any query q = (e s ,r q When the query q arrives, it is first input into the Embedded Model Group (Embs), where each Embedded Model makes a prediction on the query q, thus obtaining a set of global targets Targets = {Target1, Target2, ..., Target...}. k For the i-th set, specifically Target i =[(e1,p1),(e2,p2),...,(e L ,p L ] represents a combination of predicted answer entity IDs e and confidence levels p, in the Target i The results will be sorted in descending order according to the confidence level p, and the number of results is controlled by the hyperparameter L.

[0078] Then, each confidence score in Targets is multiplied by the weight α corresponding to each model. i The contents of the collection will be processed as: Target i =[(e1,α i p1), (e2,α i p2), ..., (e L ,α i p L After that, the prediction results of all models are integrated, the confidence scores of the predicted entities with the same answer are accumulated, and then all results are arranged in descending order of confidence score. Finally, the top L answers are taken to form a global target candidate list T0 for query q.

[0079] Since the weights of each embedding model are calculated in advance, assuming that in an extreme case, for a certain query, all embedding models output a unique answer with a confidence of 1, then after integrating the answers, the corresponding confidence in T0 will still be 1.

[0080] Let T0 = [(e1,p1), (e2,p2), ..., (e L ,p L The answer entities in the list are already sorted in descending order of confidence. Based on the pre-input confidence hyperparameter, ρ = 0.5, the answer confidence acceptance threshold th = ρp1 is calculated. Then, all answers in T0 with confidence levels lower than the acceptance threshold th are deleted, resulting in the global target list T1. The confidence hyperparameter ρ takes values ​​of [0,1]. When ρ = 0, all answers are accepted; when ρ = 1, only the answer with the highest confidence level is accepted. After processing, T1 contains n answers, i.e., T1 = [(e1,p1), (e2,p2), ..., (e...]. n ,p n )).

[0081] In step S34, the global target embedding representation e is calculated. gt The formula is:

[0082]

[0083] Among them, e gt This represents the global target embedding representation, where N represents the number of targets in the global target list, and e i p represents the embedding representation of the answer entity in the i-th embedding model. i This represents the confidence level of the entity output by the i-th embedding model.

[0084] In Example 1, Emb1, the best-performing embedding model among a set of embedding models Embs, is selected as the base embedding for entity calculation. Based on the answer entity content in the global target list T1, the embedding representation corresponding to the entity number is found in the dictionary trained on embedding model Emb1. Then, a weighted average is performed based on the confidence score of each entity answer to obtain the final global target embedding representation e. gt .

[0085] Among them, coefficient The purpose is to scale the entire embedding back to the original numerical range, because after the above steps, the sum of the confidence scores of the answers in T1 is usually not equal to 1.

[0086] The specific steps of step S4 are as follows:

[0087] S41. Based on the global target embedding representation egt Define the state of the agent when it reaches the t-th time step;

[0088] S42. Define the action space based on the knowledge graph and perform agency based on the action space;

[0089] S43. Select the Long Short-Term Memory (LSTM) network and the fully connected network as the memory network and the policy network, respectively.

[0090] S44. At time step t, the agent inputs the state into the policy network and executes a certain action;

[0091] S45. After the agent performs an action, move the agent to the entity node connected to that action;

[0092] S46. Define the reward value for the agent to complete the construction of the learning state model.

[0093] The expression defining the state of the agent at the t-th time step in step S41 is as follows:

[0094] s t =(e t ,h t ,(e s ,r q ,e gt ))

[0095] Among them, s t Let e ​​represent the state of the reinforcement learning model at time step t. t h represents the agent's position on the knowledge graph at time step t. t e represents the historical decision information obtained by the agent through the Long Short-Term Neural Network at time step t. s Represents the head entity in the query, r q Indicates the relation in the query, e gt This represents the global target embedding representation.

[0096] like Figure 3 As shown, in Example 1, the known tuple (e) in the query s ,r q ), and the global target embedding representation e gt Together, they are stored in the state as global information. Since the proposed weighted embedding model has the same embedding dimension for entities and relations, let the dimension of entities and relations be d, therefore the dimension of the reinforcement learning state representation is 5d.

[0097] The expression for defining the action space in step S42 is as follows:

[0098] A t ={(r n ,en )|(e t ,r n ,e n )∈G}

[0099] Among them, A t Let r represent the action space of the model at time step t. n Indicates the next-hop neighbor relationship of the current entity, e n Indicates the next-hop neighbor entity of the current entity, e t G represents the entity location of the agent at time step t, and G represents the knowledge graph.

[0100] In Example 1, the actions of the reinforcement learning agent are contained in a preset action space. In the link prediction scenario, the action space is defined as the set of all outgoing edges and neighbors of an entity in the knowledge graph.

[0101] The expression for the agent's reward value defined in step S46 is:

[0102] If (e) s ,r q ,e gt )∈G,R b (s t ) = 1

[0103] Otherwise, R s (s t )=f(e s ,e t )

[0104] R(s t ) = R b (s t )+(1-R b (s t ))R s (s t )

[0105] Among them, s t and e t R represents the final state of the learned state model and the final predicted answer entity. b Represents a binary reward, R s e represents the similarity reward. s Represents the head entity in the query, r q Indicates the relation in the query, e gt Let f() represent the global target embedding representation, f() represent the calculation of cosine similarity, R represent the total reward calculated based on the binary reward and similarity reward, G represent the knowledge graph, and R b (s t ) indicates that the agent's final state is s tThe binary reward given at that time, R s (s t ) indicates that the agent's final state is s t The similarity reward given at the time, R(s) t ) indicates that the agent's final state is s t The total reward given at that time.

[0106] In Example 1, R b This represents a binary reward: 1 for correct surrogate reasoning and 0 for incorrect surrogate reasoning. `f()` calculates the cosine similarity; the higher the similarity, the higher the reward.

[0107] The specific steps of step S5 are as follows:

[0108] S51. Based on the action space, search for entity nodes in the knowledge graph G, and select the neighbor of the entity node to execute an action;

[0109] S52. Based on the actions performed by the neighbors, calculate the state of the agent at each time step, and input the state of each time step into the policy network to obtain the agent's action in the next hop and the entity node connected to the next hop action;

[0110] S53. Input the entity node of the next hop into the long short-term neural network to obtain the historical information of the agent's decision-making.

[0111] S54. Input the historical information of the agent's decision-making into the policy network, calculate the agent's reward value, and evaluate the agent's decision-making at each time step based on the agent's reward value.

[0112] S55. Based on the decisions made by the agent at each time step, train the reinforcement learning state model using beam search, and determine whether the number of iterations has reached the preset threshold. If so, complete the training of the reinforcement learning state model and proceed to step S56; otherwise, return to step S51.

[0113] S56. Perform reasoning completion work on the knowledge graph based on the completed reinforcement learning state model, and complete multi-hop reasoning of the knowledge graph based on reinforcement state modeling.

[0114] In Example 1, the dimensions of both entities and relations are set to 200, the Adam gradient descent algorithm with adaptive moment estimation is selected as the optimizer for model training, and its learning rate is set to 0.001.

[0115] During training, each training batch was set to 32 data points, based on the actual GPU memory size. For the policy network, a two-layer Long Short-Term Memory (LSTM) network with 400 hidden layers was used for historical path encoding, and a dropout rate of 0.1 was set in each dropout layer. Simultaneously, a beam search with a size of 512 was used to increase the agent's chances of finding the correct entity. Finally, the dataset was split using the default training, validation, and test sets for both datasets.

[0116] Figure 4 and Figure 5 The results of Example 1 on two general datasets, the NELL-995 dataset and the WN18RR dataset, are presented and compared with the data from various baseline models. The experimental results are presented in two tables, depending on the dataset. Figure 4 and Figure 5 The data provides the hit rate metrics Hits@1, Hits@3, Hits@10, and mean reciprocal ranking (MRR) for each model on the NELL-995 and WN18RR datasets, with the data unit being 0.01.

[0117] Experimental results show that the present invention performs well in the link prediction task. The model demonstrates excellent prediction performance on both the NELL-995 and WN18RR datasets and is highly competitive with state-of-the-art baseline models.

[0118] On the NELL-995 dataset, our proposed model achieved state-of-the-art results in both the Hits@1 hit rate metric and the mean reciprocal ranking (MRR) metric. On the WN18RR dataset, it achieved the best results across all four metrics. In both experiments, the Hits@1 metric, which reflects optimal prediction performance, and the MRR metric, which reflects average overall performance, both achieved state-of-the-art results, with improvements of 4.1% and 3.1% (NELL-995) and 1.6% and 1.4% (WN18RR) respectively compared to the baseline multi-hop inference model MultiHopKG. These experimental results indicate that combining the global target embedding representation computed with a pre-trained embedding model provides the agent with more evidence for pathfinding decisions, thereby improving the accuracy of the agent's reasoning behavior on the graph.

[0119] Example 2

[0120] In Example 2, the present invention will be further illustrated through a specific example of a social friend recommendation system.

[0121] For a social user network constructed from a knowledge graph, the information in the graph consists of users and the relationships between them. Using this social graph, social platforms can mine potential friend relationships between users and recommend friends to each other. While embedding-based methods can quickly find suitable recommendations on the social graph, the results lack interpretability, reducing the probability of successful recommendations. Reinforcement-based methods are interpretable, but the state modeling information representation of mainstream methods is insufficient, leading to poor recommendation quality. Therefore, this invention provides a multi-hop inference method with intermediate processes, which, while ensuring the interpretability of recommendations, utilizes a global goal-guided agent to more effectively mine hidden friend relationships in the social graph. The specific implementation method includes the following steps:

[0122] A1. Use a set of embedding models to pre-train on social knowledge graph data to obtain the embedding representations of user entities and friend relationships, and use the test set to evaluate the prediction performance of each embedding model to obtain a set of MRR metrics, MRRs.

[0123] A2. Based on the MRR index, calculate the model weights of each embedding model, and reorder the embedding models in descending order of model weights to obtain weighted embedding models.

[0124] A3. Based on the weighted embedding model, calculate the corresponding global friend target embedding representation for each potential friend query;

[0125] A4. Construct a reinforcement learning state model based on the global target embedding representation;

[0126] A5. Train the reinforcement learning state model using a social knowledge graph, and use the trained reinforcement learning state model to complete the inference of the social knowledge graph, thus completing multi-hop inference of the social knowledge graph based on reinforcement state modeling.

[0127] The formula for calculating the model weights of each embedded model in step A2 is as follows:

[0128]

[0129] Where, α i Represents the weights of the i-th embedded model, MRR i Let represent the MRR metric of the i-th embedding model, and k' represent the total number of MRR elements in the set.

[0130] The specific steps of step A3 are as follows:

[0131] A31. Input the potential friend query q into a set of weighted embedding models, and predict the potential friend query q through each weighted embedding model to obtain a set of global friend targets Targets;

[0132] A32. Multiply each confidence score in the set of global friend targets Targets by the weight corresponding to each weighted embedding model, accumulate the confidence scores of the predicted same user entities, and arrange all results in descending order of confidence score. Take the first L users to form a global friend target candidate list T0 for the potential friend query q.

[0133] A33. Use a confidence filter to filter out users with poor quality in the global friend target candidate list T0 to obtain the global friend target list T1.

[0134] A34. Based on the global friend target list T1, calculate the global friend target embedding representation e using a weighted average. gt .

[0135] In step A34, the global friend target embedding representation e is calculated. gt The formula is:

[0136]

[0137] Among them, e gt This represents the global friend target embedding representation, where N represents the number of users in the global friend target list, and e i p represents the embedding representation of the user entity in the i-th embedding model. i This represents the confidence level of the user entity output by the i-th embedding model.

[0138] The specific steps of step A4 are as follows:

[0139] A41. Based on the global friend target embedding representation e gt Define the state of the agent when it reaches the t-th time step;

[0140] A42. Define the action space based on the social knowledge graph, and make action decisions based on the action space;

[0141] A43. Select the Long Short-Term Memory (LSTM) network and the fully connected network as the memory network and the policy network, respectively.

[0142] A44. At time step t, the agent inputs the state into the policy network and executes a certain action;

[0143] A45. After the agent performs an action, move the agent to the entity node connected to that action;

[0144] A46. Define the reward value for the agent to complete the construction of the learning state model.

[0145] The expression defining the state of the agent at the t-th time step in step A41 is as follows:

[0146] s t =(e t ,h t ,(e s ,r q ,e gt ))

[0147] Among them, s t Let e ​​represent the state of the reinforcement learning model at time step t. t h represents the position of the agent on the social knowledge graph at time step t. t e represents the historical decision information obtained by the agent through the Long Short-Term Neural Network at time step t. s Represents the user header entity in the query, r q Indicates the friend relationships in the query, e gt This represents the embedding of global friend targets.

[0148] The expression for defining the action space in step A42 is as follows:

[0149] A t ={(r n ,e n )|(e t ,r n ,e n )∈G}

[0150] Among them, A t Let r represent the action space of the model at time step t. n Indicates the next-hop neighbor friend relationship of the current user entity, e n Indicates the next-hop neighboring user entity of the current entity, e t G represents the location of the user entity at time step t, and G represents the social knowledge graph.

[0151] The expression for the agent's reward value defined in step A46 is:

[0152] If (e) s ,r q ,e gt )∈G,R b (s t ) = 1

[0153] Otherwise, R s (s t )=f(e s ,et )

[0154] R(s t ) = R b (s t )+(1-R b (s t ))R s (s t )

[0155] Among them, s t and e t R represents the final state of the learned state model and the final predicted user entity. b Represents a binary reward, R s e represents the similarity reward. s Represents the user header entity in the query, r q Indicates the friend relationships in the query, e gt Let f() represent the global friend target embedding representation, f() represent the calculation of cosine similarity, R represent the total reward calculated based on the binary reward and similarity reward, and G represent the social knowledge graph. b (s t ) indicates that the agent's final state is s t The binary reward given at that time, R s (s t ) indicates that the agent's final state is s t The similarity reward given at the time, R(s) t ) indicates that the agent's final state is s t The total reward given at that time.

[0156] The specific steps of step A5 are as follows:

[0157] A51. Based on the action space, search for user entity nodes in the social knowledge graph G, and select neighboring users of the user entity node to perform an action;

[0158] A52. Based on the actions executed, calculate the agent's state at each time step, and input the state at each time step into the policy network to obtain the agent's action in the next hop and the user entity node connected to the next hop action;

[0159] A53. Input the user entity node of the next hop into the long short-term neural network to obtain the historical information of the agent's decision-making;

[0160] A54. Input the historical information of the agent's decision-making into the policy network, calculate the agent's reward value, and evaluate the agent's decision-making at each time step based on the agent's reward value.

[0161] A55. Based on the decisions made by the agent at each time step, train the reinforcement learning state model using beam search, and determine whether the number of iterations has reached the preset threshold. If so, complete the training of the reinforcement learning state model and proceed to step A56; otherwise, return to step A51.

[0162] A56. Based on the trained reinforcement learning state model, perform reasoning and completion work on potential friend relationships in the social knowledge graph, and complete multi-hop reasoning of the social knowledge graph based on reinforcement state modeling.

[0163] In this second embodiment, firstly, the triples of the social graph are randomly divided into training and testing sets. The bilinear model DistMult, the complex model ComplEx, and the convolutional embedding model ConvE are selected as the pre-trained model set Embs to pre-train the user entities and social relationships of the social graph, obtaining their embedding representations. Simultaneously, the friend relationship prediction performance of the three embedding models in the testing set is used for verification, resulting in a set of MRR metrics, MRRs.

[0164] Based on the MRR index obtained from the test, the model weights of the three embedding models are calculated, and the embedding models are reordered in descending order of weight to obtain the weighted embedding models.

[0165] During training, a query q is constructed by combining a "user" entity and a "friend" relationship. This query is then input into the three embedding models mentioned above, resulting in a set of global friend targets, Targets. The confidence scores of each friend relationship are multiplied by the weight corresponding to each model. The confidence scores of predicted user entities are accumulated and sorted in descending order. The top L user entities are selected to form a candidate list T0 of global friend targets for query q. After filtering out low-quality answers from the candidate list T0 using a confidence filter, the global friend target embedding representation e is calculated using a weighted average method. gt .

[0166] Based on the friend's global target embedding representation e gt A reinforcement learning state model is constructed to define the state of the social reinforcement agent at time step t. Specifically, the agent's action space is defined as the set of all outgoing edges and neighboring users of a user entity in the social graph. A Long Short-Term Memory (LSTM) network and a fully connected network are selected as the memory network and policy network, respectively. The reward value is then calculated to evaluate the decisions made by the social agent at each time step, thus completing the construction of the learning state model. Finally, the model is trained again using the training set of the social graph, and its predictive performance on friend relationships is verified using the test set.

[0167] After the model is trained, the trained reinforcement learning state model is used for knowledge graph reasoning and completion. Using a similar construction method as the training model, a binary query of "user" entity and "friend" relationship is constructed to predict a user's potential friend relationships. During prediction, beam search is used to obtain a series of potential friends for a given user. Thus, the social platform can select the results with the highest confidence for user recommendations, while simultaneously outputting the intermediate process of social agent inference, thereby providing a basis for friend recommendations and increasing the probability of successful recommendations.

Claims

1. A method of knowledge graph multi-hop reasoning based on reinforcement state modeling, used for social friend recommendation, characterized in that, The method comprises the following steps: S1, pre-training the social knowledge graph data using a set of embedding models to obtain embedding representations of user entities and friend relationships, and evaluating the prediction performance of each embedding model using a test set to obtain a set of indicators ; S2、according to The index is calculated, the model weights of each embedding model are calculated, and each embedding model is reordered in descending order of the model weights of the embedding models to obtain weighted embedding models. S3, calculating a corresponding global friend target embedding representation for each potential friend query according to the weighted embedding model; S4, constructing a reinforcement learning state model according to the global friend target embedding representation; S5, training the reinforcement learning state model using the social knowledge graph, and performing inference and completion of the social knowledge graph using the trained reinforcement learning state model to complete the knowledge graph multi-hop inference based on the reinforcement state modeling. The step S3 comprises the following steps: S31, query for potential friends input into a set of weighted embedding models and through each weighted embedding model respectively to predict a set of global friend targets ;​ S32, multiply the confidence of each of the set of global friend targets in the set by the weight corresponding to each weighted embedding model respectively, accumulate the confidence of predicting the same user entity, and arrange all results in descending order of confidence, and take the first users to constitute the global friend target candidate list for the potential friend query ;​ S33, filter out the global friend target candidate list by using the confidence filter sub-optimal users, get the global friend target list ; S34, based on a global friend target list , computing a global friend target embedding representation by weighted average ; The step S4 comprises the following steps: S41, embedding representation according to global friend target , define the state of the agent at the first time step. S42, defining an action space according to the social knowledge graph, and performing an agent based on the action space; S43, selecting a long short-term memory network (LSTM) and a fully connected network as a memory network and a policy network, respectively; S44, at the first time step, the agent inputs the state to the policy network and performs an action; S45, moving the agent to an entity node connected with the action after the agent performs the action; S46, defining a reward value of the agent to complete construction of the learning state model. 2.The knowledge graph multi-hop reasoning method based on reinforcement state modeling according to claim 1, wherein, The formula for calculating the model weight of each embedding model in the step S2 is: wherein, denotes the weight of the th embedding model, denotes the weight of the th embedding model, denotes the metric of the th embedding model, denotes the total number of elements within the set. 3.The knowledge graph multi-hop reasoning method based on reinforcement state modeling according to claim 1, wherein, The step S34 calculates the global friend target embedding representation The formula is: wherein, represents a global friend target embedding representation, represents a number of targets in a global friend target list, represents an embedding representation of a user entity of an embedding model, represents a confidence corresponding to the user entity output by the embedding model. 4.The knowledge graph multi-hop reasoning method based on reinforcement state modeling according to claim 1, wherein, The expression of the state of the agent defined in step S41 at the first time step is: wherein, represents a state of the reinforcement learning model at the th time step, represents a position of the agent on the social knowledge graph at the th time step, represents historical decision information obtained by the agent via the long short-term neural network at the th time step, represents a user head entity in the query, represents a friend relationship in the query, represents a global friend target embedding representation.

5. The knowledge graph multi-hop reasoning method based on reinforcement state modeling according to claim 4, characterized in that, The expression for defining the action space in the step S42 is: in, Indicates the first The action space of the time step model This indicates the next-hop neighbor / friend relationship of the current user entity. This represents the next-hop neighboring user entity of the current user entity. Indicates at time step The entity location where the agent is located. Represents a social knowledge graph. 6.The knowledge graph multi-hop reasoning method based on reinforcement state modeling according to claim 5, wherein, The expression for defining the reward value of the agent in the step S46 is: in, and This represents the final state of the learned state model and the final predicted user entity. This indicates a binary reward. Indicates a similarity reward. This represents the user header entity in the query. This indicates the friend relationships found in the query. This represents the embedding of global friend targets. This indicates the calculation of cosine similarity. This represents the total reward calculated based on the binary reward and the similarity reward. Represents a social knowledge graph. Indicates that the agent's final state is The binary reward given at that time Indicates that the agent's final state is The similarity reward given at that time Indicates that the agent's final state is The total reward given at that time.

7. The knowledge graph multi-hop reasoning method based on reinforcement state modeling according to claim 1, characterized in that, The step S5 comprises the following steps: S51, based on the action space, searching the user entity node in the social knowledge graph and selecting a neighbor user of the user entity node to perform an action; S52, calculating a state of the agent at each time step according to an action performed by a neighbor user, and inputting the state at each time step into the policy network to obtain an action of the agent at a next hop and a user entity node connected with the action of the next hop; S53, inputting the user entity node of the next hop into a long short-term neural network to obtain historical information of the decision of the agent; S54, inputting the historical information of the decision of the agent into the policy network to calculate a reward value of the agent, and evaluating the decision made by the agent at each time step according to the reward value of the agent; S55, training the reinforcement learning state model using bundle search according to the decision made by the agent at each time step, and determining whether the number of iterations reaches a preset threshold value, if yes, completing the training of the reinforcement learning state model, and entering the step S56, otherwise, returning to the step S51; S56, performing inference and completion of the social knowledge graph using the trained reinforcement learning state model to complete the knowledge graph multi-hop inference based on the reinforcement state modeling.

Images