An Interactive Recommendation Method Based on Knowledge Graph and Reinforcement Learning

Abstract

The present invention provides an interactive recommendation method based on a knowledge graph and reinforcement learning, which relates to the technical field of interactive recommendation. From the perspective of modeling the entire interactive recommendation process, combining graph attention mechanism, semantic similarity, and structural similarity, and adding a curiosity mechanism, it effectively models the recommendation process and improves the recommendation performance index, constructing a model with stable training and small variance. It overcomes the problems in existing models such as less item-related information, lack of reference to neighbor information, and low algorithm efficiency caused by the huge candidate set in the training process. At the same time, a new reward setting mechanism is proposed. It adopts a dual-network structure for training, adds an optimization design to prevent overfitting, and sets up a reward simulator introducing the curiosity mechanism, giving reward criteria from three dimensions: scoring, node distance in the knowledge graph, and the curiosity mechanism, and adding a mechanism to reduce policy variance, making the model training stable and the efficiency significantly improved.

Description

Technical Field

[0001] The present invention relates to an interactive recommendation modeling method, and more particularly to an interactive recommendation method based on a knowledge graph and reinforcement learning. Background Art

[0002] Interactive recommendation is to model the entire interaction process between users and the recommendation system, help the recommendation system learn the recommendation strategy, and use the learned flexible recommendation strategy to derive the next recommendation decision. As a new type of recommendation method, interactive recommendation provides a solution to the problem of how to obtain the optimal long-term user experience.

[0003] Currently, most recommendation algorithms are regarded as one-step prediction tasks, which screen out personalized relevant content from a large amount of information and recommend it to users. The overall algorithm process can be regarded as a static process, that is, there is no interaction between the recommendation algorithm and the user. Therefore, the recommendation algorithm cannot be improved using the continuous immediate feedback of the user. This type of method focuses on modeling the consistent preferences of users and thus cannot reflect the high dynamics of user preferences. Therefore, some scholars have begun to study interactive recommendation systems. The core idea is to formulate the recommendation problem as a multi-step decision-making process. In each step, the recommendation system selects an item through a recommendation strategy and recommends it to the user. Then, the recommendation system receives feedback from the user, and this feedback subsequently derives the next recommendation decision in a continuous manner. The above process is repeated until the user ends the system access. One type of existing method for simulating interactive recommendation is based on multi-armed bandits (MAB), but this type of method is not explicitly designed for the long-term planning of the system, which makes the performance of this type of method unsatisfactory. Another type of method currently used to solve the interactive recommendation problem is based on deep reinforcement learning (DRL), which makes decisions in a dynamic environment and has great potential in long-term planning. However, the temporal difference-based method in deep reinforcement learning uses bootstrapping in the estimation calculation, which will cause instability and divergence problems in policy learning. At the same time, the action space and state space in each step are huge, directly increasing the time complexity of learning the recommendation strategy and resulting in low algorithm efficiency. To solve the above problems, researchers have begun to explore using rich auxiliary information and combining it with deep reinforcement learning for interactive recommendation modeling and design. This rich auxiliary information can be a knowledge graph, text, sequence information, etc.

[0004] A knowledge graph is a large heterogeneous network composed of entities and relationships. Different types of entities can be regarded as nodes, and different types of relationships can be regarded as edges. Introducing a knowledge graph into recommendation can not only generate node embeddings that combine the structural information and semantic information of the knowledge graph, but also use the knowledge graph to narrow the candidate action space. Currently, a few researchers have conducted research in this area: The KGQR model first proposed by Zhou et al. is a recommendation framework that combines a knowledge graph and extends DQN; the KGRL model subsequently proposed by Chen et al. is an Actor-Critic framework that combines a knowledge graph and is trained using the DDPG algorithm, but there is a problem of inconsistency between the continuous action representation and the true discrete action representation. Some researchers have also introduced text or item sets in recommendation. For example, in order to avoid the problem of inconsistency between continuous action representation and true discrete actions, Chen et al. proposed the TPGR model, which is a tree-structured policy gradient recommendation framework that builds a balanced hierarchical clustering tree based on the item set, but this method has the burden of constructing a clustering tree when new items frequently appear; to improve efficiency, Dulac et al. proposed to first learn action representations in a continuous hidden space and then use the k-nearest neighbor search method to find valid items.

[0005] The above methods generally have the following problems: 1. The auxiliary information introduced generally has a high reconstruction difficulty when adding new items, such as reconstructing a tree structure; 2. When the model is modeled in a discrete manner, there is a large computational amount due to the huge action and state spaces; 3. When the model is modeled in a continuous manner, there are problems of inconsistency between continuous actions and true discrete action vectors and large matching difficulties. 4. Due to the overestimation problem caused by bootstrapping and the optimal Bellman equation in the reinforcement learning model, various optimization methods have their own problems, increasing the model complexity and training difficulty. Summary of the Invention

[0006] Aiming at the problems existing in the prior art, the present invention provides an interactive recommendation method based on a knowledge graph and reinforcement learning, aiming to effectively model the interactive recommendation process and construct a model with stable training and small variance. By using the knowledge graph as auxiliary information, it overcomes the problems of less item-related information, lack of reference to neighbor information, and huge candidate sets in the training process in the existing model, resulting in low algorithm efficiency.

[0007] The technical solution of the present invention is as follows:

[0008] An interactive recommendation method based on a knowledge graph and reinforcement learning, comprising the following steps:

[0009] Step 1: Corresponding the items in the recommendation dataset with the entities in the knowledge base Freebase to construct a specific domain knowledge graph, and according to the constructed knowledge graph, pre-embedding the knowledge graph to obtain low-dimensional representations of the entities and relationships in the knowledge graph.

[0010] The mapping formula after mapping the entity to the hyperplane is as follows:

[0011]

[0012]

[0013] Among them, h represents the representation of the head entity node in the knowledge graph, and h ⊥ represents the representation of the head entity node after mapping; tail represents the representation of the tail entity node, and tail ⊥ represents the representation of the tail entity node after mapping; each relationship r has a matrix parameter W related to the hyperplane representation r .

[0014] Step 2: Calculate the attention coefficient between the entity node in the knowledge graph and its first-order neighbor entity nodes, and use the calculated attention coefficient and the low-dimensional representation of the entity obtained in Step 1 to obtain the low-dimensional representation of the central entity that propagates the first-order neighbor entity nodes in a weighted summation calculation manner.

[0015] The formula for calculating the attention coefficient is as follows:

[0016] g nj = LeakyReLu(Wi n ||Wi j ) (3)

[0017] Among them, i n and i j are the vector representations of node n and node j; g nj is the attention coefficient of node j to node n; LeakyReLU is a non-linear activation function; W is a weight matrix; || is the concatenation of two vectors; the softmax function is used to normalize the attention coefficients of all first-order neighbor entity nodes of node n, and the formula is as follows:

[0018]

[0019] Among them, α nj is the normalized attention coefficient of node j to the central node n; N is the set of all first-order neighbor entity nodes of node n; i k is the vector representation of node k.

[0020] The formula for obtaining the low-dimensional representation i' n of the central entity that propagates the first-order neighbor entity nodes in a weighted summation manner is as follows:

[0021]

[0022] Among them, σ is a non-linear function, which is a single-layer forward neural network.

[0023] Step 3: Integrate the low-dimensional representation of the entity obtained in Step 1 with the low-dimensional representation of the central entity that has propagated the first-order neighbor entity nodes obtained in Step 2. The entity node representation obtained from the integration operation will be used as the next-round low-dimensional representation of the entity. Repeat Step 2 and this step until the entity node embedding representation converges;

[0024] The integration formula is as follows:

[0025] e n = f Bi-Interaction (i n , i′ n ) = LeakyReLU(W(i n + i′ n )) + LeakyReLU(W(i n ⊙ i′ n )) (6)

[0026] Among them, f Bi-Interaction is the integration function using the Bi-Interaction integrator. The integrator can also consider using the GCN integrator or the GraphSage integrator; ⊙ is the Hadamard product; W is the weight matrix; LeakyReLU is the activation function; i n is the vector representation of node n, and i′ n is the output result of formula (5), that is, the low-dimensional representation i′ n of the central entity that has propagated the first-order neighbor entity nodes.

[0027] Define the aggregation functions of the upper layer and the lower layer. The specific calculation method is as follows:

[0028]

[0029] Among them, l is the l-th round of integration, and l - 1 is the (l - 1)-th round, that is, the previous round of the l-th round; refers to the representation of node n in the (l - 1)-th round, refers to the information sum of aggregating the first-order neighbor nodes centered on node n in the (l - 1)-th round; The Bi-Interaction integrator is used to obtain is the representation of node n in the l-th round that integrates the entity node information in the (l - 1)-th round and aggregates the first-order neighbor entity node information.

[0030] Step 4: Extract the item representations that the user has had positive interactions with from the entity node representations obtained in Step 3, and input them into the GRU model to obtain the user's immediate preference vector, that is, the state.

[0031] Step 4.1: Define the modeling process for interactive recommendation;

[0032] Given a set of users U = {u1, u2, u3,...}, a set of items I = {i1, i2, i3,...} and the knowledge graph constructed in Step 1 First, initialize the random policy π. When the user (i.e., the environment) and the recommendation policy (i.e., the agent) interact for the t-th time, under the user's specific preference S t the agent recommends an action A i to the user (u t ∈U). This action is selected based on the random policy π, and then the user's feedback (f i ∈F), including click, favorite, ignore, rating, is obtained. The feedback obtained can be used to improve the random policy π, and the user preference state also changes from S t to S t+1 .

[0033] Step 4.2: Construct an interactive recommendation agent and define the MDP tuple;

[0034] Given the observations of past interactions {u, i1, f1,..., i t-1 , f t-1}, the recommendation policy is modeled as a conditional distribution, which is used to represent the probability of recommending item i t at the t-th interaction. A series of interactions between the user and the recommendation policy will generate a trajectory ξ = {S0, i0, r0,..., S t , i t , r t ,..., S T}. Obtain the recommendation policy π * that maximizes the long-term cumulative reward for recommending items to the user, defined as follows:

[0035]

[0036] where, is the expected discounted reward under policy π; the state S t is determined by the items that the user was interested in (i.e., had positive interactions) before time t. The action A t is to recommend an item or a list of items; the reward r(S t , A t ) is the reward given by the reward simulator based on the user's immediate feedback on the recommended item. The discount factor γ ∈ [0, 1]. When γ is 0, it will focus entirely on the immediate return. At t = 1, the immediate reward is 0.9 times r(S t , A t ), and at t = 2, the immediate reward is 0.81 times r(St , A t ), and so on; T refers to the length of a trajectory; Σ refers to the summation operation; π refers to the recommendation strategy, and π * refers to the optimal recommendation strategy; Π refers to the set of all strategies.

[0037] Step 4.3: Determine the user's immediate preference vector for each step and calculate the representation after item integrated feedback;

[0038] First, convert the click item sequence {i t} into an embedding vector e t ∈R d through Steps 2 and 3, where d is the dimension of the vector. The representation after item integrated feedback is calculated as:

[0039] I n = concat(e n , tanh(W F f n + b F )) (9)

[0040] where e n is the embedding representation of entity node n obtained in the knowledge graph attention module; f n is a one-hot vector corresponding to a feedback type; tanh is used as the activation function; W F is the parameter matrix; b F is the bias vector parameter; concat is the concatenation function; I n is the representation after item integrated feedback, that is, the output of this step.

[0041] Step 4.4: Use the GRU model to track the state;

[0042] The output of the previous step is the new item representation obtained after item integrated feedback for each item interacted by the user. Input it into the GRU model to calculate the user's state. The GRU model is a sequence model composed of several GRU units. The function of one GRU unit is defined as:

[0043]

[0044]

[0045]

[0046]

[0047] where I t is the input vector, that is, the output of formula (9); z t is the update gate; rt is the reset gate; ° is the element-wise product; W z , W r , W h are the matrix parameters of the computation update gate, reset gate, and hidden state respectively, U z , U r , U h are the matrix parameters of the computation update gate, reset gate, and hidden state respectively, b z , b r , b h are the bias parameters of the computation update gate, reset gate, and hidden state respectively; Equation (13) is the update function of the hidden state h t The specific calculation is the linear interpolation of the previous hidden state and the new candidate hidden state ; σ g , σ h are the sigmoid activation function and the tanh activation function respectively.

[0048] Step 5: Centered on the items with which the user has had positive interactions, select the k-hop neighbor entity nodes of the knowledge graph as the candidate space set, that is, the action space.

[0049] The selection method is to sample all items within the k-hop neighborhood of the seed set, where the definition of the seed set is:

[0050]

[0051] where the subscript t represents the t-th step, and the superscript 0 represents the initial set, that is, the seed set. Thus, the definition of the k-hop neighborhood set is as follows:

[0052]

[0053] where the superscript number k refers to the items within the k-hop neighborhood; is the knowledge graph, (head, relation, tail) refers to the triple in the knowledge graph, head refers to the head entity node in the triple, tail refers to the tail entity node in the triple, and relation refers to the relationship connecting the head entity node and the tail entity node.

[0054] Step 6: Input the state obtained in Step 4 and the action space obtained in Step 5 into the policy network and obtain an action vector.

[0055] The policy network is composed of a multi-layer perceptron, and its output represents the high-dimensional vector of the action to be recommended.

[0056] Step 7: Input the action vector output in Step 6 into the reward simulator to obtain a reward. If the reward is greater than the preset threshold, add this action vector to the items that the user has interacted with, and reuse the calculation process in Step 4 to obtain the next state; if the reward is less than or equal to the preset threshold, the next state remains the same as the previous state.

[0057] The reward simulator receives the current state S t and the action A t , and returns the reward r t and the next state S t+1 ; where the reward function guides the model to capture the user's preferences and evaluate the quality of the recommended items. An offline log data is used to build the reward simulator to simulate the user feedback. A curiosity mechanism is introduced, and a positive reward is given to the agent when it explores unknown items; at the same time, a balance parameter is set to balance the proportion of the curiosity reward and other rewards.

[0058] The reward setting calculation is as shown in the following formula:

[0059]

[0060]

[0061] where, R t is the reward for the recommended action A t at time step t; rating t refers to the score in the offline data, and r distance refers to the distance reward; represents the curiosity reward output by the curiosity network at time step t; γ is the reward discount factor; V(s t ) is the value of the state s t ; τ is the balance factor between the curiosity reward and the distance reward, Distance refers to the distance function of the recommended item from the positive interaction item, and the Dijkstra algorithm is used to calculate the distance; ∈ approaches 0 infinitely, and W nt is the sum of the weights of the shortest paths from i n to i t .

[0062] Step 8: Store the state obtained in Step 4, the action vector obtained in Step 6, the reward obtained in Step 7, and the next state obtained in Step 4 in the experience pool. Build a recommendation agent model based on the PPO2 algorithm in reinforcement learning. The agent obtains training data in a trial-and-error manner and obtains the optimal behavior strategy, and adjusts the recommendation strategy in an interactive form using the user's immediate feedback at present.

[0063] In the recommendation agent model, the current action is A t, the value of the current state is V(S t ), and the agent takes action A t After that, the environment returns a reward r, and the curiosity neural network returns a reward which is stored in the experience pool as reusable experience. When an episode ends, the data in the experience pool is used to update the policy π. Data is sampled from it in a random sampling manner, and after repeating the update 3 times, the experience pool is emptied and the data update for the next episode starts until the policy converges;

[0064] The specific calculation method of the loss function value of the PPO2 algorithm with curiosity reward is as follows:

[0065] Adv θ′ (S t ,A t ) = R t +(γλ)R t+1 +...+(γλ) T-t+1 R T-1 (18)

[0066]

[0067] where R t is the temporal difference error at time step t; Adv θ′ (A t |S t ) is the action advantage estimation function of state s t under the policy parameter θ′; ∈ is the truncation threshold; L(θ) represents the policy gradient loss function of the PPO2 algorithm; the function clip() limits the parameter value in the function to the range [1 - ∈, 1 + ∈]; p θ (A t |S t ) is the probability distribution of action A t under policy θ and state S t ; γ is the reward discount factor, τ is the balance factor between the curiosity reward and the distance reward, ∈ approaches 0 infinitely, and T refers to the length of the trajectory.

[0068] Compared with the prior art, the present invention has the following beneficial effects:

[0069] (1) By introducing a knowledge graph, the present invention overcomes the problem of less item-related information in existing models, and uses the knowledge graph to solve the problem of large computational complexity caused by the huge action state space in interactive recommendation, reducing the action space candidate set and improving training stability.

[0070] (2) In the graph embedding part of the present invention, the structural information and semantic information between nodes in the knowledge graph are fully considered. The attention mechanism is used to assign different weights to each node, focusing on those nodes with greater effects and ignoring those with smaller effects. While processing local information, the overall information is also taken into account. The representation obtained in the graph embedding part can help to characterize the user's dynamic preference vector, thereby improving the simulation accuracy.

[0071] (3) The present invention adopts a dual-network structure for training, incorporates an optimization design to prevent overfitting, and sets up a reward simulator that introduces a curiosity mechanism. The reward criteria are given from three dimensions: scoring, node distance in the knowledge graph, and the curiosity mechanism. A mechanism for reducing policy variance is also added, making the model training stable and significantly improving the efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 is a flowchart of the interactive recommendation method based on knowledge graph and reinforcement learning of the present invention;

[0073] Figure 2 is the overall model diagram of the present invention;

[0074] Figure 3 Partial knowledge graph displayed using the Neo4j tool;

[0075] Figure 4 is the computational schematic diagram of mapping entities to a hyperplane using the TransH method in the present invention;

[0076] Figure 5 is a schematic diagram of selecting a candidate action set in the present invention;

[0077] Figure 6 is a schematic diagram of the graph attention network model structure in the graph embedding part of the present invention;

[0078] Figure 7 is a modeling scenario diagram of the dynamic interaction between users and the recommendation system in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0079] To facilitate the understanding of the present invention, the present invention will be described in more detail below with reference to the relevant drawings. Embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in various different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0080] An interactive recommendation method based on a knowledge graph and reinforcement learning, Figure 1 is a flowchart of the interactive recommendation method based on knowledge graph and reinforcement learning of the present invention. Figure 2It is the overall model diagram of the embodiment of the present invention. The following combines Figure 1 and Figure 2 to describe this method in detail. This method includes the following steps:

[0081] Step 1: Correspond the items in the recommendation dataset with the entities in the knowledge base Freebase to construct a specific domain knowledge graph. According to the constructed knowledge graph, pre-embed the knowledge graph to obtain the low-dimensional representations of the entities and relationships in the knowledge graph.

[0082] In the implementation of this step, the data to be read is divided into two categories, namely the recommendation system dataset and the knowledge base data. The recommendation datasets include MovieLens, LFM-1b, and Amazon book; the knowledge base data is Freebase. Connect the datasets from the recommendation system with the datasets from the knowledge base to form a triple set. This triple type includes (entity, relationship, entity) or (entity, attribute, value). The way to connect the datasets is: connect according to the item id in the recommendation dataset and the entity id in the knowledge graph data, similar to the inner join of a database. The obtained knowledge graph can provide rich and structured knowledge base information for the recommendation system. Such as Figure 3 is a partial knowledge graph shown using Neo4j.

[0083] After obtaining the constructed knowledge graph, use the existing knowledge graph embedding method TransH to learn the implicit representation of entity relationships. Figure 4 is the schematic diagram of the entity mapping of TransH to the hyperplane. The specific mapping formula after mapping the entity to the hyperplane is:

[0084]

[0085]

[0086] Among them, h represents the representation of the head entity node in the knowledge graph, h ⊥ represents the representation of the head entity node after mapping, tail represents the tail entity node representation, and tail ⊥ represents the representation of the tail entity node after mapping. Each relationship r has a matrix parameter W r .

[0087] Step 2: Calculate the attention coefficients between the entity nodes of the knowledge graph obtained in Step 1 and their first-order neighbor entity nodes. Use the calculated attention coefficients and the entity low-dimensional representations obtained in Step 1, that is, the entity low-dimensional representations of the previous round, to obtain the central entity low-dimensional representations that have propagated the first-order neighbor entity nodes in a weighted summation calculation method.

[0088] Specifically, the specific idea for calculating the attention coefficient of entity pairs is as follows: Different weights are assigned to each node, with greater attention paid to nodes with greater influence while ignoring some nodes with less influence. When processing local information, overall information can also be taken into account. The idea is to calculate the vector representation of each entity node in the graph by following the self-attention strategy and paying attention to its neighboring entity nodes. Based on the spatial similarity assumption, a sample has a relatively close relationship with samples within a certain range, and there are many samples, resulting in a relatively large computational amount. Therefore, for a sample, only samples within the neighborhood are used to calculate the attention coefficient and the new representation, that is, the attention is only assigned to the set of first-order neighbor entity nodes of the node. Without the need for complex matrix operations or prior knowledge of the graph structure, by stacking self-attention layers, different importance is assigned to different entity nodes within the neighborhood during the convolution process, while handling neighborhoods of different sizes. The formula for calculating the attention coefficient is:

[0089] g nj = LeakyReLU(Wi n ||Wi j ) (3)

[0090] where, i n and i j refer to the vector representations of node n and node j respectively, g nj refers to the attention coefficient of node j for node n. A single-layer feedforward neural network can apply LeakyReLU as the non-linear activation function. W refers to the weight matrix, and || refers to the concatenation of two vectors. To make the coefficients between different nodes easy to compare, the softmax function is used to normalize the attention coefficients of all first-order neighbor entity nodes of node n, and the formula is as follows:

[0091]

[0092] where, α nj refers to the normalized attention coefficient of node j for node n, and N refers to the set of all first-order neighbor entity nodes of node n. Figure 5 represents the neighborhood of the first-order neighbor entity nodes of the selected computational entity node.

[0093] Figure 6 is a schematic diagram of the graph attention network model structure. The new representation i′ n of the first-order neighbor entity nodes is obtained in the way of weighted summation, and the formula is as follows:

[0094]

[0095] where, σ is a non-linear function. As a single-layer feedforward neural network, the LeakyReLU function can be used.

[0096] Step 3: Integrate the low-dimensional representation of the entity obtained in Step 1 with the low-dimensional representation of the central entity that has propagated the first-order neighbor entity nodes obtained in Step 2. The entity node representation obtained from the integration operation will be used as the next-round low-dimensional representation of the entity. Repeat the operations of Step 2 and this step until the entity node embedding representation converges;

[0097] Use a Bi-Interaction aggregator to aggregate the entity node representation and the entity node representation based on the first-order entity node neighborhood. The calculation method is as follows:

[0098] e n = f Bi-Interaction (i n , i′ n ) = LeakyReLU(W(i n + i′ n )) + LeakyReLU(W(i n ⊙ i′ n )) (6)

[0099] where f Bi-Interaction is the integration function using the Bi-Interaction aggregator. The aggregator can also consider using a GCN aggregator or a GraphSage aggregator; ⊙ is the Hadamard product; W is the weight matrix; LeakyReLU is the activation function; i n is the vector representation of node n, and i′ n is the output result of formula (5), that is, the low-dimensional representation i′ n of the central entity that has propagated the first-order neighbor entity nodes. The advantage of using the activation function LeakyReLU is that: in the backpropagation process, for the part where the input of the LeakyReLU activation function is less than zero, the gradient can also be calculated (instead of being 0 like ReLU), thus avoiding the problem of gradient direction sawtooth.

[0100] The above description specifically refers to the integration operation process for propagating information in each layer, which propagates the information of the first-order neighbor entity nodes of the central entity node. However, to perform the high-order propagation of the central entity node, it is necessary to define the aggregation function between the previous layer and the next layer. The specific calculation method is as follows:

[0101]

[0102] where l is the l-th round of integration, l - 1 is the (l - 1)-th round, that is, the previous round of the l-th round; refers to the representation of node n in the (l - 1)-th round, refers to the sum of the information of the aggregated first-order neighbor nodes centered on node n in the (l - 1)-th round, obtained using the Bi-Interaction aggregator is the representation of node n in the l-th round that integrates the entity node information in the (l - 1)-th round and aggregates the first-order neighbor entity node information.

[0103] Step 4: Extract the representations of the items with which the user has had positive interactions from the entity node representations obtained in Step 3, and input them into the GRU model to obtain the user's immediate preference vector, i.e., the state.

[0104] To understand the training process after Step 4 of the present invention, it is necessary to introduce the entire interaction process between the recommendation system and the user before specifically introducing Step 4. Generally, the entire interaction scenario is as Figure 7 shown. Assume a typical interactive recommendation setting between the user and the recommendation strategy. Given a set of users U = {u1, u2, u3,...}, a set of items I = {i1, i2, i3,...}, and the knowledge graph constructed in Step 1 At the t-th interaction between the user and the recommendation strategy, under the user's specific preference (which can be represented by StateS t ), the recommendation agent recommends an action (ActionA i ) to the user (u t ∈ U). This action is selected based on the policy π, and then the user's feedback (Rewardf i ∈ F), such as click, favorite, ignore, rating and other behaviors, is obtained. The recommendation strategy then uses the obtained feedback to improve the policy π, and the user preference state also changes from S t to S t+1 according to the state transition function. Construct an interactive recommendation agent and define the MDP tuple. The specific method is as follows:

[0105] Given the observations of past interactions {u, i1, f1,..., i t-1 , f t-1}, the recommendation system is modeled as a conditional distribution, which is used to represent the probability of recommending item i t at the t-th interaction. A series of interactions between the user and the recommendation system will generate a trajectory ξ = {S0, i0, r0,..., S t , i t , r t ,..., S T}. The interaction between the recommendation system and the user is essentially a multi-step interaction process that lasts for a period of time, and its purpose is to obtain a recommendation strategy π * that recommends items to the user to maximize the long-term cumulative reward. The formal definition is as follows:

[0106]

[0107] Among them, is the expected discounted reward under the policy π; the state S tDetermined by the items that the user was interested in before time t, i.e., the items with positive interactions, and used to simulate the user's dynamic preferences; action A t To recommend an item or a list of items; reward r(S t ,A t ) is the reward given by the reward simulator based on the user's immediate feedback on the recommended item, and is considered an improvement to the recommendation strategy π; the discount factor γ ∈ [0, 1], which is used to balance future and immediate rewards. When γ is 0, it will focus entirely on immediate rewards. Generally, γ is taken as 0.9. When t = 1, the immediate reward is 0.9 times r(S t ,A t ), when t = 2, the immediate reward is 0.81 times r(S t ,A t ), and so on; T refers to the length of a trajectory; Σ refers to the summation operation; π refers to the recommendation strategy, and π * refers to the optimal recommendation strategy; Π refers to the set of all strategies.

[0108] This step is a key step in modeling the state, tracking the current user preferences, and encoding the user's long-term and temporary preferences into dense vectors. The state is used to characterize the user's preferences and is abstracted from the items with which the user has had positive interactions. The specific method for determining the user's immediate preference vector at each step is as follows: First, convert the click item sequence {i t} into an embedding vector e t ∈R d , where d is the dimension of the vector, and e n is the embedding representation of entity n obtained in the knowledge graph attention module (here it can refer to the item embedding representation), and f n is a one-hot vector corresponding to a feedback type. The representation after integrating the item feedback is calculated as:

[0109] I n =concat(e n ,tanh(W F f n +b F )) (9)

[0110] where tanh is used as the activation function, W F is the parameter matrix, and b F is the bias vector parameter.

[0111] After obtaining the representation of the items interacted by the user, the GRU model is used to track the state. In the real model, the user's current interests are usually related to the items with which the user has had positive interactions before. Aggregate the user's interaction history and extract the user's state. The update function of one GRU cell is defined as:

[0112]

[0113]

[0114]

[0115]

[0116] Among them, I t is the input vector, i.e. the output of formula (9); z t is the update gate, r t It is the reset gate; is the element-wise product; W z ,W r ,W h They are the matrix parameters for calculating the update gate, reset gate, hidden state, U z ,U r ,U h They are the matrix parameters for calculating the update gate, reset gate, hidden state, and b z ,b r ,b h are the calculation parameters of update gate, reset gate and hidden state respectively; Equation (13) is the hidden state h t The update function is the previous hidden state With the new candidate hidden state Linear interpolation of σ g , σ h They are sigmoid activation function and tanh activation function respectively.

[0117] Step 5: Centering on the items with which the user has had positive interactions, we select k-hop neighbor nodes of the knowledge graph based on the vector distance model as the candidate space set, i.e., the action space.

[0118] The huge action space will cause trouble in the strategy training process, low sampling efficiency and limited training speed. Therefore, it is necessary to screen the candidate set to obtain a set of related items to improve the sample efficiency of strategy learning. Figure 5 The following is a schematic diagram of candidate set screening. The method adopted is to sample all items within the k-hop range of the seed set as the candidate set, which will be used for action selection in the reinforcement model. The seed set is defined as:

[0119]

[0120] Here, t represents the tth step, and the superscript number k refers to the item in the kth hop neighborhood. From this, the k-hop neighborhood set can be defined as follows:

[0121]

[0122] Among them, it is the knowledge graph, and (head, relation, tail) refers to the large number of triples existing in the knowledge graph.

[0123] Step 6: Input the state obtained in Step 4 and the action space obtained in Step 5 into the policy network, and obtain an action vector;

[0124] As Figure 6 shown, this step is to input the state obtained in Step 4 and the set of selectable actions obtained in Step 5 into the policy network. The policy network is composed of a multi-layer perceptron, and the output is a high-dimensional vector representation of the action to be recommended. The policy network can refer to Figure 2 .

[0125] Step 7: Input the action vector output in Step 6 into the reward simulator to obtain a reward. If the reward is greater than the preset threshold, add this action vector to the items that the user has interacted with, and go back to Step 4 to calculate the next state;

[0126] One of the major innovation points of this patent is to construct a new type of reward simulator. The reward simulator receives the current state S t and the action A t , and returns the reward r t and the next state S t+1 . The reward function guides the model to capture the user's preferences and evaluate the quality of the recommended items. It is certainly the most ideal setting to obtain the user's feedback through online interaction. However, the trial-and-error nature during online interaction training will reduce the user experience, and the cost of training the reinforcement learning model in a real interaction environment is expensive and time-consuming. Therefore, offline log data is used to construct a reward simulator to simulate user feedback. The present invention introduces a curiosity mechanism when designing the reward simulator, and gives a positive reward to the agent when it explores unknown items; at the same time, a balance parameter is set to balance the proportion of the curiosity reward and other rewards. Specifically: when the recommended item given by the policy module is within the set of positive interaction items, the agent will receive a reward signal. In order to drive the agent to give positive interaction items with a high probability, the reward size of its recommended action is negatively correlated with the distance from the positive interaction item. At the same time, in order to give the agent a more definite training goal, a distance threshold is set. Within the threshold, the agent receives a positive reward, otherwise the agent receives a negative reward. The reward setting criteria are given from three dimensions: the scoring result, the distance between nodes in the knowledge graph, and the curiosity mechanism. The reward setting calculation is shown according to the following formula:

[0127]

[0128]

[0129] Among them, R t is the recommended action A at time step t t Reward; includes rating in offline data t and distance reward r distance ; represents the curiosity reward output by the curiosity network at time step t; γ is the reward discount factor; V(S t ) is state S t The value of ; τ refers to the balance factor between curiosity reward and distance reward, which is 0.8 in this embodiment; distance is the distance between the recommended item and the positive interaction item, and the distance is calculated using the Dijkstra algorithm; ∈ is to avoid the denominator being 0; W nt It means from i n to i t The sum of the weights of the shortest path.

[0130] Step 8: The state obtained in step 4, the action vector obtained in step 6, the reward obtained in step 7, and the next state obtained in step 4 are stored in the experience pool. A recommendation agent model is then constructed based on the PPO2 algorithm in reinforcement learning. The agent obtains training data and obtains the optimal behavior strategy through trial and error. This completes the interactive recommendation modeling process. Recommendation strategies can then be flexibly adjusted interactively using real-time user feedback.

[0131] The feature vectors obtained after graph embedding and state representation are input into the policy network and state value network respectively to obtain the current action A t and the value of the current state V(S t ), the agent takes action A t The environment returns the reward r, and the curiosity neural network returns the reward The data is stored in an experience pool as reusable experience. When a round ends, the policy π is updated using the data in the experience pool. Data is sampled from the pool using random sampling. After repeating the update three times, the experience pool is cleared and the next round of data update begins until the policy converges.

[0132] The specific calculation method of the loss function value of the PPO2 algorithm with curiosity reward is as follows:

[0133] Adv θ′ (S t ,A t )=R t +(γλ)R t+1 +...+(γλ) T-t+1 R T-1 (18)

[0134]

[0135] where R t is the temporal difference error at time step t; Adv θ′ (A t |S t ) is the action advantage estimate for state s t under policy parameter θ′; ∈ is the truncation threshold; L(θ) represents the policy gradient loss function of the PPO2 algorithm; the function clip() limits the parameter value in the function to the range [1 - ∈, 1 + ∈]; p θ (A t |S t ) is the probability distribution of action A t under policy θ and state S t ; γ is the reward discount factor; τ is the balance factor between the curiosity reward and the distance reward; ∈ approaches 0 infinitely; T refers to the length of the trajectory.

[0136] It should be understood that those skilled in the art can make various improvements or transformations based on the above description under the inspiration of the technical concept of the present invention without departing from the content of the present invention, and this still falls within the protection scope of the present invention.

Claims

1. An interactive recommendation method based on a knowledge graph and reinforcement learning, characterized in that, Including: Step 1: Corresponding the items in the recommendation dataset with the entities in the knowledge base Freebase to construct a specific domain knowledge graph. According to the constructed knowledge graph, pre-embed the knowledge graph to obtain the low-dimensional representations of entities and relationships in the knowledge graph, that is, the mapping of entities to the hyperplane; Step 2: Calculate the attention coefficients between the entity nodes in the knowledge graph and their first-order neighbor entity nodes. Using the calculated attention coefficients and the low-dimensional representations of entities obtained in Step 1, obtain the low-dimensional representation of the central entity that propagates the first-order neighbor entity nodes through a weighted summation calculation method; Step 3: Integrate the low-dimensional representations of entities obtained in Step 1 and the low-dimensional representation of the central entity that propagates the first-order neighbor entity nodes obtained in Step 2. The entity node representation obtained from the integration operation will be used as the next-round low-dimensional representation of entities. Repeat Step 2 and this step until the embedding representation of entity nodes converges; Step 4: Extract the representations of items that the user has had positive interactions with from the entity node representations obtained in Step 3, and input them into the GRU model to obtain the user's immediate preference vector, that is, the state; Step 5: Taking the items that the user has had positive interactions with as the center, select the k-hop neighbor entity nodes of the knowledge graph as the candidate space set, that is, the action space; Step 6: Input the state obtained in Step 4 and the action space obtained in Step 5 into the policy network, and obtain an action vector; wherein the policy network is composed of a multi-layer perceptron, and its output represents the high-dimensional vector of the action to be recommended; Step 7: Input the action vector output in Step 6 into the reward simulator to obtain a reward. If the reward is greater than the preset threshold, add this action vector to the items that the user has interacted with, and re-use the calculation process in Step 4 to obtain the next state; If the reward is less than or equal to the preset threshold, the next state remains the same as the previous state; Step 8: Store the state obtained in Step 4, the action vector obtained in Step 6, the reward obtained in Step 7, and the next state obtained in Step 4 in the experience pool. Build a recommendation agent model based on the PPO2 algorithm in reinforcement learning. The agent obtains training data in a trial-and-error manner and obtains the optimal behavior strategy, and adjusts the recommendation strategy in an interactive form using the user's current immediate feedback.

2. The interactive recommendation method based on a knowledge graph and reinforcement learning according to claim 1, wherein, The mapping formula after mapping the entity to the hyperplane in Step 1 is: Among them, h represents the representation of the head entity node in the knowledge graph, representing the representation of the head entity node after mapping; tail represents the representation of the tail entity node, representing the representation of the tail entity node after mapping; each relation r has a matrix parameter W related to the hyperplane representation r .

3. An interactive recommendation method based on a knowledge graph and reinforcement learning according to claim 1, characterized in that, The formula for calculating the attention coefficient in Step 2 is: g nj = LeakyReLU(Wi n ||Wi j ) (3) where, i n and i j are the vector representations of node n and node j; g nj is the attention coefficient of node j with respect to node n; LeakyReLU is a non-linear activation function; W is a weight matrix; || is the concatenation of two vectors; the softmax function is used to normalize the attention coefficients of all first-order neighbor entity nodes of node n, and the formula is as follows: Among them, α nj is the normalized attention coefficient of node j to the central node n; N is the set of all first-order neighbor entity nodes for node n; i k is the vector representation of node k; The low-dimensional representation i' of the central entity that has propagated the first-order neighbor entity nodes is obtained in the way of weighted summation n The calculation formula is as follows: wherein, σ is a non-linear function and is a kind of single-layer forward neural network.

4. An interactive recommendation method based on a knowledge graph and reinforcement learning according to claim 1, characterized in that The formula for the integration in Step 3 is as follows: e n = f Bi-Interaction (i n , i′ n ) = LeakyReLU(W(i n + i′ n )) + LeakyReLU(W(i n ⊙ i′ n )) (6) Among them, f Bi-Interactton is the integration function using the Bi-Interaction integrator. The integrator can also use the GCN integrator or the GraphSage integrator; ⊙ is the Hadamard product; W is the weight matrix; LeakyReLU is the activation function; i n is the vector representation of node n, and i′ n is the output result of formula (5), that is, the low-dimensional representation i′ of the central entity that has propagated the first-order neighbor entity nodes n ; The aggregation function between the upper layer and the lower layer, and the specific calculation method is as follows: where l is the l-th round of integration, l - 1 is the (l - 1)-th round, i.e., the previous round of the l-th round; refers to the representation of node n in the (l - 1)-th round, refers to the sum of the information of the aggregated first-order neighbor nodes centered on node n in the (l - 1)-th round; obtained by using a Bi-Interaction integrator is the representation of node n in the l-th round that integrates the entity node information in the (l - 1)-th round and aggregates the information of the first-order neighbor entity nodes.

5. An interactive recommendation method based on a knowledge graph and reinforcement learning according to claim 1, characterized in that, Step 4 specifically includes the following steps: Step 4.1: Define the modeling process for interactive recommendation; Given a set of users U = {u1, u2, u3,...}, a set of items I = {i1, i2, i3,...}, and the knowledge graph constructed in step 1 First, initialize the random policy π. When the user, i.e., the environment, and the recommendation policy, i.e., the agent, interact for the t-th time, under the user-specific preference S t the agent recommends an action A i ∈ U to the user u t . This action is selected based on the random policy π, and then the feedback f i ∈ F, including click, favorite, ignore, rating, is obtained. The obtained feedback is used to improve the random policy π, and the user preference state also changes from S t to S t+1 ; Step 4.2: Build an interactive recommendation agent and define the MDP tuple; Given the observations of past interactions {u, i1, f1,..., i t-1 , f t-1}, the recommendation strategy is modeled as a conditional distribution, which is used to represent the probability of recommending item i t at the t-th interaction; a series of interactions between the user and the recommendation strategy will generate a trajectory ξ = {S0, i0, r0,..., S t , i t , r t ,..., S T}, and the recommendation strategy π * that maximizes the long-term cumulative reward by recommending items to the user is defined as follows: Among them, is the expected discounted reward under policy π; the state S t is determined by the items that the user was interested in before time t, that is, the items with positive interactions. The action A t is to recommend an item or a list of items; the reward r(S t , A t ) is the reward given by the reward simulator according to the user's immediate feedback on the recommended item; the discount factor γ ∈ [0, 1]. When γ is 0, it will focus entirely on the immediate return. At t = 1, the immediate reward is 0.9 times r(S t , A t ). At t = 2, the immediate reward is 0.81 times r(S t , A t ); T refers to the length of a trajectory; ∑ refers to the summation operation; π refers to the recommendation policy, and π * refers to the optimal recommendation policy; ∏ refers to the set of all policies; Step 4.3: Determine the user's immediate preference vector for each step, and calculate the representation after item integration feedback; First, convert the click item sequence {i t} into an embedding vector e t ∈R d through Steps 2 and 3, where d is the dimension of the vector; the representation after item integration feedback is calculated as: I n = concat(e n , tanh(W F f n + b F )) (9) Among them, e n is the embedding representation of the entity node n obtained in the knowledge graph attention module; f n is a one-hot vector corresponding to a feedback type; tanh is used as the activation function; W F is the parameter matrix; b F is the bias vector parameter; concat is the concatenation function; I n is the representation after item integration feedback, that is, the output of this step; Step 4.4: Use the GRU model to track the state; The output of the previous step is a new item representation obtained after integrating the feedback of each item the user has interacted with, which is input into the GRU model to calculate the user's state; where the GRU model is a sequence model composed of several GRU units, and the function of one GRU unit is defined as: z t = σ g (W z I t + U z h t-1 + b z ) (10) r t = σ g (W r I t + U r h t-1 + b r ) (11) Among them, I t is the input vector, i.e., the output of formula (9); z t is the update gate; r t is the reset gate; is the element-wise product; W z , W r , W h are the matrix parameters for calculating the update gate, the reset gate, and the hidden state respectively. U z , U r , U h are another set of matrix parameters for calculating the update gate, the reset gate, and the hidden state respectively. b z , b r , b h are the bias parameters for calculating the update gate, the reset gate, and the hidden state respectively. Equation (13) is the update function of the hidden state h t , and the specific calculation is the linear interpolation of the previous hidden state h t-1 and the new candidate hidden state . σ g , σ h are the sigmoid activation function and the tanh activation function respectively.

6. The interactive recommendation method based on a knowledge graph and reinforcement learning according to claim 1, wherein The selection method described in step 5 is to sample all items within the k-hop neighborhood of the seed set as the candidate set, where the definition of the seed set is: Among them, the subscript t represents the t-th step, and the superscript 0 represents the initial set, that is, the seed set; thus, the definition of the k-hop neighborhood set is as follows: Among them, the superscript number k refers to the item within the k-hop neighborhood; is a knowledge graph, (head, relation, tail) refers to a triple in the knowledge graph, head refers to the head entity node in the triple, tail refers to the tail entity node in the triple, and relation refers to the relationship connecting the head entity node and the tail entity node.

7. An interactive recommendation method based on a knowledge graph and reinforcement learning according to claim 1, characterized in that In step 7, the reward simulator receives the current state S t and the action A t , and returns the reward r t and the next state S t+1 ; wherein the reward function guiding model captures the user's preferences and evaluates the quality of the recommended items, constructs a reward simulator using offline log data to simulate user feedback, introduces a curiosity mechanism, and gives a positive reward to the agent when it explores unknown items; at the same time, a balance parameter is set to balance the proportion of the curiosity reward and other rewards; Among them, the reward setting calculation is as shown in the following formula: Among them, R t is the reward for the recommended action A t at time step t; rating t refers to the score in the offline data, and r distance refers to the distance reward; represents the curiosity reward output by the curiosity network at time step t; γ is the reward discount factor; y(s t ) is the value of the state s t ; τ is the balance factor between the curiosity reward and the distance reward, Distance refers to the distance function of the recommended item from the positive interaction item, and the distance is calculated using the Dijkstra algorithm; ∈ approaches 0 infinitely, and W nt refers to the sum of the weights of the shortest path from i n to i t .

8. An interactive recommendation method based on a knowledge graph and reinforcement learning according to claim 1, characterized in that In the recommended agent model described in step 8, the current action is A t , the value of the current state is y(S t ), the agent takes action A t and then the environment returns a reward r, and the curiosity neural network returns a reward which is stored in the experience pool as reusable experience. When a round ends, the data in the experience pool is used to update the policy π. The data is sampled from it in a random sampling manner. After repeating the update 3 times, the experience pool is emptied and the data update for the next round starts until the policy converges; The specific calculation method of the loss function value of the PPO2 algorithm with curiosity reward is as follows: Adv θ′ (S t ,A t )=R t +(γλ)R t+1 +…+(γλ) T-t+1 R T-1 (18) where R t is the temporal difference error at time step t; Adv θ′ (A t |S t ) is the action advantage estimation function for state s t under the policy parameter θ′; ∈ is the truncation threshold; L(θ) represents the policy gradient loss function of the PPO2 algorithm; the function clip() limits the parameter value in the function to the range [1 - ∈, 1 + ∈]; p θ (A t |S t ) is the probability distribution of action A t under the policy θ and state S t ; γ is the reward discount factor, τ is the balance factor between the curiosity reward and the distance reward, ∈ approaches 0 infinitely, and T refers to the length of the trajectory.

Images