Multi-turn dialogue recommendation method based on rgn and reinforcement learning

Abstract

The application discloses a multi-round dialogue recommendation method based on RGCN and reinforcement learning, comprising the following steps: converting user dialogue information into knowledge graph representation, and utilizing a relation graph neural network RGCN to obtain node embedding of the knowledge graph; using multi-head self-attention to obtain good item representation to realize that each item feature embedding is better connected, then using another multi-head self-attention mechanism to capture various user preferences, and using a GRU with an attention update gate to overcome the interference of interest drift; finally, introducing reinforcement learning RL in CRS, which is used for learning a dialogue recommendation strategy, deciding the attributes asked in each dialogue round, the recommended items and when to ask or recommend, and using an improved adversarial Q network DQN to make action decisions, so as to select the optimal next action.

Description

Technical Field

[0001] This invention relates to the field of multi-turn dialogue recommendation, and in particular to a multi-turn dialogue recommendation method based on RGCN and reinforcement learning. Background Technology

[0002] 1. Dialogue Recommendation System

[0003] Unlike traditional recommender systems, conversational recommender systems (CRS) are a cutting-edge discipline that integrates dialogue systems, recommender systems, and interaction systems. They enable the system to proactively engage in dialogue with users and provide highly interpretable and accurate recommendations to meet their dynamic needs. Based on different functions and settings, current CRS methods can be broadly categorized into two types: dialogue-based recommendation and multi-turn dialogue recommendation (MCR). In this paper, we will focus on the MCR problem, as it is considered the most practically applicable setting in CRS. Unlike dialogue-based recommendation, which requires extracting information or generating responses from raw natural language, the emergence of MCR methods has brought new ideas to recommender systems. MCR not only improves the strategies for when and how to ask questions but also introduces external heuristic rules to assist in deciding which item to recommend. Specifically, EAR extends this framework to a multi-turn dialogue recommendation setting, allowing the system to make multiple recommendations. When a user rejects a recommendation, the system can retain the requested attribute until the user accepts the recommendation or the maximum number of turns is reached. Furthermore, to reduce the action space in policy learning, SCPR models dialogue recommendation as an interactive path reasoning problem on a graph, which helps to eliminate irrelevant candidate attributes. Meanwhile, SCPR also improves upon the strategy of simply deciding whether to ask or recommend, and develops an external path reasoning method to determine which attribute to ask or which item to recommend. UNICORN proposes a unified policy learning framework based on dynamic weighted graphs to solve three decision problems in CRS. In traditional MCR settings, users can typically only select one preferred attribute value at a time, which limits their ability to express needs during interaction. To overcome this limitation, Zhang et al. proposed the MIMCR setting, allowing users to select multiple options for a given attribute. MCMIPL further considers the user's multiple interests in the unified policy and develops a multi-interest policy learning module.

[0004] 2. Reinforcement learning in recommendations

[0005] Reinforcement learning (RL), as a machine learning paradigm, aims to solve the problem of how agents make decisions in specific environments to pursue long-term goals. In reinforcement learning, agents learn and improve their strategies based on reward signals provided by the environment through interaction. In recent years, reinforcement learning has demonstrated remarkable application potential in the field of recommender systems. Unlike traditional recommender systems, reinforcement learning focuses on long-term user satisfaction, not just satisfying current interests. This characteristic makes reinforcement learning exhibit superior performance in solving some key problems in recommender systems. For example, reinforcement learning can be used to generate appropriate questions to maximize user satisfaction when searching for products, or to schedule solutions to achieve efficient service and meet users' long-term needs. Furthermore, reinforcement learning can help recommender systems overcome problems such as sparse feedback and cold start, thus better adapting to constantly changing user behaviors and preferences. Driven by reinforcement learning, the recommender system field is gradually moving towards a new era of personalization and intelligence. By combining reinforcement learning with recommender systems, we can better understand users' dynamic needs, achieve more accurate and intelligent recommendations, and provide users with a better experience. In this work, we use reinforcement learning to learn a strategy that can automatically ask questions and recommend items.

[0006] 3. Graph-based recommendation

[0007] In recent years, graph-based recommendation methods have attracted increasing research attention. These methods primarily pursue two goals. First, they leverage graph structures to improve recommendation performance, including using structural information for collaborative filtering to recommend items and embedding knowledge graphs as contextual information to enrich the recommendation process. Second, they model recommendation as a path reasoning problem, aiming to build interpretable recommendation systems that enable users to understand the reasons behind the recommendations. In recent research, graph-based reinforcement learning methods have also begun to emerge in recommendation systems. These methods use graph structures to improve recommendation strategies, such as path sampling through graph search methods, enhancing path reasoning through adversarial training, or using graph convolutional networks to learn state representations, thereby improving recommendation performance. Particularly in dialogue recommendation systems, graph-based methods play a crucial role in understanding user preferences and generating more relevant recommendations. They allow systems to better understand user needs during natural language dialogues and generate recommendations that resemble human responses. This typically involves integrating different types of knowledge, such as knowledge graphs, text descriptions, and pre-trained language models, to more comprehensively model user preferences. In summary, graph-based recommendation methods have become an important research direction in the field of recommender systems. By making full use of graph structure information and reinforcement learning techniques, they have significantly improved the performance and interpretability of recommender systems.

[0008] Despite significant progress in CRS, many potential challenges remain, leading researchers to focus on specific application scenarios to effectively address these unique challenges. In this study, we focus on the setting of Multi-round Conversational Recommendation (MCR), widely considered the most practical and realistic application scenario. MCR recommends products by engaging in multiple rounds of interaction with the user, incorporating the user's click history (if any). In each round of dialogue, MCR can perform two main types of operations, such as... Figure 1 As shown, there are two main approaches: first, explicitly asking the user if they are interested in certain project attributes; and second, recommending a set of products to the user. Throughout the conversation, the MCR can flexibly switch between these two actions multiple times, aiming to minimize the number of interaction rounds while meeting user needs.

[0009] However, behind the complexity of multi-turn dialogues lie a series of challenges, including the dynamic changes in user preferences and interests, the uncertainty of the dialogue, and a complete understanding of users' true needs and preferences. These factors mean that in multi-turn dialogues, recommender systems may miss some key user preference information or fail to fully capture user needs, potentially affecting the accuracy of recommendation results and user satisfaction.

[0010] To overcome the information incompleteness problem in MCR models, this research introduces a new scenario called Multi-Interest Multi-round Conversational Recommendation (MIMCR). In the MIMCR scenario, users are allowed to have multiple preferences. For example, a user can have preferences for multiple attribute instances of a certain attribute type (e.g., documentaries and science fiction). Compared to MCR, introducing the MIMCR scenario helps the dialogue system to understand user preferences and needs more deeply, while reducing the number of dialogue interactions, thus providing users with a better recommendation experience.

[0011] However, the MIMCR model faces some significant challenges, such as Figure 2As shown on the left, the system filters the candidate item set based on the user's selection. When the user does not select black or khaki from the color attributes, all candidate items for that attribute are deleted, potentially resulting in no candidate items to recommend. Furthermore, it assumes the user has explicit preferences during the conversation, which is unrealistic. User interests can drift over time and in different contexts, making accurately capturing true user preferences quite challenging. Existing CRS methods only use attribute feedback implicitly, failing to adequately update user representations, and the dialogue data itself lacks sufficient contextual information to accurately understand user preferences. This inability to understand user preferences leads to recommendations that do not adequately meet the user's actual needs. Summary of the Invention

[0012] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multi-turn dialogue recommendation method based on RGCN and reinforcement learning. This knowledge-enhanced multi-turn dialogue recommendation method achieves more accurate recommendations by making full use of external knowledge.

[0013] To achieve the above objectives, the technical solution adopted in this invention is as follows: a multi-turn dialogue recommendation method based on RGCN and reinforcement learning, including converting user dialogue information into a knowledge graph representation and using a relational graph neural network (RGCN) to obtain node embeddings of the knowledge graph; using multi-head self-attention to obtain good item representations to better connect the feature embeddings of each item; then using another multi-head self-attention mechanism to capture multiple user preferences and using a GRU with an attention update gate to overcome the interference of interest drift; finally, reinforcement learning (RL) is introduced into the CRS to learn the dialogue recommendation strategy, determine the attributes to be queried in each dialogue round, the recommended items, and when to query or recommend; and using an improved adversarial Q network (DQN) for action decision-making, thereby selecting the optimal next action, i.e., the system ultimately successfully recommends items that satisfy the user.

[0014] The user's current and historical dialogue information is transformed into a dialogue graph, and a knowledge graph-based MDP environment is constructed. The knowledge graph-based MDP environment provides tasks and feedback. Knowledge graph-enhanced state representation learning helps to understand the environment. Action selection strategies determine the agent's behavior, and Dueling Q-Network is used to learn and optimize behavior, enabling the agent to achieve the best results in the environment.

[0015] The knowledge graph MDP environment includes states, actions, transitions, and rewards. These components interact to influence the agent's learning and behavior.

[0016] Knowledge graph augmented state representation learning integrates external knowledge graphs into the state representation of an agent to improve its performance in complex tasks. It includes two main tasks: encoding and refining user behavior and extracting and evolving user preferences.

[0017] The coding and refinement of user behavior includes:

[0018] RGCN is used to embed nodes and edges in graph data to enable machine learning models to better understand and process them. Through processing, node embedding representations corresponding to the current dialogue graph and historical dialogue graph are obtained, namely: the current weighted knowledge graph representation based on RGCN and the global heterogeneous knowledge graph representation based on RGCN. A multi-head self-attention mechanism is introduced to enhance the model's learning of the interaction relationships between different inputs. It maps the node representation to a higher-dimensional space so that the model can better capture the dependencies between different nodes. The multi-head self-attention mechanism is used to learn and represent information in the knowledge graph so that the model can understand the user's needs and preferences.

[0019] The extraction and evolution of user preferences includes: encoding user behavior and refining the output results; extracting user preference information through a second-layer multi-head self-attention mechanism to capture the user's various preferences for items and the relationships between items; then introducing a GRU with an attention update gate to further evolve user preferences; by introducing an update gate with an attention mechanism, the attention mechanism is seamlessly integrated with the GRU to more finely control and adjust the evolution of user preferences; finally, the user preference embedding representation is output.

[0020] Action selection strategies involve efficiently finding the optimal action within a vast action space to improve the effectiveness and efficiency of recommendations, based on the coarse-match score of item v in the current round. To further filter the action space, following the same calculation logic, we obtain the coarse matching score for the current turn's attribute p. Finally, we use these two scores to select candidate item set V cand and candidate attribute set P cand The top N items and top N attributes are selected and placed into the action space for the next recommendation step.

[0021] In reinforcement learning, Dueling Q-Networks can improve learning efficiency by decomposing the Q-value into a state-value function and an advantage function, allowing the network to more flexibly estimate the impact of each action. In Dueling Q-Networks, we define a Q-value function Q of policy π. π (s t a t To measure based on state s t and action a tThe cumulative reward; and we embed {h1, h2, ..., h...} based on previously learned user preferences. H} to calculate in state s t Next, action a t The scores between these user preferences are used, and the maximum value among them is taken as the final Q-score; where f V function and f A The functions represent the state value function and the dominance function, respectively;

[0022] Q π (s t a t ) = max(f V (h k )+f A (h k a t k∈{1,...,H}

[0023] The Q-function satisfies the Bellman equation, which expresses the relationship between the optimal Q-value and the current state, action, and next state. Therefore, by solving the Bellman equation, the optimal Q-function can be obtained, thereby determining the optimal action strategy π. * To maximize the cumulative reward, we define the optimal Q-function as follows: (s t a t The calculation method is as follows:

[0024]

[0025] r t It is state s t Take action a t The immediate reward obtained later, γ is a discount factor used to weigh the importance of current rewards and future rewards. It is a value between 0 and 1, used to represent the degree of decay of future rewards.

[0026] In each round, the agent interacts with the user, and the user provides a reward. t This reward is a form of feedback, telling the agent how it performed in the previous action or state. Positive rewards encourage the agent to continue taking similar actions, while negative rewards encourage the agent to avoid those actions. Based on user feedback, the environment enters a new state and a new action space after the agent takes an action. We define a replay buffer D, the purpose of which is to collect and store experience samples (s) gained by the agent in its interaction with the user. t a t r t s t+1 A t+1Experience replay is a technique that randomly samples small batches of experience samples during training to help make training more stable and efficient. Finally, the agent uses these experience samples to train the model, updating the model's parameters by minimizing a loss function. We define such a loss function as follows:

[0027]

[0028] here θ V θ A Let θ represent the dominance function and the action function, respectively. P This represents the set of parameters for obtaining user preference embeddings, y t The target value is calculated based on the optimal Q-function, and the immediate reward r for the current round is also represented. t Add the product of the maximum predicted Q value for the next round:

[0029]

[0030] The advantages of this invention are as follows: It proposes a novel user preference policy learning framework, KEUPPL, which utilizes RGCN to encode user-related external knowledge, enhancing user data representation. Then, it employs a multi-head self-attention mechanism and a GRU with an added attention update gate to extract user preferences, avoiding the interference of interest drift and effectively modeling the long-term evolution of user preferences. Finally, an improved DQN is introduced for action decision-making, selecting the optimal next action. Attached Figure Description

[0031] The following is a brief explanation of the contents of each of the accompanying drawings and the markings in the drawings:

[0032] Figure 1 This is the working process of MCR;

[0033] Figure 2 The image on the left is a schematic diagram of an existing multi-turn dialogue scenario;

[0034] Figure 2 The right side is a schematic diagram of a multi-turn dialogue scenario according to the present invention;

[0035] Figure 3 This is a diagram of the KEUPPL framework in this invention;

[0036] Figure 4 These are the results of running our model and MCMIPL on the LastFM* dataset at SR@15;

[0037] Figure 5 These are the results of our model and MCMIPL running on the Amazon-Book dataset SR@15;

[0038] Figure 6 These are the results of our model and MCMIPL running on the MovieLens dataset at SR@15. Detailed Implementation

[0039] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and the description of the preferred embodiments.

[0040] To address the issues in MIMCR and maintain diversity and accuracy in CRS, we propose a novel scenario called Knowledge-Enhanced Multi-round Conversational Recommendation (KEMCR), such as... Figure 2 As shown on the right. In this scenario, the system can prevent over-screening of candidate items, laying a solid foundation for accurately capturing user preferences by fully utilizing external knowledge (i.e., user dialogue information). The multi-turn dialogue recommendation method based on RGCN and reinforcement learning includes converting user dialogue information into a knowledge graph representation and using a relational graph neural network (RGCN) to obtain node embeddings of the knowledge graph; using multi-head self-attention to obtain good item representations to better connect the feature embeddings of each item; then using another multi-head self-attention mechanism to capture multiple user preferences; and using a GRU with an attention update gate to overcome the interference of interest drift; finally, reinforcement learning (RL) is introduced into the CRS to learn the dialogue recommendation strategy, determining the attributes to be queried, the recommended items, and when to query or recommend in each dialogue turn; and using an improved adversarial Q-network (DQN) for action decision-making to select the optimal next action, thus successfully recommending items that satisfy the user.

[0041] This paper chooses the KEMCR learning scenario as the problem setting because it better reflects real-world scenarios. It's worth noting that, within the context of KEMCR, we focus on how to achieve efficient personalized recommendations by engaging in multi-turn dialogues with users, continuously refining their preferences, and ultimately recommending items that satisfy the user. To clearly define this problem, we introduce the following key concepts and assign them corresponding symbols, as shown in Table 1: We use u∈U to represent user u in user set U, v∈V to represent item v in item set V, and p∈P to represent attribute type (instance) p in attribute set P, and an item v is associated with a set of attributes P. v Related. For example, the music genre of songs in LastFM is "folk" or "rock," etc. A new conversation begins on the user's end, which initializes the attribute p by having the user specify their preferred attributes (e.g., I like listening to instrumental music). startThen the agent will select from the attribute candidate set P cand Asking users questions about attributes or from the project candidate set V cand The agent recommends items to the user. Specifically, CRS dynamically updates the candidate set based on the user's choices and responses. In the current t-th dialogue round, the agent observes the current state S... t And according to strategy π from action space A t Take the best action:

[0042] If the action is a ask This means that the system is asking user u if they like attribute P. u (one or more), if the user has certain attributes p t Given a positive feedback signal, these p t Add to P cand In the middle, it indicates that the user is more interested in these attributes, and the remaining attributes are added to P. rej If the user is not interested in these attributes, the conversation will move on to the next round.

[0043] If the action is a rec This means that the system recommends item V to user u at this time. rec (One or more). User u will provide feedback to the system based on their preferences, if there are items v t If a user provides positive feedback, it means that user u has accepted the recommendation (at least one item), and the conversation ends. Conversely, if the user rejects all system-recommended items, then V... rej An update will occur, V rec Put V in rej The dialogue will now move on to the next round.

[0044] The system will repeat the above process until it finds an item the user likes or reaches the maximum number of dialogue rounds. The KEMCR scene differs from previous MCR scene settings in that it does not adjust the user's actions based on attribute p within a single round. t The evaluation is then completely deleted, and instead, it is recombinated with other attributes in the following rounds. Furthermore, The project is The KEMCR scenario aims to balance comprehensiveness and filtering. It ensures the system provides comprehensive recommendations covering all areas of potential user interest, while avoiding over-filtering that prevents users from finding items or attributes they like. This balance is a key characteristic of KEMCR. However, the biggest challenge facing the KEMCR scenario is the uncertainty of user preferences in multi-turn dialogue scenarios. Users may not even know what they like or dislike, and user preferences may dynamically change as the conversation progresses. Therefore, developing a learning strategy that considers these two issues is crucial.

[0045] To address the challenges in KEMCR scenarios, we propose a knowledge-enhanced user preference policy learning (KEUPPL) framework, a novel general solution framework for MCR, such as... Figure 3 As shown.

[0046] We define the KEMCR scenario as a Markov Decision Process (MDP) problem involving an agent. Decision-making is divided into two types: system query and system recommendation. The current state and the final action are communicated to the agent by the MDP environment, and the agent is then rewarded based on the corresponding action taken according to the current policy. In the KEMCR scenario, our framework aims to maximize the policy network π = (s t |a t The expected cumulative reward π * , π * The calculation formula is shown in Formula 1. Where s t Let a be the state representation of dialogue round t. t For the action taken by the agent in the t-turn dialogue, r t Represents state s t Take action a t The instant reward obtained is expressed as r. t =(s t a t ).

[0047]

[0048] The KEUPPL framework consists of three main components: a knowledge graph-based MDP environment, knowledge graph-enhanced state representation learning, and a noisy network-based action selection policy learning. These three parts together form a complete reinforcement learning system. The knowledge graph-based MDP environment provides the task and feedback, the knowledge graph-enhanced state representation learning helps in understanding the environment, and the noisy network-based action selection policy determines the agent's behavior. An improved Dueling Q-Network is used to learn and optimize behavior, enabling the agent to achieve optimal results in the environment. The goal of this architecture is to improve the agent's performance in MCR through continuous learning and optimization.

[0049] 1. Knowledge Graph-Based MDP Environment

[0050] Markov Decision Processes (MDPs) are a mathematical framework in Reinforcement Learning (RL) used to describe and model the interaction between an agent and its environment. An MDP environment comprises four basic components: State (S), Action (A), Transition (T), and Reward (R). These components interact to influence the agent's learning and behavior.

[0051] (1) State

[0052] In this paper, we define state s t The dialogue information for the current round t, as shown in Formula 2, includes the following parts: user u, and the set of attributes that user u has interacted with. User u's rejected attribute set Candidate attribute set User U rejected the project set and candidate project set

[0053]

[0054] (2) Actions

[0055] In this article, we define action a t The decision that the agent needs to make in round t, based on state s t The agent can make two decisions: a t From the candidate project set Select an item to recommend; this action is used... This can also be expressed as a set of candidate attributes; Selecting an attribute to query is the behavior used in this context. As shown in Formula 3:

[0056]

[0057] (3) Transition

[0058] We define the next state as s. t+1 According to our previous definition, s t+1 It consists of the following parts, as shown in Formula 4: user u, and the set of attributes that user u has interacted with. User u's rejected attribute set Candidate attribute set User U rejected the project set and candidate project set in, as well as Updates will occur. Specifically, It contains the set of all attributes that the user has interacted with during round t. Includes the set of attributes that the user is interested in during round t. At the same time, the user's acceptance and rejection attribute set should be included. Delete, that is To ensure that the same question is not asked twice; in round t, when the user accepts the system's recommended item... At this point, the conversation ends and no updates occur; if the user rejects all system-recommended items... Then, Then it will be updated to: and The update is similar, at this point our Updated to: in The itemset that corresponds one-to-one with the attributes in the document.

[0059]

[0060] (4) Rewards

[0061] In this paper, we define five types of rewards as follows: (1) r rec_suc This is the reward an agent receives in a recommendation system when it successfully recommends an item to a user and the user accepts the recommendation. It is a positive reward, indicating that the agent's recommendation is correct and useful to the user. (2)r rec_fail When an agent recommends an item to a user but the user rejects or is dissatisfied, it receives this reward, which is usually a negative reward, indicating that the agent's recommendation was unsuccessful and the user is dissatisfied. (3)r ask_suc This is the reward an agent receives when it asks a user a question or requests information about an attribute or item, and the user answers positively or satisfactorily. It is also a positive reward, indicating that the agent's inquiry was successful and the user is willing to provide information. (4)rask_fail This reward is given when an agent asks a question or requests information from a user, but the user fails to provide or provides an unsatisfactory answer. It is usually a negative reward, indicating that the agent's query was unsuccessful and the user's response did not meet the user's needs. (5)r over When the interaction between the user and the system reaches the maximum number of rounds T, this is usually a negative reward.

[0062] 2. Knowledge Graph Augmented State Representation Learning

[0063] Knowledge graph augmented state representation learning is an important reinforcement learning technique. Its main task is to integrate external knowledge graphs into the state representation of an agent to improve its performance in complex tasks. In this paper, knowledge graph augmented state representation learning includes the following two main tasks: encoding and refining user behavior, and extracting and evolving user preferences.

[0064] (1) Coding and refinement of user behavior

[0065] like Figure 3 As shown on the left, the Relational Graph Convolutional Network (RGCN) is a deep learning model for processing graph data, particularly suitable for handling rich and complex relational data, such as knowledge graphs, social networks, and user behavior data in recommender systems. In terms of encoding and refining user behavior, RGCN effectively embeds nodes and edges in graph data, enabling machine learning models to better understand and process this data. Therefore, we learn node representations for two graphs: a current weighted knowledge graph representation based on RGCN and a global heterogeneous knowledge graph representation based on RGCN. Following this, we introduce a multi-head self-attention mechanism to enhance the model's learning of interactions between different inputs. This maps the node representations to a higher-dimensional space, allowing the model to better capture dependencies between different nodes. Using multi-head self-attention helps us better learn and represent information in knowledge graphs, enabling the model to more accurately understand user needs and preferences, thereby providing higher-quality dialogue recommendations. Next, we will detail these three parts:

[0066] Current weighted knowledge graph representation based on RGCN

[0067] First, we construct a weighted knowledge graph of the current dialogue state. Its expression is as follows:

[0068]

[0069] in, The set of nodes that constitute the edges of the knowledge graph consists of the following parts:

[0070]

[0071] ε t This represents the weights corresponding to these edges, based on... From the definition, we can obtain ε t The value can take three cases: the edge weight between the candidate attribute and its corresponding candidate item; the edge weight between the user and the candidate attribute; and the edge weight between the user and each item. We assign the values ​​1, 1, and 1 respectively. in It is also a scalar, representing the matching score of item v in the current round. The calculation method is as follows:

[0072]

[0073] Here we define c u c v c p These represent the initial embedding user, the embedding project, and the embedding attributes, respectively. These are used to measure the similarity between a user and an item, the similarity between items that the user has interacted with and their attributes, and the similarity between items that the user interacted with but was rejected and their attributes. We use the sigmoid function to convert the final similarity into a probability value between 0 and 1.

[0074] Next, we will use a multi-layer RGCN to learn. The relationships and representations between nodes. First, we initialize the node representation, defining the initial embedding of node n as... And so on, the node embedding in the (l+1)th layer is... It is obtained by weighted summation of the embeddings of neighboring nodes and then performing a nonlinear mapping, as follows:

[0075]

[0076] in, It is the output of the embedding of the l-th layer node. These are trainable parameters. Let be the set of all neighbor nodes of node n in round t, where |Nor| is a positive number representing a normalization factor used to normalize the embeddings of neighbor nodes. The introduction of the normalization factor balances the differences in the number of neighbor nodes between nodes, ensuring that each node's contribution to its neighbors is relatively consistent during information transmission, without bias in information transmission due to differences in the number of neighbor nodes. We define the output of the last layer of node embeddings as... The final representation of C n .

[0077] Global Heterogeneous Knowledge Graph Representation Based on RGCN

[0078] Secondly, we construct a global heterogeneous knowledge graph. It includes information from the entire knowledge graph, not just the content of the current conversation. This helps the model better understand the items, attributes, and relationships between them in the knowledge base, thereby improving the quality of the dialogue recommendation.

[0079] and resemblance, The expression is as follows:

[0080]

[0081] Similarly, we use RGCN to learn The relationship between nodes and edges is important. It's worth noting that there are two types of ε in the global graph: ε... u，v and ε p，v Therefore, we define it here. and As shown in equations (10) and (11), these represent the output representations of the edges connecting the relationships between user u and item v in the (l+1)th layer, and the output representations of the edges connecting the relationships between item v and attribute p in the (l+1)th layer, respectively. middle, This represents the trainable parameters of the (l+1)th layer, which is a bias term. These are trainable parameters. This represents the neighbor node of node n under the edge corresponding to the relationship between user u and project v. |Nor_u, v| is a positive number representing the normalization factor.

[0082]

[0083]

[0084] We define the initial embedding as Node embedding at layer l+1 The calculation method is as follows:

[0085]

[0086] Here we use the average value to consider the information conveyed by different edges. Similarly, we define the output of the last layer of node embedding as... The final representation of G n .

[0087] Multi-head self-attention layer

[0088] In this mechanism, "head" represents different attention heads, each responsible for learning a different attention pattern. The input to a multi-head self-attention mechanism is typically a set of queries, keys, and values ​​derived from the same input, used to calculate a weighted sum of attention scores. Specifically, first, the embedding representation C of the current weighted knowledge graph is... n Embedded representation G of global heterogeneous knowledge graph n These are used as inputs for queries, keys, and values, respectively, meaning that for each head, there will be a corresponding C. n and G n As input, a set of attention scores is calculated for each head, used for weighted value calculation. This typically involves calculating the similarity between the query and the key, obtaining the attention weights by applying the Softmax function. For the i-th head (denoted as head)... i The attention weights are calculated as follows:

[0089]

[0090] in, and These are the query and key projection matrices used for the i-th head, respectively.

[0091] Next, we use the attention weights of each head to perform a weighted sum of the values, obtaining the output of the multi-head self-attention mechanism:

[0092]

[0093] in, This is the projection matrix used for the value of the i-th head. It's worth noting that here... With C n The reason for multiplying is to weight the current knowledge graph representation C with the attention scores obtained through the multi-head self-attention mechanism. n This approach emphasizes the parts that receive high attention while suppressing the parts that receive low attention, allowing the model to focus more on the information in the knowledge graph that is relevant to the current task or problem.

[0094] Finally, by concatenating the outputs of all heads, the final output of the multi-head self-attention mechanism is obtained.

[0095] (2) Extraction and evolution of user preferences

[0096] like Figure 3As shown in the middle, we extract user preference information through a second multi-head self-attention mechanism. This layer captures the user's various preferences for items and the relationships between items. Next, we introduce a GRU with an attentional update gate (AUGRU) to further evolve user preferences. In this part, by introducing an update gate with an attention mechanism, we seamlessly integrate the attention mechanism with the GRU to more finely control and adjust the evolution of user preferences. This attentional update gate allows for differences in importance between different dimensions, more effectively avoiding the interference of interest drift, thereby improving model performance. The multi-head self-attention layer helps capture global information, while the GRU with an attentional update gate helps to finely adjust this information to adapt to the user's different preferences and needs in the dialogue. These two parts work together to ensure that user preference information is accurately extracted and evolved, resulting in more accurate results in dialogue recommendation. Next, we will introduce these two parts in detail:

[0097] Multi-head self-attention layer

[0098] Similar to the previous section, we use another multi-head self-attention mechanism here, whose main function is to capture the user's multiple interests. Likewise, in this layer, we define the calculation of the attention weight of the i-th head as follows:

[0099] head i ≡Attention′ i (C n G n (15)

[0100]

[0101] Next, the weighted summation is calculated to obtain the output of the multi-head self-attention mechanism:

[0102]

[0103] Finally, we package all the outputs into a shorthand form: {x1, x2, ..., x...} H}, where H is the number of heads, which is also equivalent to the number of items the user likes.

[0104] AUGRU (GRU with Attentional Update gate)

[0105] This is an improved GRU architecture that combines with an attention mechanism to better control the updates of the hidden state. In this paper, we will use the output {x1, x2, ..., x...} of the second multi-head self-attention layer. HThe inputs are fed into multiple AUGRUs, allowing the model to focus more precisely on different features and relationships within the sequence, thus helping to better capture the diversity of the data. Specifically, for the i-th AUGRU (where i ranges from 1 to H), the attention update gate, reset gate, new memory unit, and output hidden state can be calculated as follows:

[0106]

[0107]

[0108]

[0109]

[0110] Here, i represents the i-th AUGRU module, and x t This represents the output from the multi-head self-attention layer. This represents the hidden state of the previous time step. This represents the query vector used to compute attention, while The attention score is used to measure the various dimensions of the update gate. The less relevant the interest, the smaller the impact on the hidden state. Through this approach, AUGRU more effectively controls the updates of hidden states across different dimensions, thus better avoiding the interference of interest drift while activating relevant needs during the evolution of user preferences. After these steps, each output hidden state has sufficient capacity to express the user's preferences and interests, so we define {h1, h2, ..., h...} H This serves as the final user preference embedding.

[0111] 3. Action selection strategy learning based on noisy networks

[0112] As the dialogue continues, the candidate attribute set... and candidate project set As the space continues to expand, if we want to include both of these parts in the action space, our candidate action space A... t As the scope expands, the challenge becomes finding the optimal action within this vast action space to improve the effectiveness and efficiency of recommendations. Here, we consider the coarse-match score of the current round's item v. To further filter the action space, following the same calculation logic, we obtain the coarse matching score for the current turn's attribute p. Finally, we use these two scores to select candidate item set V cand and candidate attribute set P candThe top N items and top N attributes are selected and placed into the action space for the next recommendation step.

[0113]

[0114] After obtaining the final user preference embedding, we need to further analyze and process these preferences to determine whether the next action should be to recommend items or query attributes. This requires a suitable strategy. In reinforcement learning, the Dueling Q-Network allows the network to more flexibly estimate the impact of each action, thereby improving learning efficiency. Secondly, it can better distinguish the merits of different actions, helping the agent to better choose actions and make optimal strategies. Specifically, the Dueling Q-Network decomposes the Q-value into a state-value function and an advantage function. The state-value function represents the expected cumulative reward in a given state, i.e., the expected reward from the current state, choosing an action according to the current policy, and continuing until the task ends. Mathematically, the state-value function is denoted as V(s), representing the value in state s. In the Dueling Q-Network, the state-value function measures the baseline value of a state, i.e., the average expected reward of the state. The advantage function represents the impact of taking an action in a given state relative to the average level, i.e., the superiority of the current action compared to other actions. Mathematically, the advantage function is denoted as A(s,a), representing the advantage of taking action a in state s. The introduction of the advantage function allows the network to estimate the impact of each action more flexibly, rather than simply assigning a Q-value to each action. Therefore, even in complex reinforcement learning tasks, the Dueling Q-Network can better handle combinations of states and actions, thus improving performance.

[0115] Meanwhile, NoisyNet in reinforcement learning introduces learnable noise into the weights of the neural network, making the network more stochastic in its exploration of the environment. Specifically, NoisyNet introduces a set of learnable noise parameters that dynamically adjust the neural network weights. These noise parameters are typically the standard deviation or scaling parameters of the network weights. Each weight is parameterized as a combination of the mean and standard deviation. By introducing learnable noise into the weights, the network's output becomes more stochastic. Thus, the same input may produce different outputs at different time steps, increasing the randomness of the exploration. In reinforcement learning, NoisyNet is primarily used for estimating the action value function (Q-function). For each action, the network outputs a corresponding Q-value. Due to the introduced noise, the Q-values ​​of different actions in the same state may differ, which helps to explore the action space more comprehensively. In dialogue recommendation systems, combining NoisyNet and Dueling Q-Network can improve the model's exploration performance, making it easier to discover different actions in the environment and thus better learn the optimal policy. This is very helpful for handling complex reinforcement learning tasks, especially those requiring extensive exploration.

[0116] In our improved Dueling Q-Network, we define a Q-value function Q of policy π. π (s t a t To measure based on state s t and action a t The cumulative rewards. And we embed {h1, h2, ..., h...} based on previously learned user preferences. H} to calculate in state s t Next, action a t The scores between these user preferences are used, and the maximum value among them is taken as the final Q-score. Where f V function and f A The functions represent the state value function and the dominance function, respectively.

[0117]

[0118] The Q-function satisfies the Bellman equation, which expresses the relationship between the optimal Q-value and the current state, action, and next state. Therefore, by solving the Bellman equation, the optimal Q-function can be obtained, thereby determining the optimal action strategy π. * To maximize cumulative reward, we define the optimal Q-function as... (s t a t The calculation method is as follows:

[0119]

[0120]

[0121] r t It is state s t Take action a t The immediate reward obtained later, γ is a discount factor used to weigh the importance of current rewards and future rewards. It is a value between 0 and 1, used to represent the degree of decay of future rewards.

[0122] The system first selects the action with the highest Q-value. If the selected action points to an item, the system will recommend the top-N items with the highest Q-values ​​to the user. If the selected action points to attribute p, the system will generate options based on the attribute type to continue prompting the user. This process is repeated until a recommendation is successful or the maximum number of rounds is reached.

[0123] 5. Model Training

[0124] In each round, the agent interacts with the user, and the user provides a reward. t This reward serves as feedback, informing the agent how well it performed in the previous action or state. Positive rewards encourage the agent to continue taking similar actions, while negative rewards encourage the agent to avoid such actions. Based on user feedback, the environment enters a new state and a new action space after the agent takes an action. We define a replay buffer D, the purpose of which is to collect and store experience samples (s) gained by the agent in its interaction with the user. t a t r t s t+1 A t+1 Experience replay is a technique that randomly samples mini-batch experiences during training to help make training more stable and efficient. Finally, the agent uses these experience samples to train the model, updating the model's parameters by minimizing a loss function. We define such a loss function as follows:

[0125]

[0126] here θ V θ A Let θ represent the dominance function and the action function, respectively. P This represents the set of parameters for obtaining user preference embeddings, y t The target value is calculated based on the optimal Q-function, and the immediate reward r for the current round is also represented. t Add the product of the maximum predicted Q value for the next round:

[0127]

[0128] During training, the agent continuously interacts with users, receiving positive and negative reward feedback. These experience samples are stored in a replay buffer D, from which mini-batch samples are randomly drawn for training. By minimizing the loss function, the parameters of our model KEUPPL are gradually adjusted to make actions more aligned with user preferences. This training process enables the agent to better adapt to user feedback, improving the performance of the recommendation system. The architecture corresponding to the method in this application is KEUPPL: a knowledge graph-based MDP environment, knowledge graph-enhanced state representation learning, and noisy network-based action selection policy learning. The various models included in the KEUPPL architecture, such as AUGRU and RGCN, require parameter tuning; therefore, training is necessary to obtain the parameters of each model within the KEUPPL architecture before recommendation processing is performed based on this architecture.

[0129] To verify the advantages of the method in this application, experiments are conducted below.

[0130] 1. Dataset

[0131] We evaluate the proposed framework on three benchmark datasets widely used to evaluate multi-turn dialogue recommendation systems.

[0132] •LastFM*: The LastFM dataset is used for music artist recommendations. Lei et al. manually classified the original attributes in LastFM into 33 coarse-grained groups. Later, Lei et al. believed that manually merging attributes was not necessarily the best practice in practical applications, so they reconstructed the dataset using the original attributes, thus forming LastFM*.

[0133] • Amazon-Book: Amazon consists of product reviews and metadata from Amazon. In our experiments, we used the Books category from the Amazon dataset.

[0134] MovieLens: MovieLens is a movie rating dataset.

[0135] 2. Experimental Setup

[0136] (1) User simulator

[0137] Taking the KEMCR scenario as an example, a user simulation strategy similar to MIMCR was adopted. During the virtual dialogue interaction, the simulated user follows a series of rules to mimic the feedback behavior of a real user. For a given observed user-item interaction pair (u, V... u We simulate a dialogue. For example, when the system asks a question, the simulated user will respond with V.u The system selects project-related attribute instances and rejects irrelevant ones; when the system recommends a list of projects to the user, it only considers instances containing V. u The simulated user will accept at least one item; however, a maximum number of rounds T is set to limit the simulated user's patience. Once this limit is reached, the simulated user will exit the system unless they accept the recommended list of items or lose interest in continuing to interact.

[0138] (2) Baseline

[0139] To evaluate the performance of our model, we compared it with the following seven representative baselines:

[0140] Abs Greedy's method does not ask for attributes during the conversation, but recommends items to the user in each round until the user accepts the recommendation.

[0141] Max Entropy is a rule-based query and recommendation strategy. It selects the attribute to query based on the maximum entropy criterion, or recommends top-ranked items with a certain probability.

[0142] The CRM method was originally designed for single-turn dialogue recommendations. It uses policy gradients

[47] to learn policies that determine when and which attributes to ask.

[0143] EAR is a reinforcement learning-based approach that uses the attributes that users are most likely to like for queries.

[0144] SCPR reformulates the CRS problem as an interaction path reasoning problem on a user-item-attribute graph. Candidate attributes and items are selected based on their relationship to user feedback collected on the graph.

[0145] UNICORN employs a reinforcement learning approach based on dynamically weighted graphs to integrate dialogue and recommendation components, and deploys a weighted entropy action selection strategy to reduce the candidate action space.

[0146] MIMCPL is a state-of-the-art CRS method that follows a graph-based unified reinforcement learning framework and employs a multi-interest encoder to learn the dialogue state. It extends the unified dialogue recommendation strategy by incorporating the user's multi-interest representation, taking into account the user's multi-interest space and extending MCR scenarios to more realistic MIMCR scenarios.

[0147] (3) Parameter settings

[0148] We divided each dataset into training, validation, and test sets in a 7:1.5:1.5 ratio. The node embedding dimension was set to 64, and the batch size to 128. In each round, the system either asks the user for two attributes or recommends 10 items, with a maximum round size T of 15. We used TransE from OpenKE to pre-train the node embeddings in the graph using the training set. The number of layers is set to 2. The number of layers is set to 1. We use the Adam optimizer with a learning rate of 1e-4 and a discount factor γ of 0.999. For the operation space A... t We selected the top 10 Ps cand and top 10 V cand During DQN training, we use r rec_suc =1, r rec_fail =-0.1, r ask_suc =0.01, r rec_fail =0.1, r over = -0.3 The same reward setting ensures fair comparison with other benchmarks.

[0149] (4) Evaluation indicators

[0150] We follow the metrics widely used in previous work to evaluate MCR. SR@T measures the percentage of successful recommendations within T rounds. The higher the SR@T, the better the performance. AT measures the average number of rounds in a conversation. A lower AT indicates higher efficiency. In addition, we also use hNDCG@(T,K) to comprehensively evaluate MCR, extending the normalized depreciation cumulative gain (NDCG@K) into a two-level stratified version, defined as shown in Equation (26).

[0151]

[0152] 3. Performance Comparison

[0153] Table 1 reports the SR@15, AT, and hDCG@(15,10) scores of KEUPPL and the baseline models. Analysis of the experimental results shows that Abs Greedy, Max Entropy, and CRM lack the ability to handle complex user interests and relationships; in complex recommendation tasks, these models may not be able to fully tap the potential of the data. EAR and SCPR may focus more on utilizing local information while neglecting the importance of global context; this locality may lead to insufficient information when dealing with large-scale knowledge graphs and multi-type data. While UNICORN and MCMIPL consider global user information, they fail to handle dynamic user preferences well, resulting in insufficient recommendation accuracy. Overall, our model achieves significant performance improvements on different datasets, especially in SR@15 and AT. This demonstrates the broad applicability of our method in recommender systems, providing better recommendation results and proving its effectiveness in the KEMCR scheme.

[0154] Table 1. Experimental results. Higher SR@15 and hDCG values ​​are better, while lower AT values ​​are better.

[0155]

[0156]

[0157] We performed a total of 100 iterations. Figure 4 , Figure 5 , Figure 6 The results of our model and MCMIPL on three datasets are presented. Based on the graphs and data performance, the following conclusions can be drawn: as the number of iterations increases, our model increasingly demonstrates its stability compared to MCMIPL. Specifically, in the LastFM* dataset, our model's SR@15 is consistently higher than MCMIPL, indicating that our model performs better in terms of recommendation success rate. In the Amazon-Book dataset, our model and MCMIPL have many overlapping parts. Although our model performs worse than MCMIPL at 50 and 60 iterations, overall, our model is more stable, with smaller differences between the maximum and minimum values ​​and a generally upward trend. The final average SR@15 is also higher than MCMIPL. Finally, in the MovieLens dataset, the stability of our model is even more evident. In contrast, as the number of iterations increases, MCMIPL's SR@15 fluctuates, and its overall graph is lower than our model's.

[0158] The features of this application are:

[0159] 1. We introduce a new multi-turn dialogue learning scenario, KEMCR, in which the system accurately captures multiple user preferences by making reasonable use of user dialogue information, overcoming the shortcomings of previous MCR scenarios.

[0160] 2. We propose a novel user preference policy learning framework, KEUPPL. It utilizes RGCN to encode user-related external knowledge, enhancing user data representation. Then, it employs a multi-head self-attention mechanism and a GRU with an added attention update gate to extract user preferences, avoiding the interference of interest drift and effectively modeling the long-term evolution of user preferences. Finally, an improved DQN is introduced for action decision-making, selecting the optimal next action.

[0161] Obviously, the specific implementation of this invention is not limited to the above-described methods. Any non-substantial improvements made using the inventive concept and technical solution of this invention are within the protection scope of this invention.

Claims

1. A multi-turn dialogue recommendation method based on RGCN and reinforcement learning, characterized by: This involves transforming user dialogue information into a knowledge graph representation and using a relational graph neural network (RGCN) to obtain node embeddings of the knowledge graph; employing multi-head self-attention to obtain good item representations to better connect the feature embeddings of each item; then using another multi-head self-attention mechanism to capture multiple user preferences and using a GRU with an attention update gate to overcome the interference of interest drift; finally, introducing reinforcement learning (RL) into CRS to learn dialogue recommendation strategies, determining the attributes to be asked, the recommended items, and when to ask or recommend in each dialogue round; and using an improved adversarial Q-network (DQN) for action decision-making to select the optimal next action. The user's current and historical dialogue information is converted into a dialogue graph, and a knowledge graph MDP environment is constructed. Knowledge graph-enhanced state representation learning is used to integrate external knowledge graphs into the agent's state representation to improve its performance in complex tasks. It includes two main tasks: encoding and refining user behavior and extracting and evolving user preferences. The coding and refinement of user behavior includes: RGCN is used to embed nodes and edges in graph data to enable machine learning models to better understand and process them. Through processing, the node embedding representations corresponding to the current dialogue graph and the historical dialogue graph are obtained, namely: the current weighted knowledge graph representation based on RGCN and the global heterogeneous knowledge graph representation based on RGCN. A multi-head self-attention mechanism is introduced to enhance the model's learning of the interaction relationships between different inputs. It maps the node representation to a higher-dimensional space so that the model can better capture the dependencies between different nodes. The multi-head self-attention mechanism is used to learn and represent information in the knowledge graph so that the model can understand the user's needs and preferences. The extraction and evolution of user preferences includes: encoding user behavior and refining the output results; extracting user preference information through a second-layer multi-head self-attention mechanism; capturing multiple user preferences for items and the relationships between items; then introducing a GRU with an attention update gate to further evolve user preferences; by introducing an update gate with an attention mechanism, the attention mechanism is seamlessly integrated with the GRU to more finely control and adjust the evolution of user preferences; finally, the user preference embedding representation is output. Improved Adversarial Q-Networks (DQNs) for action decision-making include: In reinforcement learning, the Q-value can be decomposed into a state-value function and an advantage function, allowing the network to more flexibly estimate the impact of each action, thereby improving learning efficiency. Under the Dueling Q-Network, a Q-value function for policy π is defined. ( , To measure state-based and actions Cumulative rewards; and embedding { based on previously learned user preferences. ,..., } to calculate in state Next, action The scores between these user preferences are used, and the maximum value among them is taken as the final Q-score; where functions and The functions represent the state value function and the dominance function, respectively; ,k {1,...,H}; The Q-function satisfies the Bellman equation, which expresses the relationship between the optimal Q-value and the current state, action, and next state. Therefore, by solving the Bellman equation, the optimal Q-function can be obtained, thereby determining the optimal action strategy. To maximize the cumulative reward, the optimal Q-function is defined as follows: ( , The calculation method is as follows: ； It is a state Take action The instant reward obtained afterward It is a discount factor used to weigh the importance of current rewards and future rewards. It is a value between 0 and 1 that represents the degree of decay of future rewards.

2. The multi-turn dialogue recommendation method based on RGCN and reinforcement learning as described in claim 1, characterized in that: The knowledge graph-based MDP environment provides tasks and feedback, knowledge graph-enhanced state representation learning helps to understand the environment, action selection strategies determine the agent's behavior, and Dueling Q-Network is used to learn and optimize behavior, enabling the agent to achieve the best results in the environment.

3. The multi-turn dialogue recommendation method based on RGCN and reinforcement learning as described in claim 2, characterized in that: The knowledge graph MDP environment includes states, actions, transitions, and rewards. These components interact to influence the agent's learning and behavior.

4. The multi-turn dialogue recommendation method based on RGCN and reinforcement learning as described in claim 2, characterized in that: Action selection strategies involve efficiently finding the optimal action within a vast action space to improve the effectiveness and efficiency of recommendations, based on the coarse-match score of item v in the current round. To further filter the action space, following the same calculation logic, we obtain the coarse matching score for the current round's attribute p. Finally, the candidate items were selected based on these two scores. and candidate attribute set The top N items and top N attributes are selected and placed into the action space for the next recommendation step.

5. The multi-turn dialogue recommendation method based on RGCN and reinforcement learning as described in any one of claims 1-4, characterized in that: In each round, the agent interacts with the user, and the user provides rewards. This reward is a form of feedback, telling the agent how it performed in the previous action or state. Positive rewards encourage the agent to continue taking similar actions, while negative rewards encourage the agent to avoid those actions. Based on user feedback, the environment enters a new state and a new action space after the agent takes an action. A replay buffer D is defined; the purpose of this buffer is to collect and store experience samples gained by the agent in its interaction with the user. Experience replay is a technique that randomly samples small batches of experience samples during training to help make training more stable and efficient. Finally, the agent uses these experience samples to train the model, updating the model's parameters by minimizing a loss function. A loss function is defined as follows: ； here , Let these represent the dominance function and the action function, respectively. This refers to obtaining the set of parameters embedded in user preferences. The target value, calculated using the optimal Q-function, represents the immediate reward for the current round. Add the product of the maximum predicted Q value for the next round: 。

Images