Improving dialogue recommendation system by multi-preference modeling and knowledge enhancement

Abstract

The application provides a dialogue recommendation system improved through multi-preference modeling and knowledge enhancement, comprising a semantic fusion module, a recommendation module or / and a dialogue module; a data output first end of the semantic fusion module is connected with a data input end of the recommendation module, and a data output second end of the semantic fusion module is connected with a data input end of the dialogue module; the semantic fusion module is used for fusing external knowledge and information of dialogue text; the recommendation module is used for providing recommendation by combining the preferences of a user from one or any combination of text, entity and image; the dialogue module is used for generating fluent and diverse replies by combining the recommendation result through designing an explanation template or / and highlighting dialogue keywords. According to the multi-aspect preferences of a user and external knowledge, the application improves the accuracy of system recommendation and the quality and diversity of system replies.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of recommendation systems and the technical field of dialogue systems, and particularly relates to a dialogue recommendation system improved through multi-preference modeling and knowledge enhancement. BACKGROUND

[0002] With the booming development of intelligent assistants and major e-commerce platforms, conversational recommender systems (CRS) have become an emerging research field. Traditional recommender systems rely on historical interaction data between users and items to obtain recommendation results, while CRS focuses on generating natural language to interact with users and gradually learning user preferences from it, and finally providing higher quality recommendations.

[0003] A successful CRS needs to understand user intent and user preferences, and then use appropriate sentences to reply to the user. As we can see from Figure 1 , the system uses natural and fluent sentences to ask the user and recommends items to the user after understanding the user's interests. Therefore, CRS usually contains a recommendation module and a dialogue module, the former is used to generate items that meet the user's preferences, and the latter is used to generate casual or recommended sentences.

[0004] CRS aims to complete the recommendation with fewer dialogue rounds, and the information that can be provided by the short dialogue history is often insufficient. Existing research mainly regards the movie entities involved in the dialogue history as user preferences (we call it entity preference), which come from the external knowledge graph (KG) after the entity linking task. By combining external knowledge, the representation of the user can be effectively enhanced, and the problem of insufficient information can be alleviated. However, for external KG, existing research almost always extracts a related domain (movie) subgraph from the large knowledge graph DBpedia. Although this way of implementation is convenient, there are two problems: 1) Since DBpedia contains many domains, the extracted subgraph will have noise from non-related domains, which to some extent hinders the CRS from utilizing external knowledge; 2) DBpedia does not cover the items in the CRS dataset well, so many items cannot be matched to the entities in the subgraph. In addition, even if the external knowledge is introduced to enrich the dialogue history, the number of entities mentioned in the user's utterance is still scarce, and the small number of entities makes it difficult for the model to accurately learn the user's preferences. On the other hand, using entities as the only preference of the user is one-sided, and this approach does not take into account the semantic information hidden in the user's utterance. For example, Figure 1As shown, the user expressed a negative opinion about the movie "The Avengers" initially recommended by the system. This is actually a hidden user intent (what we call semantic preference), but in the past, relying solely on entity preferences ignored the user's true intent, assuming the user liked the movie. In reality, user preferences for items (movies) should be multifaceted. For example, liking a movie isn't solely due to its content; it might also be influenced by its genre or director (what we call attribute preference). As... Figure 1 In the context of CRS (Comprehensive Scripting System), users might accept the recommendation "Interstellar (2014)" because the film is science fiction or directed by "Nolan." However, user discourse rarely mentions movie-related attributes. Although knowledge graphs contain such attribute relationships, they are only implicitly expressed, thus requiring a more explicit approach. For the dialogue module, the goal is to generate fluent and diverse responses. However, simple recommendation statements dominate the dialogue history of the training data, and some specific keywords in the dialogue have not received sufficient attention in existing research. This leads to the dialogue modules of existing CRS often generating rather monotonous and uninteresting responses, making it difficult to generate diverse responses that help explain or describe the recommendation results. Summary of the Invention

[0005] This invention aims to at least solve the technical problems existing in the prior art, and in particular, innovatively proposes an improved dialogue recommendation system through multi-preference modeling and knowledge enhancement.

[0006] To achieve the above-mentioned objectives of the present invention, the present invention provides an improved dialogue recommendation system through multi-preference modeling and knowledge enhancement, including a semantic fusion module, a recommendation module and / or a dialogue module;

[0007] The first output of the semantic fusion module is connected to the data input of the recommendation module, and the second output of the semantic fusion module is connected to the data input of the dialogue module.

[0008] The semantic fusion module is used to fuse external knowledge with information from the dialogue text;

[0009] The recommendation module is used to provide recommendations by combining user preferences from one or more of the following: text, entities, images, or any combination thereof. The recommendation module will utilize a large pre-trained model to provide recommendations.

[0010] The dialogue module is used to generate fluent and diverse responses by designing explanation templates and / or highlighting dialogue keywords, and by combining the recommendation results.

[0011] In a preferred embodiment of the present invention, the semantic fusion module includes an information encoding unit and a semantic fusion unit;

[0012] The data output of the information encoding unit is connected to the data input of the semantic fusion unit;

[0013] The information encoding unit is used for information encoding;

[0014] Semantic fusion units are used for semantic fusion.

[0015] In a preferred embodiment of the present invention, the information encoding unit includes:

[0016]

[0017] in, This represents the entity representation of entity e at level l+1.

[0018] σ(·) represents ReLU activation;

[0019] This represents the set of relations between entities e and t.

[0020] Represents the set of adjacent nodes of entity e under relation r;

[0021] Z e,r Indicates the normalization factor;

[0022] This represents the first learnable matrix;

[0023] This represents the second learnable matrix;

[0024] This represents the representation of the adjacent nodes of entity e at level l.

[0025] This represents the entity representation of entity e at level l.

[0026]

[0027] in, This represents the representation of word v at level l+1;

[0028] σ(·) represents ReLU activation;

[0029] D represents the angle matrix;

[0030] A represents the adjacency matrix of the graph;

[0031] This represents the representation of word v at level l.

[0032] This represents the transformation matrix of word v at level l.

[0033] In a preferred embodiment of the present invention, the semantic fusion unit includes:

[0034] T e,c =E e T W1E c ,

[0035] Among them, T e,c An affinity matrix representing the relationship between project entities and text;

[0036] E e Represents the entity embedding matrix;

[0037] E e T Represents the entity embedding matrix E e The transpose of the matrix;

[0038] W1 represents the first transformation matrix;

[0039] E c Represents the context embedding matrix;

[0040] T c,v =E c T W2E v ,

[0041] Among them, T c,v An affinity matrix representing the relationship between word entities and text;

[0042] E c Represents the context embedding matrix;

[0043] E c T Represents the context embedding matrix E c The transpose of the matrix;

[0044] W2 represents the second transformation matrix;

[0045] E v Represents a word embedding matrix;

[0046] E e,c =E c T e,c +E e ,

[0047] E e,c This represents the embedded entity after fusion;

[0048] E c Represents the context embedding matrix;

[0049] T e,c An affinity matrix representing the relationship between project entities and text;

[0050] E e Represents the entity embedding matrix;

[0051] E c,v =E v T c,v +E c ,

[0052] Among them, E c,v This indicates the word embedding after fusion;

[0053] E v Represents a word embedding matrix;

[0054] T c,v An affinity matrix representing the relationship between word entities and text;

[0055] E c This represents the context embedding matrix.

[0056] In a preferred embodiment of the present invention, the recommendation module includes:

[0057]

[0058] in, This represents the feature embedding matrix after fusion during the exploration phase;

[0059] E e,c This represents the embedded entity after fusion;

[0060] E c Represents the context embedding matrix;

[0061] E p-rec The characteristics of the prompt template for the recommendation task;

[0062]

[0063] in, Let represent the set of recommended results for the t-th round of exploration.

[0064] DialoGPT rec (·) represents the pre-trained model DialoGPT with frozen parameters under the recommendation task;

[0065] This represents the feature embedding matrix after fusion during the exploration phase;

[0066] E a ,I m =Lookup(m top ),

[0067] Among them, E a Represents an attribute entity;

[0068] I m Represents the original poster image;

[0069] Lookup() indicates searching for movie m top Related attributes;

[0070] m top This indicates the movie with the highest probability in the recommended result set.

[0071] In a preferred embodiment of the present invention, the recommendation module further includes:

[0072] E i =Darknet53(I m ),

[0073] Among them, E i Represents the image feature matrix;

[0074] Darknet53(·) represents a pre-trained image feature extraction model;

[0075] I m Represents the original poster image;

[0076]

[0077] in, This represents the feature embedding matrix after fusion during the integration phase;

[0078] E e,c This represents the embedded entity after fusion;

[0079] E c Represents the context embedding matrix;

[0080] E p-rec The characteristics of the prompt template for the recommendation task;

[0081] E a Represents an attribute entity;

[0082]

[0083] This represents user preferences after cross-modal fusion;

[0084] This represents the feature embedding matrix after fusion during the integration phase;

[0085] CA(·) represents a cross-modal attention layer;

[0086] E i Represents the image feature matrix;

[0087]

[0088] in, This represents the final set of recommendation results;

[0089] DialoGPT rec (·) represents the pre-trained model DialoGPT with frozen parameters under the recommendation task;

[0090] This represents user preferences after cross-modal fusion.

[0091] In a preferred embodiment of the present invention, the dialogue module includes:

[0092] E gen =E c,v +E tc +E p-gen ,

[0093] Among them, E gen This represents the fused feature embedding matrix;

[0094] E c,v This indicates the word embedding after fusion;

[0095] E tc This indicates the context embedding after the fusion interpretation template;

[0096] E p-gen The cue card features that indicate a dialogue task;

[0097] s t =DialoGPT gen (E gen ),

[0098] Among them, s t This represents the reply sentence generated in round t;

[0099] DialoGPT gen (·) represents the pre-trained model DialoGPT with frozen parameters in the dialogue task;

[0100] E gen This represents the feature embedding matrix after fusion.

[0101] In a preferred embodiment of the present invention, a knowledge graph module is also included, which is used to introduce external knowledge.

[0102] In a preferred embodiment of the present invention, an indicator evaluation module is further included, which is used to evaluate the recommendation accuracy and / or fluency and / or diversity of the dialogue recommendation system.

[0103] In summary, by adopting the above technical solutions, this invention improves the accuracy of system recommendations and the quality and diversity of system responses based on users' various preferences and external knowledge.

[0104] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0105] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0106] Figure 1 This is an example diagram of the dialogue recommendation in this invention.

[0107] Figure 2 This is a schematic block diagram of the overall architecture of the model of this invention.

[0108] Figure 3 This is a sample schematic sub-graph of an entity knowledge graph (LMKG) of the present invention.

[0109] Figure 4 This is a schematic diagram of the ablation experiment related to the recommendation task in this invention. Detailed Implementation

[0110] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0111] 1 Introduction

[0112] With the rapid development of intelligent assistants and major e-commerce platforms, conversational recommender systems (CRS) have become an emerging research field. Traditional recommender systems rely on historical interaction data between users and items to obtain recommendation results, while CRS focuses on generating natural language to interact with users and gradually learn user preferences, ultimately providing higher-quality recommendations.

[0113] A successful CRS requires understanding the user's intent and preferences, and then responding to the user with appropriate statements. As we learned from... Figure 1 As seen in the diagram, the system uses natural and fluent language to ask users questions and recommends items to them after understanding their interests. Therefore, a CRS typically includes a recommendation module and a dialogue module; the former generates items that match the user's preferences, while the latter generates casual or recommending statements.

[0114] CRS aims to complete recommendations with fewer dialogue turns, but short dialogue histories often provide insufficient information. Existing research mainly uses movie entities mentioned in the dialogue history as user preferences (which we call entity preferences). These entities come from an external knowledge graph (KG) after an entity linking task. By combining external knowledge, user representation can be effectively enhanced, alleviating the problem of insufficient information. However, for external KGs, existing research almost always extracts relevant domain (movie) subgraphs from large knowledge graphs like DBpedia. While this approach is convenient, it has two problems: 1) Since DBpedia contains many domains, the extracted subgraphs contain noise from irrelevant domains, which hinders CRS from utilizing external knowledge to some extent; 2) DBpedia does not cover the items in the CRS dataset well, so many items cannot be matched with entities in the subgraph. Furthermore, even by introducing external knowledge to enrich the dialogue history, the number of entities mentioned in user utterances remains scarce, making it difficult for the model to accurately learn user preferences. On the other hand, using only entities as the sole user preference is one-sided, as this approach does not consider the semantic information hidden in user utterances. Figure 1 As shown, the user expressed a negative opinion about the movie "The Avengers" initially recommended by the system. This is actually a hidden user intent (what we call semantic preference), but in the past, relying solely on entity preferences ignored the user's true intent, assuming the user liked the movie. In reality, user preferences for items (movies) should be multifaceted. For example, liking a movie isn't solely due to its content; it might also be influenced by its genre or director (what we call attribute preference). As... Figure 1 In the context of CRS (Comprehensive Scripting System), users might accept the recommendation "Interstellar (2014)" because the film is science fiction or directed by "Nolan." However, user discourse rarely mentions movie-related attributes. Although knowledge graphs contain such attribute relationships, they are only implicitly expressed, thus requiring a more explicit approach. For the dialogue module, the goal is to generate fluent and diverse responses. However, simple recommendation statements dominate the dialogue history of the training data, and some specific keywords in the dialogue have not received sufficient attention in existing research. This leads to the dialogue modules of existing CRS often generating rather monotonous and uninteresting responses, making it difficult to generate diverse responses that help explain or describe the recommendation results.

[0115] To address the aforementioned issues, we propose a novel model: Improving Conversational Recommender Systems via Multi-Preference Modeling and Knowledge-Enhanced (MPKE). First, to reduce interference from other domains and improve item coverage, we construct a lightweight movie knowledge graph (LMKG). Simultaneously, we use RoBERTa and R-GCN to obtain semantic preferences from the text and entity preferences from the knowledge graph, and design a semantic fusion module to align different semantic spaces. Next, we propose a two-stage recommendation mechanism (exploration phase and integration phase) to address the issues of the limited number of entities and scarce item attributes in the dialogue history. In the exploration phase, we use only semantic and entity preferences to obtain recommendation results and find relevant attributes. Notably, we also incorporate movie posters as a visually aiding attribute preference. In the integration phase, we fuse semantic, entity, and attribute preferences to generate new recommendation results. For response generation, we construct slotted explanation templates and fuse them with the original dialogue, helping the system generate more descriptive statements. Meanwhile, we leverage the word-level knowledge graph ConceptNet in the fusion module to enhance the representation of dialogue keywords, enriching the diversity of system responses. Extensive experiments on benchmark datasets demonstrate that MPKE outperforms state-of-the-art CRS models in both recommendation accuracy and response quality.

[0116] The main contributions of this patent application are as follows: 1) We construct a low-noise, high-quality lightweight movie knowledge graph (LMKG); 2) We design a semantic fusion module and a two-stage recommendation mechanism to explore and integrate multiple user preferences; 3) We use word-level knowledge graphs to enhance keyword representation and construct explanatory statement templates to improve the quality and diversity of system responses.

[0117] 2. Related work

[0118] With the rapid development of dialogue and recommendation systems, dialogue recommendation systems have become an emerging research topic. Their goal is to capture user preferences and provide high-quality recommendations using natural language within a limited number of dialogue turns. One type is attribute-based recommendation systems (CRS), which focus solely on recommendation without prioritizing dialogue generation. These systems typically use predefined, slotted response templates for user interaction and generally employ multi-armed robber models or reinforcement learning to find the optimal interaction strategy.

[0119] Recently, some research has focused on open-ended CRS, which can accurately recommend and generate fluent utterances. Unlike attribute-based CRS that builds response templates, open-ended CRS is an end-to-end model that integrates recommendation results into response statements using transformation mechanisms or copy networks.

[0120] While existing research has yielded some results, it has only superficially considered movie entities mentioned in user discourse as user preferences, neglecting the fact that user preferences are multifaceted. When a user talks about a favorite movie, the model should not only consider the movie itself but also pay attention to the implicit information in the user's discourse and related movie attributes (director, movie poster, etc.). Furthermore, existing research focuses more on the model's recommendation performance, but for open-ended CRS, generating fluent and diverse response statements is also an important goal. Therefore, we propose MPKE to model user multi-preferences and leverage knowledge augmentation to improve the model's recommendation and dialogue performance.

[0121] 3. Problem Definition

[0122] Dialogue-based recommendation systems aim to provide accurate recommendations with fewer multi-turn conversations with users. In a single conversation, the system either inquires about the user's preferences or provides recommendations based on learned user preferences. This process continues until the user accepts the recommendation or exits due to impatience. Typically, a CRS consists of a recommendation module and a dialogue module, which handle the recommendation and response generation tasks respectively.

[0123] In terms of form, we use This represents a user in the user set. This represents a movie from a collection of movies. Information such as movie titles or names mentioned in user utterances will be represented by entities e∈ε, which come from a specific knowledge graph and are part of the user's preferences. A dialogue It typically contains n rounds, where s i This represents the sentences in the i-th round of dialogue, and each sentence... It consists of k words The sequence is composed of [various elements]. In round t, the model will [comprise elements] based on the dialogue history. Learn user preferences from the information in the movie collection. Select the recommended subset Finally, a movie is generated. reply statement s t It is important to note that the recommended subset... It might be an empty set. At this point, the model generates casual conversation statements.

[0124] 4 methods

[0125] In this section, we propose a CRS model, MPKE, that improves performance by modeling user multiple preferences and knowledge enhancement. The overall model architecture is as follows: Figure 2 As shown (we first fuse and align the semantics of the knowledge graph and the context, then use a two-stage recommendation mechanism and incorporate user multiple preferences to obtain the recommendation results. Next, we use a knowledge-enhanced dialogue module to generate responses and populate the slots with the recommendation results). First, we construct a higher-quality entity-level knowledge graph (LMKG) (Section 4.1). Then, we fuse external knowledge with the information in the dialogue text through a semantic fusion module (Section 4.2). Next, we design a two-stage recommendation mechanism to explore and incorporate user multiple preferences (Section 4.3) to improve the performance of the recommendation module. Finally, we utilize knowledge enhancement and explanation templates to improve the performance of the dialogue module (Section 4.4).

[0126] 4.1 Constructing a Knowledge Graph (LMKG)

[0127] Previous work often incorporates external knowledge, such as the knowledge graph DBpedia, to enrich dialogue history and enhance user representation. However, DBpedia, as a large multi-domain knowledge graph, contains many irrelevant entities and fuzzy high-order links. Although previous work extracted subgraphs from DBpedia, significant noise from other domains remains. Furthermore, the subgraphs do not adequately cover the movies in the CRS dataset, resulting in many movies in the dialogue not matching entities in the subgraphs. To address this, we construct a higher-quality lightweight movie knowledge graph (LMKG) for the CRS domain, with sample subgraphs as shown below. Figure 3 As shown (red text indicates movies, black text indicates movie attributes, and gray text indicates relationships).

[0128] Table 1 Comparison of LMKG and DBpedia subplots

[0129]

[0130]

[0131] LMKG has a much higher number of valid entities than DBpedia.

[0132] First, we used the movies and their release dates in the dataset as keywords to collect information about the relevant movies (e.g., directors, actors) from Wikipedia and the movie website IMDb. Next, we organized the collected information and stored it in the form of triples (h, r, t), where h, t ∈ ε represent the head and tail entities. The relationship between the two is indicated. Entities (nodes) in LMKG are mainly divided into two categories: movie entities and attribute entities, where attributes include actors, directors, screenwriters, and categories. Unlike the DBpedia subgraph, we emphasize the validity of entities when constructing LMKG. A valid movie entity is one that has at least one attribute, while a valid attribute entity is one that has at least two movies possessing that attribute. An invalid movie or attribute entity is isolated, and such isolated entities also cause interference during subsequent graph aggregation learning. Table 1 shows the differences between LMKG and the DBpedia subgraph. Although the DBpedia subgraph has a large number of entities, its number of valid entities is far lower than that of LMKG, indicating that the DBpedia subgraph has some interference. Furthermore, since LMKG is specifically built for the CRS domain, it has a higher coverage of movies in the CRS dataset. This also reflects the lightweight and high-quality nature of LMKG.

[0133] 4.2 Semantic Fusion Module

[0134] Because of the significant semantic gap between knowledge graphs and dialogue texts, we need to merge their semantic spaces to make more effective use of external knowledge.

[0135] 4.2.1 Information Encoding

[0136] For knowledge graphs LMKG (hereinafter referred to as LMKG) Therefore, we must consider the semantic relationships between entities. Thus, we employ R-GCN for encoding. And obtain the entity embedding dictionary Specifically, at level (l+1), entity e is... The representation in is defined as:

[0137]

[0138] in, This represents the entity representation of entity e at level l+1.

[0139] σ(·) represents the ReLU activation function;

[0140] Represents the set of relations for entity e;

[0141] Represents the set of adjacent nodes of entity e under relation r;

[0142] Z e,r Indicates the normalization factor;

[0143] This represents the learnable matrix of relation r at the l-th layer;

[0144] This represents the learnable matrix of entity e at layer l;

[0145] This represents the representation of the adjacent nodes of entity e at level l.

[0146] This represents the entity representation of entity e at level l.

[0147] For the knowledge graph ConceptNet (hereinafter referred to as...) We removed words irrelevant to the dialogue corpus. Furthermore, since the relationships between words have no impact on subsequent tasks, we chose to use GCN for encoding. Each time an update is performed, GCN will accept... The information of the single-hop neighbor is retrieved, and the following aggregation operation is performed:

[0148]

[0149] in, This represents the representation of word v at level l+1;

[0150] σ(·) represents the ReLU activation function;

[0151] D represents the angle matrix;

[0152] A represents the adjacency matrix of the graph;

[0153] This represents the representation of word v at level l.

[0154] This represents the transformation matrix of word v at level l;

[0155] We use the semantic information in the dialogue text as the user's semantic preferences in the recommendation module, and in the dialogue module to help generate responses. To effectively extract text features, we use a pre-trained language model, RoBERTa, with frozen parameters. Based on the given context C, after performing entity linking and text encoding, we can obtain the corresponding entity embedding matrix E. e Context embedding matrix E c and word embedding matrix E v .

[0156] 4.2.2 Semantic Fusion

[0157] To mitigate the semantic discrepancies between the dialogue text and entities in the knowledge graph, we use a cross-interaction mechanism to link their semantic representations, as follows:

[0158] T e,c =Ee T W1E c (3)

[0159] Among them, T e,c An affinity matrix representing the relationship between project entities and text;

[0160] E e Represents the entity embedding matrix;

[0161] E e T Represents the entity embedding matrix E e The transpose of the matrix;

[0162] W1 represents the first transformation matrix;

[0163] E c Represents the context embedding matrix;

[0164] T c,v =E c T W2E v (4)

[0165] Among them, T c,v An affinity matrix representing the text and word entities;

[0166] E c Represents the context embedding matrix;

[0167] E c T Represents the context embedding matrix E c The transpose of the matrix;

[0168] W2 represents the second transformation matrix;

[0169] E v Represents a word embedding matrix;

[0170] E e,c =E c T e,c +E e (5)

[0171] E e,c This represents the embedded entity after fusion;

[0172] E c Represents the context embedding matrix;

[0173] T e,c An affinity matrix representing the relationship between project entities and text;

[0174] E e Represents the entity embedding matrix;

[0175] E c,v =E v T c,v +E c (6)

[0176] Among them, E c,v This indicates the word embedding after fusion;

[0177] E v Represents a word embedding matrix;

[0178] T c,v An affinity matrix representing the text and word entities;

[0179] E c Represents the context embedding matrix;

[0180] In subsequent tasks, the model uses the enhanced entity embeddings and word embeddings provided by the above formulas. However, since there are some additional parameters in the semantic fusion module, we use self-supervised signals from the dialogue to pre-train the fusion module, thereby optimizing the additional parameters.

[0181] 4.3 Preference-Enhanced Recommendation Module

[0182] In previous studies, recommendation modules primarily used movie entities from the dialogue history to represent user preferences. However, the number of movie entities is often sparse, leading to inaccurate capture of user preferences. Furthermore, user utterances often contain implicit additional information, making the representation of user preferences solely by movie entities incomplete. To address this, we first encode the dialogue text using a pre-trained model to obtain semantic preferences hidden within user utterances (Section 4.2.1). Next, we propose a two-stage recommendation mechanism (exploration phase and integration phase) for the first time to acquire user attribute preferences, compensating for the insufficient number of entities. Additionally, we consider visual information (movie posters) in item attributes as an auxiliary preference. Therefore, strictly speaking, attribute preferences consist of item attribute entities and item images.

[0183] 4.3.1 Exploration Phase

[0184] Since the recommendations generated by the model can satisfy user preferences to a certain extent, entities with related attributes can also be considered part of user preferences. In the exploration phase, we only embed entities representing entity preferences into E. e,c and context embedding E representing semantic preferences cAs model input, we then use DialoGPT as the base pre-trained model to decode user preferences and obtain initial recommendation results. DialoGPT employs an autoregressive Transformer architecture and is trained on a large-scale dialogue corpus, making it a very suitable base model for CRS tasks. However, such pre-trained language models often need to be adjusted according to downstream tasks, so we use prompt templates to guide DialoGPT to adapt to the CRS recommendation task. The specific process is as follows:

[0185]

[0186] in, This represents the feature embedding matrix after fusion during the exploration phase;

[0187] E e,c This represents the embedded entity after fusion;

[0188] E c Represents the context embedding matrix;

[0189] E p-rec The characteristics of the prompt template for the recommendation task;

[0190]

[0191] in, Let represent the set of recommended results for the t-th round of exploration.

[0192] DialoGPT rec (·) represents the pre-trained model DialoGPT with frozen parameters under the recommendation task;

[0193] This represents the feature embedding matrix after fusion during the exploration phase;

[0194] Next, the model will retrieve the relevant attributes of the initial recommendation results (such as director, actors, etc.) and the movie poster. The specific process is as follows:

[0195] E a ,I m =Lookup(m top (9)

[0196] Among them, E a Represents an attribute entity;

[0197] I m Represents the original poster image;

[0198] Lookup() indicates searching for movie m top Related attributes;

[0199] m top This indicates the movie with the highest probability in the recommended result set.

[0200] 4.3.2 Integration Phase

[0201] During the integration phase, we will combine user preferences from multiple sources, including project entities, attribute entities, textual semantics, and visual images, with visual information serving only as a secondary, auxiliary preference. For movie poster I... m We extract image features using Darknet53 and fuse them with other features using Cross-modal Attention (CA). The specific process is as follows:

[0202] E i =Darknet53(I m (10)

[0203] Among them, E i Represents the image feature matrix;

[0204] Darknet53(·) represents a pre-trained image feature extraction model;

[0205] I m Represents the original poster image;

[0206]

[0207] in, This represents the feature embedding matrix after fusion during the integration phase;

[0208] E e,c This represents the embedded entity after fusion;

[0209] E c Represents the context embedding matrix;

[0210] E p-rec The characteristics of the prompt template for the recommendation task;

[0211] E a Represents an attribute entity;

[0212]

[0213] This represents user preferences after cross-modal fusion;

[0214] This represents the feature embedding matrix after fusion during the integration phase;

[0215] CA(·) represents a cross-modal attention layer;

[0216] E i Represents the image feature matrix;

[0217]

[0218] in, This represents the final set of recommendation results;

[0219] DialoGPT rec (·) represents the pre-trained model DialoGPT with frozen parameters under the recommendation task;

[0220] This represents user preferences after cross-modal fusion;

[0221] Formula (12) indicates that the features obtained after cross-modal fusion are only used as auxiliary preferences. The specific calculation process of CA(·) is as follows:

[0222]

[0223] Where CA(·) represents a cross-modal attention layer;

[0224] softmax(·) represents the activation function;

[0225] d represents the embedding dimension;

[0226] That is

[0227] Where d represents the embedding dimension.

[0228] 4.4 Knowledge-Enhanced Dialogue Module

[0229] Previous research has focused more on improving the recommendation performance of models, while neglecting another objective of CRS: generating fluent and diverse responses. We enhance the performance of the dialogue module by integrating external knowledge and slotted explanation templates, enabling it to generate more descriptive and explanatory statements.

[0230] Based on the movie attribute information we collected, we constructed a number of explanation templates, such as "This film is a classic [genre] film directed by [director]." To facilitate the dialogue module's learning to generate explanation statements, the relevant movie attributes in this template are masked as slots (e.g., "[genre]") without any specific information. During dialogue module training, we appended the explanation templates to recommended statements from the dialogue history. When integrating the recommendation results into the response statement, the model needs to query the specific information from the recommendation results and then fill it into the slots.

[0231] DialoGPT-based methods have achieved excellent performance on many response generation tasks. Therefore, we continue to use the pre-trained language model DialoGPT with fixed parameters to complete the dialogue task. Similarly, to guide the pre-trained language model to adapt to the CRS response generation task, we leverage a cue learning approach. For the dialogue context that integrates the explained template, we still use RoBERTa to obtain the template context embedding E. tc The specific dialogue generation process is as follows:

[0232] E gen =E c,v +E tc +E p-gen (16)

[0233] Among them, E gen This represents the fused feature embedding matrix;

[0234] E c,v This indicates the word embedding after fusion;

[0235] E tc This indicates the context embedding after the fusion interpretation template;

[0236] E p-gen The cue card features that indicate a dialogue task;

[0237] s t =DialoGPT gen (E gen (17)

[0238] Among them, s t This represents the reply sentence generated in round t;

[0239] DialoGPT gen (·) represents the pre-trained model DialoGPT with frozen parameters in the dialogue task;

[0240] E gen This represents the feature embedding matrix after fusion.

[0241] 5 Experimental Environment

[0242] In this chapter, we will introduce the dataset, baseline model, evaluation metrics, and implementation details.

[0243] 5.1 Dataset

[0244] R E D IALThis is the most commonly used English CRS dataset for movie recommendations, constructed by crowdsourced workers from Amazon Mechanical Turk (AMT) according to the dialogues between seekers and recommenders. It contains 10,006 dialogues consisting of 182,150 sentences, involving 504 users and 51,699 movies. For each dialogue, we responded sequentially from the first sentence, generating casual remarks or providing recommendations. Furthermore, to facilitate training, we performed an LMKG entity linking task on the dataset and stored the relevant entities involved in each sentence in dictionary format.

[0245] 5.2 Baseline

[0246] We introduced several competitive baselines from previous research and evaluated the model's recommendation performance and dialogue performance in two ways:

[0247] TextCNN: It ranks items based on user features extracted from the dialogue history using a CNN model.

[0248] Transformer: It utilizes dialogue history and generates system responses based on an encoder-decoder architecture.

[0249] ReDial: It comes with dataset R E D IAL Released together, it consists of an HRED-based dialogue module and an autoencoder-based recommendation module.

[0250] KBRD: It uses knowledge graphs to enrich dialogue history and enhance entity semantics, and employs a Transformer-based dialogue module to integrate external knowledge.

[0251] KGSF: It introduces two knowledge graphs to enhance the semantic representation of entities and words respectively, and aligns their semantic spaces by maximizing mutual information.

[0252] RevCore: It introduces unstructured external knowledge, namely project-related comments, to improve the recommendation and conversation modules.

[0253] C2-CRS: It uses a coarse-to-fine contrastive learning method to align semantic units of different data signals, generating a more consistent fusion representation.

[0254] UniCRS: It utilizes pre-trained language models and prompt learning to build a unified CRS model, integrating recommendation and dialogue modules.

[0255] GPT-2: It is an autoregressive pre-trained language model. We concatenate historical sentences as input, use the last token as user preference to generate recommendations, and use the generated text as the system response.

[0256] DialoGPT: It is an autoregressive model trained on a large-scale dialogue corpus. The training process is similar to that of GPT-2.

[0257] BART: It is a sequence-to-sequence model pre-trained on a large-scale general corpus using a denoising autoencoder task. Training follows a similar approach to GPT-2.

[0258] In the baselines mentioned above, TextCNN is the recommendation method, and Transformer is the text generation method. Redial, KBRD, KGSF, RevCore, C2-CRS, and UniCRS are models for dialogue recommendation systems. GPT-2, DialoGPT, and BART are pre-trained language models, with GPT-2 and BART trained on a general corpus, and DialoGPT trained on a dialogue corpus.

[0259] 5.3 Evaluation Indicators

[0260] Recommendation and dialogue tasks require different evaluation metrics. We use Recall@k (k = 1, 10, 50) to evaluate recommendations, which indicates whether the top k recommendations contain true labels. To better evaluate the model's response generation, we used both automatic and human evaluation. Automatic evaluation included 1) Perplexity: this metric assesses fluency, with lower values ​​indicating more fluent sentences; and 2) Distinct-n (n = 2, 3, 4): this metric assesses diversity. For human evaluation, annotators scored based on fluency and informativeness, with scores ranging from 0 to 2.

[0261] 5.4 Implementation Details

[0262] We implemented the MPKE model in PyTorch and trained it on an NVIDIA GeForce RTX 3090 card. For fair comparison, we kept the common parameters consistent with the baseline. The maximum context length was set to 200, and the maximum number of integrated entities was 64. We used DialoGPT-small as the base pre-trained module with an embedding size of 768. For LMKG encoding of the knowledge graph, similar to KGSF, we used a 1-layer R-GCN. When optimizing the tunable parameters, we used AdamW with default parameters. The entire training process consisted of a 7-epoch fusion module, a 5-epoch recommendation module, and a 10-epoch dialogue module.

[0263] 6 Results and Analysis

[0264] In this section, we validated the effectiveness of our model through experiments and analyzed a case study.

[0265] 6.1 Recommended Task Evaluation

[0266] Results Analysis. Table 2 shows the results of different methods on the recommendation task. As we can see, dialogue recommendation methods outperform ordinary recommendation methods, thanks to CRS's ability to learn user preferences from sparse discourse environments. Among dialogue recommendation methods, KBRD introduces a knowledge graph to enhance entity semantics, KGSF aligns the semantic spaces of two knowledge graphs through MIM, and RevCore integrates item comments to enrich dialogue information. These three methods progressively enhance the introduction of external knowledge, hence the performance order RevCore > KGSF > KBRD > ReDial. C2-CR and UniCRS improve upon the model structure; the former utilizes contrastive learning to fuse multi-granular information, while the latter uses cue learning to build a unified framework. Among the three pre-trained models, BART performs better, perhaps because the dialogue understanding capabilities of the other two are limited by their unidirectional Transformer architecture.

[0267] As shown in Table 2, our model outperforms all baselines. Regarding external knowledge, we construct a higher-quality knowledge graph, LMKG, reducing interference from other domains and further enhancing entity representations. Regarding user preferences, we not only integrate hidden semantics from the user in the dialogue text but also design a secondary recommendation mechanism to explore scarce attribute entities and image representations. Compared to C2-CRS, our model achieves a 9.4% improvement in R@1 and a 9.8% improvement in R@10. On R@50, our method outperforms UniCRS by 8.8%.

[0268] Table 2 Results of the Recommendation Task

[0269]

[0270]

[0271] We will abbreviate Recall@k as R@k. Bold text indicates the best result.

[0272] Ablation Theory. In recommendation, we improve the model's recommendation performance by constructing a high-quality knowledge graph (LMKG) and integrating various user preferences. To validate the effectiveness of our method, we conducted a series of ablation experiments. See Table 3 and... Figure 4As shown ("w / o LMKG" means the external knowledge used is DBpedia; "w / o f_entity" and "w / o a_entity" mean removing the original entity and attribute entities; "w / o context" means removing the context representation; "w / o image" means removing the image representation), replacing the knowledge graph with LMKG improved the recommendation performance of both the baseline model and MPKE. This indicates that our high-quality knowledge graph effectively helps the model understand user intent and provides better entity representations. For users with multiple preferences, removing image embeddings only slightly reduced model performance, which is in line with our expectations that images are secondary preferences. Furthermore, "w / o f_entity" is slightly lower than "w / o a_entity," the former representing movie entities involved in the dialogue, and the latter representing attribute entities explored through a secondary recommendation mechanism. This indicates that movie entities have a stronger influence, but attribute entities still represent a portion of user preferences.

[0273] Table 3 Ablation experiments on LMKG

[0274]

[0275]

[0276] We replaced the external knowledge graph used in the previous CRS model with LMKG.

[0277] 6.2 Dialogue Task Evaluation

[0278] Results Analysis. Table 4 shows the automatic evaluation results of different methods on the dialogue task. As we can see, ReDial achieves better results than Transformer because it uses a pre-trained RNN to represent the dialogue. KBRD utilizes an external KG to generate word probability biases for entity words, while KGSF enhances the representation of text and entities by using MIM and KG-enhanced cross-attention. RevCore further improves the diversity of system responses by introducing unstructured comments to enrich the dialogue text. Compared to the contributions of the previous baselines in external knowledge, C2-CRS and UniCRS improve the model architecture by using contrastive learning and cue learning, respectively, thus achieving better results on the dialogue task. In addition, the performance of the three pre-trained models is better than some CRS methods (such as KGSF and KBRD), which may be due to their generative training on large-scale corpora. DialoGPT outperforms the other two because DialoGPT's corpus is about dialogue.

[0279] Our model achieved better results compared to the baseline.

[0280] We combine plain text representation with word-level knowledge graphs, emphasizing keywords and enhancing the semantic representation of the text. Furthermore, we construct explanation templates and integrate them into the dialogue history to improve the descriptive power and diversity of the system's responses. Compared to UniCRS, our model achieves a Dist-3 score of 12.3% and a Dist-4 score of 9.8%. In terms of fluency, our model achieves a PPL score comparable to C2-CRS. Table 5 also shows that MPKE-generated responses are more fluent and diverse in human evaluation.

[0281] Table 4. Automatic evaluation results of the dialogue task

[0282]

[0283]

[0284] We refer to Distinct-k as Dist-k and Perplexity as PPL. Bold text indicates the best possible result.

[0285] Table 5. Human evaluation results of the dialogue task

[0286] Models Fluency Informativeness Transformer 0.94 0.89 Redial 1.21 1.07 KBRD 1.26 1.19 KGSF 1.48 1.32 RevCore 1.51 1.37 C2-CRS 1.54 1.44 UniCRS 1.62 1.51 GPT-2 1.53 1.38 DialoGPT 1.60 1.46 BART 1.58 1.34 MPKE 1.67 1.54

[0287] The best result is to use bold text.

[0288] Table 6 Ablation experiments on dialogue tasks

[0289]

[0290] "w / o f_word" means remove the merged word embeddings. "word" means use the non-merged word embeddings. "w / o template" means remove the explanation template.

[0291] Ablation Study. In the dialogue aspect, we introduced a word-level knowledge graph and constructed explanatory templates to enhance the model's dialogue performance. To validate the effectiveness of our method, we conducted a series of ablation experiments, the results of which are shown in Table 6. As we can see, "w / o f_word" and "word" indicate that leveraging the knowledge graph to enhance keyword representation is meaningful. Furthermore, "w / o template" also demonstrates that our constructed template can improve the diversity of the model's responses. However, there are still some differences between the template statements and the dialogue, thus affecting the fluency of the responses to some extent after integrating the explanatory templates.

[0292] 6.3 Case Analysis

[0293] In this section, we will analyze how our MPKE works using a case study. First, the model extracts the movie entity "Saw (2004)" and the keyword "classic mystery" from the dialogue context using the knowledge graph LMKG and ConceptNet, respectively. Then, the semantic fusion module cross-merges the entity representation, word representation, and text representation and passes the result to the recommendation module and the dialogue module. The recommendation module calculates and generates the preliminary result "The Conjuring," which, to some extent, satisfies the user's preferences. Next, the recommendation module queries the relevant attributes of the preliminary result and uses them as auxiliary information to optimize the recommendation, obtaining the final recommendation result "Shutter Island (2010)." Based on the fused information, the dialogue module first generates a recommendation response, which includes a slotted explanation describing the recommendation result. Then, the dialogue module fills the slots with the relevant information of the final recommendation result as a complete system response.

[0294] 7 Summary

[0295] In this paper, we propose a novel dialogue recommendation model called MPKE. First, we construct a novel knowledge graph, LMKG, for the dialogue recommendation domain. Its low noise and high coverage characteristics allow the model to more fully utilize external knowledge to improve performance. Second, we construct a fusion module to semantically fuse multiple knowledge graphs and dialogue text, providing better embedding representations for recommendation and dialogue tasks. Furthermore, we design a secondary recommendation mechanism to explore scarce attribute preferences and use them to enhance user representations. Simultaneously, we utilize word-level knowledge graphs and explanation templates to improve the diversity and interpretability of the system's responses. Extensive experimental results demonstrate that our method outperforms previous state-of-the-art methods. Currently, we primarily consider direct recommendation scenarios. In future work, we hope to investigate guiding users from casual conversation scenarios to recommendation scenarios. We are also interested in exploring the use of preferences from similar users to assist in recommendation.

[0296] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. An improved dialogue recommendation system through multi-preference modeling and knowledge enhancement, characterized in that, This includes a semantic fusion module, a recommendation module, and / or a dialogue module; The first output of the semantic fusion module is connected to the data input of the recommendation module, and the second output of the semantic fusion module is connected to the data input of the dialogue module. The semantic fusion module is used to fuse external knowledge with information from the dialogue text; the semantic fusion module includes an information encoding unit and a semantic fusion unit. The data output of the information encoding unit is connected to the data input of the semantic fusion unit; The information encoding unit includes: ， in, Representing entities In the Entity representation of a layer; express Activation function; Representing entities Relationship set; Representing entities In relationship The set of adjacent nodes below; Indicates the normalization factor; Representing relations In the The learnable matrix of the layer; Representing entities In the The learnable matrix of the layer; Representing entities Adjacent nodes in the th Layer representation; Representing entities In the Entity representation of a layer; ， in, Representing words In the Layer representation; express Activation function; Represents the angle matrix; Represents the adjacency matrix of a graph; Representing words In the Layer representation; Representing words In the The learnable matrix of the layer; The information encoding unit is used for information encoding; The semantic fusion unit is used for semantic fusion; the semantic fusion unit includes: ， in, An affinity matrix representing the relationship between project entities and text; Represents the entity embedding matrix; Represents the entity embedding matrix The transpose of the matrix; This represents the first transformation matrix; Represents the context embedding matrix; ， in, An affinity matrix representing the text and word entities; Represents the context embedding matrix; Represents the context embedding matrix The transpose of the matrix; This represents the second transformation matrix; Represents a word embedding matrix; ， This represents the embedded entity after fusion; Represents the context embedding matrix; An affinity matrix representing the relationship between project entities and text; Represents the entity embedding matrix; ， in, This indicates the word embedding after fusion; Represents a word embedding matrix; An affinity matrix representing the text and word entities; Represents the context embedding matrix; The recommendation module is used to provide recommendations by combining user preferences from one or more of the following: text, entities, images, or any combination thereof. The recommendation module will utilize a large pre-trained model to provide recommendations. The dialogue module is used to generate fluent and diverse responses by designing explanation templates and / or highlighting dialogue keywords, and by combining the recommendation results.

2. The improved dialogue recommendation system based on multi-preference modeling and knowledge enhancement according to claim 1, characterized in that, The recommendation module includes: ， in, This represents the feature embedding matrix after fusion during the exploration phase; This represents the embedded entity after fusion; Represents the context embedding matrix; The characteristics of the prompt template for the recommendation task; ， in, Indicates the first The set of recommended results during the round-based exploration phase; This represents the pre-trained model DialoGPT with its parameters frozen for the recommendation task. This represents the feature embedding matrix after fusion during the exploration phase; ， in, Represents an attribute entity; Represents the original poster image; Indicates searching for movies Related attributes; This indicates the movie with the highest probability in the recommended result set.

3. The improved dialogue recommendation system based on multi-preference modeling and knowledge enhancement according to claim 2, characterized in that, The recommendation module also includes: ， in, Represents the image feature matrix; This represents the pre-trained image feature extraction model; Represents the original poster image; ， in, This represents the feature embedding matrix after fusion during the integration phase; This represents the embedded entity after fusion; Represents the context embedding matrix; The characteristics of the prompt template for the recommendation task; Represents an attribute entity; ， This represents user preferences after cross-modal fusion; This represents the feature embedding matrix after fusion during the integration phase; Indicates a cross-modal attention layer; Represents the image feature matrix; ， in, This represents the final set of recommendation results; This represents the pre-trained model DialoGPT with its parameters frozen for the recommendation task. This represents user preferences after cross-modal fusion.

4. The improved dialogue recommendation system based on multi-preference modeling and knowledge enhancement according to claim 1, characterized in that, The dialogue module includes: ， in, This represents the fused feature embedding matrix; This indicates the word embedding after fusion; This indicates the context embedding after the fusion interpretation template; The cue card features that indicate a dialogue task; ， in, Indicates the first The response sentences generated in round-robin format; DialoGPT is a pre-trained model that represents the frozen parameters in a dialogue task. This represents the feature embedding matrix after fusion.

5. The improved dialogue recommendation system based on multi-preference modeling and knowledge enhancement according to claim 1, characterized in that, It also includes a knowledge graph module, which is used to introduce external knowledge.

6. The improved dialogue recommendation system based on multi-preference modeling and knowledge enhancement according to claim 1, characterized in that, It also includes an indicator evaluation module, which is used to evaluate the fluency and / or diversity of the dialogue recommendation system.

Images