Explanatable recommendation optimization system based on user feedback driving

Abstract

The invention discloses an interpretable recommendation optimization system based on user feedback driving, and the system is characterized in that the system comprises an offline strategy reinforcement learning framework, a personalized reward prompt of target perception, multi-view interpretation optimization based on Pareto optimum, an overall target function, and a complete learning algorithm; by introducing user subjective evaluation and a multi-objective optimization strategy, the problems that recommendation interpretation quality is low and user requirements cannot be accurately reflected are solved. According to the method, the performance of the original basic model is remarkably improved in the aspects of recommendation quality and generation of the recommendation explanation text.

Description

Technical Field

[0001] The present invention relates to the field of explainable recommendation systems, and in particular to an explainable recommendation optimization system driven by user feedback. Background Art

[0002] In today's era of information explosion, recommendation systems have become an indispensable core technology for internet services. With the rapid development of e-commerce, social media, content platforms, and other fields, users' demand for personalized recommendations is increasing. However, traditional recommendation systems suffer from the "black box" problem, making it difficult for users to understand why they received certain recommendations. Explainable recommendation systems have emerged to enhance user trust and acceptance of recommendation systems by generating human-understandable justifications for recommendations.

[0003] Explainable recommendation systems have a wide range of applications and have permeated every aspect of daily life. In e-commerce, precise and persuasive recommendation explanations can directly influence users' purchasing decisions. For example, on a cosmetics e-commerce platform, simply recommending a skincare product is insufficient; users need to understand why the product is suitable for them: Is it a match for their skin type? Is it in line with their age profile? Or is it synergistic with other products they've previously purchased? In educational platforms, personalized recommendations can precisely recommend learning resources based on a learner's background, learning progress, and interests. Explainable recommendation systems go beyond simply recommending courses; they can also explain why a particular course holds particular value for the learner. On entertainment content platforms, recommendation systems must understand users' emotional needs and recommend content that not only matches their interests but also resonates with them.

[0004] Existing technologies often rely on static text fitting to generate recommended explanations, combining user comments as the target text for the recommended explanations. Specifically, this process proceeds through the following steps:

[0005] Data Collection: Collect basic interaction data between users and items. For users with sparse data, this interaction data is usually very limited. For example, a user may have only purchased or browsed a few items.

[0006] Extracting user reviews: To compensate for the lack of interactive data, many systems utilize user reviews as a form of recommendation explanatory information. User reviews provide recommendation systems with detailed feedback about products or services, helping them understand which features or functions might appeal to other similar users. For example, a user might mention in a review that "this phone is great for taking photos," and the system can use this information to infer that users with similar preferences might be recommended this phone.

[0007] Model training: The explainable recommendation model is trained on user ratings and review data. This allows the model to learn the relationship between user preferences and review content, generating recommendations that better meet user needs and providing more targeted explanations for recommendations.

[0008] Recommendation explanation generation: Ultimately, the recommendation model generates recommendation explanations that match user preferences based on review interaction data. While this generation method can provide users with some explanations, it is essentially based on supervised fitting of the target text and cannot specifically optimize model parameters based on user feedback.

[0009] The main problems faced by current recommendation systems when processing recommendation explanations are the lack of high-quality recommendation explanation data for real recommendation data, and the quality measurement methods for recommendation explanations do not conform to users' subjective evaluations. Specifically, existing technologies have the following problems:

[0010] 1. Real recommendation data lacks high-quality recommendation explanation data:

[0011] Existing recommendation systems typically rely on historical behavioral data (such as clicks, browsing, and purchases) to make personalized recommendations, but this data often lacks high-quality explanations for the recommendations. For many users, recommendation results often lack relevant explanations, which are crucial for improving user trust and satisfaction with the recommendation system. In real-world applications, recommendation explanations often adopt a templated approach, generating simple explanations based on basic product attributes or user historical behavior. This approach ignores personalization factors, resulting in low-quality recommendation explanations.

[0012] These shortcomings particularly impair performance for low-activity users and niche items: In particular, the lack of sufficient historical interaction data makes it difficult for the recommendation system to generate high-quality explanations for these users or items. Due to the low quality of the explanations, users have low trust in these recommendations, which in turn affects the activity of low-activity users, further impairing the overall performance of the system.

[0013] 2. Recommended questions to assess the quality of explanations:

[0014] Existing recommendation systems typically use text similarity metrics to measure the quality of recommended explanations. These methods primarily rely on the similarity between generated explanations and certain predefined criteria (such as templates or historical explanations) to evaluate the quality of recommended explanations. However, this approach lacks subjectivity and does not truly reflect users' true feelings and subjective evaluations of recommended explanations. In reality, the quality of recommended explanations should be determined by user subjective evaluations rather than simply measured by machine-generated similarity metrics.

[0015] Improving the quality of explanations from multiple perspectives: In an ideal recommendation system, the quality of recommendation explanations should be reflected in multiple dimensions, such as informativeness and persuasiveness. These dimensions have a significant impact on user cognition and decision-making. A single similarity metric cannot fully evaluate the effectiveness of recommendation explanations. Consequently, existing metrics fail to effectively optimize the multi-faceted quality of explanations and fail to meet the diverse needs of different users for recommendation explanations.

[0016] The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the Invention

[0017] The purpose of the present invention is to provide an explainable recommendation optimization system driven by user feedback to solve the technical problems existing in the prior art.

[0018] In order to achieve the above object, the present invention adopts the following technical solutions:

[0019] The present invention provides an explainable recommendation optimization system driven by user feedback, including: an offline policy reinforcement learning framework, target-aware personalized reward prompts, multi-perspective explanation optimization based on Pareto optimality, an overall objective function, and a complete learning algorithm. By introducing user subjective evaluation and multi-objective optimization strategies, the system solves the problem of low quality of recommendation explanations and the inability to accurately reflect user needs.

[0020] Furthermore, the offline policy reinforcement learning framework specifically includes:

[0021] 1) Difficulty-aware sampling strategy: This strategy increases the sampling opportunities for item categories with which users have fewer interactions, and reduces the sampling probability for item categories with which users have more interactions:

[0022]

[0023] in:

[0024] Represents user u and item category C i Frequency of interaction between items in

[0025] Z is the total number of item categories;

[0026] The logarithmic function prevents extreme frequency distributions from causing a few item categories to dominate the sampling process;

[0027] The constant 2 is used to avoid numerical instabilities;

[0028] 2) Offline strategy optimization process:

[0029] The specific process includes the following two stages:

[0030] (1) Data collection stage:

[0031] From the observed interactions D and the unobserved user-item interactions obtained using the difficulty-aware sampling strategy Export training data;

[0032] Generate explanation predictions using a pre-trained explainable recommendation model;

[0033] Generated explanations are formatted using predefined prompt templates and rewarded by a human simulator based on a large language model.

[0034] All trajectories are stored in a playback buffer B, the basic structure is

[0035]

[0036] (2) Model update phase:

[0037] To address the distribution difference between the behavioral policy and the target policy, an alternative objective function is used to correct the data distribution by incorporating importance sampling weights; in addition, drawing on proximal policy optimization (PPO), a clipping technique is used to discard samples that fall outside the trust region:

[0038]

[0039] in:

[0040] θ and θ b Represent the network parameters of the target strategy and the behavior strategy respectively;

[0041] ∈ is a hyperparameter of the clipping range;

[0042] All reward values ​​correspond to behavioral policies generated actions;

[0043] 3) Advantage estimation is used to stabilize model learning:

[0044] In order to solve the problem that the objective function based on reward value may cause unstable learning, an advantage function is proposed:

[0045]

[0046] in:

[0047] represents the i-th generated interpretation of the (u, v) pair;

[0048] J is the number of explanations explored for each user-item interaction pair;

[0049] The advantage value is used to better guide the policy network to distinguish the different qualities of generated sentences more quickly.

[0050] Furthermore, the goal-aware personalized reward prompt specifically includes:

[0051] 1) Target-aware user information extraction:

[0052] First, user and item information is converted into structured descriptions to facilitate processing by large language models. To extract valuable information from the lengthy and noisy user interaction history, a target-aware personalized retrieval method is introduced:

[0053] Use OpenAI embedding model to obtain sentence embeddings for all interaction items and target items;

[0054] Calculate the cosine similarity between each candidate behavior and the target item;

[0055] Select the top K items with the highest similarity scores to represent the target-aware user description;

[0056] 2) Personalized human-induced scoring mechanism:

[0057] To define a personalized reward scoring mechanism that is consistent with different human preferences, the following approach was adopted:

[0058] Based on the extracted information description, users are grouped into Y clusters using a clustering algorithm;

[0059] Randomly select a representative user from each cluster and manually create a small number of example ratings as prompt prototypes;

[0060] Leveraging prompt prototypes corresponding to user clusters to guide the large language model to create new customized reward few-shot contexts;

[0061] In addition, a three-level instruction enhancement strategy is developed to fully tap the thinking chain potential of the large language model:

[0062] (1) clearly defined quality standards as a guide for rewards;

[0063] (2) Customized few-shot contexts provide fine-grained examples with different scores;

[0064] (3) Deep reasoning is introduced to align user preferences and stimulate the logical reasoning ability of large language models.

[0065] Furthermore, the multi-perspective interpretation optimization based on Pareto optimality specifically includes:

[0066] Pareto Optimization Framework:

[0067] Assume that there are M objective functions in the explainable recommendation system, represented as {F1(θ),F2(θ),…,F M (θ); by replacing the formula:

[0068]

[0069] The rewards in are the advantages of each quality aspect to derive the respective objective functions;

[0070] Definitions related to Pareto optimization:

[0071] Pareto dominance: solution θ1 dominates solution θ2 if and only if for all targets L i (θ1)≥L i (θ2), and there exists at least one target j such that L ( (θ1)>L ( (θ2);

[0072] Pareto optimality: A solution θ is defined as Pareto optimal when it cannot be further improved on any objective without degrading other objectives;

[0073] Pareto front: For multi-objective tasks, the Pareto front represents the set of all Pareto optimal solutions;

[0074] In order to balance the conflicts between different objectives, a scalar method is adopted, using weights ω={ω1,ω2,…,ω M Average of M functions:

[0075]

[0076] The weight ω can be determined by solving the following quadratic programming problem:

[0077]

[0078] in:

[0079] 1 is a vector of all 1s;

[0080] The set {h1,h2,…,h Q} and {β1,β2,…,β Q} represent the non-negative M-dimensional prior preference vector and scalar value respectively;

[0081] h q is defined as a unit vector, β q is set to a predefined bias to quantify the relative importance of the corresponding i-th evaluation perspective.

[0082] Furthermore, the overall objective function is specifically as follows:

[0083]

[0084] in:

[0085] Indicates the generation of explanations in terms of the i-th quality perspective Advantage value of

[0086] ω i represents the weight of the i-th target loss;

[0087] M represents the number of objective functions;

[0088] By minimizing the above training objective function, the explainable recommender can optimize personalized recommendation explanations from multiple perspectives and better align with human preferences.

[0089] Furthermore, the complete learning algorithm specifically includes:

[0090] The complete training process of HF4Rec includes the following steps:

[0091] 1) Initialize the playback buffer B to an empty set;

[0092] 2) Initialize the behavior strategy network

[0093] 3) Pre-training on the observed dataset D

[0094] 4) For each iteration:

[0095] Phase 1: Data Collection:

[0096] Based on the formula Sampling unobserved data for For each (u,v):

[0097] Format item information;

[0098] Retrieve the top K similar historical items based on the target item’s information;

[0099] Format user information;

[0100] Generate human-induced few-shot context;

[0101] For j = 1 to J:

[0102] from Get and

[0103] Generate multi-view reward values

[0104] Will Store in B;

[0105] Replace all reward values ​​with advantage values;

[0106] Phase 2: Model Update:

[0107] Use θ b Initialize the target policy network M θ ;

[0108] For each round:

[0109] Use Pareto optimality to solve multi-objective weights ω;

[0110] Use the PPO formula to optimize the model parameters θ on dataset B;

[0111] Replace the behavior policy network parameters θ with θ b ;

[0112] By implementing an interactive optimization framework through this offline strategy approach, HF4Rec can effectively improve the quality of recommendation explanations.

[0113] By adopting the above technical solution, the present invention has the following beneficial effects:

[0114] This application addresses the problem of low-quality recommendation explanations that fail to accurately reflect user needs by incorporating user subjective evaluation and a multi-objective optimization strategy. This invention significantly improves the original model in both recommendation quality and the generation of recommendation explanation text. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0116] Figure 1 This is a system architecture diagram of an explainable recommendation optimization system driven by user feedback provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0117] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0118] The following describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0119] Combine Figure 1 As shown, the present application provides an explainable recommendation optimization system driven by user feedback, including: an offline policy reinforcement learning framework, target-aware personalized reward prompts, multi-perspective explanation optimization based on Pareto optimality, an overall objective function, and a complete learning algorithm; by introducing user subjective evaluation and multi-objective optimization strategies, it solves the problem of low quality of recommendation explanations and the inability to accurately reflect user needs.

[0120] This paper innovatively proposes an explainable recommendation optimization framework named HF4Rec to improve the explainable recommendation system.

[0121] The core idea of ​​the HF4Rec framework is to use large language models (LLMs) as human simulators to provide human feedback within a reinforcement learning framework to optimize the generation of recommendation explanations. This approach transforms the traditional supervised learning paradigm into an interactive optimization mechanism that better meets the human-centric need for interpretability.

[0122] The framework mainly includes the following innovations:

[0123] Use large language models to simulate human feedback;

[0124] Propose goal-aware personalized reward cues;

[0125] Multi-perspective explanation optimization based on Pareto optimality;

[0126] Design an offline policy reinforcement learning optimization process.

[0127] The following is a detailed description of each component of the explainable recommendation optimization system of the present invention:

[0128] Offline policy reinforcement learning framework, specifically including

[0129] Difficulty-aware sampling strategy

[0130] To address the data sparsity and popularity bias issues, this application proposes a difficulty-aware sampling strategy. This strategy increases the sampling chances of item categories with which users have fewer interactions, and reduces the sampling probability of item categories with which users have more interactions:

[0131]

[0132] in:

[0133] Represents user u and item category Ci Frequency of interaction between items in

[0134] Z is the total number of item categories;

[0135] The logarithmic function prevents extreme frequency distributions from causing a few item categories to dominate the sampling process;

[0136] The constant 2 is used to avoid numerical instabilities.

[0137] Offline strategy optimization process:

[0138] To address the two shortcomings of traditional online policy gradient methods (serial inference delay and the risk of convergence to local optimality), this application proposes the following two-stage process:

[0139] (1) Data collection stage:

[0140] From the observed interactions D and the unobserved user-item interactions obtained using the difficulty-aware sampling strategy Export training data;

[0141] Generate explanation predictions using a pre-trained explainable recommendation model;

[0142] Generated explanations are formatted using predefined prompt templates and rewarded by a human simulator based on a large language model.

[0143] All trajectories are stored in a playback buffer B, the basic structure is

[0144]

[0145] (2) Model update phase:

[0146] To address the distribution difference between the behavioral policy and the target policy, this application uses an alternative objective function to correct the data distribution by incorporating importance sampling weights. In addition, drawing on proximal policy optimization (PPO), a clipping technique is used to discard samples that fall outside the trust region:

[0147]

[0148] in:

[0149] θ and θ b Represent the network parameters of the target strategy and the behavior strategy respectively;

[0150] ∈ is a hyperparameter of the clipping range;

[0151] All reward values ​​correspond to behavioral policies Generated action.

[0152] Advantage estimates are used to stabilize model learning:

[0153] In order to solve the problem that the objective function based on reward value may cause unstable learning, this paper proposes an advantage function:

[0154]

[0155] in:

[0156] represents the i-th generated interpretation of the (u, v) pair;

[0157] J is the number of explanations explored for each user-item interaction pair;

[0158] The advantage value can better guide the policy network to distinguish the different qualities of generated sentences faster.

[0159] Goal-aware personalized reward prompts, including:

[0160] To effectively use large language models to predict human feedback ratings, this paper overcomes the following two challenges:

[0161] Target-aware user information extraction:

[0162] First, the present invention converts user and item information into structured descriptions, which are convenient for large language model processing. In order to extract valuable information from the lengthy and noisy user interaction history, this application introduces a target-aware personalized retrieval method:

[0163] Use OpenAI embedding model to obtain sentence embeddings for all interaction items and target items;

[0164] Calculate the cosine similarity between each candidate behavior and the target item;

[0165] The top K items with the highest similarity scores are selected to represent the target-aware user description.

[0166] Personalized human-induced scoring mechanism:

[0167] In order to define a personalized reward scoring mechanism that is consistent with different human preferences, the present invention adopts the following approach:

[0168] Group users into Y clusters using a clustering algorithm (e.g., K-means) based on the extracted information descriptions;

[0169] Randomly select a representative user from each cluster and manually create a small number of example ratings as prompt prototypes;

[0170] Leveraging prompt prototypes corresponding to user clusters, we guide the large language model to create new customized reward few-shot contexts.

[0171] In addition, this application develops a three-level instruction enhancement strategy to fully tap the thinking chain potential of the large language model:

[0172] (1) clearly defined quality standards as a guide for rewards;

[0173] (2) Customized few-shot contexts provide fine-grained examples with different scores;

[0174] (3) Deep reasoning is introduced to align user preferences and stimulate the logical reasoning ability of large language models.

[0175] Multi-perspective explanation optimization based on Pareto optimality includes:

[0176] The quality evaluation of recommendation explanations usually involves multiple perspectives (such as informativeness and persuasiveness), and these evaluation metrics are not always harmonious and consistent. Therefore, this paper regards the task of improving the quality of multi-perspective explanations as a multi-objective optimization problem.

[0177] Pareto Optimization Framework:

[0178] Assume that there are M objective functions (such as informativeness and persuasiveness) in the explainable recommendation system, represented as {F1(θ),F2(θ),…,F M (θ). The present invention replaces the formula:

[0179]

[0180] The rewards in are the advantages corresponding to each quality aspect to derive the respective objective functions.

[0181] Definitions related to Pareto optimization:

[0182] Pareto dominance: solution θ1 dominates solution θ2 if and only if for all targets L i (θ1)≥L i (θ2), and there exists at least one target j such that L ( (θ1)>L j (θ2).

[0183] Pareto Optimal: A solution θ is defined as Pareto optimal when it cannot be further improved on any objective without degrading other objectives.

[0184] Pareto front: For multi-objective tasks, the Pareto front represents the set of all Pareto optimal solutions.

[0185] In order to balance the conflicts between different objectives, this application adopts a scalarization method, using weights ω={ω1,ω2,…,ω M Average of M functions:

[0186]

[0187] The weight ω can be determined by solving the following quadratic programming problem:

[0188]

[0189] in:

[0190] 1 is a vector of all 1s;

[0191] The set {h1,h2,…,h Q} and {β1,β2,…,β Q} represent the non-negative M-dimensional prior preference vector and scalar value respectively;

[0192] h q is defined as a unit vector, β q is set to a predefined bias to quantify the relative importance of the corresponding i-th evaluation perspective.

[0193] The overall objective function is as follows:

[0194] Based on the above design, this application proposes an overall training loss function:

[0195]

[0196] in:

[0197] Indicates the generation of explanations in terms of the i-th quality perspective Advantage value of

[0198] ω i represents the weight of the i-th target loss;

[0199] M represents the number of objective functions (e.g., informativeness and persuasiveness);

[0200] By minimizing the above training objective function, the explainable recommender can optimize personalized recommendation explanations from multiple perspectives and better align with human preferences.

[0201] The complete learning algorithm is as follows:

[0202] The complete training process of HF4Rec includes the following steps:

[0203] Initialize playback buffer B to an empty set;

[0204] Initialize the behavior policy network

[0205] Pre-trained on the observed dataset D

[0206] For each iteration:

[0207] Phase 1: Data Collection:

[0208] Based on the formula Sampling unobserved data for For each (u,v):

[0209] Format item information;

[0210] Retrieve the top K similar historical items based on the target item’s information;

[0211] Format user information;

[0212] Generate human-induced few-shot context;

[0213] For j = 1 to J:

[0214] from Get and

[0215] Generate multi-view reward values

[0216] Will Store in B;

[0217] Replace all reward values ​​with advantage values;

[0218] Phase 2: Model Update:

[0219] Use θ b Initialize the target policy network M θ ;

[0220] For each round;

[0221] Use Pareto optimality to solve multi-objective weights ω;

[0222] Use the PPO formula to optimize the model parameters θ on dataset B;

[0223] Replace the behavior policy network parameters θ with θ b ;

[0224] By implementing an interactive optimization framework through this offline strategy approach, HF4Rec can effectively improve the quality of recommendation explanations.

[0225] The complete training pseudo code is as follows:

[0226]

[0227] The overall architecture of the HF4Rec method is shown in the figure below: Figure 1 shown.

[0228] The applicant conducted experimental verification using the Beauty, Sports & Outdoors, and Video & Games datasets on the Amazon platform and the Yelp dataset.

[0229] The representative explainable recommendation models PETER, PEPLER, and ERRA are used as representative basic models, and the optimization framework of the present invention is applied; for the score prediction task, the applicant selects PMF, SVD++, NRT, and NETE as comparison algorithms.

[0230] For the method of the present invention, the applicant introduces two variants:

[0231] Holistic optimization method: directly let the large model GPT3.5 perform overall quality evaluation based on user information and generated explanations according to the definitions of different indicators, and give a scalar value. The model is denoted as XH (here X can be any explainable recommendation algorithm).

[0232] Multi-angle approach: Let the large model GPT3.5 perform dimensional evaluation (such as information content and persuasiveness) based on user information and generated explanations according to the definitions of different indicators. Different angles have their own scalar values, and the model is denoted as XM (here X can be any explainable recommendation algorithm).

[0233] In terms of text evaluation, the applicant uses BERT-Score, BLEU, ROUGE (precision, recall and F1 score of ROUGE-1, ROUGE-2, ROUGE-L) as a measure of the sentence similarity between the model-generated explanation and the predefined explanation standard, and uses the feature matching rate FMR (Feature Matching Rate) to predict the feature coverage between the text and the real text. The higher these indicators are, the better the performance.

[0234] The amount of information and persuasiveness (scoring range 1-3) are used as subjective evaluation indicators, and higher indicator values ​​represent better performance.

[0235] In terms of recommendation performance evaluation, applicants choose RMSE (Root Mean Squared Error) and MAE (Mean Absolute Error) to evaluate the scoring accuracy. The lower the index value, the better the performance.

[0236] The evaluation performance of generating recommended explanation texts on these datasets is as follows:

[0237]

[0238] In terms of recommended performance, the performance is as follows:

[0239]

[0240] Judging from the experimental results, the method of this application has achieved significant performance improvements over the original basic model in terms of recommendation quality and generation of recommendation explanation text.

[0241] Compared with the prior art, the key innovations of the present invention are as follows:

[0242] Transforming explainable recommendation from supervised text fitting to a human feedback-driven framework: For the first time, we transform explainable recommendation from static text fitting to interactive optimization, using human feedback as a signal to guide explanation generation.

[0243] Multi-perspective explanation quality enhancement based on Pareto optimization: A theoretically guaranteed multi-objective optimization framework is proposed to effectively deal with the inherent conflicts between different explanation perspectives such as informativeness and persuasiveness.

[0244] Offline policy reinforcement learning framework: An efficient offline policy optimization process is designed, which addresses the application challenges of reinforcement learning in explainable recommendations through importance sampling and replay buffer mechanisms.

[0245] Solving the problems of sparse interaction data and cold start: Through difficulty-aware sampling and target-aware user information extraction, the recommendation explanation performance of long-tail users and items is effectively improved.

[0246] Preferably, this application can also be extended to human evaluation feedback based on recommendation explanations from more perspectives. This article focuses on information content and persuasiveness, and in the future, it can be further expanded to factors such as satisfaction and transparency. Other reinforcement learning techniques can be used to optimize the explainable recommendation model. This article uses PPO technology, and other reinforcement learning optimization strategies such as GRPO and DPO can be used in the future.

[0247] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An explainable recommendation optimization system driven by user feedback, characterized by: include: Offline policy reinforcement learning framework, goal-aware personalized reward prompts, multi-perspective explanation optimization based on Pareto optimality, overall objective function and complete learning algorithm; By introducing user subjective evaluation and multi-objective optimization strategies, we can solve the problem of low quality of recommendation explanations and inability to accurately reflect user needs.

2. The user feedback-driven explainable recommendation optimization system according to claim 1, characterized in that: The offline policy reinforcement learning framework specifically includes: 1) Difficulty-aware sampling strategy: This strategy increases the sampling opportunities for item categories with which users have fewer interactions, and reduces the sampling probability for item categories with which users have more interactions: in: Represents user u and item category C i Frequency of interaction between items in Z is the total number of item categories; The logarithmic function prevents extreme frequency distributions from causing a few item categories to dominate the sampling process; The constant 2 is used to avoid numerical instabilities; 2) Offline strategy optimization process: The specific process includes the following two stages: (1) Data collection phase: From the observed interactions D and the unobserved user-item interactions obtained using the difficulty-aware sampling strategy Export training data; Generate explanation predictions using a pre-trained explainable recommendation model; Generated explanations are formatted using predefined prompt templates and rewarded by a human simulator based on a large language model. All trajectories are stored in a playback buffer B, the basic structure is (2) Model update phase: To address the distribution difference between the behavioral policy and the target policy, an alternative objective function is used to correct the data distribution by incorporating importance sampling weights; in addition, drawing on proximal policy optimization (PPO), a clipping technique is used to discard samples that fall outside the trust region: in: θ and θ b Represent the network parameters of the target strategy and the behavior strategy respectively; ∈ is a hyperparameter of the clipping range; All reward values ​​correspond to behavioral policies generated actions; 3) Advantage estimation is used to stabilize model learning: In order to solve the problem that the objective function based on reward value may cause unstable learning, an advantage function is proposed: in: represents the i-th generated interpretation of the (u, v) pair; J is the number of explanations explored for each user-item interaction pair; The advantage value is used to better guide the policy network to distinguish the different qualities of generated sentences more quickly.

3. The user feedback-driven explainable recommendation optimization system according to claim 1, characterized in that: The goal-aware personalized reward prompts specifically include: 1) Target-aware user information extraction: First, user and item information is converted into structured descriptions to facilitate processing by large language models. To extract valuable information from the lengthy and noisy user interaction history, a target-aware personalized retrieval method is introduced: Use OpenAI embedding model to obtain sentence embeddings for all interaction items and target items; Calculate the cosine similarity between each candidate behavior and the target item; Select the top K items with the highest similarity scores to represent the target-aware user description; 2) Personalized human-induced scoring mechanism: To define a personalized reward scoring mechanism that is consistent with different human preferences, the following approach was adopted: Based on the extracted information description, users are grouped into Y clusters using a clustering algorithm; Randomly select a representative user from each cluster and manually create a small number of example ratings as prompt prototypes; Leveraging prompt prototypes corresponding to user clusters to guide the large language model to create new customized reward few-shot contexts; In addition, a three-level instruction enhancement strategy is developed to fully tap the thinking chain potential of the large language model: (1) clearly defined quality standards as a guide for rewards; (2) Customized few-shot contexts provide fine-grained examples with different scores; (3) Deep reasoning is introduced to align user preferences and stimulate the logical reasoning ability of large language models.

4. The user feedback-driven explainable recommendation optimization system according to claim 1, characterized in that: The multi-perspective interpretation optimization based on Pareto optimality specifically includes: Pareto Optimization Framework: Assume that there are M objective functions in the explainable recommendation system, represented as {F1(θ),F2(θ),…,F M (θ); by replacing the formula: The rewards in are the advantages of each quality aspect to derive the respective objective functions; Definitions related to Pareto optimization: Pareto dominance: solution θ1 dominates solution θ2 if and only if for all targets L i (θ1)≥L i (θ2), and there exists at least one target j such that L j (θ1)>L j (θ2); Pareto optimality: A solution θ is defined as Pareto optimal when it cannot be further improved on any objective without degrading other objectives; Pareto front: For multi-objective tasks, the Pareto front represents the set of all Pareto optimal solutions; In order to balance the conflicts between different objectives, a scalar method is adopted, using weights ω={ω1,ω2,…,ω M Average of M functions: The weight ω can be determined by solving the following quadratic programming problem: st1 T ω=1, in: 1 is a vector of all 1s; The set {h1,h2,…,h Q } and {β1,β2,…,β Q } represent the non-negative M-dimensional prior preference vector and scalar value respectively; h q is defined as a unit vector, β q is set to a predefined bias to quantify the relative importance of the corresponding i-th evaluation perspective.

5. The user feedback-driven explainable recommendation optimization system according to claim 1, characterized in that: The overall objective function is as follows: in: Indicates the generation of explanations in terms of the i-th quality perspective Advantage value of ω i represents the weight of the i-th target loss; M represents the number of objective functions; By minimizing the above training objective function, the explainable recommender can optimize personalized recommendation explanations from multiple perspectives and better align with human preferences.

6. The user feedback-driven explainable recommendation optimization system according to claim 1, characterized in that: The complete learning algorithm specifically includes: The complete training process of HF4Rec includes the following steps: 1) Initialize the playback buffer B to an empty set; 2) Initialize the behavior strategy network 3) Pre-training on the observed dataset D 4) For each iteration: Phase 1: Data Collection: Based on the formula Sampling unobserved data for For each (u,v): Format item information; Retrieve the top K similar historical items based on the target item’s information; Format user information; Generate human-induced few-shot context; For j = 1 to J: from Get and Generate multi-view reward values Will Store in B; Replace all reward values ​​with advantage values; Phase 2: Model Update: Use θ b Initialize the target policy network M θ ; For each round: Use Pareto optimality to solve multi-objective weights ω; Use the PPO formula to optimize the model parameters θ on dataset B; Replace the behavior policy network parameter θ with θ b ; By implementing an interactive optimization framework through this offline strategy approach, HF4Rec can effectively improve the quality of recommendation explanations.

Images