A dynamic feedback recommendation method and device based on reinforcement learning

Abstract

This invention discloses a dynamic feedback recommendation method and apparatus based on reinforcement learning. The method includes: pre-training a user behavior simulator using causal inference technology to generate a counterfactual user-item bipartite graph showing complete user-item interaction; weighted aggregation of the real user-item bipartite graph and the counterfactual user-item bipartite graph, using Top-K sampling to generate an interactively balanced user-item bipartite graph; using graph convolution to aggregate neighborhood user information of items to obtain graph-enhanced action representations; using gated recurrent units and self-attention mechanisms to process the interaction sequence between the user and the recommendation system to obtain user state representations at each time step; predicting Q-values ​​based on action representations and user state representations; optimizing the action decisions of the recommendation model based on user reward feedback; and finally achieving dynamic feedback recommendation of Internet information. The apparatus includes a processor and a memory. This invention utilizes causal inference and graph convolution techniques to introduce interactively balanced user-item bipartite graph information, enriching and enhancing agent action representations and user state representations through graph structure association.

Description

Technical Field

[0001] This invention relates to the field of interactive recommendation, and more particularly to a dynamic feedback recommendation method and apparatus based on reinforcement learning. Background Technology

[0002] With the rapid development of the internet, information and data have exploded. Faced with this massive amount of information, internet users struggle to find truly useful information, leading to "information overload." Recommendation systems address this problem by analyzing user preferences and providing personalized, targeted content selections. Traditional recommendation systems treat recommendation as a single-step prediction task and assume that user preferences are static and unchanging, mining user preferences from historical behavioral data. [1] This makes them unable to capture the dynamic changes in user interests over time. In contrast, interactive recommender systems consider a more realistic multi-step interaction process, refining their recommendation strategies by sequentially recommending items to the user and receiving user feedback, thereby optimizing long-term user satisfaction with the recommender system. Therefore, interactive recommender systems have recently become a hot research topic. [2] It has been applied to various personalized services such as TikTok, Pandora, and YouTube.

[0003] Some interactive recommendation systems use a multi-armed robber framework. [3] However, their utility is limited because they do not model the dynamic changes in user preferences. Given the powerful ability of deep reinforcement learning to handle dynamic interaction scenarios, it has been widely applied in recent interactive recommendation systems. Reinforcement learning-driven recommendation systems model the recommendation process as a Markov decision process. [4] Recommendation systems act as intelligent agents, sequentially recommending items (actions) to users (the environment) and receiving user satisfaction feedback (rewards). The goal is to maximize the long-term accumulated user satisfaction throughout the recommendation process. Reinforcement learning-based recommendation methods can be divided into two categories: model-free reinforcement learning methods and model-based methods. Model-free reinforcement learning methods typically include: methods based on DQN (Deep Q-Network), policy-based methods, and methods based on DDPG (Deep Deterministic Policy Gradient). DQN-based methods... [5] A deep Q-network is used to obtain the Q-value of a single item, and the item with the highest Q-value is recommended to the user. (Policy-based approach) [6] The parameterized recommendation policy is learned directly through gradient optimization, without needing to estimate the Q-value. This is a method based on DDPG. [7] Using the Actor-Critic framework [8] It outputs specific actions that can be used for continuous action prediction. Model-based reinforcement learning methods, on the other hand, utilize collected offline policy data to simulate environmental dynamics and optimize recommendation policy learning. For example: GoalRec[9] A world model with time-dense surveillance signals was trained to simulate complex environments. NRSS

[10] A reward model incorporating user preference rewards and transitional rewards was designed. KCRL

[11] A causal user model is proposed that incorporates beneficial group preferences to improve user interest estimation in sparse scenarios.

[0004] While existing deep reinforcement learning-based interactive recommendation methods have achieved good results by modeling dynamic user preferences, they have not fully utilized the structural information of the user-item bipartite graph in representing user states and recommendation system behavior. The user-item bipartite graph contains rich user-item interaction connections, which can help the agent mine user collaborative preferences from the target user's neighborhood nodes. However, most existing reinforcement learning-based recommendation methods represent the state or recommendation behavior of individual users in isolation, failing to effectively explore and utilize the potential associations between users, such as behavioral similarities, contained in the user-item bipartite graph. Although a few studies...

[12] Introducing user-item connectivity information by constructing a subgraph for each item in the interaction sequence is beneficial, but directly sampling subgraphs from observed real user-item bipartite graphs can negatively impact performance due to data imbalance. Interactions in real user-item bipartite graphs are uneven due to the long-tail effect. Highly active users or popular items have rich interactions, while most users or items have scarce interactions. Research indicates that both rich and sparse interactions lead to poor recommendation results.

[13] This is because rich interactions can contain noise such as accidental clicks, while sparse interactions cannot provide sufficient cues of preference.

[0005] Therefore, based on the above analysis, it is urgent to construct an interactively balanced user-item bipartite graph. Compared to the real user-item bipartite graph, this graph removes redundant noise edges and adds missing positive interaction edges. Building upon this, we can further extract useful graph structure information to enrich and enhance the behavioral and user state representations of the recommender system, thus promoting the effective learning of recommendation strategies. Summary of the Invention

[0006] This invention provides a dynamic feedback recommendation method based on reinforcement learning. It models the interaction between users and the recommendation system as a Markov decision process using reinforcement learning techniques, and optimizes the recommendation strategy using dynamic user feedback. It uses causal inference to predict the counterfactual user-item bipartite graph of complete user-item interaction, and combines this with the real user-item bipartite graph to generate an interactively balanced user-item bipartite graph, preventing redundant noise interference while supplementing potential user preference information. Furthermore, it utilizes graph convolution techniques to aggregate neighborhood user information of items, enriching and enhancing agent action representations and user state representations through graph structure associations, as detailed below:

[0007] A dynamic feedback recommendation method based on reinforcement learning, the method comprising:

[0008] By using causal reasoning techniques to pre-train a user behavior simulator, a counterfactual user-item bipartite graph of complete user-item interaction is generated.

[0009] We aggregate real user-item bipartite graphs and counterfactual user-item bipartite graphs using weighted aggregation, and generate an interaction-balanced user-item bipartite graph using Top-K sampling.

[0010] By utilizing the neighborhood user information of graph convolution polymer products, graph-enhanced action representations are obtained.

[0011] By using gated recurrent units and self-attention mechanisms to process the user's interaction sequence with the recommendation system, a user state representation at each time step is obtained.

[0012] Based on action representation and user state representation, the Q-value is predicted. The action decision of the recommendation model is optimized based on user reward feedback, and finally, dynamic feedback recommendation of Internet information is achieved.

[0013] The counterfactual user-item bipartite graph is a user-item bipartite graph predicted using the counterfactual concept in causal reasoning techniques.

[0014] Furthermore, the real user-item bipartite graph is specifically a user-item bipartite graph constructed based on the collected historical interaction data of users and items.

[0015] Specifically, the step of obtaining graph-enhanced action representations by utilizing the neighborhood user information of graph convolutional polymer products is as follows:

[0016] For each item in the interactively balanced user-item bipartite graph, a subgraph is constructed. The subgraph consists of the item node and its one-hop neighboring user node in the interactively balanced user-item bipartite graph. Graph convolution is applied to the constructed subgraph to aggregate the neighboring user information of the item and obtain graph-enhanced action representations.

[0017] Furthermore, the user-item bipartite graph predicted using the counterfactual approach in causal reasoning techniques is specifically as follows:

[0018] The generation process of the user-item bipartite graph is described using a structural causal model. A causal graph is constructed, and a behavioral simulator is used to analyze the structural equation f. G Modeling is performed, and the simulator is pre-trained to fit observed factual user-item interactions; a behavioral simulator is built using a graph convolutional network, and the behavioral simulator is trained and optimized using BPR loss.

[0019]

[0020] Where O represents the training sample set of the pre-trained user behavior simulator, i + i represents the positive sample interaction item for user u. - h represents negative sample interaction items that did not interact with user u. u and h i It is a user and item embedding representation encoded by a user behavior simulator, where σ(·) represents the sigmoid function, · T This represents the transpose function; it involves truncating all edges entering node T and setting T to a recommended list t containing all items under the full-exposure strategy. * To achieve causal intervention on T; for the recommended list t after intervention * Using a pre-trained behavioral simulator, predict user u's counterfactual preference scores for all items, integrating each user's... The predicted scores yielded a counterfactual user-item bipartite graph.

[0021] The weighted aggregation of the real user-item bipartite graph and the counterfactual user-item bipartite graph, using Top-K sampling to generate an interaction-equilibrium user-item bipartite graph, is as follows:

[0022] Aggregate Counterfactual User-Item Bipartite Diagram R * And real users - items two-part diagram R is the diagram

[0023]

[0024] Where β represents the weight hyperparameter, and Top-K sampling is performed on the interaction edges of each user or item in the ensemble graph. For each user u, the same hyperparameter K is used. b Controlling the number of neighborhood nodes in the bipartite graph for Top-K sampling:

[0025]

[0026] in, This represents a user-item bipartite graph. The vector in the u-th row of the vector, I, represents the set containing all items, and for each item, the hyperparameter K is used. b Generate the Top-K neighbors; the final interaction equilibrium user-item bipartite graph. The union of user-item interactions generated on both the user and item sides.

[0027] The action representation of the graph structure enhancement is as follows:

[0028]

[0029]

[0030] Among them, W a and w b e represents the trainable weight coefficients. i e represents the embedded representation of item i. u Represents the embedded representation of user u. Indicates a cascading operation. α represents the neighborhood information representation obtained through attention-based aggregation operations. iu W represents the attention score that determines how much feature information user u will pass to item i. f and b f x represents the trainable weights and biases of the fully connected layer. i This represents the graph-enhanced action representation corresponding to item i.

[0031] A second aspect is a dynamic feedback recommendation device based on reinforcement learning, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to perform the method described in any of the first aspects.

[0032] Third aspect, a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, the computer program including program instructions, which, when executed by a processor, cause the processor to perform the method described in any one of the first aspects.

[0033] The beneficial effects of the technical solution provided by this invention are:

[0034] 1. This invention uses reinforcement learning technology to model the dynamic interaction process between users and the recommendation system as a Markov decision process, and uses user feedback to dynamically optimize the recommendation strategy. It can model changes in user interests and maximize users' long-term satisfaction with the recommendation system.

[0035] 2. This invention uses causal reasoning technology to generate a counterfactual user-item bipartite graph of complete user-item interaction, and combines it with the real user-item bipartite graph to construct an interactively balanced user-item bipartite graph. This can effectively avoid noise interference caused by redundant and false interaction edges in the real user-item bipartite graph, and supplement the potential preference information of users or items with sparse interaction.

[0036] 3. This invention introduces graph structure information into interactive items and uses graph convolution technology to mine the collaborative preferences of neighboring users of the target user. This can effectively enhance and enrich the action representation and user state representation of the recommendation system, and achieve high-quality recommendation strategy learning. Attached Figure Description

[0037] Figure 1 This is a flowchart of a dynamic feedback recommendation method based on reinforcement learning;

[0038] Figure 2 The causal graph is constructed. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0040] Example 1

[0041] The algorithm involved in this embodiment of the invention is a recommendation algorithm based on reinforcement learning. In reinforcement learning, the interaction process between the recommendation system and the user can be modeled as a Markov decision process. In short, the recommendation system engages in a series of item recommendation interactions with the user. At each time step, the recommendation system (i.e., the agent) recommends an item (i.e., performs an action) to the user (i.e., the environment) based on the user's preference state (i.e., the state). After receiving the recommended item, the user provides feedback on their satisfaction with the recommended item to the recommendation system, i.e., provides satisfaction feedback (i.e., a reward). The recommendation system model updates and optimizes the recommendation strategy based on the satisfaction feedback, thereby recommending more satisfactory items to the user in the future (i.e., making better action decisions). Agent, state, environment, action, and reward are all fundamental technical terms in reinforcement learning, and this embodiment of the invention will not elaborate on them.

[0042] This invention provides a dynamic feedback recommendation method based on reinforcement learning. (See also...) Figure 1 The method includes the following steps:

[0043] 101: Construct a reinforcement learning recommendation framework that models the dynamic interaction process between users and the recommendation system as a Markov decision process;

[0044] In essence, each interaction between a user and the recommender system is recorded as a sequence. During each interaction, the recommender system is treated as an agent, sequentially recommending an item (i.e., performing an action) to the user (i.e., the environment). The user then generates corresponding satisfaction feedback (i.e., a reward) based on the recommended item. The goal of the recommender system is to maximize the long-term cumulative user satisfaction throughout the entire interaction process.

[0045] 102: Using causal reasoning techniques to pre-train a user behavior simulator to generate a counterfactual user-item bipartite graph showing the complete interaction between users and items;

[0046] The user-item bipartite graph is a graph structure used to describe the relationship between users and items. In this graph, nodes represent users or items, and edges represent the interaction between users and items. The counterfactual user-item bipartite graph, proposed in this embodiment of the invention, is a user-item bipartite graph predicted using the counterfactual concept in causal reasoning techniques.

[0047] 103: Weighted aggregation of real user-item bipartite graphs and counterfactual user-item bipartite graphs, using Top-K sampling method to generate an interaction-balanced user-item bipartite graph;

[0048] Among them, the real user-item bipartite graph refers to the user-item bipartite graph constructed based on the collected historical interaction data of users and items.

[0049] 104: Utilize graph convolution technology to collect neighborhood user information of polymer products to obtain graph-enhanced action representations;

[0050] This step involves constructing a subgraph for each item in the interactively balanced user-item bipartite graph. The subgraph consists of the item node and its one-hop neighboring user nodes in the interactively balanced user-item bipartite graph. This step then applies graph convolution techniques to the constructed subgraph to aggregate the neighboring user information of the item, obtaining a graph-enhanced action representation.

[0051] 105: Utilize gated recurrent units and self-attention mechanisms to process the user's interaction sequence with the recommendation system and obtain the user's state representation at each time step;

[0052] The gated loop unit is well known to those skilled in the art, and will not be described in detail in this embodiment of the invention.

[0053] 106: Predict Q-values ​​based on action representations and user state representations, and optimize action decisions of the recommendation model based on user reward feedback.

[0054] In this system, the recommender (i.e., the agent) uses the predicted Q-value to select an action (i.e., recommend an item to the user). This typically involves an exploration strategy, such as an ε-greedy strategy. In this strategy, the agent randomly selects an action with probability ε (called "exploration" in reinforcement learning) and selects the action with the highest Q-value with probability 1-ε (called "exploitation" in reinforcement learning). The Q-value is a technical term specific to this art, and will not be elaborated upon in the embodiments of this invention.

[0055] In summary, this embodiment of the invention, through steps 101-106, uses a reinforcement learning paradigm to model the dynamic interaction process between users and the recommendation system as a Markov decision process, dynamically optimizing the recommendation strategy using user satisfaction feedback on recommended items. During the modeling process, graph convolution technology is used to introduce user-item bipartite graph information, enhancing and enriching the action representation of the recommendation system and the state representation of the user through graph structure. The counterfactual approach in causal reasoning is used to improve the introduced user-item bipartite graph, effectively avoiding noise interference from spurious interaction edges and supplementing beneficial potential positive interaction information. The recommendation method designed in this embodiment of the invention can effectively optimize long-term user satisfaction with the recommendation system through dynamic feedback information, enhance user stickiness of the recommendation system, and fully understand user preferences through high-quality graph structure information.

[0056] Example 2

[0057] The scheme in Example 1 will be further described below with specific calculation formulas and examples:

[0058] Step 201: Construct a reinforcement learning recommendation framework, modeling the dynamic interaction process between users and the recommendation system as a Markov decision process;

[0059] The step is as follows: Assume the recommender system has a user set consisting of m users. A collection of n items The observed user-item implicit feedback matrix, i.e., the user-item bipartite graph, is as follows: If user u provides positive feedback to item i, such as clicking or viewing, then r ui =1, otherwise r ui =0. Reinforcement learning models the temporal dynamic interaction process between the user and the recommender system as a Markov decision process. In each interaction trajectory, the recommender system (i.e., the agent) interacts with the target user u at discrete time steps t = 0, ..., T-1. At each time step t, the recommender agent observes a user state s. t (i.e., the current environmental situation faced by the agent), and use its recommended strategy π θ (a t |st (This indicates how to select the recommended action given the state) Take an action a accordingly. t (i.e., recommending an item to the user) As action a) t As a result, the recommending agent receives an immediate satisfaction reward from user u. And observe the next state s t+1 The training objective of the recommendation model is to learn a recommendation policy π. θ (a t |s t This allows the system to maximize long-term cumulative user satisfaction rewards within the T-step interaction trajectory.

[0060]

[0061] Where τ={s0,a0,s1,a1,…,s T-1 ,a T-1 Let} represent a user-recommendation system interaction trajectory over T time steps, θ represent the parameters of the recommendation policy network, and γ represent the discount factor used to balance current and future rewards, with a value in the range [0,1]. In this embodiment of the invention, a widely adopted deep Q-Learning method is used to estimate the recommendation policy. This method uses a Q-network Q(s,a;θ) to approximate the optimal action-value function Q. * (s,a).

[0062] Step 202: Use causal reasoning techniques to pre-train a user behavior simulator to generate a counterfactual user-item bipartite graph showing the complete interaction between users and items;

[0063] In this embodiment of the invention, user-item bipartite graph structural information is introduced to enhance action representation and user state representation. Directly using a real user-item bipartite graph with uneven interactions is suboptimal because it introduces noise interference for users or items with rich interactions, while providing insufficient information for users or items with sparse interactions. Therefore, this embodiment of the invention constructs an interaction-balanced user-item bipartite graph for graph structure information extraction. To obtain an interaction-balanced user-item bipartite graph and supplement the potentially missing positive interaction edges between users and items, this embodiment of the invention generates a fully interactive counterfactual user-item bipartite graph by pre-training a behavior simulator using a counterfactual approach.

[0064] First, use a structural causal model.

[14] Describe the process of generating the user-project bipartite diagram. Construct a cause-effect graph. like Figure 2 As shown, Figure 2(a) shows the causal graph generated from the real user-item bipartite graph, containing four node variables: E represents a partial exposure strategy, presenting only a subset of items to the user; T represents the recommendation list under the partial exposure strategy; U represents the user; and G represents the user-item bipartite graph. The two main causal paths in the causal graph are: (1) E→T represents the fact that the recommendation list is generated according to the exposure strategy of the recommendation system. (2) (U,T)→G represents that the user-item bipartite graph is constructed based on the user u's preference for each item in the recommendation list t. Each element R in graph G = R ui This represents the interaction feedback of user u to item i, for example, R. ui =1 indicates a click, R ui =0 indicates no click. The causal relationship between graph G and its ancestor nodes U and T can be represented as G = f G (U,T), where f G This represents a structural equation.

[0065] Then, use a behavior simulator to test f G Modeling is performed, and the simulator is pre-trained to fit observed factual user-item interactions. The behavior simulator can be implemented using various established recommendation models. In this embodiment of the invention, a graph convolutional network is used.

[15] Build a behavior simulator and train and optimize it using BPR loss:

[0066]

[0067] Where O represents the training sample set of the pre-trained user behavior simulator, i + i represents the positive sample interaction item for user u. - h represents negative sample interaction items that did not interact with user u. u and h i It is a user and item embedding representation encoded by a user behavior simulator, where σ(·) represents the sigmoid function, · T This represents the transpose function.

[0068] Next, causal intervention is implemented. Under the partial exposure strategy, the number of items in the recommendation list T is limited, making it impossible to obtain unobserved positive user-item interactions. Therefore, by truncating all edges entering node T and setting T to a recommendation list t containing all items under the full exposure strategy... * To achieve causal intervention in T, such as Figure 2 As shown in (b).

[0069] Finally, counterfactual reasoning is employed. For the recommended list t after intervention... * The system uses a pre-trained behavioral simulator to predict user u's counterfactual preference scores for all items. It integrates each user's... The predicted scores yielded a counterfactual user-item bipartite graph.

[0070] Step 203: Weighted aggregation of real user-item bipartite graphs and counterfactual user-item bipartite graphs, and generation of an interaction-balanced user-item bipartite graph using the Top-K sampling method;

[0071] Comparing the real user-item bipartite graph and the counterfactual user-item bipartite graph reveals two types of edges. One type has a weight of 1 (representing a click) in the real user-item bipartite graph but a lower weight (representing dislike) in the counterfactual user-item bipartite graph. This type of edge indicates that the observed positive interaction may be spurious; that is, it is a noise edge. Another type of edge has a weight of 0 (representing no click) in the real user-item bipartite graph but a high weight (representing a like) in the counterfactual user-item bipartite graph. This type of edge may be due to insufficient exposure, resulting in unobserved potential positive user-item interactions; therefore, it is called a missing edge. The above analysis shows that considering both real and counterfactual user-item bipartite graphs helps identify noise edges and missing edges. Therefore, aggregating the counterfactual user-item bipartite graph R... * And real users - items two-part diagram R is the diagram

[0072]

[0073] Here, β represents the weight hyperparameter. To construct an interactively balanced user-item bipartite graph, Top-K sampling is performed on the interaction edges of each user or item in the ensemble graph. For each user u, the same hyperparameter K is used. b Controlling the number of neighborhood nodes in the bipartite graph for Top-K sampling:

[0074]

[0075] in, This represents a user-item bipartite graph. The vector in the u-th row of the vector, I, represents the set containing all items. For each item, a hyperparameter K is also used. b A similar process is used to generate the Top-K neighbors. This achieves denoising for users / items with a large number of neighbors in the real user-item bipartite graph, and augmentation for users / items with a small number of neighbors. The final interaction-balanced user-item bipartite graph... The union of user-item interactions generated on both the user and item sides.

[0076] Step 204: Utilize graph convolution technology to collect neighborhood user information of polymer products and obtain graph-enhanced action representations;

[0077] Representing a single user's state or behavior in isolation fails to effectively explore and utilize the potential relationships between different users. To address this issue, each item i in the user-recommendation system interaction sequence is replaced with a subgraph G(i) to fully leverage graph structure information to enrich and enhance behavioral and user state representations. The subgraph G(i) is a user-item bipartite graph with balanced interactions. Item i and its one-hop neighbors Composition. For the subgraph corresponding to item i, a graph convolutional network is used to extract graph structure information, mine potential relationships such as behavioral similarities between the target user and other users, and obtain graph-enhanced action representations:

[0078]

[0079]

[0080] Among them, W a and w b e represents the trainable weight coefficients. i e represents the embedded representation of item i. u Represents the embedded representation of user u. Indicates a cascading operation. α represents the neighborhood information representation obtained through attention-based aggregation operations. iu W represents the attention score that determines how much feature information user u will pass to item i. f and b f x represents the trainable weights and biases of the fully connected layer. i This represents the graph-enhanced action representation corresponding to item i.

[0081] Step 205: Use gated recurrent units and self-attention mechanisms to process the user's interaction sequence with the recommendation system to obtain the user state representation at each time step;

[0082] The target user u's initial state at time step t is the sequence of user-interacted items {a0,...,a1} prior to time step t. t-1 The graph structure is used to represent the user state x(s) in the sequence. After enriching each interactive item in the sequence with graph structure information, the graph-enhanced user state x(s) can be obtained. t )={x0,...,x t-1 To model the sequential information in user states, gated cyclic units are used to further process the graph-enhanced state representation, defining {x0,...,x} as follows: t-1} is converted into a hidden vector sequence {g0,...,g t-1}:

[0083] g j=GRU(g j-1 x j )

[0084] Wherein, GRU(·) represents a gated loop unit.

[0085] Then, a self-attention mechanism is used to capture the importance of different items in the state, resulting in the final user state representation:

[0086]

[0087] Where, η j w represents the attention score. at and W sa This represents the trainable weights in the self-attention mechanism. This represents the final user state representation at time step t.

[0088] Step 206: Predict the Q-value based on action representation and user state representation, and optimize the action decision of the recommendation model based on user reward feedback;

[0089] Obtain user state representation and action representation x a Using a multilayer perceptron to fuse useful feature information from both, the final action value Q(s) is predicted. t a). The network parameters are learned by training the recommendation model by minimizing the following loss function:

[0090]

[0091] in, Let θ represent the optimization function of the recommendation model, and let θ represent all trainable parameters of the Q-network. Let θ represent the optimization objective of the current iteration, r represent the user reward, γ represent the discount factor balancing the importance of future rewards and immediate rewards, and θ represent the user reward. - This indicates that the Q-network parameters from the previous iteration are fixed during the recommendation model optimization.

[0092] In actual training, the above loss function optimization can be simplified to performing stochastic gradient descent on the sampled transition (s,a,r,s′):

[0093]

[0094] in, This indicates the gradient calculation, where α represents the learning rate hyperparameter.

[0095] Example 3

[0096] The effectiveness of the schemes in Examples 1 and 2 is evaluated below based on specific experimental data, as detailed in the following description:

[0097] 301: Data Composition

[0098] This invention uses three publicly available recommendation datasets: LastFM, ML1M, and Pinterest. All datasets contain user-item feedback data. This invention focuses on implicit feedback; therefore, following common practice, different user behaviors towards items are treated as uniform implicit positive feedback. To ensure the interaction sequence length is sufficient for training the reinforcement learning model, items with fewer than 5 feedback instances in all datasets, and users with fewer than 5, 20, and 30 feedback instances in the LastFM, ML1M, and Pinterest datasets, respectively, were removed. The statistical results of the processed datasets are shown in Table 1.

[0099] Table 1 shows the statistics of the recommended dataset used.

[0100]

[0101] 302: Evaluation Indicators

[0102] The goal of interactive recommendation is to obtain the optimal long-term reward throughout the entire interaction process. Therefore, this embodiment of the invention uses the average reward, i.e., the average reward obtained in an interaction trajectory over a time step of T, as the evaluation metric for algorithm performance. In this embodiment, T is set to 20. This embodiment randomly selects 80% of the users as the training set and the remaining 20% ​​as the test set to segment each dataset. Experiments are conducted on the segmented datasets obtained under five different random seeds, and the average of the five experimental results is reported to ensure reliability.

[0103] 303: Comparison Algorithm

[0104] In comparative experiments, the embodiments of this invention were compared with nine baseline recommendation methods on the LastFM, ML1M, and Pinterest datasets. Three of these were non-reinforcement learning recommendation methods, and six were reinforcement learning-based recommendation methods. The non-reinforcement learning recommendation methods included: SVD.

[16] **Device Relationships (DRR)** is a matrix factorization model that uses the inner product of latent feature vectors to predict user-item relevance scores. **Popular** is a simple yet powerful baseline method that selects items with the most positive feedback for recommendation. **Random** randomly selects items for recommendation and can be seen as an indicator of the task's inherent difficulty. Reinforcement learning-based recommendation methods include: DEERS.

[17] , is a method based on DQN, which uses GRU to learn state representations from positive and negative feedback item sequences; LSTM-Q, utilizes LSTM

[18] Learning state representations from item sequences and learning action representations from item IDs using embedding layers; GRU-Q, similar to LSTM-Q, but different in that it uses GRUs to learn state representations; AttLSTM-Q, which extends LSTM-Q with a self-attention mechanism; AttGRU-Q, which extends GRU-Q with a self-attention mechanism; GCQN.

[19] In the DQN-based approach, action and state representations are learned by introducing user-item subgraph information into interactive items.

[0105] Table 2 shows the performance comparison of the proposed algorithm on the LastFM, ML1M, and Pinterest datasets.

[0106]

[0107]

[0108] Table 2 shows the comparison results of the embodiments of the present invention with existing state-of-the-art baseline methods on the LastFM, ML1M, and Pinterest datasets. The best algorithm performance is bolded, and the second-best algorithm performance is underlined. The experimental results lead to the following conclusions: First, compared with non-reinforcement learning recommendation methods, reinforcement learning-based recommendation methods generally exhibit better recommendation performance. This is intuitive because reinforcement learning-based recommendation methods consider and model the interaction sequence information between the user and the recommendation system, and update the recommendation strategy at each time step using user feedback, aiming to optimize the long-term reward of the entire recommendation sequence. This helps capture the user's dynamic interests and continuously optimize the recommendation system towards user satisfaction. Second, among reinforcement learning-based recommendation methods, GCQN and the present invention significantly outperform the other two methods. This indicates that introducing user-item graph structure information when learning action and user state representations is very beneficial for learning reinforcement learning-based recommendation strategies. Finally, on all three datasets, the recommendation results of the embodiments of the present invention are superior to other comparative algorithms. This is attributed to the fact that the method of this invention constructs an interactively balanced user-item bipartite graph using the concept of counterfactualism, effectively removing noise interference from spurious interaction edges in the original user-item bipartite graph, while supplementing the sparsely interacting users or items with potential preference information. Improving the quality of the user-item bipartite graph helps the graph convolutional network extract more informative subgraph information, thereby further improving the learning effect of recommendation actions and user state representations, and optimizing recommendation performance.

[0109] The above experiments verify that the dynamic feedback recommendation method based on reinforcement learning proposed in this invention can effectively model the dynamic interaction process between users and the recommendation system, and update the recommendation strategy in a timely manner according to user interaction feedback, thereby improving users' long-term satisfaction with the recommendation system. The proposed algorithm innovatively constructs a counterfactual user-item bipartite graph and combines it with the real user-item bipartite graph to generate an interactively balanced user-item bipartite graph. This provides high-quality graph structure information for modeling the dynamic interaction process, enriches the representation of recommendation actions and user states, and effectively improves the recommendation results.

[0110] Example 4

[0111] A dynamic feedback recommendation device based on reinforcement learning includes a processor and a memory. The memory stores program instructions, and the processor invokes the program instructions stored in the memory to cause the device to execute the following method steps in Embodiment 1:

[0112] By using causal reasoning techniques to pre-train a user behavior simulator, a counterfactual user-item bipartite graph of complete user-item interaction is generated.

[0113] We aggregate real user-item bipartite graphs and counterfactual user-item bipartite graphs using weighted aggregation, and generate an interaction-balanced user-item bipartite graph using Top-K sampling.

[0114] By utilizing the neighborhood user information of graph convolution polymer products, graph-enhanced action representations are obtained.

[0115] By using gated recurrent units and self-attention mechanisms to process the user's interaction sequence with the recommendation system, a user state representation at each time step is obtained.

[0116] Based on action representation and user state representation, the Q-value is predicted. The action decision of the recommendation model is optimized based on user reward feedback, and finally, dynamic feedback recommendation of Internet information is achieved.

[0117] Among them, the counterfactual user-item bipartite graph is: a user-item bipartite graph predicted using the counterfactual concept in causal reasoning techniques.

[0118] Specifically, the real user-item bipartite graph is a user-item bipartite graph constructed based on the collected historical interaction data of users and items.

[0119] The specific steps for obtaining graph-enhanced action representations using neighborhood user information from graph convolutional polymer products are as follows:

[0120] For each item in the interactively balanced user-item bipartite graph, a subgraph is constructed. The subgraph consists of the item node and its one-hop neighboring user node in the interactively balanced user-item bipartite graph. Graph convolution is applied to the constructed subgraph to aggregate the neighboring user information of the item and obtain graph-enhanced action representations.

[0121] Specifically, the user-item bipartite graph predicted using the counterfactual approach in causal reasoning is as follows:

[0122] The generation process of the user-item bipartite graph is described using a structural causal model. A causal graph is constructed, and a behavioral simulator is used to analyze the structural equation f. G Modeling is performed, and the simulator is pre-trained to fit observed factual user-item interactions; a behavioral simulator is built using a graph convolutional network, and the behavioral simulator is trained and optimized using BPR loss.

[0123]

[0124] Where O represents the training sample set of the pre-trained user behavior simulator, i + i represents the positive sample interaction item for user u. - h represents negative sample interaction items that did not interact with user u. u and h i It is a user and item embedding representation encoded by a user behavior simulator, where σ(·) represents the sigmoid function, · T This represents the transpose function; it involves truncating all edges entering node T and setting T to a recommended list t containing all items under the full-exposure strategy. * To achieve causal intervention on T; for the recommended list t after intervention * Using a pre-trained behavioral simulator, predict user u's counterfactual preference scores for all items, integrating each user's... The predicted scores yielded a counterfactual user-item bipartite graph.

[0125] The weighted aggregation of real user-item bipartite graphs and counterfactual user-item bipartite graphs, using Top-K sampling to generate an interaction-equilibrium user-item bipartite graph, is as follows:

[0126] Aggregate Counterfactual User-Item Bipartite Diagram R * And real users - items two-part diagram R is the diagram

[0127]

[0128] Where β represents the weight hyperparameter, and Top-K sampling is performed on the interaction edges of each user or item in the ensemble graph. For each user u, the same hyperparameter K is used. bControlling the number of neighborhood nodes in the bipartite graph for Top-K sampling:

[0129]

[0130] in, This represents a user-item bipartite graph. The vector in the u-th row of the vector, I, represents the set containing all items, and for each item, the hyperparameter K is used. b Generate the Top-K neighbors; the final interaction equilibrium user-item bipartite graph. The union of user-item interactions generated on both the user and item sides.

[0131] Among them, the action representation of graph structure enhancement is:

[0132]

[0133]

[0134] Among them, W a and w b e represents the trainable weight coefficients. i e represents the embedded representation of item i. u Represents the embedded representation of user u. Indicates a cascading operation. α represents the neighborhood information representation obtained through attention-based aggregation operations. iu W represents the attention score that determines how much feature information user u will pass to item i. f and b f x represents the trainable weights and biases of the fully connected layer. i This represents the graph-enhanced action representation corresponding to item i.

[0135] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0136] The aforementioned processor and memory can be devices with computing capabilities, such as computers, microcontrollers, or single-chip microcomputers. In specific implementations, this embodiment of the invention does not limit the execution entity and allows selection based on the needs of the actual application. Data signals are transmitted between the memory and processor via a bus, which will not be elaborated upon in this embodiment.

[0137] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, which includes a stored program that, when executed, controls the device where the storage medium is located to perform the method steps described in the above embodiments. This computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc. It should be noted that the description of the readable storage medium in the above embodiments corresponds to the method description in the embodiments, and will not be repeated here.

[0138] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0139] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.

[0140] References

[0141] [1]H.Guo, R.Tang, Y.Ye, Z.Li, and X.He, "DeepFM: Afactorization-machinebased neural network for CTR prediction," in Proc.26th Int.JointConf.Artif.Intell., 2017, pp.1725-1731.

[0142] [2]

[0143] [3]L.Li,W.Chu,J.Langford,and R.E.Schapire,“Acontextual-banditapproach to personalized news article recommendation,”in Proc.19thInt.Conf.World Wide Web,2010,pp.661-670.

[0144] [4]G.Shani,D.Heckerman,and R.I.Brafman,“An MDP-based recommendersystem,”J.Mach.Learn.Res.,vol.6,pp.1265-1295,2005.

[0145] [5]G.Zheng et al.,“DRN:A deep reinforcement learning framework fornews recommendation,”in Proc.World Wide Web Conf.,2018,pp.167-176.

[0146] [6]R.J.Williams,“Simple statistical gradient-following algorithms forconnectionist reinforcement learning,”Mach.Learn.,vol.8,pp.229-256,1992.

[0147] [7]X.Zhao et al.,“Deep reinforcement learning for page-wiserecommendations,”in Proc.ACM Conf.Recommender Syst.,2018,pp.95-103.

[0148] [8]V.R.Konda and J.N.Tsitsiklis,“Actor-critic algorithms,”inProc.Annu.Conf.Neural Inf.Process.Syst.,1999,pp.1008-1014.

[0149] [9]K.Wang et al.,“Reinforcement learning with a disentangleduniversal value function for item recommendation,”inProc.AAAIConf.Artif.Intell.,2021,pp.4427-4435.

[0150]

[10] D.Hong,Y.Li,and Q.Dong,“Nonintrusive-sensing and reinforcementlearning based adaptive personalized music recommendation,”in Proc.Int.ACMSIGIR Conf.Res.Develop.Inf.Retrieval,2020,pp.1721-1724.

[0151]

[11] W.Nie et al.,“Knowledge-enhanced causal reinforcement learningmodel for interactive recommendation,”IEEE Trans.Multimedia,vol.26,pp.1129-1142,Nov.2024.

[0152]

[12] Y.Lei et al.,“Reinforcement Learning based Recommendation withGraph Convolutional Q-network,”in Proc.Int.ACM SIGIRConf.Res.Develop.Inf.Retrieval,2020,pp.1757-1760.

[13] Z.Fan et al.,“GraphCollaborative Signals Denoising and Augmentation for Recommendation,”inProc.Int.ACM SIGIR Conf.Res.Develop.Inf.Retrieval,2023,pp.2037-2041.

[0153]

[14] Z.Fan et al., "Top-N Recommendation with Counterfactual UserPreference Simulation," in Proceedings of ACM International Conference on Information and Knowledge Management, 2021, pp.2342-2351.

[0154]

[15]

[0155]

[16] Y.Koren, RMBell, and C.Volinsky, "Matrix Factorization Techniques for Recommender Systems," Computer, vol.42, no.8, pp.30-37, 2009.

[0156]

[17]

[0157]

[18] S.Hochreiter and J.Schmidhuber, "Long short-term memory," NeuralComput., vol.9, no.8, pp.1735-1780, 1997.

[0158] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0159] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0160] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic feedback recommendation method based on reinforcement learning, characterized in that, The method includes: A reinforcement learning recommendation framework is constructed, which models the dynamic interaction process between users and the recommendation system as a Markov decision process. By using causal reasoning techniques to pre-train a user behavior simulator, a counterfactual user-item bipartite graph of complete user-item interaction is generated. We weighted aggregate real user-item bipartite graphs and counterfactual user-item bipartite graphs, and used Top-K sampling to generate interactively balanced user-item bipartite graphs. We then used graph convolution to aggregate neighborhood user information of items to obtain graph-enhanced action representations. By using gated recurrent units and self-attention mechanisms to process the user's interaction sequence with the recommendation system, a user state representation at each time step is obtained. Based on action representation and user state representation, predict Q value, optimize the action decision of recommendation model based on user reward feedback, and finally realize dynamic feedback recommendation of Internet information; Specifically, a reinforcement learning recommendation framework is constructed, which models the dynamic interaction process between users and the recommendation system as a Markov decision process: By using reinforcement learning, the temporal dynamic interaction process between the user and the recommender system is modeled as a Markov decision process. In each interaction trajectory, the recommender system interacts with the target user. At discrete time steps Interact on; at each time step Recommend that the agent observes a user state. and use recommendation strategies Take a corresponding action As an action As a result, the recommended agent is from the user Receive instant satisfaction reward And observe the next state. The training objective of a recommendation model is to learn a recommendation strategy. So that it can be in The interaction trajectory of each step can maximize the long-term cumulative user satisfaction reward: ； in, Indicates a paragraph User-recommendation system interaction trajectory at each time step This represents the parameters of the recommendation policy network. This represents the discount factor used to balance current and future rewards; Using causal reasoning techniques to pre-train a user behavior simulator, a counterfactual user-item bipartite graph representing the complete interaction between users and items is generated as follows: The user-item bipartite graph predicted using the counterfactual concept in causal reasoning techniques is then analyzed using a behavioral simulator to evaluate the structural equation model. Modeling is performed, and the simulator is pre-trained to fit observed factual user-item interactions; a behavioral simulator is built using a graph convolutional network, and the behavioral simulator is trained and optimized using BPR loss. ； in, This represents the set of training samples for the pre-trained user behavior simulator. Indicates user Positive sample interactive items, This indicates that there was no communication with the user. Negative samples of interactive items and It is an embedded representation of users and items encoded by a user behavior simulator. This represents the sigmoid function; it is achieved by truncating all incoming nodes. The edge and A recommended list containing all items under the full exposure strategy. To achieve Causal intervention; recommendation list after intervention Predicting user behavior using a pre-trained behavior simulator Counterfactual preference scores for all items, integrated for each user The predicted scores yielded a counterfactual user-item bipartite graph; Among them, the real user-item bipartite graph is: a user-item bipartite graph constructed based on the collected historical interaction data of users and items; Considering both true and counterfactual user-item bipartite graphs helps identify noisy edges and missing edges, and aggregates counterfactual user-item bipartite graphs. And real users - item two-part diagram For the image : ； in, This represents the weight hyperparameter, which performs Top-K sampling on the interaction edges of each user or item in the ensemble graph. For each user... Using the same hyperparameters Controlling the number of neighborhood nodes in the bipartite graph for Top-K sampling: ； in, This represents a user-item bipartite graph. The Middle The vector of rows, This represents a set containing all items, using hyperparameters for each item. Generate the Top-K neighbors; the final interaction equilibrium user-item bipartite graph. The union of user-item interactions generated on both the user and item sides; The specific steps for obtaining graph-enhanced action representations using neighborhood user information from graph convolutional polymer products are as follows: For each item in the user's interaction sequence with the recommendation system Use a subgraph Replace the subgraph User-Item Bipartite Diagram Based on Interaction Equilibrium Items in and its neighbor Composition, for items For the corresponding subgraphs, graph convolutional networks are used to extract graph structure information, uncover potential relationships such as behavioral similarities between the target user and other users, and obtain graph-enhanced action representations. The action representation of the graph structure enhancement is as follows: ； ； ； in, and This represents the trainable weight coefficients. Represents items Embedded representation, Indicates user Embedded representation, Indicates a cascading operation. This represents the neighborhood information representation obtained through attention-based aggregation operations. Indicates the decision user How much feature information will be passed to the item? Attention score and This represents the trainable weights and biases of the fully connected layer. Indicates with items The corresponding graph structure-enhanced action representation.

2. A dynamic feedback recommendation device based on reinforcement learning, characterized in that, The device includes a processor and a memory, the memory storing program instructions, and the processor calling the program instructions stored in the memory to cause the device to perform the method of claim 1.

3. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method of claim 1.

Images