Recommendation system optimization method based on user satisfaction

Abstract

The invention discloses a recommendation system optimization method based on user satisfaction, and belongs to the technical field of recommendation systems. According to the method, the decision-making process of the user is modeled into the Markov decision-making process, and it is assumed that the user always tries to maximize the satisfaction degree in the interaction process with the recommendation system. Based on this hypothesis, the user's interactive dataset may be regarded as expert behavior data. Then, the invention provides a method based on inverse reinforcement learning for training to obtain a user satisfaction model. Finally, an auxiliary alignment task is designed, so that the recommendation system maximizes the user satisfaction degree in the recommendation process, the task can be combined with any sequence recommendation model, and alignment of the recommendation system and the user satisfaction degree is achieved. The method has the advantages of being high in universality, wide in application scene and the like, and can be widely applied to various recommendation scenes such as news, music and e-commerce.

Description

Technical Field

[0001] The invention belongs to the technical field of recommendation systems and relates to a recommendation system optimization method based on user satisfaction. Background Art

[0002] The recommendation system aims to filter out content that meets the interests of users based on their historical behavior information. With the advancement of science and technology, the recommendation system has made significant progress in cold start problems, diversity assurance, and improving long-term user engagement, but there are still obvious deficiencies in understanding user behavior and needs. For example, when a user clicks on some news related to a certain topic, the system often continues to recommend more similar news. From the perspective of positive feedback, this recommendation strategy is reasonable, but in fact, the user may have lost interest in the topic because he has obtained enough information. This phenomenon shows that the recommendation strategy that only relies on the display interaction between the user and the recommendation system (such as clicking, browsing, etc.) may have serious deviations from the user's real interest preferences.

[0003] When users consume content recommended by the recommendation system, they will have a subjective feeling about the content, which is defined as user satisfaction in the present invention. Regarding user satisfaction, the present invention has the following two observations: 1. User satisfaction directly affects their interest distribution and subsequent behavior. For example, when a user clicks on a piece of news but finds that the content is repeated with the previously read content, the satisfaction may be low, and thus they are unwilling to click on similar content again; 2. Users usually tend to maximize their own satisfaction when interacting with the recommendation system. This is in line with everyday intuition, because users prefer content that can bring more emotional value or pleasure. Therefore, the recommendation system should not only pay attention to the user's explicit feedback, but also be able to maximize the user's satisfaction. However, since satisfaction is a subjective feeling of the user, it is usually unknown to the recommendation system, so new technical means are needed to help the recommendation system align with user satisfaction during the interaction with the user.

[0004] In recent years, alignment algorithms have been proposed in the field of natural language processing to guide large-scale language models (LLMs) to generate content that is more in line with human values. However, research on using alignment algorithms to improve user experience in the field of recommendation systems is still in its early stages. At the same time, the challenges faced by recommendation systems on this issue are significantly different from those of LLMs: in LLMs, due to the lack of sufficient labeled data (such as conversation quality), the research focus is on how to construct labeled data that conforms to human values ​​and input it into the model for training; while in recommendation systems, there are usually a large number of records of the interaction trajectories between users and the system as labeled data, and no manual labeling is required. Therefore, the key challenge of the alignment problem of recommendation systems is to mine user satisfaction information from the user's interaction trajectory and learn how to align with the user's actual satisfaction.

[0005] Existing recommendation algorithms include: traditional recommendation algorithms, including content-based recommendation algorithms and collaborative filtering recommendation algorithms; serialized recommendation algorithms; and recommendation algorithms based on reinforcement learning. Among them, traditional recommendation algorithms regard user-item interaction behaviors as isolated events, which can only mine users' static preferences and cannot capture dynamic changes in user interests. To solve this problem, serialized recommendation algorithms consider users' historical behavior sequences and predict items that may be of interest in the next step, significantly improving the personalization and dynamic adaptability of recommendations. The recommendation algorithm based on reinforcement learning models the recommendation process as a Markov decision process and generates reward signals to optimize the recommendation strategy to improve users' long-term satisfaction. However, existing algorithms still cannot effectively solve the following two problems:

[0006] (1) Existing algorithms can imitate user behavior based on user historical data, but they cannot understand the real motivation behind user behavior. When user interests change, the prediction performance of existing models often drops significantly.

[0007] (2) The reward signals used by existing algorithms are generally generated by complex rule-based models. These reward signals often differ from the user’s actual preferences, which may mislead the optimization direction of the recommendation system and lead to poor quality recommendation results. Summary of the invention

[0008] In view of the problem that the current recommendation system algorithm deviates from user satisfaction, the present invention proposes a recommendation system optimization method based on user satisfaction.

[0009] In the present invention, the motivation and interest behind the user behavior are firstly learned by using the interaction data between the user and the system, and the model is formed into a user satisfaction model; then the model is used to guide the training of the backbone recommendation system model, so as to realize the alignment between the recommendation system and the user satisfaction. The most critical issue in this process is how to quantify the satisfaction obtained by the user when consuming the recommended content, that is, how to train the user satisfaction model. Since the satisfaction is hidden behind the user behavior, it is very challenging to directly learn the user satisfaction model. To this end, the present invention first models the user's decision-making process as a Markov decision process (MDP), and assumes that the user always tries to maximize the satisfaction in the process of interacting with the recommendation system. Based on this assumption, the user's interaction data set can be regarded as expert behavior data. Subsequently, the present invention proposes a method based on inverse reinforcement learning, which mines the user satisfaction model hidden behind the expert behavior data. Finally, the present invention designs an auxiliary task to guide the recommendation system to maximize the user satisfaction in the process of recommendation. The task can be combined with any sequential recommendation model to realize the alignment between the recommendation system and the user satisfaction.

[0010] On this basis, the method of the present invention is roughly divided into two stages:

[0011] (1) User satisfaction model training phase: The present invention trains the user satisfaction model through inverse reinforcement learning technology. The goal of traditional reinforcement learning is to train the agent strategy to maximize its cumulative reward based on the known environment transfer function and reward function; while the goal of inverse reinforcement learning is to derive a reward function from a given expert strategy trajectory to maximize the probability that the agent will generate an expert strategy trajectory under the reward function. In the present invention, the user is regarded as an agent, the recommendation system is regarded as the environment, and the historical interaction data between the user and the recommendation system is regarded as the expert strategy trajectory. At the same time, it is assumed that the user always follows the optimal strategy when interacting with the recommendation system, that is, the user always chooses the behavior that maximizes his own reward. Based on this assumption, the present invention formalizes the user satisfaction model as a reward model in inverse reinforcement learning, and restores the implicit user satisfaction by analyzing the user's historical interaction data. This process effectively solves the problem of directly quantifying the user's subjective satisfaction and provides a reliable guidance signal for the subsequent optimization of the recommendation system.

[0012] (2) Recommendation system training and optimization stage: The present invention mainly considers the sequence recommendation system model. This type of recommendation system model takes the user interaction history sequence as input to predict the next item that will most attract the user's interest. Since the user's satisfaction cannot be directly quantified, the recommendation system cannot maximize the user's satisfaction during the training process, so the recommendation system usually deviates from the user's real interests. In order to solve this problem, the present invention designs an auxiliary task to optimize the recommendation system using the user satisfaction model trained in the first stage, so that the recommendation system can be aligned with the user's interests. Specifically, the present invention designs a new training goal to ensure that during the training stage, not only the original goal of the recommendation system can be met, but also user satisfaction can be maximized.

[0013] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0014] A recommendation system optimization method based on user satisfaction, the specific steps are as follows:

[0015] Step 1: Set the problem model and conduct mathematical modeling:

[0016] 1.1) Markov Decision Process (MDP): In reinforcement learning or inverse reinforcement learning, the problem is usually modeled using the Markov decision process, which is represented by a five-tuple<S,A,p,r,π> . Where S represents the state space, A represents the action space, p represents the environment transfer equation, r represents the reward function, and π represents the strategy function. The present invention models the Markov process from the user's perspective, that is, the user is regarded as an intelligent agent and the recommendation system is regarded as the environment. The specific modeling method is as follows:

[0017] State space S: s t ∈S represents the state of the user at time t. The present invention defines the user state as: s t =(h t-1 ,i t ), where h t-1 =(σ1,σ2,…,σ t-1 ) is the interaction history at time t-1, and each interaction σ=<u,i,a> , u is the user feature, i is the item the user interacts with, and a is the user action.

[0018] Action space A: User action a∈A represents the user's feedback on the interactive item. In different scenarios, user feedback is also varied, such as click, purchase, like, forward, etc. In this invention, in order to simplify the scale of the problem, user feedback is divided into two categories: A={a P ,a N}, where a P Represents positive feedback from users, a N Represents negative user feedback.

[0019] Transition function p: S×A×A→[0,1]. When the user performs action a at time t t After that, it will transfer to the new state s according to the transfer function t+1 .

[0020] Reward function r: The present invention models from the user's perspective, so the reward function r is a quantitative representation of the user's satisfaction after consuming the items recommended by the recommendation system. This function is also the main model to be trained in the present invention.

[0021] Strategy π: User strategy is a mapping function from state space to action space. This paper considers that the optimal strategy π is used in the interaction between the user and the recommendation system. * , that is, the user always tends to maximize his own satisfaction, which is the reward function.

[0022] 1.2) Modeling the alignment problem of recommendation system: The goal of this invention is to optimize the recommendation system model to align it with the user satisfaction model. Here, alignment means maximizing user satisfaction during the entire interaction process between the user and the recommendation system. Specifically, this invention uses the reward function r(s,a) to quantify user satisfaction, where s =<h,i> is the user state, h is the user interaction history, i is the item currently interacting with the user, and a is the user action. The goal of alignment is to maximize ∑ t η·r(s t ,a t ), where i t is the item recommended by the recommendation system at time t, a t is the user's action at time t, and η is the decay coefficient, which is used to balance short-term rewards and long-term rewards.

[0023] Step 2: User satisfaction model training

[0024] The present invention models the user satisfaction model as the reward function in the Markov decision process, and solves the reward function under the condition of known user state sequence and corresponding action. In this step, the classic algorithm IQ-Learn in inverse reinforcement learning is used to solve the reward function. The algorithm uses the soft-Q function to model the user's state-action value Q(s,a), and uses the state transition probability and strategy distribution to derive the analytical expression of the reward function, thereby avoiding the complexity of directly modeling the reward function. The specific steps are as follows:

[0025] 2.1) Data processing, building experience pool D. The existing recommendation system dataset cannot be directly used to train the user satisfaction model. The purpose of this step is to process the recommendation system dataset so that it meets the Markov decision process modeling method in step 1, which is convenient for subsequent model training.

[0026] 2.2) Randomly initialize the agent soft-Q model Q w (s,a) Network parameters w.

[0027] 2.3) Use the training data to train the model parameters w.

[0028] 2.3.1) Sample m samples from the experience pool D calculate V j (s t ) and V j (s t+1 ).in, are the state and action of the user at time t in the jth sample, is the state of the user in the jth sample at time t+1; is the action state value function value of the user at time t in the jth sample, V j (s t ), V j (s t+1 ) are the user s in the jth sample t 、s t+1 The state value function value of .

[0029] Q and V satisfy the soft-Bellman equation:

[0030]

[0031] 2.3.2) Calculate the inverse reinforcement learning loss L IRL :

[0032]

[0033] Where α is a hyperparameter.

[0034] 2.3.3) Calculate the Reward Distinction Enlargement (RDE) loss L RDE :

[0035]

[0036] This regularization term improves the explanatory power of the reward function for user behavior by amplifying the differences in reward signals between different user behaviors (such as clicks and no-clicks), so that the learned user satisfaction model can more accurately reflect the user's true preferences.

[0037] 2.3.4) Calculate the total loss and update Q by backpropagating the gradient of the neural network w (s,a) All parameters w of the network.

[0038] loss = L IRL +β·L RDE (6)

[0039] Among them, β is a hyperparameter used to balance the loss weight of the regularization term.

[0040] In this way, we can calculate the user's satisfaction r(s) when performing action a in state s. t ,a t ).

[0041] r(s t ,a t )=Q w (s t ,a t )-γV(s t+1 )(7)

[0042] Where γ is a hyperparameter, V(s t+1 ) is the user in state s t+1 The state value function value under .

[0043] Step 3: Use the user satisfaction model to optimize the recommendation system model training

[0044] The goal of this stage is to align the recommendation system with the user satisfaction model during the training process. In step 2, the user satisfaction model r(s t ,a t ). The goal of the alignment problem is to make the recommendation system recommend item i t At the same time, maximize the cumulative reward Σ t η·r(s t ,a t ).

[0045] 3.1) Initialize the recommendation system data experience pool and recommendation system model Δ ψ .

[0046] 3.2) Align the training recommendation system model and the user satisfaction model:

[0047] 3.2.1) Extract n pieces of user interaction history data from the data experience pool in is the interaction sequence of user u from time 0 to time t. Assume that the user interacts with the item at time t. If there is interaction, in is the interaction sequence of user u from time 0 to time t-1.

[0048] 3.3.2) The goal of the serialized recommendation system model is generally the click-through rate (CTR). Input into the recommendation system model and get the model's prediction of the user clicking on the item at time t Probability

[0049] 3.3.3) Using Cross Entropy to Calculate the Recommender System Loss Function L CE :

[0050]

[0051] Among them, a P =1,a N =0 represents whether the user clicks. is the estimated probability of user u clicking on the item.

[0052] 3.3.4) Use the modeling method in step 1 to model the recommendation system data into a Markov decision process data format, and input the data into the user satisfaction model in step 2 to obtain the user's satisfaction r(s) at that moment. t ,a t ),in

[0053] 3.3.5) Calculate auxiliary task loss. In order to align the recommendation system training process with user satisfaction, the present invention designs an alignment loss L with reference to the cross entropy loss used in the recommendation system. Align , to maximize user satisfaction when guiding the recommendation system training:

[0054]

[0055] 3.3.6) Calculate the final loss L Rec , update the recommendation system parameter ψ through neural network back propagation:

[0056] L Rec =L CE +κ·L Align (10)

[0057] Among them, κ is a hyperparameter used to control the proportion of alignment loss in the recommendation task.

[0058] Beneficial effects of the present invention:

[0059] The present invention adopts a new optimization method in the training process of the recommendation system. First, the present invention uses inverse reinforcement learning to mine the motivations hidden behind user behaviors from the user's interaction data, and obtains a user satisfaction model, which can quantify the user's satisfaction at the current moment according to the user's status and actions; secondly, new auxiliary tasks are designed in the training process of the recommendation system to guide the recommendation system to align with user satisfaction, thereby ensuring that user interests can be satisfied in each recommendation process. At the same time, the present invention is conducive to the optimization of recommendation systems in industry for the following reasons: (1) The optimization method proposed in the present invention can be extended to any sequential recommendation scenario, such as news recommendation, music recommendation, etc., and only requires a simple modification of the MDP process; (2) All training of the present invention is carried out in an offline environment, saving the cost of building an online user interaction environment; (3) The alignment task proposed in the present invention can be easily combined with any existing sequential recommendation system framework, and has strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a framework flow chart of the present invention.

[0061] Figure 2 Schematic diagram of the recommendation system alignment task proposed in this invention.

[0062] Figure 3 DIN network architecture diagram of the click rate prediction model used in the embodiment. DETAILED DESCRIPTION

[0063] The specific implementation of the present invention is further described below in conjunction with the accompanying drawings and technical solutions.

[0064] The method of the present invention can be used in the training process of the recommendation system in the e-commerce recommendation scenario. The flow chart of the method of the present invention is as follows: Figure 1 shown.

[0065] The alignment task optimization method designed by the present invention has a framework as follows: Figure 2 shown.

[0066] The embodiments of the present invention are described in detail below (eg Figure 1 As shown), specifically comprising the following steps:

[0067] (1) Data processing: This embodiment uses the Amazon dataset. This dataset collects user reviews of products on the e-commerce website Amazon from 1996 to 2014. It is divided into multiple sub-datasets according to different product types. Two of them are used in this invention: Amazon Electronics and Amazon Book. The specific Markov decision process model is as follows:

[0068] State space S: defines the state s of the user at time t t =(h t-1 ,i t ), where h t-1 =(σ1,σ2,…,σ t-1 ) is the interaction history at time t-1, and each interaction σ=<u,i,a> , u is the user id, i is the user interaction product, the present invention uses product id and category to represent, and a is the user action.

[0069] Action space A: A={a P ,a N}. P =1 means the user has no comment, a N =0 means the user has posted a comment.

[0070] Transition function p: S×A×A→[0,1]. When the user performs action a at time t t Then it will transfer to the new state s t+1 =s t ∪σ t+1 .

[0071] (2) User satisfaction model training process:

[0072] (2.1) Initialize the Q network parameters in the user satisfaction model and fill the data experience pool with processed data.

[0073] (2.2) Sampling 64 samples from the data experience pool Calculate the model loss.

[0074] First, calculate the corresponding state value function and state function according to the user state and action:

[0075]

[0076]

[0077] Compute the inverse reinforcement learning loss with α = 0.5 and γ = 1:

[0078]

[0079] Then calculate the reward differentiation enhancement regularization loss:

[0080]

[0081] The sum of the two is the total loss of the user satisfaction model, where β = 0.5:

[0082] loss = L IRL +β·L RDE

[0083] (2.3) Use the gradient descent method to reversely update the network parameters of the user satisfaction model.

[0084] When the loop condition is not met, that is, the loop reaches the preset number of training steps (200,000 steps), the trained soft-Q model can be obtained, and then the user satisfaction r(s) at any time can be calculated. t ,a t ), where γ=1:

[0085] r(s t ,a t )=Q w (s t ,a t )-γV(s t+1 )

[0086] This model can be used to quantitatively calculate satisfaction during the training process of the recommendation system.

[0087] (3) Recommendation system model training process: The satisfaction model proposed in this paper can be combined with any serialized recommendation system model. Here, we take the classic click-through rate prediction model DIN (Deep Interest Network) as an example. DIN is a recommendation algorithm based on user historical behavior sequences. By introducing an interest extraction mechanism, it dynamically recommends items that are most relevant to each user's current interests, thereby improving the personalization and accuracy of recommendations. The structure of the model is as follows: Figure 3 shown.

[0088] (3.1) Initialize the network parameters in the recommendation system model.

[0089] (3.2) Extract 64 user interaction history data from the data experience pool Calculate the model loss.

[0090] First, input the sample into the recommendation system model. Each sample includes: user features (user id), features of the product commented by the user (product id, product category), and features of the product to be recommended (product id, product category). Calculate the probability of commenting on the product to be recommended under this interaction history sequence. Calculate the cross entropy loss between the model estimated probability and the sample label:

[0091]

[0092] The samples are processed into s and a of the Markov decision process in step (1), and input into the user satisfaction model to obtain the user satisfaction in this state: r(s,a).

[0093] Calculate the auxiliary alignment task loss:

[0094]

[0095] Calculate the total loss, where κ = 0.6:

[0096] L Rec =L CE +κ·L Align

[0097] (3.3) Use the gradient descent method to reversely update the network parameters of the recommendation system model.

[0098] When the loop condition is not met, that is, when the loop reaches a preset number of training times or a preset indicator, a recommendation system model aligned with user satisfaction can be obtained.

[0099] In order to measure the performance of the recommendation system model, the present invention uses the following two indicators:

[0100] (1) AUC (Area Under Curve), used to measure the ranking ability of the recommendation system model:

[0101]

[0102] Among them, x and y are the number of positive and negative samples respectively. t <h t-1 ,i t >, if the user faces item i t The action is a P , that is, if the item is commented on, the sample is a positive sample, otherwise it is a negative sample. P(positive) is the recommendation system for the positive sample action a P The predicted value of the execution probability, P(negative) is the recommendation system for the negative sample action a N The predicted value of the execution probability.

[0103] (2) NCIS (Normalised Capped Importance Sampling) is used to estimate the online performance of the recommendation system model. The longer the interaction sequence generated by the user under the recommendation system model, the larger the NCIS value:

[0104]

[0105] Among them, u represents the u-th user, U is the number of users in the dataset, and ρ u is the probability of a user generating a corresponding interaction trajectory under the current recommendation system, L u is the length of the user interaction trajectory.

[0106] After experiments, the index values ​​of the recommendation system model DIN in this embodiment before and after alignment with the user satisfaction model are as follows:

[0107]

[0108] It can be seen from the experimental results that by aligning with the user satisfaction model, the AUC and NCIS indicators of the recommendation system model of this embodiment are improved, indicating that the optimization method proposed in the present invention can improve the ranking ability of the recommendation system, while increasing the user's interaction sequence length and improving user stickiness.

Claims

1. A recommendation system optimization method based on user satisfaction, characterized in that the specific steps are as follows: Step 1: Set the problem model and conduct mathematical modeling: 1.1) Markov decision process modeling: The problem is modeled using the Markov decision process, which is represented by a five-tuple<S,A,p,r,π> ; Where S represents the state space, A represents the action space, p represents the environment transfer equation, r represents the reward function, and π represents the strategy function; the present invention performs Markov process modeling from the user's perspective, that is, the user is regarded as an intelligent agent and the recommendation system is regarded as the environment; the specific modeling method is as follows: State space S: s t ∈S represents the user's state at time t; the user state is defined as: s t =(h t-1 ,i t ), where h t-1 =(σ1,σ2,…,σ t-1 ) is the interaction history at time t-1, and each interaction σ=<u,i,a> , u is the user feature, i is the item the user interacts with, and a is the user action; Action space A: User action a∈A represents the user's feedback on the interactive items. In order to simplify the problem scale, user feedback is divided into two categories: A={a P ,a N }, where a P Represents positive feedback from users, a N Represents negative user feedback; Transfer function p: S×A×A→[0,1]; when the user performs action a at time t t After that, it will transfer to the new state s according to the transfer function t+1 ; Reward function r: The reward function r is a quantitative representation of the user's satisfaction after consuming the items recommended by the recommendation system; Strategy π: User strategy is a mapping function from state space to action space; use the optimal strategy π during the interaction between the user and the recommendation system * , that is, the user always tends to maximize his own satisfaction, which is the reward function; 1.2) Modeling the alignment problem of recommendation systems: The goal is to optimize the recommendation system model to align it with the user satisfaction model. Alignment means maximizing user satisfaction during the entire interaction between the user and the recommendation system. Specifically, the reward function r(s, a) is used to quantify user satisfaction, where s =<h,i> is the user state, h is the user interaction history, i is the item currently interacting with the user, and a is the user action; the goal of alignment is to maximize ∑ t η·r(s t ,a t ), where i t is the item recommended by the recommendation system at time t, a t is the user's action at time t, and η is the decay coefficient, which is used to balance short-term rewards and long-term rewards; Step 2: User satisfaction model training The user satisfaction model is modeled as the reward function in the Markov decision process, and the reward function is solved under the condition that the user state sequence and the corresponding action are known; the reward function is solved by using the IQ-Learn algorithm in inverse reinforcement learning. The algorithm uses the soft-Q function to model the user's state-action value Q(s,a), and uses the state transition probability and strategy distribution to derive the analytical expression of the reward function; the specific steps are as follows: 2.1) Data processing, building the experience pool D; processing the recommendation system data set to satisfy the Markov decision process modeling method in step 1, which is convenient for subsequent model training; 2.2) Randomly initialize the agent soft-Q model Q w (s,a) network parameters w; 2.3) Use the training data to train the model parameters w; 2.3.1) Sample m samples from the experience pool D calculate V j (s t ) and V j (s t+1 );in, are the state and action of the user at time t in the jth sample, is the state of the user in the jth sample at time t+1; is the action state value function value of the user at time t in the jth sample, V j (s t ), V j (s t+1 ) are the user s in the jth sample t 、s t+1 The state value function value of ; Q and V satisfy the soft-Bellman equation: 2.3.2) Calculate the inverse reinforcement learning loss L IRL : Where α is a hyperparameter; 2.3.3) Calculate the reward-differentiated enhanced regularization loss L RDE : 2.3.4) Calculate the total loss and update Q by backpropagating the gradient of the neural network w (s,a) all parameters w of the network; loss=L IRL +β·L RDE (6) Among them, β is a hyperparameter used to balance the loss weight of the regularization term; By calculating the user's satisfaction r(s) when performing action a in state s, t ,a t ); r(s t ,a t )=Q w (s t ,a t )-γV(s t+1 )(7) Where γ is a hyperparameter, V(s t+1 ) is the user in state s t+1 The state value function value under ; Step 3: Use the user satisfaction model to optimize the recommendation system model training The goal is to align the recommendation system with the user satisfaction model during the training process; in step 2, the user satisfaction model r(s t ,a t ); The goal of the alignment problem is to recommend the system to recommend item i t At the same time, maximize the cumulative reward Σ t η·r(s t ,a t ); 3.1) Initialize the recommendation system data experience pool and recommendation system model Δ ψ ; 3.2) Align the training recommendation system model and the user satisfaction model: 3.2.1) Extract n pieces of user interaction history data from the data experience pool in is the interaction sequence of user u from time 0 to time t; assuming that the user interacts with the item at time t If there is interaction, in is the interaction sequence of user u from time 0 to time t-1; 3.3.2) The goal of the serialized recommendation system model is the click-through rate (CTR); the user’s interaction sequence from time 0 to time t-1 Input into the recommendation system model and get the model's prediction of the user clicking on the item at time t Probability 3.3.3) Using Cross Entropy to Calculate the Recommender System Loss Function L CE : in, a P =1,a N =0 represents whether the user clicks. is the estimated probability of user u clicking on the item; 3.3.4) Use the modeling method in step 1 to model the recommendation system data into a Markov decision process data format, and input the data into the user satisfaction model in step 2 to obtain the user's satisfaction r(s) at that moment. t ,a t ),in 3.3.5) Calculate the auxiliary task loss; In order to align the recommendation system training process with user satisfaction, the alignment loss L is designed with reference to the cross entropy loss used in the recommendation system. Align , to maximize user satisfaction when guiding the recommendation system training: 3.3.6) Calculate the final loss L Rec , update the recommendation system parameter ψ through neural network back propagation: L Rec =L CE +κ·L Align (10) in, κ is a hyperparameter used to control the weight of alignment loss in the recommendation task.

Images