A sequence recommendation method based on reinforcement learning to enhance long-term benefits

Abstract

The application provides a sequence recommendation method based on reinforcement learning for enhancing long-term benefits, and relates to the technical field of network recommendation.The application takes a reinforcement learning strategy as a teacher model and takes a supervised learning sequence recommendation model as a student model;after data preprocessing, model training is performed;in the training process, the teacher model regards the supervised learning sequence recommendation model as an environment and outputs an action for adjusting the state of the supervised learning sequence recommendation model to drive the recommendation model to focus on long-term benefits;the student model takes the new state adjusted by the teacher model as input to generate a recommendation result.The application can train the recommendation model by online interaction without affecting real users, the recommendation model will be driven to focus on long-term benefits when generating a recommendation result, and the effect of the recommendation algorithm is further improved, so that the recommendation algorithm can more accurately recommend for users.

Description

Technical Field

[0001] This invention relates to the field of network recommendation technology, and in particular to a sequence recommendation method based on reinforcement learning to enhance long-term benefits. Background Technology

[0002] Sequence recommendation uses the sequence of user interactions with items (e.g., clicks or purchases) as input to a recommendation model, capturing feature information from the interaction sequence to recommend the next series of items. This is one of the most common scenarios in recommendation systems, such as e-commerce, video, or music platforms.

[0003] In recent years, with the development of deep learning, numerous sequence recommendation algorithms based on deep learning techniques have been proposed to learn complex relationships and feature information in user-item interaction sequences. One of the most common deep learning-based methods is Self-Attentive Sequential Recommendation (SASRec), presented at the 2018 IEEE International Conference on Data Mining. This method introduces a self-attention mechanism into sequence recommendation, using it to model users' historical behavioral information (i.e., user-item interaction sequences). SASRec first uses embedding techniques to map user-item interaction sequences into a sequence of item embeddings. Then, it uses a self-attention model to encode the item embedding sequence into user embeddings representing user behavioral features. Finally, it performs inner product calculations with all item embeddings and sorts them according to their relevance to obtain Top-k recommendations. During the training phase, this method uses supervised learning on the dataset to train the model, minimizing the difference between the model's predictions and the actual recommendations.

[0004] Reinforcement learning-based recommendation algorithms are another common method for sequence recommendation. This method also uses embedding techniques to map user behavior sequences to item embedding sequences, and then uses sequence models such as recurrent neural networks (RNNs) or convolutional neural networks (CNNs) to capture features of the user behavior sequences for subsequent recommendations. Unlike supervised learning methods, reinforcement learning methods optimize by maximizing cumulative revenue. Therefore, reinforcement learning considers the long-term benefits of the system when making recommendations, making the recommendation strategy more aligned with the service provider's expectations. Furthermore, reinforcement learning can adjust the recommendation strategy in real time based on user feedback, thus better adapting to dynamic changes in user interests.

[0005] The optimization objective of sequence recommendation algorithms trained using supervised learning is to minimize the discrepancy between model predictions and the ground truth. This objective, defined based on immediate user feedback rather than explicitly optimizing for long-term cumulative returns, is difficult to align with the desired goal of maximizing cumulative returns from the system service provider's perspective. Recommendation systems should not only deliver high immediate returns but also consider long-term user satisfaction.

[0006] Therefore, reinforcement learning, with its optimization objective of maximizing cumulative returns, has been introduced into sequence recommendation. However, common reinforcement learning algorithms are trained by repeatedly making mistakes and then correcting them; that is, the policy model receives feedback from the environment after taking an action and then continuously adjusts the policy based on the feedback. This usually requires extensive online interaction with users to learn a usable policy, but this is too costly because online interaction with an undertrained policy can severely impact the user experience and even cause users to abandon the system. One solution is to train the policy using implicit feedback from offline datasets; however, this method of offline training of reinforcement learning models faces challenges such as mismatched offline data distribution and insufficient negative feedback signals. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a sequence recommendation method based on reinforcement learning to enhance long-term benefits, thereby solving the problems of impaired user experience and mismatch in offline data distribution faced by reinforcement learning strategies that directly interact with users.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] A sequence recommendation method based on reinforcement learning to enhance long-term benefits is proposed. The method uses a reinforcement learning strategy as the teacher model and a supervised sequence recommendation model as the student model. After data preprocessing, the models are trained. During training, the teacher model treats the supervised sequence recommendation model as the environment and outputs an action to adjust the state of the supervised sequence recommendation model, driving the recommendation model to focus on long-term benefits. The student model uses the new state adjusted by the teacher model as input to generate recommendation results.

[0010] Furthermore, the data preprocessing includes:

[0011] Step 1.1: The training dataset for sequence recommendation includes several sequences, each containing a series of interactions between the same user and the recommendation system, i.e., user-item interactions; in preprocessing, sequences with a length of less than 3 are deleted, and then the user-item interactions in the same sequence are sorted according to timestamps;

[0012] Step 1.2: During model training, the sequence is divided into groups of 10 interactions. The interaction with the next timestamp is used as the ground truth of the real event to be predicted by the model, providing supervision signals for the recommendation model. If the sequence has less than 10 interactions, padding items are used to pad the length of the interaction sequence to 10 before using it as input.

[0013] Furthermore, the model training includes:

[0014] Step 2.1: Initialize the model parameters using a standard normal distribution; this includes: reinforcement learning models Actorμ and Critic Q built using a multilayer perceptron, and decoder f and sequence model encoder G built using a multilayer perceptron; then create duplicate models with the exact same parameters as μ, Q, f, and G: Copy Actorμ′, Copy Critic Q′, copy decoder f′, and copy encoder G′.

[0015] Step 2.2: Obtain a batch of interaction sequences of length 10 from the dataset, as well as the interaction items for the next time step of the sequence; use the interaction sequences as input to the sequence model to obtain the representation of the interaction sequences;

[0016] Let X represent the set of candidate items; each interaction sequence is associated with an item from X, and the interaction sequence x is... 1：t ={x1, x2, ..., x t}, where x h Let X represent the interactive item at time h in the interaction sequence, where 0 < h ≤ t. In each training round, a batch of interaction sequences is obtained from the dataset. Then, each interactive item in the sequence is mapped to an item embedding. The sequence composed of item embeddings is input into the encoder G to obtain the representation state s. t ;

[0017] Step 2.3: Use the representation of the interaction sequence as the input state of the Actor in the reinforcement learning model, and then output the action;

[0018] For an Actor, it is defined as:

[0019]

[0020] Among them, o l Represents the output of the l-th layer in a neural network, where ReLU is the activation function. and These are the trainable parameters of the l-th layer of the Actor; a tis the action vector output by reinforcement learning, used to adjust the supervised learning student model; A is a hyperparameter controlling the adjustment strength. When A=0, the reinforcement learning model does not output knowledge to the student model, and the framework reverts to simple supervised learning. The larger A is, the greater the weight the reinforcement learning teacher model has in adjusting the student supervised model; the softmax function is defined as follows:

[0021]

[0022] Among them, o (j) Let represent the j-th element of vector o, and m be the dimension of vector o;

[0023] Step 2.4: After inputting the action into the softmax function, calculate the element-wise product with the state to obtain the new representation state;

[0024] Integrate the output a from the teacher model t and the state s from the student model t To obtain a new representation state as follows:

[0025]

[0026] Where ⊙ represents the element-wise product of two vectors;

[0027] Step 2.5: Input the new representation state into the decoder to obtain the recommended scores for all items in the candidate item set;

[0028] The new representation state Used as input to decoder f, we get:

[0029]

[0030] Among them, y t+1 This indicates a rating for the item; based on y t+1 The top-k items with the highest scores are selected as the recommendation results; the reward r is used to train reinforcement learning. t It is then defined as the normalized discounted cumulative return (NDCG) calculated from the recommendation results;

[0031] Step 2.6: Optimize the decoder based on the cross-entropy loss function;

[0032] For the supervised learning part, the cross-entropy loss function is calculated and trained using gradient descent:

[0033]

[0034] Among them, I iIt is an indicator function that determines if the user interacts with the i-th item in the next timestamp. i =1, otherwise I i =0;

[0035] Step 2.7: Optimize the Critic based on time difference error reinforcement learning;

[0036] For Critic, it is defined as:

[0037]

[0038] in, and These are the trainable parameters of the l-th layer in the Critic part; [·] represents the merging operation; Q(s) t a t Let Q-Value be the value of the reinforcement learning policy prediction in s. t Select to execute a under the status t The long-term benefits that can be obtained from the behavior;

[0039] The parameters of the Critic component are updated by minimizing the single-step time difference error, and the loss function L(Q, D) is:

[0040]

[0041] in,

[0042] Target = (r t +γ(1-d)Q′(s t+1 ,μ′(s t+1 )))

[0043] Here, Q′ and μ′ are replicas of Critic Q and Actor μ, serving to make the reinforcement learning model more stable during training; D is the set of state transitions, where each element is (s t a t r t+1 ,d), where s t Let a represent the state at time t. t r represents the action taken at time t. t+1 Indicates that in state s t Take action a in the following circumstances t The subsequent reward, d represents whether the next time step is the termination state. If the next time step is the termination state, then d = 1, otherwise d = 0; γ is the discount factor in reinforcement learning, representing the weight of future rewards in the reinforcement learning strategy.

[0044] Step 2.8: Optimize the reinforcement learning Actor based on the Q-Value of the Critic output;

[0045] Based on the Q-Value output by the Critic, gradient ascent is performed on the Actor according to the following formula:

[0046]

[0047] In this context, the parameters of the Critic Q function are treated as constants;

[0048] To enable the Actor policy to explore more effectively and try better solutions, noise is added to the actions taken by the Actor during training. t ,Right now

[0049]

[0050] The clip function represents when a t +∈less than a Low When, return a Low ; when a t +∈greater than a High When, return a High ; when a t +∈ is in the interval [a Low a High When between ], return a t +∈; ∈ is Gaussian noise with a mean of zero;

[0051] Then, during the training process, the supervised learning part and the reinforcement learning part are trained alternately until the model converges.

[0052] Furthermore, the method for generating the recommendation results:

[0053] Update the copy model Copy Actorμ′, Copy Critic Q′, copy decoder f′, and copy encoder G′ according to the following rules:

[0054] F′=β*F′+(1-β)F

[0055] Where F′ represents any one of the replication model μ′, Q′, f′ or G′, F represents any one of μ, Q, f or G corresponding to F′; β is a hyperparameter ranging from 0 to 1;

[0056] During training, the output of a copy of decoder f′ is used to compute the reward r. t During testing, the output of decoder f is used to generate recommendation results.

[0057] The beneficial effects of adopting the above technical solution are as follows: The sequence recommendation method based on reinforcement learning to enhance long-term benefits provided by this invention uses the reinforcement learning part as the teacher model. It treats the supervised learning-based recommendation model as the environment and outputs actions to adjust the state of the supervised learning recommendation model to drive it to focus on long-term benefits. The supervised learning-based recommendation model part acts as the student model, taking the new state adjusted by the teacher model as input to generate recommendation results. Unlike existing reinforcement learning-based methods, which typically treat the user as the environment, the reinforcement learning model proposed in this invention treats the recommendation model as the environment. This allows the reinforcement learning model to be trained through online interaction with the recommendation model without affecting real users. By combining reinforcement learning technology, the recommendation model will be driven to focus on long-term benefits when generating recommendation results, rather than just considering the benefits at the current moment, further improving the performance of the recommendation algorithm and enabling it to make more accurate recommendations to users. Attached Figure Description

[0058] Figure 1 The overall flowchart of the algorithm provided in the embodiments of the present invention;

[0059] Figure 2 The algorithm training flowchart provided in the embodiments of the present invention;

[0060] Figure 3 This is an algorithm architecture diagram provided for an embodiment of the present invention. Detailed Implementation

[0061] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0062] like Figure 1 As shown, the method of this embodiment is described below.

[0063] A sequence recommendation method based on reinforcement learning to enhance long-term benefits is proposed. The method uses a reinforcement learning strategy as the teacher model and a supervised learning sequence recommendation model as the student model. After data preprocessing, the models are trained. During training, the teacher model treats the supervised learning sequence recommendation model as the environment and outputs an action to adjust the state of the supervised learning sequence recommendation model, driving the recommendation model to focus on long-term benefits. The student model uses the adjusted state from the teacher model as input to generate recommendation results. The specific method is described below.

[0064] Step 1: Data preprocessing.

[0065] Step 1.1: The training dataset for sequence recommendation includes several sequences, each containing a series of interactions between the same user and the recommendation system, i.e., user-item interactions; in preprocessing, sequences with a length of less than 3 are deleted, and then the user-item interactions in the same sequence are sorted according to timestamps;

[0066] Step 1.2: During model training, the sequence is divided into groups of 10 interactions. The interaction with the next timestamp is used as the ground truth of the real event to be predicted by the model, providing supervision signals for the recommendation model. If the sequence has less than 10 interactions, padding items are used to pad the length of the interaction sequence to 10 before using it as input.

[0067] Step 2: Model training, such as Figure 2 As shown, the specific method is as follows.

[0068] Step 2.1: Initialize the model parameters using a standard normal distribution; this includes: reinforcement learning models Actorμ and Critic Q constructed using a multilayer perceptron, and decoder f and sequence model encoder G constructed using a multilayer perceptron (in this embodiment, a GRU network is used as an example); then create a copy model with the exact same parameters as μ, Q, f, and G: Copy Actorμ′, Copy Critic Q′, copy decoder f′, and copy encoder G′.

[0069] Step 2.2: Obtain a batch of interaction sequences of length 10 from the dataset, as well as the interaction items at the next time step of the sequence; use the interaction sequences as input to the sequence model to obtain the representation of the interaction sequences.

[0070] Let X represent the set of candidate items; each interaction sequence is associated with an item from X, and the interaction sequence x is... 1：t ={x1, x2, ..., x t}, where x h Let X represent the interactive item at time h in the interaction sequence, where 0 < h ≤ t. In each training round, a batch of interaction sequences is obtained from the dataset. Then, each interactive item in the sequence is mapped to an item embedding. The sequence composed of item embeddings is input into the encoder G to obtain the representation state s. t .

[0071] Step 2.3: Use the representation of the interaction sequence as the input state of the Actor in the reinforcement learning model, and then output the action.

[0072] For an Actor, it is defined as:

[0073]

[0074]

[0075] Where ol represents the output of the l-th layer in the neural network, and ReLU is the activation function. and These are the trainable parameters of the l-th layer of the Actor; a t is the action vector output by reinforcement learning, used to adjust the supervised learning student model; A is a hyperparameter controlling the adjustment strength. When A = 0, the reinforcement learning model does not output knowledge to the student model, and the framework reverts to simple supervised learning. A larger value means that the reinforcement learning teacher model has a greater weight in adjusting the supervised student model. The softmax function is defined as follows:

[0076]

[0077] Among them, o (j) Let m represent the j-th element of vector o, where m is the dimension of vector o.

[0078] Step 2.4: After inputting the action into the softmax function, calculate the element-wise product with the state to obtain the new representation state.

[0079] Integrate the output a from the teacher model t and the state s from the student model t To obtain a new representation state as follows:

[0080]

[0081] Here, ⊙ represents the element-wise product of two vectors.

[0082] Step 2.5: Input the new representation state into the decoder to obtain the recommended scores for all items in the candidate item set.

[0083] The new representation state Used as input to decoder f, we get:

[0084]

[0085] Among them, y t+1 This indicates a rating for the item; based on y t+1 The top k items with the highest scores are selected as the top-k recommendations; the reward r is used to train the reinforcement learning. t It is then defined as the normalized discounted cumulative return (NDCG) calculated from the recommendation results.

[0086] Step 2.6: Optimize the decoder based on the cross-entropy loss function.

[0087] For the supervised learning part, the cross-entropy loss function is calculated and trained using gradient descent:

[0088]

[0089] Among them, I i It is an indicator function that determines if the user interacts with the i-th item in the next timestamp. i =1, otherwise I i =0.

[0090] Step 2.7: Optimize reinforcement learning Critic based on time difference error.

[0091] For Critic, it is defined as:

[0092]

[0093] in, and These are the trainable parameters of the l-th layer in the Critic part; [·] represents the merging operation; Q(s) t a t Let Q-Value be the value of the reinforcement learning policy prediction in s. t Select to execute a under the status t The long-term benefits that can be obtained from the behavior.

[0094] The parameters of the Critic component are updated by minimizing the temporal difference error. The loss function L(Q, D) is:

[0095]

[0096] in,

[0097] Target = (r t +γ(1-d)Q′(s t+1 ,μ′(s t+1 )))

[0098] Here, Q′ and μ′ are replicas of Critic Q and Actor μ, serving to make the reinforcement learning model more stable during training; D is the set of state transitions, where each element is (s t a t r t+1 ,d), where s t Let a represent the state at time t. t r represents the action taken at time t. t+1 Indicates that in state s t Take action a in the following circumstancest The subsequent gain is denoted by d, which indicates whether the next time step is the termination state. If the next time step is the termination state, then d = 1; otherwise, d = 0. γ is the discount factor in reinforcement learning, which represents the weight of future gains in the reinforcement learning strategy.

[0099] Step 2.8: Optimize the reinforcement learning Actor based on the Q-Value of the Critic output.

[0100] Based on the Q-Value output by the Critic, gradient ascent is performed on the Actor according to the following formula:

[0101]

[0102] In this context, the parameters of the Critic Q function are treated as constants.

[0103] To enable the Actor policy to explore more effectively and try better solutions, noise is added to the actions taken by the Actor during training. t ,Right now

[0104]

[0105] The clip function represents when a t +∈less than a Low When, return a Low ; when a t +∈greater than a High When, return a High ; when a t +∈ is in the interval [a Low a High When between ], return a t +∈; ∈ is Gaussian noise with a mean of zero.

[0106] Then, during the training process, the supervised learning part and the reinforcement learning part are trained alternately until the model converges. Figure 3 The overall architecture of the algorithm is proposed for this embodiment.

[0107] Step 3: Generate recommendation results.

[0108] Update the copy model Copy Actorμ′, Copy Critic Q′, copy decoder f′, and copy encoder G′ according to the following rules:

[0109] F′=β*F′+(1-β)F

[0110] Where F′ represents any one of the replication models μ′, Q′, f′, or G′, and F represents any one of μ, Q, f, or G corresponding to F′; for example, when F′ and F take μ′ and μ respectively, the above formula is: μ′=β*μ′+(1-β)μ. β is a hyperparameter ranging from 0 to 1.

[0111] During training, the output of a copy of decoder f′ is used to compute the reward r. t During testing, the output of decoder f is used to generate recommendation results.

[0112] Compared to existing supervised learning-based recommendation models, the method proposed in this embodiment combines reinforcement learning to drive the recommendation model to focus on long-term gains, further improving the performance of the recommendation algorithm and enabling it to make more accurate recommendations to users. This embodiment compares with other methods on three publicly available real-world datasets: RecSys Challenge 2015 (RC15), RetailRocket (RR), and RecSysChallenge 2019 (RC19). Dataset statistics are shown in Table 1.

[0113] Table 1 Dataset Statistics

[0114] Dataset RC15 RR RC19 Total number of sequences 200,000 195,523 158,319 Total number of items 26,702 70,852 183,376 Total number of interactions 1,154,911 1,233,949 752,548

[0115] This embodiment uses cross-validation to conduct experiments on three metrics: Hit Rate (HR), Normalized Discounted Cumulative Gain (NDCG), and Continuous Normalized Discounted Cumulative Gain (CNDCG). HR evaluates whether the model-generated recommendation list matches the content the user is about to click. NDCG, based on HR, considers the position of the content the user intends to interact with within the top-k recommendation list; a higher ranking results in a higher score. CNDCG calculates the cumulative sum of NDCG using the longest consecutive sequence of user-clicked items in each session. Therefore, a higher CNDCG indicates a longer user interaction sequence with the recommendation system and a longer dwell time within the system, meaning the recommendation algorithm can better improve the system's long-term cumulative returns.

[0116] The method in this embodiment is compared with five common sequence recommendation methods, namely:

[0117] GRU: This method uses gated recursive units to simulate the user's interaction sequence with items. The final hidden state of the GRU is used as input to a fully connected layer to produce recommendation results.

[0118] Caser: This CNN-based approach treats the embedding matrix of the previous interactive project as an image, and then captures the sequential signal by applying convolution operations.

[0119] NItNet: This method uses an expanded CNN to increase the receptive field and adds depth to the network through residual connections.

[0120] SASRec: This method uses the output of the Transformer encoder as a representation of the input sequence.

[0121] DQN: Uses a recurrent neural network to capture sequences of user actions as input states for a deep Q-network. By introducing uniformly sampled, uninterrupted items to provide negative rewards, it makes Q-learning more efficient when performing recommendation ranking.

[0122] Tables 2, 3, and 4 show the performance of top-k recommendations on RC15, RetailRocket, and RC19, respectively. It can be seen that the method proposed in this embodiment significantly improves performance on all three evaluation metrics. A higher NDCG score indicates that the proposed method effectively pushes ground-truth items to the top of the recommendation list. A higher CNDCG score indicates that the proposed method can better improve the long-term cumulative reward of the recommendation system. The method in this embodiment introduces a reinforcement learning model with long-term cumulative reward as the optimization objective to fine-tune the supervised learning model. The additional information from reinforcement learning improves recommendation performance because the new state, which integrates the student model state and the teacher model behavior vector, considers not only immediate rewards but also future NDCG rewards.

[0123] Table 2 Experimental results on the RecSys Challenge 2015 dataset

[0124] Model HR@5 NDCG@5 CNDCG@5 HR@20 NDCG@20 CNDCG@20 DQN 0.1866 0.1281 4321.68 0.2906 0.1586 5350.65 GRU 0.3258 0.2236 5894.89 0.5072 0.2768 7521.47 Caser 0.3111 0.2169 6022.65 0.4806 0.2665 7631.19 NItNet 0.3182 0.2225 6010.26 0.4882 0.2722 7579.15 SASRec 0.3352 0.2308 5992.36 0.5156 0.2837 7653.22 Ours 0.3509 0.2453 6303.85 0.5251 0.2965 8119.61

[0125] Table 3 Experimental results on the RetailRocket dataset

[0126] Model HR@5 NDCG@5 CNDCG@5 HR@20 NDCG@20 CNDCG@20 DQN 0.1249 0.086 2901.36 0.2299 0.1155 3896.59 GRU 0.2701 0.2079 8132.34 0.3714 0.2374 8758.45 Caser 0.2315 0.1857 8366.95 0.3075 0.2077 8879.59 NItNet 0.2442 0.1954 8440.53 0.3301 0.2198 8956.08 SASRec 0.2814 0.2137 8210.84 0.3848 0.2438 8932.4 Ours 0.3036 0.2399 9148.85 0.4021 0.2684 9787.74

[0127] Table 4. Experimental results on the RecSys Challenge 2019 dataset.

[0128]

[0129]

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A sequence recommendation method based on reinforcement learning to enhance long-term rewards, characterized in that: The method takes a reinforcement learning strategy as a teacher model and a supervised learning sequence recommendation model as a student model; after data preprocessing, the model is trained; During the training process, the teacher model is based on the supervised learning sequence recommendation model as the environment, and outputs the action for adjusting the state of the supervised learning sequence recommendation model to drive the recommendation model to focus on long-term benefits; The student model takes the new state adjusted by the teacher model as input to generate a recommendation result; The model training includes: Step 2.1: Initialize the model parameters through a standard normal distribution; including: a reinforcement learning model Actorμ and a CriticQ built by a multilayer perception mechanism, and a decoder f and a sequence model encoder G built by a multilayer perception mechanism; then create a copy model: Copy Actorμ', Copy CriticQ', a copy decoder f', and a copy encoder G' which are exactly the same as the model parameters ofμ, Q, f, and G; Step 2.2: Obtain a batch of interaction sequences with a length of 10 from the data set, and the interaction items at the next time of the sequence; encode the interaction sequence as the input of the sequence model to obtain the representation of the interaction sequence; X denotes a candidate item set; each interaction sequence is associated with an item from X, and the interaction sequence x 1:t = {x1, x2,..., x t}, where x h ∈X represents the interaction item at h time in the interaction sequence, 0 < h < t; in each training round, a batch of interaction sequences is obtained from the data set; then each interaction item in the sequence is mapped to an item embedding; the sequence composed of item embeddings is input into the encoder G to map to a representation state s t ; Step 2.3: Take the representation of the interaction sequence as the input state state of the Actor of the reinforcement learning model, and then output the action; the specific method is: For the Actor, it is defined as: a t = softmax(o L+1 ) where o l represents the output of the l-th layer in the neural network, relu is the activation function, and are the trainable parameters of the l-th layer of the Actor; a t is the action vector output by the reinforcement learning, i.e., action, which is used to adjust the supervised learning student model; A is a hyperparameter that controls the adjustment strength, when A = 0, the reinforcement learning model will not output knowledge to the student model, and the framework reverts to simple supervised learning, and the larger A means that the reinforcement learning teacher model has a greater weight in adjusting the student supervised model; the softmax function is defined as follows: wherein o (j) represents the jth element of vector o, and m is the dimension of vector o; Step 2.4: The action is input into a softmax function and the resulting state is computed by taking the element-wise product with the state Step 2.5: Update the new representation of the state Input to the decoder gets the recommendation scores for all items in the candidate item set; Step 2.6: Optimize the decoder based on the cross-entropy loss function; Step 2.7: Optimize the Critic of the reinforcement learning based on the time difference error; Step 2.8: Optimize the Actor of the reinforcement learning based on the Q-Value output by the Critic; alternately train the supervised learning part and the reinforcement learning part during the training process until the model converges.

2. The sequential recommendation method based on reinforcement learning enhanced long-term reward according to claim 1, characterized in that: The data preprocessing includes: Step 1.1: The training data set of the sequence recommendation includes a plurality of sequences, each sequence containing a series of user-item interactions of a same user with the recommendation system; in the preprocessing, the sequences with a length less than 3 are deleted, and then the user-item interactions in the same sequence are sorted according to the time stamp; Step 1.2: During the model training, the sequence is divided into a group of inputs according to every 10 interactions, and the interaction at the next time stamp is taken as the ground-truth to be predicted by the model, providing a supervision signal for the recommendation model; if the sequence does not have 10 interactions, padding items are used to fill the length of the interaction sequence to 10 before being taken as the input.

3. The sequence recommendation method based on reinforcement learning enhanced long-term reward of claim 1, wherein: The specific method of step 2.4 is: fusing the output a from the teacher model t and the state s from the student model t to obtain a new representation state as follows: Wherein, ⊙ represents the element-wise product of two vectors.

4. The sequential recommendation method for enhancing long-term reward based on reinforcement learning according to claim 3, characterized in that: The specific method of step 2.5 is: presenting the state as input to the decoder f, yielding: wherein y t+1 represents a score of the item; according to y t+1 selects the top-k items with the highest scores as the recommendation result; a reward r t is defined as a normalized discounted cumulative gain NDCG calculated by the recommendation result.

5. The sequential recommendation method for enhancing long-term reward based on reinforcement learning according to claim 4, characterized in that: The specific method of step 2.6 is: For the supervised learning part, the cross-entropy loss function is calculated and trained by gradient descent: where I i is an indicator function that equals 1 if the user interacts with the ith item in the next time stamp, and 0 otherwise. i i is an indicator function that equals 1 if the user interacts with the ith item in the next time stamp, and 0 otherwise.​ 6. The sequential recommendation method based on reinforcement learning to enhance long-term reward according to claim 5, characterized in that: The specific method of step 2.7 is: For the Critic, it is defined as: wherein, and are trainable parameters of the Critic part of the first layer; [·] denotes a merge operation; Q(s t , a t ) is a Q-Value, representing the long-term reward that can be obtained by the reinforcement learning policy when selecting to perform action a t in state s t . The parameters of the Critic part are updated by minimizing the single-step time difference error, and the loss function L(Q, D) is: Wherein, Target = (r t + γ(1 - d)Q'(s t+1 , μ'(s t+1 ))) Wherein Q' and μ' are the copy models of Critic Q and Actor μ, which are used to make the reinforcement learning model more stable during training; D is a set of state transitions, and the elements in the set are (s t , a t , r t+1 , d), wherein s t represents the state at time t, a t represents the action taken at time t, r t+1 represents the reward obtained after taking action a t in state s t , d represents whether the next time is a terminal state, if the next time is a terminal state, d = 1, otherwise d = 0; γ is a discount factor in reinforcement learning, representing the weight of future rewards in the reinforcement learning strategy.

7. The sequential recommendation method based on reinforcement learning to enhance long-term reward according to claim 6, characterized in that: The specific method of step 2.8 is: Based on the Q-Value output by the Critic, gradient ascent is performed for the Actor according to the following formula: Wherein the Critic Q function parameters are regarded as constants here; To make the actor policy explore better, trying better solutions, add noise to the actions a that the actor takes during training t i.e. where the clip function returns a t when a Low + ∈ is less than a Low ; returns a t + ∈ when a High + ∈ is greater than a High ; and returns a t + ∈ when a Low + ∈ is between a High and a t ; ∈ is a Gaussian noise with zero mean. 8.The sequence recommendation method based on reinforcement learning enhanced long-term reward of claim 1, wherein: The method for generating a recommendation result: The copy model Copy Actor μ', Copy Critic Q', the copy decoder f', and the copy encoder G' are updated according to the following rules: F' = β * F' + (1-β) F Wherein F' represents any one of the copy models μ', Q', f', or G', F represents any one of μ, Q, f, or G corresponding to F'; β is a hyperparameter ranging from 0 to 1; During training, the output of a copy of the decoder f, f', is used to compute the reward r t ; during testing, the output of the decoder f is used to generate the recommendation result.

Images