A Multi-task Reinforcement Learning Recommendation Method
By introducing knowledge representation learning as an auxiliary task in the recommendation system and combining knowledge graph information, the problem of traditional recommendation systems being unable to optimize long-term cumulative rewards and insufficient exploration capabilities is solved, the accuracy and long-term benefits of recommendations are improved, and the generalization of the model is improved. Technique and training speed.
Patent Information
- Application Number
- CN202111450470.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-30
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2041-11-30
AI Technical Summary
Traditional recommendation systems cannot optimize long-term cumulative rewards, ignore dynamic changes in user interests, lack effective exploration capabilities, and the application of reinforcement learning in recommendation systems has problems such as weak generalization ability and slow training process.
Using a multi-task reinforcement learning recommendation method based on knowledge representation learning, by constructing Markov decision-making process and DDQN recommendation model, combining knowledge graph information, introducing knowledge representation learning as auxiliary tasks, alternately updating the recommendation model and knowledge representation learning model to improve recommendations accuracy and long-term benefits.
It improves the accuracy and long-term benefits of recommendations, enhances the generalization and training speed of the model, and can further explore the potential semantic relationship between user interests and projects.
Smart Images

Figure CN114417124B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of reinforcement learning recommendation, and relates to a multi-task reinforcement learning recommendation method, in particular to a multi-task reinforcement learning recommendation based on knowledge representation learning enhancement. Background Art
[0002] With the explosive growth of the Internet, information overload is becoming more and more serious for users. Users are often overwhelmed by massive amounts of information. It is becoming increasingly difficult for users to quickly find the information they want or are interested in at the right time. Therefore, online services increasingly rely on automatic and personalized recommendations of relevant content to a large number of users. Recommendation systems have been successfully applied to various scenarios such as online video, online music, social networks, and e-commerce. The purpose of recommendation systems is to select information spaces and guide users to select items that best meet their needs and interests.
[0003] Traditional recommendation systems have the following problems: (1) Since users’ search, click, and purchase behaviors are a continuous sequence decision-making process, traditional recommendation systems usually maximize customers’ short-term returns and are unable to model this continuous process, and therefore cannot optimize long-term cumulative rewards; (2) Most traditional recommendation systems regard the recommendation process as a static process that will not change once the model training is completed. This approach ignores the dynamic changes in user interests. In real life, user interests not only change over time, but are also affected by the recommended content; (3) Traditional recommendation systems lack effective exploration capabilities, and the system is more inclined to push information units that have occurred in the past to consumers.
[0004] In order to solve these problems, some scholars have begun to try to apply reinforcement learning (RL) to recommendation systems. Reinforcement learning regards the recommendation task as a sequential interaction between the RL agent (system) and the user (environment). It is a dynamic modeling process. The agent can continuously update the strategy according to the real-time feedback of the user during the interaction until the system converges to the best strategy and generates the object that best matches the user's dynamic preferences. The RL framework is designed to maximize the long-term cumulative rewards of users. Therefore, the agent can identify objects with small immediate rewards but large long-term contributions to rewards.
[0005] However, while solving the above problems, reinforcement learning recommendation has also generated many new problems to be solved: (1) An important part of the application modeling of reinforcement learning in recommendation systems is to effectively represent the state and action of the recommendation agent in reinforcement learning based on the specific features of the recommendation system. Currently, most studies use deep learning to learn high-level vector representations from the original state. These studies represent the state / action of each user in isolation and do not effectively explore and utilize the potential connections between different products. (2) Since reinforcement learning data is generated during the interaction between the model and the environment, it is generally believed that the generalization ability of reinforcement learning is weak and the model is more likely to overfit to the current training environment. (3) Since deep reinforcement learning requires incremental updates, when using the gradient method to update the network, large updates will make the network very unstable, so each increment needs to be very small so that the newly learned information will not be covered, which makes the reinforcement learning training process very slow.
[0006] When a multi-task learning model performs multiple tasks at the same time, it can apply the knowledge learned from other tasks to the target task, thereby improving the effect of the target task. In deep learning, multi-task models are usually divided into hard sharing of parameters and soft sharing based on constraints. In hard sharing, the lower layers of all tasks share hidden layers, and the higher layers retain the specific output layers of each task to achieve it. In soft sharing, each task has its own model and parameters, and the distance between the model and the parameters is regularized. In neural models, multi-task models obtain more general representations by averaging noise, so that the model has better generalization; it can also help the model focus on important features, because other tasks will provide additional evidence for the relevance or irrelevance of these features; it can use model eavesdropping to allow tasks that are difficult to learn features to better learn certain features from tasks that are easy to learn; it also helps related tasks to complement each other's information through shared representations, which to a certain extent alleviates the data sparsity problem in tasks. Summary of the invention
[0007] In view of the above-mentioned prior art, the technical problem to be solved by the present invention is to provide a multi-task reinforcement learning recommendation based on knowledge representation learning enhancement to improve the long-term benefits and recommendation accuracy of the recommendation.
[0008] In order to solve the above technical problems, a multi-task reinforcement learning recommendation method of the present invention comprises the following steps:
[0009] Step 1: Collect the user's rating information on the project set and construct the vector representation of the project based on the project's configuration file;
[0010] Step 2: Collect items with user ratings and all related knowledge graph information to construct corresponding triples;
[0011] Step 3: Construct a Markov decision process and establish a Q network;
[0012] Step 4: Establish a recommendation model based on DDQN;
[0013] Step 5: Establish a knowledge representation learning model;
[0014] Step 6: Train knowledge representation learning as an auxiliary task of the DDQN recommendation model, and update the DDQN recommendation model and knowledge representation learning model alternately.
[0015] Furthermore, the vector representation of the project is as follows: the structured data features are processed through a fully connected layer to obtain their low-dimensional dense vector representation, the unstructured data features are processed through a convolutional neural network or a recurrent neural network to obtain their dense vector representation, and the low-dimensional dense vector and the dense vector are concatenated to generate the feature vector representation of the project. n , the feature vectors of all items in the total item set P form the feature matrix The matrix dimension is N i ×d, where N i Indicates the number of items.
[0016] Furthermore, the corresponding triples are constructed as follows: head , relation, Entity tail ), E=Entity head ∪Entity tail , E represents all entities, and the relationship between entities is represented by such triples, and the entity vector E and relation vector R are randomly initialized, where the dimension of E vector is N e ×d, the relation vector dimension is N r ×d.
[0017] Furthermore, in step 3, the Markov decision process is constructed and the Q network is established, including:
[0018] Step 3.1: Process the user's rating to get the reward. where μ u represents the mean rating of the user, σ u Indicates the user's rating standard deviation, and rating represents the user's rating of the project;
[0019] Step 3.2: Add N items whose reward is greater than 0 in the user interaction items sItems are retained as part of the user state, using Indicates the current state of the user. A gated recurrent unit is used in function F to capture the user's sequential preference. The gated recurrent unit includes a reset gate r n and an update gate z n , r n Determines how to combine new input information with previous memory state information h n-1 Combined, z n Defines the amount of memory saved to the current time step. As the input of GRU, h n As the user's state S t ;
[0020] Step 3.3: action is represented by the feature vector of the item to be recommended to the user at that moment, i.e., a t =i rec ;
[0021] Step 3.4: After recommending an item to a user, if the user receives a positive reward, the state is changed from Convert to If no positive reward is obtained, the state remains unchanged;
[0022] Step 3.5: Construct Q network: Take the feature vector a of the item to be recommended t With the user's current status t As network input, a t The entity vector e corresponding to this item is processed through a feature cross-sharing unit: Generate a new action vector a' t , where the value of each α vector is used by the project and entity to adjust the degree of sharing of feature combinations. t With the action vector a' t After splicing, the state action value Q(s) of the recommended item is output through the fully connected layer t ,a t )=MLP([s t ,a' t ]).
[0023] Furthermore, the recommendation model based on DDQN is established in step 4 as follows: Assuming that the probability of randomly selecting an item for recommendation is ε, and the probability of selecting an item for recommendation in the Q network is 1-ε, if the Q network is used for recommendation, the items that have been recommended to the user and the initial N SItems are removed from the item set P, and the item set that is not recommended to the user is generated as the item set to be recommended. The feature vector of each item in the item set to be recommended and the current state of the user are used as the input of the Q network, and the item with the largest Q value is selected. Recommended to users as recommended items.
[0024] Furthermore, the knowledge representation learning model established in step 5 is specifically as follows: the entity e related to the recommended item is combined with the feature vector a of the item t After being processed by the feature cross-sharing unit, it becomes e': The value of each α vector is used by the project and entity to adjust the degree of feature combination sharing. After connecting e' with the relation feature representation vector r, the tail entity is predicted through the fully connected layer.
[0025] Furthermore, in step 6, knowledge representation learning is trained as an auxiliary task of the DDQN recommendation model, and alternately updating the DDQN recommendation model and the knowledge representation learning model includes:
[0026] Step 6.1: Initialize a Q network and a target Q network to generate Q estimates and Q target values respectively. The parameters of the Q network are θ and the parameters of the target Q network are θ. - , the initial parameters are the same;
[0027] Step 6.2: Observe the user's status and recommend items to them through the recommendation model. Collect (s t ,a t ,r t ,s') tuple is stored in the experience replay pool for training;
[0028] Step 6.3: Each training step randomly samples a batch of m experience tuples from the experience replay pool. For each tuple, the state action value Q is calculated through the Q network. θ (s t ,a t ), and use the Q network to select the next recommended item a based on the stored s' t+1 , calculate the target value The mean square error loss function is Update all parameters θ of the Q network and parameters α in the feature cross-sharing unit through gradient back propagation of the neural network;
[0029] Step 6.4: Update the parameters of the target Q network every C steps: θ - =θ;
[0030] Step 6.5: Perform a network update for knowledge representation learning every L steps: Collect all knowledge graph triplets related to the project trained by the recommendation system, and update the entities in these triplets. head and relation as e and r input into the knowledge representation learning model to output Then, the cosine distance between the predicted tail entity and the real tail entity is used as the loss function, and the parameters in the knowledge representation learning network and the parameter α in the feature cross-sharing unit are updated according to the loss function, where the cosine distance formula is:
[0031]
[0032] Beneficial effects of the present invention: The present invention introduces knowledge representation learning as an auxiliary task into reinforcement learning recommendation. The introduction of knowledge representation can introduce more semantic relationships for projects, dig into the potential connections between each user's state actions, discover the user's interests at a deeper level, and improve the accuracy of recommendations and long-term rewards. At the same time, the present invention introduces knowledge representation learning in a multi-task manner. In the training, reinforcement learning recommendation model training is the main method, and knowledge representation learning training is the auxiliary method, and alternating training is performed. The present invention uses knowledge representation learning as an auxiliary task for reinforcement learning recommendation, and introduces connections between projects and between recommendations for reinforcement learning recommendation, so that the recommendation model can dig deeper into the potential semantic relationships between projects and actions, thereby improving the long-term benefits of recommendations and the accuracy of recommendations; at the same time, due to the use of a multi-task model, the generalization of reinforcement learning training is improved, and the training speed is also improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 is a flow chart of the method of the present invention;
[0034] Figure 2 It is a model framework diagram of the method of the present invention;
[0035] Figure 3 This is the internal structure diagram of the DDQN part of the method of the present invention. DETAILED DESCRIPTION
[0036] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0037] Combination Figure 1 , the present invention comprises the following steps:
[0038] (1) Collect the user's rating information on the project set and construct a vector representation of the project based on the project configuration file;
[0039] (2) Collect items with user ratings and all related knowledge graph information to construct corresponding triples;
[0040] (3) Construct a Markov decision process (MDP) and establish a Q network;
[0041] (4) Establish a recommendation model based on DDQN;
[0042] (5) Establish a knowledge representation learning model;
[0043] (6) Knowledge representation learning is trained as an auxiliary task of the DDQN recommendation model, and the DDQN recommendation model and knowledge representation learning model are updated alternately.
[0044] Example:
[0045] (1) Collect the user's rating information on the item set and construct the vector representation of the item based on the item's configuration file. We use movie metadata to construct the vector representation of the item. Specifically, we convert the sparse features such as movie ID, movie year, and movie theme into low-dimensional dense vectors through embedding, process the movie name through a recurrent neural network, and concatenate them to generate the feature vector representation of the item. n , the vectors of all items form a feature matrix The matrix dimension is N i ×d, where N i Indicates the number of items;
[0046] (2) Collect the knowledge graph information of all movies and construct the corresponding triples (Entity head , relation, Entity tail ), for example (Titanic, director, James Cameron), such a triple is used to represent the relationship between entities, and the entity vector E and relation vector R are randomly initialized, where the entity vector dimension is N e ×d, the relation vector dimension is N r ×d.
[0047] (3) Construct a Markov decision process (MDP) and establish a Q network. The various parts of the MDP are constructed as follows:
[0048] ① Process the user's rating to get the reward. Specifically, where μ u represents the mean rating of the user, σ u Indicates the standard deviation of the user's rating.
[0049] ② Set N of the user interaction items whose reward is greater than 0 s Items are retained as part of the user state, using Indicates the current state of the user. In this function F, we use a gated recurrent unit (GRU) to capture the user's sequential preference. GRU has two gates, namely a reset gate r n and an update gate z n , r n Determines how to combine new input information with previous memory state information h n-1 Combined, z n Defines the amount of previous memory saved to the current time step. Our work will As the input of this GRU, h n As our user's status t .
[0050] ③action is represented by the feature vector of the item to be recommended to the user at that moment, i.e. a t =i rec .
[0051] ④ When we recommend an item to a user, if the user receives a positive reward, our state input will be Convert to If you do not get a positive reward, it is still
[0052] The construction of Q network is to transform the feature vector a of the recommended item t After a feature cross-sharing unit, a new action vector a' is generated t . The state vector s after processing by the GRU unit t With the action vector a' t After splicing, The fully connected layer of the layer calculates the state action value of the recommended item
[0053] The feature cross-sharing unit is a unit for information sharing between the recommendation model and knowledge representation learning. t To better share features with the corresponding entity features e, we first perform feature combination C = a on the relevant vectors. t e T , improve the model's expressiveness and improve the effect, where a t Both e and d are d-dimensional features. The feature combination matrix is d×d. After obtaining the feature combination matrix, we use the cross-stitch unit to automatically learn the features that need to be shared by our two tasks. The value of each α vector is used by the project and entity to adjust the degree of feature combination sharing. The feature cross-sharing unit mentioned in the knowledge representation learning below is consistent with this one.
[0054] (4) Establish a recommendation model based on DDQN such as Figure 3 In order to explore and utilize, in this algorithm, there is a probability of ε to randomly select items for recommendation, and a probability of 1-ε to select actions for recommendation in the Q network. If the Q network is used for recommendation, the feature vector of each item in the recommended item set passes through a feature cross-sharing unit to generate a new action matrix A', the dimension of which is N c ×d, where N c is the number of items in the set of items to be recommended. Each row in the matrix is an action vector of an item to be recommended. The state vector S t and the action vector of each item in A' The Q value is calculated Then select the item with the largest Q value from the items to be recommended The movie corresponding to the project is recommended to the user as a recommended project.
[0055] (5) Establish a knowledge representation learning model such as Figure 2 On the right, the knowledge representation learning model converts the entity e related to the recommended item into e' after processing it through the feature cross-sharing unit, connects e' with the relation feature representation vector r, and predicts their tail entities through the fully connected layer.
[0056] (6) Knowledge representation learning is used as an auxiliary task for training the DDQN recommendation model, and the DDQN recommendation model and the knowledge representation learning model are updated alternately. The training process is shown in Figure 2 Before training begins, we initialize a Q network and a target Q network to generate Q estimates and Q target values respectively. The parameters of the Q network are θ and the parameters of the target Q network are θ. - , their initial parameters are the same, and the specific training process is as follows:
[0057] ①Observe the user's status and recommend items to them through the recommendation model. Collect (s t ,a t ,r t ,s') tuple is stored in the experience replay pool for training;
[0058] ② Each training step randomly samples a batch of m experience tuples from the experience playback pool. For each tuple, the state action value Q is calculated through the Q network. θ (s t ,at ), and use the Q network to select the next recommended item a based on the stored s' t+1 , calculate the target value The mean square error loss function is Update all parameters θ of the Q network and parameters α in the feature cross-sharing unit through gradient back propagation of the neural network;
[0059] ③ Update the parameters of the target Q network every C steps: θ - =θ;
[0060] ④ Perform a network update for knowledge representation learning every L steps: collect all knowledge graph triplets related to the project trained by the recommendation system, and update the entities in these triplets. head and relation as e and r input into the knowledge representation learning model to output Then, the cosine distance between the predicted tail entity and the real tail entity is used as the loss function, and the parameters in the knowledge representation learning network and the parameter α in the feature cross-sharing unit are updated according to the loss function. The cosine distance formula is:
[0061]
Claims
1. A multi-task reinforcement learning recommendation method, It is characterized in that The following steps are involved: Step 1: Collect the user's rating information on the project set and construct the vector representation of the project based on the project's configuration file; Step 2: Collect items with user ratings and all related knowledge graph information to construct corresponding triples; Step 3: Construct a Markov decision process and establish a Q network; Step 3.1: Process the user's rating to get the reward. where μ u represents the mean rating of the user, σ u Indicates the user's rating standard deviation, and rating represents the user's rating of the project; Step 3.2: Add N items whose reward is greater than 0 in the user interaction items s Items are retained as part of the user state, using Indicates the current state of the user. A gated recurrent unit is used in function F to capture the user's sequential preference. The gated recurrent unit includes a reset gate r n and an update gate z n , r n Determines how to combine new input information with previous memory state information h n-1 Combined, z n Defines the amount of memory saved to the current time step. As the input of GRU, h n As the user's state S t ; Step 3.3: Action is represented by the feature vector of the item to be recommended to the user, i.e., a t =i rec ; Step 3.4: After recommending an item to a user, if the user receives a positive reward, the state is changed from Convert to If no positive reward is obtained, the state remains unchanged; Step 3.5: Construct Q network: Take the feature vector a of the item to be recommended t With the user's current status t As network input, a t The entity vector e corresponding to this item is processed through a feature cross-sharing unit: Generate a new action vector a' t , where the value of each α vector is used by the project and entity to adjust the degree of sharing of feature combinations; the state vector s t With the action vector a' t After splicing, the state action value Q(s) of the recommended item is output through the fully connected layer t ,a t )=MLP([s t ,a' t ]); Step 4: Establish a recommendation model based on DDQN; Assume that the probability of randomly selecting an item for recommendation is ε, and the probability of selecting an item for recommendation in the Q network is 1-ε. If the Q network is used for recommendation, the items that have been recommended to the user and the initial N items are first S Items are removed from the item set P, and the item set that is not recommended to the user is generated as the item set to be recommended. The feature vector of each item in the item set to be recommended and the current state of the user are used as the input of the Q network to select the item with the largest Q value. Recommended to users as recommended items; Step 5: Establish a knowledge representation learning model; Step 6: Train knowledge representation learning as an auxiliary task of the DDQN recommendation model, and update the DDQN recommendation model and knowledge representation learning model alternately.
2. A multi-task reinforcement learning recommendation method according to claim 1, Features: The vector representation of the project is specifically as follows: the structured data features are processed through a fully connected layer to obtain their low-dimensional dense vector representation, the unstructured data features are processed through a convolutional neural network or a recurrent neural network to obtain their dense vector representation, and the low-dimensional dense vector and the dense vector are concatenated to generate the feature vector representation of the project. n , the feature vectors of all items in the total item set P form a feature matrix ü, the matrix dimension is N i ×d, where N i Indicates the number of items.
3. A multi-task reinforcement learning recommendation method according to claim 1, Features: The construction of the corresponding triple is specifically as follows: head , relation, Entity tail ), E = Entity head ∪Entity tail , E represents all entities, and the relationship between entities is represented by such triples, and the entity vector E and relation vector R are randomly initialized, where the dimension of E vector is N e ×d, the dimension of the relation vector is N r ×d.
4. The multi-task reinforcement learning recommendation method according to claim 1, Features: Step 5 of establishing the knowledge representation learning model is as follows: the entity e related to the recommended item is combined with the feature vector a of the item t After being processed by the feature cross-sharing unit, it becomes e': The value of each α vector is used by the project and entity to adjust the degree of feature combination sharing; e' is connected to the relation feature representation vector r and then processed through the fully connected layer to predict the tail entity 5. The multi-task reinforcement learning recommendation method according to claim 1, Features: In step 6, the knowledge representation learning is used as an auxiliary task of the DDQN recommendation model for training, and the DDQN recommendation model and the knowledge representation learning model are alternately updated, including: Step 6.1: Initialize a Q network and a target Q network to generate Q estimates and Q target values respectively. The parameters of the Q network are θ and the parameters of the target Q network are θ. - , the initial parameters are the same; Step 6.2: Observe the user's status and recommend items to them through the recommendation model. Collect (s t ,a t ,r t ,s') tuple is stored in the experience replay pool for training; Step 6.3: Each training step randomly samples a batch of m experience tuples from the experience replay pool. For each tuple, the state action value Q is calculated through the Q network. θ (s t ,a t ), and use the Q network to select the next recommended item a based on the stored s' t+1 , calculate the target value The mean square error loss function is Update all parameters θ of the Q network and parameters α in the feature cross-sharing unit through gradient back propagation of the neural network; Step 6.4: Update the parameters of the target Q network every C steps: θ - =θ; Step 6.5: Perform a network update for knowledge representation learning every L steps: Collect all knowledge graph triplets related to the project trained by the recommendation system, and update the entities in these triplets. head and relation as e and r input into the knowledge representation learning model output Then, the cosine distance between the predicted tail entity and the real tail entity is used as the loss function, and the parameters in the knowledge representation learning network and the parameter α in the feature cross-sharing unit are updated according to the loss function, where the cosine distance formula is:
