Temporal Recommendation Method and System Based on Deep Reinforcement Learning

Abstract

The present invention provides a temporal recommendation method and system based on deep reinforcement learning. The method includes: generating a user state st at time t according to the item set It of the user at time t and user information; taking the user state st as the input information of the trained Actor model to obtain a recommendation action at under the user state st; recommending the recommendation action at to the user and obtaining corresponding user feedback information rt; obtaining the user state st+1 and item set It+1 at time t+1 according to the user feedback information rt, and storing the corresponding recommendation action set (st, at, rt, st+1) at time t into the experience pool M to update the experience pool M; taking the user state st and the recommendation action at as the input information of the trained Critic model to obtain the Q function value under the user state st, and calculating the loss function value according to the Q function value and the user feedback information rt; updating the network parameters of the Actor model and the network parameters of the Critic model with the latest experience pool M as the training data according to the Q function value and the loss function value.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular, to a temporal recommendation method and system based on deep reinforcement learning. Background Art

[0002] Currently, with the increasingly rapid development of Internet technology, the interaction between people's daily lives and the Internet has become more and more frequent. Shopping, entertainment, social networking, etc. all rely on Internet technology, and the increasingly rich content on the Internet can also provide users with a better user experience. The rich content and frequent user interactions have caused a huge increase in the amount of data. Therefore, how to provide accurate item recommendations for users based on a large amount of information has become a key problem that needs to be solved urgently.

[0003] The methods used in current recommendation systems can be divided into two categories: traditional methods and neural network-based methods. Traditional methods mainly include: content-based methods, collaborative filtering, and hybrid recommendation algorithms. Among these, the collaborative filtering method is used more frequently. Its main idea is that the products liked by users are similar, and similar products are liked by similar users. Neural network-based methods mainly include: deep neural networks, convolutional neural networks, and deep reinforcement neural networks, etc. The advantage of deep neural networks is that they can accurately extract key information in the input information. The advantage of convolutional neural networks is that the model complexity is low and they can be trained quickly. Thanks to the rapid development of deep learning, deep reinforcement learning combines the advantages of deep learning and reinforcement learning, can train model parameters in real time, continuously improve prediction accuracy, and at the same time its advantage of being able to train in real time is compatible with temporal recommendation.

[0004] Although recommendation systems are constantly evolving, their disadvantages are still obvious, mainly including the following problems: (1) Cold start problem: When a new user joins the system, it is difficult for the system to recommend items without user information. Similarly, it is also difficult for the system to decide which users to recommend new items to. (2) Long tail problem: Most existing recommendation systems are based on clustering recommendations, and popular items will become even more popular, which leads to the inability to recommend novel items to users. (3) Problem of real-time obtaining of user interest changes: Users' interests and hobbies change over time, with strong real-time and variability. How to quickly and accurately capture users is also an important issue for improving recommendation accuracy and enhancing user stickiness. Summary of the Invention

[0005] In order to solve or partially solve the cold start problem, long tail problem, and user interest change problem existing in existing recommendation methods, the present invention provides a temporal recommendation method and system based on deep reinforcement learning.

[0006] On the one hand, the present invention provides a temporal recommendation method based on deep reinforcement learning, including:

[0007] Step 1: Based on the user's item set I at time t t And user information, generate the user status s at time t t ;Wherein, the item set refers to the first n item data browsed by the user;;

[0008] Step 2: The user status s t As the input information of the trained Actor model, the user state s is obtained t Recommended action a t ;

[0009] Step 3: The recommended action a t Recommend to users and obtain corresponding user feedback information r t ;

[0010] Step 4: Based on the user feedback information r t Get the user status s at time t+1 t+1 and Project Set I t+1 And the recommended action set corresponding to time t (s t ,a t ,r t ,s t+1 ) is stored in the experience pool M to update the experience pool M;

[0011] Step 5: The user status s t and the recommended action a t As the input information of the trained Critic model, the user state s is obtained t The Q function value under the condition of t Calculate the loss function value;

[0012] Step 6: Based on the Q function value and the loss function value, the network parameters of the Actor model and the network parameters of the Critic model are updated using the latest experience pool M as training data.

[0013] Furthermore, in step 1, the user status s t Expressed as:

[0014]

[0015] Among them, g(I t )=max(I t )|I t ={i1,i2,…,i n}, Represents element-wise product, i nIt is the feature matrix of project n. g(.) represents the max pooling layer, and u represents the user feature matrix.

[0016] Further, in step 3, after recommending the recommended action a t to the user, according to the operation type of the project included in the recommended action a t , calculate the reward value of the corresponding project, and obtain the corresponding user feedback information r by integrating the reward values of all projects t .

[0017] Further, for any project in the recommended action a t , if the customer chooses to skip, the reward value corresponding to the project is the first score; if the customer chooses to click, the reward value corresponding to the project is the second score; if the customer chooses to favorite, the reward value corresponding to the project is the third score; where the third score > the second score > the first score.

[0018] Further, the calculation formula of the Q function value is:

[0019]

[0020] where γ represents the discount factor, and Q ω (s t , a t ) represents the Q function value at time t. is the selection strategy of Q ω (s t , a t ).

[0021] Further, denote the project set of the user at time t as: I t = {i1, i2, …, i n}. If the user clicks on the project i t of the recommended action a n+1 at time t + 1, then update the project set I t+1 at time t + 1 to: {i2, i3, …, i n+1}, otherwise I t+1 = {i1, i2, …, i n}.

[0022] Further, in step 5, the calculation formula of the loss function value of the Actor model is:

[0023]

[0024] where J(π θ ) is the expected value of the Q values of all policies that follow π θ , and N is the batch size.

[0025] Further, the calculation formula for the loss function value of the Critic model is as follows:

[0026]

[0027] y t = r t + γQ ω′ (s t+1 , π θ′ (s t+1 ))

[0028] where θ′ is the network parameter of the Actor model, and ω′ is the network parameter of the Critic model.

[0029] Further, the update formula for the network parameter of the Actor model is as follows:

[0030] θ u′ = τθ u + (1 - τ)θ u′

[0031] The update formula for the network parameter of the Critic model is as follows:

[0032] θ π′ = τθ π + (1 - τ)θ π′

[0033] where τ is the learning rate of the network parameter.

[0034] On the other hand, the present invention provides a time-series recommendation system based on deep reinforcement learning, including:

[0035] A user state generation module, configured to generate the user state s at time t according to the item set I of the user at time t t and the user information; wherein, the item set refers to the data of the first n items browsed by the user; t ;

[0036] An action recommendation module, configured to use the user state s t as the input information of the trained Actor model to obtain the recommended action a t under the user state s t ;

[0037] A feedback module, configured to recommend the recommended action a t to the user and obtain the corresponding user feedback information r t ;

[0038] An experience pool module, configured to according to the user feedback information r tObtain the user state s at time t+1 t+1 and the item set I t+1 , and store the set of recommended actions corresponding to time t (s t , a t , r t , s t+1 ) in the experience pool M to update the experience pool M;

[0039] Loss function calculation module, used to use the user state s t and the recommended action a t as the input information of the trained Critic model, obtain the Q function value under the user state s t , and calculate the loss function value according to the Q function value and the user feedback information r t ;

[0040] Model update module, used to update the network parameters of the Actor model and the network parameters of the Critic model with the latest experience pool M as the training data according to the Q function value and the loss function value.

[0041] Advantages of the present invention:

[0042] (1) Each time a fixed action is selected, there is no need to calculate the Q values of all actions, reducing the calculation amount, improving the response speed, and being more suitable for real-time online recommendation; (2) Adopting the Actor-Critic architecture, it is more suitable for online time-series recommendation; (3) The model of deep reinforcement learning updates the network parameters every once in a while, and can better capture the changes in user interests; (4) By using historical user data to pre-train the network model, the cold start problem can be effectively solved. Brief Description of the Drawings

[0043] Figure 1 is a schematic flowchart of a time-series recommendation method based on deep reinforcement learning provided by an embodiment of the present invention;

[0044] Figure 2 is a schematic structural diagram of a user state generation module provided by an embodiment of the present invention;

[0045] Figure 3 is a schematic network structure diagram of an Actor-Critic model provided by an embodiment of the present invention. Detailed Embodiments

[0046] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] Example 1

[0048] like Figure 1 As shown, an embodiment of the present invention provides a time series recommendation method based on deep reinforcement learning, comprising the following steps:

[0049] S101: If Figure 2 As shown, based on the first n items browsed by the user at time t, the user status s at time t is generated. t ;

[0050] Specifically, in the embodiment of the present invention, the user status s t It is expressed as shown in formula (1):

[0051]

[0052] Among them, g(I t )=max(I t )|I t ={i1,i2,…,i n}, Represents element-wise product, i n is the feature matrix of item n, g(.) represents the maximum pooling layer, and u represents the user feature matrix.

[0053] By extracting and converting the project data of user interactions and ratings into vectors, and sorting each user's behavior data by time, a "user-item-rating" matrix is generated; the user feature matrix and the item feature matrix can be obtained by performing probability matrix decomposition on the "user-item-rating" matrix.

[0054] S102: The user status s t As the input information of the trained Actor model, the user state s is obtained t Recommended action a t ;

[0055] Specifically, in the embodiment of the present invention, the recommended action a t Expressed as: a t =π θ (s t ), where π θRepresents an action selection strategy.

[0056] S103: Recommend the recommended action a t to the user and obtain the corresponding user feedback information r t ;

[0057] Specifically, to further improve the accuracy of the recommendation, the recommended action a obtained in step S102 t can be further processed as follows: Multiply the recommended action a t by the item space I t to obtain a predicted score, and then take the N items with the highest predicted scores as the final recommended action a t ; The above matrix multiplication formula is: where the item space I t represents the feature matrix of all items, represents the transpose of the recommended action a t .

[0058] After recommending the recommended action a t to the user, calculate the reward value of the corresponding item according to the operation type of the item included in the recommended action a t , and obtain the corresponding user feedback information r by integrating the reward values of all items t .

[0059] As an implementable way, the integration of the reward values of all items to obtain the corresponding user feedback information r t is specifically: where reward(*) is the reward function, i t represents the reward value of item i at time t, and N represents the total number of items included in the recommended action a t .

[0060] As an implementable way, for any item in the recommended action a t , if the customer chooses to skip, the reward value corresponding to the item is the first score; if the customer chooses to click, the reward value corresponding to the item is the second score; if the customer chooses to favorite, the reward value corresponding to the item is the third score; where the third score > the second score > the first score.

[0061] To facilitate the calculation of the user feedback information, in the embodiments of the present invention, the reward value is set in the following way: If the customer chooses to skip, the reward value r 跳过 = 0 points; if the customer chooses to click, the reward value r 点击 = 1 point; if the customer chooses to favorite, the reward value r 收藏 = 2 points.

[0062] S104: According to the user feedback information r t obtain the user state s at time t + 1 t+1 , and store the set of recommended actions (s t , a t , r t , s t+1 ) at time t into the experience pool M to update the experience pool M;

[0063] S105: Use the user state s t and the recommended action a t as the input information of the trained Critic model, obtain the Q - function value under the user state s t , and calculate the loss - function value according to the Q - function value and the user feedback information r t ;

[0064] As an implementable manner, the calculation formula of the Q - function value is shown in formula (2):

[0065]

[0066] where γ represents the discount factor, Q ω (s t , a t ) represents the Q - function value at time t, is the selection strategy of Q ω (s t , a t ). The Q - function value is an index for evaluating the recommendation performance. <\

[0067] As an implementable manner, the calculation formula of the loss - function value of the Actor model is shown in formula (3):

[0068]

[0069] where J(π θ ) is the expected value of the Q - values of all policies that comply with π θ , and a small - batch policy is used here, and N is the batch size.

[0070] At the same time, adopt the temporal - difference learning method to update the Critic network. The calculation formula of the loss - function value of the Critic model is to minimize the mean - square error of equation (4):

[0071]

[0072] y t = r t + γQ ω′ (s t+1 , πθ′ (s t+1 ))(5)

[0073] Among them, θ′ is the parameter of the Actor network, and ω′ is the parameter of the Critic network.

[0074] According to the loss function of the above Actor model, update the network parameters of the Actor model according to formula (6):

[0075] θ′ = τθ + (1 - τ)θ′(6)

[0076] According to the loss function of the above Critic model, update the network parameters of the Critic model according to formula (7):

[0077] ω′ = τω + (1 - τ)ω′(7)

[0078] Among them, τ is the learning rate of the network parameters.

[0079] Embodiment 2

[0080] When targeting social networks such as Weibo and Douyin, taking Weibo as an example, another embodiment of the method of the present invention is further given, which specifically includes the following steps:

[0081] S201: Generate the user state s at time t according to the top n topics (also called dynamics, hereinafter uniformly referred to as topics for convenience of description) browsed by the user at time t t ;

[0082] Specifically, in the embodiment of the present invention, the user state s t is expressed as shown in formula (1):

[0083]

[0084] Among them, g(I t ) = max(I t )|I t = {i1, i2,..., i n} represents the element product, i n represents the feature matrix corresponding to the feature information of the nth topic browsed by the user's history. The feature information of the topic can include the classification information of the topic and the information of the publishing user. g(.) represents the max pooling layer, and u represents the user feature matrix corresponding to the user information. The user information can include the age, gender, and ID information of the user.

[0085] S202: Use the user state s t as the input information of the trained Actor model to obtain the user state s tThe recommended action a below t ;

[0086] Specifically, in the embodiments of the present invention, the recommended action a t is represented as: a t = π θ (s t ); where π θ represents the action selection strategy.

[0087] S203: Recommend the recommended action a t to the user and obtain the corresponding user feedback information r t ;

[0088] Specifically, in order to further improve the accuracy of the recommendation, the recommended action a obtained in step S202 t can be further processed, specifically: multiplying the recommended action a t by the item space I t to obtain a predicted score, and then taking the N topics with the highest predicted scores as the final recommended action a t ; the above matrix multiplication formula is: where the item space I t represents the feature matrix of all topics, represents the transpose of the recommended action a t .

[0089] After recommending the recommended action a t to the user, calculate the reward value of the corresponding item according to the operation type of the topics included in the recommended action a t , and obtain the corresponding user feedback information r by integrating the reward values of all items t .

[0090] As an implementable manner, the integration of the reward values of all topics to obtain the corresponding user feedback information r t is specifically: where reward(*) is the reward function, i t represents the reward value of topic i at time t, and N represents the total number of topics included in the recommended action a t .

[0091] As an implementable manner, for any topic in the recommended action a t , if the customer does not perform an operation, the reward value corresponding to the topic is the first score; if the customer gives a like / forward (including quick forward, share), the reward value corresponding to the topic is the second score; if the customer makes a comment, the reward value corresponding to the topic is the third score; where the third score > the second score > the first score.

[0092] In order to facilitate the calculation of user feedback information, in the embodiment of the present invention, the reward value is set in the following way: if the customer does not perform any operation, the reward value r 跳过 =0 points; if the customer likes / reposts, the reward value is r 点赞 =1 point; if the customer chooses to comment, the reward value is r 评论 =2 points.

[0093] S204: Based on the user feedback information r t Get the user status s at time t+1 t+1 And the recommended action set corresponding to time t (s t ,a t ,r t ,s t+1 ) is stored in the experience pool M to update the experience pool M;

[0094] S205: The user status s t and the recommended action a t As the input information of the trained Critic model, the user state s is obtained t The Q function value under the condition of t Calculate the loss function value;

[0095] As an implementation method, the calculation formula of the Q function value is shown in formula (2):

[0096]

[0097] Where γ represents the discount coefficient, Q(s t ,a t ) represents the Q function value at time t, It is Q(s t ,a t ) selection strategy. The Q function value is an indicator for evaluating the recommendation performance.

[0098] As an implementation method, the calculation formula of the loss function value of the Actor model is shown in formula (3):

[0099]

[0100] Among them, J(π θ ) are all the θ The expected value of the Q value of the strategy. Here, a small batch strategy is used, and N is the batch size.

[0101] At the same time, the temporal difference learning method is used to update the critic network. The calculation formula of the loss function value of the critic model is to minimize the mean square error of equation (4):

[0102]

[0103] y t =r t +γQ ω′ (s t+1 ,π θ′ (s t+1 )) (5)

[0104] Among them, θ′ is the parameter of the Actor network, and ω′ is the parameter of the Critic network.

[0105] According to the loss function of the Actor model, the network parameters of the Actor model are updated according to formula (6):

[0106] θ′=τθ+(1-τ)θ′(6)

[0107] According to the loss function of the above Critic model, the network parameters of the Critic model are updated according to formula (7):

[0108] ω′=τω+(1-τ)ω′(7)

[0109] Among them, τ is the learning rate of network parameters.

[0110] Example 3

[0111] Regarding shopping networks such as Taobao, JD.com, and Vipshop, Taobao is used as an example to further provide another embodiment of the method of the present invention, which specifically includes the following steps:

[0112] S301: Generate the user status s at time t based on the first n items purchased by the user at time t t ;

[0113] Specifically, in the embodiment of the present invention, the user status s t It is expressed as shown in formula (1):

[0114]

[0115] Among them, g(I t )=max(I t )|I t ={i1,i2,…,i n}, represents the element-wise product, i nIt represents the feature matrix corresponding to the feature information of the nth item browsed by the user. The feature information of the item includes information such as the classification and ID of the item. g(.) represents the max pooling layer, and u represents the feature matrix corresponding to the user information, including the user's age, gender, and ID information.

[0116] S302: Use the user state s t as the input information of the trained Actor model to obtain the recommended action a t under the user state s t ;

[0117] Specifically, in the embodiment of the present invention, the recommended action a t is represented as: a t = π θ (s t ); where π θ represents the action selection strategy.

[0118] S303: Recommend the recommended action a t to the user and obtain the corresponding user feedback information r t ;

[0119] Specifically, after recommending the recommended action a t to the user, calculate the reward value of the corresponding item according to the operation type of the item included in the recommended action a t , and obtain the corresponding user feedback information r t by integrating the reward values of all items.

[0120] As an implementable manner, the integration of the reward values of all items to obtain the corresponding user feedback information r t is specifically: where reward(*) is the reward function, i t represents the reward value of item i at time t, and N represents the total number of items included in the recommended action a t .

[0121] As an implementable manner, for any item in the recommended action a t , if the customer does not perform an operation, the reward value corresponding to the item is the first score; if the customer performs a collection, the reward value corresponding to the item is the second score; if the customer makes a purchase, the reward value corresponding to the item is the third score; where the third score > the second score > the first score.

[0122] To facilitate the calculation of the user feedback information, in the embodiment of the present invention, the reward value is set in the following manner: if the customer does not perform an operation, the reward value r 跳过 = 0 points; if the customer performs a collection, the reward value r收藏 = 1 point; if the customer chooses to purchase, the reward value r 购买 = 2 points.

[0123] S304: Based on the user feedback information r t obtain the user state s at time t + 1 t+1 , and store the set of recommended actions corresponding to time t (s t , a t , r t , s t+1 ) in the experience pool M to update the experience pool M;

[0124] S305: Use the user state s t and the recommended action a t as the input information of the trained Critic model, obtain the Q-function value under the user state s t , and calculate the loss function value based on the Q-function value and the user feedback information r t ;

[0125] As an implementable manner, the calculation formula of the Q-function value is shown in formula (2):

[0126]

[0127] where γ represents the discount factor, Q(s t , a t ) represents the Q-function value at time t, is the selection strategy of Q(s t , a t ). The Q-function value is an index for evaluating the recommendation performance.

[0128] As an implementable manner, the calculation formula of the loss function value of the Actor model is shown in formula (3):

[0129]

[0130] where J(π θ ) is the expected value of the Q-values of all policies that follow π θ , and a small batch of policies are used here. N is the batch size.

[0131] At the same time, the time difference learning method is adopted to update the Critic network. The calculation formula of the loss function value of the Critic model is the mean square error that minimizes equation (4):

[0132]

[0133] y t = rt +γQ ω′ (s t+1 ,π θ′ (s t+1 )) (5)

[0134] Among them, θ′ is the parameter of the Actor network, and ω′ is the parameter of the Critic network.

[0135] According to the loss function of the above Actor model, update the network parameters of the Actor model according to formula (6):

[0136] θ′ = τθ + (1 - τ)θ′ (6)

[0137] According to the loss function of the above Critic model, update the network parameters of the Critic model according to formula (7):

[0138] ω′ = τω + (1 - τ)ω′ (7)

[0139] Among them, τ is the learning rate of the network parameters.

[0140] Embodiment 4

[0141] When it comes to the culture and entertainment network, taking movie recommendation as an example, another embodiment of the method of the present invention is further given, which specifically includes the following steps:

[0142] S401: Generate the user state s at time t according to the top n movies rated by the user at time t t ;

[0143] Specifically, in the embodiment of the present invention, the user state s t is represented as shown in formula (1):

[0144]

[0145] Among them, g(I t ) = max(I t )|I t = {i1, i2,..., i n}, represents the element product, i n represents the feature information of the nth movie in the user's historical ratings, including the classification information and ID information of the movie, g(.) represents the max pooling layer, and u represents the user feature matrix, including the user's age, gender and ID information.

[0146] S402: Use the user state s t as the input information of the trained Actor model to obtain the recommended action a under the user state s t ​t ;

[0147] Specifically, in the embodiments of the present invention, the recommended action a t is represented as: a t =π θ (s t ); where π θ represents the action selection strategy.

[0148] S403: Recommend the recommended action a t to the user and obtain the corresponding user feedback information r t ;

[0149] Specifically, after the recommended action a t is recommended to the user, calculate the reward value of the corresponding item according to the operation type of the movie included in the recommended action a t , and obtain the corresponding user feedback information r t by integrating the reward values of all items.

[0150] As an implementable manner, the integration of the reward values of all movies to obtain the corresponding user feedback information r t is specifically: where reward(*) is the reward function, i t represents the reward value of item i at time t, and N represents the total number of movies included in the recommended action a t .

[0151] As an implementable manner, for any movie in the recommended action a t , if the user rates the recommended movie, use the rating score as the reward value.

[0152] S404: Obtain the user state s t at time t + 1 according to the user feedback information r t+1 , and store the recommended action set (s t , a t , r t , s t+1 ) corresponding to time t in the experience pool M to update the experience pool M;

[0153] S405: Use the user state s t and the recommended action a t as the input information of the trained Critic model to obtain the Q function value under the user state s t , and calculate the loss function value according to the Q function value and the user feedback information r t ;

[0154] As an implementable manner, the calculation formula of the Q function value is shown in Formula (2):

[0155]

[0156] where γ represents the discount factor, and Q(s t ,a t ) represents the Q function value at time t, is the selection strategy of Q(s t ,a t ). The Q function value is an index for evaluating the recommendation performance.

[0157] As an implementable manner, the calculation formula of the loss function value of the Actor model is shown in Formula (3):

[0158]

[0159] where J(π θ ) is the expected value of the Q values of all policies that comply with π θ policy. Here, a mini-batch of policies is used, and N is the batch size.

[0160] At the same time, the time difference learning method is adopted to update the Critic network. The calculation formula of the loss function value of the Critic model is the mean square error that minimizes Equation (4):

[0161]

[0162] y t =r t +γQ ω′ (s t+1 ,π θ′ (s t+1 )) (5)

[0163] where θ′ are the parameters of the Actor network, and ω′ are the parameters of the Critic network.

[0164] According to the loss function of the above Actor model, update the network parameters of the Actor model according to Formula (6):

[0165] θ′=τθ+(1-τ)θ′(6)

[0166] According to the loss function of the above Critic model, update the network parameters of the Critic model according to Formula (7):

[0167] ω′=τω+(1-τ)ω′(7)

[0168] where τ is the learning rate of the network parameters.

[0169] Example 5

[0170] The embodiment of the present invention also provides a time series recommendation system based on deep reinforcement learning, including a user state generation module, an action recommendation module, a feedback module, an experience pool module, a loss function calculation module and a model update module;

[0171] Among them, the user state generation module is used to generate the user's item set I at time t. t And user information, generate the user status s at time t t ; The item set refers to the first n item data browsed by the user; the action recommendation module is used to convert the user state s t As the input information of the trained Actor model, the user state s is obtained t Recommended action a t ; The feedback module is used to recommend the action a t Recommend to users and obtain corresponding user feedback information r t ; The experience pool module is used to t Get the user status s at time t+1 t+1 and Project Set I t+1 And the recommended action set corresponding to time t (s t ,a t ,r t ,s t+1 ) is stored in the experience pool M to update the experience pool M; the loss function calculation module is used to store the user state s t and the recommended action a t As the input information of the trained Critic model, the user state s is obtained t The Q function value under the condition of t The loss function value is calculated; the model update module is used to update the network parameters of the Actor model and the network parameters of the Critic model based on the Q function value and the loss function value using the latest experience pool M as training data.

[0172] It should be noted that the time series recommendation system based on deep reinforcement learning provided in the embodiment of the present invention is intended to implement the above method embodiment. Its specific functions can be referred to the above method example and will not be described in detail here.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. And these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A temporal recommendation method based on deep reinforcement learning, characterized in that Including: Step 1: Generate the user status at t based on the item set at and user information; where the item set refers to the t first items data browsed by the user; and the user status n is represented as: ​ Among them, , represents element-wise product, the feature matrix of, represents the max pooling layer, represents the user feature matrix; Step 2: Use the user status as the input information of the trained Actor model to obtain the recommended action under the user status ; Step 3: Recommend the recommended action to the user, calculate the reward value of the corresponding item according to the operation type of the item included in the recommended action by the user, and obtain the corresponding user feedback information by synthesizing the reward values of all items ; Step 4: Based on the user feedback information Obtain t the user status at the +1 moment and the project set , and t store the set of recommended actions corresponding to the moment into the experience pool M to update the experience pool M ; Step 5: Take the user state and the recommended action as the input information of the trained Critic model, and obtain the Q function value under the user state According to the Q function value and the user feedback information calculate the loss function value; Step 6: Based on the Q function value and the loss function value, update the network parameters of the Actor model and the network parameters of the Critic model with the latest experience pool M as the training data.

2. The time-series recommendation method based on deep reinforcement learning according to claim 1, wherein For any item in the recommended actions if the customer chooses to skip, the reward value corresponding to the item is the first score value; If the customer chooses to click, the reward value corresponding to the item is the second score; if the customer chooses to collect, the reward value corresponding to the item is the third score; where the third score > the second score > the first score.

3. The temporal recommendation method based on deep reinforcement learning according to claim 1, wherein The calculation formula for the Q function value is: Among them, represents the discount factor, represents t the Q-function value at time is the selection strategy of 4. The temporal recommendation method based on deep reinforcement learning according to claim 1, characterized in that Denote the item set of the user t at a moment as: If, at the moment when the user t is at time +1, the user clicks on the item of the said recommended action , then update the item set t of the user at time +1 , otherwise .

5. The temporal recommendation method based on deep reinforcement learning according to claim 1, wherein In step 5, the calculation formula for the loss function value of the Actor model is: Among them, is the expected value of the Q-values that all comply with the policy, and N is the batch size.

6. The temporal recommendation method based on deep reinforcement learning according to claim 5, characterized in that The calculation formula for the loss function value of the Critic model is: Among them, are the network parameters of the Actor model, are the network parameters of the Critic model.

7. The temporal recommendation method based on deep reinforcement learning according to claim 6, characterized in that The update formula for the network parameters of the Actor model is: The update formula for the network parameters of the Critic model is: Among them, is the learning rate of the network parameters.

8. A temporal recommendation system based on deep reinforcement learning, characterized in that Including: A user status generation module, which is used to generate the user status at t based on the item set at a certain moment and user information; wherein, the item set refers to the first t items data browsed by the user; wherein, the user status is represented as: n ; where the user status is expressed as: Among them, , represents the element product, the feature matrix of represents the max pooling layer, represents the user feature matrix; An action recommendation module, which is used to use the user state as input information of the trained Actor model to obtain the recommended action under the user state ; A feedback module for sending the recommended action to the user, calculating a reward value for a corresponding item according to the operation type of the item included in the recommended action and obtaining corresponding user feedback information by integrating the reward values of all items ; An experience pool module, which is used to obtain the t user status at the +1 moment and the project set , and t the set of recommended actions corresponding to the moment is stored in the experience pool M to update M the experience pool; A loss function calculation module for using the user state and the recommended action as input information to a trained Critic model to obtain the Q function value under the user state and calculating a loss function value based on the Q function value and the user feedback information ; The model update module is used to update the network parameters of the Actor model and the network parameters of the Critic model with the latest experience pool according to the Q function value and the loss function value. M as training data.

Images