A neural bandits recommendation method fusing collaborative filtering

Abstract

The application discloses a neural Bandits recommendation method fusing collaborative filtering, and comprises the following steps: step one, constructing an exploration network and a utilization network based on a user to be recommended, and constructing a collaborative filtering module; step two, in the tth round of recommendation, giving an arm x t，i By utilizing the network to output a potential reward estimation, by the exploration network to output a potential reward gain, and by the collaborative filtering module to obtain a collaborative score of a neighbor user of the current user, step three, inputting the potential reward estimation, the potential reward gain and the collaborative score of the neighbor user of the current user into a decision maker to obtain an optimal selection strategy based on user recommendation. By constructing the utilization network and the exploration network to learn an expected reward and a potential gain, and by simultaneously considering a neighbor collaborative effect, a decision maker is constructed to make a final decision, so that the recommendation diversity is improved, and the cold start problem is relieved.

Description

TECHNICAL FIELD

[0001] The application relates to a neural Bandits recommendation method fusing collaborative filtering, and belongs to the technical field of recommendation systems. BACKGROUND

[0002] In the current unprecedented prosperity of the Internet economy, there are massive amounts of information presented to people every day. For e-commerce, video, music, news and other platforms, it is essential to promote user consumption to convert the content or goods provided by them into value. For users, it is a basic requirement to quickly find the content of interest to them by opening a certain platform to reduce the cost of searching for information. Under this situation, the research on personalized recommendation has become a hot issue.

[0003] As the most commonly used traditional recommendation algorithm, collaborative filtering finds a group of neighbor users similar to the target user through the behavior characteristics of individuals in the group, and makes recommendations by using the similarity. Through continuous collaboration, more and more accurate recommendation results can be obtained. However, the collaborative filtering algorithm itself has some key problems, namely the sparsity and cold start problems. In the actual recommendation scene, the scale of users and goods is very large, and users usually only explore a small part of goods (ratings). There are no or few common rating items between most users, and the sparsity of such data limits the calculation of similarity between users. Even if the calculation is possible, the reliability is difficult to guarantee. The cold start problem is essentially an extreme manifestation of the data sparsity problem. Frequent changes in users and goods result in insufficient rating items, which affects the recommendation effect.

[0004] In addition to the consideration of the above problems, the recommendation system should also take into account the diversity and breadth of the recommendation as much as possible, that is, the exploration-exploitation problem. Exploitation means that the known user's interest in goods should be catered to as much as possible, and exploration means that the user is recommended goods that he has not explored to explore his new interests. The multi-armed Bandit algorithm is an effective method for solving the exploration-exploitation problem in the field of reinforcement learning, which can balance the relationship between exploration and exploitation in each round of recommendation.

[0005] In the multi-armed bandit algorithm, a set of items is presented to the user in each round, and an item is selected by the selected strategy and the corresponding reward is obtained. The goal of the problem is to maximize the cumulative reward. The traditional multi-armed bandit algorithm does not consider the characteristics of users and items, and the same recommendation strategy is used for different users. In order to better utilize the content (item) and user information to adapt to individual users, the LinUCB algorithm models the personalized recommendation of news as a context multi-armed bandit problem, and calculates the expected reward based on the context information to recommend for the user, and adjusts the selection strategy according to the user feedback. However, the LinUCB algorithm assumes that the context and the reward are linearly related, which may not be correct in the real world. In order to learn the non-linear reward function, the NeuralUCB algorithm uses the powerful representation ability of the deep neural network to learn the potential reward function, and uses the UCB (Upper Confidence Bound) algorithm to select the arm. Unlike the NerualUCB algorithm, the EE-Net discards the UCB algorithm and adopts a new neural exploration strategy, which uses a neural network to learn the potential gain compared with the reward estimate.

[0006] The context-based multi-armed bandit algorithm fully adapts to the personalized needs of users, but does not consider the importance of similar users in recommendation. In view of this problem, some researches have appeared to integrate collaborative filtering and context multi-armed bandit algorithms. Based on the target item, the user is dynamically clustered, but when the user and item scale is large, frequent clustering will affect the real-time recommendation efficiency. The LinUCB algorithm is added to the collaborative effect of similar users, and the similarity between the current user and the selected user is used to control the collaborative strength, but limited by the assumption of the LinUCB algorithm on the linear reward function, it cannot adapt to the non-linear reward situation well. SUMMARY

[0007] The present application designs and develops a neural Bandits recommendation method integrating collaborative filtering, which learns the expected reward and potential gain by constructing a network and an exploration network, and simultaneously considers the neighbor collaborative effect, constructs a decision maker for final decision, improves the recommendation diversity and solves the cold start problem.

[0008] The technical scheme provided by the present application is:

[0009] A neural Bandits recommendation method integrating collaborative filtering, comprising:

[0010] Step one, based on the user to be recommended, construct an exploration network and a network, and construct a collaborative filtering module;

[0011] Step two, in the tth round of recommendation, given arm x t，iBy utilizing the potential reward estimate of the network output and exploring the potential reward gain of the network output, the collaborative score of the current user's neighboring users is obtained through the collaborative filtering module.

[0012] Step 3: Input the potential reward estimate, potential reward gain, and collaboration scores of the current user's neighboring users into the decision-maker to obtain the optimal selection strategy based on user recommendations.

[0013] Preferably,

[0014] The network is set to f1(·; θ) 1 In the t-th round of recommendations, given arm x t，i The potential reward obtained by utilizing the network is estimated to be... The tag is r t，i ;

[0015] Set the input of the exploration network as The output is Tag is

[0016] Set the input to the collaborative filtering module as follows: The output is cscore i ;

[0017] The inputs to the decision-maker are f1; f2; cscore i The output is f3(f1, f2, cscore) i ), tagged as p i ;

[0018] In the formula, These are the parameters of the last layer of the network, with the superscript representing the network identifier and the subscript indicating the round number of the last parameter update. t，i Used to represent the i-th arm in the t-th round of recommendations. To utilize the network's gradient, r t，i This represents the real reward for choosing the i-th arm in the t-th round of recommendations.

[0019] Preferably,

[0020] The decision-maker combines the potential reward estimate, the potential reward gain, and the collaboration score of the current user's neighboring users, and is defined as x. t，i The final score p t，i about cscore i The function yields:

[0021]

[0022] For m arms x provided in round tt，i ∈A t , i ∈ [m], select the arm with the highest score, the selection strategy is represented as:

[0023]

[0024] Preferably,

[0025] The step two comprises:

[0026] Set the user set as U = {u1, u2,..., u n}, for user u i , in the tth round of recommendation, by using the similarity of different users in the parameter gradient calculation using the network, the similarity formula is:

[0027]

[0028] Select the M users with the highest similarity to join the neighbor user set Neg i , user u j The collaborative score cscore t of the target item x i，j is defined as the sum of the output of the target item in the user's exploration and exploitation network and the similarity product of the user and the current user u i , the formula is:

[0029]

[0030] The collaborative score of the current user u i is defined as the sum of the collaborative scores of all neighbor users, the calculation formula is:

[0031]

[0032] Preferably,

[0033] When the decision function is a linear function,

[0034] p t，i = αf1+ βf2+ γcscore i ;

[0035] When the decision function is a nonlinear function, the decision maker is defined as a two-layer neural network, and the number of hidden layer nodes is 50;

[0036] In each round of recommendation, select the arm with the highest output of the decision maker for recommendation.

[0037] Preferably,

[0038] The utilization network and the exploration network are both two-layer neural networks, the number of hidden layer nodes is 100, and the number of neighbor users of the collaborative filtering module is 3.

[0039] The neural Bandits method COEENet fusing collaborative filtering provided by the application adopts a utilization network to learn a reward function about user and item features, adopts an exploration network to learn potential gains compared with reward estimation, simultaneously adds the collaborative effect of neighbor users, and constructs a decision maker to make a final decision. Unlike collaborative filtering, the method does not simply rely on the scores of the same items to determine the similarity between users, but calculates based on the attention of users to each feature in the context; in the recommendation problem for new users, neighbor users are found according to user features and collaborative recommendation is performed, meanwhile, the multi-armed bandit algorithm can also help users quickly explore user preferences in the least number of times, and challenge model parameters according to user feedback to improve the recommendation effect; finally, by balancing the results of the exploration network and the utilization network and controlling the strength of the collaborative effect, the decisive role of the user in the recommendation and the diversity of the recommendation can be effectively guaranteed. BRIEF DESCRIPTION OF DRAWINGS

[0040] Fig. 1 The network structure diagram of the neural Bandits recommendation method fusing collaborative filtering.

[0041] Fig. 2 The cumulative regret comparison diagram on the recipe data set by the linear decision maker.

[0042] Fig. 3 The cumulative regret comparison diagram on the recipe data set by the nonlinear decision maker. DETAILED DESCRIPTION

[0043] The application will be further described in detail below with reference to the accompanying drawings, so that those skilled in the art can implement the application according to the description.

[0044] As Figs. 1-3 shown, the application provides a neural Bandits method fusing collaborative filtering, which learns expected rewards and potential gains by constructing a utilization network and an exploration network, simultaneously considers neighbor collaborative effect, constructs a decision maker to make a final decision, improves recommendation diversity, and relieves the cold start problem, and the method comprises the following steps.

[0045] Step one, constructing an exploration network and a utilization network based on a user to be recommended, and constructing a collaborative filtering module;

[0046] Step two, in the tthround of recommendation, given arm x t，iBy using the network to output the potential reward estimation, by exploring the network to output the potential reward gain, by the collaborative filtering module to get the current user neighbor user's collaborative score;

[0047] Step three, input the potential reward estimation, the potential reward gain, the current user neighbor user's collaborative score into the decision maker, get the optimal selection strategy based on user recommendation.

[0048] The network is used for reward estimation of arms, the structure is defined as a two-layer neural network, the number of hidden layer nodes is 100; the exploration network is used for estimating the potential gain compared with the reward estimation in the network, the structure is a two-layer neural network, the number of hidden layer nodes is 100; the collaborative filtering module is used for calculating the collaborative action of the neighbor user similar to the current user, the neighbor user is selected as 3; the decision maker is used for integrating the results of the above three modules, including linear decision maker and nonlinear decision maker. The formula of linear decision maker is defined as p t，i =f1+f2+r×cscore i , wherein the weight of the output f1 of the exploitation network and the output f2 of the exploration network is 1, and the neighbor user score cscore i is adjusted by adjusting the collaborative action strength r. The nonlinear decision maker is defined as a two-layer neural network with 50 hidden layer nodes. The arm with the highest output of the decision maker is selected for recommendation in each round of recommendation.

[0049] User-based collaborative filtering

[0050] In the recommendation system, there is a user set U and an item set I, and the user and the item are represented as u i (i=1,2,...,n),i j (j=1,2,...,m),score i，j represents the rating of user i to item j. Before each recommendation, for each candidate item j:

[0051] The Euclidean distance is used to calculate the similarity sim i，j between the current user and other users, and the like of the target item of the current user u i is calculated according to the ratings of all users u∈U to the target item:

[0052]

[0053] By calculating the like of the current user to each item in the item set, the items can be ranked in descending order, and the top N items with the highest ratings are selected for recommendation.

[0054] General context multi-arm tiger machine framework

[0055] In the contextual multi-armed bandit algorithm, the total number of rounds T is known, and in the t-th (t = 1, 2,..., T) round of recommendation process:

[0056] The method observes the user feature u t and a set of arms A available for selection and presented to the user t and their feature vectors x t，a which combines the user feature and the feature of the arm, referred to as the context here;

[0057] An arm a t ∈ A t is selected for the user, and real-time feedback (reward) in the real world is obtained

[0058] The method uses new observations to improve the selection strategy;

[0059] Depending on the specific selection strategy, the reward estimation function is different, and the specific form is:

[0060]

[0061] wherein, represents the potential reward of the user u t selecting the arm a t in the t-th round, and the function can be linear or nonlinear, represents the noise parameter of and the cumulative regret in T rounds is defined as:

[0062]

[0063] wherein, The goal of the method is to minimize the cumulative regret.

[0064] The method includes four parts: the first part is to use a network to estimate the expected reward; the second part is to use an exploration network to learn the potential gain compared with the reward estimation; the third part is a collaborative filtering module for calculating the synergy of similar users; and the fourth part is a decision maker for combining the first three parts.

[0065] Use of network:

[0066] The exploitation network is a neural network for learning the potential reward function about the context, denoted by f1(·; θ 1 ). In the t-th round, the exploitation network is denoted by is the parameter of the last layer of the network, and the upper index is the network identifier, and the lower index represents the round number of the last time the parameter is updated. For the arm x t，i, i e [m], the network estimates its potential reward, after selecting the arm, will get a real-world real feedback r t , based on which the network can be used to gradient descent to fit the parameters θ 1 .

[0067] Exploration network:

[0068] Set the context of a given arm as x t，i , its expected return f1(x t，i ) and the actual return r t，i There is a 1-δ probability in the confidence interval , formula as follows:

[0069]

[0070] Where r t，i -f1(x t，i ) is defined as the potential gain compared to the reward estimate, is a confidence upper bound function about .

[0071] Unlike the traditional confidence upper bound method, which derives the confidence interval by ridge regression, etc., the exploration network uses a neural network to fit the potential gain. The exploration network takes as input

[10] , and

[0072] Collaborative filtering module

[0073] In the recommendation scenario, for each user u e U, |U| = n, build exploration and exploitation networks, and save the parameter gradient of the last update of the exploitation network. For user u i , in the tth round of recommendation, calculate the similarity of the parameter gradient of the exploitation network of different users. The Euclidean distance is used to measure the similarity, and the formula is as follows:

[0074]

[0075] Select the M users with the highest similarity to join the neighbor user set Neg i , and the collaborative score cscore j of the target item x t for user u i，j is defined as the product of the similarity of the target item in the output of the user exploitation and exploration networks and the current user u i , the formula is:

[0076]

[0077] The current user u iThe collaborative score of a user is defined as the sum of the collaborative scores of all its neighbor users, and the formula is as follows:

[0078]

[0079] Table 1 COEENet network structure

[0080]

[0081] Decision maker

[0082] In the t-th round, given arm x t，i , via the utilization network, exploration network and collaborative filtering module, three outputs will be obtained: potential reward estimate Potential reward gain And the collaborative score of the current user's neighbor users cscore i The task of the decision maker is to combine the above three outputs to balance the relationship between exploration and utilization, while controlling the strength of collaborative filtering. Let the decision maker be x t，i The final score of p t，i The function of cscore i is defined as follows:

[0083]

[0084] For the m arms x t，i ∈A t provided in the t-th round, i∈[m], select the one with the highest score, and the selection strategy is represented as:

[0085]

[0086] Considering the linear and nonlinear cases of the decision function, we will describe them separately here:

[0087] 1) Linear decision function: f3 is defined as a linear function of f1, f2, cscore i :

[0088] p t，i = αf1+ βf2+ γcscore i ;

[0089] Where, α, β, γ are weight parameters, which are set by the user.

[0090] 2) Nonlinear decision function: use a neural network to represent f3, model the problem as a binary classification problem. The network takes cscore i as input, and outputs p t，i representing arm xt，i For the probability of optimal selection, specifically includes:

[0091]

[0092]

[0093] Embodiments

[0094] The recommendation process is simulated on real data sets, and compared with other baseline methods to verify the effectiveness of the method COEENet. The cumulative regret in T rounds is used as an evaluation index, and the experiment shows that in the food recommendation problem, the method has an improvement compared with other methods.

[0095] Data sets

[0096] 3526 user information and 340734 rating information of dishes in the past period of time are crawled from the recipe website, and information of 9766 dishes involved in the rating is also collected. User information includes id, gender, age, address, custom preferences, and dish information includes id, name, cooking process (stir-fry, steam, fry, fry, etc.), difficulty level, taste, etc. The rating is defined as a three-tuple of (user id, dish id, rating), and the rating includes five levels: extreme difference, difference, general, recommendation, and very recommended.

[0097] First, the dish popularity is sorted according to the number of ratings, and the top 1000 popular dishes are selected to join the dish data set dish.dat, and the 2871 users who rated these dishes are added to the user data set user.dat, and the rating information of these users to the dishes is collected to join the rating data set rating.dat.

[0098] Feature construction

[0099] Each user is represented by a 100-class original feature vector, which includes:

[0100] 1) User basic information: id, gender (two categories), age discretized into seven levels;

[0101] 2) Geographic features: mainly distributed in more than 100 cities and regions in China;

[0102] 3) User-defined preference information: mainly including 14 categories such as sweet, spicy, and numb;

[0103] Similarly, each dish is represented by a 1000-class original feature vector, which includes: 1) dish basic information: id, cooking process, difficulty level, etc.

[0104] 2) User evaluation label: about 1000 labels are extracted from user historical evaluation data; user and dish features are encoded into fixed-length binary vectors, and user and dish features are represented by 156 and 1153 item feature vectors, respectively.

[0105] The method builds a model for each user for personalized recommendation. The context information in the method only needs to represent the dish information without including user information, so the user's rating information of the dish is defined in the form of (user id, dish feature, rating), where one data represents a user's interaction with the system. Since most users have fewer rating records, it is not possible to support independent model training. K-Means method is used for clustering users to further reduce the user scale to 20 categories.

[0106] Offline simulation of online recommendation

[0107] In the online recommendation model, the model displays the top several items with the highest expected revenue to the user, and adjusts the model parameters based on the user's real-time feedback, so that it gradually adapts to the user's needs in subsequent recommendations. Since it is not possible to obtain user feedback information in real time, the online recommendation process is simulated on the offline data set to verify the recommendation effect of the method. Through the rating information of the user on the dish in the data set, the interaction between the user and the dish is simulated. In the network and exploration network, the rating can be used as the real-time reward of the user to the corresponding dish. In the decision maker, if the rating is not more than 3, the reward is 0, otherwise the reward is 1. In each round, 10 dishes are randomly displayed to the user, including 9 dishes with a rating of not more than 3 and 1 dish with a rating of more than or equal to 3. Each dish corresponds to the feature vector of the current dish.

[0108] Offline simulation of online recommendation

[0109] To better evaluate the performance of the method in the recommendation problem, I choose four algorithms as baseline methods, which are LinUCB, NeuralUCB, Neural-Epsilon and EE-Net;

[0110] LinUCB: assumes that the reward function is a linear function of the context, and uses ridge regression and confidence upper bound algorithm to determine the selected arm together;

[0111] NeuralUCB: uses neural network to learn the reward function, and uses confidence upper bound algorithm to select the arm;

[0112] Neural-Epsilon: uses neural network to learn the reward function, selects the arm with the maximum reward with a probability of 1-ε, and explores with a probability of ε;

[0113] EE-Net: using one neural network to learn the reward function, using another neural network to learn the potential reward gain, and combining the outputs of the two networks to make a joint decision;

[0114] Evaluation index

[0115] The present application adopts the cumulative regret in the T wheel recommendation as the algorithm evaluation index, and the calculation formula is defined as:

[0116]

[0117] Among them, p t is the true reward of the tth wheel. This method minimizes the cumulative regret in the T wheel as the optimization goal, and the smaller the cumulative regret, the better the performance of this method.

[0118] Experimental result analysis

[0119] The network structure is defined as a two-layer neural network, and the number of hidden layer nodes is 100. The exploration network structure is a two-layer neural network, and the number of hidden layer nodes is also 100. In the collaborative filtering module, the synergistic effect of the three neighbor users of the current user is considered. The linear decision maker formula is defined as p t，i =f1+f2+r×cscore i , wherein the weights of the network output f1 and the exploration network output f2 are both 1, and the neighbor user score cscore i is adjusted by adjusting the synergistic effect intensity r. The nonlinear decision maker is defined as a two-layer neural network, and the number of hidden layer nodes is 50.

[0120] Fig. 2 and Fig. 3 respectively show the cumulative regret comparison results of the linear and nonlinear decision makers, and it can be seen that the performance of the present method COEENet on the recipe data set is better than that of the baseline algorithm. Since the context and reward expectation in the recipe data set are not a simple linear relationship, the LinUCB algorithm is not applicable in this problem. The NeuralUCB and Neural-Epsilon algorithms use neural networks to learn the reward expectation, and the EE-Net algorithm uses one neural network to learn the reward expectation and another neural network to learn the potential gain. The three can better adapt to the case of nonlinear reward function. The present method can well fit the nonlinear function while considering the synergistic effect of the neighbor users, further improving the recommendation effect, and for the linear decision maker, the recommendation effect is improved with the increase of the synergistic intensity.

[0121] There are challenges in personalized recommendation problem: 1) the data sparsity and cold start problem caused by the frequent changes of users and items in recommendation system limit the development of traditional collaborative filtering; 2) the existing multi-armed bandit algorithm considering the synergy is not suitable for the scene of nonlinear reward function. In view of the above problems, a neural bandit algorithm COEENet is proposed, which uses a network to learn the reward function about user and item features, uses an exploration network to learn the potential gain compared with reward estimation, and adds the synergy of neighbor users, and constructs a decision maker to make the final decision. In view of the problem of data sparsity, unlike collaborative filtering, this algorithm does not simply rely on the ratings of the same items to determine the similarity between users, but calculates based on the attention of users to each feature in the context. In the recommendation problem for new users, according to the user features, the neighbor users are found and the collaborative recommendation is carried out, and the multi-armed bandit algorithm can also help users to quickly explore user preferences in the least number of times, and challenge the model parameters according to user feedback, and improve the recommendation effect. Finally, by balancing the results of the exploration network and the utilization network and controlling the strength of the synergy, the decisive role of the user in the recommendation and the diversity of the recommendation can be effectively guaranteed. Finally, the proposed method is compared with the baseline algorithm on real data set, and the experimental results show the effectiveness of the proposed method in the recommendation problem.

[0122] Although the embodiments of the present application have been disclosed as above, it is not limited to the use listed in the specification and embodiments, and can be fully applied to various fields suitable for the present application, and additional modifications can be easily made by those skilled in the art, and therefore the present application is not limited to specific details and the figures shown and described herein, without departing from the general concept defined by the claims and the equivalent scope.

Claims

1. A neural bandits recommendation method fusing collaborative filtering, characterized in that, The method comprises the following steps: Step one, constructing an exploration network and a utility network based on a user to be recommended, and constructing a collaborative filtering module; Step two, in the tth round of recommendation, given arm x t，i By utilizing the network output potential reward estimation, the network output potential reward gain is explored, and the collaborative score of the current user neighbor user is obtained through the collaborative filtering module. Let the set of users be U = {u1, u2,..., u n}, for a user u i In the t-th round of recommendation, the similarity between different users is calculated by using the parameter gradient on the network, and the similarity formula is: Select the M users with the highest similarity to add them to the neighbor user set Neg i User u j For target item x t Collaborative score (cscore) i，j Defined as the sum of the outputs of the target item by the user's utilization and exploration of the network, plus the sum of the outputs of the user and the current user u. i The product of similarity is given by the formula: The current user u i The collaborative score of the current user u is defined as the sum of the collaborative scores of all the neighbor users of u, and the calculation formula is as follows: Step three, inputting the potential reward estimation, the potential reward gain, and the collaborative score of the current user neighbor user into a decision maker to obtain an optimal selection strategy based on user recommendation; Set the network to be used as f1(·; θ 1 ), in the t-th round of recommendation, given arm x t，i The latent reward estimate obtained by using the network is The label is r t，i ; The input to the exploration network is set to The output is The label is The input to the collaborative filtering module is set to The output is cscore i ; Inputs to the decision maker are f1; f2; cscore i Outputs from the decision maker are f3(f1, f2, cscore i ), and label p i ; In the formula, is the parameter of the last layer of the network, which is marked with the network identifier above and the round number at which the parameter was last updated below, x t，i is used to represent the i-th arm in the t-th round of recommendation, is the gradient of the network, r t，i represents the true reward of selecting the i-th arm in the t-th round of recommendation; combining the potential reward estimate, the potential reward gain the co- score of the current user neighbor user by a decision maker, set the decision maker as x t,i the final score p t,i about cscore i the function of, get: For the m arms x provided in the tth round t,i ∈A t , i ∈ [m], select the one with the highest fraction, the selection policy is represented as:

2. The collaborative filtering neural bandits recommendation method of claim 1, wherein, When the decision function is a linear function, p t,i = a f1 + b f2 + g c score i ; When the decision function is a nonlinear function, the decision maker is defined as a two-layer neural network, and the number of hidden layer nodes is 50; In each round of recommendation, the arm with the highest output of the decision maker is selected for recommendation.

3. The collaborative filtering neural bandits recommendation method of claim 1 or 2, wherein, Both the utility network and the exploration network are two-layer neural networks, the number of hidden layer nodes is 100, and the number of neighbor users of the collaborative filtering module is 3.

Images