A reward correction-based flow music recommendation method for removing attention bias

Abstract

The application realizes a stream music recommendation method for removing attention deviation based on reward correction through the method in the network security field. The core of the method contains three modules: a reward correction model, an attention prediction module and a correlation prediction module. The reward correction module obtains the unbiased reward after correction by combining the predicted user attention and correlation through the importance sampling method; the attention prediction module models the probability of user attention to each song; the correlation prediction module is used to predict the preference of the user to each candidate song, and the parameter is updated based on the corrected reward. The application eliminates the attention deviation in the user feedback to obtain unbiased reward and improve the prediction accuracy of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a flow music recommendation method for removing attention deviation based on reward correction. BACKGROUND

[0002] Automatic playlist generation is an important application in the flow recommendation scenario of music APP. Generally, the music APP will recommend a song to the user, the user will give feedback (such as playing / cutting songs), and the recommendation algorithm in the background will continuously update the model according to the user's feedback to recommend the next song until the user exits the APP. A common method to model the automatic playlist generation problem is to regard it as a sequential song selection decision problem, that is, it can be modeled based on a slot machine algorithm. In the process of sequential decision, the model needs to be continuously updated based on the user's feedback, therefore, how to use accurate user feedback is an important problem. When the user receives a recommended song, he will play (positive feedback) / skip (negative feedback) the song. However, since the system is in an automatic playing scenario, the user has no action (cutting songs) and the system will continue to play, so there are some false positive feedbacks, that is, the user's attention is not on the song (such as the user is learning, working, commuting, exercising, etc., and only regards the music playing as a background sound) leading to the system automatically playing scenario, and we define the deviation problem caused by the false positive feedback as the user's attention deviation.

[0003] In order to eliminate the user's attention deviation, it is necessary to model the user's attention and the user's music preference mechanism. The existing slot machine-based flow music recommendation algorithm usually directly learns the recommendation model from the user's feedback or solves the user's attention deviation problem through heuristic methods. There is a context slot machine model for recommending music lists for multiple users to play, which uses a slidable window to recommend personalized content to users. For music streaming recommendation, there is an efficient non-regular slot machine optimization algorithm, which updates the recommendation model using the user's negative feedback. There is also a heuristic cutting technique to solve the user's attention deviation, which only retains the K songs completely played by the user after the active action (cutting songs) as positive feedback samples.

[0004] Although the existing slot machine-based recommendation algorithm has achieved relatively excellent performance in the flow music recommendation scenario, how to solve the user's attention deviation in the online scenario is still a problem to be solved. Although there are existing heuristic solutions, they ignore the context information of the user and the song, and usually the user has different behavior models for different contexts, therefore, the user's attention deviation algorithm based on learning still needs further research. SUMMARY

[0005] To this end, the present application first proposes a reward correction-based attention bias removal method for streaming music recommendation, which is used for candidate playlists of external systems, adopts a system composed of three modules: a reward correction module, an attention prediction module, and a relevance prediction module, first recommends music to users based on the relevance prediction module, then combines the predicted user attention and relevance to obtain the corrected unbiased reward through importance sampling method, and realizes the working process of the offline training stage, the online training stage and the online recommendation stage based on the three modules.

[0006] Specifically, the system inputs the biased feedback of the user to the reward correction module and the attention prediction module, models the probability of the user's attention to each song by the attention prediction module, generates the attention probability through a recurrent neural network, and feeds the attention probability back to the reward correction module.

[0007] The relevance prediction module predicts the user's preference for each candidate song, recommends the song that the user is most likely to like and generates the relevance probability, and provides the user with further corrected recommended music by using the corrected reward generated by the reward correction module through a random weighted neural network and a duel tiger machine algorithm.

[0008] The reward correction module corrects the directly observed reward r(x n (s), c) through the importance sampling method, obtains the unbiased estimate of the true reward r(x n (s), v) under the expectation, and specifically obtains the corrected reward through the following formula:

[0009] r mod (x n (s), c) := w·r(x n (s), c)

[0010] Where w is the importance sampling weight, defined as follows:

[0011]

[0012] In the formula, Pr{o=1|x n (s)} is the probability of user attention, and Pr{v=1|x n (s)} is the relevance probability, i.e. the user's preference for the current song.

[0013] The attention prediction module aims to predict the user's attention state at each song, and the target is to predict Pr{o=1|x n (s)}. First, the context and state of the user before the nth session bth song are stored un represents the feature of the n-th session user, represents the feature of the j-th song in the n-th session, c n，j represents the feedback of the user for the j-th song in the n-th session, the probability of the attention of the user for the b-th song in the n-th session can be represented as p n，b is the estimation of Pr{o=1|x n (s)}, β n is the parameter of the model g.

[0014] The attention prediction module is implemented by a multi-layer perception and two gated recurrent units. The attention prediction model g is modeled as follows:

[0015]

[0016] where represents the operation of concatenating two vectors, and σ(·) represents the element-level Sigmoid function, and y2(c n，b ) are obtained by two GRUs, respectively.

[0017] where the input of the first gated recurrent unit is the representation of the song and the output is

[0018]

[0019] h n，b and h n，b-1 are the hidden variables of the b-th step and the b-1-th step, respectively.

[0020] The input of the second gated recurrent unit is the feedback of the user c n，b , and the output is y2(c n，b ):

[0021] y2(c n , b ), h′ n，b = GRU2(c n，b , h′ n，b-1 ).

[0022] The correlation prediction module aims to predict the preference score of the user for each song, i.e., to predict the value of Pr{v=1|x n (s)}. To make the above prediction, the context feature x n (s) is first encoded by a RWNN φ(·) [9, 10] to increase its nonlinear representation capability, which is defined as follows:

[0023]

[0024] where is a random matrix, where each element is independently sampled from a Gaussian distribution is a random vector i.i.d sampled from [-π, π], cos(·) is the element-wise cosine function. Finally, the user’s preference score prediction for each song is where is the parameter of module f, σ(·) is the Sigmoid function.

[0025] The offline training phase updates the parameters β of the attention prediction model g after each user session, using cross-entropy loss and Adam optimizer for end-to-end optimization, using partially available labels: if the user skips a song c = 0, the corresponding attention o = 1, if the user plays more than 10 songs consecutively without active action, the corresponding attention o = 0 from the 11th song until an active action occurs, in addition, the first 5 songs of a session are also considered to have attention o = 1.

[0026] The online training phase uses the Dueling Bandit to update the relevance prediction module f, specifically, at the bth song of the nth user session, given the parameters θ n，b of f, we first select a duel parameter where is a random vector sampled from the unit sphere δ > 0 is a parameter that controls the degree of exploration. To evaluate the duel model, we calculate the corresponding interpolated reward where rank θ (s) outputs the corresponding position of s according to the descending order of f(x n (·) | θ) in the candidate set Then, if the interpolated reward is greater than the corrected reward , it means that the duel model has a better reward, we update the parameters of f(· | θ n，b ) as follows:

[0027]

[0028] where γ > 0 is the step size;

[0029] For each θ n，b repeat the above update steps K times, that is, there are K duel models each time.

[0030] ​The online recommendation stage selects the song with the highest score from the candidate set according to the predicted relevance at the bth step of the nth user session n

[0031]

[0032] The technical effect to be achieved by the present application is:

[0033] The present application proposes a slot machine algorithm based on reward correction for user attention bias in music streaming recommendation: the weight of reward correction is derived according to importance sampling, which includes the prediction of user attention and user preference (relevance), so that the user attention and user preference are learned through two modules respectively, and finally the recommendation algorithm is updated based on the corrected reward. The present application eliminates the attention bias in user feedback to obtain unbiased reward and improve the accuracy of model prediction. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 Architecture diagram of the streaming music recommendation algorithm for removing user attention bias based on reward correction DETAILED DESCRIPTION

[0035] The following is a preferred embodiment of the present application, which is further described in conjunction with the drawings, but the present application is not limited to this embodiment.

[0036] The present application proposes a streaming music recommendation method for removing attention bias based on reward correction.

[0037] Streaming music recommendation algorithm for removing user attention bias based on reward correction

[0038] The present method is mainly composed of three modules: reward correction module, attention prediction module and relevance prediction module, based on the input music database, and finally realizes music recommendation for users.

[0039] The reward correction module obtains the unbiased reward after correction by combining the predicted user attention and relevance through the importance sampling method.

[0040] The attention prediction module models the probability of user attention to each song, which is realized by recurrent neural network (RNN).

[0041] The relevance prediction module predicts the preference of the user for each candidate song, so as to recommend the song that the user is most likely to like, which is realized by random weighting neural network (RWNN) and duel slot machine algorithm.

[0042] Reward correction module: ​​

[0043] The reward correction module corrects the directly observed reward r(x n (s), c) to be an unbiased estimate of the true reward r(x n (s), v) in the expectation sense by importance sampling. Specifically, it obtains the corrected reward r(x

[0044] r mod (x n (s), c) := w · r(x n (s), c)

[0045] where w is the importance sampling weight defined as follows:

[0046]

[0047] where Pr{o = 1 | x n (s)} is the probability of the user's attention on the current song, and Pr{v = 1 | x n (s)} is the relevance probability, i.e., the user's preference for the current song.

[0048] It can be theoretically proved that the corrected reward is an unbiased estimate of the true reward, i.e.,

[0049]

[0050] The attention prediction module:

[0051] The attention prediction module aims to predict Pr{o = 1 | x n (s)}, i.e., the user's attention state on each song. We first store the user's context and state before the bth song in the nth session as follows: u n denotes the user's features in the nth session, denotes the features of the jth song in the nth session, c n，j denotes the user's feedback on the jth song in the nth session. Then the probability of the user's attention on the bth song in the nth session can be represented as p n，b i.e., the estimate of Pr{o = 1 | x n (s)}, and β n is the parameter of the model g.

[0052] We implement the attention prediction module by a multi-layer perceptron (MLP) and two gated recurrent units (GRUs) [8]. Specifically, we model the attention prediction model g as follows:

[0053]

[0054] where represents the operation of concatenating two vectors, σ(·) denotes the element-wise Sigmoid function, and y2(c n，b ) are obtained from two GRUs, respectively.

[0055] a) The input of GRU1 is the representation of the song Output

[0056]

[0057] h n，b and h n，b-1 are the hidden variables of the b-th and b-1-th step, respectively.

[0058] b) The input of GRU2 is the user's feedback c n，b , and the output is y2(c n，b ):

[0059] y2(c n，b ), h' = GRU2(c n，b , h' ) n，b n，b-1

[0060] Relevance prediction module:

[0061] The goal of the relevance prediction module is to predict Pr{v = 1 | x n (s)}. We first encode the context feature x n (s) by a RWNN φ(·) [9, 10] to increase its ability of nonlinear representation, which is defined as follows:

[0062]

[0063] where is a random matrix, where each element is independently sampled from a Gaussian distribution , and is a random vector i.i.d sampled from [-π, π], and cos(·) is the element-wise cosine function. Then, the predicted relevance is where is the parameter of the module f, and σ(·) is the Sigmoid function.

[0064] Offline training phase:

[0065] ​​The parameters of the attention prediction model g are updated after each user session, i.e., offline update of the module after collecting the listening records of a user session and the corresponding user feedback. In the specific training, we use cross-entropy loss and Adam optimizer for end-to-end optimization. We use part of the available labels for training: if the user cuts the song c = 0, we consider the corresponding attention o = 1, if the user continuously plays more than 10 songs without active action (such as cutting songs), the corresponding attention o = 0 from the 11th song until the active action occurs, in addition, the first 5 songs of a session are also considered attention o = 1.

[0066] Online training phase:

[0067] In order to capture the user's preference online, we use the duel tiger to update the relevance prediction module f online. Specifically, at the bth song of the nth user session, given the parameters θ of f n，b , we first select a duel parameter where is a vector randomly sampled from the unit sphere , δ > 0 is a parameter that controls the degree of exploration. In order to evaluate the duel model, we calculate the corresponding interpolated reward where rank θ (s) outputs the corresponding position of s according to the descending order of f(x n (·) | θ) in the candidate set . Then, if the interpolated reward is greater than the corrected reward , that is, the duel model has a better reward, we update the parameters of f(· | θ n，b ) as follows:

[0068]

[0069] where γ > 0 is the step size.

[0070] In order to reduce the variance, we repeat the above update steps K times for each θ n，b , that is, there are K duel models each time.

[0071] Online recommendation phase:

[0072] At the bth step of the nth user session, we select the song with the highest score from the candidate set according to the predicted relevance and recommend it to the user u n :

[0073]

Claims

1.A method for removing attention bias of streaming music recommendation based on reward correction, characterized in that: For the candidate playlists of external systems, a system composed of three modules: reward correction module, attention prediction module, and relevance prediction module is adopted. First, music is recommended to users based on the relevance prediction module, and then the importance sampling method is used to obtain the corrected unbiased reward by combining the predicted user attention and relevance. Based on the three modules, the working process of the offline training stage, the online training stage, and the online recommendation stage is implemented; Specifically, the system inputs the user's biased feedback into the reward correction module and the attention prediction module. The attention prediction module models the probability of user attention to each song, predicts the attention probability through a recurrent neural network, and feeds the attention probability back to the reward correction module. The relevance prediction module predicts the user's preference for each candidate song, predicts the relevance score, and recommends the song that the user is most likely to like based on the relevance score. The updated relevance prediction module provides the next recommendation for the user by using the reward correction module to generate the corrected reward and the duel tiger algorithm. The reward correction module uses an importance sampling method to adjust the directly observed rewards. Make corrections to obtain the actual reward in the expected sense. The unbiased estimate, specifically, obtains the corrected reward through the following formula: Where w is the importance sampling weight, defined as follows: wherein is the probability that the user is paying attention, is the relevance probability, i.e. the user's preference for the current song; The attention prediction module aims to predict the attention state of the user for each song, i.e. the goal is to predict First, the context and state of the user before the bth song in the nth session are stored represents the features of the user in the nth session, represents the features of the jth song in the nth session, represents the feedback of the user for the jth song in the nth session, then the probability of the attention of the user for the bth song in the nth session can be represented as the estimated value of is the parameter of the model g; The attention prediction module is implemented by a multi-layer perceptron and two gated recurrent units. The specific modeling of the attention prediction model g is as follows: wherein ' represents an operation of concatenating two vectors, denotes an element-wise Sigmoid function, are obtained by two gated recurrent units, respectively; wherein the input of the first gated recurrent unit is a representation of the song , and the output : are the latent variables of the bth and b-1th steps, respectively; The input of the second gating recurrent unit is the feedback of the user , and the output is : 。 2. The method of claim 1, wherein the method is a reward correction based method for removing attention bias in streaming music recommendation. The correlation prediction module aims at predicting the user preference for each song through a correlation score, i.e. predicting , first the context features are encoded through a randomly weighted neural network [9, 10] increasing its ability to represent non-linearities, defined as follows: in It is a random matrix where each element is independently distributed from a Gaussian distribution. Generated by sampling from the middle, It is a random vector iid from Obtained by sampling in the middle, It is an element-wise cosine function. Finally, the user's preference score for each song is predicted as follows: ,in These are parameters of module f. It is the Sigmoid function. 3.The method of claim 2, wherein: parameters of the offline training phase attention prediction model g updated after each user session ends; The online training stage uses the duel tiger algorithm to update the relevance prediction module f online. The online recommendation phase selects the song with the highest relevance score from the candidate set according to the predicted relevance score at each recommendation, step b of the nth user session .​

Images