A human-in-the-loop news recommendation method based on inverse reinforcement learning

Abstract

The present invention discloses a human-in-the-loop news recommendation method based on inverse reinforcement learning, belonging to the field of data analysis and mining technology. First, a recommendation strategy is initialized for the recommendation system, and the expectation of the initial strategy is calculated. Experts provide recommendation methods based on user information and news content, and the expectation of the expert recommendation method is calculated based on the recommendation methods provided by the experts. The weight parameters of the optimal reward function are solved according to the constraints and used to update the reward function. Then, the recommendation system is allowed to perform reinforcement learning based on the updated reward function to obtain the optimal strategy under the current reward function, and the expectation of the strategy recommendation method is calculated. The expectation is compared with the expectation of the expert recommendation method. If the difference is greater than a set threshold, the reward function is continuously updated in the above manner. Otherwise, the iteration is stopped to obtain the current optimal reward function, which can be used for reinforcement learning news recommendation.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis and mining, and in particular to a human-in-the-loop news recommendation method based on inverse reinforcement learning. Background Art

[0002] With the increasing popularity of the Internet, the problem of information overload has become increasingly prominent. Simple information retrieval can no longer meet users' needs for personalized search. The application of recommendation systems in daily life is becoming more and more common. Recommendation systems can effectively filter and screen information, helping users to efficiently retrieve information that meets their personalized needs, thereby alleviating the problem of information overload. It has been widely used in e-commerce, social networks, education, music and other fields.

[0003] Early recommendation systems utilized traditional recommendation algorithms. For example, content-based recommendation methods primarily extracted user history and product reviews as user preferences. They also extracted and represented product features, then compared and calculated the degree of match between user preferences and product features to make recommendations. These traditional recommendation algorithms lacked feature extraction methods and tended to overlook typical recommendations. This led to poor performance in real-world applications due to the complex recommendation process.

[0004] Recently, researchers have also attempted to combine deep learning with recommendation technology to overcome some of the obstacles encountered by traditional recommendation technologies, thereby achieving more accurate and efficient recommendations. For example, recommendation methods based on recurrent neural networks use long short-term memory networks and gated recurrent units to calculate user characteristics and the matching degree between users and articles, and to rank recommended news, providing users with more accurate recommendation services with fewer parameters. However, the main problem of these methods is the difficulty in dynamically modeling user and item characteristics. At the same time, the difficulty in implementing recommendations when the data scale is large is also an urgent problem that needs to be solved. Therefore, how to dynamically recommend products to users based on changes in user preferences is a key issue that needs to be solved in the field of recommendation systems. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a human-in-the-loop news recommendation method based on inverse reinforcement learning. In order to dynamically provide users with real-time news recommendations according to changes in user preferences, this method first initializes a recommendation strategy for the recommendation system and calculates the expectation of the initial strategy; experts provide recommendation methods based on user information and news content, and the expectation of the expert recommendation method is calculated based on the recommendation method provided by the experts; the weight parameters of the optimal reward function are solved according to the constraints and used to update the reward function; then the recommendation system is allowed to perform reinforcement learning according to the updated reward function to obtain the optimal strategy under the current reward function, and the expectation of the strategy recommendation method is calculated, and the expectation is compared with the expectation of the expert recommendation method. If the difference is greater than the set threshold, the reward function is continued to be updated in the above manner, otherwise the iteration is stopped to obtain the current optimal reward function, which can be used for reinforcement learning news recommendation.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A human-in-the-loop news recommendation method based on inverse reinforcement learning includes the following steps:

[0008] (1) Randomly initialize a recommendation strategy for the recommendation system and calculate the expectation of the initial strategy, including the following sub-steps:

[0009] (1.1) Randomly initialize a recommendation strategy for the recommendation system. Randomly generate parameters for a reinforcement learning module to randomly initialize an initial recommendation strategy for the recommendation system. The reinforcement learning module is the module used to obtain recommended news.

[0010] (1.2) Input the 10 news articles that the user has recently browsed and the 10 news articles to be recommended to the user into the reinforcement learning module, and output a 10-dimensional vector representing the probability of the 10 news articles being recommended to the user;

[0011] (1.3) The 10 news articles to be recommended are fed into a long short-term memory network to obtain the feature vector of each news article. The feature vector of each of the 10 news articles to be recommended is multiplied and added with the corresponding probability obtained in step (1.2) to obtain the expectation of the randomly initialized recommendation strategy.

[0012] (2) Experts provide recommendations based on user information and news content. The expected recommendation method is calculated based on the recommendations provided by the experts. The specific steps are as follows:

[0013] (2.1) Provide the user information, the 10 most recently viewed news articles, and the 10 news articles to be recommended to the user to the expert, who then makes a recommendation. The recommendation is to rank the 10 recommended news articles from highest to lowest according to the degree of possible interest to the user.

[0014] (2.2) After obtaining the expert's recommendation method, the expert determines the probability of each of the 10 news articles being recommended to the user;

[0015] (2.3) Based on the feature vectors of the 10 news articles to be recommended obtained in step (1.3) and the probability of the news articles to be recommended to the user given by the experts obtained in step (2.2), the feature vectors of each of the 10 news articles to be recommended and the probability of the news articles to be recommended to the user given by the experts are multiplied and added accordingly to calculate the expectation of the expert recommendation method.

[0016] (3) Solving the optimal weight parameters of the reward function based on the constraints and updating the reward function, including the following sub-steps:

[0017] (3.1) Design the constraints for solving the reward function weight parameters. The constraints for solving the reward function weight parameters are to limit the modulus of the weight parameter vector to the range of 0-1, and to solve for the maximum distance direction vector between the expectation of the expert strategy and the optimal strategy expectation obtained by the recommendation system in each previous iteration, as the weight parameter vector;

[0018] (3.2) According to the support vector machine concept, the expert feature expected points are in the positive class, and the expected points obtained by other strategies are in the negative class. Solve for the maximum interval of the support vector. The maximum interval direction vector at this time is the weight parameter vector required. The support vector is the vector composed of the expected points of the two strategies closest to the expected point of the expert strategy, and the maximum interval direction is the normal vector of the support hyperplane. Use the above method to solve and obtain the optimal solution for the weight parameters of the reward function;

[0019] (3.3) Substitute the optimal weight parameters of the reward function obtained in step (3.2) into the reward function, update the reward function, and obtain the updated reward function of the reinforcement learning module.

[0020] (4) Let the recommendation system perform reinforcement learning based on the obtained reward function to obtain the optimal strategy under the current reward function, and calculate the expectation of the recommendation method of the strategy, including the following sub-steps:

[0021] (4.1) The 10 news articles that the user has recently browsed are fed into the long short-term memory network to obtain the features of the 10 news articles as the user's historical features. These features are then fed into the reinforcement learning module along with the features of the 10 news articles to be recommended obtained in step (1.3).

[0022] (4.2) The reinforcement learning module uses reinforcement learning to generate a 10-dimensional vector representing the probability of the 10 news articles being recommended to the user. The reinforcement learning module pushes the news list to the user in descending order based on the recommendation probability of the news articles.

[0023] (4.3) The reinforcement learning module is rewarded by the user's click or non-click on the pushed news list, the proportion of news read length, sharing behavior, and user activity. The reinforcement learning module adjusts its strategy based on the rewards brought by user behavior;

[0024] (4.4) Repeat steps (4.1) to (4.3) until the number of iterations is greater than the set number, end the iteration, and calculate the expectation of the optimal recommendation strategy obtained by the reinforcement learning module.

[0025] (5) Compare the expectation of the optimal strategy obtained by reinforcement learning with the expectation of the expert recommendation method. If the difference is greater than the set threshold, continue to update the reward function in the above manner. Otherwise, stop the iteration and obtain the current optimal reward function. This reward function can be used for reinforcement learning news recommendation, including the following sub-steps:

[0026] (5.1) Calculate the gap between the optimal strategy expectation obtained by the reinforcement learning module and the expert recommendation method expectation. If the gap is greater than the set threshold, repeat steps (3) and (4). If the gap is less than or equal to the set threshold, stop the iteration and fix the reward function weight vector to perform reinforcement learning news recommendation.

[0027] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0028] The present invention is a news recommendation method based on inverse reinforcement learning. Compared with traditional recommendation methods and methods based on deep learning, it can better update strategies in real time according to feedback and improve recommendation effects.

[0029] The present invention is a news recommendation method based on human-in-the-loop, which allows experts to participate in the decision-making process of the recommendation system. Letting the recommendation system learn the recommendation method of experts can help the recommendation system better fit the user's personalized needs for news recommendation. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flow chart of the present invention;

[0031] Figure 2 Schematic diagram of a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described below with reference to specific examples and accompanying drawings.

[0033] Such as process Figure 1 As shown, the human-in-the-loop news recommendation method based on inverse reinforcement learning of the present invention includes the following steps:

[0034] (1) Randomly initialize a recommendation strategy for the recommendation system and calculate the expectation of the initial strategy, including the following sub-steps:

[0035] (1.1) Randomly initialize a recommendation strategy for the recommendation system. Randomly generate the parameters of the reinforcement learning module to randomly initialize an initial recommendation strategy π for the recommendation system. (0) The reinforcement learning module is a module for obtaining recommended news;

[0036] (1.2) Input the 10 news articles that the user has recently browsed and the 10 news articles to be recommended to the user into the reinforcement learning module, and output a 10-dimensional vector representing the probability of the 10 news articles being recommended to the user;

[0037] (1.3) Send the 10 news articles to be recommended into a long short-term memory network to obtain the feature vector of each news article. Multiply and add the feature vector of each of the 10 news articles to be recommended with the corresponding probability obtained in step (1.2) to obtain the expected μ of the randomly initialized recommendation strategy. (0) The long short-term memory network

[0038] (2) Experts provide recommendations based on user information and news content. The expected recommendation method is calculated based on the recommendations provided by the experts. The specific steps are as follows:

[0039] (2.1) Provide the user information, the 10 most recently viewed news articles, and the 10 news articles to be recommended to the user to the expert, who then makes a recommendation. The recommendation is to rank the 10 recommended news articles from highest to lowest according to the degree of possible interest to the user.

[0040] (2.2) After obtaining the expert's recommendation method, the expert determines the probability of each of the 10 news articles being recommended to the user;

[0041] (2.3) Based on the feature vectors of the 10 news articles to be recommended obtained in step (1.3) and the probability of the news articles to be recommended to the user given by the experts obtained in step (2.2), the feature vectors of each of the 10 news articles to be recommended and the probability of the news articles to be recommended to the user given by the experts are multiplied and added together to calculate the expected μ of the expert recommendation method. E .

[0042] (3) Solving the optimal weight parameters of the reward function based on the constraints and updating the reward function, including the following sub-steps:

[0043] (3.1) Design the constraints for solving the reward function weight parameter w. The constraints for solving the reward function weight parameter are determined by limiting the modulus of the weight parameter vector to the range of 0-1, that is, 0≤||w||2≤1. The modulus of the vector is the square root of the sum of the squares of the values of each component of the vector. Solve for the expected μ of the expert strategy E and the optimal strategy expected μ obtained by the recommendation system in each previous iteration (j) ,j∈{0,1,...,i-1}, the maximum interval direction vector between them is used as the weight parameter vector w, where i is the number of iterations of steps 3 and 4;

[0044] (3.2) According to the support vector machine concept, the expert feature expected points are in the positive class, and the expected points obtained by other strategies are in the negative class. Solve for the maximum interval of the support vector. The maximum interval direction vector at this time is the weight parameter vector required. The support vector is the vector composed of the expected points of the two strategies closest to the expected point of the expert strategy, and the maximum interval direction is the normal vector of the support hyperplane. Use the above method to solve and obtain the optimal solution for the weight parameters of the reward function;

[0045] (3.3) Bring the optimal weight parameters of the reward function obtained in step (3.2) into the reward function In , update the reward function to obtain the updated reward function of the reinforcement learning module, where It is a 5-dimensional vector composed of various aspects of the user's response to the recommendation behavior of the recommendation system. Where c represents the number of news items clicked by the user among the 10 news items pushed to the user at one time, nc represents the number of news items not clicked by the user among the 10 news items pushed to the user at one time, s represents the number of news items shared by the user, t represents the ratio of the length of news read by the user to the total length of news, and l represents the user activity, which is the ratio of the number of days the user is online in a month to the total number of days in that month.

[0046] (4) Let the recommendation system perform reinforcement learning based on the obtained reward function to obtain the optimal strategy under the current reward function, and calculate the expectation of the recommendation method of the strategy, including the following sub-steps:

[0047] (4.1) The 10 news articles that the user has recently browsed are fed into the long short-term memory network to obtain the features of the 10 news articles as the user's historical features. These features are then fed into the reinforcement learning module along with the features of the 10 news articles to be recommended obtained in step (1.3).

[0048] (4.2) The reinforcement learning module uses reinforcement learning to generate a 10-dimensional vector representing the probability of the 10 news articles being recommended to the user. The reinforcement learning module pushes the news list to the user in descending order based on the recommendation probability of the news articles.

[0049] (4.3) The reinforcement learning module is rewarded by the user's click or non-click on the pushed news list, the proportion of news read length, sharing behavior, and user activity. The reinforcement learning module adjusts its strategy based on the rewards brought by user behavior;

[0050] (4.4) Repeat steps (4.1) to (4.3) until the number of iterations is greater than the set number, end the iteration, and calculate the expectation of the optimal recommendation strategy obtained by the reinforcement learning module.

[0051] (5) Compare the expectation of the optimal strategy obtained by reinforcement learning with the expectation of the expert recommendation method. If the difference is greater than the set threshold, continue to update the reward function in the above manner. Otherwise, stop the iteration and obtain the current optimal reward function. This reward function can be used for reinforcement learning news recommendation, including the following sub-steps:

[0052] (5.1) Calculate the difference between the expected optimal strategy obtained by the reinforcement learning module and the expected expert recommendation method. If the difference is greater than a set threshold, repeat steps (3) and (4). If the difference is less than or equal to the set threshold, stop the iteration and fix the reward function weight vector to perform reinforcement learning news recommendation. The threshold is a set value.

[0053] Example

[0054] See Figure 2 In this embodiment, a recommendation strategy is first initialized for the recommendation system, and the expectation of the initial strategy is calculated; the expert provides a recommendation method based on the news content browsed by the user and the news content to be recommended, and the expectation of the expert recommendation method is calculated based on the recommendation method provided by the expert; the weight parameters of the optimal reward function are solved according to the constraint conditions and used to update the reward function; then the recommendation system is allowed to perform reinforcement learning based on the updated reward function to obtain the optimal strategy under the current reward function, and the expectation of the recommendation method of the strategy is calculated, and the expectation is compared with the expectation of the expert recommendation method. If the difference is greater than the set threshold, the reward function is continued to be updated in the above manner, otherwise the iteration is stopped to obtain the current optimal reward function, which can be used for reinforcement learning news recommendation.

[0055] The above description is only a preferred embodiment of the present invention, and certain modifications may be made thereto within the scope defined by the claims of the present invention, but all modifications will fall within the scope of protection of the present invention.

Claims

1. A human-in-the-loop news recommendation method based on inverse reinforcement learning, characterized in that: The following steps are involved: (1) Randomly initialize a recommendation strategy for the recommendation system and calculate the expectation of the initial strategy; (2) Experts provide recommendations based on user information and news content, and the expectation of the expert recommendation is calculated based on the recommendations provided by the experts; (3) Solve the optimal weight parameters of the reward function based on the constraints and use them to update the reward function; (4) Let the recommendation system perform reinforcement learning based on the obtained reward function to obtain the optimal strategy under the current reward function, and calculate the expectation of the recommendation method of this strategy; (5) Calculate the gap between the optimal strategy expectation obtained by the reinforcement learning module and the expert recommendation method expectation. If the gap is greater than the set threshold, repeat steps (3) and (4). If the gap is less than or equal to the set threshold, stop the iteration and fix the reward function weight vector to perform reinforcement learning news recommendation. The step (1) is specifically as follows: (1.1) Randomly initializing a recommendation strategy for the recommendation system; randomly initializing an initial recommendation strategy for the recommendation system by randomly generating parameters of a reinforcement learning module; the reinforcement learning module is the module used to obtain recommended news; (1.2) Input the 10 most recently viewed news articles and the 10 news articles to be recommended to the user into the reinforcement learning module, and output a 10-dimensional vector representing the probability of the 10 news articles being recommended to the user; (1.3) The 10 news articles to be recommended are fed into a long short-term memory network to obtain the feature vector of each news article. The feature vector of each of the 10 news articles to be recommended is multiplied and added with the corresponding probability obtained in step (1.2) to obtain the expectation of the randomly initialized recommendation strategy.

2. The human-in-the-loop news recommendation method based on inverse reinforcement learning according to claim 1, characterized in that: The step (2) is specifically as follows: (2.1) Provide the user's information, the 10 most recently viewed news articles, and the 10 news articles to be recommended to the user to the expert. The expert will then provide a recommendation method, which is to rank the 10 recommended news articles from most to least likely to be of interest to the user. (2.2) After obtaining the expert's recommendation method, the expert determines the probability of each of the 10 news articles being recommended to the user; (2.3) Based on the feature vectors of the 10 news articles to be recommended obtained in step (1.3) and the probability of the news articles to be recommended given by the experts obtained in step (2.2), the feature vectors of each of the 10 news articles to be recommended and the probability of the news articles to be recommended given by the experts are multiplied and added accordingly to calculate the expectation of the expert recommendation method.

3. The human-in-the-loop news recommendation method based on inverse reinforcement learning according to claim 1, characterized in that: The step (3) is specifically as follows: (3.1) Establish constraints for solving the reward function weight parameters. The constraints for solving the reward function weight parameters are to limit the modulus of the weight parameter vector to the range of 0-1, and to solve for the maximum distance direction vector between the expectation of the expert strategy and the optimal strategy expectation obtained by the recommendation system in each previous iteration, which is used as the weight parameter vector. (3.2) Based on the support vector machine concept, the expert feature expected points are in the positive class, while the expected points obtained by other strategies are in the negative class. The maximum margin of the support vector is solved, and the direction vector of the maximum margin is the required weight parameter vector. The support vector is the vector composed of the expected points of the two strategies closest to the expected point of the expert strategy, and the maximum margin direction is the normal vector of the support hyperplane. (3.3) Use the method described in step (3.2) to find the optimal solution for the reward function weight parameters, substitute the obtained optimal reward function weight parameters into the reward function, update the reward function, and obtain the updated reward function of the reinforcement learning module.

4. The human-in-the-loop news recommendation method based on inverse reinforcement learning according to claim 1, characterized in that: The step (4) is specifically as follows: (4.1) The 10 news articles that the user has recently browsed are fed into the long short-term memory network to obtain the features of the 10 news articles as the user's historical features. These features are then fed into the reinforcement learning module along with the features of the 10 news articles to be recommended obtained in step (1.3). (4.2) The reinforcement learning module obtains a 10-dimensional vector through reinforcement learning method, which represents the probability of the 10 news articles to be recommended to the user; The reinforcement learning module pushes the news list to the user in descending order based on the probability of recommending the news; (4.3) The reinforcement learning module is rewarded by users' clicks or non-clicks on the pushed news list, the proportion of news read, sharing behavior, and user activity. The reinforcement learning module adjusts its strategy based on the rewards generated by user behavior. (4.4) Repeat steps (4.1) to (4.3) until the number of iterations is greater than the set number, end the iteration, and calculate the expectation of the optimal recommendation strategy obtained by the reinforcement learning module.

Images