Recommendation method based on building a computationally efficient deep reinforcement learning recommendation system

Abstract

The present invention provides a recommendation method based on building a deep reinforcement learning recommendation system with high computational efficiency, comprising the following steps: step S1, modeling the recommendation system using a Markov decision process; step S2, establishing a user rating matrix with U rows and M columns based on the historical records of all users, and obtaining a user feature matrix X and an item feature matrix Y representing the state of the interaction between the user and the recommendation system through matrix decomposition; step S3, modeling and learning and training the recommendation strategy of the recommendation system, wherein the recommendation strategy is characterized by a neural network, and the neural network output layer uses a sparse matrix for parameter modeling; step S4, deploying the recommendation system and using the trained recommendation strategy to recommend items based on the user's historical data. By improving the output layer of the neural network in deep reinforcement learning, the present invention reduces computational complexity, improves learning efficiency, and helps solve the problem of learning in a huge discrete action space.

Description

Technical Field

[0001] The present invention relates to a recommendation method based on building a deep reinforcement learning recommendation system with high computational efficiency, which belongs to the field of big data artificial intelligence and is particularly suitable for application scenarios with a huge number of recommendation items. Background Art

[0002] Recommender systems play a vital role in online life. Their primary purpose is to recommend items that users may need based on their historical data. Traditional recommendation system approaches include collaborative filtering, matrix factorization, and content-based ranking. However, these algorithms assume that user preferences remain constant throughout the entire process, and their recommendations often fail to satisfy users. Recent research has shown that modeling recommendation systems as a reinforcement learning problem, with maximizing the total possible future ratings given by users as the optimization objective, can effectively dynamically model user preferences and improve the performance of recommendation systems.

[0003] However, in real-world applications, recommendation systems often require a vast number of recommended items. For example, movie recommendation systems can potentially offer over a million movies, while online product recommendation systems often have hundreds of millions of items. When using reinforcement learning methods to address this massive decision space, modeling user historical behavior is a key factor in improving recommendation performance. Furthermore, this vast decision space can lead to low learning and decision-making efficiency, hindering the practical application of reinforcement learning methods. Summary of the Invention

[0004] The purpose of this invention is to provide a recommendation method based on building a deep reinforcement learning recommendation system with high computational efficiency. By massively reducing the parameters of the neural network in deep reinforcement learning, the decision-making time in a huge discrete space is reduced, and the efficiency of learning and decision-making is improved.

[0005] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0006] A recommendation method based on building a deep reinforcement learning recommendation system with high computational efficiency, characterized by comprising the following steps:

[0007] Step S1, using Markov decision process to model the recommendation system;

[0008] Step S2: Based on the time series data of all users' historical records, a user rating matrix with U rows and M columns is established, and the user feature matrix X and the item feature matrix Y representing the state of the user's interaction with the recommendation system are obtained through matrix decomposition;

[0009] Step S3, modeling and learning the recommendation strategy of the recommendation system, wherein the recommendation strategy is represented by a neural network, and the neural network output layer is parameterized by a sparse matrix;

[0010] In step S4, the trained recommendation strategy is used to obtain the user behavior representation based on the time series data of the user's historical browsing records (select the time series data before the current selection, such as which items have been viewed in the past period of time) using the state construction method described in S2. The behavior representation is input into the trained recommendation strategy neural network, the output is calculated, and item recommendations are made based on the output.

[0011] On the basis of adopting the above technical solutions, the present invention may also adopt the following further technical solutions, or use these further technical solutions in combination:

[0012] The Markov decision process used in step S1 is<S,A,P,R> , where S is the state set, i.e., the feature representation set of the interaction between the user and the recommendation system, A is the set of items that the user can select, and the item selected by the user at time t is recorded as a t , P(s t+1 |s t ,a t ) is the state transfer function, that is, the current state s t Next, the user selects item a t The next moment state s t+1 The probability, R(s t+1 |s t ,a t ) is the reward function, that is, the user's response to the current state s t Select Project a t For the recommendation system, the goal is to maximize the total expected rating of users received in the future, that is,

[0013]

[0014] Among them, T is the end time.

[0015] Step S2 includes a user rating matrix, where U is the total number of users, M is the total number of recommended items, and the i-th row and j-th column of the rating matrix represent the rating of the i-th user on the j-th item. If the user has not rated the item, the rating is set to 0.

[0016] Step S2 uses the matrix decomposition method of Funk's singular value matrix decomposition (Funk's SVD), focusing only on the data evaluated by the user and ignoring the data not evaluated by the user, and decomposing the user rating matrix into two matrices: the user feature matrix X and the item feature matrix Y. The X matrix has U rows and V columns, and the Y matrix has M rows and V columns. Each row of the X matrix is used as the feature vector of each user, and each row of the Y matrix is used as the feature vector of each item.

[0017] In step S2, a bidirectional recurrent neural network is used to model the user's historical behavior. The outputs of the forward and reverse recurrent neural networks are integrated with the user's own features in step S2 as the state input of the recommendation system.

[0018] In step S3, the input layer of the neural network is preceded by several fully connected layers to model the recommendation strategy.

[0019] Step S3 uses the Monte Carlo policy gradient algorithm (REINFORCE) to collect interaction data between users and the recommendation system, and learns the recommendation strategy based on the interaction data between users and the recommendation system and the gradient descent theorem.

[0020] In step S3, the recommendation strategy neural network receives the state provided in step S2 as input, and the output layer outputs the recommendation probability of each item.

[0021] In step S3, the sparse matrix is constructed by sorting and classifying each item in the recommendation system according to the user selection frequency of all items, and allocating more training parameters to low-frequency items and fewer training parameters to high-frequency items.

[0022] The item recommendation probability output by the recommendation strategy in step S3 is the output of the softmax network using the above sparse matrix as a parameter.

[0023] The present invention reduces computational complexity and improves learning efficiency by improving the output layer of the neural network in deep reinforcement learning, helping to solve the problem of learning in a huge discrete action space. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a schematic diagram of the user state modeling method adopted by the present invention;

[0025] Figure 2 Schematic diagram of the output layer parameter matrix of the strategy adopted by the present invention; DETAILED DESCRIPTION

[0026] In order to make the technical solution of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0027] The MovieLens dataset used in this application example is a user-rated movie recommendation dataset, collected by the Group-Lens research team at the University of Minnesota. It is a commonly used dataset in recommender system research. The 1MB MovieLens dataset contains over one million ratings from approximately 6,040 users for approximately 3,900 movies. Each user has watched and rated at least 20 movies. Ratings are based on a 5-point scale, with higher user ratings indicating a greater preference for the movie.

[0028] Step S1: Based on this dataset, this application uses the recommendation system as an intelligent agent for reinforcement learning and the user as the environment for reinforcement learning. At each moment, the recommendation system recommends a movie project based on the user's historical behavior and optimizes the performance of the recommendation system based on the user's rating feedback. This is represented by a mathematical model, namely, the Markov decision process quadruple<S,A,P,R> Represented as follows: where S is the state set, i.e., the feature representation set of the interaction between the user and the recommendation system. In the MovieLens dataset, it is the user's historical movie viewing records and their corresponding ratings. A is the set of items that the user can select, i.e., the total set of movie items in the movie recommendation system. The movie item selected by the user at time t is denoted as a. t , P(s t+1 |s t ,a t ) is the state transfer function, that is, the current state s t Next, the user selects item a t The next moment state s t+1 The probability, R(s t+1 |s t ,a t ) is the reward function, that is, the user's response to the current state s t Select Project a t For the recommendation system, the goal is to maximize the total expected rating of users received in the future, that is,

[0029]

[0030] Among them, T is the end time.

[0031] Step S2: Based on the historical records of all users, a user rating matrix with U rows and M columns is established, where U is the total number of users and M is the total number of recommended items. The i-th row and j-th column of the rating matrix represent the rating of the i-th user on the j-th item. If the user has not rated the item, the rating is set to 0. Using the matrix decomposition method of Funk's singular value matrix decomposition (Funk's SVD), only focus on the data that the user has rated and ignore the movie data that the user has not rated. The user rating matrix is decomposed into two matrices: the user feature matrix X and the item feature matrix Y. The X matrix has U rows and V columns, and the Y matrix has M rows and V columns, where V is a hyperparameter selected by the user. The i-th row of the X matrix is used as the eigenvector of the i-th user, and the j-th row of the Y matrix is used as the eigenvector of the j-th item.

[0032] Use attached Figure 1 The state representation method in

[15] selects the movie viewing records of the N users preceding the current time t, obtains the feature vector and corresponding rating of each movie, and converts the rating into a one-hot encoding. Each movie vector is concatenated with its rating and input into a bidirectional recurrent neural network with a GRU kernel. After processing by a convolutional neural network, a forward state and a reverse state are obtained. The user's feature vector is concatenated with these two states to form the user's behavioral feature vector.

[0033] Step S3: A neural network is used to characterize the recommendation strategy of the recommendation system. The user behavior vector is input into the policy neural network. The output layer of the policy neural network outputs the recommendation probability of each item. The output layer uses a sparse matrix for parameter modeling, and several fully connected layers precede the input layer to model the recommendation strategy. Before constructing the policy neural network, the number of user selections for all movie items in the recommendation system is counted. The number of occurrences of each item is ranked from highest to lowest, and the items are divided into several categories based on the number of occurrences. A sparse matrix is constructed by assigning more training parameters to low-frequency items and fewer training parameters to high-frequency items. The item recommendation probabilities output by the recommendation strategy are the outputs of a softmax network using the sparse matrix as parameters. Assume that the parameter matrix output by the last layer is h×M dimensional, where h is the dimension of the output of the previous layer and M is the total number of items. In recommendation systems, since M is often a large number of items, the output layer will have a large number of dimensional parameters. When the output layer needs to use softmax to output the recommendation probability of each item, it consumes a large number of parameters, resulting in increased computational efficiency. Therefore, this method adopts the method of sparsely ... Figure 2As shown in the figure, the items are divided into 4 categories according to the number of occurrences of the items. For each category, the hyperparameter d is selected. A ,d B ,d C ,d D , as its proportion in the parameter matrix. Among them, these hyperparameters satisfy the relationship:

[0034] d A +d B +d C +d D =M

[0035] According to the size of its output layer, the four parameter matrix dimensions can be obtained according to the similarity transformation:

[0036] A matrix:

[0037] B matrix:

[0038] C matrix:

[0039] D matrix:

[0040] Next, press Figure 2 The matrix arrangement method in [1] arranges the four matrices A, B, C, and D into a diagonal matrix. The parameter matrix is zero except for the four matrices A, B, C, and D. This creates a sparse matrix, significantly reducing the number of parameters in the output layer and hidden layer, improving computational efficiency. It should be noted that in actual applications, the number of categories and parameters can be adjusted as needed.

[0041] After building the policy network, we use the Monte Carlo policy gradient algorithm (REINFORCE) to collect interaction data between users and the recommendation system and learn the recommendation policy based on the gradient descent theorem. Assuming the recommendation policy is π, the parameters that the policy network needs to learn include those from user behavior characteristics and those in the policy network, which are set to θ. By periodically collecting historical records of users' interactions with the recommendation system, we can use the policy gradient theorem to obtain the gradient of the loss function J with respect to the parameter θ:

[0042]

[0043] in, is the future reward received by performing action a in state s.

[0044] By setting the learning rate α and the discount factor γ, the parameter θ can be learned and optimized through stochastic gradient ascent. The specific optimization algorithm can be referred to the following table:

[0045]

[0046] Step 4: Based on the input user's historical viewing records, follow the learned strategy π θ Recommend corresponding film projects.

[0047] The experimental results of this invention are shown in the Amazon Instant Video dataset ( https: / / jmcauley.ucsd.edu / data / amazon / We conducted experiments on a subset of data from a dataset containing 122,609 users rating 8,229 short videos, totaling 145,983 ratings. We used 80% of the data as the training set and 20% as the test set.

[0048] The experimental results of our method were compared with those of a fully connected neural network recommendation method. The evaluation criteria were the average hit rate (Hit rate@30), average reciprocal rank (Mean Reciprocal Rank@30), average learning time per step (in seconds), and average decision time per step (in seconds) of the top 30 recommended items. The experimental results are shown in Table 1. It can be seen that our method can shorten learning and decision-making time while achieving similar results, thereby improving efficiency.

[0049] Table 1 Comparison of the effects of this method with other methods

[0050] Evaluation Criteria Using a fully connected neural network Sparse matrix using this method Average hit rate @30 0.105 0.102 Average last rank @30 0.0292 0.0289 Average learning time per step (seconds) 0.03239 0.03162 Average decision time (seconds) 0.00907 0.00852

[0051] The above description is only a preferred embodiment of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements without departing from the concept of the present invention. These improvements should also be considered within the scope of protection of the present invention.

Claims

1. A recommendation method based on building a deep reinforcement learning recommendation system with high computational efficiency, characterized by The following steps are involved: Step S1, using Markov decision process to model the recommendation system; Step S2: Based on the time series data of all users' historical records, a user rating matrix with U rows and M columns is established, and the user feature matrix X and the item feature matrix Y representing the state of the user's interaction with the recommendation system are obtained through matrix decomposition; Step S3, modeling and learning the recommendation strategy of the recommendation system, wherein the recommendation strategy is represented by a neural network, and the neural network output layer is parameterized by a sparse matrix; Step S4: Deploy the recommendation system and use the trained recommendation strategy to recommend items based on the user's current browsing time series data; In step S3, the sparse matrix is constructed by sorting and classifying each item in the recommendation system according to the user selection frequency of all items, and allocating more training parameters to low-frequency items and fewer training parameters to high-frequency items.

2. The method according to claim 1, characterized in that The Markov decision process used in step S1 is<S,A,P,R> , where S is the state set, A is the set of items that the user can select, and the item selected by the user at time t is recorded as a t , P(s t+1 |s t ,a t ) is the state transfer function, R(s t+1 |s t ,a t ) is the reward function.

3. The method according to claim 1, characterized in that Step S2 includes a user rating matrix, where U is the total number of users, M is the total number of recommended items, and the i-th row and j-th column of the user rating matrix represent the rating of the i-th user on the j-th item. If the user has not rated the item, the rating is set to 0.

4. The method according to claim 1, wherein The user feature matrix X and the project feature matrix Y included in step S2 are decomposed by the matrix decomposition method of Funk singular value matrix decomposition, wherein the user feature matrix X has U rows and V columns, and the project feature matrix Y has M rows and V columns. Each row of the user feature matrix X is used as the feature vector of each user, and each row of the project feature matrix Y is used as the feature vector of each project.

5. The method according to claim 1, 3 or 4, characterized in that In step S2, a bidirectional recurrent neural network is used to model the user's historical behavior. The outputs of the forward and reverse recurrent neural networks are integrated with the user's own features in step S2 as the state input of the recommendation system.

6. The method according to claim 1, characterized in that In step S3, the input layer of the neural network is preceded by several fully connected layers to model the recommendation strategy.

7. The method according to claim 1, characterized in that Step S3 uses the Monte Carlo policy gradient algorithm to learn the recommendation strategy based on the interaction data between the user and the recommendation system and the gradient descent theorem.

8. The method according to claim 1, characterized in that In step S3, the recommendation strategy neural network receives the state provided in step S2 as input, and the output layer outputs the recommendation probability of each item.

9. The method according to claim 1, characterized in that The item recommendation probability output by the recommendation strategy in step S3 is the output of the softmax network using the above sparse matrix as a parameter.

Images