A deep reinforcement learning recommendation method integrating user preference prediction

Through the deep reinforcement learning recommendation method, combining user portrait clustering and heterogeneous graph network, user preferences and context knowledge are integrated, the user preference uncertainty and data sparse problems in the recommendation system are solved, and the user experience and accuracy of the recommendation system are improved .

CN114021024BActive Publication Date: 2024-12-06CHONGQING INST OF GREEN & INTELLIGENT TECH CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111519219.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-09
Publication Date
2024-12-06
Estimated Expiration
2041-12-09

AI Technical Summary

Technical Problem

The existing recommendation system has data sparse problems when dealing with user preference uncertainty, resulting in incomplete characterization of user preferences and directly increasing the burden through exploratory recommendations, which cannot effectively eliminate short-term impact.

Method used

Using a collaborative filtering recommendation method based on user portrait clustering, combined with deep learning and reinforcement learning, the user's current status is generated by establishing a long-term and short-term preference model, a heterogeneous graph network and attention mechanism. Status characterization of moments, optimize the recommendation system.

Benefits of technology

Effectively reduce the impact of uncertainty in user preferences on the recommendation system, improve user experience, and enhance the comfort and accuracy of the recommendation system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114021024B_ABST
    Figure CN114021024B_ABST
Patent Text Reader

Abstract

The present invention is a deep reinforcement learning recommendation method integrating user preference prediction, which belongs to the field of big data artificial intelligence deep learning. The method comprises the following steps: S1: inputting historical interaction data of user behavior and project, and pre-processing the data; S2: establishing a user's long-term and short-term preference model based on user behavior, and obtaining historical user preferences; S3: establishing a user's future preference prediction model, and predicting future user preferences at the next moment; S4: establishing contextual knowledge representation between users, projects, and time; S5: using the attention mechanism to fuse historical user preferences, future user preferences, and user contextual knowledge representation to obtain the state representation of the user at the current moment; S6: building a recommendation system based on a reinforcement learning network; S7: using the trained recommendation system to recommend projects to users. The method of the present invention can enhance the representation ability of uncertain user preferences and improve the user experience of the recommendation system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a deep reinforcement learning recommendation method integrating user preference prediction, which belongs to the field of big data artificial intelligence, and is particularly suitable for deep reinforcement learning recommendation integrating user preference prediction. Background Art

[0002] With the rapid development of the Internet and automatic technology, more and more people now have smart phones, tablets and other smart terminals, which has led to an explosive growth in data and information in production and life, which has led to the problem of information overload. When users search for information of interest, they will spend a lot of time and energy to filter out useless information, but the results are often not satisfactory to users, so personalized recommendation technology is born. Personalized recommendation technology refers to recommending content of interest to users based on certain interests and purchasing characteristics of users, which is an effective way to solve the problem of information overload.

[0003] In the field of sequential recommendation systems, such as e-commerce or online service platforms, users and recommendation systems often interact closely and continuously. In this process, user preferences usually drift dynamically with different recommendation results. This leads to uncertainty in user preferences. In order to effectively reduce the impact of user preference uncertainty on the recommendation system, the ideal method is to design and train a specific supervision model on the premise of fully obtaining user historical data, so as to obtain the user's preference for different items. However, in many real applications, user historical behavior often follows a specific long-tail distribution, that is, most users only generate limited historical data, while very few users generate relatively sufficient historical data. The data sparsity problem caused by this phenomenon leads to the existing model can only characterize part of the user's preferences. In addition, a direct response method is to actively explore user behavior, that is, by initiating a large number of trial recommendations to users to fully obtain their behavior data, so as to ensure the availability of the recommendation system. Unfortunately, this simple approach is bound to cause a huge exploration overhead, making it not feasible in reality, especially it cannot effectively eliminate the short-term impact of user preference uncertainty on the system. Summary of the invention

[0004] In view of this, the present invention provides a deep reinforcement learning recommendation method integrating user preference prediction, aiming to effectively reduce the short-term impact of user preference uncertainty on the system and improve the comfort of the recommendation system.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] The collaborative filtering recommendation method based on user portrait clustering includes the following steps:

[0007] S1: Input user behavior and project historical interaction data and pre-process the data;

[0008] S2: Based on user behavior, establish a user's long-term and short-term preference model to obtain historical user preferences;

[0009] S3: Use deep learning methods to establish a user future preference prediction model to predict the future user preferences at the next moment;

[0010] S4: Establish a heterogeneous graph network to build contextual knowledge representation between users, items, and time;

[0011] S5: Use the attention mechanism to fuse historical user preferences, future user preferences, and user contextual knowledge representations to obtain the user's current state representation;

[0012] S6: Build a recommendation system based on reinforcement learning network, use the current state representation of all users as the state space of the reinforcement learning recommendation system, and use historical interaction data to train the network parameters;

[0013] S7: Use the trained recommendation system to recommend items to users.

[0014] Furthermore, the user set is U = [u1, u2, ..., u m ], the set of items is I = [i1, i2, ..., i n ], the user behavior is an n×m×k dimensional matrix that counts the interaction between users and projects, T = [T1, T2, …, T k ] is the time dimension.

[0015] Furthermore, the recommendation system of the reinforcement learning network consists of a system including five elements: , where R is the reward function, is the state transition probability, γ is the discount factor; the state representation of the user at the current moment is abstracted into the state space S; the set of all items I is regarded as the action space A; the reinforcement learning network architecture is as follows Figure 1 shown.

[0016] Furthermore, the data preprocessing described in step S1 includes: (1) deleting duplicate data and missing or erroneous data; (2) statistically summarizing users, projects and user behaviors; and (3) eliminating data that has too little user behavior data and will cause a cold start.

[0017] Furthermore, the user behaviors P of all users need to be based on a custom time window T = [T1, T2, ..., T k ] is divided into k units, The time windows are time periods of equal length.

[0018] Preferably, the customized time is usually selected as one day, one week or one month, and it is necessary to ensure that there is historical interaction-related data in each time group, and to take into account as much as possible that the number of units k is sufficient and the historical interaction-related data in the unit is sufficient.

[0019] Further, the step S2 is specifically as follows:

[0020] (1) Use the automatic encoder to transform user u j Encode and get the user number

[0021] (2) Use the automatic encoder to transform user u j Historical user behavior The time of user u is encoded separately to obtain j Time number

[0022] (3) User u j Historical user behavior and user u j Time number The elements of are input into the bidirectional long short-term memory recurrent network (Bi-directional LSTM RNN) to extract the k time period user u j characteristic information;

[0023] (4) The user u in the first k-1 time periods is extracted j Characteristic information and user ID Use the Multi-head Attention Network to fuse and get the user u j Past user preferences

[0024] (5) Use the multi-head attention mechanism again to focus on user u in the kth time period j Feature information and past user preferences Fusion is performed to obtain user u j Historical user preferences

[0025] (6) Repeat the above steps (1) to (5) to traverse all users to obtain the historical user preferences of all users Among them, 1≤j≤m.

[0026] Furthermore, the step S3 is specifically as follows: using a deep learning method to establish a user future preference prediction model, respectively j Historical user behavior As input, predict the user u at the next moment j Future user preferences Traverse all users and get the future user preferences of all users Among them, 1≤j≤m.

[0027] Further, the step S4 is specifically as follows:

[0028] (1) Use the autoencoder to convert item i j Encode and get the project number Among them, 1≤j≤n;

[0029] (2) Use the automatic encoder to transform user u j Encode and get the user number Among them, 1≤j≤m;

[0030] (3) Use the autoencoder to convert the time window T j Encode and get the time number

[0031] (4) Use a multi-layer perceptron to transform the time window T j The user IDs corresponding to the user set that has performed the user behavior are perceptually compressed to obtain user aggregation features;

[0032] (5) Use a multi-layer perceptron to transform the time window T j In the above method, the item number of the item set where the user behavior occurs is perceptually compressed to obtain the item aggregation feature;

[0033] (6) Use the graph convolutional neural network to concatenate user aggregation features, project aggregation features, and time numbers to obtain the time window T j Contextual knowledge representation

[0034] (7) Traverse all time windows to obtain contextual knowledge representation

[0035] Furthermore, the attention mechanism described in step S5 is used to fuse the multi-source features whose weights are adaptively adjusted, and the expression representing the state of the user at the current moment is: Among them, the unknown coefficients α1, α2, and α3 are obtained by learning historical interactive correlation data through neural networks.

[0036] The beneficial effects of the present invention are: the present invention provides a deep reinforcement learning recommendation method integrating user preference prediction, proposes a contextual knowledge representation method based on heterogeneous graphs, effectively characterizes the impact of context on user behavior, proposes a user preference representation method integrating historical user preferences, future user preferences and user contextual knowledge representation, enhances the representation capability of uncertain user preferences, and improves the user experience of the recommendation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to make the purpose and technical solution of the present invention, the present invention provides the following drawings for explanation:

[0038] Figure 1 Iterative diagram of the recommendation system based on reinforcement learning network;

[0039] Figure 2 is a flow chart of the present invention;

[0040] Figure 3 This is a diagram of the user's long-term and short-term preference model in Example 1 of the present invention;

[0041] Figure 4 This is a user future preference prediction model diagram of Example 1 of the present invention;

[0042] Figure 5 This is a flow chart of a contextual knowledge characterization method for a heterogeneous graph according to Embodiment 1 of the present invention;

[0043] Figure 6 This is a fusion graph representing the user's current state based on the attention mechanism in Example 1 of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose and technical solution of the present invention more clearly understood, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0045] Example: Movie-Lens is a movie recommendation system based on ratings, created by the Group-Lens research team at the University of Minnesota, USA. It includes three data sets of 100KB, 1MB, and 10MB, which are specifically used to study recommendation technology. For the 1MB user movie data set of Movie-Lens, which includes 1,000,209 rating records of 3,900 movies by 6,040 users, and each user rates at least 20 movies, with a rating range of 1-5 points. The higher the user's rating of a movie, the more the user likes the movie. The sparsity of the user-item rating matrix constructed by this data set is 95.83%. This example provides "a deep reinforcement learning recommendation method integrating user preference prediction". In order to better demonstrate and verify the performance of the method of the present invention, the user movie data set will be split in a ratio of 8:2 for cross-validation, that is, 80% of the user movie data is used as a training set and 20% as a test set.

[0046] Among them, the set of users is U = [u1, u2, ..., u 6040 ], the set of items is I = [i1, i2, ..., i 3900 ], the user behavior is an n×m×k dimensional matrix that counts the interaction between users and projects, and the time window is divided into 30 units, T = [T1, T2, ..., T 30 ] is the time dimension.

[0047] Furthermore, the recommendation system of the reinforcement learning network consists of a system including five elements: , where R is the reward function, is the state transition probability, γ is the discount factor; the state representation of the user at the current moment is abstracted into the state space S; the set of all items I is regarded as the action space A; the reinforcement learning network architecture is as follows Figure 1 shown.

[0048] Combination Figure 2 , the method comprises the following steps:

[0049] Step 1: Combine Figure 3 , based on user behavior, establish a user's long-term and short-term preference model to obtain historical user preferences. Specifically:

[0050] (1) Use the One-Hot encoder to encode user u j Encode and get the user number Among them, 1≤j≤6040;

[0051] User jUse One-Hot coding to encode its information - 2 dimensions by gender, 7 dimensions by age, 21 dimensions by occupation, and 99 dimensions by geographic location, which is a 129-dimensional vector. Since each category is independent of each other, we use One-Hot coding to encode this information. Among them, One-Hot coding, also known as one-bit effective coding, mainly uses N-bit state registers to encode N states. Each state has its own independent register bit, and only one bit is valid at any time. One-Hot coding first requires that the classification value be mapped to an integer value, and then each integer value is represented as a binary vector. Except for the integer index, all others are zero values, which are marked as 1. For example: the One-Hot coding value of male gender is [1, 0], and the coding value of female gender is [0, 1].

[0052] (2) Use the One-Hot encoder to convert user u j The time when historical user behaviors occurred is encoded respectively to obtain user u j Time number

[0053] (3) User u j Historical user behavior and user u j Time number The elements of are input into the bidirectional long short-term memory recurrent network (Bi-directional LSTM RNN) to extract 30 time periods of user u j feature information.

[0054] (4) The user u in the first 29 time periods is extracted j Characteristic information and user ID Use the Multi-head Attention Network to fuse and get the user u j Past user preferences

[0055] (5) Use the multi-head attention mechanism again to focus on user u in the 30th time period j Feature information and past user preferences Fusion is performed to obtain user u j Historical user preferences

[0056] (6) Repeat the above steps (1) to (5) to traverse all users to obtain the historical user preferences of all users

[0057] User Behavior uaFor scoring behavior, due to r ua is a vector of dimension 3900.

[0058] Step 2: Combine Figure 4 , using deep learning methods to establish a user future preference prediction model to predict the future user preferences at the next moment. Specifically:

[0059] The deep learning method is used to establish a user future preference prediction model, and the user u j Historical user behavior As input, predict the user u at the next moment j Future user preferences Traverse all users and get the future user preferences of all users Among them, 1≤j≤6040.

[0060] Step 3: Combine Figure 5 , establish a heterogeneous graph network and establish contextual knowledge representation between users, projects, and time. Specifically:

[0061] (1) Use the One-Hot encoder to convert item i j Encode and get the project number Among them, 1≤j≤3900;

[0062] (2) Use the One-Hot encoder to convert user u j Encode and get the user number Among them, 1≤j≤6040;

[0063] (3) Use the One-Hot encoder to convert the time window T j Encode and get the time number

[0064] (4) Use a multi-layer perceptron to transform the time window T j The user IDs corresponding to the user set that has performed the user behavior are perceptually compressed to obtain user aggregation features;

[0065] (5) Use a multi-layer perceptron to transform the time window T j In the above method, the item number of the item set where the user behavior occurs is perceptually compressed to obtain the item aggregation feature;

[0066] (6) Use the graph convolutional neural network to concatenate user aggregation features, project aggregation features, and time numbers to obtain the time window T j Contextual knowledge representation

[0067] (7) Traverse all time windows to obtain contextual knowledge representation

[0068] Step 4: Combine Figure 6 , the attention mechanism is used to fuse historical user preferences, future user preferences and user contextual knowledge representation to obtain the user's current state representation.

[0069] The attention mechanism is used to fuse the multi-source features with adaptive weight adjustment. The expression representing the user's current state is: Among them, the unknown coefficients α1, α2, and α3 are obtained by learning historical interactive correlation data through neural networks.

[0070] Step 5: Build a recommendation system based on reinforcement learning network, use the current state representation of all users as the state space of the reinforcement learning recommendation system, use the reinforcement learning recommendation system to calculate the state in the next time window, and use the training set and test set to train and test it.

[0071] Step 6: Use the trained recommendation system to recommend items to users, and make Top-N recommendations to users in the corresponding time window by sorting the state transition probabilities of the movies.

[0072] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A deep reinforcement learning recommendation method integrating user preference prediction, characterized in that: The method comprises the following steps: S1: Input user behavior and project historical interaction data and pre-process the data; S2: Based on user behavior, establish a user's long-term and short-term preference model to obtain historical user preferences; S3: Use deep learning methods to establish a user future preference prediction model to predict the future user preferences at the next moment; S4: Establish a heterogeneous graph network to build contextual knowledge representation between users, items, and time; S5: Use the attention mechanism to fuse historical user preferences, future user preferences, and user contextual knowledge representations to obtain the user's current state representation; S6: Build a recommendation system based on reinforcement learning network, use the current state representation of all users as the state space of the reinforcement learning recommendation system, and use historical interaction data to train the network parameters; S7: Use the trained recommendation system to recommend items to users; The user set is U = [u1, u2, ..., u m ], the set of items is I = [i1, i2, ..., i n ], the user behavior is an n×m×k dimensional matrix that counts the interaction between users and projects, T = [T1, T2, …, T k ] is the time dimension.

2. According to claim 1, a deep reinforcement learning recommendation method integrating user preference prediction is characterized in that: The reinforcement learning network recommendation system consists of five elements: , where R is the reward function, is the state transition probability, γ is the discount factor; the state representation of the user at the current moment is abstracted into the state space S; the set of all items I is regarded as the action space A.

3. The deep reinforcement learning recommendation method integrating user preference prediction according to claim 1, characterized in that: The data preprocessing described in step S1 includes: (1) deleting duplicate data and missing or erroneous data; (2) summarizing users, projects and user behaviors; and (3) eliminating data that has too little user behavior data and will cause a cold start.

4. The deep reinforcement learning recommendation method integrating user preference prediction according to claim 1, characterized in that: The user behaviors P of all users need to be based on the customized time window T = [T1, T2, ..., T k ] is divided into k units, The time windows are time periods of equal length.

5. The deep reinforcement learning recommendation method integrating user preference prediction according to claim 1, characterized in that: The step S2 is specifically as follows: (1) using an automatic encoder to convert user u j Encode and get the user number (2) Use the automatic encoder to transform user u j Historical user behavior The time of user u is encoded separately to obtain j Time number (3) User u j Historical user behavior and user u j Time number The elements of are input into the bidirectional long short-term memory recurrent network to extract the k time period user u j characteristic information; (4) The user u extracted in the first k-1 time periods j Characteristic information and user ID Use the multi-head attention mechanism to fuse and get user u j Past user preferences (5) Use the multi-head attention mechanism again to focus on user u in the kth time period j Feature information and past user preferences Fusion is performed to obtain user u j Historical user preferences (6) Repeat the above steps (1) to (5) to traverse all users to obtain the historical user preferences of all users Among them, 1≤j≤m.

6. The deep reinforcement learning recommendation method integrating user preference prediction according to claim 1, characterized in that: The step S3 is specifically as follows: using deep learning methods to establish a user future preference prediction model, respectively j Historical user behavior As input, predict the user u at the next moment j Future user preferences Traverse all users and get the future user preferences of all users Among them, 1≤j≤m.

7. The deep reinforcement learning recommendation method integrating user preference prediction according to claim 1, characterized in that: The step S4 is specifically as follows: (1) using an automatic encoder to convert item i j Encode and get the project number Where 1≤j≤n; (2) Use the automatic encoder to transform user u j Encode and get the user number Where 1≤j≤m; (3) Use the automatic encoder to convert the time window T j Encode and get the time number (4) Use a multi-layer perceptron to transform the time window T j The user IDs corresponding to the user set that has performed user behaviors are perceptually compressed to obtain user aggregation features; (5) A multi-layer perceptron is used to convert the time window T j In the example, the item set in which the user behavior occurs is subjected to perceptual compression of the item numbers used to obtain the item aggregation features; (6) the user aggregation features, item aggregation features, and time numbers are connected in series using a graph convolutional neural network to obtain the time window T j Contextual knowledge representation (7) Traverse all time windows to obtain contextual knowledge representation 8. The deep reinforcement learning recommendation method integrating user preference prediction according to claim 1, characterized in that: The attention mechanism described in step S5 is used to fuse the multi-source features with adaptively adjusted weights. The expression representing the state of the user at the current moment is: Among them, the unknown coefficients α1, α2, and α3 are obtained by learning historical interactive correlation data through neural networks.