Interactive movie recommendation system and method based on graph neural network and reinforcement learning

Abstract

The present invention discloses an interactive movie recommendation system and method based on graph neural network and reinforcement learning, with the purpose of optimizing user experience within a period of time. At the same time, an undirected graph of movie similarity is constructed through the user's historical interaction data to obtain a more accurate movie expression and improve the accuracy of movie recommendation. The technical scheme is as follows: the design of the present invention is divided into 4 modules: a composition module, a movie vector generation module, a user vector generation module and a recommendation module. The steps of the present invention include constructing an undirected graph of movie similarity, constructing a graph neural network to obtain a movie vector representation matrix, constructing an attention module to extract and fuse the information contained in the user's movie viewing history to obtain a user vector representation, constructing a multi-layer perceptron model to fit the recommendation strategy, and sorting and generating recommended movies by state action value. The present invention provides a method for constructing an undirected graph of movie similarity, and effectively improves the accuracy of movie recommendations by introducing a graph neural network.

Description

Technical Field

[0001] The present invention belongs to the technical field of interactive recommendation, and in particular relates to an interactive movie recommendation system and method based on graph neural network and reinforcement learning. Background Art

[0002] With the advent of the big data era, both users and platforms are facing the problem of information overload. Therefore, the platform hopes to filter effective information for users through personalized recommendation systems, improve user experience and promote platform revenue. Personalized recommendation systems have been widely used in various fields of the information industry, such as e-commerce platforms, video websites, and social media.

[0003] Traditional personalized recommendation systems can be divided into user-based personalized recommendation, content-based personalized recommendation, and collaborative filtering-based personalized recommendation. Traditional personalized recommendation systems cannot model users' dynamic interests and do not have enough data for accurate prediction in cold start scenarios. Therefore, research has begun to focus on interactive recommender systems (IRS), which optimize the model during the interaction process and have advantages in dynamic modeling of user interests and user cold start recommendations.

[0004] At present, the research on interactive recommendation systems mainly focuses on two technical directions, namely contextual bandit and reinforcement learning.

[0005] The contextual slot machine method has been widely used in news recommendation, collaborative filtering, online advertising push, and e-commerce recommendation. However, the contextual slot machine method has certain limitations: (1) The model of this method only has a good fitting prediction effect on the linear model; (2) Since the slot machine method attempts to constrain the upper limit between the actual feedback and the ideal feedback, it is a constraint on the worst case and the method is too pessimistic. The reinforcement learning method is an optimization of the method in the Markov decision process, and the recommendation process is a typical Markov decision process. Therefore, the reinforcement learning method is suitable for recommendation systems. At present, the recommendation method based on reinforcement learning still has the following problems in practical applications: (1) The recommendation accuracy is not high; (2) The action space is too large; (3) It is difficult to build an offline simulation environment; (4) The online application and optimization process design of the reinforcement learning framework are complex. Summary of the invention

[0006] Purpose of the invention: To address the problem of low recommendation accuracy of recommendation methods based on reinforcement learning, the present invention designs a new type of movie similarity undirected graph, and combines it with a graph neural network to realize an interactive movie recommendation system and method based on graph neural network and reinforcement learning.

[0007] The above purpose is achieved through the following technical solutions:

[0008] The technical solution adopted by the present invention to solve the technical problem is: an interactive recommendation system based on graph neural network and reinforcement learning, the system includes a composition module, a movie vector generation module, a user vector generation module and a recommendation module;

[0009] Graph construction module: used to construct an undirected graph of movie similarity based on the historical data of user-movie interactions in the database, and obtain the adjacency matrix A of the undirected graph of movie similarity;

[0010] Movie vector generation module: used to take the adjacency matrix A of the movie similarity undirected graph and the randomly initialized movie vector representation matrix E as input, and obtain the updated movie vector representation matrix E′ through the constructed graph neural network model;

[0011] User vector generation module: takes the updated movie vector representation matrix and the user's current historical movie viewing record in the simulated interactive environment as input, and outputs the user feature vector representation after calculation by the constructed self-attention network model;

[0012] Recommendation module: takes the current user feature vector as input, passes through the multi-layer perceptron network model fitting strategy, and outputs the final movie recommendation for the user in the current state.

[0013] Furthermore, the movie vector generation module includes: a vector initialization module and a graph neural network module;

[0014] Vector initialization module: used to randomly initialize the movie vector. Let the dimension of the movie vector be d, then the initialized movie vector matrix is N is the total number of movies;

[0015] Graph neural network module: used to mine collaborative information in offline data based on the movie similarity undirected graph, fuse information on the initialized movie vector matrix, and generate an updated movie vector matrix E′.

[0016] Furthermore, the user vector generation module includes a feature extraction module and a feature synthesis module;

[0017] Feature extraction module: used to convert the user's current state of historical movie viewing into a movie vector representation updated by the graph neural network, and classify them according to the user's historical scores. In each category, the built self-attention network is used to calculate and output the feature vectors of each category;

[0018] Feature synthesis module: takes the feature vector generated by each category as input and outputs the user feature vector.

[0019] Furthermore, the recommendation module is divided into a strategy fitting module and a recommendation generation module;

[0020] The strategy fitting module is used to input the user feature vector expression, fit the recommendation strategy, and output the state action value of each movie to be recommended;

[0021] The recommendation generation module sorts the calculated state-action values ​​and generates a movie recommendation for the target user.

[0022] A method for recommending movies using the interactive movie recommendation system based on graph neural network and reinforcement learning comprises the following steps:

[0023] Step S1: constructing an undirected graph of movie similarity based on historical data of user-movie interactions in a database;

[0024] Step S2: taking the adjacency matrix A of the movie similarity undirected graph and the randomly initialized movie vector representation matrix E as input, and obtaining the updated movie vector representation matrix E′ through the constructed graph neural network model;

[0025] Step S3: taking the updated movie vector representation matrix and the user's current historical movie viewing record in the simulated interactive environment as input, and outputting the user feature vector representation through calculation by the constructed self-attention network model;

[0026] Step S4: taking the current user feature vector as input, and outputting the state action value of each movie to be recommended through the multi-layer perceptron network model fitting strategy;

[0027] Step S5: Sort the movies to be recommended by their state action values ​​from large to small to generate the final movie recommendation.

[0028] Furthermore, the movie similarity undirected graph in S1 is represented by an adjacency matrix A.

[0029] Furthermore, the movie similarity undirected graph is composed of offline interaction data of a training set in a data set.

[0030] Furthermore, the data set is divided into a training set and a test set by user, of which 85% of the users and their interaction data are the training set, and 15% of the users and their interaction data are the test set.

[0031] Furthermore, when two movies appear in the viewing records of n users at the same time, and n ≥ 10, there is an edge between the two movies.

[0032] Furthermore, when there is an edge between nodes m1 and m2 in the graph, the value of the position (m1, m2) in the adjacency matrix is ​​1.

[0033] Furthermore, the steps for constructing the graph neural network model in S2 are specifically as follows:

[0034] Step S2.1 Each graph neural network layer is composed of a GAT graph neural network structure, and each graph neural network layer is divided into 5 heads for parallel operation:

[0035] E 1i =GAT(A, E), 1≤i≤5

[0036] Step S2.2 concatenates the vectors output by the five heads to obtain the output of the first layer of the graph neural network:

[0037] E1=concat(E1,...,E5)

[0038] Step S2.3 uses the output of the first layer of graph neural network as the input of the second layer of graph neural network.

[0039] In step S2.4, after the same operation as the first layer, the output of the second layer of graph neural network is used as the updated movie vector representation matrix E′.

[0040] Furthermore, the specific steps of calculating the user feature vector in S3 are:

[0041] Step S3.1 uses the vectors in the updated movie vector representation matrix E′ to represent the user's historical movie viewing records according to the index;

[0042] Step S3.2 classifies the movies in the viewing record according to the user's ratings. The vector matrix composed of each category of movies is

[0043] Step S3.3 uses each type of movie vector matrix as the input of the self-attention (SA) network, and the self-attention network calculates:

[0044]

[0045] Furthermore, is the query sentence weight of the self-attention network, is the self-attention network matching landmark weight, is the weight of the sentence to be matched. The three weights are trained by the network.

[0046] Furthermore,

[0047]

[0048] Furthermore, h is the dimension parameter, usually equal to the matrix Dimensions. Used to prevent the desired Too large.

[0049] Step S3.4 uses the output of the self-attention network as the input of the feed forward network to further mine feature information and increase the degree of model nonlinearity:

[0050]

[0051] Furthermore, represents the output of the self-attention network, W (1) , b (1) Represent the weight and bias of the first layer of neural network, W (2) , b (2) They represent the weight and bias of the second layer of the neural network respectively, and the ReLU (Rectified Linear Unit) function is a linear rectification function.

[0052] Furthermore, the self-attention network and the feedforward network form a self-attention module. The self-attention modules of each category together form a feature extraction module.

[0053] In step S3.5, the feature extraction module outputs the movie feature vectors under each category, and the feature synthesis module generates a user feature vector by merging the movie feature vectors under each category.

[0054] Beneficial effects:

[0055] Compared with the existing modules, the present invention has the following beneficial effects:

[0056] 1. The present invention provides an interactive recommendation method based on graph neural network and reinforcement learning, proposes a design method for an undirected graph of movie similarity, and introduces a graph neural network based on the framework of reinforcement learning, which significantly increases the recommendation accuracy of the method;

[0057] 2. The reinforcement learning-based recommendation model provided by the present invention promotes a balance between exploration and utilization in the recommendation process, improves the user portrait as quickly as possible without significantly harming the user experience in a cold start scenario, and achieves the highest user recommendation accuracy within a period of time.

[0058] Other features and advantages of the present invention will become more apparent after reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a schematic diagram of the structure of the recommended method provided by the present invention;

[0060] Figure 2 It is a flow chart of the recommended method provided by the present invention;

[0061] Figure 3It is a schematic diagram of calculating the movie feature representation matrix provided by the present invention;

[0062] Figure 4 It is a schematic diagram of calculating the user feature representation vector provided by the present invention;

[0063] In the figure: 1-composition module, 2-movie vector generation module, 3-user vector generation module, 4-recommendation module; among them: 21-vector initialization module, 22-graph neural network module, 31-feature extraction module, 32-feature synthesis module, 41-strategy fitting module, 42-recommendation generation module. DETAILED DESCRIPTION

[0064] The present invention is further described below with reference to the accompanying drawings and examples.

[0065] like Figure 1-4 As shown, an interactive recommendation method based on graph neural network and reinforcement learning includes a composition module 1, a movie vector generation module, a user vector generation module and a recommendation module. The composition module is connected to the movie vector generation module, the movie vector generation module is connected to the user vector generation module, and the user vector generation module is connected to the recommendation module;

[0066] Graph construction module 1: used to construct an undirected graph of movie similarity based on the historical data of user-movie interactions in the database, and obtain the adjacency matrix A of the undirected graph of movie similarity;

[0067] Movie vector generation module 2: used to take the adjacency matrix A of the movie similarity undirected graph and the randomly initialized movie vector representation matrix E as input, and obtain the updated movie vector representation matrix E′ through the constructed graph neural network model;

[0068] User vector generation module 3: takes the updated movie vector representation matrix and the user's current historical movie viewing record in the simulated interactive environment as input, and outputs the user feature vector representation after calculation by the constructed self-attention network model;

[0069] Recommendation module 4: takes the current user feature vector as input, passes through the multi-layer perceptron network model fitting strategy, and outputs the final movie recommendation for the user in the current state.

[0070] Furthermore, the movie vector generation module includes: a vector initialization module 21 and a graph neural network module 22. The vector initialization module 21 is connected to the graph neural network module 22;

[0071] Vector initialization module 21: used to randomly initialize the movie vector. Let the dimension of the movie vector be d, then the initialized movie vector matrix is N is the total number of movies;

[0072] Graph neural network module 22: used to mine collaborative information in offline data based on the movie similarity undirected graph, perform information fusion on the initialized movie vector matrix, and generate an updated movie vector matrix E′.

[0073] Furthermore, the user vector generation module 3 includes a feature extraction module 31 and a feature synthesis module 32. The feature extraction module 31 and the feature synthesis module 32 are connected;

[0074] Feature extraction module 31: used to convert the historical movies watched by the user in the current state into a movie vector representation updated by the graph neural network, and classify them according to the user's historical scores, and use the constructed self-attention network to calculate and output the feature vectors of each category in each category;

[0075] Feature synthesis module 32: takes the feature vector generated by each category as input and outputs the user feature vector.

[0076] Furthermore, the recommendation module 4 is divided into a strategy fitting module 41 and a recommendation generation module 42. The strategy fitting module 41 is connected to the recommendation generation module 42;

[0077] The strategy fitting module 41 is used to input the user feature vector expression, fit the recommendation strategy, and output the state action value of each movie to be recommended;

[0078] The recommendation generation module 42 sorts the calculated state action values ​​and generates a movie recommendation for the target user.

[0079] Furthermore, an interactive movie recommendation method based on graph neural network and reinforcement learning includes the following steps:

[0080] Step S1, constructing an undirected graph of movie similarity: It is used to construct an undirected graph of movie similarity based on the historical data of user-movie interactions in the database. The undirected graph of movie similarity is represented by the adjacency matrix A. The undirected graph of movie similarity is composed of offline interaction data of the training set in the dataset. The dataset is divided into training set and test set according to users. 85% of the users and their interaction data are training sets, and 15% of the users and their interaction data are test sets. When two movies appear in the viewing records of n users at the same time, and n≥10, there is an edge between the two movies. When there is an edge between nodes (i, j) in the graph, the value of position (i, j) in the adjacency matrix is ​​1;

[0081] Step S2, constructing a graph neural network model to obtain a movie vector representation matrix: using the adjacency matrix A of the movie similarity undirected graph and the randomly initialized movie vector representation matrix E as input, and obtaining an updated movie vector representation matrix E′ through the constructed graph neural network model;

[0082] like Figure 3 As shown in Figure 2, the construction steps of the graph neural network model in S2 are as follows:

[0083] Each graph neural network layer is composed of the GAT graph neural network structure. Each graph neural network layer is divided into 5 heads for parallel operation:

[0084] E 1i =GAT(A, E), 1≤i≤5

[0085] The vectors output by the five heads are concatenated to get the output of the first layer of the graph neural network:

[0086] E1=concat(E1,...,E5)

[0087] The output of the first layer of graph neural network is then used as the input of the second layer of graph neural network.

[0088] Finally, after the same operation as the first layer, the output of the second layer of graph neural network is used as the updated movie vector representation matrix E′;

[0089] Step S3, grouping the user's historical movie-watching rating records, and using the attention mechanism to fuse the information to obtain the user vector representation: taking the updated movie vector representation matrix and the user's current historical movie-watching records in the simulated interactive environment as input, and calculating through the constructed self-attention network model, outputting the user feature vector representation;

[0090] like Figure 4 As shown, the specific steps for calculating the user feature vector are:

[0091] Step S3.1 uses the vectors in the updated movie vector representation matrix E′ to represent the user's historical movie viewing records according to the index;

[0092] Step S3.2 classifies the movies in the viewing record according to the user's ratings. The vector matrix composed of each category of movies is

[0093] Step S3.3 uses each type of movie vector matrix as the input of the self-attention (SA) network, and the self-attention network calculates:

[0094]

[0095] Furthermore, is the query sentence weight of the self-attention network, is the self-attention network matching landmark weight, is the weight of the sentence to be matched. These three weights are trained by the network.

[0096] Furthermore,

[0097]

[0098] Furthermore, h is the dimension parameter, usually equal to the matrix Dimensions. Used to prevent the desired Too large.

[0099] Step S3.4 uses the output of the self-attention network as the input of the feed forward network to further mine feature information and increase the degree of model nonlinearity:

[0100]

[0101] Furthermore, represents the output of the self-attention network, W (1) , b (1) Represent the weight and bias of the first layer of neural network, W (2) , b (2) They represent the weight and bias of the second layer of the neural network respectively, and the ReLU (Rectified Linear Unit) function is a linear rectification function.

[0102] Furthermore, the self-attention network and the feedforward network form a self-attention module. The self-attention modules of each category together form a feature extraction module.

[0103] In step S3.5, the feature extraction module outputs the movie feature vectors under each category, and the feature synthesis module generates a user feature vector by merging the movie feature vectors under each category.

[0104] Step S4, construct a multi-layer perceptron model and fit the recommendation strategy: take the current user feature vector as input, fit the strategy through the multi-layer perceptron network model, and output the state action value of each movie to be recommended.

[0105] Step S5: Generate recommended movies: Sort the movies to be recommended by their state action values ​​from large to small to generate the final movie recommendation.

[0106] The present invention conducts experiments in a virtual simulation environment of an offline data set. For the known user and movie interaction data in the training set, an undirected graph of movie similarity is designed and constructed, and a graph neural network is introduced to mine the similarity information between movies contained in the collaborative data, so as to obtain a more accurate expression of the movie vector. After the movie vector expression is obtained, the user's viewing history is classified, and the self-attention mechanism is used to further mine the user's historical behavior information, so as to obtain the user feature vector expression. In addition, the present invention is trained under the reinforcement learning framework, balancing the use and exploration problems, which does not obviously harm the user's experience, but quickly obtains the exploration of user interests, improves the user portrait, and optimizes the user's experience over a period of time.

[0107] The above embodiments are only preferred implementation modes of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and equivalent substitutions without departing from the principles of the present invention. These technical solutions after improvements and equivalent substitutions to the claims of the present invention all fall within the protection scope of the present invention.

Claims

1. An interactive movie recommendation method based on graph neural network and reinforcement learning, characterized in that: The method comprises the following steps: Step S1: construct an undirected graph of movie similarity based on the historical data of user-movie interactions in the database; Step S2: taking the adjacency matrix A of the movie similarity undirected graph and the randomly initialized movie vector representation matrix E as input, and passing through the constructed graph neural network model to obtain the updated movie vector representation matrix E′; Step S3: taking the updated movie vector representation matrix and the user's current historical movie viewing record in the simulated interactive environment as input, and outputting the user feature vector representation through calculation by the constructed self-attention network model; Step S4: taking the current user feature vector as input, and outputting the state action value of each movie to be recommended through the multi-layer perceptron network model fitting strategy; Step S5: Sort the movies to be recommended by their state action values ​​from large to small to generate the final movie recommendation; The specific steps of calculating the user feature vector in S3 are: Step S3.1 uses the vectors in the updated movie vector representation matrix E′ to represent the user's historical movie viewing records according to the index; Step S3.2 classifies the movies in the viewing record according to the user's ratings. The vector matrix composed of each category of movies is Step S3.3 takes each type of movie vector matrix as the input of the self-attention network, and obtains the following after calculation by the self-attention network: in, is the query sentence weight of the self-attention network, is the self-attention network matching landmark weight, is the weight of the sentence to be matched. These three weights are trained by the network. h is the dimension parameter, which is equal to the matrix Dimensions to prevent the desired Too large; Step S3.4 uses the output of the self-attention network as the input of the forward network to further mine feature information and increase the degree of nonlinearity of the model: in, represents the output of the self-attention network, W (1) , b (1) Represent the weight and bias of the first layer of neural network, W (2) , b (2) They represent the weight and bias of the second layer of the neural network respectively, and the ReLU function is a linear rectification function; the self-attention network and the forward network form a self-attention module, and the self-attention modules of each category together form a feature extraction module; In step S3.5, the feature extraction module outputs the movie feature vectors under each category, and the feature synthesis module generates a user feature vector by merging the movie feature vectors under each category.

2. The interactive movie recommendation method based on graph neural network and reinforcement learning according to claim 1, characterized in that: The movie similarity undirected graph described in step S1 is composed of offline interaction data of the training set in the data set. The data set is divided into a training set and a test set according to users, wherein 85% of the users and their interaction data are the training set, and 15% of the users and their interaction data are the test set. When two movies appear in the viewing records of n users at the same time, and n≥10, there is an edge between the two movies. When there is an edge between nodes m1 and m2 in the graph, the value of the position (m1, m2) in the adjacency matrix is ​​1.

3. The interactive movie recommendation method based on graph neural network and reinforcement learning according to claim 1, characterized in that: The specific steps for constructing the graph neural network model described in S2 are: Step S2.1 Each graph neural network layer is composed of a GAT graph neural network structure, and each graph neural network layer is divided into 5 heads for parallel operation: AND 1i =GAT(A,E),1≤i≤5 Step S2.2 concatenates the vectors output by the five heads to obtain the output of the first layer of the graph neural network: E1=concat(E1...,E5) Step S2.3 uses the output of the first layer of graph neural network as the input of the second layer of graph neural network; In step S2.4, after the same operation as the first layer, the output of the second layer of graph neural network is used as the updated movie vector representation matrix E′.

4. An interactive movie recommendation system based on graph neural network and reinforcement learning, used to implement any method of claims 1-3, characterized in that: It includes a composition module (1), a movie vector generation module (2), a user vector generation module (3), and a recommendation module (4); Graph construction module (1): used to construct a movie similarity undirected graph based on the historical data of user-movie interactions in the database, and obtain an adjacency matrix A of the movie similarity undirected graph; Movie vector generation module (2): used to take the adjacency matrix A of the movie similarity undirected graph and the randomly initialized movie vector representation matrix E as input, and obtain the updated movie vector representation matrix E′ through the constructed graph neural network model; User vector generation module (3): takes the updated movie vector representation matrix and the user's current historical movie viewing record in the simulated interactive environment as input, and outputs the user feature vector representation after calculation by the constructed self-attention network model; Recommendation module (4): takes the current user feature vector as input, passes through the multi-layer perceptron network model fitting strategy, and outputs the final movie recommendation for the user in the current state.

5. The interactive movie recommendation system based on graph neural network and reinforcement learning according to claim 4, characterized in that: The movie vector generation module (2) comprises: a vector initialization module (21) and a graph neural network module (22); Vector initialization module (21): used to randomly initialize the movie vector. Let the dimension of the movie vector be d, then the initialized movie vector matrix is N is the total number of movies; Graph neural network module (22): used to mine collaborative information in offline data based on the movie similarity undirected graph, perform information fusion on the initialized movie vector matrix, and generate an updated movie vector matrix E′.

6. The interactive movie recommendation system based on graph neural network and reinforcement learning according to claim 4, characterized in that: The user vector generation module (3) comprises a feature extraction module (31) and a feature synthesis module (32); Feature extraction module (31): used to convert the user's current state of historical movie viewing into a movie vector representation updated by the graph neural network, and classify according to the user's historical scores, and use the constructed self-attention network to calculate and output the feature vector of each category; Feature synthesis module (32): takes the feature vector generated by each category as input and outputs the user feature vector.

7. The interactive movie recommendation system based on graph neural network and reinforcement learning according to claim 4, characterized in that: The recommendation module (4) is divided into a strategy fitting module (41) and a recommendation generation module (42); The strategy fitting module (41) is used to input the user feature vector expression, fit the recommendation strategy, and output the state action value of each movie to be recommended; The recommendation generation module (42) sorts the calculated state action values ​​and generates a movie recommendation for the target user.

Images