A Graph Convolutional Interest Decoupling Method Based on Transformer Model

Abstract

The present invention discloses a graph convolution interest decoupling method based on the Transformer model. Traditional recommendation algorithms mostly adopt the graph convolution method, which considers the information transmission between users and their neighbor nodes. However, the generated user feature representation is single, ignoring that the user's interaction is generated by multiple aspects of interests, that is, the problem of implicit vector entanglement, resulting in the performance of the recommendation not reaching the optimal. The present invention proposes a new type of fine-grained interest decoupling method. First, the graph convolution method is used to study the features of users within different interest spaces, then the Transformer model is used to explore the internal connections between different interests of users, and finally the features between different interests are fused to obtain the user feature expression after interest decoupling. For the above method, we use the publicly available datasets of social networks, e-commerce platforms, and review websites to train and test respectively to optimize the performance of the model and verify the effectiveness of the method.

Description

Technical Field

[0001] The present invention relates to collaborative filtering technology, specifically a model-based collaborative filtering algorithm; the present invention relates to a representation learning method based on graph convolution, specifically the LightGCN algorithm; the present invention relates to a representation learning method based on Transformer; the present invention also relates to related technologies such as data mining and deep learning. Background Art

[0002] The emergence and popularization of the Internet have brought a large amount of information to users, meeting the users' information needs in the information age. However, with the rapid development of the network, the substantial growth of online information has made it impossible for users to obtain the truly useful part of the information from a large amount of information, and the utilization efficiency of information has instead decreased. This is the so-called information overload problem.

[0003] A very important way to solve the information overload problem is the recommendation system, which is a personalized information recommendation system that recommends information, items, etc. that users are interested in according to users' information needs, interests, etc. Currently, the most widely used method in the recommendation system is the collaborative filtering (CF) idea, that is, analyzing the historical interaction records of users-items to predict the next possible item that a user may interact with, and making full use of the existing user interaction data to find the content that best meets the user's interests. And the most commonly used in the CF idea is the model-based collaborative filtering (MCF) algorithm. Its core idea is that both users and items have certain characteristics that affect users' choices; the reason why a user selects a certain commodity is that the user characteristics match the item characteristics. Based on this idea, during the process of model establishment, if there are no explicit user / item characteristics to extract, then we have to mine potential user / item characteristics based on the existing preference data, perform model training, and user-item matching and recommendation.

[0004] Learning high-quality user and item representations has always been the key to MCF. Traditional MCF recommendation algorithms mostly use the method of Graph Convolutional Network (GCN) to explore the information transfer between the target node and its high-order neighbor nodes. However, the generated user feature representation is single, ignoring that users' interactions are generated by multiple aspects of interests, that is, the problem of latent vector entanglement. In the current research on interest decoupling, such as Disentangled Graph Collaborative Filtering (DGCF) by the School of Computing, National University of Singapore, only the independence modeling between different interest features is considered, without considering the internal connection, that is, compatibility, between different interest features, which will lead to the recommendation performance not reaching the optimal.

[0005] The Transformer model was proposed by the Google team in 2017. Initially, its emergence was to solve problems in Natural Language Processing (NLP), such as language modeling, machine translation, etc. The core module in the Transformer model is the Self-attention mechanism. In NLP tasks, its role is to represent local features by studying the dependence relationship between local features and global features. For example, in the sentence 'The animal didn't cross the street because it was tired', for the word vector 'it', without considering its position encoding information, that is, the degree of dependence on the global feature, it is impossible to determine whether 'it' represents 'animal' or'street' in this sentence. Therefore, the Transformer model can add position encoding information to the local feature representation to solve this problem. Currently, the application of the Transformer model has also expanded from the NLP field to computer vision. For example: 'End to End Object Detection With Transformer (DETR)' for object detection tasks in computer vision, 'Vision Transformer (VIT)' for video classification tasks, etc. In 'Multiplex Behavioral Relation Learning for Recommendation via Memory Augmented Transformer Network (MATN)' proposed by South China University of Technology and JD.com in 2020, the Transformer model was applied to the field of recommendation systems to explore the dependence between different user behaviors.

[0006] In summary, most of the existing methods based on graph neural networks can only generate unified representations and rarely study the representations generated by multiple user interests at a fine-grained level. And only a few existing interest decoupling methods only focus on the independence between different interests and do not consider the internal relationship between interests, that is, compatibility. So the performance of the recommendation cannot reach the optimal level.

[0007] Starting from this, the present invention proposes a graph convolutional interest decoupling method based on the Transformer model. SUMMARY OF THE INVENTION

[0008] Existing recommendation algorithms usually use graph neural network methods for user and item feature representation learning. However, the generated user feature representation is single, ignoring that user interactions are generated by multiple interests. The user buys a keyboard driven by Interest 1, a mouse driven by Interest 2, perfume driven by Interest 3, and lipstick driven by Interest 4. Obviously, using only a single user feature is not enough to describe all interests. We will explore the feature representations of users under different interests with a graph neural network at a fine-grained level. At the same time, it can also be seen from the above example that the four different interests are not independent of each other. There is a certain internal relationship, that is, compatibility, between Interest 1 and Interest 2. Similarly, there is also a certain internal relationship between Interest 3 and Interest 4. Therefore, we will use the Transformer model to add the position information of the corresponding other interests to the features under each interest, that is, consider the compatibility between different interests.

[0009] In order to explore the features of users under different interests and the compatibility of features between different interests, the present invention discloses a graph convolutional interest decoupling method based on the Transformer model. The overall framework is as Figure 1As shown in the figure. The datasets used in this method are all from publicly available datasets in academia and industry (social network: Gowalla, e-commerce platform: Amazon-book, review website: Yelp2018). This method includes four modules, namely the mapping module, the multi-interest space GCN module, the Transformer-based interest compatibility module, and the gated fusion module. The input is the historical interaction between users and items. First, it is regarded as the structure of a bipartite graph, with users and items as nodes and historical interactions as undirected edges between user nodes and items. Each node in the graph has its own feature vector. In the mapping module, the interaction graph needs to be mapped to different interest spaces. Although the graph topologies in different interest spaces are the same, the feature representations of the same node in different interest spaces are already different. In the multi-interest space GCN module, we use a routing mechanism to iteratively update our adjacency matrix in different interest spaces, making the adjacency matrices in different spaces different, and the adjacency values are updated using the affinity between the node itself and its first-order neighbors. At this time, the graph topologies in different spaces are already different. We use the LightGCN algorithm to update the features of users and items in different spaces, making the features of users and items in the space closely related to the interactions driven by a certain interest. In the Transformer-based interest compatibility module, we will study the correlation between different interest spaces, regard the node features in a certain space as local features, and regard the corresponding node features in the remaining spaces as global features, and generate attention values through the Transformer model to update the node features in each space. Finally, through the gated mechanism, the node features in different interest spaces are fused into a single feature. At this time, the node contains the feature information after decoupling the interests, and also contains the feature information of the compatibility between different interests. After passing through the above network, it is necessary to add a constraint on the independence of different spaces to the loss function in the training stage, making different interest spaces tend to be orthogonal. After training, in the test stage, the inner product of the user feature and the target item feature is calculated, and the inner product value is used as the prediction score between the user and the target item, and the item with the largest score value is recommended to the user.

[0010] The invention content of each main module of this method is as follows:

[0011] 1. Mapping module

[0012] First, regard the input user-item interaction history as a graph structure, with the IDs of users and items as nodes and historical interactions as undirected edges of the graph structure, and the weight of each edge is regarded as 1. At the same time, randomly initialize the respective feature vectors for each node. As Figure 2As shown in the figure, the same graph structure is mapped to different interest spaces using different fully connected layers. At this time, the topological structure of the graph is the same, but the features of the same node in different interest spaces are already different.

[0013] 2. Multi-Interest GCN Module

[0014] In traditional GCN, if a user has a historical interaction with an item, the adjacency value in the adjacency matrix is 1. In this way, the information transfer between nodes and neighbor nodes is only considered at a coarse-grained level. If we consider from the perspective of the user's multiple interests, the adjacency values in different spaces should be different, that is, the interaction between the user and the item is mainly driven by a certain interest, and the association with other interests is relatively small. Therefore, we can use a routing mechanism, such as Figure 3 As shown in the figure, set the number of iterations T. In each iteration, the adjacency values in the adjacency matrix in different spaces are updated through the affinity calculation formula between the target node and the neighbor nodes in each space, so that the weights of the undirected edges in different spaces are no longer 1, but update their weights according to different interests. Then, through the updated adjacency matrix, the graph structures in different spaces update the features of their respective users and items through the LightGCN algorithm. After T iterations, it can be considered that all nodes in the graph have passed through 1 layer of multi-interest GCN.

[0015] 3. Transformer-Based Interest Compatibility Module

[0016] As Figure 4 shown in the figure, after passing through L layers of multi-interest GCN, we use the node features in a certain space as local features, and the node features in other spaces as global features and send them into the Transformer model. The node features in this space are updated through the self-attention scores. At this time, the information on the compatibility between different interests has been added to the node features.

[0017] 4. Gated Fusion Module and Prediction Recommendation

[0018] The gated mechanism fuses the features under different interests into a single feature in an adaptive way. The gated mechanism can learn the importance of the features under specific interests. At the same time, the fused features include both the information under specific interests and the information on the internal connection with other interests. Finally, the learned user features and the target item features are used to calculate the inner product, and the inner product value is used as the matching score between the user and the item. The item with the highest score is recommended to the user. Brief Description of the Drawings

[0019] Figure 1 This is the overall framework of the graph convolutional interest decoupling method based on the Transformer model of the present invention.

[0020] Figure 2 Schematic diagram of the mapping module

[0021] Figure 3 Schematic diagram of the multi-interest GCN module

[0022] Figure 4 Schematic diagram of interest compatibility modeling based on Transformer Specific implementation manner

[0023] The present invention discloses a graph convolutional interest decoupling method based on the Transformer model. The specific implementation steps of the invention are as follows:

[0024] Step 1: Data preprocessing and division of the training set and the test set:

[0025] First, select appropriate datasets: Gowalla (social network), Amazon - Book (e - commerce platform), Yelp2018 (review website). Number the users and items in the datasets sequentially from 0, and record the historical interactions between users and items in the form of implicit feedback. That is, if a user has interacted with an item (browsed, purchased, collected, etc.), the label is 1 (positive sample); if there is no interaction, the label is 0 (negative sample). If the interactions in the dataset are in the form of ratings, the items with ratings can be labeled as 1, and otherwise as 0. After obtaining the processed dataset, for each user, divide the dataset of the items it has interacted with (positive samples) into a training set and a test set according to a ratio of 4:1. Finally, in the loss optimization stage, keep the ratio of positive and negative samples at 1:1.

[0026] Step 2: Input the model and map it to different interest spaces:

[0027] Read all users, items and their interaction histories in the dataset, and convert them into an undirected graph structure G=(V, E), where V = {v1, v2...v M ...v M+N} represents node information, M and N represent the total number of users and items respectively, and E is the edge information of the historical interactions between users and items. Randomly generate feature vectors for each node through Xavier initialization, X i ∈R d (i = 1…(M + N), R represents a matrix or a vector, and the vector dimension d is set to 64. Map the graph structure to different interest spaces through a fully - connected layer where W k and b k represent the weight matrix and the bias vector under the k - th interest space respectively, which are generated through Xavier initialization, W k ∈R d*d , bk ∈R d 。The activation function uses the Relu function, X = {X1, X2…X (M+N)}. Y k represents the feature vectors of all nodes in the k-th interest space, where k = 1, 2, 3, 4.)

[0028] Step 3: Multi-Interest GCN:

[0029] Set the number of layers L of the multi-interest GCN to 0, 1, 2, and the number of iterations t of the routing mechanism for each layer to 0, 1, 2. The initial adjacency matrix for each interest space is the same, which is the interaction value between users and items, with a value of 1 for existing historical interactions and 0 for no interaction. That is A is the adjacency matrix generated according to the input interaction data, A ∈ R (M+N)*(M+N) . represents the adjacency matrix updated by the t-th iteration algorithm in the k-th interest space. In the routing mechanism iteration, first use the softmax function to obtain the interest distribution in each space, and the formula is K = 4, represents the interaction value between user u and item i in the adjacency matrix. Then use the LightGCN algorithm to update the node features through different normalized adjacency matrices in each space, and the specific formula is is the adjacency matrix corresponding degree matrix, represents the feature vector of the k-th feature space after the t-th iteration. Finally, calculate the adjacency value of the updated adjacency matrix through the affinity between the self-node and the first-order neighbor nodes, and the specific formula is tanh() is the hyperbolic tangent function. Among them represents the feature of user u node in the k-th interest space after the t-th iteration, represents the initial feature of neighbor node i of user u after passing through the fully connected layer in the k-th interest space. After t = 2 iterations, it is equivalent to all nodes in the graph passing through 1 layer of LightGCN. Finally, sum the node features generated in each layer after L layers, that is Y k = Y k (0) + Y k (1) ... + Y k (L-1) , L = 3, Y k (L-1) represents the feature vector of the k-th feature space in the L-1 layer.)

[0030] Step 4: Transformer Interest Compatibility:

[0031] The node features of the same node in different spaces are fed into the Transformer model. The node features under a specific interest type are used as local features, and the node features of other interest types are used as global features to generate self-attention values to update the features of the node. The specific formula is

[0032]

[0033]

[0034]

[0035] Among them, is the mapping matrix. h = 0, 1, both are Xavier randomly initialized matrices. H is the number of multi-head self-attention, set to 2. is the attention value between features of different channels, and || is the concatenation operation.

[0036] Step Five: Gated Fusion:

[0037] The node features in different interest spaces are generated through the above steps and fused into a single feature through the gated module. The formula is as follows:

[0038]

[0039] Y = ω1 * Y 1 ... + ω k * Y k

[0040] Among them, the value of ω i is Xavier randomly initialized, where i = 1, 2, 3, 4. Y is the fused user and item features.

[0041] Y = {Y1, Y2... Y (M+N)} where Y i ∈R d (i = 1, 2... M + N) is the feature vector of each node after passing through the model. Yk represents the feature vector of all nodes in the kth interest space.

[0042] Step Six: Loss Function and Optimization Method

[0043] The loss function in the present invention includes two parts. The first part is the loss function of the interaction prediction part. Two commonly used learning strategies in the recommendation system are pointwise and pairwise optimization methods. The present invention selects the pairwise BPR loss function, the purpose of which is to make the scores of the user's historical interaction items higher than the scores of the unobserved items, and minimize the following loss function:

[0044]

[0045] y ui = E u * E i

[0046] where O = {(u, i, j)|(u, i) ∈ O + , (u, j) ∈ O -} represents the dataset, E u = {Y1...Y M}, E i = {Y M+1 ...Y (M+N)}, O + represents the interactions observed by the user, and O - represents the interactions not observed by the user. σ() is the sigmoid function, and λ = 0.01 is a hyperparameter that controls the strength of L2 regularization, aiming to control the overfitting problem of the model. E u and E i are the feature mappings of user u and item i after fusion. In this method, the inner product between the user and the item is used as the interaction prediction score y ui . θ = {X, Q h , K h , V h , W k , b k}

[0047] On the other hand, to orthogonalize different interest mapping spaces and minimize their correlation with each other, it is necessary to minimize the distance correlation loss function between different spaces, as follows:

[0048]

[0049]

[0050] where dCov() is the distance correlation function of the feature matrices between two spaces, and dVar() is the distance variance function of each feature matrix.

[0051] Finally, jointly minimize the loss functions Loss of these two parts to optimize the model parameters.

[0052] Loss = loss BPR + loss inv

[0053] Step 7: Verify the effectiveness of the method:

[0054] After the model training is completed, in order to verify the effectiveness of the method of the present invention, the method of the present invention is applied to the public datasets Gowalla: social network, Amazon-book: e-commerce platform, and Yelp2018: review website. After obtaining the interaction prediction scores of the target users for the items to be recommended, for each target user, the top 20, 40, 60, 80, and 100 items with the highest scores are selected to form a Top-N personalized recommendation list. The evaluation metrics used in the experiment are Recall (the accuracy rate relative to all positive samples), Precision (the accuracy rate relative to the Top-N list), Hit Rate (the probability of detecting any positive sample in the Top-N list), and Normalized Discounted Cumulative Gain (NDCG focuses on the position of the positive sample in the Top-N recommendation list. The earlier the position, the larger the NDCG).

Claims

1. A graph convolutional interest decoupling method based on the Transformer model; characterized in that: It includes the following steps: S1. Data preprocessing: Obtain a dataset including user IDs, item IDs, and labels of user-item interactions; the positive sample ratio of the training set and the test set in the dataset is 4:1; the positive and negative sample ratio in the training set is 1:1; S2. Model input and mapping to different interest spaces: Convert user IDs, item IDs, and historical interaction records into node and edge information in a graph model respectively, and initialize feature vectors for each node; Map the graph structure to different interest spaces through a fully connected layer, i.e., MLP operation, so that the feature representations of the same node are different in different spaces; S3. Multi-interest GCN: Use a routing mechanism to iteratively update the adjacency matrix in different interest spaces, making the adjacency matrices in different spaces different, and the adjacency values are updated using the affinity between the node itself and its first-order neighbors; At this time, the graph topologies in different spaces are already different. Use the LightGCN algorithm, use the softmax function to obtain the interest distribution in each space, update the node features through different normalized adjacency matrices in each space, then calculate the adjacency values of the updated adjacency matrix using the affinity between the node itself and the first-order neighbor nodes, and finally obtain the feature vectors of each interest space, and update the user and item features in different spaces, making the features of users and items in the space closely related to the interactions driven by a certain interest; S4. Transformer interest compatibility: To study the correlation between different interest spaces, regard the node features in a certain space as local features, and regard the corresponding node features in the remaining spaces as global features, and generate attention values through the Transformer model to update the node features in each space; S5. Gated fusion: Fuse the node features in different interest spaces into a single feature. At this time, the nodes contain the feature information after decoupling the interests, and also contain the feature information of the compatibility between different interests; S6. Optimization method and loss function: The loss function consists of two parts. One part is the BPR loss function used for recommendation prediction and the L2 regularization term composed of initialization parameters, and the other part is the distance correlation function to ensure that different interest spaces tend to be orthogonal. The two are jointly optimized; S7. Generate a recommendation list: After obtaining the predicted user-item interaction scores, for each user, sort all items in descending order according to the scores, and generate a list of the top N items to recommend to the user.

2. The graph convolutional interest decoupling method based on the Transformer model according to claim 1, characterized in that; Record the user-item interaction records in the form of implicit feedback: That is, if there is an interaction between a user and an item, the label is recorded as 1 and is regarded as a positive sample, otherwise it is 0 and is regarded as a negative sample. If the interaction in the dataset is in the form of a score, the items with a score and a 5-point scale score greater than or equal to 4 are labeled as 1 and regarded as positive samples, and the rest are 0 and regarded as negative samples; for each user, the positive sample ratio of the training set and the test set is 4:1, and the positive and negative sample ratio in the training set is 1:

1.

3. A graph convolutional interest decoupling method based on the Transformer model according to claim 1, characterized in that ； Read all users, items, and their interaction histories in the dataset and convert them into an undirected graph structure \(G=(V, E)\), where \(V = \{v_1, v_2, \ldots, v M+N \}\) represents node information, \(M\) and \(N\) represent the total number of users and items respectively, and \(E\) is the edge information of the historical interactions between users and items; randomly generate a feature vector \(X i \in\mathbb{R} d \) for each node by Xavier initialization, \(i = 1, 2, \ldots, M + N\), and the vector dimension \(d\) is set to 64; map the graph structure into different interest spaces through a fully connected layer where \(W k \) and \(b k \) represent the weight matrix and bias vector under the \(k\)-th interest space respectively, which are generated by Xavier initialization, \(W k \in\mathbb{R} d*d \), \(b k \in\mathbb{R} d \); use the ReLU function as the activation function, \(X=\{X_1, X_2, \ldots, X M+N \}\); \(Y k \) represents the feature vectors of all nodes under the \(k\)-th interest space, \(k = 1, 2, 3, 4\).

4. The graph convolutional interest decoupling method based on the Transformer model according to claim 3, wherein: Set the number of layers L of the multi-interest GCN to 3, and the number of iterations t of the routing mechanism for each layer to 3; the initial adjacency matrix of each interest space is the same, which is the interaction value between users and items. If there is a historical interaction, it is 1, and if there is no interaction, it is 0. That is A is the adjacency matrix generated according to the input interaction data, A ∈ R (M+N)*(M+N) ; represents the adjacency matrix updated by the algorithm at the t-th iteration under the k-th interest space; In the routing mechanism iteration, first use the softmax function to obtain the interest distribution in each space. The formula is K = 4, represents the interaction value between user u and item i in the adjacency matrix; then use the LightGCN algorithm to update the node features through different normalized adjacency matrices in each space. The specific formula is is the adjacency matrix corresponding degree matrix, represents the feature vector of the k-th interest space after the t-th iteration; finally, calculate the adjacency value of the updated adjacency matrix through the affinity between the self-node and the first-order neighbor nodes. The specific formula is tanh() is the hyperbolic tangent function; where represents the feature of user u node in the k-th interest space after the t-th iteration, represents the initial feature of neighbor node i of user u after passing through the fully connected layer in the k-th interest space; After t = 2 iterations, it is equivalent to all nodes in the figure passing through 1 layer of LightGCN; finally, the node features of each layer generated after L layers are summed, that is, Y k = Y k (0) + Y k (1) ... + Y k (L-1) , L = 3, Y k (L-1) represents the feature vector of the k-th interest space in the (L - 1)-th layer.

5. A graph convolutional interest decoupling method based on the Transformer model according to claim 4, wherein ； The node features of the same node in different spaces are fed into the Transformer model. The node features under a specific interest type are used as local features, and the node features of other interest types are used as global features to generate self-attention values to update the node features. The specific formula is Among them, is the mapping matrix, h = 0, 1, both are Xavier randomly initialized matrices, H is the number of multi-head self-attention, set to 2, is the attention value of features between different channels, || is the concatenation operation.

6. The graph convolutional interest decoupling method based on the Transformer model according to claim 5, characterized in that ； The node features of different interest spaces are generated through the calculation method in claim 5 and fused into a single feature through a gating module. The formula is as follows: Y = ω1 * Y 1 ... + ω k * Y k Among them, Y = {Y1, Y2,..., Y M+N} is the fused user and item feature, where Y * ∈R d is the feature vector of each node after passing through the model.

7. A graph convolutional interest decoupling method based on the Transformer model according to claim 6, characterized in that ； The loss function includes two parts. The first part is the loss function of the interaction prediction part; the pairwise BPR loss function is selected, and its purpose is to make the scores of the user's historical interaction items higher than the scores of the unobserved items. Minimize the following loss function: y ui = E u * E i where, O = {(u, i, j)|(u, i) ∈ O + , (u, j) ∈ O -} represents the dataset, E u = {Y1...Y M}, E i = {Y M+ 1...Y (M+N)}, O + represents the interactions observed by the user, and O - represents the interactions not observed by the user; σ() is the sigmoid function, λ = 0.01 is the hyperparameter controlling the strength of L2 regularization, aiming to control the overfitting problem of the model; E u and E i are the feature mappings of user u and item i after fusion; in this method, the inner product between the user and the item is used as the interaction prediction scores y ui , y u,j ; θ = {X, Q h , K h , V h , W k , b k} For the other part, to orthogonalize different interest mapping spaces and minimize their correlation with each other, it is necessary to minimize the distance correlation loss function of different spaces, as follows: where dCov() is the distance correlation function of the feature matrices between two spaces, and dVar() is the distance variance function of each feature matrix; Finally, jointly minimize the loss functions Loss of these two parts to optimize the model parameters; Loss=loss BPR +loss inv 。 8. A graph convolutional interest decoupling method based on the Transformer model according to claim 1, characterized in that ； After obtaining the interaction prediction scores for all items to be recommended, for each user, sort all items in descending order according to the scores, and generate lists for the top 20, 40, 60, 80, or 100 items with the highest scores and recommend them to the user.

Images