A recommendation method based on user implicit hierarchical interest

Abstract

The application discloses a recommendation method based on implicit hierarchical interests of users. Most of the existing graph-based recommendations are graph learning in the traditional Euclidean space, but when learning hierarchical structure from the graph, the grid characteristics of the Euclidean space can introduce serious structure distortion. Compared with Euclidean geometry, hyperbolic geometry is suitable for modeling data with potential non-Euclidean topology. The application proposes a recommendation method based on implicit hierarchical interests of users, which first uses a hyperbolic graph neural network to mine the implicit hierarchical interests of users in the interaction graph, and designs a self-distillation loss to embed the hierarchical interests into Euclidean representation learning to improve the reasoning efficiency of the model. The experimental results on multiple data sets verify the effectiveness and reasoning efficiency of the method. For the above method, we use public data sets to train and test respectively to optimize the performance of the model and verify the effectiveness of the method.

Description

TECHNICAL FIELD

[0001] The present application relates to a knowledge distillation model, in particular to a geometric-enhanced self-distillation model; the present application relates to information transmission technology based on a graph convolution network; the present application relates to hyperbolic geometry technology, in particular to Poincaré ball model for representation learning; the present application also relates to data mining and deep learning and other related technologies. BACKGROUND

[0002] With the development of the Internet and the explosive growth of information, users are faced with a vast amount of information and choices, and traditional information retrieval methods have been unable to meet user needs. The recommendation system can filter information for users through personalized recommendation, reduce the distress of information overload, and improve the efficiency of users' information acquisition, so as to better meet the needs of users. Recommendation systems are widely used in e-commerce, social networks, music, video, news and other fields, and have become an important tool to help users discover more valuable information.

[0003] The recommendation paradigm based on collaborative filtering (Collaborative Filtering, CF for short) usually maps users and items into latent space through embedding learning, and predicts user-item interaction scores based on learned embeddings using an interaction function. Early models such as matrix factorization only model single user-item pairs, ignoring the CF signals between interactions. Therefore, recent research has made in-depth study of the structured signals in the interaction graph. They effectively capture high-order CF signals in the user-item interaction graph using graph neural networks, and this technology has been proven to significantly improve recommendation performance. In addition to the traditional flat graph structure, the interaction graph presents a tree-like structure. This tree-like structure is manifested in the power-law degree distribution of all items in the interaction data, so the graph structure exhibits obvious hierarchical structure. Although graph learning in traditional Euclidean space is good at capturing local structure signals in terms of computation, when learning hierarchical structure from the graph, the grid characteristics of Euclidean space will introduce serious structure distortion, such as Figure 1 as shown.

[0004] Compared with Euclidean geometry, hyperbolic geometry is suitable for modeling data with potential non-Euclidean topology, and the advantages of hyperbolic geometry are as follows: 1) Hyperbolic geometry can embed a complex structure graph in a smaller space, which means that hyperbolic embedding can capture structured information with lower dimension representation; 2) Hyperbolic space can extract tree-like structure in the graph and preserve the global hierarchical structure. In order to take advantage of the above advantages, some researchers apply graph neural networks to hyperbolic space. Hierarchical signals are mined from user interactions to complete interest modeling, but these studies are still insufficient to capture complex geometric structures from interaction graphs. Hierarchical signals can only describe part of the user's interest, resulting in suboptimal performance. In addition, the inevitable nonlinear transformation in hyperbolic space brings huge computational cost, which greatly burdens model inference, which is unbearable in real scenarios with millions of users.

[0005] In order to take advantage of the complementarity of the two, an intuitive way is to expand the network structure from Euclidean space and hyperbolic space to learn user interest. In order to make the two cooperate with each other, following the knowledge transfer paradigm, a recommendation method based on user implicit hierarchical interest is proposed. SUMMARY

[0006] The interaction graph data in the recommendation scenario presents a tree-like structure. This tree-like structure is manifested in that all items in the interaction data present a power-law degree distribution, so the graph structure exhibits obvious hierarchical structure. The hierarchical structure in the user-item interaction graph also implies the user's hierarchical interest implied in the structured data. Existing methods mostly use graph convolution networks in Euclidean space to capture user interest, but when they learn the hierarchical graph structure, serious distortion occurs, making it difficult to accurately mine the user's hierarchical interest implied in the structure, and thus affecting the recommendation performance. While other studies combine the ability of hyperbolic space to represent hierarchical structure with graph convolution networks and apply them to the recommendation system to mine hierarchical relationships, but such models often take a long time to infer when learning interactions, which is unbearable in real scenarios with millions of users. The optimization strategy module can not only affect the quality of representation learning to achieve accurate interest modeling, but also affect the form of interaction learning, which plays an important role in model inference efficiency. Therefore, how to effectively integrate the user's hierarchical interest into the optimization strategy is of great significance for accurate modeling of user interest and reducing model inference efficiency

[0007] In order to explore how to integrate the relationship between interests into the optimization strategy, the present application discloses a recommendation method based on user implicit hierarchical interest. The overall framework is as follows Figure 2The method uses the following public datasets: 1) social network: Gowalla, 2) e-commerce platform: Amazon-book, 3) review website: Yelp2018). The method uses the hyperbolic geometry to mine the implicit hierarchical interest as knowledge migration to the Euclidean space of the backbone student network through the parameter sharing of the self-teacher network, and then enriches the collaborative filtering signal to enhance the interest modeling. At the same time, the feature fusion method is abandoned in the representation learning, and a response-based adaptive restriction is designed on the optimization strategy to migrate the hierarchical knowledge in the hyperbolic space to the Euclidean space to improve the efficiency of model reasoning. The invention consists of two key components: 1) interest propagation layer, which uses a graph convolution network (GCN) to extract high-order neighborhood information in the Euclidean space and the hyperbolic space, respectively. 2) prediction and optimization, which uses different space metrics to calculate the interaction score of the user-item pair, and sends them to the recommendation loss and self-distillation loss to realize geometric-enhanced embedding learning in the Euclidean space, and to mine complex geometric structure signals from the interaction graph to learn user interest comprehensively.

[0008] The invention contents of the modules of the method are as follows:

[0009] 1. Interest propagation layer

[0010] Firstly, the representation of users and items is mapped to the hyperbolic space through exponential mapping, and the corresponding interest propagation layer is established as a self-teacher network to naturally capture the hierarchical structure. Secondly, the hyperbolic embedding of users and items is mapped to the tangent space, which is approximately Euclidean space. Then, the traditional GCN aggregation is performed in the tangent space, and the exponential mapping is used to project it back to the hyperbolic space to obtain the hyperbolic representation of interest propagation.

[0011] 2. Prediction and optimization

[0012] After L-layer interest propagation, the embedding of each layer can be obtained in different spaces, which contains different structure information. In order to obtain better quality representation, average pooling is used to fuse the collaborative filtering signals from multiple layers. For hyperbolic embedding, the hyperbolic distance is used to evaluate the similarity between user-item pairs. In order to combine the recommendation task with the structured knowledge migration task, a multi-task training strategy is adopted to optimize the whole model. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 An example diagram of the hierarchical relationship implied in user interaction and the interaction graph structure in Euclidean space and hyperbolic space.

[0014] Figure 2 The overall framework of the recommendation method based on user implicit hierarchical interest of the invention. DETAILED DESCRIPTION

[0015] The application discloses a recommendation method based on implicit hierarchical interests of users.

[0016] Step one: data preprocessing and division of training set and test set:

[0017] Firstly, appropriate data sets are selected: 1) social network: Gowalla, 2) e-commerce platform: Amazon-book, 3) review website: Yelp2018, and the users and items in the data sets are sequentially numbered from 0, and the historical interaction of the user-item is recorded in the form of implicit feedback, that is, if the user has interacted with the item (browsed, purchased, collected, etc.), the label is 1 (positive sample), and if no interaction has occurred, the label is 0 (negative sample). On the three data sets, for each user, the interaction records are randomly divided into training set and test set according to the ratio of 4:1. The observed user-item interaction is regarded as a positive example, and the unobserved interaction is regarded as a negative example.

[0018] Step two: hyperbolic space interest propagation:

[0019] Firstly, the representation of the user and the item is mapped to the hyperbolic space by exponential mapping, and the corresponding interest propagation layer is established as a self-teaching network to naturally capture the hierarchical structure. u i u i} and the corresponding interest propagation layer is established as a self-teaching network to naturally capture the hierarchical structure. The interest propagation mechanism in the hyperbolic space is defined as follows:

[0020]

[0021] Wherein, N u represents the neighbor node set of the user u, N i represents the adjacent node set of the item i, and respectively represent the hyperbolic embedding of the user u and the item i in the lth interest propagation layer. g B (·) represents the interest propagation operation in the d-dimensional hyperbolic space B d , which extracts and aggregates the information of the target node and its neighborhood in the hyperbolic manifold.

[0022] The specific process of the aggregation operation g B (·) is as follows:

[0023]

[0024] ​​​where exp0and log0are the exponential and logarithmic mapping functions described above. To avoid complex averaging operations in hyperbolic space, we first map the user and item hyperbolic embeddings to the tangent space, which is approximately Euclidean. Then, we perform the traditional GCN aggregation in the tangent space and project it back to the hyperbolic space using the exponential mapping to obtain the hyperbolic representation at the l-th layer of interest propagation and Therefore, this aggregation operation can incorporate neighborhood information into the tangent representation in hyperbolic space through a space mapping function, while avoiding complex hyperbolic average pooling.

[0025] Step three: layer combination and prediction:

[0026] After L layers of interest propagation, we can obtain the embedding at each layer under different spaces, which contains different structural information. To obtain better quality representation, we use average pooling to fuse the collaborative filtering signals from multiple layers. When performing layer combination in hyperbolic space, we use the same mapping strategy as the interest propagation module to extract information. In addition, this method removes the feature vector of the initial input layer and retains the skip-connection structure from all intermediate layers to the final layer to obtain better recommendation performance. The layer combination operation under different spaces is represented as:

[0027]

[0028]

[0029] where, and represent the final embedding of user u and item i in Euclidean and hyperbolic space, respectively.

[0030] Step four: recommendation prediction:

[0031] For Euclidean embedding, we use inner product operation as the prediction function to measure the similarity between users and items, which is widely used in collaborative filtering tasks. For hyperbolic embedding, we use hyperbolic distance to evaluate the similarity between user-item pairs. Specifically, the similarity between two embeddings is the inverse of the geodesic distance connecting the two points:

[0032]

[0033]

[0034] where d B is the geodesic distance function described above. and represent the prediction scores in Euclidean and hyperbolic space, respectively.

[0035] Step five: self-distillation and model training:

[0036] To combine the recommendation task with the structured knowledge transfer task, a multi-task training strategy is used to optimize the entire model. For the recommendation task, the pairwise Bayesian personalized ranking (BPR) is adopted as the main loss function, which encourages the score of user u and the positive example item i interacting with it to be higher than the unobserved negative example item j.

[0037]

[0038] where O = {(u, i, j) | (u, i) ∈ O + ,(u, i) ∈ O -}, O + represents the observed interaction set between user u and item i, O - represents the unobserved interaction set between user u and item j. σ is the sigmoid function.

[0039] Intuitively, the predicted score directly reflects the user's preference for the item, regardless of the inconsistency in space. Therefore, inspired by response-based knowledge distillation, an adaptive self-distillation loss is designed as an auxiliary task, which aims to transfer the missing hierarchical information from the hyperbolic space to the Euclidean space by taking the predicted score in the hyperbolic space as a soft label, so that the score in the Euclidean space can not only reflect the hierarchical structure in the data, but also strengthen the score distribution in the Euclidean space through joint learning with the main function. Specifically, the score response-based geometric-enhanced self-distillation is realized by minimizing the following loss function:

[0040]

[0041] where σ is the sigmoid function. τ is the temperature factor, which is used to adjust the output of the sigmoid function, thereby affecting the final recommendation result. By combining the adaptive self-distillation loss with the BPR loss, the model parameters are optimized by minimizing the following objective function:

[0042] loss = loss Rec + λloss SD

[0043] λ controls the self-distillation loss adjustment of this parameter to minimize the loss.

[0044] Step six: verify the effectiveness of the method:

[0045] After the model training is completed, in order to verify the effectiveness of the method, the method is used in the public data set: 1) social network: Gowalla, 2) e-commerce platform: Amazon-book, 3) review website: Yelp2018). After obtaining the interaction prediction score of the target user to the recommended item, for each target user, the top 20, 40, 60, 80, 100 items with the highest score are selected to form a Top-N personalized recommendation list. The evaluation index used in the experiment is Recall (correctness relative to all positive samples), Precision (correctness relative to Top-N list), Hit Rate (probability of hitting any positive sample in Top-N list) and Normalized Discounted Cumulative Gain (NDCG, which focuses on the position of the positive sample in the Top-N recommendation list. The earlier the position is, the larger the NDCG is).

Claims

1. A recommendation method based on users' implicit hierarchical interests; characterized in that: Includes the following steps: S1. Data Preprocessing: Obtain the dataset, including user IDs, item IDs, and user-item interaction labels; divide it into training set and test set; S2. Interest propagation: First, the representations of users and items are mapped to hyperbolic space through exponential mapping, and a corresponding interest propagation layer is established as a self-teacher network to naturally capture hierarchical structures; Second, the hyperbolic embeddings of users and items are mapped to tangent space, which approximates Euclidean space. Subsequently, traditional GCN aggregation is performed in the tangent space, and exponential mapping is used to project it back into the hyperbolic space to obtain the hyperbolic representation of interest propagation. S3, Inter-layer aggregation: After interest propagation through L layers, L=3, the embeddings of each layer are obtained in different spaces, and these embeddings contain different structural information; average pooling is used to fuse the collaborative filtering signals from multiple layers; when combining layers in hyperbolic space, the same mapping strategy as the interest propagation module is used to extract information; the feature vectors of the initial input layer are removed, and the skip connection structure from all intermediate layers to the final layer is preserved; S4. Recommendation Prediction: For hyperbolic embeddings, hyperbolic distance is used to evaluate the similarity between user-item pairs; the similarity between two embeddings is the reciprocal of the geodesic distance connecting the two points; S5. Optimization Method and Loss Function: The loss function consists of two parts. One part is the BPR loss function used for recommendation prediction and the L2 regularization term composed of initialization parameters. The other part is the adaptive self-distillation loss, which is jointly optimized. S6. Generate a recommendation list: After obtaining the predicted user-item interaction score, for each user, sort all items in descending order according to the score, and generate a list of the top N items to recommend to the user.

2. The recommendation method based on users' implicit hierarchical interests according to claim 1, characterized in that, User-item interaction records are recorded using implicit feedback: if a user interacts with an item, the label is 1 (positive sample), and if not, it is 0 (negative sample). For each dataset, the ratio of training set to test set is 4:

1.

3. The recommendation method based on users' implicit hierarchical interests according to claim 1, characterized in that, Representations of users and items {E} through exponential mapping u E i Mapped to hyperbolic space {H u H i }, and establish a corresponding interest propagation layer as a self-teacher network to naturally capture hierarchical structures; define the interest propagation mechanism in hyperbolic space as follows: in, and The hyperbolic embeddings representing user u and item i in the l-th interest propagation layer, respectively; g B (·) represents the hyperbolic space B in d-dimensional space. d The interest propagation operation in N extracts and aggregates information about the target node and its neighborhood in the hyperbolic manifold; u N represents the set of neighboring nodes of user u, consisting of the ID information of neighboring users. i This represents the set of adjacent nodes of item i, consisting of the ID information of neighboring items; Aggregate operation g B The specific process of (·) is as follows: Here, exp0 and log0 are the exponential and logarithmic mapping functions mentioned above. To avoid complex averaging operations in the hyperbolic space, the hyperbolic embeddings of users and items are first mapped to the tangent space, which approximates Euclidean space. Then, traditional GCN aggregation is performed in the tangent space, and the exponential mapping is used to project it back into the hyperbolic space to obtain the hyperbolic representation of interest propagation at the l-th layer. and 4. The recommendation method based on users' implicit hierarchical interests according to claim 1, characterized in that, Layer combination operations in different spaces are represented as follows: in, and This represents the final embedding of user u and item i in Euclidean space and hyperbolic space, respectively.

5. The recommendation method based on users' implicit hierarchical interests according to claim 1, characterized in that, For Euclidean embeddings, the inner product is used as the prediction function to measure the similarity between users and items; this is widely used in collaborative filtering tasks. For hyperbolic embeddings, hyperbolic distance is used to evaluate the similarity between user-item pairs. Specifically, the similarity between two embeddings is the reciprocal of the geodesic distance connecting the two points. Where, d B It is the geodesic distance function mentioned above; and These represent the prediction scores in Euclidean space and hyperbolic space, respectively.

6. The recommendation method based on users' implicit hierarchical interests according to claim 1, characterized in that, For recommendation tasks, Paired Bayesian Personalized Ranking (BPR) is used as the main loss function, which encourages user u and the positive sample item i that they interact with to score higher than the unobserved negative sample item j. Among them, O={(u,i,j)|(u,i)∈O + ,(u,i)∈O - }, O + P represents the set of interactions observed between user u and item i. - Represents the set of unobserved interactions between user u and item j; This represents the model's interactive prediction score for positive examples in Euclidean space. σ represents the model's interactive prediction score for negative examples in Euclidean space; σ is the sigmoid function.

7. The recommendation method based on users' implicit hierarchical interests according to claim 1, characterized in that, Geometrically enhanced self-distillation based on score response is achieved by minimizing the following loss function: Where σ is the sigmoid function; This represents the model's interactive prediction score for positive examples in hyperbolic space. This represents the model's interactive prediction score for negative examples in hyperbolic space; This represents the model's interactive prediction score for positive examples in Euclidean space. This represents the model's interactive prediction score for negative examples in Euclidean space; τ is a temperature factor used to adjust the output of the sigmoid function, thus affecting the final recommendation result; the sigmoid function transforms the model's original prediction score into a probability distribution, and the temperature factor τ is used to control this transformation process. By combining the adaptive self-distillation loss with the BPR loss, the following objective function is minimized to optimize the model parameters: loss=loss Rec +λloss SD Here, λ controls the self-distillation loss, and this parameter is adjusted to minimize the loss. In order to systematically find the optimal values ​​of λ and τ, a grid search algorithm is used. Within a predefined range, different combinations of λ and τ values ​​are tried, and the combination of λ and τ with the lowest verification loss is selected.

8. The recommendation method based on users' implicit hierarchical interests according to claim 1, characterized in that, After calculating the interaction prediction scores for all recommended items, for each user, all items are sorted in descending order according to their scores, and a list of the top N items is generated and recommended to the user.

Images