Recommendation system based on training and / or operation graph neural network

Abstract

The invention discloses a method for training a recommendation system based on a graph neural network. The method includes receiving and processing a data set to generate graph characterization data. The dataset includes user data associated with a user, item data associated with an item, and user-item interaction data associated with an interaction between the user and the item. The method further includes processing the graph characterization data to obtain modified graph characterization data, which helps to learn or determine representations of at least some of the item nodes with less user-item interaction. The method further includes training a graph neural network-based recommendation system based at least in part on the modified graph characterization data. Some embodiments of the present invention provide a more effective, more efficient and / or more practical solution for improving the recommendation performance of tail items in user behavior long-tail distribution.

Description

Technical Field

[0001] The present invention generally relates to the training and / or operation of recommender systems based on graph neural networks. Background Art

[0002] Machine learning-based recommender systems are information processing and filtering systems that use machine learning techniques to provide suggestions or recommend items to users. One problem associated with some existing machine learning-based recommender systems is that they can be susceptible to biases such as popularity bias. Reducing such biases can improve the fairness and overall performance of machine learning-based recommender systems. Summary of the invention

[0003] In a first aspect, the present invention provides a computer-implemented method for training a recommendation system based on a graph neural network. The method includes receiving a data set. The data set includes user data associated with a user, item data associated with an item, and user-item interaction data associated with a user-item interaction (i.e., an interaction between a user and an item), wherein some items have less user-item interaction than some other items. The method also includes processing the data set to generate graph representation data of the data set. The graph representation data includes data associated with nodes and data associated with edges connecting the nodes, the nodes include user nodes and item nodes, the edges include user-item interaction edges, each edge is connected to an item node and a user node, wherein some item nodes have less user-item interaction edges than some other item nodes. The method further includes processing the graph representation data of the data set to modify (or enhance) the graph representation data of the data set to obtain modified graph representation data. The modified graph representation data can facilitate learning or determining the representation of at least some item nodes and reduce the interaction between users and items. The method further includes training one or more graph neural network-based recommendation models of a recommendation system based on a graph neural network based at least in part on the modified graph representation data.

[0004] The interaction between users and items may include feedback provided by users to items (such as ratings, scores, etc.). Some users may provide feedback on more items; some users may provide feedback on fewer items.

[0005] Representations may correspond to embeddings, the learning or determination of which may be performed by one or more graph neural network based recommendation models of a graph neural network based recommendation system.

[0006] In the data set, the distribution of user-item interactions is unbalanced or skewed. In some embodiments, in the data set, the amount of user-item interactions associated with items is generally heavy-tailed or long-tailed, such that some items have much higher user interactions than other items.

[0007] In some embodiments, processing the graph representation data of the data set includes performing an edge addition operation to add one or more edges to the graph representation data.

[0008] In some embodiments, performing an edge adding operation includes: performing a homogenous edge adding operation to add one or more item-item edges to the graph representation data for at least some item nodes with fewer user-item interaction edges. Each of the one or more item-item edges is respectively connected to two item nodes, at least one of the two item nodes is one of the item nodes with fewer user-item interaction edges.

[0009] In some embodiments, performing a homogeneous edge adding operation includes, for each of at least some item nodes with fewer user-item interactions, determining one or more structurally adjacent item nodes from the item nodes, respectively, and adding one or more item-item edges between the corresponding item node and its one or more structurally adjacent item nodes to the graph representation data.

[0010] In some embodiments, determining one or more structural adjacent item nodes includes: determining an item similarity matrix based on user-item interactions (the item similarity matrix includes common interaction values ​​of two respective items), and determining one or more structural adjacent item nodes based on the common interaction values ​​in the item similarity matrix.

[0011] In some embodiments, performing a homogeneous edge adding operation includes, for each of at least some item nodes with fewer user-item interactions, determining one or more semi-neighboring item nodes from the item nodes, respectively, and adding one or more item-item edges between the corresponding item node and its one or more semi-neighboring item nodes to the graph representation data.

[0012] In some embodiments, determining one or more semi-adjacent item nodes includes: clustering the items according to a clustering method to form a plurality of item clusters. The adding includes adding one or more item-item edges between the corresponding item node and an item in the same cluster as the corresponding item node.

[0013] In some embodiments, clustering is based on a K-means method or a mean shift method.

[0014] In some embodiments, performing a homogeneous edge adding operation includes performing a message passing operation based at least in part on the graph representation data that adds one or more item-item edges between a corresponding item node and one of its one or more structural neighboring item nodes and one or more item-item edges between the corresponding item node and one of its one or more semi-structural neighboring item nodes.

[0015] In some embodiments, processing the graph representation data of the data set includes or further includes: performing an edge drop operation to drop one or more user-item interaction edges. In some embodiments in which an edge add operation and an edge drop operation are performed simultaneously, the edge drop operation can be performed after the edge add operation.

[0016] In some embodiments, performing an edge discarding operation includes: performing an adaptive heterogeneous edge discarding operation to discard some user-item interaction edges in such a manner that the number of user-item interaction edges discarded by at least some item nodes with less user-item interaction is less than the number of user-item interaction edges discarded by item nodes with more user-item interaction.

[0017] In some embodiments, performing an adaptive heterogeneous edge discarding operation includes, for each of at least some item nodes, determining the degree to which the item node has insufficient user interaction, and discarding user-item interaction edges associated with the item node according to the determined degree, thereby discarding more user-item interaction edges for item nodes with less insufficient user interaction, and discarding fewer user-item interaction edges for item nodes with more insufficient user interaction.

[0018] In some embodiments, processing the graph representation data of the data set further includes: performing a node synthesis operation to add at least one or more synthetic item nodes to the graph representation data. In some embodiments, the node synthesis operation also adds one or more corresponding synthetic user-item interaction edges to the graph representation data.

[0019] In some embodiments, performing a node synthesis operation includes: processing graph representation data of a data set using one or more graph neural network-based recommendation models of a graph neural network-based recommendation system to determine embeddings associated with user nodes and embeddings associated with item nodes; performing a data blending operation to generate one or more synthetic item nodes based at least in part on the embeddings associated with the user nodes and the embeddings associated with the item nodes; and for each of the one or more synthetic item nodes, generating a corresponding synthetic user-item interaction edge based at least in part on a hyperparameter.

[0020] In some other embodiments, performing a node synthesis operation includes processing graph representation data of a dataset using one or more graph neural network-based recommendation models of a graph neural network-based recommendation system to determine a set of data including user-item-interaction triples and corresponding embeddings associated with the graph representation data; performing a first data augmentation operation on the set of data to generate a first synthetic dataset having one or more synthetic user-item-interaction triples and one or more corresponding embeddings; processing the dataset and the first synthetic dataset using a bilateral branching network model to compare or determine the performance of the item node; based on the comparison or determination, selecting the item node to perform data augmentation; and performing a second data augmentation operation on the dataset for the selected item node to generate a second synthetic dataset including one or more synthetic item nodes and one or more corresponding synthetic user-item interaction edges. In some embodiments, training includes training one or more graph neural network-based recommendation models of a graph neural network-based recommendation system using a combination of the dataset and the second synthetic dataset.

[0021] In some embodiments, the bilateral branch network model includes two substantially identical branches, each branch including one or more graph neural network-based recommendation models. In some embodiments, processing the dataset and the first synthetic dataset includes: processing the dataset using one branch of the bilateral branch network model, and processing the combination of the dataset and the first synthetic dataset using another branch of the bilateral branch network model.

[0022] In some embodiments, the first data augmentation operation includes a data blending operation and / or a data resampling operation. In some embodiments, the data blending operation includes blending one or more pairs of user-item-interaction triples and their corresponding embeddings to generate synthetic data. In some embodiments, the data resampling operation includes selectively drawing and discarding at least some user-item-interaction triples and corresponding embeddings from the dataset.

[0023] In some embodiments, the second data augmentation operation includes a data blending operation and / or a data resampling operation. In some embodiments, the data blending operation includes blending one or more pairs of user-item-interaction triples and their corresponding embeddings to generate synthetic data. In some embodiments, the data resampling operation includes selectively extracting and discarding at least some user-item-interaction triples and corresponding embeddings from the dataset.

[0024] In some embodiments, the training includes training one or more graph neural network-based recommendation models of a graph neural network-based recommendation system using a graph contrastive learning technique. In some embodiments, the graph contrastive learning technique includes an enhanced contrastive learning technique, which includes graph representation data modification and noise injection. In some embodiments, the training includes optimizing a loss function that takes into account recommendation loss, regularization loss, and contrast loss.

[0025] In some embodiments, the training includes: in a first training phase, using the graph representation data to train one or more graph neural network-based recommendation models of the graph neural network-based recommendation system; in a second training phase, using the modified graph representation data to train one or more graph neural network-based recommendation models of the graph neural network-based recommendation system. In some embodiments, the training includes optimizing a loss function that takes into account at least a recommendation loss and a regularization loss.

[0026] In a second aspect of the present invention, a system for training a recommendation system based on a graph neural network is provided, the system comprising: one or more processors, and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising instructions for executing or facilitating the execution of the computer-implemented method of the first aspect.

[0027] In a third aspect of the present invention, a non-transitory computer-readable storage medium is provided, in which one or more programs are stored, which are configured to be executed by one or more processors, and the one or more programs include instructions for performing or facilitating the execution of the computer-implemented method of the first aspect.

[0028] In a fourth aspect of the present invention, a computer-implemented method for operating a graph neural network-based recommendation system is provided, comprising: one or more graph neural network-based recommendation models of the graph neural network-based recommendation system trained by the computer-implemented method of the first aspect process user data and item data to determine item recommendations for the user. In some embodiments, the computer-implemented method further includes presenting (e.g., displaying) the item recommendations. The item recommendations may include one or more items recommended to the user.

[0029] In a fifth aspect of the invention, there is provided a system for operating a neural network-based ranking model, the system comprising: one or more processors, and a memory storing one or more programs, the one or more programs being configured to be executed by the one or more processors, the one or more programs comprising instructions for performing or facilitating the performance of the computer-implemented method of the fourth aspect. In some embodiments, the system further comprises a display.

[0030] In a sixth aspect of the present invention, a non-transitory computer-readable storage medium is provided, in which one or more programs are stored, which are configured to be executed by one or more processors, and the one or more programs include instructions for executing or facilitating the execution of the computer-implemented method of the fourth aspect.

[0031] Other features and aspects of the present invention will become apparent from the following detailed description of specific embodiments and the accompanying drawings. Any feature described herein in relation to one aspect or embodiment may be combined with any other feature described herein in relation to any other aspect or embodiment, as long as appropriate and applicable.

[0032] Terms of degree, such as "generally," "approximately," "substantially," or the like, are used as the context requires to account for manufacturing tolerances, degradation, trends, tendencies, imperfect physical conditions, etc. For example, "generally increasing" refers to a general trend of increasing (not necessarily strictly or monotonically increasing). For example, "generally decreasing" refers to a general trend of decreasing (not necessarily strictly or monotonically decreasing).

[0033] The term "item" as used in this document, unless otherwise specified, generally refers to information items such as digital images, photographs, electronic documents or files, web pages, portions of web pages, maps, electronic links, commercial products, non-commercial products, multimedia files, songs, books, albums, articles, database records, people, objects, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Embodiments of the present invention will now be described, by way of example, with reference to the accompanying drawings.

[0035] Figure 1A The following is a diagram of the graph representation of a dataset in an example.

[0036] Figure 1B The following is a diagram of the graph representation of a dataset in an example.

[0037] Figure 2 It is a schematic diagram of a recommendation system training framework based on graph neural network in certain embodiments of the present invention.

[0038] Figure 3 It is a flowchart of a recommendation system training method based on graph neural network in certain embodiments of the present invention.

[0039] Figure 4 is a flowchart of a method for modifying graph representation data in certain embodiments of the present invention.

[0040] Figure 5 is a flowchart of a method for modifying graph representation data in certain embodiments of the present invention.

[0041] Figure 6is a flow chart illustrating a method for modifying graph representation data in certain embodiments of the present invention.

[0042] Figure 7 is a flow chart illustrating a method for modifying graph representation data in certain embodiments of the present invention.

[0043] Figure 8 is a flow chart illustrating a method for modifying graph representation data in certain embodiments of the present invention.

[0044] Fig. 9A is a flow chart illustrating a method for modifying graph representation data in certain embodiments of the present invention.

[0045] Fig. 9B is a flow chart illustrating a method for modifying graph representation data in certain embodiments of the present invention.

[0046] Fig. 10A It is a flowchart illustrating a training method of a recommendation system based on a graph neural network in certain embodiments of the present invention.

[0047] Fig. 10B It is a schematic diagram of a training method for a recommendation system based on a graph neural network in certain embodiments of the present invention.

[0048] Fig.11 It is a schematic diagram of a framework based on graph contrastive learning and adaptive enhancement of long-tail recommendation (GALR) in certain embodiments of the present invention.

[0049] Fig.12 is a schematic diagram illustrating the message passing operation in GALR in certain embodiments of the present invention.

[0050] Fig.13 is a schematic diagram illustrating the training data augmentation operation in GALR in certain embodiments of the present invention.

[0051] Fig.14 is a table listing the details of four datasets (ML-1M, Douban Books, Yelp2018, and Amazon Books) used in some example experiments.

[0052] Fig.15A is a table listing the recommendation performance of GALR embodiments and other existing methods on two datasets (ML-1M and Douban Books) in some example experiments.

[0053] Fig. 15B is a table listing the recommendation performance of GALR embodiments and other existing methods on two other datasets (Yelp2018 and Amazon books) in some example experiments.

[0054] Fig.16 is a table listing the recommendation performance of GALR implementations on the ML-1M dataset for some ablation study experimental examples.

[0055] Fig.17 is a table listing the performance of GALR embodiments on the ML-1M dataset for different edge drop rates in some example experiments.

[0056] Fig.18 is a table listing the performance of GALR embodiments on the ML-1M dataset for different numbers of clusters in some example experiments.

[0057] Fig.19 is a table listing the performance of GALR embodiments on the ML-1M dataset for different numbers of structural neighbors in some example experiments.

[0058] Fig. 20A is a graph showing the recommendation performance of all items on the ML-1M dataset at different embedding dimensions in some example experiments.

[0059] Fig. 20B is a graph showing the recommendation performance of head items in the ML-1M dataset at different embedding dimensions in some example experiments.

[0060] Fig. 20C is a graph showing the recommendation performance of tail items in the ML-1M dataset with different embedding dimensions in some example experiments.

[0061] Fig.21A is a two-dimensional (2D) visualization showing the tail and head item embeddings obtained by LightGCN on the ML-1M dataset in an example experiment.

[0062] Fig.21B is a 2D visualization showing the tail and head item embeddings obtained by a GALR embodiment on the ML-1M dataset in an example experiment.

[0063] Fig. 22 Schematic diagram of a graph-enhanced long-tail recommendation framework (GALORE) in certain embodiments of the present invention.

[0064] Fig.23 is a schematic diagram illustrating the message passing operation in GALORE in certain embodiments of the present invention.

[0065] Fig.24 is a table listing the details of four datasets (ML-1M, Douban Books, Yelp2018, and Gowalla) used in some example experiments.

[0066] Fig.25A is a graph illustrating the cumulative distribution function (CDF) of the distribution of items in the ML-1M dataset used in some example experiments.

[0067] Fig.25B This is a CDF plot of the item distribution in the Douban Books dataset, used in some example experiments.

[0068] Fig.25C is a graph illustrating the CDF of the distribution of items in the Yelp2018 dataset used in some example experiments.

[0069] Fig.25D is a graph illustrating the CDF of the distribution of items in the Gowalla dataset used in some example experiments.

[0070] Fig.26A This is a table listing the recommendation performance of the GALORE invention and other existing methods on two datasets (ML-1M and Douban Books) in some example experiments.

[0071] Fig.26B is a table listing the recommendation performance of the GALORE embodiment and other existing methods on two datasets (ML-1M and Gowalla) in some example experiments.

[0072] Fig. 27 is a table that lists the recommendation performance of the GALORE implementation on the Douban Books dataset in some ablation study experimental examples.

[0073] Fig.28 is a table listing the performance of the GALORE implementation on the Douban Books dataset at different edge drop rates in some example experiments.

[0074] Fig.29A A graph showing the recommendation performance (metrics: recall and normalized discounted cumulative gain (NDCG)) of a GALORE embodiment for different synthesis node rates for all items (head and tail items) in an example experiment.

[0075] Fig.29B is a graph showing the recommendation performance (metrics: call rate and normalized discounted cumulative gain) of the GALORE embodiment for different synthesis node rates of the head item in an example experiment.

[0076] Fig.29C A graph showing the recommendation performance (metrics: call rate and normalized discounted cumulative gain) of the GALORE embodiment for different synthesis node rates of tail items in an example experiment is shown.

[0077] Fig. 30A A graph showing the recommendation performance (metrics: recall and normalized discounted cumulative gain) of a GALORE implementation for different embedding dimensions for all items (head and tail items) in an example experiment.

[0078] Fig. 30B A graph showing the recommendation performance (metrics: recall rate and normalized discounted cumulative gain) of a GALORE embodiment for different embedding dimensions of the head items in an example experiment is shown.

[0079] Fig. 30C A graph showing the recommendation performance (metrics: recall rate and normalized discounted cumulative gain) of the GALORE implementation for different embedding dimensions of tail items in an example experiment.

[0080] Fig.31 A schematic diagram of an exemplary data processing system is shown, which can be used to perform one or more methods in embodiments of the present invention. DETAILED DESCRIPTION

[0081] Figure 1AA graph representation 100A of a dataset related to recommendations in an example is shown. The dataset related to recommendations includes user data associated with three users, item data associated with four items, and user-item interaction data associated with interactions between users and items, such as feedback provided by users on items (such as ratings, scores, etc.). In this example, the amount of feedback provided by different users is different. The graph representation uses nodes and edges (lines connecting two nodes) to represent datasets related to recommendations. Specifically, the graph representation includes three user nodes U1, U2, U3 representing users, four item nodes I1, I2, I3, I4 representing items, and multiple user-item interaction edges connecting user nodes U1, U2, U3 and item nodes I1, I2, I3, I4 and representing interactions between users and items. In this example, the user represented by user node U1 interacted with all four items represented by item nodes I1, I2, I3, and I4, so there is an edge between user node U1 and item node I1, an edge between user node U1 and item node I2, an edge between user node U1 and item node I3, and an edge between user node U1 and item node I4. In this example, the user represented by user node U2 only interacted with one item represented by item node I1, so there is an edge between user node U2 and item node I1. In this example, the user represented by user node U3 interacted with two items represented by item nodes I1 and I4, so there is an edge between user node U3 and item node I1, and an edge between user node U3 and item node I4. Although not shown, attributes or information related to users can be embedded in user nodes U1, U2, U3; attributes or information related to items can be embedded in item nodes I1, I2, I3, I4; attributes or information related to user-item interactions can be embedded in user-item interaction edges. The graph representation 100A can be used as or form a graph, such as an input graph, for processing by a graph neural network based recommendation system.

[0082] Figure 1B A diagram 100B is shown showing an example of recommending related data sets. In this example, the recommended related data sets are Figure 1A The data set in is the same, that is, the graph representation method 100B is another method to represent the same recommendation-related data set.

[0083] It should be understood that the graph representations 100A and 100B are only examples. In other examples, the dataset related to the recommendation may have a different number of users (not limited to three), a different number of items (not limited to four), and / or a different number of user-item interactions than the illustrated diagram, and thus have a different graph representation than the illustrated diagram.

[0084] Figure 2 A recommendation system training framework 200 based on a graph neural network (GNN) in certain embodiments of the present invention is shown. The framework 200 applies various computer-implemented processing and / or training strategies to improve the performance of recommendation systems based on graph neural networks. These strategies include graph enhancement 202 (which may include graph topology enhancement 202A and / or graph data enhancement 202B), graph comparative learning 204, multi-stage training 206, and the like. These strategies can be used to address problems such as insufficient or unbalanced available training data (such as long-tail or heavy-tail data) or popularity deviations to improve the fairness or overall performance of recommendation systems based on graph neural networks. In some examples, all of these strategies are applied to collaboratively improve the fairness or overall performance of recommendation systems based on graph neural networks. In other examples, only one or some (not all) of these strategies are applied to improve the fairness or overall performance of recommendation systems based on graph neural networks.

[0085] Figure 3 A general method 300 for training a graph neural network based recommendation system in some embodiments of the present invention is shown. The method 300 is a computer-implemented method.

[0086] The method 300 includes receiving a data set, in particular a data set related to recommendation, in step 302, including user data, item data, and user-item interaction data, such as reference Figure 1A The data set described herein includes user data related to users. Item data includes data related to items. User-item interaction data includes data related to user-item interaction (i.e., interaction between users and items). User-item interaction can be feedback (such as ratings, scores, etc.) provided by users to items. In some examples, some users may provide feedback on more items, while some users may provide feedback on fewer items. In other words, some items may have less user-item interaction (or user interaction) than other items. Therefore, the distribution of user-item interaction in terms of items may be unbalanced or skewed. In some examples, the amount of user-item interaction related to items generally follows a heavy-tail or long-tail distribution, so that the amount of user interaction for some items (e.g., more feedback from more users) is much greater than the amount of user interaction for some other items.

[0087] In step 304, method 300 also includes processing the data set to generate graph representation data of the data set. The graph representation data includes data associated with nodes and data associated with edges connecting the nodes. Specifically, the nodes include user nodes and item nodes, and the edges include user-item interaction edges, each edge connected to an item node and a user node. The user nodes can represent users in the data set. The item nodes can represent items in the data set. The user-item interaction edges can represent user-item interactions in the data set. In some examples, because the distribution of user-item interactions in terms of items may be unbalanced or skewed, some item nodes may have fewer user-item interaction edges than some other item nodes.

[0088] Method 300 also includes modifying the graph representation data of the data set in step 306 to obtain modified graph representation data of the data set. Modifying the graph representation data of the data set helps to improve the representation of the data set. In some embodiments, the modification of the graph representation data of the data set can facilitate learning or determining representations or embeddings associated with nodes, examples of which are item nodes with less user-item interactions. In some examples, the learning or determination of representations or embeddings associated with at least some item nodes with less user-item interactions can be performed by one or more graph neural network-based recommendation models of the graph neural network-based recommendation system.

[0089] The method 300 further includes, in step 308, training one or more graph neural network-based recommendation models of the graph neural network-based recommendation system based at least in part on the modified graph representation data. In some embodiments, the training is further based on at least in part on the graph representation data.

[0090] Figure 4 A method 400 for modifying graph representation data in some embodiments of the present invention is shown. In some embodiments, the method 400 may be considered as belonging to Figure 2 Graph topology enhancement operations in 202A or Figure 3 Step 306 of method 300. Method 400 includes, in step 402, processing graph representation data by performing an edge addition operation, e.g. Figure 3The graph representation data obtained in step 304 of method 300 is added to the graph representation data to form the modified graph representation data. In some embodiments, the edge adding operation includes a homogeneous edge adding operation to add one or more item-item edges (each edge is connected to two item nodes), for example, for at least some item nodes with fewer user-item interaction edges. In some examples, each item-item edge is connected to two item nodes, wherein at least one item node may be an item node with fewer user-item interaction edges. In some embodiments, the edge adding operation is a homogeneous edge adding operation to add one or more user-user edges (each user-user edge is connected to two user nodes), for example, adding a user-user edge for at least some user nodes with fewer user-item interaction edges. In some examples, each user-user edge is connected to two user nodes, wherein at least one user node may be a user node with fewer user-item interaction edges. In some embodiments, performing the homogeneous edge adding operation includes: performing a message passing operation based at least in part on the graph representation data added together with the one or more item-item edges. In some embodiments, performing a homogeneous edge adding operation includes performing a message passing operation based at least in part on the graph representation data added along with the one or more user-user edges.

[0091] Figure 5 A method 500 for modifying graph representation data in some embodiments of the present invention is shown. In some embodiments, the method 400 may be considered as belonging to Figure 4 Step 402 of method 400.

[0092] Method 500 includes, in step 502A, determining a respective structurally neighboring item node for each of at least some item nodes (e.g., at least some item nodes with few user-item interactions). Method 500 also includes, in step 504A, adding a respective item-item edge between the item node and one of its structurally neighboring item nodes for each of at least some item nodes (e.g., at least some item nodes with few user-item interactions). In some examples, determining the structurally neighboring item nodes includes: determining an item similarity matrix based on the user-item interactions, the similarity matrix including common interaction values ​​of two respective items, and determining the structurally neighboring item nodes based on the common interaction values ​​in the item similarity matrix.

[0093] Method 500 includes, in step 502B, determining a respective semantically adjacent item node for each of at least some item nodes (e.g., at least some item nodes with less user-item interactions). Method 500 also includes, in step 504B, adding a respective item-item edge between the item node and one of its semantically adjacent item nodes for each of at least some item nodes (e.g., at least some item nodes with less user-item interactions). In some examples, the determination of semantically adjacent item nodes may include clustering items according to a clustering method (e.g., K-means, mean shift, etc.) to form item clusters. The adding in step 504B may include adding an item-item edge between the corresponding item node and an item in the same cluster as the corresponding item node.

[0094] In some embodiments, method 400 further includes performing a message passing operation based at least in part on the graph representation data of the item-item edge added between the item node and one of its structural neighboring item nodes and the item-item edge added between the item node and one of its semantic neighboring item nodes.

[0095] In some embodiments of the method 500, steps 502A, 504A and steps 502B, 504B are all performed. In some embodiments of the method 500, steps 502A, 504A are performed, and steps 502B, 504B are not performed. In some embodiments of the method 500, steps 502A, 504A are not performed, and steps 502B, 504B are performed. The modified graph representation data obtained from the method 500 can be used to facilitate the training of one or more graph neural network-based recommendation models of a graph neural network-based recommendation system.

[0096] Figure 6 A method 600 for modifying graph representation data in some embodiments of the present invention is shown. In some embodiments, the method 600 may be considered as belonging to Figure 2 Graph topology enhancement operations in 202A or Figure 3 Method 600 includes, in step 602, processing the graph representation data by performing an edge discarding operation, such as in Figure 3 The graph representation data obtained in step 304 of method 300 or in an edge adding operation (e.g. Figure 4 and Figure 5In some embodiments, the edge discarding operation is an adaptive heterogeneous edge discarding operation to discard some user-item interaction edges, so that the number of user-item interaction edges discarded by at least some item nodes with less user-item interaction is less than the number of user-item interaction edges discarded by item nodes with more user-item interaction. The edges of nodes with more user-item interaction edges are more likely to be discarded.

[0097] Figure 7 A method 700 for modifying graph representation data in some embodiments of the present invention is shown. In some embodiments, the method 700 may be considered as belonging to Figure 6 Step 602 in method 600.

[0098] Method 700 includes, at step 702, determining, for each of at least some of the item nodes, a user-item interaction edge to be discarded. In some embodiments, such determination may include determining a degree to which each respective item node is insufficiently user interactive. The degree may be directly or indirectly related to (not necessarily equal to) the number of user-item interaction edges associated with the item node. Item nodes connected to more user-item interaction edges are considered to have more sufficient user interactions than item nodes connected to fewer user-item interaction edges.

[0099] Method 700 includes, in step 704, discarding some user-item interaction edges based on the determination in step 702. In some embodiments, for each of at least a portion of the item nodes, the discarding is based on the degree to which the item node is determined to have insufficient user interaction. Item nodes that are determined to have insufficient user-item interaction edges will have more edges discarded than item nodes that are determined to have insufficient user-item interaction edges. In some embodiments, user-item interaction edges of one or more item nodes that have insufficient user-item interaction are not discarded (i.e., all are retained).

[0100] The modified graph representation data obtained from method 700 may be used to facilitate the training of one or more graph neural network-based recommendation models of a graph neural network-based recommendation system.

[0101] Figure 8 A method 800 for modifying graph representation data in some embodiments of the present invention is shown. In some embodiments, the method 800 may be considered as belonging to Figure 2 Graph data augmentation operations in 202B or Figure 3 Step 306 of method 300. Method 800 includes, in step 802, processing graph representation data by performing a node synthesis operation, such as Figure 3The graph representation data obtained in step 304 of method 300 or in an edge adding operation (e.g. Figure 4 and Figure 5 ) or after an edge discarding operation (e.g. Figure 6 and Figure 7 The graph representation data obtained after the edge discarding operation in the graph representation data is modified to add at least one or more synthetic item nodes and preferably one or more corresponding synthetic user-item interaction edges to the graph representation data, thereby forming modified graph representation data.

[0102] Fig. 9A A method 900A for modifying graph representation data in some embodiments of the present invention is shown. In some embodiments, the method 900A is considered to be part of Figure 8 Step 802 of method 800.

[0103] Method 900A includes, in step 902A, using one or more graph neural network-based recommendation models of a graph neural network-based recommendation system to process graph representation data, such as in Figure 3 The graph representation data obtained in step 304 of method 300 is used to determine the embedding of the user node and the embedding of the item node.

[0104] The method 900A includes performing a data blending operation in step 904A to generate one or more synthetic item nodes based at least in part on the embedding of the user node and the embedding of the item node. In some embodiments, for generating each synthetic item node, the data blending operation includes: selecting two item nodes from the item nodes, and generating the synthetic item node based on the embedding of the two item nodes. In some embodiments, the selection of the two item nodes can be random.

[0105] The method 900A includes, at step 906A, for each synthetic item node, generating a corresponding synthetic user-item interaction edge based at least in part on a hyperparameter. In some embodiments, depending on the hyperparameter, the synthetic item node either inherits a user node connection from one of the corresponding two item nodes or inherits a user node connection from the other of the corresponding two item nodes.

[0106] The modified graph representation data obtained from method 900A may be used to facilitate the training of one or more graph neural network-based recommendation models of a graph neural network-based recommendation system.

[0107] Fig. 9B Another method 900B for modifying graph representation data in some embodiments of the present invention is shown. In some embodiments, method 900B may be considered as belonging to Figure 8 Step 802 of method 800.

[0108] Method 900B includes, in step 902B, performing a first data augmentation operation on a set of data to generate a first synthetic data set, the data set comprising one or more synthetic user-item-interaction triples and one or more corresponding synthetic embeddings. The data set on which the first data augmentation operation is performed can be obtained by processing graph representation data using one or more graph neural network-based recommendation models of a graph neural network-based recommendation system, for example, Figure 3 The first data augmentation operation may include a data mixing operation and / or a data resampling operation. For example, the data mixing operation may include mixing one or more pairs of user-item-interaction triples and their corresponding embeddings to generate synthetic data. For example, the data resampling operation may include selectively extracting and discarding at least some user-item-interaction triples and corresponding embeddings from the data set.

[0109] Method 900B includes, in step 904B, processing the data set and the first synthetic data set using a bilateral branch network model to compare or determine the performance of the item node. In some embodiments, the bilateral branch network model includes two substantially identical branches, each branch including one or more graph neural network-based recommendation models, which may be or may be substantially the same as one or more graph neural network-based recommendation models of a graph neural network-based recommendation system. In some embodiments, step 904B includes: processing the data set using one branch of the bilateral branch network model, and processing the data set and the first synthetic data set in combination using another branch of the bilateral branch network model. Comparing or determining the performance of the item nodes helps to identify item nodes that can achieve performance improvements based on the first data enhancement operation. In some embodiments, comparing or determining the performance of the item nodes helps to identify item nodes that can achieve performance improvements exceeding a certain threshold performance based on the first data enhancement operation.

[0110] Method 900B includes, in step 906B, selecting an item node for performing data augmentation based on the comparison or determination of the item node performance in step 904B. In some embodiments, item nodes that are determined to be capable of achieving performance improvement are selected to perform training data augmentation. In some embodiments, item nodes that are determined to be capable of achieving performance improvement exceeding a certain threshold are selected to perform training data augmentation. In some embodiments, item nodes that are determined to be more capable of achieving performance improvement are more likely to be selected to perform training data augmentation.

[0111] Method 900B includes, in step 908B, performing a second data augmentation operation on the data set of the selected item nodes to generate a second synthetic data set, including one or more synthetic item nodes and one or more corresponding synthetic user-item interaction edges. The second data augmentation operation may include a data blending operation and / or a data resampling operation. For example, the data blending operation may include blending one or more pairs of user-item-interaction triples and their corresponding embeddings to generate synthetic data. For example, the data resampling operation may include selectively extracting and discarding at least a portion of the user-item-interaction triples and corresponding embeddings from the data set.

[0112] The modified graph representation data obtained from method 900B may be used to facilitate the training of one or more graph neural network-based recommendation models of a graph neural network-based recommendation system.

[0113] Fig. 10A A method 1000A for training a recommendation system based on a graph neural network in some embodiments of the present invention is shown. The method 1000A is a computer-implemented method. The method 1000A includes multi-stage (e.g., two-stage) training. In some embodiments, the method 1000A can be considered as belonging to Figure 2 The multi-stage training operation 206 or Figure 3 Step 308 of method 300. Method 1000A includes: in step 1002A, using a graph to represent data (e.g., in Figure 3 The method 300 of FIG1004 is used to train one or more graph neural network-based recommendation models of the graph neural network-based recommendation system using the data obtained in step 304 of the method 300; in step 1004A, one or more graph neural network-based recommender models of the graph neural network-based recommendation system are trained using the modified graph representation data, for example, after an edge addition operation (such as Figure 4 and Figure 5 ) or after edge discarding operations (such as Figure 6 and Figure 7 edge discarding operation in ) or after node synthesis operation (such as Figure 8-9BSince the graph representation data can better represent the head items (items with more user interactions), and the modified graph representation can better represent the tail items (items with less interactions), method 1000A can enable one or more graph neural network-based recommendation models of the graph neural network-based recommendation system to first learn high-quality representations of the head items (items with more user interactions), which are easier to learn, and then learn high-quality representations of the tail items (items with less interactions), which are more difficult to learn. This can improve the training quality of one or more graph neural network-based recommendation models of the graph neural network-based recommendation system. In some embodiments, as part of the training, method 1000A includes optimizing a loss function that at least or only considers recommendation loss and regularization loss.

[0114] Fig. 10B A method 1000B for performing graph contrast learning to train a graph neural network-based recommendation system in certain embodiments of the present invention is shown. The method 1000B is a computer-implemented method. The method 1000B includes, in step 1002B, training one or more graph neural network-based recommendation models of a graph neural network-based recommendation system using a graph contrast learning technique. In some embodiments, the graph contrast learning technique includes graph representation data modification and noise injection operations. In some embodiments, the method 1000A includes, as part of the training, optimizing a loss function that at least or only considers contrast loss, recommendation loss, and regularization loss.

[0115] Although not shown, some embodiments of the present invention involve using or operating a graph neural network-based recommendation system. The use or operation includes processing user data and item data using one or more graph neural network-based recommendation models of the graph neural network-based recommendation system, the recommendation model being trained using any one or more methods of the present invention to determine item recommendations for the user. The use or operation may also include presenting (e.g., displaying) item recommendations. The item recommendations may include one or more items recommended to the user.

[0116] The following description provides some example frameworks for training a graph neural network-based recommendation system in some embodiments of the present invention. In some embodiments, the example frameworks can be viewed as Figure 2 Frame 200 and / or Figures 3 to 10B A specific example implementation of one or more operations in .

[0117] The inventors of the present invention have realized through research that recommendation systems have been widely used in various fields, and data sparsity problems (i.e., lack of or limited availability of certain data in the real world) may reduce the performance of recommendation systems. The inventors of the present invention have found that in recommendation applications, user behavior usually follows a long-tail distribution, that is, users may provide much more feedback on popular items (i.e., head items) than on unpopular items (i.e., tail items), although tail items may also be useful, for example, for enhancing user experience and increasing service provider revenue. Therefore, traditional recommendation systems and methods may not be able to make relatively accurate predictions for tail items.

[0118] Example 1

[0119] To date, there are various approaches to address the long-tail recommendation problem. These approaches include resampling strategies, transfer learning or meta-learning methods, and graph contrastive learning methods. Resampling strategies use undersampling and oversampling strategies to modify the data distribution. Transfer learning-based methods aim to transfer knowledge from head items to tail items. Graph contrastive learning-based methods apply a contrastive learning-based approach to build more robust representation learning for features to handle highly skewed data, which may benefit the performance of items located in the long-tail part. Although these existing methods can improve the long-tail recommendation performance, they may still have limitations. For example, data resampling methods may affect the performance of head items because they change the data distribution during the training phase. For example, transfer learning or meta-learning methods may rely heavily on user / item features to obtain rich information, so if there is not enough side information (such as user / item features), these methods may not be able to extract enough knowledge for knowledge transfer. In fact, in practice, such side information is not always available, which limits the application of such methods. For example, some existing graph contrastive learning methods only use one enhancement (such as edge perturbation, adding noise), which may not be enough to learn good or useful representations. In addition, augmentation-based graph contrastive learning methods usually build random uniform graph augmentations. However, these methods may fail to learn expression representations for tail items due to the insufficient number of edges for tail items.

[0120] In view of this, some embodiments of the present invention provide a framework for solving the long-tail recommendation problem. The framework is called a long-tail recommendation adaptive enhancement framework based on graph contrast learning ("GALR"). In general, GALR includes graph topology enhancement (adaptive edge discarding and adaptive edge addition) and graph data (training data) enhancement (training triple resampling and mixing). Through adaptive edge addition, GALR can make the tail items aggregate information from homogeneous head items and heterogeneous users, and the information of head items and heterogeneous users is richer. Therefore, the recommendation performance of tail items can be improved. Compared with random edge discarding, adaptive edge discarding can retain more important edges for tail items and discard more unimportant edges for head items, thereby promoting the characterization learning of tail items. Enhanced training data can provide more training triples, thereby alleviating the problem of data sparsity. In addition, combining graph contrast learning helps to learn more unified and robust representations. In some embodiments, an adaptive amplification framework is provided for solving the long-tail recommendation problem using edge addition and graph topology amplification. In some embodiments, training data enhancement is applied to force the neural network model to learn more information about the tail. In some embodiments, enhancement nodes are selected based on performance improvements through a bilateral branching network.

[0121] Regarding long-tail recommendations, the inventors of the present invention have determined through research that methods for long-tail recommendations can be divided into several types, including clustering, deep learning, and graph-based methods. For example, Huang et al. disclosed a rebalancing solution in "Correcting sample selection bias by unlabeled data" (2006) to solve the long-tail problem by directly generating resampling weights to select samples. For example, Park et al. disclosed a clustering tail method in "The long tail of recommendation systems and how to leverage it" (2008), which uses clustering techniques to group tail items and then recommends tail items based on the scores within the cluster. For example, Grozin et al. disclosed a clustering method in "Similar Product Clustering for Long-Tail Cross-Sell Recommendations" (2017), which uses user and item data and some distance indicators to perform cross-selling recommendations based on association rule mining. For example, Zhang et al. disclosed a dual transfer learning framework MIRec in "A model of twotales: Dual transfer learning framework for improved long-tail item recommendation" (2021) for transferring model-level and item-level knowledge from the head to the tail. For example, Liu et al.'s "Long-tail session-based recommendation" (2020) disclosed a preference mechanism, Tail-Net, which includes session representation and correction factor generation, which can soft-adjust the recommendation model and recommend more long-tail items. For example, Yin et al. disclosed a random walk solution for long-tail recommendation in "Challenging the long tail recommendation" (2012). For example, Yao et al.'s "Self-supervised learning for large-scale item recommendations" (2021) discloses a method of using contrastive learning to enhance feature representation learning and improve recommendation performance.

[0122] However, the problem is that these existing methods related to long-tail recommendation either require rich (large) side information or ignore the individual characteristics of the tail items (i.e., recommending a cluster of target tail items rather than a single item). Unlike these existing methods, in some GALR embodiments described in detail below, graph enhancement is performed to transfer the information of the head items to the tail items to help learn better tail item representations.

[0123] Regarding data enhancement, the inventors of the present invention have determined through research that data enhancement methods may include data resampling and data mixing. For example, as shown in Estabrooks' "A multiple resampling method for learning from imbalanced data sets" (2004) and Huang et al.'s "Correcting sample selection bias by unlabeled data" (2006), data resampling methods can be used to process unbalanced data by changing the distribution of training data. For example, as described in Zhang et al.'s "Mixup: Beyond empirical risk minimization" (2017), data mixing methods can be used to synthesize new data samples by combining existing data samples and their corresponding labels. The inventors of the present invention further determined that graph data enhancement learning helps to alleviate incomplete / imbalanced data or noisy data in the graph structure, and example techniques for graph enhancement learning include node discarding, edge addition / discarding, and attribution completion. However, most of these techniques are used for homogeneous graphs and cannot be directly used for recommendation applications where the user-item graph is a double rectangle. For example, Rong et al. introduced DropEdge in DropEdge: Towards Deep Graph Convolutional Networks on Node Classification (2019), which randomly drops graph edges in a message passing mechanism to alleviate over-smoothing. For example, Volkovs et al. introduced DropoutNet in DropoutNet: Addressing cold start in recommendation systems (2017), which applies dropout during training to address the cold start problem in recommendation systems. For example, Wang et al.’s NodeAug: Semi-supervised node classification with data augmentation (2020) publicly teaches NodeAug to create a data-augmented parallel universe for each node to handle the negative influence from other nodes. For example, Zhao et al.’s Data augmentation for graph neural networks (2021) introduced GAUG, which introduces a graph data augmentation framework to improve the performance of GNN-based node classification through edge prediction.For example, Zhou et al., "Data augmentation for graph classification" (2020), introduced graph data augmentation and proposed two heuristic algorithms, random mapping and pattern similarity mapping, which generate more weakly labeled data for small-scale benchmark data sets by heuristically modifying the graph structure. The inventors of the present invention further determined that some other techniques combine graph data augmentation with contrastive learning. For example, Zhu et al., in "Graph contrastive learning with adaptive augmentation" (2021), disclosed a graph contrastive representation learning method with adaptive augmentation, which combines various priors on the topology and semantics of the graph. For example, Suresh et al., "Adversarial graph augmentation to improve graph contrastive learning" (2021), disclosed a principle called adversarial-GCL, which enables graph neural networks (GNNs) to avoid capturing redundant information during training by optimizing the adversarial graph augmentation strategy used in graph contrastive learning (GCL).

[0124] Despite the teachings of these prior arts, the inventors of the present invention have found that there is no research or limited research on graph data enhancement learning in these prior art suggestions. The inventors of the present invention have found through their own research and experiments that graph data enhancement learning may be particularly useful for alleviating the adverse effects of long-tail distributions. Therefore, in some GALR embodiments described in detail below, an adaptive enhancement method is applied to solve the long-tail recommendation problem.

[0125] Further details of GALR are now provided in some of the Examples.

[0126] Without loss of generality, in an example, a triple is defined<U,I,R> , where U={u1,u2,…,u m} represents a set of m users, I = {i1,i2,…,i n} represents a set of n items, R = {r1,…,r p} represents the user's feedback on the item. In this example, the cumulative item occurrence counts in all user-item interactions are distributed in a long-tail distribution. A user-item bidirectional graph G = (V, E) can be constructed (similar to Figure 1A), where the node set V includes two types of nodes: user nodes and item nodes, and E represents the set of edges. If the user represented by the user node provides feedback to the item represented by the item node, there is an edge between the user node and the item node. In one example, the Top-K recommendation model (K is an integer) aims to recommend the K most relevant items to each user and maximize the average recommendation performance.

[0127] In some embodiments, GALR is configured to improve the recommendation performance of tail items (in addition to the average recommendation performance of all items).

[0128] Fig.11 A GALR in certain embodiments is shown, which generally includes various functions / modules.

[0129] In summary, in certain embodiments of GALR, graph topology enhancement, i.e., adaptive homogeneous edge addition and adaptive heterogeneous edge discarding, is performed. By adaptive homogeneous edge addition, high-order homogeneous neighbors (neibours) are explicitly introduced for the tail items, thereby enhancing information transfer in the graph model. By adaptive heterogeneous edge discarding, representation learning can become more robust and can avoid or improve the problem of over-smoothing. In certain embodiments, the critical graph structure of the tail node is preserved, which is very advantageous. In certain embodiments, a bilateral branch network is applied to compare the performance of each item node. Based on the performance comparison results, appropriate item nodes are selected for enhancement. By enhancing the selected nodes, the graph model is encouraged to learn more about the tail or tail nodes. In addition, in certain embodiments, self-supervised learning is also incorporated, and contrast loss is increased by comparing two enhanced graph views, thereby enhancing representation learning through self-discrimination.

[0130] In certain embodiments of GALR, adaptive homogeneous edge addition is applied.

[0131] In certain embodiments of GALR, graph augmentation learning is applied to long-tail recommendations so that tail items can get some information from their neighbors. Since the head item nodes have more connections in the graph than the tail item nodes, it can be assumed that the head item nodes can learn better graph structural information in their embeddings. Therefore, certain embodiments aim to transfer information from the head item nodes to the tail item nodes. In certain embodiments of GALR, homogeneous edges (i.e., item-item edges connecting item nodes) are added to improve the graph connectivity of the tail item nodes, thereby improving long-tail performance. To this end, an item-item similarity graph is first constructed to find structural neighbors for the tail item nodes. Then, the same item nodes are clustered into different groups to find semantic neighbors for the tail item nodes. In one example, it is assumed that nodes in the same group have similar interests, while nodes in different groups have different / diverse interests. It should be noted that in certain embodiments of GALR, edges are added to both structural and semantic neighbors (item nodes) to improve the connectivity of the graph. In certain embodiments of GALR, message passing and aggregation are further performed on the enhanced graph.

[0132] In some embodiments, adaptive homogeneous edge addition includes finding structural neighbors. An item-item similarity graph can be constructed to find structural neighbors. First, an item similarity matrix is ​​calculated based only on the interactions Y between users and items. The item similarity matrix is ​​an M×M matrix, and each element is the common interaction value between two items. In some embodiments, the common interaction value between two users is defined as the number of items they have both interacted with, and the common interaction value between two items is the number of users they have both interacted with. Mathematically, the calculation formula of the item similarity matrix is

[0133] S I =Y T ·Y∈R M×M (1)

[0134] It should be noted that in some other embodiments, the framework may use other defined common interaction values, such as Pearson correlation, cosine distance, Jaccard similarity, etc. Then, the common interaction values ​​of each tail item may be sorted to obtain the structural neighbors.

[0135] In some embodiments, adaptive homogeneous edge addition includes finding semantic neighbors. Some studies have shown that high-order neighbors may have a negative impact on performance if interests are different. Therefore, simply adding homogeneous edges to all items may not be appropriate, because items of different categories or interests may convey irrelevant information and interfere with representation learning. Therefore, in some embodiments, it is best to find semantic neighbors (in addition to structural neighbors). To this end, in some embodiments, clustering is performed to aggregate similar items. Clustering can use K-means (for example, MacQueen et al.'s "Some methods for classification and analysis of multivariate observations" (1967)), mean shift (for example, Comaniciu et al.'s "Mean shift: A robust approach toward featurespace analysis" (2002)), etc. In this example, the K-means method is used to cluster items into several groups or clusters. Items in the same cluster are considered semantic neighbors.

[0136] In some embodiments, message passing is performed.After graph construction and enhancement, neighbor information can be aggregated to enhance self-node representation. Fig.12 The information transfer operation of GALR in some embodiments of the present invention is shown. Take the tail item node as an example. In this example, the structural neighbors of the tail item are represented as The semantic neighbors of the tail item are represented as Then the homogeneous neighbors of the tail item are At the same time, the heterogeneous neighbors of the item are defined as N e (i). Likewise, N o (u) and N e (u) represent the homogeneous neighbors and heterogeneous neighbors of a user, respectively.

[0137] To update the representation of the t-th layer item node (of a GNN), we first aggregate the representations of the t-1-th layer item node neighbors and then combine them with the item node’s own representation. This process can be expressed as follows

[0138]

[0139] in, is the ID embedding of item node i at layer t, AGGREGATE() represents an adjacency aggregation function, such as an average or maxpooling operation, and COMBINE() represents a function that merges the aggregated adjacency representation with the node representation. One example is to use an average operation, which is relatively simple. Another example is to use an attention mechanism, such as:

[0140]

[0141] in, is the normalized attention weight of the homogeneous neighbor node p in the tth layer, a i.t It means that each layer uses its own attention parameters, the operator || indicates connection, and LeakyReLU() is used as the activation function.

[0142] After obtaining the T-layer representation, the readout function is used to generate the final representation for prediction:

[0143]

[0144] In this example, a weighted sum operation is used as READOUT().

[0145] In some embodiments, the user node also adopts similar steps. However, since the user node does not have homogeneous neighbor nodes, the process is described as follows:

[0146]

[0147] In certain embodiments of GALR, an adaptive heterogeneous edge dropping technique is applied.

[0148] In DropEdge: Towards Deep Graph Convolutional Networks on Node Classification by Rong et al. (2019), it is shown that edge dropping is simple and effective in enhancing graph representation learning and alleviating the over-smoothing problem. However, it may not be appropriate to treat every edge equally. In long-tail recommendations, the tail item nodes are connected by fewer edges than the head item nodes. Therefore, in long-tail recommendations, the edges of the tail item nodes are more important than the edges of the head item nodes. Discarding too many edges of the tail item nodes may destroy the local graph structure, adversely affect the information transfer process, and degrade the long-tail performance.

[0149] To solve this problem, some embodiments use an adaptive heterogeneous edge discarding technique, that is, discarding more edges for the head items and discarding fewer edges for the tail items. In some embodiments, the degree to which the node is located in the tail position must first be defined. One example is to use the node degree. However, if the node degree is used, there may be a relatively large gap between the node at the head and the node at the tail. To solve this problem, in another example, a long tail coefficient describing the degree to which the node is located in the tail position is first defined. Specifically, in one example, the item node i k The long tail coefficient LC ik It is expressed as:

[0150]

[0151] in, is item node i k The larger the long-tail coefficient value is, the more serious the long-tail phenomenon of the node is. c is the cutoff value. Then the adaptive edge discarding process is formulated. Formally, the edge discarding process is expressed as

[0152] s(G)=(V,M⊙ε) (7)

[0153] Among them, s() is randomly selected, M∈{0,1} |ε| is the shielding vector on the edge set ε. Assume that the edge discard probability of user u and item i is It is defined as

[0154]

[0155] Among them, p e is a hyperparameter that controls the overall probability of discarding an edge; and yes The maximum and minimum values ​​of p c <1 is a cutoff probability.

[0156] In certain embodiments of GALR, training triplet resampling and mixing are applied.

[0157] Data augmentation (such as data resampling, data mixing, etc.) is a model-independent technique for dealing with data imbalance problems. By repeatedly extracting samples or synthesizing samples from the training data, the model can be arranged to pay more attention to the tail. For long-tail recommendations, one way to select nodes for data augmentation is to split them according to the degree of the item node, that is, to split the item node into a head and a tail, and only amplify the tail. However, this method may not be optimal in some cases, because not all low-degree nodes have poor performance, so it may not be optimal to treat every tail item equally. The inventor believes that there may be two reasons for the poor performance of item nodes: low frequency and high noise. In terms of low frequency, if the item node appears more times, its performance may be better. As for noise, even if the item node appears more times, its performance may not improve because it is noise. Therefore, it is very useful to choose an appropriate method because noisy nodes cannot be improved by data augmentation.

[0158] To this end, in some implementations, item nodes that improve performance are selected over low-degree nodes.

[0159] Fig.13 FIG. 4 shows the training data augmentation operation of GALR in some embodiments of the present invention. In some embodiments, a bilateral branch network is designed to select appropriate nodes for augmentation. Fig.13 As shown in Figure 1, the bilateral branch network includes two roughly identical branches, each with its own GNN-based recommendation model. In one branch, the original data remains unchanged. In the other branch, some nodes are sampled and data augmentation is performed. In some embodiments, two methods for data augmentation are used: data mixing and data resampling. For data mixing, given two interacting user-item-rating triples, they are represented as p ,i m ,r k > and p ,i n ,r l >, where u p is the one with the corresponding embedding point e p User, i m and i n is the one with the corresponding embedding point e m and e n Items. Mix these two pairs of data and get Items The embedding of And λ is a hyperparameter. For data resampling, some embodiments repeatedly extract or remove samples from the training data. i ∈D and d i = <u​​a ,i b ,r c >. For data resampling, d′ i = a ,i b ,r c > For data mixing, d′ i = <u′ a ,i′ b ,r′ c >. Assume that w is increased o data items, the added data is recorded as

[0160] Then, the GNN-based recommendation model of each branch is trained. One branch is trained with the original training data, while the other branch is trained with the enhanced training data (original training data + synthetic training data). The GNN-based recommendation model can be NGCF, PinSage, LightGCN, etc. The process is expressed as

[0161]

[0162] Among them, Θ init are the initial parameters of the model, D is the training data, G is the graph, D′ is the enhanced training data, and f() is the calculation result of a specific GNN.

[0163] In some embodiments, the node performance of each item node is compared, and the performance improvement is represented by Δp, which is defined as:

[0164] Δp=p′ i -p i (10)

[0165] Among them, p i is the performance of item node i in the branch trained with the original data, p′ i is the performance of item node i in another branch. Then, the item nodes whose performance improvement is greater than a threshold value θ are selected, that is, Δp>θ. θ is a hyperparameter, which is set to 0 in this case. The sampling process is repeated until all candidate nodes are covered. Then a selected set of items (set of item nodes) is obtained for data enhancement. In certain embodiments, an adaptive training data enhancement scheme is designed. The inventors believe that item nodes with less improvement require more data enhancement to achieve satisfactory performance. Assuming w is a weight that controls the degree of data augmentation, its definition is

[0166]

[0167] Where w is a hyperparameter that controls the overall probability of adding an edge; Δp​max and Δp min is the maximum and minimum value of Δp; w c is the cutoff value. Then, the adaptive augmented data D″ is used to train the GNN model (the same as the model of any branch in the bilateral branch network), expressed as:

[0168] Θ final =f(Θ1,G,D″) (12)

[0169] In certain embodiments of GALR, contrastive learning is applied.

[0170] Contrastive learning can improve representation learning by learning a more uniform distribution, thereby improving model performance. For graph contrastive learning, one approach is to generate two enhanced views of the graph. This approach may include edge / node perturbations. For example, Wu et al.'s Self-supervised graph learning for recommendation (2021) discloses SGL with graph enhancements such as random edge discarding. For example, Zhu et al.'s Graph contrastive learning with adaptive augmentation (2021) discloses GCA, which retains more central and critical graph structures. However, for tail items, since they have fewer edges, treating them as the same or less important than head items may not be suitable for improving long-tail performance. Since the number of edges of tail items is quite limited, and each edge bears a higher responsibility for information transfer, abandoning the edges of tail items may lose relatively important information. Therefore, in some embodiments, a contrastive learning method based on adaptive graph enhancement is adopted. Specifically, the head items are more likely to have edge perturbations than the tail items to maintain the graph structure and enable the tail items to transfer information.

[0171] Formally, in some embodiments, a mixture of two augmentation methods is used to generate graph views, because a single augmentation method may not be enough. One augmentation method is graph perturbation, including the adaptive edge addition and adaptive edge discarding described. Another augmentation method is to add noise to the embedding and compare it with the embeddings of different layers, such as SimGCL (as disclosed by Yu et al. in "Are graph augmentations necessary? Simple graph contrastive learning for recommendation" (2022)) to learn a more uniform embedding distribution. In some embodiments, contrast loss, InfoNCE (as disclosed by Gutmann et al. in "Noise-contrastive estimation: A new estimation principle for unnormalized statistical models" (2010)) is used to maximize the consistency of positive pairs and minimize the consistency of negative pairs. Hypothesis

[0172] ″″″

[0173] Assume e u and e u are user embeddings of two views, e i and e i is the object embedding of the two views generated by adaptive edge dropping, then the contrast loss It is expressed as:

[0174]

[0175] Among them, s() represents the similarity between two vectors, which is set as the cosine similarity function; τ is a hyperparameter, which is called "temperature" in softmax. In this way, representation learning can be enhanced and model training can be promoted.

[0176] In certain embodiments of GALR, specific training strategies are employed.

[0177] After obtaining user embedding and item embedding through the GNN-based model, the inner product of user embedding and item embedding is used to estimate the user's preference for the target item. Specifically, user u j For item i k The prediction model is expressed as

[0178]

[0179] In some embodiments, a BPR loss (as disclosed in Rendle et al., BPR: Bayesian personalized ranking from implicit feedback (2012)) is used to encourage predictions for observed user-item pairs to be higher than predictions for unobserved user-item pairs:

[0180]

[0181] In some embodiments, the overall loss is the sum of the recommendation loss, the regularization loss, and the contrastive loss, expressed as

[0182]

[0183] Wherein, Θ is a model parameter, γ1 and γ2 are hyperparameters, respectively controlling the strength of contrast loss and L2 regularization loss. In some embodiments, an Adam optimizer (as described in Kingma et al., A method for stochastic optimization (2014)) is applied to optimize the loss function to minimize the loss.

[0184] To evaluate the performance of GALR in certain implementations and analyze its components, experiments were conducted on four real-world datasets from different domains. Specifically, the experiments aim to determine how GALR performs in terms of overall and long-tail performance compared to existing baseline methods, determine how adding adaptive enhancement and contrastive loss to GALR affects recommendation performance, and determine how hyperparameters affect the performance of GALR.

[0185] In this example, the experiments are conducted on four public real-world datasets, as detailed in Fig.14 As shown:

[0186] (1)MovieLens(https: / / grouplens.org / datasets / movielens / )

[0187] MovieLens is a movie rating dataset used to evaluate recommendation algorithms. This example uses MovieLens-1M (ML-1M), which contains about 1,000,000 user ratings.

[0188] (2) Douban Books (https: / / github.com / librahu / HIN-Datasets-for-

[0189] Recommendation-and-Network-Embedding)

[0190] Douban Books contains user ratings of books.

[0191] (3)Yelp2018(https: / / www.yelp.com / dataset / challenge)

[0192] Yelp2018 comes from the Yelp-2018 Challenge, which is related to the task of recommending business web pages to users.

[0193] (4) Amazon Books (https: / / jmcauley.ucsd.edu / data / amazon / )

[0194] Amazon Books comes from Amazon review data and is used to recommend books to users.

[0195] In the experiments, GALR in certain implementations is compared with existing model-agnostic baseline methods, including data reweighting, loss function refinement, and various meta-learning, graph reinforcement learning, and contrastive learning models. For fair comparison, LightGCN is used as the backbone model. All models use the same layer and embedding dimensions. Specifically, the baseline methods used in the experiments are as follows:

[0196] (1) Backbone model:

[0197] LightGCN (see He et al., LightGCN: Simplifying and powering

[0198] graph convolution network for recommendation》(2020)):

[0199] A lightweight model that simplifies the weight parameter aggregation step in graph convolutional networks.

[0200] (2) Resampling strategy (such as Brownlee's "Imbalanced classification with Python: better metrics, balance skewed classes, cost-

[0201] Sensitive Learning (2020):

[0202] Oversampling: Sample more from the tail items to rebalance the head and tail items in the training data distribution.

[0203] ● Undersampling: The opposite of oversampling, keeping the tail items unchanged and reducing the sampling of the head items.

[0204] (3) Loss function improvement method:

[0205] Focal Loss (FL) (as published by Lin et al. in Focal loss for dense object detection (2017)): reweights the standard loss to assign less weight to well-classified examples.

[0206] ClassBalance (as published in Cui et al., Class-balanced loss based on effective number of samples (2019)): Calculate the effective number of samples and reweight the loss for each class.

[0207] (4) Meta-learning model:

[0208] MeLu (as published by Lee et al. in “MeLU: Meta-learned user preference estimator for cold-start recommendation” (2019)): applies optimization-based meta-learning (MAML) for cold-start item rating prediction.

[0209] (5) Graph Enhanced Learning Model:

[0210] DropEdge (as in Rong et al., DropEdge: Towards Deep Graph Convolutional Networks on Node Classification (2019)

[0211] ): Randomly discard graph edges in the message passing mechanism to alleviate over-smoothing.

[0212] Tail-GNN (as published by Liu et al. in Tail-GNN: Tail-node graph neural networks (2021)): adding tail node-specific components to the original GNN.

[0213] (6) Contrastive learning model:

[0214] SGL (such as Wu et al.'s Self-supervised graph learning for

[0215] recommendations”(2021)): Incorporating self-supervised learning into recommendations through layer augmentation such as edge dropout and random walks.

[0216] xSimGCL (as in Yu et al., XSimGCL: Towards extremely simple

[0217] graph contrastive learning for recommendation》(2022)

[0218] Injecting random noise into embeddings as a non-augmented graph contrastive learning approach.

[0219] The experiments in this example use a variety of settings.

[0220] One of the settings is related to preprocessing. In this example, following the experimental settings published by Yu et al. in "Graph Augmentation Learning" (2022), the ratings less than 4 in the Movielens and Douban Books datasets (the rating scale is 1-5) are discarded, and the remaining ratings are reset to 1. The Pareto principle (as described in the book "An analysis for unreplicated fractional factorials" (1986) by Box et al.) is used as the criterion for dividing head and tail items. The items with the top 20% of the occurrences are set as head items, and the rest are set as tail items. Evaluation metrics for the tail and head item sets are reported separately. The average performance of five runs is reported.

[0221] One of the settings is related to the evaluation metrics. In this example, two evaluation metrics are used to evaluate the performance: (1) Recall, which measures the probability that the recommended list contains the items that the user is interested in; (2) a weighted version of the hit rate (HR), called the normalized discounted cumulative gain (NDCG), which is used to increase the weight of the items ranked higher in the recommended list. In this example, Recall@K (or R@K for short) and NDCG@K (or N@K for short), K∈{20,50}, are evaluated.

[0222] One of the settings is related to hyperparameters. In this case, for a fair comparison, the best hyperparameter settings reported in publications related to the benchmark methods were consulted and all hyperparameters of the benchmark methods were fine-tuned via grid search. For the general settings for all benchmark methods, Xavier initialization (as published by Glorot et al. in Understanding the difficulty of training deep feedforward neural networks (2010)) was used for all embeddings. In this case, the embedding size was 64 and the L2 regularization parameter was 10 -4 , with a batch size of 2048. All models are optimized using the Adam optimizer with a learning rate of 0.001. For GALR, the number of structural neighbors is selected from {3, 5, 10} and the number of clusters is selected from {3, 5, 10, 20}.

[0223] In certain embodiments, the performance of the baseline method and GALR is evaluated using four datasets to explore the performance of the model in different situations. Fig.15A and 15B The tables in summarize the recommendation performance for all items, head items, and tail items on these datasets.

[0224] Based on the obtained recommendation performance, some conclusions can be drawn.

[0225] First, compared with the backbone model LightGCN, the GALR implementation generally improves on both tail items and head items. This indicates that in long-tail item distributions, it is important to consider adaptive graph perturbations and inter-item training data augmentation. The results show that graph augmentation in the GALR implementation can benefit long-tail distribution recommendation data.

[0226] Secondly, the GALR implementation outperforms the reweighting methods (i.e., oversampling, undersampling) on ​​all three splits. Among these methods, oversampling performs worse than undersampling on the tail item split. However, the resampling method may change the distribution of the original data, which may have a negative impact on the performance of the entire model. The methods based on meta-learning and loss function refinement strategies do not perform well on the head and tail items. This verifies that the meta-learning method cannot achieve good performance without model parameters and user / item features. The reason why the loss function refinement strategy cannot achieve excellent performance may be that the trade-off between the head and tail items is unhealthy or undesirable. Graph enhancement methods (i.e., DropEdge and Tail-GNN) successfully improved the performance of the head and tail items. This shows that by perturbing the graph structure, the model can learn more robust representations.

[0227] Third, GALR achieves better performance than contrastive learning-based models. This suggests that combining adaptive graph perturbations with contrastive learning can learn more unified and robust representations, which will be beneficial to improving model performance.

[0228] Ablation studies were also performed to analyze different components of the GALR embodiment. Fig.16 Results of ablation studies are shown. The full GALR embodiment outperforms all GALR variants on three segments, indicating that all major components in this GALR embodiment can be used to improve performance.

[0229] The four GALR variants and their consequences are as follows.

[0230] A variant of GALR is one that does not include contrast loss In this variant, the contrast loss is removed Only BPR loss and L2 normalization loss are used to train the GNN-based model. Fig.16 As can be seen, there is no contrast loss The GALR performance of The representation can be effectively enhanced to learn a more unified and robust embedding, which can improve the recommendation performance.

[0231] A variant of GALR is without adaptive edge drop. In this variant, adaptive edge drop is replaced by random uniform edge drop (other parts remain unchanged). The GNN model without adaptive edge drop performs worse on tail items. This suggests that adaptive edge drop can improve the recommendation performance of tail items because it can preserve more critical graph structure information for tail items.

[0232] A variant of GALR is without adaptive edge addition. In this variant, adaptive edge addition is removed (other components remain the same). Without adaptive edge addition, the performance of tail items degrades significantly. This suggests that head items can pass information to neighboring tail items.

[0233] A variant of GALR is without training triplet resampling and mixing. In this variant, training triplet resampling and mixing are not applied (other parts remain the same). It can be seen that the GNN-based model performs poorly.

[0234] Hyperparameter studies were also performed.

[0235] One of the hyperparameter studies concerns the effect of edge drop rate. Fig.17The performance of all three split points varies according to the edge drop rate. When edge drop rate = 0.9, the graph structure is basically destroyed. In this case, graph contrastive learning cannot learn any useful information and it is difficult to optimize the contrast objective.

[0236] One of the hyperparameter studies concerned the effect of the number of clusters. Fig.18 The results of the study are shown in Figure 2. It can be seen that as the number of clusters increases, both the overall model performance ("all items") and the tail performance ("tail items") increase first and then decrease. This is because when the number of clusters is too large or too small, it is difficult to reflect the characteristics of the items. This verifies that the number of clusters should be chosen carefully.

[0237] One of the hyperparameter studies concerns the effect of the number of structural neighbors. Fig.19 The performance of GNN-based models with different numbers of structural neighbors is shown. The results show that, in general, when a larger number of structural neighbors is selected, more information is transferred from the head item to the tail item, thereby improving the performance of long-tail items.

[0238] One of the hyperparameter studies involves the effect of embedding size or dimensionality. The effect of embedding size on the performance of GNN-based models was studied. Figures 20A to 20C The results are shown. It can be seen that the performance of GNN-based models generally improves with increasing embedding size or dimensionality. However, the performance improvement is limited when the embedding size is larger than 64. This may be because in this case, an embedding size of 64 is sufficient to represent the items in the latent space. For the tail items, the effect of embedding size is more obvious. This may be because a larger embedding size can better represent the tail items.

[0239] Fig.21A and 21B It shows the results from LightGCN ( Fig.21A ) and GALR Example ( Fig.21B ) and visualized using t-SNE (as published by Maaten et al., “Data Visualization with t-SNE (2008)”). It can be seen that in LightGCN, the embeddings of the head and tail items are sparse, while in GALR, the embeddings of the head and tail items are close. Fig.21A and 21B In the figure, the lighter gray dots correspond to the head items, while the darker blocky dots correspond to the tail items. This indicates that information has been successfully transferred from the head items to the tail items, thereby facilitating the training process. Therefore, the method in this embodiment can improve the performance of the tail items.

[0240] Although the GALR embodiment described above mainly focuses on long-tail distribution from the item side, it should be understood that in other embodiments, GALR can also be used for long-tail user recommendation after appropriate modification. For example, a user-user similarity graph can be constructed and / or user-user homogeneous edges can be added. The integration and addition of user-user homogeneous edges and item-item homogeneous edges can make long-tail recommendations have two sides, namely, user side and item side.

[0241] The above embodiment provides an implementation example of a long-tail recommendation model named GALR (i.e., a long-tail recommendation adaptive enhancement framework based on graph contrastive learning), which is used to improve the recommendation performance of tail items. In some embodiments, the purpose of adaptive edge addition and edge discarding is to allow the tail items to learn some missing information. In some embodiments, data-level enhancement is to make the GNN-based model pay more attention to the tail. In certain embodiments, contrast loss through edge perturbation is used to learn a more uniform representation. Experiments conducted on multiple benchmark data sets have demonstrated the effectiveness of GALR.

[0242] As mentioned above, the long-tail distribution of user behaviors (user-item interactions) may result in reduced recommendation performance for items with fewer user records (i.e., tail items) (compared to items with more user records (i.e., head items)). In order to improve the recommendation performance of tail items, some methods have been provided, such as transferring the knowledge of head items to tail items through transfer learning, combining graph contrast learning to learn better representations for tail items, etc. However, these existing methods are not without problems. For example, some existing transfer learning methods rely heavily on rich item / user features, which may not exist in practice. For example, some existing graph contrast learning-based recommendation methods may only use one enhancement (in feature space or graph structure space), which may be suboptimal. For example, some existing graph contrast learning-based recommendation methods may only perform unified graph enhancement, which makes it difficult to learn the representation of tail items. In some embodiments of the present invention, a graph contrast learning-based framework is provided, which has an adaptive enhancement function (referred to herein as "GALR") for long-tail recommendation. GALR can solve one or more of the above problems. In some embodiments, GALR may include various features: performing adaptive edge addition by directly introducing high-order homogeneous information for tail items; performing adaptive edge discarding to retain more critical graph structure information of tail items; information can be transferred from well-learned head items to tail items to improve long-tail performance; applying a bilateral branch network to select appropriate item nodes for training data augmentation; incorporating contrastive learning to achieve more robust and unified representation learning, etc.

[0243] Example 2

[0244] To date, there are various approaches to solve the long-tail recommendation problem. These approaches include data-level approaches and algorithm-level approaches. For example, some data-level approaches may include resampling strategies to modify the data distribution through undersampling and oversampling techniques. For example, some algorithm-level approaches may include transfer learning-based methods to transfer the knowledge of head items to tail items to improve the recommendation performance of tail items. For example, some algorithm-level approaches may include modifying the network structure to solve the long-tail recommendation problem. For example, some algorithm-level approaches may utilize multi-objective optimization and adversarial methods to improve the long-tail recommendation performance.

[0245] The inventors of the present invention have found through research that existing solutions to the long-tail recommendation problem mainly focus on traditional neural network-based models, without utilizing graph-based recommendation models. The inventors of the present invention have found through research that existing graph-based recommendation models have limitations. For example, tail items may have limited graph connections, which may limit their learning ability due to low connectivity. During the propagation stage of the graph neural network (GNN), the sparse connectivity of the tail items in the graph inhibits sufficient information flow and limits the potential for learning meaningful representations. For example, the training loss is mainly caused by the head items, so learning the tail items is challenging. The optimization loss during training is often determined by more head nodes, which marginalizes the tail nodes and exacerbates the learning bias towards the head items. For example, an imbalance between head and tail items may lead to overfitting of head items and poor generalization of tail items.

[0246] Based on the above situation, the inventors of the present invention believe that it is necessary to develop an effective graph-based method to solve the long-tail recommendation problem.

[0247] In view of this, some embodiments of the present invention provide a framework for solving the long-tail recommendation problem. The framework is called the long-tail recommendation graph enhancement framework ("GALORE"). In some GALORE embodiments, the connectivity of the tail items is improved by adding item-to-item edges of the tail items, so that the tail items can receive information from nearby head items with stronger learning capabilities. This can improve the recommendation performance of the tail items. In some GALORE embodiments, a degree-aware edge discarding process is applied to retain relatively more important edges for the tail items and discard relatively unimportant edges for the head items. This process can promote the characterization learning of the tail items. In some GALORE embodiments, a node synthesis method for synthesizing new data is applied to alleviate the data sparsity of the tail items, thereby providing more training samples and alleviating the data sparsity problem. Synthetic data can be used as hard negative mining to improve model performance. In some GALORE embodiments, a multi-stage (such as two-stage) training strategy is used to promote the training process. In some embodiments, a graph enhancement framework is provided, which solves the long-tail recommendation problem by graph enhancement, including edge addition and edge discarding. By enhancing the graph, the graph connectivity of the tail items can be improved, thereby better learning the representation method of the tail items. In some embodiments, a node synthesis technique is provided to alleviate the data imbalance problem and allow the model to learn from more tail item data. In some embodiments, a two-stage training strategy is provided to enable the model to learn the representation of the head items and the tail items at different stages.

[0248] Regarding long-tail recommendation, the inventors of the present invention have expressed appreciation for various existing solutions through research. For example, Huang et al. disclosed a rebalancing solution in "Correcting sample selection bias by unlabeled data" (2006), which can directly generate resampling weights to select samples. For example, Park et al. disclosed a clustering tail method in "., The long tail of recommender systems and how to leverage it" (2008), which uses clustering technology to group tail items and recommend them according to the scores within the cluster. For example, Grozin et al.'s "Similar Product Clustering for Long-Tail Cross-Sell Recommendations" (2017) discloses a clustering method that uses user and item data and distance metrics to perform cross-sale recommendations based on association rule mining. For example, Yin et al.'s "Challenging the long tail recommendation" (2012) solves the long-tail recommendation problem by exploring the item-item similarity graph and using random walk technology to capture the preference similarity between items. For example, Volkovs et al. published DropoutNet in Dropoutnet: Addressing coldstart in recommender systems. Advances in neural information processing systems (2017), which applies dropout during training to address the cold start problem in recommender systems. For example, Liu et al. published TailNet in Long-tail session-based recommendation (2020), which uses a preference mechanism consisting of session representations and generative correction factors to adjust the recommendation model and recommend more long-tail items. For example, MIRec introduced by Zhang et al. in A model of two tales: Dual transfer learning framework for improved long-tail item recommendation (2021) includes a dual transfer learning framework that transfers model-level and item-level knowledge from the head to the tail.

[0249] The problem is that these existing long-tail recommendation algorithms are mainly designed for traditional neural networks and are therefore not suitable for graph-based recommendation models. Unlike these existing methods, in some GALORE embodiments detailed below, graph augmentation techniques are used to transfer information from the well-learned head items to the poorly learned tail items in the graph view. By promoting better representation learning for tail items, some GALORE embodiments can improve the recommendation performance of tail items.

[0250] In terms of graph enhancement, the inventors of the present invention have found through research that graph enhancement learning helps to alleviate the problem of incomplete or noisy data in the graph structure. For example, as disclosed in the book "Graph data augmentation for graph machine learning: A survey" (2022) by Zhao et al., the techniques of graph enhancement learning include node dropping, edge addition / dropping, and attribute completion. For example, Rong et al. disclosed DropEdge in "DropEdge: Towards Deep Graph Convolutional Networks on Node Classification" (2019), which randomly drops graph edges in a message passing mechanism to alleviate over-smoothing. For example, Wang et al.'s "Nodeaug: Semi-supervised node classification with data augmentation" (2020) creates a parallel universe for data enhancement for each node to deal with the negative impact from other nodes. For example, Zhao et al.'s "Data augmentation for graph neural networks" (2021) discloses GAUG, which introduces a graph data augmentation framework to improve the performance of GNN-based node classification through edge prediction. For example, Zhou et al.’s “Data augmentation for graph classification (2020)” discloses graph data augmentation and two heuristic algorithms, random mapping and pattern similarity mapping, which generate more weakly labeled data for small-scale benchmark datasets by heuristically modifying the graph structure.

[0251] However, the problem is that most of these existing technologies are used for homogeneous graphs, but in the recommendation scenario, the user-item graph in the recommendation scenario is a double square, so these existing technologies cannot be directly applied. The inventors of the present invention believe that it is necessary to study graph enhancement learning in recommendation, which may help to alleviate the adverse effects of long-tail distribution.

[0252] Further details of GALORE are now provided in some examples.

[0253] Without loss of generality, in one example, consider a set of users (denoted by U = {u}) and a set of items (denoted by I = {i}). Based on the interaction between users and items, a bipartite graph G = (V, E) can be constructed, where V = U∪I is the set of all users and items, and ε = U∪I represents the set of edges. If user u gives feedback to item i, there is an edge between user u∈U and item i∈I, that is, (u,i)∈ε. The interaction between users and items can also be represented by the matrix Y∈{0,1} |U|×|I| To represent, if user u gives feedback to item i (i.e. (u,i)∈ε, then Y u,i =1, otherwise Y u,i = 0. In this example, the cumulative item occurrences across all user-item interactions yield a long-tailed distribution.

[0254] The Top-K recommendation model aims to recommend the most relevant items to each user and maximize the average recommendation performance. However, the long-tail distribution means that some (or many) items may have almost no feedback, resulting in poor recommendation performance for these tail items. In addition to improving the average recommendation performance of all items, GALORE can also improve the recommendation performance of tail items. To this end, in the GALORE embodiment, graph enhancement is performed to transfer information from the head items to the tail items.

[0255] Fig. 22 The GALORE in certain embodiments is shown, which in its entirety includes various functions / modules.

[0256] In summary, in certain embodiments of GALORE, the graph enhancement method includes three operations / modules: edge addition, edge discarding, and node synthesis. The purpose of adding edges is to add item-item edges to strengthen the knowledge transfer from the head items to the tail items. The edge discarding module is used to selectively discard user-item edges to promote robust representation learning. The node synthesis module is used to generate synthetic item nodes to alleviate the long tail problem. In some embodiments, in GALORE, the training process includes two stages: the first stage is trained on the original bipartite graph to initialize item embeddings (especially better embeddings of head items); the second stage is trained on the enhanced graph to improve the representation of tail items. This training strategy can prompt the model to focus on the head and tail items respectively at different training stages.

[0257] In certain embodiments of GALORE, edge addition is applied.

[0258] In certain embodiments of GALORE, graph enhancement learning is applied to long-tail recommendations so that tail items can obtain some missing information from other items. Since head items have more connections than tail items, the model can usually learn better representations for head items. By connecting tail items to head items, the knowledge of head items can be transferred to tail items. More specifically, in certain embodiments, homogeneous edges, i.e., item-item edges, are added to improve the graph connectivity of tail items, thereby improving long-tail performance. To this end, an item-item similarity graph is first constructed to find structural neighbors for tail item nodes. Then, the embeddings of homogeneous neighbor nodes are clustered into different groups in the latent space to find semantic neighbors for tail item nodes. In this example, it is assumed that nodes in the same group have similar interests, while nodes in different groups have different interests. It should be noted that in certain embodiments of GALORE, both structural and semantic neighbors can improve the connectivity of the graph. In certain embodiments, message passing and aggregation are further performed on the enhanced graph.

[0259] In some embodiments, edge addition includes finding structural adjacencies. An item-item similarity graph can be constructed to find structural adjacencies. First, an item similarity matrix can be calculated based only on the interactions Y between users and items. The item similarity matrix is ​​an |I|×|I| matrix, and each element is the common interaction value between two items. In some embodiments, the common interaction value between two users is defined as the number of items they both interact with, and the common interaction value between two items is defined as the number of users they both interact with. Mathematically, the calculation formula for the item similarity matrix is

[0260] S I =Y T ·Y∈R |I|×|I| (17)

[0261] Note that in some other embodiments, the framework may use other defined co-interaction values, such as Pearson correlation, cosine distance, Jaccard similarity, etc. The co-interaction values ​​of each tail item may then be ranked to obtain structural neighbors.

[0262] In some embodiments, edge addition includes finding semantic neighbors. Some studies have shown that adding high-order neighbors may have an adverse effect on performance, especially when the interests of the neighbors are different. Therefore, naively adding homogeneous edges to all items may be suboptimal because it may result in the transfer of irrelevant information between items with different features, thereby hindering representation learning. Therefore, in some embodiments, it is better to identify semantic neighbors, that is, items with similar features and more likely to have meaningful relationships. To this end, in some embodiments, clustering is performed to group similar items together in the latent space. Clustering methods may include K-means (such as the K-means method disclosed by MacQueen et al. in "Some methods for classification and analysis of multivariate observations" (1967)), mean shift methods (such as Comaniciu et al. in "Mean shift: A robust approach toward featurespace analysis" (2002)), etc. In this example, the K-means method is applied to cluster the item embeddings into a set of groups, and the items in the same group are regarded as semantic neighbors.

[0263] In some embodiments, message passing is performed. After graph construction and enhancement, neighbor information is aggregated to strengthen the self-node representation. Fig.23 The information transmission operation of GALORE in some embodiments of the present invention is shown. Take the tail item i as an example. In this example, the structural neighbors of item i are first represented as Semantic neighbors are represented as Then, the homogeneous neighbors of item i are In this example, the heterogeneous neighbors of item i are defined as N e (i). Likewise, N o (u) is defined as the homogeneous neighbors of user u, N e (u) is defined as the heterogeneous neighbors of user u.

[0264] In one embodiment, to update the representation of an item node at the tth layer of the GNN, the GNN first aggregates the representations of its neighbors at the t-1th layer and then combines them with its own representation. This process can be expressed as follows

[0265]

[0266] in, is the ID embedding of item node i at layer t, AGGREGATE() represents the neighbor aggregation function, such as the average maxpooling operation, and COMBINE() represents the function that merges the aggregated neighbor representation with the node representation. One example is to use the average operation. Another example is to use the attention mechanism, such as

[0267]

[0268] in, is the normalized attention weight of the homogeneous neighbor node p in the tth layer, a i.t It means that each layer uses its own attention parameters, the operator || indicates connection, and LeakyReLU() indicates the activation function.

[0269] After obtaining the T-layer representation, the readout function can be used to generate the final representation for prediction:

[0270]

[0271] In one example, a weighted sum operation is used as READOUT().

[0272] In some embodiments, the user node also adopts similar steps. However, since the user node does not have homogeneous neighbor nodes, the process is described as follows:

[0273]

[0274] In certain embodiments of GALORE, edge discarding is applied.

[0275] In DropEdge: Towards Deep Graph Convolutional Networks on Node Classification by Rong et al. (2019), the efficacy of edge dropping in improving graph representation learning and alleviating the over-smoothing problem has been demonstrated. However, in the long-tail recommendation scenario, treating all edges equally may not be the best choice. In the long-tail recommendation setting, the connections of the tail item nodes are usually less than the head item nodes, so the edge connections of the tail items are more critical to achieve effective recommendation performance. However, indiscriminately dropping the edges of the tail items may destroy the local graph structure and negatively affect the information transfer process, resulting in degraded long-tail performance.

[0276] To solve this problem, some embodiments use adaptive heterogeneous edge discarding technology to discard more edges for head items and fewer edges for tail items. In some embodiments, the discrimination of indicators such as node degree may not be high enough, and the long tail coefficient l iis defined for item i and is used to quantify the degree to which a node is at the tail of the distribution. Specifically, in one example, the long-tail coefficient is defined as follows:

[0277] l i =min(log(deg(i)+1) -1 ,l c ) (twenty two)

[0278] Among them, deg(i) is the degree of item node i, l c is a cutoff value. The larger the long-tail coefficient value,

[0279] The less user feedback the item has.

[0280] The probability that the edge between user u and item i is dropped is p ui , the calculation formula is:

[0281]

[0282] Among them, l max and l min are the maximum and minimum values ​​of the item long tail coefficient, p e is a hyperparameter that controls the overall probability of discarding an edge, p c <1 is the cutoff probability.

[0283] In these embodiments, the recommendation performance is improved by discarding more edges for the head items and fewer edges for the tail items, while the local graph structure of the tail items is preserved, thereby solving the specific difficulties of the long-tail recommendation scenario.

[0284] In certain embodiments of GALORE, node synthesis is applied.

[0285] The inventors of the present invention have realized that synthetic minority oversampling, such as SMOTE (as disclosed by Chawla et al. in SMOTE: synthetic minority over-sampling technique (2002)) and Embed-SMOTE (as disclosed by Ando et al. in Deep over-sampling framework for classifying imbalanced data (2017)), is a model-independent technique for solving the problem of data imbalance, while the GraphSMOTE method (as disclosed by Zhao et al. in GraphSMOTE: Imbalanced node classification on graphs with graph neural networks (2021)) extends its application to graph data. The inventors of the present invention have realized that directly applying GraphSMOTE to recommendation tasks may not be optimal because in recommendation systems, graph edges have special meaning because they represent user preferences for items, and the edge generator in GraphSMOTE generates edges based on model predictions, which introduces noise and negatively affects model performance, especially for tail nodes with inaccurate predictions.

[0286] To this end, in some embodiments, after obtaining user / item representations through GNN, but before the optimization process, synthetic nodes need to be added. This is because adding synthetic nodes after obtaining embeddings will negatively affect the connectivity of existing tail item nodes. To address the problem of GraphSMOTE, some embodiments use data mixing for node synthesis. In some examples, two items i and j are first randomly selected, and then synthetic item nodes are generated based on the selected items. Crafting Item Nodes The embedding of is the embedding e of item i and item j i and e j The convex combination of

[0287]

[0288] Where λ∈[0,1] is a hyperparameter. In one example, when λ>0.5, the synthetic items Connected to the same user as item i; otherwise, synthesize the item are connected to the same user as item j. When λ = 0 or 1, the mixing operation is equivalent to the data resampling operation.

[0289] In some embodiments, node synthesis involves hard negative mining. The basic idea of ​​hard negative mining is to pay more attention to examples that are difficult to classify or rank correctly. In a recommendation system, hard negative mining can be used to identify negative samples that are difficult for the recommendation model to distinguish from positive samples. Therefore, these samples require more attention from the model, thereby improving model performance.

[0290] In some embodiments, synthetic nodes can be used as hard negative mining to improve model performance. For example, assuming that user u interacts with item i but not with item j, item i is a positive sample and item j is a negative sample. By mixing the embeddings of i and j, a synthetic node can be generated. In this case, the composite node is harder to tell whether it is positive or negative than item i or item j because it contains information from both items. Therefore, the model must be more nuanced in distinguishing such hard negative samples. This in turn improves the model's ability to distinguish between positive and negative samples.

[0291] By using synthetic nodes as negative samples, examples that the model has difficulty correctly classifying can be identified. These synthetic nodes may contain information about both positive and negative samples, making them informative false negatives. By including these synthetic nodes in the training data, the model is forced to better distinguish between positive and negative samples, thereby improving performance.

[0292] In certain embodiments of GALORE, a two-stage training strategy is employed.

[0293] In some embodiments, the two-stage training strategy includes: in the first stage, following the standard graph-based recommendation model training process, the main goal is to promote the learning of high-quality representations of head items; in the second stage, graph enhancement (edge ​​addition, edge discarding and / or node synthesis) is adopted. Since the learning of tail items is usually more challenging, the tail of the graph is enhanced when the model is trained in the second stage.

[0294] One advantage of this two-stage training strategy is that it allows the model to focus on different aspects of the representation learning problem at different training stages. Specifically, the model first masters the learning method of the head, which is relatively easy to learn, and then learns the more complex tail in the next stage.

[0295] In certain embodiments of GALORE, an optimization strategy is applied.

[0296] In some embodiments, after obtaining user embedding and item embedding through GNN, the inner product (of user and item embedding) is obtained to estimate the user's preference for the target item. Specifically, the model prediction of user u for item i is expressed as:

[0297]

[0298] Among them, e u and e i are the embeddings of user u and item i respectively.

[0299] To encourage the prediction value of observed user-item pairs to be higher than the prediction value of unobserved user-item pairs, some implementations use the Bayesian Personalized Ranking (BPR) loss (as disclosed in Rendle et al., BPR: Bayesian Personalized Ranking from implicit feedback (2012)), which is defined as follows:

[0300]

[0301] Among them, N u is the set of items observed to interact with user u, and σ is the sigmoid function.

[0302] In some embodiments, the overall loss is the sum of the recommendation loss and the regularization loss, defined as follows

[0303]

[0304] Wherein, Θ is a model parameter vector, and γ is a hyperparameter controlling the strength of the regularization loss L2. In some embodiments, an Adam optimizer (as described in the book Adam: A method for stochastic optimization (2014) by Kingma et al.) is used to optimize the loss function to minimize the loss.

[0305] In order to evaluate the performance of GALORE in certain embodiments and analyze its components, experiments were conducted on four real-world datasets from different domains. Specifically, the experiments aim to determine how GALORE performs compared to existing baseline methods in terms of all items and tail items, determine how adding graph augmentation and training strategies to GALORE affects the recommendation performance, and determine how hyperparameters affect the performance of GALORE.

[0306] In this example, the experiments are conducted on four public real-world datasets, as detailed in Fig.24 As shown:

[0307] (1)MovieLens(https: / / grouplens.org / datasets / movielens / )

[0308] MovieLens is a movie rating dataset used to evaluate recommendation algorithms. This example uses MovieLens-1M (ML-1M), which contains about 1,000,000 user ratings.

[0309] (2) Douban Books (https: / / github.com / librahu / HIN-Datasets-for-

[0310] Recommendation-and-Network-Embedding)

[0311] Douban Books contains user ratings of books.

[0312] (3)Yelp2018(https: / / www.yelp.com / dataset / challenge)

[0313] Yelp2018 comes from the Yelp-2018 Challenge, which is related to the task of recommending business web pages to users.

[0314] (4) Gowalla (mentioned in He et al.’s LightGCN: Simplifying and powering graph convolution network for recommendation (2020))

[0315] Gowalla is a dataset that contains check-in information of users who have shared their locations.

[0316] Figures 25A to 25D The distribution of items in all four datasets is shown ( Fig.25A : ML-1M dataset; Fig.25B : Douban book dataset; Fig.25C : Yelp2018 dataset; Figure 25: Gowalla dataset). FIG. 25A to FIG. 25D It can be seen that the item distributions in these four datasets all exhibit a highly skewed long-tail pattern. This feature makes these datasets particularly suitable for experiments to address the challenge of improving recommendation quality for long-tail distributions.

[0317] In the experiments, GALORE in certain embodiments is compared with existing model-independent baseline methods, including data reweighting, loss function refinement, meta-learning, and graph reinforcement learning methods. To ensure fair comparison, LightGCN is used as the backbone model for all methods. All methods use the same layer and embedding dimensions. Specifically, the baseline methods used in the experiments are as follows:

[0318] (1) Backbone model:

[0319] LightGCN (see He et al., “LightGCN: Simplifying and powering graph convolution network for recommendation” (2020)):

[0320] A lightweight model that simplifies the weight parameter aggregation step in graph convolutional networks.

[0321] (2) Resampling strategy (such as Brownlee's "Imbalanced classification with Python: better metrics, balance skewed classes, cost-

[0322] Sensitive Learning (2020):

[0323] Oversampling: Sample more from the tail items to rebalance the head and tail items in the training data distribution.

[0324] Undersampling: The opposite of oversampling, keeping the tail items unchanged and reducing the sampling of the head items.

[0325] (3) Loss function improvement method:

[0326] Focal Loss (FL) (as published by Lin et al. in Focal loss for dense object detection (2017)): reweights the standard loss to assign less weight to well-classified examples.

[0327] ClassBalance (as published in Cui et al., Class-balanced loss based on effective number of samples (2019)): Calculate the effective number of samples and reweight the loss for each class.

[0328] (4) Meta-learning model:

[0329] MeLu (as published by Lee et al. in “MeLU: Meta-learned user preference estimator for cold-start recommendation” (2019)): applies optimization-based meta-learning (MAML) for cold-start item rating prediction.

[0330] (5) Graph Enhanced Learning Model:

[0331] DropEdge (as in Rong et al., DropEdge: Towards Deep Graph Convolutional Networks on Node Classification (2019)

[0332] ): Randomly discard graph edges in the message passing mechanism to alleviate over-smoothing.

[0333] ·GraphSMOTE (such as Zhou et al. in "GraphSMOTE:Imbalanced node classification on graphs with graph neural networks"

[0334] (2021)): A supersampling technique that uses graph-based synthetic minority supersampling to address the class imbalance problem in graph-structured data.

[0335] The experiments in this example use a variety of settings. In this example, following the experimental settings published in Yu et al.'s "GraphAugmentation Learning" (2022), the ratings with a rating of less than 4 in the Movielens and Douban Books datasets (the rating scale is 1-5) are discarded, and the remaining ratings are reset to 1. The dataset is divided into three parts (training set, validation set, and test set) with a ratio of 7:1:2. The Pareto principle (as described in Box et al.'s book "An analysis for unreplicated fractional factorials" (1986)) is used as a criterion for dividing head and tail items. The items with the top 20% of the number of occurrences are set as head items, and the rest of the items are set as tail items. Evaluation metrics for the tail and head item sets are reported separately. The average performance of five runs is reported.

[0336] One of the settings is related to the evaluation metrics. In this example, two evaluation metrics are used to evaluate the performance: (1) Recall, which measures the probability that the recommended list contains the items that the user is interested in; (2) a weighted version of the hit rate (HR), called the normalized discounted cumulative gain (NDCG), which is used to increase the weight of the items ranked higher in the recommended list. In this example, Recall@K (or R@K for short) and NDCG@K (or N@K for short), K∈{20,50}, are evaluated.

[0337] One of the settings is related to hyperparameters. In this case, for a fair comparison, the best hyperparameter settings reported in publications related to the benchmark methods were consulted and all hyperparameters of the benchmark methods were fine-tuned via grid search. For the general settings for all benchmark methods, Xavier initialization (as published by Glorot et al. in Understanding the difficulty of training deep feedforward neural networks (2010)) was used for all embeddings. In this case, the embedding size was 64 and the L2 regularization parameter was 10 -4 , with a batch size of 2048. All models are optimized using the Adam optimizer with a learning rate of 0.001. For GALR, the number of structural neighbors is selected from {3, 5, 10}, and the number of clusters is selected from {3, 5, 10, 20}. In the experiments, for GALORE, the number of structural neighbors is selected from {1, 3, 5, 10}, the overall edge drop rate is selected from {0.1, 0.3, 0.5}, and λ is selected from {0.8, 0.9, 1.0}.

[0338] The performance of the baseline methods and GALORE in certain embodiments are evaluated using these four datasets to explore the performance of the models in different situations. Fig.26A and 26B The tables in summarize the recommendation performance for all items, head items, and tail items on these datasets.

[0339] Based on the recommended performance obtained, the following conclusions can be drawn:

[0340] First, the GALORE implementation outperforms the LightGCN model by a large margin, which shows that it is effective in enhancing both tail and head item recommendations. This emphasizes the importance of considering graph augmentation in long-tail item distributions. The results also show that GALORE's training strategy can improve recommendation performance on long-tail distribution data.

[0341] Second, in almost all cases, GALORE's embodiments outperform reweighting methods, including undersampling and oversampling. Notably, oversampling performs slightly worse than undersampling on the tail item split. However, resampling methods may change the distribution of the original data, which may negatively affect the overall performance of the model. In addition, experiments show that meta-learning and loss function refinement strategies cannot achieve excellent performance on both head and tail items. This suggests that meta-learning methods may get into trouble without model parameters and user / item characteristics. As for the loss function refinement strategy, its trade-off between head and tail items may have an adverse impact on the overall performance.

[0342] Third, the performance of the GALORE implementation is better than that of graph enhancement models such as DropEdge and GraphSMOTE in most cases. These findings show that the customized adaptive graph enhancement technology in the GALORE implementation is very effective in improving the performance of long-tail recommendations. In addition, the two-stage training method in the GALORE implementation can enhance the representativeness of learning by assigning different learning objectives to each stage.

[0343] Ablation studies were also performed to analyze different components of the GALORE embodiment. Fig. 27 Results of ablation studies are shown. The full GALORE embodiment outperforms all GALORE variants on three splits, indicating that all major components in this GALORE embodiment can be used to improve performance.

[0344] The four GALORE variants and their results are described below.

[0345] One of the GALORE variants has no edge addition. In this variant, adaptive edge addition is not used, while other components / operations remain the same. The performance of the model without adaptive edge addition drops significantly on the tail items, suggesting that the head items can pass useful information to the adjacent tail items through adaptive edge addition as hypothesized.

[0346] One of the GALORE variants does not have edge drop. In this variant, adaptive edge drop is replaced by random uniform edge drop, while other components / operations remain the same. Without adaptive edge drop, the model performs relatively poorly on tail items. This suggests that adaptive edge drop is very useful for improving tail item recommendations as it preserves key graph structure information of tail items.

[0347] One of the GALORE variants does not have a synthesis node. In this variant, node synthesis is not used, while other components / operations remain the same. The model's performance is significantly degraded in the tail, highlighting the effectiveness of the synthesis node in alleviating the data imbalance problem. Without the synthesis node, the model tends to overfit the head data, resulting in poor and unfair performance on the tail data.

[0348] One of the GALORE variants does not have two-stage training. In this variant, the first training stage is removed and only the second training stage is used (training the model on the augmented graph). The results show that the performance of the head items is severely degraded, which shows the role of two-stage learning. With two-stage learning, the model can learn different parts of the graph in each stage, thus avoiding overfitting on the tail which is difficult to learn.

[0349] These ablation studies show that all major components of the GALORE implementation contribute to its performance. Edge addition, edge drop, synthetic nodes, and two-stage learning all contribute to improving the recommendation performance, especially for tail items.

[0350] In addition, a hyperparameter study was performed.

[0351] One of the hyperparameter studies was related to the impact of edge dropout rate. Fig.28 A clear relationship between the total edge drop rate and the performance of the model in all three splits is shown. Specifically, as the edge drop rate increases, the overall performance of the model decreases. For example, when the edge drop rate is set to 0.9, the generated graph structure is highly chaotic. In this case, it becomes very difficult for the graph neural network to obtain meaningful information and effectively optimize the model. In addition, an interesting trend in the tail performance can be observed, that is, as the edge drop rate increases, the tail performance first increases and then decreases. This may be due to the fact that discarding the edges of the head item nodes can improve the graph connectivity of the tail item nodes, thereby promoting graph information transfer and representation learning. This finding suggests that the edge drop rate is a useful parameter to consider when training graph-based models because it affects the performance of the model. Careful selection of the edge drop rate can help strike a balance between maintaining graph connectivity and promoting effective representation learning.

[0352] One of the hyperparameter studies is related to the impact of synthetic node rate. This study was conducted on the Douban Books dataset. Figures 29A to 29C The experimental results of studying the effect of synthetic node rate on model performance are presented ( Fig.29A , all items; Fig.29B , head items; Fig.29C , tail items). The synthetic node rate indicates the ratio of synthetic tail nodes to real tail nodes. The higher the synthetic node rate, the more synthetic nodes are generated. This study reveals the overall performance (all items, Fig.29A ), that is, as the synthesis node rate increases, the overall performance first increases and then decreases. In contrast, the tail performance (tail items, Fig.29C ) continues to rise with the increase of synthesis node rate, while head performance (head items, Fig.29B ) shows the opposite trend. This observation is in line with expectations, as generating more synthetic nodes can generally alleviate the data imbalance problem in the graph and improve long-tail performance. Specifically, an increase in the number of synthetic nodes helps provide more training examples for tail items, which are often underrepresented in the data. By facilitating more effective representation learning for such items, synthetic nodes help improve the model's tail performance.

[0353] One of the hyperparameter studies is related to the impact of embedding size. This study investigates the impact of embedding size as a hyperparameter on the performance of the model in three splits. FIG. 30A to FIG. 30C Presents experimental results on the effect of embedding size on model performance ( Fig. 30A , all items; Fig. 30B , head items; Fig. 30C , tail items). It can be seen from the figure that as the embedding size increases, the model performance continues to improve in all three splits. One possible explanation for this phenomenon is that larger embedding sizes can capture more complex patterns and richer semantic information, thereby improving expressiveness and performance. There is a clear improvement in performance when the embedding size increases from 32 to 64. Nonetheless, it should be noted that increasing the embedding size comes at the cost of requiring more computational resources, which may not always be feasible or desirable in practice. Therefore, it is very useful to choose an appropriate embedding size that strikes a balance between model performance and computational cost.

[0354] Although the GALORE embodiments described above mainly focus on solving the long-tail distribution problem from the perspective of items, it should be understood that in other embodiments, GALORE can also be applied to long-tail user recommendations by appropriate modifications. For example, homogeneous user-user edges and / or synthetic user nodes can be added to facilitate long-tail user recommendations. Integrating these modifications can achieve long-tail recommendations for users and items, thereby improving overall recommendation performance.

[0355] In some embodiments, GALORE is a plug-and-play and model-agnostic framework, so more powerful graph-based backbone models (e.g., SGL (as disclosed by Wu et al. in "Self-supervised graph learning for recommendation" (2021)), SimGCL (as disclosed by Yu et al. in "Are graph augmentations necessary? Simple graph contrastive learning for recommendation" (2022)) can be used to achieve better performance.

[0356] The above embodiments provide some implementation examples of the long-tail recommendation model GALORE (i.e., graph enhancement for long-tail recommendation) for improving the recommendation performance of tail items. In some embodiments, edge addition and adaptive edge discarding are applied to improve the graph connectivity of tail items and enhance the representation learning process. In some embodiments, node synthesis is applied to modify the data distribution so that the model pays more attention to the tail. In some embodiments, in order to further improve the performance of the model, a two-stage training scheme is used to supervise the model learning process to prevent overfitting of the tail. This method allows the model to learn better representations at different stages, thereby improving the performance of head and tail items. Experiments conducted on several benchmark datasets have demonstrated the effectiveness of GALORE.

[0357] As mentioned above, the long-tail distribution of inherent user behaviors will result in the recommendation performance of items with fewer user records (i.e., tail items) being inferior to that of items with more user records (i.e., head items). Existing technologies for alleviating the long-tail recommendation problem mainly focus on traditional methods, and lack graph-based methods that can be used to efficiently handle the long-tail recommendation problem. In some embodiments of the present invention, a graph enhancement framework (GALORE) for long-tail recommendation is proposed, which can be inserted into (applied to or appear simultaneously with) various graph-based recommendation models to improve the performance of tail items. In some embodiments, GALORE may include various functions: edge addition, which enriches the connectivity of the tail item graph by injecting additional items into the item edges; degree-aware edge discarding, which retains the more valuable edges in the tail items while selectively discarding the less informative edges in the head items; synthesizing new data samples to solve the data scarcity problem of tail items; applying a two-stage training strategy to promote the learning of head and tail items, etc.

[0358] Example System

[0359] Fig.31An example of a data processing system 3100 in some embodiments of the present invention is shown. The data processing system 3100 may be used to implement at least partially or fully one or more operations, methods, models, frameworks, etc. of the present invention. For example, the data processing system 3100 may be used to implement any operations, methods, models, frameworks, etc. described with reference to the accompanying drawings. The data processing system 3100 generally includes appropriate components required to receive, store, and execute appropriate computer instructions, commands, and / or codes. The main components of the data processing system 3100 are a processor 3102 and a memory (memory) 3104. The processor 3102 may include one or more: one or more CPUs, one or more MCUs, one or more GPUs, one or more logic circuits, one or more Raspberry Pi chips, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or any other digital or analog circuits / circuits that are configured to interpret and / or execute program instructions and / or process signals and / or information and / or data. In some embodiments, the processor 3102 may be used to train and / or operate a GNN-based recommendation model and system. The processor 3102 may be used to partially or fully execute any one or more method embodiments of the present invention. The memory 3104 may include one or more volatile memories (such as RAM, DRAM, SRAM, etc.), one or more non-volatile memories (such as ROM, PROM, EPROM, EEPROM, FRAM, MRAM, FLASH, SSD, NAND, NVDIMM, etc.), or any combination thereof. Appropriate computer instructions, commands, codes, information and / or data may be stored in the memory 3104. For example, the memory 3104 may store training / testing / validation data for a GNN-based recommendation model and system. For example, the memory 3104 may store original or trained GNN-based recommendation models and systems. Computer instructions for executing or facilitating the execution of the method embodiments of the present invention may be stored in the memory 3104. For example, the memory 3104 may partially or completely store program instructions or codes for executing any one or more methods in the method embodiments of the present invention. The processor 3102 and the memory (storage) 3104 may be integrated or separated (and operably connected). In some embodiments, the data processing system 3100 also includes one or more input devices 3106. Examples of such input devices 3106 include: keyboard, mouse, stylus, image scanner, microphone, tactile / touch input device (such as touch sensitive screen), image / video input device (such as camera), etc. In some embodiments, data processing system 3100 also includes one or more output devices 3108. Examples of such output devices 3108 include: display (such as monitor, screen, projector, etc.), speaker, earphone, headset, printer, additive manufacturing machine (such as 3D printer), etc.The display screen may include an LCD display screen, an LED / OLED display screen, or other suitable display screen, which may be touch-sensitive or non-touch-sensitive. The display may display the training and / or running results of the GNN-based recommendation model and system. The data processing system 3100 may further include one or more disk drives 3112, which may include one or more of a solid-state drive, a hard disk drive, an optical drive, a flash drive, a tape drive, etc. A suitable operating system may be installed in the data processing system 3100, for example, on the disk drive 3112 or in the memory 3104. The memory 3104 and the disk drive 3112 may be operated by the processor 3102. In some embodiments, the data processing system 3100 also includes a communication device 3110 for establishing one or more communication links (not shown) with one or more other computing devices (such as servers, personal computers, terminals, tablet computers, mobile phones, watches, Internet of Things devices, or other wireless computing devices). The communication device 3110 may include one or more of the following devices: a modem, a network interface card (NIC), an integrated network interface, an NFC transceiver, a ZigBee transceiver, a Wi-Fi transceiver, etc. Transceiver, RF transceiver, cellular (2G, 3G, 4G, 5G, 5G and above or similar) transceiver, optical port, infrared port, USB connection or other wired or wireless communication interface. The transceiver can be implemented by one or more devices (integrated transmitter and receiver, independent transmitter and receiver, etc.). The communication link can be wired or wireless and is used to communicate commands, instructions, information and / or data. In one example, the processor 3102, memory 3104 (in some embodiments, input device 3106, output device 3108, communication device 3110 and disk drive 3112 (if any)) are directly or indirectly connected to each other via a bus, peripheral component interconnect (PCI) (such as PCI Express), universal serial bus (USB), optical bus or other similar bus structure. In some embodiments, at least some of these components can be connected wirelessly, for example through a network such as the Internet or a cloud computing network. It will be appreciated by those skilled in the art that Fig.31 The data processing system 3100 shown is merely an example, and in other embodiments, the data processing system 3100 may have a different configuration (eg, include additional components, have fewer components, include alternative components, etc.).

[0360] Although not required, the embodiments described with reference to the accompanying drawings may be implemented as an application programming interface (API) or a series of libraries for use by developers, or may be included in another software application, such as a terminal or computer operating system or a portable computing device operating system. Generally speaking, since program modules include routines, programs, objects, components, and data files that assist in performing specific functions, technicians will understand that the functionality of a software application may be distributed among multiple routines, objects, and / or components to achieve the same functionality required herein.

[0361] In addition, it is also understood that if the methods and systems of the present invention are implemented entirely or in part by a computing system, any suitable computing system architecture may be used. This includes stand-alone computers, network computers, dedicated or non-dedicated hardware devices. When the terms "computing system" and "computing device" are used, these terms are intended to include (but not limited to) any suitable arrangement of computers or information processing hardware capable of implementing the described functions.

[0362] Embodiments of the present invention provide various functions.

[0363] For example, some embodiments provide an adaptive enhancement framework for solving the long-tail recommendation problem using edge addition and graph topology enhancement. For example, some embodiments use training data enhancement to force GNN-based models to learn more about the tail. For example, in some embodiments, instead of selecting low-degree nodes for enhancement, nodes are selected based on performance improvement through a bilateral branch network.

[0364] For example, some embodiments provide a graph enhancement framework that solves the long-tail recommendation problem through graph enhancement, which includes edge addition and edge discarding. Graph enhancement can improve the graph connectivity of tail items and promote learning better tail item representations. For example, some embodiments use node synthesis technology to alleviate the data imbalance problem and allow the model to learn from more tail item data. For example, some implementation schemes use a two-stage training strategy to enable the model to learn the representation of head items and tail items at different stages.

[0365] Embodiments of the present invention provide various functions. For example, some embodiments improve the recommendation performance of tail items in the long-tail distribution of user behavior. For example, in some embodiments, the proposed framework can enhance representation learning by using adaptive edge dropping and adding, introducing high-order homogeneous information for tail items, and combining contrastive learning to achieve improved recommendation performance.

[0366] Embodiments of the present invention can be applied to various recommendation systems / applications, such as e-commerce, online streaming, social media, and personalized content delivery. These systems may encounter the problem of long-tail distribution of user behavior, in which case there are few user records for many tail items, making it challenging to make accurate recommendations. The present invention can enhance the overall user experience of these systems by improving the recommendation performance of tail items. In some embodiments of the present invention, "items" can include digital images, photos, electronic documents or files, web pages, parts of web pages, maps, electronic links, commercial products, non-commercial products, multimedia files, songs, books, albums, articles, database records, people, objects, etc.

[0367] Embodiments of the present invention have various advantages. For example, certain embodiments of the present invention improve the performance of tail items. Some embodiments of the present invention provide a more effective, efficient and / or practical solution for improving the recommendation performance of tail items in the long tail distribution of user behavior.

[0368] Those skilled in the art will appreciate that the described and / or illustrated embodiments of the present invention may be varied and / or modified to provide other embodiments of the present invention. Therefore, the embodiments described and / or illustrated in the present invention should be considered in all respects as exemplary and not restrictive. Examples of optional features of certain embodiments of the present invention are provided in the summary and description. Some embodiments of the present invention may include one or more of these optional features (some of which are not specifically described in the drawings). Some embodiments of the present invention may lack one or more of these optional features (some of which are not specifically described in the drawings). For example, the neural network-based ranking model of the present invention can have different network architectures, i.e., not limited to the architecture specifically described or illustrated.

Claims

1. A computer-implemented method for training a graph neural network-based recommendation system, comprising: receiving a data set comprising user data associated with users, item data associated with items, and user-item interaction data associated with user-item interactions, wherein some of said items have fewer said user-item interactions than some other of said items; Processing the data set to generate graph representation data of the data set; the graph representation data includes data associated with nodes and data associated with edges connecting the nodes; the nodes include user nodes corresponding to the users and item nodes corresponding to the items; the edges include user-item interaction edges corresponding to the user-item interactions; each user-item interaction edge is connected to one item node and one user node, wherein some item nodes have fewer user-item interaction edges than other item nodes; Processing the graph representation data of the data set to modify the graph representation data of the data set and obtain modified graph representation data of the data set, which helps to learn or determine representations of at least some of the item nodes with less user-item interactions; as well as Based at least in part on the modified graph representation data, one or more graph neural network based recommendation models of the graph neural network based recommendation system are trained.

2. The computer-implemented method of claim 1, wherein the amount of the user-item interaction associated with the item substantially follows a heavy-tailed or long-tailed distribution. 3 . The computer-implemented method of claim 1 , wherein processing the graph representation data of the data set comprises performing an edge addition operation to add one or more edges to the graph representation data.

4. The computer-implemented method of claim 3, wherein performing an edge addition operation comprises: A homogeneous edge adding operation is performed to add one or more item-item edges of at least some of the item nodes having fewer of the user-item interaction edges to the graph representation data, wherein the one or more item-item edges are respectively connected to two item nodes, at least one of the two item nodes is one of the item nodes having fewer of the user-item interaction edges.

5. The computer-implemented method of claim 4, wherein performing a homogeneous edge addition operation comprises: For each node in at least part of the item nodes with less user-item interaction: determining one or more structurally adjacent item nodes from the item nodes; as well as One or more item-item edges between a corresponding item node and its one or more structurally adjacent item nodes are added to the graph representation data.

6. The computer-implemented method of claim 5, wherein said determining one or more structural neighboring item nodes comprises: Determining an item similarity matrix according to the user-item interaction, the item similarity matrix comprising common interaction values, each of the common interaction values ​​corresponding to two of the items; as well as The one or more structurally adjacent item nodes are determined according to the common interaction values ​​in the item similarity matrix.

7. The computer-implemented method of claim 4, wherein performing the homogeneous edge adding operation comprises: For each node in at least some of the item nodes with less user-item interaction: determining one or more semi-adjacent item nodes from the item nodes; as well as One or more item-item edges between a corresponding item node and one or more semi-neighboring item nodes are added to the graph representation data.

8. The computer-implemented method of claim 7, wherein determining the one or more semi-neighboring item nodes comprises: Clustering the objects according to a clustering method to form a cluster of multiple objects; as well as One or more item-item edges are added between the corresponding item node and an item in the same cluster as the corresponding item node.

9. The computer-implemented method of claim 8, wherein the clustering is based on a K-means method or a mean shift method.

10. The computer-implemented method of claim 8, wherein performing the homogeneous edge adding operation comprises: The message passing operation is performed based at least in part on the graph representation data that adds one or more item-item edges between the corresponding item node and its one or more structural neighboring item nodes, and one or more item-item edges between the corresponding item node and its one or more semi-structural neighboring item nodes.

11. The computer-implemented method of claim 3, wherein processing the graph representation data of the data set further comprises: An edge discarding operation is performed to discard one or more of the user-item interaction edges.

12. The computer-implemented method of claim 11, wherein: Executing the edge discard operation includes: An adaptive heterogeneous edge discarding operation is performed to discard part of the user-item interaction edges, so that the number of the user-item interaction edges discarded for at least part of the item nodes with less user-item interaction is less than the number of the user-item interaction edges discarded for the item nodes with more user-item interaction.

13. The computer-implemented method of claim 12, wherein performing the adaptive heterogeneous edge discarding operation comprises: For each of the at least some item nodes, respectively: determining a degree of insufficient user interaction for the item node; as well as According to the determined degree, user-item interaction edges associated with the item nodes are discarded, so that more of the user-item interaction edges are discarded for the item nodes with less insufficient user interactions, and fewer of the user-item interaction edges are discarded for the item nodes with more insufficient user interactions.

14. The computer-implemented method of claim 11 , wherein processing the graph representation data of the data set further comprises: A node synthesis operation is performed to add one or more synthetic item nodes and one or more corresponding synthetic user-item interaction edges to the graph representation data.

15. The computer-implemented method of claim 14, wherein performing the node composition operation comprises: Processing the graph representation data of the dataset using one or more graph neural network based recommendation models of the graph neural network based recommendation system to determine embeddings associated with the user nodes and embeddings associated with the item nodes; performing a data blending operation to generate one or more composite item nodes based at least in part on the embedding associated with the user node and the embedding associated with the item node; as well as For each of the one or more synthetic item nodes, a corresponding synthetic user-item interaction edge is generated based at least in part on the hyperparameter.

16. The computer-implemented method of claim 14, wherein performing the node composition operation comprises: Processing the graph representation data of the dataset using one or more graph neural network based recommendation models of the graph neural network based recommendation system to determine a set of data including user-item-interaction triples associated with the graph representation data and corresponding embeddings; performing a first data augmentation operation on the dataset to generate a first synthetic dataset having one or more synthetic user-item-interaction triples and one or more corresponding synthetic embeddings; Processing the dataset and the first synthetic dataset using a bilateral branching network model to compare or determine the performance of the item node; Selecting an item node for performing data augmentation based on the comparison or determination result; as well as A second data augmentation operation is performed on the dataset of the selected item nodes to generate a second synthetic dataset including one or more synthetic item nodes and one or more corresponding synthetic user-item interaction edges.

17. The computer-implemented method of claim 16, wherein the bilateral branch network model comprises two substantially identical branches, each branch comprising one or more graph neural network-based recommendation models; as well as The processing of the data set and the first synthetic data set includes: Processing the data set using a branch of the bilateral branch network model; as well as The combination of the dataset and the first synthetic dataset is processed using another branch of the bilateral branch network model.

18. The computer-implemented method of claim 16, wherein: The first data enhancement operation includes a data mixing operation and / or a data resampling operation; The second data enhancement operation includes a data mixing operation and / or a data resampling operation; The data mixing operation includes mixing one or more pairs of user-item-interaction triplets and their corresponding embeddings to generate synthetic data; and The data resampling operation includes selectively extracting and discarding at least some of the user-item-interaction triplets and corresponding embeddings from the dataset.

19. The computer-implemented method of claim 1, wherein the training comprises: The one or more graph neural network-based recommendation models of the graph neural network-based recommendation system are trained using graph contrastive learning technology.

20. The computer-implemented method of claim 19, wherein the training comprises optimizing a loss function that takes into account a recommendation loss, a regularization loss, and a contrastive loss.

21. The computer-implemented method of claim 1, wherein the training comprises: In a first training phase, one or more graph neural network-based recommendation models of the graph neural network-based recommendation system are trained using the graph representation data; as well as In the second training stage, the modified graph representation data is used to train one or more graph neural network-based recommendation models of the graph neural network-based recommendation system.

22. The computer-implemented method of claim 21, wherein the training comprises optimizing a loss function that considers at least a recommendation loss and a regularization loss.

23. A system for training a recommendation system based on a graph neural network, comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by one or more processors, the one or more programs comprising instructions for performing or facilitating the performance of the computer-implemented method of claim 1.

24. A computer-implemented method for a recommendation system based on a graph neural network, comprising: One or more graph neural network-based recommendation models of the graph neural network-based recommendation system are used to process user data and item data, and the recommendation model is trained using the computer-implemented method of claim 1 to determine item recommendations for the user.

25. An operating system for a recommendation system based on a graph neural network, comprising: one or more processors; as well as A memory storing one or more programs configured to be executed by one or more processors, the one or more programs comprising instructions for performing or facilitating the performance of the computer-implemented method of claim 24.

Images