Apparatus and method for network-embedded structured graph convolutional network with random mask

Abstract

A method includes receiving a first data set including first and second types of embeddings, generating a fixed adjacency matrix from the first data set, and applying a first random binary mask to the fixed adjacency matrix to obtain a first subgraph of the fixed adjacency matrix. The method also includes processing the first subgraph through a first layer of a graph convolution network (GCN) to obtain a first embedding matrix, and applying a second random binary mask to the fixed adjacency matrix to obtain a second subgraph of the fixed adjacency matrix. The method includes processing the first embedding matrix and the second subgraph through a second layer of the GCN to obtain a second embedding matrix, and then determining a plurality of gradients of a loss function, and using at least one gradient of the plurality of gradients to modify the first random binary mask and the second random binary mask.

Description

[0001] Cross-references to related applications

[0002] This application is a PCT application that claims priority and benefit to U.S. Provisional Patent Application No. 63 / 146,089, filed February 5, 2021, the entire contents of which are incorporated herein by reference. Background Technology

[0003] Recommender systems can be used to provide users with suggestions based on item ratings. They are widely used across various fields. For example, social media networks can use recommender systems to provide users with lists of other users on the platform based on ratings that include features such as location, educational history, and workplace. Another example could be online marketplaces, which can use recommender systems to users based on data such as previous purchases and search history. A prominent technique for implementing such recommender systems is collaborative filtering, which takes users' historical data and assumes that users who share similar preferences in previous data are likely to make similar decisions in the future. However, many recommender systems are unable to learn high-order user-item feature interactions.

[0004] Graph convolutional networks (GCNNs) are a powerful representation learning framework for collaborative filtering. A key component is exploring message passing, or some neighborhood aggregation mechanism, to extract high-level representations of users and items. However, many real-world user-item graphs are often incomplete and noisy, which can lead to suboptimal performance of GCNNs if not properly regularized.

[0005] The implementation scheme disclosed herein addresses these and other issues individually and collectively. Summary of the Invention

[0006] One implementation relates to a method comprising: receiving a first dataset including a plurality of embeddings of a first type and a second type, the first type of embeddings being associated with a first node type and the second type of embeddings being associated with a second node type; generating a fixed adjacency matrix from the first dataset; applying a first random binary mask to the fixed adjacency matrix to obtain a first subgraph of the fixed adjacency matrix; processing the first subgraph of the fixed adjacency matrix through a first layer of a graph convolutional network to obtain a first embedding matrix; applying a second random binary mask to the fixed adjacency matrix to obtain a second subgraph of the fixed adjacency matrix; processing the first embedding matrix and the second subgraph of the fixed adjacency matrix through a second layer of the graph convolutional network to obtain a second embedding matrix; determining a plurality of gradients of a loss function; and using at least one of the plurality of gradients to modify the first random binary mask and the second random binary mask. The process involves: creating a mask; generating a second fixed adjacency matrix from a second dataset; applying a modified first random binary mask to the second fixed adjacency matrix to obtain a third subgraph of the second fixed adjacency matrix; processing the third subgraph of the second fixed adjacency matrix through a first layer of the graph convolutional network to obtain a third embedding matrix associated with the second dataset; applying a modified second random binary mask to the second fixed adjacency matrix to obtain a fourth subgraph of the second fixed adjacency matrix; processing the third embedding matrix and the fourth subgraph of the second fixed adjacency matrix associated with the second dataset through a second layer of the graph convolutional network to obtain a fourth embedding matrix associated with the second dataset; and generating a recommendation score between the first type of embedding and the second type of embedding from the second dataset using at least the third embedding matrix and the fourth embedding matrix associated with the second dataset.

[0007] Another embodiment relates to a server computer comprising: a processor; and a non-transitory computer-readable medium including code executable by the processor to perform: receiving a first dataset comprising a plurality of embeddings of a first type and a second type, the first type of embeddings being associated with a first node type and the second type of embeddings being associated with a second node type; generating a fixed adjacency matrix from the first dataset; applying a first random binary mask to the fixed adjacency matrix to obtain a first subgraph of the fixed adjacency matrix; processing the first subgraph of the fixed adjacency matrix through a first layer of a graph convolutional network to obtain a first embedding matrix; applying a second random binary mask to the fixed adjacency matrix to obtain a second subgraph of the fixed adjacency matrix; processing the first embedding matrix and the second subgraph of the fixed adjacency matrix through a second layer of the graph convolutional network to obtain a second embedding matrix; determining a plurality of gradients of a loss function; and using the plurality of gradients... The graph convolutional network modifies the first and second random binary masks using at least one gradient in the degree of the first random binary mask; generates a second fixed adjacency matrix from the second dataset; applies the modified first random binary mask to the second fixed adjacency matrix to obtain a third subgraph of the second fixed adjacency matrix; processes the third subgraph of the second fixed adjacency matrix through the first layer of the graph convolutional network to obtain a third embedding matrix associated with the second dataset; applies the modified second random binary mask to the second fixed adjacency matrix to obtain a fourth subgraph of the second fixed adjacency matrix; processes the third embedding matrix and the fourth subgraph of the second fixed adjacency matrix associated with the second dataset through the second layer of the graph convolutional network to obtain a fourth embedding matrix associated with the second dataset; and generates a recommendation score between the first type of embedding and the second type of embedding from the second dataset using at least the third and fourth embedding matrices associated with the second dataset. Attached Figure Description

[0008] A better understanding of the nature and advantages of embodiments of the present invention can be obtained by referring to the following detailed description and accompanying drawings. Attached Figure Description

[0010] Figure 1 A block diagram of a system for generating user-project relationship data is shown.

[0011] Figure 2 This shows noisy user-project data that generates misleading information for the target user.

[0012] Figure 3 A routine for imposing low-rank constraints on a structure graph convolutional network is shown.

[0013] Figure 4 A two-layer graph convolutional network with a random binary mask is shown.

[0014] Figure 5 A trained two-layer graph convolutional network with a random binary mask is shown for generating recommendation scores.

[0015] Figure 7 and Figure 7 (Continued) This paper demonstrates the robustness of structured graph convolutional networks compared to vanilla graph convolutional networks for various noise levels.

[0016] Figure 8 Training curves for training loss and test recall are shown for the MovieLens dataset with 25% noise.

[0017] Figure 9 Oversmoothing is demonstrated for both structured graph convolutional networks and ordinary graph convolutional networks.

[0018] Figure 10 and Figure 10 (Continued) The parameter sensitivity for the dataset is shown. Detailed Implementation

[0019] Before discussing the embodiments of this disclosure, some terms may be described in further detail.

[0020] A "random binary mask" can be a binary data item used in a mask matrix. In some implementations, a random binary mask can be a matrix whose elements consist of 0s or 1s. A random binary mask can be learned or randomly generated. In some implementations, a random binary mask can be applied to other matrices to effectively mask the values ​​of those matrices by setting them to 0.

[0021] "Embedding" can be a low-dimensional representation of data. The embedding process can be used to transform high-dimensional vectors into low-dimensional representations.

[0022] A "node" can be a point where a line or path interacts or branches. Nodes can have different types. For example, a node can have a first type, such as a "user node". A node can have a second type, which could be a "project node".

[0023] A "dataset" can be a collection of data. For example, a dataset can be data that includes connections between two types of nodes. Nodes can be users and items; in this case, the dataset includes interactions between users and items.

[0024] An "adjacency matrix" can be a matrix that includes connections between nodes in a dataset. For example, an adjacency matrix can indicate which nodes in a dataset are connected. In some implementations, the adjacency matrix includes connections between sets of users and sets of items, and the elements of the adjacency matrix can be connections.

[0025] A "server computer" is typically a powerful computer or cluster of computers. For example, a server computer can be a mainframe, a small cluster of computers, or a group of servers acting as a unit. In one example, a server computer could be a database server coupled to a web server.

[0026] "Processor" can include any suitable one or more data computing devices. A processor can include one or more microprocessors that work together to achieve a desired function. A processor can include a CPU, which includes at least one high-speed data processor sufficient to execute program components for performing requests generated by the user and / or the system. A CPU can be a microprocessor such as AMD's Athlon, Duron, and / or Opteron; IBM and / or Motorola's PowerPC; IBM and Sony's Cell processors; Intel's Celeron, Itanium, Pentium, Xeon, and / or XScale; and / or similar processors.

[0027] "Memory" can be any suitable one or more devices capable of storing electronic data. Suitable memory can include non-transitory computer-readable media whose storage can be executed by a processor to implement desired methods. Examples of memory can include one or more memory chips, disk drives, etc. Such memory can be operated using any suitable electrical, optical, and / or magnetic modes of operation.

[0028] I. Introduction

[0029] Personalized recommendation systems have been widely deployed in many online services to meet user interests and address the problem of information overload [Paul Covington, Jay Adams, and Emre Sargin, 2016, published in...]. RecSys , 191–198; Jizhe Wang et al., 2018, in KDD, 839–848; Rex Ying et al., 2018, in KDD[974–983]. One of the most prominent techniques is collaborative filtering, which takes into account users' historical interactions and assumes that users who have shared similar preferences in the past are likely to make similar decisions in the future. In particular, the decomposition machine has achieved great success by using the inner product of user embeddings and item embeddings as preference scores [Yehuda Koren, Robert Bell, and Chris Volinsky, 2009, 974–983]. Computer (2009), 30–37; Steffen Rendle, 2012, ACM Transactions on Intelligent Systems and Technology [(2012), 1–22]. Although they are effective, their recommendation performance is unsatisfactory due to the lack of strategies for learning high-order user-item feature interactions [Huifeng Guo et al., 2017, published in...]. IJCAI; Xiangnan He et al., 2017, published in WWW , 173–182; Jianxun Lian et al., 2018, published in KDD , 1754–1763; Dawen Liang et al., 2018, published in WWW , 689–698; Guorui Zhou et al., 2018, published in KDD [1059–1068]. Deep learning technology has therefore begun to dominate the prospects of recommender systems [Shuai Zhang et al., 2019, 1059–1068]. ACM Computing Surveys (CSUR) (2019), 1–38.

[0030] Recently, Graph Convolutional Networks (GCNs) have become increasingly powerful in representation learning for graph-structured data [Will Hamilton, Zhitao Ying, and Jure Leskovec, 2017, published in...]. NeurIPS , 1024–1034; Thomas N. Kipf and Max Welling, 2017, published in ICLR; Felix Wu et al., 2019, published in ICML [6861-6871]. Graph convolutional networks use a message-passing mechanism on the input graph, which can be summarized in three steps: 1) initializing node representations using initial properties or structural features of the node representations (such as the degree of the node); 2) updating the representation of each node by recursively aggregating and transforming the representations of each node's neighboring nodes; and 3) reading out the final representation of a single node or the entire graph required for downstream tasks. By treating user-item interactions as bipartite graphs, some have attempted to use graph convolutional networks for recommendation due to the theoretical elegance and performance of graph convolutional networks [Wenqi Fan et al., 2019, published in...]. WWW, 417–426; Xiangnan He et al., 2020, published in SIGIR , 639–648; Federico Monti, Michael Bronstein and Xavier Bresson, 2017, published in NeurIPS , 3697–3707; Xiang Wang et al., 2019, published in SIGIR , 165–174; Rex Ying et al., 2018, published in KDD [974–983]. For example, PinSage [Rex Ying et al., 2018, published in...] KDD [974–983] combines efficient random walks and graph convolutions to generate item embeddings. NGCF [Xiang Wang et al., 2019, published in...] SIGIR [165-174] proposed an embedded propagation layer to study higher-order connectivity in bipartite graphs. LightGCN [Xiangnan He et al., 2020, published in...] SIGIR [639–648] Recently, the design of graph convolutional networks has been simplified to make them more concise for recommendation.

[0031] Although encouraging performance has been achieved, graph convolutional networks are known to be susceptible to the quality of the input graph due to their recursive message-passing pattern [Hanjun Dai et al., 2018, published in...]. ICML , 1115–1124; Dingyuan Zhu et al., 2019, published in KDD [1399–1407]. Unfortunately, real-world user-item graphs are often noisy. This is especially true for implicit behaviors, as they are not necessarily consistent with user preferences [Yuta Saito, 2020, published in...]. WSDM [501–509]. If graph convolutional networks are not properly regularized, aggregating misleading neighborhood information can lead to suboptimal performance. See [reference]. Figure 1 and Figure 2 The following examples can be used to further explain the above issues.

[0032] Figure 1 A block diagram of a system 100 for generating user-project relationship data is shown. Figure 1The system includes a first embedded computer 102, a second embedded computer 104, a third embedded computer 106, and a server computer 108. The first embedded computer 102, the second embedded computer 104, the third embedded computer 106, and the server computer 108 can all operatively communicate with each other via any suitable communication channel or communication network. A suitable communication network can be any and / or a combination of the following: direct interconnection, the Internet, a local area network (LAN), a metropolitan area network (MAN), an Operational Mission as an Internet node (OMNI), a secure custom connection, a wide area network (WAN), a wireless network (e.g., employing protocols such as, but not limited to, Wireless Application Protocol (WAP), I-mode, etc.). Secure communication protocols, such as, but not limited to, File Transfer Protocol (FTP), Hypertext Transfer Protocol (HTTP), Secure Hypertext Transfer Protocol (HTTPS), Secure Sockets Layer (SSL), ISO (e.g., ISO 8583), etc., can be used to transmit messages between computers, networks, and devices. It should be understood that embodiments of the invention may include... Figure 1 Showing more or fewer computers.

[0033] In step S100, the first embedded computer 102 can provide a first dataset containing data associated with multiple users. For each user in the dataset, the data can include a list of items associated with that user. This list can include data associated with the items. For example, if the item is a movie, then the item's data can include the movie's title, genre, length, director, etc. The first embedded computer 102 can provide the server computer 108 with a list of movies watched by multiple users, and data associated with the movies in the list. The second embedded computer 104 and the third embedded computer 106 can provide similar data to the server computer 108. For example, the second embedded computer 104 can provide a second dataset associated with the same or different users. Similarly, the third embedded computer 106 can provide similar data to the server computer 108. For example, the third embedded computer 106 can provide a third dataset associated with the same or different users. Although Figure 1 Three embedded computers are shown, but in other embodiments there may be more or fewer embedded computers.

[0034] After receiving a dataset from one or more embedded computers, server computer 108 can analyze the provided data. For example, server computer 108 can use at least one of the received datasets to generate a user-item bipartite graph 110. User-item bipartite graph 110 can be a graphical representation of the dataset. User-item bipartite graph 110 can illustrate the user... and projects The connection between the representations. A line between nodes (e.g., a node could be a user or an item) can represent that two nodes are connected (e.g., a user). I have already watched the movie. In an example of a user's list of movies watched, the user... You may have already watched the movie. Some of them. Users And movies The lines between them can represent users. I have already watched the movie. At least some of them. However, the connection between a user and different items may have different strengths (e.g., a user may only watch a portion of a movie or the entire movie). The dataset may or may not provide information related to the strength of the connection between the user and the items.

[0035] Figure 2 This illustrates noisy user-item data that generates misleading information for the target user. The user-item bipartite graph 200 may contain noisy edges. For example, such as user... The nodes may have similarities to the project. Noise connections. For The corresponding message passing node for 202 may include noisy connections. When the user and projects Noisy connections may occur when the relationship between them is weak, or when they are unintentional in other cases.

[0036] The key idea behind graph convolutional networks is to fully discover higher-order relations in a bipartite graph. Therefore, even and Even without explicit connections, nodes can be aggregated. The representation of the path Update target node The characterization of noise connections (such as...). and Noise connections between nodes may lead to misleading information about the user. First-hop neighbor. Or the second hop neighbor It can also be connected via noise. Transmitted to the target node This degrades the performance of the analysis. As graph convolutional networks deepen, this misleading information continues to propagate and contaminate the entire graph. For example, noisy connections might recommend items from another, essentially unrelated user to a user.

[0037] Therefore, it is preferable to remove irrelevant neighbors during message passing. Otherwise, including less useful messages will complicate model training, increase the risk of overfitting, and even impair model effectiveness. The key challenge then becomes determining the criteria for ignoring irrelevant neighbors during the training phase. Fortunately, real-world graphs are typically... sparse and low-rank [Santo Fortunato, 2010] Physics reports 486, 3-5 (2010), 75–174]. Sparsity means that during message passing, only the most important neighbors should be locally connected to the target node. Low rank indicates that the entire graph is globally structured and that only a few factors influence user preferences. These two intrinsic graph properties are widely used in linear matrix completion models [Yehuda Koren, Robert Bell, and Chris Volinsky, 2009, Computer (2009), 30–37; Xia Ning and George Karypis, 2011, in ICDM, 497–506], for example, Norm regularization or matrix rank minimization are used, but have been rarely studied in graph neural models. One approach is to first create a clean [structure / structure] based on some similarity function. Nearest neighbor graph. This is a common strategy used in shaded graph models, such as LLE [Sam T Roweis and Lawrence K Saul, 2000, science 290, 5500 (2000), 2323-2326] and Isomap [Joshua B Tenenbaum, Vin De Silva and John C Langford, 2000, science [290, 5500 (2000), 2319-2323], and has recently been revisited in depth graph models [Zheng et al., 2020, published in ICML [11458–11468]. However, The expressive power of the nearest neighbor is affected The choice of similarity function in the embedding space and the constraints thereof.

[0038] To address the aforementioned issues, Structured Graph Convolutional Networks (SGCNs) can be used to enhance the performance of graph convolutional networks by leveraging the sparsity and low-rank graph structure properties. To achieve sparsity within the graph, random binary masks are appended to each layer of the graph convolutional network to prune noisy and meaningless edges, such as noisy connections, within the framework of the graph convolutional network. Intuitively, a random binary mask (i.e., Sampling and The sparse edges (discarded) can be viewed as graph generators to support high-quality sparse graphs for each layer of a graph convolutional network. The motivation for doing so is twofold: 1) it is possible to learn to discard noisy edges with parameterized masks in a data-driven manner, thus resulting in a less complex sparse messaging strategy with better generalization ability; 2) overfitting and oversmoothing are two of the main bottlenecks in developing deeper graph convolutional networks [YuRong et al., 2019, published in...]. ICLR These problems can be mitigated by sampling the subgraph using a random binary mask mechanism. However, directly training a random binary mask is challenging due to the combinatorial nature of discrete samples. To make the samples differentiable, the discrete space needs to be further reformulated into a continuous space via probability reparameterization in the optimization problem [Eric Jang, Shixiang Gu, and Ben Poole, 2017, published in...]. ICLR ;Mingzhang Yin and Mingyuan Zhou, 2019, published in ICLR To better backpropagate the gradients of binary variables, an unbiased gradient estimator is also proposed. Inspired by adversarial machine learning [Negin Entezari et al., 2020, published in...] WSDM , 169–177; Wei Jin et al., 2020, published in KDD [66–74], low-rank constraints are also imposed on the sparse adjacency matrix for each layer of the graph convolutional network. This regularization forces the graph to be globally structured, which has been shown to defend against adversarial attacks [Negin Entezari et al., 2020, published in...]. WSDM , 169–177; Wei Jin et al., 2020, published in KDD [66–74] It has been very successful in terms of noise in defense recommendations.

[0039] Extensive experiments were conducted to evaluate the effectiveness and robustness of the proposed structured graph convolutional network (SBR) method. The embodiments of this invention offer several advantages over conventional systems. The SBR according to the embodiments of this invention employs a method that explicitly prunes irrelevant neighbors during the message-passing phase of the graph convolutional network, which significantly reduces the negative impact of noise on the recommendation system. The purpose of developing a random binary mask is to select sparse and high-quality subgraphs for each layer of the graph convolutional network. Low-rank constraints are also imposed to enhance the robustness and generalization of the graph convolutional network. Furthermore, an unbiased gradient estimator for random binary optimization is created by transforming the unbiased gradient estimator into an equivalent unbiased gradient estimator in continuous space. Thus, the parameters of the random binary mask and the parameters of the graph convolutional network can be learned jointly. Extensive experiments were conducted on four public datasets. The results demonstrate the benefits of the structured graph convolutional network in terms of the effectiveness of pruning noisy edges and the use of low-rank constraints, resulting in performance gains of 4.92%–26.23% compared to conventional methods.

[0040] II. Related Works

[0041] This chapter reviews relevant works on recommender systems and graph convolutional networks. It describes the differences between existing achievements and structured graph convolutional networks.

[0042] A. Collaborative Filtering

[0043] Recommender systems often employ collaborative filtering (CF) to learn complex feature interactions between users and items based on users' historical profiles [Yifan Hu, Yehuda Koren, and Chris Volinsky, 2008, published in...]. ICDM , 263–272; Steffen Rendle, in UAI [452-461]. Matrix factorization is an early approach that learns the latent embeddings of users and items from the user-item rating matrix and uses the inner product to predict user preferences [Yehuda Koren, Robert Bell, and Chris Volinsky, 2009, 452-461]. Computer (2009), 30–37; Steffen Rendle, 2012, ACM Transactions on Intelligent Systems and Technology [(2012), 1–22]. Inspired by the expressive power of deep neural networks, modern recommender systems have been further improved using deep learning techniques to leverage more complex and non-linear feature interactions between users and items [Shuai Zhang et al., 2019, 1–22]. ACM Computing Surveys (CSUR) (2019), 1–38]. Some representative models include: Wide & Deep [Heng-Tze Cheng et al., 2016, published in DLRS [7–10], NCF [Xiangnan He et al., 2017, published in WWW [173–182], LRML [Yi Tay, Luu Anh Tuan and Siu CheungHui, 2018, published in WWW [–739], DeepFM [Huifeng Guo et al., 2017, published in IJCAI xDeepFM [Jianxun Lian et al., 2018, published in...] KDD [1754–1763], CDAE [Dawen Liang et al., 2018, published in WWW [689–698] etc. However, these CF-based models are often designed as first-order approximations (e.g., direct connections between users and items). By forming user-item interactions as a bipartite graph, graph-based models are able to explore implicit higher-order proximity between nodes, which helps to discover deeper connections between users and items in personalized recommendation systems [Ming Gao et al., 2018, published in...]. SIGIR , 715–724; Jheng-Hong Yang et al., 2018, published in RecSys , 140–144; Lei Zheng et al., 2018, published in RecSys ,311–319).

[0044] B. Graph Convolutional Networks

[0045] Graph Convolutional Networks (GCNs) are a special instance of convolutional neural networks used for structured data. They have received considerable attention due to their performance in graph embeddings [Will Hamilton, Zhitao Ying, and Jure Leskovec, 2017, published in...]. NeurIPS , 1024–1034; Thomas N. Kipf and Max Welling, 2017, published in ICLR; Felix Wu et al., 2019, published in ICML [6861-6871]. Some researchers have already deployed graph convolutional networks in network-scale recommender systems [Rianne van den Berg, Thomas N Kipf, and Max Welling, 2017, 6861-6871]. Published on KDD Workshop on Deep Learning Day; Wenqi Fan et al., 2019, published in WWW , 417–426; Xiangnan He et al., 2020, published in SIGIR, 639–648; Federico Monti, Michael Bronstein and Xavier Bresson, 2017, published in NeurIPS , 3697–3707; Hongwei Wang et al., 2019, published in WWW , 3307–3313; Xiang Wang et al., 2019, published in SIGIR , 165–174; Rex Ying et al., 2018, published in KDD , 974–983; Lei Zheng et al., 2018, published in RecSys [311–319]. For example, GC-MC [Rianne van den Berg, Thomas N Kipf and MaxWelling, 2017, ] Published in KDD Workshop on Deep Learning Day [and RMGCNN [Federico Monti, Michael Bronstein and Xavier Bresson, 2017, published in...] NeurIPS [3697–3707] expresses the recommender system as matrix completion and designs a graph convolutional network on the user-item bipartite graph. SpectralCF [Lei Zheng et al., 2018, published in...] RecSys [311–319] developed spectral convolution to identify all possible connectivity between users and items in the spectral domain. PinSage [Rex Ying et al., 2018, published in...] KDD [974–983] combines efficient random walks and graph convolutions to generate item embeddings in Pinterest. GraphRec [Wenqi Fan et al., 2019, published in...] WWW [417–426] proposed a heterogeneous graph convolutional network for social recommendation. NGCF [Xiang Wang et al., 2019, published in...] SIGIR [165-174] proposed embedding a propagation layer to harvest higher-order cooperative signals in bipartite graphs. LightGCN [Xiangnan He et al., 2020, published in...] SIGIR [639–648] simplified the design of graph convolutional networks to make them more concise for recommendation purposes.

[0046] While the above methods have proven effective in generating user and item embeddings, it is known that graph convolutional networks are sensitive to the quality of the input graph due to their recursive message-passing patterns [Hanjun Dai et al., 2018, published in...]. ICML , 1115–1124; Dingyuan Zhu et al., 2019, published in KDD[1399–1407]. In other words, slight perturbations to the user-item bipartite graph can mislead graph convolutional networks into producing incorrect predictions.

[0047] C. Overfitting and oversmoothing

[0048] Two major obstacles encountered in developing deeper graph convolutional networks are overfitting and oversmoothing [Guohao Li et al., 2019, published in...]. CVPR , 9267–9276; Qimai Li, Zhichao Han, and Xiao-Ming Wu, 2018, in AAAI; Dongsheng Luo et al., 2021, published in WSDM Given only limited training data, overfitting stems from overparameterized graph convolutional networks used to fit the distribution. As the number of convolutional layers increases, oversmoothing causes the features of graph nodes to gradually converge to the same value [Qimai Li, Zhichao Han, and Xiao-Ming Wu, 2018, published in...]. AAAI Both of the above problems can be mitigated by using the dropout trick in graph convolutional networks. For example, ordinary dropout [Nitish Srivastava et al., 2014, The journal of machine learning research [(2014), 1929–1958] Randomly masking elements in the weight matrix can reduce the effects of overfitting. However, dropout cannot prevent oversmoothing because it does not make any changes to the graph adjacency matrix. DropNode [Will Hamilton, Zhitao Ying, and Jure Leskovec, 2017, published in...] NeurIPS [1024–1034] is a node-oriented method that randomly selects nodes for mini-batch training. DropEdge [Yu Rong et al., 2019, published in...] ICLR [This is an edge-oriented method that randomly removes a certain number of edges from the input graph, acting as a data augmenter. Message discarding [Xiang Wang et al., 2019, published in...] SIGIR [165–174] Randomly discard outgoing messages in each propagation layer to refine the representation. DropoutNet [Maksims Volkovs, Guangwei Yu and Tomi Poutanen, 2017, published in NeurIPS [4957–4966] Input dropping during training is used to address the cold start problem in recommender systems. However, these dropping techniques often randomly remove a subset of nodes, edges, or features, which can lead to suboptimal performance.

[0049] The mechanism of random binary masking differs slightly from the aforementioned discarding methods, but is more relevant to recent developments in graph sparsity [Luca Franceschi et al., 2019, published in...]. ICML , 1972–1982; Cheng Zheng et al., 2020, published in ICML [11458–11468]. Alternative optimization algorithms to random sampling were used to determine which edges would be removed in a data-driven manner. Therefore, sparse graphs that best preserve desired properties (e.g., sparsity and low rank) can benefit graph convolutional networks in terms of better robustness and superior generalization.

[0050] III. Further Details

[0051] This chapter introduces notation and defines the problem setting for recommendation. Then, it describes in detail the proposed structured graph convolutional network model. The structured graph convolutional network mainly consists of three components: a well-designed graph convolutional network, a random binary mask, and rank approximation. Finally, the loss function for model optimization is introduced.

[0052] A. Problem Formulation

[0053] This disclosure focuses on learning user preferences from implicit feedback. Specifically, behavioral data (e.g., in the case of a project marketplace, behavioral data could be clicks on projects, reviews of projects, purchases of projects, etc.) involves users. and projects A set such that the set Indicates user Projects that have already been interacted with before, and This indicates unobserved items. Unobserved interactions are not necessarily negative. Rather, users may simply be unaware of them.

[0054] When the view forms a bipartite graph During user-project interactions, an implicit feedback matrix can be constructed. ,in and These represent the total number of users and projects, respectively. If users... Already with the project Interactive, then each item ,otherwise Its corresponding adjacency matrix for a bipartite graph. It can be obtained as: (1) The adjacency matrix This can then be used as the input graph for graph convolutional networks. The purpose is to... Recommended users A sorted list of items of interest, in the same sense, is a bipartite graph. Perform link prediction on the above.

[0055] B. GCNs for Recommendation

[0056] 1. Overfitting and oversmoothing

[0057] Following the mainstream graph convolutional recommendation system [Xiangnan He et al., 2020, published in...] SIGIR , 639–648; Xiangnan He et al., 2017, published in WWW , 173–182; Xiang Wang et al., 2019, published in SIGIR [165–174] describes the user and projects Representation via embedded lookup table: (2) in and These represent the IDs for "user" and "project" respectively; and They are users and projects The embedding, and This refers to the embedding size. These embeddings are expected to remember the initial characteristics of the item and the user. Next, two state-of-the-art GCN-based recommendation models are introduced.

[0058] 2.NGCF

[0059] In standard graph convolutional networks [Thomas N. Kipf and Max Welling, 2017, published in...] ICLR Following this, NGCF [Xiang Wang et al., 2019, published in...] SIGIR [165–174] The user-item bipartite graph is used to perform embedding propagation and feature transformation, as follows: (3) The initialization is performed in equation (2). and In this case, and These represent the first and second digits of the graph convolutional network, respectively. Users in the layer and projects Detailed characterization; It is a non-linear activation function, and Represents element-wise product; and It is a trainable weight matrix; Indicated by user A collection of projects that interact directly, and Indicates connection to the project The model is able to explore higher-order collaboration signals between users and items by adding more convolutional layers.

[0060] 3. LightGCN

[0061] Several studies have shown that simpler, sometimes linear, graph convolutional networks are very effective for representation learning [Felix Wu et al., 2019, published in...]. ICML [6861-6871]. LightGCN [Xiangnan He et al., 2020, published in...] SIGIR [639–648] aims to simplify the design of NGCF to make it more concise for use in recommendations.

[0062] Compared to NGCF, LightGCN uses a weighted sum aggregator instead of feature transformations and nonlinear activations. Therefore, the propagation seen in equation (3) can be simplified to: (4) The above equation can be rewritten in compact matrix form. Let the first... The layer embedding matrix is It collects all embeddings of users and items from equation (2). Then, the matrix equivalent form of equation (4) is: (5) in It is the adjacency matrix of the use-item graph as shown in equation (1); It is the corresponding angle matrix, where each entry Representation matrix The The number of non-zero numbers in the row.

[0063] 4. Model optimization for NGCF and LightGCN

[0064] Through dissemination Layers, graph convolutional networks obtain Embedded to represent users and projects :

[0065] NGCF is implemented through cascading. LightGCN, on the other hand, uses a weighted sum. The inner product between the two embeddings can be used to predict preference scores:

[0066] Both methods can employ Bayesian Personalized Ranking (BPR) loss [Steffen Rendle, published in...]. UAI [452-461] to optimize model parameters, i.e. minimize: (6) in Represents paired training data; It is the sigmoid function; Represents model parameters, and control Norms are used to prevent overfitting.

[0067] 5. Restrictions

[0068] Despite the success of NGCF and LightGCN, they are insufficient to solve noise in bipartite graphs. For example, LightGCN relies entirely on the adjacency matrix. To refine the representation of users and items in equation (5). However, the adjacency matrix It can contain the noisy edges discussed in Chapter I. These misleading messages continue to propagate as LightGCN goes through more layers. The situation worsens when the noise signal contains low-frequency components. Therefore, these graph convolutional networks have a high risk of overfitting to noise [Hoang NT and Takanori Maehara, 2019, arXiv preprint arXiv:1905.09550 (2019)).

[0069] Vanilla Dropout [Nitish Srivastava et al., 2014, The Journal of Machine learning research [(2014), 1929–1958] Randomly mask the elements of the weight matrix (e.g., in equation (3)). and This may have limited noise prevention capabilities because it does not address the adjacency matrix. Make no changes. NGCF attempts to mitigate this problem by randomly removing a subset of messages or nodes. However, this weakens the interpretability and understanding of which edges should be retained or removed during training (see Section IV.C for details). To address this challenge, as an alternative to random sampling, a simple yet effective data-driven principle is proposed: masking edges using random binary masks.

[0070] C. Random binary mask

[0071] 1. Graph sparsification

[0072] To filter out noise, each layer of the graph convolutional network is appended with a random binary mask to remove meaningless edges, while the network's parameters are trained simultaneously. The overall network architecture is as follows: Figure 3 As shown. Formally, for each convolutional layer in equation (5), a binary matrix is ​​introduced. ,in Represents a node and nodes Does the edge between them include the first? In the layer. Replacing equation (5) fixed Adjacency matrix, for the th The input graph adjacency matrix of the layer becomes: (7) in This represents element-wise product. Intuitively, a random binary mask. (Right now Sampling and The sparse graphs (those that are discarded) can be viewed as graph generators to support high-quality sparse graphs for each layer of a graph convolutional network. These sparse graphs achieve neighbor aggregation rather than full aggregation of subsets during training, thus avoiding overly smooth graph convolutional networks when there are many layers. The idea of ​​this graph sparsification is known, and its initial goal was to remove unnecessary edges for graph compression while preserving almost all the information of the input graph [David Eppstein et al., 1997, ]. Journal of The ACM (JACM) (1997), 669–696]. This point has recently been revisited in depth graph models [Luca Franceschi et al., 2019, published in...]. ICML , 1972–1982; Cheng Zheng et al., 2020, published in ICML ,11458–11468).

[0073] To promote the adjacency matrix The sparsity of the regularization term is explicitly penalized by minimizing the following equation. The number of non-zero entries: (8) in express Norm, which can drive meaningless edges as accurate zero. It is an indicator function, if the condition If true, then the indicator function is equal to Otherwise equals However, due to binary masks of The nondifferentiability, discreteness, and combinatoriality of possible states make optimization under this penalty computationally difficult. To address this challenge, these discrete variables need to be reparameterized as deterministic transformations of the underlying continuous variables, and then dual sampling is applied to generate low-variance and unbiased gradients. Next, an efficient algorithm is introduced to improve backpropagation of the reparameterized gradients through a stochastic binary layer.

[0074] 2. Reparameterization and gradient

[0075] binary mask Joint optimization with the original graph convolutional network (e.g., NGCF or LightGCN), and then combining equations (6) and (8) into a unified objective: (9) in The sparsity of the control plot. Therefore, equation (9) involves stochastic gradient estimation, which requires... Marginalization of binary sequences. For this purpose, each... Experiencing parameters The Bernoulli distribution makes Equation (9) can be restated as follows: (10) in It is the expected value, and the target in equation (10) It is for the target in equation (9) In parameters Variational upper limit (This can be derived using Jensen's inequality). Now, the second term in equation (1) is relative to the new parameter. It is differentiable. However, because The discrete nature of the first term still has problems.

[0076] To efficiently compute the gradient, [Eric Jang, Shixiang Gu and Ben Poole, 2017, published in...] was used. ICLR The reparameterization technique shown in the image will... Reparameterization Deterministic functions .

[0077]

[0078] Because of deterministic functions It should be limited to Therefore, the standard sigmoid function is a good candidate function (e.g., Furthermore, augmentation-reinforcement-merge (ARM), a recently proposed unbiased gradient estimator, is suitable for solving stochastic binary optimization problems [Mingzhang Yin and Mingyuan Zhou, 2019, published in...]. ICLR ].

[0079] The following key theorem can be introduced: For Binary random variables vectors and any function ,

[0080] Compared to The gradient of (the fraction of the Bernoulli probability parameter) can be expressed as:

[0081] in ,and It is the sigmoid function.

[0082] Due to the linearity of the expected value, ARM can directly optimize the Bernoulli variables without introducing any bias, resulting in a highly competitive estimator. Furthermore, the expected value can be estimated using only a pair of dual-coupled samples, allowing for efficient gradient computation.

[0083] According to the above theorem, let BPR loss function: And reparameterize (here we point out: for the K layers of the GCN, Z:={Z (1) ,…,Z (K)}, Π:={Π (1) ,…,Π (K)}, and Φ:={Φ (1) ,…,Φ (K)}): Now we can calculate equation (10) in the following matrix form. Compared to gradient: (11) in By if in the forward pass of a graph convolutional network Then the binary mask Set to Otherwise set to The resulting BPR loss. Then, the same strategy can be applied... .

[0084] Therefore, gradients can be effectively backpropagated using random binary masks for the following three reasons: 1) Sampling from the Bernoulli distribution is performed from... and The sampling performed by the uniform distribution between them is replaced; 2); the first term of equation (11) only involves the forward pass of the graph convolutional network to calculate the gradient; 3) the second term Differentiable and computationally easy. These properties allow the computation of gradients, which typically originate from discrete spaces, to be performed in continuous spaces.

[0085] During the inference phase, The expectation can be used as a mask in equation (7), i.e. However, this will not be obtained from the adjacency matrix. This produces a sparse graph because the sigmoid function in the ARM theorem is smooth, and no mask element is exactly zero (unless a hard sigmoid function is used). Here, the values ​​are simply... Setting the amplitude to zero guarantees a sparse graph and eliminates any corresponding noisy edges.

[0086] It is worth mentioning that several studies have been proposed to estimate the gradients for discrete variables in equations such as equation (10). Some of these studies include REINFORCE [Ronald J Williams, 1992, Machine learning [8, 3-4 (1992), 229–256], Gumbel-Softmax [Eric Jang, Shixiang Gu and BenPoole, 2017, published in ICLR [], Through-pass estimator [Yoshua Bengio, Nicholas Léonard and Aaron Courville, 2013, arXiv preprint arXiv:1308.3432 (2013), and a rigid concrete estimator [Christos Louizos, Max Welling and Diederik P Kingma, 2019, published in ICLR However, these methods either suffer from biased gradients or high variance, while the ARM estimator is unbiased, exhibits low variance, and has low computational complexity, as shown in [Mingzhang Yin and Mingyuan Zhou, 2019, published in...]. ICLR As shown in the image.

[0087] The ARM estimator for equation (11) is straightforward, but requires two forward passes of the graph convolutional network to compute the BPR loss. [In [Mingzhang Yin and Mingyuan Zhou, 2019, published in...] ICLR In [the paper], the authors also introduced a variant, Augment-Reinforce (AR), to overcome the problem of double forward propagation; however, this resulted in higher variance. Fortunately, unlike convolutional neural networks (CNNs), graph convolutional networks typically have very few layers (e.g., the number of layers in NGCF and LightGCN). Furthermore, the complexity of the double forward pass is acceptable. Therefore, standard ARM is used in the experiments. Other advanced techniques, such as DisARM [Zhe Dong, Andriy Mnih, and George Tucker, 2020, published in...], can be combined to further improve the training of random binary masks. NeurIPS ].

[0088] D. Low-rank approximation

[0089] Besides implementing sparse graphs via binary masks, graph convolutional networks themselves also suffer from vulnerabilities in resisting small perturbations [Hanjun Dai et al., 2018, published in...]. ICML [1115–1124]. A change to one node (e.g., a user or an item) can affect other nodes in the same local community. Several studies have shown that graphs with low-rank constraints are more robust to perturbations [Negin Entezari et al., 2020, published in...]. WSDM , 169–177; Wei Jin et al., 2020, published in KDD [66–74]. By minimizing the following terms, the adjacency matrix is... , Apply low-rank constraints: (12) in Let represent the nuclear norm of a convex proxy for rank minimization. Representing the adjacency matrix The The kernel norm is the largest singular value. Singular value decomposition (SVD) is typically needed to optimize the kernel norm [Negin Entezari et al., 2020, published in...]. WSDM , 169–177.

[0090] Singular value decomposition can be readily implemented, but it is typically numerically unstable during backpropagation [CatalinIonescu, Orestis Vantzos, and Cristian Sminchisescu, 2015, published in...]. CVPR , 2965–2973; WeiWang et al., 2019, published in NeurIPS[3162–3170]. This is because the partial derivatives of the nuclear norm depend on a matrix with the following elements. [Catalin Ionescu, Orestis Vantzos, and Cristian Sminchisescu, 2015, in CVPR [2965–2973]:

[0091] When two singular values ​​are close, the partial derivatives become very large, causing arithmetic overflow. This is especially true for large matrices, where the probability of two singular values ​​being nearly equal is much higher than in smaller matrices. Power iteration is one approach to address this problem. Power iteration relies on an iterative process to approximate the principal eigenvalues ​​and eigenvectors. However, power iteration is sensitive to how the singular vectors are initialized at the beginning of each contraction step [Wei Wang et al., 2019, published in...]. NeurIPS ,3162–3170).

[0092] To address these issues, a user-friendly algorithm combining singular value decomposition and power iteration was explored [WeiWang et al., 2019, published in...]. NeurIPS [3162–3170]. For the nuclear norm, the previous Each singular value provides more information, and therefore the truncated singular value decomposition can be used to approximate equation (12) as For example, as cited by Wei Wang et al., 2019, published in […]. NeurIPS As suggested in [3162–3170], the hybrid strategy is as follows: 1) In the forward pass, a truncated singular value decomposition is used to compute for each adjacency matrix. and based on 1) Calculate the nuclear norm; 2) In backpropagation, calculate the gradient based on the power-law iterative derivative, but use the vector calculated using singular value decomposition. Used for initialization purposes. Figure 3 This shows the entire computational graph.

[0093] Figure 3 Routines for imposing low-rank constraints on structured graph convolutional networks are shown. These routines seek to minimize the difference between the input adjacency matrix and the ideal adjacency matrix determined by equation (12). This routine can be derived from the adjacency matrix A. (k) Starting with 300, it can be the adjacency matrix of the k-th layer of a structured graph convolutional network. As output, it can form a matrix such as A. (k) The new adjacency matrix. (See reference) Figures 4 to 5 , A (k) This can correspond to the first masking bipartite graph 404, while A (k)’This can correspond to the third masked bipartite graph 504. The above algorithm, combining singular value decomposition and power iteration, can be used for... Figure 3 In the method.

[0094] In step S300, the singular value decomposition algorithm 302 can be applied to the adjacency matrix A. (k) 300. Singular Value Decomposition Algorithm 302 can decompose the adjacency matrix A (k) 300 factorization to receive the corresponding adjacency matrix A (k) A list of singular values ​​and singular vectors for 300. In some implementations, the singular values ​​may then correspond to the strength of the connections between nodes in a bipartite graph. For example, two directly connected nodes may then have higher values ​​than two nodes indirectly connected through one or more other nodes. Singular vectors may be the vector components of an adjacency matrix decomposition.

[0095] In step S302, the first n singular values ​​304 can be truncated, leaving only the largest singular value. The first n singular values ​​304 can be the values ​​that have the greatest influence in the singular value decomposition. A list.

[0096] In step S304, the first n singular values ​​304 can be used to approximate equation (12) as a loss 306. This completes the first routine. The approximate loss can be used to apply the exponential interaction algorithm (i.e., Figure 3 Data initialization of PI 310 in the middle.

[0097] In step S306, the first n singular vectors 308 can be truncated, leaving only the largest singular vector. The first n singular vectors 308 can be the vectors that have the greatest influence in the singular value decomposition algorithm 302. A list.

[0098] In step S308, the power iteration algorithm 310 can be used to generate approximate first n singular values ​​312. The approximate first n singular values ​​312 are approximations of the first n singular values ​​304 and are used for numerical stability.

[0099] In step S310, approximating the first n singular vectors 312 can be used to approximate equation (12) as an approximate loss 314. This completes the second routine.

[0100] In step S320, after calculating the approximate loss 314, the backpropagation approximate loss 314 can be calculated. Compared to The gradient is used to update the approximate first n singular values ​​312 generated by the power iteration algorithm 310.

[0101] In step S322, the approximate first n singular values ​​312 can be calculated. Compared to The gradient. Remember. Furthermore, Z(k) is a trainable random binary mask whose gradient can be minimized by changing the values ​​of the random binary mask (e.g., changing some values ​​of the random binary mask to 0 or 1).

[0102] In summary, singular value decomposition does not involve backpropagation (e.g., in...). Figure 3 In the next steps S306-S322 (referred to as the second routine), singular value decomposition only involves forward propagation to calculate the nuclear norm loss for equation (12). And initialize the state of the exponentiation iteration. In contrast, the exponentiation iteration does not involve forward propagation (e.g., in...). Figure 3 In the upper steps S302-S304 (referred to as the first routine), it is only used to compute gradients during backpropagation (e.g., the steps of the second routine). For imposing low-rank constraints in a graph convolutional network, the resulting computation graph is numerically stable and differentiable.

[0103] E. Joint Training

[0104] 1. Mixed loss

[0105] Therefore, a graph structure and graph convolutional network model can be jointly learned for the recommendation task. By combining the losses in equations (6), (8), and (12), the overall objective function of the structured graph convolutional network is given as: (13) in and These are the hyperparameters that control sparsity and low-rank constraints, respectively. Hyperparameters can be regularization coefficients. Algorithm 1 summarizes the overall training of the structured graph convolutional network.

[0106]

[0107] 2. Model Complexity

[0108] The complexity of structured graph convolutional networks (SCRs) comes from three components: the basic graph convolutional network (NGCF or LightGCN), a random binary mask, and low-rank constraints. The basic graph convolutional network has the same complexity as NGCF or LightGCN, which is represented as... The complexity of the random binary mask is derived from the ARM in equation (11), which requires two forward passes of the graph convolutional network. As mentioned earlier, the number of layers in a graph convolutional network is typically small. Therefore, the complexity of the ARM is approximately O(n). It is much cheaper than standard gradient backpropagation [Mingzhang Yin and Mingyuan Zhou, 2019, published in...]. ICLR Furthermore, the main complexity of low-rank constraints lies in the computation of singular value decomposition. For - Singular value decomposition has yielded some breakthroughs, such as the block Krylov method [Cameron Musco and Christopher Musco, 2015, NeurIPS [(2015), 1396–1404] or Lazy singular value decomposition [Zeyuan Allen-Zhu and Yuanzhi Li, 2016, Published in NeurIPS [974–982]. Due to the adjacency matrix It is naturally sparse, so it only needs Before calculation There are 10 singular values ​​and their corresponding singular vectors. Although structured graph convolutional networks combine sparse and low-rank information, their computational complexity remains the same as that of state-of-the-art graph convolutional networks used for recommendation.

[0109] Figure 4 A two-layer graph convolutional network with random binary masks is illustrated. The input to the graph convolutional network can be a first dataset comprising embeddings of two node types, such as first-type nodes and second-type nodes (e.g., a first-type node could be a user, and a second-type node could be an item such as a product or service, and the first dataset can include interactions between users and items). More specifically, the first dataset can include multiple first-type embeddings and multiple second-type embeddings. For example, the first dataset could include first-type nodes such as "movie" and second-type nodes such as "Star Wars". ® The embeddings for the movie "Joe Smith" can have associated data items such as science fiction, action, production year, and actors. The first dataset can also include a second type of node such as "user," and the embeddings associated with users such as "Joe Smith" can include data items associated with age, gender, income, and residence. The first dataset can be transformed into a user-item bipartite graph 400, which contains nodes corresponding to user and item sets, as well as edges representing the interactions between them. The bipartite graph can have a basic adjacency matrix, which can be considered equivalent. The user-item bipartite graph 400 can therefore have an equivalent fixed adjacency matrix representation. Thus, a fixed adjacency matrix can be generated from the first dataset. Figure 4 The graph convolutional network shown only has two layers; however, this method can be applied to n-layer graph convolutional networks. Any suitable type of graph convolutional network, such as NGCF or LightGCN, can be used as... Figure 4Graph convolutional networks. Graph convolutional networks and random binary masks can be derived from, for example... Figure 1 The server computer is trained on 108 processors. The server computer may have the capability to generate graph convolutional networks and random binary masks.

[0110] In step S400, a first random binary mask 402 may be applied to the user-item bipartite graph 400, or its equivalent fixed adjacency matrix. Initially, the first random binary mask 402 may be generated randomly, or by minimizing equation (8) or its derivative (e.g., a reparameterized form of equation (8)). By applying the first random binary mask 402 to the user-item bipartite graph 400, or its equivalent fixed adjacency matrix, a first masked bipartite graph 404 may be generated. The first masked bipartite graph 404 may be an example of a first subgraph of the fixed adjacency matrix associated with the user-item bipartite graph 400. The first masked bipartite graph 404 includes the same set of users and items as the input user-item bipartite graph 400, and may be a subgraph of the user-item bipartite graph 400. However, some edges among the edges between users and items are masked according to the first random binary mask 402. For example, the input user-item bipartite graph 400 may be the first random binary mask 402 of equation (7). And the first random binary mask 402 can be of the same equation. Therefore, the output of the first masking bipartite graph 404 is... .

[0111] In step S402, the first masking bipartite graph 404 can be processed by the first layer of a graph convolutional network or the first GCN layer 406. The first GCN layer 406 can process the first masking bipartite graph 404 to generate a first embedding matrix. Using the first masking bipartite graph 404 as A, the first embedding matrix can be generated by using equation (5). The first embedding matrix contains the embedding state of the user and item sets in the first masking bipartite graph 404.

[0112] In step S404, after the second GCN layer 412 is ready for processing, the first embedding matrix can be sent to the second layer of the graph convolutional network, the second GCN layer 412.

[0113] In step S406, a second random binary mask 408 can be applied to the user-item bipartite graph 400. The second random binary mask 408 can be generated similarly to the first random binary mask by minimizing equation (8). The result can be a second masked bipartite graph 410, which can be formally similar to the first masked bipartite graph 404. That is, the masked graph contains the same set of users and items. However, due to the difference between the first random binary mask 402 and the second random binary mask 408, the edges between the users and items may differ. The second masked bipartite graph 410 can be an example of a second subgraph of a fixed adjacency matrix associated with the user-item bipartite graph 400.

[0114] In step S408, the second GCN layer 412 can process the first embedding matrix (from step S404) and the second subgraph (e.g., the second masking bipartite graph 410) with a fixed adjacency matrix to obtain the second embedding matrix. Using the second masking bipartite graph 410 as A, the second embedding matrix can be generated similarly to the first embedding matrix via equation (5).

[0115] In step S410, the loss 414 of the structured graph convolutional network can be calculated. The loss 414 can be calculated via equation (13), which is reproduced as follows.

[0116] (14)

[0117] in and These are the hyperparameters controlling sparsity and low-rank constraints, respectively. Loss 414 comprises three terms from equations (6), (8), and (12). In the case of a graph convolutional network with more layers, loss 414 is calculated after all layers of the graph convolutional network have been applied, along with the associated random binary mask. The loss function (e.g., ...) can be determined in steps S412 and S414 (described below). Multiple gradients of the loss 414 can then be used to modify random binary masks or layers in graph convolutional networks.

[0118] In step S412, it can be relative to (Model parameters) are used to differentiate the loss 414 to obtain... The resulting gradient This gradient can be backpropagated to the second GCN layer 412. This gradient may be associated with the graph convolutional network. The parameters of the graph convolutional network can then be modified. To minimize the loss of 414.

[0119] In step S414, it can be relative to Differentiating the loss 414 yields the following result. The resulting gradient can be associated with either the first or second random binary mask. It can be backpropagated to the second random binary mask 408, and optionally backpropagated to the first random binary mask 402. Figure 4 Then, the parameters of the second random binary mask 408 can be modified to minimize the loss. For example, the binary value of the second random binary mask 408 can be updated to remove or re-add the edge between specific users and items.

[0120] Furthermore, in utilizing gradients After modifying the parameters of both the graph convolutional network and the random binary mask, the regularization coefficient can be adjusted. and (See Section III.E.1) to control the shape of the subgraph (e.g., the shape of the masked bipartite graph). This can be at least one gradient in the gradient determined according to loss 414. This process can be repeated any number of times using any number of masks and GCN layers. The resulting structured graph convolutional network will include, for example, at least random binary masks 402, 408, and also include at least GCN layers 406, 412.

[0121] The resulting structured graph convolutional network is trained and can be used to generate recommendations. A second dataset, which may be the same as or different from the first dataset, can be fed into the trained structured graph convolutional network. The result of applying the trained structured graph convolutional network can be recommendation scores between embeddings in the second dataset. For example, the second dataset can be another set of embeddings of users and items, and the resulting recommendation scores can be between individual user-item pairs in the second dataset.

[0122] Figure 5 A trained two-layer graph convolutional network with a random binary mask is shown for generating recommendation scores. After computation and minimization of the loss, it can be modified... Figure 4 The parameters of the graph convolutional network shown are used to generate the trained two-layer graph convolutional network. A second dataset can be used as input to the trained two-layer graph convolutional network. In some implementations, the second dataset may be... Figure 4 The second dataset can be a subset of the first dataset, or it can be new. The second dataset can be related to... Figure 5 The user-project two-part diagram 500 is associated with this. It can be seen from... Figure 5 The second fixed adjacency matrix is ​​generated by associating the second dataset with the user-project bipartite graph 500.

[0123] In some implementations, the second dataset can be transformed into a user-item bipartite graph 500, or an equivalent adjacency matrix. If corresponding to... Figure 4If the adjacency matrix of the user-item bipartite graph 400 is the first fixed adjacency matrix, then the adjacency matrix corresponding to the user-item bipartite graph 500 can be the second fixed adjacency matrix.

[0124] Steps S500 to S508 can be combined with Figure 4 Steps S400-S408 are executed in a similar manner. Figure 5 In the process, the user-project bipartite graph 500 can be used as input, and the third random binary mask 502 (which can be...) Figure 4 The modified first random binary mask 402 can be applied to the user-item bipartite graph to generate a third masked bipartite graph 504 (which may be an example of a third subgraph of the second fixed adjacency matrix). Similarly, a fourth random binary mask 508 (which may be...) can be applied to the user-item bipartite graph to generate a third masked bipartite graph 504. Figure 4 The modified second random binary mask 408 can be applied to the user-item bipartite graph 500 to generate a fourth masked bipartite graph 510 (which can be an example of a fourth subgraph of the second fixed adjacency matrix). The first GCN layer 506 can be Figure 4 A first GCN layer 406 with modified parameters is used. The first GCN layer 506 can be used to process a third masked bipartite graph 504 (which may be an example of a third subgraph of the second fixed adjacency matrix) to obtain a third embedding matrix associated with the second dataset. A second GCN layer 512 can be used to process a fourth masked bipartite graph 510 (which may be an example of a fourth subgraph of the second fixed adjacency matrix) using the third embedding matrix to generate a fourth embedding matrix.

[0125] In step S510, candidate embedding pairs (e.g., u1 and i4) can be selected from the second dataset. For example, see Section III.B.4, through propagation. Layers, graph convolutional networks obtain Embedded to represent users and projects :

[0126] Therefore, in an embodiment of the present invention, the user embeds e u 1 It can exist in the third embedding matrix derived from the third masking bipartite graph 504 (which can be an example of the third subgraph of the second fixed adjacency matrix), and the user embedding e u 2 It can exist in the fourth embedding matrix derived from the fourth masking bipartite graph 510 (which can be an example of the fourth subgraph of the second fixed adjacency matrix). User embedding e u 1 and user-embedded e u 2These can be generated by the first GCN layer 506 and the second GCN layer 512 respectively, targeting a specific user (e.g., Figure 5 Vectors representing the characteristics of u1 in the context (e.g., interactions with various items). These embeddings or vectors can then be combined to form aggregate embeddings or vectors (e.g., ... ).

[0127] Data from the third and fourth embedding matrices can be used to generate recommendation 514, which can be a recommendation score. For example, the inner product between two candidate embeddings can be used as the basis for generating the recommendation score:

[0128] Recommendation scores can be the probability that an item is relevant to a user. Recommendation scores can be generated between any pair of user-item embeddings and can be generated simultaneously with or separately from other recommendation scores. In one example, in... Figure 1 In the case where the first embedded computer 102 operates for a service that allows users to watch movies, recommendation scores can be sent to the first embedded computer 102. In another example, a computer that trains a structured graph convolutional network can directly use recommendation scores to recommend items to the user. The computer that trains the structured graph convolutional network can generate recommendation scores for multiple user-item pairs and store the recommendation scores in a database. The recommendation scores can be retrieved at a later time and transmitted to the embedded computer or an external device operated by the user who requested the recommendation scores (whose data is in the analyzed dataset).

[0129] Figure 6 A block diagram of server computer 600 is shown. Server computer 600 can be operated by an entity (such as a marketplace) that generates recommendation scores. Server computer 600 can train a structured graph convolutional network and use the structured graph convolutional network to generate recommendation scores. Server computer 600 may include processor 602. Processor 602 may be coupled to memory 604, network interface 606, and computer-readable medium 608. The computer-readable medium may include any suitable number and type of software modules.

[0130] Memory 604 can be used to store data and code. Memory 604 can be coupled internally or externally to processor 602 (e.g., via a cloud-based data storage device) and can include any combination of volatile and / or non-volatile memory (such as RAM, DRAM, ROM, flash memory, or any other suitable memory device). In some embodiments, memory 604 can store datasets including interactions between two types of nodes (e.g., interactions between users and projects). Memory 604 can additionally store data such as a list of recommended scores between users and projects.

[0131] Network interface 606 may include an interface that allows supervisory computer 600 to communicate with external computers and / or devices. Network interface 606 enables server computer 600 to transfer data to and from another device, such as an embedded computer. Some examples of network interface 606 may include a modem, a physical network interface (such as an Ethernet card or other network interface card (NIC)), a virtual network interface, a communication port, a PCMCIA slot and card, etc. Wireless protocols enabled by network interface 906 may include Wi-Fi™. Data transferred via network interface 906 may be in the form of signals, which may be electrical, electromagnetic, optical, or any other signal that can be received by an external communication interface (collectively, “electronic signals” or “electronic messages”). These electronic messages, which may include data or instructions, may be provided between network interface 906 and other devices via a communication path or channel. As described above, any suitable communication path or channel may be used, such as, for example, wires or cables, fiber optics, telephone lines, cellular links, radio frequency (RF) links, WAN or LAN networks, the Internet, or any other suitable medium.

[0132] Computer-readable medium 608 may include code executable by processor 602 to perform operations causing processor 602 to: receive a first dataset comprising a plurality of embeddings of a first type and a second type, the first type of embeddings being associated with a first node type and the second type of embeddings being associated with a second node type; generate a fixed adjacency matrix from the first dataset; apply a first random binary mask to the fixed adjacency matrix to obtain a first subgraph of the fixed adjacency matrix; process the first subgraph of the fixed adjacency matrix through a first layer of a graph convolutional network to obtain a first embedding matrix; apply a second random binary mask to the fixed adjacency matrix to obtain a second subgraph of the fixed adjacency matrix; process the first embedding matrix and the second subgraph of the fixed adjacency matrix through a second layer of the graph convolutional network to obtain a second embedding matrix; determine a plurality of gradients of a loss function; and modify the first random binary mask using at least one of the plurality of gradients. The graph convolutional network is used to generate a second fixed adjacency matrix from a second dataset, using a modified first random binary mask and a second random binary mask. The modified first random binary mask is then applied to the second fixed adjacency matrix to obtain a third subgraph of the second fixed adjacency matrix. The third subgraph of the second fixed adjacency matrix is ​​processed through a first layer of the graph convolutional network to obtain a third embedding matrix associated with the second dataset. The modified second random binary mask is then applied to the second fixed adjacency matrix to obtain a fourth subgraph of the second fixed adjacency matrix. The third embedding matrix and the fourth subgraph of the second fixed adjacency matrix associated with the second dataset are processed through a second layer of the graph convolutional network to obtain a fourth embedding matrix associated with the second dataset. Finally, using at least the third and fourth embedding matrices associated with the second dataset, a recommendation score is generated from the second dataset between the first type of embeddings and the second type of embeddings.

[0133] The computer-readable medium 608 may include multiple software modules, including but not limited to recommendation module 608A and communication module 608B.

[0134] Recommendation module 608A may include code that enables processor 602 to generate recommendation scores. For example, recommendation module 608A may receive a dataset as input, comprising a set of interactions between multiple users and items. Recommendation module 608A may transform the dataset into a user-item bipartite graph and / or adjacency matrix, generate a structured graph convolutional network, train the structured graph convolutional network, and generate recommendation scores. Recommendation module 608A may perform computations based on singular value decomposition, power iteration, differentiation, and matrix multiplication.

[0135] The communication module 608B, in conjunction with the processor 602, can generate, forward, and reformat messages and / or otherwise communicate with other devices. For example, the communication module 608B enables the server computer 600 to receive datasets from an embedded computer. The communication module 608B also enables the server computer 600 to transmit recommendation scores to external devices, such as computers operated by users requesting to view recommendation scores or the top n recommendations (which may be included in the analyzed dataset).

[0136] IV. Experiment

[0137] This section presents experiments evaluating the effectiveness and robustness of the proposed structured graph convolutional network. Specifically, it answers the following questions: RQ1: How effective is the proposed structured graph convolutional network compared to state-of-the-art baselines? RQ2: How does the structured graph convolutional network mitigate the problem of noisy edges? RQ3: How do different components (e.g., random binary masks and low-rank constraints) affect the performance of the structured graph convolutional network?

[0138] Table 1: Data set statistics.

[0139]

[0140] A. Experimental Setup

[0141] 1. Dataset

[0142] Recommendation performance was evaluated using four public benchmark datasets. Movielens-1M is a widely used benchmark for evaluating collaborative filtering algorithms. This dataset contains 1 million user-movie ratings. Gowalla is a dataset from location-based social networking sites. Gowalla The obtained check-in dataset shows that users share their location by checking in [Dawen Liang et al., 2016, published in...]. WWW [951–961]. Yelp is... Yelp Challenge posting. It includes various metadata about local businesses (e.g., category, location, opening hours, etc.) and user reviews. Used in the experiment. Yelp 2018 Versions. Amazon includes versions from... Amazon website A large amount of user reviews, ratings, and product metadata (e.g., price, brand, and images) have been collected. The largest categories are used. books This dataset is known for its high sparsity.

[0143] For MovieLens, all ratings are treated as implicit feedback (e.g., each rating score is converted to...). or (to indicate whether users have rated a movie). For sparse datasets: Gowalla, Yelp, and Amazon, graphs... The kernel settings are used to ensure that all users and projects have at least 10 interactions [Xiangnan He et al., 2020, published in...]. SIGIR , 639–648; Xiang Wang et al., 2019, published in SIGIR [165–174]. The statistical data of the dataset are summarized in Table 1.

[0144] For each dataset, 80% of each user's historical interactions were randomly selected to construct the training set, and the remainder was used as the test set. From the training set, 10% of the randomized interactions were selected as the validation set to tune the hyperparameters. For each observed user-item interaction, it was treated as a positive instance, and the triples were then sorted by sampling from negative items that the user had not previously interacted with. Five random splits were performed independently, and the average results were used in the experiments.

[0145] 2. Baseline

[0146] The following baselines are used for comparison with structured graph convolutional networks. BPR-MF [Steffen Rendle, published in...] UAI [452-461]: Seeking a classic model to optimize Bayesian personalized ranking loss. Matrix factorization is used as its preference predictor. NeuMF [Xiangnan He et al., 2017, published in WWW [173–182]: NeuMF learns nonlinear interactions between user and item embeddings via a multilayer perceptron and generalized matrix factorization components. GC-MC [Rianne van den Berg, Thomas N Kipf, and Max Welling, 2017, ] Published in KDD Workshop on Deep Learning Day GC-MC employs a graph autoencoder approach to learn the embeddings of users and items. A bilinear decoder is then used to predict preference scores. [HOP-Rec [Jheng-Hong Yang et al., 2018, published in...] RecSys [140–144]: HOP-Rec discovers higher-order indirect information about neighboring items for each user from a bipartite graph by randomly surfing the graph. BiNE [Ming Gao et al., 2018, published in SIGIR [715–724]: BiNE learns explicit and implicit user-item relationships by performing biased random walks on bipartite graphs. NGCF [Xiang Wang et al., 2019, published in SIGIR [165–174] and LightGCN [Xiangnan He et al., 2020, published in SIGIR [639–648]: Two state-of-the-art collaborative filtering models based on graph convolutional networks. They are briefly introduced in Section III.b. S-NGCF: The Structured Graph Convolutional Network model is a general framework compatible with various graph convolutional network models. Using NGCF as its basic backbone, the Structured NGCF (S-NGCF) aims to improve its performance and robustness. S-LightGCN: Similarly, within the Structured Graph Convolutional Network framework, LightGCN is used as the backbone.

[0147] 3. Implementation Details

[0148] Structured graph convolutional network models are implemented in TensorFlow. For all models, in... Embedded dimensions of search users and projects (For example, in equation (2)). For the baselines BPR-MF, NeuMF, GC-MC, HOP-Rec, and BiNE, their hyperparameters are initialized as in their original literature and then carefully tuned for optimal performance. For the graph convolutional network components within the proposed structured graph convolutional network, we use the same hyperparameters as the original NGCF and LightGCN, such as batch size, stopping criteria, learning rate in the Adam optimizer, etc. Additionally, the structured graph convolutional network has two hyperparameters. and To control sparsity and low-rank structure respectively. Internal adjustment and To study the parameter sensitivity of our model.

[0149] To evaluate the performance of the top n recommendations, two widely used evaluation metrics were used [Xiangnan He et al., 2020, In, 639–648; Wang et al., 2019, in SIGIR, 165–174]: the performance of the top n recommendations on a variable number of items. Recall rate and Normalized Discount Cumulative Gain (NDCG) .

[0150] Table 2: Comparison of recommendation performance for different models. Note that R and N are abbreviations for recall and NDCG, respectively. % improvement indicates the relative improvement of SGCN over its corresponding GCN. The best results are highlighted in bold, and the second best results are underlined.

[0151]

[0152] B. Performance Comparison (RQ1)

[0153] In this chapter, the proposed structured graph convolutional network is based on all four datasets. Recall rate @ and NDCG @ Compared with the baseline, where Set to 50 and 100. For different front... The performance of each value was similar in the experiments, so they were omitted. Table 2 summarizes the performance for the first... The proposed structured graph convolutional network model consistently delivers the best performance in all cases. The following observations were made from Table 2.

[0154] Compared to collaborative filtering-based methods (e.g., BPR-MF, NeuMF, and GC-MC), graph-based methods consistently achieve better performance in most cases. This demonstrates the effectiveness of leveraging higher-order proximity between users and items in a bipartite graph. Consequently, users are able to receive a wider range of messages from items beyond their immediate field of vision.

[0155] Among graph-based methods, graph convolutional network-based methods (e.g., NGCF, LightGCN, and SGCN) outperform HOP-Rec and BiNE across all datasets. This is because graph convolutional network-based methods allow for end-to-end gradient-based training, and they can directly accept the original graph as input without any preprocessing. In contrast, HOP-Rec and BiNE first require random walks to generate the graph. The random walk algorithm iterates through a sequence of nodes and then leverages downstream tasks to optimize node embeddings. However, the random walk algorithm cannot be trained end-to-end, which can lead to suboptimal performance.

[0156] By comparing S-NGCF and NGCF, S-NGCF has relative advantages over NGCF. Recall rate average Improvements, and relative to NDCG An improvement of over 15.74%. Similarly, S-LightGCN outperforms the best baseline LightGCN in... Recall rate The average was 9.85% and in NDCG In this aspect, the performance was 13.12%. The results show that structured graph convolutional networks (GCNNs) outperform their standard counterparts. In real-world applications, users may implicitly interact with millions of items, and the implicit feedback may not perfectly match user preferences. Therefore, the original GCNNs cannot handle noisy interactions (e.g., spurious positive interactions). In contrast, structured GCNNs, by jointly learning a sparse and low-rank graph structure within the architecture of a graph convolutional network, possess the ability to denoise the user's implicit feedback.

[0157] Typically, it is assumed that the observations contain some noise, and then the resilience to the edges of noise is explored, thus providing some insights into the design of structured graph convolutional networks.

[0158] C. Robustness Analysis (RQ2)

[0159] 1. Noise edge injection

[0160] As mentioned earlier, the performance of graph convolutional networks is highly sensitive to noise, as misleading information can propagate extensively between nodes via noisy edges. In this section, simulation experiments investigate the robustness of structured graph convolutional networks to noisy edges on graphs. For each dataset, random connections from unobserved edges are created to serve as false positive interactions for each user in the training set. The performance of different models on simulated graphs with varying noisy edge ratios from 0% to 25% is then evaluated. For better comparison, the focus is on structured graph convolutional networks and graph convolutional networks, while results from other baselines are omitted due to their poor performance. All simulation experiments are repeated five times, and the average results are presented in [data missing]. Figure 7 As shown in the image.

[0161] Figure 7 The robustness of structured graph convolutional networks compared to ordinary graph convolutional networks is shown for various noise levels. According to... Figure 7 On all datasets with varying noise ratios, structured graph convolutional networks consistently outperform graph convolutional networks. As the number of noisy edges increases, the margin achieved by structured graph convolutional networks relative to graph convolutional networks becomes even larger. For example, S-LightGCN achieves an improvement of over 13% over LightGCN on the MovieLens dataset with a 25% noise ratio. These comparisons demonstrate that the random message / node sampling strategy used in NGCF / LightGCN is susceptible to noisy edges.

[0162] Figure 8 Training curves for training loss and test recall are shown for the MovieLens dataset with 25% noise. Clearly, the original graph convolutional networks (GCNNs) risk overfitting to noise. For example, GCNNs achieve stable training errors but introduce significant uncertainty during validation. This means that the performance of GCNNs degrades slightly with more training epochs. In contrast, structured graph convolutional networks work well for both training and validation.

[0163] Structured graph convolutional networks address noise by introducing trainable random binary masks and low-rank constraints. Random binary masks have the potential to be used as... The potential of regularization, which drives meaningless or noisy edges to accurateZero. By sparsifying the graph, unnecessary computations in the message passing phase can be avoided, thereby mitigating overfitting and improving generalization ability. Simultaneously, low-rank constraints ensure the preservation of the graph's structural information by optimizing its principal singular values. Therefore, structured graph convolutional networks can significantly reduce the impact of noisy edges, thus improving the robustness of ordinary graph convolutional networks.

[0164] D. Parameter sensitivity (RQ3)

[0165] The sensitivity of structured graph convolutional networks to the following hyperparameters: number of layers The two regularization parameters in equation (13) and the approximate nuclear norm in equation (12) The number of singular values. The MovieLens dataset is used for hyperparameter studies, and results for other datasets show the same trend, which is ignored.

[0166] 1. Noise edge injection

[0167] Oversmoothing occurs when training deeper graph convolutional networks [Qimai Li, Zhichao Han and Xiao-MingWu, 2018, published in...]. AAAI To illustrate its impact, see [Heng-Tze Cheng et al., 2016, published in...]. DLRS , 7–10; Negin Entezari et al., 2020, published in WSDM , 169–177; Xiangnan He et al., 2020, published in SIGIR , 639–648; Cameron Musco and Christopher Musco, 2015, NeurIPS

[2015] 1396–1404] utilizes a variable number of graph convolutional network layers. To conduct the experiment. The results are presented in... Figure 9 middle.

[0168] Figure 9 Oversmoothing is demonstrated for both Structured Graph Convolutional Networks (SGMs) and the original Graph Convolutional Networks (GCNs). Significant performance degradation is observed for NGCF and LightGCN by increasing the number of layers. The Structured Graph Convolutional Networks successfully mitigate the oversmoothing problem. This is because during training, a random binary mask achieves a subset of neighbor aggregations, rather than full aggregations. This strategy prevents all node representations from converging to the same value as the graph convolutional network deepens, which improves generalization ability during the testing phase. This finding is consistent with the recent work DropEdge [Yu Rong et al., 2019, published in...]. ICLR Consistent. In fact, if Furthermore, by allowing random masks to randomly drop a specific ratio of edges (e.g., simply separating the mask from the computation graph), structured graph convolutional networks can degenerate into DropEdge. This is observed in structured graph convolutional networks. and Non-zero settings are generally superior to DropEdge. DropEdge is a random dropping method that cannot distinguish between real and noisy edges, while structured graph convolutional networks can accurately remove noisy edges using parameterized masks.

[0169] 2. Regularization term

[0170] For sparsity and low-rank constraints There are two main regularization parameters. and . Figure 10 The parameter sensitivity for the MovieLens dataset is shown. Figure 10 Part (A) in 10 and Part (B) in 10 show how to fix one parameter by changing another. The performance. It can be seen that, and The non-zero selection indicates the importance of the regularization term in our model. Even in or Even in the worst-case settings, the structured graph convolutional network still outperforms the baseline. In extreme cases, i.e., setting... By enabling all matrices with masks set to all 1s, the structured graph convolutional network becomes a graph convolutional network. Figure 10 (Continued) Also shown is the approximate nuclear norm. The impact of the number of singular values. Performance increases with larger... And increase. However, larger This results in a longer runtime. In the experiment, [the following will be used]. Settings The internal reasoning is reasonable.

[0171] V. Conclusion

[0172] A structured graph convolutional network (SGCN) is proposed to mitigate the negative impact of noise in user-item bipartite graphs. Specifically, forced sparsity and a low-rank structure of the input graph are employed during the training of the SGCN parameters. The proposed SGCN is compatible with various SGCN models, such as NGCF and LightGCN, which improves their robustness and generalization performance. Extensive experiments using real-world datasets demonstrate that SGCN outperforms existing baselines.

[0173] Any software component or function described in this application may be implemented as software code executed by a processor using any suitable computer language, such as Java, C++, or Perl using conventional or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer-readable medium such as random access memory (RAM), read-only memory (ROM), magnetic media (such as hard disk drives or floppy disks), or optical media (such as CD-ROMs). Any such computer-readable medium may reside on or within a single computing device, and may exist on or within different computing devices within a system or network.

[0174] The above description is illustrative and not restrictive. Many variations of the invention will become apparent to those skilled in the art upon review of this disclosure. Therefore, the scope of the invention may be determined not with reference to the above description, but rather with reference to the pending claims and their full scope or equivalents.

[0175] Without departing from the scope of the invention, one or more features of any embodiment may be combined with one or more features of any other embodiment.

[0176] Unless specifically indicated to the contrary, the use of “a / an” or “the” is intended to mean “one or more”.

[0177] All patents, patent applications, publications, and descriptions mentioned above are incorporated herein by reference in their entirety for all purposes. This does not constitute an admission that they are prior art.

Claims

1. A method comprising: Receive a first dataset comprising multiple embeddings of a first type and a second type, wherein the first type of embeddings is associated with a first node type and the second type of embeddings is associated with a second node type, wherein the first dataset comprises at least one of user reviews, ratings and product metadata; Generate a first fixed adjacency matrix from the first dataset; A first random binary mask is applied to the first fixed adjacency matrix to obtain a first subgraph of the first fixed adjacency matrix; The first subgraph of the first fixed adjacency matrix is ​​processed by the first layer of the graph convolutional network to obtain the first embedding matrix; A second random binary mask is applied to the first fixed adjacency matrix to obtain a second subgraph of the first fixed adjacency matrix; The second subgraph of the first embedding matrix and the first fixed adjacency matrix is ​​processed by the second layer of the graph convolutional network to obtain the second embedding matrix; Determine multiple gradients of the loss function; Use at least one of the plurality of gradients to modify the first random binary mask and the second random binary mask; Generate a second fixed adjacency matrix from the second dataset; The modified first random binary mask is applied to the second fixed adjacency matrix to obtain the third subgraph of the second fixed adjacency matrix; The third subgraph of the second fixed adjacency matrix is ​​processed by the first layer of the graph convolutional network to obtain a third embedding matrix associated with the second dataset; The modified second random binary mask is applied to the second fixed adjacency matrix to obtain the fourth subgraph of the second fixed adjacency matrix; The fourth subgraph associated with the second dataset is obtained by processing the third embedding matrix and the second fixed adjacency matrix associated with the second dataset through the second layer of the graph convolutional network. as well as Using at least the third and fourth embedding matrices associated with the second dataset, a recommendation score is generated from the second dataset between the first type of embeddings and the second type of embeddings.

2. The method of claim 1, wherein the first fixed adjacency matrix and the second fixed adjacency matrix are derived from the first bipartite graph and the second bipartite graph, respectively.

3. The method of claim 1, wherein the second dataset is received from an external computer.

4. The method of claim 1, wherein the second dataset is a subset of the first dataset.

5. The method of claim 1, wherein the first type of node is a user, and the second type of node is a project.

6. The method of claim 1, wherein modifying one or more of the first random binary mask and the second random binary mask comprises: Backpropagate at least one of the plurality of gradients to the second random binary mask; as well as Modify the value of the second random binary mask to minimize at least one of the plurality of gradients.

7. The method of claim 1, wherein modifying one or more of the first random binary mask and the second random binary mask includes modifying both the first binary random mask and the second binary random mask.

8. The method of claim 1, further comprising: At least one of the plurality of gradients is backpropagated to the second layer of the graph convolutional network; as well as The second layer of the graph convolutional network is modified using at least one of the plurality of gradients.

9. The method of claim 1, wherein the graph convolutional network is a graph convolutional network based on collaborative filtering.

10. The method of claim 1, wherein the graph convolutional network is a graph-based graph convolutional network.

11. The method of claim 1, further comprising: The recommended scores are stored in the database; as well as The recommended score is transmitted to an external computer.

12. A server computer, comprising: processor; and A non-transitory computer-readable medium, the non-transitory computer-readable medium comprising code, the code being executable by the processor to cause the processor to: Receive a first dataset comprising multiple embeddings of a first type and a second type, wherein the first type of embeddings is associated with a first node type and the second type of embeddings is associated with a second node type, wherein the first dataset comprises at least one of user reviews, ratings and product metadata; Generate a first fixed adjacency matrix from the first dataset; A first random binary mask is applied to the first fixed adjacency matrix to obtain a first subgraph of the first fixed adjacency matrix; The first subgraph of the first fixed adjacency matrix is ​​processed by the first layer of the graph convolutional network to obtain the first embedding matrix; A second random binary mask is applied to the first fixed adjacency matrix to obtain a second subgraph of the first fixed adjacency matrix; The second subgraph of the first embedding matrix and the first fixed adjacency matrix is ​​processed by the second layer of the graph convolutional network to obtain the second embedding matrix; Determine multiple gradients of the loss function; Use at least one of the plurality of gradients to modify the first random binary mask and the second random binary mask; Generate a second fixed adjacency matrix from the second dataset; The modified first random binary mask is applied to the second fixed adjacency matrix to obtain the third subgraph of the second fixed adjacency matrix; The third subgraph of the second fixed adjacency matrix is ​​processed by the first layer of the graph convolutional network to obtain a third embedding matrix associated with the second dataset; The modified second random binary mask is applied to the second fixed adjacency matrix to obtain the fourth subgraph of the second fixed adjacency matrix; The fourth subgraph associated with the second dataset is obtained by processing the third embedding matrix and the second fixed adjacency matrix associated with the second dataset through the second layer of the graph convolutional network. as well as Using at least the third and fourth embedding matrices associated with the second dataset, a recommendation score is generated from the second dataset between the first type of embeddings and the second type of embeddings.

13. The server computer of claim 12, wherein the first fixed adjacency matrix and the second fixed adjacency matrix are derived from the first bipartite graph and the second bipartite graph, respectively.

14. The server computer of claim 12, wherein the first type of node is a user and the second type of node is a project.

15. The server computer of claim 12, wherein the code further causes the processor to: Modify the regularization coefficients to control the shape of the subgraph associated with the adjacency matrix.

16. The server computer of claim 12, wherein the code further causes the processor to: Backpropagation of at least one of the plurality of gradients to the second layer of the graph convolutional network; and The second layer of the graph convolutional network is modified using at least one of the plurality of gradients.

17. The server computer of claim 12, wherein the second dataset is received from an external computer.

18. The server computer of claim 12, wherein the second dataset is a subset of the first dataset.

19. The server computer of claim 12, further comprising a memory storing the recommended scores.

20. The server computer of claim 19, wherein the code further causes the processor to: The recommended score is transmitted to an external computer.

Images