A distributed graph neural network training method supporting cross-node automatic differentiation

Abstract

The application provides a distributed graph neural network training method supporting cross-node automatic differentiation, and relates to the fields of distributed computing and deep learning; the graph neural network training process is decoupled into two parts of communication and calculation, and according to the calculation characteristics of forward and reverse calculation of the graph neural network, in forward calculation, each node collects information of neighbor nodes from a cluster, and the generated intermediate variable is cached and used in reverse calculation; in reverse calculation, each node calculates the gradient of neighbor nodes based on the result of forward propagation and sends the result to the neighbor, that is, a local or remote machine; the application decouples communication and calculation, and according to the calculation characteristics of forward and reverse calculation of the graph neural network, a synchronize-compute and compute-synchronize dual-mode calculation engine is designed and implemented to flexibly support forward and reverse calculation, thereby greatly reducing the design difficulty, and a master-mirror synchronization mechanism is used to support efficient node attribute communication.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of distributed computing and deep learning, and particularly relates to a distributed graph neural network training method supporting cross-node automatic differentiation. BACKGROUND

[0002] Although the traditional deep learning method has achieved revolutionary success in target recognition, natural language processing, speech recognition and other fields, and has promoted the development and research of pattern recognition and data mining, it can only process data in Euclidean space, while data in many practical application scenarios is generated from non-Euclidean space. Graph data is a typical structured data with non-Euclidean structure, which can well reflect the correlation between objects in many scenes in life. It is composed of a series of objects (nodes) and relationship types (edges) and has the characteristic of local connection. The traditional machine learning method cannot effectively process such graph data. Therefore, a new research hotspot "graph neural network (GNN)" has emerged, and many graph neural network algorithms that can apply deep learning methods to graph data have appeared, such as GCN, GAT, GraphSage, etc.

[0003] The graph neural network system AliGraph published in VLDB 2019 "AliGraph: A Comprehensive Graph Neural Network Platform" adopts a strategy of caching the neighbors of important nodes to reduce communication overhead, but this strategy causes a large amount of redundant computation and huge storage space cost. Because a node is likely to be the neighbor node of multiple nodes, when its neighbor nodes are distributed on different machines, the node will be copied to multiple machines, thereby causing repeated computation overhead. For example, when there is a two-layer neural network model, it is necessary to copy a sub-tree with the neighbor node of the node as the root node and the one-hop neighbor nodes of the neighbor node as the leaf nodes on the machine of the node, and if the neighbor node has multiple neighbors, it may be necessary to copy multiple copies of the sub-tree to different machines, resulting in a large amount of redundant computation. Storing such a sub-tree also occupies a huge storage space.

[0004] The graph neural network training system Roc proposed in "Improving the accuracy,scalability,and performance of graph neural networks with roc" published in MLSys adopts a non-cached node embedding value, and obtains the node embedding value of the remote neighbor node through communication. However, Roc adopts a single processing engine, that is, one-way communication, and the master node sends messages to the mirror node in the forward and backward propagation stages. Such a design results in inflexible forward and backward calculation. Because the forward propagation calculation and the backward propagation calculation of the graph neural network have different calculation logics. In the forward calculation, each node collects the information of the neighbor nodes from the cluster, and the generated intermediate variable will be cached and used in the backward calculation. In the backward calculation, each point calculates the gradient of the neighbor node based on the result of the forward propagation and sends the result to its neighbor (local or remote machine). These two calculations have different characteristics, but if a single graph propagation model is used for processing, in the backward propagation calculation, each master node first sends its gradient value to the mirror node, and then the mirror node receives the gradient value sent by the master node and calculates the gradient value to be transmitted to the neighbor node according to the dependency relationship. Such a processing strategy causes the backward calculation graph and the forward calculation graph to be unable to be executed on the same machine, so the automatic differentiation library of the existing deep learning library cannot be used, and only the operator of the backward propagation can be manually implemented. At the same time, in order to support the training of different graph neural network algorithms on Roc, the forward and backward calculation operators need to be manually implemented according to the characteristics of each algorithm, especially requiring the user to manually derive the reverse calculation logic, which is extremely difficult for the user. At the same time, the manually implemented operator is difficult to guarantee its efficiency and ease of use.

[0005] The strategy of caching important nodes adopted by AliGraph will generate a large amount of redundant calculation and occupy a large amount of storage space, and the single processing engine of ROC causes the forward and backward calculation to be inflexible, and the operators of the corresponding logic need to be manually implemented for different algorithms. SUMMARY

[0006] In view of the deficiencies of the prior art, the present application provides a distributed graph neural network training method supporting cross-node automatic differentiation;

[0007] A distributed graph neural network training method supporting cross-node automatic differentiation, specifically comprising the following steps:

[0008] Step 1: forward training of the distributed graph neural network;

[0009] Step 1.1: first, divide the graph data provided by the user using a graph division method such as MEITS, and divide a part of points and their incoming edges to each machine;

[0010] Step 1.2: Each machine establishes a master-mirror mechanism based on the divided graph data;

[0011] Step 1.2.1: Each machine represents the divided nodes as master nodes, and each machine is responsible for performing the calculation of the master nodes;

[0012] Step 1.2.2: These master nodes are embodied as source nodes in other machines, which are called mirror nodes, representing edge relationships. A master node may become a mirror node in multiple other machines;

[0013] Step 1.3: Analyze, decouple, and train the training process of graph neural network forward training; that is, use the synchronize-compute engine to perform graph computation, and use existing deep learning libraries to perform neural network computation;

[0014] Step 1.3.1: The formula for forward calculation of each node in each layer of the graph neural network is:

[0015]

[0016] wherein, respectively represent the embedding values of nodes v in the i-th layer and the i+1-th layer; N(v) represents the neighbor nodes of node v, represents the node representation of the neighbor nodes of node v; W i+1 respectively represent the learnable parameters of the i+1-th layer; σ represents a function introducing a nonlinear transformation; AGGREGATE i+1 represents the calculation method of combining the embedding values of neighbor nodes and the node itself in the i+1-th layer; COMBINE i+1 represents the aggregation calculation of the i+1-th layer;

[0017] Step 1.3.2: Based on the forward propagation calculation formula in step 1.3.1, further decouple it into graph propagation calculation and neural network calculation:

[0018]

[0019]

[0020] wherein, represents the aggregated neighbor representation of node v after graph propagation calculation in the i+1-th layer; represents the embedding value of node v in the i-th layer; represents the node representation of the neighbor nodes of node v; AGGREGATE i+1This indicates the calculation method for the embedding value of the (i+1)th layer, combining neighboring nodes and the node itself; COMBINE i+1 W represents the aggregation calculation at level i+1; i+1 represent the learnable parameters of the (i+1)th layer; σ represents the function that introduces a nonlinear transformation;

[0021] Step 1.3.3: Use the synchronize-compute mode computing engine to perform calculations on each node in the graph data according to the decoupled graph propagation calculation part, i.e., Formula 2; where synchronize means synchronizing the values ​​of the primary backup and mirror backup of a node, and compute means performing calculations on the node;

[0022] Step 1.3.3.1: First, perform Synchronize; each mirror node synchronizes its node embedding by pulling the latest node embedding value from its master node, i.e., the machine where the corresponding remote master node is located.

[0023] Step 1.3.3.2: Then perform Compute; because all the data that the master node depends on for computation is cached locally through Sychronzie communication, the master node performs computation locally according to the graph structure, that is, it pulls the node embedding values ​​of its in-neighbors to perform computation and obtain the neighbor representation;

[0024] Step 1.3.4: Using existing deep learning libraries such as PyTorch, Tensorflow, and MindSpore, calculate the next layer embedding value of the node according to the decoupled neural network computation part, i.e., Formula 3.

[0025] Step 1.4: Step 1.3 is the forward training process of a single layer of graph neural network. The forward training logic of each layer of graph neural network is the same. If the model has N layers, then the node representation of the Nth layer will be obtained after the forward training is completed.

[0026] Step 1.5: Then, based on the node representation of layer N and the real node representation provided by the dataset, the loss function L is used to obtain the loss value Loss; this part can be done using existing deep learning libraries such as PyTorch, Tensorflow, and MindSpore; next, back-training is performed based on the loss function and the loss value;

[0027] Step 2: Reverse train the distributed graph neural network;

[0028] Step 2.1: Analysis, decoupling, training of the training process of the backpropagation of the graph neural network; that is, using the compute-synchronize engine to perform graph computation, and the Autograd library of the existing deep learning library to perform neural network computation;

[0029] The gradient of each layer in the graph neural network contains two parts, one part is the node embedding gradient, and the other part is the gradient of the learnable parameter;

[0030] Step 2.1.1: Based on the training formula of the forward training process, that is, step 1.3.1, the gradient evaluation derivation is performed using the chain rule to obtain the gradient of each layer of the learnable parameter in the graph neural network—formula 4 and the embedding gradient of each layer of the node—formula 5.

[0031]

[0032]

[0033]

[0034] Wherein, L represents the loss function; represents the gradient of the i+1 layer learnable parameter; represents the embedding gradient of the i layer node; represents the evaluation formula of the embedding gradient of the i+1 layer node; represents the node embedding H i+1 of the i+1 layer based on the i+1 layer; represents the node embedding H i +1 of the i+1 layer based on the i+1 layer; represents the aggregation formula of the embedding gradient of the i layer node, which is the backpropagation process of the graph propagation calculation;

[0035] Because the node embedding value H i+1 of each layer is obtained by multiplying the neighbor representation embedding value and the learnable parameter W i+1 , so the evaluation of the learnable parameter gradient of each layer and the output H i+1 of each layer node are related; the third term on the right side of formula 5 corresponds to the graph propagation calculation part formula 2 in the forward training, which is a graph calculation involving communication operation;

[0036] Therefore, the backpropagation process of a layer of graph neural network is decoupled into neural network operation— in formula 4 and formula 5, and graph propagation operation ​

[0037] Step 2.1.2: Use the autograd library of the existing deep learning library, such as PyTorch, Tensorflow, MindSpore, etc. to call the automatic differentiation library to execute the first two terms on the right side of equations 4 and 5 to obtain the gradient of the current layer learnable parameters and the gradient of the neighbor representation

[0038] Step 2.1.3: Use the compute-synchronize mode computing engine to calculate each node in the graph data according to the decoupled graph propagation calculation part, that is, the in equation 5, and perform the calculation;

[0039] Step 2.1.3.1: First, compute each node to perform backward calculation locally, calculate the gradient value it needs to pass to each incoming neighbor;

[0040] Step 2.1.3.2: Then, synchronize each mirror, that is, mirror point, to send its gradient value to the remote master point, and the master point aggregates the gradient values sent by multiple mirror points to obtain the node embedding gradient of the next layer

[0041] Step 2.2: The above is the backward training process of one layer of graph neural network, and the backward training steps of each layer of graph neural network are the same; if the model has N layers, then after the backward training is completed, the parameter gradient of N layers will be obtained, then the gradients are synchronized between machines through communication, and finally the parameters are updated, completing a round of training;

[0042] Step 3: After the forward propagation and backward propagation training of the distributed graph neural network, a round of training is completed; the number of rounds is determined by the user according to different desired accuracies.

[0043] The beneficial technical effects of the present application: The purpose of the present application is to design and implement a distributed graph neural network training technology supporting efficient cross-node automatic differentiation technology, avoiding the burden of manually deriving and implementing the backward propagation training of the graph neural network model. Compared with the Aligraph and Roc systems in the background art, the present application reduces the redundant calculation overhead and improves the throughput of the system. In terms of communication and computation design, the present application decouples communication and computation, and according to the forward and backward calculation characteristics of the graph neural network, a synchronize-compute and compute-synchronize dual-mode computing engine is designed and implemented to flexibly support forward and backward calculation, greatly reducing the design difficulty, and a master-mirror synchronization mechanism is used to support efficient node attribute communication.

[0044] The method of the present application decouples the graph neural network training process into two parts of communication and calculation, and according to the calculation characteristics of the forward and backward of the graph neural network, in the forward calculation, each node collects the information of the neighbor nodes from the cluster, and the generated intermediate variable will be cached and used in the backward calculation. In the backward calculation, each point calculates the gradient of the neighbor nodes based on the results of the forward propagation and sends the results to its neighbors, i.e. local or remote machines. The key point of implementing automatic differentiation is to use the synchronize-compute dual mode calculation to obtain the required dependencies before performing forward and backward calculations, to ensure that all dependent data is local when calculating the gradient, so that the existing deep learning automatic differentiation library can be combined to realize automatic differentiation across computing nodes. Based on this logic and the decoupling of the training process, the present application designs and implements a synchronize-compute dual mode calculation engine and a master-mirror synchronization mechanism.

[0045] Using the master-mirror synchronization mechanism supports efficient inter-cluster data communication and avoids redundant calculation. The design of the synchronize-compute and compute-synchronize dual mode calculation engine realizes in-place execution of forward and backward training to support efficient automatic differentiation across computing nodes by combining the graph engine and existing mature automatic differentiation libraries, and avoids manual implementation of operators by users. At the same time, based on the design of the present application, the model training can flexibly select a specific hardware optimized deep learning library, such as PyTorch optimized for execution on GPU, Tensorflow optimized for execution on TPU, and MindSore optimized for execution on Ascend artificial intelligence chip, to support more efficient model training. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The system model training example flowchart of the embodiment of the present application;

[0047] Figure 2 The compute-synchronize calculation mode of the embodiment of the present application;

[0048] Figure 3 Comparison of the embodiment of the present application with the DistDGL and Roc system experimental data; wherein, NeutronStar is the system designed and implemented by the present application. DETAILED DESCRIPTION

[0049] The present application will be further described below in conjunction with the drawings and embodiments;

[0050] The present application describes the process of training a graph neural network model composed of 2 layers of graph neural network layers using the system designed by the present application in the form of an illustration, as shown in Figure 1 A distributed graph neural network training method supporting cross-node automatic differentiation, the specific steps are:

[0051] Forward training process:

[0052] Step 1: First, divide the graph data provided by the user: as shown in Figure 2 The example graph data consists of 2 nodes, node 1 is divided to machine 0, and node 2 is divided to machine 1.

[0053] Step 2: Based on the divided graph, establish a master-mirror mechanism:

[0054] Step 2.1: The master point of machine 0 is node 1, and the master point of machine 1 is node 2.

[0055] Step 2.2: The mirror point of machine 0 is node 2, and the mirror point of machine 1 is node 1.

[0056] Step 3: First layer graph neural network forward training;

[0057] Step 3.1: The forward calculation formula of node 1 in the master point of machine 0 in the first layer graph neural network is:

[0058]

[0059] Where, respectively represent the embedding values of node 1 in the 0th and 1st layers; N(1) represents the neighbor nodes {1, 2} of node 1. represent the embedding values of the neighbor nodes 1 and 2 of node 1 in the 0th layer; W 1 respectively represent the learnable parameters in the 1st layer; σ represents a function introducing a nonlinear transformation; AGGREGATE 1 represent the calculation method of combining the embedding values of the neighbor nodes and the node itself in the 1st layer; COMBINE 1 represent the aggregation calculation in the 1st layer. Similarly, the forward calculation formula of node 2 in the master point of machine 1 in the 1st layer graph neural network is:

[0060]

[0061] Where, respectively represent the embedding values of node 2 in the 0th and 1st layers; N(2) represents the neighbor nodes {1, 2} of node 2.

[0062] Step 3.2: Based on the forward propagation calculation formula in Step 3.1, further decouple it into two parts: graph propagation calculation and neural network calculation:

[0063]

[0064]

[0065] wherein, represents the aggregated neighbor representation of node 1 after the graph propagation calculation of the 1st layer. Similarly, the calculation formula of node 2 after decoupling is:

[0066]

[0067]

[0068] wherein, represents the aggregated neighbor representation of node 2 after the graph propagation calculation of the 1st layer.

[0069] Step 3.3: Use the synchronize-compute mode calculation engine to perform calculation on node 1 and node 2 according to the decoupled graph propagation calculation part (3), (5);

[0070] Step 3.3.1: Synchronize: As shown in the forward direction, first, mirror node 2 on machine 0 pulls the embedding value of the 0th layer from machine 1 through communication Figure 2 and caches it locally;

[0071] Step 3.3.2: Compute: Because all the data that node 1 depends on - the embedding values of node 1 and node 2 of the 0th layer are already local (on machine 0), node 1 performs calculation according to the graph structure, pulling the embedding values of its in-neighbors node 1 and node 2 to execute calculation, obtaining the neighbor representation of the 1st layer

[0072] Similarly, in the calculation of machine 1, first perform Synchronize, mirror node 1 on machine 1 pulls the embedding value of the 0th layer from machine 0 through communication and caches it locally; then perform Compute, node 2 performs calculation according to the graph structure, pulling the embedding values of its in-neighbors node 1 and node 2 of the 0th layer to execute calculation, obtaining the neighbor representation of the 1st layer

[0073] ​Step 3.4: Using existing deep learning libraries such as PyTorch, Tensorflow, and MindSpore, perform calculations according to the decoupled neural network computation parts (4) and (6) to obtain the first-layer embedding values ​​of nodes 1 and 2.

[0074] Step 4: Forward training of the second layer of the graph neural network;

[0075] Step 4.1: The forward calculation formula for master node 1 in machine 0 in the second layer of the graph neural network is:

[0076]

[0077] in, These represent the embedding values ​​of node 1 in layers 1 and 2, respectively; N(1) represents the neighboring nodes {1, 2} of node 1. W represents the embedding values ​​of the first-level neighbors, nodes 1 and 2, of node 1. 2 These represent the learnable parameters of the second layer; σ represents the function that introduces the nonlinear transformation; AGGREGATE 2 This indicates the calculation method for the embedding values ​​of the second layer, combining those of neighboring nodes and the node itself; COMBINE 2 This represents the aggregation calculation in the second layer. Similarly, the forward calculation formula for the master node 2 in machine 1 in the second layer of the graph neural network is:

[0078]

[0079] in, represents the embedding values ​​of node 2 in layer 1 and layer 2, respectively; N(2) represents the neighboring nodes {1, 2} of node 2.

[0080] Step 4.2: Based on the forward propagation calculation formula in Step 4.1, further decouple it into two parts with finer granularity: graph propagation calculation and neural network calculation.

[0081]

[0082]

[0083] in, This represents the aggregated neighbor representation of node 1 obtained after graph propagation computation in layer 2. Similarly, the calculation formula for node 2 after decoupling is:

[0084]

[0085]

[0086] in, represents the aggregated neighbor representation of node 2 after the 2nd layer graph propagation computation.

[0087] Step 4.3: The synchronize-compute mode computing engine performs computation on node 1 and node 2 according to the decoupled graph propagation computation parts (9), (11) respectively;

[0088] Step 4.3.1: Synchronize: As Figure 2 As shown in the forward direction, first, the mirror node 2 on machine 0 pulls the embedding values of the 1st layer from machine 1 through communication and caches them locally;

[0089] Step 4.3.2: Compute: Because all the data that node 1 depends on, the embedding values of node 1 and node 2, are already locally (on machine 0), node 1 performs computation according to the graph structure, pulling the embedding values of its in-neighbors node 1 and node 2, to obtain the neighbor representation

[0090] Similarly, in the computation on machine 1, first, Synchronize is performed, and the mirror node 1 on machine 1 pulls the embedding values of the 0th layer from machine 0 through communication and caches them locally; then Compute is performed, and node 2 performs computation according to the graph structure, pulling the embedding values of its in-neighbors node 1 and node 2 of the 0th layer, to obtain the neighbor representation of the 1st layer

[0091] Step 4.4: Using existing deep learning libraries such as PyTorch, Tensorflow, MindSpore, etc., perform computation according to the decoupled neural network computation parts (4), (6) to obtain the embedding values of node 1 and node 2 of the 2nd layer

[0092] Then, based on the node representation of the 1st layer and the real node representation provided by the dataset, use the loss function L to obtain the loss value Loss. This part can be completed using existing deep learning libraries such as PyTorch, Tensorflow, MinSpore, etc.; next, based on the loss function and the loss value, perform backpropagation training.

[0093] Backpropagation training process:

[0094] Step 1: Backpropagation training of the 2nd layer graph neural network;

[0095] Step 1.1: The backpropagation computation formula of the master node 1 in machine 0 in the 2nd layer graph neural network is:

[0096]

[0097]

[0098] where L denotes the loss function; denotes the gradient of the 2nd layer learnable parameters; denotes the embedding gradient of node 1 in the 1st layer; denotes the evaluation formula of the embedding gradient of node 1 in the 2nd layer nodes; denotes the node embedding H1 2 the evaluation formula of the gradient of the 2nd layer learnable parameters; denotes the node embedding H1 2 the evaluation formula of the gradient of the 2nd layer learnable parameters; denotes the aggregation formula of the embedding gradient of the 1st layer nodes. Similarly, the forward calculation formula of the master node 2 in machine 1 in the 2nd layer graph neural network is:

[0099]

[0100]

[0101] where, denotes the embedding gradient of node 2 in the 1st layer; denotes the evaluation formula of the embedding gradient of node 2 in the 2nd layer nodes; denotes the node embedding H1 the evaluation formula of the gradient of the 2nd layer learnable parameters; denotes the node embedding H1 the evaluation formula of the gradient of the 2nd layer learnable parameters; denotes the aggregation formula of the embedding gradient of the 1st layer nodes.

[0102] Step 1.2: Use the Autograd library of the existing deep learning library, such as PyTorch, Tensorflow, MindSpore, etc., to call the automatic differentiation library to execute the right two terms of formulas (13), (15), (14) and (16) to obtain the gradient of the current layer learnable parameters and the gradient of the neighbor representation

[0103] Step 1.3: Use the compute-synchronize mode computing engine to respectively calculate the 1st node and the 2nd node according to the decoupled graph propagation calculation part (the third term on the right side of formulas (14), (16) ) Perform computation:

[0104] Step 1.3.1: Compute as Figure 2 As shown in the reverse direction, first the master node 1 on machine 0 performs the reverse computation locally, computing the gradient values it needs to pass to each of its incoming neighbors, nodes 1 and 2;

[0105] Step 1.3.2: Synchronize the mirror node 2 sends its gradient values to the remote master, node 2 on machine 1, via communication. Node 2 on machine 1 aggregates the gradient values from multiple mirror nodes (from both machine 0 and machine 1) to obtain the node embedding gradient for the next layer

[0106] Similarly, in the computation on machine 1, first perform Compute, the master node 2 on machine 1 performs the reverse computation locally, computing the gradient values it needs to pass to each of its incoming neighbors, nodes 1 and 2; then perform Synchronize the mirror node 1 sends its gradient values to the remote master, node 1 on machine 0, via communication. Node 1 on machine 0 aggregates the gradient values from multiple mirror nodes (from both machine 0 and machine 1) to obtain the node embedding gradient for the next layer

[0107] Machine 1 and machine 2 communicate to synchronize the parameter gradients for the 2nd layer Then synchronize to update the parameters for the current layer.

[0108] Step 2: Backward training of the first layer graph neural network;

[0109] Step 2.1: The reverse computation formula for the master node 1 in machine 0 in the first layer graph neural network is:

[0110]

[0111]

[0112] where L denotes the loss function; denotes the gradient of the first layer learnable parameter; denotes the embedding gradient of node 1 in the first layer; denotes the evaluation formula for the embedding gradient of node 1 in the first layer; denotes the node embedding of node 1 in the first layer denotes the evaluation formula for the learnable parameter gradient of the first layer; denotes the node embedding of node 1 in the first layer The gradient evaluation formula for the neighbor node of the first layer node 1; The embedding gradient aggregation formula for the 0th layer node is represented. Similarly, the forward calculation formula of the master node 2 in machine 1 in the 2nd layer graph neural network is:

[0113]

[0114]

[0115] where, represents the embedding gradient of the 2nd node in the 1st layer; represents the embedding gradient evaluation formula for the 2nd node in the 1st layer node; represents the embedding gradient of the 2nd node in the 1st layer The gradient evaluation formula for the learnable parameter of the 1st layer; represents the embedding gradient of the 2nd node in the 1st layer The gradient evaluation formula for the neighbor node of the 2nd layer node 2; The embedding gradient aggregation formula for the 0th layer node is represented.

[0116] Step 2.2: Use the existing Autograd library of the deep learning library, such as PyTorch, Tensorflow, MindSpore, etc. on the two machines to call the automatic differentiation library to execute the right two items of formulas (17), (19), (18) and (20) to obtain the gradient of the learnable parameter of the current layer and the gradient of the neighbor representation

[0117] Step 2.3: Because the current layer is the 1st layer of the model, and the gradient of the 1st layer has been calculated in machine 1 and machine 2, it is not necessary to use compute-synchronize to perform the calculation on the graph again, so the parameter gradient of the 2nd layer is synchronized Then update to obtain the parameter of the current layer.

[0118] From now on, a round of training is completed.

[0119] Through the master-mirror synchronization mechanism, the forward and the backward can communicate to obtain the value of the dependent item before calculation. At the same time, the design of the double-mode computing engine can perform forward and backward calculation on a node locally, that is, the forward and backward calculation graphs are executed on one machine. Therefore, this design can naturally use the optimization operators (including automatic differentiation operators) in the existing mature deep learning library combined with the graph propagation engine to perform forward and backward calculation, without the need for users to manually implement the operators, realizing automatic differentiation across computing nodes. At the same time, it is also because of this design that model training can flexibly switch to use a specific hardware-optimized deep learning library, such as PyTorch optimized for execution on GPU, Tensorflow optimized for execution on TPU, and MindSpore optimized for execution on Ascend artificial intelligence chip.

[0120] Based on the above analysis and design, the present application combines the existing deep learning library and the synchronize-compute double-mode computing engine to design an automatic differentiation tool chain across computing nodes, which is abstracted into an algorithm as follows:

[0121]

[0122] The input of the algorithm is the transposed graph (i.e. the structure relationship of the original input graph is turned), the edge weight, the node representation of the current layer, the gradient of the next layer, the neighbor representation of the current layer, and the learnable parameter of the current layer. The output is the node embedding gradient of the previous layer.

[0123] Algorithm explanation: lines 1-5 represent the calculation of the gradients of the learnable parameters and the neighbor representation of the current layer according to the node representation of the current layer. The gradients of the learnable parameters W and the neighbor node representation can be calculated using the automatic differentiation library of the existing deep learning library, such as PyTorch, Tensorflow, MindSpore, etc. Corresponding to the second term on the right side of equation 4 and the second term on the right side of equation 5, respectively. Lines 6-11 represent the calculation of the global gradient of the parameter W and the node embedding gradient of the previous layer according to the chain rule, corresponding to the first term of equation 4 and equation 5, respectively. Lines 12-16 represent the calculation of the gradient that each node needs to pass to the neighbor node, and the aggregation of the gradient after communication to obtain the node embedding gradient of the previous layer. Line 17 represents the synchronization of the gradient of the learnable parameter through communication, and then the update. Line 18 represents the output of the node embedding gradient of the previous layer.

[0124] The experiment of the application is carried out under a cluster composed of 16 Aliyun servers. The configuration of each machine is 16-core CPU, 62GB memory, Nvidia Tesla T4 GPU, and the operating system is Ubuntu 18.04. The network bandwidth is 6Gbps / s.

[0125] Dataset and GNN algorithm: the experimental test uses 6 datasets: Google, Pokec, LiveJ, Reddi, Orkut, Wiki. At the same time, three representative graph neural network models are selected: GCN, GIN, GAT, and each model is two layers in actual experiment.

[0126] Comparison system: DistDGL, Roc.NeutronStar is the system designed and implemented by the application; Figure 3 Comparison of the application embodiment and the experimental data of the DistDGL and Roc system;

[0127] The application uses 6 real world graph data for experimental comparison, and compared with the DistDGL and ROC two graph neural network systems, the system of the application can obtain an acceleration ratio of 1.8 to 15.8 times.

Claims

1. A method for training a distributed graph neural network that supports automatic differentiation across nodes, characterized in that, Specifically, the following steps are included: Step 1: Perform forward training on the distributed graph neural network; Step 2: Reverse train the distributed graph neural network; Step 3: After performing forward and backward propagation training on the distributed graph neural network, one round of training is complete; the number of rounds is determined by the user based on the desired accuracy. Specifically, step 1 is as follows: Step 1.1: First, the graph data provided by the user is partitioned using the MEITS graph partitioning method, and a portion of the points and their incoming edges are assigned to each machine; Step 1.2: Each machine establishes a master-mirror mechanism based on the partitioned graph data; Step 1.3: Analyze, decouple, and train the graph neural network forward training process; use the synchronize-compute engine to perform graph computation and existing deep learning libraries to perform neural network computation; Step 1.4: Step 1.3 is the forward training process of a single layer of graph neural network. The forward training logic of each layer of graph neural network is the same. The model has N layers. After the forward training is completed, the node representation of the Nth layer is obtained. Step 1.5: Based on the node representation of layer N and the real node representation provided by the dataset, obtain the loss value Loss using the loss function L; this part is completed using existing deep learning libraries PyTorch, Tensorflow, and MindSpore; next, back-training is performed based on the loss function L and the loss value Loss. Specifically, step 1.2 is as follows: Step 1.2.1: Each machine will designate the node it is assigned as the master node, and each machine will be responsible for executing the computation of the master node; Step 1.2.2: These master points, i.e., the main nodes, serve as source points in other machines to represent edge relationships. They are called mirror points, i.e., mirror points. One master point is a mirror point of multiple other machines. Specifically, step 1.3 is as follows: Step 1.3.1: The formula for the forward computation of each node in each layer of the graph neural network is: （1） in, , Let represent the embedding values ​​of node v in the i-th layer and the (i+1)-th layer, respectively; Indicates the neighboring nodes of node v. Represents the node representation of node v's neighbor node u; This represents the learnable parameters of the (i+1)th layer; This represents a function that introduces a nonlinear transformation; This indicates the calculation method for the embedding value of the (i+1)th layer, which combines the embedding values ​​of neighboring nodes and the node itself. This represents the aggregation calculation at the (i+1)th level; Step 1.3.2: Based on the forward propagation calculation formula in Step 1.3.1, further decouple it into two parts: graph propagation calculation and neural network calculation. （2） （3） in, This represents the aggregated neighbor representation of node v obtained after graph propagation computation at layer i+1; Step 1.3.3: Use the synchronize-compute mode computing engine to perform calculations on each node in the graph data according to the decoupled graph propagation calculation part, i.e., Formula 2; where synchronize means synchronizing the values ​​of the primary backup and mirror backup of a node, and compute means performing calculations on the node; Step 1.3.3.1: First, synchronize; each mirror node synchronizes its node embedding by pulling the latest node embedding value from its master node, i.e., the machine where the corresponding remote master node is located. Step 1.3.3.2: Then perform compute; all the data that the master node depends on for computation is cached locally through synchronzie communication. The master node performs computation locally according to the graph structure, that is, it pulls the node embedding values ​​of its in-neighbors and performs computation to obtain the neighbor representation. Step 1.3.4: Using existing deep learning libraries PyTorch, Tensorflow, and MindSpore, calculate the next layer embedding value of the node according to the decoupled neural network computation part, i.e., Formula 3.

2. The distributed graph neural network training method supporting automatic differentiation across nodes as described in claim 1, characterized in that... It lies in, Step 2 specifically involves: Step 2.1: Analyze, decouple, and train the graph neural network in reverse training; use the compute-synchronize engine to perform graph computation, and the Autograd library of the existing deep learning library to perform neural network computation; The gradient of each layer in a graph neural network consists of two parts: the node embedding gradient and the gradient of the learnable parameters. Step 2.2: Step 2.1 is the reverse training process of a single layer of graph neural network. The reverse training steps for each layer of graph neural network are the same. The model has N layers. After the reverse training is completed, the gradients of the parameters of the N layers are obtained. Then, the gradients are synchronized between the machines through communication. Finally, the parameters are updated to complete one round of training.

3. The distributed graph neural network training method supporting automatic differentiation across nodes as described in claim 2, characterized in that... It lies in, Step 2.1 specifically involves: Step 2.1: Analyze, decouple, and train the training process of graph neural network reverse training; that is, use the compute-synchronize engine to perform graph computation and the Autograd library of the existing deep learning library to perform neural network computation. The gradient of each layer in a graph neural network consists of two parts: the node embedding gradient and the gradient of the learnable parameters. Step 2.1.1: Based on the training formula in the forward training process (i.e., Step 1.3.1), the gradient calculation is derived using the chain rule, resulting in the gradient formula 4 for each learnable parameter in the graph neural network and the embedding gradient formula 5 for each node in the layer. （4） （5） Where L represents the loss function; This represents the gradient of the learnable parameters of the (i+1)th layer; This represents the embedding gradient of the i-th layer node; This represents the formula for evaluating the embedding gradient of the (i+1)th layer node; Represents node embedding based on layer i+1 Formula for calculating the gradient of the learnable parameters of the (i+1)th layer; Represents node embedding based on layer i+1 Formula for evaluating the gradient of the (i+1)th layer neighbor node representation; This represents the formula for the embedding gradient aggregation of nodes in the i-th layer, which is the backpropagation process of graph propagation computation; Because the node embedding value of each layer The neighbor representation embedding values ​​are all calculated using graph propagation. With learnable parameters The result is obtained by multiplication, so the gradient of the learnable parameters for each layer and the output of each node in each layer are calculated. Related to; the third term on the right side of equation 5 Formula 2, which corresponds to the graph propagation computation part in forward training, is a graph computation involving communication operations; Therefore, the reverse training process of a one-layer graph neural network is decoupled into neural network operations—Equations 4 and 5. , And graph propagation operation ; Step 2.1.2: Using existing deep learning libraries such as PyTorch, Tensorflow, and MindSpore, call the Autograd library to execute the first two terms on the right side of equations 4 and 5 to obtain the gradient of the learnable parameters of the current layer. gradient represented by neighbors ; Step 2.1.3: Use the compute-synchronize mode computing engine to compute the part of the graph data for each node according to the decoupled graph propagation, i.e., in Formula 5. Perform the calculation.

4. The distributed graph neural network training method supporting automatic differentiation across nodes as described in claim 3, characterized in that... It lies in, Step 2.1.3 specifically involves: Step 2.1.3.1: First, compute each node locally to perform reverse computation, calculating the gradient values ​​it needs to pass to each incoming neighbor; Step 2.1.3.2: Then, synchronize each mirror point and send its gradient value to the remote master point. The master point aggregates the gradient values ​​from multiple mirror points to obtain the node embedding gradient for the next layer. .

Images